Merge pull request 'fix(regex): special missing handling' (#286) from regexfix into additional-validations-regex

Reviewed-on: #286

.agent/docs/regex-validations.md

@@ -0,0 +1,86 @@
+# Regex Validations (HARDREGEX / SOFTREGEX)
+
+This document describes how the regex validation rules work internally. For user-facing documentation see `docs/dcc-validations.md` in the `docs.datacontroller.io` repo.
+
+## Overview
+
+Two validation rule types in `MPE_VALIDATIONS` validate cell values against a regular expression supplied in `RULE_VALUE`:
+
+- `HARDREGEX` - submission-blocking. A failing value is rejected by the cell validator and painted red (HOT's own `htInvalid` class), so the row cannot be submitted.
+- `SOFTREGEX` - display-only warning. A failing value is painted yellow (`dc-warning-cell` class) with a tooltip, but submission is not blocked.
+
+Both rule types are selectable in the MPE_VALIDATIONS RULE_TYPE dropdown; they were added to the selectbox seed data in `sas/sasjs/macros/mpe_makedata.sas` and via the optional migration `sas/sasjs/db/migrations/20260720_v7.12_release.sas`. `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.
+
+## Config-time validation (SAS side)
+
+`sas/sasjs/services/hooks/mpe_validations_postedit.sas` runs `prxparse()` on any staged HARDREGEX/SOFTREGEX rule value and aborts the edit with a list of offending `libref.table.column` references if the pattern is invalid. This is a best-effort syntax check to catch typos at config time; an empty pattern is treated as valid (it matches everything in JS). Rows marked for delete are skipped.
+
+Because patterns must pass `prxparse`, rule values are authored in the SAS PRX delimiter form `/pattern/flags` (although a bare pattern is also accepted for backwards compatibility).
+
+## Frontend evaluation
+
+All regex handling lives in the client; there is no server-side re-validation of data values.  The per-cell decision flow:
+
+```mermaid
+flowchart TD
+    A[Cell value] --> B{Row marked for delete<br/>and not a PK column?}
+    B -- Yes --> Z[No validation / no warning]
+    B -- No --> C{isRegexRuleExempt?<br/>blank, or "." on a numeric column}
+    C -- Yes --> Z
+    C -- No --> D{HARDREGEX rule on column?}
+    D -- Yes --> E{Pattern matches?}
+    E -- No --> F[Invalid: submission blocked,<br/>red htInvalid + REGEX tooltip]
+    E -- Yes --> J[Valid]
+    D -- No --> G{SOFTREGEX rule on column?}
+    G -- Yes --> H{Pattern matches?}
+    H -- No --> I[Warning: yellow dc-warning-cell<br/>+ REGEX tooltip, submission allowed]
+    H -- Yes --> J[Valid]
+    G -- No --> J
+```
+
+A malformed pattern never reaches this flow: it is treated as always-valid (HARDREGEX) / never-warn (SOFTREGEX) with a `console.warn`, rather than breaking the editor.
+
+### `client/src/app/shared/dc-validator/utils/parseRegexRule.ts`
+
+Converts an authored SAS PRX pattern into a JavaScript `RegExp`:
+
+1. If the value matches `/^\/(.*)\/([a-z]*)$/s`, the delimiters are stripped and the trailing flags are passed to `new RegExp(body, flags)`. Otherwise the value is used as-is (backwards compatibility with bare patterns).
+2. Three mechanical Perl→JS translations are applied to the body:
+   - a leading `(?i)` inline modifier is removed and folded into the `i` flag;
+   - `\Q...\E` literal sequences are replaced with escaped literal text;
+   - `\A` → `^` and `\z` → `(?![\s\S])` (end-of-string anchor).
+3. Perl-only constructs that would need a capture-group-renumbering rewrite (atomic groups `(?>...)`, possessive quantifiers `a++`) are deliberately NOT translated. They throw from `new RegExp`, and every caller treats a throw as "always valid / never warn" (see below) rather than breaking the editor.
+
+### `client/src/app/shared/dc-validator/utils/isRegexRuleExempt.ts`
+
+Blank values (`undefined`, `null`, `''`) are exempt from pattern matching on any column type - use a separate NOTNULL rule if populated values must also be enforced. On numeric columns the plain SAS missing (`.`) is also exempt; special missings (`.A`-`.Z`, `._`, bare letters) are NOT exempt anywhere - being deliberately set, they are real values the pattern must match (and on a character column even `.` is real text). `isSpecialMissing` from `@sasjs/utils` is deliberately not used: its optional-dot regex would exempt any single-letter character value ("d", "z") before the regex ever ran. The check takes an `isNumeric` flag, passed by all callers (the dq validator via `dqValidate(rules, value, colType === 'numeric')`, the warning renderer via a `makeRegexWarningRenderer` argument, and `failsSoftRegex` via the column's HOT type).
+
+### HARDREGEX - blocking validation
+
+`HARDREGEX` is implemented as a cell validator in `client/src/app/shared/dc-validator/validations/dq-validation.ts`. It returns `true` (valid) for exempt values and for patterns that fail to compile (with a `console.warn`), and otherwise returns `parseRegexRule(ruleValue).test(value.toString())`. Returning `false` makes HOT mark the cell invalid, block submission, and paint it red via its standard `htInvalid` styling.
+
+### SOFTREGEX - warning renderer
+
+`client/src/app/editor/utils/regex-warning-renderer.ts` builds a display-only Handsontable renderer (registered per-column by `DcValidator.setupRules` in `client/src/app/shared/dc-validator/dc-validator.ts`). It never returns false; it only:
+
+- adds a `REGEX: <pattern>` tooltip (`td.title`) when a rule fails;
+- adds the yellow `dc-warning-cell` class when only SOFTREGEX fails.
+
+`DcValidator.failsSoftRegex(col, value)` provides the same logic outside the grid (e.g. the edit-record screen).
+
+### Precedence: HARD and SOFT on the same column
+
+Only one regex ever runs per column. If a HARDREGEX rule exists, SOFTREGEX is ignored entirely - never compiled, never evaluated - regardless of whether individual cell values pass or fail the hard rule. A value failing HARDREGEX gets the red invalid styling (blocking submission) plus a `REGEX: <pattern>` tooltip; a SOFTREGEX-only column warns in yellow without blocking. This holds in both the renderer and `failsSoftRegex`.
+
+### Other behaviour
+
+- Rows marked for delete (`_____DELETE__THIS__RECORD_____ = 'Yes'`) are not warned/validated by the renderer (except primary key columns, which still validate).
+- A malformed pattern never breaks the editor: the dq validator treats it as always-valid and the renderer disables the warning, logging to the console instead.
+- The pattern is used as authored - it is NOT auto-anchored. Authors must include `^`/`$` to match the entire cell value.
+- Column info: `client/src/app/shared/utils/col-info-html.ts` shows the applied pattern in the column-info dropdown - the HARDREGEX pattern if one exists (it is the rule actually applied when both are present), otherwise the SOFTREGEX pattern, otherwise nothing.
+
+## Testing
+
+- Unit tests: `parseRegexRule.spec.ts`, `isRegexRuleExempt.spec.ts`, `dq-validation.spec.ts`, `dc-validator.spec.ts` (under `client/src/app/shared/dc-validator/`), `client/src/app/editor/utils/regex-warning-renderer.spec.ts`, `client/src/app/shared/utils/col-info-html.spec.ts`.
+- E2E: `client/cypress/e2e/editor.cy.ts`.
+- SAS side: `sas/sasjs/services/editors/stagedata.test.3.sas`, plus seed data in `mpe_makedata.sas`: HARDREGEX "SOME_CHAR must contain 'the' or 'data'" (`/the|data/i`), SOFTREGEX "SOME_CHAR should contain the letter 't'", SOFTREGEX on PRIMARY_KEY_FIELD (`/^\d+$/` - yellow if the key contains a decimal), and HARDREGEX on SOME_SHORTNUM (`/^(?![1-5](\.\d+)?$).*/` - values 1-5 blocked; generated data starts at 6 so demos aren't blocked accidentally).

AGENTS.md

@@ -12,9 +12,12 @@ This is different from `.sas` files, where a maximum line length applies. The no
 
 Rationale: hard-wrapped prose produces noisy diffs when sentences are edited and reflowed, and Markdown renderers already handle wrapping.
 
-## SAS files
+## Linting (required before "done")
 
-After creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
+Never consider a change complete until the relevant linters pass on the files you touched — do not rely on the user's pre-commit hooks to catch it:
+
+- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
+- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
 
 ## No external assets
 
@@ -23,3 +26,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
 ## The .agent folder
 
 Agent-related content lives in `.agent/`: technical/agent-facing documentation goes in `.agent/docs/` (not `docs/`), and skills in `.agent/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agent/docs/`.
+
+## Code comments and test names
+
+Never reference items that are not active parts of the repository — no "the original bug", "regression from this fix", "this session/PR/commit", or similar ephemeral context. Comments and test names must be self-contained: describe the behaviour being asserted, not the history of how it was discovered. The one exception is a literal link to a ticket/issue tracker.

client/cypress/e2e/editor.cy.ts

@@ -285,21 +285,23 @@ context('editor tests: ', function () {
 
         openColumnDropdown('REGEX_HARD_COL')
         cy.get('.htDropdownMenu').should(($menu) => {
-          expect($menu.text()).to.include('REGEX: /[\\w.]+@[\\w]+\\.[a-z]{2,}/')
+          expect($menu.text()).to.include(
+            'HARDREGEX: /[\\w.]+@[\\w]+\\.[a-z]{2,}/'
+          )
         })
         cy.get('body').click(0, 0) // close menu
 
         openColumnDropdown('REGEX_SOFT_COL')
         cy.get('.htDropdownMenu').should(($menu) => {
           expect($menu.text()).to.include(
-            'REGEX: /[A-Z]{1,2}\\d{1,2}[A-Z]?\\s?\\d[A-Z]{2}/'
+            'SOFTREGEX: /[A-Z]{1,2}\\d{1,2}[A-Z]?\\s?\\d[A-Z]{2}/'
           )
         })
       })
     })
   })
 
-  it('10 | Info dropdown labels HARDREGEX/SOFTREGEX separately when a column has both', () => {
+  it('10 | Info dropdown shows only the applied HARDREGEX pattern when a column has both', () => {
     openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
 
     clickOnEdit(() => {
@@ -313,7 +315,7 @@ context('editor tests: ', function () {
           const text = $menu.text()
 
           expect(text).to.include('HARDREGEX: /^[A-Z0-9_-]+$/')
-          expect(text).to.include('SOFTREGEX: /^.{5,10}$/')
+          expect(text).to.not.include('SOFTREGEX: /^.{5,10}$/')
         })
       })
     })
@@ -358,7 +360,7 @@ context('editor tests: ', function () {
     })
   })
 
-  it('12 | REGEX_BOTH_COL: passing HARDREGEX but failing SOFTREGEX warns without blocking', (done) => {
+  it('12 | REGEX_BOTH_COL: SOFTREGEX is ignored entirely when HARDREGEX is present', (done) => {
     openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
 
     clickOnEdit(() => {
@@ -368,7 +370,7 @@ context('editor tests: ', function () {
         scrollGridRight()
 
         // 'AB' passes HARDREGEX (uppercase only) but fails SOFTREGEX (too
-        // short) - previously always inert whenever a column had both rules.
+        // short) - only one regex runs per column, so no warning is shown.
         getCellByHeaderAndRow(1, 'REGEX_BOTH_COL')
           .dblclick({ force: true })
           .then(() => {
@@ -377,8 +379,8 @@ context('editor tests: ', function () {
               .type('AB{enter}')
               .then(() => {
                 getCellByHeaderAndRow(1, 'REGEX_BOTH_COL')
-                  .should('have.class', 'dc-warning-cell')
+                  .should('not.have.class', 'dc-warning-cell')
-                  .and('have.attr', 'title', 'REGEX: /^.{5,10}$/')
+                  .and('not.have.attr', 'title')
 
                 submitTable(() => {
                   cy.get('#submitBtn', { timeout: longerCommandTimeout })

client/src/app/editor/utils/regex-warning-renderer.spec.ts

@@ -19,6 +19,26 @@ describe('makeRegexWarningRenderer', () => {
     return { hot, container }
   }
 
+  const buildHotNumeric = (data: any[]) => {
+    const container = document.createElement('div')
+    document.body.appendChild(container)
+
+    const hot = new Handsontable(container, {
+      data,
+      columns: [
+        {
+          data: 'val',
+          renderer: makeRegexWarningRenderer('^[A-Z]{3}$', undefined, true)
+        },
+        { data: '_____DELETE__THIS__RECORD_____' }
+      ],
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    return { hot, container }
+  }
+
   it('adds dc-warning-cell and a REGEX: title when the value fails the pattern', () => {
     const { hot, container } = buildHot([
       { val: 'abc', _____DELETE__THIS__RECORD_____: 'No' }
@@ -56,9 +76,9 @@ describe('makeRegexWarningRenderer', () => {
     container.remove()
   })
 
-  it('does not add dc-warning-cell for a SAS special missing value', () => {
+  it('does not add dc-warning-cell for the plain SAS missing (".") on a numeric column', () => {
-    const { hot, container } = buildHot([
+    const { hot, container } = buildHotNumeric([
-      { val: '.a', _____DELETE__THIS__RECORD_____: 'No' }
+      { val: '.', _____DELETE__THIS__RECORD_____: 'No' }
     ])
 
     const td = hot.getCell(0, 0)
@@ -68,6 +88,18 @@ describe('makeRegexWarningRenderer', () => {
     container.remove()
   })
 
+  it('DOES add dc-warning-cell for a special missing on a numeric column (a deliberately-set value)', () => {
+    const { hot, container } = buildHotNumeric([
+      { val: '.a', _____DELETE__THIS__RECORD_____: 'No' }
+    ])
+
+    const td = hot.getCell(0, 0)
+    expect(td?.classList.contains('dc-warning-cell')).toBeTrue()
+
+    hot.destroy()
+    container.remove()
+  })
+
   it('does not throw and never warns on a malformed pattern', () => {
     const container = document.createElement('div')
     document.body.appendChild(container)
@@ -228,7 +260,7 @@ describe('makeRegexWarningRenderer', () => {
       container.remove()
     })
 
-    it('falls through to SOFTREGEX when the value passes HARDREGEX but fails SOFTREGEX', () => {
+    it('never warns for SOFTREGEX when the column also has HARDREGEX, even for a value that passes HARDREGEX', () => {
       const container = document.createElement('div')
       document.body.appendChild(container)
 
@@ -247,10 +279,10 @@ describe('makeRegexWarningRenderer', () => {
 
       const td = hot.getCell(0, 0)
       // 'AB' passes HARDREGEX (uppercase/digits) but fails SOFTREGEX (too
-      // short) - this is the case that was previously always inert (no
+      // short) - only one regex runs per column, so the soft rule is
-      // warning ever shown for a column with both rules).
+      // ignored entirely: no warning, no title.
-      expect(td?.classList.contains('dc-warning-cell')).toBeTrue()
+      expect(td?.classList.contains('dc-warning-cell')).toBeFalse()
-      expect(td?.title).toEqual('REGEX: ^.{5,10}$')
+      expect(td?.title).toEqual('')
 
       hot.destroy()
       container.remove()

client/src/app/editor/utils/regex-warning-renderer.ts

@@ -22,10 +22,12 @@ const compileRegex = (
  * `REGEX: <pattern>` title, plus a yellow `dc-warning-cell` class when only
  * SOFTREGEX fails, same split as makeNumberFormatRenderer.
  *
- * A column can carry both rules at once. Hard is validated first: a value
+ * Only one regex ever runs per column: when a HARDREGEX rule exists,
- * failing HARDREGEX gets its title (no yellow class - red htInvalid already
+ * SOFTREGEX is ignored entirely (never compiled, never evaluated) -
- * covers the color), and SOFTREGEX is only evaluated once HARDREGEX passes.
+ * regardless of whether individual cell values pass or fail the hard
- * This mirrors DcValidator.failsSoftRegex's own precedence.
+ * rule.  A value failing HARDREGEX gets its title (no yellow class - red
+ * htInvalid already covers the color).  This mirrors
+ * DcValidator.failsSoftRegex's own precedence.
  *
  * Suppressed on rows marked for delete (_____DELETE__THIS__RECORD_____ =
  * 'Yes') — a warning about data about to be removed is just noise.
@@ -35,10 +37,11 @@ const compileRegex = (
  */
 export const makeRegexWarningRenderer = (
   softPattern?: string,
-  hardPattern?: string
+  hardPattern?: string,
+  isNumeric: boolean = false
 ) => {
-  const softRegex = compileRegex(softPattern, 'SOFTREGEX')
   const hardRegex = compileRegex(hardPattern, 'HARDREGEX')
+  const softRegex = hardRegex ? null : compileRegex(softPattern, 'SOFTREGEX')
 
   const baseRenderer = Handsontable.renderers.getRenderer('text')
 
@@ -55,7 +58,7 @@ export const makeRegexWarningRenderer = (
 
     const markedForDelete =
       instance.getDataAtRowProp(row, '_____DELETE__THIS__RECORD_____') === 'Yes'
-    const exempt = isRegexRuleExempt(value)
+    const exempt = isRegexRuleExempt(value, isNumeric)
 
     const failsHard =
       !!hardRegex && !exempt && !hardRegex.test(value.toString())

client/src/app/shared/dc-validator/dc-validator.ts

@@ -206,10 +206,9 @@ export class DcValidator {
 
   /**
    * Returns the RULE_VALUEs of a HARDREGEX/SOFTREGEX rule on the given
-   * column, for display in the column-header info dropdown. A column can
+   * column, for display in the column-header info dropdown. Both are
-   * carry both rules at once (HARDREGEX blocks submission, SOFTREGEX only
+   * returned raw - the caller (buildColInfoHtml) decides which one is
-   * warns), so both are surfaced independently rather than one taking
+   * actually applied (HARDREGEX wins when both exist).
-   * precedence over the other.
    *
    * @param col column name
    */
@@ -275,28 +274,21 @@ export class DcValidator {
    * edit-record modal, which has no grid renderer to hook into, uses this
    * directly to show the same warning outside the grid.
    *
-   * A column can carry both HARDREGEX and SOFTREGEX at once. HARDREGEX is
+   * A column can carry both HARDREGEX and SOFTREGEX at once, but only one
-   * checked first: if this value fails it, SOFTREGEX is never evaluated —
+   * regex ever runs per column: if a HARDREGEX rule exists, SOFTREGEX is
-   * the cell is already red/blocked, so a yellow warning on top would be
+   * ignored entirely - the cell is already governed by the blocking rule,
-   * redundant. If HARDREGEX passes (or doesn't apply), SOFTREGEX is checked
+   * so a yellow warning on top would be redundant, even for values that
-   * independently — same precedence as makeRegexWarningRenderer.
+   * pass the hard rule.  Same precedence as makeRegexWarningRenderer.
    */
   failsSoftRegex(col: string, value: any): boolean {
-    if (isRegexRuleExempt(value)) return false
+    const isNumeric =
+      this.rules.find((rule) => rule.data === col)?.type === 'numeric'
+    if (isRegexRuleExempt(value, isNumeric)) return false
 
     const hardRegexRule = this.dqrules.find(
       (rule: DQRule) => rule.BASE_COL === col && rule.RULE_TYPE === 'HARDREGEX'
     )
-    if (hardRegexRule) {
+    if (hardRegexRule) return false
-      try {
-        if (!parseRegexRule(hardRegexRule.RULE_VALUE).test(value.toString())) {
-          return false
-        }
-      } catch (e) {
-        // Malformed HARDREGEX is treated as always-valid (see dqValidate) -
-        // fall through to SOFTREGEX.
-      }
-    }
 
     const softRegexRule = this.dqrules.find(
       (rule: DQRule) => rule.BASE_COL === col && rule.RULE_TYPE === 'SOFTREGEX'
@@ -480,7 +472,8 @@ export class DcValidator {
             this.getRegexRuleValues(ruleColName)
           this.rules[i].renderer = makeRegexWarningRenderer(
             softRegexValue,
-            hardRegexValue
+            hardRegexValue,
+            this.rules[i].type === 'numeric'
           )
         }
       }
@@ -569,7 +562,11 @@ export class DcValidator {
       }
 
       if (self.isDqCol(col || '')) {
-        const dqValid = dqValidate(self.getDqDetails(col || ''), value)
+        const dqValid = dqValidate(
+          self.getDqDetails(col || ''),
+          value,
+          colType === 'numeric'
+        )
 
         if (!dqValid) {
           console.warn(`DQ Validation - invalid (Value: ${value})`)

client/src/app/shared/dc-validator/tests/dc-validator.spec.ts

@@ -724,7 +724,7 @@ describe('DC Validator', () => {
       ).toBeFalse()
     })
 
-    it('is false for blank/special-missing values, same exemption as HARDREGEX', () => {
+    it('is false for blank values; special-missing-looking values are real text on a character column', () => {
       const dcValidator = buildValidator([
         {
           BASE_COL: 'SOME_CHAR_ANY',
@@ -735,7 +735,23 @@ describe('DC Validator', () => {
       ])
 
       expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', '')).toBeFalse()
-      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', '.a')).toBeFalse()
+      // ".a" is real text here and fails ^[A-Z]+$
+      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', '.a')).toBeTrue()
+    })
+
+    it('exempts the plain missing (".") but not special missings on a numeric column', () => {
+      const dcValidator = buildValidator([
+        {
+          BASE_COL: 'SOME_NUM',
+          RULE_TYPE: 'SOFTREGEX',
+          RULE_VALUE: '^[0-9]+$',
+          X: 0
+        }
+      ])
+
+      expect(dcValidator.failsSoftRegex('SOME_NUM', '.')).toBeFalse()
+      // Special missings are deliberately-set values and face the pattern.
+      expect(dcValidator.failsSoftRegex('SOME_NUM', '.a')).toBeTrue()
     })
 
     it('is false when a value fails both HARDREGEX and SOFTREGEX (precedence — no yellow on a red cell)', () => {
@@ -760,7 +776,7 @@ describe('DC Validator', () => {
       expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', 'ab')).toBeFalse()
     })
 
-    it('is true when a value passes HARDREGEX but fails SOFTREGEX', () => {
+    it('is false for every value when the column has HARDREGEX - SOFTREGEX never runs, even if HARDREGEX passes', () => {
       const dcValidator = buildValidator([
         {
           BASE_COL: 'SOME_CHAR_ANY',
@@ -777,9 +793,9 @@ describe('DC Validator', () => {
       ])
 
       // 'AB' passes HARDREGEX (uppercase/digits) but fails SOFTREGEX (too
-      // short) -
+      // short) - only one regex runs per column, so the soft rule is
-      // evaluated independently once HARDREGEX passes.
+      // ignored entirely.
-      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', 'AB')).toBeTrue()
+      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', 'AB')).toBeFalse()
     })
   })
 

client/src/app/shared/dc-validator/utils/applyNumericFormats.ts

@@ -7,6 +7,10 @@ import { DcValidation } from '../models/dc-validation.model'
  * Uses Intl.NumberFormat options (HOT 17+) instead of the deprecated numbro
  * `pattern`/`culture`. `maximumFractionDigits: 20` preserves all natural
  * decimals (Intl's default of 3 would round); `locale` replaces `culture`.
+ * `useGrouping: false` keeps raw digits (no thousands separator) - a
+ * separator would leak into anything that pattern-matches the displayed
+ * value (eg HARDREGEX/SOFTREGEX), and grouping can be opted into per-column
+ * with the NUMBER_FORMAT rule.
  *
  * @param rules Cell Validation rules to be updated
  * Those rules are passed in the `columns` property Of handsontable settings.
@@ -14,7 +18,7 @@ import { DcValidation } from '../models/dc-validation.model'
 export const applyNumericFormats = (rules: DcValidation[]): DcValidation[] => {
   for (let rule of rules) {
     if (rule.type === 'numeric') {
-      rule.numericFormat = { useGrouping: true, maximumFractionDigits: 20 }
+      rule.numericFormat = { useGrouping: false, maximumFractionDigits: 20 }
       rule.locale = window.navigator.language
     }
   }

client/src/app/shared/dc-validator/utils/isRegexRuleExempt.spec.ts

@@ -1,19 +1,40 @@
 import { isRegexRuleExempt } from './isRegexRuleExempt'
 
 describe('isRegexRuleExempt', () => {
-  it('exempts blank/undefined/null', () => {
+  it('exempts blank/undefined/null on any column type', () => {
     expect(isRegexRuleExempt('')).toBeTrue()
     expect(isRegexRuleExempt(undefined)).toBeTrue()
     expect(isRegexRuleExempt(null)).toBeTrue()
+    expect(isRegexRuleExempt('', true)).toBeTrue()
   })
 
-  it('exempts SAS special missing values', () => {
+  it('exempts the plain SAS missing (".") on numeric columns', () => {
-    expect(isRegexRuleExempt('.')).toBeTrue()
+    expect(isRegexRuleExempt('.', true)).toBeTrue()
-    expect(isRegexRuleExempt('.a')).toBeTrue()
+  })
-    expect(isRegexRuleExempt('_')).toBeTrue()
+
+  it('does not exempt special missings on numeric columns (they are deliberately-set values)', () => {
+    expect(isRegexRuleExempt('.a', true)).toBeFalse()
+    expect(isRegexRuleExempt('._', true)).toBeFalse()
+    expect(isRegexRuleExempt('_', true)).toBeFalse()
+    expect(isRegexRuleExempt('d', true)).toBeFalse()
+  })
+
+  it('exempts nothing but blank on character columns', () => {
+    expect(isRegexRuleExempt('.')).toBeFalse()
+    expect(isRegexRuleExempt('.a')).toBeFalse()
+    expect(isRegexRuleExempt('_')).toBeFalse()
+  })
+
+  it('does not exempt a bare single letter on character columns (real value, not a missing)', () => {
+    // A bare single letter is real character data, not a SAS special
+    // missing - it must reach the pattern, not be exempted.
+    expect(isRegexRuleExempt('d')).toBeFalse()
+    expect(isRegexRuleExempt('z')).toBeFalse()
+    expect(isRegexRuleExempt('A')).toBeFalse()
   })
 
   it('does not exempt an ordinary value', () => {
     expect(isRegexRuleExempt('ABC123')).toBeFalse()
+    expect(isRegexRuleExempt('ABC123', true)).toBeFalse()
   })
 })

client/src/app/shared/dc-validator/utils/isRegexRuleExempt.ts

@@ -1,13 +1,22 @@
-import { isSpecialMissing } from '@sasjs/utils/input/validators'
-
 /**
- * HARDREGEX/SOFTREGEX both skip pattern-matching for blank and SAS special
+ * HARDREGEX/SOFTREGEX both skip pattern-matching for:
- * missing values (., .a-.z, _) — the same exemption NOTNULL/MINVAL/MAXVAL
+ *
- * already apply elsewhere, since neither convention represents a real
+ * - blank values (undefined, null, '') on any column type - enforcing
- * formatted value the pattern is meant to check.
+ *   populated values is NOTNULL's job, not the pattern's; and
+ * - the plain SAS numeric missing (".") on numeric columns - it
+ *   represents the absence of a value, same as blank.
+ *
+ * SPECIAL missings (.a-.z, ._, bare letters) are NOT exempt, even on
+ * numeric columns: being deliberately set, they are real values the
+ * pattern is meant to check.  This also means isSpecialMissing from
+ * @sasjs/utils (which would match them, with an optional dot) is
+ * deliberately not used here.
  */
-export const isRegexRuleExempt = (value: any): boolean => {
+export const isRegexRuleExempt = (
+  value: any,
+  isNumeric: boolean = false
+): boolean => {
   if (value === undefined || value === null || value === '') return true
 
-  return isSpecialMissing(value)
+  return isNumeric && value === '.'
 }

client/src/app/shared/dc-validator/validations/dq-validation.spec.ts

@@ -30,12 +30,28 @@ describe('dqValidate - HARDREGEX', () => {
     expect(dqValidate(rules, null)).toBeTrue()
   })
 
-  it('treats SAS special missing values as valid regardless of the pattern', () => {
+  it('treats the plain SAS missing (".") as valid on numeric columns, regardless of the pattern', () => {
     const rules = [rule({ RULE_VALUE: '^[A-Z]{3}\\d{4}$' })]
 
-    expect(dqValidate(rules, '.')).toBeTrue()
+    expect(dqValidate(rules, '.', true)).toBeTrue()
-    expect(dqValidate(rules, '.a')).toBeTrue()
+  })
-    expect(dqValidate(rules, '_')).toBeTrue()
+
+  it('applies the pattern to special missings on numeric columns (deliberately-set values)', () => {
+    const rules = [rule({ RULE_VALUE: '^[A-Z]{3}\\d{4}$' })]
+
+    expect(dqValidate(rules, '.a', true)).toBeFalse()
+    expect(dqValidate(rules, '_', true)).toBeFalse()
+    expect(dqValidate(rules, 'd', true)).toBeFalse()
+  })
+
+  it('applies the pattern to special-missing-looking values on character columns', () => {
+    const rules = [rule({ RULE_VALUE: '^[A-Z]{3}\\d{4}$' })]
+
+    expect(dqValidate(rules, '.')).toBeFalse()
+    expect(dqValidate(rules, '.a')).toBeFalse()
+    expect(dqValidate(rules, '_')).toBeFalse()
+    // A bare single letter is a real character value and must be matched.
+    expect(dqValidate(rules, 'd')).toBeFalse()
   })
 
   it('fails open (treats as valid) when the pattern is malformed', () => {
@@ -44,6 +60,19 @@ describe('dqValidate - HARDREGEX', () => {
     expect(dqValidate(rules, 'anything')).toBeTrue()
   })
 
+  it('validates a single letter on a character column against the pattern', () => {
+    // A bare single letter ("d", "z") is real character data, not a SAS
+    // special missing - it must reach the pattern, not be exempted.
+    const rules = [rule({ RULE_VALUE: '/the|data/i' })]
+
+    expect(dqValidate(rules, 'd')).toBeFalse()
+    expect(dqValidate(rules, 'z')).toBeFalse()
+    expect(dqValidate(rules, 'the')).toBeTrue()
+    expect(dqValidate(rules, 'DATA')).toBeTrue()
+    expect(dqValidate(rules, 'some data here')).toBeTrue()
+    expect(dqValidate(rules, '')).toBeTrue() // blank stays exempt
+  })
+
   it('handles a more elaborate pattern (multiple character classes, quantifiers, an escaped literal dot)', () => {
     // Same email pattern used in the getdata.js mock's REGEX_HARD_COL demo.
     const rules = [rule({ RULE_VALUE: '[\\w.]+@[\\w]+\\.[a-z]{2,}' })]

client/src/app/shared/dc-validator/validations/dq-validation.ts

@@ -4,7 +4,11 @@ import { isRegexRuleExempt } from '../utils/isRegexRuleExempt'
 import { parseRegexRule } from '../utils/parseRegexRule'
 
 const dqValidation: {
-  [key: string]: (value: any, ruleValue: string | number) => boolean
+  [key: string]: (
+    value: any,
+    ruleValue: string | number,
+    isNumeric?: boolean
+  ) => boolean
 } = {
   CASE: (value: any, ruleValue: string | number): boolean => {
     switch (ruleValue) {
@@ -51,8 +55,12 @@ const dqValidation: {
   },
   // Pattern is used as authored, not auto-anchored — a rule author who
   // wants a full-value match must write ^...$ themselves.
-  HARDREGEX: (value: any, ruleValue: string | number): boolean => {
+  HARDREGEX: (
-    if (isRegexRuleExempt(value)) return true
+    value: any,
+    ruleValue: string | number,
+    isNumeric: boolean = false
+  ): boolean => {
+    if (isRegexRuleExempt(value, isNumeric)) return true
 
     try {
       return parseRegexRule(ruleValue.toString()).test(value.toString())
@@ -65,10 +73,16 @@ const dqValidation: {
   }
 }
 
-export const dqValidate = (dqRules: DQRule[], value: any): boolean => {
+export const dqValidate = (
+  dqRules: DQRule[],
+  value: any,
+  isNumeric: boolean = false
+): boolean => {
   for (let detail of dqRules) {
     if (dqValidation[detail.RULE_TYPE]) {
-      if (!dqValidation[detail.RULE_TYPE](value, detail.RULE_VALUE)) {
+      if (
+        !dqValidation[detail.RULE_TYPE](value, detail.RULE_VALUE, isNumeric)
+      ) {
         console.warn(
           `DQ Invalid Reason: ${
             detail.RULE_TYPE

client/src/app/shared/utils/col-info-html.spec.ts

@@ -19,7 +19,7 @@ describe('buildColInfoHtml', () => {
     )
   })
 
-  it('appends a REGEX line when only a HARDREGEX value is provided', () => {
+  it('appends a HARDREGEX line when only a HARDREGEX value is provided', () => {
     const colInfo: DataFormat = {
       label: 'Some Character Column',
       type: 'char',
@@ -30,11 +30,11 @@ describe('buildColInfoHtml', () => {
     expect(
       buildColInfoHtml('SOME_CHAR', colInfo, '/^[A-Z]+$/i', undefined)
     ).toBe(
-      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>REGEX: /^[A-Z]+$/i'
+      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>HARDREGEX: /^[A-Z]+$/i'
     )
   })
 
-  it('appends a REGEX line when only a SOFTREGEX value is provided', () => {
+  it('appends a SOFTREGEX line when only a SOFTREGEX value is provided', () => {
     const colInfo: DataFormat = {
       label: 'Some Character Column',
       type: 'char',
@@ -45,11 +45,11 @@ describe('buildColInfoHtml', () => {
     expect(
       buildColInfoHtml('SOME_CHAR', colInfo, undefined, '/^[a-z]+$/')
     ).toBe(
-      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>REGEX: /^[a-z]+$/'
+      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>SOFTREGEX: /^[a-z]+$/'
     )
   })
 
-  it('appends separate HARDREGEX/SOFTREGEX lines when a column has both', () => {
+  it('appends only the HARDREGEX line when a column has both rules, since HARDREGEX is the one applied', () => {
     const colInfo: DataFormat = {
       label: 'Some Character Column',
       type: 'char',
@@ -60,7 +60,7 @@ describe('buildColInfoHtml', () => {
     expect(
       buildColInfoHtml('SOME_CHAR', colInfo, '/^[A-Z]+$/i', '/^.{5,10}$/')
     ).toBe(
-      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>HARDREGEX: /^[A-Z]+$/i<br>SOFTREGEX: /^.{5,10}$/'
+      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>HARDREGEX: /^[A-Z]+$/i'
     )
   })
 

client/src/app/shared/utils/col-info-html.ts

@@ -15,13 +15,14 @@ export function buildColInfoHtml(
 
   let html = `NAME: ${colName}<br>LABEL: ${colInfo.label}<br>TYPE: ${colInfo.type}<br>LENGTH: ${colInfo.length}<br>FORMAT: ${colInfo.format}`
 
-  // HARDREGEX/SOFTREGEX are only distinguished by name when a column has
+  // Only ever one REGEX rule is applied per column: when both HARDREGEX
-  // both - with just one rule, which one it is is already implied, so the
+  // and SOFTREGEX exist, SOFTREGEX is ignored entirely (same precedence as
-  // generic REGEX label keeps the common case uncluttered.
+  // makeRegexWarningRenderer / DcValidator.failsSoftRegex).  Show only the
-  if (hardRegexValue && softRegexValue) {
+  // rule that is applied.
-    html += `<br>HARDREGEX: ${hardRegexValue}<br>SOFTREGEX: ${softRegexValue}`
+  if (hardRegexValue) {
-  } else if (hardRegexValue || softRegexValue) {
+    html += `<br>HARDREGEX: ${hardRegexValue}`
-    html += `<br>REGEX: ${hardRegexValue || softRegexValue}`
+  } else if (softRegexValue) {
+    html += `<br>SOFTREGEX: ${softRegexValue}`
   }
 
   return html

sas/sasjs/macros/mpe_makedata.sas

@@ -2012,6 +2012,29 @@ insert into &lib..MPE_VALIDATIONS set
     ,rule_value='/t/'
     ,rule_active=1
     ,tx_to='31DEC5999:23:59:59'dt;
+/* test soft regex - PRIMARY_KEY_FIELD should be an integer (yellow if it
+contains a decimal point).  All generated keys are integers, so no
+warning shows until a demo user enters a decimal. */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0
+    ,base_lib="&lib"
+    ,base_ds="MPE_X_TEST"
+    ,base_col="PRIMARY_KEY_FIELD"
+    ,rule_type='SOFTREGEX'
+    ,rule_value='/^\d+$/'
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* test hard regex - SOME_SHORTNUM must not be between 1 and 5.  Generated
+data starts at 6, so nothing is blocked until a demo user enters 1-5. */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0
+    ,base_lib="&lib"
+    ,base_ds="MPE_X_TEST"
+    ,base_col="SOME_SHORTNUM"
+    ,rule_type='HARDREGEX'
+    ,rule_value='/^(?![1-5](\.\d+)?$).*/'
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
 insert into &lib..MPE_VALIDATIONS set
     tx_from=0
     ,base_lib="&lib"
@@ -2061,7 +2084,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=42
       ,some_datetime=42
       ,some_time=42
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=1
@@ -2071,7 +2094,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=42
       ,some_datetime=42
       ,some_time=422
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=2
@@ -2081,7 +2104,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=42
       ,some_datetime=42
       ,some_time=142
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=3
@@ -2093,7 +2116,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=423
       ,some_datetime=423
       ,some_time=44
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=4
@@ -2103,7 +2126,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=4231
       ,some_datetime=423123123
       ,some_time=412
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
 %do x=10 %to 500;
   insert into &lib..mpe_x_test
@@ -2114,7 +2137,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=round(ranuni(0)*1000,1)
       ,some_datetime=round(ranuni(0)*50000,1)
       ,some_time=round(ranuni(0)*100,1)
-      ,some_shortnum=round(ranuni(0)*100,1)
+      ,some_shortnum=6+round(ranuni(0)*94,1)
       ,some_bestnum=round(ranuni(0)*100,1);
 %end;