dc/.agent/docs/regex-validations.md

Regex Validations (HARDREGEX / SOFTREGEX)

This document describes how the regex validation rules work internally. For user-facing documentation see docs/dcc-validations.md in the docs.datacontroller.io repo.

Overview

Two validation rule types in MPE_VALIDATIONS validate cell values against a regular expression supplied in RULE_VALUE:

• HARDREGEX - submission-blocking. A failing value is rejected by the cell validator and painted red (HOT's own htInvalid class), so the row cannot be submitted.
• SOFTREGEX - display-only warning. A failing value is painted yellow (dc-warning-cell class) with a tooltip, but submission is not blocked.

Both rule types are selectable in the MPE_VALIDATIONS RULE_TYPE dropdown; they were added to the selectbox seed data in sas/sasjs/macros/mpe_makedata.sas and via the optional migration sas/sasjs/db/migrations/20260720_v7.12_release.sas. RULE_VALUE is limited to 128 characters, which constrains very long patterns.

Config-time validation (SAS side)

sas/sasjs/services/hooks/mpe_validations_postedit.sas runs prxparse() on any staged HARDREGEX/SOFTREGEX rule value and aborts the edit with a list of offending libref.table.column references if the pattern is invalid. This is a best-effort syntax check to catch typos at config time; an empty pattern is treated as valid (it matches everything in JS). Rows marked for delete are skipped.

Because patterns must pass prxparse, rule values are authored in the SAS PRX delimiter form /pattern/flags (although a bare pattern is also accepted for backwards compatibility).

Frontend evaluation

All regex handling lives in the client; there is no server-side re-validation of data values. The per-cell decision flow:

flowchart TD
    A[Cell value] --> B{Row marked for delete<br/>and not a PK column?}
    B -- Yes --> Z[No validation / no warning]
    B -- No --> C{isRegexRuleExempt?<br/>blank, or "." on a numeric column}
    C -- Yes --> Z
    C -- No --> D{HARDREGEX rule on column?}
    D -- Yes --> E{Pattern matches?}
    E -- No --> F[Invalid: submission blocked,<br/>red htInvalid + REGEX tooltip]
    E -- Yes --> J[Valid]
    D -- No --> G{SOFTREGEX rule on column?}
    G -- Yes --> H{Pattern matches?}
    H -- No --> I[Warning: yellow dc-warning-cell<br/>+ REGEX tooltip, submission allowed]
    H -- Yes --> J[Valid]
    G -- No --> J

A malformed pattern never reaches this flow: it is treated as always-valid (HARDREGEX) / never-warn (SOFTREGEX) with a console.warn, rather than breaking the editor.

client/src/app/shared/dc-validator/utils/parseRegexRule.ts

Converts an authored SAS PRX pattern into a JavaScript RegExp:

1. If the value matches /^\/(.*)\/([a-z]*)$/s, the delimiters are stripped and the trailing flags are passed to new RegExp(body, flags). Otherwise the value is used as-is (backwards compatibility with bare patterns).
2. Three mechanical Perl→JS translations are applied to the body:
   • a leading (?i) inline modifier is removed and folded into the i flag;
   • \Q...\E literal sequences are replaced with escaped literal text;
   • \A → ^ and \z → (?![\s\S]) (end-of-string anchor).
3. Perl-only constructs that would need a capture-group-renumbering rewrite (atomic groups (?>...), possessive quantifiers a++) are deliberately NOT translated. They throw from new RegExp, and every caller treats a throw as "always valid / never warn" (see below) rather than breaking the editor.

client/src/app/shared/dc-validator/utils/isRegexRuleExempt.ts

Blank values (undefined, null, '') are exempt from pattern matching on any column type - use a separate NOTNULL rule if populated values must also be enforced. On numeric columns the plain SAS missing (.) is also exempt; special missings (.A-.Z, ._, bare letters) are NOT exempt anywhere - being deliberately set, they are real values the pattern must match (and on a character column even . is real text). isSpecialMissing from @sasjs/utils is deliberately not used: its optional-dot regex would exempt any single-letter character value ("d", "z") before the regex ever ran. The check takes an isNumeric flag, passed by all callers (the dq validator via dqValidate(rules, value, colType === 'numeric'), the warning renderer via a makeRegexWarningRenderer argument, and failsSoftRegex via the column's HOT type).

HARDREGEX - blocking validation

HARDREGEX is implemented as a cell validator in client/src/app/shared/dc-validator/validations/dq-validation.ts. It returns true (valid) for exempt values and for patterns that fail to compile (with a console.warn), and otherwise returns parseRegexRule(ruleValue).test(value.toString()). Returning false makes HOT mark the cell invalid, block submission, and paint it red via its standard htInvalid styling.

SOFTREGEX - warning renderer

client/src/app/editor/utils/regex-warning-renderer.ts builds a display-only Handsontable renderer (registered per-column by DcValidator.setupRules in client/src/app/shared/dc-validator/dc-validator.ts). It never returns false; it only:

• adds a REGEX: <pattern> tooltip (td.title) when a rule fails;
• adds the yellow dc-warning-cell class when only SOFTREGEX fails.

DcValidator.failsSoftRegex(col, value) provides the same logic outside the grid (e.g. the edit-record screen).

Precedence: HARD and SOFT on the same column

Only one regex ever runs per column. If a HARDREGEX rule exists, SOFTREGEX is ignored entirely - never compiled, never evaluated - regardless of whether individual cell values pass or fail the hard rule. A value failing HARDREGEX gets the red invalid styling (blocking submission) plus a REGEX: <pattern> tooltip; a SOFTREGEX-only column warns in yellow without blocking. This holds in both the renderer and failsSoftRegex.

Other behaviour

• Rows marked for delete (_____DELETE__THIS__RECORD_____ = 'Yes') are not warned/validated by the renderer (except primary key columns, which still validate).
• A malformed pattern never breaks the editor: the dq validator treats it as always-valid and the renderer disables the warning, logging to the console instead.
• The pattern is used as authored - it is NOT auto-anchored. Authors must include ^/$ to match the entire cell value.
• Column info: client/src/app/shared/utils/col-info-html.ts shows the applied pattern in the column-info dropdown - the HARDREGEX pattern if one exists (it is the rule actually applied when both are present), otherwise the SOFTREGEX pattern, otherwise nothing.

Testing

• Unit tests: parseRegexRule.spec.ts, isRegexRuleExempt.spec.ts, dq-validation.spec.ts, dc-validator.spec.ts (under client/src/app/shared/dc-validator/), client/src/app/editor/utils/regex-warning-renderer.spec.ts, client/src/app/shared/utils/col-info-html.spec.ts.
• E2E: client/cypress/e2e/editor.cy.ts.
• SAS side: sas/sasjs/services/editors/stagedata.test.3.sas, plus seed data in mpe_makedata.sas: HARDREGEX "SOME_CHAR must contain 'the' or 'data'" (/the|data/i), SOFTREGEX "SOME_CHAR should contain the letter 't'", SOFTREGEX on PRIMARY_KEY_FIELD (/^\d+$/ - yellow if the key contains a decimal), and HARDREGEX on SOME_SHORTNUM (/^(?![1-5](\.\d+)?$).*/ - values 1-5 blocked; generated data starts at 6 so demos aren't blocked accidentally).