# Regex Validations (HARDREGEX / SOFTREGEX)

This document describes how the regex validation rules work internally. For user-facing documentation see `docs/dcc-validations.md` in the `docs.datacontroller.io` repo.

## Overview

Two validation rule types in `MPE_VALIDATIONS` validate cell values against a regular expression supplied in `RULE_VALUE`:

- `HARDREGEX` - submission-blocking. A failing value is rejected by the cell validator and painted red (HOT's own `htInvalid` class), so the row cannot be submitted.
- `SOFTREGEX` - display-only warning. A failing value is painted yellow (`dc-warning-cell` class) with a tooltip, but submission is not blocked.

Both rule types are selectable in the MPE_VALIDATIONS RULE_TYPE dropdown; they were added to the selectbox seed data in `sas/sasjs/macros/mpe_makedata.sas` and via the optional migration `sas/sasjs/db/migrations/20260720_v7.12_release.sas`. `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.

## Config-time validation (SAS side)

`sas/sasjs/services/hooks/mpe_validations_postedit.sas` runs `prxparse()` on any staged HARDREGEX/SOFTREGEX rule value and aborts the edit with a list of offending `libref.table.column` references if the pattern is invalid. This is a best-effort syntax check to catch typos at config time; an empty pattern is treated as valid (it matches everything in JS). Rows marked for delete are skipped.

Because patterns must pass `prxparse`, rule values are authored in the SAS PRX delimiter form `/pattern/flags` (although a bare pattern is also accepted for backwards compatibility).

## Frontend evaluation

All regex handling lives in the client; there is no server-side re-validation of data values.  The per-cell decision flow:

```mermaid
flowchart TD
    A[Cell value] --> B{Row marked for delete<br/>and not a PK column?}
    B -- Yes --> Z[No validation / no warning]
    B -- No --> C{isRegexRuleExempt?<br/>blank, or "." on a numeric column}
    C -- Yes --> Z
    C -- No --> D{HARDREGEX rule on column?}
    D -- Yes --> E{Pattern matches?}
    E -- No --> F[Invalid: submission blocked,<br/>red htInvalid + REGEX tooltip]
    E -- Yes --> J[Valid]
    D -- No --> G{SOFTREGEX rule on column?}
    G -- Yes --> H{Pattern matches?}
    H -- No --> I[Warning: yellow dc-warning-cell<br/>+ REGEX tooltip, submission allowed]
    H -- Yes --> J[Valid]
    G -- No --> J
```

A malformed pattern never reaches this flow: it is treated as always-valid (HARDREGEX) / never-warn (SOFTREGEX) with a `console.warn`, rather than breaking the editor.

### `client/src/app/shared/dc-validator/utils/parseRegexRule.ts`

Converts an authored SAS PRX pattern into a JavaScript `RegExp`:

1. If the value matches `/^\/(.*)\/([a-z]*)$/s`, the delimiters are stripped and the trailing flags are passed to `new RegExp(body, flags)`. Otherwise the value is used as-is (backwards compatibility with bare patterns).
2. Three mechanical Perl→JS translations are applied to the body:
   - a leading `(?i)` inline modifier is removed and folded into the `i` flag;
   - `\Q...\E` literal sequences are replaced with escaped literal text;
   - `\A` → `^` and `\z` → `(?![\s\S])` (end-of-string anchor).
3. Perl-only constructs that would need a capture-group-renumbering rewrite (atomic groups `(?>...)`, possessive quantifiers `a++`) are deliberately NOT translated. They throw from `new RegExp`, and every caller treats a throw as "always valid / never warn" (see below) rather than breaking the editor.

### `client/src/app/shared/dc-validator/utils/isRegexRuleExempt.ts`

Blank values (`undefined`, `null`, `''`) are exempt from pattern matching on any column type - use a separate NOTNULL rule if populated values must also be enforced. On numeric columns the plain SAS missing (`.`) is also exempt; special missings (`.A`-`.Z`, `._`, bare letters) are NOT exempt anywhere - being deliberately set, they are real values the pattern must match (and on a character column even `.` is real text). `isSpecialMissing` from `@sasjs/utils` is deliberately not used: its optional-dot regex would exempt any single-letter character value ("d", "z") before the regex ever ran. The check takes an `isNumeric` flag, passed by all callers (the dq validator via `dqValidate(rules, value, colType === 'numeric')`, the warning renderer via a `makeRegexWarningRenderer` argument, and `failsSoftRegex` via the column's HOT type).

### HARDREGEX - blocking validation

`HARDREGEX` is implemented as a cell validator in `client/src/app/shared/dc-validator/validations/dq-validation.ts`. It returns `true` (valid) for exempt values and for patterns that fail to compile (with a `console.warn`), and otherwise returns `parseRegexRule(ruleValue).test(value.toString())`. Returning `false` makes HOT mark the cell invalid, block submission, and paint it red via its standard `htInvalid` styling.

### SOFTREGEX - warning renderer

`client/src/app/editor/utils/regex-warning-renderer.ts` builds a display-only Handsontable renderer (registered per-column by `DcValidator.setupRules` in `client/src/app/shared/dc-validator/dc-validator.ts`). It never returns false; it only:

- adds a `REGEX: <pattern>` tooltip (`td.title`) when a rule fails;
- adds the yellow `dc-warning-cell` class when only SOFTREGEX fails.

`DcValidator.failsSoftRegex(col, value)` provides the same logic outside the grid (e.g. the edit-record screen).

### Precedence: HARD and SOFT on the same column

Only one regex ever runs per column. If a HARDREGEX rule exists, SOFTREGEX is ignored entirely - never compiled, never evaluated - regardless of whether individual cell values pass or fail the hard rule. A value failing HARDREGEX gets the red invalid styling (blocking submission) plus a `REGEX: <pattern>` tooltip; a SOFTREGEX-only column warns in yellow without blocking. This holds in both the renderer and `failsSoftRegex`.

### Other behaviour

- Rows marked for delete (`_____DELETE__THIS__RECORD_____ = 'Yes'`) are not warned/validated by the renderer (except primary key columns, which still validate).
- A malformed pattern never breaks the editor: the dq validator treats it as always-valid and the renderer disables the warning, logging to the console instead.
- The pattern is used as authored - it is NOT auto-anchored. Authors must include `^`/`$` to match the entire cell value.
- Column info: `client/src/app/shared/utils/col-info-html.ts` shows the applied pattern in the column-info dropdown - the HARDREGEX pattern if one exists (it is the rule actually applied when both are present), otherwise the SOFTREGEX pattern, otherwise nothing.

## Testing

- Unit tests: `parseRegexRule.spec.ts`, `isRegexRuleExempt.spec.ts`, `dq-validation.spec.ts`, `dc-validator.spec.ts` (under `client/src/app/shared/dc-validator/`), `client/src/app/editor/utils/regex-warning-renderer.spec.ts`, `client/src/app/shared/utils/col-info-html.spec.ts`.
- E2E: `client/cypress/e2e/editor.cy.ts`.
- SAS side: `sas/sasjs/services/editors/stagedata.test.3.sas`, plus seed data in `mpe_makedata.sas`: HARDREGEX "SOME_CHAR must contain 'the' or 'data'" (`/the|data/i`), SOFTREGEX "SOME_CHAR should contain the letter 't'", SOFTREGEX on PRIMARY_KEY_FIELD (`/^\d+$/` - yellow if the key contains a decimal), and HARDREGEX on SOME_SHORTNUM (`/^(?![1-5](\.\d+)?$).*/` - values 1-5 blocked; generated data starts at 6 so demos aren't blocked accidentally).