Two new validation rule types apply regular expressions to cell values, configured via MPE_VALIDATIONS:
HARDREGEX — a non-matching value blocks submission (cell renders red), on the same path as the existing CASE / MINVAL / MAXVAL / NOTNULL rules.
SOFTREGEX — a non-matching value shows a yellow warning but submission is still allowed. Since the cell validator would block submission, this is wired as a display-only grid renderer instead, and mirrored in the edit-record modal (which has no grid renderer to hook into) via DcValidator.failsSoftRegex.
Behaviour details:
SAS-side validation — a new post-edit hook on MPE_VALIDATIONS (mpe_validations_postedit.sas) validates regex patterns with PRXPARSE when the rule is saved; invalid patterns are rejected with the offending columns listed. Patterns may be written in SAS PRX /pattern/flags syntax or bare form — the frontend parses both (parseRegexRule).
Frontend evaluation — patterns are evaluated with the JavaScript regex engine. They are used as authored (not auto-anchored), so rule authors write ^...$ themselves for a full-value match. Malformed-in-the-browser patterns fail open (treated as valid, logged via console.warn) rather than blocking every submission on that column.
Exemptions — blank values are always exempt; on numeric columns the plain SAS missing (.) is also exempt. Special missings (.A-.Z, ._) are deliberately-set values and are validated against the pattern. Cells in rows marked for deletion are not validated/warned (except primary-key columns).
One regex per column — if a column has both rules, the SOFTREGEX is ignored entirely (the column behaves as HARDREGEX-only); the column-header info dropdown and cell tooltip show the applied pattern (col-info-html).
New dc-warning-cell CSS class for the SOFTREGEX warning, reusing the app's existing Clarity/CDS warning colour tokens rather than a hardcoded colour.
Mock coverage: two new demo columns on MPE_X_TEST / MPE_X_NEW (REGEX_HARD_COL — email pattern, REGEX_SOFT_COL — UK postcode pattern), following the existing RULE_DEMO_COLS convention.
Documentation: new "Regex Rules" section in docs/dcc-validations.md and updated RULE_TYPE values in docs/tables/mpe_validations.md (docs.datacontroller.io).
Row-level security with EDIT (or ALL) scope is incompatible with tables configured with the REPLACE loadtype. This is now enforced in the backend post-edit hooks:
mpe_row_level_security_postedit.sas rejects submissions that would apply EDIT-scope RLS to a REPLACE-loadtype table.
mpe_tables_postedit.sas performs the symmetric check, preventing a table being switched to REPLACE while EDIT-scope RLS rules exist.
Fixed viewbox filter state leaking into (and from) the base table's filter state — each viewbox now maintains its own isolated filters. Includes Cypress fixes for the previously flaky viewbox.cy.ts assertions, now added to the CI test suite.
Angular 19 → 20 and Clarity bump, with a crypto/browserify shim for sheetJS.
Bumped @sasjs/core and patched audit vulnerabilities across client and sas lockfiles — npm audit is now at 0 vulnerabilities, and the CI check has been made strict (fails on any audit finding).
Lockfile fixes to keep npm ci working in the pipeline (peer-dependency drift).
Ensured no assets (including og: meta links) are fetched from external sources, keeping DC fully offline/on-prem capable.
Test plan
npm run test:headless — unit tests green, incl. new dq-validation.spec.ts, parseRegexRule.spec.ts, regex-warning-renderer.spec.ts, isRegexRuleExempt.spec.ts, col-info-html.spec.ts, and new cases in dc-validator.spec.ts (matching/non-matching values, blank/missing exemptions, malformed patterns, dual-rule behaviour, modal support, regression coverage using real HARDREGEX/SOFTREGEX rule values)
npx tsc --noEmit — clean
npm run lint:check / sasjs lint — clean on touched files
npm run build — production build succeeds
npm audit — 0 vulnerabilities (now enforced in CI)
CI: build + Cypress green (editor.cy.ts incl. new HARDREGEX/SOFTREGEX cases against MPE_X_TEST, and viewbox.cy.ts)
SAS tests: sasjs test — incl. new stagedata.test.3.sas and updated RLS/validations hook tests
Manual: on MPE_X_TEST, an invalid email in REGEX_HARD_COL blocks submission (red); an invalid postcode in REGEX_SOFT_COL shows a yellow warning but still submits; marking that row for delete clears the warning; the same behaviour is visible in the edit-record modal; the applied pattern shows in the column info dropdown
Manual: attempting to save an EDIT-scope RLS rule against a REPLACE-loadtype table (or switching such a table to REPLACE) is rejected with a clear message
## Issue
Closes #273 (HARDREGEX + SOFTREGEX validation rules)
Closes #279 (display all libs in MPE_SECURITY)
Closes #211 (prevent incompatible RLS + REPLACE loadtype combinations)
Closes #239 (viewbox filter state leaking into the base table's filters)
Closes #276 (Angular / Clarity upgrade + sheetJS shim to clear audit CVEs)
## Summary
### New HARDREGEX / SOFTREGEX DQ rules (#273)
Two new validation rule types apply regular expressions to cell values, configured via `MPE_VALIDATIONS`:
- **`HARDREGEX`** — a non-matching value blocks submission (cell renders red), on the same path as the existing `CASE` / `MINVAL` / `MAXVAL` / `NOTNULL` rules.
- **`SOFTREGEX`** — a non-matching value shows a yellow warning but submission is still allowed. Since the cell validator would block submission, this is wired as a display-only grid renderer instead, and mirrored in the edit-record modal (which has no grid renderer to hook into) via `DcValidator.failsSoftRegex`.
Behaviour details:
- **SAS-side validation** — a new post-edit hook on `MPE_VALIDATIONS` (`mpe_validations_postedit.sas`) validates regex patterns with `PRXPARSE` when the rule is saved; invalid patterns are rejected with the offending columns listed. Patterns may be written in SAS PRX `/pattern/flags` syntax or bare form — the frontend parses both (`parseRegexRule`).
- **Frontend evaluation** — patterns are evaluated with the JavaScript regex engine. They are used as authored (not auto-anchored), so rule authors write `^...$` themselves for a full-value match. Malformed-in-the-browser patterns fail open (treated as valid, logged via `console.warn`) rather than blocking every submission on that column.
- **Exemptions** — blank values are always exempt; on numeric columns the plain SAS missing (`.`) is also exempt. Special missings (`.A`-`.Z`, `._`) are deliberately-set values and **are** validated against the pattern. Cells in rows marked for deletion are not validated/warned (except primary-key columns).
- **One regex per column** — if a column has both rules, the SOFTREGEX is ignored entirely (the column behaves as HARDREGEX-only); the column-header info dropdown and cell tooltip show the applied pattern (`col-info-html`).
- New `dc-warning-cell` CSS class for the SOFTREGEX warning, reusing the app's existing Clarity/CDS warning colour tokens rather than a hardcoded colour.
- Mock coverage: two new demo columns on `MPE_X_TEST` / `MPE_X_NEW` (`REGEX_HARD_COL` — email pattern, `REGEX_SOFT_COL` — UK postcode pattern), following the existing `RULE_DEMO_COLS` convention.
- Documentation: new "Regex Rules" section in `docs/dcc-validations.md` and updated `RULE_TYPE` values in `docs/tables/mpe_validations.md` (docs.datacontroller.io).
### RLS / REPLACE loadtype validation (#211)
Row-level security with EDIT (or ALL) scope is incompatible with tables configured with the `REPLACE` loadtype. This is now enforced in the backend post-edit hooks:
- `mpe_row_level_security_postedit.sas` rejects submissions that would apply EDIT-scope RLS to a REPLACE-loadtype table.
- `mpe_tables_postedit.sas` performs the symmetric check, preventing a table being switched to REPLACE while EDIT-scope RLS rules exist.
### MPE_SECURITY shows all libraries (#279)
The LIBREF validation dropdown in `MPE_SECURITY` now lists all assigned libraries instead of only libraries that contain editable tables.
### Viewbox filter isolation (#239)
Fixed viewbox filter state leaking into (and from) the base table's filter state — each viewbox now maintains its own isolated filters. Includes Cypress fixes for the previously flaky `viewbox.cy.ts` assertions, now added to the CI test suite.
### Dependency upgrades / security (#276)
- Angular 19 → 20 and Clarity bump, with a crypto/browserify shim for sheetJS.
- Bumped `@sasjs/core` and patched audit vulnerabilities across `client` and `sas` lockfiles — `npm audit` is now at **0 vulnerabilities**, and the CI check has been made strict (fails on any audit finding).
- Lockfile fixes to keep `npm ci` working in the pipeline (peer-dependency drift).
- Ensured no assets (including `og:` meta links) are fetched from external sources, keeping DC fully offline/on-prem capable.
## Test plan
- [ ] `npm run test:headless` — unit tests green, incl. new `dq-validation.spec.ts`, `parseRegexRule.spec.ts`, `regex-warning-renderer.spec.ts`, `isRegexRuleExempt.spec.ts`, `col-info-html.spec.ts`, and new cases in `dc-validator.spec.ts` (matching/non-matching values, blank/missing exemptions, malformed patterns, dual-rule behaviour, modal support, regression coverage using real HARDREGEX/SOFTREGEX rule values)
- [ ] `npx tsc --noEmit` — clean
- [ ] `npm run lint:check` / `sasjs lint` — clean on touched files
- [ ] `npm run build` — production build succeeds
- [ ] `npm audit` — 0 vulnerabilities (now enforced in CI)
- [ ] CI: build + Cypress green (`editor.cy.ts` incl. new HARDREGEX/SOFTREGEX cases against `MPE_X_TEST`, and `viewbox.cy.ts`)
- [ ] SAS tests: `sasjs test` — incl. new `stagedata.test.3.sas` and updated RLS/validations hook tests
- [ ] Manual: on `MPE_X_TEST`, an invalid email in `REGEX_HARD_COL` blocks submission (red); an invalid postcode in `REGEX_SOFT_COL` shows a yellow warning but still submits; marking that row for delete clears the warning; the same behaviour is visible in the edit-record modal; the applied pattern shows in the column info dropdown
- [ ] Manual: attempting to save an EDIT-scope RLS rule against a REPLACE-loadtype table (or switching such a table to REPLACE) is rejected with a clear message
Two new DQ rule types apply regular expressions to cell values:
HARDREGEX blocks submission on a non-matching value (same path as the
existing CASE/MINVAL/MAXVAL rules); SOFTREGEX is display-only — a
non-matching value gets a yellow warning cell but can still submit,
so it's wired as a grid renderer rather than a validator, and mirrored
in the edit-record modal (which has no grid renderer to hook into) via
DcValidator.failsSoftRegex. Both rules exempt blank and SAS special
missing values, and fail open on a malformed pattern rather than
blocking every submission on that column. HARDREGEX takes precedence
when both rules apply to the same column, so a failing value renders
red/blocked, never yellow.
Yury
requested review from allan 2026-07-20 15:20:27 +00:00
Yury
requested review from sead 2026-07-20 15:20:27 +00:00
Yury
requested review from trevor 2026-07-20 15:20:27 +00:00
RULE_VALUE is authored in PRX delimiter form because prxparse() requires
it, but HARDREGEX, the SOFTREGEX grid renderer, and failsSoftRegex were
all passing that string straight into `new RegExp()`, so the delimiters
and flags were matched as literal characters instead of applied - making
these rules silently never match real data.
Adds parseRegexRule (extracted, tested independently) to strip the
delimiters, apply flags, hoist a leading (?i) modifier, and translate
\Q...\E and \A/\z to their JS equivalents. Atomic groups and possessive
quantifiers are left unfixed (documented, fail-safe) - translating them
risks renumbering the pattern's own capture groups.
Also switches the REGEX_HARD_COL/REGEX_SOFT_COL mock rules to the
delimited form so editor.cy.ts's existing e2e coverage actually exercises
this path.
Bumps @angular/* to 21 (passing through 20 as a transient step, since
vendored Clarity has no Angular-20-compatible release) and
@typescript-eslint to 8.65.0 for TS 5.9 compat. Applies the required
ng update migrations, including *ngIf/*ngFor/*ngSwitch -> control-flow
syntax across 37 templates.
Verifies parseRegexRule against a corpus of RULE_VALUEs collected from an
existing MPE_VALIDATIONS table - none use PRX-only syntax, so behavior is
confirmed to match SAS PRX exactly, not just "doesn't throw". Pins two
data quirks found along the way (a range vs. literal-hyphen character
class, and a double-escaped lookahead that's a no-op in both engines)
rather than silently treating either as a bug to fix.
Replace one-shot .then()/for-loop matching with retrying .should()/cy.contains()
lookups, drop stale cached DOM node references (viyaLib) in favor of live re-queries.
Both rules can now apply to one column: HARDREGEX still blocks submission
and takes its own tooltip, but SOFTREGEX is evaluated (and shown) whenever
HARDREGEX passes, instead of being silently suppressed whenever HARDREGEX
was merely present. Column-header info dropdown labels each rule separately
when a column has both.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
sead
trevor
allan
Issue
Closes #273 (HARDREGEX + SOFTREGEX validation rules)
Closes #279 (display all libs in MPE_SECURITY)
Closes #211 (prevent incompatible RLS + REPLACE loadtype combinations)
Closes #239 (viewbox filter state leaking into the base table's filters)
Closes #276 (Angular / Clarity upgrade + sheetJS shim to clear audit CVEs)
Summary
New HARDREGEX / SOFTREGEX DQ rules (#273)
Two new validation rule types apply regular expressions to cell values, configured via
MPE_VALIDATIONS:HARDREGEX— a non-matching value blocks submission (cell renders red), on the same path as the existingCASE/MINVAL/MAXVAL/NOTNULLrules.SOFTREGEX— a non-matching value shows a yellow warning but submission is still allowed. Since the cell validator would block submission, this is wired as a display-only grid renderer instead, and mirrored in the edit-record modal (which has no grid renderer to hook into) viaDcValidator.failsSoftRegex.Behaviour details:
MPE_VALIDATIONS(mpe_validations_postedit.sas) validates regex patterns withPRXPARSEwhen the rule is saved; invalid patterns are rejected with the offending columns listed. Patterns may be written in SAS PRX/pattern/flagssyntax or bare form — the frontend parses both (parseRegexRule).^...$themselves for a full-value match. Malformed-in-the-browser patterns fail open (treated as valid, logged viaconsole.warn) rather than blocking every submission on that column..) is also exempt. Special missings (.A-.Z,._) are deliberately-set values and are validated against the pattern. Cells in rows marked for deletion are not validated/warned (except primary-key columns).col-info-html).dc-warning-cellCSS class for the SOFTREGEX warning, reusing the app's existing Clarity/CDS warning colour tokens rather than a hardcoded colour.MPE_X_TEST/MPE_X_NEW(REGEX_HARD_COL— email pattern,REGEX_SOFT_COL— UK postcode pattern), following the existingRULE_DEMO_COLSconvention.docs/dcc-validations.mdand updatedRULE_TYPEvalues indocs/tables/mpe_validations.md(docs.datacontroller.io).RLS / REPLACE loadtype validation (#211)
Row-level security with EDIT (or ALL) scope is incompatible with tables configured with the
REPLACEloadtype. This is now enforced in the backend post-edit hooks:mpe_row_level_security_postedit.sasrejects submissions that would apply EDIT-scope RLS to a REPLACE-loadtype table.mpe_tables_postedit.sasperforms the symmetric check, preventing a table being switched to REPLACE while EDIT-scope RLS rules exist.MPE_SECURITY shows all libraries (#279)
The LIBREF validation dropdown in
MPE_SECURITYnow lists all assigned libraries instead of only libraries that contain editable tables.Viewbox filter isolation (#239)
Fixed viewbox filter state leaking into (and from) the base table's filter state — each viewbox now maintains its own isolated filters. Includes Cypress fixes for the previously flaky
viewbox.cy.tsassertions, now added to the CI test suite.Dependency upgrades / security (#276)
@sasjs/coreand patched audit vulnerabilities acrossclientandsaslockfiles —npm auditis now at 0 vulnerabilities, and the CI check has been made strict (fails on any audit finding).npm ciworking in the pipeline (peer-dependency drift).og:meta links) are fetched from external sources, keeping DC fully offline/on-prem capable.Test plan
npm run test:headless— unit tests green, incl. newdq-validation.spec.ts,parseRegexRule.spec.ts,regex-warning-renderer.spec.ts,isRegexRuleExempt.spec.ts,col-info-html.spec.ts, and new cases indc-validator.spec.ts(matching/non-matching values, blank/missing exemptions, malformed patterns, dual-rule behaviour, modal support, regression coverage using real HARDREGEX/SOFTREGEX rule values)npx tsc --noEmit— cleannpm run lint:check/sasjs lint— clean on touched filesnpm run build— production build succeedsnpm audit— 0 vulnerabilities (now enforced in CI)editor.cy.tsincl. new HARDREGEX/SOFTREGEX cases againstMPE_X_TEST, andviewbox.cy.ts)sasjs test— incl. newstagedata.test.3.sasand updated RLS/validations hook testsMPE_X_TEST, an invalid email inREGEX_HARD_COLblocks submission (red); an invalid postcode inREGEX_SOFT_COLshows a yellow warning but still submits; marking that row for delete clears the warning; the same behaviour is visible in the edit-record modal; the applied pattern shows in the column info dropdownfeat(editor): add HARDREGEX/SOFTREGEX validation rules##to v7.12 release