chore(release): 7.12.0 [skip ci]

# [7.12.0](https://git.datacontroller.io/dc/dc/compare/v7.11.0...v7.12.0) (2026-07-28)

### Bug Fixes

* adding REGEX validations to mpe_x_test ([bd798b4](bd798b424a))
* default value for label ([a3e46a9](a3e46a968e))
* **deps:** retarget Angular upgrade to 20, not 21 (CI install was broken) ([cac9244](cac9244f92))
* ensure only one REGEX applies at a time ([8fb58eb](8fb58eb36e))
* ensure that no assets (including og links) ever fetch from external sources ([33dcb98](33dcb989d3))
* include peer dependencies in package-lock for npm ci in pipeline ([b51c770](b51c770782))
* licensecheker ([f60bcef](f60bcef583))
* **lint:** remove redundant optional chaining ([d881290](d881290618))
* optimisation, renamed values for DDTYPE to save space ([cfb60e5](cfb60e5e4b))
* patch npm audit vulnerabilities in sas and client dependencies ([e22edf7](e22edf7ed3))
* **query:** isolate viewbox filter state from the base table's ([7a35cf4](7a35cf4a45))
* regenerate client lockfile to resolve Angular peer-dependency drift breaking npm ci ([05fe474](05fe4744d5))
* **regex:** special missing handling ([180c247](180c2477ed))
* removing low severity warning in npm audit ([2a771bb](2a771bb91a))
* removing thousand seperator from plain numerics in EDIT mode ([0392a81](0392a81cbd))
* **validations:** parse SAS PRX /pattern/flags syntax in HARDREGEX/SOFTREGEX ([7ed3730](7ed3730ae3))

### Features

* **docs:** adding agents.md and docs for RLS ([359d833](359d833406))
* **editor:** add HARDREGEX/SOFTREGEX validation rules ([17e4802](17e4802895))
* **editor:** evaluate HARDREGEX/SOFTREGEX independently instead of hard-wins precedence ([57db117](57db1179a9))
* **editor:** show applied HARDREGEX/SOFTREGEX pattern in column info dropdown ([39c8855](39c8855f37))
* **regex:** backend validations on regex strings ([d2c93a4](d2c93a46fa))
* using ALL libraries as validation in MPE_SECURITY.  Closes [#279](#279) ([62ff0ae](62ff0aee4a))
* validation checks to prevent incompatible RLS rules (eg REPLACE load type).  Closes [#211](#211) ([ea00c5a](ea00c5afad))
* validation on RLS for REPLACE, + docs + tests.  Closes [#211](#211) ([7378f3b](7378f3ba30))

.agent/docs/dependency-updates.md

@@ -0,0 +1,19 @@
+# Dependency Updates Checklist
+
+Whenever any `package.json` (root, `client/`, or `sas/`) or lockfile is modified, run the same checks CI runs before pushing:
+
+1. **npm audit** (must be clean for prod deps):
+   ```bash
+   npm audit --omit=dev            # in repo root
+   cd sas && npm audit --omit=dev
+   cd ../client && npm audit --omit=dev
+   ```
+   Fix with `npm audit fix`, targeted `overrides` in `package.json`, or version bumps — never `npm audit fix --force` blindly, as it can introduce breaking changes.
+
+2. **License checker** (client only):
+   ```bash
+   cd client && npm run license-checker
+   ```
+   If a new dependency fails, either add its SPDX id to the `onlyAllow` list in `client/licenseChecker.js` (if the license is acceptable, e.g. permissive ones like `BlueOak-1.0.0`) or add the specific package to `excludePackages` with justification. Data Controller ships on-prem, so only OSI-approved permissive licenses are acceptable for production dependencies.
+
+Both checks run in `.gitea/workflows/build.yaml` (`Check audit` and `Licence checker` steps) and will fail the build if skipped locally.

.agent/docs/regex-validations.md

@@ -0,0 +1,86 @@
+# Regex Validations (HARDREGEX / SOFTREGEX)
+
+This document describes how the regex validation rules work internally. For user-facing documentation see `docs/dcc-validations.md` in the `docs.datacontroller.io` repo.
+
+## Overview
+
+Two validation rule types in `MPE_VALIDATIONS` validate cell values against a regular expression supplied in `RULE_VALUE`:
+
+- `HARDREGEX` - submission-blocking. A failing value is rejected by the cell validator and painted red (HOT's own `htInvalid` class), so the row cannot be submitted.
+- `SOFTREGEX` - display-only warning. A failing value is painted yellow (`dc-warning-cell` class) with a tooltip, but submission is not blocked.
+
+Both rule types are selectable in the MPE_VALIDATIONS RULE_TYPE dropdown; they were added to the selectbox seed data in `sas/sasjs/macros/mpe_makedata.sas` and via the optional migration `sas/sasjs/db/migrations/20260720_v7.12_release.sas`. `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.
+
+## Config-time validation (SAS side)
+
+`sas/sasjs/services/hooks/mpe_validations_postedit.sas` runs `prxparse()` on any staged HARDREGEX/SOFTREGEX rule value and aborts the edit with a list of offending `libref.table.column` references if the pattern is invalid. This is a best-effort syntax check to catch typos at config time; an empty pattern is treated as valid (it matches everything in JS). Rows marked for delete are skipped.
+
+Because patterns must pass `prxparse`, rule values are authored in the SAS PRX delimiter form `/pattern/flags` (although a bare pattern is also accepted for backwards compatibility).
+
+## Frontend evaluation
+
+All regex handling lives in the client; there is no server-side re-validation of data values.  The per-cell decision flow:
+
+```mermaid
+flowchart TD
+    A[Cell value] --> B{Row marked for delete<br/>and not a PK column?}
+    B -- Yes --> Z[No validation / no warning]
+    B -- No --> C{isRegexRuleExempt?<br/>blank, or "." on a numeric column}
+    C -- Yes --> Z
+    C -- No --> D{HARDREGEX rule on column?}
+    D -- Yes --> E{Pattern matches?}
+    E -- No --> F[Invalid: submission blocked,<br/>red htInvalid + REGEX tooltip]
+    E -- Yes --> J[Valid]
+    D -- No --> G{SOFTREGEX rule on column?}
+    G -- Yes --> H{Pattern matches?}
+    H -- No --> I[Warning: yellow dc-warning-cell<br/>+ REGEX tooltip, submission allowed]
+    H -- Yes --> J[Valid]
+    G -- No --> J
+```
+
+A malformed pattern never reaches this flow: it is treated as always-valid (HARDREGEX) / never-warn (SOFTREGEX) with a `console.warn`, rather than breaking the editor.
+
+### `client/src/app/shared/dc-validator/utils/parseRegexRule.ts`
+
+Converts an authored SAS PRX pattern into a JavaScript `RegExp`:
+
+1. If the value matches `/^\/(.*)\/([a-z]*)$/s`, the delimiters are stripped and the trailing flags are passed to `new RegExp(body, flags)`. Otherwise the value is used as-is (backwards compatibility with bare patterns).
+2. Three mechanical Perl→JS translations are applied to the body:
+   - a leading `(?i)` inline modifier is removed and folded into the `i` flag;
+   - `\Q...\E` literal sequences are replaced with escaped literal text;
+   - `\A` → `^` and `\z` → `(?![\s\S])` (end-of-string anchor).
+3. Perl-only constructs that would need a capture-group-renumbering rewrite (atomic groups `(?>...)`, possessive quantifiers `a++`) are deliberately NOT translated. They throw from `new RegExp`, and every caller treats a throw as "always valid / never warn" (see below) rather than breaking the editor.
+
+### `client/src/app/shared/dc-validator/utils/isRegexRuleExempt.ts`
+
+Blank values (`undefined`, `null`, `''`) are exempt from pattern matching on any column type - use a separate NOTNULL rule if populated values must also be enforced. On numeric columns the plain SAS missing (`.`) is also exempt; special missings (`.A`-`.Z`, `._`, bare letters) are NOT exempt anywhere - being deliberately set, they are real values the pattern must match (and on a character column even `.` is real text). `isSpecialMissing` from `@sasjs/utils` is deliberately not used: its optional-dot regex would exempt any single-letter character value ("d", "z") before the regex ever ran. The check takes an `isNumeric` flag, passed by all callers (the dq validator via `dqValidate(rules, value, colType === 'numeric')`, the warning renderer via a `makeRegexWarningRenderer` argument, and `failsSoftRegex` via the column's HOT type).
+
+### HARDREGEX - blocking validation
+
+`HARDREGEX` is implemented as a cell validator in `client/src/app/shared/dc-validator/validations/dq-validation.ts`. It returns `true` (valid) for exempt values and for patterns that fail to compile (with a `console.warn`), and otherwise returns `parseRegexRule(ruleValue).test(value.toString())`. Returning `false` makes HOT mark the cell invalid, block submission, and paint it red via its standard `htInvalid` styling.
+
+### SOFTREGEX - warning renderer
+
+`client/src/app/editor/utils/regex-warning-renderer.ts` builds a display-only Handsontable renderer (registered per-column by `DcValidator.setupRules` in `client/src/app/shared/dc-validator/dc-validator.ts`). It never returns false; it only:
+
+- adds a `REGEX: <pattern>` tooltip (`td.title`) when a rule fails;
+- adds the yellow `dc-warning-cell` class when only SOFTREGEX fails.
+
+`DcValidator.failsSoftRegex(col, value)` provides the same logic outside the grid (e.g. the edit-record screen).
+
+### Precedence: HARD and SOFT on the same column
+
+Only one regex ever runs per column. If a HARDREGEX rule exists, SOFTREGEX is ignored entirely - never compiled, never evaluated - regardless of whether individual cell values pass or fail the hard rule. A value failing HARDREGEX gets the red invalid styling (blocking submission) plus a `REGEX: <pattern>` tooltip; a SOFTREGEX-only column warns in yellow without blocking. This holds in both the renderer and `failsSoftRegex`.
+
+### Other behaviour
+
+- Rows marked for delete (`_____DELETE__THIS__RECORD_____ = 'Yes'`) are not warned/validated by the renderer (except primary key columns, which still validate).
+- A malformed pattern never breaks the editor: the dq validator treats it as always-valid and the renderer disables the warning, logging to the console instead.
+- The pattern is used as authored - it is NOT auto-anchored. Authors must include `^`/`$` to match the entire cell value.
+- Column info: `client/src/app/shared/utils/col-info-html.ts` shows the applied pattern in the column-info dropdown - the HARDREGEX pattern if one exists (it is the rule actually applied when both are present), otherwise the SOFTREGEX pattern, otherwise nothing.
+
+## Testing
+
+- Unit tests: `parseRegexRule.spec.ts`, `isRegexRuleExempt.spec.ts`, `dq-validation.spec.ts`, `dc-validator.spec.ts` (under `client/src/app/shared/dc-validator/`), `client/src/app/editor/utils/regex-warning-renderer.spec.ts`, `client/src/app/shared/utils/col-info-html.spec.ts`.
+- E2E: `client/cypress/e2e/editor.cy.ts`.
+- SAS side: `sas/sasjs/services/editors/stagedata.test.3.sas`, plus seed data in `mpe_makedata.sas`: HARDREGEX "SOME_CHAR must contain 'the' or 'data'" (`/the|data/i`), SOFTREGEX "SOME_CHAR should contain the letter 't'", SOFTREGEX on PRIMARY_KEY_FIELD (`/^\d+$/` - yellow if the key contains a decimal), and HARDREGEX on SOME_SHORTNUM (`/^(?![1-5](\.\d+)?$).*/` - values 1-5 blocked; generated data starts at 6 so demos aren't blocked accidentally).

.gitea/workflows/build.yaml

@@ -44,7 +44,7 @@ jobs:
           cd ./sas
           npm audit --omit=dev
           cd ../client
-          npm audit --audit-level=low --omit=dev
+          npm audit --omit=dev
 
       - name: Lint check
         run: npm run lint:check

.gitea/workflows/release.yaml

@@ -51,7 +51,7 @@ jobs:
           cd ./sas
           npm audit --omit=dev
           cd ../client
-          npm audit --audit-level=low --omit=dev
+          npm audit --omit=dev
 
       - name: Angular Tests
         run: |

AGENTS.md

@@ -1,5 +1,9 @@
 # Agent Instructions
 
+## Git
+
+Do NOT auto-commit. Never run `git commit` (or `git push`) unless the user explicitly asks. Leave changes in the working tree for the user to review and commit themselves.
+
 ## Markdown files
 
 Do NOT hard-wrap Markdown files (no fixed-width line wrapping / carriage returns inside paragraphs). Each paragraph, list item, and heading should be a single logical line, regardless of length. Let the editor/viewer soft-wrap. This includes AGENTS.md itself.
@@ -8,10 +12,21 @@ This is different from `.sas` files, where a maximum line length applies. The no
 
 Rationale: hard-wrapped prose produces noisy diffs when sentences are edited and reflowed, and Markdown renderers already handle wrapping.
 
-## SAS files
+## Linting (required before "done")
 
-After creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
+Never consider a change complete until the relevant linters pass on the files you touched — do not rely on the user's pre-commit hooks to catch it:
+
+- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
+- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
+
+## No external assets
+
+Data Controller must run entirely locally (offline / on-prem, no internet access). The built product must never fetch assets from remote servers — no external fonts, images, scripts, stylesheets, CDN links, or remote URLs in meta tags (e.g. `og:image`, `og:url`, `itemprop="image"`). All assets must be bundled and served locally.
 
 ## The .agent folder
 
 Agent-related content lives in `.agent/`: technical/agent-facing documentation goes in `.agent/docs/` (not `docs/`), and skills in `.agent/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agent/docs/`.
+
+## Code comments and test names
+
+Never reference items that are not active parts of the repository — no "the original bug", "regression from this fix", "this session/PR/commit", or similar ephemeral context. Comments and test names must be self-contained: describe the behaviour being asserted, not the history of how it was discovered. The one exception is a literal link to a ticket/issue tracker.

CHANGELOG.md

@@ -1,3 +1,37 @@
+# [7.12.0](https://git.datacontroller.io/dc/dc/compare/v7.11.0...v7.12.0) (2026-07-28)
+
+
+### Bug Fixes
+
+* adding REGEX validations to mpe_x_test ([bd798b4](https://git.datacontroller.io/dc/dc/commit/bd798b424a7c4b90d613425348d3f99eeb49b025))
+* default value for label ([a3e46a9](https://git.datacontroller.io/dc/dc/commit/a3e46a968ef08295347b230235f1e79dfa51c5a2))
+* **deps:** retarget Angular upgrade to 20, not 21 (CI install was broken) ([cac9244](https://git.datacontroller.io/dc/dc/commit/cac9244f92544f432078125c312d05c36ed4b855))
+* ensure only one REGEX applies at a time ([8fb58eb](https://git.datacontroller.io/dc/dc/commit/8fb58eb36e05fdf7262459b05e53643b3ca672f7))
+* ensure that no assets (including og links) ever fetch from external sources ([33dcb98](https://git.datacontroller.io/dc/dc/commit/33dcb989d3ae0c5ee0a13fe64195862a945c4348))
+* include peer dependencies in package-lock for npm ci in pipeline ([b51c770](https://git.datacontroller.io/dc/dc/commit/b51c770782a8b62816e5145496dbea9fb83059e0))
+* licensecheker ([f60bcef](https://git.datacontroller.io/dc/dc/commit/f60bcef58381ba269415640fa19a27305274939a))
+* **lint:** remove redundant optional chaining ([d881290](https://git.datacontroller.io/dc/dc/commit/d881290618f25ddb6335125f1a08b5aaf56a8c48))
+* optimisation, renamed values for DDTYPE to save space ([cfb60e5](https://git.datacontroller.io/dc/dc/commit/cfb60e5e4bfd37dc79fa4e9a6c6649ab1a61f2e4))
+* patch npm audit vulnerabilities in sas and client dependencies ([e22edf7](https://git.datacontroller.io/dc/dc/commit/e22edf7ed3aa54fc08411cf90d51df9ec30877c3))
+* **query:** isolate viewbox filter state from the base table's ([7a35cf4](https://git.datacontroller.io/dc/dc/commit/7a35cf4a458791b26ec8141c0c84d97fcb555145))
+* regenerate client lockfile to resolve Angular peer-dependency drift breaking npm ci ([05fe474](https://git.datacontroller.io/dc/dc/commit/05fe4744d58e7985ecc5ef5f9c1969dbab2d2efb))
+* **regex:** special missing handling ([180c247](https://git.datacontroller.io/dc/dc/commit/180c2477ed9ae44b4e479e81ec642f40580bcbc8))
+* removing low severity warning in npm audit ([2a771bb](https://git.datacontroller.io/dc/dc/commit/2a771bb91acaaa16aba44e2d7ddb51b8b7123c10))
+* removing thousand seperator from plain numerics in EDIT mode ([0392a81](https://git.datacontroller.io/dc/dc/commit/0392a81cbd9a634590ca7a60a316cd12c690a5a5))
+* **validations:** parse SAS PRX /pattern/flags syntax in HARDREGEX/SOFTREGEX ([7ed3730](https://git.datacontroller.io/dc/dc/commit/7ed3730ae3145dffd639ce3d2f831447d78f4e85))
+
+
+### Features
+
+* **docs:** adding agents.md and docs for RLS ([359d833](https://git.datacontroller.io/dc/dc/commit/359d833406ace79f96d03c08a5c72fbd3df29442))
+* **editor:** add HARDREGEX/SOFTREGEX validation rules ([17e4802](https://git.datacontroller.io/dc/dc/commit/17e48028955d7b091e53bad9d7f9e3a089b53af9))
+* **editor:** evaluate HARDREGEX/SOFTREGEX independently instead of hard-wins precedence ([57db117](https://git.datacontroller.io/dc/dc/commit/57db1179a97973ecf0f711bda6057383de546dd3))
+* **editor:** show applied HARDREGEX/SOFTREGEX pattern in column info dropdown ([39c8855](https://git.datacontroller.io/dc/dc/commit/39c8855f37477f06d8b8cf5c97e04543070e8385))
+* **regex:** backend validations on regex strings ([d2c93a4](https://git.datacontroller.io/dc/dc/commit/d2c93a46facb386fd31661c02c0d3d34f38f6e43))
+* using ALL libraries as validation in MPE_SECURITY.  Closes [#279](https://git.datacontroller.io/dc/dc/issues/279) ([62ff0ae](https://git.datacontroller.io/dc/dc/commit/62ff0aee4a976184de65d87ea3c8b8b7cb777938))
+* validation checks to prevent incompatible RLS rules (eg REPLACE load type).  Closes [#211](https://git.datacontroller.io/dc/dc/issues/211) ([ea00c5a](https://git.datacontroller.io/dc/dc/commit/ea00c5afad0daf2a66cdb206f2a7481b992192bf))
+* validation on RLS for REPLACE, + docs + tests.  Closes [#211](https://git.datacontroller.io/dc/dc/issues/211) ([7378f3b](https://git.datacontroller.io/dc/dc/commit/7378f3ba3014141008b4dbde0b89c21a6ce02f69))
+
 # [7.11.0](https://git.datacontroller.io/dc/dc/compare/v7.10.1...v7.11.0) (2026-07-20)
 
 

client/cypress/e2e/editor.cy.ts

@@ -212,6 +212,12 @@ context('editor tests: ', function () {
               .clear()
               .type('not-an-email{enter}')
               .then(() => {
+                getCellByHeaderAndRow(1, 'REGEX_HARD_COL').should(
+                  'have.attr',
+                  'title',
+                  'REGEX: /[\\w.]+@[\\w]+\\.[a-z]{2,}/'
+                )
+
                 submitTable(() => {
                   cy.get('.modal-body').then((modalBody: any) => {
                     if (
@@ -245,10 +251,13 @@ context('editor tests: ', function () {
               .clear()
               .type('not a postcode{enter}')
               .then(() => {
-                getCellByHeaderAndRow(1, 'REGEX_SOFT_COL').should(
+                getCellByHeaderAndRow(1, 'REGEX_SOFT_COL')
-                  'have.class',
+                  .should('have.class', 'dc-warning-cell')
-                  'dc-warning-cell'
+                  .and(
-                )
+                    'have.attr',
+                    'title',
+                    'REGEX: /[A-Z]{1,2}\\d{1,2}[A-Z]?\\s?\\d[A-Z]{2}/'
+                  )
 
                 submitTable(() => {
                   // Validation passed despite the SOFTREGEX warning: the
@@ -264,6 +273,126 @@ context('editor tests: ', function () {
       })
     })
   })
+
+  it('9 | Info dropdown shows the applied HARDREGEX/SOFTREGEX pattern', () => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        scrollGridRight()
+
+        openColumnDropdown('REGEX_HARD_COL')
+        cy.get('.htDropdownMenu').should(($menu) => {
+          expect($menu.text()).to.include(
+            'HARDREGEX: /[\\w.]+@[\\w]+\\.[a-z]{2,}/'
+          )
+        })
+        cy.get('body').click(0, 0) // close menu
+
+        openColumnDropdown('REGEX_SOFT_COL')
+        cy.get('.htDropdownMenu').should(($menu) => {
+          expect($menu.text()).to.include(
+            'SOFTREGEX: /[A-Z]{1,2}\\d{1,2}[A-Z]?\\s?\\d[A-Z]{2}/'
+          )
+        })
+      })
+    })
+  })
+
+  it('10 | Info dropdown shows only the applied HARDREGEX pattern when a column has both', () => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        scrollGridRight()
+
+        openColumnDropdown('REGEX_BOTH_COL')
+        cy.get('.htDropdownMenu').should(($menu) => {
+          const text = $menu.text()
+
+          expect(text).to.include('HARDREGEX: /^[A-Z0-9_-]+$/')
+          expect(text).to.not.include('SOFTREGEX: /^.{5,10}$/')
+        })
+      })
+    })
+  })
+
+  it('11 | REGEX_BOTH_COL: a HARDREGEX failure blocks submission and sets its own tooltip, not yellow', (done) => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        scrollGridRight()
+
+        // 'bad value' fails HARDREGEX (lowercase + space) but is 9 chars,
+        // within SOFTREGEX's 5-10 range - isolates the hard-only failure.
+        getCellByHeaderAndRow(1, 'REGEX_BOTH_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused()
+              .clear()
+              .type('bad value{enter}')
+              .then(() => {
+                getCellByHeaderAndRow(1, 'REGEX_BOTH_COL')
+                  .should('not.have.class', 'dc-warning-cell')
+                  .and('have.attr', 'title', 'REGEX: /^[A-Z0-9_-]+$/')
+
+                submitTable(() => {
+                  cy.get('.modal-body').then((modalBody: any) => {
+                    if (
+                      modalBody[0].innerHTML
+                        .toLowerCase()
+                        .includes(`invalid values are present`)
+                    ) {
+                      done()
+                    }
+                  })
+                })
+              })
+          })
+      })
+    })
+  })
+
+  it('12 | REGEX_BOTH_COL: SOFTREGEX is ignored entirely when HARDREGEX is present', (done) => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        scrollGridRight()
+
+        // 'AB' passes HARDREGEX (uppercase only) but fails SOFTREGEX (too
+        // short) - only one regex runs per column, so no warning is shown.
+        getCellByHeaderAndRow(1, 'REGEX_BOTH_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused()
+              .clear()
+              .type('AB{enter}')
+              .then(() => {
+                getCellByHeaderAndRow(1, 'REGEX_BOTH_COL')
+                  .should('not.have.class', 'dc-warning-cell')
+                  .and('not.have.attr', 'title')
+
+                submitTable(() => {
+                  cy.get('#submitBtn', { timeout: longerCommandTimeout })
+                    .should('exist')
+                    .should('not.be.disabled')
+                    .then(() => done())
+                })
+              })
+          })
+      })
+    })
+  })
 })
 
 // Handsontable virtualizes columns — with 17 columns on MPE_X_NEW, only
@@ -280,6 +409,24 @@ const scrollGridRight = () => {
     .scrollTo('right')
 }
 
+// Opens a column header's dropdown menu to reach its `info` item, which
+// has a custom renderer showing NAME/LABEL/TYPE/LENGTH/FORMAT and, when the
+// column has a HARDREGEX/SOFTREGEX rule, the applied pattern. Clicking the
+// header text first selects the column - the renderer reads
+// hot.getSelected() to decide which column to describe.
+const openColumnDropdown = (headerText: string) => {
+  cy.get('#hotTable .ht_clone_top .htCore thead button.changeType', {
+    timeout: longerCommandTimeout
+  })
+    .parents('th')
+    .filter((_, th) => Cypress.$(th).text().includes(headerText))
+    .last()
+    .as('targetHeader')
+
+  cy.get('@targetHeader').click()
+  cy.get('@targetHeader').find('button.changeType').click({ force: true })
+}
+
 // Locates a body cell by its column's header text rather than a hardcoded
 // childNodes index. Handsontable's hiddenColumns plugin (e.g. HIDDEN_COL,
 // used by several demo columns ahead of REGEX_HARD_COL/REGEX_SOFT_COL in

client/licenseChecker.js

@@ -8,7 +8,7 @@ const check = (cwd) => {
         start: cwd,
         excludePrivatePackages: true,
         onlyAllow:
-          'AFLv2.1;Apache 2.0;Apache-2.0;Apache*;Artistic-2.0;0BSD;BSD*;BSD-2-Clause;BSD-3-Clause;CC0-1.0;CC-BY-3.0;CC-BY-4.0;ISC;MIT;MPL-2.0;ODC-By-1.0;Python-2.0;Unlicense;',
+          'AFLv2.1;Apache 2.0;Apache-2.0;Apache*;Artistic-2.0;BlueOak-1.0.0;0BSD;BSD*;BSD-2-Clause;BSD-3-Clause;CC0-1.0;CC-BY-3.0;CC-BY-4.0;ISC;MIT;MPL-2.0;ODC-By-1.0;Python-2.0;Unlicense;',
         excludePackages:
           '@cds/city@1.1.0;@handsontable/angular-wrapper@16.0.1;@handsontable/angular-wrapper@17.1.0;@handsontable/angular-wrapper@18.0.0;handsontable@^16.0.1;handsontable@16.2.0;handsontable@17.1.0;handsontable@18.0.0;hyperformula@2.7.1;hyperformula@3.0.0;hyperformula@3.1.0;hyperformula@3.2.0;hyperformula@3.3.0;jackspeak@3.4.3;path-scurry@1.11.1;package-json-from-dist@1.0.1;buffers@0.1.1'
       },

client/package.json

@@ -61,8 +61,10 @@
     "@types/d3-graphviz": "^2.6.7",
     "@types/text-encoding": "0.0.35",
     "base64-arraybuffer": "^0.2.0",
+    "browserify-cipher": "^1.0.1",
     "buffer": "^5.4.3",
-    "crypto-browserify": "^3.12.1",
+    "create-hash": "^1.2.0",
+    "create-hmac": "^1.1.7",
     "crypto-js": "^4.2.0",
     "d3-graphviz": "^5.0.2",
     "exceljs": "^4.4.0",
@@ -78,6 +80,7 @@
     "ngx-clipboard": "^16.0.0",
     "ngx-json-viewer": "file:libraries/ngx-json-viewer-3.2.1.tgz",
     "os-browserify": "0.3.0",
+    "randombytes": "^2.1.0",
     "rxjs": "^7.8.0",
     "save-svg-as-png": "^1.4.17",
     "stream-browserify": "3.0.0",
@@ -103,12 +106,15 @@
     "@cypress/webpack-preprocessor": "^5.17.1",
     "@lhci/cli": "^0.15.1",
     "@types/core-js": "^2.5.5",
+    "@types/create-hash": "^1.2.6",
+    "@types/create-hmac": "^1.1.3",
     "@types/crypto-js": "^4.2.1",
     "@types/es6-shim": "^0.31.39",
     "@types/jasmine": "~5.1.4",
     "@types/lodash-es": "^4.17.3",
     "@types/marked": "^4.3.0",
     "@types/node": "12.20.50",
+    "@types/randombytes": "^2.0.3",
     "@typescript-eslint/eslint-plugin": "8.65.0",
     "@typescript-eslint/parser": "8.65.0",
     "core-js": "^2.5.4",
@@ -141,6 +147,10 @@
   "overrides": {
     "ajv": "8.18.0",
     "uuid": "11.1.1",
-    "lighthouse": "13.4.0"
+    "lighthouse": "13.4.0",
+    "exceljs": {
+      "archiver": "^8.0.0",
+      "unzipper": "^0.12.5"
+    }
   }
-}
+}

client/src/app/editor/editor.component.ts

@@ -3254,7 +3254,15 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
                   colName = this.hotInstance?.colToProp(selectedCol) as string
                   colInfo = this.$dataFormats?.vars[colName]
 
-                  textInfo = buildColInfoHtml(colName, colInfo)
+                  const { hardRegexValue, softRegexValue } =
+                    this.dcValidator?.getRegexRuleValues(colName) || {}
+
+                  textInfo = buildColInfoHtml(
+                    colName,
+                    colInfo,
+                    hardRegexValue,
+                    softRegexValue
+                  )
                 }
 
                 elem.innerHTML = textInfo

client/src/app/editor/utils/regex-warning-renderer.spec.ts

@@ -19,13 +19,34 @@ describe('makeRegexWarningRenderer', () => {
     return { hot, container }
   }
 
-  it('adds dc-warning-cell when the value fails the pattern', () => {
+  const buildHotNumeric = (data: any[]) => {
+    const container = document.createElement('div')
+    document.body.appendChild(container)
+
+    const hot = new Handsontable(container, {
+      data,
+      columns: [
+        {
+          data: 'val',
+          renderer: makeRegexWarningRenderer('^[A-Z]{3}$', undefined, true)
+        },
+        { data: '_____DELETE__THIS__RECORD_____' }
+      ],
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    return { hot, container }
+  }
+
+  it('adds dc-warning-cell and a REGEX: title when the value fails the pattern', () => {
     const { hot, container } = buildHot([
       { val: 'abc', _____DELETE__THIS__RECORD_____: 'No' }
     ])
 
     const td = hot.getCell(0, 0)
     expect(td?.classList.contains('dc-warning-cell')).toBeTrue()
+    expect(td?.title).toEqual('REGEX: ^[A-Z]{3}$')
 
     hot.destroy()
     container.remove()
@@ -55,9 +76,9 @@ describe('makeRegexWarningRenderer', () => {
     container.remove()
   })
 
-  it('does not add dc-warning-cell for a SAS special missing value', () => {
+  it('does not add dc-warning-cell for the plain SAS missing (".") on a numeric column', () => {
-    const { hot, container } = buildHot([
+    const { hot, container } = buildHotNumeric([
-      { val: '.a', _____DELETE__THIS__RECORD_____: 'No' }
+      { val: '.', _____DELETE__THIS__RECORD_____: 'No' }
     ])
 
     const td = hot.getCell(0, 0)
@@ -67,6 +88,18 @@ describe('makeRegexWarningRenderer', () => {
     container.remove()
   })
 
+  it('DOES add dc-warning-cell for a special missing on a numeric column (a deliberately-set value)', () => {
+    const { hot, container } = buildHotNumeric([
+      { val: '.a', _____DELETE__THIS__RECORD_____: 'No' }
+    ])
+
+    const td = hot.getCell(0, 0)
+    expect(td?.classList.contains('dc-warning-cell')).toBeTrue()
+
+    hot.destroy()
+    container.remove()
+  })
+
   it('does not throw and never warns on a malformed pattern', () => {
     const container = document.createElement('div')
     document.body.appendChild(container)
@@ -128,4 +161,154 @@ describe('makeRegexWarningRenderer', () => {
     hot.destroy()
     container.remove()
   })
+
+  describe('HARDREGEX (second, optional pattern)', () => {
+    // HARDREGEX already blocks submission via dqValidate/HOT's own
+    // htInvalid (unchanged, untouched here) - this renderer only adds the
+    // matching 'REGEX: <pattern>' title on top, so a column with only a
+    // HARDREGEX rule (no SOFTREGEX at all) still tells the user why a cell
+    // is red, not just that it is.
+    const buildHardOnlyHot = (value: string) => {
+      const container = document.createElement('div')
+      document.body.appendChild(container)
+
+      const hot = new Handsontable(container, {
+        data: [{ val: value, _____DELETE__THIS__RECORD_____: 'No' }],
+        columns: [
+          {
+            data: 'val',
+            renderer: makeRegexWarningRenderer(undefined, '^[A-Z]{3}$')
+          },
+          { data: '_____DELETE__THIS__RECORD_____' }
+        ],
+        licenseKey: 'non-commercial-and-evaluation'
+      })
+      hot.render()
+
+      return { hot, container }
+    }
+
+    it('sets a REGEX: title (no dc-warning-cell) when only HARDREGEX fails', () => {
+      const { hot, container } = buildHardOnlyHot('abc')
+
+      const td = hot.getCell(0, 0)
+      expect(td?.classList.contains('dc-warning-cell')).toBeFalse()
+      expect(td?.title).toEqual('REGEX: ^[A-Z]{3}$')
+
+      hot.destroy()
+      container.remove()
+    })
+
+    it('sets no title when the value passes HARDREGEX', () => {
+      const { hot, container } = buildHardOnlyHot('ABC')
+
+      const td = hot.getCell(0, 0)
+      expect(td?.title).toEqual('')
+
+      hot.destroy()
+      container.remove()
+    })
+
+    it('suppresses the HARDREGEX title on a row marked for delete', () => {
+      const container = document.createElement('div')
+      document.body.appendChild(container)
+
+      const hot = new Handsontable(container, {
+        data: [{ val: 'abc', _____DELETE__THIS__RECORD_____: 'Yes' }],
+        columns: [
+          {
+            data: 'val',
+            renderer: makeRegexWarningRenderer(undefined, '^[A-Z]{3}$')
+          },
+          { data: '_____DELETE__THIS__RECORD_____' }
+        ],
+        licenseKey: 'non-commercial-and-evaluation'
+      })
+      hot.render()
+
+      expect(hot.getCell(0, 0)?.title).toEqual('')
+
+      hot.destroy()
+      container.remove()
+    })
+
+    it('prefers the HARDREGEX title over SOFTREGEX when a value fails both', () => {
+      const container = document.createElement('div')
+      document.body.appendChild(container)
+
+      const hot = new Handsontable(container, {
+        data: [{ val: 'ab', _____DELETE__THIS__RECORD_____: 'No' }],
+        columns: [
+          {
+            data: 'val',
+            renderer: makeRegexWarningRenderer('^.{5,10}$', '^[A-Z]{3}$')
+          },
+          { data: '_____DELETE__THIS__RECORD_____' }
+        ],
+        licenseKey: 'non-commercial-and-evaluation'
+      })
+      hot.render()
+
+      const td = hot.getCell(0, 0)
+      // 'ab' fails HARDREGEX (not 3 uppercase letters) AND SOFTREGEX (too
+      // short) - HARDREGEX wins: its title shows, and no yellow class is
+      // added (HOT's own red htInvalid governs this cell instead).
+      expect(td?.classList.contains('dc-warning-cell')).toBeFalse()
+      expect(td?.title).toEqual('REGEX: ^[A-Z]{3}$')
+
+      hot.destroy()
+      container.remove()
+    })
+
+    it('never warns for SOFTREGEX when the column also has HARDREGEX, even for a value that passes HARDREGEX', () => {
+      const container = document.createElement('div')
+      document.body.appendChild(container)
+
+      const hot = new Handsontable(container, {
+        data: [{ val: 'AB', _____DELETE__THIS__RECORD_____: 'No' }],
+        columns: [
+          {
+            data: 'val',
+            renderer: makeRegexWarningRenderer('^.{5,10}$', '^[A-Z0-9]+$')
+          },
+          { data: '_____DELETE__THIS__RECORD_____' }
+        ],
+        licenseKey: 'non-commercial-and-evaluation'
+      })
+      hot.render()
+
+      const td = hot.getCell(0, 0)
+      // 'AB' passes HARDREGEX (uppercase/digits) but fails SOFTREGEX (too
+      // short) - only one regex runs per column, so the soft rule is
+      // ignored entirely: no warning, no title.
+      expect(td?.classList.contains('dc-warning-cell')).toBeFalse()
+      expect(td?.title).toEqual('')
+
+      hot.destroy()
+      container.remove()
+    })
+
+    it('does not throw and never warns on a malformed HARDREGEX pattern', () => {
+      const container = document.createElement('div')
+      document.body.appendChild(container)
+
+      const hot = new Handsontable(container, {
+        data: [{ val: 'abc', _____DELETE__THIS__RECORD_____: 'No' }],
+        columns: [
+          {
+            data: 'val',
+            renderer: makeRegexWarningRenderer(undefined, '[unterminated')
+          },
+          { data: '_____DELETE__THIS__RECORD_____' }
+        ],
+        licenseKey: 'non-commercial-and-evaluation'
+      })
+
+      expect(() => hot.render()).not.toThrow()
+      expect(hot.getCell(0, 0)?.title).toEqual('')
+
+      hot.destroy()
+      container.remove()
+    })
+  })
 })

client/src/app/editor/utils/regex-warning-renderer.ts

@@ -2,28 +2,46 @@ import Handsontable from 'handsontable'
 import { isRegexRuleExempt } from '../../shared/dc-validator/utils/isRegexRuleExempt'
 import { parseRegexRule } from '../../shared/dc-validator/utils/parseRegexRule'
 
+const compileRegex = (
+  pattern: string | undefined,
+  ruleType: 'SOFTREGEX' | 'HARDREGEX'
+): RegExp | null => {
+  try {
+    return pattern ? parseRegexRule(pattern) : null
+  } catch (e) {
+    console.warn(`${ruleType} - invalid pattern, warning disabled: ${pattern}`)
+    return null
+  }
+}
+
 /**
- * Builds a display-only HOT renderer for SOFTREGEX: a value that fails the
+ * Builds a display-only HOT renderer for HARDREGEX/SOFTREGEX: neither ever
- * pattern gets a yellow `dc-warning-cell` class (styles.scss) instead of
+ * returns false from the cell validator here (that's dqValidate's job for
- * blocking submission — SOFTREGEX must never return false from the cell
+ * HARDREGEX, which blocks submission and paints HOT's own red htInvalid
- * validator (that would paint htInvalid and block submit), so the warning
+ * independently of this renderer) - this renderer only adds the matching
- * lives purely at the render layer, same split as makeNumberFormatRenderer.
+ * `REGEX: <pattern>` title, plus a yellow `dc-warning-cell` class when only
+ * SOFTREGEX fails, same split as makeNumberFormatRenderer.
+ *
+ * Only one regex ever runs per column: when a HARDREGEX rule exists,
+ * SOFTREGEX is ignored entirely (never compiled, never evaluated) -
+ * regardless of whether individual cell values pass or fail the hard
+ * rule.  A value failing HARDREGEX gets its title (no yellow class - red
+ * htInvalid already covers the color).  This mirrors
+ * DcValidator.failsSoftRegex's own precedence.
  *
  * Suppressed on rows marked for delete (_____DELETE__THIS__RECORD_____ =
  * 'Yes') — a warning about data about to be removed is just noise.
  *
- * Falls back to no warning ever showing when the pattern itself is
+ * Falls back to no warning ever showing when a pattern itself is malformed,
- * malformed, rather than breaking the cell.
+ * rather than breaking the cell.
  */
-export const makeRegexWarningRenderer = (pattern?: string) => {
+export const makeRegexWarningRenderer = (
-  let regex: RegExp | null = null
+  softPattern?: string,
-
+  hardPattern?: string,
-  try {
+  isNumeric: boolean = false
-    if (pattern) regex = parseRegexRule(pattern)
+) => {
-  } catch (e) {
+  const hardRegex = compileRegex(hardPattern, 'HARDREGEX')
-    console.warn(`SOFTREGEX - invalid pattern, warning disabled: ${pattern}`)
+  const softRegex = hardRegex ? null : compileRegex(softPattern, 'SOFTREGEX')
-    regex = null
-  }
 
   const baseRenderer = Handsontable.renderers.getRenderer('text')
 
@@ -40,13 +58,20 @@ export const makeRegexWarningRenderer = (pattern?: string) => {
 
     const markedForDelete =
       instance.getDataAtRowProp(row, '_____DELETE__THIS__RECORD_____') === 'Yes'
+    const exempt = isRegexRuleExempt(value, isNumeric)
 
-    const failsPattern =
+    const failsHard =
-      !!regex && !isRegexRuleExempt(value) && !regex.test(value.toString())
+      !!hardRegex && !exempt && !hardRegex.test(value.toString())
+    const failsSoft =
+      !!softRegex && !exempt && !softRegex.test(value.toString())
 
-    if (failsPattern && !markedForDelete) {
+    if (markedForDelete) return td
+
+    if (failsHard) {
+      td.title = `REGEX: ${hardPattern}`
+    } else if (failsSoft) {
       td.classList.add('dc-warning-cell')
-      td.title = 'Value does not match the expected pattern'
+      td.title = `REGEX: ${softPattern}`
     }
 
     return td

client/src/app/models/TableData.ts

@@ -14,7 +14,8 @@ export interface ColumnDetail {
   LENGTH: number
   NAME: string
   TYPE: string
-  VARNUM: number
+  // No longer sent by viewdata.sas — optional for legacy responses.
+  VARNUM?: number
 }
 
 export interface Approver {

client/src/app/services/va-filter.service.spec.ts

@@ -55,8 +55,8 @@ describe('VaFilterService', () => {
     'SOME_DATETIME'
   ]
   const cols = [
-    { NAME: 'SOME_CHAR', DDTYPE: 'CHARACTER' },
+    { NAME: 'SOME_CHAR', DDTYPE: 'C' },
-    { NAME: 'SOME_NUM', DDTYPE: 'NUMERIC' },
+    { NAME: 'SOME_NUM', DDTYPE: 'N' },
     { NAME: 'SOME_TIME', DDTYPE: 'TIME' },
     { NAME: 'SOME_DATE', DDTYPE: 'DATE' },
     { NAME: 'SOME_DATETIME', DDTYPE: 'DATETIME' }

client/src/app/services/va-filter.service.ts

@@ -163,14 +163,20 @@ export class VaFilterService {
 
   /**
    * SAS data-type kind of a column spec. DDTYPE carries
-   * TIME/DATE/DATETIME/NUMERIC/CHARACTER; checked DATETIME-before-DATE (substring).
+   * TIME/DATE/DATETIME in full, with C/N for CHARACTER/NUMERIC (legacy
+   * responses may still carry the full strings); checked DATETIME-before-DATE
+   * (substring).
    */
   private columnKind(col: any): VaColumnKind {
     const ddtype = (col?.DDTYPE ?? '').toString().toUpperCase()
     if (ddtype.includes('DATETIME')) return 'datetime'
     if (ddtype.includes('DATE')) return 'date'
     if (ddtype.includes('TIME')) return 'time'
-    if (ddtype.includes('NUMERIC') || (col?.TYPE ?? '') === 'num') {
+    if (
+      ddtype === 'N' ||
+      ddtype.includes('NUMERIC') ||
+      (col?.TYPE ?? '') === 'num'
+    ) {
       return 'numeric'
     }
     return 'char'

client/src/app/shared/dc-validator/dc-validator.ts

@@ -204,6 +204,31 @@ export class DcValidator {
     return isNaN(digits) ? undefined : digits
   }
 
+  /**
+   * Returns the RULE_VALUEs of a HARDREGEX/SOFTREGEX rule on the given
+   * column, for display in the column-header info dropdown. Both are
+   * returned raw - the caller (buildColInfoHtml) decides which one is
+   * actually applied (HARDREGEX wins when both exist).
+   *
+   * @param col column name
+   */
+  getRegexRuleValues(col: string): {
+    hardRegexValue: string | undefined
+    softRegexValue: string | undefined
+  } {
+    const hardRegexRule = this.dqrules.find(
+      (rule: DQRule) => rule.BASE_COL === col && rule.RULE_TYPE === 'HARDREGEX'
+    )
+    const softRegexRule = this.dqrules.find(
+      (rule: DQRule) => rule.BASE_COL === col && rule.RULE_TYPE === 'SOFTREGEX'
+    )
+
+    return {
+      hardRegexValue: hardRegexRule?.RULE_VALUE,
+      softRegexValue: softRegexRule?.RULE_VALUE
+    }
+  }
+
   /**
    * Retrieves dropdown source for given dc validation rule
    * The values comes from MPE_SELECTBOX table
@@ -247,14 +272,23 @@ export class DcValidator {
    * exists. SOFTREGEX never goes through dqValidate/the cell validator (see
    * setupValidations — it's a display-only grid renderer instead), so the
    * edit-record modal, which has no grid renderer to hook into, uses this
-   * directly to show the same warning outside the grid. HARDREGEX takes
+   * directly to show the same warning outside the grid.
-   * precedence when both rules apply to the column, matching the grid's own
+   *
-   * behaviour (setupValidations skips wiring the SOFTREGEX renderer there
+   * A column can carry both HARDREGEX and SOFTREGEX at once, but only one
-   * too) — a dual-rule column should render red/blocked, never yellow.
+   * regex ever runs per column: if a HARDREGEX rule exists, SOFTREGEX is
+   * ignored entirely - the cell is already governed by the blocking rule,
+   * so a yellow warning on top would be redundant, even for values that
+   * pass the hard rule.  Same precedence as makeRegexWarningRenderer.
    */
   failsSoftRegex(col: string, value: any): boolean {
-    if (this.hasDqRules(col, ['HARDREGEX'])) return false
+    const isNumeric =
-    if (isRegexRuleExempt(value)) return false
+      this.rules.find((rule) => rule.data === col)?.type === 'numeric'
+    if (isRegexRuleExempt(value, isNumeric)) return false
+
+    const hardRegexRule = this.dqrules.find(
+      (rule: DQRule) => rule.BASE_COL === col && rule.RULE_TYPE === 'HARDREGEX'
+    )
+    if (hardRegexRule) return false
 
     const softRegexRule = this.dqrules.find(
       (rule: DQRule) => rule.BASE_COL === col && rule.RULE_TYPE === 'SOFTREGEX'
@@ -425,25 +459,21 @@ export class DcValidator {
           this.rules[i].numericFormat = undefined
         }
 
-        // SOFTREGEX: display-only warning (yellow), never blocks submission —
+        // HARDREGEX/SOFTREGEX: submission-blocking for HARDREGEX still goes
-        // unlike HARDREGEX (see dq-validation.ts), which goes through the
+        // through the normal validator/dqValidate path (see dq-validation.ts)
-        // normal validator/dqValidate path instead. Last-wins against
+        // and is unaffected by this renderer. This only wires the display
-        // NUMBER_FORMAT if a column somehow carried both (not expected in
+        // layer - a 'REGEX: <pattern>' title, plus a yellow dc-warning-cell
-        // practice — one formats numbers, the other pattern-matches text).
+        // when only SOFTREGEX fails (never for HARDREGEX, which relies on
-        //
+        // HOT's own red htInvalid instead). Last-wins against NUMBER_FORMAT
-        // HARDREGEX takes precedence when both apply to the same column: skip
+        // if a column somehow carried both (not expected in practice — one
-        // wiring this renderer entirely, so a failing value stays HOT's own
+        // formats numbers, the other pattern-matches text).
-        // red htInvalid (from HARDREGEX/dqValidate) rather than being
+        if (this.hasDqRules(ruleColName, ['HARDREGEX', 'SOFTREGEX'])) {
-        // visually overridden by this renderer's yellow dc-warning-cell.
+          const { hardRegexValue, softRegexValue } =
-        if (
+            this.getRegexRuleValues(ruleColName)
-          this.hasDqRules(ruleColName, ['SOFTREGEX']) &&
-          !this.hasDqRules(ruleColName, ['HARDREGEX'])
-        ) {
-          const softRegexRule = this.getDqDetails(ruleColName).find(
-            (rule: DQRule) => rule.RULE_TYPE === 'SOFTREGEX'
-          )
           this.rules[i].renderer = makeRegexWarningRenderer(
-            softRegexRule?.RULE_VALUE
+            softRegexValue,
+            hardRegexValue,
+            this.rules[i].type === 'numeric'
           )
         }
       }
@@ -532,7 +562,11 @@ export class DcValidator {
       }
 
       if (self.isDqCol(col || '')) {
-        const dqValid = dqValidate(self.getDqDetails(col || ''), value)
+        const dqValid = dqValidate(
+          self.getDqDetails(col || ''),
+          value,
+          colType === 'numeric'
+        )
 
         if (!dqValid) {
           console.warn(`DQ Validation - invalid (Value: ${value})`)

client/src/app/shared/dc-validator/models/col.model.ts

@@ -1,6 +1,8 @@
 export interface Col {
   NAME: string
-  VARNUM: number
+  // No longer sent by getdata.sas (dropped to trim the COLS payload) —
+  // column order in the array is authoritative. Optional for legacy responses.
+  VARNUM?: number
   LABEL: string
   FMTNAME: string
   DDTYPE: string

client/src/app/shared/dc-validator/tests/dc-validator.spec.ts

@@ -555,7 +555,32 @@ describe('DC Validator', () => {
     })
   })
 
-  it('11 | wires a function renderer for a SOFTREGEX rule, without blocking submission', () => {
+  it('11 | wires a function renderer for a HARDREGEX-only rule too (for the REGEX: tooltip)', () => {
+    // HARDREGEX blocking itself is covered by test 10 above - this isolates
+    // the newer addition: even with no SOFTREGEX at all, a renderer must
+    // still be wired so a failing cell gets a 'REGEX: <pattern>' title on
+    // top of HOT's own red htInvalid, not just silence.
+    const dcValidator: DcValidator = new DcValidator(
+      example_sasparams,
+      example_dataformats,
+      example_cols,
+      [
+        ...example_dqRules,
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'HARDREGEX',
+          RULE_VALUE: '^[A-Z]+$',
+          X: 0
+        }
+      ],
+      example_dqData
+    )
+    const someCharAnyRule = dcValidator.getRule('SOME_CHAR_ANY')
+
+    expect(typeof someCharAnyRule?.renderer).toEqual('function')
+  })
+
+  it('12 | wires a function renderer for a SOFTREGEX rule, without blocking submission', () => {
     // SOME_CHAR_ANY carries no other DQ rules in the shared fixture, so this
     // isolates SOFTREGEX's own wiring. The renderer's own pass/fail/delete-
     // suppression behaviour is covered by regex-warning-renderer.spec.ts —
@@ -589,11 +614,11 @@ describe('DC Validator', () => {
     )
   })
 
-  it('12 | HARDREGEX takes precedence over SOFTREGEX on a dual-rule column', () => {
+  it('13 | wires a renderer for a dual-rule column, and HARDREGEX still blocks submission', () => {
-    // Both rules on the same column: HARDREGEX must win, so the cell
+    // Both rules on the same column: submission blocking is still governed
-    // renders red (blocked) via HOT's own htInvalid, not yellow. No
+    // entirely by HARDREGEX/dqValidate (unchanged). A renderer is wired so
-    // SOFTREGEX renderer is wired at all here, so there's nothing that could
+    // the cell gets a 'REGEX: <pattern>' title - its own hard-vs-soft
-    // visually compete with htInvalid for that column.
+    // precedence and coloring are covered by regex-warning-renderer.spec.ts.
     const dcValidator: DcValidator = new DcValidator(
       example_sasparams,
       example_dataformats,
@@ -624,10 +649,10 @@ describe('DC Validator', () => {
         expect(valid).toBeFalse()
       }
     )
-    expect(rule?.renderer).toBeUndefined()
+    expect(typeof rule?.renderer).toEqual('function')
   })
 
-  describe('13 | failsSoftRegex (edit-record modal support for SOFTREGEX)', () => {
+  describe('14 | failsSoftRegex (edit-record modal support for SOFTREGEX)', () => {
     // SOFTREGEX never goes through dqValidate (see the wiring in
     // setupValidations), so the edit-record modal — which has no grid
     // renderer to hook into — calls this directly to show the same warning.
@@ -699,7 +724,7 @@ describe('DC Validator', () => {
       ).toBeFalse()
     })
 
-    it('is false for blank/special-missing values, same exemption as HARDREGEX', () => {
+    it('is false for blank values; special-missing-looking values are real text on a character column', () => {
       const dcValidator = buildValidator([
         {
           BASE_COL: 'SOME_CHAR_ANY',
@@ -710,28 +735,141 @@ describe('DC Validator', () => {
       ])
 
       expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', '')).toBeFalse()
-      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', '.a')).toBeFalse()
+      // ".a" is real text here and fails ^[A-Z]+$
+      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', '.a')).toBeTrue()
     })
 
-    it('is false when the column also has HARDREGEX (precedence — no yellow on a red cell)', () => {
+    it('exempts the plain missing (".") but not special missings on a numeric column', () => {
+      const dcValidator = buildValidator([
+        {
+          BASE_COL: 'SOME_NUM',
+          RULE_TYPE: 'SOFTREGEX',
+          RULE_VALUE: '^[0-9]+$',
+          X: 0
+        }
+      ])
+
+      expect(dcValidator.failsSoftRegex('SOME_NUM', '.')).toBeFalse()
+      // Special missings are deliberately-set values and face the pattern.
+      expect(dcValidator.failsSoftRegex('SOME_NUM', '.a')).toBeTrue()
+    })
+
+    it('is false when a value fails both HARDREGEX and SOFTREGEX (precedence — no yellow on a red cell)', () => {
+      const dcValidator = buildValidator([
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'HARDREGEX',
+          RULE_VALUE: '^[A-Z]{3}$',
+          X: 0
+        },
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'SOFTREGEX',
+          RULE_VALUE: '^.{5,10}$',
+          X: 0
+        }
+      ])
+
+      // 'ab' fails both HARDREGEX (not 3 uppercase letters) and SOFTREGEX
+      // (too short) - HARDREGEX wins, so this must stay false rather than
+      // report a SOFTREGEX warning on a value that's already blocked.
+      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', 'ab')).toBeFalse()
+    })
+
+    it('is false for every value when the column has HARDREGEX - SOFTREGEX never runs, even if HARDREGEX passes', () => {
+      const dcValidator = buildValidator([
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'HARDREGEX',
+          RULE_VALUE: '^[A-Z0-9]+$',
+          X: 0
+        },
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'SOFTREGEX',
+          RULE_VALUE: '^.{5,10}$',
+          X: 0
+        }
+      ])
+
+      // 'AB' passes HARDREGEX (uppercase/digits) but fails SOFTREGEX (too
+      // short) - only one regex runs per column, so the soft rule is
+      // ignored entirely.
+      expect(dcValidator.failsSoftRegex('SOME_CHAR_ANY', 'AB')).toBeFalse()
+    })
+  })
+
+  describe('15 | getRegexRuleValues (column-header info display)', () => {
+    const buildValidator = (dqRules: DQRule[]) =>
+      new DcValidator(
+        example_sasparams,
+        example_dataformats,
+        example_cols,
+        dqRules,
+        example_dqData
+      )
+
+    it('returns only hardRegexValue for a column with just HARDREGEX', () => {
       const dcValidator = buildValidator([
         {
           BASE_COL: 'SOME_CHAR_ANY',
           RULE_TYPE: 'HARDREGEX',
           RULE_VALUE: '^[A-Z]+$',
           X: 0
-        },
+        }
+      ])
+
+      expect(dcValidator.getRegexRuleValues('SOME_CHAR_ANY')).toEqual({
+        hardRegexValue: '^[A-Z]+$',
+        softRegexValue: undefined
+      })
+    })
+
+    it('returns only softRegexValue for a column with just SOFTREGEX', () => {
+      const dcValidator = buildValidator([
         {
           BASE_COL: 'SOME_CHAR_ANY',
           RULE_TYPE: 'SOFTREGEX',
-          RULE_VALUE: '^[A-Z]+$',
+          RULE_VALUE: '/\\b(the|data)\\b/i',
           X: 0
         }
       ])
 
-      expect(
+      expect(dcValidator.getRegexRuleValues('SOME_CHAR_ANY')).toEqual({
-        dcValidator.failsSoftRegex('SOME_CHAR_ANY', 'lowercase')
+        hardRegexValue: undefined,
-      ).toBeFalse()
+        softRegexValue: '/\\b(the|data)\\b/i'
+      })
+    })
+
+    it('returns both values for a column with both HARDREGEX and SOFTREGEX', () => {
+      const dcValidator = buildValidator([
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'HARDREGEX',
+          RULE_VALUE: '^HARD$',
+          X: 0
+        },
+        {
+          BASE_COL: 'SOME_CHAR_ANY',
+          RULE_TYPE: 'SOFTREGEX',
+          RULE_VALUE: '^SOFT$',
+          X: 0
+        }
+      ])
+
+      expect(dcValidator.getRegexRuleValues('SOME_CHAR_ANY')).toEqual({
+        hardRegexValue: '^HARD$',
+        softRegexValue: '^SOFT$'
+      })
+    })
+
+    it('returns both undefined for a column with no regex rule', () => {
+      const dcValidator = buildValidator([])
+
+      expect(dcValidator.getRegexRuleValues('SOME_CHAR_ANY')).toEqual({
+        hardRegexValue: undefined,
+        softRegexValue: undefined
+      })
     })
   })
 })
@@ -743,7 +881,7 @@ const makeCol = (name: string, varnum: number): Col =>
     VARNUM: varnum,
     LABEL: name,
     FMTNAME: '',
-    DDTYPE: 'CHARACTER',
+    DDTYPE: 'C',
     TYPE: '',
     CLS_RULE: 'READ',
     MEMLABEL: '',
@@ -865,7 +1003,7 @@ const example_dqRules: any = [
 const example_cols = [
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'CHARACTER',
+    DDTYPE: 'C',
     DESC: 'dropdown_desc',
     TYPE: '',
     FMTNAME: '',
@@ -878,7 +1016,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'NUMERIC',
+    DDTYPE: 'N',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -891,7 +1029,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'NUMERIC',
+    DDTYPE: 'N',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -904,7 +1042,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'CHARACTER',
+    DDTYPE: 'C',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -917,7 +1055,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'CHARACTER',
+    DDTYPE: 'C',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -930,7 +1068,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'CHARACTER',
+    DDTYPE: 'C',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -943,7 +1081,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'CHARACTER',
+    DDTYPE: 'C',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -995,7 +1133,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'NUMERIC',
+    DDTYPE: 'N',
     DESC: '',
     TYPE: '',
     FMTNAME: '',
@@ -1008,7 +1146,7 @@ const example_cols = [
   },
   {
     CLS_RULE: 'READ',
-    DDTYPE: 'NUMERIC',
+    DDTYPE: 'N',
     DESC: '',
     TYPE: '',
     FMTNAME: '',

client/src/app/shared/dc-validator/utils/applyNumericFormats.ts

@@ -7,6 +7,10 @@ import { DcValidation } from '../models/dc-validation.model'
  * Uses Intl.NumberFormat options (HOT 17+) instead of the deprecated numbro
  * `pattern`/`culture`. `maximumFractionDigits: 20` preserves all natural
  * decimals (Intl's default of 3 would round); `locale` replaces `culture`.
+ * `useGrouping: false` keeps raw digits (no thousands separator) - a
+ * separator would leak into anything that pattern-matches the displayed
+ * value (eg HARDREGEX/SOFTREGEX), and grouping can be opted into per-column
+ * with the NUMBER_FORMAT rule.
  *
  * @param rules Cell Validation rules to be updated
  * Those rules are passed in the `columns` property Of handsontable settings.
@@ -14,7 +18,7 @@ import { DcValidation } from '../models/dc-validation.model'
 export const applyNumericFormats = (rules: DcValidation[]): DcValidation[] => {
   for (let rule of rules) {
     if (rule.type === 'numeric') {
-      rule.numericFormat = { useGrouping: true, maximumFractionDigits: 20 }
+      rule.numericFormat = { useGrouping: false, maximumFractionDigits: 20 }
       rule.locale = window.navigator.language
     }
   }

client/src/app/shared/dc-validator/utils/isRegexRuleExempt.spec.ts

@@ -1,19 +1,40 @@
 import { isRegexRuleExempt } from './isRegexRuleExempt'
 
 describe('isRegexRuleExempt', () => {
-  it('exempts blank/undefined/null', () => {
+  it('exempts blank/undefined/null on any column type', () => {
     expect(isRegexRuleExempt('')).toBeTrue()
     expect(isRegexRuleExempt(undefined)).toBeTrue()
     expect(isRegexRuleExempt(null)).toBeTrue()
+    expect(isRegexRuleExempt('', true)).toBeTrue()
   })
 
-  it('exempts SAS special missing values', () => {
+  it('exempts the plain SAS missing (".") on numeric columns', () => {
-    expect(isRegexRuleExempt('.')).toBeTrue()
+    expect(isRegexRuleExempt('.', true)).toBeTrue()
-    expect(isRegexRuleExempt('.a')).toBeTrue()
+  })
-    expect(isRegexRuleExempt('_')).toBeTrue()
+
+  it('does not exempt special missings on numeric columns (they are deliberately-set values)', () => {
+    expect(isRegexRuleExempt('.a', true)).toBeFalse()
+    expect(isRegexRuleExempt('._', true)).toBeFalse()
+    expect(isRegexRuleExempt('_', true)).toBeFalse()
+    expect(isRegexRuleExempt('d', true)).toBeFalse()
+  })
+
+  it('exempts nothing but blank on character columns', () => {
+    expect(isRegexRuleExempt('.')).toBeFalse()
+    expect(isRegexRuleExempt('.a')).toBeFalse()
+    expect(isRegexRuleExempt('_')).toBeFalse()
+  })
+
+  it('does not exempt a bare single letter on character columns (real value, not a missing)', () => {
+    // A bare single letter is real character data, not a SAS special
+    // missing - it must reach the pattern, not be exempted.
+    expect(isRegexRuleExempt('d')).toBeFalse()
+    expect(isRegexRuleExempt('z')).toBeFalse()
+    expect(isRegexRuleExempt('A')).toBeFalse()
   })
 
   it('does not exempt an ordinary value', () => {
     expect(isRegexRuleExempt('ABC123')).toBeFalse()
+    expect(isRegexRuleExempt('ABC123', true)).toBeFalse()
   })
 })

client/src/app/shared/dc-validator/utils/isRegexRuleExempt.ts

@@ -1,13 +1,22 @@
-import { isSpecialMissing } from '@sasjs/utils/input/validators'
-
 /**
- * HARDREGEX/SOFTREGEX both skip pattern-matching for blank and SAS special
+ * HARDREGEX/SOFTREGEX both skip pattern-matching for:
- * missing values (., .a-.z, _) — the same exemption NOTNULL/MINVAL/MAXVAL
+ *
- * already apply elsewhere, since neither convention represents a real
+ * - blank values (undefined, null, '') on any column type - enforcing
- * formatted value the pattern is meant to check.
+ *   populated values is NOTNULL's job, not the pattern's; and
+ * - the plain SAS numeric missing (".") on numeric columns - it
+ *   represents the absence of a value, same as blank.
+ *
+ * SPECIAL missings (.a-.z, ._, bare letters) are NOT exempt, even on
+ * numeric columns: being deliberately set, they are real values the
+ * pattern is meant to check.  This also means isSpecialMissing from
+ * @sasjs/utils (which would match them, with an optional dot) is
+ * deliberately not used here.
  */
-export const isRegexRuleExempt = (value: any): boolean => {
+export const isRegexRuleExempt = (
+  value: any,
+  isNumeric: boolean = false
+): boolean => {
   if (value === undefined || value === null || value === '') return true
 
-  return isSpecialMissing(value)
+  return isNumeric && value === '.'
 }

client/src/app/shared/dc-validator/validations/dq-validation.spec.ts

@@ -30,12 +30,28 @@ describe('dqValidate - HARDREGEX', () => {
     expect(dqValidate(rules, null)).toBeTrue()
   })
 
-  it('treats SAS special missing values as valid regardless of the pattern', () => {
+  it('treats the plain SAS missing (".") as valid on numeric columns, regardless of the pattern', () => {
     const rules = [rule({ RULE_VALUE: '^[A-Z]{3}\\d{4}$' })]
 
-    expect(dqValidate(rules, '.')).toBeTrue()
+    expect(dqValidate(rules, '.', true)).toBeTrue()
-    expect(dqValidate(rules, '.a')).toBeTrue()
+  })
-    expect(dqValidate(rules, '_')).toBeTrue()
+
+  it('applies the pattern to special missings on numeric columns (deliberately-set values)', () => {
+    const rules = [rule({ RULE_VALUE: '^[A-Z]{3}\\d{4}$' })]
+
+    expect(dqValidate(rules, '.a', true)).toBeFalse()
+    expect(dqValidate(rules, '_', true)).toBeFalse()
+    expect(dqValidate(rules, 'd', true)).toBeFalse()
+  })
+
+  it('applies the pattern to special-missing-looking values on character columns', () => {
+    const rules = [rule({ RULE_VALUE: '^[A-Z]{3}\\d{4}$' })]
+
+    expect(dqValidate(rules, '.')).toBeFalse()
+    expect(dqValidate(rules, '.a')).toBeFalse()
+    expect(dqValidate(rules, '_')).toBeFalse()
+    // A bare single letter is a real character value and must be matched.
+    expect(dqValidate(rules, 'd')).toBeFalse()
   })
 
   it('fails open (treats as valid) when the pattern is malformed', () => {
@@ -44,6 +60,19 @@ describe('dqValidate - HARDREGEX', () => {
     expect(dqValidate(rules, 'anything')).toBeTrue()
   })
 
+  it('validates a single letter on a character column against the pattern', () => {
+    // A bare single letter ("d", "z") is real character data, not a SAS
+    // special missing - it must reach the pattern, not be exempted.
+    const rules = [rule({ RULE_VALUE: '/the|data/i' })]
+
+    expect(dqValidate(rules, 'd')).toBeFalse()
+    expect(dqValidate(rules, 'z')).toBeFalse()
+    expect(dqValidate(rules, 'the')).toBeTrue()
+    expect(dqValidate(rules, 'DATA')).toBeTrue()
+    expect(dqValidate(rules, 'some data here')).toBeTrue()
+    expect(dqValidate(rules, '')).toBeTrue() // blank stays exempt
+  })
+
   it('handles a more elaborate pattern (multiple character classes, quantifiers, an escaped literal dot)', () => {
     // Same email pattern used in the getdata.js mock's REGEX_HARD_COL demo.
     const rules = [rule({ RULE_VALUE: '[\\w.]+@[\\w]+\\.[a-z]{2,}' })]

client/src/app/shared/dc-validator/validations/dq-validation.ts

@@ -4,7 +4,11 @@ import { isRegexRuleExempt } from '../utils/isRegexRuleExempt'
 import { parseRegexRule } from '../utils/parseRegexRule'
 
 const dqValidation: {
-  [key: string]: (value: any, ruleValue: string | number) => boolean
+  [key: string]: (
+    value: any,
+    ruleValue: string | number,
+    isNumeric?: boolean
+  ) => boolean
 } = {
   CASE: (value: any, ruleValue: string | number): boolean => {
     switch (ruleValue) {
@@ -51,8 +55,12 @@ const dqValidation: {
   },
   // Pattern is used as authored, not auto-anchored — a rule author who
   // wants a full-value match must write ^...$ themselves.
-  HARDREGEX: (value: any, ruleValue: string | number): boolean => {
+  HARDREGEX: (
-    if (isRegexRuleExempt(value)) return true
+    value: any,
+    ruleValue: string | number,
+    isNumeric: boolean = false
+  ): boolean => {
+    if (isRegexRuleExempt(value, isNumeric)) return true
 
     try {
       return parseRegexRule(ruleValue.toString()).test(value.toString())
@@ -65,10 +73,16 @@ const dqValidation: {
   }
 }
 
-export const dqValidate = (dqRules: DQRule[], value: any): boolean => {
+export const dqValidate = (
+  dqRules: DQRule[],
+  value: any,
+  isNumeric: boolean = false
+): boolean => {
   for (let detail of dqRules) {
     if (dqValidation[detail.RULE_TYPE]) {
-      if (!dqValidation[detail.RULE_TYPE](value, detail.RULE_VALUE)) {
+      if (
+        !dqValidation[detail.RULE_TYPE](value, detail.RULE_VALUE, isNumeric)
+      ) {
         console.warn(
           `DQ Invalid Reason: ${
             detail.RULE_TYPE

client/src/app/shared/utils/col-info-html.spec.ts

@@ -18,4 +18,60 @@ describe('buildColInfoHtml', () => {
       'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.'
     )
   })
+
+  it('appends a HARDREGEX line when only a HARDREGEX value is provided', () => {
+    const colInfo: DataFormat = {
+      label: 'Some Character Column',
+      type: 'char',
+      length: '1024',
+      format: '$1024.'
+    }
+
+    expect(
+      buildColInfoHtml('SOME_CHAR', colInfo, '/^[A-Z]+$/i', undefined)
+    ).toBe(
+      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>HARDREGEX: /^[A-Z]+$/i'
+    )
+  })
+
+  it('appends a SOFTREGEX line when only a SOFTREGEX value is provided', () => {
+    const colInfo: DataFormat = {
+      label: 'Some Character Column',
+      type: 'char',
+      length: '1024',
+      format: '$1024.'
+    }
+
+    expect(
+      buildColInfoHtml('SOME_CHAR', colInfo, undefined, '/^[a-z]+$/')
+    ).toBe(
+      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>SOFTREGEX: /^[a-z]+$/'
+    )
+  })
+
+  it('appends only the HARDREGEX line when a column has both rules, since HARDREGEX is the one applied', () => {
+    const colInfo: DataFormat = {
+      label: 'Some Character Column',
+      type: 'char',
+      length: '1024',
+      format: '$1024.'
+    }
+
+    expect(
+      buildColInfoHtml('SOME_CHAR', colInfo, '/^[A-Z]+$/i', '/^.{5,10}$/')
+    ).toBe(
+      'NAME: SOME_CHAR<br>LABEL: Some Character Column<br>TYPE: char<br>LENGTH: 1024<br>FORMAT: $1024.<br>HARDREGEX: /^[A-Z]+$/i'
+    )
+  })
+
+  it('omits the REGEX line when no regex rule value is provided', () => {
+    const colInfo: DataFormat = {
+      label: 'Some Character Column',
+      type: 'char',
+      length: '1024',
+      format: '$1024.'
+    }
+
+    expect(buildColInfoHtml('SOME_CHAR', colInfo)).not.toContain('REGEX:')
+  })
 })

client/src/app/shared/utils/col-info-html.ts

@@ -7,9 +7,23 @@ import { DataFormat } from '../../models/sas/common/DateFormat'
  */
 export function buildColInfoHtml(
   colName: string,
-  colInfo?: DataFormat
+  colInfo?: DataFormat,
+  hardRegexValue?: string,
+  softRegexValue?: string
 ): string {
   if (!colInfo) return 'No info found'
 
-  return `NAME: ${colName}<br>LABEL: ${colInfo.label}<br>TYPE: ${colInfo.type}<br>LENGTH: ${colInfo.length}<br>FORMAT: ${colInfo.format}`
+  let html = `NAME: ${colName}<br>LABEL: ${colInfo.label}<br>TYPE: ${colInfo.type}<br>LENGTH: ${colInfo.length}<br>FORMAT: ${colInfo.format}`
+
+  // Only ever one REGEX rule is applied per column: when both HARDREGEX
+  // and SOFTREGEX exist, SOFTREGEX is ignored entirely (same precedence as
+  // makeRegexWarningRenderer / DcValidator.failsSoftRegex).  Show only the
+  // rule that is applied.
+  if (hardRegexValue) {
+    html += `<br>HARDREGEX: ${hardRegexValue}`
+  } else if (softRegexValue) {
+    html += `<br>SOFTREGEX: ${softRegexValue}`
+  }
+
+  return html
 }

client/src/browserify-cipher.d.ts

@@ -0,0 +1,23 @@
+/**
+ * Type declarations for the untyped `browserify-cipher` package.
+ * Only the cipher functions used by `src/crypto-shim.ts` are declared.
+ */
+declare module 'browserify-cipher' {
+  export interface Cipheriv {
+    update(data: unknown): unknown
+    final(): unknown
+    setAutoPadding(autoPadding?: boolean): void
+  }
+
+  export function createCipheriv(
+    algorithm: string,
+    key: unknown,
+    iv: unknown
+  ): Cipheriv
+
+  export function createDecipheriv(
+    algorithm: string,
+    key: unknown,
+    iv: unknown
+  ): Cipheriv
+}

client/src/crypto-shim.ts

@@ -0,0 +1,30 @@
+/**
+ * Minimal browser shim for the node `crypto` module.
+ *
+ * Replaces `crypto-browserify`, which pulls in `elliptic` (via
+ * `browserify-sign` and `create-ecdh`) - a package with open low-severity
+ * advisories and no fixed release.  None of the signing/ECDH functionality
+ * is needed here: `@sheet/crypto` (see tsconfig `paths` mapping for
+ * "crypto") only uses hashing, HMAC, AES ciphers and random bytes.
+ */
+import createHash from 'create-hash'
+import createHmac from 'create-hmac'
+import { createCipheriv, createDecipheriv } from 'browserify-cipher'
+import randomBytes from 'randombytes'
+
+const HASHES = [
+  'md4',
+  'md5',
+  'ripemd160',
+  'sha1',
+  'sha224',
+  'sha256',
+  'sha384',
+  'sha512'
+]
+
+export function getHashes(): string[] {
+  return [...HASHES]
+}
+
+export { createHash, createHmac, createCipheriv, createDecipheriv, randomBytes }

client/src/index.html

@@ -15,21 +15,18 @@
     -->
 
     <!-- meta tags -->
+    <!--
+      NOTE: Data Controller must run entirely offline / on-prem. Never reference
+      external assets (fonts, images, scripts, styles, CDN links, or remote
+      og:/itemprop URLs) here or anywhere in the built product. All assets must
+      be bundled and served locally.
+    -->
     <meta name="description" content="Capture, Review, and Approve" />
     <meta itemprop="name" content="Data Controller for SAS®" />
     <meta itemprop="description" content="Capture, Review, and Approve" />
-    <meta
-      itemprop="image"
-      content="https://docs.datacontroller.io/img/dc_bg_Asset-5@2x.png"
-    />
-    <meta property="og:url" content="http://demo.datacontroller.io" />
     <meta property="og:type" content="website" />
     <meta property="og:title" content="Data Controller for SAS®" />
     <meta property="og:description" content="Capture, Review, and Approve" />
-    <meta
-      property="og:image"
-      content="https://docs.datacontroller.io/img/dc_bg_Asset-5@2x.png"
-    />
 
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <meta http-equiv="X-UA-Compatible" content="IE=edge" />
@@ -54,7 +51,7 @@
 
   <sasjs
     serverUrl=""
-    appLoc="/Public/app/devtest"
+    appLoc="/Public/app/dc"
     serverType="SASJS"
     loginMechanism="Redirected"
     debug="false"

client/tsconfig.app.json

@@ -10,6 +10,6 @@
     "outDir": "./app",
     "types": []
   },
-  "files": ["src/polyfills.ts", "src/main.ts", "src/app/app.d.ts"],
+  "files": ["src/polyfills.ts", "src/main.ts", "src/app/app.d.ts", "src/crypto-shim.ts"],
   "include": ["src/**/*.d.ts"]
 }

client/tsconfig.json

@@ -17,7 +17,7 @@
     "resolveJsonModule": true,
     "target": "ES2022",
     "paths": {
-      "crypto": ["./node_modules/crypto-browserify"],
+      "crypto": ["./src/crypto-shim"],
       "stream": ["./node_modules/stream-browserify"],
       "assert": ["./node_modules/assert"],
       "http": ["./node_modules/stream-http"],

client/tsconfig.spec.json

@@ -5,6 +5,6 @@
     "outDir": "./out-tsc/spec",
     "types": ["jasmine"]
   },
-  "files": ["src/polyfills.ts"],
+  "files": ["src/polyfills.ts", "src/crypto-shim.ts"],
   "include": ["src/**/*.spec.ts", "src/**/*.d.ts"]
 }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "dcfrontend",
-  "version": "7.11.0",
+  "version": "7.12.0",
   "description": "Data Controller",
   "devDependencies": {
     "@saithodev/semantic-release-gitea": "^2.1.0",

sas/mocks/sas9/sasjs/services/editors/getdata.js

@@ -44,10 +44,9 @@ const data = {
     cols: [
         {
             NAME: "PRIMARY_KEY_FIELD",
-            VARNUM: 1,
             LABEL: "PRIMARY_KEY_FIELD",
             FMTNAME: "",
-            DDTYPE: "NUMERIC",
+            DDTYPE: "N",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -55,10 +54,9 @@ const data = {
         },
         {
             NAME: "SOME_BESTNUM",
-            VARNUM: 9,
             LABEL: "SOME_BESTNUM",
             FMTNAME: "BEST",
-            DDTYPE: "NUMERIC",
+            DDTYPE: "N",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -66,10 +64,9 @@ const data = {
         },
         {
             NAME: "SOME_CHAR",
-            VARNUM: 2,
             LABEL: "SOME_CHAR",
             FMTNAME: "",
-            DDTYPE: "CHARACTER",
+            DDTYPE: "C",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -77,7 +74,6 @@ const data = {
         },
         {
             NAME: "SOME_DATE",
-            VARNUM: 5,
             LABEL: "SOME_DATE",
             FMTNAME: "DATE",
             DDTYPE: "DATE",
@@ -88,7 +84,6 @@ const data = {
         },
         {
             NAME: "SOME_DATETIME",
-            VARNUM: 6,
             LABEL: "SOME_DATETIME",
             FMTNAME: "DATETIME",
             DDTYPE: "DATETIME",
@@ -99,10 +94,9 @@ const data = {
         },
         {
             NAME: "SOME_DROPDOWN",
-            VARNUM: 3,
             LABEL: "SOME_DROPDOWN",
             FMTNAME: "",
-            DDTYPE: "CHARACTER",
+            DDTYPE: "C",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -110,10 +104,9 @@ const data = {
         },
         {
             NAME: "SOME_HARDSELECT",
-            VARNUM: 10,
             LABEL: "SOME_HARDSELECT",
             FMTNAME: "",
-            DDTYPE: "CHARACTER",
+            DDTYPE: "C",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -121,10 +114,9 @@ const data = {
         },
         {
             NAME: "SOME_NUM",
-            VARNUM: 4,
             LABEL: "SOME_NUM",
             FMTNAME: "",
-            DDTYPE: "NUMERIC",
+            DDTYPE: "N",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -132,10 +124,9 @@ const data = {
         },
         {
             NAME: "SOME_SHORTNUM",
-            VARNUM: 8,
             LABEL: "SOME_SHORTNUM",
             FMTNAME: "",
-            DDTYPE: "NUMERIC",
+            DDTYPE: "N",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "",
@@ -143,7 +134,6 @@ const data = {
         },
         {
             NAME: "SOME_TIME",
-            VARNUM: 7,
             LABEL: "SOME_TIME",
             FMTNAME: "TIME",
             DDTYPE: "TIME",
@@ -154,10 +144,9 @@ const data = {
         },
         {
             NAME: "READONLY_COL",
-            VARNUM: 11,
             LABEL: "READONLY_COL",
             FMTNAME: "",
-            DDTYPE: "CHARACTER",
+            DDTYPE: "C",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "Read-only: default value inserted on add-row, not editable",
@@ -165,10 +154,9 @@ const data = {
         },
         {
             NAME: "HIDDEN_COL",
-            VARNUM: 12,
             LABEL: "HIDDEN_COL",
             FMTNAME: "",
-            DDTYPE: "CHARACTER",
+            DDTYPE: "C",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "Hidden: invisible in grid but submitted; default on add-row",
@@ -176,10 +164,9 @@ const data = {
         },
         {
             NAME: "ROUND_COL",
-            VARNUM: 13,
             LABEL: "ROUND_COL",
             FMTNAME: "",
-            DDTYPE: "NUMERIC",
+            DDTYPE: "N",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "Round: edited values rounded Excel-style to 2 decimals",
@@ -187,10 +174,9 @@ const data = {
         },
         {
             NAME: "NUMFMT_COL",
-            VARNUM: 14,
             LABEL: "NUMFMT_COL",
             FMTNAME: "",
-            DDTYPE: "NUMERIC",
+            DDTYPE: "N",
             CLS_RULE: "READ",
             MEMLABEL: "",
             DESC: "Number format: displayed as EUR currency (value unchanged)",

sas/mocks/sas9/sasjs/services/public/viewdata.js

@@ -9,37 +9,33 @@ _webout=`{"SYSDATE" : "26SEP22"
 {
 "NAME": "PRIMARY_KEY_FIELD",
 "LENGTH": 8,
-"VARNUM": 1,
 "LABEL": "PRIMARY_KEY_FIELD",
 "FMTNAME": "",
 "FORMAT": "8.",
 "TYPE": "N",
-"DDTYPE": "NUMERIC"
+"DDTYPE": "N"
 },
 {
 "NAME": "SOME_BESTNUM",
 "LENGTH": 8,
-"VARNUM": 9,
 "LABEL": "SOME_BESTNUM",
 "FMTNAME": "BEST",
 "FORMAT": "BEST.",
 "TYPE": "N",
-"DDTYPE": "NUMERIC"
+"DDTYPE": "N"
 },
 {
 "NAME": "SOME_CHAR",
 "LENGTH": 32767,
-"VARNUM": 2,
 "LABEL": "SOME_CHAR",
 "FMTNAME": "",
 "FORMAT": "$32767.",
 "TYPE": "C",
-"DDTYPE": "CHARACTER"
+"DDTYPE": "C"
 },
 {
 "NAME": "SOME_DATE",
 "LENGTH": 8,
-"VARNUM": 5,
 "LABEL": "SOME_DATE",
 "FMTNAME": "DATE",
 "FORMAT": "DATE9.",
@@ -49,7 +45,6 @@ _webout=`{"SYSDATE" : "26SEP22"
 {
 "NAME": "SOME_DATETIME",
 "LENGTH": 8,
-"VARNUM": 6,
 "LABEL": "SOME_DATETIME",
 "FMTNAME": "DATETIME",
 "FORMAT": "DATETIME19.",
@@ -59,37 +54,33 @@ _webout=`{"SYSDATE" : "26SEP22"
 {
 "NAME": "SOME_DROPDOWN",
 "LENGTH": 128,
-"VARNUM": 3,
 "LABEL": "SOME_DROPDOWN",
 "FMTNAME": "",
 "FORMAT": "$128.",
 "TYPE": "C",
-"DDTYPE": "CHARACTER"
+"DDTYPE": "C"
 },
 {
 "NAME": "SOME_NUM",
 "LENGTH": 8,
-"VARNUM": 4,
 "LABEL": "SOME_NUM",
 "FMTNAME": "",
 "FORMAT": "8.",
 "TYPE": "N",
-"DDTYPE": "NUMERIC"
+"DDTYPE": "N"
 },
 {
 "NAME": "SOME_SHORTNUM",
 "LENGTH": 4,
-"VARNUM": 8,
 "LABEL": "SOME_SHORTNUM",
 "FMTNAME": "",
 "FORMAT": "4.",
 "TYPE": "N",
-"DDTYPE": "NUMERIC"
+"DDTYPE": "N"
 },
 {
 "NAME": "SOME_TIME",
 "LENGTH": 8,
-"VARNUM": 7,
 "LABEL": "SOME_TIME",
 "FMTNAME": "TIME",
 "FORMAT": "TIME8.",

sas/mocks/sasjs/services/editors/getdata.js

@@ -52,7 +52,8 @@ function makeRows(n) {
             ROUND_COL: Number((i + 1 + i / 7).toFixed(5)), // rounds on edit
             NUMFMT_COL: 1000 + i * 12.5,                    // shown as EUR
             REGEX_HARD_COL: "user@example.com",  // HARDREGEX: email — starts valid
-            REGEX_SOFT_COL: "SW1A 1AA"            // SOFTREGEX: UK postcode — starts valid
+            REGEX_SOFT_COL: "SW1A 1AA",           // SOFTREGEX: UK postcode — starts valid
+            REGEX_BOTH_COL: "ABC-123"             // HARDREGEX + SOFTREGEX together — starts valid against both
         })
     }
     return rows
@@ -68,10 +69,9 @@ let webouts = {
         cols: [
             {
                 NAME: "PRIMARY_KEY_FIELD",
-                VARNUM: 1,
                 LABEL: "PRIMARY_KEY_FIELD",
                 FMTNAME: "",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -80,10 +80,9 @@ let webouts = {
             },
             {
                 NAME: "SOME_BESTNUM",
-                VARNUM: 9,
                 LABEL: "SOME_BESTNUM",
                 FMTNAME: "BEST",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -92,10 +91,9 @@ let webouts = {
             },
             {
                 NAME: "SOME_CHAR",
-                VARNUM: 2,
                 LABEL: "Some Character Column",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -104,7 +102,6 @@ let webouts = {
             },
             {
                 NAME: "SOME_DATE",
-                VARNUM: 5,
                 LABEL: "Some Date",
                 FMTNAME: "DATE",
                 DDTYPE: "DATE",
@@ -116,7 +113,6 @@ let webouts = {
             },
             {
                 NAME: "SOME_DATETIME",
-                VARNUM: 6,
                 LABEL: "SOME_DATETIME",
                 FMTNAME: "DATETIME",
                 DDTYPE: "DATETIME",
@@ -128,10 +124,9 @@ let webouts = {
             },
             {
                 NAME: "SOME_DROPDOWN",
-                VARNUM: 3,
                 LABEL: "SOME_DROPDOWN",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -140,10 +135,9 @@ let webouts = {
             },
             {
                 NAME: "SOME_HARDSELECT",
-                VARNUM: 10,
                 LABEL: "SOME_HARDSELECT",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -152,10 +146,9 @@ let webouts = {
             },
             {
                 NAME: "SOME_NUM",
-                VARNUM: 4,
                 LABEL: "",
                 FMTNAME: "",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -164,10 +157,9 @@ let webouts = {
             },
             {
                 NAME: "SOME_SHORTNUM",
-                VARNUM: 8,
                 LABEL: "SOME_SHORTNUM",
                 FMTNAME: "",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -176,7 +168,6 @@ let webouts = {
             },
             {
                 NAME: "SOME_TIME",
-                VARNUM: 7,
                 LABEL: "SOME_TIME",
                 FMTNAME: "TIME",
                 DDTYPE: "TIME",
@@ -188,10 +179,9 @@ let webouts = {
             },
             {
                 NAME: "READONLY_COL",
-                VARNUM: 11,
                 LABEL: "READONLY_COL",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Read-only: default value inserted on add-row, not editable",
@@ -200,10 +190,9 @@ let webouts = {
             },
             {
                 NAME: "HIDDEN_COL",
-                VARNUM: 12,
                 LABEL: "HIDDEN_COL",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Hidden: invisible in grid but submitted; default on add-row",
@@ -212,10 +201,9 @@ let webouts = {
             },
             {
                 NAME: "ROUND_COL",
-                VARNUM: 13,
                 LABEL: "ROUND_COL",
                 FMTNAME: "",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Round: edited values rounded Excel-style to 2 decimals",
@@ -224,10 +212,9 @@ let webouts = {
             },
             {
                 NAME: "NUMFMT_COL",
-                VARNUM: 14,
                 LABEL: "NUMFMT_COL",
                 FMTNAME: "",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Number format: displayed as EUR currency (value unchanged)",
@@ -236,10 +223,9 @@ let webouts = {
             },
             {
                 NAME: "REGEX_HARD_COL",
-                VARNUM: 15,
                 LABEL: "REGEX_HARD_COL",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "HARDREGEX: must be a valid email address or submission is blocked",
@@ -248,15 +234,27 @@ let webouts = {
             },
             {
                 NAME: "REGEX_SOFT_COL",
-                VARNUM: 16,
                 LABEL: "REGEX_SOFT_COL",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "SOFTREGEX: should be a valid UK postcode, shown as a yellow warning if not, but can still submit",
                 LONGDESC: "",
                 COLTYPE: "{\"data\":\"REGEX_SOFT_COL\"}"
+            },
+            {
+                NAME: "REGEX_BOTH_COL",
+                LABEL: "REGEX_BOTH_COL",
+                FMTNAME: "",
+                DDTYPE: "C",
+                CLS_RULE: "READ",
+                MEMLABEL: "",
+                DESC: "HARDREGEX + SOFTREGEX together: must be uppercase/digits/-/_ (blocking), recommended 5-10 chars long (warning)",
+                LONGDESC: ""
+                // No COLTYPE - DcValidator falls back to a plain { data: name }
+                // rule when COLTYPE is absent, same shape this trivial JSON
+                // would produce anyway (see parseColTypeRow).
             }
         ],
         dqdata: [
@@ -282,7 +280,9 @@ let webouts = {
             { BASE_COL: "ROUND_COL", RULE_TYPE: "ROUND", RULE_VALUE: "2" },
             { BASE_COL: "NUMFMT_COL", RULE_TYPE: "NUMBER_FORMAT", RULE_VALUE: '{"style":"currency","currency":"EUR"}' },
             { BASE_COL: "REGEX_HARD_COL", RULE_TYPE: "HARDREGEX", RULE_VALUE: "/[\\w.]+@[\\w]+\\.[a-z]{2,}/" },
-            { BASE_COL: "REGEX_SOFT_COL", RULE_TYPE: "SOFTREGEX", RULE_VALUE: "/[A-Z]{1,2}\\d{1,2}[A-Z]?\\s?\\d[A-Z]{2}/" }
+            { BASE_COL: "REGEX_SOFT_COL", RULE_TYPE: "SOFTREGEX", RULE_VALUE: "/[A-Z]{1,2}\\d{1,2}[A-Z]?\\s?\\d[A-Z]{2}/" },
+            { BASE_COL: "REGEX_BOTH_COL", RULE_TYPE: "HARDREGEX", RULE_VALUE: "/^[A-Z0-9_-]+$/" },
+            { BASE_COL: "REGEX_BOTH_COL", RULE_TYPE: "SOFTREGEX", RULE_VALUE: "/^.{5,10}$/" }
         ],
         dsmeta: [
             { ODS_TABLE: "ATTRIBUTES", NAME: "Data Set Name", VALUE: "DC996664.MPE_X_TEST" },
@@ -335,7 +335,8 @@ let webouts = {
             { NAME: "round_col", MAXLEN: 8 },
             { NAME: "numfmt_col", MAXLEN: 8 },
             { NAME: "regex_hard_col", MAXLEN: 128 },
-            { NAME: "regex_soft_col", MAXLEN: 128 }
+            { NAME: "regex_soft_col", MAXLEN: 128 },
+            { NAME: "regex_both_col", MAXLEN: 128 }
         ],
         query: [],
         sasdata: makeRows(100),
@@ -357,12 +358,13 @@ let webouts = {
                 ROUND_COL: { format: "best.", label: "ROUND_COL", length: "8", type: "num" },
                 NUMFMT_COL: { format: "best.", label: "NUMFMT_COL", length: "8", type: "num" },
                 REGEX_HARD_COL: { format: "$128.", label: "REGEX_HARD_COL", length: "128", type: "char" },
-                REGEX_SOFT_COL: { format: "$128.", label: "REGEX_SOFT_COL", length: "128", type: "char" }
+                REGEX_SOFT_COL: { format: "$128.", label: "REGEX_SOFT_COL", length: "128", type: "char" },
+                REGEX_BOTH_COL: { format: "$128.", label: "REGEX_BOTH_COL", length: "128", type: "char" }
             }
         },
         sasparams: [
             {
-                COLHEADERS: "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,SOME_HARDSELECT,SOME_NUM,SOME_DATE,SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM,READONLY_COL,HIDDEN_COL,ROUND_COL,NUMFMT_COL,REGEX_HARD_COL,REGEX_SOFT_COL",
+                COLHEADERS: "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,SOME_HARDSELECT,SOME_NUM,SOME_DATE,SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM,READONLY_COL,HIDDEN_COL,ROUND_COL,NUMFMT_COL,REGEX_HARD_COL,REGEX_SOFT_COL,REGEX_BOTH_COL",
                 FILTER_TEXT: FILTER_TEXT,
                 PKCNT: 1,
                 PK: "PRIMARY_KEY_FIELD",
@@ -404,10 +406,9 @@ let webouts = {
         cols: [
             {
                 NAME: "DD_LONGDESC",
-                VARNUM: 6,
                 LABEL: "DD_LONGDESC",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -416,10 +417,9 @@ let webouts = {
             },
             {
                 NAME: "DD_OWNER",
-                VARNUM: 7,
                 LABEL: "DD_OWNER",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -428,10 +428,9 @@ let webouts = {
             },
             {
                 NAME: "DD_RESPONSIBLE",
-                VARNUM: 8,
                 LABEL: "DD_RESPONSIBLE",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -440,10 +439,9 @@ let webouts = {
             },
             {
                 NAME: "DD_SENSITIVITY",
-                VARNUM: 9,
                 LABEL: "DD_SENSITIVITY",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -452,10 +450,9 @@ let webouts = {
             },
             {
                 NAME: "DD_SHORTDESC",
-                VARNUM: 5,
                 LABEL: "DD_SHORTDESC",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -464,10 +461,9 @@ let webouts = {
             },
             {
                 NAME: "DD_SOURCE",
-                VARNUM: 4,
                 LABEL: "DD_SOURCE",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -476,10 +472,9 @@ let webouts = {
             },
             {
                 NAME: "DD_TYPE",
-                VARNUM: 3,
                 LABEL: "DD_TYPE",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -488,10 +483,9 @@ let webouts = {
             },
             {
                 NAME: "TX_FROM",
-                VARNUM: 1,
                 LABEL: "TX_FROM",
                 FMTNAME: "datetime",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -499,10 +493,9 @@ let webouts = {
             },
             {
                 NAME: "TX_TO",
-                VARNUM: 2,
                 LABEL: "TX_TO",
                 FMTNAME: "datetime",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -682,10 +675,9 @@ let webouts = {
         cols: [
             {
                 NAME: "LAST_SEEN_DT",
-                VARNUM: 2,
                 LABEL: "LAST_SEEN_DT",
                 FMTNAME: "date",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -693,10 +685,9 @@ let webouts = {
             },
             {
                 NAME: "REGISTERED_DT",
-                VARNUM: 3,
                 LABEL: "REGISTERED_DT",
                 FMTNAME: "date",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -704,10 +695,9 @@ let webouts = {
             },
             {
                 NAME: "USER_ID",
-                VARNUM: 1,
                 LABEL: "USER_ID",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -809,10 +799,9 @@ let webouts = {
         cols: [
             {
                 NAME: "AUDIT_LIBDS",
-                VARNUM: 22,
                 LABEL: "AUDIT_LIBDS",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -821,10 +810,9 @@ let webouts = {
             },
             {
                 NAME: "BUSKEY",
-                VARNUM: 7,
                 LABEL: "BUSKEY",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -833,10 +821,9 @@ let webouts = {
             },
             {
                 NAME: "CLOSE_VARS",
-                VARNUM: 13,
                 LABEL: "CLOSE_VARS",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -845,10 +832,9 @@ let webouts = {
             },
             {
                 NAME: "DSN",
-                VARNUM: 4,
                 LABEL: "DSN",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -857,10 +843,9 @@ let webouts = {
             },
             {
                 NAME: "LIBREF",
-                VARNUM: 3,
                 LABEL: "LIBREF",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -869,10 +854,9 @@ let webouts = {
             },
             {
                 NAME: "LOADTYPE",
-                VARNUM: 6,
                 LABEL: "LOADTYPE",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -881,10 +865,9 @@ let webouts = {
             },
             {
                 NAME: "NOTES",
-                VARNUM: 20,
                 LABEL: "NOTES",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -893,10 +876,9 @@ let webouts = {
             },
             {
                 NAME: "NUM_OF_APPROVALS_REQUIRED",
-                VARNUM: 5,
                 LABEL: "NUM_OF_APPROVALS_REQUIRED",
                 FMTNAME: "",
-                DDTYPE: "NUMERIC",
+                DDTYPE: "N",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -905,10 +887,9 @@ let webouts = {
             },
             {
                 NAME: "POST_APPROVE_HOOK",
-                VARNUM: 17,
                 LABEL: "POST_APPROVE_HOOK",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -917,10 +898,9 @@ let webouts = {
             },
             {
                 NAME: "POST_EDIT_HOOK",
-                VARNUM: 15,
                 LABEL: "POST_EDIT_HOOK",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -929,10 +909,9 @@ let webouts = {
             },
             {
                 NAME: "PRE_APPROVE_HOOK",
-                VARNUM: 16,
                 LABEL: "PRE_APPROVE_HOOK",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -941,10 +920,9 @@ let webouts = {
             },
             {
                 NAME: "PRE_EDIT_HOOK",
-                VARNUM: 14,
                 LABEL: "PRE_EDIT_HOOK",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -953,10 +931,9 @@ let webouts = {
             },
             {
                 NAME: "RK_UNDERLYING",
-                VARNUM: 21,
                 LABEL: "RK_UNDERLYING",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -965,10 +942,9 @@ let webouts = {
             },
             {
                 NAME: "SIGNOFF_COLS",
-                VARNUM: 18,
                 LABEL: "SIGNOFF_COLS",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -977,10 +953,9 @@ let webouts = {
             },
             {
                 NAME: "SIGNOFF_HOOK",
-                VARNUM: 19,
                 LABEL: "SIGNOFF_HOOK",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -989,7 +964,6 @@ let webouts = {
             },
             {
                 NAME: "TX_FROM",
-                VARNUM: 1,
                 LABEL: "TX_FROM",
                 FMTNAME: "DATETIME",
                 DDTYPE: "DATETIME",
@@ -1000,7 +974,6 @@ let webouts = {
             },
             {
                 NAME: "TX_TO",
-                VARNUM: 2,
                 LABEL: "TX_TO",
                 FMTNAME: "DATETIME",
                 DDTYPE: "DATETIME",
@@ -1011,10 +984,9 @@ let webouts = {
             },
             {
                 NAME: "VAR_BUSFROM",
-                VARNUM: 10,
                 LABEL: "VAR_BUSFROM",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -1023,10 +995,9 @@ let webouts = {
             },
             {
                 NAME: "VAR_BUSTO",
-                VARNUM: 11,
                 LABEL: "VAR_BUSTO",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -1035,10 +1006,9 @@ let webouts = {
             },
             {
                 NAME: "VAR_PROCESSED",
-                VARNUM: 12,
                 LABEL: "VAR_PROCESSED",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -1047,10 +1017,9 @@ let webouts = {
             },
             {
                 NAME: "VAR_TXFROM",
-                VARNUM: 8,
                 LABEL: "VAR_TXFROM",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -1059,10 +1028,9 @@ let webouts = {
             },
             {
                 NAME: "VAR_TXTO",
-                VARNUM: 9,
                 LABEL: "VAR_TXTO",
                 FMTNAME: "",
-                DDTYPE: "CHARACTER",
+                DDTYPE: "C",
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
@@ -1551,7 +1519,7 @@ let webouts = {
 // MPE_X_TEST keeps matching the excel upload fixtures (which predate these
 // columns). The demo tables are built lazily (below) so we don't clone the large
 // MPE_X_TEST payload unless MPE_X_NEW is actually requested.
-const RULE_DEMO_COLS = ['SOME_HARDSELECT', 'READONLY_COL', 'HIDDEN_COL', 'ROUND_COL', 'NUMFMT_COL', 'REGEX_HARD_COL', 'REGEX_SOFT_COL']
+const RULE_DEMO_COLS = ['SOME_HARDSELECT', 'READONLY_COL', 'HIDDEN_COL', 'ROUND_COL', 'NUMFMT_COL', 'REGEX_HARD_COL', 'REGEX_SOFT_COL', 'REGEX_BOTH_COL']
 
 function stripRuleCols(t) {
     const uc = new Set(RULE_DEMO_COLS)

sas/mocks/sasjs/services/public/viewdata.js

@@ -31,37 +31,33 @@ let webouts = {
     {
     "NAME": "PRIMARY_KEY_FIELD",
     "LENGTH": 8,
-    "VARNUM": 1,
     "LABEL": "PRIMARY_KEY_FIELD",
     "FMTNAME": "",
     "FORMAT": "8.",
     "TYPE": "N",
-    "DDTYPE": "NUMERIC"
+    "DDTYPE": "N"
     },
     {
     "NAME": "SOME_BESTNUM",
     "LENGTH": 8,
-    "VARNUM": 9,
     "LABEL": "SOME_BESTNUM",
     "FMTNAME": "BEST",
     "FORMAT": "BEST.",
     "TYPE": "N",
-    "DDTYPE": "NUMERIC"
+    "DDTYPE": "N"
     },
     {
     "NAME": "SOME_CHAR",
     "LENGTH": 32767,
-    "VARNUM": 2,
     "LABEL": "Some Character Column",
     "FMTNAME": "",
     "FORMAT": "$32767.",
     "TYPE": "C",
-    "DDTYPE": "CHARACTER"
+    "DDTYPE": "C"
     },
     {
     "NAME": "SOME_DATE",
     "LENGTH": 8,
-    "VARNUM": 5,
     "LABEL": "Some Date",
     "FMTNAME": "DATE",
     "FORMAT": "DATE9.",
@@ -71,7 +67,6 @@ let webouts = {
     {
     "NAME": "SOME_DATETIME",
     "LENGTH": 8,
-    "VARNUM": 6,
     "LABEL": "SOME_DATETIME",
     "FMTNAME": "DATETIME",
     "FORMAT": "DATETIME19.",
@@ -81,37 +76,33 @@ let webouts = {
     {
     "NAME": "SOME_DROPDOWN",
     "LENGTH": 128,
-    "VARNUM": 3,
     "LABEL": "SOME_DROPDOWN",
     "FMTNAME": "",
     "FORMAT": "$128.",
     "TYPE": "C",
-    "DDTYPE": "CHARACTER"
+    "DDTYPE": "C"
     },
     {
     "NAME": "SOME_NUM",
     "LENGTH": 8,
-    "VARNUM": 4,
     "LABEL": "",
     "FMTNAME": "",
     "FORMAT": "8.",
     "TYPE": "N",
-    "DDTYPE": "NUMERIC"
+    "DDTYPE": "N"
     },
     {
     "NAME": "SOME_SHORTNUM",
     "LENGTH": 4,
-    "VARNUM": 8,
     "LABEL": "SOME_SHORTNUM",
     "FMTNAME": "",
     "FORMAT": "4.",
     "TYPE": "N",
-    "DDTYPE": "NUMERIC"
+    "DDTYPE": "N"
     },
     {
     "NAME": "SOME_TIME",
     "LENGTH": 8,
-    "VARNUM": 7,
     "LABEL": "SOME_TIME",
     "FMTNAME": "TIME",
     "FORMAT": "TIME8.",
@@ -3093,117 +3084,105 @@ let webouts = {
             {
                 "NAME": "DSN",
                 "LENGTH": 32,
-                "VARNUM": 3,
                 "LABEL": "Dataset Name (32 chars)",
                 "FMTNAME": "",
                 "FORMAT": "$32.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "IS_DIFF",
                 "LENGTH": 8,
-                "VARNUM": 9,
                 "LABEL": "Did value change? (1/0/-1).  Always -1 for appends and deletes.",
                 "FMTNAME": "",
                 "FORMAT": "8.",
                 "TYPE": "N",
-                "DDTYPE": "NUMERIC"
+                "DDTYPE": "N"
             },
             {
                 "NAME": "IS_PK",
                 "LENGTH": 8,
-                "VARNUM": 8,
                 "LABEL": "Is Primary Key Field? (1/0)",
                 "FMTNAME": "",
                 "FORMAT": "8.",
                 "TYPE": "N",
-                "DDTYPE": "NUMERIC"
+                "DDTYPE": "N"
             },
             {
                 "NAME": "KEY_HASH",
                 "LENGTH": 32,
-                "VARNUM": 4,
                 "LABEL": "MD5 Hash of primary key values (pipe seperated)",
                 "FMTNAME": "",
                 "FORMAT": "$32.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "LIBREF",
                 "LENGTH": 8,
-                "VARNUM": 2,
                 "LABEL": "Library Reference (8 chars)",
                 "FMTNAME": "",
                 "FORMAT": "$8.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "LOAD_REF",
                 "LENGTH": 36,
-                "VARNUM": 1,
                 "LABEL": "unique load reference",
                 "FMTNAME": "",
                 "FORMAT": "$36.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "MOVE_TYPE",
                 "LENGTH": 1,
-                "VARNUM": 6,
                 "LABEL": "Either (A)ppended, (D)eleted or (M)odified",
                 "FMTNAME": "",
                 "FORMAT": "$1.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "NEWVAL_CHAR",
                 "LENGTH": 32765,
-                "VARNUM": 14,
                 "LABEL": "New (character) value",
                 "FMTNAME": "",
                 "FORMAT": "$32765.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "NEWVAL_NUM",
                 "LENGTH": 8,
-                "VARNUM": 12,
                 "LABEL": "New (numeric) value",
                 "FMTNAME": "BEST",
                 "FORMAT": "BEST32.",
                 "TYPE": "N",
-                "DDTYPE": "NUMERIC"
+                "DDTYPE": "N"
             },
             {
                 "NAME": "OLDVAL_CHAR",
                 "LENGTH": 32765,
-                "VARNUM": 13,
                 "LABEL": "Old (character) value",
                 "FMTNAME": "",
                 "FORMAT": "$32765.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "OLDVAL_NUM",
                 "LENGTH": 8,
-                "VARNUM": 11,
                 "LABEL": "Old (numeric) value",
                 "FMTNAME": "BEST",
                 "FORMAT": "BEST32.",
                 "TYPE": "N",
-                "DDTYPE": "NUMERIC"
+                "DDTYPE": "N"
             },
             {
                 "NAME": "PROCESSED_DTTM",
                 "LENGTH": 8,
-                "VARNUM": 7,
                 "LABEL": "Processed at timestamp",
                 "FMTNAME": "E8601DT",
                 "FORMAT": "E8601DT26.6",
@@ -3213,22 +3192,20 @@ let webouts = {
             {
                 "NAME": "TGTVAR_NM",
                 "LENGTH": 32,
-                "VARNUM": 5,
                 "LABEL": "Target variable name (32 chars)",
                 "FMTNAME": "",
                 "FORMAT": "$32.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "TGTVAR_TYPE",
                 "LENGTH": 1,
-                "VARNUM": 10,
                 "LABEL": "Either (C)haracter or (N)umeric",
                 "FMTNAME": "",
                 "FORMAT": "$1.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             }
         ],
         "dsmeta": [
@@ -4426,12 +4403,12 @@ let webouts = {
     ]
     , "cols":
     [
-    {"NAME":"ALERT_DS" ,"LENGTH":32 ,"VARNUM":4 ,"LABEL":"ALERT_DS" ,"FMTNAME":"" ,"FORMAT":"$32." ,"TYPE":"C" ,"DDTYPE":"CHARACTER" }
+    {"NAME":"ALERT_DS" ,"LENGTH":32 ,"LABEL":"ALERT_DS" ,"FMTNAME":"" ,"FORMAT":"$32." ,"TYPE":"C" ,"DDTYPE":"C" }
-    ,{"NAME":"ALERT_EVENT" ,"LENGTH":20 ,"VARNUM":2 ,"LABEL":"ALERT_EVENT" ,"FMTNAME":"" ,"FORMAT":"$20." ,"TYPE":"C" ,"DDTYPE":"CHARACTER" }
+    ,{"NAME":"ALERT_EVENT" ,"LENGTH":20 ,"LABEL":"ALERT_EVENT" ,"FMTNAME":"" ,"FORMAT":"$20." ,"TYPE":"C" ,"DDTYPE":"C" }
-    ,{"NAME":"ALERT_LIB" ,"LENGTH":8 ,"VARNUM":3 ,"LABEL":"ALERT_LIB" ,"FMTNAME":"" ,"FORMAT":"$8." ,"TYPE":"C" ,"DDTYPE":"CHARACTER" }
+    ,{"NAME":"ALERT_LIB" ,"LENGTH":8 ,"LABEL":"ALERT_LIB" ,"FMTNAME":"" ,"FORMAT":"$8." ,"TYPE":"C" ,"DDTYPE":"C" }
-    ,{"NAME":"ALERT_USER" ,"LENGTH":100 ,"VARNUM":5 ,"LABEL":"ALERT_USER" ,"FMTNAME":"" ,"FORMAT":"$100." ,"TYPE":"C" ,"DDTYPE":"CHARACTER" }
+    ,{"NAME":"ALERT_USER" ,"LENGTH":100 ,"LABEL":"ALERT_USER" ,"FMTNAME":"" ,"FORMAT":"$100." ,"TYPE":"C" ,"DDTYPE":"C" }
-    ,{"NAME":"TX_FROM" ,"LENGTH":8 ,"VARNUM":1 ,"LABEL":"TX_FROM" ,"FMTNAME":"DATETIME" ,"FORMAT":"DATETIME19.3" ,"TYPE":"N" ,"DDTYPE":"DATETIME" }
+    ,{"NAME":"TX_FROM" ,"LENGTH":8 ,"LABEL":"TX_FROM" ,"FMTNAME":"DATETIME" ,"FORMAT":"DATETIME19.3" ,"TYPE":"N" ,"DDTYPE":"DATETIME" }
-    ,{"NAME":"TX_TO" ,"LENGTH":8 ,"VARNUM":6 ,"LABEL":"TX_TO" ,"FMTNAME":"DATETIME" ,"FORMAT":"DATETIME19.3" ,"TYPE":"N" ,"DDTYPE":"DATETIME" }
+    ,{"NAME":"TX_TO" ,"LENGTH":8 ,"LABEL":"TX_TO" ,"FMTNAME":"DATETIME" ,"FORMAT":"DATETIME19.3" ,"TYPE":"N" ,"DDTYPE":"DATETIME" }
     ]
     , "dsmeta":
     [
@@ -4520,67 +4497,60 @@ let webouts = {
             {
                 "NAME": "BASE_COL",
                 "LENGTH": 32,
-                "VARNUM": 4,
                 "LABEL": "BASE_COL",
                 "FMTNAME": "",
                 "FORMAT": "$32.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "BASE_DS",
                 "LENGTH": 32,
-                "VARNUM": 3,
                 "LABEL": "BASE_DS",
                 "FMTNAME": "",
                 "FORMAT": "$32.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "BASE_LIB",
                 "LENGTH": 8,
-                "VARNUM": 2,
                 "LABEL": "BASE_LIB",
                 "FMTNAME": "",
                 "FORMAT": "$8.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "RULE_ACTIVE",
                 "LENGTH": 8,
-                "VARNUM": 7,
                 "LABEL": "RULE_ACTIVE",
                 "FMTNAME": "",
                 "FORMAT": "8.",
                 "TYPE": "N",
-                "DDTYPE": "NUMERIC"
+                "DDTYPE": "N"
             },
             {
                 "NAME": "RULE_TYPE",
                 "LENGTH": 32,
-                "VARNUM": 5,
                 "LABEL": "RULE_TYPE",
                 "FMTNAME": "",
                 "FORMAT": "$32.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "RULE_VALUE",
                 "LENGTH": 128,
-                "VARNUM": 6,
                 "LABEL": "RULE_VALUE",
                 "FMTNAME": "",
                 "FORMAT": "$128.",
                 "TYPE": "C",
-                "DDTYPE": "CHARACTER"
+                "DDTYPE": "C"
             },
             {
                 "NAME": "TX_FROM",
                 "LENGTH": 8,
-                "VARNUM": 1,
                 "LABEL": "TX_FROM",
                 "FMTNAME": "DATETIME",
                 "FORMAT": "DATETIME19.3",
@@ -4590,7 +4560,6 @@ let webouts = {
             {
                 "NAME": "TX_TO",
                 "LENGTH": 8,
-                "VARNUM": 8,
                 "LABEL": "TX_TO",
                 "FMTNAME": "DATETIME",
                 "FORMAT": "DATETIME19.3",
@@ -5395,10 +5364,10 @@ let webouts = {
     const v = JSON.parse(webouts.MPE_X_TEST)
 
     v.cols.push(
-        { NAME: "READONLY_COL", LENGTH: 200, VARNUM: 11, LABEL: "READONLY_COL", FMTNAME: "", FORMAT: "$200.", TYPE: "C", DDTYPE: "CHARACTER" },
+        { NAME: "READONLY_COL", LENGTH: 200, LABEL: "READONLY_COL", FMTNAME: "", FORMAT: "$200.", TYPE: "C", DDTYPE: "C" },
-        { NAME: "HIDDEN_COL", LENGTH: 200, VARNUM: 12, LABEL: "HIDDEN_COL", FMTNAME: "", FORMAT: "$200.", TYPE: "C", DDTYPE: "CHARACTER" },
+        { NAME: "HIDDEN_COL", LENGTH: 200, LABEL: "HIDDEN_COL", FMTNAME: "", FORMAT: "$200.", TYPE: "C", DDTYPE: "C" },
-        { NAME: "ROUND_COL", LENGTH: 8, VARNUM: 13, LABEL: "ROUND_COL", FMTNAME: "", FORMAT: "BEST.", TYPE: "N", DDTYPE: "NUMERIC" },
+        { NAME: "ROUND_COL", LENGTH: 8, LABEL: "ROUND_COL", FMTNAME: "", FORMAT: "BEST.", TYPE: "N", DDTYPE: "N" },
-        { NAME: "NUMFMT_COL", LENGTH: 8, VARNUM: 14, LABEL: "NUMFMT_COL", FMTNAME: "", FORMAT: "BEST.", TYPE: "N", DDTYPE: "NUMERIC" }
+        { NAME: "NUMFMT_COL", LENGTH: 8, LABEL: "NUMFMT_COL", FMTNAME: "", FORMAT: "BEST.", TYPE: "N", DDTYPE: "N" }
     )
 
     v.viewdata = v.viewdata.map((row, i) => ({

sas/sasjs/macros/mpe_makedata.sas

@@ -2012,6 +2012,29 @@ insert into &lib..MPE_VALIDATIONS set
     ,rule_value='/t/'
     ,rule_active=1
     ,tx_to='31DEC5999:23:59:59'dt;
+/* test soft regex - PRIMARY_KEY_FIELD should be an integer (yellow if it
+contains a decimal point).  All generated keys are integers, so no
+warning shows until a demo user enters a decimal. */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0
+    ,base_lib="&lib"
+    ,base_ds="MPE_X_TEST"
+    ,base_col="PRIMARY_KEY_FIELD"
+    ,rule_type='SOFTREGEX'
+    ,rule_value='/^\d+$/'
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* test hard regex - SOME_SHORTNUM must not be between 1 and 5.  Generated
+data starts at 6, so nothing is blocked until a demo user enters 1-5. */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0
+    ,base_lib="&lib"
+    ,base_ds="MPE_X_TEST"
+    ,base_col="SOME_SHORTNUM"
+    ,rule_type='HARDREGEX'
+    ,rule_value='/^(?![1-5](\.\d+)?$).*/'
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
 insert into &lib..MPE_VALIDATIONS set
     tx_from=0
     ,base_lib="&lib"
@@ -2061,7 +2084,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=42
       ,some_datetime=42
       ,some_time=42
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=1
@@ -2071,7 +2094,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=42
       ,some_datetime=42
       ,some_time=422
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=2
@@ -2081,7 +2104,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=42
       ,some_datetime=42
       ,some_time=142
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=3
@@ -2093,7 +2116,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=423
       ,some_datetime=423
       ,some_time=44
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
   insert into &lib..mpe_x_test
     set primary_key_field=4
@@ -2103,7 +2126,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=4231
       ,some_datetime=423123123
       ,some_time=412
-      ,some_shortnum=3
+      ,some_shortnum=8
       ,some_bestnum=44;
 %do x=10 %to 500;
   insert into &lib..mpe_x_test
@@ -2114,7 +2137,7 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_date=round(ranuni(0)*1000,1)
       ,some_datetime=round(ranuni(0)*50000,1)
       ,some_time=round(ranuni(0)*100,1)
-      ,some_shortnum=round(ranuni(0)*100,1)
+      ,some_shortnum=6+round(ranuni(0)*94,1)
       ,some_bestnum=round(ranuni(0)*100,1);
 %end;
 

sas/sasjs/services/editors/getdata.sas

@@ -24,10 +24,10 @@
   <h5> cols </h5>
   Contains column level attributes.
   @li NAME - column name
-  @li VARNUM - var position. https://core.sasjs.io/mp__getcols_8sas.html
   @li LABEL - var label. https://core.sasjs.io/mp__getcols_8sas.html
   @li FMTNAME - derived format. https://core.sasjs.io/mp__getcols_8sas.html
-  @li DDTYPE - derived dropdown. https://core.sasjs.io/mp__getcols_8sas.html
+  @li DDTYPE - derived dropdown.  C=CHARACTER, N=NUMERIC, else DATE / TIME /
+    DATETIME. https://core.sasjs.io/mp__getcols_8sas.html
   @li CLS_RULE - values include:
     - EDIT - the column is editable
     - READ - the column should be readonly
@@ -488,14 +488,18 @@ select upcase(loadtype)
 /* extract col info */
 %mp_getcols(&libds, outds=cols1)
 
-/* join with cls rules */
+/* join with cls rules.  Trim COLS payload in the SQL below - DDTYPE
+  C=CHARACTER N=NUMERIC (dates kept in full), VARNUM dropped (client
+  relies on column order, not position) */
 proc sql;
 create table work.cols as
   select a.NAME
-    ,a.VARNUM
     ,coalesce(c.desc,a.NAME) as LABEL
     ,a.FMTNAME
-    ,a.DDTYPE
+    ,case a.DDTYPE
+      when 'CHARACTER' then 'C'
+      when 'NUMERIC' then 'N'
+      else a.DDTYPE end as DDTYPE
     ,case b.cls_hide
       when 1 then 'HIDE'
       when 0 then 'EDIT'
@@ -503,7 +507,7 @@ create table work.cols as
     ,c.memlabel
     ,c.longdesc
     ,d.colType
-  from work.cols1 a
+  from work.cols1(drop=varnum) a
   left join work.cls_rules b
   on a.NAME=b.CLS_VARIABLE_NM
   left join work.spec c

sas/sasjs/services/public/getcols.sas

@@ -33,7 +33,7 @@ data cols(keep=name type length varnum format label);
     else if formatl=0 then format=cats(format2,'.');
     else format=cats(format2,formatl,'.');
     type='C';
-    ddtype='CHARACTER';
+    ddtype='C';
   end;
   else do;
     if format2='' then format=cats(length,'.');
@@ -44,7 +44,7 @@ data cols(keep=name type length varnum format label);
     if format=:'DATETIME' then ddtype='DATETIME';
     else if format=:'DATE' then ddtype='DATE';
     else if format=:'TIME' then ddtype='TIME';
-    else ddtype='NUMERIC';
+    else ddtype='N';
   end;
   if label='' then label=name;
 run;

sas/sasjs/services/public/viewdata.sas

@@ -15,13 +15,12 @@
   <h4> Service Outputs </h4>
 
   <h5> cols </h5>
-  @li DDTYPE
+  @li DDTYPE - C=CHARACTER, N=NUMERIC, else DATE / TIME / DATETIME
   @li FORMAT
   @li LABEL
   @li LENGTH
   @li NAME
   @li TYPE
-  @li VARNUM
 
   <h5> sasparams </h5>
   @li FILTER_TEXT
@@ -361,11 +360,23 @@ run;
 )
 
 %mp_getcols(&libds, outds=cols1)
+
+/* trim COLS payload in the SQL below - DDTYPE C=CHARACTER N=NUMERIC
+  (dates kept in full), VARNUM dropped (client relies on column order,
+  not position) */
 proc sql;
-create table cols(drop=srclabel) as
+create table cols as
-  select a.*
+  select a.name
-      ,coalesce(b.dd_shortdesc,a.srclabel,a.name) as label
+      ,a.type
-  from cols1(rename=(label=srclabel)) a
+      ,a.length
+      ,a.format
+      ,a.fmtname
+      ,case a.ddtype
+        when 'CHARACTER' then 'C'
+        when 'NUMERIC' then 'N'
+        else a.ddtype end as ddtype
+      ,coalesce(b.dd_shortdesc,a.label,a.name) as label
+  from cols1 a
   left join &mpelib..mpe_datadictionary
       (where=(&dc_dttmtfmt. < tx_to
       and dd_source ? %upcase("&orig_libds")