chore(release): 7.15.0 [skip ci]

# [7.15.0](https://git.datacontroller.io/dc/dc/compare/v7.14.2...v7.15.0) (2026-09-23)

### Bug Fixes

* **mocks:** keep special missings in the DIFF, and add the clip recording spec ([d124ce3](d124ce3b36))
* **mocks:** list a column's own special missing in its dropdown ([b8f16a6](b8f16a6382))
* **mocks:** make the approval path work, and record the approval scene ([12847cf](12847cf071))
* **validator:** a range rule ignores a missing value ([70dae4b](70dae4b701))
* **validator:** keep the regular missing in a numeric dropdown source too ([a639bca](a639bca702))
* **validator:** primary keys are NOT NULL, and a strict dropdown can match a special missing ([586a41a](586a41ad49))
* **validator:** reject a special missing on a NOT NULL column ([6d85bce](6d85bce2a0))

### Features

* **validator:** compare MINVAL and MAXVAL in SAS's own order ([3467322](3467322e99))

.agent/skills/sas/SKILL.md

@@ -1,73 +0,0 @@
----
-name: sas
-description: >
-  Use this skill whenever writing, modifying, or debugging SAS code in this repository — SAS
-  macros, sasjs services, hooks, or test files (*.sas). Triggers include: SAS macro compilation
-  errors, parameter parsing issues, writing sasjs tests with mp_assert / mp_assertdsobs /
-  mp_assertscope, macro-quoting questions (%str, %nrstr, %superq), %mp_abort usage, or unexpected
-  behaviour when passing free-text values (descriptions, messages) to macros. Also covers repo
-  conventions such as running `sasjs lint` after touching .sas files and the maximum line length
-  rule.
----
-
-# SAS Development
-
-Conventions and pitfalls for SAS code in this repository (macros in `sas/sasjs/macros/`, services
-in `sas/sasjs/services/`, tests alongside them as `*.test.sas`).
-
-## Always `%str()` free-text macro parameters
-
-A comma inside a macro parameter value is parsed as a **parameter delimiter**. Free-text
-parameters such as `desc=` (in `%mp_assert`, `%mp_assertdsobs`, `%mp_assertscope`) and `msg=` (in
-`%mp_abort`) must be wrapped in `%str()` — even when they currently contain no comma, so later
-edits cannot silently break the call.
-
-Wrong — the text after the comma becomes an unexpected positional parameter and the compilation
-fails (or worse, is misparsed):
-
-```sas
-%mp_assert(iftrue=(&oldrows=0),
-  desc=Test 2 - all staged records loaded, delete flags ignored,
-  outds=work.test_results
-)
-```
-
-Right:
-
-```sas
-%mp_assert(iftrue=(&oldrows=0),
-  desc=%str(Test 2 - all staged records loaded, delete flags ignored),
-  outds=work.test_results
-)
-```
-
-The same applies to any macro parameter that carries user-facing text: `%mp_abort(msg=%str(...))`,
-`etlsource=` values containing punctuation (use `%superq()` when passing macro variables that may
-contain special characters), etc.
-
-Related quoting rules of thumb:
-
-- `%str()` masks commas, parentheses, semicolons and quotes at compile time — sufficient for
-  static text like `desc=`.
-- Use `%nrstr()` if the text must also mask `%` and `&` (rare in descriptions).
-- Use `%superq(var)` when forwarding a macro **variable** whose value may contain special
-  characters (e.g. `etlsource=` in `mpe_targetloader.sas`).
-
-## Tests must be idempotent
-
-A test file must pass when run repeatedly (including after a run that failed partway). Conventions:
-
-- Start the file with `%let syscc=0;` — many macros (e.g. `%mp_lockanytable(LOCK)`) abort on entry if `&syscc>0`, and any `WARNING` in a previous test bumps `syscc` to 4.
-- Clean up **all** persistent state, not just the obvious one: `MPE_TABLES` registrations, `MPE_LOCKANYTABLE` lock records (a run dying between LOCK and UNLOCK leaves a stale `LOCKED` row), physical tables in `dctest` (`proc datasets ... delete`), and global macro variables created via `select ... into:` (`%symdel`).
-- Make prep defensive: delete-then-insert config records (handles leftovers from an aborted run), and recreate physical tables rather than assuming they are absent.
-
-## Repo conventions for .sas files
-
-- Run `npx sasjs lint` from the `sas/` directory after creating or modifying any `.sas` file; the
-  files you touched must have zero warnings (pre-existing warnings in other files can be ignored).
-- Maximum line length is 80 characters (lint-enforced) — this is why long `msg=%str(...)` values
-  sometimes need shortening.
-- Test files are named `<thing>.test.sas` (or `<thing>.test.N.sas`) and run via
-  `npm run 4gl && sasjs test -t 4gl` from `sas/` — see `.agent/docs/testing.md`.
-- `sas/sasjsbuild/` is generated build output — never hand-edit it; edit sources under
-  `sas/sasjs/` only.

.agents/skills/dc-cypress/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: dc-cypress
+description: >
+  Use this skill when writing or debugging Cypress E2E specs in the dc repo, or when working on the
+  JS mock backend they run against. Covers the SASjs Server mock environment (deploy, seed, ports),
+  the spec conventions the CI harness expects, the viewer/editor DOM selectors that the existing
+  specs get wrong, the fixed 1280x720 video capture, and the rule that a mock service must mirror
+  the real SAS service it stands in for.
+---
+
+# Cypress E2E and the JS mock backend
+
+The specs under `client/cypress/e2e/` run against a mock backend: JS stored programs under `sas/mocks/sasjs/services/` executed by SASjs Server in JS-only mode. No SAS licence, no SAS server. CI (`.gitea/workflows/build.yaml`) builds the same environment - SASjs Server on :5000, `sasjs cbd -t server-ci`, makedata, then `ng serve` on :4200 - and runs a fixed list of specs against it.
+
+This skill covers the parts that are not obvious from reading a spec file: how to stand the environment up, the selectors that actually work, and the traps in the mock services. It does not cover the SAS-side tests (see `.agents/docs/testing.md` and the `dc-sas` skill).
+
+## When to Use
+
+- Adding or changing a spec under `client/cypress/e2e/`.
+- A mock service needs a new behaviour, or a spec "does nothing" because the mock ignores an input.
+- Recording a demo video from a spec run.
+- Debugging a spec that passes locally but fails in CI (usually a viewport or timing assumption).
+
+Don't use for: SAS backend tests (`sasjs test -t 4gl`), Angular unit tests (`npm run test:headless`), or general SAS development (use the `dc-sas` skill).
+
+## Stand up the mock environment
+
+1. **SASjs Server, JS mode.** Download the linux build from the sasjs/server releases, then create a `.env` next to the binary:
+
+       RUN_TIMES=js
+       NODE_PATH=node
+       CORS=enable
+       WHITELIST=http://localhost:4200
+
+   Start it (`./api-linux`). It runs in desktop mode on :5000 with no authentication.
+
+2. **Deploy the mocks.**
+
+       cd sas/mocks/sasjs && sasjs cbd -t server-ci
+
+   To start from a clean appLoc, delete it first:
+
+       curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
+
+3. **Seed the mock database.**
+
+       cd sas/mocks && sasjs request services/admin/makedata -t server-ci \
+           -d deploy/makedata.json -o ./makedata_out.json
+
+   `makedata` replies with HTML (it is normally called as a URL redirect), so the CLI reports `invalid Json string`. That is expected - check the drive instead.
+
+4. **Frontend.** Point the `sasjs` tag in `client/src/index.html` at the mock backend (`serverUrl="http://localhost:5000"`, `appLoc="/Public/app/dc"`, `serverType="SASJS"` - a local-only edit, the repo default is `serverUrl=""`), then `npx ng serve --host 0.0.0.0 --port 4200`. `client/cypress.env.json` needs `username`/`password` entries; any values work, `loginAndUpdateValidKey()` no-ops when there is no login form.
+
+5. **Run.**
+
+       cd client && npx cypress run --browser electron --spec cypress/e2e/<spec>.cy.ts
+
+   Only Electron is installed here (no system Chrome), so always pass `--browser electron`.
+
+## Spec conventions
+
+- Start the file with `export {}` so its top-level consts do not collide with other spec files under the TS type-checker.
+- `beforeAll`: `cy.visit(\`${hostUrl}/SASLogon/logout\`)` then `cy.loginAndUpdateValidKey(true)`. The `true` forces a licence key instead of relying on an earlier spec in the same run having applied one.
+- `beforeEach`: `cy.visit(hostUrl + appLocation)`, `visitPage('home')`, `visitPage('view/data')`. The `home` visit is what triggers the startup-data fetch; going straight to `view/data` leaves the nav tree empty in slower environments.
+- Declare helpers at the **bottom of each spec file** (that is the house pattern) - do not introduce a shared helper module.
+- Use `Cypress.env('longerCommandTimeout')` for waits and gate on `.app-loading` disappearing.
+- **No artificial delays.** These specs are the CI regression suite, so `cy.wait(n)` to "let a result land" is not acceptable - assert on the resulting state instead (Cypress retries assertions). Replace a sleep with a state assertion: e.g. after picking an option from a soft-select, wait for `#datalist_<inputId>` to have class `hidden` rather than sleeping.
+- Lint: `cd client && npm run lint:check` (prettier covers `cypress/e2e/*.cy.ts`).
+
+## Selectors that work
+
+- Viewer search box: `input[name="search-input"]` (Enter or the magnifier icon submits); the numeric toggle is `input[name="numeric_check"]`.
+- Row/column count in the header: `.title-col p` renders `(1,000 rows, 9 cols)` - and `(1 row, 9 cols)` when the result is a single row, so an assertion on the count has to pluralise.
+- Filter info bar: `.infoBar b` - only rendered when the backend's `sasparams.FILTER_TEXT` is neither blank, `' '`, nor `'1=1'`.
+- Empty result: `.noData h3` with `No data found with given conditions`.
+- Grid: `#hotTable .ht_master.handsontable .htCore tbody tr` (body rows), `#hotTable .ht_clone_top .htCore thead` (headers, and `button.changeType` confirms Handsontable has finished rendering them).
+- Nav tree: `.nav-tree clr-tree > clr-tree-node` -> `.clr-tree-node-content-container .clr-treenode-content p` (expands a library) -> `.clr-treenode-link` (table links).
+- **Viewer filter modal** - `.filterSide` (the options button) -> `.dropdown-menu` contains `Filter` -> `#vals_var_id0_0` (variable, pick from `#datalist_vals_var_id0_0 option`) and `#vals_0_0` (value) -> `.filter-modal button[aria-label="Ok"]`.
+  - `filtering.cy.ts` opens the filter with `.btnCtrl .btnView`, which is an **editor** control and does not exist in the viewer. Do not copy that helper into a viewer spec.
+  - The soft-select inputs drop a transparent click-catcher over the modal while their suggestion list is open, so Cypress reports the Ok button (and the next input) as covered by `.overlay`. Send the Escape keyup a user would send (`trigger('keyup', { key: 'Escape', force: true })`) and wait for the list to close before clicking on.
+
+## Recording a demo video
+
+`video: true` in `cypress.config.ts` is all that is needed. Run the spec and the MP4 lands in `client/cypress/videos/<spec>.mp4`.
+
+**The capture is the browser window's content area, and it contains the Cypress runner** - the command log takes the left ~450px and the app is rendered in the remaining pane, so the pane is `(capture width - 450) x (capture height - 96)` and the app is zoomed to fit it.
+
+That makes the window size the only lever that matters:
+
+- Electron is stuck at 1280x720 (it ignores `--window-size`), i.e. an 830x624 pane, and the repo's 1600x900 viewport renders at 50% - the app only occupies ~806x452 of it.
+- A real Chromium browser honours `--window-size`, so the capture can be sized to whatever the pane needs. `cypress.config.ts` passes it through from `RECORD_WINDOW_SIZE` (inert in CI, which sets no env).
+
+For a 16:9 pane, solve `(W - 450) / (H - 96) = 16/9`: W = 1920 gives H = 923 of window *content*, and Chrome adds ~87px of window chrome above that, so pass 1920x1010:
+
+    RECORD_WINDOW_SIZE=1920,1010 npx cypress run \
+      --browser /opt/data/profiles/dc/tools/chrome-linux64/chrome \
+      --spec cypress/e2e/full-table-search.cy.ts \
+      --env skipNegative=true,demoPause=1500
+
+That yields a 1920x922 capture with a 1470x826 pane (16:9) and, at the repo's 1600x900 viewport, an app at ~0.90 zoom occupying 1441x812 - about 1.9x the linear size of the 16:9 crop Electron can give. Crop the app out:
+
+    ffmpeg -ss <start> -i cypress/videos/<spec>.mp4 -vf "crop=1440:812:464:80" \
+      -an -c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p \
+      -movflags +faststart <spec>-cut.mp4
+
+Measure the crop rect from the frames rather than guessing: the pane's flat background is rgb(225,227,236), so a row/column scan for pixels that differ from it gives the app's bounding box (for the 1920x922 capture: runner pane ends x=449, app x=463-1903, app y=80-893; for the 1280x720 Electron capture: app x=468-1259, y=80-675). The app's own background is nearly the same grey, so verify the result on the first, middle and last frames rather than trusting a pixel test alone.
+
+To find where to cut, scan the mean *and* standard deviation of the content area: the app boots mid-run (a dark screen, mean ~65, sd ~2.5), the "Loading Table Viewer" spinner is flat and light (mean ~252, sd ~0), and the rendered table is mean ~245, sd ~31. Cut on the first table frame - cutting a fraction early shows the boot screen or the spinner.
+
+Cypress starts its own Xvfb for the run (a second display appears in `/tmp/.X11-unix`), so an external `DISPLAY` is ignored; `--force-device-scale-factor` and a CSS `zoom` also do not change the capture.
+
+### Grid columns that are clipped
+
+Handsontable sizes columns to content, so a narrow result can be *wider* than the grid viewport: the horizontal scrollbar appears and the rightmost column (NOTES on `MPE_X_SEARCH`) is cut off. The cell text is still in the DOM, so an assertion passes either way - the column only has to be scrolled into view for the *video* to show why the rows matched.
+
+    const scrollGridTo = (position: 'left' | 'right') => {
+      cy.get('#hotTable .wtHolder').first().then(($holder: any) => {
+        const el = $holder[0]
+        el.scrollLeft = position === 'right' ? el.scrollWidth : 0
+        expect(el.scrollLeft).to.equal(position === 'right' ? el.scrollWidth - el.clientWidth : 0)
+      })
+    }
+
+Setting `scrollLeft` is enough - Handsontable keeps the header clone in step - and the browser clamps it to `scrollWidth - clientWidth`, so that is what scrolling right has to land on. This is a legitimate spec step (it runs in CI), not a recording flag.
+
+### A longer, more readable demo
+
+A spec that doubles as a walkthrough should be a single `it()`: open the table once and make each variation a search in place. The recording then reads as a demo rather than a sequence of page loads, and it is far faster - every extra `it()` pays for the `beforeEach` navigation again.
+
+Two recording-only env flags, both inert in CI because CI passes no env:
+
+    const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
+    const negativeStep = (step: () => void) => {
+      if (!skipNegative) step()
+    }
+    const demoPauseMs = Number(Cypress.env('demoPause')) || 0
+
+    negativeStep(() => {
+      searchFor('trench')
+      assertNoData()
+    })
+
+- `skipNegative=true` drops the steps that legitimately match nothing (wrong case, partial number, value not present), so the recording never shows an empty screen.
+- `demoPause=1500` pauses 1.5s at each row-count assertion, so each result can be read. Apply it in one place (the row-count assertion, which every beat makes exactly once) rather than sprinkling `cy.wait()` through the test - CI still runs the file with no waits at all.
+
+More beats, not a slower take, is what makes a demo longer: a walkthrough of 12 row-count beats takes 52s against 30s for 7.
+
+## Mock services
+
+- One JS file per service under `sas/mocks/sasjs/services/**`, deployed as SASjs stored programs. Shared helpers live in `sas/mocks/sasjs/services/dcMockUtils.js` (`fetchTable`/`parseCsv` for `%webout(FETCH)`, `webOutOpen`/`webOutObj`/`webOutClose` for `%webout`, `mpeFilterMaster`, `mpFilterGenerate`, `mpFilterStore`, `getDdType`, `formatSasValue`, `sasVarsEntry`).
+- Mock data is one JSON file per table at `<drive>/files/<appLoc>/data/<LIBREF>/<table>.json`, written by `services/admin/makedata.js` (which also writes `services/settings.js`). The drive lives under the SASjs Server directory (`sasjs_root/drive`).
+- `makedata.js` self-destructs after a successful run (it deletes itself from the drive so the frontend can detect completion), so re-deploy before re-seeding.
+- **A mock must mirror the real service.** `viewdata.js` once ignored `FILTER_RK` and hard-coded `FILTER_TEXT` to blank, so the viewer's filter silently filtered nothing and its info bar never showed a clause, while the real `viewdata.sas` applies `%mpe_filtermaster` and then `%mp_searchdata`. If a mock is a deliberate stub, say so in a comment and make the divergence explicit - a mock that quietly drops an input turns an E2E spec into a test of nothing.
+
+## Pitfalls
+
+- `sasjs cbd -t server-ci` rebuilds the *mock services* and re-deploys the streamed web app from `client/dist`; it does not rebuild the Angular app.
+- Deleting the drive folder before deploying avoids stale services and stale data.
+- Captured `_webout` JSON, request logs and iteration snapshots belong in `tmp/` (gitignored), never in `sas/mocks/` - see the `dc-sas` skill.
+- The mock server has no auth, so `cy.visit` on `/SASLogon/logout` returns 404 - that is fine, Cypress does not fail on the status code.
+
+## Verification
+
+- `npx cypress run --browser electron --spec <specs>` - all green, and run it more than once when you have just removed a wait.
+- `cd client && npm run lint:check` passes.
+- For a mock change, hit the service directly before writing UI assertions:
+
+      curl -sS -H "Content-Type: application/json" -X POST \
+        --data '{"_program":"/Public/app/dc/services/public/viewdata","SASControlTable":"LIBDS:$41. FILTER_RK:best. SEARCHTYPE:$4 SEARCHVAL:$1000\nDC_JSLIB.MPE_X_SEARCH,0,CHAR,Trench"}' \
+        http://localhost:5000/SASjsApi/stp/execute
+
+  then check `sasparams[0].NOBS`, the row count and `FILTER_TEXT`. Pin those exact numbers in the spec.

.agents/skills/dc-sas/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: dc-sas
+description: >
+  Use this skill alongside the sasjs/skills collection (sas, sasjs-core, sasjs-cli) when working on
+  Data Controller SAS code. Covers the repo-specific SAS knowledge that is NOT in those skills —
+  DC test-state conventions (MPE control tables, dctest, %let syscc) and the chunked-deploy
+  strategy for the huge generated build script (sasjsbuild/viya.sas). For general SAS language,
+  @sasjs/core macro, and @sasjs/cli guidance, rely on the sas, sasjs-core and sasjs-cli skills.
+---
+
+# SAS Development in Data Controller
+
+The sas, sasjs-core and sasjs-cli skills cover general SAS language and SASjs conventions (macro
+quoting, `%mp_assert*` testing style, `sasjs lint`, `sasjs run`, `sasjsbuild/` being generated
+output). These notes only cover the parts specific to Data Controller.
+
+
+## Tests must be idempotent
+
+A test file must pass when run repeatedly (including after a run that failed partway).
+
+- Clean up **all** persistent DC state, not just the obvious one: `MPE_TABLES` registrations,
+  `MPE_LOCKANYTABLE` lock records (a run dying between LOCK and UNLOCK leaves a stale `LOCKED`
+  row), physical tables in `dctest` (`proc datasets ... delete`), and global macro variables
+  created via `select ... into:` (`%symdel`).
+
+## Chunked deploy to Viya
+
+The full DC build script (`sasjsbuild/viya.sas`) is very large (11+ MB / 300k+ lines). The Viya
+compute API can hang or fail when submitting a file this size in a single request.
+
+**Chunked deploy.** Split the build script at `%let path=<folder>` boundaries — these are clean
+section breaks between service/test folders (always preceded by blank lines, never inside a data
+step). Each sub-chunk must be prepended with the shared header (the first ~3827 lines containing
+macro definitions, `appLoc` setup, and logging options:
+
+`options ps=max nonotes nosgen nomprint nomlogic nosource2 nosource noquotelenmax;`
+
+Logging is already suppressed in the header — no extra action needed. Keep each chunk under ~1.5
+MB. The `tests/macros`, `tests/services`, and streaming-app tail sections are the largest and may
+need further sub-splitting at `%let service=` boundaries.
+
+When splitting mid-folder (between services in the same folder), the sub-chunk that starts
+mid-folder must re-assert `%let path=<folder>;` before its first `%let service=` line — otherwise
+`&path` is unresolved and the `%mv_createwebservice(path=&appLoc/&path, ...)` call will abort with
+a recursive reference error.
+
+Deploy each chunk in order with `sasjs run <chunk>.sas -t <target>`. After all chunks are
+deployed, run `sasjs request services/admin/makedata -d deploy/makeDataViya.json -t <target>` to
+create the database.
+
+**Non-fatal errors.** `ERROR: Unauthorized` on `/dataSources/providers/Compute/` lines in the
+log are expected for non-admin Viya users — they come from the Data Sources API during
+lineage/catalog refresh and do not affect the deploy.
+
+## Repo conventions for .sas files
+
+- Test files are named `<thing>.test.sas` (or `<thing>.test.N.sas`) and run via
+  `npm run 4gl && sasjs test -t 4gl` from `sas/` — see `.agents/docs/testing.md`.
+- `sas/sasjsbuild/` is generated build output — never hand-edit it; edit sources under
+  `sas/sasjs/` only.
+
+## Temp and scratch files stay out of the repo
+
+Transient artifacts from manual test runs against sasjs-server — captured `_webout` JSON
+responses (e.g. `*_out.json`), request/response logs (e.g. `*.log`), and iteration snapshots
+(`vd_out2.json`, `stage_check2.json`, etc.) — must not be left in the working tree. They
+clutter `git status`, are never referenced by source or tests, and easily leak local
+filesystem paths (the sasjs-server session dir, `/home/.../sasjs_root/sessions/.../`) into
+committed history.
+
+Write them to a gitignored tmp location instead. The repo already provides two:
+
+- `tmp/` (gitignored at repo root) — general scratch space.
+- `sas/tmp/` (gitignored via the root `tmp/` rule) — for SAS-side artifacts.
+
+If a helper script dumps captured output for comparison during a debug session, point it at
+`tmp/` (or `sas/tmp/` for SAS-local paths) rather than alongside the tracked mocks in
+`sas/mocks/`. The tracked mock fixtures live under `sas/mocks/sas9/` and
+`sas/mocks/sasjs/services/`; loose `*_out.json` / `*.log` files at the `sas/mocks/` root are
+not mocks, they are leftover captures — remove them with `git clean -fd sas/mocks/` or
+write to `tmp/` in the first place.

.agents/skills/sas/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: sas
+description: Expert guidance for the SAS programming language — DATA step, PROC SQL, macro language, formats, ODS, and common procedures. Pure SAS syntax only, no SASjs-framework content. Use when writing, reviewing, or debugging .sas programs, or answering SAS language questions.
+---
+
+# SAS Language
+
+Write idiomatic, production-quality SAS code. This skill covers the SAS language itself, independent of any framework.
+
+## Scope
+
+- DATA step programming (SET/MERGE/BY, RETAIN, arrays, DO loops, hash objects, first./last. processing)
+- PROC SQL (joins, subqueries, views, pass-through with CONNECT TO)
+- Macro language (%macro/%mend, macro variables, %local/%global, conditional %IF logic, quoting functions %STR/%NRSTR/%BQUOTE, CALL SYMPUTX)
+- Common PROCs: SORT, MEANS/SUMMARY, FREQ, TRANSPOSE, REPORT, PRINT, CONTENTS, DATASETS, FORMAT, IMPORT/EXPORT
+- Formats and informats (user-defined via PROC FORMAT, picture formats)
+- ODS output (HTML, PDF, RTF, Excel, OUTPUT destination)
+- File I/O: LIBNAME, FILENAME, INFILE/INPUT, FILE/PUT
+
+## Non-negotiable: no WARNINGs
+
+Generated SAS code must run cleanly — **zero WARNINGs (and zero ERRORs) in the log**. Treat every `WARNING:` as a defect: uninitialized variables, implicit type conversions, truncation notes that should be warnings, "no observations", unresolved macro references, etc. If a warning is truly unavoidable, suppress it deliberately (e.g. an explicit option) and comment why. Likewise avoid superfluous NOTEs where reasonable.
+
+## Style rules
+
+These follow the @sasjs/core coding standards — apply them to all SAS code:
+
+- One statement per line; indentation = 2 spaces, no tabs, no trailing whitespace
+- Lines no longer than 80 characters; unix (LF) line endings; UTF-8
+- Avoid non-ASCII / special characters entirely — maximum compatibility across SAS installations and encodings
+- Always end steps with `run;`; for `proc sql` (and CAS-connected procs) `quit;` is essential to avoid `WARNING: You cannot disconnect or terminate session ...` on Viya
+- All dataset references must be 2-level (`work.blah`, not `blah`) — protects against `DATASTMTCHK=ALLKEYWORDS` and an active `USER` library
+- Explicit `length` / `attrib` for character variables rather than relying on defaults (avoids implicit length=8 truncation)
+- Use literal suffixes for clarity (`'01JAN2020'd`, `'12:30't`)
+- Prefer `proc sort` with `nodupkey` over manual dedup logic
+- Macros:
+  - Define with parentheses, even with no parameters: `%macro x();` not `%macro x;`
+  - Closing `%mend;` must repeat the macro name
+  - Macro calls are not terminated with a semicolon: `%my_macro()` not `%my_macro();`
+  - Mandatory parameters positional; optional parameters keyword (`var=`) style
+  - Macro names lowercase, verb-noun convention
+  - Macro variables without trailing dot (`&var` not `&var.`) unless needed to prevent incorrect resolution
+  - Macro variable NAMES are case-insensitive: `&Foo`, `&FOO`, and `&foo` all resolve to the same symbol (same for `%symexist`/`%superq`/`symget` arguments, which take a NAME not a value). Don't chase case mismatches as a bug — it's never the cause.
+  - ALL macro variables must be `%local` unless deliberately global (globals should use an application prefix to avoid collisions); use `call symputx` (not `symput`) in DATA steps
+  - Comment with `/* */` inside macros (asterisk comments are compiled into the macro)
+  - Guard macro logic with `%length(&var)=0` checks rather than `&var=` (empty comparisons are unsafe)
+- Avoid naming collisions: use `%sysfunc`-/`&syslast`-based work tables (e.g. `data &output; set &syslast; run;`) rather than hard-coded names
+- No open (non-macro) conditional code: wrap platform-branching or conditionally-executed blocks (e.g. `%if %mf_getplatform()=VIYA %then %do; ... %end;`) in a `%macro ... %mend` and invoke the macro. Open `%if` at program level fails in some execution contexts (job/scheduler/test harnesses) and hides scope leaks.
+
+## Portability awareness
+
+- Note when code differs between SAS 9.4 and Viya (e.g. CAS actions vs procs, `proc casutil` for sashdat loading, no X command on locked-down servers)
+- Avoid hard-coded physical paths and engine-specific options unless asked
+- No open macro code with if/else logic: wrap branching blocks in `%macro ... %mend` and call them — open `%if`/`%else` does not behave as expected in all SAS environments.
+
+## Common pitfalls to flag when reviewing
+
+- Unintended many-to-many MERGEs
+- Automatic variable `_ERROR_` / implicit RETAIN surprises
+- Macro timing issues: referencing `&macrovar` before it exists, `%if` evaluating data-step variables (use `if`/`symget` instead)
+- Automatic macro variables (`&syscc`, `&syswarningtext`, `&syserrortext`, `&sysdate`, `&sysuserid`, etc.) are READ-ONLY — attempting to overwrite them (e.g. `%let syswarningtext=;` or `call symput('syscc',...)`) raises `ERROR: Unable to assign value to a macro variable that is read only` (or similar). Never try to "clear" them.
+- Truncation from implicit length=8 on first assignment
+- `proc sql` cartesian product warnings

.agents/skills/sasjs-adapter/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: sasjs-adapter
+description: Frontend/Node integration with SAS backends using @sasjs/adapter — configuring the SASjs class, authentication (SAS 9, Viya, SASjs server), requests with input/output tables, file upload, and session management. Use when writing TypeScript/JavaScript that calls SAS services or jobs.
+---
+
+# @sasjs/adapter
+
+`@sasjs/adapter` is the TypeScript library for calling SAS services/jobs from browsers or Node, with a unified API across three server types: `SAS9`, `SASVIYA`, `SASJS`.
+
+## Basic setup
+
+```ts
+import SASjs from '@sasjs/adapter'
+
+const sasjs = new SASjs({
+  serverUrl: 'https://sas.example.com',
+  serverType: 'SASVIYA',      // SAS9 | SASVIYA | SASJS
+  appLoc: '/Public/app/myapp', // root folder of deployed services
+  contextName: 'SAS Job Execution compute context', // Viya only
+  debug: false
+})
+```
+
+## Executing a request
+
+```ts
+const response = await sasjs.request('services/common/getdata', {
+  mytable: [{ col1: 'value', col2: 42 }]   // input tables as JS arrays of objects
+})
+// response.result contains output tables sent back from SAS (_webout JSON)
+```
+
+- Input tables become SAS datasets via the `sasjs_tables` mechanism (work tables named after the JS keys).
+- The SAS service must write JSON to `_webout` — conventionally with the `mp_jsonout` macro from @sasjs/core, wrapped in `proc stp`-style begin/end macros.
+- Responses follow the `SASjsRequest`/`SASjsResponse` types; check `response.result` for tables and `response.log` where available.
+
+## Authentication
+
+- **SAS 9**: `sasjs.logIn(username, password)` (form-based against the stored process server). Session cookie is managed automatically.
+- **Viya**: OAuth client/secret (client credentials grant) or authorization code flow; tokens are refreshed automatically. Configure via CLI (`sasjs add cred`) for Node usage.
+- **SASJS server**: token-based auth against the sasjs/server API.
+
+## Key classes / modules
+
+- `SASjs` — main facade: `request()`, `logIn()/logOut()`, `uploadFile()`, `executeScript()`
+- `SessionManager` — Viya compute session lifecycle
+- `ContextManager` — Viya compute context selection
+- `SASViyaApiClient` / `SAS9ApiClient` / `SASjsApiClient` — low-level per-platform clients (rarely needed directly)
+- `file/` utilities — file upload to SAS (binary content handling)
+
+## Tips
+
+- Set `debug: true` to surface the SAS log in responses while developing.
+- Always handle `response.status` / error responses — SAS-side errors (e.g. from `%mp_abort`) come back in the JSON, not necessarily as HTTP errors.
+- For large payloads prefer CSV upload or streamed files over JSON input tables.
+- Keep `appLoc` consistent with the `appLoc` in `sasjsconfig.json` used to deploy.
+
+## Important: request() inputs are ALWAYS tables
+
+Every key in the `data` object of `sasjs.request(path, data)` is serialized via the `sasjs_tables` CSV mechanism and arrives in SAS as a **work dataset named after the key** — even scalar values. You cannot pass ad-hoc macro variables this way; services must read inputs from the work table (e.g. `data _null_; set work.config; call symputx('rootdir', rootdir); run;`). Output column names in `response.result.<table>` come back UPPERCASE (SAS dataset semantics).
+
+## Using the adapter without a bundler (zero-build / strict CSP frontends)
+
+The package root `index.js` is a UMD bundle exposing a global `SASjs`. Pattern (from the minimal seed app):
+
+1. `"prepare": "cp node_modules/@sasjs/adapter/index.js src/sasjs.js"` in package.json (runs on `npm i`).
+2. `<script src="sasjs.js"></script>` before your app script.
+3. Configure via a hidden custom element: `<sasjs serverType="SASJS" appLoc="/Public/app/myapp" debug="false"></sasjs>` and read attributes with `document.querySelector('sasjs')`. When the app is streamed by SAS itself, omit `serverUrl` — same-origin requests just work (CSP `default-src 'self'` safe).

.agents/skills/sasjs-cli/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: sasjs-cli
+description: Using the SASjs CLI (@sasjs/cli) to create, compile, build, deploy, run, and test SASjs projects against SAS 9, Viya, and SASjs server targets. Use for any `sasjs <command>` usage, CI/CD pipelines, target/auth config, sasjsconfig.json, service packs, or frontend streaming builds.
+---
+
+# @sasjs/cli
+
+The SASjs CLI (`npm i -g @sasjs/cli`, invoked as `sasjs`) automates compiling, building, and deploying SAS projects. All commands support `-t <target>` to select a target from `sasjsconfig.json`.
+
+## Targets and auth
+
+- A **target** = `{ name, serverUrl, serverType, appLoc }`. `serverType` is one of `SAS9`, `SASVIYA`, `SASJS`.
+- Credentials: `sasjs add cred` (or `.env` file). Viya uses client/secret **or** `sasjs auth login` (user/pass, no client/secret needed — see below); SAS 9 uses user/pass; SASJS server uses an access token.
+- `sasjs context` manages Viya compute contexts; `sasjs add target` adds a new target.
+
+## Core workflow
+
+```
+sasjs create myapp        # scaffold a new app (templates available)
+sasjs compile             # gather macros/services/jobs into per-file build outputs (sasjsbuild/)
+sasjs build               # produce deployable artefacts: JSON + .sas per target
+sasjs deploy              # deploy compiled/built artefacts to the target server
+sasjs cbd                 # compile + build + deploy in one step (-t viya etc.)
+```
+
+## Command reference
+
+| Command | Purpose |
+|---|---|
+| `sasjs create / init` | Scaffold new app or add SASjs to existing repo |
+| `sasjs compile` | Resolve dependencies (`<h4> SAS Macros </h4>` etc.) into `sasjsbuild/` |
+| `sasjs build` | Create build JSON / service pack per target |
+| `sasjs deploy` / `cbd` | Deploy to server (servicepack or direct) |
+| `sasjs run <file.sas>` | Execute an arbitrary SAS file on the server, return log |
+| `sasjs request <path>` | Execute a deployed service/job with input data (`-d`) |
+| `sasjs job execute` | Run a deployed job |
+| `sasjs flow execute` | Run a sequence of jobs with dependencies (CSV-defined flows) |
+| `sasjs servicepack deploy` | Deploy from a JSON service pack |
+| `sasjs web` | Build the frontend and stream it into the SAS web root (streamConfig) |
+| `sasjs db` | Build database DDL/data load scripts from `sasjs/db` |
+| `sasjs doc` | Generate doxygen documentation + lineage |
+| `sasjs folder create/delete/move` | Manage folders in SAS metadata / SAS Drive |
+| `sasjs fs` | File-system operations against the server (sync folders) |
+| `sasjs test` | Execute tests defined in `sasjs/tests` with init/term programs |
+| `sasjs lint` | Lint `.sas` files per `.sasjslint` config |
+| `sasjs version` | Print/set version info |
+
+## Conventions
+
+- Test coverage is generated only from a `sasjs compile` (or `sasjs c`). It accepts a target (`-t <target>`), but nothing is deployed to that target — compilation and coverage are fully local/offline, so no server needs to be available or reachable. Missing macro dependencies (e.g. `mp_ds2csv.sas`) mean `@sasjs/core` isn't installed — run `npm i` first.
+
+- Dependencies are declared in doxygen headers: `<h4> SAS Macros </h4>`, `<h4> SAS Files </h4>`, `<h4> SAS Folders </h4>`, and `@li item` entries — the CLI builds the dependency tree from these.
+- `sasjs compile` output goes to the `sasjsbuild/` folder (git-ignore it); `sasjsresults/` holds test/run outputs.
+- CI/CD: `sasjs cbd -t viya` is the standard deploy step; combine with `sasjs servicepack deploy` for artefact-based releases.
+- Exit codes are non-zero on failure — safe for pipelines.
+
+## Gotchas
+
+- Run `npm i` before `sasjs cb` — macro dependency resolution needs `node_modules/@sasjs/core` present, and `@sasjs/core` (and `@sasjs/adapter` if used) must be listed in `package.json`.
+- Credentials files are per-target: `.env.<targetname>` (e.g. `.env.server`) with `CLIENT`, `ACCESS_TOKEN`, `REFRESH_TOKEN`. Never commit them — gitignore `.env*`.
+
+## Viya auth without a client/secret (`sasjs auth login`)
+
+`sasjs auth login -t <target>` authenticates with a regular SAS username/password via the OAuth2 password grant against the built-in, secret-less `sas.cli` public client. No admin-registered OAuth client is needed — the fastest way to get `sasjs run`/`deploy` working on dev/demo estates. The password is never stored; the minted ACCESS_TOKEN/REFRESH_TOKEN pair is persisted to `.env.<target>` (local) or `~/.sasjsrc` (global) and verified via `/identities/users/@currentUser` (`Logged in as <id> (<name>)`). Bare `sasjs auth` is still an alias for `sasjs add cred`.
+
+- Token expiry: the CLI silently refreshes via the stored refresh token (works with and without a client/secret), and re-persists the rotated pair — Viya refresh tokens are **single-use/rotating**, so this persistence is what keeps later invocations working. If refresh fails, re-run `sasjs auth login`.
+- Some estates give `sas.cli` a short access-token TTL (e.g. 1h); a refresh on most invocations is normal.
+- Opaque (non-JWT) tokens are treated as usable — the server is the authority on expiry.
+- Limitations: local/LDAP accounts only (no SSO/SAML/MFA estates); password grant must be enabled for `sas.cli` (default on Viya 3.5+/4); ROPC is deprecated in OAuth 2.1 — use a registered client/secret for CI/production.
+- `sasjs run` 403 on session creation = the account isn't authorised for the configured compute context — set `contextName: "SAS Studio compute context"` on the target. First run on a cold estate can take many minutes (compute pod spin-up) and may appear to hang.
+- Self-signed estates: use `--insecure` on `auth login`, or configure `httpsAgentOptions` on the target.
+
+## Viya streaming apps (streamConfig.streamWeb)
+
+With `streamWeb: true`, `sasjs web`/`cbd -t viya` deploys the frontend **into SAS Files Service**: a streaming job at `<appLoc>/services/<streamServiceName>.html` serves `index.html`, and assets land in `<appLoc>/services/web/...`. At build time every asset/script/css URL in the HTML is rewritten to `/SASJobExecution?_FILE=<appLoc>/services/web/...` and the adapter config element (`<sasjs apploc=...>`) is stamped with the target `appLoc`.
+
+**Consequence: the app only works when served from the exact `appLoc` it was deployed against.** If the streaming HTML is placed anywhere else (e.g. manually uploaded to a user home folder like `/Users/<id>/myapp/...`), all rewritten `/Public/app/...` asset links 404 and the adapter calls the wrong service paths — the page renders with no CSS/JS and no backend. Fix by redeploying.  The original build (`sasjs cb`) has the apploc from the sasjsconfig.json - when the app is deployed as a SAS program, the supplied `%let apploc = ` (runtime value) is swapped with the `compiled_apploc` (build time value) to allow apps to be dynamically deployed to a given apploc at deploy time.
+
+### Verifying a Viya deployment headlessly (no browser)
+
+1. Get a token (password grant works out of the box with the built-in `sas.ec` client, empty secret):
+   `curl -X POST <server>/SASLogon/oauth/token -u 'sas.ec:' -d 'grant_type=password&username=U&password=P'`
+2. Fetch the app: `GET /SASJobExecution/?_FILE=<appLoc>/services/<name>.html` with `Authorization: Bearer` → expect `200` and the full `index.html`. `401` = auth, anything else = not deployed there.
+3. Fetch each asset the HTML references. Response tells you what's wrong:
+   - `200` with file content → asset deployed correctly.
+   - `202` + a tiny plain-text body (e.g. `Parameter Error\nFile error`) → **file does not exist at that Drive path** (classic appLoc-mismatch symptom). Note `_FILE` responses can be async: add `&_action=wait` to get content synchronously.
+   - Services: `POST /SASJobExecution/?_program=<appLoc>/services/<svc>&_action=wait` → `Parameter Error / Unable to get job definition` means the service was never deployed as a JES job (files on Drive alone are not enough — only `sasjs deploy`/`cbd`/`servicepack deploy` registers them).
+
+### Talking to SAS — ALWAYS use the adapter or the CLI
+
+When executing a SAS service or job for any purpose (debugging, reproduction, CI), **always go through the `@sasjs/adapter` or `sasjs` CLI** — never hand-roll curl against `SASJobExecution`. The adapter and CLI handle the things that are trivial to get wrong by hand: the input-data CSV format (space-separated `name:$format.` headers, CRLF, double-quoted special values, `%nrstr(...)` wrapping), the execution-mode routing (`_executionTasks=true` reads `sasjs<N>data` as **macro variables**, not file uploads), the `_debug`/`>>weboutBEGIN<<`/`>>weboutEND<<` wrapper parsing, the token refresh, and the `_contextname` URL param. A hand-built comma-separated CSV will read as blanks, silently skip guarded macro blocks, and send you down a rabbit hole of phantom bugs.
+
+- Build input data as a JSON file: `{ "<table>": [{ "<col>": "<val>", ... }] }`, e.g. `{ "config": [{ "rootdir": "/export/...", "runastask": "true", "usecomputeapi": "null", "contextname": "Compute Reusable" }] }`.
+- Run: `sasjs request '<appLoc>/services/common/<svc>' -t viya -d data.json -l <path>.log -o <path>.json`. The CLI hardcodes `debug: true`, so the `-l` flag always captures the full MPRINT/NOTE/`&syscc` log; `-o` saves the parsed webout. **Always pass `-l`** — reproducing a bug without the log means re-running the whole thing.
+- The Viya Folders/JES REST APIs ARE fine to hit directly with curl (token + `Authorization: Bearer`) for folder/file/member management and for `GET /SASJobExecution/?_FILE=...` asset checks — just not for *executing your own services with input data*.
+
+#### Adapter execution-context and input-data facts
+
+- The adapter automatically appends `_contextname=<value>` as a **URL parameter** to every Viya JES request (you do NOT need to pass it yourself). BUT JES request params are **NOT auto-promoted to SAS macro variables** — `%symexist(_contextname)` is false inside the service. If the service needs the chosen context name (e.g. to stamp it into the streamed HTML), pass it in the **input data table** (a `contextname` column) and read it with `call symputx` — that is the reliable channel.
+- The target's `contextName` in `sasjsconfig.json` decides which compute context the service runs under (and thus the `runAs` identity). The adapter's URL `_contextname` param is the same value. To reproduce a service under a batch/reusable context via `sasjs request`, set the target's `contextName` to that reusable context (e.g. `Compute Reusable`, runAs=sasbatch) — otherwise it runs as your own identity and write-test steps to batch-owned folders fail with `User does not have appropriate authorization level`.
+- Adapter CSV format (so you can read the `NOTE: The infile ... is:` RULE in the log correctly): row 1 is the header with `name:$informat.` pairs **space-separated**; data rows are **comma-separated** (CRLF), with values double-quoted only if they contain a special char (`,`, `"`, tab, newline). The sasjs/core webout reader reads it back with `dsd` + `firstobs=2` + an `input <name>:$informat.;` statement derived from the header.
+- `_executionTasks=true` (runAsTask) changes how input data arrives: as `sasjs<N>data` **macro variables** (chunked into `sasjs<N>data0..N`), NOT as `_WEBIN_FILE` uploads. The core webout `mv_webout` macro handles both, but it branches on `_EXECUTIONTASKS` — be aware when reading logs.
+
+### Redeploying cleanly on Viya (the 409 Conflict problem)
+
+Re-running `sasjs cbd`/`sasjs run viya.sas` against an **existing** appLoc often fails mid-deploy with `409 Conflict` (and an `mp_abort` → `abort cancel`): `mv_createfile` DELETEs the old file id then tries to recreate it, but when an intermediate **folder** already exists (e.g. `<appLoc>/services/web/js`) the recreate step conflicts and the whole deploy aborts — leaving a half-deployed app (services present, some assets missing). The `?recursive=true` folder DELETE also returns `409 You cannot delete the folder because it is not empty`, and individual member/folder DELETEs can return `403` even as the owner, so you cannot easily tear the tree down by hand.
+
+The reliable workaround is to **move the top appLoc folder out of the way** and redeploy to the original path — the deploy creates a fresh folder tree with no conflicts, and the old folder is retained as a backup. This is far faster and more reliable than fighting per-member deletes, and is the recommended pre-deploy step for any non-CI redeploy on Viya. Two ways to move it:
+
+- **Rename in place** (simplest): `PATCH /folders/folders/{id}` with `{"name":"<old>.bak.<ts>","version":2}` — frees the original name, keeps the old folder as a sibling backup.
+- **MOVE into a backup parent** (tidier — keeps all backups in one place): create (once) a backup parent folder e.g. `/Users/<id>/macrodash-backups`, then `POST /folders/folders/{backupParentId}/members/{appLocFolderId}?action=move` moves the whole tree (with contents) into it, freeing the original path for the fresh deploy.
+
+(On sasjs/server just `sasjs fs delete` the appLoc first — Drive there supports clean recursive deletes.)
+
+JES applies its own CSP header when streaming (includes `unsafe-inline`/`unsafe-eval`); a strict CSP meta tag in the app's HTML still applies and is the one that matters for the app code.

.agents/skills/sasjs-core/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: sasjs-core
+description: Standards and conventions for the @sasjs/core SAS macro library (mf_*, mp_*, mm*, ms_*, mv_* macros). Use when writing or editing SAS macros in a sasjs/core-style repo, picking an existing macro over reinventing one, or the sasjs/core build, lint, doxygen, and testing conventions.
+---
+
+# @sasjs/core — SAS Macro Library
+
+@sasjs/core is an MIT-licensed library of production-quality SAS macros for SAS application development, portable across SAS 9 (meta), Viya, and SASjs server.
+
+## Coding standards (mandatory)
+
+- One macro per file; filename must match the macro name (lowercase, no spaces)
+- Macro definitions must use parentheses: `%macro x();` not `%macro x;`
+- Macro *calls* are NOT terminated with a semicolon: `%my_macro()` not `%my_macro();`
+- All macro variables must be declared `%local` to prevent scope leakage
+- Always use `mf_getuniquefileref` when assigning filerefs, and `mf_getuniquelibref` when assigning librefs (never hardcode or hand-roll unique references)
+- 2-space indentation, no tabs, no trailing spaces, no invisible characters, max line length 300 (hard lint limit) but keep lines to 80 chars max where possible
+- Every file must have a Doxygen header:
+
+```sas
+/**
+  @file
+  @brief One-line description of the macro
+
+  <h4> SAS Macros </h4>
+  @li mf_othermacro.sas
+
+  @param [in] paramname Description
+  @param [out] outparam Description
+
+  <h4> Related Macros </h4>
+  @li mp_related.sas
+
+  @version 9.4
+  @author Your Name
+**/
+```
+
+## Folder / prefix conventions
+
+| Folder | Prefix | Platform |
+|---|---|---|
+| `base/` | `mf_` (function-style), `mp_` (procedure-style) | All platforms |
+| `meta/` | `mm_` | SAS 9 metadata |
+| `metax/` | `mmx_` | SAS 9 metadata (OS command dependent) |
+| `viya/` | `mv_` | Viya |
+| `server/` | `ms_` | SASjs server |
+| `xplatform/` | `mx_` | Runtime platform detection |
+| `fcmp/`, `lua/`, `ddl/` | — | PROC FCMP functions, LUA wrappers, DDL |
+
+Use `mf_` macros when the macro returns a value usable in an expression; use `mp_` for procedural macros that generate code/statements.
+
+**Cross-suite rule:** `mp_` macros must never reference `mx_` macros. Platform dispatching (SAS 9 / Viya / SASjs server) belongs in the `mx_` suite, which delegates to `ms_`/`mv_`/PROC STP per platform. If an `mp_` macro seems to need platform-specific behaviour, the macro itself belongs in `xplatform/` as an `mx_` macro instead.
+
+## Reuse before writing
+
+Before writing a new macro, check the library for an existing one — common utilities already exist, e.g. `mp_abort` (the deprecated `mf_abort` is retained for backwards compatibility — don't use it in new code), `mf_existds`, `mf_existvar`, `mf_existfileref`, `mf_getuser`, `mp_jsonout` (SAS datasets → JSON for `_webout`), `mp_ds2ddl`, `mp_hashdataset`. Platform-specific variants exist under `meta/`, `viya/`, `server/` and are selected at compile time by the CLI.
+
+## Aborting safely
+
+Never invoke `%mp_abort` from inside an `%if/%else` block — as a procedural macro, the macro processor can continue executing statements after it before the abort takes effect. Use the `iftrue=` condition parameter instead:
+
+```sas
+%mp_abort(iftrue= (&syscc ne 0)
+  ,mac=&_program
+  ,msg=%str(Something went wrong)
+)
+```
+
+When `%mp_abort` is called from within a `%include` block, SAS cannot exit cleanly (e.g. to `_webout`). Call `%mp_abort(mode=INCLUDE)` after the include (OUTSIDE any macro wrapper) — it checks `work.mp_abort_errds` for an abort status:
+
+```sas
+%mp_abort(mode=INCLUDE)
+```
+
+Note: `%include`s inside macros should be performed with `%mp_include()` so the `_SYSINCLUDEFILEDEVICE` indicator is set and the abort dataset (`work.mp_abort_errds`) is passed back to the calling program.
+
+## Testing macros (mandatory conventions)
+
+- **Always apply `%mp_assertscope` around the macro under test** to catch scope leakage (macro variables must stay `%local`):
+
+```sas
+%mp_assertscope(SNAPSHOT)
+%mx_foo(args)
+%mp_assertscope(COMPARE,
+  desc=Test 1: mx_foo does not leak scope,
+  outds=work.test_results
+)
+```
+
+- Assertions go to `work.test_results` via `%mp_assert(iftrue=(...), desc=..., outds=work.test_results)`.
+
+## Lint and build
+
+- Run `sasjs lint` after every change; do not consider work done until it passes
+- NEVER bump the version in `package.json` (semantic-release handles it)
+- Do NOT edit generated files by hand: `all.sas`, `mc_*.sas`, the `lua/` wrappers, and `sasjsbuild/` outputs are produced by the CI build
+- Markdown files: never hard-wrap; one paragraph per line
+

.agents/skills/sasjs-framework/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: sasjs-framework
+description: Building full SASjs applications — project structure, sasjsconfig.json, services/jobs/macros folders, multi-target (SAS 9 / Viya / SASjs server) configuration, streaming frontends, mocks and tests. Use when creating or modifying a SASjs app, editing sasjsconfig.json, or writing backend services returning JSON to a web frontend.
+---
+
+# SASjs Framework — Building SASjs Applications
+
+A SASjs app = a web frontend (any framework: Angular, React, vanilla) + SAS backend code organised in a standard layout, compiled and deployed by `@sasjs/cli` to SAS 9, Viya, or SASjs server. Frontend talks to SAS via `@sasjs/adapter`; backend services return JSON via `_webout`.
+
+## Standard project layout
+
+```
+sasjs/
+  sasjsconfig.json     # project + target configuration
+  macros/              # project-specific macros (macroFolders)
+  services/            # web services called from the frontend
+  jobs/                # jobs (scheduled / flow / long-running)
+  programs/            # plain programs (initProgram, termProgram, utilities)
+  db/                  # DDL + static data per library (sasjs db)
+  tests/               # tests run by `sasjs test`
+  mocks/               # mock responses for offline frontend dev (syncFolder)
+  doxy/                # extra doxygen content for `sasjs doc`
+```
+
+## sasjsconfig.json
+
+Root config holds defaults; each entry in `targets[]` can override them. Key sections:
+
+- `macroFolders`, `binaryFolders` — where the CLI finds macros/binaries
+- `serviceConfig.serviceFolders` — service source folders; `initProgram` runs before every service (set up libnames, options)
+- `jobConfig.jobFolders` — job source folders
+- `programFolders` — programs compiled/deployed with the app
+- `streamConfig` — `streamWeb: true` streams the built frontend into SAS so it is served by the platform itself (no separate web server needed); `webSourcePath` points at the frontend build output
+- `syncFolder` — folder synced to the server (e.g. mocks)
+- `testConfig` — init/term programs for `sasjs test`
+- `targets[]` — per-environment overrides: `serverUrl`, `serverType` (`SAS9`/`SASVIYA`/`SASJS`), `appLoc` (deploy root, e.g. `/Public/app/myapp`), target-specific macroFolders (e.g. `targets/viya/macros_viya` for platform shims), `httpsAgentOptions`, `deployConfig`
+
+The full JSON schema is bundled at `sasjsconfig-schema.json` next to this file — validate config changes against it. Reference it with `"$schema": "https://cli.sasjs.io/sasjsconfig-schema.json"`.
+
+## Streamed frontend files on Viya (mime types)
+
+When `streamWeb: true`, the CLI uploads the frontend (`index.html`, renamed per `streamServiceName`, plus css/js) to the Viya Files service using the `%mv_createfile` macro. That macro creates the file in a very particular way to ensure it streams correctly:
+
+- POSTs to `/files/files` with the content type derived from the extension (`%mf_mimetype`)
+- sets `typeDefName=file_html` (via `%mv_getViyaFileExtParms`) so the file is recognised as HTML
+- sends `Content-Disposition` **without** `attachment` for HTML/SVG so it renders in the browser
+
+**Never update a streamed frontend file in place** with a `filename filesrvc` fileref + data step rewrite — the Files service then treats it as a generic blob and the mime type is lost, so the app no longer streams (browser downloads it or shows raw text). To modify a streamed file at runtime (eg patching the compute `contextname` in the html), read it (a `filesrvc` fileref is fine for *reading*), write the modified content to a temp fileref, and **re-create the file with `%mv_createfile(path=..., name=..., inref=...)`** (it deletes the old file and re-POSTs with the correct mime type).
+
+## Service contract (frontend ↔ SAS)
+
+1. Adapter POSTs to `services/<folder>/<name>` with input tables (arrays of objects) → work datasets named after the JS keys.
+2. Service SAS code runs after `initProgram`; it reads inputs, does work, and writes output JSON to `_webout`.
+3. Conventional pattern using @sasjs/core macros:
+
+```sas
+/**
+  @file
+  @brief Example service returning data
+  <h4> SAS Macros </h4>
+  @li mp_jsonout.sas
+  @li mp_abort.sas
+**/
+
+/* validation / logic here */
+
+%mp_jsonout(OPEN)
+%mp_jsonout(OBJ,results,dslabel=results)
+%mp_jsonout(CLOSE)
+```
+
+4. On error, abort cleanly with `%mp_abort(...)` (`mf_abort` is deprecated) so the adapter receives a structured error in the JSON, not a half-written response. Do **not** call `%mp_abort` inside an `%if/%else` block — the macro processor may keep executing beyond the abort. Use the conditional `iftrue=` parameter instead, e.g.:
+
+```sas
+%mp_abort(iftrue= (%mf_existds(work.results)=0)
+  ,mac=&_program
+  ,msg=%str(No results found)
+)
+```
+
+If the abort happens inside a `%include` block, SAS cannot exit to `_webout` cleanly — after the include, call `%mp_abort(mode=INCLUDE)` (outside any macro wrapper), which checks `work.mp_abort_errds` for an abort status.
+
+## Multi-target discipline
+
+- Keep backend code platform-neutral in shared folders; put platform-specific shims in `targets/<name>/macros_*` folders and register them only on that target.
+- Platform capability macros exist in @sasjs/core (`mm_*` metadata, `mv_*` Viya, `ms_*` server) — don't branch on server type by hand.
+
+## Quality gates (follow the conventions of mature apps like Data Controller)
+
+- Run `sasjs lint` after touching any `.sas` file; fix all warnings in files you touched.
+- The linter enforces 2-space indentation everywhere, including continuation lines inside `/* ... */` block comments — never align comment text with 3+ spaces.
+- Add tests and run `sasjs test` for backend logic changes. When testing macros, always wrap the macro under test with `%mp_assertscope(SNAPSHOT)` / `%mp_assertscope(COMPARE, ...)` to catch macro-variable scope leakage, and wrap any platform-branching code in `%macro` wrappers (no open conditional macro code in test programs).
+- Provide mocks in `sasjs/mocks` so the frontend can be developed without a live SAS server.
+- Never auto-commit or bump versions; releases are pipeline-driven (conventional commits).
+- Markdown files: no hard wrapping — one paragraph per line.
+- Apps must work offline/on-prem: no external CDN assets in the frontend bundle.
+
+## Reference implementations
+
+Look at existing apps for patterns: folder layouts, `sasjsconfig.json` multi-target setups, service structure, streaming builds, and test/mock conventions, eg:
+* https://git.datacontroller.io/dc/dc
+* https://github.com/sasjs/react-seed-app
+* https://github.com/sasjs/macro-dash

.agents/skills/sasjs-framework/sasjsconfig-schema.json

@@ -0,0 +1,966 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/sasjs/utils/blob/main/src/types/sasjsconfig-schema.json",
+  "type": "object",
+  "title": "SASjs Config File",
+  "description": "The SASjs Config file provides the settings and structure for your SASjs project. ",
+  "default": {},
+  "examples": [
+    {
+      "macroFolders": ["macros"],
+      "programFolders": ["programs"],
+      "binaryFolders": ["binaries"],
+      "defaultTarget": "viya",
+      "targets": [
+        {
+          "name": "viya",
+          "serverType": "SASVIYA",
+          "serverUrl": "https://sas.sasjs.com",
+          "appLoc": "/Public/app",
+          "contextName": "SAS Job Execution compute context",
+          "deployConfig": {
+            "deployServicePack": true,
+            "deployScripts": ["sasjsbuild/myviyadeploy.sas"]
+          },
+          "serviceConfig": {
+            "serviceFolders": ["targets/viya/services/admin"],
+            "initProgram": "build/serviceinit.sas",
+            "termProgram": "build/serviceinit.sas",
+            "macroVars": {
+              "name": "viyavalue",
+              "extravar": "this too"
+            }
+          },
+          "jobConfig": {
+            "jobFolders": [],
+            "initProgram": "",
+            "termProgram": "",
+            "macroVars": {}
+          },
+          "streamConfig": {
+            "assetPaths": [],
+            "streamWeb": false,
+            "streamWebFolder": "webv",
+            "webSourcePath": "dist"
+          },
+          "testConfig": {
+            "initProgram": "sasjs/tests/testinit.sas",
+            "termProgram": "sasjs/tests/testterm.sas",
+            "macroVars": {
+              "testVar": "testValue"
+            },
+            "testSetUp": "sasjs/tests/testsetup.sas",
+            "testTearDown": "sasjs/tests/testteardown.sas"
+          },
+          "macroFolders": ["targets/viya/macros"],
+          "programFolders": [],
+          "binaryFolders": ["binaries"]
+        },
+        {
+          "name": "sas9",
+          "serverType": "SAS9",
+          "serverUrl": "https://sas.sasjs.com:7980",
+          "appLoc": "/User Folders/&sysuserid/My Folder",
+          "serverName": "Foundation",
+          "repositoryName": "SASApp",
+          "buildConfig": {
+            "buildOutputFileName": "mysas9deploy.sas",
+            "initProgram": "",
+            "termProgram": "",
+            "macroVars": {}
+          },
+          "deployConfig": {
+            "deployScripts": ["build/deploysas9.sh"],
+            "deployServicePack": false
+          },
+          "serviceConfig": {
+            "serviceFolders": ["targets/sas9/services/admin"],
+            "initProgram": "",
+            "termProgram": "build/servicetermother.sas",
+            "macroVars": {}
+          },
+          "streamConfig": {
+            "assetPaths": [],
+            "streamWeb": false,
+            "streamWebFolder": "web9",
+            "webSourcePath": "dist"
+          },
+          "testConfig": {
+            "initProgram": "sasjs/tests/testinit.sas",
+            "termProgram": "sasjs/tests/testterm.sas",
+            "macroVars": {
+              "testVar": "testValue"
+            },
+            "testSetUp": "sasjs/tests/testsetup.sas",
+            "testTearDown": "sasjs/tests/testteardown.sas"
+          },
+          "macroFolders": ["targets/sas9/macros"],
+          "programFolders": [],
+          "binaryFolders": ["binaries"]
+        }
+      ]
+    }
+  ],
+  "properties": {
+    "binaryFolders": {
+      "$id": "#/properties/binaryFolders",
+      "type": "array",
+      "title": "The binaryFolders array",
+      "description": "These local folders are searched for Binary Files when running sasjs compile.  Folders are relative to the sasjs/sasjsconfig.json file.",
+      "examples": [["binaries", "../../more_binaries"]]
+    },
+    "sasjsBuildFolder": {
+      "$id": "#/properties/sasjsBuildFolder",
+      "type": "string",
+      "title": "sasjsBuildFolder",
+      "description": "The name of the folder containing the compiled output.  The `sasjs build` command will take all of the subfolders here as inputs to create the build pack.  By default this will be named `sasjsbuild`.  In global, the default is `~/.sasjsbuild`.",
+      "default": "sasjsbuild",
+      "examples": ["sasjsbuild", ".sasjsbuild"]
+    },
+    "sasjsResultsFolder": {
+      "$id": "#/properties/sasjsResultsFolder",
+      "type": "string",
+      "title": "sasjsResultsFolder",
+      "description": "The name of the folder containing the output (eg logs, ODS output) from `sasjs run`.  By default this will be named `sasjsresults`.  In global, the default is `~/.sasjsresults`.",
+      "default": "sasjsresults",
+      "examples": ["sasjsresults", ".sasjsresults"]
+    },
+    "defaultTarget": {
+      "$id": "#/properties/defaultTarget",
+      "type": "string",
+      "title": "Default Target",
+      "description": "If a target is not specified, this target is used by default.  The default target must exist in the (local) targets array.",
+      "default": "viya",
+      "examples": ["viya"]
+    },
+    "docConfig": {
+      "$id": "#/properties/docConfig",
+      "type": "object",
+      "title": "The docConfig schema",
+      "description": "SASjs uses doxygen to auto-generate HTML documentation using the headers in your SAS programs, macros, services & jobs. For more info, see [https://cli.sasjs.io/doc](https://cli.sasjs.io/doc).  Some properties are taken from package.json (such as the project Name).",
+      "default": {
+        "dataControllerUrl": "https://mysasserver.com/web/datacontroller/#",
+        "enableLineage": true,
+        "doxyContent": {
+          "readMe": "../../README.md"
+        }
+      },
+      "examples": [
+        {
+          "displayMacroCore": true,
+          "outDirectory": "/some/output/directory",
+          "dataControllerUrl": "https://mysasserver.com/web/datacontroller/#",
+          "enableLineage": true,
+          "doxyContent": {
+            "readMe": "../../my/custom/homepage.md",
+            "path": "my/custom/doxy/folder"
+          }
+        }
+      ],
+      "properties": {
+        "displayMacroCore": {
+          "$id": "#/properties/docConfig/properties/displayMacroCore",
+          "type": "boolean",
+          "title": "The displayMacroCore docConfig option",
+          "description": "The CLI will autocompile macro dependencies that exist in the SASjs Macro Core library.  These will also show in the documentation under 'node_modules'.  If you'd prefer not to show these in the rendered docs, set this value to false.",
+          "default": true,
+          "examples": [true]
+        },
+        "outDirectory": {
+          "$id": "#/properties/docConfig/properties/outDirectory",
+          "type": "string",
+          "title": "The outDirectory docConfig option",
+          "description": "The location to which the generated HTML SAS documentation is written.  If missing, or left blank, the files will be written to the `sasjsbuild/doc` directory (default behaviour).",
+          "default": "sasjsbuild/doc",
+          "examples": ["/my/preferred/docs/directory"]
+        },
+        "dataControllerUrl": {
+          "$id": "#/properties/docConfig/properties/dataControllerUrl",
+          "type": "string",
+          "title": "The dataControllerUrl docConfig option",
+          "description": "Provide the full URL to Data Controller so that `sasjs doc` can link the lineage diagram directly to the table viewer in [Data Controller](https://datacontroller.io).\nIf left blank, or undefined, no links will be generated.",
+          "default": "",
+          "examples": ["https://yourserver.co.uk/dcviya/#"]
+        },
+        "enableLineage": {
+          "$id": "#/properties/docConfig/properties/enableLineage",
+          "type": "boolean",
+          "title": "Enable Lineage",
+          "description": "If true, sasjs doc will generate and display Data Lineage from Jobs and Services.",
+          "default": true
+        },
+        "doxyContent": {
+          "$id": "#/properties/docConfig/properties/doxyContent",
+          "type": "object",
+          "title": "doxyContent",
+          "description": "Configuration of the Doxyfile variables",
+          "default": {},
+          "examples": [
+            {
+              "favIcon": "favicon.ico",
+              "footer": "new_footer.html",
+              "header": "new_header.html",
+              "layout": "DoxygenLayout.xml",
+              "logo": "logo.png",
+              "readMe": "../../README.md",
+              "stylesheet": "new_stylesheet.css",
+              "path": "sasjs/doxy"
+            }
+          ],
+          "properties": {
+            "favIcon": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/favIcon",
+              "type": "string",
+              "title": "Doxygen favicon",
+              "description": "The favicon used in the doxygen documentation",
+              "default": "favicon.ico"
+            },
+            "footer": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/footer",
+              "type": "string",
+              "title": "Doxygen footer",
+              "description": "The footer HTML file used in the doxygen documentation",
+              "default": "new_footer.html"
+            },
+            "header": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/header",
+              "type": "string",
+              "title": "Doxygen header",
+              "description": "The header HTML file used in the doxygen documentation",
+              "default": "new_header.html"
+            },
+            "layout": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/layout",
+              "type": "string",
+              "title": "Doxygen Layout XML File",
+              "description": "The layout XML file used to build the doxygen documentation",
+              "default": "DoxygenLayout.xml"
+            },
+            "logo": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/logo",
+              "type": "string",
+              "title": "Doxygen Logo",
+              "description": "The logo file used by doxygen",
+              "default": "logo.png"
+            },
+            "readMe": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/readMe",
+              "type": "string",
+              "title": "Doxygen Homepage",
+              "description": "The file used to generate the doxygen homepage (defaults to the readme.md)",
+              "default": "../../README.md"
+            },
+            "stylesheet": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/stylesheet",
+              "type": "string",
+              "title": "Doxygen CSS",
+              "description": "The CSS file used to extend Doxygen",
+              "default": "new_stylesheet.css"
+            },
+            "path": {
+              "$id": "#/properties/docConfig/properties/doxyContent/properties/path",
+              "type": "string",
+              "title": "Doxygen Path",
+              "description": "The path to the Doxygen configuration files (relative to sasjs/sasjsconfig.json)",
+              "default": "sasjs/doxy"
+            }
+          }
+        }
+      }
+    },
+    "httpsAgentOptions": {
+      "$id": "#/properties/httpsAgentOptions",
+      "type": "object",
+      "title": "httpsAgentOptions",
+      "description": "Configure https agent by setting all supported attribute such as `key`, `cert`, `ca`, `rejectUnauthorized` and `requestCert`",
+      "examples": [
+        {
+          "allowInsecureRequests": false,
+          "caPath": "path/to/caFile",
+          "keyPath": "path/to/keyFile",
+          "certPath": "path/to/certFile",
+          "requestCert": false,
+          "rejectUnauthorized": true
+        }
+      ],
+      "properties": {
+        "allowInsecureRequests": {
+          "$id": "#/properties/httpsAgentOptions/properties/allowInsecureRequests",
+          "type": "boolean",
+          "title": "allowInsecureRequests",
+          "description": "If you are having certificate errors connecting to SAS, that cannot be properly resolved, try setting this value to true. This option only has an effect if rejectUnauthorized is not present.",
+          "default": false,
+          "examples": [true, false]
+        },
+        "caPath": {
+          "$id": "#/properties/httpsAgentOptions/properties/caPath",
+          "type": "string",
+          "title": "caPath",
+          "description": "Optionally override the trusted CA certificates. Default is to trust the well-known CAs curated by Mozilla. Mozilla's CAs are completely replaced when CAs are explicitly specified using this option.",
+          "examples": ["path/to/caFile"]
+        },
+        "keyFile": {
+          "$id": "#/properties/httpsAgentOptions/properties/keyFile",
+          "type": "string",
+          "title": "keyFile",
+          "description": "Private keys in PEM format. PEM allows the option of private keys being encrypted. Encrypted keys will be decrypted with options.passphrase. Multiple keys using different algorithms can be provided either as an array of unencrypted key strings or buffers, or an array of objects in the form {pem: <string|buffer>[, passphrase: <string>]}. The object form can only occur in an array. object.passphrase is optional. Encrypted keys will be decrypted with object.passphrase if provided, or options.passphrase if it is not.",
+          "examples": ["path/to/keyFile"]
+        },
+        "certFile": {
+          "$id": "#/properties/httpsAgentOptions/properties/certFile",
+          "type": "string",
+          "title": "certFile",
+          "description": "Cert chains in PEM format. One cert chain should be provided per private key. Each cert chain should consist of the PEM formatted certificate for a provided private key, followed by the PEM formatted intermediate certificates (if any), in order, and not including the root CA (the root CA must be pre-known to the peer, see ca). When providing multiple cert chains, they do not have to be in the same order as their private keys in key. If the intermediate certificates are not provided, the peer will not be able to validate the certificate, and the handshake will fail.",
+          "examples": ["path/to/certFile"]
+        },
+        "requestCert": {
+          "$id": "#/properties/httpsAgentOptions/properties/requestCert",
+          "type": "boolean",
+          "title": "requestCert",
+          "description": "If true the server will request a certificate from clients that connect and attempt to verify that certificate. Defaults to false.",
+          "default": false,
+          "examples": [true, false]
+        },
+        "rejectUnauthorized": {
+          "$id": "#/properties/httpsAgentOptions/properties/rejectUnauthorized",
+          "type": "boolean",
+          "title": "rejectUnauthorized",
+          "description": "If true the server will reject any connection which is not authorized with the list of supplied CAs. This option only has an effect if requestCert is true.",
+          "default": true,
+          "examples": [true, false]
+        }
+      }
+    },
+    "buildConfig": {
+      "$id": "#/properties/buildConfig",
+      "type": "object",
+      "title": "buildConfig",
+      "description": "Dictates which files get compiled into the build program (.sas), used to deploy services into SAS 9 or Viya environments (without a client/secret). You may use this config to include build specific macros, programs or macro variables - which is run a single time, on deployment - for things like database creation, or exporting a SAS 9 SPK after service creation.",
+      "examples": [
+        {
+          "initProgram": "build/buildinit.sas",
+          "termProgram": "build/buildterm.sas",
+          "macroVars": {
+            "name": "value",
+            "numvar": "42"
+          },
+          "buildOutputFileName": "buildpack.sas"
+        }
+      ],
+      "properties": {
+        "buildOutputFileName": {
+          "$id": "#/properties/buildConfig/properties/buildOutputFileName",
+          "type": "string",
+          "title": "buildOutputFileName",
+          "description": "The name of the generated .sas program, which can be used to deploy the app using only SAS Studio. By default, this will be the name of the target.",
+          "examples": ["viya.sas", "sas9.sas"]
+        },
+        "initProgram": {
+          "$id": "#/properties/buildConfig/properties/initProgram",
+          "type": "string",
+          "title": "The buildConfig initProgram",
+          "description": "The path to a .sas program that will be inserted at the start of the build .sas program (created when running `sasjs build`).",
+          "default": "sasjs/buildinit.sas"
+        },
+        "termProgram": {
+          "$id": "#/properties/buildConfig/properties/termProgram",
+          "type": "string",
+          "title": "The buildConfig termProgram",
+          "description": "The path to a .sas program that will be inserted at the end of the build .sas program (created when running `sasjs build`).",
+          "default": "sasjs/buildterm.sas"
+        },
+        "macroVars": {
+          "$id": "#/properties/buildConfig/properties/macroVars",
+          "type": "object",
+          "title": "The buildConfig macro variables",
+          "description": "A series of name value pairs that will be turned into SAS macro variables in the build .sas program (generated when running `sasjs build`).  The example provided will generate sas code as follows:\n```\n%let name=value;\n%let numvar=42;\n```",
+          "default": {},
+          "examples": [
+            {
+              "name": "value",
+              "numvar": "42"
+            }
+          ]
+        }
+      }
+    },
+    "deployConfig": {
+      "$id": "#/properties/deployConfig",
+      "type": "object",
+      "title": "The deployConfig schema",
+      "description": "The deployConfig object enables settings that relate to the deployment of a SAS app - be that Viya, or SAS 9, or a pure Base environment.",
+      "default": {},
+      "examples": [
+        {
+          "deployScripts": ["build/deployscript.sh"]
+        },
+        {
+          "deployScripts": ["build/deployscript.sh"],
+          "deployServicePack": true
+        }
+      ],
+      "properties": {
+        "deployScripts": {
+          "$id": "#/properties/deployConfig/properties/deployScripts",
+          "type": "array",
+          "title": "The deployConfig deployScripts array",
+          "description": "These scripts are executed when running `sasjs deploy`.  If the file is a .sas file, it is executed on the SAS server (Viya only).  Otherwise it is executed locally.  These scripts are run AFTER the deployment of the servicepack, if `deployServicePack:true` (Viya only).",
+          "default": [],
+          "examples": [["build/deployscript.sh", "build/myprogram.sas"]]
+        },
+        "deployServicePack": {
+          "$id": "#/properties/deployConfig/properties/deployServicePack",
+          "type": "boolean",
+          "title": "The deployConfig deployServicePack flag",
+          "description": "If set to `true` the json pack produced by `sasjs build` will be auto-deployed to the `appLoc` of the specified target (creating all jobs and services in the SAS folder tree).  Currently only Viya is supported for this flag.",
+          "default": false
+        }
+      }
+    },
+    "serviceConfig": {
+      "$id": "#/properties/serviceConfig",
+      "type": "object",
+      "title": "The serviceConfig schema",
+      "description": "The serviceConfig object defines how SASjs web services are compiled.  Web services differ from jobs in that they include some fixed pre-code (eg the macros to stream out the result json).",
+      "default": {},
+      "examples": [
+        {
+          "serviceFolders": ["services/common", "services/admin"],
+          "initProgram": "build/serviceinit.sas",
+          "termProgram": "build/serviceterm.sas",
+          "macroVars": {
+            "mac1": "value",
+            "mac2": "42"
+          }
+        }
+      ],
+      "required": [],
+      "properties": {
+        "serviceFolders": {
+          "$id": "#/properties/serviceConfig/properties/serviceFolders",
+          "type": "array",
+          "title": "The serviceConfig serviceFolders array",
+          "description": "When running `sasjs compile`, all programs in the folders defined in this array are compiled and placed into same-named folders under `sasjsbuild/services`.  They will be compiled as services (so, with the service pre-code).  Folders can be absolute, or relative to the `sasjs` folder.",
+          "default": [],
+          "examples": [["services/common", "services/admin"]]
+        },
+        "initProgram": {
+          "$id": "#/properties/serviceConfig/properties/initProgram",
+          "type": "string",
+          "title": "The serviceConfig initProgram",
+          "description": "The serviceConfig `initProgram` is a .sas file that is inserted at the start of every SAS service (after compiled macros and any `macroVars`, and before the service itself). ",
+          "default": "",
+          "examples": ["build/serviceinit.sas"]
+        },
+        "termProgram": {
+          "$id": "#/properties/serviceConfig/properties/termProgram",
+          "type": "string",
+          "title": "The serviceConfig termProgram",
+          "description": "The serviceConfig termProgram is inserted at the end of every service as part of `sasjs compile`.",
+          "default": "",
+          "examples": ["build/serviceterm.sas"]
+        },
+        "macroVars": {
+          "$id": "#/properties/serviceConfig/properties/macroVars",
+          "type": "object",
+          "title": "The `serviceConfig` macroVars",
+          "description": "This object allows `sasjs compile` to insert specific macro variables at the start of every service.  In this case, the code generated would be:\n```\n%let mac1=value;\n%let mac2=42;\n```",
+          "default": {},
+          "examples": [
+            {
+              "mac1": "value",
+              "mac2": "42"
+            }
+          ]
+        }
+      }
+    },
+    "jobConfig": {
+      "$id": "#/properties/jobConfig",
+      "type": "object",
+      "title": "The jobConfig schema",
+      "description": "The jobConfig object defines how SASjs Jobs are compiled. ",
+      "default": {},
+      "examples": [
+        {
+          "jobFolders": ["jobs/extract", "jobs/load"],
+          "initProgram": "jobs/jobinit.sas",
+          "termProgram": "jobs/jobterm.sas",
+          "macroVars": {
+            "mac1": "value",
+            "mac2": "42"
+          }
+        }
+      ],
+      "properties": {
+        "jobFolders": {
+          "$id": "#/properties/jobConfig/properties/jobFolders",
+          "type": "array",
+          "title": "The jobConfig jobFolders array",
+          "description": "When running `sasjs compile`, all programs in the local folders defined in this array are compiled and placed into same-named folders under `sasjsbuild/jobs`. Folders can be absolute, or relative to the local project `/sasjs` folder.",
+          "default": [],
+          "examples": [["jobs/extract", "jobs/transform", "jobs/load"]]
+        },
+        "initProgram": {
+          "$id": "#/properties/jobConfig/properties/initProgram",
+          "type": "string",
+          "title": "The jobConfig initProgram",
+          "description": "The jobConfig `initProgram` is a local .sas file that is inserted at the start of every SAS Job (after compiled macros and any `macroVars`, and before the Job itself). ",
+          "default": "",
+          "examples": ["jobs/jobinit.sas"]
+        },
+        "termProgram": {
+          "$id": "#/properties/jobConfig/properties/termProgram",
+          "type": "string",
+          "title": "The jobConfig termProgram",
+          "description": "The jobConfig termProgram is inserted at the end of every Job as part of `sasjs compile`.",
+          "default": "",
+          "examples": ["jobs/jobterm.sas"]
+        },
+        "macroVars": {
+          "$id": "#/properties/jobConfig/properties/macroVars",
+          "type": "object",
+          "title": "The `jobConfig` macroVars",
+          "description": "This object allows `sasjs compile` to insert specific macro variables at the start of every Job.  In this case, the code generated would be:\n```\n%let mac1=value;\n%let mac2=42;\n```",
+          "default": {},
+          "examples": [
+            {
+              "mac1": "value",
+              "mac2": "42"
+            }
+          ]
+        }
+      }
+    },
+    "streamConfig": {
+      "$id": "#/properties/streamConfig",
+      "type": "object",
+      "title": "streamConfig",
+      "description": "SASjs allows a local web app to be compiled such that all html, css, javascript, and other assets such as png or mp4 are converted into web services and streamed directly from SAS.\n\nThis approach is convenient as it bypasses the need to deploy to a web server. ",
+      "default": {},
+      "examples": [
+        {
+          "assetPaths": [],
+          "streamWeb": false,
+          "streamWebFolder": "webv",
+          "webSourcePath": "dist",
+          "streamLogo": "logo.png"
+        }
+      ],
+      "required": ["streamWeb"],
+      "properties": {
+        "assetPaths": {
+          "$id": "#/properties/streamConfig/properties/assetPaths",
+          "type": "array",
+          "title": "The streamConfig assetPaths array",
+          "description": "An array of local folders.  All assets placed in these folders are converted into web services - example file types could be png, svg, mp3, mp4, excel - anything really.",
+          "default": [],
+          "examples": [["/myassets"]]
+        },
+        "streamLogo": {
+          "$id": "#/properties/streamConfig/properties/streamLogo",
+          "type": "string",
+          "title": "Icon shown in AppStream (sasjs/server)",
+          "description": "Provide the location of a square image, under the webSourcePath.  Used as the display icon on the appStream page.",
+          "default": "logo.png",
+          "examples": ["logo.png", "favicon.ico"]
+        },
+        "streamWeb": {
+          "$id": "#/properties/streamConfig/properties/streamWeb",
+          "type": "boolean",
+          "title": "The streamConfig streamWeb flag",
+          "description": "When set to `true`, frontend files saved in the `webSourcePath` will be converted to streaming services in the `streamWebFolder` in SAS.",
+          "default": false,
+          "examples": [true]
+        },
+        "streamWebFolder": {
+          "$id": "#/properties/streamConfig/properties/streamWebFolder",
+          "type": "string",
+          "title": "The streamConfig streamWebFolder",
+          "description": "This is the target SAS folder (relative to the appLoc) where the compiled web assets will be created.",
+          "default": "webv"
+        },
+        "webSourcePath": {
+          "$id": "#/properties/streamConfig/properties/webSourcePath",
+          "type": "string",
+          "title": "The webSourcePath schema",
+          "description": "Active when `streamConfig` is `true`. Is the source (or build, or dist) LOCAL folder, relative to the `sasjs` folder, which contains the frontend to be deployed.  All assets (PNG, JS, CSS, HTML etc) are taken from here and converted to streaming services in `streamWebFolder`.  In Viya and SAS 9, any relative URLS will be modified such that the links still work and the assets still load.",
+          "default": "dist",
+          "examples": ["dist", "build"]
+        },
+        "streamServiceName": {
+          "$id": "#/properties/streamConfig/properties/streamServiceName",
+          "type": "string",
+          "title": "streamServiceName Schema",
+          "description": "The name of the service containing the `index.html` for a streaming web app.  Defaults to `clickme` and is always deployed under the appLoc/services SAS Folder.",
+          "default": "clickme.sas"
+        }
+      }
+    },
+    "testConfig": {
+      "$id": "#/properties/testConfig",
+      "type": "object",
+      "title": "testConfig",
+      "description": "Create tests for Macros, Services & Jobs by simply adding a '.test.sas' extension. ",
+      "default": {
+        "initProgram": "sasjs/tests/testinit.sas",
+        "termProgram": "sasjs/tests/testterm.sas",
+        "macroVars": {
+          "testVar": "testValue"
+        },
+        "testSetUp": "sasjs/tests/testsetup.sas",
+        "testTearDown": "sasjs/tests/testteardown.sas"
+      },
+      "properties": {
+        "initProgram": {
+          "$id": "#/properties/testConfig/properties/initProgram",
+          "type": "string",
+          "title": "The jobConfig initProgram",
+          "description": "The testConfig `initProgram` is a local .sas file that is inserted at the start of every Test (after compiled macros and any `macroVars`, and before the Test itself). ",
+          "default": "",
+          "examples": ["sasjs/tests/testinit.sas"]
+        },
+        "termProgram": {
+          "$id": "#/properties/testConfig/properties/termProgram",
+          "type": "string",
+          "title": "The jobConfig termProgram",
+          "description": "The testConfig termProgram is inserted at the end of every Test as part of `sasjs compile`.",
+          "default": "",
+          "examples": ["jobs/jobterm.sas"]
+        },
+        "macroVars": {
+          "$id": "#/properties/testConfig/properties/macroVars",
+          "type": "object",
+          "title": "The `jobConfig` macroVars",
+          "description": "This object allows `sasjs compile` to insert specific macro variables at the start of every Test.  In this case, the code generated would be:\n```\n%let mac1=value;\n%let mac2=42;\n```",
+          "default": {},
+          "examples": [
+            {
+              "mac1": "value",
+              "mac2": "42"
+            }
+          ]
+        },
+        "testSetUp": {
+          "$id": "#/properties/testConfig/properties/testsetup",
+          "type": "string",
+          "title": "The jobConfig testSetUp",
+          "description": "This program is the first to execute as part of 'sasjs test'.  It does not contain the testInit, testTerm or macroVariables.  It IS compiled.",
+          "default": "sasjs/tests/testsetup.sas",
+          "examples": ["sasjs/tests/testsetup.sas"]
+        },
+        "testTearDown": {
+          "$id": "#/properties/testConfig/properties/testTearDown",
+          "type": "string",
+          "title": "The jobConfig testTearDown",
+          "description": "The last program to execute as part of 'sasjs test'.",
+          "default": "sasjs/tests/testteardown.sas",
+          "examples": ["sasjs/tests/testteardown.sas"]
+        }
+      }
+    },
+    "macroFolders": {
+      "$id": "#/properties/macroFolders",
+      "type": "array",
+      "title": "The macroFolders array",
+      "description": "These local folders are searched for SAS Macros when running `sasjs compile`.  Folders are relative to the `sasjs/sasjsconfig.json` file.",
+      "default": [],
+      "examples": [["macros", "../../more_macros"]]
+    },
+    "programFolders": {
+      "$id": "#/properties/programFolders",
+      "type": "array",
+      "title": "The programFolders array",
+      "description": "These local folders are searched for SAS Programs when running `sasjs compile`.  Folders are relative to the `sasjs/sasjsconfig.json` file.",
+      "default": [],
+      "examples": [["programs", "../../more_programs"]]
+    },
+    "syncFolder": {
+      "$id": "#/properties/syncFolder",
+      "type": "string",
+      "title": "Sync Folder",
+      "description": "The contents of this folder are simply copied to the sasjsbuild directory AFTER the rest of the project is compiled.  Useful for synchronising random / generic content with SAS logical folders.",
+      "default": "sasjs/static_files"
+    },
+    "syncDirectories": {
+      "$id": "#/properties/syncDirectories",
+      "type": "array",
+      "title": "syncDirectories",
+      "description": "Maps the local filesystem to remote (SAS) physical directories.",
+      "examples": [
+        [
+          {
+            "local": "C:\\temp\\local\\fs1",
+            "remote": "/opt/data/fs1"
+          },
+          {
+            "local": "C:\\temp\\elsewhere",
+            "remote": "/opt/somewhere"
+          }
+        ]
+      ]
+    },
+    "targets": {
+      "$id": "#/properties/targets",
+      "type": "array",
+      "title": "The targets array",
+      "description": "A target is an alias for a deployment location, and includes at a minimum, the `serverUrl`, `serverType` and `appLoc`.  This array allows multiple targets to be defined (eg dev / test / prod).  Any properties defined here will override same-named properties in the sasjsconfig root.",
+      "default": [],
+      "examples": [
+        [
+          {
+            "name": "sas9target",
+            "serverType": "SAS9",
+            "serverUrl": "https://mysas9server",
+            "appLoc": "/Shared Folders/myApp"
+          }
+        ],
+        [
+          {
+            "name": "viya",
+            "serverType": "SASVIYA",
+            "serverUrl": "https://sas.sasjs.com",
+            "appLoc": "/Public/app",
+            "contextName": "SAS Job Execution compute context",
+            "buildConfig": {
+              "buildOutputFileName": "myviyadeploy.sas",
+              "initProgram": "build/buildinitviya.sas",
+              "termProgram": "targets/viya/viyabuildterm.sas",
+              "macroVars": {
+                "name": "viyavalue",
+                "extravar": "this too"
+              }
+            },
+            "deployConfig": {
+              "deployServicePack": true,
+              "deployScripts": ["sasjsbuild/myviyadeploy.sas"]
+            },
+            "serviceConfig": {
+              "serviceFolders": ["targets/viya/services/admin"],
+              "initProgram": "build/serviceinit.sas",
+              "termProgram": "build/serviceinit.sas",
+              "macroVars": {
+                "name": "viyavalue",
+                "extravar": "this too"
+              }
+            },
+            "streamConfig": {
+              "assetPaths": [],
+              "streamWeb": false,
+              "streamWebFolder": "webv",
+              "webSourcePath": "dist"
+            },
+            "testConfig": {
+              "initProgram": "sasjs/tests/testinit.sas",
+              "termProgram": "sasjs/tests/testterm.sas",
+              "macroVars": {
+                "testVar": "testValue"
+              },
+              "testSetUp": "sasjs/tests/testsetup.sas",
+              "testTearDown": "sasjs/tests/testteardown.sas"
+            },
+            "macroFolders": ["targets/viya/macros"]
+          },
+          {
+            "name": "sas9",
+            "serverType": "SAS9",
+            "serverUrl": "https://sas.sasjs.com:7980",
+            "appLoc": "/User Folders/&sysuserid/My Folder",
+            "serverName": "Foundation",
+            "repositoryName": "SASApp",
+            "buildConfig": {
+              "buildOutputFileName": "mysas9deploy.sas",
+              "initProgram": "",
+              "termProgram": "",
+              "macroVars": {}
+            },
+            "deployConfig": {
+              "deployScripts": ["build/deploysas9.sh"],
+              "deployServicePack": false
+            },
+            "serviceConfig": {
+              "serviceFolders": ["targets/sas9/services/admin"],
+              "initProgram": "",
+              "termProgram": "build/servicetermother.sas",
+              "macroVars": {}
+            },
+            "streamConfig": {
+              "assetPaths": [],
+              "streamWeb": false,
+              "streamWebFolder": "web9",
+              "webSourcePath": "dist"
+            },
+            "testConfig": {
+              "initProgram": "sasjs/tests/testinit.sas",
+              "termProgram": "sasjs/tests/testterm.sas",
+              "macroVars": {
+                "testVar": "testValue"
+              },
+              "testSetUp": "sasjs/tests/testsetup.sas",
+              "testTearDown": "sasjs/tests/testteardown.sas"
+            },
+            "macroFolders": ["targets/sas9/macros"],
+            "programFolders": []
+          }
+        ]
+      ],
+      "items": {
+        "$id": "#/properties/targets/items",
+        "anyOf": [
+          {
+            "$id": "#/properties/targets/items/anyOf/0",
+            "type": "object",
+            "title": "SASjs Targets",
+            "description": "A target provides the configuration specific to a particular deployment, eg DEV / TEST, or SAS9 / SASVIYA.",
+            "default": {},
+            "examples": [
+              {
+                "name": "viya",
+                "serverType": "SASVIYA",
+                "serverUrl": "https://sas.sasjs.com",
+                "appLoc": "/Public/app",
+                "contextName": "SAS Job Execution compute context",
+                "buildConfig": {
+                  "buildOutputFileName": "myviyadeploy.sas",
+                  "initProgram": "build/buildinitviya.sas",
+                  "termProgram": "targets/viya/viyabuildterm.sas",
+                  "macroVars": {
+                    "name": "viyavalue",
+                    "extravar": "this too"
+                  }
+                },
+                "deployConfig": {
+                  "deployServicePack": true,
+                  "deployScripts": ["sasjsbuild/myviyadeploy.sas"]
+                },
+                "serviceConfig": {
+                  "serviceFolders": ["targets/viya/services/admin"],
+                  "initProgram": "build/serviceinit.sas",
+                  "termProgram": "build/serviceinit.sas",
+                  "macroVars": {
+                    "name": "viyavalue",
+                    "extravar": "this too"
+                  }
+                },
+                "jobConfig": {
+                  "jobFolders": [],
+                  "initProgram": "",
+                  "termProgram": "",
+                  "macroVars": {}
+                },
+                "streamConfig": {
+                  "assetPaths": [],
+                  "streamWeb": false,
+                  "streamWebFolder": "webv",
+                  "webSourcePath": "dist"
+                },
+                "testConfig": {
+                  "initProgram": "sasjs/tests/testinit.sas",
+                  "termProgram": "sasjs/tests/testterm.sas",
+                  "macroVars": {
+                    "testVar": "testValue"
+                  },
+                  "testSetUp": "sasjs/tests/testsetup.sas",
+                  "testTearDown": "sasjs/tests/testteardown.sas"
+                },
+                "macroFolders": ["targets/viya/macros"],
+                "programFolders": []
+              }
+            ],
+            "required": ["name", "serverType", "appLoc"],
+            "properties": {
+              "name": {
+                "$id": "#/properties/targets/items/anyOf/0/properties/name",
+                "type": "string",
+                "title": "Target name property",
+                "description": "A target name can only contain alphanumeric characters and dashes.  It cannot contain spaces. It is used as the alias when referencing the target using the `-t` attribute in many of the SASjs commands.",
+                "default": "",
+                "examples": ["viya"]
+              },
+              "serverType": {
+                "$id": "#/properties/targets/items/anyOf/0/properties/serverType",
+                "type": "string",
+                "title": "The Target serverType",
+                "description": "The serverType can be either SAS9, SASVIYA or SASJS.",
+                "default": "",
+                "examples": ["SASVIYA"]
+              },
+              "appLoc": {
+                "$id": "#/properties/targets/items/anyOf/0/properties/appLoc",
+                "type": "string",
+                "title": "The Target appLoc",
+                "description": "The appLoc provides the root SAS folder location under which all jobs and services are deployed and executed.  The SAS folder could be metadata in SAS 9, or SAS Drive in Viya. ",
+                "default": "",
+                "examples": ["/Public/app"]
+              },
+              "binaryFolders": {
+                "$ref": "#/properties/binaryFolders"
+              },
+              "buildConfig": {
+                "$ref": "#/properties/buildConfig"
+              },
+              "contextName": {
+                "$id": "#/properties/targets/items/anyOf/0/properties/contextName",
+                "type": "string",
+                "title": "The Target contextName",
+                "description": "The name of the compute context used to execute SAS code.  The context determines the way in which the SAS session is spawned (eg user credentials, autoexec code, system options etc).\nContexts can be created / modified / deleted using the `sasjs context` command.",
+                "default": "SAS Job Execution compute context",
+                "examples": ["SAS Job Execution compute context"]
+              },
+              "deployConfig": {
+                "$ref": "#/properties/deployConfig"
+              },
+              "httpsAgentOptions": {
+                "$ref": "#/properties/httpsAgentOptions"
+              },
+              "serverUrl": {
+                "$id": "#/properties/targets/items/anyOf/0/properties/serverUrl",
+                "type": "string",
+                "title": "The Target serverUrl",
+                "description": "The serverUrl is the location to which the app is deployed, and against which any server based operations are performed.  If SAS is served from a particular port, that port should also be included here.",
+                "default": "",
+                "examples": [
+                  "https://sas.sasjs.com",
+                  "https://sas.sasjs.com:8080"
+                ]
+              },
+              "serviceConfig": {
+                "$ref": "#/properties/serviceConfig"
+              },
+              "jobConfig": {
+                "$ref": "#/properties/jobConfig"
+              },
+              "docConfig": {
+                "$ref": "#/properties/docConfig"
+              },
+              "streamConfig": {
+                "$ref": "#/properties/streamConfig"
+              },
+              "syncDirectories": {
+                "$ref": "#/properties/syncDirectories"
+              },
+              "syncFolder": {
+                "$ref": "#/properties/syncFolder"
+              },
+              "testConfig": {
+                "$ref": "#/properties/testConfig"
+              },
+              "macroFolders": {
+                "$ref": "#/properties/macroFolders"
+              },
+              "programFolders": {
+                "$ref": "#/properties/programFolders"
+              },
+              "sasjsBuildFolder": {
+                "$ref": "#properties/sasjsBuildFolder"
+              },
+              "sasjsResultsFolder": {
+                "$ref": "#properties/sasjsResultsFolder"
+              }
+            }
+          }
+        ]
+      }
+    }
+  }
+}

.agents/skills/sasjs-server/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: sasjs-server
+description: Installing, configuring, and running @sasjs/server — the open-source NodeJS wrapper around the SAS binary that provides a REST API, filesystem (SASjs Drive), Stored Program execution, and web app streaming. Covers desktop vs server modes, runtimes (SAS/JS/Python/R), env vars, auth (tokens, LDAP), and mock servers. Use when deploying, troubleshooting, or developing against sasjs/server.
+---
+
+# @sasjs/server
+
+SASjs Server is an open-source NodeJS wrapper for calling the SAS binary executable. It runs on a real SAS server or a local desktop and provides:
+
+- A filesystem (SASjs Drive) for storing SAS programs and content
+- Execution of Stored Programs from a URL (equivalent to SAS 9 Stored Processes / Viya Jobs)
+- Web app streaming (serve frontend apps straight from SAS content)
+- A REST API with Swagger docs
+- Portability: apps built for SASjs Server deploy unchanged to SAS 9 / Viya via @sasjs/cli
+
+## Modes
+
+- **Desktop mode** (`MODE=desktop`, default): single-user, no authentication, no database. CORS enabled by default.
+- **Server mode** (`MODE=server`): multi-user with authentication, requires a database (`DB_CONNECT`, `DB_TYPE=mongodb|cosmos_mongodb`). CORS disabled by default — configure `WHITELIST` if enabling.
+
+## Installation
+
+Download the relevant zip from GitHub releases and run the packaged executable (`api-linux`, etc.):
+
+```bash
+curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
+unzip linux.zip && ./api-linux
+```
+
+On first run it prompts (unless set as env vars) for the SAS executable path and the filesystem location for Stored Programs/temp files. Docker is also supported (`DockerfileApi`, docker-compose files in the repo).
+
+## Configuration via environment variables
+
+Set in `/etc/environment`, exported, prepended to the command, or in a `.env` file alongside the executable. Key variables:
+
+| Variable | Purpose |
+|---|---|
+| `MODE` | `desktop` (default) or `server` |
+| `SAS_PATH` | Path to `sas.exe` / `sas.sh` |
+| `RUN_TIMES` | Comma-separated runtime priority, e.g. `sas,js,py` — options: `sas`, `js`, `py`, `r`. Each needs its path: `SAS_PATH`, `NODE_PATH`, `PYTHON_PATH`, `R_PATH` |
+| `SASJS_ROOT` | Working directory: SAS WORK, staged files, drive, config |
+| `DRIVE_LOCATION` | Location for files, sasjs packages, `appStreamConfig.json` |
+| `PROTOCOL` / `PORT` | `http` (default) or `https` (needs `PRIVATE_KEY`, `CERT_CHAIN`, optional `CA_ROOT`); default port 5000 |
+| `SAS_OPTIONS` / `SASV9_OPTIONS` | Extra SAS system options auto-applied to sessions (Windows vs Unix), e.g. `-NOXCMD` |
+| `DB_CONNECT` / `DB_TYPE` | MongoDB connection string / type — required for server mode |
+| `AUTH_PROVIDERS` + `LDAP_*` | LDAP auth: `LDAP_URL`, `LDAP_BIND_DN`, `LDAP_BIND_PASSWORD`, `LDAP_USERS_BASE_DN`, `LDAP_GROUPS_BASE_DN` |
+| `CORS` / `WHITELIST` | CORS is only applied when `CORS=enable`, and only origins in `WHITELIST` (space-separated) receive `Access-Control-Allow-Origin` — an empty whitelist means NO cross-origin calls work |
+| `MOCK_SERVERTYPE` / `STATIC_MOCK_LOCATION` | Emulate `sas9`/`sasviya` API responses for frontend testing against a sasjs server (static canned files only — no logic) |
+
+## Developing against the API
+
+- Server type for @sasjs/adapter / CLI targets is `SASJS` (`serverType: 'SASJS'`); auth is token-based.
+- The REST API is self-documented via Swagger on the running instance.
+- Server-side execution uses `ms_*` macros from @sasjs/core (e.g. `ms_createfile`, `ms_adduser2group`) — services/jobs deployed by the CLI work as on other platforms.
+- Repo layout (for contributors): `api/` (Express/TypeScript backend: controllers, routes, middlewares, model), `web/` (frontend), `restClient/` (REST examples), `mongo-seed/` (server-mode DB seed).
+
+## Mock services with the JS runtime (no SAS required)
+
+With `RUN_TIMES=js` (and `NODE_PATH` set), any `.js` file on SASjs Drive is an executable Stored Program — this is how react-seed-app / Data Controller provide **mock backends** for frontend development.  Desktop mode (`MODE=desktop`) has no auth, which makes local mocking trivial.
+
+Writing a JS stored program (docs: https://server.sasjs.io/storedprograms/#js-programs):
+
+- The runtime template predeclares `const fs = require('fs')`, `_program`, `weboutPath`, `_SASJS_TOKENFILE`, `_SASJS_WEBOUT_HEADERS`, `_SASJS_USERNAME` / `_SASJS_USERID` / `_SASJS_DISPLAYNAME`, `_METAPERSON`, `_METAUSER`, `SASJSPROCESSMODE`.  **Do NOT redeclare `fs`** — `const fs = require('fs')` in your program crashes it with `Identifier 'fs' has already been declared`.
+- Output: assign a JSON **string** to `_webout` (e.g. `_webout = JSON.stringify({...})`); it is written back only if non-empty.  `console.log()` output is returned in the response `log` (like a SAS log).  Custom response headers can be written as lines to the `_SASJS_WEBOUT_HEADERS` file.
+- Mimic real services by including the standard SASjs automatic fields in the JSON: `_PROGRAM` (from `_program`), `SYSDATE` / `SYSTIME` (format `DDMMMYY` / `HH:mm`), `_METAUSER`, `SASJSPROCESSMODE`.
+- URL/body parameters arrive as `const <name> = \`<value>\`` strings.
+- Input tables (the `sasjs_tables` mechanism) arrive **either** as an inline CSV const **or** — when the adapter sends multipart — as an uploaded `<name>.csv` file in the session folder, referenced by generated module-scope consts (handle BOTH):
+  - `_WEBIN_FILE_COUNT` (always created), `_WEBIN_NAME<n>` (table/field name), `_WEBIN_FILENAME<n>` (original filename), `_WEBIN_FILEREF<n>` (file **contents**, a Buffer from `fs.readFileSync` — call `.toString('utf8')`)
+  - these consts are **not on `globalThis`** — look them up with `typeof` guards or direct `eval()` in module scope (server-side JS, no CSP)
+  - adapter CSV quirks: header row is **space-separated** `name:format.` entries (e.g. `rootdir:$char256.`) — strip the `:format` suffix; lines end CRLF; values containing special characters are wrapped in double quotes with `""` escaping
+- Adapter response shape: `sasjs.request()` resolves with the webout JSON **already unwrapped** — output tables are arrays of row objects directly on the response (`res.mytable[0].COL`).  A table named `result` is perfectly fine (`res.result` is then that array); do NOT add your own `res.result`-unwrapping layer, it breaks exactly that case.
+- Third-party npm packages are NOT resolvable at runtime — bundle the service first (e.g. `npx webpack --mode none --target node --entry <file> --output-path sasjsbuild/... --output-filename <name>.js`), then `sasjs build` / `sasjs deploy`.
+
+Deploying mocks:
+
+- `sasjs fs sync` does NOT work on a JS-only server (it generates and executes SAS code to hash remote files).  Upload files directly via the Drive API instead: `DELETE` then `POST /SASjsApi/drive/file?_filePath=<appLoc>/services/<folder>/<name>.js` (multipart `file` field).  In desktop mode no auth headers are needed; in server mode read the `Authorization` header line from `_SASJS_TOKENFILE`.
+- Mocks can be stateful with the predeclared `fs`.  Prefer real locations over `/tmp`: the SASjs Drive root is derivable from `weboutPath` (`<root>/sessions/<id>/webout.txt` → `path.resolve(weboutPath, '..', '..', '..', 'drive')`), and a mock `configure`-style service can treat a configured folder as a real local path (the server IS local).  `require('path')` and other core modules work (only `fs` is predeclared).
+- A JS program can even call the server's own REST API (`http://127.0.0.1:$PORT/SASjsApi/...`) — e.g. to rewrite a streamed `index.html` on the Drive (`GET` + `PATCH /SASjsApi/drive/file`).
+
+Gotchas:
+
+- The packaged binaries (`api-linux` etc.) reject some globally-exported `NODE_OPTIONS` (e.g. `--network-family-autoselection`) — start with `NODE_OPTIONS="" ./api-linux`.
+- AppStream URLs redirect to a trailing slash (`/AppStream/MyApp` → 301 → `/AppStream/MyApp/`) — test/automation scripts should use the trailing-slash URL directly.

.git-hooks/pre-commit

@@ -3,6 +3,42 @@
 # Using `--silent` helps for showing any errs in the first line of the response
 # The first line is picked up by the VS Code GIT UI popup when rc is not 0
 
+# Scan staged changes for secrets before anything else runs.
+# The binary is pinned by the @nogoo9/gitleaks devDependency and lands
+# in node_modules/.bin after npm i. Commits are blocked until it is
+# installed so no clone silently skips the secrets scan.
+gitleaks_bin=node_modules/.bin/gitleaks
+
+if [ ! -x "$gitleaks_bin" ]; then
+    echo "❌ gitleaks not found - run 'npm i' to install it"
+    exit 1
+fi
+
+if ! "$gitleaks_bin" protect --staged --redact; then
+    echo "❌ gitleaks detected a potential secret in your staged changes"
+    echo "Remove the secret, or add a .gitleaksignore entry if it is a false positive"
+    exit 1
+fi
+
+## Avoid large commits
+# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
+size_limit=$((2 * 1024 * 1024)) # 2MB
+# Sum the sizes of the blobs this commit would introduce: the staged
+# added/copied/modified/renamed/typechanged entries in the index.
+# https://git-scm.com/docs/git-cat-file#Documentation/git-cat-file.txt---batch-check
+# Renames are treated as add+delete (--no-renames) and pure mode changes
+# (same blob, different permissions) add no new blob, so both are safe.
+commit_size=$(git diff --cached --raw --no-renames --diff-filter=ACMRT |
+  awk '$3 != $4 { print $4 }' |
+  git cat-file --batch-check='%(objectsize)' |
+  awk '{ s += $1 } END { print s + 0 }')
+
+if [ "$commit_size" -ge "$size_limit" ]; then
+  echo "Commit size is too large: $commit_size > $size_limit"
+  echo "Force commit using --no-verify"
+  exit 1
+fi
+
 if npm run --silent lint:check:silent ; then
     exit 0
 else
@@ -10,14 +46,3 @@ else
     echo "❌ Prettier check failed! We ran lint:fix for you. Please add & commit again."
     exit 1
 fi
-
-## Avoid large commits
-# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
-size_limit=$((2 * 2**20)) # 2mbs
-# https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---disk-usage
-commit_size=$(git rev-list --disk-usage HEAD^..HEAD)
-test "$commit_size" -lt "$size_limit" || (
-  echo "Commit size is too large: $commit_size > $size_limit"
-  echo "Force commit using --no-verify"
-  exit 1
-)

.gitea/workflows/build.yaml

@@ -125,8 +125,16 @@ jobs:
           cd ./sas/mocks/sasjs
           npm install -g @sasjs/cli
           npm install -g replace-in-files-cli
+          # Remove any previous deployment (drive folder delete API) so the
+          # deploy and makedata start from a clean appLoc
+          curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
           sasjs cbd -t server-ci
-          # sasjs request services/admin/makedata -t server-ci -d ./deploy/makeData4GL.json -c ./deploy/requestConfig.json -o ./output.json
+          # Seed the DC database (mock data files on the drive).
+          # makedata replies with HTML (it is normally called as a URL redirect),
+          # which the CLI reports as "invalid Json string" - ignore that.
+          # NOTE: -d paths resolve from the sasjs project root (sas/mocks),
+          # not the cwd (sas/mocks/sasjs).
+          sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
 
       - name: Prepare and run frontend and cypress
         timeout-minutes: 35
@@ -139,14 +147,14 @@ jobs:
           npm run postinstall
           # Prepare index.html to SASJS local
           replace-in-files --regex='serverUrl=".*?"' --replacement='serverUrl="http://localhost:5000"' ./src/index.html
-          replace-in-files --regex='appLoc=".*?"' --replacement='appLoc="/Public/app/devtest"' ./src/index.html
+          replace-in-files --regex='appLoc=".*?"' --replacement='appLoc="/Public/app/dc"' ./src/index.html
           replace-in-files --regex='serverType=".*?"' --replacement='serverType="SASJS"' ./src/index.html
           replace-in-files --regex='"hosturl".*' --replacement='hosturl:"http://localhost:4200",' ./cypress.config.ts
           cat ./cypress.config.ts
           # Start frontend and run cypress
           # timeout 1800: SIGTERM after 30 min so Cypress can flush video/screenshots
           # before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
-          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
+          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
 
       - name: Zip Cypress videos
         if: always()

.gitea/workflows/release.yaml

@@ -125,8 +125,16 @@ jobs:
           cd ./sas/mocks/sasjs
           npm install -g @sasjs/cli
           npm install -g replace-in-files-cli
+          # Remove any previous deployment (drive folder delete API) so the
+          # deploy and makedata start from a clean appLoc
+          curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
           sasjs cbd -t server-ci
-          # sasjs request services/admin/makedata -t server-ci -d ./deploy/makeData4GL.json -c ./deploy/requestConfig.json -o ./output.json
+          # Seed the DC database (mock data files on the drive).
+          # makedata replies with HTML (it is normally called as a URL redirect),
+          # which the CLI reports as "invalid Json string" - ignore that.
+          # NOTE: -d paths resolve from the sasjs project root (sas/mocks),
+          # not the cwd (sas/mocks/sasjs).
+          sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
 
       - name: Prepare and run frontend and cypress
         run: |
@@ -138,7 +146,7 @@ jobs:
           npm run postinstall
           # Prepare index.html to SASJS local
           replace-in-files --regex='serverUrl=".*?"' --replacement='serverUrl="http://localhost:5000"' ./src/index.html
-          replace-in-files --regex='appLoc=".*?"' --replacement='appLoc="/Public/app/devtest"' ./src/index.html
+          replace-in-files --regex='appLoc=".*?"' --replacement='appLoc="/Public/app/dc"' ./src/index.html
           replace-in-files --regex='serverType=".*?"' --replacement='serverType="SASJS"' ./src/index.html
           replace-in-files --regex='"hosturl".*' --replacement='hosturl:"http://localhost:4200",' ./cypress.config.ts
           cat ./cypress.config.ts
@@ -294,8 +302,10 @@ jobs:
         run: |
           cd sas
           cp sasjsbuild/viya.json ../client/dist/viya.json
-          cd ..
-          zip -r frontend.zip ./client/dist
+          # Zip from *inside* dist so the archive holds the frontend files at its
+          # root rather than under a client/dist/ prefix (see #147).
+          cd ../client/dist
+          zip -r ../../frontend.zip .
 
       - name: Release Typedoc
         run: |

.gitignore

@@ -22,4 +22,6 @@ sasjsresults
 .sasjsrc
 client/.npmrc
 *~
-.lighthouseci
+.lighthouseci
+audit.log
+tmp/

AGENTS.md

@@ -7,7 +7,7 @@ Read **`CONTEXT.md`** at the repo root first - it is the domain glossary and ori
 Data Controller spans three sibling repos (usually checked out side by side):
 
 - **`dc`** (this repo) - the product source (Angular client + SAS backend).
-- **`docs.datacontroller.io`** - the user-facing product documentation (MkDocs). User-facing behaviour is documented there; the `.agent/docs/` deep-dives here link out to it.
+- **`docs.datacontroller.io`** - the user-facing product documentation (MkDocs). User-facing behaviour is documented there; the `.agents/docs/` deep-dives here link out to it.
 - **`datacontroller.io`** - the marketing site, blog and feed (Gatsby).
 
 ## Git
@@ -16,7 +16,7 @@ Do NOT auto-commit. Never run `git commit` (or `git push`) unless the user expli
 
 ## CHANGELOG and versioning
 
-Do NOT edit `CHANGELOG.md` by hand, and do NOT bump the `version` in `package.json`. The release pipeline (semantic-release, in `.gitea/workflows/release.yaml`) generates both automatically from Conventional Commit messages on every push to `main`. Manual entries collide with the pipeline's release commit and are overwritten anyway. Control the changelog through your commit messages instead. See `.agent/docs/releases-and-changelog.md`.
+Do NOT edit `CHANGELOG.md` by hand, and do NOT bump the `version` in `package.json`. The release pipeline (semantic-release, in `.gitea/workflows/release.yaml`) generates both automatically from Conventional Commit messages on every push to `main`. Manual entries collide with the pipeline's release commit and are overwritten anyway. Control the changelog through your commit messages instead. See `.agents/docs/releases-and-changelog.md`.
 
 ## Markdown files
 
@@ -37,9 +37,11 @@ Never consider a change complete until the relevant linters pass on the files yo
 
 Data Controller must run entirely locally (offline / on-prem, no internet access). The built product must never fetch assets from remote servers — no external fonts, images, scripts, stylesheets, CDN links, or remote URLs in meta tags (e.g. `og:image`, `og:url`, `itemprop="image"`). All assets must be bundled and served locally.
 
-## The .agent folder
+## The .agents folder
 
-Agent-related content lives in `.agent/`: technical/agent-facing documentation goes in `.agent/docs/` (not `docs/`), and skills in `.agent/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agent/docs/`.
+Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
+
+Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
 
 ## Code comments and test names
 

CHANGELOG.md

@@ -1,3 +1,141 @@
+# [7.15.0](https://git.datacontroller.io/dc/dc/compare/v7.14.2...v7.15.0) (2026-09-23)
+
+
+### Bug Fixes
+
+* **mocks:** keep special missings in the DIFF, and add the clip recording spec ([d124ce3](https://git.datacontroller.io/dc/dc/commit/d124ce3b36f8f5153e8d7a576d4bec3189524ed5))
+* **mocks:** list a column's own special missing in its dropdown ([b8f16a6](https://git.datacontroller.io/dc/dc/commit/b8f16a63827310f68629f2e016a4fb876593c37c))
+* **mocks:** make the approval path work, and record the approval scene ([12847cf](https://git.datacontroller.io/dc/dc/commit/12847cf07165d5aefa61592815cab0a37245e31f))
+* **validator:** a range rule ignores a missing value ([70dae4b](https://git.datacontroller.io/dc/dc/commit/70dae4b7013ac825eb77d3dc2b622980c0e4af05))
+* **validator:** keep the regular missing in a numeric dropdown source too ([a639bca](https://git.datacontroller.io/dc/dc/commit/a639bca70264e8a6a81795892b9ed63ba58d5f73))
+* **validator:** primary keys are NOT NULL, and a strict dropdown can match a special missing ([586a41a](https://git.datacontroller.io/dc/dc/commit/586a41ad49670ee795284a83cf8535711ce14d2c))
+* **validator:** reject a special missing on a NOT NULL column ([6d85bce](https://git.datacontroller.io/dc/dc/commit/6d85bce2a0c528e8eab9ee4abbade8c682ec3a1b))
+
+
+### Features
+
+* **validator:** compare MINVAL and MAXVAL in SAS's own order ([3467322](https://git.datacontroller.io/dc/dc/commit/3467322e99b0c3f6fda5654d35b507fdb2cb1c22))
+
+## [7.14.2](https://git.datacontroller.io/dc/dc/compare/v7.14.1...v7.14.2) (2026-09-22)
+
+
+### Bug Fixes
+
+* **release:** put frontend files at the root of frontend.zip ([7771a24](https://git.datacontroller.io/dc/dc/commit/7771a24b9ce55751253686885e4848a292817a38)), closes [#147](https://git.datacontroller.io/dc/dc/issues/147)
+* **security:** accept the format-catalog form when validating a libds ([0fa5da8](https://git.datacontroller.io/dc/dc/commit/0fa5da8abfc303ce3b18beb4c01484dcce508439))
+* **security:** escape col-info dropdown and origin-check VA replay ([c9eed6d](https://git.datacontroller.io/dc/dc/commit/c9eed6dae717414b874ef6f3d60312ea0bb5a0da))
+* validate request inputs and add admin gates to public services ([be86000](https://git.datacontroller.io/dc/dc/commit/be86000fe0f0858edcbecf71d426a2ff81257247))
+
+## [7.14.1](https://git.datacontroller.io/dc/dc/compare/v7.14.0...v7.14.1) (2026-09-17)
+
+
+### Bug Fixes
+
+* **admin:** return refreshcatalog output as webout JSON ([1dccf61](https://git.datacontroller.io/dc/dc/commit/1dccf61f6290e50ebc365964fab864cb93f8d1b5))
+* **deps:** bump @sasjs/cli to 4.20.3 ([d301fc2](https://git.datacontroller.io/dc/dc/commit/d301fc2b4dc86b7eb3027b8c8965dd7d1079363e))
+* **deps:** bump @sasjs/core to 5.2.7 ([09f31ce](https://git.datacontroller.io/dc/dc/commit/09f31ce6da1e6a286c4e05ac55fb1d0c801d62f5))
+* **deps:** bump @sasjs/core to 5.2.8 ([c73817f](https://git.datacontroller.io/dc/dc/commit/c73817fb6a8c19d970a19b98d5ef79c05243cc8d))
+* **editors:** declare the mp_validatecol dependency in stagedata and loadfile ([a556be4](https://git.datacontroller.io/dc/dc/commit/a556be4c577020c8e3147de4d3e71ab84fb0d174))
+* **hooks:** make the pre-commit size check actually run ([5383803](https://git.datacontroller.io/dc/dc/commit/53838036e635cd6f33dccc4534d6f258d3b3da5d))
+* **hooks:** require the repo-pinned gitleaks, drop the fallbacks ([fbcde41](https://git.datacontroller.io/dc/dc/commit/fbcde4132147240fed21a88d44f62b68f58cf67c))
+* **security:** escape cell values in status renderers to prevent DOM XSS ([cfd8f06](https://git.datacontroller.io/dc/dc/commit/cfd8f064352940cd57cf19c86504036f0a7edd8d))
+* **security:** validate libds in stagedata and loadfile ([2d31e5a](https://git.datacontroller.io/dc/dc/commit/2d31e5a8b957d30057eb749c2d7f246170af8c8f))
+* **viewdata:** return the full row cap from a table search ([199b57d](https://git.datacontroller.io/dc/dc/commit/199b57d9f9fb4b00a229d23e25e22e5dee471d2a))
+
+# [7.14.0](https://git.datacontroller.io/dc/dc/compare/v7.13.0...v7.14.0) (2026-09-04)
+
+
+### Features
+
+* **system:** show session timezone on system information screen ([966dc07](https://git.datacontroller.io/dc/dc/commit/966dc072e0cb8bb44bd93a7682c3d7335428d688))
+
+# [7.13.0](https://git.datacontroller.io/dc/dc/compare/v7.12.0...v7.13.0) (2026-09-03)
+
+
+### Bug Fixes
+
+*  dynamic js mocks ([3b9344c](https://git.datacontroller.io/dc/dc/commit/3b9344cb602b609774737dcbb6df6756a2a08edb))
+* adding formula types ([c07a01a](https://git.datacontroller.io/dc/dc/commit/c07a01a38da6a6e4edaf014ef2fd4d2691b29698))
+* address hermes review feedback (formula quoting, cell revert, addRow guard) ([36963aa](https://git.datacontroller.io/dc/dc/commit/36963aa74627b861aee9ec91428a8dbee0946dfc))
+* agent skills and nextviya deploys ([f7db871](https://git.datacontroller.io/dc/dc/commit/f7db8719f5fe860605f4e6da81bad925d7412599))
+* bump core for mp_validate fix ([5a44b28](https://git.datacontroller.io/dc/dc/commit/5a44b2804feddbd616ece1d721865e02bfbedda8))
+* CAS support for REPLACE type plus docs ([ea05f07](https://git.datacontroller.io/dc/dc/commit/ea05f0718047a0ac54cffc58938b6d06f5a112e2))
+* clarify debug comment is permanent, extend ComputeContextDetails with Viya response fields ([72484ae](https://git.datacontroller.io/dc/dc/commit/72484ae8441715813983faa221be348cf421b735))
+* core major bump plus autofix of viya context on deploy ([6527c10](https://git.datacontroller.io/dc/dc/commit/6527c10f262249187c27723d983edd32c3848d72))
+* **core:** bump to v5 (breaking change) ([59e9e96](https://git.datacontroller.io/dc/dc/commit/59e9e96f5a6a6c9c0560cbb8387982611a0c8ca4))
+* **cypress:** correct mock data, test expectations, and abort modal ([13c10be](https://git.datacontroller.io/dc/dc/commit/13c10be30faae736d27b584e0507eaab702d2d32))
+* **cypress:** make licensing combined-key tests order-independent ([4db8d62](https://git.datacontroller.io/dc/dc/commit/4db8d62dca9257764438b00ef034103a0703111a))
+* **deploy:** Viya deploy checks, startup diagnostics, and chunked deploy script ([917925f](https://git.datacontroller.io/dc/dc/commit/917925f55393aca919c2a7c7cd7dbecadbc24ace)), closes [#303](https://git.datacontroller.io/dc/dc/issues/303) [#200](https://git.datacontroller.io/dc/dc/issues/200) [#125](https://git.datacontroller.io/dc/dc/issues/125)
+* **deps:** align @angular/* packages to the same lockstep version ([24e6297](https://git.datacontroller.io/dc/dc/commit/24e62971879e61eef0fc3ea9f07796b92231dbe3))
+* **deps:** override nanoid to 3.3.18 in sas/ ([d57ae03](https://git.datacontroller.io/dc/dc/commit/d57ae03fc44770f41de5dbe0389d33cd11fc0971))
+* **deps:** pin @handsontable/angular-wrapper to 18.0.0 ([2d7df66](https://git.datacontroller.io/dc/dc/commit/2d7df66fc28ffe07e859920c052b9818ed58647f))
+* **deps:** pin babel-loader to resolve an unresolvable peer conflict ([3bf3bf0](https://git.datacontroller.io/dc/dc/commit/3bf3bf0dde943246380528d5af7c46ad1f518c79))
+* **deps:** regenerate lockfile with Node 24 and pin handsontable to 18.0.0 ([7da29bd](https://git.datacontroller.io/dc/dc/commit/7da29bdb1180a9f50055380c1df3649283e9cdf6))
+* **deps:** regenerate lockfile with strict peer-dep resolution to fix npm ci in CI ([efd2e18](https://git.datacontroller.io/dc/dc/commit/efd2e18c7b09f80655528a2f6985034f5a41dcf3))
+* **deps:** regenerate package-lock.json to resolve npm ci sync errors ([a501903](https://git.datacontroller.io/dc/dc/commit/a501903e6d4276f1b4eba8f8ae0feccafcfbcf86))
+* **deps:** resolve npm audit findings and align @angular/* to the same lockstep version ([b9e4b27](https://git.datacontroller.io/dc/dc/commit/b9e4b2733fa01e2b8060dab54277000a2c6252b8))
+* **deps:** resolve npm audit findings via direct lockfile patch, not regeneration ([4d1bfa6](https://git.datacontroller.io/dc/dc/commit/4d1bfa6343342a167bdc315204866a18bcbca8ed))
+* **deps:** resolve npm audit vulnerabilities in production dependencies ([257f69c](https://git.datacontroller.io/dc/dc/commit/257f69ccc6dd070daf8c73f597a08d01e8ea6888))
+* **deps:** scope the brace-expansion override to its actual vulnerable chain ([f9d061c](https://git.datacontroller.io/dc/dc/commit/f9d061c48901d9ef131303940f0c2e2948b12fdf))
+* dynamic getdynamiccolvals.js ([d59ef44](https://git.datacontroller.io/dc/dc/commit/d59ef442266bd4823f5cad3f7f4985e3df9afb48))
+* **editor:** clear sort before reading cells to preserve on cancel ([4137ebd](https://git.datacontroller.io/dc/dc/commit/4137ebdf6b6563eeac4f805708c3b4ffcd099fb1))
+* **editor:** escape formula-looking values in uploaded Excel data ([e69df3d](https://git.datacontroller.io/dc/dc/commit/e69df3deb22d80b1371b63a5ae1c6a44538b00bc))
+* **editor:** fix formula $-substitution corruption and sorted-grid row/formula desync ([e139c37](https://git.datacontroller.io/dc/dc/commit/e139c37fbfe46123cce2cbfcf7e919bc68e19a2f))
+* **editor:** let the column info dropdown's text be selected and copied ([1e516f4](https://git.datacontroller.io/dc/dc/commit/1e516f4012947c3f42dec1f3a46fa364c39bfe3e))
+* **editor:** re-mark a reverted cell as auto-escaped so "Apply as formula" works again ([12091c4](https://git.datacontroller.io/dc/dc/commit/12091c4044b0982c36b7d6011b51625bfb6b8cb5))
+* **editor:** resolve a primary key's live formula to its computed value, not the raw formula text ([d72e19a](https://git.datacontroller.io/dc/dc/commit/d72e19a3086ef578a1304fe032e1088c6ef60cf7))
+* **editor:** resolve computed value for a live formula in numeric columns too ([2cc8489](https://git.datacontroller.io/dc/dc/commit/2cc8489892638aaf97fd5999ebc6086ffd674a76))
+* **editor:** restore multi-column sort with one call, not a loop ([8fafeb7](https://git.datacontroller.io/dc/dc/commit/8fafeb7923bc7dae178c4fed0898195d5b5825a6))
+* **editor:** size table-header buttons and title to content, not fixed grid thirds ([d0a7561](https://git.datacontroller.io/dc/dc/commit/d0a7561f1aeb67b214e6a473cc6235509daa80bb))
+* **editor:** stop dc.row_status from being submitted to the backend ([dc0f6a7](https://git.datacontroller.io/dc/dc/commit/dc0f6a7baa128e0b75d7341ac7c03dcd4b1ffee5))
+* **editor:** stop turning missing columns into undefined, and fix column lookup for sparse rows ([a104f78](https://git.datacontroller.io/dc/dc/commit/a104f78645bc7777fee672956d3f028ff52ed9ba))
+* **editor:** submit a live formula's computed value, not its raw text ([b7d3106](https://git.datacontroller.io/dc/dc/commit/b7d31066ae5272856ea442d0f76d2f0752015dbb))
+* ensure cleanup of casuser temp table in error condition ([c4123e5](https://git.datacontroller.io/dc/dc/commit/c4123e5c965ba3f78ec8cf32f915f263d218ca56))
+* ensuring formats arrive from backend in getstagetable.sas ([daacb49](https://git.datacontroller.io/dc/dc/commit/daacb49c8a2c6a266dd842a8b7ccdebaa992a1d7))
+* escape regex metacharacters in context name before prxchange ([9023eb2](https://git.datacontroller.io/dc/dc/commit/9023eb2a393ee8441bab5b9c5ff4f53b500e0e91))
+* failing test ([f213d24](https://git.datacontroller.io/dc/dc/commit/f213d24f096d814d286bfdac5a28748e97e7293f))
+* fixed npm vuln ([26b55b1](https://git.datacontroller.io/dc/dc/commit/26b55b1bde5e28afde2d4f9889cb452985410c1d))
+* **formulas:** avoid EDIT_STATUS colliding with a real column of that name ([acb97f4](https://git.datacontroller.io/dc/dc/commit/acb97f4bfb1c7d6a0b38dd33207170f6ec2bbd1d))
+* hardening following PR review feedback ([1f4aa6f](https://git.datacontroller.io/dc/dc/commit/1f4aa6fceafc4b51ecc5a19f4789cf5bd018a5d3))
+* history mock ([0082828](https://git.datacontroller.io/dc/dc/commit/00828285435a412e3e5dccb045b053c1096e8876))
+* **licensing:** bridge async proceed() through cy.then and replace Response.arrayBuffer with manual stream reader ([8e51fde](https://git.datacontroller.io/dc/dc/commit/8e51fde524e799652fbd1f0b0763d667cb3a4845))
+* **mocks:** add fixture staged data and MPE_SUBMIT row for stage.cy.ts ([6e2548d](https://git.datacontroller.io/dc/dc/commit/6e2548d78c613d684fc8d874ba11b30fc9b4e32e))
+* **mocks:** add LIBRARYNAME to startupservice, fix viewbox column order ([c0af86e](https://git.datacontroller.io/dc/dc/commit/c0af86ef297334ee531e979754d4d289ed3900ea))
+* **mocks:** limit stage fixture to 1 row, fix column order ([f25ecd8](https://git.datacontroller.io/dc/dc/commit/f25ecd88283433770f56167774617f0e2089c6ef))
+* **mocks:** look up PK_FIELDS from MPE_TABLES in viewdata, fallback for MPE_AUDIT ([94d2a79](https://git.datacontroller.io/dc/dc/commit/94d2a7970e49845b235c71523418e4a5d2a70396))
+* **mocks:** remove MPE_VALIDATIONS and MPE_ALERTS from DC996664 libref ([4f7b845](https://git.datacontroller.io/dc/dc/commit/4f7b845390d50db50114afa7fa851ddf3e853ed7))
+* **mocks:** remove shadowed saveTableData in rejection.js ([fed0396](https://git.datacontroller.io/dc/dc/commit/fed039673777659594feb9a92a5d9c2dca2cc544))
+* **mocks:** remove SOME_SHORTNUM HARDREGEX from MPE_VALIDATIONS ([34bd660](https://git.datacontroller.io/dc/dc/commit/34bd6607c547c3d40c1129395a28e4cbf0fa1475))
+* **mocks:** rename saveTableData in postdata.js to avoid shadowing ([e4b04b1](https://git.datacontroller.io/dc/dc/commit/e4b04b1d447640542c122dce93d65219719cb094))
+* **mocks:** SAS datetime format in getchangeinfo, login in stage.cy.ts ([c180934](https://git.datacontroller.io/dc/dc/commit/c1809341672ad51baa603f104176fdc302e4c0c1))
+* more mock improvements ([a80927c](https://git.datacontroller.io/dc/dc/commit/a80927cbb6c8a5ce26a1d251dc10cae4bae1eedd))
+* mp_execute dep ([023c29f](https://git.datacontroller.io/dc/dc/commit/023c29f00f74f964825e0ee01768baf44aead332))
+* pin fast-uri to 3.1.7 to close high-severity SSRF/host-confusion advisories ([f4325b2](https://git.datacontroller.io/dc/dc/commit/f4325b205699bf02158c232ffd8c279489778512)), closes [hi#severity](https://git.datacontroller.io/hi/issues/severity)
+* spy for startup ([69b4155](https://git.datacontroller.io/dc/dc/commit/69b41559f7aa66a537b685e26e00c44f1f2568c4))
+* **stage:** submit data via editor flow, use dynamic date assertion ([490d013](https://git.datacontroller.io/dc/dc/commit/490d0137b921c150b431083b847d471aa379bc42))
+* **startup:** show the real startupservice response text on a malformed reply ([9a1b7d0](https://git.datacontroller.io/dc/dc/commit/9a1b7d0f52eebe6bea229dedbcb8f5c19bc96141))
+* tests and realistic getcolvals.js ([5d03164](https://git.datacontroller.io/dc/dc/commit/5d03164b6a3245b47f1278fe962c537f72a8f473))
+* **tests:** new libref ([c0bc87d](https://git.datacontroller.io/dc/dc/commit/c0bc87d331b081d787b4ed3d04c639c139d0fa1b))
+* unnecessary getsubmits call removed ([cc2ff87](https://git.datacontroller.io/dc/dc/commit/cc2ff873fae58f41332dcbe725a49f0815b219b3))
+* using utility macro for webout ([3595bb9](https://git.datacontroller.io/dc/dc/commit/3595bb94993aa054235cce3f41b9d83711907c46))
+
+
+### Features
+
+* **editor:** color row-header status cells and switch modified symbol to ± ([e15f2c2](https://git.datacontroller.io/dc/dc/commit/e15f2c2a3488f43ed4e5814fdc22527961cc5c22))
+* **editor:** generalize cell revert to any overwritten value, not just formulas ([07d586d](https://git.datacontroller.io/dc/dc/commit/07d586da5235a2d94b2b689c774c40851b9a57e2))
+* **editor:** scope live formulas to character columns, not just HARDFORMULA/SOFTFORMULA ones ([92d0d91](https://git.datacontroller.io/dc/dc/commit/92d0d91440ff6d58d10f1af291386a1813cb07d4))
+* **editor:** translate column names to cell references on formula paste ([1031ea7](https://git.datacontroller.io/dc/dc/commit/1031ea7ed7b1a74ecc77bf155f6d329feeaf4b13))
+* **formulas:** flag formula-overwritten cells with revert, harden dc.row_status against SAS name collisions ([a8237b2](https://git.datacontroller.io/dc/dc/commit/a8237b2881b85854bb52c85ca2f17e3368f91f4a))
+* **formulas:** live DC.ROW_STATUS, insert-row formula fixes, UserService singleton fix ([4a8c39b](https://git.datacontroller.io/dc/dc/commit/4a8c39b4c0d6e0edb3493ac3d26d86788ebfdf90))
+* **formulas:** wire HyperFormula into HARDFORMULA/SOFTFORMULA rules ([cbea04c](https://git.datacontroller.io/dc/dc/commit/cbea04c8e129d7d28a626215aae3b67735a036d5))
+* functional JS mocks ([ee52bb8](https://git.datacontroller.io/dc/dc/commit/ee52bb8967e5bcdcfd17500678295e6bffbe2cf0))
+* **licensing:** support combined-key paste, object-format features, and a live key preview ([20ba633](https://git.datacontroller.io/dc/dc/commit/20ba63308717dbde929dbe5974b76f160343de29))
+* **licensing:** warn and block applying a licence key generated for the wrong protocol ([41680e2](https://git.datacontroller.io/dc/dc/commit/41680e2eccad82584bf9fd71a1efd1fd2d8f1a39))
+* **stage:** add Formatted/Unformatted toggle to the staging page ([74c38e1](https://git.datacontroller.io/dc/dc/commit/74c38e16416dcd8be20faa66d9f282a6438522b5))
+* updated configurator for Viya deploy ([df2027d](https://git.datacontroller.io/dc/dc/commit/df2027dab6ae51fb9cabf21e35d1fa5d6e1f7957))
+* **viewboxes:** add drag-to-resize on all four edges and corners ([e5a5bf2](https://git.datacontroller.io/dc/dc/commit/e5a5bf2144d7f35c62368924c7e59f55eeb246ad))
+
 # [7.12.0](https://git.datacontroller.io/dc/dc/compare/v7.11.0...v7.12.0) (2026-07-28)
 
 

CONTEXT.md

@@ -6,9 +6,9 @@ This file is the shared glossary and orientation for the repo. When your output
 
 ## Repository layout
 
-- `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agent/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agent/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
+- `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
 - `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
-- `.agent/docs/` - technical deep-dives (see below). `.agent/skills/` - task skills (handsontable, hyperformula, sas).
+- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
 
 ## Roles
 
@@ -32,10 +32,10 @@ Admins are listed in `&mpeadmins`; admin membership bypasses Row Level Security.
 
 ## Load types
 
-Set per table in `MPE_TABLES.LOADTYPE`. Determines the loader and how history is kept (see [.agent/docs/bitemporal-dataloader.md](.agent/docs/bitemporal-dataloader.md)):
+Set per table in `MPE_TABLES.LOADTYPE`. Determines the loader and how history is kept (see [.agents/docs/bitemporal-dataloader.md](.agents/docs/bitemporal-dataloader.md)):
 
 - **UPDATE** - no history; changed records deleted and re-appended (via `%bitemporal_dataloader`).
-- **REPLACE** - no history; whole table wiped and reloaded. Implemented inline in `%mpe_targetloader`, deliberately *not* via `%bitemporal_dataloader` (see [.agent/docs/replace-load-type.md](.agent/docs/replace-load-type.md)).
+- **REPLACE** - no history; whole table wiped and reloaded. Implemented inline in `%mpe_targetloader`, deliberately *not* via `%bitemporal_dataloader` (see [.agents/docs/replace-load-type.md](.agents/docs/replace-load-type.md)).
 - **TXTEMPORAL** - SCD2-style history on technical (transaction) time.
 - **BITEMPORAL** - full two-dimensional history (business time + technical time).
 - **FORMAT_CAT** - format-catalog load (via `%mp_loadformat`).
@@ -64,11 +64,11 @@ Selectbox seed values for these tables are defined in `sas/sasjs/macros/mpe_make
 - **HARDREGEX** - submission-blocking regex; failing cells are painted red and cannot be submitted.
 - **SOFTREGEX** - display-only warning regex; failing cells are painted yellow but submission is allowed.
 
-Regex rule values are authored in SAS PRX form `/pattern/flags` and limited to 128 chars. See [.agent/docs/regex-validations.md](.agent/docs/regex-validations.md).
+Regex rule values are authored in SAS PRX form `/pattern/flags` and limited to 128 chars. See [.agents/docs/regex-validations.md](.agents/docs/regex-validations.md).
 
 ## Security
 
-- **Row Level Security (RLS)** - server-side `WHERE`-clause generation via `%mpe_filtermaster` / `%mp_filtergenerate`; no data leaves SAS without passing the filter. Works on any engine because it's expressed as a standard SAS `WHERE`. On write, the *inverse* filter is applied to reject out-of-scope rows. See [.agent/docs/row-level-security.md](.agent/docs/row-level-security.md).
+- **Row Level Security (RLS)** - server-side `WHERE`-clause generation via `%mpe_filtermaster` / `%mp_filtergenerate`; no data leaves SAS without passing the filter. Works on any engine because it's expressed as a standard SAS `WHERE`. On write, the *inverse* filter is applied to reject out-of-scope rows. See [.agents/docs/row-level-security.md](.agents/docs/row-level-security.md).
 - **Column Level Security (CLS)** - restricts visibility/editability at column level.
 
 ## Key macros / services
@@ -79,7 +79,7 @@ Regex rule values are authored in SAS PRX form `/pattern/flags` and limited to 1
 
 ## Testing
 
-Backend tests run with the sasjs CLI from `sas/`. `npm run 4gl` compiles+deploys+seeds then `sasjs test -t 4gl` runs tests. Assertions use `%mp_assert()`. See [.agent/docs/testing.md](.agent/docs/testing.md).
+Backend tests run with the sasjs CLI from `sas/`. `npm run 4gl` compiles+deploys+seeds then `sasjs test -t 4gl` runs tests. Assertions use `%mp_assert()`. See [.agents/docs/testing.md](.agents/docs/testing.md).
 
 ## Conventions
 
@@ -88,6 +88,6 @@ Backend tests run with the sasjs CLI from `sas/`. `npm run 4gl` compiles+deploys
 - No external assets - everything must be bundled and served locally.
 - Comments and test names must be self-contained (no "original bug", "this PR", etc.); the only exception is a literal issue-tracker link.
 - Do not auto-commit or push.
-- Never edit `CHANGELOG.md` or bump `package.json` version by hand - the release pipeline generates both from Conventional Commit messages. See [.agent/docs/releases-and-changelog.md](.agent/docs/releases-and-changelog.md).
+- Never edit `CHANGELOG.md` or bump `package.json` version by hand - the release pipeline generates both from Conventional Commit messages. See [.agents/docs/releases-and-changelog.md](.agents/docs/releases-and-changelog.md).
 
-See `AGENTS.md` for the full set of enforced rules (git, linting, no-wrap, no-external-assets, `.agent/` layout).
+See `AGENTS.md` for the full set of enforced rules (git, linting, no-wrap, no-external-assets, `.agents/` layout).

CONTRIBUTING.md

@@ -4,6 +4,21 @@
 ## Workflow guidelines
 [Wiki Page](https://git.datacontroller.io/dc/dc/wiki/Git-Workflow)
 
+## Git hooks
+
+The repo ships its own git hooks in [`.git-hooks/`](./.git-hooks):
+
+- `pre-commit` - scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)
+- `commit-msg` - verifies the commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) standard
+
+The repo `.npmrc` sets `ignore-scripts=true`, so the `prepare` and `postinstall` lifecycle scripts that would normally set `core.hooksPath` during `npm i` never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
+
+```bash
+git config core.hooksPath ./.git-hooks
+```
+
+The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
+
 ## Dependencies that requires licences
 
 [SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)
@@ -54,6 +69,71 @@ cd client
 npm start
 ```
 
+## Mocked backend (frontend development without SAS)
+
+Data Controller ships a full set of JS mock services under `sas/mocks/sasjs/` that mimic the SAS backend using [SASjs Server](https://server.sasjs.io) in JS-only mode. This lets you run the entire frontend (editor grid, approvals, viewer, admin, Excel maps, etc.) without a SAS licence or SAS server. The Cypress E2E tests and Lighthouse CI both run against this setup.
+
+### Prerequisites
+
+- Node.js (same version used for the frontend build)
+- The frontend dependencies installed (`cd client && npm ci` after decrypting sheet-crypto as described above)
+
+### 1. Start SASjs Server
+
+Download the latest Linux (or macOS/Windows) binary from [SASjs Server releases](https://github.com/sasjs/server/releases), unzip it, and create a `.env` file alongside the executable:
+
+```
+RUN_TIMES=js
+NODE_PATH=node
+CORS=enable
+WHITELIST=http://localhost:4200
+```
+
+`RUN_TIMES=js` means the server executes `.js` files as Stored Programs (no SAS binary needed). `CORS=enable` plus `WHITELIST` allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
+
+```
+./api-linux    # or api-macos / api-win.exe depending on your platform
+```
+
+The server runs in desktop mode (no auth) on `http://localhost:5000` by default.
+
+### 2. Deploy the mock services and streaming frontend
+
+From `sas/mocks/sasjs/`:
+
+```
+npm ci
+sasjs cbd -t server-ci
+```
+
+This compiles, builds, and deploys the JS mock services plus the Angular frontend (as a streaming app) to the SASjs Server. The `server-ci` target in `sas/mocks/sasjs/sasjsconfig.json` points to `http://localhost:5000` with appLoc `/Public/app/dc` and stream service name `clickme`.
+
+After deploy, the streamed app is available at: `http://localhost:5000/AppStream/clickme/`
+
+### 3. Run the Angular dev server (optional, for hot reload)
+
+For frontend development with live reload, point `client/src/index.html` at the local SASjs Server and start the dev server:
+
+```
+cd client
+# Edit src/index.html: set serverUrl="http://localhost:5000", appLoc="/Public/app/dc", serverType="SASJS"
+npm start
+```
+
+The app will be at `http://localhost:4200`. It calls the JS mock services on port 5000 via the SASjs adapter (CORS is enabled by the `.env` above).
+
+### How the mocks work
+
+Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). The JS runtime predeclares `fs` and `_webout`; the mock assigns a JSON string to `_webout` that matches the structure the real SAS service would return. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
+
+The mock data includes sample tables (`DC996664.MPE_X_TEST`, `MPE_X_NEW`, `MPE_X_FORMULA_TEST`, etc.) with realistic columns, validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT), and dropdown values. This is sufficient to exercise the full editor grid, approval workflow, viewer, and admin screens.
+
+There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`).
+
+### Re-deploying after changes
+
+After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sasjs/`. After changing the frontend, rebuild (`cd client && npm run build`) then re-deploy so the streaming app picks up the new `client/dist/`. If you are using `ng serve` (step 3), only the mock services need re-deploying; the frontend hot-reloads automatically.
+
 ## GUI Elements
 
 For documentation on the Clarity Design System, including a list of components and example usage, see [our website](https://vmware.github.io/clarity).

client/cypress.config.ts

@@ -9,6 +9,7 @@ export default defineConfig({
     html: true,
     json: false
   },
+
   viewportHeight: 900,
   viewportWidth: 1600,
 
@@ -24,7 +25,7 @@ export default defineConfig({
     serverType: 'SASJS',
     libraryToOpenIncludes_SASVIYA: 'viya',
     libraryToOpenIncludes_SAS9: 'dc',
-    libraryToOpenIncludes_SASJS: 'dc',
+    libraryToOpenIncludes_SASJS: 'dc_jslib',
     debug: false,
     screenshotOnRunFailure: false,
     longerCommandTimeout: 50000,
@@ -37,9 +38,24 @@ export default defineConfig({
       // Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
       // render deterministically regardless of the runner's system locale.
       on('before:browser:launch', (browser, launchOptions) => {
+        // Recording only: the video capture is the browser window, which
+        // contains the Cypress runner - the command log takes the left ~450px
+        // and the app sits in the remaining pane.  Sizing the window is the
+        // only lever that moves the capture, and only for a real Chromium
+        // browser (Electron ignores it).  For a 16:9 pane, solve
+        // (W - 450) / (H - 96) = 16/9: W = 1920 -> H = 923 of window content,
+        // so pass RECORD_WINDOW_SIZE=1920,1010 and crop the pane.
+        // Inert unless the env var is set, so CI is unaffected.
+        if (process.env.RECORD_WINDOW_SIZE && browser.family === 'chromium') {
+          launchOptions.args.push(
+            `--window-size=${process.env.RECORD_WINDOW_SIZE}`
+          )
+        }
+
         if (browser.family === 'chromium' && browser.name !== 'electron') {
           launchOptions.args.push('--lang=en-GB')
         }
+
         return launchOptions
       })
     }

client/cypress/clips/special-missings-clip.cy.ts

@@ -0,0 +1,592 @@
+// Clip script: special missings inside Data Controller's validation rules.
+//
+// This is not a test - it is the recording script for the companion clip, and
+// it deliberately pauses between steps so each beat is readable on video. It
+// lives in cypress/clips (outside cypress/e2e) so the default spec pattern does
+// not pick it up in CI.
+//
+// Record it with:
+//
+//   npx cypress run --spec cypress/clips/special-missings-clip.cy.ts \
+//     --config video=true,viewportWidth=1280,viewportHeight=720
+//
+// The table is TESTDATA.DEMO_01 (seven columns, one rule each):
+//   ID (NOTNULL), AMOUNT (MINVAL 1), SCORE (MAXVAL 100),
+//   REF (SOFTREGEX /^[0-9]+$/), STATUS (SOFTSELECT TESTDATA.DEMO_01.STATUS),
+//   RATING (no rule), LAST_REVIEWED (date9.)
+//
+// One editing session, one submission. The rule scenes are played on row 1 and
+// each value is put back afterwards, so the DIFF that follows carries only the
+// two changes the clip is about; the review screens are reached through the
+// app's own navigation rather than cy.visit, so nothing reloads mid-clip.
+//
+// The beats are deliberately short - the take is played back at ~1.4x on the
+// way out, so a hold that looks tight here reads as a normal pause on the
+// finished clip.
+
+// Caption track: each mark opens a caption and carries the text it shows.
+// The encoder turns consecutive marks into subtitle cues, so the captions are
+// timed by the recording itself rather than by guessed offsets.
+const BEATS = '/tmp/clip-beats.tsv'
+
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+
+// A beat long enough to read on video.
+const beat = (ms = 1500) => cy.wait(ms)
+
+context('special missings clip (DEMO_01)', function () {
+  this.beforeEach(() => {
+    cy.visit(hostUrl + appLocation)
+
+    // The mock estate carries a valid licence key (mock-storage/licence.json),
+    // so the app activates directly with no free-tier banner in shot.
+    cy.get('.nav-tree', { timeout: longerCommandTimeout }).should('exist')
+  })
+
+  it('records the demo in one editing session', () => {
+    cy.writeFile(BEATS, '', { flag: 'w' })
+
+    // ---- Scene 1: the table and its rules ----------------------------------
+    openTableFromTree('testdata', 'demo_01')
+    beat(1200)
+
+    clickOnEdit(() => {
+      // Seven columns, five rules: ID is the key, AMOUNT the floor, SCORE the
+      // ceiling, REF the pattern, STATUS the dropdown, and RATING and
+      // LAST_REVIEWED carry nothing.
+      cy.get('.ht_master tbody tr', { timeout: longerCommandTimeout }).should(
+        'have.length.greaterThan',
+        3
+      )
+      mark(
+        'grid',
+        'TESTDATA.DEMO_01 - eight columns, six rules. ID is the key, and RATING and LAST_REVIEWED carry none.'
+      )
+      beat(2600)
+
+      // ---- Scene 2: typing a special missing ------------------------------
+      // RATING has no rule, so this shows entry on its own. Every beat asserts
+      // the settled cell state before holding, so the recording always rests on
+      // the outcome (flagged or accepted) rather than on a blind pause that
+      // might land before the rule engine has run.
+      mark(
+        'entry',
+        'A numeric cell takes a special missing as a letter or an underscore, with or without a period - row 2 already holds .a.'
+      )
+      typeAndHold(0, 'RATING', 'a', 'accepted') // a single letter is taken
+      mark(
+        'reject',
+        'Two letters, or a letter mixed with a number, are refused. Red means it will not submit.'
+      )
+      typeAndHold(0, 'RATING', 'AB', 'rejected', 2200) // two letters are not
+      typeAndHold(0, 'RATING', '1a', 'rejected', 2200) // nor a number and a letter
+      typeAndHold(0, 'RATING', 'a', 'accepted') // leave it as a special missing
+
+      // ---- Scene 3: NOTNULL refuses both --------------------------------
+      // ID is the key, so NOTNULL applies. A special missing is not a value
+      // here: like a blank, it fails the rule (a real SAS NOT NULL constraint
+      // rejects a special missing as well), and only a number satisfies it -
+      // the original key, put back so the row carries no change into the DIFF.
+      mark(
+        'notnull_blank',
+        "ID is the table's primary key, so NOT NULL is applied to it automatically - a blank fails..."
+      )
+      typeAndHold(0, 'ID', '', 'rejected', 2200) // a blank fails NOTNULL
+      mark(
+        'notnull_missing',
+        '...and so does a special missing: a missing is not a value to NOT NULL.'
+      )
+      typeAndHold(0, 'ID', 'A', 'rejected', 2200) // so does a special missing
+      mark(
+        'notnull_number',
+        'A number satisfies it, and the row key goes back.'
+      )
+      typeAndHold(0, 'ID', '1', 'accepted') // a number is what it wants
+
+      // ---- Scene 4: the pattern still applies -----------------------------
+      // REF carries SOFTREGEX /^[0-9]+$/, and a special missing is not exempt
+      // from it. The amber cell is a soft rule warning rather than a block,
+      // which the caption on this beat says out loud - the pattern itself is
+      // only visible in the cell's native title, which a screencast does not
+      // capture.
+      mark(
+        'softregex',
+        'REF carries SOFTREGEX /^[0-9]+$/. Amber is a soft warning - it warns, it does not block.'
+      )
+      typeIntoCell(0, 'REF', 'A')
+      getCellByHeaderAndRow(0, 'REF').should('have.class', 'dc-warning-cell')
+      beat(3400)
+      typeIntoCell(0, 'REF', '1001') // put the reference back
+      getCellByHeaderAndRow(0, 'REF').should(
+        'not.have.class',
+        'dc-warning-cell'
+      )
+      beat(600)
+
+      // ---- Scene 5: the dropdown lists the missing -------------------------
+      // STATUS carries a SOFTSELECT whose list is taken from the column itself
+      // (the library.member.column form), and that column holds a special
+      // missing - so the dropdown offers it alongside the ordinary values, as
+      // the bare letter SAS produces for it.
+      mark(
+        'dropdown',
+        "STATUS carries a SOFTSELECT, and its list is the column's own values."
+      )
+      openDropdown(0, 'STATUS')
+        .should('have.length', 4)
+        .then(($items: any) => {
+          const texts = [...$items].map((td: any) => td.innerText.trim())
+          expect(texts).to.include('A')
+        })
+      beat(2400)
+
+      mark(
+        'dropdown_pick',
+        'So the special missing the column holds is offered as a bare letter, first - a missing sorts below every number.'
+      )
+      pickFromDropdown('A')
+      getCellByHeaderAndRow(0, 'STATUS').should('contain.text', 'A')
+      beat(1800)
+      typeIntoCell(0, 'STATUS', '1') // put the status back
+      beat(600)
+
+      // ---- Scene 6: a range rule compares in SAS order, missings included
+      mark(
+        'minval',
+        'AMOUNT has MINVAL 1. A missing sorts below every number, so it is below the floor.'
+      )
+      typeAndHold(0, 'AMOUNT', 'A', 'rejected', 2200) // a missing is below the floor
+      mark(
+        'maxval',
+        'SCORE has MAXVAL 100 - the same missing is below the ceiling, so it passes.'
+      )
+      typeAndHold(0, 'SCORE', 'A', 'accepted', 2200) // a missing is below the ceiling
+      mark(
+        'grade',
+        'GRADE takes MINVAL .A and MAXVAL .C. The missings have an order of their own: .B is inside the range, .D is outside it.'
+      )
+      typeAndHold(0, 'GRADE', '.B', 'accepted', 2400) // .B is between .A and .C
+      typeAndHold(0, 'GRADE', '.D', 'rejected', 2400) // .D is above .C
+
+      mark('abort', 'Submitting with an invalid cell aborts.')
+
+      // Submit while AMOUNT is invalid - the modal reports it.
+      submitTable(() => {
+        cy.get('.modal-body', { timeout: longerCommandTimeout }).should(
+          'contain.text',
+          'Invalid Values are Present'
+        )
+        beat(2400)
+        // Close the abort so the editor is clean for the next scene.
+        cy.get('clr-modal.clr-abort-modal .modal-footer button')
+          .contains('Close')
+          .click({ force: true })
+        beat(800)
+      })
+
+      // ---- Scene 7: put the test values back, then make the real change ----
+      mark(
+        'clean',
+        'Back to a clean row. Row 2 already holds .a, so this changes one special missing to another.'
+      )
+      typeIntoCell(0, 'AMOUNT', '120')
+      typeIntoCell(0, 'SCORE', '82')
+      typeIntoCell(0, 'GRADE', '.a')
+      typeIntoCell(0, 'RATING', '4')
+      beat(800)
+
+      // Row 2 (ID 2) already carries a special missing in RATING, so this is a
+      // change from one special missing to another, and the date column gives
+      // the formatted / unformatted switch something to switch.
+      typeAndHold(1, 'RATING', 'B', 'accepted')
+      typeIntoDateCell(1, 'LAST_REVIEWED', '2026-01-15')
+      beat(1500)
+
+      submitTable(() => {
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+          .click()
+        beat(2500)
+      })
+    })
+
+    // ---- Scene 8: the queue, then the DIFF (in-app navigation) ------------
+    mark('submitted', 'Submitted - the queue shows it waiting for approval.')
+    goToReviewNav()
+    beat(1800)
+
+    // The submit queue lists oldest first, so the row we just created is last.
+    cy.get('app-submitter clr-datagrid clr-dg-row', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .last()
+      .click({ force: true })
+    beat(1800)
+
+    cy.get('app-approve-details .card', { timeout: longerCommandTimeout })
+      .should('exist')
+      .should('be.visible')
+    beat(1000)
+
+    // The DIFF table is wider than the frame, so the two columns that matter
+    // (RATING and LAST_REVIEWED) sit off the right edge until it is scrolled.
+    scrollDiffToEnd()
+    beat(1200)
+
+    // Only those two cells changed: the rule scenes were put back, so the DIFF
+    // is the two changes the clip is about and nothing else.
+    getDiffCell('RATING')
+      .should('contain.text', '.b')
+      .should('have.class', 'ch')
+    getDiffCell('LAST_REVIEWED')
+      .should('contain.text', '15JAN2026')
+      .should('have.class', 'ch')
+    cy.get('app-approve-details .tableCont tbody tr td:not(.ch)').should(
+      'have.length.greaterThan',
+      3
+    )
+    mark(
+      'diff',
+      'The DIFF compares staged with base: one changed row, two changed cells.'
+    )
+    beat(2600)
+
+    // ---- Scene 9: the staged data -----------------------------------------
+    // What the approval is actually acting on: the staged row, still holding
+    // the special missing, before it reaches the base table.
+    mark(
+      'staged',
+      'The staged row, before approval - still holding the special missing.'
+    )
+    clickButton('VIEW STAGED DATA')
+    // 'Basic Submitted Details' is on the staged screen only - asserting
+    // 'Staged Data' alone would be satisfied by the button that was just
+    // clicked, which is how a beat can pass without ever leaving the DIFF.
+    cy.get('body', { timeout: longerCommandTimeout })
+      .should('contain.text', 'Basic Submitted Details')
+      .and('contain.text', 'Base Table')
+    beat(4200)
+    mark('staged_end', '')
+
+    // ---- Scene 10: the approver opens it, and switches the format ----------
+    goToReviewNav()
+    beat(1500)
+    openApproveTab()
+    beat(1500)
+
+    cy.get('app-approve clr-datagrid clr-dg-row a.color-green', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .last()
+      .click({ force: true })
+    beat(1800)
+
+    cy.get('#acceptBtn', { timeout: longerCommandTimeout })
+      .should('exist')
+      .should('not.be.disabled')
+    beat(1000)
+
+    // Same scroll as the submitter view, so the date column is in frame when
+    // the format is switched.
+    scrollDiffToEnd()
+    beat(1200)
+
+    mark(
+      'approve',
+      'The approver opens the same submission. Hovering a changed cell shows the value it replaced.'
+    )
+
+    // Hovering the two changed cells shows what each replaced - that is how the
+    // two special missings are told apart, not just the before/after of one.
+    hoverDiffCell('RATING', 'Original value is: .a')
+    beat(2800)
+
+    mark('hover_date', '...including the date it replaced.')
+    hoverDiffCell('LAST_REVIEWED', 'Original value is: 29FEB2024')
+    beat(2800)
+
+    mark(
+      'toggle',
+      'The formatted / unformatted switch shows the value as SAS stores it: 24121.'
+    )
+    cy.get('.formatted-values-toggle').should('have.text', 'Formatted').click()
+    cy.get('.formatted-values-toggle').should('have.text', 'Unformatted')
+    getDiffCell('LAST_REVIEWED').should('contain.text', '24121')
+    getDiffCell('RATING').should('contain.text', '.b')
+    beat(2800)
+
+    cy.get('.formatted-values-toggle').click()
+    cy.get('.formatted-values-toggle').should('have.text', 'Formatted')
+    getDiffCell('LAST_REVIEWED').should('contain.text', '15JAN2026')
+    beat(1000)
+
+    // ---- Scene 11: approve, and the change is in the history --------------
+    mark('accepted', 'Accepted - the history records it as APPROVED.')
+    cy.get('#acceptBtn').click()
+    cy.url({ timeout: longerCommandTimeout }).should(
+      'include',
+      '/review/history'
+    )
+    cy.get('app-history clr-datagrid clr-dg-row', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .first()
+      .should('contain.text', 'APPROVED')
+    beat(2800)
+  })
+})
+
+// ---------------------------------------------------------------------------
+// Helpers (mirrored from the e2e specs so the clip drives the same UI paths)
+// ---------------------------------------------------------------------------
+
+/**
+ * Caption track. Each mark opens a caption and carries the text it shows; the
+ * encoder turns consecutive marks into subtitle cues, so the captions are timed
+ * by the recording itself rather than by guessed offsets.
+ *
+ * The timestamp has to be taken inside a `cy.then()`: Cypress evaluates a
+ * command's arguments when the command is *queued*, so `Date.now()` passed to
+ * `cy.writeFile` directly would give every mark the same value - the moment the
+ * spec body ran.
+ */
+const mark = (label: string, text: string) => {
+  cy.then(() => {
+    cy.writeFile(BEATS, `${label}\t${text}\t${Date.now()}\n`, { flag: 'a+' })
+  })
+}
+
+const typeIntoCell = (rowIndex: number, header: string, value: string) => {
+  getCellByHeaderAndRow(rowIndex, header)
+    .dblclick({ force: true })
+    .then(() => {
+      cy.focused().clear().type(`${value}{enter}`)
+    })
+}
+
+/**
+ * A date-formatted column edits through an HTML date input, where cy.type()
+ * refuses anything but a bare YYYY-MM-DD string (so no {enter} in the same
+ * call). Set the value through the DOM and commit it with the Enter keydown
+ * that Handsontable listens for.
+ */
+const typeIntoDateCell = (rowIndex: number, header: string, value: string) => {
+  getCellByHeaderAndRow(rowIndex, header)
+    .dblclick({ force: true })
+    .then(() => {
+      cy.focused()
+        .then(($i: any) => {
+          const el = $i[0]
+          el.value = value
+          el.dispatchEvent(new Event('input', { bubbles: true }))
+          el.dispatchEvent(new Event('change', { bubbles: true }))
+        })
+        .trigger('keydown', { key: 'Enter', keyCode: 13, which: 13 })
+    })
+}
+
+/**
+ * Opens the selectbox on a cell and returns its list entries. Handsontable
+ * renders the arrow itself (`.htAutocompleteArrow`); the list is a
+ * `.handsontable.listbox` in the app document.
+ */
+const openDropdown = (rowIndex: number, header: string) => {
+  getCellByHeaderAndRow(rowIndex, header).within(() => {
+    cy.get('.htAutocompleteArrow').click({ force: true })
+  })
+
+  return cy.get('.handsontable.listbox td', { timeout: longerCommandTimeout })
+}
+
+/** Picks an entry from the open selectbox. */
+const pickFromDropdown = (value: string) => {
+  cy.get('.handsontable.listbox td').contains(value).click({ force: true })
+}
+
+/** Reaches the review area through the app's own navigation (no reload). */
+const goToReviewNav = () => {
+  cy.get('.nav-link', { timeout: longerCommandTimeout })
+    .contains('REVIEW')
+    .click({ force: true })
+}
+
+/** Opens the APPROVE tab within the review area. */
+const openApproveTab = () => {
+  cy.get('.nav-link', { timeout: longerCommandTimeout })
+    .contains('APPROVE')
+    .click({ force: true })
+}
+
+/**
+ * Clicks a button by its visible text. The match is case-insensitive: the app
+ * uppercases button labels in CSS, so the rendered text and the DOM's
+ * textContent differ.
+ */
+const clickButton = (text: string) => {
+  cy.contains('button', new RegExp(text, 'i'), {
+    timeout: longerCommandTimeout
+  }).click({ force: true })
+}
+
+/**
+ * The DIFF table is wider than the recording frame, so scroll its container to
+ * the end - that is what puts the changed RATING and the date column on screen.
+ */
+const scrollDiffToEnd = () => {
+  cy.get('app-approve-details .tableCont', { timeout: longerCommandTimeout })
+    .should('exist')
+    .scrollTo('right', { duration: 1200 })
+}
+
+/**
+ * The DIFF on the review screen is a plain HTML table (`.tableCont`), not
+ * Handsontable: headers are `th`, cells `td`, and a changed cell also carries
+ * the `ch` class whose tooltip holds the value it replaced.
+ */
+const getDiffCell = (headerText: string) => {
+  return cy
+    .get('app-approve-details .tableCont thead tr th', {
+      timeout: longerCommandTimeout
+    })
+    .should(($ths) => {
+      const texts = [...$ths].map((th) => th.innerText.trim())
+      expect(texts).to.include(headerText)
+    })
+    .then(($ths) => {
+      const index = [...$ths].findIndex(
+        (th) => th.innerText.trim() === headerText
+      )
+
+      return cy
+        .get('app-approve-details .tableCont tbody tr')
+        .first()
+        .then(($tr: any) => $tr[0].childNodes[index])
+        .then((cell) => cy.get(cell))
+    })
+}
+
+/**
+ * Reveals the value a changed DIFF cell replaced, and waits until it is really
+ * on screen.
+ *
+ * Clarity shows the tooltip through CSS :hover, which a synthetic
+ * `trigger('mouseover')` does NOT activate - the tooltip stays
+ * `visibility: hidden`, and a `contain.text` assertion still passes because the
+ * text is in the DOM. So this moves the real mouse (cypress-real-events) and
+ * then asserts the computed style. The first real move after another action can
+ * be swallowed, hence the repeat; if the tooltip ever fails to appear the
+ * recording fails rather than quietly showing nothing.
+ */
+const hoverDiffCell = (headerText: string, expected: string) => {
+  getDiffCell(headerText).realHover()
+  beat(300)
+  getDiffCell(headerText).realHover()
+
+  getDiffCell(headerText)
+    .find('.tooltip-content')
+    .should(($t) => {
+      const style = getComputedStyle($t[0] as HTMLElement)
+      expect(style.visibility, 'tooltip visibility').to.eq('visible')
+      expect(Number(style.opacity), 'tooltip opacity').to.be.greaterThan(0)
+      expect($t[0].textContent || '', 'tooltip text').to.contain(expected)
+    })
+}
+
+/**
+ * Types a value into a cell and holds on the settled result. The rule engine
+ * flags the cell (htInvalid) once the edit commits, so asserting the expected
+ * state before the hold means the recording always rests on the outcome, and
+ * waits for it however long the engine takes.
+ *
+ * @param expected 'rejected' when the rule engine should flag the cell,
+ *   'accepted' when the value should settle unflagged.
+ */
+const typeAndHold = (
+  rowIndex: number,
+  header: string,
+  value: string,
+  expected: 'accepted' | 'rejected',
+  hold = 1800
+) => {
+  typeIntoCell(rowIndex, header, value)
+
+  getCellByHeaderAndRow(rowIndex, header).should(
+    expected === 'rejected' ? 'have.class' : 'not.have.class',
+    'htInvalid'
+  )
+
+  beat(hold)
+}
+
+const getCellByHeaderAndRow = (rowIndex: number, headerText: string) => {
+  return cy
+    .get('.ht_clone_top .htCore thead tr th')
+    .should(($ths) => {
+      const texts = [...$ths].map((th) => th.innerText.trim())
+      expect(texts).to.include(headerText)
+    })
+    .then(($ths) => {
+      const index = [...$ths].findIndex(
+        (th) => th.innerText.trim() === headerText
+      )
+
+      return cy
+        .get('.ht_master tbody tr')
+        .then((rows: any) => rows[rowIndex].childNodes[index])
+        .then((cell) => cy.get(cell))
+    })
+}
+
+const clickOnEdit = (callback?: any) => {
+  cy.get('.btnCtrl button.btn-primary', { timeout: longerCommandTimeout })
+    .click()
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
+const submitTable = (callback?: any) => {
+  cy.get('.btnCtrl button.btn-primary')
+    .click()
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let libNode
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            libNode = node
+            break
+          }
+        }
+
+        cy.get(libNode).within(() => {
+          cy.get('.clr-tree-node-content-container > button').click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+}

client/cypress/e2e/editor.cy.ts

@@ -197,7 +197,7 @@ context('editor tests: ', function () {
   // RULE_DEMO_COLS in the getdata.js mock) — MPE_X_TEST stays clean to
   // match the excel fixtures.
   it('7 | Blocks submission of a value that fails HARDREGEX', (done) => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+    openTableFromTree('testdata', 'mpe_x_new')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -236,7 +236,7 @@ context('editor tests: ', function () {
   })
 
   it('8 | Warns (but still submits) a value that fails SOFTREGEX', (done) => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+    openTableFromTree('testdata', 'mpe_x_new')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -280,7 +280,7 @@ context('editor tests: ', function () {
   // A_COL=2, B_COL=10: HARDFORMULA (A_COL * B_COL) computes to 20,
   // SOFTFORMULA (A_COL + B_COL) computes to 12.
   it('9 | HARDFORMULA shows the computed value and blocks direct edits', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -305,7 +305,7 @@ context('editor tests: ', function () {
   })
 
   it('10 | SOFTFORMULA shows the computed default but accepts an override', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -328,7 +328,7 @@ context('editor tests: ', function () {
   })
 
   it('11 | Info dropdown shows the applied formula as √x=<formula>', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -349,7 +349,7 @@ context('editor tests: ', function () {
   })
 
   it('12 | Info dropdown shows the applied HARDREGEX/SOFTREGEX pattern', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+    openTableFromTree('testdata', 'mpe_x_new')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -376,7 +376,7 @@ context('editor tests: ', function () {
   })
 
   it('13 | Info dropdown shows only the applied HARDREGEX pattern when a column has both', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+    openTableFromTree('testdata', 'mpe_x_new')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -396,7 +396,7 @@ context('editor tests: ', function () {
   })
 
   it('14 | REGEX_BOTH_COL: a HARDREGEX failure blocks submission and sets its own tooltip, not yellow', (done) => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+    openTableFromTree('testdata', 'mpe_x_new')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -435,7 +435,7 @@ context('editor tests: ', function () {
   })
 
   it('15 | REGEX_BOTH_COL: SOFTREGEX is ignored entirely when HARDREGEX is present', (done) => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_new')
+    openTableFromTree('testdata', 'mpe_x_new')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -469,7 +469,7 @@ context('editor tests: ', function () {
   })
 
   it("16 | Insert Row above keeps existing rows' formulas aligned with their own shifted data", () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -499,7 +499,7 @@ context('editor tests: ', function () {
   })
 
   it("17 | Insert Row below keeps existing rows' formulas aligned with their own shifted data", () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -526,7 +526,7 @@ context('editor tests: ', function () {
   })
 
   it('18 | Insert Row seeds formulas that live-recalculate once A_COL/B_COL are filled in', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -562,7 +562,7 @@ context('editor tests: ', function () {
   })
 
   it('19 | Row header shows blank for unchanged, ± for a modified row', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -587,7 +587,7 @@ context('editor tests: ', function () {
   })
 
   it('20 | Row header shows - for a delete-marked row and + for a newly inserted row', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -614,7 +614,7 @@ context('editor tests: ', function () {
   })
 
   it('21 | Inserting a row does not falsely mark unrelated rows as modified or revert an existing override', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -644,7 +644,7 @@ context('editor tests: ', function () {
   })
 
   it("22 | DC.ROW_STATUS resolves to a live cell reference, reacting to this row's own edit status", () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -673,19 +673,19 @@ context('editor tests: ', function () {
   })
 
   it('23 | DC.USER_NAME resolves to the current logged-in user', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
         timeout: longerCommandTimeout
       }).then(() => {
-        getCellByHeaderAndRow(0, 'USER_NAME_COL').should('have.text', 'sasdemo')
+        getCellByHeaderAndRow(0, 'USER_NAME_COL').should('not.be.empty')
       })
     })
   })
 
   it("24 | DC.ORIG_VALUE echoes this row's pre-edit value, and is blank for a newly-inserted row", () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -703,7 +703,7 @@ context('editor tests: ', function () {
   })
 
   it('25 | Row header symbols stay aligned with their own row after sorting by another column', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -752,7 +752,7 @@ context('editor tests: ', function () {
   // doesn't recompute the "changed from" text, it just flips which
   // already-computed branch IF() reveals.
   it("26 | Combining DC.ROW_STATUS/DC.USER_NAME/DC.ORIG_VALUE in one formula reveals the frozen 'changed from' text once the row is edited", () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -769,10 +769,11 @@ context('editor tests: ', function () {
             cy.focused().clear().type('999{enter}')
           })
 
-        getCellByHeaderAndRow(0, 'CHANGE_SUMMARY_COL').should(
-          'have.text',
-          'sasdemo changed from orig-1'
-        )
+        getCellByHeaderAndRow(0, 'CHANGE_SUMMARY_COL')
+          .invoke('text')
+          .should((text) => {
+            expect(text).to.match(/^.+ changed from orig-1$/)
+          })
       })
     })
   })
@@ -785,7 +786,7 @@ context('editor tests: ', function () {
   // and proves the live-recalculated value - not a stale one - is what
   // gets sent.
   it('27 | Submits the computed formula value, not the raw formula string', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -835,7 +836,7 @@ context('editor tests: ', function () {
   // other row's formula columns are seeded blank, so only this row is
   // affected.
   it('28 | A formula overwriting real pre-existing data marks the row modified, adds a comment, and can be reverted', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     // Already true on the very first render, before Edit is ever clicked -
     // dataSourceUnchanged is seeded with the formula overlay at load time
@@ -893,7 +894,7 @@ context('editor tests: ', function () {
   // "Revert") must keep showing the live computed value and the modified
   // marker, both in the cancelled edit session and back in read-only view.
   it('29 | Cancelling an edit session after a formula silently overwrote real data keeps the computed value and modified marker, not the stale raw one', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -919,7 +920,7 @@ context('editor tests: ', function () {
   // getStableFormulaBaseCols, which excludes any DC.*-referencing formula
   // rule from the "did a formula overwrite real data" comparison entirely.
   it('30 | DC.*-referencing formula columns never get a "changed value" comment, even when their own raw seed differs from the computed result', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -937,7 +938,7 @@ context('editor tests: ', function () {
   })
 
   it('31 | Reverting a formula value then cancelling keeps it plain, and it stays plain on the next edit session', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -986,7 +987,7 @@ context('editor tests: ', function () {
   // actually sent, not only formula columns - A_COL is a plain numeric
   // column with no HARDFORMULA/SOFTFORMULA rule of its own.
   it('32 | Editing a plain (non-formula) cell marks it overwritten, and Revert restores it', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1015,7 +1016,7 @@ context('editor tests: ', function () {
   })
 
   it('33 | Selecting a range containing one overwritten cell shows Revert and only reverts that cell', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1053,7 +1054,7 @@ context('editor tests: ', function () {
   })
 
   it('34 | Selecting a range with no overwritten cells does not show Revert', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1074,7 +1075,7 @@ context('editor tests: ', function () {
   })
 
   it('35 | A selection spanning a newly-inserted row and an existing overwritten cell only reverts the existing row', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1117,7 +1118,7 @@ context('editor tests: ', function () {
   // already show their raw-vs-computed mismatch on load, with no manual
   // edit needed.
   it('36 | Selecting a block spanning multiple pre-loaded overwritten cells across different rows and columns reverts only the actually-overwritten ones', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1175,7 +1176,7 @@ context('editor tests: ', function () {
   })
 
   it('37 | Selecting an entire column reverts every pre-loaded overwritten cell in it, leaving other columns and out-of-range rows untouched', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1229,7 +1230,7 @@ context('editor tests: ', function () {
   // proves the general revert path still works via a plain live edit on a
   // column that's a string, not a number.
   it('38 | Editing a plain character column with no DQ rule at all marks it overwritten, and Revert restores it', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1265,7 +1266,7 @@ context('editor tests: ', function () {
   // the dropdown were still closing, `.htDropdownMenu` would no longer
   // exist and the `.should()` below would time out.
   it('39 | Left-click or right-click inside the info dropdown does not close it', () => {
-    openTableFromTree(libraryToOpenIncludes, 'mpe_x_formula_test')
+    openTableFromTree('testdata', 'mpe_x_formula_test')
 
     clickOnEdit(() => {
       cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
@@ -1291,6 +1292,613 @@ context('editor tests: ', function () {
       })
     })
   })
+
+  // Row 1 (0-indexed): A_COL=2, B_COL=10 - see the HARDFORMULA/SOFTFORMULA
+  // tests' own comment above. FORMULA_SOFT_COL is a character column (see
+  // character-column-formula-plan.md), so it's formula-capable for pasted
+  // input; unlike FORMULA_HARD_COL it also accepts a user override (test 10).
+  it("40 | Pasting a formula with column names translates them to this row's cell references and evaluates it, on a formula-designated column", () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        getCellByHeaderAndRow(1, 'FORMULA_SOFT_COL').click()
+
+        pasteTextIntoFocusedCell('=A_COL * B_COL')
+
+        getCellByHeaderAndRow(1, 'FORMULA_SOFT_COL').should('have.text', '20')
+      })
+    })
+  })
+
+  // mpe_x_test has zero HARDFORMULA/SOFTFORMULA rules, but formulas are on
+  // for every table, not just ones that declare a formula rule - so
+  // paste-translation still works here too. Uses SOME_BESTNUM * 0 + 1936
+  // so the result is deterministic regardless of the column's actual value.
+  it('41 | Pasting a formula with column names on a table with zero formula rules still translates and evaluates it', () => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        // force: true - same overlay-clone-layer reason as sortByColumn's
+        // own comment below; SOME_CHAR's header sort icon covers the cell.
+        getCellByHeaderAndRow(1, 'SOME_CHAR').click({ force: true })
+
+        pasteTextIntoFocusedCell('=SOME_BESTNUM * 0 + 1936')
+
+        getCellByHeaderAndRow(1, 'SOME_CHAR').should('have.text', '1936')
+      })
+    })
+  })
+
+  it('42 | Pasting a formula referencing a column name inside a quoted string leaves the quoted text untouched', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        getCellByHeaderAndRow(1, 'FORMULA_SOFT_COL').click()
+
+        // A_COL needs a leading/trailing blank to be recognised as a
+        // column-name token (see substituteColumnReferences' own "no
+        // surrounding blanks" test) - matches the space-after-paren
+        // convention already used by this fixture's own CHANGE_SUMMARY_COL
+        // rule value ('=IF( DC.ROW_STATUS ="U",...)').
+        pasteTextIntoFocusedCell(
+          '=IF( A_COL > 0, "A_COL is positive", "negative")'
+        )
+
+        getCellByHeaderAndRow(1, 'FORMULA_SOFT_COL').should(
+          'have.text',
+          'A_COL is positive'
+        )
+      })
+    })
+  })
+
+  it('43 | Pasting a formula directly into an open cell editor (double-click, then paste) also translates and evaluates it', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        getCellByHeaderAndRow(1, 'FORMULA_SOFT_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            pasteTextIntoFocusedCell('=A_COL * B_COL')
+
+            cy.focused().type('{enter}')
+
+            getCellByHeaderAndRow(1, 'FORMULA_SOFT_COL').should(
+              'have.text',
+              '20'
+            )
+          })
+      })
+    })
+  })
+
+  // PLAIN_TEXT_COL has no DQ rule at all - not HARDFORMULA/SOFTFORMULA-
+  // designated, and not referenced by any formula either. Only
+  // backend-sourced `=`-led values are ever auto-escaped - anything the
+  // user pastes or types is left exactly as entered, which genuinely
+  // evaluates as a live formula since formulas are on for every column.
+  it('44 | Pasting a formula-looking value into any character column evaluates it as a real formula - user input is never auto-escaped', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        getCellByHeaderAndRow(1, 'PLAIN_TEXT_COL').click()
+
+        pasteTextIntoFocusedCell('=100 * 100')
+
+        getCellByHeaderAndRow(1, 'PLAIN_TEXT_COL').should('have.text', '10000')
+      })
+    })
+  })
+
+  it('44b | Typing a formula-looking value directly into a character column evaluates it as a real formula too', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        getCellByHeaderAndRow(2, 'PLAIN_TEXT_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused().clear().type('=100 * 100{enter}')
+          })
+
+        getCellByHeaderAndRow(2, 'PLAIN_TEXT_COL').should('have.text', '10000')
+      })
+    })
+  })
+
+  // Rows 5/6/7 (0-indexed 4/5/6) come straight from the mock backend, never
+  // touched by a user - see getdata.js's own comment. Rows 5-6 are a
+  // `=`-led value that must be auto-escaped to literal text on load (not
+  // evaluated - only a *user's* input evaluates live, per test 44 above);
+  // row 7 is already apostrophe-escaped straight from the backend, which
+  // the app must never mistake for its own marker.
+  it('46 | A formula-looking value seeded from the backend into a character column is escaped to literal text on load; a genuinely-literal backend value is left untouched', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        // Row 5 (PK=5): references a column name (B_COL) - never
+        // translated (that only happens for a pasted value), so this is
+        // purely proving the escape, not evaluation.
+        getCellByHeaderAndRow(4, 'PLAIN_TEXT_COL').should(
+          'have.text',
+          '=100 + B_COL'
+        )
+
+        // Row 6 (PK=6): a bare arithmetic expression.
+        getCellByHeaderAndRow(5, 'PLAIN_TEXT_COL').should(
+          'have.text',
+          '=100 + 200'
+        )
+
+        // Row 7 (PK=7): already apostrophe-escaped in the backend data
+        // itself - Handsontable's own formulas-plugin display logic strips
+        // exactly one leading `'` for display regardless of who put it
+        // there, so this also reads as literal text with no leading `'`
+        // visible - the difference from rows 5-6 only shows up in what
+        // gets submitted (test 45) and in "Apply as formula" (test 47).
+        getCellByHeaderAndRow(6, 'PLAIN_TEXT_COL').should(
+          'have.text',
+          '=genuine literal'
+        )
+      })
+    })
+  })
+
+  // Proves requirements 2/5/6 together: an untouched auto-escaped cell
+  // (row 5) submits with the marker stripped back off; a genuinely-literal
+  // backend value (row 7) - never marked, since it never started with `=`
+  // in the first place - submits completely unchanged, leading `'` and all.
+  it('45 | Submits an untouched auto-escaped cell unmarked, and a genuinely-literal backend value completely unchanged', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.intercept('POST', '**/SASjsApi/stp/execute*', (req) => {
+          const bodyText = JSON.stringify(req.body || '')
+
+          if (bodyText.includes('stagedata')) {
+            expect(bodyText).to.include('"PLAIN_TEXT_COL":"=100 + 200"')
+            expect(bodyText).not.to.include("'=100 + 200")
+            expect(bodyText).to.include('"PLAIN_TEXT_COL":"\'=genuine literal"')
+          }
+
+          req.continue()
+        }).as('stpExecute')
+
+        // PLAIN_TEXT_COL itself is never touched by hand on either row -
+        // editing each row's own A_COL instead marks both rows submittable
+        // (row 6 = PK 6, "=100 + 200"; row 7 = PK 7, the genuine literal)
+        // without disturbing PLAIN_TEXT_COL's auto-escaped/untouched state.
+        getCellByHeaderAndRow(5, 'A_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused().clear().type('60{enter}')
+          })
+        getCellByHeaderAndRow(6, 'A_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused().clear().type('70{enter}')
+          })
+
+        submitTable()
+
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+
+        cy.get('#formFields_8').type('round-trip submission test')
+
+        submitTableMessage()
+      })
+    })
+  })
+
+  // Requirement 7: "Apply as formula" only ever appears for / acts on a
+  // cell the app itself auto-escaped.
+  it('47 | "Apply as formula" is hidden when the selection has no auto-escaped cell', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        // Row 7 (PK=7) is genuinely-literal, never auto-escaped.
+        getCellByHeaderAndRow(6, 'PLAIN_TEXT_COL').click({ force: true })
+        getCellByHeaderAndRow(6, 'PLAIN_TEXT_COL').rightclick({ force: true })
+
+        cy.get('.htContextMenu').should(($menu) => {
+          expect($menu.text()).not.to.include('Apply as formula')
+        })
+      })
+    })
+  })
+
+  it('48 | "Apply as formula" appears when the selection has an auto-escaped cell, and only strips the auto-inserted marker from it', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        // Select rows 6-7 (PK 6-7) of PLAIN_TEXT_COL - row 6 auto-escaped
+        // (bare expression, no column-name translation needed), row 7
+        // genuinely literal from the backend.
+        getCellByHeaderAndRow(5, 'PLAIN_TEXT_COL').click({ force: true })
+        getCellByHeaderAndRow(6, 'PLAIN_TEXT_COL').click({
+          force: true,
+          shiftKey: true
+        })
+        getCellByHeaderAndRow(5, 'PLAIN_TEXT_COL').rightclick({ force: true })
+
+        cy.get('.htContextMenu').contains('Apply as formula').click()
+
+        // Row 6: marker stripped, now a real live formula.
+        getCellByHeaderAndRow(5, 'PLAIN_TEXT_COL').should('have.text', '300')
+        // Row 7: never marked, left completely untouched.
+        getCellByHeaderAndRow(6, 'PLAIN_TEXT_COL').should(
+          'have.text',
+          '=genuine literal'
+        )
+      })
+    })
+  })
+
+  // A user-typed formula in any character column (not just HARDFORMULA/
+  // SOFTFORMULA ones) is never auto-escaped and evaluates live (test 44b)
+  // - submission must send that computed value, not the raw formula text,
+  // the same way it already does for HARDFORMULA/SOFTFORMULA columns
+  // (test 27).
+  it('49 | Submits the computed value for a user-typed formula in a non-formula-rule column, not the raw formula text', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.intercept('POST', '**/SASjsApi/stp/execute*', (req) => {
+          const bodyText = JSON.stringify(req.body || '')
+
+          if (bodyText.includes('stagedata')) {
+            expect(bodyText).to.include('"PLAIN_TEXT_COL":2')
+            expect(bodyText).not.to.match(/"PLAIN_TEXT_COL":"?=1\+1/)
+          }
+
+          req.continue()
+        }).as('stpExecute')
+
+        getCellByHeaderAndRow(1, 'PLAIN_TEXT_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused().clear().type('=1+1{enter}')
+          })
+
+        getCellByHeaderAndRow(1, 'PLAIN_TEXT_COL').should('have.text', '2')
+
+        submitTable()
+
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+
+        cy.get('#formFields_8').type('computed formula value submission test')
+
+        submitTableMessage()
+      })
+    })
+  })
+
+  // B_COL is `type: numeric` (DDTYPE N) - coerceNumericRow (beforePaste)
+  // deliberately leaves a non-numeric, formula-looking pasted value alone
+  // rather than forcing it to NaN, so a live formula can end up in a
+  // numeric column too, not just a character one. Submission must resolve
+  // its computed value the same way (test 49) - the raw `=2+2` text would
+  // otherwise reach the backend, which a numeric SAS field can't parse.
+  it('50 | Submits the computed value for a live formula pasted into a numeric column, not the raw formula text', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.intercept('POST', '**/SASjsApi/stp/execute*', (req) => {
+          const bodyText = JSON.stringify(req.body || '')
+
+          if (bodyText.includes('stagedata')) {
+            expect(bodyText).to.include('"B_COL":4')
+            expect(bodyText).not.to.match(/"B_COL":"?=2\+2/)
+          }
+
+          req.continue()
+        }).as('stpExecute')
+
+        getCellByHeaderAndRow(1, 'B_COL').click()
+
+        pasteTextIntoFocusedCell('=2+2')
+
+        getCellByHeaderAndRow(1, 'B_COL').should('have.text', '4')
+
+        submitTable()
+
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+
+        cy.get('#formFields_8').type(
+          'numeric column computed formula value submission test'
+        )
+
+        submitTableMessage()
+      })
+    })
+  })
+
+  // A primary key identifies its own row for the rest of the edit session
+  // (dataModified/classifyRow match rows by PK) - a live, still-
+  // recalculating formula left sitting in a PK cell would let that
+  // identity drift between when it's entered and when submission finally
+  // resolves it, silently dropping the row from what's submitted. So
+  // unlike B_COL above, a formula pasted into PRIMARY_KEY_FIELD resolves
+  // to its computed value immediately, the moment the paste completes,
+  // rather than staying live until submit. Row 4 (PK=5): pasting
+  // `=999+1` evaluates to 1000, a value clearly different from both the
+  // original and every other row's PK, so a passing assertion here can't
+  // be a coincidence (nor a PK collision with another seeded row).
+  it('51 | Resolves a formula pasted into the primary key column to its computed value immediately', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.intercept('POST', '**/SASjsApi/stp/execute*', (req) => {
+          const bodyText = JSON.stringify(req.body || '')
+
+          if (bodyText.includes('stagedata')) {
+            expect(bodyText).to.include('"PRIMARY_KEY_FIELD":1000')
+          }
+
+          req.continue()
+        }).as('stpExecute')
+
+        getCellByHeaderAndRow(4, 'PRIMARY_KEY_FIELD').should('have.text', '5')
+
+        getCellByHeaderAndRow(4, 'PRIMARY_KEY_FIELD').click()
+
+        pasteTextIntoFocusedCell('=999+1')
+
+        getCellByHeaderAndRow(4, 'PRIMARY_KEY_FIELD').should(
+          'have.text',
+          '1000'
+        )
+
+        submitTable()
+
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+
+        cy.get('#formFields_8').type(
+          'primary key computed formula value submission test'
+        )
+
+        submitTableMessage()
+      })
+    })
+  })
+
+  // Nothing gates a HARDFORMULA/SOFTFORMULA rule's BASE_COL by column type
+  // or PK-ness - MPE_X_FORMULA_PK_TEST's composite PK is itself made up of
+  // a HARDFORMULA numeric column and a SOFTFORMULA character column. Left
+  // unresolved, that reproduces the same PK-identity bug test 51 fixes -
+  // just triggered by the rule seeding instead of a user edit, and present
+  // on every row from the very first render rather than only an edited
+  // one. resolvePkFormulaSeedValues (editor.component.ts) freezes both PK
+  // cells to their computed values once, right after the initial table
+  // load, so they display correctly immediately and the row still submits
+  // - editing A_COL afterward (which the PK formulas reference) must NOT
+  // retroactively recompute either PK cell, proving they're frozen rather
+  // than still-live.
+  it('52 | A primary key made of HARDFORMULA/SOFTFORMULA columns freezes to its computed value at load and still submits correctly', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_pk_test')
+
+    // Row 0: A_COL=2, B_COL=3 -> PK_HARDFORMULA_COL=6, PK_SOFTFORMULA_COL="PK-2".
+    getCellByHeaderAndRow(0, 'PK_HARDFORMULA_COL').should('have.text', '6')
+    getCellByHeaderAndRow(0, 'PK_SOFTFORMULA_COL').should('have.text', 'PK-2')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.intercept('POST', '**/SASjsApi/stp/execute*', (req) => {
+          const bodyText = JSON.stringify(req.body || '')
+
+          if (bodyText.includes('stagedata')) {
+            // The row must not be silently dropped from what's submitted
+            // - both frozen PK values and the actually-edited column must
+            // be present.
+            expect(bodyText).to.include('"PK_HARDFORMULA_COL":6')
+            expect(bodyText).to.include('"PK_SOFTFORMULA_COL":"PK-2"')
+            expect(bodyText).to.include('"A_COL":9')
+          }
+
+          req.continue()
+        }).as('stpExecute')
+
+        getCellByHeaderAndRow(0, 'A_COL')
+          .dblclick({ force: true })
+          .then(() => {
+            cy.focused().clear().type('9{enter}')
+          })
+
+        // Frozen, not live - A_COL changing from 2 to 9 would recompute
+        // PK_HARDFORMULA_COL to 27 if it were still a live formula.
+        getCellByHeaderAndRow(0, 'PK_HARDFORMULA_COL').should('have.text', '6')
+
+        submitTable()
+
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+
+        cy.get('#formFields_8').type(
+          'formula-driven composite primary key submission test'
+        )
+
+        submitTableMessage()
+      })
+    })
+  })
+
+  // afterChange delivers Handsontable's own VISUAL row - the EDIT_STATUS/
+  // overwritten-comment sync hook must translate that to physical before
+  // touching dataSource (classifyRow, PK-matching), and the two methods it
+  // calls must translate back to visual for their own Handsontable/
+  // comments-plugin writes. ROW_STATUS_COL (=DC.ROW_STATUS) makes the
+  // EDIT_STATUS write directly observable, so sorting first (breaking
+  // visual/physical alignment) and editing the row that moved proves the
+  // fix end-to-end, not just that an unsorted grid still works.
+  it('53 | Editing a row after sorting updates DC.ROW_STATUS on that same row, not whatever row now visually sits in its old position', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        sortByColumn('PLAIN_TEXT_COL')
+
+        cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
+          const pkColIndex = [...$ths].findIndex(
+            (th) => th.innerText.trim() === 'PRIMARY_KEY_FIELD'
+          )
+          const statusColIndex = [...$ths].findIndex(
+            (th) => th.innerText.trim() === 'ROW_STATUS_COL'
+          )
+
+          cy.get('.ht_master tbody tr').then((rows: any) => {
+            const visualRow = [...rows].findIndex(
+              (row: any) => row.childNodes[pkColIndex].innerText.trim() === '1'
+            )
+
+            // Rows 7-10 (their own pre-existing FORMULA_HARD_COL/
+            // FORMULA_SOFT_COL data, silently overwritten by the DQ rule -
+            // see test 28) already read 'M' from the very first render, so
+            // asserting a specific OTHER row is 'U' would be a coincidence
+            // of seed data, not a proof of anything. Snapshot every other
+            // row's own status instead, and assert none of them changed -
+            // that's what actually distinguishes "the edit landed on the
+            // right row" from "it landed on whatever moved into that
+            // visual slot".
+            const beforeStatuses = [...rows].map((row: any, i: number) =>
+              i === visualRow
+                ? null
+                : row.childNodes[statusColIndex].innerText.trim()
+            )
+
+            getCellByHeaderAndRow(visualRow, 'ROW_STATUS_COL').should(
+              'have.text',
+              'U'
+            )
+
+            getCellByHeaderAndRow(visualRow, 'A_COL')
+              .dblclick({ force: true })
+              .then(() => {
+                cy.focused().clear().type('999{enter}')
+              })
+
+            // The edited row (PK=1) shows 'M' at its own visual position...
+            getCellByHeaderAndRow(visualRow, 'ROW_STATUS_COL').should(
+              'have.text',
+              'M'
+            )
+
+            // ...and every other row's status is exactly what it was
+            // before - none of them got the write meant for row PK=1.
+            cy.get('.ht_master tbody tr').should((rows2: any) => {
+              ;[...rows2].forEach((row: any, i: number) => {
+                if (i === visualRow) return
+
+                const status = row.childNodes[statusColIndex].innerText.trim()
+                expect(status, `row ${i} status unchanged`).to.equal(
+                  beforeStatuses[i]
+                )
+              })
+            })
+          })
+        })
+      })
+    })
+  })
+
+  // multiColumnSorting's own sort() replaces its entire sort state on every
+  // call rather than accumulating (see its own JSDoc) - restoring a
+  // captured multi-column sort with a loop of single-column sort() calls
+  // therefore silently drops every column but the last one restored. This
+  // hits cancelEdit() specifically once a user has shift-clicked a second
+  // header to sort by two columns, then clicks Cancel: the underlying bug
+  // is already covered directly against the Handsontable plugin in
+  // sortedGridRowSync.integration.spec.ts, but only a real shift-click
+  // through the actual grid proves editor.component.ts's own cancelEdit()
+  // restores it correctly end-to-end.
+  it('54 | Cancelling an edit after sorting by two columns keeps both columns sorted, not just the last one clicked', () => {
+    openTableFromTree('testdata', 'mpe_x_formula_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        sortByColumn('A_COL')
+        sortByColumnAdditional('B_COL')
+
+        cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
+          const pkColIndex = [...$ths].findIndex(
+            (th) => th.innerText.trim() === 'PRIMARY_KEY_FIELD'
+          )
+
+          cy.get('.ht_master tbody tr').then((rows: any) => {
+            const beforeCancelPkOrder = [...rows].map((row: any) =>
+              row.childNodes[pkColIndex].innerText.trim()
+            )
+
+            cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+              timeout: longerCommandTimeout
+            }).click()
+
+            // cancelEdit() re-renders read-only with the same dataSource,
+            // still ordered by the restored sort - the row order itself is
+            // the only observable signal that both columns' sort survived.
+            cy.get('.ht_master tbody tr').should((rows2: any) => {
+              const afterCancelPkOrder = [...rows2].map((row: any) =>
+                row.childNodes[pkColIndex].innerText.trim()
+              )
+              expect(afterCancelPkOrder).to.deep.equal(beforeCancelPkOrder)
+            })
+          })
+        })
+      })
+    })
+  })
 })
 
 // Handsontable virtualizes columns — with 18 columns on MPE_X_NEW, only
@@ -1320,6 +1928,19 @@ const sortByColumn = (headerText: string) => {
     .click({ force: true })
 }
 
+// Shift-clicks a column header's sort indicator to ADD it to whatever sort
+// is already active, rather than replacing it - Handsontable's own
+// multiColumnSorting UI convention for building a multi-column sort (a
+// plain click, sortByColumn above, always starts a fresh single-column
+// sort instead).
+const sortByColumnAdditional = (headerText: string) => {
+  cy.get('.ht_clone_top .htCore thead tr th')
+    .filter((_, th) => Cypress.$(th).text().includes(headerText))
+    .last()
+    .find('.columnSorting')
+    .click({ force: true, shiftKey: true })
+}
+
 // Opens a column header's dropdown menu to reach its `info` item, which
 // has a custom renderer showing NAME/LABEL/TYPE/LENGTH/FORMAT and, when the
 // column has a HARDREGEX/SOFTREGEX rule, the applied pattern. Clicking the
@@ -1352,6 +1973,27 @@ const openColumnDropdown = (headerText: string) => {
 // re-renders virtualized columns asynchronously after a scroll event, on
 // its own schedule outside Cypress's command queue, so scrollTo() settling
 // does not mean the target column has been rendered yet.
+// Simulates a real clipboard paste into whichever element currently has
+// DOM focus (Handsontable moves focus to its own internal textarea once a
+// cell is selected/clicked) - Cypress has no built-in clipboard simulation,
+// and Handsontable's CopyPaste plugin only listens for a native 'paste'
+// DOM event with clipboardData, so a synthetic ClipboardEvent is
+// dispatched directly rather than trying to drive the OS clipboard.
+const pasteTextIntoFocusedCell = (text: string) => {
+  cy.focused().then(($el) => {
+    const dataTransfer = new DataTransfer()
+    dataTransfer.setData('text/plain', text)
+
+    const pasteEvent = new ClipboardEvent('paste', {
+      clipboardData: dataTransfer,
+      bubbles: true,
+      cancelable: true
+    })
+
+    $el[0].dispatchEvent(pasteEvent)
+  })
+}
+
 const getCellByHeaderAndRow = (rowIndex: number, headerText: string) => {
   return cy
     .get('.ht_clone_top .htCore thead tr th')

client/cypress/e2e/excel.cy.ts

@@ -546,7 +546,7 @@ const checkResultOfFormulaUpload = (callback?: any) => {
     .find('tbody')
     .then((data) => {
       const cell: any = data[0].children[0].children[5]
-      expect(cell.innerText).to.equal('=1+1')
+      expect(cell.innerText).to.equal('2')
       if (callback) callback()
     })
 }

client/cypress/e2e/filtering.cy.ts

@@ -25,7 +25,7 @@ context('filtering tests: ', function () {
     openFilterPopup(() => {
       setFilterWithValue('SOME_CHAR', 'this is dummy data', 'value', () => {
         checkInfoBarIncludes(
-          `AND,AND,0,SOME_CHAR,=,"'this is dummy data'"`,
+          `(( SOME_CHAR = 'this is dummy data' ))`,
           (includes: boolean) => {
             if (includes) done()
           }
@@ -39,7 +39,7 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_NUM', '42', 'value', () => {
-        checkInfoBarIncludes(`AND,AND,0,SOME_NUM,=,42`, (includes: boolean) => {
+        checkInfoBarIncludes(`(( SOME_NUM = 42 ))`, (includes: boolean) => {
           if (includes) done()
         })
       })
@@ -51,12 +51,9 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_TIME', '00:00:42', 'time', () => {
-        checkInfoBarIncludes(
-          `AND,AND,0,SOME_TIME,=,42`,
-          (includes: boolean) => {
-            if (includes) done()
-          }
-        )
+        checkInfoBarIncludes(`(( SOME_TIME = 42 ))`, (includes: boolean) => {
+          if (includes) done()
+        })
       })
     })
   })
@@ -66,12 +63,9 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_TIME', '42', 'value', () => {
-        checkInfoBarIncludes(
-          `AND,AND,0,SOME_TIME,=,42`,
-          (includes: boolean) => {
-            if (includes) done()
-          }
-        )
+        checkInfoBarIncludes(`(( SOME_TIME = 42 ))`, (includes: boolean) => {
+          if (includes) done()
+        })
       })
     }, false)
   })
@@ -81,12 +75,9 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATE', '12/02/1960', 'date', () => {
-        checkInfoBarIncludes(
-          `AND,AND,0,SOME_DATE,=,42`,
-          (includes: boolean) => {
-            if (includes) done()
-          }
-        )
+        checkInfoBarIncludes(`(( SOME_DATE = 42 ))`, (includes: boolean) => {
+          if (includes) done()
+        })
       })
     })
   })
@@ -96,12 +87,9 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATE', '42', 'value', () => {
-        checkInfoBarIncludes(
-          `AND,AND,0,SOME_DATE,=,42`,
-          (includes: boolean) => {
-            if (includes) done()
-          }
-        )
+        checkInfoBarIncludes(`(( SOME_DATE = 42 ))`, (includes: boolean) => {
+          if (includes) done()
+        })
       })
     }, false)
   })
@@ -116,7 +104,7 @@ context('filtering tests: ', function () {
         'datetime',
         () => {
           checkInfoBarIncludes(
-            `AND,AND,0,SOME_DATETIME,=,42`,
+            `(( SOME_DATETIME = 42 ))`,
             (includes: boolean) => {
               if (includes) done()
             }
@@ -132,7 +120,7 @@ context('filtering tests: ', function () {
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATETIME', '42', 'value', () => {
         checkInfoBarIncludes(
-          `AND,AND,0,SOME_DATETIME,=,42`,
+          `(( SOME_DATETIME = 42 ))`,
           (includes: boolean) => {
             if (includes) done()
           }
@@ -146,12 +134,9 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATE', '', 'in', () => {
-        checkInfoBarIncludes(
-          `AND,AND,0,SOME_DATE,IN,(0)`,
-          (includes: boolean) => {
-            if (includes) done()
-          }
-        )
+        checkInfoBarIncludes(`(( SOME_DATE IN (42) ))`, (includes: boolean) => {
+          if (includes) done()
+        })
       })
     })
   })
@@ -311,7 +296,11 @@ const setFilterWithValue = (
           })
       })
 
-      break
+      // The IN clause is submitted by the modal footer button above; the
+      // trailing tab/click sequence below is for the plain value fields and
+      // must not run here (focus is gone once the modal closes and the
+      // editor reloads with the stored filter).
+      return
     }
     case 'between': {
       cy.focused().tab()

client/cypress/e2e/full-table-search.cy.ts

@@ -0,0 +1,385 @@
+// Marks this file as an ES module (rather than a global script) so its
+// top-level consts don't collide, under the TS type-checker, with the same
+// names declared in other spec files — see e.g. viewer-labels.cy.ts.
+export {}
+
+const username = Cypress.env('username')
+const password = Cypress.env('password')
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+const serverType = Cypress.env('serverType')
+const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
+
+// Fixture: MPE_X_SEARCH, a 1000-row demo table built by the mock data builder
+// (sas/mocks/sasjs/services/admin/makedata.js).  It is deterministic, so every
+// count below can be derived by hand:
+//   - 1000 rows, PK 1001-2000, columns in demo order: PRIMARY_KEY_FIELD,
+//     SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE,
+//     EXPEDITION_ID, NOTES
+//   - 300 rows have a site name containing "Trench" (case sensitive, so
+//     "trench" matches nothing)
+//   - 280 rows contain "Halcyon": the site Halcyon Trench (100 rows) plus the
+//     vessel RV Halcyon (200 rows) less the 20 rows that are both
+//   - 3 rows carry "siphonophore" in the middle of their NOTES text, in the
+//     sites Oceanus Rise, Nereid Trench and Halcyon Trench
+//   - 100 rows observed Vampire squid (species is picked by (site+vessel)%10)
+//   - 100 rows are at Oceanus Rise (site is picked by (n%50)/5)
+//   - 9 rows have an expedition id starting EXP-000 (EXP-0001..EXP-0009)
+//   - the first row (PK 1001) is the only one with EXPEDITION_ID EXP-0001
+//   - 12 rows were sounded at exactly 1000m (DEPTH_M = 1000 + (n%90)*37)
+//   - 2 rows have SAMPLE_COUNT 4210 (the only numeric value that is not a
+//     primary key, a depth or a date)
+//   - PK 2000 is the last row of the table
+//   - 200 rows were surveyed by RV Halcyon, 60 of them at a Trench site
+const demoTable = 'mpe_x_search'
+
+// Full-table search: the viewer's search box searches EVERY column of the
+// table, character columns by case-sensitive CONTAINS and numeric columns by
+// exact equality (see %mp_searchdata in the SASjs macro core, and
+// viewdata.sas / services/public/viewdata.js).
+//
+// The whole walkthrough is one test - the table is opened once and every
+// variation is a search in place, which is how a user works and what makes the
+// recording read as a demo rather than a set of page loads.
+//
+// Some searches legitimately match nothing: they pin the case sensitivity, the
+// exact-match numeric semantics and the "no data found" handling.  Those steps
+// are wrapped in negativeStep() and can be skipped when recording the demo, so
+// the video only ever shows searches that return rows:
+//
+//   npx cypress run --browser electron \
+//     --spec cypress/e2e/full-table-search.cy.ts --env skipNegative=true
+//
+// CI runs the file with no env, so every step runs there.
+const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
+const negativeStep = (step: () => void) => {
+  if (!skipNegative) step()
+}
+
+context('full table search tests: ', function () {
+  this.beforeAll(() => {
+    cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
+    // forceLicenceKey=true: applies a valid licence key instead of relying on
+    // an earlier spec in the same run having already done so.
+    cy.loginAndUpdateValidKey(true)
+  })
+
+  this.beforeEach(() => {
+    cy.visit(hostUrl + appLocation, { timeout: longerCommandTimeout })
+
+    // Visit 'home' first, matching filtering.cy.ts/viewer-labels.cy.ts — going
+    // straight to 'view/data' skips the startup-data fetch 'home' triggers,
+    // which leaves the nav tree empty in slower environments.
+    visitPage('home')
+    visitPage('view/data')
+  })
+
+  it('searches the whole table from the viewer', () => {
+    // Full view: every column, and the true row count for the table.
+    openTableFromTree(libraryToOpenIncludes, demoTable)
+    assertRowCount('1,000')
+    assertGridContains(['Nereid Trench', 'RV Pelagia', 'Vampire squid'])
+
+    // Partial character search: "siphonophore" only ever appears in the middle
+    // of the NOTES text, so a match proves the search is not column-specific.
+    // Three rows is narrow enough that the grid grows a horizontal scrollbar
+    // and clips NOTES - the column the match is actually in - so scroll it
+    // into view, assert on what is now on screen, then put the grid back.
+    searchFor('siphonophore')
+    assertRowCount('3')
+    scrollGridTo('right')
+    pauseForDemo()
+    assertGridContains(['Oceanus Rise', 'Nereid Trench', 'Halcyon Trench'])
+    assertGridExcludes(['Kraken Hollow', 'Vostok Deep'])
+    assertGridContains(['bioluminescent siphonophore was filmed here'])
+    scrollGridTo('left')
+
+    // A search spans every column at once: "Halcyon" is both a site (Halcyon
+    // Trench) and a vessel (RV Halcyon), and the 280 matches are the union.
+    searchFor('Halcyon')
+    assertRowCount('280')
+    assertGridContains(['Halcyon Trench', 'RV Halcyon'])
+
+    // A multi-word site name: the search is not limited to single tokens.
+    searchFor('Oceanus Rise')
+    assertRowCount('100')
+    assertGridContains(['Oceanus Rise'])
+
+    // Multi-word values work the same way - species names are two words.
+    searchFor('Vampire squid')
+    assertRowCount('100')
+    assertGridContains(['Vampire squid'])
+    assertGridExcludes(['Yeti crab'])
+
+    // A partial value in a short code column: EXP-000 matches the first nine
+    // expedition ids.
+    searchFor('EXP-000')
+    assertRowCount('9')
+    assertGridContains(['1001', '1009'])
+
+    // A value that is unique in the table: the first row's expedition id.
+    searchFor('EXP-0001')
+    assertRowCount('1')
+    assertGridContains(['1001'])
+
+    // Character search is case sensitive: the same word in the wrong case
+    // matches nothing.
+    negativeStep(() => {
+      searchFor('trench')
+      assertNoData()
+    })
+
+    searchFor('Trench')
+    assertRowCount('300')
+    assertGridContains(['Halcyon Trench'])
+    assertGridExcludes(['Kraken Hollow'])
+
+    // Numeric search is an exact match across the numeric columns.
+    setNumericSearch(true)
+    searchFor('4210')
+    assertRowCount('2')
+    assertGridContains(['1221', '1781', '4210'])
+
+    // The numeric search covers every numeric column, not just the primary
+    // key: 12 rows were sounded at exactly 1000m.
+    searchFor('1000')
+    assertRowCount('12')
+    assertGridContains(['1091'])
+
+    // ...and an exact primary key is a single row.
+    searchFor('2000')
+    assertRowCount('1')
+    assertGridContains(['2000'])
+
+    negativeStep(() => {
+      searchFor('421')
+      assertNoData()
+    })
+
+    // Back to a character search: clearing the box restores the full table.
+    setNumericSearch(false)
+    searchFor('')
+    assertRowCount('1,000')
+
+    // A search runs inside the current filter, not the whole table.
+    openFilterPopup(() => {
+      setFilterWithValue('VESSEL', 'RV Halcyon', () => {
+        assertRowCount('200')
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+
+        searchFor('Trench')
+        assertRowCount('60')
+        assertGridContains(['Halcyon Trench'])
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+
+        // Clearing the search leaves the filter in place.
+        searchFor('')
+        assertRowCount('200')
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+      })
+    })
+
+    negativeStep(() => {
+      searchFor('not-in-this-table')
+      assertNoData()
+    })
+  })
+})
+
+const visitPage = (url: string) => {
+  cy.visit(`${hostUrl}${appLocation}/#/${url}`)
+}
+
+// Recording-only linger.  A beat's row count is the moment its result is on
+// screen, so that is where a demo recording pauses to let it be read.  CI runs
+// with no env, so demoPauseMs is 0 and the spec contains no waits at all:
+//
+//   npx cypress run --browser electron --spec cypress/e2e/full-table-search.cy.ts \
+//     --env skipNegative=true,demoPause=1500
+const demoPauseMs = Number(Cypress.env('demoPause')) || 0
+
+// Linger for a recording.  A no-op unless demoPause is set, which only the
+// recording run does - CI passes no env.
+const pauseForDemo = () => {
+  if (demoPauseMs > 0) cy.wait(demoPauseMs)
+}
+
+// The row/column count the viewer shows next to the table name, e.g.
+// "(1,000 rows, 9 cols)".  The viewer pluralises: a single-row result reads
+// "(1 row, 9 cols)".
+const assertRowCount = (rows: string) => {
+  cy.get('.title-col p', { timeout: longerCommandTimeout }).should(
+    ($el: any) => {
+      const unit = rows === '1' ? 'row' : 'rows'
+      expect($el.text().replace(/\s+/g, ' ')).to.contain(`(${rows} ${unit},`)
+    }
+  )
+
+  pauseForDemo()
+}
+
+// The grid body is a Handsontable .wtHolder, and a narrow result is wider than
+// the grid viewport, which clips the rightmost column (NOTES).  Setting
+// scrollLeft is enough - Handsontable keeps the header clone in step - and the
+// assertion right after proves the scroll landed rather than assuming it.  The
+// browser clamps scrollLeft to (scrollWidth - clientWidth), so that is what
+// scrolling right has to land on.
+const scrollGridTo = (position: 'left' | 'right') => {
+  cy.get('#hotTable .wtHolder')
+    .first()
+    .then(($holder: any) => {
+      const el = $holder[0]
+      const maxScroll = el.scrollWidth - el.clientWidth
+
+      el.scrollLeft = position === 'right' ? el.scrollWidth : 0
+
+      if (position === 'right') {
+        // This step only means anything if the grid really is clipping a
+        // column - fail loudly rather than silently doing nothing.
+        expect(el.scrollWidth).to.be.greaterThan(el.clientWidth)
+        expect(el.scrollLeft).to.equal(maxScroll)
+      } else {
+        expect(el.scrollLeft).to.equal(0)
+      }
+    })
+}
+
+// A search that matches nothing replaces the grid with the viewer's "no data"
+// panel - and must not raise a request error.
+const assertNoData = () => {
+  cy.get('.noData h3', { timeout: longerCommandTimeout }).should(
+    'contain.text',
+    'No data found with given conditions'
+  )
+}
+
+const assertGridContains = (values: string[]) => {
+  cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
+    ($grid: any) => {
+      const text = $grid.text().replace(/\s+/g, ' ')
+      for (const value of values) expect(text).to.contain(value)
+    }
+  )
+}
+
+const assertGridExcludes = (values: string[]) => {
+  cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
+    ($grid: any) => {
+      const text = $grid.text().replace(/\s+/g, ' ')
+      for (const value of values) expect(text).to.not.contain(value)
+    }
+  )
+}
+
+const searchFor = (value: string) => {
+  cy.get('input[name="search-input"]', { timeout: longerCommandTimeout })
+    .should('be.visible')
+    .clear()
+
+  if (value.length > 0) cy.get('input[name="search-input"]').type(value)
+
+  cy.get('input[name="search-input"]').type('{enter}')
+}
+
+const setNumericSearch = (enabled: boolean) => {
+  cy.get('input[name="numeric_check"]', { timeout: longerCommandTimeout }).then(
+    (checkbox: any) => {
+      if (checkbox[0].checked !== enabled) checkbox[0].click()
+    }
+  )
+}
+
+const checkInfoBarIncludes = (text: string) => {
+  cy.get('.infoBar b', { timeout: longerCommandTimeout }).should(($el: any) => {
+    expect($el.text().toLowerCase()).to.contain(text.toLowerCase())
+  })
+}
+
+// Opens the viewer's options menu and clicks through to the filter modal.
+const openFilterPopup = (callback?: any) => {
+  cy.get('.filterSide', { timeout: longerCommandTimeout }).first().click()
+
+  cy.get('.dropdown-menu', { timeout: longerCommandTimeout })
+    .contains('Filter')
+    .click()
+
+  cy.get('#vals_var_id0_0', { timeout: longerCommandTimeout }).should('exist')
+
+  if (callback) callback()
+}
+
+// Fills the first clause of the filter modal (variable, then value) and
+// submits it - the viewer stores the clause, reloads the table with the new
+// filter RK and shows the generated WHERE text in the info bar.
+const setFilterWithValue = (
+  variableValue: string,
+  valueString: string,
+  callback?: any
+) => {
+  cy.get('#vals_var_id0_0').type(variableValue)
+  cy.get('#datalist_vals_var_id0_0 option')
+    .contains(variableValue)
+    .click({ force: true })
+
+  // The soft-select closes its suggestion list shortly after a selection -
+  // wait for that closed state instead of sleeping.
+  cy.get('#datalist_vals_var_id0_0').should('have.class', 'hidden')
+
+  cy.get('#vals_0_0').type(valueString)
+
+  closeAutocompleteList('#vals_0_0')
+  cy.get('#datalist_vals_0_0').should('have.class', 'hidden')
+
+  cy.get('.filter-modal button[aria-label="Ok"]').click()
+
+  if (callback) callback()
+}
+
+// The soft-select inputs open a suggestion list that drops a transparent
+// click-catcher over the modal until it closes, so send the same Escape key a
+// user would press.  `force` is needed because the input sits underneath that
+// click-catcher while the list is open.
+const closeAutocompleteList = (inputSelector: string) => {
+  cy.get(inputSelector).trigger('keyup', { key: 'Escape', force: true })
+}
+
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let viyaLib
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            viyaLib = node
+            break
+          }
+        }
+
+        cy.get(viyaLib).within(() => {
+          cy.get(
+            '.clr-tree-node-content-container .clr-treenode-content p'
+          ).click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+
+  // Selecting the table triggers async SPA routing + a viewdata fetch; wait
+  // for the grid to actually render before any subsequent action.
+  cy.get('#hotTable .ht_clone_top .htCore thead button.changeType', {
+    timeout: longerCommandTimeout
+  }).should('exist')
+}

client/cypress/e2e/licensing.cy.ts

@@ -249,6 +249,10 @@ context('licensing tests: ', function () {
 
     cy.get('button').contains('Paste licence').click()
 
+    // Defaults to the combined single-field form - switch to the legacy
+    // two-field layout, since this test targets that path specifically.
+    cy.get('button').contains('Paste as two separate keys instead').click()
+
     // HTTP-format keys are generated as the exact same string for both
     // fields (see detectLicenceKeyProtocolMismatch's own doc comment) -
     // any non-empty matching pair triggers the warning, no real key needed.
@@ -273,6 +277,261 @@ context('licensing tests: ', function () {
       })
   })
 
+  it('7 | Combined single-string key (default paste format) activates successfully', (done) => {
+    let keyData = {
+      valid_until: moment().add(1, 'year').format('YYYY-MM-DD'),
+      users_allowed: 4,
+      hot_license_key: '',
+      demo: false,
+      site_id: site_id
+    }
+
+    generateKeys(keyData, (keysGen: any) => {
+      cy.wait(2000)
+
+      // Navigate there explicitly rather than only acting when already on
+      // the licensing page - earlier tests may have left the app on it (a
+      // bad key) or already activated (a good one), and this test should
+      // pass either way rather than silently no-op when it's the latter.
+      isLicensingPage((result: boolean) => {
+        const proceed = () =>
+          generateCombinedKey(keysGen.licenseKey, keysGen.activationKey).then(
+            (combinedKey) => {
+              inputCombinedKeyPage(combinedKey)
+
+              cy.wait(2000)
+
+              acceptTermsIfPresented((termsResult: boolean) => {
+                if (termsResult) {
+                  cy.wait(10000)
+                }
+
+                visitPage('home')
+
+                cy.get('.nav-tree clr-tree > clr-tree-node', {
+                  timeout: longerCommandTimeout
+                }).then(() => {
+                  done()
+                })
+              })
+            }
+          )
+
+        if (!result) {
+          visitPage('licensing/update')
+          cy.wait(2000)
+        }
+
+        // proceed() generates the combined key asynchronously, then
+        // enqueues the actual page commands only once its promise
+        // resolves. Bridged through cy.then() so the command chain stays
+        // open across that async gap: a bare proceed() call returns a
+        // promise that nothing awaits, and once the surrounding queue
+        // has drained, commands enqueued from that promise's continuation
+        // are never run.
+        cy.then(() => proceed())
+      })
+    })
+  })
+
+  it('8 | Pasting a combined key into the legacy licence-key field alone still auto-detects and activates', (done) => {
+    let keyData = {
+      valid_until: moment().add(1, 'year').format('YYYY-MM-DD'),
+      users_allowed: 4,
+      hot_license_key: '',
+      demo: false,
+      site_id: site_id
+    }
+
+    generateKeys(keyData, (keysGen: any) => {
+      cy.wait(2000)
+
+      // Navigate there explicitly rather than only acting when already on
+      // the licensing page - see test 7's own comment for why.
+      isLicensingPage((result: boolean) => {
+        const proceed = () =>
+          generateCombinedKey(keysGen.licenseKey, keysGen.activationKey).then(
+            (combinedKey) => {
+              cy.get('button').contains('Paste licence').click()
+              cy.get('button')
+                .contains('Paste as two separate keys instead')
+                .click()
+
+              // Only the licence-key field gets the combined string - the
+              // activation-key field is left empty, proving the split
+              // populates both fields from this one paste.
+              cy.get('.license-key-form textarea', {
+                timeout: longerCommandTimeout
+              })
+                .invoke('val', combinedKey)
+                .trigger('input')
+                .trigger('mouseleave')
+
+              cy.get('.activation-key-form textarea', {
+                timeout: longerCommandTimeout
+              }).should(($textarea) => {
+                expect($textarea.val()).to.equal(keysGen.activationKey)
+              })
+
+              cy.get('button.apply-keys').click()
+
+              cy.wait(2000)
+
+              acceptTermsIfPresented((termsResult: boolean) => {
+                if (termsResult) {
+                  cy.wait(10000)
+                }
+
+                visitPage('home')
+
+                cy.get('.nav-tree clr-tree > clr-tree-node', {
+                  timeout: longerCommandTimeout
+                }).then(() => {
+                  done()
+                })
+              })
+            }
+          )
+
+        // See test 7's own comment for why proceed() is bridged through
+        // cy.then() rather than called as a plain statement.
+        if (!result) {
+          visitPage('licensing/update')
+          cy.wait(2000)
+        }
+        cy.then(() => proceed())
+      })
+    })
+  })
+
+  it('9 | Uploading a single-line combined-format file activates successfully', (done) => {
+    let keyData = {
+      valid_until: moment().add(1, 'year').format('YYYY-MM-DD'),
+      users_allowed: 4,
+      hot_license_key: '',
+      demo: false,
+      site_id: site_id
+    }
+
+    generateKeys(keyData, (keysGen: any) => {
+      cy.wait(2000)
+
+      // Navigate there explicitly rather than only acting when already on
+      // the licensing page - see test 7's own comment for why.
+      isLicensingPage((result: boolean) => {
+        const proceed = () =>
+          generateCombinedKey(keysGen.licenseKey, keysGen.activationKey).then(
+            (combinedKey) => {
+              cy.get('input[type="file"]').attachFile({
+                fileContent: new Blob([combinedKey], { type: 'text/plain' }),
+                fileName: 'datacontroller-licence-combined.txt',
+                mimeType: 'text/plain'
+              })
+
+              cy.get('button.apply-keys', {
+                timeout: longerCommandTimeout
+              }).click()
+
+              cy.wait(2000)
+
+              acceptTermsIfPresented((termsResult: boolean) => {
+                if (termsResult) {
+                  cy.wait(10000)
+                }
+
+                visitPage('home')
+
+                cy.get('.nav-tree clr-tree > clr-tree-node', {
+                  timeout: longerCommandTimeout
+                }).then(() => {
+                  done()
+                })
+              })
+            }
+          )
+
+        // See test 7's own comment for why proceed() is bridged through
+        // cy.then() rather than called as a plain statement.
+        if (!result) {
+          visitPage('licensing/update')
+          cy.wait(2000)
+        }
+        cy.then(() => proceed())
+      })
+    })
+  })
+
+  it('10 | Key details preview shows for a validly-pasted key and disappears once the key input is cleared', (done) => {
+    let keyData = {
+      valid_until: moment().add(1, 'year').format('YYYY-MM-DD'),
+      users_allowed: 4,
+      hot_license_key: '',
+      demo: false,
+      site_id: site_id,
+      // Kept to just the two enabled toggles (relying on
+      // decodeLicenceFeatures' fallback for the rest) to keep the
+      // encrypted payload well under this suite's RSA-OAEP plaintext
+      // ceiling - see generateKeys' modulusLength below.
+      features: { vb: true, fu: true }
+    }
+
+    generateKeys(keyData, (keysGen: any) => {
+      cy.wait(2000)
+
+      // Navigate there explicitly rather than only acting when already on
+      // the licensing page - see test 7's own comment for why.
+      isLicensingPage((result: boolean) => {
+        const proceed = () =>
+          generateCombinedKey(keysGen.licenseKey, keysGen.activationKey).then(
+            (combinedKey) => {
+              cy.get('button').contains('Paste licence').click()
+
+              cy.get('.combined-key-form textarea', {
+                timeout: longerCommandTimeout
+              })
+                .invoke('val', combinedKey)
+                .trigger('input')
+                .trigger('mouseleave')
+
+              cy.get('.key-details')
+                .should('contain', 'Valid until:')
+                .and('contain', 'Allowed users:')
+                .and('contain', '4')
+                .and('contain', 'Site ID(s) in this key:')
+                .and('contain', 'Enabled features:')
+                .and('contain', 'Viewbox')
+                .and('contain', 'File Upload')
+                .invoke('text')
+                .then((text) => {
+                  expect(text).to.not.contain('Edit Record')
+                  expect(text).to.not.contain('Add Record')
+                })
+
+              cy.get('.combined-key-form textarea')
+                .invoke('val', '')
+                .trigger('input')
+                .trigger('mouseleave')
+
+              cy.get('.key-details')
+                .should('not.exist')
+                .then(() => {
+                  done()
+                })
+            }
+          )
+
+        // See test 7's own comment for why this is chained via .then()
+        // rather than called as the next plain statement.
+        if (!result) {
+          visitPage('licensing/update')
+          cy.wait(2000).then(() => proceed())
+        } else {
+          proceed()
+        }
+      })
+    })
+  })
+
   if (testLicenceUserLimits) {
     it('4 | User try to register when limit is reached', (done) => {
       let keyData = {
@@ -446,7 +705,7 @@ const verifyLicensingPage = (text: string, callback: any) => {
   cy.wait(1000)
   isLicensingPage((result: boolean) => {
     if (result) {
-      cy.get('p.key-error')
+      cy.get('.key-error')
         .should('contain', text)
         .then((treeNodes: any) => {
           callback(true)
@@ -470,6 +729,10 @@ const verifyLicensingWarning = (text: string, callback: any) => {
 const inputLicenseKeyPage = (licenseKey: string, activationKey: string) => {
   cy.get('button').contains('Paste licence').click()
 
+  // Defaults to the combined single-field form - this helper exercises the
+  // two-part path specifically, so switch to it first.
+  cy.get('button').contains('Paste as two separate keys instead').click()
+
   cy.get('.license-key-form textarea', { timeout: longerCommandTimeout })
     .invoke('val', licenseKey)
     .trigger('input')
@@ -481,6 +744,30 @@ const inputLicenseKeyPage = (licenseKey: string, activationKey: string) => {
   cy.get('button.apply-keys').click()
 }
 
+const inputCombinedKeyPage = (combinedKey: string) => {
+  cy.get('button').contains('Paste licence').click()
+
+  // Combined is the default paste format - no toggle click needed here,
+  // unlike inputLicenseKeyPage's legacy two-field path.
+  cy.get('.combined-key-form textarea', { timeout: longerCommandTimeout })
+    .invoke('val', combinedKey)
+    .trigger('input')
+    .trigger('mouseleave')
+    .should('not.be.undefined')
+
+  // mouseleave's handler (onCombinedKeyInput) does async work - gzip
+  // decompress the split, then decrypt for the key-details preview -
+  // before licenceKeyValue/activationKeyValue are ready to submit.
+  // Wait for that preview to land rather than racing "Apply licence
+  // keys" against the still-in-flight promise.
+  cy.get('.key-details', { timeout: longerCommandTimeout }).should(
+    'contain',
+    'Valid until:'
+  )
+
+  cy.get('button.apply-keys').click()
+}
+
 const updateUsersTable = (options: any, callback?: any) => {
   visitPage('home')
   openTableFromTree(libraryToOpenIncludes, 'mpe_users')
@@ -625,6 +912,62 @@ const generateKeys = async (licenseData: any, resultCallback?: any) => {
     })
 }
 
+// Mirrors dckey's own encodeCombinedKey()/gzipCompress() (main.js) and
+// DC's own splitCombinedLicenceKey() decode side, so this stays a genuine
+// end-to-end check of the real format rather than a fixture the app
+// happens to accept.
+const COMBINED_KEY_PREFIX = 'DCKEY1:'
+
+// Reads a ReadableStream directly via its own reader, rather than through
+// `new Response(readable).arrayBuffer()` - Cypress patches fetch/Response
+// globally for its network-interception features, which appears to hang
+// a Response built locally around a stream that never touches the network
+// (writer.write()/close() resolve fine; only the Response-based read never
+// settles).
+const streamToArrayBuffer = async (
+  readable: ReadableStream<Uint8Array>
+): Promise<ArrayBuffer> => {
+  const reader = readable.getReader()
+  const chunks: Uint8Array[] = []
+  let totalLength = 0
+
+  while (true) {
+    const { done, value } = await reader.read()
+    if (done) break
+    chunks.push(value)
+    totalLength += value.length
+  }
+
+  const result = new Uint8Array(totalLength)
+  let offset = 0
+  for (const chunk of chunks) {
+    result.set(chunk, offset)
+    offset += chunk.length
+  }
+
+  return result.buffer
+}
+
+const generateCombinedKey = async (
+  licenseKey: string,
+  activationKey: string
+): Promise<string> => {
+  const payloadBytes = new TextEncoder().encode(
+    `${licenseKey} ${activationKey}`
+  )
+
+  const compressionStream = new CompressionStream('gzip')
+  const writer = compressionStream.writable.getWriter()
+  const [, compressedBuffer] = await Promise.all([
+    writer.write(payloadBytes).then(() => writer.close()),
+    streamToArrayBuffer(compressionStream.readable)
+  ])
+
+  const compressedBase64 = await arrayBufferToBase64(compressedBuffer)
+
+  return COMBINED_KEY_PREFIX + compressedBase64
+}
+
 const editTableField = (edits: EditConfigTableCells[], callback?: any) => {
   cy.get('td').then((tdNodes: any) => {
     for (let edit of edits) {

client/cypress/e2e/stage.cy.ts

@@ -1,37 +1,81 @@
+const username = Cypress.env('username')
+const password = Cypress.env('password')
 const hostUrl = Cypress.env('hosturl')
 const appLocation = Cypress.env('appLocation')
 const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+const serverType = Cypress.env('serverType')
+const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
+const fixturePath = 'excels/'
 
 context('stage tests: ', function () {
   this.beforeAll(() => {
     cy.visit(`${hostUrl}/SASLogon/logout`)
-    cy.loginAndUpdateValidKey()
+    cy.loginAndUpdateValidKey(true)
   })
 
   this.beforeEach(() => {
     cy.visit(hostUrl + appLocation)
 
-    visitPage('stage/DC20221007T122326121_612316_7259')
+    visitPage('home')
   })
 
-  // getstagetable's mock ignores the table_id param entirely, so any id in
-  // the URL resolves to the same fixture row - no need to submit a real
-  // table first just to reach this page.
   it('1 | Formatted/Unformatted toggle switches between fmt_stagetable and stagetable, defaulting to formatted', () => {
-    cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
-      'not.exist'
-    )
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
 
-    getCellByHeaderAndRow(0, 'SOME_DATE').should('have.text', '12FEB1960')
+    attachExcelFile('regular_excel.xlsx', () => {
+      submitExcel()
 
-    cy.get('.formatted-values-toggle').click()
+      // The app navigates to /stage/{tableId}.
+      cy.url({ timeout: longerCommandTimeout }).should('include', '/stage/')
 
-    getCellByHeaderAndRow(0, 'SOME_DATE').should('have.text', '42')
+      cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
+        'not.exist'
+      )
 
-    // Toggling back reverts to the formatted view.
-    cy.get('.formatted-values-toggle').click()
+      // Toggle to unformatted first to read the raw SOME_DATE value.
+      cy.get('.formatted-values-toggle').click()
 
-    getCellByHeaderAndRow(0, 'SOME_DATE').should('have.text', '12FEB1960')
+      getCellByHeaderAndRow(0, 'SOME_DATE')
+        .invoke('text')
+        .then((rawText) => {
+          const rawDate = parseInt(rawText, 10)
+
+          // Toggle back to formatted view.
+          cy.get('.formatted-values-toggle').click()
+
+          // Compute the expected SAS date9. format from the raw value.
+          const sasEpoch = new Date(Date.UTC(1960, 0, 1))
+          const formatted = new Date(sasEpoch.getTime() + rawDate * 86400000)
+          const months = [
+            'JAN',
+            'FEB',
+            'MAR',
+            'APR',
+            'MAY',
+            'JUN',
+            'JUL',
+            'AUG',
+            'SEP',
+            'OCT',
+            'NOV',
+            'DEC'
+          ]
+          const expected =
+            String(formatted.getUTCDate()).padStart(2, '0') +
+            months[formatted.getUTCMonth()] +
+            formatted.getUTCFullYear()
+
+          getCellByHeaderAndRow(0, 'SOME_DATE').should('have.text', expected)
+
+          // Toggle to unformatted again to verify round-trip.
+          cy.get('.formatted-values-toggle').click()
+
+          getCellByHeaderAndRow(0, 'SOME_DATE').should(
+            'have.text',
+            String(rawDate)
+          )
+        })
+    })
   })
 })
 
@@ -39,6 +83,68 @@ const visitPage = (url: string) => {
   cy.visit(`${hostUrl}${appLocation}/#/${url}`)
 }
 
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let viyaLib
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            viyaLib = node
+            break
+          }
+        }
+
+        cy.get(viyaLib).within(() => {
+          cy.get('.clr-tree-node-content-container > button').click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+}
+
+const attachExcelFile = (excelFilename: string, callback?: any) => {
+  cy.get('.buttonBar button:last-child')
+    .should('exist')
+    .click()
+    .then(() => {
+      cy.get('input[type="file"]#file-upload')
+        .attachFile(`/${fixturePath}/${excelFilename}`)
+        .then(() => {
+          cy.get('.clr-abort-modal .modal-title').then((modalTitle) => {
+            if (!modalTitle[0].innerHTML.includes('Abort Message')) {
+              cy.get('.modal-footer .btn.btn-primary').then((modalBtn) => {
+                modalBtn.click()
+                if (callback) callback()
+              })
+            } else {
+              if (callback) callback()
+            }
+          })
+        })
+    })
+}
+
+const submitExcel = (callback?: any) => {
+  cy.get('.buttonBar button.preview-submit', { timeout: longerCommandTimeout })
+    .click()
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
 // Locates a body cell by its column's header text rather than a hardcoded
 // childNodes index - same helper as editor.cy.ts's own.
 const getCellByHeaderAndRow = (rowIndex: number, headerText: string) => {

client/cypress/e2e/submitted-details.cy.ts

@@ -0,0 +1,73 @@
+// Marks this file as an ES module (rather than a global script) so its
+// top-level consts don't collide, under the TS type-checker, with the same
+// names declared in other spec files — see e.g. stage.cy.ts, which gets
+// this for free via a real import.
+export {}
+
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+
+context('submitted-details tests: ', function () {
+  this.beforeAll(() => {
+    cy.loginAndUpdateValidKey(true)
+  })
+
+  it('1 | clicking an open submit loads its diffs with a single postdata request', () => {
+    // Count every STP request the app makes during the test.
+    // The STP URL is `stp/execute/?_program=...` - glob segments don't match
+    // past `execute/` with a single `*`, so keep the glob within the path.
+    cy.intercept('POST', '**/SASjsApi/stp/**').as('stpExecute')
+
+    cy.visit(`${hostUrl}${appLocation}/#/review/submitted`)
+    cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
+      'not.exist'
+    )
+
+    // the SUBMIT queue list must be rendered with at least one row
+    cy.get('app-submitter clr-datagrid clr-dg-row', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .and('have.length.greaterThan', 0)
+
+    // route-based flow: clicking a row navigates to /review/submitted/:tableId
+    // and triggers exactly one SHOW_DIFFS (auditors/postdata) request
+    cy.get('app-submitter clr-datagrid clr-dg-row')
+      .first()
+      .click()
+      .then(() => {
+        cy.url().should('include', '/review/submitted/')
+      })
+
+    // wait for the diff table to render (the response was applied)
+    cy.get('app-approve-details .card', { timeout: longerCommandTimeout })
+      .should('exist')
+      .should('be.visible')
+
+    // give any (incorrect) duplicate request time to surface
+    cy.wait(3000)
+
+    cy.get('@stpExecute.all').then((allRequests: any) => {
+      // the adapter posts multipart form data, so match on the _program
+      // query param rather than the request body
+      const postdataRequests = allRequests.filter((req: any) =>
+        JSON.stringify(req.request.query || {}).includes('auditors/postdata')
+      )
+      expect(
+        postdataRequests.length,
+        'exactly one auditors/postdata SHOW_DIFFS request'
+      ).to.equal(1)
+
+      // the submit queue was already fetched to render the list - opening a
+      // row must not fetch it again
+      const getsubmitsRequests = allRequests.filter((req: any) =>
+        JSON.stringify(req.request.query || {}).includes('editors/getsubmits')
+      )
+      expect(
+        getsubmitsRequests.length,
+        'no editors/getsubmits request after opening a submit'
+      ).to.equal(1)
+    })
+  })
+})

client/cypress/support/commands.ts

@@ -183,6 +183,12 @@ const isLicensingPage = (callback: any) => {
 
 const inputLicenseKeyPage = (licenseKey: string, activationKey: string) => {
   cy.get('button').contains('Paste licence').click()
+
+  // Combined single-string paste is the default format now - switch to
+  // the legacy two-field layout, since this helper fills licenseKey and
+  // activationKey as two separate values.
+  cy.get('button').contains('Paste as two separate keys instead').click()
+
   cy.get('.license-key-form textarea', { timeout: longerCommandTimeout })
     .invoke('val', licenseKey)
     .trigger('input')

client/cypress/util/helper-functions.ts

@@ -1,32 +1,30 @@
+import * as base64Converter from 'base64-arraybuffer'
+
 export const base64ToArrayBuffer = (base64: string) => {
   return new Promise(async (resolve, reject) => {
-    const dataUrl = "data:application/octet-binary;base64," + base64;
-  
+    const dataUrl = 'data:application/octet-binary;base64,' + base64
+
     fetch(dataUrl)
-      .then(res => res.arrayBuffer())
-      .then(buffer => {
+      .then((res) => res.arrayBuffer())
+      .then((buffer) => {
         resolve(new Uint8Array(buffer))
-      }).catch((err) => {
+      })
+      .catch((err) => {
         reject(err)
       })
   })
 }
 
-export const arrayBufferToBase64 = (arrayBuffer: any) => {
-  return new Promise((resolve, reject) => {
-    const blob = new Blob([arrayBuffer])
-
-    const reader = new FileReader();
-
-    reader.onload = async function(event){
-      if (event.target) {
-        var base64: any = event.target.result
-        base64 = base64.substring(37, base64.length)
-
-        resolve(base64)
-      }
-    };
-
-    reader.readAsDataURL(blob);
-  })
-}
+// Blob + FileReader.readAsDataURL (a natural alternative here) is
+// unreliable in the Cypress runner's iframe - onload/onerror can simply
+// never fire, leaving the promise permanently unsettled (a silent hang,
+// surfacing only as Mocha's "done() was never invoked" 30s later with no
+// clue why). base64-arraybuffer's encode() is synchronous and
+// dependency-free - no browser API involved, so this class of hang isn't
+// possible - and it's the same library the app itself uses for this exact
+// conversion.
+export const arrayBufferToBase64 = (
+  arrayBuffer: ArrayBuffer
+): Promise<string> => {
+  return Promise.resolve(base64Converter.encode(arrayBuffer))
+}

client/package.json

@@ -54,9 +54,9 @@
     "@clr/angular": "file:libraries/clr-angular-17.9.0.tgz",
     "@clr/icons": "^13.0.2",
     "@clr/ui": "file:libraries/clr-ui-17.9.0.tgz",
-    "@handsontable/angular-wrapper": "^18.0.0",
-    "@sasjs/adapter": "^4.17.3",
-    "@sasjs/utils": "^3.5.9",
+    "@handsontable/angular-wrapper": "18.0.0",
+    "@sasjs/adapter": "^4.18.0",
+    "@sasjs/utils": "^3.6.0",
     "@sheet/crypto": "file:libraries/sheet-crypto.tgz",
     "@types/d3-graphviz": "^2.6.7",
     "@types/text-encoding": "0.0.35",
@@ -69,7 +69,7 @@
     "d3-graphviz": "^5.0.2",
     "exceljs": "^4.4.0",
     "fs-extra": "^7.0.1",
-    "handsontable": "^18.0.0",
+    "handsontable": "18.0.0",
     "https-browserify": "1.0.0",
     "hyperformula": "^2.5.0",
     "iconv-lite": "^0.5.0",
@@ -148,6 +148,7 @@
     "ajv": "8.18.0",
     "uuid": "11.1.1",
     "lighthouse": "13.4.0",
+    "fast-uri": "3.1.7",
     "readdir-glob": {
       "brace-expansion": "^5.0.9"
     },

client/src/app/app.component.html

@@ -130,25 +130,18 @@
         </div>
       }
       <div class="logo d-flex clr-align-items-center">
-        @if (!router.url.includes('deploy')) {
-          <a href="#" [routerLink]="['/']" class="nav-link">
-            <img
-              class="without-text d-block d-md-none"
-              src="images/dc-logo.svg"
-              alt="datacontroller logo without text"
-            />
-            <img
-              class="with-text d-none d-md-block"
-              src="images/datacontroller.svg"
-              alt="datacontroller logo"
-            />
-          </a>
-        }
-        @if (router.url.includes('deploy')) {
-          <a>
-            <span class="clr-icon header-logo ml-10"></span>
-          </a>
-        }
+        <a href="#" [routerLink]="['/']" class="nav-link">
+          <img
+            class="without-text d-block d-md-none"
+            src="images/dc-logo.svg"
+            alt="datacontroller logo without text"
+          />
+          <img
+            class="with-text d-none d-md-block"
+            src="images/datacontroller.svg"
+            alt="datacontroller logo"
+          />
+        </a>
       </div>
       @if (
         !router.url.includes('deploy') && !router.url.includes('licensing')
@@ -318,6 +311,45 @@
         <div class="subline dec"></div>
       </div>
     }
+    @if (startupSteps.length > 0) {
+      <div class="startup-checks">
+        @for (step of startupSteps; track step.label) {
+          <div
+            class="startup-check"
+            [class.startup-check--error]="step.status === 'error'"
+            [class.startup-check--done]="step.status === 'done'"
+          >
+            @switch (step.status) {
+              @case ('pending') {
+                <clr-icon
+                  shape="circle"
+                  class="is-solid startup-check__icon--pending"
+                ></clr-icon>
+              }
+              @case ('in_progress') {
+                <clr-spinner clrSmall></clr-spinner>
+              }
+              @case ('done') {
+                <clr-icon
+                  shape="check-circle"
+                  class="is-solid startup-check__icon--done"
+                ></clr-icon>
+              }
+              @case ('error') {
+                <clr-icon
+                  shape="exclamation-circle"
+                  class="is-solid startup-check__icon--error"
+                ></clr-icon>
+              }
+            }
+            <span class="startup-check__label">{{ step.label }}</span>
+            @if (step.detail) {
+              <span class="startup-check__detail">{{ step.detail }}</span>
+            }
+          </div>
+        }
+      </div>
+    }
   </div>
 }
 <!-- /App Loading Page -->

client/src/app/app.component.ts

@@ -19,6 +19,10 @@ import { InfoModal } from './models/InfoModal'
 import { DcAdapterSettings } from './models/DcAdapterSettings'
 import { AppStoreService } from './services/app-store.service'
 import { LicenceService } from './services/licence.service'
+import {
+  StartupCheckService,
+  StartupStep
+} from './services/startup-check.service'
 import '@cds/core/icon/register.js'
 import {
   ClarityIcons,
@@ -64,6 +68,7 @@ export class AppComponent {
   public requestsModal: boolean = false
   public showRegistration: boolean = true
   public startupDataLoaded: boolean = false
+  public startupSteps: StartupStep[] = []
   public demoLimitNotice: { open: boolean; featureName: string } = {
     open: false,
     featureName: ''
@@ -81,11 +86,16 @@ export class AppComponent {
     private location: Location,
     private eventService: EventService,
     private appStoreService: AppStoreService,
+    private startupCheckService: StartupCheckService,
     private cdr: ChangeDetectorRef,
     private elementRef: ElementRef
   ) {
     this.parseDcAdapterSettings()
 
+    this.startupCheckService.steps$.subscribe((steps) => {
+      this.startupSteps = steps
+    })
+
     /**
      * Prints app info in the console such as:
      * - Adapter versions

client/src/app/deploy/deploy.component.ts

@@ -21,8 +21,6 @@ export class DeployComponent implements OnInit {
   public step: number = 0
   public adminGroups: any = []
 
-  public client_id: string = ''
-  public client_secret: string = ''
   public appLoc: string = ''
   public dcPath: string = ''
   public selectedAdminGroup: string = ''
@@ -52,9 +50,6 @@ export class DeployComponent implements OnInit {
     this.sasJs = this.sasService.getSasjsInstance()
     this.sasJsConfig = this.sasService.getSasjsConfig()
     this.appLoc = this.dcAdapterSettings?.appLoc || ''
-    this.client_id = localStorage.getItem('deploy_client_id') || ''
-    this.client_secret = localStorage.getItem('deploy_secret_key') || ''
-    this.dcPath = localStorage.getItem('deploy_dc_loc') || ''
   }
 
   ngOnInit() {

client/src/app/deploy/sections/automatic/automatic.component.html

@@ -77,69 +77,173 @@
 }
 
 <h4 class="text-center my-15">Viya Deploy</h4>
+<p class="deploy-intro">
+  Configure and deploy Data Controller to your Viya environment. Deployment
+  creates the backend SAS services and the control library used by the
+  application.
+</p>
 <hr />
-<label for="dcloc" class="mt-20 clr-control-label">App Loc</label>
-<div class="mb-10 clr-control-container">
-  <div class="clr-input-wrapper">
-    <p class="mt-0">{{ appLoc }}</p>
+
+<div class="deploy-field">
+  <label class="clr-control-label">App Loc</label>
+  <p class="deploy-field-description">
+    The SAS drive folder in which the Data Controller frontend and services are
+    deployed. Derived from your deployment configuration &mdash; it cannot be
+    changed here.
+  </p>
+  <div class="clr-control-container">
+    <div class="clr-input-wrapper">
+      <p class="m-0">{{ appLoc }}</p>
+    </div>
   </div>
 </div>
 
-<label for="dcloc" class="mt-20 clr-control-label">DC Loc</label>
-<div class="mb-10 clr-control-container dc-loc-input-wrapper">
-  <div class="clr-input-wrapper small-mt">
-    <input clrInput name="dcloc" [(ngModel)]="dcPath" />
+<div class="deploy-field">
+  <label for="dcloc" class="clr-control-label">DC Loc</label>
+  <p class="deploy-field-description">
+    A physical directory on the compute server to which the deployment user can
+    write. Data Controller stores its control tables, staging area and audit
+    history here.
+  </p>
+  <div class="clr-control-container dc-loc-input-wrapper">
+    <div class="clr-input-wrapper small-mt">
+      <input clrInput name="dcloc" [(ngModel)]="dcPath" />
+    </div>
   </div>
 </div>
 
-<label for="dcloc" class="mt-20 clr-control-label">SAS Admin group</label>
-<div class="mb-10 clr-control-container">
-  <div class="clr-input-wrapper small-mt">
-    @if (!adminGroupsLoading) {
-      <select clrSelect name="options" [(ngModel)]="selectedAdminGroup">
-        @for (adminGroup of adminGroups; track adminGroup) {
-          <option [value]="adminGroup.id">
-            {{ adminGroup.name }}
-          </option>
+<div class="deploy-field">
+  <label for="adminGroup" class="clr-control-label">SAS Admin group</label>
+  <p class="deploy-field-description">
+    The group you select becomes the Data Controller Admin Group &mdash;
+    everyone in it will have unrestricted access to Data Controller. By default
+    the list is filtered to the groups your identity
+    @if (currentUserInfo?.id) {
+      (<strong>{{ currentUserInfo?.id }}</strong
+      >)
+    }
+    is a member of. Toggle below to list every group on the Viya server.
+  </p>
+  <clr-toggle-container class="mt-0">
+    <clr-toggle-wrapper>
+      <input
+        id="show-all-groups-toggle"
+        type="checkbox"
+        [(ngModel)]="showAllGroups"
+        (ngModelChange)="onShowAllGroupsChange()"
+        clrToggle
+      />
+      <label for="show-all-groups-toggle">Show all groups</label>
+    </clr-toggle-wrapper>
+  </clr-toggle-container>
+  <div class="clr-control-container">
+    <div class="clr-input-wrapper small-mt">
+      @if (!adminGroupsLoading) {
+        <select clrSelect name="adminGroup" [(ngModel)]="selectedAdminGroup">
+          @for (adminGroup of adminGroups; track adminGroup) {
+            <option [value]="adminGroup.id">
+              {{ adminGroup.name }}
+            </option>
+          }
+        </select>
+      }
+      @if (adminGroupsLoading) {
+        <clr-spinner clrInline class="spinner-sm"></clr-spinner>
+      }
+    </div>
+  </div>
+</div>
+
+@if (!showAllContexts) {
+  <div class="deploy-field">
+    <label for="batchId" class="clr-control-label">Batch ID</label>
+    <p class="deploy-field-description">
+      The batch identity (<code>runServerAs</code>) under which Data Controller
+      jobs will execute. It is used to filter the Compute Context dropdown below
+      to only the contexts running as this identity.
+    </p>
+    <div class="clr-control-container">
+      <div class="clr-input-wrapper small-mt">
+        @if (!computeContextsLoading) {
+          <select
+            clrSelect
+            name="batchId"
+            [(ngModel)]="selectedBatchId"
+            (ngModelChange)="onBatchIdChange()"
+          >
+            @for (batchId of batchIds; track batchId) {
+              <option [value]="batchId">
+                {{ batchId }}
+              </option>
+            }
+          </select>
         }
-      </select>
-    }
-    @if (adminGroupsLoading) {
-      <clr-spinner clrInline class="spinner-sm"></clr-spinner>
-    }
-  </div>
-</div>
-
-<label for="computeContext" class="mt-20 clr-control-label"
-  >Compute Context</label
->
-<div class="mb-10 clr-control-container">
-  <div class="clr-input-wrapper small-mt">
-    @if (!computeContextsLoading) {
-      <select
-        clrSelect
-        name="options"
-        (ngModelChange)="onComputeContextChange($event)"
-        [(ngModel)]="selectedComputeContext"
-      >
-        @for (computeContext of computeContexts; track computeContext) {
-          <option [value]="computeContext.id">
-            {{ computeContext.name }}
-          </option>
+        @if (computeContextsLoading) {
+          <clr-spinner clrInline class="spinner-sm"></clr-spinner>
         }
-      </select>
-    }
-    @if (computeContextsLoading) {
-      <clr-spinner clrInline class="spinner-sm"></clr-spinner>
-    }
+      </div>
+    </div>
+  </div>
+}
+
+<div class="deploy-field">
+  <label for="computeContext" class="clr-control-label">Compute Context</label>
+  <p class="deploy-field-description">
+    The compute context used to run the Data Controller services (including the
+    <code>makedata</code> deployment job). Only reusable contexts matching the
+    selected Batch ID are shown, unless the toggle below is enabled.
+  </p>
+  <clr-toggle-container class="mt-0">
+    <clr-toggle-wrapper>
+      <input
+        id="show-all-contexts-toggle"
+        type="checkbox"
+        [(ngModel)]="showAllContexts"
+        (ngModelChange)="onShowAllContextsChange()"
+        clrToggle
+      />
+      <label for="show-all-contexts-toggle">Show all compute contexts</label>
+    </clr-toggle-wrapper>
+  </clr-toggle-container>
+  <div class="clr-control-container">
+    <div class="clr-input-wrapper small-mt">
+      @if (!computeContextsLoading) {
+        <select
+          clrSelect
+          name="computeContext"
+          (ngModelChange)="onComputeContextChange($event)"
+          [(ngModel)]="selectedComputeContext"
+        >
+          @for (
+            computeContext of getFilteredComputeContexts();
+            track computeContext
+          ) {
+            <option [value]="computeContext.id">
+              {{ computeContext.name }}
+            </option>
+          }
+        </select>
+      }
+      @if (computeContextsLoading) {
+        <clr-spinner clrInline class="spinner-sm"></clr-spinner>
+      }
+    </div>
   </div>
 </div>
 
 @if (runningAsUser) {
-  <label for="dcloc" class="mt-20 clr-control-label">Running as user:</label>
-  <div class="mb-10 clr-control-container">
-    <div class="clr-input-wrapper">
-      <p class="mt-0">{{ runningAsUser }}</p>
+  <div class="deploy-field">
+    <label class="clr-control-label">Running as user</label>
+    <p class="deploy-field-description">
+      The identity under which Data Controller jobs will run with the selected
+      compute context. This user must be able to write to the DC Loc above.
+    </p>
+    <div class="clr-control-container">
+      <div class="clr-input-wrapper">
+        <p class="m-0">
+          <strong>{{ runningAsUser }}</strong>
+        </p>
+      </div>
     </div>
   </div>
 }

client/src/app/deploy/sections/automatic/automatic.component.ts

@@ -25,6 +25,13 @@ import {
   Item as ComputeContextItem
 } from 'src/app/viya-api-explorer/models/viya-compute-contexts.model'
 
+export interface ComputeContextInfo {
+  id: string
+  name: string
+  runAs: string | null
+  reusable: boolean
+}
+
 @Component({
   selector: 'app-automatic-deploy',
   templateUrl: './automatic.component.html',
@@ -60,6 +67,21 @@ export class AutomaticComponent implements OnInit {
   public currentUserInfoLoading: boolean = false
   public computeContextsLoading: boolean = false
   public adminGroups: { id: string; name: string }[] = []
+  /** Compute contexts enriched with their batch identity and reusability */
+  public computeContextInfos: ComputeContextInfo[] = []
+  /** Distinct batch identities (runServerAs) across reusable contexts */
+  public batchIds: string[] = []
+  public selectedBatchId: string = ''
+  /**
+   * When enabled, any compute context can be chosen (the batch id dropdown
+   * is hidden and no reusability / batch id filtering is applied)
+   */
+  public showAllContexts: boolean = false
+  /** All groups on the Viya server */
+  public allAdminGroups: { id: string; name: string }[] = []
+  /** Groups the current user is a member of */
+  public memberAdminGroups: { id: string; name: string }[] = []
+  public showAllGroups: boolean = false
   public runningAsUser: string | undefined
   public currentUserInfo: ViyaApiCurrentUser | null = null
   public computeContexts: ComputeContextItem[] = []
@@ -104,9 +126,9 @@ export class AutomaticComponent implements OnInit {
   }
 
   public async loadData() {
+    await this.getCurrentUser()
     await this.getAdminGroups()
     await this.getComputeContexts()
-    await this.getCurrentUser()
 
     setTimeout(() => {
       if (this.selectedComputeContext) {
@@ -120,20 +142,13 @@ export class AutomaticComponent implements OnInit {
       this.computeContextsLoading = true
 
       this.sasViyaService.getComputeContexts().subscribe(
-        (res: ViyaComputeContexts) => {
-          this.computeContextsLoading = false
-
-          const defaultContext = res.items.find(
-            (item: ComputeContextItem) =>
-              item.name === 'SAS Job Execution compute context'
-          )
-
-          if (defaultContext) {
-            this.selectedComputeContext = defaultContext.id
-          }
-
+        async (res: ViyaComputeContexts) => {
           this.computeContexts = res.items
 
+          await this.getComputeContextDetails()
+
+          this.computeContextsLoading = false
+
           resolve()
         },
         (err) => {
@@ -143,6 +158,137 @@ export class AutomaticComponent implements OnInit {
     })
   }
 
+  /**
+   * The `/compute/contexts` list view is minimal, so each context is fetched
+   * in full (in parallel) to pick up its batch identity (runServerAs) and
+   * whether it reuses server processes.  Only reusable contexts with a batch
+   * identity feed the batch id dropdown.
+   */
+  public async getComputeContextDetails() {
+    const infos = await Promise.all(
+      this.computeContexts.map(
+        (context: ComputeContextItem) =>
+          new Promise<ComputeContextInfo>((resolveInfo) => {
+            this.sasViyaService.getComputeContextById(context.id).subscribe(
+              (details: ComputeContextDetails) => {
+                resolveInfo({
+                  id: context.id,
+                  name: context.name,
+                  runAs: this.extractRunAs(details),
+                  reusable: details.attributes?.reuseServerProcesses === 'true'
+                })
+              },
+              (err: any) => {
+                this.loggerService.error(
+                  `Error while getting compute context ${context.name}`,
+                  err
+                )
+
+                resolveInfo({
+                  id: context.id,
+                  name: context.name,
+                  runAs: null,
+                  reusable: false
+                })
+              }
+            )
+          })
+      )
+    )
+
+    this.computeContextInfos = infos
+
+    this.batchIds = [
+      ...new Set(
+        infos
+          .filter((info) => info.reusable && info.runAs)
+          .map((info) => info.runAs as string)
+      )
+    ]
+
+    if (this.batchIds.length) {
+      this.selectedBatchId = this.batchIds[0]
+    }
+
+    const filteredContexts = this.getFilteredComputeContexts()
+
+    const defaultContext =
+      filteredContexts.find(
+        (info: ComputeContextInfo) =>
+          info.name === 'SAS Job Execution compute context'
+      ) || filteredContexts[0]
+
+    if (defaultContext) {
+      this.selectedComputeContext = defaultContext.id
+    }
+  }
+
+  /**
+   * Pulls the runAs (batch) identity out of a compute context. The batch user
+   * is exposed as `attributes.runServerAs` in the Viya compute context API
+   * response. Not all contexts expose one.
+   */
+  public extractRunAs(details: ComputeContextDetails): string | null {
+    return details.attributes?.runServerAs || null
+  }
+
+  /**
+   * The contexts offered in the dropdown.  By default only reusable contexts
+   * (`reuseServerProcesses`) matching the selected batch id are shown.  When
+   * the `showAllContexts` toggle is on, all contexts are shown.
+   */
+  public getFilteredComputeContexts(): ComputeContextInfo[] {
+    if (this.showAllContexts) {
+      return this.computeContextInfos
+    }
+
+    return this.computeContextInfos.filter(
+      (info: ComputeContextInfo) =>
+        info.reusable && info.runAs === this.selectedBatchId
+    )
+  }
+
+  /**
+   * Changing the batch id re-filters the compute context dropdown and makes
+   * sure the currently selected context exists in the displayed list.
+   */
+  public onBatchIdChange() {
+    this.ensureSelectedComputeContext()
+  }
+
+  /**
+   * Toggles the compute context dropdown between contexts matching the
+   * selected batch id and all contexts on the Viya server.
+   */
+  public onShowAllContextsChange() {
+    this.ensureSelectedComputeContext()
+  }
+
+  private ensureSelectedComputeContext() {
+    const filteredContexts = this.getFilteredComputeContexts()
+
+    const selectedContextExists = filteredContexts.some(
+      (info: ComputeContextInfo) => info.id === this.selectedComputeContext
+    )
+
+    if (!selectedContextExists) {
+      this.selectedComputeContext = filteredContexts.length
+        ? filteredContexts[0].id
+        : ''
+    }
+
+    this.updateRunningAsUser()
+  }
+
+  private updateRunningAsUser() {
+    const selectedContext = this.computeContextInfos.find(
+      (info: ComputeContextInfo) => info.id === this.selectedComputeContext
+    )
+
+    this.runningAsUser =
+      selectedContext?.runAs || this.currentUserInfo?.id || 'unknown'
+  }
+
   public async getCurrentUser() {
     return new Promise<void>((resolve, reject) => {
       this.currentUserInfoLoading = true
@@ -170,16 +316,21 @@ export class AutomaticComponent implements OnInit {
       this.adminGroupsLoading = true
       ;(this.sasViyaService
         .getAdminGroups()
-        .subscribe((res: ViyaApiIdentities) => {
-          this.adminGroupsLoading = false
+        .subscribe(async (res: ViyaApiIdentities) => {
           // Map admin groups with only needed fields
-          this.adminGroups = res.items.map((item: Item) => {
+          this.allAdminGroups = res.items.map((item: Item) => {
             return {
               id: item.id,
               name: item.name
             }
           })
 
+          await this.getMemberAdminGroups()
+
+          this.applyAdminGroupFilter()
+
+          this.adminGroupsLoading = false
+
           resolve()
         }),
         (err: any) => {
@@ -192,16 +343,72 @@ export class AutomaticComponent implements OnInit {
     })
   }
 
-  public async onComputeContextChange(computeContextId: string) {
-    this.sasViyaService
-      .getComputeContextById(computeContextId)
-      .subscribe((res: ComputeContextDetails) => {
-        if (res.attributes && res.attributes.runServerAs) {
-          this.runningAsUser = res.attributes.runServerAs
-        } else {
-          this.runningAsUser = this.currentUserInfo?.id || 'unknown'
+  /**
+   * Fetches the groups the current user is a member of, from the Viya
+   * identities API.  If the memberships cannot be fetched, the member list
+   * falls back to all groups.
+   */
+  public async getMemberAdminGroups() {
+    return new Promise<void>((resolve) => {
+      this.sasViyaService.getCurrentUserGroupMemberships().subscribe(
+        (res: ViyaApiIdentities) => {
+          const memberIds = res.items.map((item: Item) => item.id)
+
+          this.memberAdminGroups = this.allAdminGroups.filter(
+            (group: { id: string; name: string }) =>
+              memberIds.includes(group.id)
+          )
+
+          resolve()
+        },
+        (err: any) => {
+          this.loggerService.error(
+            'Error while getting user group memberships',
+            err
+          )
+          this.memberAdminGroups = this.allAdminGroups
+
+          resolve()
         }
-      })
+      )
+    })
+  }
+
+  /**
+   * Toggles the SAS Admin group dropdown between the groups the current user
+   * is a member of and all groups on the Viya server.
+   */
+  public onShowAllGroupsChange() {
+    this.applyAdminGroupFilter()
+  }
+
+  /**
+   * Applies the member/all groups filter to the dropdown and makes sure the
+   * currently selected group exists in the displayed list.
+   */
+  private applyAdminGroupFilter() {
+    this.adminGroups =
+      this.showAllGroups || !this.memberAdminGroups.length
+        ? this.allAdminGroups
+        : this.memberAdminGroups
+
+    const selectedGroupExists = this.adminGroups.some(
+      (group: { id: string; name: string }) =>
+        group.id === this.selectedAdminGroup
+    )
+
+    if (!selectedGroupExists && this.adminGroups.length) {
+      this.selectedAdminGroup = this.adminGroups[0].id
+    }
+  }
+
+  public async onComputeContextChange(computeContextId: string) {
+    // The explicit (ngModelChange) handler runs before the [(ngModel)]
+    // assignment updates the field, so synchronise the selected context id
+    // here before deriving the running user. Otherwise the displayed user
+    // lags behind the dropdown by one change.
+    this.selectedComputeContext = computeContextId
+    this.updateRunningAsUser()
   }
 
   public getComputeContextName(id: string): string | undefined {
@@ -341,7 +548,8 @@ export class AutomaticComponent implements OnInit {
             this.eventService.showAbortModal('makedata', abortMsg, {
               SYSWARNINGTEXT: abortRes.SYSWARNINGTEXT,
               SYSERRORTEXT: abortRes.SYSERRORTEXT,
-              MAC: macMsg
+              MAC: macMsg,
+              _PROGRAM: abortRes._PROGRAM
             })
           }
 
@@ -373,7 +581,9 @@ export class AutomaticComponent implements OnInit {
     let contextname = `&_contextname=${params.contextName}`
     let admin = `&admin=${params.admin}`
     let dcPath = `&dcpath=${params.dcPath}`
-    let debug = this.sasService.getDebugUrlParam()
+    // Debug is ALWAYS enabled for makedata — it runs only once during
+    // deployment and the full log is needed to diagnose any issues.
+    let debug = '&_debug=131'
 
     let programUrl =
       serverUrl +
@@ -387,6 +597,53 @@ export class AutomaticComponent implements OnInit {
       debug
 
     window.open(programUrl)
+
+    // Poll for makedata completion. The makedata service self-deletes
+    // when it finishes successfully, so we check if it's still present
+    // in the admin folder. When it's gone, reload the app fresh.
+    this.pollMakedataCompletion()
+  }
+
+  private makedataPollInterval: any = null
+
+  private pollMakedataCompletion() {
+    if (this.makedataPollInterval) {
+      clearInterval(this.makedataPollInterval)
+    }
+
+    let attempts = 0
+    const maxAttempts = 120 // 10 minutes at 5s intervals
+
+    this.makedataPollInterval = setInterval(async () => {
+      attempts++
+      console.log(
+        `pollMakedataCompletion: checking if makedata is gone (attempt ${attempts}/${maxAttempts})`
+      )
+
+      try {
+        const makedataGone = await this.sasService.viyaMakedataSuccessfull()
+        if (makedataGone) {
+          console.log(
+            'pollMakedataCompletion: makedata job is gone, reloading app'
+          )
+          clearInterval(this.makedataPollInterval)
+          this.makedataPollInterval = null
+
+          // Reload the app fresh from the SASJobExecution URL
+          window.location.href = window.location.href.split('#')[0]
+        }
+      } catch (err) {
+        console.error('pollMakedataCompletion: error checking', err)
+      }
+
+      if (attempts >= maxAttempts) {
+        console.warn(
+          'pollMakedataCompletion: timed out after 10 minutes, stopping poll'
+        )
+        clearInterval(this.makedataPollInterval)
+        this.makedataPollInterval = null
+      }
+    }, 5000)
   }
 
   /**
@@ -428,9 +685,12 @@ export class AutomaticComponent implements OnInit {
       return
     }
 
+    /* Use a function replacement so $ in computeContextName is treated as a
+      literal character. In a string replacement, $ has special meaning
+      ($&, $1, $` etc.); a function return value is used verbatim. */
     const updatedContent = indexHtmlContent.replace(
       /contextname="[^"]*"/g,
-      `contextname="${computeContextName}"`
+      () => `contextname="${computeContextName}"`
     )
 
     await this.sasService

client/src/app/deploy/sections/manual/manual.component.ts

@@ -242,6 +242,9 @@ export class ManualComponent implements OnInit {
    */
   public createDatabase(newTab: boolean = true) {
     if (newTab) {
+      // Debug is ALWAYS enabled for makedata — it runs only once during
+      // deployment and the full log is needed to diagnose any issues.
+      const _debug = '&_debug=131'
       let url =
         this.sasService.getSasjsConfig().serverUrl +
           '/SASJobExecution/?_program=' +
@@ -252,7 +255,7 @@ export class ManualComponent implements OnInit {
           this.selectedAdminGroup +
           '&DCPATH=' +
           this.dcPath +
-          this.sasService.getDebugUrlParam()
+          _debug
 
       window.open(url, '_blank')
 

client/src/app/deploy/sections/sasjs-configurator/sasjs-configurator.component.ts

@@ -120,8 +120,10 @@ export class SasjsConfiguratorComponent implements OnInit {
    * Creating database
    */
   makeData() {
-    // const _debug = "&_debug=131"; //debug on
-    const _debug = ' ' //debug off
+    // Debug is ALWAYS enabled for this service. The makedata service runs only once
+    // during deployment configuration, so full debug output (131) is always wanted
+    // to help diagnose any issues. This is NOT temporary — do not remove or gate it.
+    const _debug = '&_debug=131'
 
     let executor = this.sasService.getExecutionPath()
     const root = this.sasJsConfig.appLoc

client/src/app/editor/editor.component.ts

@@ -46,9 +46,11 @@ import { preventMenuItemAutoClose } from './utils/preventMenuItemAutoClose'
 import { findOverwrittenCells } from '../shared/dc-validator/utils/findOverwrittenCells'
 import { getFormulaCellsToPreserveOnCancel } from '../shared/dc-validator/utils/getFormulaCellsToPreserveOnCancel'
 import { getRevertableCols } from '../shared/dc-validator/utils/getRevertableCols'
+import { resolveRevertedCellValue } from '../shared/dc-validator/utils/resolveRevertedCellValue'
 import { getStableFormulaBaseCols } from '../shared/dc-validator/utils/getStableFormulaBaseCols'
 import { syncOverwrittenCellComment } from '../shared/dc-validator/utils/syncOverwrittenCellComment'
 import { parseFormulaRule } from '../shared/dc-validator/utils/parseFormulaRule'
+import { substituteColumnReferences } from '../shared/dc-validator/utils/substituteColumnReferences'
 import { DcValidator } from '../shared/dc-validator/dc-validator'
 import { Col } from '../shared/dc-validator/models/col.model'
 import { DcValidation } from '../shared/dc-validator/models/dc-validation.model'
@@ -57,6 +59,17 @@ import { getHotDataSchema } from '../shared/dc-validator/utils/getHotDataSchema'
 import { excelRound } from '../shared/dc-validator/utils/excelRound'
 import { isEmpty } from '../shared/dc-validator/utils/isEmpty'
 import { hasFormulaRules } from '../shared/dc-validator/utils/hasFormulaRules'
+import { getFormulaColumnNames } from '../shared/dc-validator/utils/getFormulaColumnNames'
+import { isCharacterColumn } from '../shared/dc-validator/utils/isCharacterColumn'
+import { escapeCharacterColumnValue } from '../shared/dc-validator/utils/escapeCharacterColumnValue'
+import { unescapeFormula } from '../shared/dc-validator/utils/unescapeFormula'
+import {
+  AutoEscapedCellMap,
+  getRowKey,
+  markAutoEscaped,
+  isAutoEscaped,
+  clearAutoEscaped
+} from '../shared/dc-validator/utils/autoEscapedCellTracker'
 import { HyperFormula } from 'hyperformula'
 import { parseLabelsParam } from '../shared/utils/parse-labels-param'
 import { getDisplayColHeaders } from '../shared/utils/display-col-headers'
@@ -73,6 +86,7 @@ import { parseTableColumns } from './utils/grid.utils'
 import { classifyRow } from './utils/classifyRow'
 import { withoutEditStatus } from './utils/withoutEditStatus'
 import { getEditStatusSymbol } from './utils/getEditStatusSymbol'
+import { normalizeSortConfig } from './utils/normalizeSortConfig'
 import { EDIT_STATUS_COLUMN_NAME } from '../shared/dc-validator/utils/editStatusColumnRule'
 import {
   errorRenderer,
@@ -156,6 +170,12 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       licenseKey: this.hotTable.licenseKey,
       readOnly: this.hotTable.readOnly,
       copyPaste: this.hotTable.copyPaste,
+      // Must carry the current formulas config through - this rebuilds
+      // hotTableSettings (and re-applies it via the [settings] binding)
+      // whenever hot_license_key emits, which can happen after a table has
+      // already loaded and turned formulas on; omitting the key here would
+      // let that rebuild silently drop it again.
+      formulas: this.hotTable.formulas,
       contextMenu: true,
       className: 'htDark',
       theme: 'ht-theme-classic'
@@ -170,6 +190,11 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     height: 'calc(100vh - 160px)',
     licenseKey: undefined,
     readOnly: true,
+    // Every table gets the formulas engine, regardless of whether it
+    // declares any HARDFORMULA/SOFTFORMULA rules - set here (not only
+    // inside initSetup) so it's already present in hotTableSettings for
+    // the very first [settings] binding, before any table has loaded.
+    formulas: { engine: HyperFormula, licenseKey: 'gpl-v3' },
     copyPaste: {
       copyColumnHeaders: true,
       copyColumnHeadersOnly: true
@@ -262,12 +287,71 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
                 hot.setDataAtRowProp(
                   row,
                   prop,
-                  isNumericCol ? Number(rawValueText) : rawValueText
+                  resolveRevertedCellValue(rawValueText, isNumericCol),
+                  'revert'
                 )
                 commentsPlugin.removeCommentAtCell(row, col)
               }
             }
           },
+          // Only ever shown when the selection contains at least one cell
+          // this app itself auto-escaped (see character-column-formula-
+          // plan.md) - a genuinely literal leading `'` the backend sent, or
+          // one the user typed themselves, is never in autoEscapedCells and
+          // so never makes this item appear or gets touched by it.
+          apply_as_formula: {
+            name: 'Apply as formula',
+            hidden: (): boolean => {
+              const hot = this.hotInstance
+              if (hot.getSettings().readOnly) return true
+
+              const ranges: CellRange[] | undefined = hot.getSelectedRange()
+              if (!ranges || ranges.length === 0) return true
+
+              const cells = expandCellRanges(
+                ranges,
+                hot.countRows(),
+                hot.countCols()
+              )
+
+              return !cells.some(({ row, col }) => {
+                const dataRow = this.dataSource[hot.toPhysicalRow(row)]
+                if (!dataRow) return false
+
+                const colName = hot.colToProp(col) as string
+                return isAutoEscaped(
+                  this.autoEscapedCells,
+                  getRowKey(dataRow, this.headerPks),
+                  colName
+                )
+              })
+            },
+            callback: (key: string, selection: any[]) => {
+              const hot = this.hotInstance
+              const cells = expandCellRanges(
+                selection.map((sel) => ({ from: sel.start, to: sel.end })),
+                hot.countRows(),
+                hot.countCols()
+              )
+
+              for (const { row, col } of cells) {
+                const dataRow = this.dataSource[hot.toPhysicalRow(row)]
+                if (!dataRow) continue
+
+                const colName = hot.colToProp(col) as string
+                const rowKey = getRowKey(dataRow, this.headerPks)
+                if (!isAutoEscaped(this.autoEscapedCells, rowKey, colName))
+                  continue
+
+                hot.setDataAtRowProp(
+                  row,
+                  colName,
+                  unescapeFormula(hot.getDataAtRowProp(row, colName))
+                )
+                clearAutoEscaped(this.autoEscapedCells, rowKey, colName)
+              }
+            }
+          },
           row_above: {
             name: 'Insert Row above',
             hidden: () => this.hotTable.readOnly === true,
@@ -492,6 +576,12 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
   dataSourceRaw!: any[]
   dataSourceBeforeSubmit!: any[]
   dataModified!: any[]
+  // Cells the app itself auto-escaped (see escapeCharacterColumnValue) -
+  // PK-keyed, not a property on the row objects, so it can never leak into
+  // the submit payload and survives insert/delete/sort. Reset on every
+  // fresh load, same lifecycle as dataSourceRaw/dataSourceUnchanged. See
+  // character-column-formula-plan.md.
+  private autoEscapedCells: AutoEscapedCellMap = new Map()
 
   public filePasswordSubject: Subject<string | undefined> = new Subject()
   public fileUnlockError = false
@@ -560,6 +650,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
   private ariaObserver: MutationObserver | undefined
   private ariaCheckInterval: any | undefined
   private gridResizeObserver: ResizeObserver | undefined
+  private pasteListener: ((event: ClipboardEvent) => void) | undefined
 
   constructor(
     private licenceService: LicenceService,
@@ -805,7 +896,8 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
             this.eventService.showAbortModal('', abortMsg, {
               SYSWARNINGTEXT: abortRes.SYSWARNINGTEXT,
               SYSERRORTEXT: abortRes.SYSERRORTEXT,
-              MAC: macMsg
+              MAC: macMsg,
+              _PROGRAM: abortRes._PROGRAM
             })
           }
         },
@@ -855,6 +947,36 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       previewDatasource.push(itemObject)
     })
 
+    // An uploaded file is bulk, external data - not something the user
+    // typed/pasted cell-by-cell - so a `=`-led value in a character column
+    // must be escaped to inert text, the same way a fresh backend load is
+    // (see initSetup's own escape loop). Formulas are enabled for every
+    // table, so without this a spreadsheet cell that merely looks like a
+    // formula (an inventory code, a serial number) would silently start
+    // evaluating as a live one.
+    const formulaBaseCols = getFormulaColumnNames(
+      this.dcValidator?.getDqDetails() ?? []
+    )
+    this.autoEscapedCells = new Map()
+    for (const row of previewDatasource) {
+      const rowKey = getRowKey(row, this.headerPks)
+      for (const colName of this.headerColumns) {
+        if (formulaBaseCols.includes(colName)) continue
+        if (!isCharacterColumn(this.cols, colName)) continue
+
+        const { value, wasEscaped } = escapeCharacterColumnValue(row[colName])
+        // A row whose backend source never had this column at all (a
+        // sparse row) must stay that way - writing back unconditionally
+        // would materialize a new own property set to undefined, which
+        // the SASjs adapter rejects at submit time even though nothing
+        // about this cell ever changed.
+        if (!wasEscaped) continue
+
+        row[colName] = value
+        markAutoEscaped(this.autoEscapedCells, rowKey, colName)
+      }
+    }
+
     this.dataSourceUnchanged = this.helperService.deepClone(this.dataSource)
 
     this.dataSource = previewDatasource
@@ -864,7 +986,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
 
     this.excelUploadState = 'Validating-HOT'
 
-    hot.updateSettings(
+    this.updateSettingsSortSafe(
       {
         data: this.dataSource,
         maxRows: Infinity
@@ -889,7 +1011,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
    * @param discardData wheter to discard data parsed from the file or to keep it in the table after dropping a attached excel file
    */
   public discardPendingExcel(discardData?: boolean) {
-    this.hotInstance.updateSettings({
+    this.updateSettingsSortSafe({
       maxRows: this.licenceState.value.editor_rows_allowed
     })
 
@@ -1083,13 +1205,49 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     if (!hot) return []
     try {
       const plugin = hot.getPlugin('multiColumnSorting')
-      const cfg = plugin?.getSortConfig()
-      return Array.isArray(cfg) ? cfg : cfg ? [cfg] : []
+      return normalizeSortConfig(plugin?.getSortConfig())
     } catch {
       return []
     }
   }
 
+  /**
+   * Wraps hot.updateSettings() to work around a genuine Handsontable
+   * multiColumnSorting + formulas plugin bug: calling updateSettings() -
+   * with ANY settings, even a bare {} - while a sort is active corrupts
+   * formula cell references, rendering '#REF!' instead of their computed
+   * value (reproduced directly: a bare {} settings object alone is enough
+   * to trigger it, independent of what it actually changes). Clearing the
+   * sort first and restoring it immediately afterward avoids the
+   * corruption entirely. Not needed for the very first updateSettings()
+   * call at initial table load, since nothing can be sorted yet then.
+   *
+   * editTable()/cancelEdit() deliberately do NOT call this wrapper - they
+   * manage their own clear/restore inline, because their restore point is
+   * later in the method (after other logic like syncOverwrittenComments()),
+   * not immediately after their own updateSettings() call the way this
+   * wrapper restores it. Don't "simplify" those two to use this instead
+   * without preserving that later restore point.
+   */
+  private updateSettingsSortSafe(
+    settings: Handsontable.GridSettings,
+    render?: boolean
+  ): void {
+    const hot = this.hotInstance
+    const columnSorting = hot.getPlugin('multiColumnSorting')
+    const sortConfigs = this.getCurrentSortConfigs()
+
+    if (sortConfigs.length > 0) columnSorting.clearSort()
+
+    hot.updateSettings(settings, render)
+
+    // sort() replaces the entire sort state on every call rather than
+    // accumulating (see its own JSDoc) - restoring a multi-column sort
+    // needs one call with the whole array, not a loop of single-column
+    // calls, or every column but the last silently loses its sort.
+    if (sortConfigs.length > 0) columnSorting.sort(sortConfigs)
+  }
+
   editTable(previewEdit?: boolean, newRow?: boolean) {
     this.toggleHotPlugin('contextMenu', true)
 
@@ -1119,6 +1277,13 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       this.hotTable.readOnly = false
       this.hotTable.data = this.dataSource
 
+      // Handsontable's multiColumnSorting + formulas integration corrupts
+      // formula cell references (renders '#REF!') if updateSettings() runs
+      // while a sort is still active, even for an unrelated setting -
+      // clear it first and restore the captured sortConfigs afterward
+      // (below), rather than leaving the sort applied across the call.
+      if (sortConfigs.length > 0) columnSorting.clearSort()
+
       hot.updateSettings(
         {
           readOnly: this.hotTable.readOnly
@@ -1139,9 +1304,11 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       // same as cancelEdit() does for the read-only return path.
       this.syncOverwrittenComments()
 
-      for (const sortConfig of sortConfigs) {
-        columnSorting.sort(sortConfig)
-      }
+      // sort() replaces the entire sort state on every call rather than
+      // accumulating (see its own JSDoc) - a multi-column sort needs one
+      // call with the whole array, not a loop of single-column calls, or
+      // every column but the last silently loses its sort.
+      if (sortConfigs.length > 0) columnSorting.sort(sortConfigs)
 
       this.reSetCellValidationValues()
 
@@ -1184,10 +1351,16 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
 
     const hot = this.hotInstance
     const columnSorting = hot.getPlugin('multiColumnSorting')
-    const columnSortConfig = columnSorting.getSortConfig()
-    const sortConfigs = Array.isArray(columnSortConfig)
-      ? columnSortConfig
-      : [columnSortConfig]
+    const sortConfigs = this.getCurrentSortConfigs()
+
+    // Cleared here, before anything below reads a row by index - both the
+    // getFormulaCellsToPreserveOnCancel callbacks just below (physical
+    // rowIndex passed straight to visual-row-expecting Handsontable APIs)
+    // and the later updateSettings() call (see editTable()'s own comment -
+    // corrupts formula cell references while a sort is active) need the
+    // grid unsorted. Restored below, after the re-render, from the
+    // sortConfigs captured here.
+    if (sortConfigs.length > 0) columnSorting.clearSort()
 
     if (this.dataSourceUnchanged) {
       // dataSourceUnchanged deliberately holds the RAW pre-formula value for
@@ -1223,6 +1396,10 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     this.hotTable.data = this.dataSource
     this.hotTable.readOnly = true
 
+    // Sort was already cleared above (before the getFormulaCellsToPreserveOnCancel
+    // callbacks); still cleared here going into updateSettings() - see
+    // editTable()'s own comment: it corrupts formula cell references
+    // ('#REF!') while a sort is active. Restored below via sortConfigs.
     hot.updateSettings(
       {
         readOnly: this.hotTable.readOnly,
@@ -1242,9 +1419,11 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     this.modifedRowsIndexes = []
     hot.validateCells()
     // this.editRecordListeners();
-    for (const sortConfig of sortConfigs) {
-      columnSorting.sort(sortConfig)
-    }
+    // sort() replaces the entire sort state on every call rather than
+    // accumulating (see its own JSDoc) - a multi-column sort needs one
+    // call with the whole array, not a loop of single-column calls, or
+    // every column but the last silently loses its sort.
+    if (sortConfigs.length > 0) columnSorting.sort(sortConfigs)
 
     this.checkRowLimit()
 
@@ -1449,7 +1628,14 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       // about the new row and HARDFORMULA/SOFTFORMULA columns silently
       // fall out of sync with the grid's own data.
       hot.alter('insert_row_below', newIndex - 1, 1)
-      this.dataSource[newIndex].noLinkOption = true
+
+      // alter() is expected to splice synchronously into dataSource, but
+      // this guards against relying on that if the data binding ever
+      // becomes async.
+      if (this.dataSource[newIndex]) {
+        this.dataSource[newIndex].noLinkOption = true
+      }
+
       this.seedFormulaValuesForRow(newIndex)
       this.updateEditStatusForRow(newIndex)
 
@@ -1600,6 +1786,13 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
    * call after a row insert/delete/sort has moved things around). No-op if
    * the row has no PK match in dataSourceRaw (a newly-inserted row) - see
    * findOverwrittenCells for why that's never revertable.
+   *
+   * rowIndex is physical (matches dataSource's own order, same contract as
+   * syncOverwrittenComments()'s own dataSource.forEach caller below) -
+   * getDataAtRowProp and the comments plugin's own cell API all document a
+   * VISUAL row instead, so it's translated via hot.toVisualRow() for every
+   * Handsontable/plugin call, while dataSource itself stays indexed by the
+   * physical rowIndex throughout.
    */
   private syncOverwrittenCommentForCell(rowIndex: number, prop: string): void {
     const hot = this.hotInstance
@@ -1611,11 +1804,14 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     )
     if (!rawRow) return
 
+    const visualRow = hot.toVisualRow(rowIndex)
+    if (visualRow === null) return
+
     const colIndex = hot.propToCol(prop) as number
     const commentsPlugin = hot.getPlugin('comments')
-    const currentValue = hot.getDataAtRowProp(rowIndex, prop)
+    const currentValue = hot.getDataAtRowProp(visualRow, prop)
     const hasCommentAlready = !!commentsPlugin.getCommentAtCell(
-      rowIndex,
+      visualRow,
       colIndex
     )
 
@@ -1627,12 +1823,12 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
 
     if (action === 'set') {
       commentsPlugin.setCommentAtCell(
-        rowIndex,
+        visualRow,
         colIndex,
         `${EditorComponent.ORIGINAL_VALUE_COMMENT_PREFIX}${rawRow[prop]}`
       )
     } else if (action === 'remove') {
-      commentsPlugin.removeCommentAtCell(rowIndex, colIndex)
+      commentsPlugin.removeCommentAtCell(visualRow, colIndex)
     }
   }
 
@@ -1727,6 +1923,53 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     }
   }
 
+  /**
+   * A primary key identifies its own row for the rest of the edit session
+   * (getRowKey, dataModified, classifyRow all match rows by comparing PK
+   * values) - a HARDFORMULA/SOFTFORMULA rule on a PK column gets seeded by
+   * applyFormulaRules as a live `=...` formula string exactly like any
+   * other formula column, but a PK can never be allowed to keep
+   * recalculating for the rest of the session the way a normal formula
+   * column does: saveTable's own formulaRules loop only resolves a formula
+   * column's computed value once, at submit, so anything that changed a
+   * column the PK's formula depends on in between would silently shift the
+   * PK out from under dataModified/classifyRow's PK-matching - the same
+   * failure mode the afterChange hook below already prevents for a
+   * user-typed PK formula. So a PK-as-formula-column is frozen to its
+   * computed value once, right here at load. Both dataSource and
+   * dataSourceUnchanged get the same frozen value - overriding whatever
+   * overlayFormulaRawValuesOnUnchanged set for it, since a PK's identity
+   * role takes priority over the normal "did a formula overwrite real
+   * data" modified-detection every other formula column gets.
+   */
+  private resolvePkFormulaSeedValues(): void {
+    const hot = this.hotInstance
+    const formulaBaseCols = getFormulaColumnNames(
+      this.dcValidator?.getDqDetails() ?? []
+    )
+    const pkFormulaCols = this.headerPks.filter((pk) =>
+      formulaBaseCols.includes(pk)
+    )
+    if (pkFormulaCols.length === 0) return
+
+    // Deferred for the same reason as this method's own caller
+    // (markOverwrittenCells) defers its setDataAtRowProp call above: right
+    // after the initial hot.updateSettings(), the Formulas plugin's
+    // hidden-column index mapping hasn't settled yet, and setDataAtRowProp
+    // throws ExpectedValueOfTypeError before it has.
+    setTimeout(() => {
+      this.dataSource.forEach((_row, rowIndex) => {
+        for (const pkCol of pkFormulaCols) {
+          const computed = hot.getDataAtRowProp(rowIndex, pkCol)
+          hot.setDataAtRowProp(rowIndex, pkCol, computed, 'resolvePkFormula')
+
+          const unchangedRow = this.dataSourceUnchanged?.[rowIndex]
+          if (unchangedRow) unchangedRow[pkCol] = computed
+        }
+      })
+    }, 0)
+  }
+
   /**
    * Recomputes this row's classification (M/A/D/U) and writes it into the
    * EDIT_STATUS cell - the same classification the row-header symbol shows,
@@ -1736,6 +1979,12 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
    * HyperFormula is notified and dependents recalculate - the source string
    * also lets the afterChange hook below recognise and ignore its own
    * writes, avoiding infinite recursion.
+   *
+   * rowIndex is physical (matches dataSource's own order, same contract as
+   * addRow()/insertRowAtPosition()'s own callers) - setDataAtRowProp
+   * documents a VISUAL row instead, so it's translated via
+   * hot.toVisualRow() right before that one call, while dataSource itself
+   * stays indexed by the physical rowIndex.
    */
   private updateEditStatusForRow(rowIndex: number): void {
     const dataRow = this.dataSource[rowIndex]
@@ -1747,8 +1996,11 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       this.headerPks
     )
 
+    const visualRow = this.hotInstance.toVisualRow(rowIndex)
+    if (visualRow === null) return
+
     this.hotInstance.setDataAtRowProp(
-      rowIndex,
+      visualRow,
       EDIT_STATUS_COLUMN_NAME,
       status,
       'editStatus'
@@ -1761,7 +2013,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     this.hotTable.data = this.dataSource
 
     const hot = this.hotInstance
-    hot.updateSettings(
+    this.updateSettingsSortSafe(
       {
         data: this.dataSource,
         colHeaders: getDisplayColHeaders(
@@ -2227,7 +2479,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       }
     }
 
-    hot.updateSettings(
+    this.updateSettingsSortSafe(
       {
         data: this.dataSource,
         colHeaders: getDisplayColHeaders(
@@ -2310,6 +2562,54 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       })
     }
 
+    // Two things every other column (HARDFORMULA/SOFTFORMULA ones are
+    // already fully handled above) still needs before submission:
+    //
+    // 1. Strip the escape marker this app added (see
+    //    escapeCharacterColumnValue/character-column-formula-plan.md) from
+    //    every cell the app itself marked, so it never reaches the
+    //    backend. A cell the user has since edited, or one whose leading
+    //    `'` was already genuine backend data, is never in
+    //    autoEscapedCells and so is never touched here - it submits
+    //    exactly as it currently reads. autoEscapedCells is only ever
+    //    populated for character columns, so this is naturally a no-op
+    //    for any other column.
+    // 2. Resolve the computed value of a genuine, currently-live formula
+    //    (typed, pasted, or "Apply as formula"'d - never auto-escaped, so
+    //    its raw dataSource string still starts with a bare `=`) the same
+    //    way the HARDFORMULA/SOFTFORMULA loop above does - otherwise the
+    //    raw formula text itself would reach the backend instead of what
+    //    the grid actually displays. Not scoped to character columns -
+    //    HyperFormula evaluates a `=`-led value the same way regardless
+    //    of the column's own declared type, and coerceNumericRow
+    //    (beforePaste/beforeAutofill) deliberately leaves a non-numeric,
+    //    formula-looking paste alone rather than forcing it to NaN, so a
+    //    numeric column can end up holding a live formula too.
+    //
+    // Runs regardless of whether this table has any HARDFORMULA/
+    // SOFTFORMULA rules - formulas (and both of the above) are enabled
+    // everywhere.
+    const formulaBaseCols = getFormulaColumnNames(
+      this.dcValidator?.getDqDetails() ?? []
+    )
+    data.forEach((row: any, physicalRowIndex: number) => {
+      const visualRowIndex = hot.toVisualRow(physicalRowIndex)
+      const rowKey = getRowKey(row, this.headerPks)
+
+      for (const colName of this.headerColumns) {
+        if (formulaBaseCols.includes(colName)) continue
+
+        if (isAutoEscaped(this.autoEscapedCells, rowKey, colName)) {
+          row[colName] = unescapeFormula(row[colName])
+        } else if (
+          typeof row[colName] === 'string' &&
+          row[colName].startsWith('=')
+        ) {
+          row[colName] = hot.getDataAtRowProp(visualRowIndex, colName)
+        }
+      }
+    })
+
     data = data.filter((dataRow: any) => {
       const elModified = this.dataModified.find((row) => {
         for (const pkCol of this.headerPks) {
@@ -2489,9 +2789,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     if (this.recordAction === 'ADD' && !confirmButtonClicked) {
       this.dataSource = this.helperService.deepClone(this.prevDataSource)
 
-      const hot = this.hotInstance
-
-      hot.updateSettings(
+      this.updateSettingsSortSafe(
         {
           data: this.dataSource
         },
@@ -2518,9 +2816,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     this.dataSource[closingRecordIndex] = this.currentEditRecord
     this.hotTable.data[closingRecordIndex] = this.currentEditRecord
 
-    const hot = this.hotInstance
-
-    hot.updateSettings(
+    this.updateSettingsSortSafe(
       {
         data: this.dataSource
       },
@@ -2692,7 +2988,13 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
 
     const cellData = hot.getDataAtCell(row, column)
     const clickedRow = this.helperService.deepClone(this.dataSource[row])
-    const clickedColumnKey = Object.keys(clickedRow)[column]
+    // hot.colToProp resolves the column's actual configured data key -
+    // Object.keys(clickedRow)[column] only worked by coincidence for a
+    // fully-populated row; a sparse row (one whose backend source never
+    // populated some earlier column) has fewer real keys than the grid has
+    // columns, so counting into its own key list drifts out of alignment
+    // (or runs out of bounds entirely) once column indexes past the gap.
+    const clickedColumnKey = hot.colToProp(column) as string
     const skipRender = !!opts?.skipRender
     const myEpoch = this.validationEpoch
 
@@ -3287,7 +3589,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       this.useLabels
     )
 
-    this.hotInstance.updateSettings({ colHeaders }, false)
+    this.updateSettingsSortSafe({ colHeaders }, false)
   }
 
   ngOnDestroy() {
@@ -3311,6 +3613,16 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       this.gridResizeObserver = undefined
     }
 
+    // Remove the native paste listener (see its own registration comment -
+    // attached once on rootElement rather than per edit session)
+    if (this.pasteListener && this.hotInstance?.rootElement) {
+      this.hotInstance.rootElement.removeEventListener(
+        'paste',
+        this.pasteListener
+      )
+      this.pasteListener = undefined
+    }
+
     // Cancel any pending debounced VA apply
     if (this.vaDebounceTimer) {
       clearTimeout(this.vaDebounceTimer)
@@ -3336,7 +3648,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       requestAnimationFrame(() => {
         const hot = this.hotInstance
         if (!hot || hot.isDestroyed) return
-        hot.updateSettings({ height: this.hotTable.height }, false)
+        this.updateSettingsSortSafe({ height: this.hotTable.height }, false)
         hot.render()
       })
     })
@@ -3588,13 +3900,15 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       response.data.dqdata
     )
 
-    // Only turn on Handsontable's formulas plugin (HyperFormula engine) for
-    // tables that actually use HARDFORMULA/SOFTFORMULA - it's not free to
-    // run for every grid. gpl-v3: this app embeds HyperFormula under its
-    // GPLv3 free-tier terms, not a purchased commercial key.
-    this.hotTable.formulas = hasFormulaRules(response.data.dqrules)
-      ? { engine: HyperFormula, licenseKey: 'gpl-v3' }
-      : false
+    // this.hotTable.formulas is set once, on the field itself, and applies
+    // to every table regardless of whether it has any HARDFORMULA/
+    // SOFTFORMULA rules.
+
+    // BASE_COL names of every HARDFORMULA/SOFTFORMULA rule - applyFormulaRules
+    // below legitimately writes a live `=...` formula string into these, so
+    // the escape loop further down must skip them (they're never
+    // backend-sourced text that merely looks like a formula).
+    const formulaBaseCols = getFormulaColumnNames(response.data.dqrules)
 
     this.cellValidation = this.dcValidator.getRules()
 
@@ -3631,6 +3945,48 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       this.userService.user?.username ?? ''
     )
 
+    // Prevent a `=`-led value from the backend in any character column from
+    // being treated as a live formula - see character-column-formula-plan.md.
+    // Must run after applyFormulaRules above (which legitimately writes a
+    // live formula string into HARDFORMULA/SOFTFORMULA columns) and before
+    // dataSourceUnchanged is cloned below, so the "unchanged baseline"
+    // already reflects the escaped form rather than flagging these cells as
+    // modified. Only backend-sourced values are ever escaped here - a value
+    // the user later types or pastes is never touched (see beforeChange).
+    //
+    // Also escapes the matching dataSourceRaw cell (still index-aligned
+    // with dataSource here, before any sort/insert reorders things) for the
+    // same non-formula columns, so it stays in sync with what the user
+    // actually sees. dataSourceRaw drives the Revert feature - if it kept
+    // the true unescaped backend string instead, reverting a cell after
+    // "Apply as formula" would restore the raw `=...` text, which
+    // (formulas being enabled everywhere) evaluates live again instead of
+    // going back to the escaped text the user started from. Formula base
+    // columns are still skipped, so dataSourceRaw keeps meaning "value
+    // before the formula overwrote it" for them, same as before.
+    this.autoEscapedCells = new Map()
+    this.dataSource.forEach((row, rowIndex) => {
+      const rowKey = getRowKey(row, this.headerPks)
+      for (const colName of this.headerColumns) {
+        if (formulaBaseCols.includes(colName)) continue
+        if (!isCharacterColumn(this.cols, colName)) continue
+
+        const { value, wasEscaped } = escapeCharacterColumnValue(row[colName])
+        // A row whose backend source never had this column at all (a
+        // sparse row) must stay that way - writing back unconditionally
+        // would materialize a new own property set to undefined, which
+        // the SASjs adapter rejects at submit time even though nothing
+        // about this cell ever changed.
+        if (!wasEscaped) continue
+
+        row[colName] = value
+        if (this.dataSourceRaw[rowIndex]) {
+          this.dataSourceRaw[rowIndex][colName] = value
+        }
+        markAutoEscaped(this.autoEscapedCells, rowKey, colName)
+      }
+    })
+
     // Seeded here too (not just editTable()) so a HARDFORMULA/SOFTFORMULA
     // rule that silently overwrote real pre-existing data (see
     // markOverwrittenCells) already shows the row as modified - both the
@@ -3769,7 +4125,22 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
         },
         rowHeaderWidth: 20,
         rowHeights: 24,
-        maxRows: this.licenceState.value.editor_rows_allowed || Infinity,
+        // Handsontable's own maxRows (a UI/licensing cap on how many NEW
+        // rows can be ADDED - see checkRowLimit/restrictAddRow) silently
+        // becomes HyperFormula's OWN sheet-size limit too (see
+        // getEngineSettingsOverrides in Handsontable's formulas plugin,
+        // which maps hotSettings.maxRows straight into the engine config).
+        // editor_rows_allowed can be smaller than a table's ALREADY-
+        // EXISTING row count, and formulas are on for every table - capping
+        // at editor_rows_allowed alone would make HyperFormula reject the
+        // sheet outright for any table bigger than that cap. Never let it
+        // go below what's actually loaded - the add-row restriction itself
+        // is still enforced separately, by checkRowLimit's own comparison
+        // against dataSource.length.
+        maxRows: Math.max(
+          this.dataSource.length,
+          this.licenceState.value.editor_rows_allowed || Infinity
+        ),
         invalidCellClassName: 'htInvalid',
         // Prevent automatic row creation
         autoWrapRow: false,
@@ -3925,6 +4296,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     // actually evaluated them against the data/formulas settings just
     // applied.
     this.markOverwrittenCells()
+    this.resolvePkFormulaSeedValues()
 
     this.hotTable.hidden = false
     // Keep the context menu enabled in view mode too so Copy/Export remain
@@ -4002,24 +4374,56 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     // ROUND: round numeric values Excel-style before they are written.
     // Mutating `changes` in place (rather than setDataAtRowProp) avoids
     // re-entrancy and uniformly covers edit, paste and autofill.
-    hot.addHook('beforeChange', (changes: any[]) => {
+    hot.addHook('beforeChange', (changes: any[], source: any) => {
       if (!changes) return
 
       for (const change of changes) {
         if (!change) continue
 
-        const [, prop, , newValue] = change
+        const [changeRow, prop, , newValue] = change
         const colName =
           typeof prop === 'string'
             ? prop
             : (hot.colToProp(prop as number) as string)
 
         const digits = this.dcValidator?.getRoundDigits(colName)
-        if (digits === undefined) continue
+        if (digits !== undefined) {
+          const num = Number(newValue)
+          if (newValue !== null && newValue !== '' && !isNaN(num)) {
+            change[3] = excelRound(num, digits)
+          }
+        }
 
-        const num = Number(newValue)
-        if (newValue !== null && newValue !== '' && !isNaN(num)) {
-          change[3] = excelRound(num, digits)
+        // A `=`-led value the user types or pastes is never auto-escaped
+        // (see character-column-formula-plan.md) - it's left exactly as
+        // entered, which genuinely evaluates as a live formula since
+        // formulas are enabled for every column. If this cell was
+        // previously auto-escaped from backend data, the user is now
+        // supplying their own content, so that marker no longer describes
+        // what's here - clear it so submit/"Apply as formula" leave this
+        // cell alone.
+        const row = this.dataSource[hot.toPhysicalRow(changeRow)]
+        if (!row) continue
+
+        const rowKey = getRowKey(row, this.headerPks)
+
+        // Revert (see the revert_cells context-menu item, which tags its
+        // own write with this source) restores dataSourceRaw's value
+        // verbatim - for a character column that's the same escaped form
+        // the initial-load pass would have produced, so it's correct to
+        // treat it the same way and re-mark it, letting "Apply as
+        // formula" and the submit-time strip work on it again. A value
+        // that merely happens to start with `'=` because the user typed
+        // it themselves never takes this path, since only revert_cells
+        // uses this source.
+        if (
+          source === 'revert' &&
+          typeof newValue === 'string' &&
+          newValue.startsWith("'=")
+        ) {
+          markAutoEscaped(this.autoEscapedCells, rowKey, colName)
+        } else {
+          clearAutoEscaped(this.autoEscapedCells, rowKey, colName)
         }
       }
     })
@@ -4072,9 +4476,18 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       for (const change of changes) {
         if (!change) continue
 
-        const [row, prop] = change
+        // changes gives the VISUAL row (Handsontable's own documented
+        // afterChange shape) - translate to physical before indexing
+        // dataSource, same as beforeChange's own hot.toPhysicalRow(changeRow)
+        // a few lines up. syncOverwrittenCommentForCell/updateEditStatusForRow
+        // both expect physical (they translate back to visual internally,
+        // only where a Handsontable API actually needs it).
+        const [visualRow, prop] = change
         if (prop === EDIT_STATUS_COLUMN_NAME) continue
 
+        const row = hot.toPhysicalRow(visualRow)
+        if (row === null) continue
+
         changedRows.add(row)
 
         if (revertableCols.includes(prop)) {
@@ -4130,7 +4543,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       }, 50)
     })
 
-    hot.addHook('afterCreateRow', (source: any, change: any) => {
+    hot.addHook('afterCreateRow', (source: any) => {
       if (source > this.dataSource.length) {
         // don't scroll if row is not added to the end (bottom)
         const wtHolder = document.querySelector('.wtHolder')
@@ -4197,13 +4610,117 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
         return value
       })
 
-    hot.addHook('beforePaste', (data: any, cords: any) => {
-      const startCol = cords[0].startCol
-      for (let r = 0; r < data.length; r++) {
-        data[r] = coerceNumericRow(data[r], startCol)
+    // A primary key identifies its own row for the rest of the edit session
+    // (getRowKey, dataModified, classifyRow all match rows by comparing PK
+    // values) - if it's left holding a live formula, HyperFormula's
+    // recalculation can shift that identity out from under those matches
+    // between whenever the formula was entered and whenever saveTable()
+    // finally resolves it. Unlike any other column, resolve a PK's formula
+    // to its computed value immediately, right when it's entered (typed,
+    // pasted, autofilled, or "Apply as formula"'d), so the PK is never
+    // anything other than a stable, already-computed value going forward.
+    // Deferred via setTimeout, same as the NOTNULL default-population hook
+    // above, since writing back through setDataAtRowProp from inside
+    // afterChange itself would re-enter change processing immediately.
+    hot.addHook('afterChange', (changes: any[], source: any) => {
+      if (!changes || source === 'loadData' || source === 'resolvePkFormula')
+        return
+
+      for (const change of changes) {
+        if (!change) continue
+
+        const [row, prop, , newValue] = change
+        if (typeof newValue !== 'string' || !newValue.startsWith('=')) continue
+
+        const colName =
+          typeof prop === 'string'
+            ? prop
+            : (hot.colToProp(prop as number) as string)
+        if (!this.headerPks.includes(colName)) continue
+
+        setTimeout(() => {
+          const computed = hot.getDataAtRowProp(row, colName)
+          hot.setDataAtRowProp(row, colName, computed, 'resolvePkFormula')
+        }, 0)
       }
     })
 
+    hot.addHook('beforePaste', (data: any, cords: any) => {
+      const startCol = cords[0].startCol
+      const startRow = cords[0].startRow
+
+      for (let r = 0; r < data.length; r++) {
+        data[r] = coerceNumericRow(data[r], startCol)
+
+        // Translate column names in a pasted formula (e.g. "=PRICE * VOLUME")
+        // into this row's cell references ("=B4 * C4") - only meaningful on
+        // a character column, since that's what's formula-capable for user
+        // input (see character-column-formula-plan.md). Translating for a
+        // numeric column would be pointless and would leave a confusing
+        // half-translated string sitting there instead of the original,
+        // readable pasted text.
+        data[r] = data[r].map((value: any, index: number) => {
+          const colName = this.columnHeader[startCol + index]
+          if (!isCharacterColumn(this.cols, colName)) return value
+
+          return typeof value === 'string' && value.trim().startsWith('=')
+            ? substituteColumnReferences(
+                value.trim(),
+                this.headerColumns,
+                startRow + r
+              )
+            : value
+        })
+      }
+    })
+
+    // Same column-name -> cell-reference translation as beforePaste above,
+    // but for pasting directly into an open cell editor (double-click,
+    // then Cmd+V/Ctrl+V or right-click > Paste) - Handsontable's CopyPaste
+    // plugin only intercepts a grid-level paste (cell selected but not
+    // being edited); pasting into the editor's own textarea is a native
+    // browser paste event the plugin never sees. Listens once on the root
+    // element (paste events bubble) rather than attaching a new listener
+    // per edit session, to avoid accumulating listeners across repeated
+    // edits.
+    //
+    // This whole registration only ever runs once per component instance
+    // in practice (initSetup's own retry-until-ready loop returns before
+    // reaching this point on every attempt except the one that succeeds,
+    // and table-switching navigates to a fresh component instance rather
+    // than reloading in place) - but removing any previous listener before
+    // replacing the reference costs nothing and keeps that guarantee from
+    // becoming a silent leak if this assumption ever changes.
+    if (this.pasteListener) {
+      hot.rootElement.removeEventListener('paste', this.pasteListener)
+    }
+    this.pasteListener = (event: ClipboardEvent) => {
+      const editor: any = hot.getActiveEditor()
+      if (!editor || editor.row === null || event.target !== editor.TEXTAREA)
+        return
+
+      // Only on a character column - see beforePaste's own comment above
+      // for why.
+      const colName = this.columnHeader[editor.col]
+      if (!isCharacterColumn(this.cols, colName)) return
+
+      const pastedText = event.clipboardData?.getData('text/plain') ?? ''
+      if (!pastedText.trim().startsWith('=')) return
+
+      event.preventDefault()
+      editor.TEXTAREA.value = substituteColumnReferences(
+        pastedText.trim(),
+        this.headerColumns,
+        editor.row
+      )
+      // A real paste's default action would fire this itself once the
+      // text landed - since preventDefault() above skips that, dispatch it
+      // manually so the editor's own input-tracking (autosize, its
+      // internal notion of the current value) picks up the change.
+      editor.TEXTAREA.dispatchEvent(new Event('input', { bubbles: true }))
+    }
+    hot.rootElement.addEventListener('paste', this.pasteListener)
+
     hot.addHook(
       'beforeAutofill',
       (selectionData: any[][], sourceRange: any) => {

client/src/app/editor/utils/normalizeSortConfig.spec.ts

@@ -0,0 +1,20 @@
+import { normalizeSortConfig } from './normalizeSortConfig'
+
+describe('normalizeSortConfig', () => {
+  it('wraps a single config in an array', () => {
+    const cfg = { column: 0, sortOrder: 'asc' as const }
+    expect(normalizeSortConfig(cfg)).toEqual([cfg])
+  })
+
+  it('passes an array of configs through unchanged', () => {
+    const cfgs = [
+      { column: 0, sortOrder: 'asc' as const },
+      { column: 1, sortOrder: 'desc' as const }
+    ]
+    expect(normalizeSortConfig(cfgs)).toEqual(cfgs)
+  })
+
+  it('returns an empty array for undefined (nothing currently sorted)', () => {
+    expect(normalizeSortConfig(undefined)).toEqual([])
+  })
+})

client/src/app/editor/utils/normalizeSortConfig.ts

@@ -0,0 +1,13 @@
+/**
+ * multiColumnSorting's getSortConfig() returns a single config, an array of
+ * them, or undefined depending on how many columns are currently sorted -
+ * normalizes all three shapes into one array so callers never need three
+ * branches to handle it. `SortConfig` isn't part of Handsontable's public
+ * API surface in this version (only re-exported internally), and its own
+ * `.d.ts` shape isn't structurally assignable to `Record<string, unknown>`
+ * either (a specific interface isn't assignable to an index-signature type
+ * without one of its own), so `any` is the pragmatic choice here rather
+ * than fighting the type system for a type this file can't even name.
+ */
+export const normalizeSortConfig = (cfg: any): any[] =>
+  Array.isArray(cfg) ? cfg : cfg ? [cfg] : []

client/src/app/editor/utils/renderers.utils.spec.ts

@@ -1,5 +1,10 @@
 import Handsontable from 'handsontable'
-import { makeNumberFormatRenderer } from './renderers.utils'
+import {
+  makeNumberFormatRenderer,
+  errorRenderer,
+  noSpinnerRenderer,
+  spinnerRenderer
+} from './renderers.utils'
 
 describe('makeNumberFormatRenderer', () => {
   it('renders a numeric cell as EUR currency without changing the value', () => {
@@ -86,3 +91,67 @@ describe('makeNumberFormatRenderer', () => {
     container.remove()
   })
 })
+
+/**
+ * DOM-injection reproduction mirroring the editor's cell-render cycle.
+ * During dynamic cell validation the editor applies one of the status
+ * renderers to a cell via setCellMeta + hot.render().  Those renderers paint
+ * the cell value with td.innerHTML, so a value containing markup is injected
+ * and executed (the <img onerror> fires in the browser).  The value can come
+ * straight from a dataset row served by the getdata stored program, or from a
+ * typed edit.  These fail on the vulnerable implementation and pass once the
+ * renderer escapes the value.
+ */
+describe('grid cell renderers do not inject raw HTML', () => {
+  const maliciousValue = '<img src=x onerror=alert(1)>'
+
+  // Seed a real Handsontable grid with the payload as a loaded cell value,
+  // then apply the given status renderer and render — exactly the sequence the
+  // editor uses during the dynamic-validation cycle.
+  const renderWith = (
+    renderer: (
+      i: any,
+      td: any,
+      r: number,
+      c: number,
+      p: any,
+      v: any,
+      cp: any
+    ) => any
+  ) => {
+    const container = document.createElement('div')
+    document.body.appendChild(container)
+    const hot = new Handsontable(container, {
+      data: [{ SOME_CHAR: maliciousValue }],
+      columns: [{ data: 'SOME_CHAR', type: 'text' }],
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+    hot.setCellMeta(0, 0, 'renderer', renderer)
+    hot.render()
+    const td: HTMLTableCellElement | null = hot.getCell(0, 0)
+    hot.destroy()
+    container.remove()
+    return td
+  }
+
+  // A vulnerable renderer turns the value into a real <img> element with an
+  // onerror handler (proven by the browser firing alert(1)).  A safe
+  // renderer leaves no such element.  Asserting on the parsed DOM rather
+  // than the raw string avoids false passes from browser attribute normalising.
+  const assertNoInjectedElement = (td: HTMLTableCellElement | null) => {
+    expect(td?.querySelector('img[onerror]')).toBeNull()
+  }
+
+  it('noSpinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(noSpinnerRenderer))
+  })
+
+  it('errorRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(errorRenderer))
+  })
+
+  it('spinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(spinnerRenderer))
+  })
+})

client/src/app/editor/utils/renderers.utils.ts

@@ -1,5 +1,23 @@
 import Handsontable from 'handsontable'
 
+/**
+ * Returns string-safe text of any value so it can be assigned to innerHTML.
+ * The cell values painted by the status renderers are user/DB-controlled,
+ * so they must never be parsed as HTML by the browser — escaping turns any
+ * embedded markup into inert text.
+ */
+const escapeHtml = (value: any): string =>
+  String(value ?? '').replace(/[&<>"']/g, (char) => {
+    const entities: Record<string, string> = {
+      '&': '&amp;',
+      '<': '&lt;',
+      '>': '&gt;',
+      '"': '&quot;',
+      "'": '&#39;'
+    }
+    return entities[char]
+  })
+
 /**
  * Builds a display-only HOT renderer that formats numeric cell values using
  * Intl.NumberFormat. The stored/submitted value is never changed — only the
@@ -67,9 +85,7 @@ export const errorRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
+  td.innerHTML = `${escapeHtml(value)} <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
 
   return td
 }
@@ -89,7 +105,7 @@ export const noSpinnerRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = value ? value : ''
+  td.innerHTML = escapeHtml(value)
 
   return td
 }
@@ -110,9 +126,7 @@ export const spinnerRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <span class="spinner spinner-sm vertical-align-middle"></span>`
+  td.innerHTML = `${escapeHtml(value)} <span class="spinner spinner-sm vertical-align-middle"></span>`
 
   return td
 }

client/src/app/editor/utils/sortedGridRowSync.integration.spec.ts

@@ -0,0 +1,540 @@
+import Handsontable from 'handsontable'
+import { HyperFormula } from 'hyperformula'
+import { classifyRow } from './classifyRow'
+import { normalizeSortConfig } from './normalizeSortConfig'
+import { EDIT_STATUS_COLUMN_NAME } from '../../shared/dc-validator/utils/editStatusColumnRule'
+import { parseFormulaRule } from '../../shared/dc-validator/utils/parseFormulaRule'
+import { getFormulaCellsToPreserveOnCancel } from '../../shared/dc-validator/utils/getFormulaCellsToPreserveOnCancel'
+
+/**
+ * afterChange delivers changes as [visualRow, prop, oldValue, newValue] -
+ * Handsontable's own documented behaviour. Once multiColumnSorting reorders
+ * the grid, visual row order no longer matches dataSource's physical order.
+ * editor.component.ts's afterChange hook (the one syncing EDIT_STATUS and
+ * the "overwritten" comment) passes that raw visual row straight into
+ * updateEditStatusForRow/syncOverwrittenCommentForCell, both of which index
+ * dataSource directly - the same [visualRow] shape the beforeChange hook a
+ * few lines up in the same file already translates via
+ * hot.toPhysicalRow(changeRow) before doing the same kind of dataSource
+ * access.
+ *
+ * Mirrors the row extraction only (not the full hook), against a real
+ * Handsontable + multiColumnSorting instance, since proving visual/physical
+ * divergence requires an actual sort to have happened.
+ */
+describe('afterChange must translate the changed row to physical before indexing dataSource, on a sorted grid', () => {
+  const setup = (): { hot: Handsontable; dataSource: any[] } => {
+    // Physical row 0: PK=1. Physical row 1: PK=2.
+    const dataSource = [
+      { PRIMARY_KEY_FIELD: 1, SOME_CHAR: 'b' },
+      { PRIMARY_KEY_FIELD: 2, SOME_CHAR: 'a' }
+    ]
+    const hot = new Handsontable(document.createElement('div'), {
+      data: dataSource,
+      columns: [{ data: 'PRIMARY_KEY_FIELD' }, { data: 'SOME_CHAR' }],
+      multiColumnSorting: true,
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    // Ascending by SOME_CHAR flips the order: physical row 1 (SOME_CHAR
+    // 'a') becomes visual row 0; physical row 0 (SOME_CHAR 'b') becomes
+    // visual row 1.
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    sortPlugin.sort({ column: 1, sortOrder: 'asc' })
+    hot.render()
+
+    return { hot, dataSource }
+  }
+
+  it('demonstrates the failure mode: indexing dataSource with the raw visual row reads the wrong row', () => {
+    const { hot, dataSource } = setup()
+
+    // The user edits what they see as the FIRST row on screen - visual
+    // row 0, which after sorting is physical row 1 (PK=2).
+    const visualRow = 0
+
+    // Mirrors the current (buggy) afterChange hook: uses the visual row
+    // straight from `changes` to index dataSource.
+    const wronglyReadRow = dataSource[visualRow]
+    expect(wronglyReadRow.PRIMARY_KEY_FIELD).not.toEqual(2)
+    expect(wronglyReadRow.PRIMARY_KEY_FIELD).toEqual(1) // the OTHER row
+
+    // Mirrors the fix: translate visual -> physical first, matching
+    // beforeChange's own hot.toPhysicalRow(changeRow) pattern.
+    const physicalRow = hot.toPhysicalRow(visualRow)
+    const correctlyReadRow = dataSource[physicalRow as number]
+    expect(correctlyReadRow.PRIMARY_KEY_FIELD).toEqual(2)
+
+    hot.destroy()
+  })
+})
+
+/**
+ * updateEditStatusForRow/syncOverwrittenCommentForCell's own established
+ * contract (per their other existing callers - addRow/insertRowAtPosition,
+ * and syncOverwrittenComments()'s own `dataSource.forEach((_row, rowIndex)
+ * => ...)`) is a PHYSICAL row index, used to index dataSource. But both
+ * also call Handsontable APIs - hot.setDataAtRowProp/getDataAtRowProp
+ * (updateEditStatusForRow) and the comments plugin's setCommentAtCell/
+ * getCommentAtCell/removeCommentAtCell (syncOverwrittenCommentForCell) -
+ * which Handsontable's own JSDoc documents as expecting the VISUAL row.
+ * Passing the physical row straight through to those (the current code)
+ * only happens to work when visual and physical coincide - i.e. an
+ * unsorted grid. On a sorted one, translating the afterChange hook's own
+ * row (see the describe block above) is necessary but not sufficient -
+ * these two methods must also convert their own physical rowIndex
+ * parameter to visual before calling any Handsontable API, or the fix
+ * above just moves the bug rather than closing it.
+ */
+describe('updateEditStatusForRow resolves the correct visual cell on a sorted grid', () => {
+  const headerPks = ['PRIMARY_KEY_FIELD']
+
+  const setup = (): { hot: Handsontable; dataSource: any[] } => {
+    const dataSource = [
+      {
+        PRIMARY_KEY_FIELD: 1,
+        SOME_CHAR: 'b',
+        [EDIT_STATUS_COLUMN_NAME]: 'U'
+      },
+      {
+        PRIMARY_KEY_FIELD: 2,
+        SOME_CHAR: 'a',
+        [EDIT_STATUS_COLUMN_NAME]: 'U'
+      }
+    ]
+    const hot = new Handsontable(document.createElement('div'), {
+      data: dataSource,
+      columns: [
+        { data: 'PRIMARY_KEY_FIELD' },
+        { data: 'SOME_CHAR' },
+        { data: EDIT_STATUS_COLUMN_NAME }
+      ],
+      multiColumnSorting: true,
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    // Ascending by SOME_CHAR: physical row 1 (PK=2) -> visual row 0;
+    // physical row 0 (PK=1) -> visual row 1.
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    sortPlugin.sort({ column: 1, sortOrder: 'asc' })
+    hot.render()
+
+    return { hot, dataSource }
+  }
+
+  // Mirrors the CURRENT updateEditStatusForRow - uses the (correct,
+  // physical) rowIndex directly for the Handsontable write too.
+  const updateEditStatusForRowBuggy = (
+    hot: Handsontable,
+    dataSource: any[],
+    dataSourceUnchanged: any[],
+    rowIndex: number
+  ): void => {
+    const dataRow = dataSource[rowIndex]
+    if (!dataRow) return
+
+    const status = classifyRow(dataRow, dataSourceUnchanged, headerPks)
+    hot.setDataAtRowProp(
+      rowIndex,
+      EDIT_STATUS_COLUMN_NAME,
+      status,
+      'editStatus'
+    )
+  }
+
+  // Mirrors the fix - dataSource access stays physical; the Handsontable
+  // write is translated to visual first.
+  const updateEditStatusForRowFixed = (
+    hot: Handsontable,
+    dataSource: any[],
+    dataSourceUnchanged: any[],
+    rowIndex: number
+  ): void => {
+    const dataRow = dataSource[rowIndex]
+    if (!dataRow) return
+
+    const status = classifyRow(dataRow, dataSourceUnchanged, headerPks)
+    const visualRow = hot.toVisualRow(rowIndex)
+    if (visualRow === null) return
+
+    hot.setDataAtRowProp(
+      visualRow,
+      EDIT_STATUS_COLUMN_NAME,
+      status,
+      'editStatus'
+    )
+  }
+
+  it('demonstrates the failure mode: the buggy version writes the status to the wrong cell', () => {
+    const { hot, dataSource } = setup()
+    // Physical row 1 (PK=2) is the one that actually changed vs its
+    // unchanged baseline (SOME_CHAR 'a' vs 'ORIGINAL') - classifyRow
+    // should mark it 'M'. It's currently displayed at visual row 0.
+    const dataSourceUnchanged = [
+      {
+        PRIMARY_KEY_FIELD: 1,
+        SOME_CHAR: 'b',
+        [EDIT_STATUS_COLUMN_NAME]: 'U'
+      },
+      {
+        PRIMARY_KEY_FIELD: 2,
+        SOME_CHAR: 'ORIGINAL',
+        [EDIT_STATUS_COLUMN_NAME]: 'U'
+      }
+    ]
+
+    updateEditStatusForRowBuggy(hot, dataSource, dataSourceUnchanged, 1)
+
+    // Buggy: wrote 'M' to visual row 1 (PK=1's cell, unaffected), not
+    // visual row 0 (PK=2's actual current position).
+    expect(hot.getDataAtRowProp(1, EDIT_STATUS_COLUMN_NAME)).toEqual('M')
+    expect(hot.getDataAtRowProp(0, EDIT_STATUS_COLUMN_NAME)).toEqual('U')
+
+    hot.destroy()
+  })
+
+  it('writes the status to the correct (visually current) cell once fixed', () => {
+    const { hot, dataSource } = setup()
+    const dataSourceUnchanged = [
+      {
+        PRIMARY_KEY_FIELD: 1,
+        SOME_CHAR: 'b',
+        [EDIT_STATUS_COLUMN_NAME]: 'U'
+      },
+      {
+        PRIMARY_KEY_FIELD: 2,
+        SOME_CHAR: 'ORIGINAL',
+        [EDIT_STATUS_COLUMN_NAME]: 'U'
+      }
+    ]
+
+    updateEditStatusForRowFixed(hot, dataSource, dataSourceUnchanged, 1)
+
+    expect(hot.getDataAtRowProp(0, EDIT_STATUS_COLUMN_NAME)).toEqual('M')
+    expect(hot.getDataAtRowProp(1, EDIT_STATUS_COLUMN_NAME)).toEqual('U')
+
+    hot.destroy()
+  })
+})
+
+/**
+ * Same physical/visual split as updateEditStatusForRow above, for the
+ * comments plugin's own row-indexed API (setCommentAtCell/
+ * getCommentAtCell/removeCommentAtCell all document a VISUAL row -
+ * propToCol, unlike the row side, already returns a visual column, so
+ * only the row needs translating).
+ */
+describe('syncOverwrittenCommentForCell resolves the correct visual cell on a sorted grid', () => {
+  const setup = (): { hot: Handsontable } => {
+    const dataSource = [
+      { PRIMARY_KEY_FIELD: 1, SOME_CHAR: 'b' },
+      { PRIMARY_KEY_FIELD: 2, SOME_CHAR: 'a' }
+    ]
+    const hot = new Handsontable(document.createElement('div'), {
+      data: dataSource,
+      columns: [{ data: 'PRIMARY_KEY_FIELD' }, { data: 'SOME_CHAR' }],
+      multiColumnSorting: true,
+      comments: true,
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    sortPlugin.sort({ column: 1, sortOrder: 'asc' })
+    hot.render()
+
+    return { hot }
+  }
+
+  it('demonstrates the failure mode: setCommentAtCell(physicalRow, ...) lands on the wrong visual cell', () => {
+    const { hot } = setup()
+    const commentsPlugin: any = hot.getPlugin('comments')
+
+    // Mirrors the buggy call: the established-physical rowIndex (1) used
+    // directly against a Handsontable API that wants visual.
+    commentsPlugin.setCommentAtCell(1, 1, 'Original value: b')
+
+    // Landed on visual row 1 (PK=1's cell), not visual row 0 (PK=2's
+    // actual current position, physical row 1).
+    expect(commentsPlugin.getCommentAtCell(1, 1)).toEqual('Original value: b')
+    expect(commentsPlugin.getCommentAtCell(0, 1)).toBeUndefined()
+
+    hot.destroy()
+  })
+
+  it('lands on the correct visual cell once translated to visual first', () => {
+    const { hot } = setup()
+    const commentsPlugin: any = hot.getPlugin('comments')
+
+    const visualRow = hot.toVisualRow(1)
+    commentsPlugin.setCommentAtCell(visualRow, 1, 'Original value: a')
+
+    expect(commentsPlugin.getCommentAtCell(0, 1)).toEqual('Original value: a')
+    expect(commentsPlugin.getCommentAtCell(1, 1)).toBeUndefined()
+
+    hot.destroy()
+  })
+})
+
+/**
+ * A genuine Handsontable multiColumnSorting + formulas plugin bug (not
+ * specific to any of DC's own hooks): calling hot.updateSettings() - with
+ * ANY settings, even a bare {} - while a sort is active corrupts formula
+ * cell references, rendering '#REF!' instead of their computed value.
+ * Reproduced directly: a no-op {} settings object alone is enough to
+ * trigger it, independent of what it actually changes. editTable(),
+ * cancelEdit(), and every other place editor.component.ts calls
+ * updateSettings() after the grid could already be sorted (e.g. the user
+ * sorted while still read-only, then clicked Edit) hits this. The fix
+ * (updateSettingsSortSafe) clears the sort before the call and restores it
+ * immediately afterward.
+ */
+describe('updateSettings must not run while a sort is active - it corrupts formula cell references', () => {
+  const buildSortedGrid = (): Handsontable => {
+    const columnNames = [
+      'PRIMARY_KEY_FIELD',
+      'A_COL',
+      'B_COL',
+      'FORMULA_HARD_COL',
+      'PLAIN_TEXT_COL'
+    ]
+    const data = Array.from({ length: 10 }, (_, i) => ({
+      PRIMARY_KEY_FIELD: i + 1,
+      A_COL: i + 1,
+      B_COL: 10,
+      FORMULA_HARD_COL: '',
+      PLAIN_TEXT_COL: `note-${i + 1}`
+    }))
+    data.forEach((row: any, i) => {
+      row.FORMULA_HARD_COL = parseFormulaRule('=A_COL * B_COL', {
+        columnNames,
+        rowIndex: i,
+        userName: 'test',
+        origValue: undefined
+      })
+    })
+
+    const hot = new Handsontable(document.createElement('div'), {
+      data,
+      columns: columnNames.map((c) => ({ data: c })),
+      formulas: { engine: HyperFormula, licenseKey: 'gpl-v3' },
+      multiColumnSorting: true,
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    sortPlugin.sort({
+      column: columnNames.indexOf('PLAIN_TEXT_COL'),
+      sortOrder: 'desc'
+    })
+    hot.render()
+
+    return hot
+  }
+
+  const formulaHardColValues = (hot: Handsontable): any[] =>
+    Array.from({ length: 10 }, (_, r) =>
+      hot.getDataAtRowProp(r, 'FORMULA_HARD_COL')
+    )
+
+  it('demonstrates the failure mode: updateSettings while sorted corrupts formula cells', () => {
+    const hot = buildSortedGrid()
+
+    hot.updateSettings({}, false)
+    hot.render()
+
+    expect(formulaHardColValues(hot)).toContain('#REF!')
+
+    hot.destroy()
+  })
+
+  it('clearing the sort before updateSettings and restoring it after avoids the corruption', () => {
+    const hot = buildSortedGrid()
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    const sortConfigs = normalizeSortConfig(sortPlugin.getSortConfig())
+
+    sortPlugin.clearSort()
+    hot.updateSettings({}, false)
+    hot.render()
+    if (sortConfigs.length > 0) sortPlugin.sort(sortConfigs)
+    hot.render()
+
+    expect(formulaHardColValues(hot)).not.toContain('#REF!')
+
+    hot.destroy()
+  })
+})
+
+/**
+ * multiColumnSorting's own JSDoc: "every call of `sort` function set an
+ * entirely new sort order. Previous sort configs aren't preserved." A loop
+ * calling sort() once per captured config - the shape editor.component.ts
+ * used before this fix, in updateSettingsSortSafe/editTable/cancelEdit's
+ * restore step - therefore doesn't accumulate a multi-column sort: each
+ * call replaces the last, so only the FINAL config in the array survives.
+ * Restoring a captured multi-column sort must pass the whole array to a
+ * single sort() call instead.
+ */
+describe('restoring a multi-column sort must apply as a single call, not a loop', () => {
+  const columnNames = ['PRIMARY_KEY_FIELD', 'GROUP_COL', 'VALUE_COL']
+
+  const buildGrid = (): Handsontable => {
+    const data = [
+      { PRIMARY_KEY_FIELD: 1, GROUP_COL: 'B', VALUE_COL: 2 },
+      { PRIMARY_KEY_FIELD: 2, GROUP_COL: 'A', VALUE_COL: 1 },
+      { PRIMARY_KEY_FIELD: 3, GROUP_COL: 'A', VALUE_COL: 3 },
+      { PRIMARY_KEY_FIELD: 4, GROUP_COL: 'B', VALUE_COL: 4 }
+    ]
+    const hot = new Handsontable(document.createElement('div'), {
+      data,
+      columns: columnNames.map((c) => ({ data: c })),
+      multiColumnSorting: true,
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    return hot
+  }
+
+  // Sort by GROUP_COL asc, then VALUE_COL desc within each group.
+  const sortConfigs = [
+    { column: 1, sortOrder: 'asc' },
+    { column: 2, sortOrder: 'desc' }
+  ]
+
+  const visiblePkOrder = (hot: Handsontable): number[] =>
+    Array.from(
+      { length: 4 },
+      (_, r) => hot.getDataAtRowProp(r, 'PRIMARY_KEY_FIELD') as number
+    )
+
+  it('demonstrates the failure mode: looping sort() per column loses every column but the last', () => {
+    const hot = buildGrid()
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+
+    for (const sc of sortConfigs) sortPlugin.sort(sc)
+    hot.render()
+
+    // GROUP_COL asc got wiped out by the second call - only VALUE_COL desc
+    // (the last config) survived, sorting the whole table by value alone.
+    expect(visiblePkOrder(hot)).toEqual([4, 3, 1, 2])
+
+    hot.destroy()
+  })
+
+  it('applying the full config array in one call restores the actual multi-column order', () => {
+    const hot = buildGrid()
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+
+    sortPlugin.sort(sortConfigs)
+    hot.render()
+
+    // Group A first (value desc: 3, 1), then group B (value desc: 4, 2).
+    expect(visiblePkOrder(hot)).toEqual([3, 2, 4, 1])
+
+    hot.destroy()
+  })
+})
+
+/**
+ * getFormulaCellsToPreserveOnCancel's rowIndex parameter is established as
+ * PHYSICAL (it iterates 0..dataSource.length, matching dataSource's own
+ * order - see the file's own docblock). cancelEdit() passes that rowIndex
+ * straight into commentsPlugin.getCommentAtCell and hot.getDataAtRowProp,
+ * both of which - same as updateEditStatusForRow/syncOverwrittenCommentForCell
+ * above - document a VISUAL row. On a sorted grid this both checks and
+ * reads the wrong row's cell: a row that actually has a comment can go
+ * unflagged while a different row gets wrongly flagged, carrying that other
+ * row's live value onto its own prop - corrupting cancelEdit()'s restore
+ * rather than just skipping it. The fix is to clear the sort before this
+ * runs at all (cancelEdit() now does this - see its own comment), rather
+ * than translate physical->visual per call, since dataSourceUnchanged's
+ * restore right after also needs an unsorted physical dataSource to index
+ * into.
+ */
+describe('getFormulaCellsToPreserveOnCancel callbacks resolve the correct cell on a sorted grid', () => {
+  const formulaBaseCols = ['FORMULA_HARD_COL']
+
+  const setup = (): { hot: Handsontable } => {
+    const dataSource = [
+      { PRIMARY_KEY_FIELD: 1, SOME_CHAR: 'b', FORMULA_HARD_COL: 10 },
+      { PRIMARY_KEY_FIELD: 2, SOME_CHAR: 'a', FORMULA_HARD_COL: 20 }
+    ]
+    const hot = new Handsontable(document.createElement('div'), {
+      data: dataSource,
+      columns: [
+        { data: 'PRIMARY_KEY_FIELD' },
+        { data: 'SOME_CHAR' },
+        { data: 'FORMULA_HARD_COL' }
+      ],
+      multiColumnSorting: true,
+      comments: true,
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+
+    // Physical row 0 (PK=1) has the comment/live value that must be
+    // preserved. Ascending by SOME_CHAR flips it to visual row 1; physical
+    // row 1 (PK=2, no comment) becomes visual row 0.
+    const commentsPlugin: any = hot.getPlugin('comments')
+    commentsPlugin.setCommentAtCell(0, 2, 'Original value: 5')
+
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    sortPlugin.sort({ column: 1, sortOrder: 'asc' })
+    hot.render()
+
+    return { hot }
+  }
+
+  it('demonstrates the failure mode: physical rowIndex passed straight to visual-row APIs resolves the wrong cell', () => {
+    const { hot } = setup()
+    const commentsPlugin: any = hot.getPlugin('comments')
+
+    const toPreserve = getFormulaCellsToPreserveOnCancel(
+      2,
+      formulaBaseCols,
+      (rowIndex, baseCol) =>
+        !!commentsPlugin.getCommentAtCell(rowIndex, hot.propToCol(baseCol)),
+      (rowIndex, prop) => hot.getDataAtRowProp(rowIndex, prop)
+    )
+
+    // Physical row 0's comment lives at visual row 1 post-sort, but the
+    // buggy call checked physical rowIndex 0 straight as a visual row -
+    // landing on physical row 1's cell (no comment, value 20) instead.
+    // rowIndex 1 then wrongly reads AS visual row 1, which is physical row
+    // 0's actual commented cell (value 10) - so the wrong row (1, not 0)
+    // gets flagged, carrying the wrong row's value.
+    expect(toPreserve).toEqual([
+      { rowIndex: 1, prop: 'FORMULA_HARD_COL', value: 10 }
+    ])
+
+    hot.destroy()
+  })
+
+  it('resolves the correct cell once the sort is cleared before reading', () => {
+    const { hot } = setup()
+    const commentsPlugin: any = hot.getPlugin('comments')
+    const sortPlugin: any = hot.getPlugin('multiColumnSorting')
+    const sortConfigs = normalizeSortConfig(sortPlugin.getSortConfig())
+
+    sortPlugin.clearSort()
+
+    const toPreserve = getFormulaCellsToPreserveOnCancel(
+      2,
+      formulaBaseCols,
+      (rowIndex, baseCol) =>
+        !!commentsPlugin.getCommentAtCell(rowIndex, hot.propToCol(baseCol)),
+      (rowIndex, prop) => hot.getDataAtRowProp(rowIndex, prop)
+    )
+
+    expect(toPreserve).toEqual([
+      { rowIndex: 0, prop: 'FORMULA_HARD_COL', value: 10 }
+    ])
+
+    for (const sc of sortConfigs) sortPlugin.sort(sc)
+    hot.destroy()
+  })
+})

client/src/app/licensing/licensing.component.html

@@ -5,6 +5,11 @@
     <div class="card-text">
       <ng-container *ngSwitchCase="'key'">
         <p class="key-error" *ngIf="!keyError">
+          <clr-icon
+            class="is-error"
+            shape="exclamation-circle"
+            size="26"
+          ></clr-icon>
           Licence key is invalid. We can't provide you more details at the
           moment
         </p>
@@ -20,6 +25,11 @@
 
       <ng-container *ngSwitchCase="'limit'">
         <p class="key-error">
+          <clr-icon
+            class="is-error"
+            shape="exclamation-circle"
+            size="26"
+          ></clr-icon>
           The registered number of users reached the limit specified for your
           licence. Please contact
           <contact-link classes="color-green" />
@@ -34,9 +44,9 @@
         </p>
       </ng-container>
 
-      <p><strong>Protocol:</strong> {{ protocol }}</p>
+      <p class="m-0 mt-10"><strong>Protocol:</strong> {{ protocol }}</p>
 
-      <p>
+      <p class="m-0">
         <strong>SYSSITE:</strong>
         <span
           *ngFor="let id of syssite.value; let i = index"
@@ -59,11 +69,6 @@
         </a>
       </p>
 
-      <p *ngIf="licenseKeyData && userCountLimitation" class="m-0">
-        <strong>Allowed users:</strong>
-        {{ licenseKeyData.users_allowed }}
-      </p>
-
       <clr-tabs>
         <clr-tab>
           <button clrTabLink>Upload licence</button>
@@ -102,56 +107,102 @@
         <clr-tab>
           <button clrTabLink>Paste licence</button>
           <clr-tab-content>
-            <form class="clr-form license-key-form">
-              <p>Licence key:</p>
-              <div class="clr-control-container">
-                <textarea
-                  [(ngModel)]="licenceKeyValue"
-                  (mouseleave)="trimKeys()"
-                  name="license-key-area"
-                  placeholder="Paste licence key here"
-                  class="clr-textarea"
-                ></textarea>
-              </div>
-            </form>
+            <ng-container *ngIf="keyFormat === 'combined'">
+              <form class="clr-form combined-key-form">
+                <p>Licence key:</p>
+                <div class="clr-control-container">
+                  <textarea
+                    [(ngModel)]="combinedKeyValue"
+                    (mouseleave)="onCombinedKeyInput()"
+                    name="combined-key-area"
+                    placeholder="Paste licence key here"
+                    class="clr-textarea"
+                  ></textarea>
+                </div>
+              </form>
 
-            <form class="clr-form activation-key-form">
-              <p>Activation key:</p>
-              <div class="clr-control-container">
-                <textarea
-                  [(ngModel)]="activationKeyValue"
-                  (mouseleave)="trimKeys()"
-                  name="activation-key-area"
-                  placeholder="Paste activation key here"
-                  class="clr-textarea"
-                ></textarea>
-              </div>
-            </form>
+              <button
+                type="button"
+                class="btn btn-sm btn-link p-0"
+                (click)="toggleKeyFormat()"
+              >
+                Paste as two separate keys instead
+              </button>
+            </ng-container>
+
+            <ng-container *ngIf="keyFormat === 'legacy'">
+              <form class="clr-form license-key-form">
+                <p>Licence key:</p>
+                <div class="clr-control-container">
+                  <textarea
+                    [(ngModel)]="licenceKeyValue"
+                    (mouseleave)="trimKeys()"
+                    name="license-key-area"
+                    placeholder="Paste licence key here"
+                    class="clr-textarea"
+                  ></textarea>
+                </div>
+              </form>
+
+              <form class="clr-form activation-key-form">
+                <p>Activation key:</p>
+                <div class="clr-control-container">
+                  <textarea
+                    [(ngModel)]="activationKeyValue"
+                    (mouseleave)="trimKeys()"
+                    name="activation-key-area"
+                    placeholder="Paste activation key here"
+                    class="clr-textarea"
+                  ></textarea>
+                </div>
+              </form>
+
+              <button
+                type="button"
+                class="btn btn-sm btn-link p-0"
+                (click)="toggleKeyFormat()"
+              >
+                Paste as a single key instead
+              </button>
+            </ng-container>
           </clr-tab-content>
         </clr-tab>
       </clr-tabs>
 
-      <p
-        class="key-error protocol-mismatch-warning"
-        *ngIf="protocolMismatch === 'requiresHttps'"
-      >
-        This key was generated for a secure (HTTPS) connection, but
-        DataController is currently running over HTTP - it will not activate
-        here. Access DataController via HTTPS, or contact
-        <contact-link classes="color-green" />
-        for an HTTP-compatible key.
-      </p>
+      <!--
+        licenseKeyData starts as whatever key is currently active (set once
+        in ngOnInit), then live-updates to preview a pasted-but-not-yet-
+        applied key's own details instead - see refreshKeyPreview().
+      -->
+      <div *ngIf="licenseKeyData" class="key-details">
+        <h6 cds-text="subsection" class="mt-15 mb-10">Key Details</h6>
 
-      <p
-        class="key-error protocol-mismatch-warning"
-        *ngIf="protocolMismatch === 'requiresHttp'"
-      >
-        This key was generated for an insecure (HTTP) connection, but this page
-        is running in a secure browsing context (HTTPS, or localhost) - it will
-        not activate here. Contact
-        <contact-link classes="color-green" />
-        for an HTTPS-compatible key.
-      </p>
+        <p class="m-0">
+          <strong>Valid until:</strong>
+          {{ licenseKeyData.valid_until }}
+          <span *ngIf="licenseKeyData.demo">(demo/free tier key)</span>
+        </p>
+
+        <p class="m-0">
+          <strong>Allowed users:</strong>
+          {{ licenseKeyData.users_allowed }}
+        </p>
+
+        <p class="m-0">
+          <strong>Site ID(s) in this key:</strong>
+          {{
+            (licenseKeyData.site_id_multiple?.length
+              ? licenseKeyData.site_id_multiple
+              : [licenseKeyData.site_id]
+            ).join(', ')
+          }}
+        </p>
+
+        <p *ngIf="enabledFeatures.length" class="m-0">
+          <strong>Enabled features:</strong>
+          {{ enabledFeatures.join(', ') }}
+        </p>
+      </div>
     </div>
 
     <div class="card-footer d-flex clr-align-items-center">
@@ -172,6 +223,38 @@
         Continue with free tier
       </button>
     </div>
+
+    <p
+      class="key-error protocol-mismatch-warning"
+      *ngIf="protocolMismatch === 'requiresHttps'"
+    >
+      <clr-icon
+        class="color-orange"
+        shape="exclamation-triangle"
+        size="26"
+      ></clr-icon>
+      This key was generated for a secure (HTTPS) connection, but DataController
+      is currently running over HTTP - it will not activate here. Access
+      DataController via HTTPS, or contact
+      <contact-link classes="color-green" />
+      for an HTTP-compatible key.
+    </p>
+
+    <p
+      class="key-error protocol-mismatch-warning"
+      *ngIf="protocolMismatch === 'requiresHttp'"
+    >
+      <clr-icon
+        class="color-orange"
+        shape="exclamation-triangle"
+        size="26"
+      ></clr-icon>
+      This key was generated for an insecure (HTTP) connection, but this page is
+      running in a secure browsing context (HTTPS, or localhost) - it will not
+      activate here. Contact
+      <contact-link classes="color-green" />
+      for an HTTPS-compatible key.
+    </p>
   </div>
 </div>
 

client/src/app/licensing/licensing.component.scss

@@ -0,0 +1,7 @@
+.key-error clr-icon {
+  // clr-icon defaults to vertical-align: middle, which sits it noticeably
+  // above the text baseline next to it - align its bottom edge with the
+  // text's instead.
+  vertical-align: bottom;
+  margin-right: 10px;
+}

client/src/app/licensing/licensing.component.ts

@@ -1,4 +1,5 @@
 import { Component, OnInit, ViewEncapsulation } from '@angular/core'
+import { DomSanitizer, SafeHtml } from '@angular/platform-browser'
 import { ActivatedRoute, Router } from '@angular/router'
 import { AppService, LicenceService, SasService } from '../services'
 import { LicenseKeyData } from '../models/LicenseKeyData'
@@ -7,6 +8,11 @@ import {
   detectLicenceKeyProtocolMismatch,
   LicenceKeyProtocolMismatch
 } from './utils/detectLicenceKeyProtocolMismatch'
+import {
+  isCombinedLicenceKey,
+  splitCombinedLicenceKey
+} from './utils/combinedLicenceKey'
+import { getEnabledFeatureLabels } from './utils/getEnabledFeatureLabels'
 
 enum LicenseActions {
   key = 'key',
@@ -25,12 +31,19 @@ enum LicenseActions {
 export class LicensingComponent implements OnInit {
   public action: LicenseActions | null = null
 
-  public licenseErrors: { [key: string]: string } = {
-    missing: `Licence key is missing - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`,
-    expired: `Licence key is expired - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`,
-    invalid: `Licence key is invalid - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`,
-    missmatch: `Your SYSSITE (below) is not found in the licence key - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`
-  }
+  // Each message carries its own icon markup, rather than relying on a
+  // sibling element in the template - these render via [innerHTML], which
+  // replaces all of the host element's children, so a real <clr-icon>
+  // placed in the template around the binding would never survive.
+  private readonly errorIcon = `<clr-icon class="is-error" shape="exclamation-circle" size="26"></clr-icon>`
+
+  // SafeHtml, built via bypassSecurityTrustHtml in the constructor (needs
+  // DomSanitizer, not yet available in a field initializer) - Angular's
+  // default [innerHTML] sanitizer strips unrecognised tags like <clr-icon>
+  // (a custom element, not on its standard-tags allowlist), even though it
+  // leaves the plain <a> in these same strings alone. Safe to bypass since
+  // this content is entirely hardcoded here, never derived from user input.
+  public licenseErrors: { [key: string]: SafeHtml }
 
   public keyError: string | undefined
   public errorDetails: string | undefined
@@ -38,6 +51,12 @@ export class LicensingComponent implements OnInit {
   public licenceKeyValue: string = ''
   public activationKeyValue: string = ''
 
+  // Purely which INPUT LAYOUT is shown (one field vs two) - not a strict
+  // parsing gate. Recognising a combined key doesn't depend on this: see
+  // maybeSplitCombinedKey(), called from both layouts' own input handlers.
+  public keyFormat: 'combined' | 'legacy' = 'combined'
+  public combinedKeyValue: string = ''
+
   public applyingKeys: boolean = false
   public protocol: string =
     location.protocol === 'https:'
@@ -48,7 +67,6 @@ export class LicensingComponent implements OnInit {
   public currentLicenceKey = this.licenceService.licenceKey
   public currentActivationKey = this.licenceService.activationKey
   public isAppFreeTier = this.licenceService.isAppFreeTier
-  public userCountLimitation = this.licenceService.userCountLimitation
 
   public licenseKeyData: LicenseKeyData | null = null
 
@@ -64,8 +82,24 @@ export class LicensingComponent implements OnInit {
     private router: Router,
     private licenceService: LicenceService,
     private sasService: SasService,
-    private appService: AppService
-  ) {}
+    private appService: AppService,
+    private sanitizer: DomSanitizer
+  ) {
+    this.licenseErrors = {
+      missing: this.sanitizer.bypassSecurityTrustHtml(
+        `${this.errorIcon} Licence key is missing - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`
+      ),
+      expired: this.sanitizer.bypassSecurityTrustHtml(
+        `${this.errorIcon} Licence key is expired - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`
+      ),
+      invalid: this.sanitizer.bypassSecurityTrustHtml(
+        `${this.errorIcon} Licence key is invalid - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`
+      ),
+      missmatch: this.sanitizer.bypassSecurityTrustHtml(
+        `${this.errorIcon} Your SYSSITE (below) is not found in the licence key - please contact <a class="color-green" href="mailto: support@datacontroller.io">support@datacontroller.io</a> and enter valid keys below.`
+      )
+    }
+  }
 
   ngOnInit(): void {
     this.licenceKeyValue = this.currentLicenceKey || ''
@@ -93,9 +127,73 @@ export class LicensingComponent implements OnInit {
     this.licenseKeyData = this.licenceService.getLicenseKeyData()
   }
 
-  public trimKeys() {
+  public async trimKeys() {
     this.licenceKeyValue = this.licenceKeyValue.trim()
     this.activationKeyValue = this.activationKeyValue.trim()
+
+    // Auto-detect regardless of keyFormat - a combined key pasted into the
+    // legacy "Licence key" field on its own still gets recognised and
+    // splits into both fields, rather than requiring the toggle to match.
+    await this.maybeSplitCombinedKey(this.licenceKeyValue)
+    await this.refreshKeyPreview()
+  }
+
+  public async onCombinedKeyInput() {
+    this.combinedKeyValue = this.combinedKeyValue.trim()
+
+    // The combined field is the sole source of truth for
+    // licenceKeyValue/activationKeyValue in this layout - if it's been
+    // cleared (or isn't a recognisable combined key), there's no key data
+    // to preview, so clear rather than leave a stale split from an earlier
+    // paste (which would otherwise keep decrypting to that old key).
+    const wasSplit = await this.maybeSplitCombinedKey(this.combinedKeyValue)
+    if (!wasSplit) {
+      this.licenceKeyValue = ''
+      this.activationKeyValue = ''
+    }
+
+    await this.refreshKeyPreview()
+  }
+
+  // Decrypts whatever's currently in licenceKeyValue/activationKeyValue
+  // purely to preview its details (valid_until, users_allowed, ...) before
+  // the user ever clicks Apply - decryptLicenseKey() has no side effects,
+  // so this is safe to call speculatively on every input change. Clears
+  // rather than leaves stale data on failure/incomplete input, since
+  // showing a previous key's details next to a since-changed paste would
+  // be actively misleading, not just uninformative.
+  private async refreshKeyPreview(): Promise<void> {
+    if (!this.licenceKeyValue || !this.activationKeyValue) {
+      this.licenseKeyData = null
+      return
+    }
+
+    try {
+      this.licenseKeyData = await this.licenceService.decryptLicenseKey(
+        this.licenceKeyValue,
+        this.activationKeyValue
+      )
+    } catch {
+      this.licenseKeyData = null
+    }
+  }
+
+  public toggleKeyFormat() {
+    this.keyFormat = this.keyFormat === 'combined' ? 'legacy' : 'combined'
+  }
+
+  // Shared by both input layouts' own handlers above - populates
+  // licenceKeyValue/activationKeyValue when value is recognisably a
+  // combined key, regardless of which field it was typed/pasted into.
+  private async maybeSplitCombinedKey(value: string): Promise<boolean> {
+    if (!isCombinedLicenceKey(value)) return false
+
+    const split = await splitCombinedLicenceKey(value)
+    if (!split) return false
+
+    this.licenceKeyValue = split.licenceKey
+    this.activationKeyValue = split.activationKey
+    return true
   }
 
   public copySyssite(copyIconRef: any, copyTooltip: any, syssite: string[]) {
@@ -154,7 +252,7 @@ export class LicensingComponent implements OnInit {
 
     const reader = new FileReader()
 
-    reader.onload = (evt) => {
+    reader.onload = async (evt) => {
       this.licenceFileError = 'Error reading file.'
 
       if (!evt || !evt.target) return
@@ -164,9 +262,20 @@ export class LicensingComponent implements OnInit {
 
       this.licenceFileLoading = false
       this.licenceFileError = undefined
-      const fileArr = evt.target.result.toString().split('\n')
-      this.activationKeyValue = fileArr[1]
-      this.licenceKeyValue = fileArr[0]
+
+      const fileArr = evt.target.result.toString().trim().split('\n')
+
+      // A combined-key file is a single line - splits into both fields.
+      // Anything else keeps the existing 2-line legacy file behaviour.
+      if (
+        fileArr.length !== 1 ||
+        !(await this.maybeSplitCombinedKey(fileArr[0]))
+      ) {
+        this.activationKeyValue = fileArr[1]
+        this.licenceKeyValue = fileArr[0]
+      }
+
+      await this.refreshKeyPreview()
     }
 
     reader.readAsText(file)
@@ -205,4 +314,8 @@ export class LicensingComponent implements OnInit {
       !!(window.crypto && window.crypto.subtle)
     )
   }
+
+  get enabledFeatures(): string[] {
+    return getEnabledFeatureLabels(this.licenseKeyData?.features)
+  }
 }

client/src/app/licensing/utils/combinedLicenceKey.spec.ts

@@ -0,0 +1,90 @@
+import * as base64Converter from 'base64-arraybuffer'
+import {
+  COMBINED_KEY_PREFIX,
+  isCombinedLicenceKey,
+  splitCombinedLicenceKey
+} from './combinedLicenceKey'
+
+// Mirrors dckey's own encodeCombinedKey() (main.js) exactly, so these tests
+// build fixtures the same way a real generated key would be produced,
+// rather than relying on a hardcoded string that could go stale if either
+// side's algorithm ever changes.
+const gzipCompress = async (bytes: Uint8Array): Promise<ArrayBuffer> => {
+  const compressionStream = new CompressionStream('gzip')
+  const writer = compressionStream.writable.getWriter()
+  writer.write(new Uint8Array(bytes))
+  writer.close()
+
+  return new Response(compressionStream.readable).arrayBuffer()
+}
+
+const encodeCombinedLicenceKeyFixture = async (
+  licenceKey: string,
+  activationKey: string
+): Promise<string> => {
+  const payloadBytes = new TextEncoder().encode(
+    `${licenceKey} ${activationKey}`
+  )
+  const compressedBytes = await gzipCompress(payloadBytes)
+
+  return COMBINED_KEY_PREFIX + base64Converter.encode(compressedBytes)
+}
+
+describe('isCombinedLicenceKey', () => {
+  it('is true for prefixed text', () => {
+    expect(isCombinedLicenceKey('DCKEY1:abc123')).toBeTrue()
+  })
+
+  it('is false for a plain legacy key', () => {
+    expect(isCombinedLicenceKey('some-legacy-licence-key')).toBeFalse()
+  })
+
+  it('is false for an empty string', () => {
+    expect(isCombinedLicenceKey('')).toBeFalse()
+  })
+
+  it('is true even with leading/trailing whitespace around the prefix', () => {
+    expect(isCombinedLicenceKey('  DCKEY1:abc123  ')).toBeTrue()
+  })
+})
+
+describe('splitCombinedLicenceKey', () => {
+  it('round-trips a fixture built the same way dckey encodes one', async () => {
+    const combined = await encodeCombinedLicenceKeyFixture(
+      'licence-key-value',
+      'activation-key-value'
+    )
+
+    expect(await splitCombinedLicenceKey(combined)).toEqual({
+      licenceKey: 'licence-key-value',
+      activationKey: 'activation-key-value'
+    })
+  })
+
+  it('returns null for text without the prefix', async () => {
+    expect(
+      await splitCombinedLicenceKey('a-plain-legacy-licence-key')
+    ).toBeNull()
+  })
+
+  it('rejects for prefixed text that is not valid compressed data', async () => {
+    await expectAsync(
+      splitCombinedLicenceKey('DCKEY1:not-valid-base64-gzip-data')
+    ).toBeRejected()
+  })
+
+  it('splits on the first space only, not every space', async () => {
+    // Not a real-world value today (licence/activation keys are base64,
+    // which never contains a space) - guards the split logic itself rather
+    // than that assumption.
+    const combined = await encodeCombinedLicenceKeyFixture(
+      'licence key with spaces',
+      'activation-key-value'
+    )
+
+    expect(await splitCombinedLicenceKey(combined)).toEqual({
+      licenceKey: 'licence',
+      activationKey: 'key with spaces activation-key-value'
+    })
+  })
+})

client/src/app/licensing/utils/combinedLicenceKey.ts

@@ -0,0 +1,83 @@
+import * as base64Converter from 'base64-arraybuffer'
+
+/**
+ * Format produced by dckey's own encodeCombinedKey() (main.js):
+ * "DCKEY1:" + base64(gzip(licenceKey + " " + activationKey)). The prefix is
+ * plain ASCII, not itself compressed/encoded - base64's alphabet never
+ * contains ":", so it can never collide with the start of a legacy licence
+ * key or activation key, making format detection unambiguous.
+ */
+export const COMBINED_KEY_PREFIX = 'DCKEY1:'
+
+export const isCombinedLicenceKey = (text: string): boolean =>
+  text.trim().startsWith(COMBINED_KEY_PREFIX)
+
+const streamToArrayBuffer = async (
+  readable: ReadableStream<Uint8Array>
+): Promise<ArrayBuffer> => {
+  const reader = readable.getReader()
+  const chunks: Uint8Array[] = []
+  let totalLength = 0
+
+  while (true) {
+    const { done, value } = await reader.read()
+    if (done) break
+    chunks.push(value)
+    totalLength += value.length
+  }
+
+  const result = new Uint8Array(totalLength)
+  let offset = 0
+  for (const chunk of chunks) {
+    result.set(chunk, offset)
+    offset += chunk.length
+  }
+
+  return result.buffer
+}
+
+const gzipDecompress = async (bytes: ArrayBuffer): Promise<ArrayBuffer> => {
+  const decompressionStream = new DecompressionStream('gzip')
+  const writer = decompressionStream.writable.getWriter()
+
+  // Awaited (unlike a fire-and-forget write) so invalid gzip data rejects
+  // through this function's own returned promise, not as a separate
+  // unhandled rejection racing the readable side below.
+  const result = await Promise.all([
+    writer.write(new Uint8Array(bytes)).then(() => writer.close()),
+    streamToArrayBuffer(decompressionStream.readable)
+  ])
+
+  return result[1]
+}
+
+/**
+ * Splits a combined licence key back into its two parts. Returns null (not
+ * a rejected promise) when text isn't a combined key at all, so callers can
+ * use it as a "try this, then fall back to the legacy two-field input" check
+ * without a try/catch for that common case - a prefixed-but-corrupted
+ * string still rejects, since that's a real error, not a format mismatch.
+ */
+export const splitCombinedLicenceKey = async (
+  text: string
+): Promise<{ licenceKey: string; activationKey: string } | null> => {
+  const trimmed = text.trim()
+
+  if (!isCombinedLicenceKey(trimmed)) return null
+
+  const compressedBytes = base64Converter.decode(
+    trimmed.slice(COMBINED_KEY_PREFIX.length)
+  )
+  const decompressedBytes = await gzipDecompress(compressedBytes)
+  const payload = new TextDecoder().decode(decompressedBytes)
+  const separatorIndex = payload.indexOf(' ')
+
+  if (separatorIndex === -1) {
+    throw new Error('Invalid combined licence key: missing separator')
+  }
+
+  return {
+    licenceKey: payload.slice(0, separatorIndex),
+    activationKey: payload.slice(separatorIndex + 1)
+  }
+}

client/src/app/licensing/utils/detectLicenceKeyProtocolMismatch.ts

@@ -11,6 +11,10 @@ export type LicenceKeyProtocolMismatch = 'requiresHttps' | 'requiresHttp' | null
  * alone tells us which format a pasted key is, without needing to attempt
  * decryption first.
  *
+ * This is a structural assumption about key-generation logic
+ * (a separate repo) - if that ever changes to generate distinct values for
+ * HTTP-format keys too, this detection would silently misclassify keys.
+ *
  * isSecureContext must reflect whether this browsing context can actually
  * decrypt a secure-connection key (mirrors the check licence.service.ts's
  * own decryptLicenseKey() makes) - not simply `location.protocol ===

client/src/app/licensing/utils/getEnabledFeatureLabels.spec.ts

@@ -0,0 +1,73 @@
+import { getEnabledFeatureLabels } from './getEnabledFeatureLabels'
+
+describe('getEnabledFeatureLabels', () => {
+  it('returns an empty list when features is undefined', () => {
+    expect(getEnabledFeatureLabels(undefined)).toEqual([])
+  })
+
+  it('lists only the enabled toggle features from a new-format object', () => {
+    expect(
+      getEnabledFeatureLabels({
+        vra: null,
+        era: null,
+        sra: null,
+        hra: null,
+        srl: null,
+        till: null,
+        vb: true,
+        vbl: null,
+        ldl: null,
+        fu: false,
+        er: true,
+        ar: false
+      })
+    ).toEqual(['Viewbox', 'Edit Record'])
+  })
+
+  it('lists all four when every toggle is enabled', () => {
+    expect(
+      getEnabledFeatureLabels({
+        vra: null,
+        era: null,
+        sra: null,
+        hra: null,
+        srl: null,
+        till: null,
+        vb: true,
+        vbl: null,
+        ldl: null,
+        fu: true,
+        er: true,
+        ar: true
+      })
+    ).toEqual(['Viewbox', 'File Upload', 'Edit Record', 'Add Record'])
+  })
+
+  it('returns an empty list when every toggle is disabled', () => {
+    expect(
+      getEnabledFeatureLabels({
+        vra: null,
+        era: null,
+        sra: null,
+        hra: null,
+        srl: null,
+        till: null,
+        vb: false,
+        vbl: null,
+        ldl: null,
+        fu: false,
+        er: false,
+        ar: false
+      })
+    ).toEqual([])
+  })
+
+  it('decodes a legacy positional string the same way', () => {
+    // viewbox=1 (position 6), fileUpload=0 (position 9), editRecord=1
+    // (position 10), addRecord=0 (position 11) - see LicenceFeaturesMap.
+    expect(getEnabledFeatureLabels('-,-,-,-,-,-,1,-,-,0,1,0')).toEqual([
+      'Viewbox',
+      'Edit Record'
+    ])
+  })
+})

client/src/app/licensing/utils/getEnabledFeatureLabels.ts

@@ -0,0 +1,24 @@
+import {
+  decodeLicenceFeatures,
+  LicenceFeaturesObject
+} from '../../services/utils/decodeLicenceFeatures'
+import { LicenceState } from '../../models/LicenceState'
+
+const TOGGLE_FEATURE_LABELS: { key: keyof LicenceState; label: string }[] = [
+  { key: 'viewbox', label: 'Viewbox' },
+  { key: 'fileUpload', label: 'File Upload' },
+  { key: 'editRecord', label: 'Edit Record' },
+  { key: 'addRecord', label: 'Add Record' }
+]
+
+export const getEnabledFeatureLabels = (
+  features: string | LicenceFeaturesObject | undefined
+): string[] => {
+  if (!features) return []
+
+  const decoded = decodeLicenceFeatures(features)
+
+  return TOGGLE_FEATURE_LABELS.filter(({ key }) => decoded[key]).map(
+    ({ label }) => label
+  )
+}

client/src/app/models/InfoModal.ts

@@ -10,5 +10,6 @@ export class AbortDetails {
   SYSWARNINGTEXT?: string
   SYSERRORTEXT?: string
   MAC?: string
+  _PROGRAM?: string
   LOG?: string
 }

client/src/app/models/LicenceState.ts

@@ -15,6 +15,11 @@ export interface LicenceState {
 }
 
 /**
+ * Legacy-format decode table only - newly-issued keys use a named
+ * LicenceFeaturesObject instead (see decodeLicenceFeatures.ts). Kept
+ * unchanged and permanently, since every already-issued key still relies
+ * on these ordinal positions to decode.
+ *
  * '-' means unset
  * '0' disabled
  * '1' enabled

client/src/app/models/LicenseKeyData.ts

@@ -1,3 +1,5 @@
+import { LicenceFeaturesObject } from '../services/utils/decodeLicenceFeatures'
+
 export interface LicenseKeyData {
   valid_until: string
   users_allowed: number
@@ -5,5 +7,5 @@ export interface LicenseKeyData {
   site_id_multiple: string[]
   demo: boolean
   hot_license_key: string | undefined
-  features?: string
+  features?: string | LicenceFeaturesObject
 }

client/src/app/review/approve-details/approve-details.component.ts

@@ -217,7 +217,9 @@ export class ApproveDetailsComponent implements AfterViewInit, OnDestroy {
   }
 
   public calcDiff() {
-    if (!this.response) return
+    // params-only responses (e.g. already reviewed submissions) carry no diff
+    // tables - nothing to calculate, the details header still renders
+    if (!this.response || !this.response.cols) return
 
     let news = this.response.new
     let updates = this.response.updates
@@ -355,26 +357,6 @@ export class ApproveDetailsComponent implements AfterViewInit, OnDestroy {
             this.submitArr.push(item)
           }
         }
-
-        let diffs = {
-          ACTION: 'SHOW_DIFFS',
-          TABLE: this.tableId,
-          DIFFTIME: new Date().toUTCString()
-        }
-        // show diffs and changes info in a same call
-        this.sasStoreService
-          .showDiffs(diffs, 'SASControlTable', 'auditors/postdata')
-          .then((res: AuditorsPostdataSASResponse) => {
-            let param = res.params[0]
-            this.params = param
-            this.response = res
-            this.calcDiff()
-            this.callChangesInfo(this.tableId)
-          })
-          .catch((err: any) => err)
-          .finally(() => {
-            this.loadingTable = true
-          })
       }
     )
     if (typeof this.router.snapshot.params['tableId'] === 'undefined') {

client/src/app/review/submitter/submitter.component.ts

@@ -68,6 +68,9 @@ export class SubmitterComponent implements OnInit, AfterViewInit {
   }
 
   public goToDetails(table_id: any) {
+    // the details URL re-creates this component - carry the queue payload
+    // over so it is not fetched again on mount
+    this.sasStoreService.handoffSubmits(this.submitData)
     this.router.navigateByUrl('/review/submitted/' + table_id)
   }
 
@@ -83,14 +86,19 @@ export class SubmitterComponent implements OnInit, AfterViewInit {
 
     this.itemsNum = 10
     try {
-      let res = await this.sasStoreService.getSubmitts()
+      // navigating from the submit list to a details URL re-creates this
+      // component - take the queue payload carried over by that navigation
+      // instead of fetching it again
+      let fromsas: any = this.sasStoreService.takeSubmitsHandoff()
 
-      this.remained = res.fromsas.length
+      if (!fromsas) {
+        fromsas = (await this.sasStoreService.getSubmitts()).fromsas
+      }
+
+      this.remained = fromsas.length
       if (this.remained > 0) {
-        this.submitter = res.fromsas[0].SUBMITTED_BY_NM
-        let submitterList: SubmitterData[] = res.fromsas.map(function (
-          item: any
-        ) {
+        this.submitter = fromsas[0].SUBMITTED_BY_NM
+        let submitterList: SubmitterData[] = fromsas.map(function (item: any) {
           return {
             tableId: item.TABLE_ID,
             base: item.BASE_TABLE,
@@ -100,7 +108,7 @@ export class SubmitterComponent implements OnInit, AfterViewInit {
           }
         })
         this.submitterList = submitterList
-        this.submitData = res.fromsas
+        this.submitData = fromsas
 
         // Details page
         if (typeof tableIdParam !== 'undefined') {

client/src/app/services/app.service.spec.ts

@@ -38,6 +38,9 @@ const buildDeps = () => {
   const appStoreService: any = {
     getDcAdapterSettings: () => undefined
   }
+  const startupCheckService: any = {
+    updateStep: jasmine.createSpy('updateStep')
+  }
 
   return {
     licenceService,
@@ -46,7 +49,8 @@ const buildDeps = () => {
     loggerService,
     appSettingsService,
     router,
-    appStoreService
+    appStoreService,
+    startupCheckService
   }
 }
 
@@ -58,7 +62,8 @@ const buildAppService = (deps: ReturnType<typeof buildDeps>) =>
     deps.loggerService,
     deps.appSettingsService,
     deps.router,
-    deps.appStoreService
+    deps.appStoreService,
+    deps.startupCheckService
   )
 
 // Minimal valid startupservice payload — just enough to pass startUpData()'s

client/src/app/services/app.service.ts

@@ -11,6 +11,7 @@ import { AppSettingsService } from './app-settings.service'
 import { AppThemes } from '../models/AppSettings'
 import { RequestWrapperResponse } from '../models/request-wrapper/RequestWrapperResponse'
 import { AppStoreService } from './app-store.service'
+import { StartupCheckService } from './startup-check.service'
 import { retryOnce, RetryOnceOptions } from '../shared/utils/retry-once'
 import { getMalformedAdapterResponseMessage } from '../shared/utils/get-malformed-adapter-response-message'
 
@@ -29,7 +30,8 @@ export class AppService {
     private loggerService: LoggerService,
     private appSettingsService: AppSettingsService,
     private router: Router,
-    private appStoreService: AppStoreService
+    private appStoreService: AppStoreService,
+    private startupCheckService: StartupCheckService
   ) {
     this.subscribe()
 
@@ -100,6 +102,11 @@ export class AppService {
         )
         if (malformedMessage) {
           startupServiceError = true
+          console.error(
+            'startUpData: startupservice returned malformed response:',
+            malformedMessage
+          )
+          this.startupCheckService.updateStep(2, 'error', 'Malformed response')
           this.eventService.showInfoModal('Error', malformedMessage)
           this.licenceService.isAppActivated.next(false)
 
@@ -121,6 +128,15 @@ export class AppService {
 
         if (missingProps.length > 0) {
           startupServiceError = true
+          console.error(
+            'startUpData: startupservice missing properties:',
+            missingProps.join(', ')
+          )
+          this.startupCheckService.updateStep(
+            2,
+            'error',
+            `Missing: ${missingProps.join(', ')}`
+          )
           this.eventService.showInfoModal(
             'Error',
             `${missingProps.join(', ')} are not present in the startupservice`
@@ -144,6 +160,9 @@ export class AppService {
           SYSHOSTINFOLONG: res.adapterResponse.SYSHOSTINFOLONG,
           SYSENCODING: res.adapterResponse.SYSENCODING,
           AUTOEXEC: res.adapterResponse.AUTOEXEC,
+          TIMEZONE: res.adapterResponse.globvars[0].TIMEZONE,
+          SYSTIMEZONEIDENT: res.adapterResponse.globvars[0].SYSTIMEZONEIDENT,
+          SYSTIMEZONEOFFSET: res.adapterResponse.globvars[0].SYSTIMEZONEOFFSET,
           ISADMIN: res.adapterResponse.globvars[0].ISADMIN,
           DC_ADMIN_GROUP: res.adapterResponse.globvars[0].DC_ADMIN_GROUP,
           APP_LOC: dcAdapterSettings?.appLoc
@@ -198,6 +217,8 @@ export class AppService {
       })
       .catch((err: any) => {
         startupServiceError = true
+        console.error('startUpData: startupservice request failed:', err)
+        this.startupCheckService.updateStep(2, 'error', 'Request failed')
         this.eventService.showInfoModal(
           'Error',
           'There is an issue with startupservice response'

client/src/app/services/licence.service.ts

@@ -10,7 +10,8 @@ import { Globvar } from '../models/sas/public-startupservice.model'
 import { freeTierConfig } from '../free-tier.config'
 import { AppStoreService } from './app-store.service'
 import { HelperService } from './helper.service'
-import { LicenceFeaturesMap, LicenceState } from '../models/LicenceState'
+import { LicenceState } from '../models/LicenceState'
+import { decodeLicenceFeatures as decodeFeatures } from './utils/decodeLicenceFeatures'
 import { LoggerService } from './logger.service'
 import { EventService } from './event.service'
 
@@ -117,6 +118,13 @@ export class LicenceService {
 
     let variables = globvars[0]
 
+    // Trim whitespace — the SAS makedata service initialises licence keys
+    // as a single space, which is truthy in JS but not a valid key.
+    if (variables.LICENCE_KEY)
+      variables.LICENCE_KEY = variables.LICENCE_KEY.trim()
+    if (variables.ACTIVATION_KEY)
+      variables.ACTIVATION_KEY = variables.ACTIVATION_KEY.trim()
+
     if (
       variables.LICENCE_KEY === undefined ||
       variables.ACTIVATION_KEY === undefined ||
@@ -255,9 +263,9 @@ export class LicenceService {
   }
 
   /**
-   * Decode and set features that are encoded in the key
-   * featureValue - used as number/Infinity for limit set
-   * featureToggle - used as boolean, based on 1 or 0 encoded in the key
+   * Decode and set features that are encoded in the key - accepts either
+   * an already-issued key's legacy positional string or a newly-issued
+   * key's named object, see decodeLicenceFeatures for the format contract.
    * @param licenseData from the licence key
    */
   private decodeLicenceFeatures(licenseData: LicenseKeyData) {
@@ -270,72 +278,21 @@ export class LicenceService {
       }
     }
 
-    const featuresMap = licenseData.features.split(',')
     this._licenceState = {
       ...this._licenceState,
-      viewer_rows_allowed: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.viewer_rows_allowed]
-      ),
-      editor_rows_allowed: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.editor_rows_allowed]
-      ),
-      stage_rows_allowed: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.stage_rows_allowed]
-      ),
-      history_rows_allowed: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.history_rows_allowed]
-      ),
-      submit_rows_limit: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.submit_rows_limit]
-      ),
-      tables_in_library_limit: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.tables_in_library_limit]
-      ),
-      viewbox_limit: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.viewbox_limit]
-      ),
-      lineage_daily_limit: this.parseFeatureValue(
-        featuresMap[LicenceFeaturesMap.lineage_daily_limit]
-      ),
-      viewbox: this.parseFeatureToggle(featuresMap[LicenceFeaturesMap.viewbox]),
-      fileUpload: this.parseFeatureToggle(
-        featuresMap[LicenceFeaturesMap.fileUpload]
-      ),
-      editRecord: this.parseFeatureToggle(
-        featuresMap[LicenceFeaturesMap.editRecord]
-      ),
-      addRecord: this.parseFeatureToggle(
-        featuresMap[LicenceFeaturesMap.addRecord]
-      )
+      ...decodeFeatures(licenseData.features)
     }
 
     this.loggerService.log('Licence state:', this._licenceState)
   }
 
-  /**
-   * Converts licence key feature code value to the number or Infinity
-   * Used for limiting rows, submits etc.
-   * @param codeBit from licence key encoded features. Every bit is separated by comma(,) Eg. 5,10,15
-   * @returns number
-   */
-  private parseFeatureValue(codeBit: string): number {
-    if (codeBit === '-') return Infinity
-    return parseInt(codeBit)
-  }
-
-  /**
-   * Converts licence key feature code value to the boolean. Depending on if it's 0 or 1
-   * @param codeBit from licence key encoded features. Every bit is separated by comma(,) Eg. 1,1,0
-   * @returns boolean to turn on or off the value
-   */
-  private parseFeatureToggle(codeBit: string): boolean {
-    return !!parseInt(codeBit)
-  }
-
   /**
    * If decryption fails, key will be marked as invalid and app not activated.
+   * Public (not just used internally by licensing()) since it's a pure
+   * decrypt with no side effects - also used to preview a pasted key's
+   * details before it's actually applied.
    */
-  private decryptLicenseKey(
+  public decryptLicenseKey(
     licenseKey: string,
     activationKey: string
   ): Promise<LicenseKeyData> {

client/src/app/services/sas-store.service.ts

@@ -35,6 +35,13 @@ export class SasStoreService {
   public setSubmit: Subject<any> = new Subject<any>()
   public setSubmitList: Subject<any> = new Subject<any>()
 
+  /**
+   * Submit queue payload carried across the navigation from the submit
+   * list to a details URL - the re-created list component renders it
+   * without fetching the queue again.
+   */
+  private submitsHandoff: Array<any> | null = null
+
   constructor(
     private sasService: SasService,
     private helperService: HelperService,
@@ -155,6 +162,24 @@ export class SasStoreService {
       .adapterResponse
   }
 
+  /**
+   * Carries the fetched submit queue over to the details URL navigation,
+   * so the re-created submit list component does not fetch it again.
+   */
+  public handoffSubmits(fromsas: Array<any> | undefined) {
+    this.submitsHandoff = fromsas || null
+  }
+
+  /**
+   * Returns the submit queue carried by the current navigation, if any,
+   * and clears it - it is only valid for the navigation that set it.
+   */
+  public takeSubmitsHandoff(): Array<any> | null {
+    const fromsas = this.submitsHandoff
+    this.submitsHandoff = null
+    return fromsas
+  }
+
   private libsPromise: Promise<any> | null = null
 
   /**

client/src/app/services/sas-viya.service.ts

@@ -150,9 +150,12 @@ export class SasViyaService {
     )
   }
 
-  getFolderMembers(folderId: string): Observable<ViyaApiFolderMembers> {
+  getFolderMembers(
+    folderId: string,
+    limit: number = 500
+  ): Observable<ViyaApiFolderMembers> {
     return this.get<ViyaApiFolderMembers>(
-      `${this.serverUrl}/folders/folders/${folderId}/members`,
+      `${this.serverUrl}/folders/folders/${folderId}/members?limit=${limit}`,
       {
         withCredentials: true
       }
@@ -168,6 +171,20 @@ export class SasViyaService {
     )
   }
 
+  /**
+   * @returns The groups the current user is a member of
+   */
+  getCurrentUserGroupMemberships(
+    limit: number = 10000
+  ): Observable<ViyaApiIdentities> {
+    return this.get<ViyaApiIdentities>(
+      `${this.serverUrl}/identities/users/@currentUser/memberships?limit=${limit}`,
+      {
+        withCredentials: true
+      }
+    )
+  }
+
   getCurrentUser(): Observable<ViyaApiCurrentUser> {
     return this.get<ViyaApiCurrentUser>(
       `${this.serverUrl}/identities/users/@currentUser`,

client/src/app/services/sas.service.ts

@@ -3,6 +3,7 @@ import { Injectable, EventEmitter } from '@angular/core'
 import SASjs, { UploadFile } from '@sasjs/adapter'
 import { BehaviorSubject } from 'rxjs'
 import { UserService } from '../shared/user.service'
+import { StartupCheckService } from './startup-check.service'
 
 import { Router } from '@angular/router'
 import { EventService } from './event.service'
@@ -43,6 +44,7 @@ export class SasService {
     private sasjsService: SasjsService,
     private sasViyaService: SasViyaService,
     private loggerService: LoggerService,
+    private startupCheckService: StartupCheckService,
     private router: Router
   ) {}
 
@@ -195,7 +197,8 @@ export class SasService {
                   {
                     SYSWARNINGTEXT: abortRes.SYSWARNINGTEXT,
                     SYSERRORTEXT: abortRes.SYSERRORTEXT,
-                    MAC: macMsg
+                    MAC: macMsg,
+                    _PROGRAM: abortRes._PROGRAM
                   }
                 )
               }
@@ -355,7 +358,10 @@ export class SasService {
 
     this.sasjsService.getFolderContentsFromDrive(configuratorFolder).subscribe(
       (contents: SASjsApiDriveFolderContents) => {
-        if (contents.files.includes('makedata.sas')) {
+        if (
+          contents.files.includes('makedata.sas') ||
+          contents.files.includes('makedata.js')
+        ) {
           this.eventService.startupDataLoaded()
           this.router.navigateByUrl('/deploy')
         } else {
@@ -417,7 +423,10 @@ export class SasService {
         .getFolderContentsFromDrive(configuratorFolder)
         .subscribe(
           (contents: SASjsApiDriveFolderContents) => {
-            if (!contents.files.includes('makedata.sas')) {
+            if (
+              !contents.files.includes('makedata.sas') &&
+              !contents.files.includes('makedata.js')
+            ) {
               resolve(true)
             } else {
               resolve(false)
@@ -437,6 +446,9 @@ export class SasService {
   }
 
   public async checkViyaDeploy(path: string) {
+    console.log('checkViyaDeploy: checking appLoc', path)
+    this.startupCheckService.updateStep(0, 'in_progress')
+
     const getFolderExistsInAdapter =
       typeof this.sasjsAdapter.getFolder !== 'undefined'
 
@@ -452,25 +464,58 @@ export class SasService {
       appLocExists = await this.appLocCheckPreAxiosdAdapter(path)
     }
 
+    console.log(
+      'checkViyaDeploy: appLoc exists?',
+      appLocExists,
+      'error?',
+      errorMessage
+    )
+
     if (appLocExists) {
+      this.startupCheckService.updateStep(0, 'done', path)
+
       // Check if there is appLoc/services/admin/makedata.sas present
       // if yes, it needs to be run, so we redirect to /deploy
       // if not, we load the startup service
 
+      this.startupCheckService.updateStep(1, 'in_progress')
       this.viyaMakedataSuccessfull().then(
         (success: boolean) => {
+          console.log(
+            'checkViyaDeploy: makedata job already run?',
+            success,
+            success
+              ? '-> loading startupservice'
+              : '-> redirecting to /deploy (setup screen)'
+          )
           if (success) {
+            this.startupCheckService.updateStep(1, 'done', 'makedata complete')
+            this.startupCheckService.updateStep(2, 'in_progress')
             this.loadStartupServiceEmitter.emit()
           } else {
+            this.startupCheckService.updateStep(
+              1,
+              'done',
+              'makedata job found - setup needed'
+            )
             this.eventService.startupDataLoaded()
             this.router.navigateByUrl('/deploy')
           }
         },
         (error: any) => {
-          console.error('Error while looking for the file: makedata.sas', error)
+          console.error(
+            'checkViyaDeploy: error while looking for makedata job',
+            error
+          )
+          this.startupCheckService.updateStep(1, 'error', String(error))
         }
       )
     } else {
+      this.startupCheckService.updateStep(
+        0,
+        'error',
+        errorMessage || 'not found'
+      )
       const errorMessageToShow =
         (errorMessage ||
           'Viya services are not present on the current appLoc, or API not reachable. Check the ADAPTER configuration.') +
@@ -480,43 +525,71 @@ export class SasService {
     }
   }
 
-  private async viyaMakedataSuccessfull(): Promise<boolean> {
+  public async viyaMakedataSuccessfull(): Promise<boolean> {
     return new Promise((resolve, reject) => {
       const sasjsConfig = this.getSasjsConfig()
       const configuratorFolder = `${sasjsConfig.appLoc}/services/admin`
 
+      console.log(
+        'viyaMakedataSuccessfull: checking folder',
+        configuratorFolder
+      )
+
       this.sasViyaService.getFolderByPath(configuratorFolder).subscribe(
         (folderInfo: ViyaApiFolder) => {
           const folderId = folderInfo.id
 
           if (!folderId) {
             console.error(
-              `Folder ID is not present. ${configuratorFolder}`,
+              `viyaMakedataSuccessfull: folder ID not present for ${configuratorFolder}`,
               sasjsConfig
             )
             resolve(false)
+            return
           }
 
           this.sasViyaService.getFolderMembers(folderId).subscribe(
             (members: ViyaApiFolderMembers) => {
-              if (
-                !members.items.some((item: any) => item.name === 'makedata')
-              ) {
-                // Makedata.sas is not present, which means it was run
+              const memberNames = members.items.map((item: any) => item.name)
+              const hasMakedata = members.items.some(
+                (item: any) => item.name === 'makedata'
+              )
+
+              console.log(
+                'viyaMakedataSuccessfull: folder members:',
+                memberNames.join(', ') || '(none)'
+              )
+              console.log(
+                'viyaMakedataSuccessfull: makedata job present?',
+                hasMakedata,
+                hasMakedata
+                  ? '-> setup needed (redirect to /deploy)'
+                  : '-> setup already done (load startupservice)'
+              )
+
+              if (!hasMakedata) {
                 resolve(true)
               } else {
-                // Makedata.sas is present, which means it was not run
                 resolve(false)
               }
             },
             (err: any) => {
-              console.error('Error getting folder contents', err)
-              reject()
+              console.error(
+                'viyaMakedataSuccessfull: error getting folder members for',
+                configuratorFolder,
+                err
+              )
+              // On error, assume setup is needed rather than skipping it
+              resolve(false)
             }
           )
         },
         (err: any) => {
-          console.warn('Error getting folder info', err)
+          console.warn(
+            'viyaMakedataSuccessfull: error getting folder info for',
+            configuratorFolder,
+            err
+          )
           reject(err)
         }
       )

client/src/app/services/startup-check.service.ts

@@ -0,0 +1,37 @@
+import { Injectable } from '@angular/core'
+import { BehaviorSubject } from 'rxjs'
+
+export type StartupStepStatus = 'pending' | 'in_progress' | 'done' | 'error'
+
+export interface StartupStep {
+  label: string
+  status: StartupStepStatus
+  detail?: string
+}
+
+@Injectable({ providedIn: 'root' })
+export class StartupCheckService {
+  private steps: StartupStep[] = [
+    { label: 'Checking app location', status: 'pending' },
+    { label: 'Checking Viya deploy', status: 'pending' },
+    { label: 'Loading startup service', status: 'pending' }
+  ]
+
+  public steps$ = new BehaviorSubject<StartupStep[]>([...this.steps])
+
+  public updateStep(index: number, status: StartupStepStatus, detail?: string) {
+    if (index >= 0 && index < this.steps.length) {
+      this.steps[index] = {
+        ...this.steps[index],
+        status,
+        detail: detail ?? this.steps[index].detail
+      }
+      this.steps$.next([...this.steps])
+    }
+  }
+
+  public setSteps(steps: StartupStep[]) {
+    this.steps = steps
+    this.steps$.next([...this.steps])
+  }
+}

client/src/app/services/utils/decodeLicenceFeatures.spec.ts

@@ -0,0 +1,79 @@
+import { decodeLicenceFeatures } from './decodeLicenceFeatures'
+
+describe('decodeLicenceFeatures', () => {
+  // Positional order per LicenceFeaturesMap: viewer_rows_allowed,
+  // editor_rows_allowed, stage_rows_allowed, history_rows_allowed,
+  // submit_rows_limit, tables_in_library_limit, viewbox, viewbox_limit,
+  // lineage_daily_limit, fileUpload, editRecord, addRecord.
+  const legacyFeaturesString = '10,15,20,-,5,35,1,3,7,1,0,1'
+
+  const newFeaturesObject = {
+    vra: 10,
+    era: 15,
+    sra: 20,
+    hra: null,
+    srl: 5,
+    till: 35,
+    vb: true,
+    vbl: 3,
+    ldl: 7,
+    fu: true,
+    er: false,
+    ar: true
+  }
+
+  const expectedDecodedState = {
+    viewer_rows_allowed: 10,
+    editor_rows_allowed: 15,
+    stage_rows_allowed: 20,
+    history_rows_allowed: Infinity,
+    submit_rows_limit: 5,
+    tables_in_library_limit: 35,
+    viewbox: true,
+    viewbox_limit: 3,
+    lineage_daily_limit: 7,
+    fileUpload: true,
+    editRecord: false,
+    addRecord: true
+  }
+
+  it('decodes a legacy positional string exactly as before', () => {
+    expect(decodeLicenceFeatures(legacyFeaturesString)).toEqual(
+      expectedDecodedState
+    )
+  })
+
+  it('decodes a new named object', () => {
+    expect(decodeLicenceFeatures(newFeaturesObject)).toEqual(
+      expectedDecodedState
+    )
+  })
+
+  it('produces an identical result for the same licence expressed in either format', () => {
+    expect(decodeLicenceFeatures(legacyFeaturesString)).toEqual(
+      decodeLicenceFeatures(newFeaturesObject)
+    )
+  })
+
+  it('maps "-" in a legacy limit position to Infinity', () => {
+    const result = decodeLicenceFeatures(legacyFeaturesString)
+    expect(result.history_rows_allowed).toBe(Infinity)
+  })
+
+  it('maps null in a new-format limit field to Infinity', () => {
+    const result = decodeLicenceFeatures(newFeaturesObject)
+    expect(result.history_rows_allowed).toBe(Infinity)
+  })
+
+  it('maps "1"/"0" in legacy toggle positions to true/false', () => {
+    const result = decodeLicenceFeatures(legacyFeaturesString)
+    expect(result.viewbox).toBe(true)
+    expect(result.editRecord).toBe(false)
+  })
+
+  it('passes new-format boolean toggle fields straight through', () => {
+    const result = decodeLicenceFeatures(newFeaturesObject)
+    expect(result.viewbox).toBe(true)
+    expect(result.editRecord).toBe(false)
+  })
+})

client/src/app/services/utils/decodeLicenceFeatures.ts

@@ -0,0 +1,91 @@
+import { LicenceFeaturesMap, LicenceState } from '../../models/LicenceState'
+
+// Field names are abbreviated on the wire to keep the RSA-OAEP-encrypted
+// payload small - see "Why abbreviated field names" in
+// licence-features-object-plan.md for the full name each one stands for
+// and why this is a fixed, append-only contract (same invariant as
+// LicenceFeaturesMap's ordinals).
+export type LicenceFeaturesObject = {
+  vra: number | null // viewer_rows_allowed
+  era: number | null // editor_rows_allowed
+  sra: number | null // stage_rows_allowed
+  hra: number | null // history_rows_allowed
+  srl: number | null // submit_rows_limit
+  till: number | null // tables_in_library_limit
+  vb: boolean // viewbox
+  vbl: number | null // viewbox_limit
+  ldl: number | null // lineage_daily_limit
+  fu: boolean // fileUpload
+  er: boolean // editRecord
+  ar: boolean // addRecord
+}
+
+// Accepts either an already-issued key's positional string
+// ("-,-,-,-,-,-,1,-,-,1,1,1", decoded via LicenceFeaturesMap) or a
+// newly-issued key's named object - see licence-features-object-plan.md
+// for why these two shapes are how format versioning is done here (no
+// separate version field: the shapes are already distinguishable).
+export const decodeLicenceFeatures = (
+  features: string | LicenceFeaturesObject
+): Partial<LicenceState> => {
+  if (typeof features === 'object') {
+    return decodeFeaturesObject(features)
+  }
+
+  return decodeFeaturesString(features)
+}
+
+const decodeFeaturesObject = (
+  features: LicenceFeaturesObject
+): Partial<LicenceState> => ({
+  viewer_rows_allowed: features.vra ?? Infinity,
+  editor_rows_allowed: features.era ?? Infinity,
+  stage_rows_allowed: features.sra ?? Infinity,
+  history_rows_allowed: features.hra ?? Infinity,
+  submit_rows_limit: features.srl ?? Infinity,
+  tables_in_library_limit: features.till ?? Infinity,
+  viewbox: features.vb,
+  viewbox_limit: features.vbl ?? Infinity,
+  lineage_daily_limit: features.ldl ?? Infinity,
+  fileUpload: features.fu,
+  editRecord: features.er,
+  addRecord: features.ar
+})
+
+// Existing positional-string decode, moved here unchanged from
+// LicenceService.decodeLicenceFeatures/parseFeatureValue/parseFeatureToggle.
+const decodeFeaturesString = (features: string): Partial<LicenceState> => {
+  const featuresMap = features.split(',')
+  const parseValue = (codeBit: string) =>
+    codeBit === '-' ? Infinity : parseInt(codeBit)
+  const parseToggle = (codeBit: string) => !!parseInt(codeBit)
+
+  return {
+    viewer_rows_allowed: parseValue(
+      featuresMap[LicenceFeaturesMap.viewer_rows_allowed]
+    ),
+    editor_rows_allowed: parseValue(
+      featuresMap[LicenceFeaturesMap.editor_rows_allowed]
+    ),
+    stage_rows_allowed: parseValue(
+      featuresMap[LicenceFeaturesMap.stage_rows_allowed]
+    ),
+    history_rows_allowed: parseValue(
+      featuresMap[LicenceFeaturesMap.history_rows_allowed]
+    ),
+    submit_rows_limit: parseValue(
+      featuresMap[LicenceFeaturesMap.submit_rows_limit]
+    ),
+    tables_in_library_limit: parseValue(
+      featuresMap[LicenceFeaturesMap.tables_in_library_limit]
+    ),
+    viewbox_limit: parseValue(featuresMap[LicenceFeaturesMap.viewbox_limit]),
+    lineage_daily_limit: parseValue(
+      featuresMap[LicenceFeaturesMap.lineage_daily_limit]
+    ),
+    viewbox: parseToggle(featuresMap[LicenceFeaturesMap.viewbox]),
+    fileUpload: parseToggle(featuresMap[LicenceFeaturesMap.fileUpload]),
+    editRecord: parseToggle(featuresMap[LicenceFeaturesMap.editRecord]),
+    addRecord: parseToggle(featuresMap[LicenceFeaturesMap.addRecord])
+  }
+}

client/src/app/services/va-messaging.service.ts

@@ -138,13 +138,38 @@ export class VaMessagingService {
     this.earlyDrained = true
     const captured = (window as unknown as { __vaLastMessage?: any })
       .__vaLastMessage
-    const parsed = this.parseData(captured && captured.data)
+    // The early listener (va-early.js) captures from any origin, so re-apply
+    // the live-path trust rule here before acting on it: only replay a message
+    // that came from our own origin or from the frame that embedded us (whose
+    // URL is document.referrer). Without this, a same-origin sibling frame
+    // could inject a crafted DDC message that the live isTrustedSource check
+    // would have rejected.
+    if (!captured || typeof captured.origin !== 'string') return
+    if (!this.isTrustedEarlyOrigin(captured.origin)) return
+    const parsed = this.parseData(captured.data)
     if (!parsed) return
     this.resultName = parsed.resultName
-    if (captured.origin) this.parentOrigin = captured.origin
+    this.parentOrigin = captured.origin
     callback(parsed)
   }
 
+  /**
+   * Origin check for the pre-bootstrap replay. Mirrors isTrustedSource: the
+   * live path trusts a message whose event.source IS the parent frame; for a
+   * captured message we cannot reference its source Window, so we trust an
+   * origin that is this window's origin or the embedding frame's origin
+   * (document.referrer). Unverifiable/absent referrer -> reject.
+   */
+  private isTrustedEarlyOrigin(origin: string): boolean {
+    if (origin === window.location.origin) return true
+    if (!document.referrer) return false
+    try {
+      return new URL(document.referrer).origin === origin
+    } catch {
+      return false
+    }
+  }
+
   /**
    * Parses a raw window MessageEvent into a VaMessage, or null when it is not a
    * recognisable DDC message (e.g. unrelated postMessage traffic).

client/src/app/shared/abort-modal/info-modal.component.html

@@ -23,6 +23,7 @@
         </p>
         <p><strong>SYSERRORTEXT:</strong> {{ data.details.SYSERRORTEXT }}</p>
         <p><strong>MAC:</strong> {{ data.details.MAC }}</p>
+        <p><strong>_PROGRAM:</strong> {{ data.details._PROGRAM }}</p>
       </div>
     }
   </div>

client/src/app/shared/dc-validator/dc-validator.ts

@@ -92,14 +92,49 @@ export class DcValidator {
     this.rules.push({ ...EDIT_STATUS_COLUMN_RULE })
     this.hiddenColumns.push(this.rules.length - 1)
 
-    this.dqrules = dqRules
+    this.dqrules = [...dqRules]
     this.dqdata = dqData
     this.primaryKeys = sasparams.PK.split(' ')
 
+    // A primary key is NOT NULL by definition, so it must reject a blank and
+    // a special missing (".A"-".Z", "._") even when the target table carries
+    // no physical NOT NULL constraint and MPE_VALIDATIONS has no NOTNULL rule
+    // for it.  Synthesised here so every keyed table gets it, and so the grid,
+    // the edit-record modal and Excel upload validation all see the same rule.
+    this.addPrimaryKeyNotNullRules()
+
     this.updateDqData()
     this.setupValidations()
   }
 
+  /**
+   * Adds a NOTNULL rule for each primary key column that does not already
+   * have one.  A primary key identifies the row, so a blank or a special
+   * missing there is never valid.
+   */
+  private addPrimaryKeyNotNullRules(): void {
+    for (const pk of this.primaryKeys) {
+      if (!pk) continue
+
+      // A buskey can name a column the table no longer has - do not
+      // synthesise a rule for a column that is not in the grid.
+      if (!this.rules.some((rule) => rule.data === pk)) continue
+
+      const hasNotNull = this.dqrules.some(
+        (rule) => rule.BASE_COL === pk && rule.RULE_TYPE === 'NOTNULL'
+      )
+
+      if (!hasNotNull) {
+        this.dqrules.push({
+          BASE_COL: pk,
+          RULE_TYPE: 'NOTNULL',
+          RULE_VALUE: '',
+          X: 1
+        })
+      }
+    }
+  }
+
   registerCustomEditors() {
     Handsontable.editors.registerEditor(
       'autocomplete.custom',
@@ -283,7 +318,14 @@ export class DcValidator {
              * So we will convert it before pushing to array.
              */
             if (rule.type && rule.type === 'numeric') {
-              details.push(Number(data['RULE_DATA']))
+              // A special missing reaches us as a bare letter ("A", "_"), and
+              // the regular missing as "." - Number() would turn either into
+              // NaN, so a strict (HARDSELECT) dropdown could never accept a
+              // value the column actually holds.  Keep them as they are.
+              const rawValue = data['RULE_DATA']
+              details.push(
+                isSpecialMissing(rawValue) ? rawValue : Number(rawValue)
+              )
             } else {
               details.push(data['RULE_DATA'])
             }

client/src/app/shared/dc-validator/tests/dc-validator.spec.ts

@@ -118,7 +118,12 @@ describe('DC Validator', () => {
     expect(dcValidator.getRule('SOME_TIME')).toBeUndefined()
 
     // Test data quality functions
-    expect(dcValidator.getDqDetails()).toHaveSize(dqRules.length)
+    // dqRules + 2 synthesised rules: the SOFTSELECT rule updateDqData()
+    // derives for SOME_DROPDOWN out of dqdata, and the NOTNULL rule
+    // addPrimaryKeyNotNullRules() gives the primary key column (example_dqRules
+    // has none for it).  Note the constructor copies dqRules rather than
+    // aliasing it, so this array is no longer mutated by construction.
+    expect(dcValidator.getDqDetails()).toHaveSize(dqRules.length + 2)
     expect(dcValidator.getDqDetails('non_existant')).toHaveSize(0)
     expect(dcValidator.getDqDetails('SOME_NUM')).toHaveSize(2)
     expect(dcValidator.isDqCol('SOME_NUM')).toBeTrue()
@@ -133,6 +138,91 @@ describe('DC Validator', () => {
     ])
   })
 
+  it('treats the primary key column as NOT NULL, even with no NOTNULL rule configured', () => {
+    const sasparams: SASParam = example_sasparams
+    const cols: Col[] = example_cols
+    const dqRules: DQRule[] = example_dqRules // no NOTNULL for PRIMARY_KEY_FIELD
+    const dqData: DQData[] = example_dqData
+    const $dataFormats: $DataFormats = example_dataformats
+
+    const dcValidator: DcValidator = new DcValidator(
+      sasparams,
+      $dataFormats,
+      cols,
+      dqRules,
+      dqData
+    )
+
+    const pkRules = dcValidator.getDqDetails('PRIMARY_KEY_FIELD')
+    expect(pkRules.some((rule) => rule.RULE_TYPE === 'NOTNULL')).toBeTrue()
+
+    const pkRule = dcValidator.getRule('PRIMARY_KEY_FIELD')
+
+    // A primary key identifies the row, so neither a blank nor a special
+    // missing can satisfy it.
+    dcValidator.executeHotValidator(pkRule!, null, (valid: boolean) => {
+      expect(valid).toBeFalse()
+    })
+    dcValidator.executeHotValidator(pkRule!, 'A', (valid: boolean) => {
+      expect(valid).toBeFalse()
+    })
+    dcValidator.executeHotValidator(pkRule!, 5, (valid: boolean) => {
+      expect(valid).toBeTrue()
+    })
+  })
+
+  it('keeps a special missing in a numeric dropdown source, so a strict rule can match it', () => {
+    const dqData: DQData[] = [
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: 1,
+        SELECTBOX_ORDER: 1
+      },
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: 'A',
+        SELECTBOX_ORDER: 2
+      },
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: '_',
+        SELECTBOX_ORDER: 3
+      },
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: '.',
+        SELECTBOX_ORDER: 4
+      }
+    ] as DQData[]
+
+    const dcValidator: DcValidator = new DcValidator(
+      example_sasparams,
+      example_dataformats,
+      example_cols,
+      example_dqRules,
+      dqData
+    )
+
+    // SOME_NUM is numeric and carries a HARDSELECT_HOOK, so its dropdown
+    // source comes from dqdata.  Number() would have NaN'd the special
+    // missings, leaving the strict membership test unable to match a value
+    // the column actually holds.
+    const source = dcValidator.getDqDropdownSource(
+      dcValidator.getRule('SOME_NUM')!
+    )
+
+    expect(source[0]).toEqual(1)
+    expect(source[1]).toEqual('A')
+    expect(source[2]).toEqual('_')
+    // the regular missing is a dropdown option too, not a NaN
+    expect(source[3]).toEqual('.')
+    expect(source.some((value) => Number.isNaN(value as number))).toBeFalse()
+  })
+
   it('should test hot validator', () => {
     const sasparams: SASParam = example_sasparams
     const cols: Col[] = example_cols
@@ -163,21 +253,22 @@ describe('DC Validator', () => {
     dcValidator.executeHotValidator(someNumRule!, 'ss', (valid: boolean) => {
       expect(valid).toBeFalse()
     })
-    //Special missings
+    // Special missings - a SAS NOT NULL (or primary key) constraint rejects
+    // these, so the rule must reject them too: they are NULL, not values.
     dcValidator.executeHotValidator(someNumRule!, 's', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '.s', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '.', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '..', (valid: boolean) => {
       expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '._', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
 
     // MINVAL, MAXVAL Validation
@@ -193,7 +284,7 @@ describe('DC Validator', () => {
       expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(shortNumRule!, 's', (valid: boolean) => {
-      expect(valid).toBeFalse() // Special missings are lowest numbers, if any MINVAL is set, special missing is always lower
+      expect(valid).toBeFalse() // Special missings are the lowest numbers, so any MINVAL is above them
     })
 
     // CASE validation

client/src/app/shared/dc-validator/tests/dqValidation.spec.ts

@@ -24,7 +24,44 @@ describe('DC Validator - dq validation', () => {
     expect(dqValidate(dqRules, invalidValue)).toBeFalse()
     expect(dqValidate(dqRules, invalidStringValue)).toBeFalse()
     expect(dqValidate(dqRules, numericStringValue)).toBeTrue()
+    // A missing sorts below every number, so it is below the floor.
     expect(dqValidate(dqRules, numericSpecialMissingValue)).toBeFalse()
+    expect(dqValidate(dqRules, null)).toBeFalse()
+    expect(dqValidate(dqRules, undefined)).toBeFalse()
+  })
+
+  it('should order the missing values in a range rule as SAS does', () => {
+    const missingRange: DQRule[] = [
+      { BASE_COL: 'test', RULE_TYPE: 'MINVAL', RULE_VALUE: '.A', X: 0 },
+      { BASE_COL: 'test', RULE_TYPE: 'MAXVAL', RULE_VALUE: '.C', X: 0 }
+    ]
+
+    // Inside the range of missings
+    expect(dqValidate(missingRange, '.A')).toBeTrue()
+    expect(dqValidate(missingRange, '.B')).toBeTrue()
+    expect(dqValidate(missingRange, '.C')).toBeTrue()
+    expect(dqValidate(missingRange, 'b')).toBeTrue()
+
+    // Outside it - above the ceiling
+    expect(dqValidate(missingRange, '.D')).toBeFalse()
+    expect(dqValidate(missingRange, 'z')).toBeFalse()
+
+    // Outside it - below the floor.  The regular missing sits between ._ and .A
+    expect(dqValidate(missingRange, '._')).toBeFalse()
+    expect(dqValidate(missingRange, null)).toBeFalse()
+
+    // Every number sorts above every missing, so it is above the ceiling
+    expect(dqValidate(missingRange, 0)).toBeFalse()
+    expect(dqValidate(missingRange, 5)).toBeFalse()
+
+    // A floor of .A alone still lets the numbers through: they are above it
+    const missingFloor: DQRule[] = [
+      { BASE_COL: 'test', RULE_TYPE: 'MINVAL', RULE_VALUE: '.A', X: 0 }
+    ]
+    expect(dqValidate(missingFloor, '.A')).toBeTrue()
+    expect(dqValidate(missingFloor, '.Z')).toBeTrue()
+    expect(dqValidate(missingFloor, 5)).toBeTrue()
+    expect(dqValidate(missingFloor, '._')).toBeFalse()
   })
 
   it('should validate MAXVAL value', () => {
@@ -49,7 +86,10 @@ describe('DC Validator - dq validation', () => {
     expect(dqValidate(dqRules, invalidValue)).toBeFalse()
     expect(dqValidate(dqRules, invalidStringValue)).toBeFalse()
     expect(dqValidate(dqRules, numericStringValue)).toBeTrue()
+    // A missing sorts below every number, so it is below the ceiling
     expect(dqValidate(dqRules, numericSpecialMissingValue)).toBeTrue()
+    expect(dqValidate(dqRules, null)).toBeTrue()
+    expect(dqValidate(dqRules, undefined)).toBeTrue()
   })
 
   it('should validate UPCASE value', () => {
@@ -111,6 +151,43 @@ describe('DC Validator - dq validation', () => {
     expect(dqValidate(dqRules, invalidValue2)).toBeFalse()
   })
 
+  it('should reject a special missing on a numeric column (a SAS NOT NULL constraint does)', () => {
+    const dqRules: DQRule[] = [
+      {
+        BASE_COL: 'test',
+        RULE_TYPE: 'NOTNULL',
+        RULE_VALUE: ' ',
+        X: 0
+      }
+    ]
+
+    // The values a real SAS service delivers for a numeric column.
+    expect(dqValidate(dqRules, 'A', true)).toBeFalse()
+    expect(dqValidate(dqRules, '_', true)).toBeFalse()
+    expect(dqValidate(dqRules, '.', true)).toBeFalse()
+    // Ordinary numbers and numeric strings still pass.
+    expect(dqValidate(dqRules, 5, true)).toBeTrue()
+    expect(dqValidate(dqRules, '5', true)).toBeTrue()
+  })
+
+  it('should not reject a single letter on a character column', () => {
+    const dqRules: DQRule[] = [
+      {
+        BASE_COL: 'test',
+        RULE_TYPE: 'NOTNULL',
+        RULE_VALUE: ' ',
+        X: 0
+      }
+    ]
+
+    // There is no special-missing concept on a character column - a lone
+    // letter is ordinary data.
+    expect(dqValidate(dqRules, 'A')).toBeTrue()
+    expect(dqValidate(dqRules, '_')).toBeTrue()
+    expect(dqValidate(dqRules, '.')).toBeTrue()
+    expect(dqValidate(dqRules, '', false)).toBeFalse()
+  })
+
   it('should return true if rule not found', () => {
     const validValue = 5
 

client/src/app/shared/dc-validator/utils/autoEscapedCellTracker.spec.ts

@@ -0,0 +1,83 @@
+import {
+  getRowKey,
+  markAutoEscaped,
+  isAutoEscaped,
+  clearAutoEscaped
+} from './autoEscapedCellTracker'
+
+describe('autoEscapedCellTracker', () => {
+  describe('getRowKey', () => {
+    it('derives the same key for two different row objects sharing the same PK values', () => {
+      const rowA = { PRIMARY_KEY_FIELD: 3, A_COL: 'x' }
+      const rowB = { PRIMARY_KEY_FIELD: 3, A_COL: 'y' }
+
+      expect(getRowKey(rowA, ['PRIMARY_KEY_FIELD'])).toEqual(
+        getRowKey(rowB, ['PRIMARY_KEY_FIELD'])
+      )
+    })
+
+    it('derives different keys for different PK values', () => {
+      const rowA = { PRIMARY_KEY_FIELD: 3 }
+      const rowB = { PRIMARY_KEY_FIELD: 4 }
+
+      expect(getRowKey(rowA, ['PRIMARY_KEY_FIELD'])).not.toEqual(
+        getRowKey(rowB, ['PRIMARY_KEY_FIELD'])
+      )
+    })
+
+    it('supports composite primary keys', () => {
+      const rowA = { LIBREF: 'DC1', DSN: 'A' }
+      const rowB = { LIBREF: 'DC1', DSN: 'B' }
+
+      expect(getRowKey(rowA, ['LIBREF', 'DSN'])).not.toEqual(
+        getRowKey(rowB, ['LIBREF', 'DSN'])
+      )
+    })
+  })
+
+  describe('markAutoEscaped / isAutoEscaped / clearAutoEscaped', () => {
+    it('is not auto-escaped before being marked', () => {
+      const map = new Map<string, Set<string>>()
+      expect(isAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')).toEqual(false)
+    })
+
+    it('is auto-escaped after being marked', () => {
+      const map = new Map<string, Set<string>>()
+      markAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+      expect(isAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')).toEqual(true)
+    })
+
+    it('is not auto-escaped after being cleared', () => {
+      const map = new Map<string, Set<string>>()
+      markAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+      clearAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+      expect(isAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')).toEqual(false)
+    })
+
+    it('clearing an unmarked cell is a no-op, not a throw', () => {
+      const map = new Map<string, Set<string>>()
+      expect(() =>
+        clearAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+      ).not.toThrow()
+    })
+
+    it('tracks two columns on the same row independently', () => {
+      const map = new Map<string, Set<string>>()
+      markAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+      markAutoEscaped(map, 'row-1', 'OTHER_COL')
+
+      clearAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+
+      expect(isAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')).toEqual(false)
+      expect(isAutoEscaped(map, 'row-1', 'OTHER_COL')).toEqual(true)
+    })
+
+    it('tracks the same column on two different rows independently', () => {
+      const map = new Map<string, Set<string>>()
+      markAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')
+
+      expect(isAutoEscaped(map, 'row-1', 'PLAIN_TEXT_COL')).toEqual(true)
+      expect(isAutoEscaped(map, 'row-2', 'PLAIN_TEXT_COL')).toEqual(false)
+    })
+  })
+})

client/src/app/shared/dc-validator/utils/autoEscapedCellTracker.ts

@@ -0,0 +1,36 @@
+/**
+ * Tracks which cells got a `'` the app itself inserted (see
+ * escapeCharacterColumnValue), so submission and the "Apply as formula"
+ * context-menu action only ever touch cells the app marked - never a
+ * genuine leading `'` that was already part of the data. Kept as a
+ * standalone Map (not a property on the row object) so it can never leak
+ * into the submit payload, and keyed by primary key rather than row index
+ * so it survives insert/delete/sort.
+ */
+export type AutoEscapedCellMap = Map<string, Set<string>>
+
+export const getRowKey = (row: any, headerPks: string[]): string =>
+  headerPks.map((pk) => String(row[pk])).join(' ')
+
+export const markAutoEscaped = (
+  map: AutoEscapedCellMap,
+  rowKey: string,
+  colName: string
+): void => {
+  if (!map.has(rowKey)) map.set(rowKey, new Set())
+  map.get(rowKey)!.add(colName)
+}
+
+export const isAutoEscaped = (
+  map: AutoEscapedCellMap,
+  rowKey: string,
+  colName: string
+): boolean => map.get(rowKey)?.has(colName) ?? false
+
+export const clearAutoEscaped = (
+  map: AutoEscapedCellMap,
+  rowKey: string,
+  colName: string
+): void => {
+  map.get(rowKey)?.delete(colName)
+}

client/src/app/shared/dc-validator/utils/escapeCharacterColumnValue.spec.ts

@@ -0,0 +1,46 @@
+import { escapeCharacterColumnValue } from './escapeCharacterColumnValue'
+
+describe('escapeCharacterColumnValue', () => {
+  it('escapes a =-led value', () => {
+    expect(escapeCharacterColumnValue('=SUM(1)')).toEqual({
+      value: "'=SUM(1)",
+      wasEscaped: true
+    })
+  })
+
+  it('escapes a =-led value referencing a column name', () => {
+    expect(escapeCharacterColumnValue('=100 + B_COL')).toEqual({
+      value: "'=100 + B_COL",
+      wasEscaped: true
+    })
+  })
+
+  it('leaves an already-escaped value unchanged (does not start with =)', () => {
+    expect(escapeCharacterColumnValue("'=already")).toEqual({
+      value: "'=already",
+      wasEscaped: false
+    })
+  })
+
+  it('leaves a plain string unchanged', () => {
+    expect(escapeCharacterColumnValue('note-1')).toEqual({
+      value: 'note-1',
+      wasEscaped: false
+    })
+  })
+
+  it('leaves a non-string value unchanged', () => {
+    expect(escapeCharacterColumnValue(123)).toEqual({
+      value: 123,
+      wasEscaped: false
+    })
+    expect(escapeCharacterColumnValue(null)).toEqual({
+      value: null,
+      wasEscaped: false
+    })
+    expect(escapeCharacterColumnValue(undefined)).toEqual({
+      value: undefined,
+      wasEscaped: false
+    })
+  })
+})

client/src/app/shared/dc-validator/utils/escapeCharacterColumnValue.ts

@@ -0,0 +1,15 @@
+/**
+ * Prepends `'` to a `=`-led value so Handsontable/HyperFormula treats it as
+ * inert text rather than a live formula - Handsontable's own escape
+ * convention, which it strips automatically when displaying the value.
+ * Caller decides eligibility (character column? not a formula base column?)
+ * before calling this - it only ever looks at the value itself.
+ */
+export const escapeCharacterColumnValue = (
+  value: unknown
+): { value: unknown; wasEscaped: boolean } => {
+  if (typeof value !== 'string' || !value.startsWith('=')) {
+    return { value, wasEscaped: false }
+  }
+  return { value: `'${value}`, wasEscaped: true }
+}

client/src/app/shared/dc-validator/utils/getFormulaColumnNames.spec.ts

@@ -0,0 +1,51 @@
+import { getFormulaColumnNames } from './getFormulaColumnNames'
+import { DQRule } from '../models/dq-rules.model'
+
+const rule = (overrides: Partial<DQRule> = {}): DQRule => ({
+  BASE_COL: 'SOME_COL',
+  RULE_TYPE: 'NOTNULL',
+  RULE_VALUE: '',
+  X: 0,
+  ...overrides
+})
+
+describe('getFormulaColumnNames', () => {
+  it('returns the BASE_COL of every HARDFORMULA rule', () => {
+    expect(
+      getFormulaColumnNames([
+        rule({ BASE_COL: 'FORMULA_HARD_COL', RULE_TYPE: 'HARDFORMULA' })
+      ])
+    ).toEqual(['FORMULA_HARD_COL'])
+  })
+
+  it('returns the BASE_COL of every SOFTFORMULA rule', () => {
+    expect(
+      getFormulaColumnNames([
+        rule({ BASE_COL: 'FORMULA_SOFT_COL', RULE_TYPE: 'SOFTFORMULA' })
+      ])
+    ).toEqual(['FORMULA_SOFT_COL'])
+  })
+
+  it('excludes non-formula rule types', () => {
+    expect(
+      getFormulaColumnNames([
+        rule({ BASE_COL: 'PRIMARY_KEY_FIELD', RULE_TYPE: 'NOTNULL' }),
+        rule({ BASE_COL: 'READONLY_COL', RULE_TYPE: 'READONLY' })
+      ])
+    ).toEqual([])
+  })
+
+  it('returns [] for an empty rule set', () => {
+    expect(getFormulaColumnNames([])).toEqual([])
+  })
+
+  it('mixes formula and non-formula rules correctly', () => {
+    expect(
+      getFormulaColumnNames([
+        rule({ BASE_COL: 'PRIMARY_KEY_FIELD', RULE_TYPE: 'NOTNULL' }),
+        rule({ BASE_COL: 'FORMULA_HARD_COL', RULE_TYPE: 'HARDFORMULA' }),
+        rule({ BASE_COL: 'FORMULA_SOFT_COL', RULE_TYPE: 'SOFTFORMULA' })
+      ])
+    ).toEqual(['FORMULA_HARD_COL', 'FORMULA_SOFT_COL'])
+  })
+})

client/src/app/shared/dc-validator/utils/getFormulaColumnNames.ts

@@ -0,0 +1,9 @@
+import { DQRule } from '../models/dq-rules.model'
+
+export const getFormulaColumnNames = (dqRules: DQRule[]): string[] =>
+  dqRules
+    .filter(
+      (rule) =>
+        rule.RULE_TYPE === 'HARDFORMULA' || rule.RULE_TYPE === 'SOFTFORMULA'
+    )
+    .map((rule) => rule.BASE_COL)

client/src/app/shared/dc-validator/utils/isCharacterColumn.spec.ts

@@ -0,0 +1,21 @@
+import { isCharacterColumn } from './isCharacterColumn'
+import { Col } from '../models/col.model'
+
+const col = (NAME: string, DDTYPE: string): Col => ({ NAME, DDTYPE }) as Col
+
+describe('isCharacterColumn', () => {
+  it('returns true for a column whose DDTYPE is C', () => {
+    const cols = [col('PLAIN_TEXT_COL', 'C')]
+    expect(isCharacterColumn(cols, 'PLAIN_TEXT_COL')).toEqual(true)
+  })
+
+  it('returns false for a column whose DDTYPE is N', () => {
+    const cols = [col('A_COL', 'N')]
+    expect(isCharacterColumn(cols, 'A_COL')).toEqual(false)
+  })
+
+  it('returns false for a column not present in cols at all', () => {
+    const cols = [col('A_COL', 'N')]
+    expect(isCharacterColumn(cols, 'EDIT_STATUS')).toEqual(false)
+  })
+})

client/src/app/shared/dc-validator/utils/isCharacterColumn.ts

@@ -0,0 +1,9 @@
+import { Col } from '../models/col.model'
+
+/**
+ * DDTYPE is the backend's own char/numeric classification per column ('C'
+ * or 'N') - distinct from Col.TYPE, which mergeColsRules derives separately
+ * from $sasdata.vars (values 'char'/'num', not 'C'/'N').
+ */
+export const isCharacterColumn = (cols: Col[], colName: string): boolean =>
+  cols.find((col) => col.NAME === colName)?.DDTYPE === 'C'