chore(release): 7.15.0 [skip ci]

# [7.15.0](https://git.datacontroller.io/dc/dc/compare/v7.14.2...v7.15.0) (2026-09-23)

### Bug Fixes

* **mocks:** keep special missings in the DIFF, and add the clip recording spec ([d124ce3](d124ce3b36))
* **mocks:** list a column's own special missing in its dropdown ([b8f16a6](b8f16a6382))
* **mocks:** make the approval path work, and record the approval scene ([12847cf](12847cf071))
* **validator:** a range rule ignores a missing value ([70dae4b](70dae4b701))
* **validator:** keep the regular missing in a numeric dropdown source too ([a639bca](a639bca702))
* **validator:** primary keys are NOT NULL, and a strict dropdown can match a special missing ([586a41a](586a41ad49))
* **validator:** reject a special missing on a NOT NULL column ([6d85bce](6d85bce2a0))

### Features

* **validator:** compare MINVAL and MAXVAL in SAS's own order ([3467322](3467322e99))

.agents/skills/dc-cypress/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: dc-cypress
+description: >
+  Use this skill when writing or debugging Cypress E2E specs in the dc repo, or when working on the
+  JS mock backend they run against. Covers the SASjs Server mock environment (deploy, seed, ports),
+  the spec conventions the CI harness expects, the viewer/editor DOM selectors that the existing
+  specs get wrong, the fixed 1280x720 video capture, and the rule that a mock service must mirror
+  the real SAS service it stands in for.
+---
+
+# Cypress E2E and the JS mock backend
+
+The specs under `client/cypress/e2e/` run against a mock backend: JS stored programs under `sas/mocks/sasjs/services/` executed by SASjs Server in JS-only mode. No SAS licence, no SAS server. CI (`.gitea/workflows/build.yaml`) builds the same environment - SASjs Server on :5000, `sasjs cbd -t server-ci`, makedata, then `ng serve` on :4200 - and runs a fixed list of specs against it.
+
+This skill covers the parts that are not obvious from reading a spec file: how to stand the environment up, the selectors that actually work, and the traps in the mock services. It does not cover the SAS-side tests (see `.agents/docs/testing.md` and the `dc-sas` skill).
+
+## When to Use
+
+- Adding or changing a spec under `client/cypress/e2e/`.
+- A mock service needs a new behaviour, or a spec "does nothing" because the mock ignores an input.
+- Recording a demo video from a spec run.
+- Debugging a spec that passes locally but fails in CI (usually a viewport or timing assumption).
+
+Don't use for: SAS backend tests (`sasjs test -t 4gl`), Angular unit tests (`npm run test:headless`), or general SAS development (use the `dc-sas` skill).
+
+## Stand up the mock environment
+
+1. **SASjs Server, JS mode.** Download the linux build from the sasjs/server releases, then create a `.env` next to the binary:
+
+       RUN_TIMES=js
+       NODE_PATH=node
+       CORS=enable
+       WHITELIST=http://localhost:4200
+
+   Start it (`./api-linux`). It runs in desktop mode on :5000 with no authentication.
+
+2. **Deploy the mocks.**
+
+       cd sas/mocks/sasjs && sasjs cbd -t server-ci
+
+   To start from a clean appLoc, delete it first:
+
+       curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
+
+3. **Seed the mock database.**
+
+       cd sas/mocks && sasjs request services/admin/makedata -t server-ci \
+           -d deploy/makedata.json -o ./makedata_out.json
+
+   `makedata` replies with HTML (it is normally called as a URL redirect), so the CLI reports `invalid Json string`. That is expected - check the drive instead.
+
+4. **Frontend.** Point the `sasjs` tag in `client/src/index.html` at the mock backend (`serverUrl="http://localhost:5000"`, `appLoc="/Public/app/dc"`, `serverType="SASJS"` - a local-only edit, the repo default is `serverUrl=""`), then `npx ng serve --host 0.0.0.0 --port 4200`. `client/cypress.env.json` needs `username`/`password` entries; any values work, `loginAndUpdateValidKey()` no-ops when there is no login form.
+
+5. **Run.**
+
+       cd client && npx cypress run --browser electron --spec cypress/e2e/<spec>.cy.ts
+
+   Only Electron is installed here (no system Chrome), so always pass `--browser electron`.
+
+## Spec conventions
+
+- Start the file with `export {}` so its top-level consts do not collide with other spec files under the TS type-checker.
+- `beforeAll`: `cy.visit(\`${hostUrl}/SASLogon/logout\`)` then `cy.loginAndUpdateValidKey(true)`. The `true` forces a licence key instead of relying on an earlier spec in the same run having applied one.
+- `beforeEach`: `cy.visit(hostUrl + appLocation)`, `visitPage('home')`, `visitPage('view/data')`. The `home` visit is what triggers the startup-data fetch; going straight to `view/data` leaves the nav tree empty in slower environments.
+- Declare helpers at the **bottom of each spec file** (that is the house pattern) - do not introduce a shared helper module.
+- Use `Cypress.env('longerCommandTimeout')` for waits and gate on `.app-loading` disappearing.
+- **No artificial delays.** These specs are the CI regression suite, so `cy.wait(n)` to "let a result land" is not acceptable - assert on the resulting state instead (Cypress retries assertions). Replace a sleep with a state assertion: e.g. after picking an option from a soft-select, wait for `#datalist_<inputId>` to have class `hidden` rather than sleeping.
+- Lint: `cd client && npm run lint:check` (prettier covers `cypress/e2e/*.cy.ts`).
+
+## Selectors that work
+
+- Viewer search box: `input[name="search-input"]` (Enter or the magnifier icon submits); the numeric toggle is `input[name="numeric_check"]`.
+- Row/column count in the header: `.title-col p` renders `(1,000 rows, 9 cols)` - and `(1 row, 9 cols)` when the result is a single row, so an assertion on the count has to pluralise.
+- Filter info bar: `.infoBar b` - only rendered when the backend's `sasparams.FILTER_TEXT` is neither blank, `' '`, nor `'1=1'`.
+- Empty result: `.noData h3` with `No data found with given conditions`.
+- Grid: `#hotTable .ht_master.handsontable .htCore tbody tr` (body rows), `#hotTable .ht_clone_top .htCore thead` (headers, and `button.changeType` confirms Handsontable has finished rendering them).
+- Nav tree: `.nav-tree clr-tree > clr-tree-node` -> `.clr-tree-node-content-container .clr-treenode-content p` (expands a library) -> `.clr-treenode-link` (table links).
+- **Viewer filter modal** - `.filterSide` (the options button) -> `.dropdown-menu` contains `Filter` -> `#vals_var_id0_0` (variable, pick from `#datalist_vals_var_id0_0 option`) and `#vals_0_0` (value) -> `.filter-modal button[aria-label="Ok"]`.
+  - `filtering.cy.ts` opens the filter with `.btnCtrl .btnView`, which is an **editor** control and does not exist in the viewer. Do not copy that helper into a viewer spec.
+  - The soft-select inputs drop a transparent click-catcher over the modal while their suggestion list is open, so Cypress reports the Ok button (and the next input) as covered by `.overlay`. Send the Escape keyup a user would send (`trigger('keyup', { key: 'Escape', force: true })`) and wait for the list to close before clicking on.
+
+## Recording a demo video
+
+`video: true` in `cypress.config.ts` is all that is needed. Run the spec and the MP4 lands in `client/cypress/videos/<spec>.mp4`.
+
+**The capture is the browser window's content area, and it contains the Cypress runner** - the command log takes the left ~450px and the app is rendered in the remaining pane, so the pane is `(capture width - 450) x (capture height - 96)` and the app is zoomed to fit it.
+
+That makes the window size the only lever that matters:
+
+- Electron is stuck at 1280x720 (it ignores `--window-size`), i.e. an 830x624 pane, and the repo's 1600x900 viewport renders at 50% - the app only occupies ~806x452 of it.
+- A real Chromium browser honours `--window-size`, so the capture can be sized to whatever the pane needs. `cypress.config.ts` passes it through from `RECORD_WINDOW_SIZE` (inert in CI, which sets no env).
+
+For a 16:9 pane, solve `(W - 450) / (H - 96) = 16/9`: W = 1920 gives H = 923 of window *content*, and Chrome adds ~87px of window chrome above that, so pass 1920x1010:
+
+    RECORD_WINDOW_SIZE=1920,1010 npx cypress run \
+      --browser /opt/data/profiles/dc/tools/chrome-linux64/chrome \
+      --spec cypress/e2e/full-table-search.cy.ts \
+      --env skipNegative=true,demoPause=1500
+
+That yields a 1920x922 capture with a 1470x826 pane (16:9) and, at the repo's 1600x900 viewport, an app at ~0.90 zoom occupying 1441x812 - about 1.9x the linear size of the 16:9 crop Electron can give. Crop the app out:
+
+    ffmpeg -ss <start> -i cypress/videos/<spec>.mp4 -vf "crop=1440:812:464:80" \
+      -an -c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p \
+      -movflags +faststart <spec>-cut.mp4
+
+Measure the crop rect from the frames rather than guessing: the pane's flat background is rgb(225,227,236), so a row/column scan for pixels that differ from it gives the app's bounding box (for the 1920x922 capture: runner pane ends x=449, app x=463-1903, app y=80-893; for the 1280x720 Electron capture: app x=468-1259, y=80-675). The app's own background is nearly the same grey, so verify the result on the first, middle and last frames rather than trusting a pixel test alone.
+
+To find where to cut, scan the mean *and* standard deviation of the content area: the app boots mid-run (a dark screen, mean ~65, sd ~2.5), the "Loading Table Viewer" spinner is flat and light (mean ~252, sd ~0), and the rendered table is mean ~245, sd ~31. Cut on the first table frame - cutting a fraction early shows the boot screen or the spinner.
+
+Cypress starts its own Xvfb for the run (a second display appears in `/tmp/.X11-unix`), so an external `DISPLAY` is ignored; `--force-device-scale-factor` and a CSS `zoom` also do not change the capture.
+
+### Grid columns that are clipped
+
+Handsontable sizes columns to content, so a narrow result can be *wider* than the grid viewport: the horizontal scrollbar appears and the rightmost column (NOTES on `MPE_X_SEARCH`) is cut off. The cell text is still in the DOM, so an assertion passes either way - the column only has to be scrolled into view for the *video* to show why the rows matched.
+
+    const scrollGridTo = (position: 'left' | 'right') => {
+      cy.get('#hotTable .wtHolder').first().then(($holder: any) => {
+        const el = $holder[0]
+        el.scrollLeft = position === 'right' ? el.scrollWidth : 0
+        expect(el.scrollLeft).to.equal(position === 'right' ? el.scrollWidth - el.clientWidth : 0)
+      })
+    }
+
+Setting `scrollLeft` is enough - Handsontable keeps the header clone in step - and the browser clamps it to `scrollWidth - clientWidth`, so that is what scrolling right has to land on. This is a legitimate spec step (it runs in CI), not a recording flag.
+
+### A longer, more readable demo
+
+A spec that doubles as a walkthrough should be a single `it()`: open the table once and make each variation a search in place. The recording then reads as a demo rather than a sequence of page loads, and it is far faster - every extra `it()` pays for the `beforeEach` navigation again.
+
+Two recording-only env flags, both inert in CI because CI passes no env:
+
+    const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
+    const negativeStep = (step: () => void) => {
+      if (!skipNegative) step()
+    }
+    const demoPauseMs = Number(Cypress.env('demoPause')) || 0
+
+    negativeStep(() => {
+      searchFor('trench')
+      assertNoData()
+    })
+
+- `skipNegative=true` drops the steps that legitimately match nothing (wrong case, partial number, value not present), so the recording never shows an empty screen.
+- `demoPause=1500` pauses 1.5s at each row-count assertion, so each result can be read. Apply it in one place (the row-count assertion, which every beat makes exactly once) rather than sprinkling `cy.wait()` through the test - CI still runs the file with no waits at all.
+
+More beats, not a slower take, is what makes a demo longer: a walkthrough of 12 row-count beats takes 52s against 30s for 7.
+
+## Mock services
+
+- One JS file per service under `sas/mocks/sasjs/services/**`, deployed as SASjs stored programs. Shared helpers live in `sas/mocks/sasjs/services/dcMockUtils.js` (`fetchTable`/`parseCsv` for `%webout(FETCH)`, `webOutOpen`/`webOutObj`/`webOutClose` for `%webout`, `mpeFilterMaster`, `mpFilterGenerate`, `mpFilterStore`, `getDdType`, `formatSasValue`, `sasVarsEntry`).
+- Mock data is one JSON file per table at `<drive>/files/<appLoc>/data/<LIBREF>/<table>.json`, written by `services/admin/makedata.js` (which also writes `services/settings.js`). The drive lives under the SASjs Server directory (`sasjs_root/drive`).
+- `makedata.js` self-destructs after a successful run (it deletes itself from the drive so the frontend can detect completion), so re-deploy before re-seeding.
+- **A mock must mirror the real service.** `viewdata.js` once ignored `FILTER_RK` and hard-coded `FILTER_TEXT` to blank, so the viewer's filter silently filtered nothing and its info bar never showed a clause, while the real `viewdata.sas` applies `%mpe_filtermaster` and then `%mp_searchdata`. If a mock is a deliberate stub, say so in a comment and make the divergence explicit - a mock that quietly drops an input turns an E2E spec into a test of nothing.
+
+## Pitfalls
+
+- `sasjs cbd -t server-ci` rebuilds the *mock services* and re-deploys the streamed web app from `client/dist`; it does not rebuild the Angular app.
+- Deleting the drive folder before deploying avoids stale services and stale data.
+- Captured `_webout` JSON, request logs and iteration snapshots belong in `tmp/` (gitignored), never in `sas/mocks/` - see the `dc-sas` skill.
+- The mock server has no auth, so `cy.visit` on `/SASLogon/logout` returns 404 - that is fine, Cypress does not fail on the status code.
+
+## Verification
+
+- `npx cypress run --browser electron --spec <specs>` - all green, and run it more than once when you have just removed a wait.
+- `cd client && npm run lint:check` passes.
+- For a mock change, hit the service directly before writing UI assertions:
+
+      curl -sS -H "Content-Type: application/json" -X POST \
+        --data '{"_program":"/Public/app/dc/services/public/viewdata","SASControlTable":"LIBDS:$41. FILTER_RK:best. SEARCHTYPE:$4 SEARCHVAL:$1000\nDC_JSLIB.MPE_X_SEARCH,0,CHAR,Trench"}' \
+        http://localhost:5000/SASjsApi/stp/execute
+
+  then check `sasparams[0].NOBS`, the row count and `FILTER_TEXT`. Pin those exact numbers in the spec.

.git-hooks/pre-commit

@@ -3,6 +3,42 @@
 # Using `--silent` helps for showing any errs in the first line of the response
 # The first line is picked up by the VS Code GIT UI popup when rc is not 0
 
+# Scan staged changes for secrets before anything else runs.
+# The binary is pinned by the @nogoo9/gitleaks devDependency and lands
+# in node_modules/.bin after npm i. Commits are blocked until it is
+# installed so no clone silently skips the secrets scan.
+gitleaks_bin=node_modules/.bin/gitleaks
+
+if [ ! -x "$gitleaks_bin" ]; then
+    echo "❌ gitleaks not found - run 'npm i' to install it"
+    exit 1
+fi
+
+if ! "$gitleaks_bin" protect --staged --redact; then
+    echo "❌ gitleaks detected a potential secret in your staged changes"
+    echo "Remove the secret, or add a .gitleaksignore entry if it is a false positive"
+    exit 1
+fi
+
+## Avoid large commits
+# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
+size_limit=$((2 * 1024 * 1024)) # 2MB
+# Sum the sizes of the blobs this commit would introduce: the staged
+# added/copied/modified/renamed/typechanged entries in the index.
+# https://git-scm.com/docs/git-cat-file#Documentation/git-cat-file.txt---batch-check
+# Renames are treated as add+delete (--no-renames) and pure mode changes
+# (same blob, different permissions) add no new blob, so both are safe.
+commit_size=$(git diff --cached --raw --no-renames --diff-filter=ACMRT |
+  awk '$3 != $4 { print $4 }' |
+  git cat-file --batch-check='%(objectsize)' |
+  awk '{ s += $1 } END { print s + 0 }')
+
+if [ "$commit_size" -ge "$size_limit" ]; then
+  echo "Commit size is too large: $commit_size > $size_limit"
+  echo "Force commit using --no-verify"
+  exit 1
+fi
+
 if npm run --silent lint:check:silent ; then
     exit 0
 else
@@ -10,14 +46,3 @@ else
     echo "❌ Prettier check failed! We ran lint:fix for you. Please add & commit again."
     exit 1
 fi
-
-## Avoid large commits
-# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
-size_limit=$((2 * 2**20)) # 2mbs
-# https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---disk-usage
-commit_size=$(git rev-list --disk-usage HEAD^..HEAD)
-test "$commit_size" -lt "$size_limit" || (
-  echo "Commit size is too large: $commit_size > $size_limit"
-  echo "Force commit using --no-verify"
-  exit 1
-)

.gitea/workflows/build.yaml

@@ -154,7 +154,7 @@ jobs:
           # Start frontend and run cypress
           # timeout 1800: SIGTERM after 30 min so Cypress can flush video/screenshots
           # before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
-          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
+          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
 
       - name: Zip Cypress videos
         if: always()

.gitea/workflows/release.yaml

@@ -302,8 +302,10 @@ jobs:
         run: |
           cd sas
           cp sasjsbuild/viya.json ../client/dist/viya.json
-          cd ..
-          zip -r frontend.zip ./client/dist
+          # Zip from *inside* dist so the archive holds the frontend files at its
+          # root rather than under a client/dist/ prefix (see #147).
+          cd ../client/dist
+          zip -r ../../frontend.zip .
 
       - name: Release Typedoc
         run: |

CHANGELOG.md

@@ -1,3 +1,47 @@
+# [7.15.0](https://git.datacontroller.io/dc/dc/compare/v7.14.2...v7.15.0) (2026-09-23)
+
+
+### Bug Fixes
+
+* **mocks:** keep special missings in the DIFF, and add the clip recording spec ([d124ce3](https://git.datacontroller.io/dc/dc/commit/d124ce3b36f8f5153e8d7a576d4bec3189524ed5))
+* **mocks:** list a column's own special missing in its dropdown ([b8f16a6](https://git.datacontroller.io/dc/dc/commit/b8f16a63827310f68629f2e016a4fb876593c37c))
+* **mocks:** make the approval path work, and record the approval scene ([12847cf](https://git.datacontroller.io/dc/dc/commit/12847cf07165d5aefa61592815cab0a37245e31f))
+* **validator:** a range rule ignores a missing value ([70dae4b](https://git.datacontroller.io/dc/dc/commit/70dae4b7013ac825eb77d3dc2b622980c0e4af05))
+* **validator:** keep the regular missing in a numeric dropdown source too ([a639bca](https://git.datacontroller.io/dc/dc/commit/a639bca70264e8a6a81795892b9ed63ba58d5f73))
+* **validator:** primary keys are NOT NULL, and a strict dropdown can match a special missing ([586a41a](https://git.datacontroller.io/dc/dc/commit/586a41ad49670ee795284a83cf8535711ce14d2c))
+* **validator:** reject a special missing on a NOT NULL column ([6d85bce](https://git.datacontroller.io/dc/dc/commit/6d85bce2a0c528e8eab9ee4abbade8c682ec3a1b))
+
+
+### Features
+
+* **validator:** compare MINVAL and MAXVAL in SAS's own order ([3467322](https://git.datacontroller.io/dc/dc/commit/3467322e99b0c3f6fda5654d35b507fdb2cb1c22))
+
+## [7.14.2](https://git.datacontroller.io/dc/dc/compare/v7.14.1...v7.14.2) (2026-09-22)
+
+
+### Bug Fixes
+
+* **release:** put frontend files at the root of frontend.zip ([7771a24](https://git.datacontroller.io/dc/dc/commit/7771a24b9ce55751253686885e4848a292817a38)), closes [#147](https://git.datacontroller.io/dc/dc/issues/147)
+* **security:** accept the format-catalog form when validating a libds ([0fa5da8](https://git.datacontroller.io/dc/dc/commit/0fa5da8abfc303ce3b18beb4c01484dcce508439))
+* **security:** escape col-info dropdown and origin-check VA replay ([c9eed6d](https://git.datacontroller.io/dc/dc/commit/c9eed6dae717414b874ef6f3d60312ea0bb5a0da))
+* validate request inputs and add admin gates to public services ([be86000](https://git.datacontroller.io/dc/dc/commit/be86000fe0f0858edcbecf71d426a2ff81257247))
+
+## [7.14.1](https://git.datacontroller.io/dc/dc/compare/v7.14.0...v7.14.1) (2026-09-17)
+
+
+### Bug Fixes
+
+* **admin:** return refreshcatalog output as webout JSON ([1dccf61](https://git.datacontroller.io/dc/dc/commit/1dccf61f6290e50ebc365964fab864cb93f8d1b5))
+* **deps:** bump @sasjs/cli to 4.20.3 ([d301fc2](https://git.datacontroller.io/dc/dc/commit/d301fc2b4dc86b7eb3027b8c8965dd7d1079363e))
+* **deps:** bump @sasjs/core to 5.2.7 ([09f31ce](https://git.datacontroller.io/dc/dc/commit/09f31ce6da1e6a286c4e05ac55fb1d0c801d62f5))
+* **deps:** bump @sasjs/core to 5.2.8 ([c73817f](https://git.datacontroller.io/dc/dc/commit/c73817fb6a8c19d970a19b98d5ef79c05243cc8d))
+* **editors:** declare the mp_validatecol dependency in stagedata and loadfile ([a556be4](https://git.datacontroller.io/dc/dc/commit/a556be4c577020c8e3147de4d3e71ab84fb0d174))
+* **hooks:** make the pre-commit size check actually run ([5383803](https://git.datacontroller.io/dc/dc/commit/53838036e635cd6f33dccc4534d6f258d3b3da5d))
+* **hooks:** require the repo-pinned gitleaks, drop the fallbacks ([fbcde41](https://git.datacontroller.io/dc/dc/commit/fbcde4132147240fed21a88d44f62b68f58cf67c))
+* **security:** escape cell values in status renderers to prevent DOM XSS ([cfd8f06](https://git.datacontroller.io/dc/dc/commit/cfd8f064352940cd57cf19c86504036f0a7edd8d))
+* **security:** validate libds in stagedata and loadfile ([2d31e5a](https://git.datacontroller.io/dc/dc/commit/2d31e5a8b957d30057eb749c2d7f246170af8c8f))
+* **viewdata:** return the full row cap from a table search ([199b57d](https://git.datacontroller.io/dc/dc/commit/199b57d9f9fb4b00a229d23e25e22e5dee471d2a))
+
 # [7.14.0](https://git.datacontroller.io/dc/dc/compare/v7.13.0...v7.14.0) (2026-09-04)
 
 

CONTRIBUTING.md

@@ -4,6 +4,21 @@
 ## Workflow guidelines
 [Wiki Page](https://git.datacontroller.io/dc/dc/wiki/Git-Workflow)
 
+## Git hooks
+
+The repo ships its own git hooks in [`.git-hooks/`](./.git-hooks):
+
+- `pre-commit` - scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)
+- `commit-msg` - verifies the commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) standard
+
+The repo `.npmrc` sets `ignore-scripts=true`, so the `prepare` and `postinstall` lifecycle scripts that would normally set `core.hooksPath` during `npm i` never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
+
+```bash
+git config core.hooksPath ./.git-hooks
+```
+
+The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
+
 ## Dependencies that requires licences
 
 [SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)

client/cypress.config.ts

@@ -38,9 +38,24 @@ export default defineConfig({
       // Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
       // render deterministically regardless of the runner's system locale.
       on('before:browser:launch', (browser, launchOptions) => {
+        // Recording only: the video capture is the browser window, which
+        // contains the Cypress runner - the command log takes the left ~450px
+        // and the app sits in the remaining pane.  Sizing the window is the
+        // only lever that moves the capture, and only for a real Chromium
+        // browser (Electron ignores it).  For a 16:9 pane, solve
+        // (W - 450) / (H - 96) = 16/9: W = 1920 -> H = 923 of window content,
+        // so pass RECORD_WINDOW_SIZE=1920,1010 and crop the pane.
+        // Inert unless the env var is set, so CI is unaffected.
+        if (process.env.RECORD_WINDOW_SIZE && browser.family === 'chromium') {
+          launchOptions.args.push(
+            `--window-size=${process.env.RECORD_WINDOW_SIZE}`
+          )
+        }
+
         if (browser.family === 'chromium' && browser.name !== 'electron') {
           launchOptions.args.push('--lang=en-GB')
         }
+
         return launchOptions
       })
     }

client/cypress/clips/special-missings-clip.cy.ts

@@ -0,0 +1,592 @@
+// Clip script: special missings inside Data Controller's validation rules.
+//
+// This is not a test - it is the recording script for the companion clip, and
+// it deliberately pauses between steps so each beat is readable on video. It
+// lives in cypress/clips (outside cypress/e2e) so the default spec pattern does
+// not pick it up in CI.
+//
+// Record it with:
+//
+//   npx cypress run --spec cypress/clips/special-missings-clip.cy.ts \
+//     --config video=true,viewportWidth=1280,viewportHeight=720
+//
+// The table is TESTDATA.DEMO_01 (seven columns, one rule each):
+//   ID (NOTNULL), AMOUNT (MINVAL 1), SCORE (MAXVAL 100),
+//   REF (SOFTREGEX /^[0-9]+$/), STATUS (SOFTSELECT TESTDATA.DEMO_01.STATUS),
+//   RATING (no rule), LAST_REVIEWED (date9.)
+//
+// One editing session, one submission. The rule scenes are played on row 1 and
+// each value is put back afterwards, so the DIFF that follows carries only the
+// two changes the clip is about; the review screens are reached through the
+// app's own navigation rather than cy.visit, so nothing reloads mid-clip.
+//
+// The beats are deliberately short - the take is played back at ~1.4x on the
+// way out, so a hold that looks tight here reads as a normal pause on the
+// finished clip.
+
+// Caption track: each mark opens a caption and carries the text it shows.
+// The encoder turns consecutive marks into subtitle cues, so the captions are
+// timed by the recording itself rather than by guessed offsets.
+const BEATS = '/tmp/clip-beats.tsv'
+
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+
+// A beat long enough to read on video.
+const beat = (ms = 1500) => cy.wait(ms)
+
+context('special missings clip (DEMO_01)', function () {
+  this.beforeEach(() => {
+    cy.visit(hostUrl + appLocation)
+
+    // The mock estate carries a valid licence key (mock-storage/licence.json),
+    // so the app activates directly with no free-tier banner in shot.
+    cy.get('.nav-tree', { timeout: longerCommandTimeout }).should('exist')
+  })
+
+  it('records the demo in one editing session', () => {
+    cy.writeFile(BEATS, '', { flag: 'w' })
+
+    // ---- Scene 1: the table and its rules ----------------------------------
+    openTableFromTree('testdata', 'demo_01')
+    beat(1200)
+
+    clickOnEdit(() => {
+      // Seven columns, five rules: ID is the key, AMOUNT the floor, SCORE the
+      // ceiling, REF the pattern, STATUS the dropdown, and RATING and
+      // LAST_REVIEWED carry nothing.
+      cy.get('.ht_master tbody tr', { timeout: longerCommandTimeout }).should(
+        'have.length.greaterThan',
+        3
+      )
+      mark(
+        'grid',
+        'TESTDATA.DEMO_01 - eight columns, six rules. ID is the key, and RATING and LAST_REVIEWED carry none.'
+      )
+      beat(2600)
+
+      // ---- Scene 2: typing a special missing ------------------------------
+      // RATING has no rule, so this shows entry on its own. Every beat asserts
+      // the settled cell state before holding, so the recording always rests on
+      // the outcome (flagged or accepted) rather than on a blind pause that
+      // might land before the rule engine has run.
+      mark(
+        'entry',
+        'A numeric cell takes a special missing as a letter or an underscore, with or without a period - row 2 already holds .a.'
+      )
+      typeAndHold(0, 'RATING', 'a', 'accepted') // a single letter is taken
+      mark(
+        'reject',
+        'Two letters, or a letter mixed with a number, are refused. Red means it will not submit.'
+      )
+      typeAndHold(0, 'RATING', 'AB', 'rejected', 2200) // two letters are not
+      typeAndHold(0, 'RATING', '1a', 'rejected', 2200) // nor a number and a letter
+      typeAndHold(0, 'RATING', 'a', 'accepted') // leave it as a special missing
+
+      // ---- Scene 3: NOTNULL refuses both --------------------------------
+      // ID is the key, so NOTNULL applies. A special missing is not a value
+      // here: like a blank, it fails the rule (a real SAS NOT NULL constraint
+      // rejects a special missing as well), and only a number satisfies it -
+      // the original key, put back so the row carries no change into the DIFF.
+      mark(
+        'notnull_blank',
+        "ID is the table's primary key, so NOT NULL is applied to it automatically - a blank fails..."
+      )
+      typeAndHold(0, 'ID', '', 'rejected', 2200) // a blank fails NOTNULL
+      mark(
+        'notnull_missing',
+        '...and so does a special missing: a missing is not a value to NOT NULL.'
+      )
+      typeAndHold(0, 'ID', 'A', 'rejected', 2200) // so does a special missing
+      mark(
+        'notnull_number',
+        'A number satisfies it, and the row key goes back.'
+      )
+      typeAndHold(0, 'ID', '1', 'accepted') // a number is what it wants
+
+      // ---- Scene 4: the pattern still applies -----------------------------
+      // REF carries SOFTREGEX /^[0-9]+$/, and a special missing is not exempt
+      // from it. The amber cell is a soft rule warning rather than a block,
+      // which the caption on this beat says out loud - the pattern itself is
+      // only visible in the cell's native title, which a screencast does not
+      // capture.
+      mark(
+        'softregex',
+        'REF carries SOFTREGEX /^[0-9]+$/. Amber is a soft warning - it warns, it does not block.'
+      )
+      typeIntoCell(0, 'REF', 'A')
+      getCellByHeaderAndRow(0, 'REF').should('have.class', 'dc-warning-cell')
+      beat(3400)
+      typeIntoCell(0, 'REF', '1001') // put the reference back
+      getCellByHeaderAndRow(0, 'REF').should(
+        'not.have.class',
+        'dc-warning-cell'
+      )
+      beat(600)
+
+      // ---- Scene 5: the dropdown lists the missing -------------------------
+      // STATUS carries a SOFTSELECT whose list is taken from the column itself
+      // (the library.member.column form), and that column holds a special
+      // missing - so the dropdown offers it alongside the ordinary values, as
+      // the bare letter SAS produces for it.
+      mark(
+        'dropdown',
+        "STATUS carries a SOFTSELECT, and its list is the column's own values."
+      )
+      openDropdown(0, 'STATUS')
+        .should('have.length', 4)
+        .then(($items: any) => {
+          const texts = [...$items].map((td: any) => td.innerText.trim())
+          expect(texts).to.include('A')
+        })
+      beat(2400)
+
+      mark(
+        'dropdown_pick',
+        'So the special missing the column holds is offered as a bare letter, first - a missing sorts below every number.'
+      )
+      pickFromDropdown('A')
+      getCellByHeaderAndRow(0, 'STATUS').should('contain.text', 'A')
+      beat(1800)
+      typeIntoCell(0, 'STATUS', '1') // put the status back
+      beat(600)
+
+      // ---- Scene 6: a range rule compares in SAS order, missings included
+      mark(
+        'minval',
+        'AMOUNT has MINVAL 1. A missing sorts below every number, so it is below the floor.'
+      )
+      typeAndHold(0, 'AMOUNT', 'A', 'rejected', 2200) // a missing is below the floor
+      mark(
+        'maxval',
+        'SCORE has MAXVAL 100 - the same missing is below the ceiling, so it passes.'
+      )
+      typeAndHold(0, 'SCORE', 'A', 'accepted', 2200) // a missing is below the ceiling
+      mark(
+        'grade',
+        'GRADE takes MINVAL .A and MAXVAL .C. The missings have an order of their own: .B is inside the range, .D is outside it.'
+      )
+      typeAndHold(0, 'GRADE', '.B', 'accepted', 2400) // .B is between .A and .C
+      typeAndHold(0, 'GRADE', '.D', 'rejected', 2400) // .D is above .C
+
+      mark('abort', 'Submitting with an invalid cell aborts.')
+
+      // Submit while AMOUNT is invalid - the modal reports it.
+      submitTable(() => {
+        cy.get('.modal-body', { timeout: longerCommandTimeout }).should(
+          'contain.text',
+          'Invalid Values are Present'
+        )
+        beat(2400)
+        // Close the abort so the editor is clean for the next scene.
+        cy.get('clr-modal.clr-abort-modal .modal-footer button')
+          .contains('Close')
+          .click({ force: true })
+        beat(800)
+      })
+
+      // ---- Scene 7: put the test values back, then make the real change ----
+      mark(
+        'clean',
+        'Back to a clean row. Row 2 already holds .a, so this changes one special missing to another.'
+      )
+      typeIntoCell(0, 'AMOUNT', '120')
+      typeIntoCell(0, 'SCORE', '82')
+      typeIntoCell(0, 'GRADE', '.a')
+      typeIntoCell(0, 'RATING', '4')
+      beat(800)
+
+      // Row 2 (ID 2) already carries a special missing in RATING, so this is a
+      // change from one special missing to another, and the date column gives
+      // the formatted / unformatted switch something to switch.
+      typeAndHold(1, 'RATING', 'B', 'accepted')
+      typeIntoDateCell(1, 'LAST_REVIEWED', '2026-01-15')
+      beat(1500)
+
+      submitTable(() => {
+        cy.get('#submitBtn', { timeout: longerCommandTimeout })
+          .should('exist')
+          .should('not.be.disabled')
+          .click()
+        beat(2500)
+      })
+    })
+
+    // ---- Scene 8: the queue, then the DIFF (in-app navigation) ------------
+    mark('submitted', 'Submitted - the queue shows it waiting for approval.')
+    goToReviewNav()
+    beat(1800)
+
+    // The submit queue lists oldest first, so the row we just created is last.
+    cy.get('app-submitter clr-datagrid clr-dg-row', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .last()
+      .click({ force: true })
+    beat(1800)
+
+    cy.get('app-approve-details .card', { timeout: longerCommandTimeout })
+      .should('exist')
+      .should('be.visible')
+    beat(1000)
+
+    // The DIFF table is wider than the frame, so the two columns that matter
+    // (RATING and LAST_REVIEWED) sit off the right edge until it is scrolled.
+    scrollDiffToEnd()
+    beat(1200)
+
+    // Only those two cells changed: the rule scenes were put back, so the DIFF
+    // is the two changes the clip is about and nothing else.
+    getDiffCell('RATING')
+      .should('contain.text', '.b')
+      .should('have.class', 'ch')
+    getDiffCell('LAST_REVIEWED')
+      .should('contain.text', '15JAN2026')
+      .should('have.class', 'ch')
+    cy.get('app-approve-details .tableCont tbody tr td:not(.ch)').should(
+      'have.length.greaterThan',
+      3
+    )
+    mark(
+      'diff',
+      'The DIFF compares staged with base: one changed row, two changed cells.'
+    )
+    beat(2600)
+
+    // ---- Scene 9: the staged data -----------------------------------------
+    // What the approval is actually acting on: the staged row, still holding
+    // the special missing, before it reaches the base table.
+    mark(
+      'staged',
+      'The staged row, before approval - still holding the special missing.'
+    )
+    clickButton('VIEW STAGED DATA')
+    // 'Basic Submitted Details' is on the staged screen only - asserting
+    // 'Staged Data' alone would be satisfied by the button that was just
+    // clicked, which is how a beat can pass without ever leaving the DIFF.
+    cy.get('body', { timeout: longerCommandTimeout })
+      .should('contain.text', 'Basic Submitted Details')
+      .and('contain.text', 'Base Table')
+    beat(4200)
+    mark('staged_end', '')
+
+    // ---- Scene 10: the approver opens it, and switches the format ----------
+    goToReviewNav()
+    beat(1500)
+    openApproveTab()
+    beat(1500)
+
+    cy.get('app-approve clr-datagrid clr-dg-row a.color-green', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .last()
+      .click({ force: true })
+    beat(1800)
+
+    cy.get('#acceptBtn', { timeout: longerCommandTimeout })
+      .should('exist')
+      .should('not.be.disabled')
+    beat(1000)
+
+    // Same scroll as the submitter view, so the date column is in frame when
+    // the format is switched.
+    scrollDiffToEnd()
+    beat(1200)
+
+    mark(
+      'approve',
+      'The approver opens the same submission. Hovering a changed cell shows the value it replaced.'
+    )
+
+    // Hovering the two changed cells shows what each replaced - that is how the
+    // two special missings are told apart, not just the before/after of one.
+    hoverDiffCell('RATING', 'Original value is: .a')
+    beat(2800)
+
+    mark('hover_date', '...including the date it replaced.')
+    hoverDiffCell('LAST_REVIEWED', 'Original value is: 29FEB2024')
+    beat(2800)
+
+    mark(
+      'toggle',
+      'The formatted / unformatted switch shows the value as SAS stores it: 24121.'
+    )
+    cy.get('.formatted-values-toggle').should('have.text', 'Formatted').click()
+    cy.get('.formatted-values-toggle').should('have.text', 'Unformatted')
+    getDiffCell('LAST_REVIEWED').should('contain.text', '24121')
+    getDiffCell('RATING').should('contain.text', '.b')
+    beat(2800)
+
+    cy.get('.formatted-values-toggle').click()
+    cy.get('.formatted-values-toggle').should('have.text', 'Formatted')
+    getDiffCell('LAST_REVIEWED').should('contain.text', '15JAN2026')
+    beat(1000)
+
+    // ---- Scene 11: approve, and the change is in the history --------------
+    mark('accepted', 'Accepted - the history records it as APPROVED.')
+    cy.get('#acceptBtn').click()
+    cy.url({ timeout: longerCommandTimeout }).should(
+      'include',
+      '/review/history'
+    )
+    cy.get('app-history clr-datagrid clr-dg-row', {
+      timeout: longerCommandTimeout
+    })
+      .should('exist')
+      .first()
+      .should('contain.text', 'APPROVED')
+    beat(2800)
+  })
+})
+
+// ---------------------------------------------------------------------------
+// Helpers (mirrored from the e2e specs so the clip drives the same UI paths)
+// ---------------------------------------------------------------------------
+
+/**
+ * Caption track. Each mark opens a caption and carries the text it shows; the
+ * encoder turns consecutive marks into subtitle cues, so the captions are timed
+ * by the recording itself rather than by guessed offsets.
+ *
+ * The timestamp has to be taken inside a `cy.then()`: Cypress evaluates a
+ * command's arguments when the command is *queued*, so `Date.now()` passed to
+ * `cy.writeFile` directly would give every mark the same value - the moment the
+ * spec body ran.
+ */
+const mark = (label: string, text: string) => {
+  cy.then(() => {
+    cy.writeFile(BEATS, `${label}\t${text}\t${Date.now()}\n`, { flag: 'a+' })
+  })
+}
+
+const typeIntoCell = (rowIndex: number, header: string, value: string) => {
+  getCellByHeaderAndRow(rowIndex, header)
+    .dblclick({ force: true })
+    .then(() => {
+      cy.focused().clear().type(`${value}{enter}`)
+    })
+}
+
+/**
+ * A date-formatted column edits through an HTML date input, where cy.type()
+ * refuses anything but a bare YYYY-MM-DD string (so no {enter} in the same
+ * call). Set the value through the DOM and commit it with the Enter keydown
+ * that Handsontable listens for.
+ */
+const typeIntoDateCell = (rowIndex: number, header: string, value: string) => {
+  getCellByHeaderAndRow(rowIndex, header)
+    .dblclick({ force: true })
+    .then(() => {
+      cy.focused()
+        .then(($i: any) => {
+          const el = $i[0]
+          el.value = value
+          el.dispatchEvent(new Event('input', { bubbles: true }))
+          el.dispatchEvent(new Event('change', { bubbles: true }))
+        })
+        .trigger('keydown', { key: 'Enter', keyCode: 13, which: 13 })
+    })
+}
+
+/**
+ * Opens the selectbox on a cell and returns its list entries. Handsontable
+ * renders the arrow itself (`.htAutocompleteArrow`); the list is a
+ * `.handsontable.listbox` in the app document.
+ */
+const openDropdown = (rowIndex: number, header: string) => {
+  getCellByHeaderAndRow(rowIndex, header).within(() => {
+    cy.get('.htAutocompleteArrow').click({ force: true })
+  })
+
+  return cy.get('.handsontable.listbox td', { timeout: longerCommandTimeout })
+}
+
+/** Picks an entry from the open selectbox. */
+const pickFromDropdown = (value: string) => {
+  cy.get('.handsontable.listbox td').contains(value).click({ force: true })
+}
+
+/** Reaches the review area through the app's own navigation (no reload). */
+const goToReviewNav = () => {
+  cy.get('.nav-link', { timeout: longerCommandTimeout })
+    .contains('REVIEW')
+    .click({ force: true })
+}
+
+/** Opens the APPROVE tab within the review area. */
+const openApproveTab = () => {
+  cy.get('.nav-link', { timeout: longerCommandTimeout })
+    .contains('APPROVE')
+    .click({ force: true })
+}
+
+/**
+ * Clicks a button by its visible text. The match is case-insensitive: the app
+ * uppercases button labels in CSS, so the rendered text and the DOM's
+ * textContent differ.
+ */
+const clickButton = (text: string) => {
+  cy.contains('button', new RegExp(text, 'i'), {
+    timeout: longerCommandTimeout
+  }).click({ force: true })
+}
+
+/**
+ * The DIFF table is wider than the recording frame, so scroll its container to
+ * the end - that is what puts the changed RATING and the date column on screen.
+ */
+const scrollDiffToEnd = () => {
+  cy.get('app-approve-details .tableCont', { timeout: longerCommandTimeout })
+    .should('exist')
+    .scrollTo('right', { duration: 1200 })
+}
+
+/**
+ * The DIFF on the review screen is a plain HTML table (`.tableCont`), not
+ * Handsontable: headers are `th`, cells `td`, and a changed cell also carries
+ * the `ch` class whose tooltip holds the value it replaced.
+ */
+const getDiffCell = (headerText: string) => {
+  return cy
+    .get('app-approve-details .tableCont thead tr th', {
+      timeout: longerCommandTimeout
+    })
+    .should(($ths) => {
+      const texts = [...$ths].map((th) => th.innerText.trim())
+      expect(texts).to.include(headerText)
+    })
+    .then(($ths) => {
+      const index = [...$ths].findIndex(
+        (th) => th.innerText.trim() === headerText
+      )
+
+      return cy
+        .get('app-approve-details .tableCont tbody tr')
+        .first()
+        .then(($tr: any) => $tr[0].childNodes[index])
+        .then((cell) => cy.get(cell))
+    })
+}
+
+/**
+ * Reveals the value a changed DIFF cell replaced, and waits until it is really
+ * on screen.
+ *
+ * Clarity shows the tooltip through CSS :hover, which a synthetic
+ * `trigger('mouseover')` does NOT activate - the tooltip stays
+ * `visibility: hidden`, and a `contain.text` assertion still passes because the
+ * text is in the DOM. So this moves the real mouse (cypress-real-events) and
+ * then asserts the computed style. The first real move after another action can
+ * be swallowed, hence the repeat; if the tooltip ever fails to appear the
+ * recording fails rather than quietly showing nothing.
+ */
+const hoverDiffCell = (headerText: string, expected: string) => {
+  getDiffCell(headerText).realHover()
+  beat(300)
+  getDiffCell(headerText).realHover()
+
+  getDiffCell(headerText)
+    .find('.tooltip-content')
+    .should(($t) => {
+      const style = getComputedStyle($t[0] as HTMLElement)
+      expect(style.visibility, 'tooltip visibility').to.eq('visible')
+      expect(Number(style.opacity), 'tooltip opacity').to.be.greaterThan(0)
+      expect($t[0].textContent || '', 'tooltip text').to.contain(expected)
+    })
+}
+
+/**
+ * Types a value into a cell and holds on the settled result. The rule engine
+ * flags the cell (htInvalid) once the edit commits, so asserting the expected
+ * state before the hold means the recording always rests on the outcome, and
+ * waits for it however long the engine takes.
+ *
+ * @param expected 'rejected' when the rule engine should flag the cell,
+ *   'accepted' when the value should settle unflagged.
+ */
+const typeAndHold = (
+  rowIndex: number,
+  header: string,
+  value: string,
+  expected: 'accepted' | 'rejected',
+  hold = 1800
+) => {
+  typeIntoCell(rowIndex, header, value)
+
+  getCellByHeaderAndRow(rowIndex, header).should(
+    expected === 'rejected' ? 'have.class' : 'not.have.class',
+    'htInvalid'
+  )
+
+  beat(hold)
+}
+
+const getCellByHeaderAndRow = (rowIndex: number, headerText: string) => {
+  return cy
+    .get('.ht_clone_top .htCore thead tr th')
+    .should(($ths) => {
+      const texts = [...$ths].map((th) => th.innerText.trim())
+      expect(texts).to.include(headerText)
+    })
+    .then(($ths) => {
+      const index = [...$ths].findIndex(
+        (th) => th.innerText.trim() === headerText
+      )
+
+      return cy
+        .get('.ht_master tbody tr')
+        .then((rows: any) => rows[rowIndex].childNodes[index])
+        .then((cell) => cy.get(cell))
+    })
+}
+
+const clickOnEdit = (callback?: any) => {
+  cy.get('.btnCtrl button.btn-primary', { timeout: longerCommandTimeout })
+    .click()
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
+const submitTable = (callback?: any) => {
+  cy.get('.btnCtrl button.btn-primary')
+    .click()
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let libNode
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            libNode = node
+            break
+          }
+        }
+
+        cy.get(libNode).within(() => {
+          cy.get('.clr-tree-node-content-container > button').click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+}

client/cypress/e2e/full-table-search.cy.ts

@@ -0,0 +1,385 @@
+// Marks this file as an ES module (rather than a global script) so its
+// top-level consts don't collide, under the TS type-checker, with the same
+// names declared in other spec files — see e.g. viewer-labels.cy.ts.
+export {}
+
+const username = Cypress.env('username')
+const password = Cypress.env('password')
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+const serverType = Cypress.env('serverType')
+const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
+
+// Fixture: MPE_X_SEARCH, a 1000-row demo table built by the mock data builder
+// (sas/mocks/sasjs/services/admin/makedata.js).  It is deterministic, so every
+// count below can be derived by hand:
+//   - 1000 rows, PK 1001-2000, columns in demo order: PRIMARY_KEY_FIELD,
+//     SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE,
+//     EXPEDITION_ID, NOTES
+//   - 300 rows have a site name containing "Trench" (case sensitive, so
+//     "trench" matches nothing)
+//   - 280 rows contain "Halcyon": the site Halcyon Trench (100 rows) plus the
+//     vessel RV Halcyon (200 rows) less the 20 rows that are both
+//   - 3 rows carry "siphonophore" in the middle of their NOTES text, in the
+//     sites Oceanus Rise, Nereid Trench and Halcyon Trench
+//   - 100 rows observed Vampire squid (species is picked by (site+vessel)%10)
+//   - 100 rows are at Oceanus Rise (site is picked by (n%50)/5)
+//   - 9 rows have an expedition id starting EXP-000 (EXP-0001..EXP-0009)
+//   - the first row (PK 1001) is the only one with EXPEDITION_ID EXP-0001
+//   - 12 rows were sounded at exactly 1000m (DEPTH_M = 1000 + (n%90)*37)
+//   - 2 rows have SAMPLE_COUNT 4210 (the only numeric value that is not a
+//     primary key, a depth or a date)
+//   - PK 2000 is the last row of the table
+//   - 200 rows were surveyed by RV Halcyon, 60 of them at a Trench site
+const demoTable = 'mpe_x_search'
+
+// Full-table search: the viewer's search box searches EVERY column of the
+// table, character columns by case-sensitive CONTAINS and numeric columns by
+// exact equality (see %mp_searchdata in the SASjs macro core, and
+// viewdata.sas / services/public/viewdata.js).
+//
+// The whole walkthrough is one test - the table is opened once and every
+// variation is a search in place, which is how a user works and what makes the
+// recording read as a demo rather than a set of page loads.
+//
+// Some searches legitimately match nothing: they pin the case sensitivity, the
+// exact-match numeric semantics and the "no data found" handling.  Those steps
+// are wrapped in negativeStep() and can be skipped when recording the demo, so
+// the video only ever shows searches that return rows:
+//
+//   npx cypress run --browser electron \
+//     --spec cypress/e2e/full-table-search.cy.ts --env skipNegative=true
+//
+// CI runs the file with no env, so every step runs there.
+const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
+const negativeStep = (step: () => void) => {
+  if (!skipNegative) step()
+}
+
+context('full table search tests: ', function () {
+  this.beforeAll(() => {
+    cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
+    // forceLicenceKey=true: applies a valid licence key instead of relying on
+    // an earlier spec in the same run having already done so.
+    cy.loginAndUpdateValidKey(true)
+  })
+
+  this.beforeEach(() => {
+    cy.visit(hostUrl + appLocation, { timeout: longerCommandTimeout })
+
+    // Visit 'home' first, matching filtering.cy.ts/viewer-labels.cy.ts — going
+    // straight to 'view/data' skips the startup-data fetch 'home' triggers,
+    // which leaves the nav tree empty in slower environments.
+    visitPage('home')
+    visitPage('view/data')
+  })
+
+  it('searches the whole table from the viewer', () => {
+    // Full view: every column, and the true row count for the table.
+    openTableFromTree(libraryToOpenIncludes, demoTable)
+    assertRowCount('1,000')
+    assertGridContains(['Nereid Trench', 'RV Pelagia', 'Vampire squid'])
+
+    // Partial character search: "siphonophore" only ever appears in the middle
+    // of the NOTES text, so a match proves the search is not column-specific.
+    // Three rows is narrow enough that the grid grows a horizontal scrollbar
+    // and clips NOTES - the column the match is actually in - so scroll it
+    // into view, assert on what is now on screen, then put the grid back.
+    searchFor('siphonophore')
+    assertRowCount('3')
+    scrollGridTo('right')
+    pauseForDemo()
+    assertGridContains(['Oceanus Rise', 'Nereid Trench', 'Halcyon Trench'])
+    assertGridExcludes(['Kraken Hollow', 'Vostok Deep'])
+    assertGridContains(['bioluminescent siphonophore was filmed here'])
+    scrollGridTo('left')
+
+    // A search spans every column at once: "Halcyon" is both a site (Halcyon
+    // Trench) and a vessel (RV Halcyon), and the 280 matches are the union.
+    searchFor('Halcyon')
+    assertRowCount('280')
+    assertGridContains(['Halcyon Trench', 'RV Halcyon'])
+
+    // A multi-word site name: the search is not limited to single tokens.
+    searchFor('Oceanus Rise')
+    assertRowCount('100')
+    assertGridContains(['Oceanus Rise'])
+
+    // Multi-word values work the same way - species names are two words.
+    searchFor('Vampire squid')
+    assertRowCount('100')
+    assertGridContains(['Vampire squid'])
+    assertGridExcludes(['Yeti crab'])
+
+    // A partial value in a short code column: EXP-000 matches the first nine
+    // expedition ids.
+    searchFor('EXP-000')
+    assertRowCount('9')
+    assertGridContains(['1001', '1009'])
+
+    // A value that is unique in the table: the first row's expedition id.
+    searchFor('EXP-0001')
+    assertRowCount('1')
+    assertGridContains(['1001'])
+
+    // Character search is case sensitive: the same word in the wrong case
+    // matches nothing.
+    negativeStep(() => {
+      searchFor('trench')
+      assertNoData()
+    })
+
+    searchFor('Trench')
+    assertRowCount('300')
+    assertGridContains(['Halcyon Trench'])
+    assertGridExcludes(['Kraken Hollow'])
+
+    // Numeric search is an exact match across the numeric columns.
+    setNumericSearch(true)
+    searchFor('4210')
+    assertRowCount('2')
+    assertGridContains(['1221', '1781', '4210'])
+
+    // The numeric search covers every numeric column, not just the primary
+    // key: 12 rows were sounded at exactly 1000m.
+    searchFor('1000')
+    assertRowCount('12')
+    assertGridContains(['1091'])
+
+    // ...and an exact primary key is a single row.
+    searchFor('2000')
+    assertRowCount('1')
+    assertGridContains(['2000'])
+
+    negativeStep(() => {
+      searchFor('421')
+      assertNoData()
+    })
+
+    // Back to a character search: clearing the box restores the full table.
+    setNumericSearch(false)
+    searchFor('')
+    assertRowCount('1,000')
+
+    // A search runs inside the current filter, not the whole table.
+    openFilterPopup(() => {
+      setFilterWithValue('VESSEL', 'RV Halcyon', () => {
+        assertRowCount('200')
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+
+        searchFor('Trench')
+        assertRowCount('60')
+        assertGridContains(['Halcyon Trench'])
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+
+        // Clearing the search leaves the filter in place.
+        searchFor('')
+        assertRowCount('200')
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+      })
+    })
+
+    negativeStep(() => {
+      searchFor('not-in-this-table')
+      assertNoData()
+    })
+  })
+})
+
+const visitPage = (url: string) => {
+  cy.visit(`${hostUrl}${appLocation}/#/${url}`)
+}
+
+// Recording-only linger.  A beat's row count is the moment its result is on
+// screen, so that is where a demo recording pauses to let it be read.  CI runs
+// with no env, so demoPauseMs is 0 and the spec contains no waits at all:
+//
+//   npx cypress run --browser electron --spec cypress/e2e/full-table-search.cy.ts \
+//     --env skipNegative=true,demoPause=1500
+const demoPauseMs = Number(Cypress.env('demoPause')) || 0
+
+// Linger for a recording.  A no-op unless demoPause is set, which only the
+// recording run does - CI passes no env.
+const pauseForDemo = () => {
+  if (demoPauseMs > 0) cy.wait(demoPauseMs)
+}
+
+// The row/column count the viewer shows next to the table name, e.g.
+// "(1,000 rows, 9 cols)".  The viewer pluralises: a single-row result reads
+// "(1 row, 9 cols)".
+const assertRowCount = (rows: string) => {
+  cy.get('.title-col p', { timeout: longerCommandTimeout }).should(
+    ($el: any) => {
+      const unit = rows === '1' ? 'row' : 'rows'
+      expect($el.text().replace(/\s+/g, ' ')).to.contain(`(${rows} ${unit},`)
+    }
+  )
+
+  pauseForDemo()
+}
+
+// The grid body is a Handsontable .wtHolder, and a narrow result is wider than
+// the grid viewport, which clips the rightmost column (NOTES).  Setting
+// scrollLeft is enough - Handsontable keeps the header clone in step - and the
+// assertion right after proves the scroll landed rather than assuming it.  The
+// browser clamps scrollLeft to (scrollWidth - clientWidth), so that is what
+// scrolling right has to land on.
+const scrollGridTo = (position: 'left' | 'right') => {
+  cy.get('#hotTable .wtHolder')
+    .first()
+    .then(($holder: any) => {
+      const el = $holder[0]
+      const maxScroll = el.scrollWidth - el.clientWidth
+
+      el.scrollLeft = position === 'right' ? el.scrollWidth : 0
+
+      if (position === 'right') {
+        // This step only means anything if the grid really is clipping a
+        // column - fail loudly rather than silently doing nothing.
+        expect(el.scrollWidth).to.be.greaterThan(el.clientWidth)
+        expect(el.scrollLeft).to.equal(maxScroll)
+      } else {
+        expect(el.scrollLeft).to.equal(0)
+      }
+    })
+}
+
+// A search that matches nothing replaces the grid with the viewer's "no data"
+// panel - and must not raise a request error.
+const assertNoData = () => {
+  cy.get('.noData h3', { timeout: longerCommandTimeout }).should(
+    'contain.text',
+    'No data found with given conditions'
+  )
+}
+
+const assertGridContains = (values: string[]) => {
+  cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
+    ($grid: any) => {
+      const text = $grid.text().replace(/\s+/g, ' ')
+      for (const value of values) expect(text).to.contain(value)
+    }
+  )
+}
+
+const assertGridExcludes = (values: string[]) => {
+  cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
+    ($grid: any) => {
+      const text = $grid.text().replace(/\s+/g, ' ')
+      for (const value of values) expect(text).to.not.contain(value)
+    }
+  )
+}
+
+const searchFor = (value: string) => {
+  cy.get('input[name="search-input"]', { timeout: longerCommandTimeout })
+    .should('be.visible')
+    .clear()
+
+  if (value.length > 0) cy.get('input[name="search-input"]').type(value)
+
+  cy.get('input[name="search-input"]').type('{enter}')
+}
+
+const setNumericSearch = (enabled: boolean) => {
+  cy.get('input[name="numeric_check"]', { timeout: longerCommandTimeout }).then(
+    (checkbox: any) => {
+      if (checkbox[0].checked !== enabled) checkbox[0].click()
+    }
+  )
+}
+
+const checkInfoBarIncludes = (text: string) => {
+  cy.get('.infoBar b', { timeout: longerCommandTimeout }).should(($el: any) => {
+    expect($el.text().toLowerCase()).to.contain(text.toLowerCase())
+  })
+}
+
+// Opens the viewer's options menu and clicks through to the filter modal.
+const openFilterPopup = (callback?: any) => {
+  cy.get('.filterSide', { timeout: longerCommandTimeout }).first().click()
+
+  cy.get('.dropdown-menu', { timeout: longerCommandTimeout })
+    .contains('Filter')
+    .click()
+
+  cy.get('#vals_var_id0_0', { timeout: longerCommandTimeout }).should('exist')
+
+  if (callback) callback()
+}
+
+// Fills the first clause of the filter modal (variable, then value) and
+// submits it - the viewer stores the clause, reloads the table with the new
+// filter RK and shows the generated WHERE text in the info bar.
+const setFilterWithValue = (
+  variableValue: string,
+  valueString: string,
+  callback?: any
+) => {
+  cy.get('#vals_var_id0_0').type(variableValue)
+  cy.get('#datalist_vals_var_id0_0 option')
+    .contains(variableValue)
+    .click({ force: true })
+
+  // The soft-select closes its suggestion list shortly after a selection -
+  // wait for that closed state instead of sleeping.
+  cy.get('#datalist_vals_var_id0_0').should('have.class', 'hidden')
+
+  cy.get('#vals_0_0').type(valueString)
+
+  closeAutocompleteList('#vals_0_0')
+  cy.get('#datalist_vals_0_0').should('have.class', 'hidden')
+
+  cy.get('.filter-modal button[aria-label="Ok"]').click()
+
+  if (callback) callback()
+}
+
+// The soft-select inputs open a suggestion list that drops a transparent
+// click-catcher over the modal until it closes, so send the same Escape key a
+// user would press.  `force` is needed because the input sits underneath that
+// click-catcher while the list is open.
+const closeAutocompleteList = (inputSelector: string) => {
+  cy.get(inputSelector).trigger('keyup', { key: 'Escape', force: true })
+}
+
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let viyaLib
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            viyaLib = node
+            break
+          }
+        }
+
+        cy.get(viyaLib).within(() => {
+          cy.get(
+            '.clr-tree-node-content-container .clr-treenode-content p'
+          ).click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+
+  // Selecting the table triggers async SPA routing + a viewdata fetch; wait
+  // for the grid to actually render before any subsequent action.
+  cy.get('#hotTable .ht_clone_top .htCore thead button.changeType', {
+    timeout: longerCommandTimeout
+  }).should('exist')
+}

client/src/app/deploy/deploy.component.ts

@@ -21,8 +21,6 @@ export class DeployComponent implements OnInit {
   public step: number = 0
   public adminGroups: any = []
 
-  public client_id: string = ''
-  public client_secret: string = ''
   public appLoc: string = ''
   public dcPath: string = ''
   public selectedAdminGroup: string = ''
@@ -52,9 +50,6 @@ export class DeployComponent implements OnInit {
     this.sasJs = this.sasService.getSasjsInstance()
     this.sasJsConfig = this.sasService.getSasjsConfig()
     this.appLoc = this.dcAdapterSettings?.appLoc || ''
-    this.client_id = localStorage.getItem('deploy_client_id') || ''
-    this.client_secret = localStorage.getItem('deploy_secret_key') || ''
-    this.dcPath = localStorage.getItem('deploy_dc_loc') || ''
   }
 
   ngOnInit() {

client/src/app/editor/utils/renderers.utils.spec.ts

@@ -1,5 +1,10 @@
 import Handsontable from 'handsontable'
-import { makeNumberFormatRenderer } from './renderers.utils'
+import {
+  makeNumberFormatRenderer,
+  errorRenderer,
+  noSpinnerRenderer,
+  spinnerRenderer
+} from './renderers.utils'
 
 describe('makeNumberFormatRenderer', () => {
   it('renders a numeric cell as EUR currency without changing the value', () => {
@@ -86,3 +91,67 @@ describe('makeNumberFormatRenderer', () => {
     container.remove()
   })
 })
+
+/**
+ * DOM-injection reproduction mirroring the editor's cell-render cycle.
+ * During dynamic cell validation the editor applies one of the status
+ * renderers to a cell via setCellMeta + hot.render().  Those renderers paint
+ * the cell value with td.innerHTML, so a value containing markup is injected
+ * and executed (the <img onerror> fires in the browser).  The value can come
+ * straight from a dataset row served by the getdata stored program, or from a
+ * typed edit.  These fail on the vulnerable implementation and pass once the
+ * renderer escapes the value.
+ */
+describe('grid cell renderers do not inject raw HTML', () => {
+  const maliciousValue = '<img src=x onerror=alert(1)>'
+
+  // Seed a real Handsontable grid with the payload as a loaded cell value,
+  // then apply the given status renderer and render — exactly the sequence the
+  // editor uses during the dynamic-validation cycle.
+  const renderWith = (
+    renderer: (
+      i: any,
+      td: any,
+      r: number,
+      c: number,
+      p: any,
+      v: any,
+      cp: any
+    ) => any
+  ) => {
+    const container = document.createElement('div')
+    document.body.appendChild(container)
+    const hot = new Handsontable(container, {
+      data: [{ SOME_CHAR: maliciousValue }],
+      columns: [{ data: 'SOME_CHAR', type: 'text' }],
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+    hot.setCellMeta(0, 0, 'renderer', renderer)
+    hot.render()
+    const td: HTMLTableCellElement | null = hot.getCell(0, 0)
+    hot.destroy()
+    container.remove()
+    return td
+  }
+
+  // A vulnerable renderer turns the value into a real <img> element with an
+  // onerror handler (proven by the browser firing alert(1)).  A safe
+  // renderer leaves no such element.  Asserting on the parsed DOM rather
+  // than the raw string avoids false passes from browser attribute normalising.
+  const assertNoInjectedElement = (td: HTMLTableCellElement | null) => {
+    expect(td?.querySelector('img[onerror]')).toBeNull()
+  }
+
+  it('noSpinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(noSpinnerRenderer))
+  })
+
+  it('errorRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(errorRenderer))
+  })
+
+  it('spinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(spinnerRenderer))
+  })
+})

client/src/app/editor/utils/renderers.utils.ts

@@ -1,5 +1,23 @@
 import Handsontable from 'handsontable'
 
+/**
+ * Returns string-safe text of any value so it can be assigned to innerHTML.
+ * The cell values painted by the status renderers are user/DB-controlled,
+ * so they must never be parsed as HTML by the browser — escaping turns any
+ * embedded markup into inert text.
+ */
+const escapeHtml = (value: any): string =>
+  String(value ?? '').replace(/[&<>"']/g, (char) => {
+    const entities: Record<string, string> = {
+      '&': '&amp;',
+      '<': '&lt;',
+      '>': '&gt;',
+      '"': '&quot;',
+      "'": '&#39;'
+    }
+    return entities[char]
+  })
+
 /**
  * Builds a display-only HOT renderer that formats numeric cell values using
  * Intl.NumberFormat. The stored/submitted value is never changed — only the
@@ -67,9 +85,7 @@ export const errorRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
+  td.innerHTML = `${escapeHtml(value)} <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
 
   return td
 }
@@ -89,7 +105,7 @@ export const noSpinnerRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = value ? value : ''
+  td.innerHTML = escapeHtml(value)
 
   return td
 }
@@ -110,9 +126,7 @@ export const spinnerRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <span class="spinner spinner-sm vertical-align-middle"></span>`
+  td.innerHTML = `${escapeHtml(value)} <span class="spinner spinner-sm vertical-align-middle"></span>`
 
   return td
 }

client/src/app/services/va-messaging.service.ts

@@ -138,13 +138,38 @@ export class VaMessagingService {
     this.earlyDrained = true
     const captured = (window as unknown as { __vaLastMessage?: any })
       .__vaLastMessage
-    const parsed = this.parseData(captured && captured.data)
+    // The early listener (va-early.js) captures from any origin, so re-apply
+    // the live-path trust rule here before acting on it: only replay a message
+    // that came from our own origin or from the frame that embedded us (whose
+    // URL is document.referrer). Without this, a same-origin sibling frame
+    // could inject a crafted DDC message that the live isTrustedSource check
+    // would have rejected.
+    if (!captured || typeof captured.origin !== 'string') return
+    if (!this.isTrustedEarlyOrigin(captured.origin)) return
+    const parsed = this.parseData(captured.data)
     if (!parsed) return
     this.resultName = parsed.resultName
-    if (captured.origin) this.parentOrigin = captured.origin
+    this.parentOrigin = captured.origin
     callback(parsed)
   }
 
+  /**
+   * Origin check for the pre-bootstrap replay. Mirrors isTrustedSource: the
+   * live path trusts a message whose event.source IS the parent frame; for a
+   * captured message we cannot reference its source Window, so we trust an
+   * origin that is this window's origin or the embedding frame's origin
+   * (document.referrer). Unverifiable/absent referrer -> reject.
+   */
+  private isTrustedEarlyOrigin(origin: string): boolean {
+    if (origin === window.location.origin) return true
+    if (!document.referrer) return false
+    try {
+      return new URL(document.referrer).origin === origin
+    } catch {
+      return false
+    }
+  }
+
   /**
    * Parses a raw window MessageEvent into a VaMessage, or null when it is not a
    * recognisable DDC message (e.g. unrelated postMessage traffic).

client/src/app/shared/dc-validator/dc-validator.ts

@@ -92,14 +92,49 @@ export class DcValidator {
     this.rules.push({ ...EDIT_STATUS_COLUMN_RULE })
     this.hiddenColumns.push(this.rules.length - 1)
 
-    this.dqrules = dqRules
+    this.dqrules = [...dqRules]
     this.dqdata = dqData
     this.primaryKeys = sasparams.PK.split(' ')
 
+    // A primary key is NOT NULL by definition, so it must reject a blank and
+    // a special missing (".A"-".Z", "._") even when the target table carries
+    // no physical NOT NULL constraint and MPE_VALIDATIONS has no NOTNULL rule
+    // for it.  Synthesised here so every keyed table gets it, and so the grid,
+    // the edit-record modal and Excel upload validation all see the same rule.
+    this.addPrimaryKeyNotNullRules()
+
     this.updateDqData()
     this.setupValidations()
   }
 
+  /**
+   * Adds a NOTNULL rule for each primary key column that does not already
+   * have one.  A primary key identifies the row, so a blank or a special
+   * missing there is never valid.
+   */
+  private addPrimaryKeyNotNullRules(): void {
+    for (const pk of this.primaryKeys) {
+      if (!pk) continue
+
+      // A buskey can name a column the table no longer has - do not
+      // synthesise a rule for a column that is not in the grid.
+      if (!this.rules.some((rule) => rule.data === pk)) continue
+
+      const hasNotNull = this.dqrules.some(
+        (rule) => rule.BASE_COL === pk && rule.RULE_TYPE === 'NOTNULL'
+      )
+
+      if (!hasNotNull) {
+        this.dqrules.push({
+          BASE_COL: pk,
+          RULE_TYPE: 'NOTNULL',
+          RULE_VALUE: '',
+          X: 1
+        })
+      }
+    }
+  }
+
   registerCustomEditors() {
     Handsontable.editors.registerEditor(
       'autocomplete.custom',
@@ -283,7 +318,14 @@ export class DcValidator {
              * So we will convert it before pushing to array.
              */
             if (rule.type && rule.type === 'numeric') {
-              details.push(Number(data['RULE_DATA']))
+              // A special missing reaches us as a bare letter ("A", "_"), and
+              // the regular missing as "." - Number() would turn either into
+              // NaN, so a strict (HARDSELECT) dropdown could never accept a
+              // value the column actually holds.  Keep them as they are.
+              const rawValue = data['RULE_DATA']
+              details.push(
+                isSpecialMissing(rawValue) ? rawValue : Number(rawValue)
+              )
             } else {
               details.push(data['RULE_DATA'])
             }

client/src/app/shared/dc-validator/tests/dc-validator.spec.ts

@@ -118,7 +118,12 @@ describe('DC Validator', () => {
     expect(dcValidator.getRule('SOME_TIME')).toBeUndefined()
 
     // Test data quality functions
-    expect(dcValidator.getDqDetails()).toHaveSize(dqRules.length)
+    // dqRules + 2 synthesised rules: the SOFTSELECT rule updateDqData()
+    // derives for SOME_DROPDOWN out of dqdata, and the NOTNULL rule
+    // addPrimaryKeyNotNullRules() gives the primary key column (example_dqRules
+    // has none for it).  Note the constructor copies dqRules rather than
+    // aliasing it, so this array is no longer mutated by construction.
+    expect(dcValidator.getDqDetails()).toHaveSize(dqRules.length + 2)
     expect(dcValidator.getDqDetails('non_existant')).toHaveSize(0)
     expect(dcValidator.getDqDetails('SOME_NUM')).toHaveSize(2)
     expect(dcValidator.isDqCol('SOME_NUM')).toBeTrue()
@@ -133,6 +138,91 @@ describe('DC Validator', () => {
     ])
   })
 
+  it('treats the primary key column as NOT NULL, even with no NOTNULL rule configured', () => {
+    const sasparams: SASParam = example_sasparams
+    const cols: Col[] = example_cols
+    const dqRules: DQRule[] = example_dqRules // no NOTNULL for PRIMARY_KEY_FIELD
+    const dqData: DQData[] = example_dqData
+    const $dataFormats: $DataFormats = example_dataformats
+
+    const dcValidator: DcValidator = new DcValidator(
+      sasparams,
+      $dataFormats,
+      cols,
+      dqRules,
+      dqData
+    )
+
+    const pkRules = dcValidator.getDqDetails('PRIMARY_KEY_FIELD')
+    expect(pkRules.some((rule) => rule.RULE_TYPE === 'NOTNULL')).toBeTrue()
+
+    const pkRule = dcValidator.getRule('PRIMARY_KEY_FIELD')
+
+    // A primary key identifies the row, so neither a blank nor a special
+    // missing can satisfy it.
+    dcValidator.executeHotValidator(pkRule!, null, (valid: boolean) => {
+      expect(valid).toBeFalse()
+    })
+    dcValidator.executeHotValidator(pkRule!, 'A', (valid: boolean) => {
+      expect(valid).toBeFalse()
+    })
+    dcValidator.executeHotValidator(pkRule!, 5, (valid: boolean) => {
+      expect(valid).toBeTrue()
+    })
+  })
+
+  it('keeps a special missing in a numeric dropdown source, so a strict rule can match it', () => {
+    const dqData: DQData[] = [
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: 1,
+        SELECTBOX_ORDER: 1
+      },
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: 'A',
+        SELECTBOX_ORDER: 2
+      },
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: '_',
+        SELECTBOX_ORDER: 3
+      },
+      {
+        BASE_COL: 'SOME_NUM',
+        RULE_VALUE: 'SOME_NUM',
+        RULE_DATA: '.',
+        SELECTBOX_ORDER: 4
+      }
+    ] as DQData[]
+
+    const dcValidator: DcValidator = new DcValidator(
+      example_sasparams,
+      example_dataformats,
+      example_cols,
+      example_dqRules,
+      dqData
+    )
+
+    // SOME_NUM is numeric and carries a HARDSELECT_HOOK, so its dropdown
+    // source comes from dqdata.  Number() would have NaN'd the special
+    // missings, leaving the strict membership test unable to match a value
+    // the column actually holds.
+    const source = dcValidator.getDqDropdownSource(
+      dcValidator.getRule('SOME_NUM')!
+    )
+
+    expect(source[0]).toEqual(1)
+    expect(source[1]).toEqual('A')
+    expect(source[2]).toEqual('_')
+    // the regular missing is a dropdown option too, not a NaN
+    expect(source[3]).toEqual('.')
+    expect(source.some((value) => Number.isNaN(value as number))).toBeFalse()
+  })
+
   it('should test hot validator', () => {
     const sasparams: SASParam = example_sasparams
     const cols: Col[] = example_cols
@@ -163,21 +253,22 @@ describe('DC Validator', () => {
     dcValidator.executeHotValidator(someNumRule!, 'ss', (valid: boolean) => {
       expect(valid).toBeFalse()
     })
-    //Special missings
+    // Special missings - a SAS NOT NULL (or primary key) constraint rejects
+    // these, so the rule must reject them too: they are NULL, not values.
     dcValidator.executeHotValidator(someNumRule!, 's', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '.s', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '.', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '..', (valid: boolean) => {
       expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(someNumRule!, '._', (valid: boolean) => {
-      expect(valid).toBeTrue()
+      expect(valid).toBeFalse()
     })
 
     // MINVAL, MAXVAL Validation
@@ -193,7 +284,7 @@ describe('DC Validator', () => {
       expect(valid).toBeFalse()
     })
     dcValidator.executeHotValidator(shortNumRule!, 's', (valid: boolean) => {
-      expect(valid).toBeFalse() // Special missings are lowest numbers, if any MINVAL is set, special missing is always lower
+      expect(valid).toBeFalse() // Special missings are the lowest numbers, so any MINVAL is above them
     })
 
     // CASE validation

client/src/app/shared/dc-validator/tests/dqValidation.spec.ts

@@ -24,7 +24,44 @@ describe('DC Validator - dq validation', () => {
     expect(dqValidate(dqRules, invalidValue)).toBeFalse()
     expect(dqValidate(dqRules, invalidStringValue)).toBeFalse()
     expect(dqValidate(dqRules, numericStringValue)).toBeTrue()
+    // A missing sorts below every number, so it is below the floor.
     expect(dqValidate(dqRules, numericSpecialMissingValue)).toBeFalse()
+    expect(dqValidate(dqRules, null)).toBeFalse()
+    expect(dqValidate(dqRules, undefined)).toBeFalse()
+  })
+
+  it('should order the missing values in a range rule as SAS does', () => {
+    const missingRange: DQRule[] = [
+      { BASE_COL: 'test', RULE_TYPE: 'MINVAL', RULE_VALUE: '.A', X: 0 },
+      { BASE_COL: 'test', RULE_TYPE: 'MAXVAL', RULE_VALUE: '.C', X: 0 }
+    ]
+
+    // Inside the range of missings
+    expect(dqValidate(missingRange, '.A')).toBeTrue()
+    expect(dqValidate(missingRange, '.B')).toBeTrue()
+    expect(dqValidate(missingRange, '.C')).toBeTrue()
+    expect(dqValidate(missingRange, 'b')).toBeTrue()
+
+    // Outside it - above the ceiling
+    expect(dqValidate(missingRange, '.D')).toBeFalse()
+    expect(dqValidate(missingRange, 'z')).toBeFalse()
+
+    // Outside it - below the floor.  The regular missing sits between ._ and .A
+    expect(dqValidate(missingRange, '._')).toBeFalse()
+    expect(dqValidate(missingRange, null)).toBeFalse()
+
+    // Every number sorts above every missing, so it is above the ceiling
+    expect(dqValidate(missingRange, 0)).toBeFalse()
+    expect(dqValidate(missingRange, 5)).toBeFalse()
+
+    // A floor of .A alone still lets the numbers through: they are above it
+    const missingFloor: DQRule[] = [
+      { BASE_COL: 'test', RULE_TYPE: 'MINVAL', RULE_VALUE: '.A', X: 0 }
+    ]
+    expect(dqValidate(missingFloor, '.A')).toBeTrue()
+    expect(dqValidate(missingFloor, '.Z')).toBeTrue()
+    expect(dqValidate(missingFloor, 5)).toBeTrue()
+    expect(dqValidate(missingFloor, '._')).toBeFalse()
   })
 
   it('should validate MAXVAL value', () => {
@@ -49,7 +86,10 @@ describe('DC Validator - dq validation', () => {
     expect(dqValidate(dqRules, invalidValue)).toBeFalse()
     expect(dqValidate(dqRules, invalidStringValue)).toBeFalse()
     expect(dqValidate(dqRules, numericStringValue)).toBeTrue()
+    // A missing sorts below every number, so it is below the ceiling
     expect(dqValidate(dqRules, numericSpecialMissingValue)).toBeTrue()
+    expect(dqValidate(dqRules, null)).toBeTrue()
+    expect(dqValidate(dqRules, undefined)).toBeTrue()
   })
 
   it('should validate UPCASE value', () => {
@@ -111,6 +151,43 @@ describe('DC Validator - dq validation', () => {
     expect(dqValidate(dqRules, invalidValue2)).toBeFalse()
   })
 
+  it('should reject a special missing on a numeric column (a SAS NOT NULL constraint does)', () => {
+    const dqRules: DQRule[] = [
+      {
+        BASE_COL: 'test',
+        RULE_TYPE: 'NOTNULL',
+        RULE_VALUE: ' ',
+        X: 0
+      }
+    ]
+
+    // The values a real SAS service delivers for a numeric column.
+    expect(dqValidate(dqRules, 'A', true)).toBeFalse()
+    expect(dqValidate(dqRules, '_', true)).toBeFalse()
+    expect(dqValidate(dqRules, '.', true)).toBeFalse()
+    // Ordinary numbers and numeric strings still pass.
+    expect(dqValidate(dqRules, 5, true)).toBeTrue()
+    expect(dqValidate(dqRules, '5', true)).toBeTrue()
+  })
+
+  it('should not reject a single letter on a character column', () => {
+    const dqRules: DQRule[] = [
+      {
+        BASE_COL: 'test',
+        RULE_TYPE: 'NOTNULL',
+        RULE_VALUE: ' ',
+        X: 0
+      }
+    ]
+
+    // There is no special-missing concept on a character column - a lone
+    // letter is ordinary data.
+    expect(dqValidate(dqRules, 'A')).toBeTrue()
+    expect(dqValidate(dqRules, '_')).toBeTrue()
+    expect(dqValidate(dqRules, '.')).toBeTrue()
+    expect(dqValidate(dqRules, '', false)).toBeFalse()
+  })
+
   it('should return true if rule not found', () => {
     const validValue = 5
 

client/src/app/shared/dc-validator/validations/dq-validation.ts

@@ -1,8 +1,49 @@
 import { DQRule } from '../models/dq-rules.model'
-import { specialMissingNumericValidator } from './hot-custom-validators'
+import { isSpecialMissing } from '@sasjs/utils/input/validators'
 import { isRegexRuleExempt } from '../utils/isRegexRuleExempt'
 import { parseRegexRule } from '../utils/parseRegexRule'
 
+/**
+ * A SAS numeric variable's values have a total order, and its missing values sit
+ * below every non-missing value.  The missing values are themselves ordered:
+ * `._` is the lowest, then the regular missing, then `.A` through `.Z`.
+ *
+ * A range rule compares in that order.  That is what makes a range of missings
+ * meaningful - with `MINVAL .A` and `MAXVAL .C`, `.B` is inside the range and `.D`
+ * is outside it - and it is also why a special missing fails a numeric floor: it
+ * sorts below every number.
+ *
+ * The key is a pair: the class (0 for a missing, 1 for a number, so that every
+ * number sorts above every missing) and the position within that class.  A value
+ * that is neither a number nor a missing has no place in the order and gets null,
+ * which no rule accepts.
+ */
+const sasNumericOrderKey = (value: any): [number, number] | null => {
+  if (value === undefined || value === null || value === '') return [0, 1] // regular missing
+
+  if (typeof value === 'string') {
+    const upper = value.trim().toUpperCase()
+
+    if (upper === '.' || upper === '') return [0, 1] // regular missing
+    if (upper === '._' || upper === '_') return [0, 0] // the lowest missing
+    if (/^\.?[A-Z]$/.test(upper))
+      return [0, 2 + (upper.charCodeAt(upper.length - 1) - 65)] // .A .. .Z
+
+    const numValue = parseFloat(upper)
+
+    return isNaN(numValue) ? null : [1, numValue]
+  }
+
+  const numValue = Number(value)
+
+  return isNaN(numValue) ? null : [1, numValue]
+}
+
+const compareSasNumericOrder = (
+  a: [number, number],
+  b: [number, number]
+): number => (a[0] !== b[0] ? a[0] - b[0] : a[1] - b[1])
+
 const dqValidation: {
   [key: string]: (
     value: any,
@@ -33,25 +74,39 @@ const dqValidation: {
     return true
   },
   MINVAL: (value: any, ruleValue: string | number): boolean => {
-    const isValidNumeric = specialMissingNumericValidator(value)
-    const numValue = parseFloat(value)
+    const valueKey = sasNumericOrderKey(value)
+    const ruleKey = sasNumericOrderKey(ruleValue)
 
-    // If it's validNumeric and it is NaN it means it is special numeric, and those are always less then any
-    // min value set
-    if (isValidNumeric && isNaN(numValue)) return false
+    // A value that is neither a number nor a missing has no place in the order,
+    // so nothing satisfies the rule.
+    if (!valueKey || !ruleKey) return false
 
-    return numValue >= Number(ruleValue.toString())
+    return compareSasNumericOrder(valueKey, ruleKey) >= 0
   },
   MAXVAL: (value: any, ruleValue: string | number): boolean => {
-    const isValidNumeric = specialMissingNumericValidator(value)
-    const numValue = parseFloat(value)
+    const valueKey = sasNumericOrderKey(value)
+    const ruleKey = sasNumericOrderKey(ruleValue)
 
-    if (isValidNumeric && isNaN(numValue)) return true
+    if (!valueKey || !ruleKey) return false
 
-    return numValue <= Number(ruleValue.toString())
+    return compareSasNumericOrder(valueKey, ruleKey) <= 0
   },
-  NOTNULL: (value: any, ruleValue: string | number): boolean => {
-    return value !== undefined && value !== null && value.toString().length > 0
+  NOTNULL: (
+    value: any,
+    ruleValue: string | number,
+    isNumeric: boolean = false
+  ): boolean => {
+    if (value === undefined || value === null) return false
+
+    // A special missing (.A-.Z, ._) is NULL as far as a SAS NOT NULL (or
+    // primary key) constraint is concerned - an insert carrying one is
+    // rejected with an integrity constraint error - so the rule must reject
+    // it too, or the editor would accept a value the target table refuses.
+    // Numeric columns only: a lone letter is ordinary data on a character
+    // column.
+    if (isNumeric && isSpecialMissing(value)) return false
+
+    return value.toString().length > 0
   },
   // Pattern is used as authored, not auto-anchored — a rule author who
   // wants a full-value match must write ^...$ themselves.

client/src/app/shared/utils/col-info-html.spec.ts

@@ -149,3 +149,75 @@ describe('buildColInfoHtml', () => {
     )
   })
 })
+
+/**
+ * DOM-injection reproduction for the column-info dropdown.
+ * buildColInfoHtml interpolates server/DB-controlled values (column label,
+ * format, and DQ RULE_VALUE regex/formula strings) into a string that the
+ * viewer/editor assign to raw DOM `elem.innerHTML` - so a value containing
+ * markup (e.g. a HARDREGEX RULE_VALUE of `<img src=x onerror=alert(1)>`) is
+ * parsed and executed in the browser of whoever opens the info dropdown.
+ * These tests fail on the vulnerable implementation and pass once each field
+ * is escaped.
+ */
+describe('buildColInfoHtml escapes rather than injecting raw HTML', () => {
+  const malicious = '<img src=x onerror=alert(1)>'
+
+  const info: DataFormat = {
+    format: malicious,
+    label: malicious,
+    length: '8',
+    type: 'N'
+  }
+
+  // Parse the returned string the same way the callers do (innerHTML on a
+  // real element) and assert no scriptable element survived.
+  const parseInto = (html: string): HTMLElement => {
+    const host = document.createElement('div')
+    host.innerHTML = html
+    return host
+  }
+
+  const assertNoInjectedElement = (html: string) => {
+    const host = parseInto(html)
+    expect(host.querySelector('img[onerror]')).toBeNull()
+    host.remove()
+  }
+
+  it('is inert for a colInfo whose label and format carry markup', () => {
+    assertNoInjectedElement(buildColInfoHtml('SOMECHAR', info))
+  })
+
+  it('escapes the column NAME', () => {
+    const html = buildColInfoHtml(malicious, {
+      format: '$8.',
+      label: 'safe',
+      length: '8',
+      type: 'C'
+    })
+    // < and > must not survive as markup in the NAME position
+    expect(html).not.toContain(malicious)
+    assertNoInjectedElement(html)
+  })
+
+  it('escapes a HARDREGEX RULE_VALUE', () => {
+    assertNoInjectedElement(
+      buildColInfoHtml('SOMECHAR', info, malicious, undefined, undefined)
+    )
+  })
+
+  it('escapes a SOFTREGEX RULE_VALUE', () => {
+    assertNoInjectedElement(
+      buildColInfoHtml('SOMECHAR', info, undefined, malicious, undefined)
+    )
+  })
+
+  it('escapes a formula RULE_VALUE (with and without a leading =)', () => {
+    assertNoInjectedElement(
+      buildColInfoHtml('SOMECHAR', info, undefined, undefined, malicious)
+    )
+    assertNoInjectedElement(
+      buildColInfoHtml('SOMECHAR', info, undefined, undefined, `=${malicious}`)
+    )
+  })
+})

client/src/app/shared/utils/col-info-html.ts

@@ -1,9 +1,35 @@
 import { DataFormat } from '../../models/sas/common/DateFormat'
 
+/**
+ * Returns string-safe text of any value so it can be concatenated into a
+ * string that is later assigned to raw DOM innerHTML.  The column metadata
+ * (label/format) and DQ RULE_VALUE strings (regex/formula) are DB-controlled -
+ * a validation-rule author can store markup such as
+ * `<img src=x onerror=...>` in a HARDREGEX value or a column label - so they
+ * must never be parsed as HTML by the browser.  Escaping turns any embedded
+ * markup into inert text.
+ */
+const escapeHtml = (value: any): string =>
+  String(value ?? '').replace(/[&<>"']/g, (char) => {
+    const entities: Record<string, string> = {
+      '&': '&amp;',
+      '<': '&lt;',
+      '>': '&gt;',
+      '"': '&quot;',
+      "'": '&#39;'
+    }
+    return entities[char]
+  })
+
 /**
  * Builds the HTML shown in a column-header "info" dropdown item (viewer and
  * editor). NAME is listed first so it's visible regardless of whether
  * headers are currently displayed as NAME or LABEL.
+ *
+ * The returned string is assigned to raw DOM `elem.innerHTML` by both callers
+ * (viewer.component.ts / editor.component.ts) - every field interpolated below
+ * is therefore escaped via escapeHtml, since no Angular sanitizer runs on a
+ * raw innerHTML assignment.
  */
 export function buildColInfoHtml(
   colName: string,
@@ -14,16 +40,16 @@ export function buildColInfoHtml(
 ): string {
   if (!colInfo) return 'No info found'
 
-  let html = `NAME: ${colName}<br>LABEL: ${colInfo.label}<br>TYPE: ${colInfo.type}<br>LENGTH: ${colInfo.length}<br>FORMAT: ${colInfo.format}`
+  let html = `NAME: ${escapeHtml(colName)}<br>LABEL: ${escapeHtml(colInfo.label)}<br>TYPE: ${escapeHtml(colInfo.type)}<br>LENGTH: ${escapeHtml(colInfo.length)}<br>FORMAT: ${escapeHtml(colInfo.format)}`
 
   // Only ever one REGEX rule is applied per column: when both HARDREGEX
   // and SOFTREGEX exist, SOFTREGEX is ignored entirely (same precedence as
   // makeRegexWarningRenderer / DcValidator.failsSoftRegex).  Show only the
   // rule that is applied.
   if (hardRegexValue) {
-    html += `<br>HARDREGEX: ${hardRegexValue}`
+    html += `<br>HARDREGEX: ${escapeHtml(hardRegexValue)}`
   } else if (softRegexValue) {
-    html += `<br>SOFTREGEX: ${softRegexValue}`
+    html += `<br>SOFTREGEX: ${escapeHtml(softRegexValue)}`
   }
 
   // '√x=' stands in for a text label here - HARDFORMULA vs SOFTFORMULA is
@@ -36,7 +62,7 @@ export function buildColInfoHtml(
     const formula = formulaValue.startsWith('=')
       ? formulaValue.slice(1)
       : formulaValue
-    html += `<br>√x=${formula}`
+    html += `<br>√x=${escapeHtml(formula)}`
   }
 
   return html

package-lock.json

@@ -1,14 +1,15 @@
 {
   "name": "dcfrontend",
-  "version": "7.12.0",
+  "version": "7.14.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "dcfrontend",
-      "version": "7.12.0",
+      "version": "7.14.0",
       "hasInstallScript": true,
       "devDependencies": {
+        "@nogoo9/gitleaks": "8.30.1-post.2",
         "@saithodev/semantic-release-gitea": "^2.1.0",
         "@semantic-release/changelog": "^6.0.3",
         "@semantic-release/commit-analyzer": "13.0.1",
@@ -119,6 +120,156 @@
         "node": ">= 8"
       }
     },
+    "node_modules/@nogoo9/gitleaks": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks/-/gitleaks-8.30.1-post.2.tgz",
+      "integrity": "sha512-WkQPfryKH8w9DEqJ1Bxw3ex9aXl0M8cxDDt1ATvyieaMvWiGamGXHNLZaJeqECInVfxNPEiZA4Few8Uwtuj1wQ==",
+      "dev": true,
+      "license": "MIT",
+      "bin": {
+        "gitleaks": "src/index.js"
+      },
+      "engines": {
+        "node": ">=18"
+      },
+      "optionalDependencies": {
+        "@nogoo9/gitleaks-darwin-arm64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-darwin-x64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-arm": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-arm64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-x32": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-x64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-windows-arm64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-windows-x32": "8.30.1-post.2",
+        "@nogoo9/gitleaks-windows-x64": "8.30.1-post.2"
+      }
+    },
+    "node_modules/@nogoo9/gitleaks-darwin-arm64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-darwin-arm64/-/gitleaks-darwin-arm64-8.30.1-post.2.tgz",
+      "integrity": "sha512-3j1zSvCKY7N4RnSl+oJrnrLZAY6HihD3uyi32qYrdIXH5hgVjmKiJhvGyoaV7DNw/4NdN3UsadrTM1u5M7iCuw==",
+      "cpu": [
+        "arm64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-darwin-x64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-darwin-x64/-/gitleaks-darwin-x64-8.30.1-post.2.tgz",
+      "integrity": "sha512-EbWRmrWdwRtzfO4ivYXA2ZKcf4aqVZFGQVu2HJjCFjRNryyMfou4Hw3tdZc5tRbdSE0yiwaAXJ8lPazApd3iGQ==",
+      "cpu": [
+        "x64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-arm": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-arm/-/gitleaks-linux-arm-8.30.1-post.2.tgz",
+      "integrity": "sha512-a5aZc5WQnBmNz5CCOPd0MT75csSBUEYMEVIHR6p56TOt4lj8h+IeQ5VTKG+f9j9nU+hmG2e//j4hLcbZZkCspw==",
+      "cpu": [
+        "arm"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-arm64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-arm64/-/gitleaks-linux-arm64-8.30.1-post.2.tgz",
+      "integrity": "sha512-XJzvhOEsqr6nJrBuLzDb+qBfdLjssKv/7gqQNhojd8LPM0/aEiC380kYuM9mFCXF3o3PpD4IXv/Jvemw5JJQGA==",
+      "cpu": [
+        "arm64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-x32": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-x32/-/gitleaks-linux-x32-8.30.1-post.2.tgz",
+      "integrity": "sha512-d20jRC6HCzc9J79Ti8JHFc3X3iJktF78NaUEg0EPGLHRsVpxVObtiCQAJzCgH0uXNMOVFrnsf/+dSDFwjTLU9Q==",
+      "cpu": [
+        "ia32"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-x64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-x64/-/gitleaks-linux-x64-8.30.1-post.2.tgz",
+      "integrity": "sha512-GSJ7RU0xuaF9Blf2AmDeytxyiZb4W7/Urd5Cw5LlNK+IF/C+Nd/Fp5pKbaAF4osdxADbCZ7d4XMYUMSdlpjQFA==",
+      "cpu": [
+        "x64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-windows-arm64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-arm64/-/gitleaks-windows-arm64-8.30.1-post.2.tgz",
+      "integrity": "sha512-cTu+rMASv3BuNJcPj57PhfiOHwapmOVH6s5PFBqAXzJomUnHdB/ZMrTDNseW78UZQQ/c0RPI3VXwieyMC23w7w==",
+      "cpu": [
+        "arm64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-windows-x32": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-x32/-/gitleaks-windows-x32-8.30.1-post.2.tgz",
+      "integrity": "sha512-MGMtKVEOf0TsXIGBu/1eGb94mAZRX2rzZmgNRYqzZBSYPnmyl3w1n4tic9e6rPOewyGfuxnweCDnu0Lu/qkaaw==",
+      "cpu": [
+        "ia32"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-windows-x64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-x64/-/gitleaks-windows-x64-8.30.1-post.2.tgz",
+      "integrity": "sha512-whDUUm3+J5YRQUJS0mcE72JaPb2N0Njmjaj6NYTcG1OJS3v42Y/JYen6R9FuQ5jUCfsVGcblmSBmDb9uZVdwrw==",
+      "cpu": [
+        "x64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
     "node_modules/@pnpm/config.env-replace": {
       "version": "1.1.0",
       "resolved": "https://registry.npmjs.org/@pnpm/config.env-replace/-/config.env-replace-1.1.0.tgz",

package.json

@@ -1,8 +1,9 @@
 {
   "name": "dcfrontend",
-  "version": "7.14.0",
+  "version": "7.15.0",
   "description": "Data Controller",
   "devDependencies": {
+    "@nogoo9/gitleaks": "8.30.1-post.2",
     "@saithodev/semantic-release-gitea": "^2.1.0",
     "@semantic-release/changelog": "^6.0.3",
     "@semantic-release/commit-analyzer": "13.0.1",

sas/mocks/sasjs/services/auditors/postdata.js

@@ -152,10 +152,19 @@ for (const col of diffCols) colDdtypes[col.name] = getDdType(col)
 // Staged values arrive from CSV as strings ("42"), base values are native JSON
 // (42). Compare numerics by value and strings case-sensitively.
 
+// A SAS special missing (._ , .A-.Z) reaches us as a string. Special missings
+// are numeric-only, and real DC writes the DIFF with `missing=STRING` - whose
+// format maps ._ and .a-.z to a string but leaves a bare `.` as null (see the
+// `bart` format in mp_jsonout.sas). Coercing a special missing with Number()
+// would yield NaN and blank the cell out of the DIFF.
+const SPECIAL_MISSING_RE = /^\.(_|[a-z])$/i
+
 function normVal(value, colName) {
   const col = diffCols.find((c) => c.name === colName)
   if (col && col.type === 'N') {
     if (value === null || value === undefined || value === '') return null
+    const str = String(value).trim()
+    if (SPECIAL_MISSING_RE.test(str)) return str
     const num = Number(value)
     return isNaN(num) ? null : num
   }
@@ -543,7 +552,7 @@ if (action === 'SHOW_DIFFS') {
   const stageFolder = getStageFolder(loadRef)
 
   // check: has this user already approved? (mirrors prev_upload_check in postdata.sas)
-  const reviewData = loadTableData('MPE_REVIEW') || { rows: [] }
+  const reviewData = mpeLoadTableData('MPE_REVIEW') || { rows: [] }
   const alreadyApproved = reviewData.rows.some(
     (r) =>
       r.TABLE_ID === loadRef &&

sas/mocks/sasjs/services/editors/getdata.js

@@ -8,6 +8,28 @@ const dcLibref = 'DC_JSLIB'
 // Load shared DC mock utilities
 eval(fs.readFileSync(nodePath.resolve(driveRoot, 'files', appLoc, 'services', 'dcMockUtils.js'), 'utf8'))
 
+// Mirrors SAS cats() for the values a dropdown source can hold: a special
+// missing stored in its period form (".a") becomes the bare uppercase letter
+// ("A") that cats() returns, so the mock's dropdown list matches the real one.
+function cats(value) {
+    if (typeof value === 'string') {
+        const m = /^\.(_|[a-z])$/i.exec(value.trim())
+        if (m) return m[1].toUpperCase()
+        return value.trim()
+    }
+    return String(value)
+}
+
+/**
+ * SAS sort position of a special missing, or null when the value is ordinary.
+ * `._` sorts first, then `.a` to `.z` - all of them below every number.
+ */
+function missingRank(value) {
+    const m = /^\.(_|[a-z])$/i.exec(String(value).trim())
+    if (!m) return null
+    return m[1] === '_' ? 0 : m[1].toUpperCase().charCodeAt(0) - 64
+}
+
 // ─── Parse input ──────────────────────────────────────────────────────────────
 
 const _sctRow = fetchTable('SASControlTable')[0] || {}
@@ -130,7 +152,7 @@ if (tableData && tableData.rows && tableData.columns) {
         const colName = parts[parts.length - 1]
         const lcName = colName.toLowerCase()
         const seen = new Set()
-        let order = 1
+        const values = []
         for (const row of tableData.rows) {
             let val = row[colName]
             if (val === undefined) val = row[lcName]
@@ -139,13 +161,30 @@ if (tableData && tableData.rows && tableData.columns) {
                 if (key) val = row[key]
             }
             if (val !== undefined && val !== null) {
-                const strVal = String(val)
+                const strVal = cats(val)
                 if (!seen.has(strVal)) {
                     seen.add(strVal)
-                    dqdata.push({ BASE_COL: rule.BASE_COL, RULE_VALUE: rule.RULE_VALUE, RULE_DATA: strVal, SELECTBOX_ORDER: order++ })
+                    values.push({ raw: val, str: strVal })
                 }
             }
         }
+        // getdata.sas orders the source by the column itself, and a special
+        // missing sorts below every number - so the list reads `._`, `.a`-`.z`,
+        // then the numbers ascending.
+        values.sort((a, b) => {
+            const ra = missingRank(a.raw)
+            const rb = missingRank(b.raw)
+            if (ra !== null && rb !== null) return ra - rb
+            if (ra !== null) return -1
+            if (rb !== null) return 1
+            const na = Number(a.str)
+            const nb = Number(b.str)
+            if (!isNaN(na) && !isNaN(nb)) return na - nb
+            return a.str < b.str ? -1 : a.str > b.str ? 1 : 0
+        })
+        values.forEach((v, i) => {
+            dqdata.push({ BASE_COL: rule.BASE_COL, RULE_VALUE: rule.RULE_VALUE, RULE_DATA: v.str, SELECTBOX_ORDER: i + 1 })
+        })
     }
 }
 

sas/mocks/sasjs/services/public/viewdata.js

@@ -4,10 +4,13 @@ let appLoc = nodePath.join(..._program.split('services')[0].split('/'))
 const sasjsRoot = nodePath.resolve(weboutPath, '..', '..', '..')
 const driveRoot = nodePath.resolve(sasjsRoot, 'drive')
 const dcLibref = 'DC_JSLIB'
-const dataDir = nodePath.resolve(driveRoot, 'files', appLoc, 'data', dcLibref)
+const mpeDataDir = nodePath.resolve(driveRoot, 'files', appLoc, 'data', dcLibref)
 
 eval(fs.readFileSync(nodePath.resolve(driveRoot, 'files', appLoc, 'services', 'dcMockUtils.js'), 'utf8'))
-const loadTableData = makeTableLoader(dataDir)
+
+// Row cap for the viewer - mirrors DC_MAXOBS_WEBVIEW in viewdata.sas, and the
+// MAXROWS value reported in sasparams.
+const maxObsWebView = 500
 
 // ─── Parse input ──────────────────────────────────────────────────────────────
 
@@ -16,11 +19,65 @@ let libds = _sctRow.LIBDS || ''
 const libref = libds.split('.')[0] || dcLibref
 const dsn = libds.split('.')[1] || ''
 
+// FILTER_RK (matching viewdata.sas: if filter_rk le 0 then filter_rk=-1)
+const filterRk = Number(colVal(_sctRow, 'FILTER_RK')) > 0 ? Number(colVal(_sctRow, 'FILTER_RK')) : -1
+
+// SEARCHTYPE / SEARCHVAL - only CHAR or NUM triggers a search, anything else
+// (blank, NONE) is a normal view.  Search values are stripped of the same
+// characters viewdata.sas removes before handing them to mp_searchdata.
+const rawSearchType = String(colVal(_sctRow, 'SEARCHTYPE') || '').toUpperCase()
+const searchType = (rawSearchType === 'CHAR' || rawSearchType === 'NUM') ? rawSearchType : ''
+const rawSearchVal = colVal(_sctRow, 'SEARCHVAL')
+let searchVal = (rawSearchVal === undefined || rawSearchVal === null) ? '' : String(rawSearchVal)
+if (searchType) searchVal = searchVal.replace(/[%&;"]/g, '')
+
+// ─── Load the target table ───────────────────────────────────────────────────
+
+const dataDir = libDataDir(libref)
+const loadTableData = makeTableLoader(dataDir)
+const loadMpeTable = makeTableLoader(mpeDataDir)
+
+let tableData = loadTableData(dsn)
+if (!tableData && dsn) {
+    const found = loadTableAnyLib(dsn)
+    if (found) tableData = found.data
+}
+
+// ─── Apply the stored filter (mpe_filtermaster, VIEW mode) ───────────────────
+// Mirrors viewdata.sas: %mpe_filtermaster(VIEW,&libds,filter_rk=&filter_rk)
+// Row Level Security and any saved filter are applied here, before the search.
+
+const { predicate: filterPredicate, filterText } = mpeFilterMaster({
+    mode: 'VIEW',
+    libds: (libref + '.' + dsn).toUpperCase(),
+    filterRk: filterRk,
+    dataDir: mpeDataDir,
+    columns: tableData && tableData.columns ? tableData.columns : []
+})
+
+// ─── Search predicate (mp_searchdata.sas) ────────────────────────────────────
+// mp_searchdata builds one WHERE clause covering every column of the table:
+//   or ("COL1"n ? "the string")   /* character columns: case sensitive CONTAINS */
+//   or ("COL2"n = the number)     /* numeric columns: exact equality           */
+
+function rowMatchesSearch(row, columns, type, value) {
+    for (const col of columns) {
+        const val = colVal(row, col.name)
+        if (val === undefined || val === null) continue
+        if (type === 'CHAR') {
+            if (col.type !== 'N' && String(val).includes(value)) return true
+        } else {
+            if (col.type === 'N' && Number(val) === Number(value)) return true
+        }
+    }
+    return false
+}
+
 // ─── Build response ──────────────────────────────────────────────────────────
 
 // Look up PK fields from MPE_TABLES (mirrors mp_getpk).
 // MPE_AUDIT is a standard DC audit table whose PK is always the first 5 columns.
-const mpeTables = loadTableData('MPE_TABLES')
+const mpeTables = loadMpeTable('MPE_TABLES')
 let pkFields = ' '
 if (mpeTables && mpeTables.rows) {
     const reg = mpeTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -32,8 +89,6 @@ if (dsn.toUpperCase() === 'MPE_AUDIT' && pkFields.trim() === '') {
     pkFields = 'LOAD_REF LIBREF DSN KEY_HASH TGTVAR_NM'
 }
 
-const tableData = loadTableData(dsn)
-
 let cols = []
 let viewdata = []
 let vars = {}
@@ -53,8 +108,26 @@ if (tableData && tableData.columns) {
         }
     })
 
+    // Rows in scope: the stored filter is applied first, then the search
+    // (matching viewdata.sas, which searches a filtered view of the table).
+    const filteredRows = (tableData.rows || []).filter(r => filterPredicate(r))
+
+    let rows
+    if (searchType) {
+        const matches = filteredRows.filter(row => rowMatchesSearch(row, tableData.columns, searchType, searchVal))
+        // mp_searchdata caps its output at outobs=&DC_MAXOBS_WEBVIEW, and
+        // viewdata.sas then keeps only `if _n_<&DC_MAXOBS_WEBVIEW` of them.
+        nobs = Math.min(matches.length, maxObsWebView)
+        rows = matches.slice(0, maxObsWebView - 1)
+    } else {
+        // proc sql select count(*) ... where <filter> (uncapped)
+        nobs = filteredRows.length
+        // data work.viewdata; set &libds; where <filter>; if _n_>&DC_MAXOBS_WEBVIEW then stop;
+        rows = filteredRows.slice(0, maxObsWebView)
+    }
+
     // viewdata rows: all values as strings (formatted, SAS proc-print style)
-    viewdata = (tableData.rows || []).map(r => {
+    viewdata = rows.map(r => {
         const row = {}
         for (const col of tableData.columns) {
             const lcName = col.name.toLowerCase()
@@ -94,7 +167,9 @@ if (tableData && tableData.columns) {
     // the client can derive the column set even when empty (the UI iterates
     // viewdata[0] to build the grid headers). Numeric columns get "."
     // (SAS missing), character columns get "".
-    if (viewdata.length === 0 && tableData.columns && tableData.columns.length > 0) {
+    // A search that matches nothing sends no rows at all (the client shows
+    // its "no data found" panel instead of an empty grid).
+    if (viewdata.length === 0 && !searchType && tableData.columns.length > 0) {
         const emptyRow = {}
         for (const col of tableData.columns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
         viewdata = [emptyRow]
@@ -105,8 +180,6 @@ if (tableData && tableData.columns) {
     for (const col of tableData.columns) {
         vars[col.name] = sasVarsEntry(col)
     }
-
-    nobs = tableData.rows ? tableData.rows.length : 0
 }
 
 webOutOpen()
@@ -118,8 +191,8 @@ webOutObj([{ ODS_TABLE: 'ATTRIBUTES', NAME: 'Data Set Name', VALUE: libref + '.'
     { ODS_TABLE: 'ATTRIBUTES', NAME: 'Variables', VALUE: String(cols.length) },
     { ODS_TABLE: 'ATTRIBUTES', NAME: 'Engine', VALUE: 'V9' }], 'dsmeta')
 webOutObj([], 'query')
-webOutObj([{ TABLEURI: ' ', TABLENAME: dsn, FILTER_TEXT: ' ',
-    PK_FIELDS: pkFields, NOBS: nobs, VARS: cols.length, MAXROWS: 500 }], 'sasparams')
+webOutObj([{ TABLEURI: ' ', TABLENAME: dsn, FILTER_TEXT: filterText === '' ? ' ' : filterText,
+    PK_FIELDS: pkFields, NOBS: nobs, VARS: cols.length, MAXROWS: maxObsWebView }], 'sasparams')
 webOutObj([], 'versions')
 webOutObj(viewdata, 'viewdata', { vars })
 webOutClose()

sas/package-lock.json

@@ -6,8 +6,8 @@
     "": {
       "name": "dc-sas",
       "dependencies": {
-        "@sasjs/cli": "4.20.1",
-        "@sasjs/core": "5.2.0"
+        "@sasjs/cli": "4.20.4",
+        "@sasjs/core": "5.2.8"
       }
     },
     "node_modules/@asamuzakjp/css-color": {
@@ -23,17 +23,6 @@
         "lru-cache": "^10.4.3"
       }
     },
-    "node_modules/@asamuzakjp/dom-selector": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-2.0.2.tgz",
-      "integrity": "sha512-x1KXOatwofR6ZAYzXRBL5wrdV0vwNxlTCK9NCuLqAzQYARqGcvFwiJA6A1ERuh+dgeA4Dxm3JBYictIes+SqUQ==",
-      "license": "MIT",
-      "dependencies": {
-        "bidi-js": "^1.0.3",
-        "css-tree": "^2.3.1",
-        "is-potential-custom-element-name": "^1.0.1"
-      }
-    },
     "node_modules/@coolaj86/urequest": {
       "version": "1.3.7",
       "resolved": "https://registry.npmjs.org/@coolaj86/urequest/-/urequest-1.3.7.tgz",
@@ -214,94 +203,59 @@
       }
     },
     "node_modules/@sasjs/cli": {
-      "version": "4.20.1",
-      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.1.tgz",
-      "integrity": "sha512-482xPlEuyBKqGlg2ArCiREu3Q9+8TsTj0dwy5ewNURi7pWEcwJH0A4c9Yw4a3akUtPAphPEzs8/9BkcS/v+3WQ==",
+      "version": "4.20.4",
+      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.4.tgz",
+      "integrity": "sha512-hKE002Dm9AzWU62qpRyYpjQv8BIXG951Z0ToNdG9J1wvvJNYH7h6rGv7kdQMbgvXff/KOQvVdnGJZ023SrG+Jw==",
       "license": "ISC",
       "dependencies": {
         "@sasjs/adapter": "^4.19.0",
-        "@sasjs/core": "4.68.1",
-        "@sasjs/lint": "2.4.3",
-        "@sasjs/utils": "^3.6.0",
-        "adm-zip": "0.6.0",
+        "@sasjs/core": "5.2.9",
+        "@sasjs/lint": "2.5.0",
+        "@sasjs/utils": "3.6.2",
         "chalk": "4.1.2",
-        "dotenv": "16.0.3",
+        "dotenv": "17.4.2",
         "find": "0.3.0",
-        "js-base64": "3.7.5",
-        "jsdom": "23.2.0",
+        "js-base64": "3.9.3",
+        "jsdom": "26.0.0",
         "jwt-decode": "3.1.2",
         "lodash.groupby": "4.6.0",
         "lodash.uniqby": "4.7.0",
         "node-graphviz": "0.1.1",
         "node-powershell": "5.0.1",
         "ora": "5.4.1",
-        "prompts": "2.4.1",
+        "prompts": "2.4.2",
         "shelljs": "0.10.0",
         "ssl-root-cas": "1.3.1",
         "xml": "1.0.1",
-        "yargs": "17.6.2"
+        "yargs": "17.6.2",
+        "yauzl": "3.4.0",
+        "yazl": "3.3.1"
       },
       "bin": {
         "sasjs": "build/index.js"
       }
     },
     "node_modules/@sasjs/cli/node_modules/@sasjs/core": {
-      "version": "4.68.1",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-4.68.1.tgz",
-      "integrity": "sha512-BjoOzV7h7guCShW4JsFHvB1NNLCKHhPI5708OtEk29itQbgVCMRlEXKFRbnMiuNklktDQDi8dR4XZZ0aK7/4Wg==",
+      "version": "5.2.9",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.9.tgz",
+      "integrity": "sha512-R5wtJF0ANHchaURqTF2t1vRXrhUJ+uDfj21ewzdOq4Lp14CWW22+yYeRQooY4QOpyHUyFDjReA5q6SoJt2Tz1g==",
       "license": "MIT"
     },
     "node_modules/@sasjs/core": {
-      "version": "5.2.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.0.tgz",
-      "integrity": "sha512-YJAVckql9o2lbldo4xFbu/Vkja0DfSw8w8hrnhnCov8iqqzkM84rHnqqKzdD0+sa52+AoPoHAxt4nCBoQfH9wA==",
+      "version": "5.2.8",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.8.tgz",
+      "integrity": "sha512-+K8tXTrsw52NR1y5MFHW0+Jf9euL+GukwqA79FIHHu4Oj+BhHBm1uZiyB5tPrvwl3V8mY1CQhMOT83//bV+jWg==",
       "license": "MIT"
     },
     "node_modules/@sasjs/lint": {
-      "version": "2.4.3",
-      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-2.4.3.tgz",
-      "integrity": "sha512-J0fYJmDfnhXDeMkZI8vHo8zjIsByIPcKHSQbiQEuRMXbNJXjqgTqoCpN0L6SlhyrrXUtMQvk/XErJ3OQGVEjrA==",
+      "version": "2.5.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-2.5.0.tgz",
+      "integrity": "sha512-bVUtYnKhCigXylCCvQ6dsAIywpPKkX3BXatMqOWjDVw9R30jfj4M3/qn05fcR45L42Y7e5Tuzlt+LYs+9k95Yg==",
       "hasInstallScript": true,
       "license": "MIT",
       "dependencies": {
-        "@sasjs/utils": "3.5.2",
-        "ignore": "5.2.4"
-      }
-    },
-    "node_modules/@sasjs/lint/node_modules/@sasjs/utils": {
-      "version": "3.5.2",
-      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-3.5.2.tgz",
-      "integrity": "sha512-LBpBDx0T7G/eO15Gb+r3DR1LfBnoqagWT3HiHabojFziA4ej4ePORo8chrk0zHLIzrjI2ljAnDabyJEwC5KtIA==",
-      "hasInstallScript": true,
-      "license": "ISC",
-      "dependencies": {
-        "@fast-csv/format": "4.3.5",
-        "@types/fs-extra": "11.0.4",
-        "@types/prompts": "2.0.13",
-        "chalk": "4.1.1",
-        "cli-table": "0.3.6",
-        "consola": "2.15.0",
-        "find": "0.3.0",
-        "fs-extra": "11.3.0",
-        "jwt-decode": "3.1.2",
-        "prompts": "2.4.1",
-        "valid-url": "1.0.9"
-      }
-    },
-    "node_modules/@sasjs/lint/node_modules/chalk": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
-      "integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
-      "license": "MIT",
-      "dependencies": {
-        "ansi-styles": "^4.1.0",
-        "supports-color": "^7.1.0"
-      },
-      "engines": {
-        "node": ">=10"
-      },
-      "funding": {
-        "url": "https://github.com/chalk/chalk?sponsor=1"
+        "@sasjs/utils": "3.6.2",
+        "ignore": "7.0.8"
       }
     },
     "node_modules/@sasjs/utils": {
@@ -339,6 +293,19 @@
         "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
+    "node_modules/@sasjs/utils/node_modules/prompts": {
+      "version": "2.4.1",
+      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
+      "integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
+      "license": "MIT",
+      "dependencies": {
+        "kleur": "^3.0.3",
+        "sisteransi": "^1.0.5"
+      },
+      "engines": {
+        "node": ">= 6"
+      }
+    },
     "node_modules/@types/fs-extra": {
       "version": "11.0.4",
       "resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-11.0.4.tgz",
@@ -383,15 +350,6 @@
         "ms": "^2.1.3"
       }
     },
-    "node_modules/adm-zip": {
-      "version": "0.6.0",
-      "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.0.tgz",
-      "integrity": "sha512-XleryMhbuksdKtofnWZ9Sk+4CUTbms4Mb/EU32SZwToAyZ5RgVos/ki8n+yr0LWHOGKuakbXTuuYNHLQjhddgg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=14.0"
-      }
-    },
     "node_modules/agent-base": {
       "version": "7.1.4",
       "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz",
@@ -482,15 +440,6 @@
       ],
       "license": "MIT"
     },
-    "node_modules/bidi-js": {
-      "version": "1.0.3",
-      "resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz",
-      "integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==",
-      "license": "MIT",
-      "dependencies": {
-        "require-from-string": "^2.0.2"
-      }
-    },
     "node_modules/bl": {
       "version": "4.1.0",
       "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
@@ -538,6 +487,15 @@
         "ieee754": "^1.1.13"
       }
     },
+    "node_modules/buffer-crc32": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-1.0.0.tgz",
+      "integrity": "sha512-Db1SbgBS/fg/392AblrMJk97KggmvYhr4pB5ZIMTWtaivCPMWLkmb7m21cJvpvgK+J3nsU2CmmixNBZx4vFj/w==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=8.0.0"
+      }
+    },
     "node_modules/bytes": {
       "version": "3.1.2",
       "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
@@ -708,19 +666,6 @@
         "node": ">= 8"
       }
     },
-    "node_modules/css-tree": {
-      "version": "2.3.1",
-      "resolved": "https://registry.npmjs.org/css-tree/-/css-tree-2.3.1.tgz",
-      "integrity": "sha512-6Fv1DV/TYw//QF5IzQdqsNDjx/wc8TrMBZsqjL9eW01tWb7R7k/mq+/VXfJCl7SoD5emsJop9cOByJZfs8hYIw==",
-      "license": "MIT",
-      "dependencies": {
-        "mdn-data": "2.0.30",
-        "source-map-js": "^1.0.1"
-      },
-      "engines": {
-        "node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0"
-      }
-    },
     "node_modules/cssstyle": {
       "version": "4.6.0",
       "resolved": "https://registry.npmjs.org/cssstyle/-/cssstyle-4.6.0.tgz",
@@ -734,12 +679,6 @@
         "node": ">=18"
       }
     },
-    "node_modules/cssstyle/node_modules/rrweb-cssom": {
-      "version": "0.8.0",
-      "resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
-      "integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
-      "license": "MIT"
-    },
     "node_modules/data-urls": {
       "version": "5.0.0",
       "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-5.0.0.tgz",
@@ -798,12 +737,15 @@
       }
     },
     "node_modules/dotenv": {
-      "version": "16.0.3",
-      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.3.tgz",
-      "integrity": "sha512-7GO6HghkA5fYG9TYnNxi14/7K9f5occMlp3zXAuSxn7CKCxt9xbNWG7yF8hTCSUchlfWSe3uLmlPfigevRItzQ==",
+      "version": "17.4.2",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
+      "integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
       "license": "BSD-2-Clause",
       "engines": {
         "node": ">=12"
+      },
+      "funding": {
+        "url": "https://dotenvx.com"
       }
     },
     "node_modules/dunder-proto": {
@@ -938,9 +880,9 @@
       }
     },
     "node_modules/fastq": {
-      "version": "1.20.1",
-      "resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.1.tgz",
-      "integrity": "sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw==",
+      "version": "1.20.3",
+      "resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.3.tgz",
+      "integrity": "sha512-XKv5nnLs6nLF71NgiKJLIZFLkPyIEuOselLG7ujZnGrRfQK8HpvY+WqKhAJUAdLomwVHErVS4LfxFlPq0/FTAw==",
       "license": "ISC",
       "dependencies": {
         "reusify": "^1.0.4"
@@ -1284,9 +1226,9 @@
       "license": "BSD-3-Clause"
     },
     "node_modules/ignore": {
-      "version": "5.2.4",
-      "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.2.4.tgz",
-      "integrity": "sha512-MAb38BcSbH0eHNBxn7ql2NH/kX33OkB3lZ1BNdh7ENeRChHTYsTvWrMubiIAMNS2llXEEgZ1MUOBtXChP3kaFQ==",
+      "version": "7.0.8",
+      "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.8.tgz",
+      "integrity": "sha512-YYNsSlXBjMk92SKnkwvB5LOVSa6OznlFUGcsvrFgNJbJCd0M1XKeFVRc8ZByeCqz32FivYNHJVooLmdqrmvp/Q==",
       "license": "MIT",
       "engines": {
         "node": ">= 4"
@@ -1410,44 +1352,44 @@
       "license": "ISC"
     },
     "node_modules/js-base64": {
-      "version": "3.7.5",
-      "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.7.5.tgz",
-      "integrity": "sha512-3MEt5DTINKqfScXKfJFrRbxkrnk2AxPWGBL/ycjz4dK8iqiSJ06UxD8jh8xuh6p10TX4t2+7FsBYVxxQbMg+qA==",
+      "version": "3.9.3",
+      "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.3.tgz",
+      "integrity": "sha512-uwYQp+VJ38FVvtim6qNbit6e9uT6dwWQ4Y1+H9TxhW5hcHjpHwoxlR0nMpqUmIFOmu4VqMxwdJA88gIVuZJQ/g==",
       "license": "BSD-3-Clause"
     },
     "node_modules/jsdom": {
-      "version": "23.2.0",
-      "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-23.2.0.tgz",
-      "integrity": "sha512-L88oL7D/8ufIES+Zjz7v0aes+oBMh2Xnh3ygWvL0OaICOomKEPKuPnIfBJekiXr+BHbbMjrWn/xqrDQuxFTeyA==",
+      "version": "26.0.0",
+      "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-26.0.0.tgz",
+      "integrity": "sha512-BZYDGVAIriBWTpIxYzrXjv3E/4u8+/pSG5bQdIYCbNCGOvsPkDQfTVLAIXAf9ETdCpduCVTkDe2NNZ8NIwUVzw==",
       "license": "MIT",
       "dependencies": {
-        "@asamuzakjp/dom-selector": "^2.0.1",
-        "cssstyle": "^4.0.1",
+        "cssstyle": "^4.2.1",
         "data-urls": "^5.0.0",
         "decimal.js": "^10.4.3",
-        "form-data": "^4.0.0",
+        "form-data": "^4.0.1",
         "html-encoding-sniffer": "^4.0.0",
-        "http-proxy-agent": "^7.0.0",
-        "https-proxy-agent": "^7.0.2",
+        "http-proxy-agent": "^7.0.2",
+        "https-proxy-agent": "^7.0.6",
         "is-potential-custom-element-name": "^1.0.1",
-        "parse5": "^7.1.2",
-        "rrweb-cssom": "^0.6.0",
+        "nwsapi": "^2.2.16",
+        "parse5": "^7.2.1",
+        "rrweb-cssom": "^0.8.0",
         "saxes": "^6.0.0",
         "symbol-tree": "^3.2.4",
-        "tough-cookie": "^4.1.3",
+        "tough-cookie": "^5.0.0",
         "w3c-xmlserializer": "^5.0.0",
         "webidl-conversions": "^7.0.0",
         "whatwg-encoding": "^3.1.1",
         "whatwg-mimetype": "^4.0.0",
-        "whatwg-url": "^14.0.0",
-        "ws": "^8.16.0",
+        "whatwg-url": "^14.1.0",
+        "ws": "^8.18.0",
         "xml-name-validator": "^5.0.0"
       },
       "engines": {
         "node": ">=18"
       },
       "peerDependencies": {
-        "canvas": "^2.11.2"
+        "canvas": "^3.0.0"
       },
       "peerDependenciesMeta": {
         "canvas": {
@@ -1468,6 +1410,18 @@
         "node": ">= 14"
       }
     },
+    "node_modules/jsdom/node_modules/tough-cookie": {
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
+      "integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
+      "license": "BSD-3-Clause",
+      "dependencies": {
+        "tldts": "^6.1.32"
+      },
+      "engines": {
+        "node": ">=16"
+      }
+    },
     "node_modules/jsonfile": {
       "version": "6.2.1",
       "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.1.tgz",
@@ -1578,12 +1532,6 @@
         "node": ">= 0.4"
       }
     },
-    "node_modules/mdn-data": {
-      "version": "2.0.30",
-      "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.0.30.tgz",
-      "integrity": "sha512-GaqWWShW4kv/G9IEucWScBx9G1/vsFZZJUO+tD26M8J8z3Kw5RDQjaoZe03YAClgeS/SWPOcb4nkFBTEi5DUEA==",
-      "license": "CC0-1.0"
-    },
     "node_modules/merge-stream": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz",
@@ -1694,6 +1642,12 @@
         "node": ">=8"
       }
     },
+    "node_modules/nwsapi": {
+      "version": "2.2.27",
+      "resolved": "https://registry.npmjs.org/nwsapi/-/nwsapi-2.2.27.tgz",
+      "integrity": "sha512-gQPNF78qebCQ6tvVFBYrvJdBNOrYZm90ZlXgpIFm06p6qHDHq/XC4TnJftN6OMbxVE0UTBAoRgcsDeJBBooITw==",
+      "license": "MIT"
+    },
     "node_modules/onetime": {
       "version": "5.1.2",
       "resolved": "https://registry.npmjs.org/onetime/-/onetime-5.1.2.tgz",
@@ -1799,6 +1753,12 @@
         "node": ">=8"
       }
     },
+    "node_modules/pend": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz",
+      "integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==",
+      "license": "MIT"
+    },
     "node_modules/picomatch": {
       "version": "2.3.2",
       "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
@@ -1812,9 +1772,9 @@
       }
     },
     "node_modules/prompts": {
-      "version": "2.4.1",
-      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
-      "integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
+      "version": "2.4.2",
+      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
+      "integrity": "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q==",
       "license": "MIT",
       "dependencies": {
         "kleur": "^3.0.3",
@@ -1903,15 +1863,6 @@
         "node": ">=0.10.0"
       }
     },
-    "node_modules/require-from-string": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
-      "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
     "node_modules/requires-port": {
       "version": "1.0.0",
       "resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
@@ -1942,9 +1893,9 @@
       }
     },
     "node_modules/rrweb-cssom": {
-      "version": "0.6.0",
-      "resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.6.0.tgz",
-      "integrity": "sha512-APM0Gt1KoXBz0iIkkdB/kfvGOwC4UuJFeG/c+yV7wSc7q96cG/kJ0HiYCnzivD9SB53cLV1MlHFNfOuPaadYSw==",
+      "version": "0.8.0",
+      "resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
+      "integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
       "license": "MIT"
     },
     "node_modules/run-parallel": {
@@ -2054,15 +2005,6 @@
       "integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==",
       "license": "MIT"
     },
-    "node_modules/source-map-js": {
-      "version": "1.2.1",
-      "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
-      "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
-      "license": "BSD-3-Clause",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
     "node_modules/ssl-root-cas": {
       "version": "1.3.1",
       "resolved": "https://registry.npmjs.org/ssl-root-cas/-/ssl-root-cas-1.3.1.tgz",
@@ -2134,6 +2076,24 @@
       "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==",
       "license": "MIT"
     },
+    "node_modules/tldts": {
+      "version": "6.1.86",
+      "resolved": "https://registry.npmjs.org/tldts/-/tldts-6.1.86.tgz",
+      "integrity": "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ==",
+      "license": "MIT",
+      "dependencies": {
+        "tldts-core": "^6.1.86"
+      },
+      "bin": {
+        "tldts": "bin/cli.js"
+      }
+    },
+    "node_modules/tldts-core": {
+      "version": "6.1.86",
+      "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-6.1.86.tgz",
+      "integrity": "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA==",
+      "license": "MIT"
+    },
     "node_modules/to-regex-range": {
       "version": "5.0.1",
       "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz",
@@ -2322,9 +2282,9 @@
       }
     },
     "node_modules/ws": {
-      "version": "8.21.1",
-      "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.1.tgz",
-      "integrity": "sha512-+0NTnW77fFN/DjQi6k/Sq/Yvk4Sgajw7urW8V+asjXnRgDs9gyGkdb7EzgfhA4goXsRIZKE28fzIXBHEzhuiWw==",
+      "version": "8.21.3",
+      "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
+      "integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==",
       "license": "MIT",
       "engines": {
         "node": ">=10.0.0"
@@ -2398,6 +2358,27 @@
       "engines": {
         "node": ">=12"
       }
+    },
+    "node_modules/yauzl": {
+      "version": "3.4.0",
+      "resolved": "https://registry.npmjs.org/yauzl/-/yauzl-3.4.0.tgz",
+      "integrity": "sha512-jIH9yLR9wqr0wOS0TpBvo/g/2UgZH5qePVbjgRliiF0BYvOZyaBknKsF+x9Iht0O6sqgnB93rCICdOZFecJuDw==",
+      "license": "MIT",
+      "dependencies": {
+        "pend": "~1.2.0"
+      },
+      "engines": {
+        "node": ">=12"
+      }
+    },
+    "node_modules/yazl": {
+      "version": "3.3.1",
+      "resolved": "https://registry.npmjs.org/yazl/-/yazl-3.3.1.tgz",
+      "integrity": "sha512-BbETDVWG+VcMUle37k5Fqp//7SDOK2/1+T7X8TD96M3D9G8jK5VLUdQVdVjGi8im7FGkazX7kk5hkU8X4L5Bng==",
+      "license": "MIT",
+      "dependencies": {
+        "buffer-crc32": "^1.0.0"
+      }
     }
   }
 }

sas/package.json

@@ -30,8 +30,8 @@
   },
   "private": true,
   "dependencies": {
-    "@sasjs/cli": "4.20.1",
-    "@sasjs/core": "5.2.0"
+    "@sasjs/cli": "4.20.4",
+    "@sasjs/core": "5.2.8"
   },
   "overrides": {
     "nanoid": "3.3.18"

sas/sasjs/macros/mpe_accesscheck.sas

@@ -23,7 +23,7 @@
   @li mp_abort.sas
   @li mf_getuniquename.sas
   @li mf_getuser.sas
-  @li mf_verifymacvars.sas
+  @li mpe_validatecol.sas
   @li mpe_getgroups.sas
 
   <h4> Related Macros </h4>
@@ -52,10 +52,52 @@
     ,msg=%str(outds should be a WORK table)
   )
 
+  /**
+    * Validate inputs before they reach executable code.  base_table is
+    * interpolated into a SQL where clause (and callers may pass raw request
+    * input), so it must be a well-formed LIBREF.DATASET (or the
+    * LIBREF.CATALOGNAME-FC form of a format catalog) and access_level must
+    * be one of the known levels - anything else aborts before the query is
+    * built.  Values are read with symget (never re-resolved) and scanned in
+    * a data step so no macro content in the input can execute.
+    */
+  %local is_libds is_level;
+  %let is_libds=0;
+  %let is_level=0;
+  data _null_;
+    length _bt $64 _lvl $16;
+    _bt=symget('base_table');
+    _lvl=upcase(symget('access_level'));
+    %mpe_validatecol(_bt,LIBDS,is_libds)
+    if is_libds=0 then do;
+      call symputx('is_libds',0,'l');
+      putlog 'ERR' 'OR: Invalid base_table:' _bt;
+      stop;
+    end;
+    if _lvl not in ('EDIT','APPROVE','VIEW','SIGNOFF','AUDIT') then do;
+      call symputx('is_level',0,'l');
+      putlog 'ERR' 'OR: Invalid access_level:' _lvl;
+      stop;
+    end;
+    /* escape any embedded quotes so the value cannot break out of the
+      * double-quoted SQL literals below (defence in depth - the LIBDS
+      * check above already rejects quotes) */
+    _bt=tranwrd(_bt,'"','');
+    call symputx('base_table',_bt,'l');
+    call symputx('access_level',_lvl,'l');
+    call symputx('is_libds',is_libds,'l');
+    call symputx('is_level',1,'l');
+  run;
+
   %mp_abort(
-    iftrue=(%mf_verifymacvars(base_table user access_level)=0)
+    iftrue=(&is_libds ne 1)
     ,mac=mpe_accesscheck
-    ,msg=%str(Missing base_table/user access_level variables)
+    ,msg=%str(Invalid base_table)
+  )
+  %mp_abort(
+    iftrue=(&is_level ne 1)
+    ,mac=mpe_accesscheck
+    ,msg=%str(Invalid access_level)
   )
 
   /* make unique temp table vars */

sas/sasjs/macros/mpe_validatecol.sas

@@ -0,0 +1,47 @@
+/**
+  @file
+  @brief Validates a column of values, including the Data Controller
+    format-catalog form of a libds reference
+  @details Wrapper around mp_validatecol() that adds the Data Controller
+    convention of addressing a format catalog as `LIBREF.CATALOGNAME-FC`
+    to the LIBDS rule.
+
+    The `-FC` suffix is matched exactly and the remainder is validated as
+    a strict LIBREF.DATASET, so the *whole* value is covered by the
+    validation - a bare scan on the dash would leave anything after it
+    unvalidated.  The input column itself is never modified, so a caller
+    that needs the catalog reference downstream (to match MPE_SECURITY,
+    for instance) still receives it.
+
+  @param [in] incol Input column (a data step variable)
+  @param [in] rule Validation rule, as per mp_validatecol()
+  @param [out] outcol 1 when the value is valid, else 0
+
+  <h4> SAS Macros </h4>
+  @li mp_validatecol.sas
+  @li mf_getuniquename.sas
+
+  @version 9.2
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+**/
+
+%macro mpe_validatecol(incol,rule,outcol);
+
+%if &rule=LIBDS %then %do;
+  /* tempcol is given a unique name with every invocation */
+  %local tempcol;
+  %let tempcol=%mf_getuniquename(prefix=cat);
+  &tempcol=strip(&incol);
+  /* permit the format-catalog form: LIBREF.CATALOGNAME-FC, exactly */
+  if length(&tempcol)>3
+    and upcase(substr(&tempcol,length(&tempcol)-2,3))='-FC'
+  then &tempcol=substr(&tempcol,1,length(&tempcol)-3);
+  %mp_validatecol(&tempcol,LIBDS,&outcol)
+  drop &tempcol;
+%end;
+%else %mp_validatecol(&incol,&rule,&outcol);
+
+%mend mpe_validatecol;

sas/sasjs/macros/mpe_validatecol.test.sas

@@ -0,0 +1,72 @@
+/**
+  @file
+  @brief testing the mpe_validatecol macro (LIBDS rule)
+  @details The LIBDS rule must accept the Data Controller format-catalog
+  reference (LIBREF.CATALOGNAME-FC) and reject everything else - including
+  a value that merely *starts* with a valid libds and continues past the
+  dash, which a bare scan on the dash would let through.
+
+  <h4> SAS Macros </h4>
+  @li mpe_validatecol.sas
+  @li mp_assertdsobs.sas
+
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+data work.check;
+  length val $64;
+  /* valid: a plain libds */
+  exp=1; val='WORK.CLASS';              output;
+  exp=1; val='DC.MPE_TABLES';           output;
+  exp=1; val='_A._B';                   output;
+  /* valid: the format-catalog form, however the suffix is cased */
+  exp=1; val='DCTEST.DCFMTS-FC';        output;
+  exp=1; val='dctest.dcfmts-fc';        output;
+  exp=1; val='WORK.CLASS-FC';           output;
+  /* valid: padded values are trimmed before the check */
+  exp=1; val='WORK.CLASS   ';           output;
+  /* invalid: not a libds at all */
+  exp=0; val='WORK';                    output;
+  exp=0; val='WORK.CLASS.NOPE';         output;
+  exp=0; val='';                        output;
+  exp=0; val='-FC';                     output;
+  exp=0; val='../secprobe';             output;
+  /* invalid: the dash suffix is not exactly -FC */
+  exp=0; val='WORK.CLASS-FCX';          output;
+  exp=0; val='WORK.CLASS-C';            output;
+  exp=0; val='WORK.CLASS-FC X';         output;
+  /* invalid: content smuggled past a valid libds prefix */
+  exp=0; val="WORK.CLASS-FC'";          output;
+  exp=0; val='WORK.CLASS-FC;proc sql;'; output;
+  exp=0; val='WORK.CLASS-FC) or 1=1';   output;
+  exp=0; val='WORK.CLASS-fc-';          output;
+run;
+
+data work.check;
+  set work.check;
+  is_libds=0;
+  %mpe_validatecol(val,LIBDS,is_libds)
+  got=is_libds;
+  if got ne exp then putlog 'ERR' 'OR: unexpected result for [' val +(-1) ']';
+run;
+
+data work.mismatch;
+  set work.check;
+  where got ne exp;
+run;
+
+%mp_assertdsobs(work.mismatch,
+  desc=Every LIBDS value validated as expected (catalog form permitted, nothing smuggled past the dash),
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+/* dump for offline inspection */
+data _null_;
+  set work.check;
+  putlog 'TEST_RESULT_LINE: [' val +(-1) '] exp=' exp 'got=' got;
+run;

sas/sasjs/services/admin/dirlist.sas

@@ -5,8 +5,13 @@
 
     @li &parent= (parent path)
 
+  Requires membership of the DC administrators group.
+
   <h4> SAS Macros </h4>
+  @li mf_getuser.sas
+  @li mp_abort.sas
   @li mp_dirlist.sas
+  @li mpe_getgroups.sas
 
   @version 9.2
   @author 4GL Apps Ltd
@@ -17,8 +22,37 @@
 **/
 
 %global parent;
-/* if no flavour is specified, default to root */
-%let parent=%sysfunc(coalescec(&parent,/));
+%mpeinit()
+
+/* check user is in admin group */
+%let cnt=0;
+%mpe_getgroups(user=%mf_getuser(),outds=work.usergroups)
+proc sql noprint;
+select count(*) into:cnt
+  from usergroups
+  where groupname="&mpeadmins";
+%mp_abort(iftrue= (&cnt=0)
+  ,mac=&_program
+  ,msg=%str(This service is only available to &mpeadmins members)
+)
+
+/* if no parent is specified, default to root - read with symget
+  * (never re-resolved) so macro content in the param cannot execute */
+%let is_bad=0;
+data _null_;
+length _parent $512;
+_parent=coalescec(symget('parent'),'/');
+  if index(_parent,'%')>0 or index(_parent,'&')>0 then do;
+    putlog 'ERR' 'OR: Invalid parent:' _parent;
+    call symputx('is_bad',1,'l');
+  end;
+  else call symputx('parent',_parent,'g');
+run;
+
+%mp_abort(iftrue=(&is_bad=1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid parent)
+)
 
 %mp_dirlist(path=&parent,outds=dirlist, maxdepth=2)
 

sas/sasjs/services/admin/dirlist.test.sas

@@ -0,0 +1,45 @@
+/**
+  @file
+  @brief testing admin dirlist service - admin gate (security)
+  @details The service requires membership of the DC administrators
+  group.  The test suite runs as a member of that group, so the gate
+  passes and the directory listing is returned.  (A non-admin negative
+  test would need a second user, which this suite does not have.)
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+
+**/
+
+%let _program=&appLoc/services/admin/dirlist;
+
+data work.params;
+  length name $32 value $1000;
+  name='parent';value='/tmp';
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params,
+  outlib=web1
+)
+
+%let nobs=0;
+proc sql noprint;
+select count(*) into: nobs from web1.dirlist;
+quit;
+
+%mp_assert(
+  iftrue=(&nobs>0),
+  desc=Admin user gets a directory listing,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/admin/exportconfig.sas

@@ -15,6 +15,7 @@
   @li mp_ds2csv.sas
   @li mp_streamfile.sas
   @li mp_validatecol.sas
+  @li mpe_getgroups.sas
 
   @author 4GL Apps Ltd
   @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
@@ -26,6 +27,18 @@
 %global dclib islib newlib;
 %mpeinit()
 
+/* check user is in admin group */
+%let cnt=0;
+%mpe_getgroups(user=%mf_getuser(),outds=work.usergroups)
+proc sql noprint;
+select count(*) into:cnt
+  from usergroups
+  where groupname="&mpeadmins";
+%mp_abort(iftrue= (&cnt=0)
+  ,mac=&_program
+  ,msg=%str(The DC configuration can only be exported by &mpeadmins members)
+)
+
 data _null_;
   newlib=coalescec(symget('dclib'),"&mpelib");
   %mp_validatecol(newlib,ISLIB,islib)

sas/sasjs/services/admin/refreshcatalog.sas

@@ -2,10 +2,16 @@
   @file refreshcatalog.sas
   @brief Refreshes the library data catalog
   @details  A library may be passed in a LIBREF url param.
+  Requires membership of the DC administrators group.
 
   <h4> SAS Macros </h4>
   @li mpeinit.sas
   @li dc_refreshcatalog.sas
+  @li mf_getuser.sas
+  @li mpe_getgroups.sas
+  @li mp_abort.sas
+  @li mp_validatecol.sas
+  @li mpeterm.sas
 
   @version 9.3
   @author 4GL Apps Ltd
@@ -17,10 +23,54 @@
 %global libref;
 %mpeinit()
 
+/**
+  * libref is a request input used in dc_assignlib and catalog queries -
+  * it must be a well-formed libref before use.  Read with symget (never
+  * re-resolved) and validated in a data step.
+  */
+%let is_lib=0;
+data _null_;
+  length _libref $8;
+  _libref=coalescec(symget('libref'),'');
+  /* an absent libref is a valid, full catalog refresh */
+  if missing(_libref) then do;
+    call symputx('is_lib',1,'l');
+    call symputx('libref','','g');
+    stop;
+  end;
+  %mp_validatecol(_libref,ISLIB,is_lib)
+  if is_lib=0 then putlog 'ERR' 'OR: Invalid libref:' _libref;
+  call symputx('is_lib',is_lib,'l');
+  if is_lib=1 then call symputx('libref',upcase(_libref),'g');
+run;
+
+%mp_abort(iftrue= (&is_lib ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid libref)
+)
+
+/* check user is in admin group */
+%let cnt=0;
+%mpe_getgroups(user=%mf_getuser(),outds=work.usergroups)
+proc sql noprint;
+select count(*) into:cnt
+  from usergroups
+  where groupname="&mpeadmins";
+%mp_abort(iftrue= (&cnt=0)
+  ,mac=&_program
+  ,msg=%str(This service is only available to &mpeadmins members)
+)
+
 %dc_refreshcatalog(&libref)
 
 
-data _null_;
-  file _webout;
-  put '<h1> Catalog Refresh Complete </h1>';
+data sasparams;
+  length msg $64;
+  msg='Catalog Refresh Complete';
 run;
+
+%webout(OPEN)
+%webout(OBJ,sasparams)
+%webout(CLOSE)
+
+%mpeterm()

sas/sasjs/services/admin/refreshcatalog.test.1.sas

@@ -0,0 +1,78 @@
+/**
+  @file
+  @brief testing admin refreshcatalog service - admin gate + libref validation (security)
+  @details The service requires membership of the DC administrators
+  group and a well-formed libref (or none).  An invalid libref aborts
+  the service, which shows up as a canceled child job (an aborted
+  service registers no webout).  The test suite runs as a member of
+  the admin group, so the gate passes here.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+
+**/
+
+%let _program=&appLoc/services/admin/refreshcatalog;
+
+/**
+  * Test 1 - an invalid libref must abort the service
+  */
+data work.params1;
+  length name $32 value $1000;
+  name='libref';value='A.%sysevalf(3+4)B';output;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params1,
+  outref=web1,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort1=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort1',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort1=1),
+  desc=Macro content in libref aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - a valid libref still refreshes the catalog (admin user)
+  */
+data work.params2;
+  length name $32 value $1000;
+  name='libref';value='DCTEST';output;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params2,
+  outlib=web2
+)
+
+%let msgcheck=0;
+data _null_;
+  set web2.sasparams;
+  putlog (_all_)(=);
+  if index(msg,'Catalog Refresh Complete') then call symputx('msgcheck',1);
+run;
+
+%mp_assert(
+  iftrue=(&msgcheck=1),
+  desc=Valid libref refresh completes for admin user,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/admin/refreshcatalog.test.sas

@@ -20,25 +20,20 @@ run;
 %mx_execute(&_program,
   viyacontext=&defaultcontext,
   inputparams=work.params,
-  outref=webout,
-  viyaresult=WEBOUT_TXT,
+  outlib=web1,
   mdebug=&sasjs_mdebug
 )
 
-data work.results;
-  infile webout;
-  input;
-  putlog _infile_;
-  if index(upcase(_infile_),'CATALOG REFRESH COMPLETE') then do;
-    putlog 'test passed';
-    output;
-    stop;
-  end;
+%let msgcheck=0;
+data _null_;
+  set web1.sasparams;
+  putlog (_all_)(=);
+  if index(msg,'Catalog Refresh Complete') then call symputx('msgcheck',1);
 run;
 
-%mp_assertdsobs(work.results,
+%mp_assert(
+  iftrue=(&msgcheck=1),
   desc=Refresh catalog confirmation message is returned,
-  test=EQUALS 1,
   outds=work.test_results
 )
 

sas/sasjs/services/admin/refreshlibs.sas

@@ -1,11 +1,14 @@
 /**
   @file refreshlibs.sas
   @brief Refreshes the library data catalog
-  @details
+  @details Requires membership of the DC administrators group.
 
   <h4> SAS Macros </h4>
   @li mpeinit.sas
   @li mpe_refreshlibs.sas
+  @li mf_getuser.sas
+  @li mpe_getgroups.sas
+  @li mp_abort.sas
 
   @version 9.3
   @author 4GL Apps Ltd
@@ -17,4 +20,16 @@
 
 %mpeinit()
 
+/* check user is in admin group */
+%let cnt=0;
+%mpe_getgroups(user=%mf_getuser(),outds=work.usergroups)
+proc sql noprint;
+select count(*) into:cnt
+  from usergroups
+  where groupname="&mpeadmins";
+%mp_abort(iftrue= (&cnt=0)
+  ,mac=&_program
+  ,msg=%str(This service is only available to &mpeadmins members)
+)
+
 %mpe_refreshlibs()

sas/sasjs/services/auditors/getdiffs.sas

@@ -7,9 +7,11 @@
   @li mpe_getvars.sas
   @li mpe_accesscheck.sas
   @li mf_getattrn.sas
+  @li mf_getuser.sas
   @li mp_abort.sas
   @li mp_binarycopy.sas
   @li mp_streamfile.sas
+  @li mp_validatecol.sas
 
   @version 9.2
   @author 4GL Apps Ltd
@@ -22,6 +24,52 @@
 %mpeinit()
 %mpe_getvars(BrowserParams, BrowserParams);
 
+/**
+  * Validate inputs before they reach executable code.  libds is passed to
+  * the access check (which interpolates it into SQL) and table is used in
+  * the staging file path, so both must be well formed before use.  Values
+  * are re-read with symget (never re-resolved) and validated in a data
+  * step so no macro content in the request can execute.
+  */
+%let is_libds=0;
+%let is_table=0;
+%let is_csv=0;
+data _null_;
+  length _libds $64 _table $64 _csv $128;
+  _libds=symget('libds');
+  _table=symget('table');
+  _csv=coalescec(symget('stp_diffs_csv'),'tempDiffs.csv');
+  %mp_validatecol(_libds,LIBDS,is_libds)
+  %mp_validatecol(_table,ISNAME,is_table)
+  /* the diffs csv filename must stay inside the staging directory */
+  if findc(_csv,'/\')>0 or index(_csv,'..')>0 then do;
+    is_csv=0;
+    putlog 'ERR' 'OR: Invalid stp_diffs_csv:' _csv;
+  end;
+  else is_csv=1;
+  if is_libds=0 then putlog 'ERR' 'OR: Invalid libds:' _libds;
+  if is_table=0 then putlog 'ERR' 'OR: Invalid table:' _table;
+  call symputx('is_libds',is_libds,'l');
+  call symputx('is_table',is_table,'l');
+  call symputx('is_csv',is_csv,'l');
+  if is_libds=1 then call symputx('libds',_libds,'g');
+  if is_table=1 then call symputx('table',_table,'g');
+  if is_csv=1 then call symputx('stp_diffs_csv',_csv,'g');
+run;
+
+%mp_abort(iftrue= (&is_libds ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid libds)
+)
+%mp_abort(iftrue= (&is_table ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid table)
+)
+%mp_abort(iftrue= (&is_csv ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid stp_diffs_csv)
+)
+
 /* security checks */
 %let user=%mf_getuser();
 %mpe_accesscheck(&libds,outds=authEDIT,user=&user,access_level=EDIT)
@@ -51,5 +99,4 @@
 %mpestp_diffs()
 
 
-
 %mpeterm()

sas/sasjs/services/auditors/getdiffs.test.sas

@@ -0,0 +1,194 @@
+/**
+  @file
+  @brief testing getdiffs service - input validation (security)
+  @details The libds, table and stp_diffs_csv request params must be
+  well-formed before they reach the access check and the staging file
+  path.  An invalid value aborts the service, which shows up as a
+  canceled child job (an aborted service registers no webout).
+
+  A real load is staged first (stagedata) and a diffs csv written into
+  the staging directory, so every payload below resolves to that REAL
+  file when executed - on a vulnerable service the job completes, and
+  only the validating service cancels it.  The assertions cannot pass
+  against a service that does not validate.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+  @li mf_getuniquefileref.sas
+
+**/
+
+%let _program=&appLoc/services/auditors/getdiffs;
+
+/**
+  * Stage a real load so a real staging directory exists
+  */
+data work.sascontroltable;
+  action='LOAD';
+  message="getdiffs test prep";
+  libds="&dclib..MPE_X_TEST";
+  output;
+  stop;
+run;
+
+proc sql noprint;
+select max(primary_key_field) into: maxpk
+  from &dclib..mpe_x_test;
+quit;
+
+data work.jsdata;
+  set &dclib..mpe_x_test(rename=(
+    some_date=dt2 SOME_DATETIME=dttm2 some_time=tm2)
+  );
+  some_date=put(dt2,date9.);
+  SOME_DATETIME=put(dttm2,datetime19.);
+  some_time=put(tm2,time.);
+  drop dt2 dttm2 tm2;
+  if _n_=1 then do;
+    _____DELETE__THIS__RECORD_____='No';
+    some_char='getdiffs security test';
+    some_num=&maxpk+1;
+  end;
+  else stop;
+run;
+
+%mx_execute(&appLoc/services/editors/stagedata,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.jsdata work.sascontroltable,
+  outlib=webstage,
+  mdebug=&sasjs_mdebug
+)
+
+%let stagetest=0;
+data _null_;
+  set webstage.sasparams;
+  putlog (_all_)(=);
+  if status='SUCCESS' then call symputx('stagetest',1);
+  call symputx('loadref',dsid);
+run;
+
+%mp_assert(
+  iftrue=(&stagetest=1 and &syscc=0),
+  desc=stagedata succeeded in getdiffs prep,
+  outds=work.test_results
+)
+
+/**
+  * Write the diffs csv into the real staging directory
+  */
+%let diffscsv=tempDiffs_secrev.csv;
+data _null_;
+  file "&dc_staging_area/&loadref./&diffscsv";
+  put 'SOME_CHAR,_____STATUS_____';
+  put 'getdiffs security test,UPDATED';
+run;
+
+/**
+  * Test 1 - the mpe_accesscheck SQL injection payload in libds must
+  * abort the service (validation fires before the authz query, so
+  * the authz bypass cannot happen).  The payload is sent through the
+  * sasjs table channel (BrowserParams) like the frontend does - the
+  * raw-quote form is masked in plain URL params on this platform.
+  */
+%let fb1=%mf_getuniquefileref();
+data _null_;
+  file &fb1 termstr=crlf;
+  length _row $400.;
+  put 'TABLE:$41. STP_DIFFS_CSV:$100. libds:$41.';
+  _row=cats(symget('loadref'),',',symget('diffscsv'),',',
+    'SOMELIB.SOMEDS',"'22'x"," or ","'22'x",'1',"'22'x",' ne ',"'22'x",'2');
+  put _row;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&fb1:BrowserParams,
+  outref=web1,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort1=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort1',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort1=1),
+  desc=SQL injection payload in libds aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - path traversal in table must abort the service (the
+  * payload resolves to the real staged file through a .. detour)
+  */
+data _null_;
+  length _dir $512;
+  _dir=scan(symget('dc_staging_area'),-1,'/');
+  call symputx('travtable',cats('../',_dir,'/','&loadref'));
+run;
+
+%let fb2=%mf_getuniquefileref();
+data _null_;
+  file &fb2 termstr=crlf;
+  put 'TABLE:$41. STP_DIFFS_CSV:$100. libds:$41.';
+  put "&travtable.,&diffscsv.,&dclib..MPE_X_TEST";
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&fb2:BrowserParams,
+  outref=web2,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort2=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort2',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort2=1),
+  desc=Path traversal in table aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 3 - path traversal in stp_diffs_csv must abort the service
+  */
+%let fb3=%mf_getuniquefileref();
+data _null_;
+  file &fb3 termstr=crlf;
+  put 'TABLE:$41. STP_DIFFS_CSV:$100. libds:$41.';
+  put "&loadref.,../&loadref./&diffscsv.,&dclib..MPE_X_TEST";
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&fb3:BrowserParams,
+  outref=web3,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort3=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort3',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort3=1),
+  desc=Path traversal in stp_diffs_csv aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/auditors/getstagetable.test.sas

@@ -25,7 +25,14 @@ select max(primary_key_field) into: maxpk
   from &dclib..mpe_x_test;
 
 data work.jsdata;
-  set &dclib..mpe_x_test;
+  set &dclib..mpe_x_test(rename=(
+    some_date=dt2 SOME_DATETIME=dttm2 SOME_TIME=tm2)
+  );
+  /* the adapter sends these as strings - see postdata.test.1 */
+  some_date=put(dt2,date9.);
+  SOME_DATETIME=put(dttm2,datetime19.);
+  some_time=put(tm2,time.);
+  drop dt2 dttm2 tm2;
   if _n_=1 then do;
     _____DELETE__THIS__RECORD_____='No';
     some_char='getstagetable test';
@@ -84,14 +91,16 @@ data _null_;
   if some_char='getstagetable test' then call symputx('fmtcheck',1);
 run;
 
-%mp_assertdsobs(web2.stagetable,
+/* the json libname can only be read once, so assert on the flags set
+   above rather than re-reading the tables */
+%mp_assert(
+  iftrue=(&stagecheck=1),
   desc=getstagetable returns the staged table,
-  test=EQUALS 1,
   outds=work.test_results
 )
-%mp_assertdsobs(web2.fmt_stagetable,
+%mp_assert(
+  iftrue=(&fmtcheck=1),
   desc=getstagetable returns the formatted staged table,
-  test=EQUALS 1,
   outds=work.test_results
 )
 %mp_assert(

sas/sasjs/services/auditors/postdata.test.2.sas

@@ -25,9 +25,9 @@ proc format lib=DCTEST.DCFMTS cntlout=work.fmtextract;
 run;
 data work.jsdata;
   set work.fmtextract;
+  fmtrow=_n_;
   if _n_<5 then _____DELETE__THIS__RECORD_____='Yes';
   else _____DELETE__THIS__RECORD_____='No';
-  if _n_>12 then label=cats('new!',label);
   if _n_>20 then stop;
 run;
 

sas/sasjs/services/editors/loadfile.sas

@@ -18,6 +18,7 @@
   @li mpe_loader.sas
   @li mp_cleancsv.sas
   @li mp_binarycopy.sas
+  @li mp_validatecol.sas
   @li mpeinit.sas
 
   @version 9.2
@@ -113,9 +114,21 @@ data _null_;
   end;
   else call symputx('libds',libds);
   call symputx('is_fmt',is_fmt);
+  /* validate libds to prevent code injection */
+  %mp_validatecol(LIBDS,LIBDS,is_libds)
+  if is_libds=0 then do;
+    putlog 'ERR' 'OR: Invalid libds:' libds;
+    call symputx('bad_libds',1);
+  end;
+  else call symputx('bad_libds',0);
   putlog (_all_)(=);
 run;
 
+%mp_abort(iftrue= (&bad_libds=1)
+  ,mac=&_program
+  ,msg=%str(Invalid libds supplied)
+)
+
 /* check that the user has the requisite access */
 %mpe_getgroups(user=&user,outds=groups)
 

sas/sasjs/services/editors/stagedata.sas

@@ -26,6 +26,7 @@
   @li mp_binarycopy.sas
   @li mp_cntlout.sas
   @li mp_ds2csv.sas
+  @li mp_validatecol.sas
   @li mf_getplatform.sas
   @li removecolsfromwork.sas
   @li mpeinit.sas
@@ -66,9 +67,21 @@ data _null_;
   end;
   else call symputx('libds',libds);
   call symputx('is_fmt',is_fmt);
+  /* validate libds to prevent code injection */
+  %mp_validatecol(LIBDS,LIBDS,is_libds)
+  if is_libds=0 then do;
+    putlog 'ERR' 'OR: Invalid libds:' libds;
+    call symputx('bad_libds',1);
+  end;
+  else call symputx('bad_libds',0);
   putlog (_all_)(=);
 run;
 
+%mp_abort(iftrue= (&bad_libds=1)
+  ,mac=&_program
+  ,msg=%str(Invalid libds supplied)
+)
+
 %mp_cntlout(
   iftrue=(&is_fmt=1)
   ,libcat=&orig_libds

sas/sasjs/services/editors/stagedata.test.3.sas

@@ -5,6 +5,11 @@
     1. loading row level EDIT security against a table with LOADTYPE=REPLACE
     2. applying LOADTYPE=REPLACE to a table with row level EDIT security
 
+  The staged loads are aborted by the post edit hooks inside the child
+  service.  The abort is recorded as a FAILED load in MPE_LOADS (via
+  mpe_loadfail), so the assertions query MPE_LOADS rather than the child
+  response - a canceled child does not always return a webout or log.
+
   <h4> SAS Macros </h4>
   @li mx_execute.sas
   @li mp_assert.sas
@@ -31,11 +36,42 @@ insert into &dc_libref..mpe_tables
     ,buskey='X'
     ,loadtype='REPLACE'
     ,num_of_approvals_required=1;
+/* ensure the RLS table's post edit hook is registered (matches the
+  * mpe_makedata registration - some estates were initialised before the
+  * hook existed, which would silently skip the test 1 validation) */
+update &dc_libref..mpe_tables
+  set post_edit_hook='services/hooks/mpe_row_level_security_postedit'
+  where upcase(dsn)='MPE_ROW_LEVEL_SECURITY'
+    and post_edit_hook='';
+/* insert an active EDIT rule for DCTEST.WIDEBOY up front - the child
+  * services run in separate (possibly pooled) compute sessions, so the
+  * rule must be committed to the shared filesystem well before the test
+  * 2 child reads it */
+delete from &dc_libref..mpe_row_level_security
+  where rls_libref="DCTEST" and rls_table='WIDEBOY';
+select coalesce(max(rls_rk),0)+1 into: rlsrk
+  from &dc_libref..mpe_row_level_security;
+insert into &dc_libref..mpe_row_level_security
+  set tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,rls_rk=&rlsrk
+    ,rls_scope='EDIT'
+    ,rls_group='SASAdministrators'
+    ,rls_libref='DCTEST'
+    ,rls_table='WIDEBOY'
+    ,rls_group_logic='AND'
+    ,rls_subgroup_logic='OR'
+    ,rls_subgroup_id=0
+    ,rls_variable_nm='ROW_ID'
+    ,rls_operator_nm='NE'
+    ,rls_raw_value='1'
+    ,rls_active=1;
 quit;
 
 /**
   * Test 1 - submitting an RLS EDIT rule against a REPLACE table should abort
   */
+%let t1=%sysfunc(datetime());
 data work.sascontroltable;
   action='LOAD';
   message='RLS EDIT rule on REPLACE table should abort';
@@ -72,49 +108,36 @@ run;
   mdebug=&sasjs_mdebug
 )
 
-/* search terms may span multiple lines, so track matches across records */
+/* a hook-aborted load is terminated before mpe_loadfail can record a
+  * FAILED status, so the load row stays IN PROGRESS - the abort is
+  * proved by the load NOT completing successfully.  Tests in this suite run
+  * in parallel sessions, so match the row for THIS test by its unique
+  * reason_txt (the submitted message) rather than any load in the time
+  * window - an unrelated aborted/IN PROGRESS load would otherwise satisfy
+  * the assertion even if this load completed.  The row id (csv_dir/mperef)
+  * is generated inside the aborted child service, which returns no webout,
+  * so reason_txt is the only stable identifier the test controls. */
 %let abort1=0;
-data _null_;
-  retain foundabort foundmsg 0;
-  infile wb1;
-  input;
-  putlog _infile_;
-  if index(_infile_,'sasjsAbort') then foundabort=1;
-  if index(_infile_,'REPLACE loadtype') then foundmsg=1;
-  if foundabort=1 and foundmsg=1 then call symputx('abort1',1);
-run;
+proc sql noprint;
+select count(*) into: abort1
+  from &dc_libref..mpe_loads
+  where processed_dttm>&t1
+    and reason_txt="RLS EDIT rule on REPLACE table should abort"
+    and upcase(status) ne 'SUCCESS';
+quit;
 
 %mp_assert(
-  iftrue=(&abort1=1),
+  iftrue=(&abort1>0),
   desc=Checking RLS EDIT rule is rejected for REPLACE loadtype table (#211),
   outds=work.test_results
 )
 
 /**
   * Test 2 - applying REPLACE loadtype to a table with RLS EDIT security
-  * should abort.  First insert an active EDIT rule directly.
+  * should abort.  The active EDIT rule was inserted in the prep section,
+  * long before this child service reads it.
   */
-proc sql noprint;
-delete from &dc_libref..mpe_row_level_security
-  where rls_libref="DCTEST" and rls_table='WIDEBOY';
-select coalesce(max(rls_rk),0)+1 into: rlsrk
-  from &dc_libref..mpe_row_level_security;
-insert into &dc_libref..mpe_row_level_security
-  set tx_from=0
-    ,tx_to='31DEC5999:23:59:59'dt
-    ,rls_rk=&rlsrk
-    ,rls_scope='EDIT'
-    ,rls_group='SASAdministrators'
-    ,rls_libref='DCTEST'
-    ,rls_table='WIDEBOY'
-    ,rls_group_logic='AND'
-    ,rls_subgroup_logic='OR'
-    ,rls_subgroup_id=0
-    ,rls_variable_nm='ROW_ID'
-    ,rls_operator_nm='NE'
-    ,rls_raw_value='1'
-    ,rls_active=1;
-quit;
+%let t2=%sysfunc(datetime());
 
 /* now stage a REPLACE loadtype for DCTEST.WIDEBOY */
 data work.sascontroltable;
@@ -141,18 +164,16 @@ run;
 )
 
 %let abort2=0;
-data _null_;
-  retain foundabort foundmsg 0;
-  infile wb2;
-  input;
-  putlog _infile_;
-  if index(_infile_,'sasjsAbort') then foundabort=1;
-  if index(_infile_,'EDIT security') then foundmsg=1;
-  if foundabort=1 and foundmsg=1 then call symputx('abort2',1);
-run;
+proc sql noprint;
+select count(*) into: abort2
+  from &dc_libref..mpe_loads
+  where processed_dttm>&t2
+    and reason_txt="REPLACE loadtype on RLS-secured table should abort"
+    and upcase(status) ne 'SUCCESS';
+quit;
 
 %mp_assert(
-  iftrue=(&abort2=1),
+  iftrue=(&abort2>0),
   desc=Checking REPLACE loadtype is rejected for RLS EDIT-secured table (#211),
   outds=work.test_results
 )

sas/sasjs/services/public/getcols.sas

@@ -5,8 +5,8 @@
 
   <h4> SAS Macros </h4>
   @li dc_assignlib.sas
-  @li mf_getvalue.sas
   @li mp_abort.sas
+  @li mp_validatecol.sas
 
   @version 9.2
   @author 4GL Apps Ltd
@@ -18,7 +18,29 @@
 %mpeinit()
 
 
-%let ds=%mf_getvalue(work.iwant,libds);
+/**
+  * The libds is read from the IWANT input table.  Reading it with
+  * mf_getvalue would re-resolve any macro content in the value, so it is
+  * read with symget in a data step and validated (LIBREF.DATASET) before
+  * it is used in proc contents.
+  */
+%let is_libds=0;
+data _null_;
+  length _libds $64;
+  set work.iwant;
+  _libds=libds;
+  %mp_validatecol(_libds,LIBDS,is_libds)
+  if is_libds=0 then putlog 'ERR' 'OR: Invalid libds:' _libds;
+  call symputx('is_libds',is_libds,'l');
+  if is_libds=1 then call symputx('ds',upcase(_libds),'l');
+  stop;
+run;
+
+%mp_abort(iftrue= (&is_libds ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid libds)
+)
+
 %dc_assignlib(READ,%scan(&ds,1,.))
 
 proc contents noprint data=&ds

sas/sasjs/services/public/getcols.test.sas

@@ -0,0 +1,85 @@
+/**
+  @file
+  @brief testing getcols service - input validation (security)
+  @details The libds in the IWANT input table must be a well-formed
+  LIBREF.DATASET.  An invalid value aborts the service before it
+  reaches proc contents.  The abort shows up as a canceled child job
+  (an aborted service registers no webout).
+
+  The payload in test 1 resolves to a REAL table when the request
+  content is executed as macro code, so on a vulnerable service the
+  job completes, and only the validating service cancels it - the
+  assertion cannot pass against a service that does not validate.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+  @li mf_getuniquefileref.sas
+
+**/
+
+%let _program=&appLoc/services/public/getcols;
+
+/**
+  * Test 1 - macro content in libds must abort the service
+  */
+%let f1=%mf_getuniquefileref();
+data _null_;
+  file &f1 termstr=crlf;
+  put 'LIBDS:$41.';
+  put '%sysfunc(coalescec(&dclib..MPE_X_TEST,))';
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:iwant,
+  outref=web1,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort1=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort1',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort1=1),
+  desc=Macro content in libds aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - valid libds still returns columns
+  */
+%let f2=%mf_getuniquefileref();
+data _null_;
+  file &f2 termstr=crlf;
+  put 'LIBDS:$41.';
+  put "&dclib..MPE_X_TEST";
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f2:iwant,
+  outlib=web2
+)
+
+%let nobs=0;
+proc sql noprint;
+select count(*) into: nobs from web2.cols;
+quit;
+
+%mp_assert(
+  iftrue=(&nobs>0),
+  desc=Valid libds returns columns,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/public/getcolvals.sas

@@ -35,7 +35,6 @@
 
   <h4> SAS Macros </h4>
   @li mf_existds.sas
-  @li mf_getvalue.sas
   @li mf_verifymacvars.sas
   @li dc_assignlib.sas
   @li mf_getvarformat.sas
@@ -43,6 +42,8 @@
   @li mp_cntlout.sas
   @li mp_filtercheck.sas
   @li mp_filtergenerate.sas
+  @li mp_validatecol.sas
+  @li mpe_validatecol.sas
 
   @version 9.2
   @author 4GL Apps Ltd.
@@ -77,18 +78,48 @@ data _null_;
   put (_all_)(=);
 run;
 
-%let libds=%mf_getvalue(work.iwant,libds);
-%let col2=%mf_getvalue(work.iwant,col);
+/**
+  * libds and col are request inputs that flow into executable positions
+  * (set &libds, proc sql select &col2).  They are read from the IWANT
+  * table with symget in a data step (never re-resolved) and validated
+  * here before use - mf_getvalue would re-resolve any macro content in
+  * the value before this code ran.
+  */
+%let libds=;
+%let col2=;
+%let is_libds=0;
+%let is_col=0;
+data _null_;
+  length _libds $64 _col $32;
+  set work.iwant;
+  _libds=libds;
+  _col=col;
+  %mpe_validatecol(_libds,LIBDS,is_libds)
+  %mp_validatecol(_col,ISNAME,is_col)
+  if is_libds=0 then putlog 'ERR' 'OR: Invalid libds:' _libds;
+  if is_col=0 then putlog 'ERR' 'OR: Invalid col:' _col;
+  call symputx('is_libds',is_libds,'l');
+  call symputx('is_col',is_col,'l');
+  if is_libds=1 then call symputx('libds',upcase(_libds),'l');
+  if is_col=1 then call symputx('col2',upcase(_col),'l');
+  stop;
+run;
+
 %let is_fmt=0;
 %let startrow=1;
 %let rows=4000;
 
-%put &=libds;
-%put &=col2;
-
 %mp_abort(iftrue= (%mf_verifymacvars(libds col2)=0)
   ,mac=&_program..sas
-  ,msg=%str(Missing inputs from iwant. Libds=&libds col=&col2 )
+  ,msg=%str(Missing inputs from iwant)
+)
+%mp_abort(iftrue= (&is_libds ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid libds)
+)
+%mp_abort(iftrue= (&is_col ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid col)
 )
 
 %dc_assignlib(WRITE,%scan(&libds,1,.))

sas/sasjs/services/public/getcolvals.test.4.sas

@@ -0,0 +1,115 @@
+/**
+  @file
+  @brief testing getcolvals service - input validation (security)
+  @details The libds and col in the IWANT input table must be
+  well-formed (LIBREF.DATASET and SAS name).  An invalid value aborts
+  the service, which shows up as a canceled child job (an aborted
+  service registers no webout).
+
+  The payloads in tests 1-2 resolve to a REAL table / column when the
+  request content is executed as macro code, so on a vulnerable
+  service the job completes, and only the validating service cancels
+  it - the assertion cannot pass against a service that does not
+  validate.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+  @li mf_getuniquefileref.sas
+
+**/
+
+%let _program=&appLoc/services/public/getcolvals;
+
+/**
+  * Test 1 - macro content in libds must abort the service
+  */
+%let f1=%mf_getuniquefileref();
+data _null_;
+  file &f1 termstr=crlf;
+  put 'LIBDS:$19. COL:$9.';
+  put '%sysfunc(coalescec(&dclib..MPE_X_TEST,)),SOME_TIME';
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:iwant,
+  outref=web1,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort1=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort1',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort1=1),
+  desc=Macro content in libds aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - macro content in col must abort the service
+  */
+%let f2=%mf_getuniquefileref();
+data _null_;
+  file &f2 termstr=crlf;
+  put 'LIBDS:$19. COL:$9.';
+  put '&dclib..MPE_X_TEST,%sysfunc(coalescec(SOME_TIME,))';
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f2:iwant,
+  outref=web2,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort2=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort2',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort2=1),
+  desc=Macro content in col aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 3 - valid inputs still return values
+  */
+%let f3=%mf_getuniquefileref();
+data _null_;
+  file &f3 termstr=crlf;
+  put 'LIBDS:$19. COL:$9.';
+  put "&dclib..MPE_X_TEST,SOME_TIME";
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f3:iwant,
+  outlib=web3
+)
+
+%let nobs=0;
+proc sql noprint;
+select count(*) into: nobs from web3.vals;
+quit;
+
+%mp_assert(
+  iftrue=(&nobs>0),
+  desc=Valid inputs return values,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/public/getrawdata.sas

@@ -11,16 +11,17 @@
   @li filter - the filter RK if used
 
   <h4> SAS Macros </h4>
-  @li mf_verifymacvars.sas
   @li mf_getuser.sas
   @li mf_existfeature.sas
   @li dc_assignlib.sas
-  @li mp_ds2cards.sas
-  @li mp_ds2csv.sas
   @li mp_abort.sas
   @li mp_binarycopy.sas
   @li mp_cntlout.sas
+  @li mp_ds2cards.sas
+  @li mp_ds2csv.sas
   @li mp_streamfile.sas
+  @li mp_validatecol.sas
+  @li mpe_validatecol.sas
   @li mpe_filtermaster.sas
 
 
@@ -37,9 +38,57 @@
 %let user=%mf_getuser();
 %let is_fmt=0;
 
-%mp_abort(iftrue= (%mf_verifymacvars(type table)=0)
+/**
+  * Validate inputs before they reach executable code.  table is used as a
+  * dataset reference, in the output file path, and in the download filename,
+  * so it must be a well-formed LIBREF.DATASET (the trailing -FC catalog
+  * suffix is permitted); filter must be an integer.  Values are read with
+  * symget (never re-resolved) and validated in a data step so no macro
+  * content in the input can execute.
+  */
+%let is_libds=0;
+%let is_int=0;
+%let is_type=0;
+data _null_;
+  length _table $64 _filter $16 _type $16;
+  _type=upcase(coalescec(symget('type'),''));
+  _table=coalescec(symget('table'),'');
+  _filter=coalescec(symget('filter'),'0');
+  if missing(_table) then do;
+    putlog 'ERR' 'OR: Missing table';
+    stop;
+  end;
+  %mpe_validatecol(_table,LIBDS,is_libds)
+  /* an absent filter is a valid, unfiltered download */
+  if missing(_filter) then _filter='0';
+  %mp_validatecol(_filter,ISINT,is_int)
+  /* type is validated against a fixed list of download formats */
+  length _types_ok 8;
+  _types_ok=0;
+  if _type in ('SAS','CSV','EXCEL','MARKDOWN','WEBCSV','WEBTAB')
+  then _types_ok=1;
+  else putlog 'ERR' 'OR: Invalid type:' _type;
+  if is_libds=0 then putlog 'ERR' 'OR: Invalid table:' _table;
+  if is_int=0 then putlog 'ERR' 'OR: Invalid filter:' _filter;
+  call symputx('is_libds',is_libds,'l');
+  call symputx('is_int',is_int,'l');
+  call symputx('is_type',_types_ok,'l');
+  call symputx('filter',_filter,'l');
+  if is_libds=1 then call symputx('table',upcase(_table),'l');
+  if _types_ok=1 then call symputx('type',_type,'l');
+run;
+
+%mp_abort(iftrue= (&is_libds ne 1)
   ,mac=&_program..sas
-  ,msg=%str(Invalid inputs: type table)
+  ,msg=%str(Invalid table)
+)
+%mp_abort(iftrue= (&is_int ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid filter)
+)
+%mp_abort(iftrue= (&is_type ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid type)
 )
 
 %let libds=%upcase(&table); /* actual source */

sas/sasjs/services/public/getrawdata.test.1.sas

@@ -0,0 +1,152 @@
+/**
+  @file
+  @brief testing getrawdata service - input validation (security)
+  @details table must be a well-formed LIBREF.DATASET (the trailing
+  format-catalog suffix is permitted), filter must be an integer and
+  type one of the supported download types.  An invalid value aborts
+  the service before any request content can execute.  The abort is
+  asserted from the child job state: the payloads either resolve to a
+  real table when executed as macro code, or write a file outside the
+  WORK directory (verified by live probe) - so on the vulnerable
+  service the job completes, and only the validating service cancels
+  it.  The assertion cannot pass against a service that does not
+  validate.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+
+**/
+
+%let _program=&appLoc/services/public/getrawdata;
+
+/**
+  * Test 1 - macro content in table must abort the service
+  * (the payload resolves to a real table when executed as macro code)
+  */
+data work.params1;
+  length name $32 value $1000;
+  name='type';value='CSV';output;
+  name='table';value='%sysfunc(coalescec(&mpelib..MPE_X_TEST,))';output;
+  name='filter';value='0';output;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params1,
+  outref=web1,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort1=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort1',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort1=1),
+  desc=Macro content in table aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - sql injection in filter must abort the service
+  */
+data work.params2;
+  length name $32 value $1000;
+  name='type';value='CSV';output;
+  name='table';value="&dclib..MPE_X_TEST";output;
+  name='filter';value='0 or 1=1';output;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params2,
+  outref=web2,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort2=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort2',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort2=1),
+  desc=SQL injection in filter aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 3 - path traversal in table must abort the service
+  * (verified by live probe: on a vulnerable service this writes
+  * ../SECPROBE.csv outside the WORK directory and completes)
+  */
+data work.params3;
+  length name $32 value $1000;
+  name='type';value='CSV';output;
+  name='table';value='../secprobe';output;
+  name='filter';value='0';output;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params3,
+  outref=web3,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort3=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort3',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort3=1),
+  desc=Path traversal in table aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 4 - the valid request must still work (positive control)
+  */
+data work.params4;
+  length name $32 value $1000;
+  name='type';value='CSV';output;
+  name='table';value="&dclib..MPE_X_TEST";output;
+  name='filter';value='0';output;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputparams=work.params4,
+  outref=web4,
+  viyaresult=WEBOUT_TXT
+)
+
+%let ok4=0;
+data _null_;
+  infile web4;
+  input;
+  if _infile_=:'PRIMARY_KEY_FIELD' then do;
+    call symputx('ok4',1);
+    stop;
+  end;
+run;
+
+%mp_assert(
+  iftrue=(&ok4=1),
+  desc=Valid table request still returns data,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/public/validatefilter.sas

@@ -26,8 +26,9 @@
 
   <h4> SAS Macros </h4>
   @li dc_assignlib.sas
-  @li mf_getvalue.sas
+  @li mp_abort.sas
   @li mp_filterstore.sas
+  @li mpe_validatecol.sas
   @li removecolsfromwork.sas
 
   @version 9.2
@@ -40,7 +41,32 @@
 
 %mpeinit()
 
-%let ds=%upcase(%mf_getvalue(work.iwant,filter_table));
+/**
+  * filter_table is a request input that flows into executable positions
+  * (mp_filterstore libds=, which interpolates it into SQL).  It is read
+  * from the IWANT table in a data step (never re-resolved) and validated
+  * before use - mf_getvalue would re-resolve any macro content in the
+  * value before this code ran.  A format catalog is referenced as
+  * LIBREF.CATALOGNAME-FC, so that form is accepted too.
+  */
+%let ds=;
+%let is_libds=0;
+data _null_;
+  length _ds $64;
+  set work.iwant;
+  _ds=filter_table;
+  %mpe_validatecol(_ds,LIBDS,is_libds)
+  if is_libds=0 then putlog 'ERR' 'OR: Invalid filter_table:' _ds;
+  call symputx('is_libds',is_libds,'l');
+  if is_libds=1 then call symputx('ds',upcase(_ds),'l');
+  stop;
+run;
+
+%mp_abort(iftrue= (&is_libds ne 1)
+  ,mac=&_program..sas
+  ,msg=%str(Invalid filter_table)
+)
+
 %dc_assignlib(WRITE,%scan(&ds,1,.))
 
 %mp_filterstore(

sas/sasjs/services/public/validatefilter.test.1.sas

@@ -0,0 +1,97 @@
+/**
+  @file
+  @brief testing validatefilter service - input validation (security)
+  @details The filter_table in the IWANT input table must be a
+  well-formed LIBREF.DATASET.  An invalid value aborts the service
+  before it reaches mp_filterstore.  The abort shows up as a canceled
+  child job (an aborted service registers no webout).
+
+  The payload in test 1 resolves to a REAL table when the request
+  content is executed as macro code, so on a vulnerable service the
+  job completes, and only the validating service cancels it - the
+  assertion cannot pass against a service that does not validate.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+  @li mf_getuniquefileref.sas
+
+**/
+
+%let _program=&appLoc/services/public/validatefilter;
+
+/**
+  * Test 1 - macro content in filter_table must abort the service
+  */
+%let f1=%mf_getuniquefileref();
+data _null_;
+  file &f1 termstr=crlf;
+  put 'FILTER_TABLE:$41.';
+  put '%sysfunc(coalescec(&dclib..MPE_TABLES,))';
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:iwant,
+  outref=web1,
+  viyaresult=WEBOUT_TXT
+)
+
+%let abort1=0;
+data _null_;
+  set work.results;
+  if state='canceled' then call symputx('abort1',1);
+run;
+
+%mp_assert(
+  iftrue=(&abort1=1),
+  desc=Macro content in filter_table aborts the service,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - valid filter_table still stores a filter
+  */
+%let f2=%mf_getuniquefileref();
+data _null_;
+  file &f2 termstr=crlf;
+  put 'FILTER_TABLE:$41.';
+  put "&dclib..MPE_TABLES";
+run;
+%let f3=%mf_getuniquefileref();
+data _null_;
+  file &f3 termstr=crlf;
+  infile datalines4 dsd;
+  input;
+  put _infile_;
+datalines4;
+GROUP_LOGIC:$3. SUBGROUP_LOGIC:$3. SUBGROUP_ID:8. VARIABLE_NM:$32. OPERATOR_NM:$10. RAW_VALUE:$4000.
+AND,AND,1,LIBREF,CONTAINS,"'DC'"
+AND,OR,2,DSN,=,"'MPE_LOCK_ANYTABLE'"
+;;;;
+run;
+
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f2:iwant &f3:filterquery,
+  outlib=web2
+)
+
+%let nobs=0;
+proc sql noprint;
+select count(*) into: nobs from web2.result;
+quit;
+
+%mp_assert(
+  iftrue=(&nobs>0),
+  desc=Valid filter_table returns a filter result,
+  outds=work.test_results
+)
+
+/**
+  * dump results to the log for offline inspection
+  */
+data _null_;
+  set work.test_results;
+  putlog 'TEST_RESULT_LINE: ' test_result ' - ' test_description;
+run;

sas/sasjs/services/public/viewdata.sas

@@ -8,10 +8,16 @@
   <h4> Service Inputs </h4>
 
   <h5> SASCONTROLTABLE </h5>
-  |LIBDS:$41.|FILTER_RK:$5.|SEARCHTYPE:$4|SEARCHVAL:$1000
+  |LIBDS:$41.|FILTER_RK:best.|SEARCHTYPE:$4|SEARCHVAL:$100|
   |---|---|---|---
   |DC258467.MPE_X_TEST|-1|CHAR|Some String|
 
+  No input column may be longer than the length declared in this service -
+  LIBDS 41, SEARCHTYPE 4, SEARCHVAL 100.  An input column longer than the
+  declared length stops the step with "Multiple lengths were specified for
+  the variable ...", so the SASjs adapter sends the $char form (eg $char4.),
+  which also preserves any leading blanks.
+
   <h4> Service Outputs </h4>
 
   <h5> cols </h5>
@@ -271,7 +277,7 @@ run;
       %let dsobs=%mf_nobs(MPSEARCH.vwsearch);
       data viewdata;
         set MPSEARCH.vwsearch;
-        if _n_<&DC_MAXOBS_WEBVIEW;
+        if _n_<=&DC_MAXOBS_WEBVIEW;
       run;
     %end;
     %else %do;

sas/sasjs/services/public/viewdata.test.3.sas

@@ -0,0 +1,123 @@
+/**
+  @file
+  @brief testing public/viewdata service full table search semantics
+  @details The viewer's search box runs a full table search through
+  %mp_searchdata.  A character search is a partial (CONTAINS) match against
+  every character column of the table and is case sensitive; a numeric search
+  is an exact match against every numeric column.
+
+  The fixture below holds both cases of the same substring so the assertions
+  can show the behaviour in both directions - a value with a capital letter is
+  not found by a lowercase search and vice versa.
+
+  <h4> SAS Macros </h4>
+  @li mf_getuniquefileref.sas
+  @li mf_getuniquelibref.sas
+  @li mp_assert.sas
+  @li mx_execute.sas
+
+**/
+
+%let _program=&appLoc/services/public/viewdata;
+
+/* deterministic fixture - mixed case, and a value that only exists in the
+  second character column, so the search must cover every character column */
+data &dclib..mpe_x_search;
+  length NAME $32 NOTES $64;
+  name='Selkie';        qty=12345; notes='grey seal folklore';   output;
+  name='Vampire squid'; qty=23456; notes='deep sea cephalopod';  output;
+  name='siphonophore';  qty=34567; notes='colonial organism';    output;
+  name='Smithson';      qty=45678; notes='capital S surname';    output;
+  name='Goldsmith';     qty=56789; notes='lowercase substring';  output;
+run;
+
+/**
+  * Run one full table search and assert the row count the viewer would show.
+  */
+%macro search(searchtype,searchval,expected,desc);
+%local f1 outlib nobs;
+%let f1=%mf_getuniquefileref();
+%let outlib=%mf_getuniquelibref(prefix=web);
+data _null_;
+  file &f1 termstr=crlf;
+  put 'LIBDS:$char41. FILTER_RK:best. SEARCHTYPE:$char4. SEARCHVAL:$char100.';
+  put "&dclib..MPE_X_SEARCH,-1,&searchtype,&searchval";
+run;
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:SASControlTable ,
+  outlib=&outlib,
+  mdebug=&sasjs_mdebug
+)
+%let nobs=0;
+data _null_;
+  set &outlib..sasparams;
+  putlog (_all_)(=);
+  call symputx('nobs',nobs);
+run;
+%mp_assert(
+  iftrue=(&nobs=&expected),
+  desc=&desc (&searchtype "%superq(searchval)" returned &nobs rows, expected &expected),
+  outds=work.test_results
+)
+%mend search;
+
+/* a character search matches part of a value, in the same case */
+%search(CHAR,Selkie,1,Checking a same-case partial match returns the row)
+%search(CHAR,quid,1,Checking a partial match on the middle of a value)
+%search(CHAR,seal,1,Checking the search covers every character column)
+
+/* the same term in another case does not match */
+%search(CHAR,selkie,0,Checking a lowercase search does not match a capital S)
+%search(CHAR,SELKIE,0,Checking an uppercase search does not match mixed case)
+%search(CHAR,Quid,0,Checking a capital Q does not match a lowercase value)
+
+/* a search in the case the value is stored in finds it, whether the match
+  starts the value or sits inside it */
+%search(CHAR,Smith,1,Checking Smith finds Smithson)
+%search(CHAR,smith,1,Checking smith finds Goldsmith)
+
+/* a numeric search is an exact match, not a partial one */
+%search(NUM,12345,1,Checking an exact numeric search returns the row)
+%search(NUM,1234,0,Checking a partial numeric search returns nothing)
+
+/* a search that matches more rows than the web view cap returns the cap,
+  and reports the same number of rows as it returns */
+data &dclib..mpe_x_cap;
+  length txt $24;
+  do i=1 to 600;
+    txt='needle'!!cats(i);
+    output;
+  end;
+run;
+
+%let f1=%mf_getuniquefileref();
+%let outlib=%mf_getuniquelibref(prefix=web);
+data _null_;
+  file &f1 termstr=crlf;
+  put 'LIBDS:$char41. FILTER_RK:best. SEARCHTYPE:$char4. SEARCHVAL:$char100.';
+  put "&dclib..MPE_X_CAP,-1,CHAR,needle";
+run;
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:SASControlTable ,
+  outlib=&outlib,
+  mdebug=&sasjs_mdebug
+)
+%let capnobs=0;
+%let capmax=0;
+data _null_;
+  set &outlib..sasparams;
+  putlog (_all_)(=);
+  call symputx('capnobs',nobs);
+  call symputx('capmax',maxrows);
+run;
+%let caprows=0;
+proc sql noprint;
+  select count(*) into :caprows from &outlib..viewdata;
+quit;
+%mp_assert(
+  iftrue=(&caprows=&capnobs and &capnobs gt 0 and &capnobs le &capmax),
+  desc=Checking a search beyond the cap returns as many rows as it reports (&capnobs reported, &caprows returned, cap &capmax),
+  outds=work.test_results
+)

sas/sasjs/tests/testinit.sas

@@ -16,7 +16,7 @@
 %let syscc=0;
 %global apploc _program dclib defaultcontext _debug sasjs_mdebug dc_dttmtfmt;
 
-%let defaultcontext=SAS Job Execution compute context;
+%let defaultcontext=Compute Reusable;
 %let sasjs_mdebug=0;
 
 options mprint mprintnest nobomfile lrecl=32767;

sas/sasjs/tests/testsetup.sas

@@ -29,7 +29,7 @@
 
 %global apploc _program;
 
-%let defaultcontext=SAS Job Execution compute context;
+%let defaultcontext=Compute Reusable;
 
 data _null_;
   length _pgm $1000;

sas/sasjs/utils/chunk_deploy.py

@@ -141,10 +141,19 @@ print(f'\nDeploying {len(chunk_files)} chunks to nextviya...')
 for idx, cf in enumerate(chunk_files, 1):
     print(f'\n[{idx}/{len(chunk_files)}] Deploying {os.path.basename(cf)} ...')
     timeout = 600 if os.path.getsize(cf) > 5 * 1024 * 1024 else 300
-    result = subprocess.run(
-        ['npx', 'sasjs', 'run', cf, '-t', 'nextviya'],
-        capture_output=True, text=True, timeout=timeout
-    )
+    result = None
+    for attempt in range(1, 4):
+        try:
+            result = subprocess.run(
+                ['npx', 'sasjs', 'run', cf, '-t', 'nextviya'],
+                capture_output=True, text=True, timeout=timeout
+            )
+            break
+        except subprocess.TimeoutExpired:
+            print(f'  attempt {attempt} timed out after {timeout}s'
+                  + (' - retrying...' if attempt < 3 else ' - GIVING UP'))
+    if result is None:
+        sys.exit(1)
     out = '\n'.join([l for l in result.stdout.splitlines() if not l.startswith('isTokenExpiring')])
     tail_lines = out.splitlines()[-40:]
     print('\n'.join(tail_lines))
@@ -153,5 +162,4 @@ for idx, cf in enumerate(chunk_files, 1):
         print(result.stderr[-1000:] if len(result.stderr) > 1000 else result.stderr)
         sys.exit(1)
     print(f'  -> OK')
-
 print('\nAll chunks deployed successfully!')