fix(lint): make noBannedStrings an error and clear its 13 findings

Rung 3 of the #376 ratchet: `sas/.sasjslint` promotes noBannedStrings to
"error". The rule flags the literal words "error " / "warning " anywhere in
a file, comments included, because they turn up in a grep of the SAS log
and bury the real messages. Its 13 findings are cleared:

- Nine are prose and are reworded, so the sentence both reads normally and
  stops matching: a truncation that aborts a build, a syntax fault, an
  abort condition, "only records the failure in work.mp_abort_errds" (four
  copies of the mp_abort-inside-%include comment), a test description, a
  hook comment.
- Two are the text the user sees. loadfile.sas reports
  "(mf_abort.error file found)" - the marker file carries the word in its
  name - and buildtermsas9.sas reports "Some final error in &_program";
  both are reworded to say the same thing without the literal, since the
  text sits in a macro argument where the %str(WARN)ING split does not
  survive the quoting.
- Three keep the word because the word is the content: the doxygen
  @warning tags in macros/bitemporal_dataloader.sas and the two SAS 9
  admin STPs (makedata, makelib). Each carries
  @sasjslint {"noBannedStrings": false}, as
  targets/server/services_server/admin/makedata.sas already did.

VERIFICATION

- `npm run lint:sas`: 5 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- Deployed to sas.4gl.io and ran the SAS suite there:
  `sasjs test` reports "Tests that pass: 64/64 (100%)". The run used a
  scratch appLoc (/Public/app/dc-lint-ratchet) seeded with makedata, so
  the shared /Public/app/dc deployment was left alone; the scratch target
  was removed from sasjsconfig.json before this commit.

5 warnings remain: macro name length (2) and case (2),
indentationMultiple (1) - the last rung.

.agents/skills/dc-estate-testing/SKILL.md

@@ -44,7 +44,7 @@ An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check
 
 `npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
 
-- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `AllUsers`, which is not a group on the shared estate - the account belongs to `all-users` (check with a call to `services/public/getgroups`, or read the `MF_GETUSER` the response echoes back). With a group the account is not in, the seed still answers `SUCCESS` and every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Override `ADMIN` to `all-users` for the estate rather than editing the shipped JSON.
+- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
 - **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
 - **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
 - **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.

.git-hooks/pre-commit

@@ -20,6 +20,16 @@ if ! "$gitleaks_bin" protect --staged --redact; then
     exit 1
 fi
 
+# Every tracked .npmrc is a policy file - root, sas/ and client/ - so a registry
+# credential pasted into any of them is one `git add -A` from being committed.
+# Each staged one is checked, not just the client.
+for npmrc in $(git diff --cached --name-only --diff-filter=ACMR | grep -E '(^|/)\.npmrc$'); do
+    if git show ":$npmrc" | grep -Eq '(_authToken|_auth[^a-z]|_password|:_auth)'; then
+        echo "ERROR: $npmrc holds a registry credential - keep it in ~/.npmrc instead"
+        exit 1
+    fi
+done
+
 ## Avoid large commits
 # https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
 size_limit=$((2 * 1024 * 1024)) # 2MB

.gitea/scripts/cypress-specs.sh

@@ -9,21 +9,27 @@
 #   - hook-programs asserts that an edit raises an approval row against its
 #     target table; it runs early, before other specs have added rows to the
 #     approval tables.
+#   - metanav only reads the metadata tree, and applies a licence key so it can
+#     run on its own; it goes last so that key cannot change another spec.
 #
 # Specs added later are appended rather than inserted, so the sequence that was
-# known green keeps its shape. metanav is deliberately absent - see
-# https://git.datacontroller.io/dc/dc/issues/395.
+# known green keeps its shape.
 #
-# download is held out for a different reason - its download button is covered by
-# a fixed-position element, see
-# https://git.datacontroller.io/dc/dc/issues/398.
+# download was held out while its selectors were stale - see
+# https://git.datacontroller.io/dc/dc/issues/398. Its first case uploads a fixture
+# to mpe_x_test and submits it, so it creates the approval it downloads instead of
+# depending on what earlier specs leave behind. That approval stays: the CI job
+# deletes the appLoc and re-runs makedata before the specs, so nothing accumulates
+# between runs, but a hand-run against a reused estate leaves one pending row per
+# run. It is appended because a spec added later goes on the end rather than
+# reordering the sequence that was known green.
 set -euo pipefail
 
-SPECS="csv-limited hook-programs liveness editor excel-multi-load excel csv filtering filter-panel licensing viewer-labels full-table-search viewbox stage browser-info ddl-export row-cell-limits derived-variables startupservice-launcher coltype-delete-record submitted-details embed-va"
+SPECS="csv-limited hook-programs liveness editor excel-multi-load excel csv filtering filter-panel licensing viewer-labels full-table-search viewbox stage browser-info ddl-export row-cell-limits derived-variables startupservice-launcher coltype-delete-record submitted-details embed-va download metanav"
 
 # Specs that exist but are held out of CI, each with the reason. Listed here so
 # the coverage check below can tell a deliberate exclusion from a forgotten file.
-EXCLUDED="metanav download"
+EXCLUDED=""
 
 case "${1:-}" in
   all)

.gitea/workflows/build.yaml

@@ -3,6 +3,8 @@ run-name: Running Lint Check and Licence checker on Pull Request
 on: [pull_request]
 
 env:
+  # the server the mocked services and app tests run against - bump deliberately
+  SASJS_SERVER_VERSION: '3.6.2'
   NODE_VERSION: '24.15.0'
 
 jobs:
@@ -22,9 +24,8 @@ jobs:
         # credential must fail the build, not just a commit.
         # Uses the same @nogoo9/gitleaks binary the pre-commit hook pins - the
         # gitleaks-action requires a license key for organization repos.
-        # Runs before client/.npmrc is written below, because that file is
-        # created from the NPMRC secret and a later scan would flag the
-        # registry token this job injects itself.
+        # Runs before the registry config is written, so the scan cannot see
+        # the token this job injects.
         # --no-git scans the working tree, which is what a pull request gate
         # is about.
         run: |
@@ -38,7 +39,11 @@ jobs:
           apt install -y ./google-chrome*.deb
 
       - name: Write .npmrc file
-        run: echo "$NPMRC" >> client/.npmrc
+        run: |
+          # the registry secret stays outside the repo: client/.npmrc is
+          # tracked now, and one git add -A would commit a live token
+          echo "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
         shell: bash
         env:
           NPMRC: ${{ secrets.NPMRC}}
@@ -51,7 +56,13 @@ jobs:
             gpg --batch --yes --passphrase-fd 0 \
             --output ./libraries/sheet-crypto.tgz \
             --decrypt ./libraries/sheet-crypto.tgz.gpg
-          npm ci
+          # dependency hooks stay off (client/.npmrc); the client's own
+          # generator runs explicitly, because the app imports the file it makes
+          npm ci --ignore-scripts
+          npm run postinstall
+          # cypress downloads its binary in a dependency install hook, which the
+          # policy skips, so it is installed explicitly here
+          npx cypress install
 
       - name: Check audit
         # Audit should fail and stop the CI on any vulnerability in root and sas, and on low+ in client
@@ -62,9 +73,27 @@ jobs:
           cd ../client
           npm audit --omit=dev
 
+      - name: Check archiver directory walk
+        # the readdir-glob minimatch override in client/package.json is a major
+        # bump; archiver's directory() is the only runtime caller of that path,
+        # so exercise it rather than trusting that the install succeeded
+        run: |
+          cd client
+          npm run check:archiver
+
       - name: Lint check
         run: npm run lint:check
 
+      - name: SAS lint check
+        # sas/.sasjslint promotes each rule to "error" as its findings are
+        # cleared, and only errors change the exit code - so this gate fails on
+        # a regression in a cleared rule and passes while other rules still
+        # carry warnings.
+        run: |
+          cd sas
+          npm ci --ignore-scripts
+          npm run lint:sas
+
       - name: Licence checker
         run: |
           cd client
@@ -102,6 +131,11 @@ jobs:
   # limits how far sharding helps.  The specs it runs come from
   # .gitea/scripts/cypress-specs.sh.
 
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"
+
   Build-and-test-development:
     runs-on: ubuntu-latest
     needs: Build-and-ng-test
@@ -119,8 +153,13 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          touch client/.npmrc
-          echo '${{ secrets.NPMRC}}' > client/.npmrc
+          # the secret arrives through the environment, not interpolated into
+          # this script: a quote or $( in the value would otherwise break the
+          # step or execute
+          printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
+        env:
+          NPMRC: ${{ secrets.NPMRC}}
 
       - name: Install system dependencies
         run: |
@@ -143,7 +182,13 @@ jobs:
           cd client
           # Decrypt and Install sheet
           echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
-          npm ci
+          # dependency hooks stay off (client/.npmrc); the client's own
+          # generator runs explicitly, because the app imports the file it makes
+          npm ci --ignore-scripts
+          npm run postinstall
+          # cypress downloads its binary in a dependency install hook, which the
+          # policy skips, so it is installed explicitly here
+          npx cypress install
 
       - name: Build frontend for web streaming
         # The server-ci target streams client/dist as the 'clickme' web app (see
@@ -154,11 +199,21 @@ jobs:
           cd client
           npm run build
 
+      - name: Cache the npm downloads
+        # The server is installed from npm at a pinned version, so the tarball is
+        # immutable and the key never needs invalidating by hand: a hit skips
+        # the 10.7MB download, a miss costs exactly what the zip cost before.
+        # continue-on-error, because a runner without cache support should still
+        # run the pipeline - this step only ever saves time.
+        continue-on-error: true
+        uses: actions/cache@v4
+        with:
+          path: ~/.npm
+          key: npm-sasjs-server-${{ env.SASJS_SERVER_VERSION }}
+
       - name: Setup and start SASjs server
         run: |
           npm i -g pm2
-          curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
-          unzip linux.zip
           touch .env
           echo RUN_TIMES=js >> .env
           echo NODE_PATH=node >> .env
@@ -170,11 +225,10 @@ jobs:
           # in a throwaway container, so opt in explicitly.
           echo RUN_AS=root >> .env
           cat .env
-          pm2 start api-linux --wait-ready
-          # `pm2 start --wait-ready` can return while the app is still
-          # launching (it does not always wait for the ready signal), and the
-          # next step hits :5000 immediately. Wait for the server to answer,
-          # and dump its log if it never does.
+          pm2 start npx --name sasjs-server -- --yes @sasjs/server@${{ env.SASJS_SERVER_VERSION }}
+          # `pm2 start` returns before the app is listening, and the next step
+          # hits :5000 immediately. Wait for the server to answer, and dump its
+          # log if it never does.
           for i in $(seq 1 60); do
             if curl -sf -o /dev/null http://localhost:5000/; then
               echo "SASjs server answered on attempt ${i}"
@@ -184,7 +238,7 @@ jobs:
           done
           echo "SASjs server did not come up within 120s"
           pm2 list
-          pm2 logs api-linux --lines 100 --nostream
+          pm2 logs sasjs-server --lines 100 --nostream
           exit 1
 
       - name: Deploy mocked services
@@ -252,3 +306,8 @@ jobs:
         with:
           name: cypress-videos.zip
           path: cypress-videos.zip
+
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"

.gitea/workflows/docs.yaml

@@ -28,8 +28,9 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          printf '%s\n' "$NPMRC" >> client/.npmrc
-          echo "legacy-peer-deps=true" >> client/.npmrc
+          printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "legacy-peer-deps=true" >> "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
         shell: bash
         env:
           NPMRC: ${{ secrets.NPMRC }}
@@ -40,6 +41,9 @@ jobs:
           # Decrypt and Install sheet
           echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
         shell: bash
         env:
           SHEET_PWD: ${{ secrets.SHEET_PWD }}
@@ -76,6 +80,11 @@ jobs:
             --password "$SURFER_PASSWORD" \
             --message "Typedoc from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
 
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"
+
   sas-docs:
     runs-on: ubuntu-latest
     concurrency:

.gitea/workflows/lighthouse.yaml

@@ -3,6 +3,8 @@ run-name: Running Lighthouse Performance and Accessibility Checks on Pull Reques
 on: [pull_request]
 
 env:
+  # the server the mocked services and app tests run against - bump deliberately
+  SASJS_SERVER_VERSION: '3.6.2'
   NODE_VERSION: '24.15.0'
 
 jobs:
@@ -26,6 +28,18 @@ jobs:
       - name: Install global packages
         run: npm i -g pm2 @sasjs/cli wait-on
 
+      - name: Cache the npm downloads
+        # The server is installed from npm at a pinned version, so the tarball is
+        # immutable and the key never needs invalidating by hand: a hit skips
+        # the 10.7MB download, a miss costs exactly what the zip cost before.
+        # continue-on-error, because a runner without cache support should still
+        # run the pipeline - this step only ever saves time.
+        continue-on-error: true
+        uses: actions/cache@v4
+        with:
+          path: ~/.npm
+          key: npm-sasjs-server-${{ env.SASJS_SERVER_VERSION }}
+
       - name: Setup and start SASjs server
         run: |
           touch .env
@@ -39,13 +53,10 @@ jobs:
           # in a throwaway container, so opt in explicitly.
           echo RUN_AS=root >> .env
           cat .env
-          curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
-          unzip linux.zip
-          pm2 start api-linux --wait-ready
-          # `pm2 start --wait-ready` can return while the app is still
-          # launching (it does not always wait for the ready signal), and the
-          # next step hits :5000 immediately. Wait for the server to answer,
-          # and dump its log if it never does.
+          pm2 start npx --name sasjs-server -- --yes @sasjs/server@${{ env.SASJS_SERVER_VERSION }}
+          # `pm2 start` returns before the app is listening, and the next step
+          # hits :5000 immediately. Wait for the server to answer, and dump its
+          # log if it never does.
           for i in $(seq 1 60); do
             if curl -sf -o /dev/null http://localhost:5000/; then
               echo "SASjs server answered on attempt ${i}"
@@ -55,11 +66,15 @@ jobs:
           done
           echo "SASjs server did not come up within 120s"
           pm2 list
-          pm2 logs api-linux --lines 100 --nostream
+          pm2 logs sasjs-server --lines 100 --nostream
           exit 1
 
       - name: Write .npmrc file
-        run: echo "$NPMRC" > client/.npmrc
+        run: |
+          # the registry secret stays outside the repo: client/.npmrc is
+          # tracked now, and one git add -A would commit a live token
+          echo "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
         shell: bash
         env:
           NPMRC: ${{ secrets.NPMRC}}
@@ -73,6 +88,9 @@ jobs:
             --output ./libraries/sheet-crypto.tgz \
             --decrypt ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
           npm install -g replace-in-files-cli
 
       - name: Update appLoc in index.html
@@ -109,3 +127,8 @@ jobs:
           name: Lighthouse results
           path: client/lighthouse-reports
           include-hidden-files: true
+
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"

.gitea/workflows/release-assets.yaml

@@ -40,8 +40,9 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          printf '%s\n' "$NPMRC" >> client/.npmrc
-          echo "legacy-peer-deps=true" >> client/.npmrc
+          printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "legacy-peer-deps=true" >> "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
         shell: bash
         env:
           NPMRC: ${{ secrets.NPMRC }}
@@ -91,6 +92,9 @@ jobs:
           # Decrypt and Install sheet
           echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
           npm i webpack
           npm run build
         shell: bash
@@ -207,3 +211,8 @@ jobs:
           done
           echo "Uploading SHA256SUMS ..."
           curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
+
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"

.gitea/workflows/release.yaml

@@ -6,6 +6,8 @@ on:
       - main
 
 env:
+  # the server the mocked services and app tests run against - bump deliberately
+  SASJS_SERVER_VERSION: '3.6.2'
   NODE_VERSION: '24.5.0'
 
 jobs:
@@ -22,8 +24,13 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          touch client/.npmrc
-          echo '${{ secrets.NPMRC}}' > client/.npmrc
+          # the secret arrives through the environment, not interpolated into
+          # this script: a quote or $( in the value would otherwise break the
+          # step or execute
+          printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
+        env:
+          NPMRC: ${{ secrets.NPMRC}}
 
       - name: Install Chrome for Angular tests
         run: |
@@ -43,6 +50,9 @@ jobs:
           # Decrypt and Install sheet
           echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
 
       - name: Check audit
         # Audit should fail and stop the CI on any vulnerability in root and sas, and on low+ in client
@@ -62,8 +72,14 @@ jobs:
         run: |
           cd client
           npm run postinstall
+          npm run check:archiver
           npm run build
 
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"
+
   Build-and-test-development:
     runs-on: ubuntu-latest
     needs: Build-production-and-ng-test
@@ -81,8 +97,13 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          touch client/.npmrc
-          echo '${{ secrets.NPMRC}}' > client/.npmrc
+          # the secret arrives through the environment, not interpolated into
+          # this script: a quote or $( in the value would otherwise break the
+          # step or execute
+          printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
+        env:
+          NPMRC: ${{ secrets.NPMRC}}
 
       - name: Install system dependencies
         run: |
@@ -106,6 +127,9 @@ jobs:
           # Decrypt and Install sheet
           echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
 
       - name: Build frontend for web streaming
         # The server-ci target streams client/dist as the 'clickme' web app, so
@@ -117,11 +141,21 @@ jobs:
           cd client
           npm run build
 
+      - name: Cache the npm downloads
+        # The server is installed from npm at a pinned version, so the tarball is
+        # immutable and the key never needs invalidating by hand: a hit skips
+        # the 10.7MB download, a miss costs exactly what the zip cost before.
+        # continue-on-error, because a runner without cache support should still
+        # run the pipeline - this step only ever saves time.
+        continue-on-error: true
+        uses: actions/cache@v4
+        with:
+          path: ~/.npm
+          key: npm-sasjs-server-${{ env.SASJS_SERVER_VERSION }}
+
       - name: Setup and start SASjs server
         run: |
           npm i -g pm2
-          curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
-          unzip linux.zip
           touch .env
           echo RUN_TIMES=js >> .env
           echo NODE_PATH=node >> .env
@@ -133,11 +167,10 @@ jobs:
           # in a throwaway container, so opt in explicitly.
           echo RUN_AS=root >> .env
           cat .env
-          pm2 start api-linux --wait-ready
-          # `pm2 start --wait-ready` can return while the app is still
-          # launching (it does not always wait for the ready signal), and the
-          # next step hits :5000 immediately. Wait for the server to answer,
-          # and dump its log if it never does.
+          pm2 start npx --name sasjs-server -- --yes @sasjs/server@${{ env.SASJS_SERVER_VERSION }}
+          # `pm2 start` returns before the app is listening, and the next step
+          # hits :5000 immediately. Wait for the server to answer, and dump its
+          # log if it never does.
           for i in $(seq 1 60); do
             if curl -sf -o /dev/null http://localhost:5000/; then
               echo "SASjs server answered on attempt ${i}"
@@ -147,7 +180,7 @@ jobs:
           done
           echo "SASjs server did not come up within 120s"
           pm2 list
-          pm2 logs api-linux --lines 100 --nostream
+          pm2 logs sasjs-server --lines 100 --nostream
           exit 1
 
       - name: Deploy mocked services
@@ -174,6 +207,9 @@ jobs:
           replace-in-files --regex='"password".*' --replacement='"password":"'${{ secrets.CYPRESS_PWD_SASJS }}'" ' ./cypress.env.json
           cat ./cypress.env.json
           npm run postinstall
+          # cypress downloads its binary in a dependency install hook, which the
+          # client policy skips
+          npx cypress install
           # Prepare index.html to SASJS local
           replace-in-files --regex='serverUrl=".*?"' --replacement='serverUrl="http://localhost:5000"' ./src/index.html
           replace-in-files --regex='appLoc=".*?"' --replacement='appLoc="/Public/app/dc"' ./src/index.html
@@ -196,6 +232,11 @@ jobs:
           name: cypress-videos.zip
           path: cypress-videos.zip
 
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"
+
   release:
     runs-on: ubuntu-latest
     needs: [Build-production-and-ng-test, Build-and-test-development]
@@ -222,8 +263,9 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          echo "$NPMRC" > client/.npmrc
-          echo "legacy-peer-deps=true" >> client/.npmrc
+          echo "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "legacy-peer-deps=true" >> "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
         shell: bash
         env:
           NPMRC: ${{ secrets.NPMRC}}
@@ -241,6 +283,9 @@ jobs:
           # Decrypt and Install sheet
           echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
           npm i webpack
           npm run build
 
@@ -459,3 +504,8 @@ jobs:
             -H 'Content-Type: application/json' \
             --data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
             "$BASE/actions/workflows/sites.yaml/dispatches"
+
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"

.gitea/workflows/sites.yaml

@@ -10,6 +10,8 @@ on:
   workflow_dispatch:
 
 env:
+  # the server the mocked services and app tests run against - bump deliberately
+  SASJS_SERVER_VERSION: '3.6.2'
   NODE_VERSION: '24.15.0'
   APP_LOC: /Public/app/dc
   SURFER_HOST: https://demo.datacontroller.io
@@ -42,8 +44,13 @@ jobs:
 
       - name: Write .npmrc file
         run: |
-          touch client/.npmrc
-          echo '${{ secrets.NPMRC}}' > client/.npmrc
+          # the secret arrives through the environment, not interpolated into
+          # this script: a quote or $( in the value would otherwise break the
+          # step or execute
+          printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
+          echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
+        env:
+          NPMRC: ${{ secrets.NPMRC}}
 
       - name: Install client dependencies
         run: |
@@ -51,6 +58,9 @@ jobs:
           # Decrypt and Install sheet
           echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
+          # client install scripts are off (client/.npmrc); the generator that
+          # creates src/environments/version.ts runs explicitly
+          npm run postinstall
 
       - name: Build frontend
         # The site serves this build. `sasjs build` rewrites its index.html to
@@ -66,19 +76,29 @@ jobs:
           cd sas
           npm ci
 
+      - name: Cache the npm downloads
+        # The server is installed from npm at a pinned version, so the tarball is
+        # immutable and the key never needs invalidating by hand: a hit skips
+        # the 10.7MB download, a miss costs exactly what the zip cost before.
+        # continue-on-error, because a runner without cache support should still
+        # run the pipeline - this step only ever saves time.
+        continue-on-error: true
+        uses: actions/cache@v4
+        with:
+          path: ~/.npm
+          key: npm-sasjs-server-${{ env.SASJS_SERVER_VERSION }}
+
       - name: Start SASjs server
         # Needed only to run makedata: the drive the site ships with is captured
         # from a real server (see the snapshot step below).
         run: |
-          curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
-          unzip -q linux.zip
           touch .env
           echo RUN_TIMES=js >> .env
           echo NODE_PATH=node >> .env
           echo HTTP_PORT=5000 >> .env
           # The server refuses to run as root unless RUN_AS names an account
           echo RUN_AS=root >> .env
-          nohup ./api-linux > server.log 2>&1 &
+          nohup npx --yes @sasjs/server@${{ env.SASJS_SERVER_VERSION }} > server.log 2>&1 &
           for i in $(seq 1 60); do
             if curl -sf -o /dev/null http://localhost:5000/; then
               echo "SASjs server answered on attempt ${i}"
@@ -181,3 +201,8 @@ jobs:
             --message "$MESSAGE" \
             "${SITE_ARGS[@]}" \
             sas/mocks/sasjsbuild/mock
+
+      - name: Remove registry config
+        # written earlier in this job; nothing deletes it on the way out
+        if: always()
+        run: rm -f "$RUNNER_TEMP/npmrc"

.gitignore

@@ -20,7 +20,6 @@ sasjsbuild
 sasjsresults
 .env*
 .sasjsrc
-client/.npmrc
 *~
 .lighthouseci
 audit.log

AGENTS.md

@@ -31,9 +31,11 @@ Rationale: hard-wrapped prose produces noisy diffs when sentences are edited and
 Never consider a change complete until the relevant linters pass on the files you touched — do not rely on the user's pre-commit hooks to catch it:
 
 - **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
-- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
+- **SAS**: after creating or modifying any `.sas` files, run `npm run lint:sas` from the `sas/` directory. That script runs the `@sasjs/cli` version pinned in `sas/package.json`, so it matches CI; a global or `npx` `sasjs` may resolve a different `@sasjs/lint` and report a different rule set.
+  Rules are promoted to `error` in `sas/.sasjslint` one at a time as their findings reach zero, and only errors change the exit code - so the whole project must stay clean of the promoted rules, not just the files you touched. Warnings under rules not yet promoted are the remaining backlog; do not add to them in files you touch, and do not clear other files' findings in the same change unless that rule is the point of the change.
   `sas/.sasjslint` carries `ignoredMacros: ["webout", "mpeinit", "mpeterm"]`. `webout` is the linter's own default - `sasjs compile` injects `%webout` into every service - and an explicit list replaces that default, so it has to be restated there. `mpeinit` and `mpeterm` are listed because every service is compiled with a serviceInit program (`sasjs/utils/serviceinitsas9|viya|server.sas`) that declares both macros in its own header, so a service body may call them without declaring them. Add a macro to that list only when it reaches a file the same way - injected by the compiler, through serviceInit, testSetUp or another implicit provider - never to silence a genuinely missing declaration.
   `noBannedStrings` flags the literal words `error ` and `warning ` anywhere in a file, comments included. Reword prose; where the text has to stay (a real `%put`, a doxygen `@warning` tag), keep the log output identical with `%put %str(WARN)ING:` rather than a per-file override, and use `@sasjslint {"noBannedStrings": false}` only when the word itself is the content.
+  Hook scripts (`sasjs/services/hooks/*.sas`) run inside the service that includes them, so the macros they call come from that service's session. They carry a file-local `@sasjslint {"ignoredMacros": [...]}` rather than a macro list of their own, because end users copy and edit these scripts and should not have to maintain a declaration section.
 
 ## No external assets
 

CHANGELOG.md

@@ -1,3 +1,17 @@
+# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)
+
+
+### Bug Fixes
+
+* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
+* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
+* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
+
+
+### Features
+
+* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
+
 # [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
 
 

CONTRIBUTING.md

@@ -46,8 +46,31 @@ echo "YOUR_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
   --decrypt ./libraries/sheet-crypto.tgz.gpg
 
 npm i
+npm run postinstall
 ```
 
+The root and `sas/` `.npmrc` files set `ignore-scripts=true` too, so the composite `install`
+script in the root `package.json` (which installs both subprojects) does not run on `npm i` at the
+root. Install each part explicitly:
+
+```bash
+npm ci              # root tooling: gitleaks, prettier, semantic-release
+cd client && npm ci && npm run postinstall
+cd ../sas && npm ci
+```
+
+The client `.npmrc` is now tracked and sets `ignore-scripts=true`, so dependency install hooks do not
+run. Two of them matter here: the client's own `postinstall` - which generates
+`src/environments/version.ts`, imported throughout the app - and cypress's binary download. Run
+`npm run postinstall` after installing (and `npx cypress install` if you are running the Cypress
+suite).
+
+Registry credentials do **not** belong in `client/.npmrc`: it is a tracked policy file, and a token
+written there is one `git add -A` away from being committed. Keep the token in `~/.npmrc`, npm's
+default user config, or point npm at a file outside the repo with
+`NPM_CONFIG_USERCONFIG=/path/to/npmrc`. The pre-commit hook rejects a staged `client/.npmrc` that
+holds `_authToken`, `_auth` or `_password`.
+
 [Handsontable](https://www.npmjs.com/package/handsontable)
 
 The licence key comes from the backend licence: the decrypted licence carries `hot_license_key`,

client/.npmrc

@@ -0,0 +1,5 @@
+ignore-scripts=true
+save-exact=true
+# dependency install hooks are skipped: the client's own postinstall generates
+# src/environments/version.ts and the hot icons, so run `npm run postinstall`
+# after installing (the root install script does this for you)

client/cypress.config.ts

@@ -54,6 +54,43 @@ export default defineConfig({
               err ? reject(err) : resolve(null)
             )
           })
+        },
+
+        /**
+         * Writes a copy of a fixture whose primary keys are renumbered from a
+         * random base, and returns the path to write it to.
+         *
+         * An upload that matches what the table already holds produces no
+         * changes, and a changeset with no changes has no approval waiting - so
+         * re-running a spec that uploads the same file finds nothing on the
+         * approve screen.  That is not a UI fault, and a retry after a partial
+         * first attempt hits it every time.  New keys make the upload a real
+         * insert whichever attempt gets there.
+         */
+        makeUniqueExcel({ source, dest }: { source: string; dest: string }) {
+          // eslint-disable-next-line @typescript-eslint/no-var-requires
+          const ExcelJS = require('exceljs')
+          const workbook = new ExcelJS.Workbook()
+          const base = 1000000 + Math.floor(Math.random() * 8000000)
+
+          return workbook.xlsx.readFile(source).then(() => {
+            const sheet = workbook.worksheets[0]
+            let rowNumber = 0
+
+            sheet.eachRow((row: any, index: number) => {
+              if (index === 1) return // header
+              row.getCell(1).value = base + rowNumber++
+            })
+
+            return workbook.xlsx.writeFile(dest).then(() => dest)
+          })
+        },
+
+        /** removes one file - deleteFolder would take the source fixture too */
+        deleteFile(filePath: string) {
+          return new Promise((resolve) => {
+            ;(fs as any).rm(filePath, { force: true }, () => resolve(null))
+          })
         }
       })
 

client/cypress/e2e/download.cy.ts

@@ -7,12 +7,21 @@ const serverType = Cypress.env('serverType')
 const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
 const downloadsFolder = Cypress.config('downloadsFolder')
 
+/* set by test 1 and removed afterwards, since it is written per attempt */
+let generatedExcel = ''
+
 import { deleteDownloadsFolder } from '../util/deleteDownloadFolder'
 
 context('download files test: ', function () {
   this.beforeAll(() => {
     cy.visit(`${hostUrl}/SASLogon/logout`)
-    cy.loginAndUpdateValidKey()
+    /* the estate ships no licence, so it runs on the free tier - and the free
+      tier disables file upload, which is the whole point of the audit case.
+      Asking for a licensed estate here means the spec does not depend on
+      another spec having planted a key first: the key persists in the estate's
+      config, so whether this case could run depended on the order the specs
+      were sharded into. */
+    cy.loginAndUpdateValidKey(true)
   })
 
   this.beforeEach(() => {
@@ -37,22 +46,84 @@ context('download files test: ', function () {
 
   this.afterEach(() => {
     deleteDownloadsFolder()
+
+    if (generatedExcel) {
+      cy.task('deleteFile', 'cypress/fixtures/excels/' + generatedExcel)
+      generatedExcel = ''
+    }
   })
 
   it('1 | downloads audit file', (done) => {
-    visitPage('approve/toapprove')
+    /* The approve grid renders a download button per table that is waiting for
+      approval, and a freshly seeded estate has none, so create one: upload a
+      fixture to the test table and submit it from the stage screen.  The row is
+      then found on the approve screen by the table it names, because the stage
+      route does not carry that table's id.
+      The submitted row is left behind: CI recreates the appLoc and re-runs
+      makedata before the specs.  On an estate reused by hand the rows accumulate
+      and the approve grid paginates, so a row pushed past the first page would
+      not be found - the table name still anchors the assertion, but it cannot
+      rescue a row that is not on the page. */
+    /* the fixture is renumbered before every attempt: an upload identical to
+      what the table already holds produces no changes, and a changeset with no
+      changes leaves nothing waiting for approval.  On a retry that is exactly
+      what happens - the first attempt's rows are already in the table - so the
+      keys come from a random base and the upload is always an insert. */
+    generatedExcel = 'generated_' + Date.now() + '.xlsx'
 
-    cy.get('.app-loading', { timeout: longerCommandTimeout })
-      .should('not.exist')
-      .then(() => {
-        cy.get('.btn.btn-success')
-          .should('be.visible')
-          .then((buttons) => {
-            buttons[0].click()
-            const id = buttons[0].id
+    cy.task('makeUniqueExcel', {
+      source: 'cypress/fixtures/excels/regular_excel.xlsx',
+      dest: 'cypress/fixtures/excels/' + generatedExcel
+    })
 
-            checkForFileDownloaded(id, 'zip', () => done())
-          })
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    attachExcelFile(generatedExcel)
+
+    submitExcel()
+
+    /* the stage route carries a load reference, not the table id: an upload
+      lands on /stage/DC20261031-201657.470_601010_4449 rather than on
+      /stage/<table id>, so taking the id from the URL looked for a button that
+      could never exist.  The row is found by the table it names, and the id the
+      download is named after is read from the button itself. */
+    visitPage('review/approve')
+
+    cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
+      'not.exist'
+    )
+
+    cy.get('clr-datagrid clr-dg-row', { timeout: longerCommandTimeout }).should(
+      ($rows) => {
+        const texts = Array.from($rows).map((row: any) =>
+          row.innerText.replace(/\s+/g, ' ')
+        )
+
+        expect(
+          texts.some((text: string) => text.includes('MPE_X_TEST')),
+          `no approval row names MPE_X_TEST - rows: ${texts
+            .slice(0, 5)
+            .join(' | ')
+            .slice(0, 300)}`
+        ).to.be.true
+      }
+    )
+
+    cy.get('clr-datagrid clr-dg-row')
+      .filter(':contains("MPE_X_TEST")')
+      .first()
+      .find('[aria-label="Download audit file"]')
+      .should('have.length', 1)
+      .then(($btn) => {
+        const tableId = $btn.attr('id') as string
+
+        expect(tableId, 'the approve button carries the table id').to.be.a(
+          'string'
+        )
+
+        cy.wrap($btn).click()
+
+        checkForFileDownloaded(tableId, 'zip', () => done())
       })
   })
 
@@ -83,7 +154,7 @@ context('download files test: ', function () {
 
     openDownloadModal(() => {
       cy.get('select')
-        .select('Excel')
+        .select('EXCEL')
         .then(() => {
           cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
             button.trigger('click')
@@ -103,7 +174,7 @@ context('download files test: ', function () {
 
     openDownloadModal(() => {
       cy.get('select')
-        .select('SAS Datalines')
+        .select('SAS')
         .then(() => {
           cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
             button.trigger('click')
@@ -123,7 +194,7 @@ context('download files test: ', function () {
 
     openDownloadModal(() => {
       cy.get('select')
-        .select('SAS DDL')
+        .select('SAS_DDL')
         .then(() => {
           cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
             button.trigger('click')
@@ -143,7 +214,7 @@ context('download files test: ', function () {
 
     openDownloadModal(() => {
       cy.get('select')
-        .select('TSQL DDL')
+        .select('TSQL_DDL')
         .then(() => {
           cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
             button.trigger('click')
@@ -163,7 +234,7 @@ context('download files test: ', function () {
 
     openDownloadModal(() => {
       cy.get('select')
-        .select('PGSQL DDL')
+        .select('PGSQL_DDL')
         .then(() => {
           cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
             button.trigger('click')
@@ -209,18 +280,23 @@ const checkForFileDownloaded = (
 }
 
 const openDownloadModal = (callback?: any) => {
-  cy.get('.btn.btn-sm.btn-outline.filterSide.dropdown-toggle')
+  /* filterSide is the viewer's options dropdown; it no longer carries a
+    dropdown-toggle class - see filter-panel.cy.ts and row-cell-limits.cy.ts.
+    Its entries are div[clrDropdownItem], not buttons, so select Download by its
+    text the way filter-panel.cy.ts selects Filter. */
+  cy.get('.btn.btn-sm.btn-outline.filterSide', {
+    timeout: longerCommandTimeout
+  })
     .click()
     .then(() => {
-      cy.get('clr-dropdown-menu button').then((buttons) => {
-        for (let button of buttons) {
-          if (button.innerText.toLowerCase().includes('download')) {
-            button.click()
+      /* the click handler is on the clrDropdownItem element, not on the span
+        holding the text, and the whole label is matched: more than one entry
+        may mention Download, and an ambiguous cy.contains fails outright */
+      cy.contains('clr-dropdown-menu [clrDropdownItem]', /^\s*Download\s*$/, {
+        timeout: longerCommandTimeout
+      }).click()
 
-            if (callback) callback()
-          }
-        }
-      })
+      if (callback) callback()
     })
 }
 
@@ -259,3 +335,45 @@ const openTableFromTree = (libNameIncludes: string, tablename: string) => {
       })
     })
 }
+
+const fixturePath = 'excels'
+
+/**
+ * Uploads a fixture to the open table.  The editor renders the file input inside
+ * its upload modal, which the Upload button in the button bar opens, so that
+ * button is clicked first - by its label, not by position, so a change to the
+ * bar's button order cannot break this.
+ */
+const attachExcelFile = (excelFilename: string) => {
+  cy.get('.buttonBar button', { timeout: longerCommandTimeout })
+    .contains('Upload')
+    .click()
+
+  cy.get('input[type="file"]#file-upload', { timeout: longerCommandTimeout })
+    .should('have.length', 1)
+    .attachFile(`/${fixturePath}/${excelFilename}`)
+
+  /* a parsed upload raises the info modal with this title.  The modal component
+    is the same one aborts use, so the title is what distinguishes them: any
+    other text here - an abort, a validation error - fails the case with its own
+    message rather than being clicked through. */
+  cy.get('.clr-abort-modal .modal-title', { timeout: longerCommandTimeout })
+    .should('be.visible')
+    .invoke('text')
+    .should('include', 'Table Found')
+
+  cy.get('.clr-abort-modal .modal-footer button').contains('Close').click()
+}
+
+const submitExcel = () => {
+  cy.get('.buttonBar button.preview-submit', {
+    timeout: longerCommandTimeout
+  }).click()
+
+  /* the URL is already a stage route before the submit lands, so waiting on it
+    proves nothing; the editor leaving preview mode is the signal that the
+    submit was accepted */
+  cy.get('.buttonBar button.preview-submit', {
+    timeout: longerCommandTimeout
+  }).should('not.exist')
+}

client/cypress/e2e/metanav.cy.ts

@@ -1,16 +1,13 @@
-const username = Cypress.env('username')
-const password = Cypress.env('password')
+export {}
+
 const hostUrl = Cypress.env('hosturl')
 const appLocation = Cypress.env('appLocation')
 const longerCommandTimeout = Cypress.env('longerCommandTimeout')
-const serverType = Cypress.env('serverType')
-const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
-const fixturePath = 'excels_general/'
 
 context('metanav tests: ', function () {
   this.beforeAll(() => {
     cy.visit(`${hostUrl}/SASLogon/logout`)
-    cy.loginAndUpdateValidKey()
+    cy.loginAndUpdateValidKey(true)
   })
 
   this.beforeEach(() => {
@@ -21,14 +18,34 @@ context('metanav tests: ', function () {
     visitPage('view/metadata')
   })
 
-  it('1 | Opens metadata object', (done) => {
-    openFirstMetadataFromTree(() => {
-      // BLOCKER
-      // For unkown reasons, .clr-accordion-header-button always null although it is present on the page.
-      cy.get('.clr-accordion-header-button').then((panelNodes: any) => {
-        panelNodes[0].querySelector('button').click()
-      })
-    })
+  it('1 | Opens metadata object', () => {
+    openFirstMetadataType()
+
+    /**
+     * Query the accordion at document level. The panel renders in the content
+     * area, while `.within(<tree node>)` scopes every following `cy.get` to
+     * that node alone - which finds nothing however long it retries.
+     */
+    cy.get('.clr-accordion-header-button', { timeout: longerCommandTimeout })
+      .should('have.length.greaterThan', 0)
+      .first()
+      // Clarity puts the class on the button itself, so it is the click target.
+      .click()
+
+    // The panel reports its own open state, so the assertion is on the app's
+    // marker rather than on the content happening to be present.
+    cy.get('.clr-accordion-panel-open').should('have.length', 1)
+    cy.get('.clr-accordion-header-button')
+      .first()
+      .should('have.attr', 'aria-expanded', 'true')
+
+    // Opening the panel loads the object's details: the association tree and
+    // the attribute grid both come from `metanav/metadetails`.
+    cy.get('.clr-accordion-inner-content clr-tree-node', {
+      timeout: longerCommandTimeout
+    }).should('have.length.greaterThan', 0)
+    cy.get('clr-dg-row').should('have.length.greaterThan', 0)
+    cy.get('clr-dg-cell').contains('Name')
   })
 
   this.afterEach(() => {
@@ -36,23 +53,16 @@ context('metanav tests: ', function () {
   })
 })
 
-const openFirstMetadataFromTree = (callback?: any) => {
-  cy.get('.app-loading', { timeout: longerCommandTimeout })
-    .should('not.exist')
-    .then(() => {
-      cy.get('.nav-tree clr-tree > clr-tree-node', {
-        timeout: longerCommandTimeout
-      }).then((treeNodes: any) => {
-        let firstMetaNode
-
-        firstMetaNode = treeNodes[1]
-
-        cy.get(firstMetaNode).within(() => {
-          cy.get('.clr-treenode-content').click()
-          callback()
-        })
-      })
-    })
+const openFirstMetadataType = () => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
+  // The metadata types are the tree nodes after the search box; the search box
+  // is its own node (`.search-node`), so selecting by position would tie the
+  // spec to that detail. The <p> is the element that owns the click handler.
+  cy.get('.nav-tree clr-tree > clr-tree-node:not(.search-node) p', {
+    timeout: longerCommandTimeout
+  })
+    .first()
+    .click()
 }
 
 const visitPage = (url: string) => {

client/package-lock.json

@@ -82,6 +82,7 @@
         "@types/randombytes": "^2.0.3",
         "@typescript-eslint/eslint-plugin": "8.65.0",
         "@typescript-eslint/parser": "8.65.0",
+        "archiver": "8.0.0",
         "core-js": "^2.5.4",
         "cypress": "^15.19.0",
         "cypress-file-upload": "^5.0.8",
@@ -5208,6 +5209,12 @@
         "node": ">=18.0.0"
       }
     },
+    "node_modules/@isaacs/nonrecursive-glob-match": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/@isaacs/nonrecursive-glob-match/-/nonrecursive-glob-match-1.0.3.tgz",
+      "integrity": "sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==",
+      "license": "BlueOak-1.0.0"
+    },
     "node_modules/@istanbuljs/schema": {
       "version": "0.1.6",
       "resolved": "https://registry.npmjs.org/@istanbuljs/schema/-/schema-0.1.6.tgz",
@@ -15246,9 +15253,9 @@
       "license": "ISC"
     },
     "node_modules/follow-redirects": {
-      "version": "1.16.1",
-      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.1.tgz",
-      "integrity": "sha512-FNvFGzoMLWmE6Yj9spb/zjd7yiNCHiAW9/Tg9CXrQ8wuu32HtlJOwWO11OJafl5FfY3DxTdQ0vj42zU1kvv5jg==",
+      "version": "1.16.2",
+      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.2.tgz",
+      "integrity": "sha512-/kVP2U8z596Ia0gwFrwyIeo74ta5aVmSeU1FfAUMAmcEP5EP7zydmag3SLiN64NaRKef3lxsq7i/g7ltJQDGow==",
       "funding": [
         {
           "type": "individual",
@@ -19280,6 +19287,7 @@
       "version": "10.2.6",
       "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.6.tgz",
       "integrity": "sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==",
+      "dev": true,
       "license": "BlueOak-1.0.0",
       "dependencies": {
         "brace-expansion": "^5.0.8"
@@ -22269,6 +22277,22 @@
         "url": "https://github.com/sponsors/yqnn"
       }
     },
+    "node_modules/readdir-glob/node_modules/minimatch": {
+      "version": "11.0.3",
+      "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-11.0.3.tgz",
+      "integrity": "sha512-iny/40PEGkNJQ5/Pv6z3nm6FbRAby32GeZRjCUJk2WnNDbgPSTcdI+DDvLqsgBdD4cD5sMtMheqPPnehCCa8Nw==",
+      "license": "BlueOak-1.0.0",
+      "dependencies": {
+        "@isaacs/nonrecursive-glob-match": "^1.0.3",
+        "brace-expansion": "^5.0.12"
+      },
+      "engines": {
+        "node": "18 || 20 || >=22"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/isaacs"
+      }
+    },
     "node_modules/readdir-scoped-modules": {
       "version": "1.1.0",
       "resolved": "https://registry.npmjs.org/readdir-scoped-modules/-/readdir-scoped-modules-1.1.0.tgz",

client/package.json

@@ -3,43 +3,44 @@
   "description": "DataController Client",
   "angular-cli": {},
   "scripts": {
-    "start": "node --max_old_space_size=4096 node_modules/@angular/cli/bin/ng serve",
-    "prestart": "npm run generate-eula",
-    "generate-eula": "node ./src/eula.ts",
-    "license-checker": "node licenseChecker.js",
-    "prebuild": "node ./src/version.ts && npm run generate-eula && npm run license-checker",
+    "add-githook": "[ -d ../.git ] && git config core.hooksPath ./.git-hooks || true",
+    "audit:prod": "npm audit --omit=dev",
     "build": "node --max_old_space_size=4096 node_modules/@angular/cli/bin/ng build --configuration production",
-    "postbuild": "rimraf dist/3rdpartylicenses.txt",
     "build-dev": "node --max_old_space_size=4096 node_modules/@angular/cli/bin/ng build --configuration development",
     "build-watch": "node --max_old_space_size=4096 node_modules/@angular/cli/bin/ng build --configuration development --watch",
-    "sync": "./node_modules/.bin/watch --wait=3 \"echo Account: ${npm_config_account} && npm run deploy_${npm_config_server} --account=${npm_config_account} && notify-send 'Data Controller' 'App is synced!' ; echo 'App is synced!'\" dist",
-    "deploy_sas9": "rsync -avhe ssh ./dist/* --delete ${npm_config_account}@sas.4gl.io:/opt/sas/sas9/config/Lev1/Web/WebServer/htdocs/${npm_config_account}/dc/dev",
-    "deploy_viya": "rsync -avhe ssh ./dist/* --delete ${npm_config_account}@sas.4gl.io:/var/www/html/${npm_config_account}/dc/dev",
-    "deploy_sasjs": "rsync -avhe ssh ./dist/* --delete root@${npm_config_account}.4gl.io:/var/www/html/dc/dev",
-    "viyabuild": "cd build; ./viyabuild.sh",
-    "lint": "npm run lint:fix",
-    "lint:fix": "npx prettier --write \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
-    "lint:fix:silent": "npx prettier --log-level silent --write \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
-    "lint:check": "npx prettier --check \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
-    "lint:check:silent": "npx prettier --log-level silent --check \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
-    "test": "npx ng test",
-    "test:headless": "npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI",
-    "watch": "ng test watch=true",
-    "pree2e": "webdriver-manager update",
-    "e2e": "protractor protractor.config.js",
-    "postinstall": "node ./src/version.ts && npm run add-githook && node ./scripts/strip-clr-base64-fonts.mjs && node ./scripts/gen-hot-icons.mjs",
-    "add-githook": "[ -d ../.git ] && git config core.hooksPath ./.git-hooks || true",
-    "cypress": "cypress open",
-    "cy:run": "cypress run",
-    "audit:prod": "npm audit --omit=dev",
-    "sasdocs": "sasjs doc && ./sasjs/utils/deploydocs.sh",
+    "check:archiver": "node scripts/check-archiver.mjs",
     "compodoc:build": "compodoc -p tsconfig.doc.json --name 'Data Controller Client'",
     "compodoc:build-and-serve": "compodoc -p tsconfig.doc.json -s --name 'Data Controller Client'",
     "compodoc:serve": "compodoc -s --name 'Data Controller Client'",
+    "cy:run": "cypress run",
+    "cypress": "cypress open",
+    "deploy_sas9": "rsync -avhe ssh ./dist/* --delete ${npm_config_account}@sas.4gl.io:/opt/sas/sas9/config/Lev1/Web/WebServer/htdocs/${npm_config_account}/dc/dev",
+    "deploy_sasjs": "rsync -avhe ssh ./dist/* --delete root@${npm_config_account}.4gl.io:/var/www/html/dc/dev",
+    "deploy_viya": "rsync -avhe ssh ./dist/* --delete ${npm_config_account}@sas.4gl.io:/var/www/html/${npm_config_account}/dc/dev",
+    "e2e": "protractor protractor.config.js",
+    "generate-eula": "node ./src/eula.ts",
+    "license-checker": "node licenseChecker.js",
     "lighthouse": "lhci autorun",
+    "lint": "npm run lint:fix",
+    "lint:check": "npx prettier --check \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
+    "lint:check:silent": "npx prettier --log-level silent --check \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
+    "lint:fix": "npx prettier --write \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
+    "lint:fix:silent": "npx prettier --log-level silent --write \"{src,test}/**/*.{ts,tsx,js,jsx,html,css,sass,less,yml,md,graphql}\" \"cypress/e2e/*.cy.ts\"",
     "ng": "ng",
+    "postbuild": "rimraf dist/3rdpartylicenses.txt",
+    "postinstall": "node ./src/version.ts && npm run add-githook && node ./scripts/strip-clr-base64-fonts.mjs && node ./scripts/gen-hot-icons.mjs",
+    "prebuild": "node ./src/version.ts && npm run generate-eula && npm run license-checker",
+    "pree2e": "webdriver-manager update",
+    "prestart": "npm run generate-eula",
+    "sasdocs": "sasjs doc && ./sasjs/utils/deploydocs.sh",
+    "start": "node --max_old_space_size=4096 node_modules/@angular/cli/bin/ng serve",
+    "sync": "./node_modules/.bin/watch --wait=3 \"echo Account: ${npm_config_account} && npm run deploy_${npm_config_server} --account=${npm_config_account} && notify-send 'Data Controller' 'App is synced!' ; echo 'App is synced!'\" dist",
+    "test": "npx ng test",
+    "test:headless": "npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI",
+    "test:tz-east": "TZ=Pacific/Auckland npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'",
     "test:tz-west": "TZ=America/New_York npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'",
-    "test:tz-east": "TZ=Pacific/Auckland npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'"
+    "viyabuild": "cd build; ./viyabuild.sh",
+    "watch": "ng test watch=true"
   },
   "private": true,
   "dependencies": {
@@ -118,6 +119,7 @@
     "@types/randombytes": "^2.0.3",
     "@typescript-eslint/eslint-plugin": "8.65.0",
     "@typescript-eslint/parser": "8.65.0",
+    "archiver": "8.0.0",
     "core-js": "^2.5.4",
     "cypress": "^15.19.0",
     "cypress-file-upload": "^5.0.8",
@@ -151,11 +153,12 @@
     "lighthouse": "13.4.0",
     "fast-uri": "4.2.3",
     "readdir-glob": {
-      "brace-expansion": "^5.0.9"
+      "brace-expansion": "5.0.12",
+      "minimatch": "11.0.3"
     },
     "exceljs": {
-      "archiver": "^8.0.0",
-      "unzipper": "^0.12.5"
+      "archiver": "8.0.0",
+      "unzipper": "0.12.5"
     }
   }
 }

client/scripts/check-archiver.mjs

@@ -0,0 +1,74 @@
+/* Guards the readdir-glob -> minimatch override in client/package.json.
+ *
+ * The override forces minimatch 11 where readdir-glob asks for ^10, and
+ * readdir-glob reaches minimatch on two paths: the directory walk, and the
+ * pattern/ignore matching behind archiver's glob(). Both are exercised here,
+ * because a matcher that breaks shows up as files silently skipped rather than
+ * as an error.
+ *
+ * Entries are read from archiver's own 'entry' events, so nothing has to be
+ * unzipped and the check needs no reader dependency.
+ */
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
+import { ZipArchive } from 'archiver'
+
+const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dc-arch-'))
+const source = path.join(root, 'source')
+
+const cleanup = () => fs.rmSync(root, { recursive: true, force: true })
+const fail = (message) => {
+  console.error(message)
+  cleanup()
+  process.exit(1)
+}
+
+fs.mkdirSync(source)
+fs.writeFileSync(path.join(source, 'a.txt'), 'hello')
+fs.mkdirSync(path.join(source, 'sub'))
+fs.writeFileSync(path.join(source, 'sub', 'b.txt'), 'world')
+fs.writeFileSync(path.join(source, 'sub', 'c.md'), 'ignored by pattern')
+
+let runs = 0
+
+const archiveWith = (configure) =>
+  new Promise((resolve, reject) => {
+    const out = path.join(root, `test-${++runs}.zip`)
+    const sink = fs.createWriteStream(out)
+    const archive = new ZipArchive({ zlib: { level: 9 } })
+    const names = []
+    archive.on('entry', (entry) => names.push(entry.name))
+    archive.on('error', reject)
+    sink.on('error', reject)
+    sink.on('close', () => resolve(names))
+    archive.pipe(sink)
+    configure(archive)
+    archive.finalize()
+  })
+
+try {
+  /* the directory walk: every file, nested directories included */
+  const walked = await archiveWith((archive) => archive.directory(source, false))
+  const missing = ['a.txt', 'sub/b.txt'].filter((n) => !walked.includes(n))
+  if (missing.length) {
+    fail('directory walk missed: ' + missing.join(', ') + ' (got: ' + walked.join(', ') + ')')
+  }
+
+  /* pattern and ignore matching, which is minimatch used directly */
+  const globbed = await archiveWith((archive) =>
+    archive.glob('**/*.txt', { cwd: source, ignore: ['sub/**'] })
+  )
+  if (!globbed.includes('a.txt')) {
+    fail('glob did not match a.txt (got: ' + globbed.join(', ') + ')')
+  }
+  if (globbed.some((n) => n.startsWith('sub/'))) {
+    fail('glob ignore did not exclude sub/ (got: ' + globbed.join(', ') + ')')
+  }
+
+  console.log('archiver walk ok: ' + walked.join(', '))
+  console.log('archiver glob ok: ' + globbed.join(', '))
+  cleanup()
+} catch (err) {
+  fail('archiver error: ' + err.message)
+}

client/src/app/deploy/sections/automatic/automatic.component.html

@@ -8,20 +8,6 @@
       @if (autodeployDone) {
         <p class="m-0 align-self-start">Done</p>
         <hr class="w-100" />
-        @if (autoDeployStatus.deployServicePack !== null) {
-          <div class="deploy-status-row">
-            @if (autoDeployStatus.deployServicePack === true) {
-              <clr-icon
-                class="deploy-success"
-                shape="success-standard"
-              ></clr-icon>
-            }
-            @if (!autoDeployStatus.deployServicePack === false) {
-              <clr-icon class="deploy-error" shape="times-circle"></clr-icon>
-            }
-            <p>Deploy SAS Jobs</p>
-          </div>
-        }
         @if (autoDeployStatus.runMakeData !== null) {
           <div class="deploy-status-row">
             @if (autoDeployStatus.runMakeData) {
@@ -33,16 +19,13 @@
             @if (autoDeployStatus.runMakeData === false) {
               <clr-icon class="deploy-error" shape="times-circle"></clr-icon>
             }
-            <p>Create database</p>
+            <p>Create control library</p>
           </div>
         }
         <hr class="w-100" />
         <div class="buttons">
           <button (click)="navigateToHome()" class="btn btn-primary mt-15 mr-0">
-            @if (
-              autoDeployStatus.deployServicePack === false ||
-              autoDeployStatus.runMakeData === false
-            ) {
+            @if (autoDeployStatus.runMakeData === false) {
               <clr-icon class="deploy-error" shape="times-circle"></clr-icon>
             }
             LAUNCH
@@ -78,9 +61,9 @@
 
 <h4 class="text-center my-15">Viya Deploy</h4>
 <p class="deploy-intro">
-  Configure and deploy Data Controller to your Viya environment. Deployment
-  creates the backend SAS services and the control library used by the
-  application.
+  Configure Data Controller for your Viya environment. The frontend and services
+  are already in place; this runs the one-off job that creates the control
+  library and its configuration data, then removes itself.
 </p>
 <hr />
 
@@ -248,19 +231,8 @@
   </div>
 }
 
-<!-- Keeping this for a reference in case future VIYA changes and starts allowing separate backend and frontend) -->
-
-<!-- <clr-checkbox-wrapper>
-  <input
-    clrCheckbox
-    [(ngModel)]="recreateDatabase"
-    (click)="recreateDatabaseClicked($event)"
-    type="checkbox"
-    checked
-    />
-  <label>Recreate database</label>
-</clr-checkbox-wrapper> -->
-
+<!-- Runs the one-off makedata job. This is the screen's only action: the
+     frontend and services are already deployed. -->
 <hr />
 
 <button
@@ -269,53 +241,3 @@
 >
   Deploy
 </button>
-
-<!-- Keeping this for a reference in case future VIYA changes and starts allowing separate backend and frontend) -->
-
-<!-- <button
-(click)="executeJson()"
-class="btn-autodeploy btn btn-primary d-inline-block mr-10"
-[disabled]="!jsonFile"
->
-Deploy {{ !jsonFile ? '(json file is not available)' : '' }}
-</button> -->
-
-<!-- <button
-(click)="uploadJsonAuto.click()"
-class="btn-autodeploy btn btn-primary d-inline-block mr-10"
->
-Upload different file to deploy
-</button>
-<input
-  #uploadJsonAuto
-  type="file"
-  hidden
-  (click)="clearUploadInput($event)"
-  (change)="onJsonFileChange($event)"
-  /> -->
-
-<clr-modal [(clrModalOpen)]="recreateDatabaseModal" [clrModalClosable]="false">
-  <h3 class="modal-title">Warning</h3>
-  <div class="modal-body">
-    This action will recreate the database (if it exists). For an initial
-    deployment, this is expected. If this is a re-deployment, you will lose any
-    existing tables in <strong>{{ dcPath }}</strong
-    >.
-  </div>
-  <div class="modal-footer">
-    <button
-      type="button"
-      class="btn btn-sm btn-primary"
-      (click)="recreateDatabaseModal = false; recreateDatabase = false"
-    >
-      Cancel
-    </button>
-    <button
-      type="button"
-      class="btn btn-sm btn-primary"
-      (click)="recreateDatabaseModal = false; recreateDatabase = true"
-    >
-      Ok
-    </button>
-  </div>
-</clr-modal>

client/src/app/deploy/sections/automatic/automatic.component.spec.ts

@@ -0,0 +1,153 @@
+import { SASjsConfig } from '@sasjs/adapter'
+import { AutomaticComponent } from './automatic.component'
+
+/**
+ * The Viya setup screen has one job: run `makedata` with the selections the
+ * form collected - the admin group, the DC path, and the compute context the
+ * job runs in. These tests pin that payload and the new-window URL, because a
+ * screen whose only action silently stopped calling the service would still
+ * compile, still render, and still look like it worked.
+ */
+describe('AutomaticComponent (Viya setup)', () => {
+  const build = () => {
+    const requests: any[][] = []
+
+    const sasJs: any = {
+      request: (...args: any[]) => {
+        requests.push(args)
+        return Promise.resolve({ result: [{}] })
+      }
+    }
+
+    const sasService: any = {
+      getSasjsConfig: () => new SASjsConfig(),
+      getExecutionPath: () => '/SASJobExecution/',
+      shouldLogin: { next: () => undefined },
+      viyaMakedataSuccessfull: () => Promise.resolve(true),
+      updateFileContent: () => Promise.resolve()
+    }
+
+    const component = new AutomaticComponent(
+      {
+        showAbortModal: () => undefined,
+        showInfoModal: () => undefined
+      } as any,
+      { downloadFile: () => undefined } as any,
+      sasService,
+      {} as any,
+      { error: () => undefined } as any,
+      {} as any
+    )
+
+    component.sasJs = sasJs
+    const config = {
+      contextName: 'Compute Reusable',
+      serverUrl: 'https://viya.example.com',
+      appLoc: '/Users/demo/dc'
+    } as any
+    component.sasJsConfig = config
+    // The new-window URL is built from the adapter config the component holds,
+    // while the request path reads the @Input one.
+    component.sasjsConfig = config
+    component.appLoc = '/Users/demo/dc'
+    component.dcPath = '/export/dc'
+    component.selectedAdminGroup = 'viyagroup18'
+
+    return { component, requests, sasService }
+  }
+
+  it('posts makedata with the admin group and the DC path', async () => {
+    const { component, requests } = build()
+    component.deployInNewWindow = false
+
+    await component.createDatabase()
+
+    expect(requests[0][0]).toBe('services/admin/makedata')
+    expect(requests[0][1]).toEqual({
+      fromjs: [{ ADMIN: 'viyagroup18', DCPATH: '/export/dc' }]
+    })
+  })
+
+  it('runs the job in the configured context, on the web method', async () => {
+    const { component, requests } = build()
+    component.deployInNewWindow = false
+
+    await component.createDatabase()
+
+    expect(requests[0][2]).toEqual({
+      useComputeApi: null,
+      contextName: 'Compute Reusable',
+      debug: true
+    })
+  })
+
+  it('prefers the context chosen in the form over the configured one', async () => {
+    const { component, requests } = build()
+    component.deployInNewWindow = false
+    component.computeContexts = [
+      { id: 'ctx-1', name: 'Selected Context' } as any
+    ]
+    component.selectedComputeContext = 'ctx-1'
+
+    await component.createDatabase()
+
+    expect(requests[0][2].contextName).toBe('Selected Context')
+  })
+
+  it('opens makedata in a new window with the selections as URL parameters', () => {
+    const { component } = build()
+    const opened: string[] = []
+    spyOn(window, 'open').and.callFake((url?: any) => {
+      opened.push(String(url))
+      return null
+    })
+    spyOn(component as any, 'pollMakedataCompletion')
+    component.deployInNewWindow = true
+
+    component.createDatabase()
+
+    expect(opened.length).toBe(1)
+    expect(opened[0]).toContain('/Users/demo/dc/services/admin/makedata')
+    expect(opened[0]).toContain('_contextname=Compute%20Reusable')
+    expect(opened[0]).toContain('admin=viyagroup18')
+    expect(opened[0]).toContain('dcpath=%2Fexport%2Fdc')
+    expect(opened[0]).toContain('_debug=131')
+  })
+
+  it('percent-encodes the values that need it in the new-window URL', () => {
+    // A context name or admin group can hold characters that would otherwise
+    // split the query string - an ampersand would start a new parameter and a
+    // space would arrive truncated.
+    const { component } = build()
+    const opened: string[] = []
+    spyOn(window, 'open').and.callFake((url?: any) => {
+      opened.push(String(url))
+      return null
+    })
+
+    component.runMakedataInNewWindow({
+      contextName: 'SAS Job Execution & co',
+      admin: 'group 1+2',
+      dcPath: '/export/dc data'
+    })
+
+    expect(opened[0]).toContain('_contextname=SAS%20Job%20Execution%20%26%20co')
+    expect(opened[0]).toContain('admin=group%201%2B2')
+    expect(opened[0]).toContain('dcpath=%2Fexport%2Fdc%20data')
+    // the parameter names themselves are not encoded
+    expect(opened[0]).toContain('&admin=group%201%2B2')
+  })
+
+  it('starts the job when deploy is pressed', () => {
+    // The screen's only button calls this; a version where it stopped doing so
+    // would leave the setup page unable to do anything at all.
+    const { component } = build()
+    const spy = spyOn(component, 'createDatabase').and.returnValue(
+      Promise.resolve() as any
+    )
+
+    component.runAutoDeploy()
+
+    expect(spy).toHaveBeenCalled()
+  })
+})

client/src/app/deploy/sections/automatic/automatic.component.ts

@@ -51,17 +51,8 @@ export class AutomaticComponent implements OnInit {
 
   public selectedComputeContext: string = ''
   public makeDataResponse: string = ''
-  public jsonFile: any = null
   public autodeploying: boolean = false
   public autodeployDone: boolean = false
-  public recreateDatabaseModal: boolean = false
-  public isSubmittingJson: boolean = false
-  public isJsonSubmitted: boolean = false
-  /**
-   * Default was `false` when deploy was done with frontend and backend separately.
-   * Now we are using only streaming app, so we always want to recreate database (makedata)
-   */
-  public recreateDatabase: boolean = true
   public createDatabaseLoading: boolean = false
   public adminGroupsLoading: boolean = false
   public currentUserInfoLoading: boolean = false
@@ -86,18 +77,14 @@ export class AutomaticComponent implements OnInit {
   public currentUserInfo: ViyaApiCurrentUser | null = null
   public computeContexts: ComputeContextItem[] = []
 
-  /** autoDeployStatus
-   * This object presents the status for two steps that we have for deploy.
-   * `deployServicePack` - Creating services based on `viya.json`
-   * `runMakeData` - Running `makedata` service
-   * If any of them is `null` or `false` that means step failed
-   * and will be shown to user on deploy done modal.
+  /**
+   * Status of the one step this screen runs: `makedata`, which creates the
+   * control library and its configuration data and then deletes itself.
+   * `null` means it has not run yet, `false` that it failed.
    */
   public autoDeployStatus: {
-    deployServicePack: any
     runMakeData: any
   } = {
-    deployServicePack: null,
     runMakeData: null
   }
 
@@ -420,61 +407,15 @@ export class AutomaticComponent implements OnInit {
   }
 
   /**
-   * Executes sas.json file to deploy the backend
-   * Method will first try to run the `auto deploy`
-   * If that fails the rest of the code is ignored.
-   * If request is successfull, method will continue to try
-   * to create database if checkbox is toggled on
+   * Runs the deployment. The control library and its configuration data are
+   * created by `makedata`, which self-deletes once it has run; the frontend and
+   * services arrived with the sasjs deploy, so there is nothing else to deploy
+   * from here.
    */
-  public async executeJson() {
-    this.isSubmittingJson = true
-
-    try {
-      let uploadJsonFile = await this.sasJs.deployServicePack(
-        this.jsonFile,
-        this.dcAdapterSettings?.appLoc,
-        undefined,
-        undefined,
-        true
-      )
-
-      this.autoDeployStatus.deployServicePack = true
-      this.isJsonSubmitted = true
-    } catch (ex: any) {
-      let textEx = ''
-
-      if (typeof ex.message !== 'string') {
-        textEx = JSON.stringify(ex).replace(/\\/gm, '')
-      } else {
-        textEx = ex.message
-      }
-
-      this.autoDeployStatus.deployServicePack = false
-      this.eventService.showInfoModal(
-        'Deploy error',
-        `Exception: \n ${textEx !== '' ? textEx : ex}`
-      )
-      this.autodeploying = false
-      this.autodeployDone = false
-
-      return
-    }
-
-    this.isSubmittingJson = false
-  }
-
-  public async runAutoDeploy(executeJson: boolean = false) {
+  public runAutoDeploy() {
     if (!this.deployInNewWindow) this.autodeploying = true
 
-    if (executeJson) {
-      this.executeJson()
-    }
-
-    if (this.recreateDatabase) {
-      this.createDatabase()
-    } else {
-      if (!this.deployInNewWindow) this.autodeployDone = true
-    }
+    this.createDatabase()
   }
 
   /**
@@ -578,9 +519,9 @@ export class AutomaticComponent implements OnInit {
     let serverUrl = this.sasjsConfig.serverUrl
     let appLoc = this.sasjsConfig.appLoc
     const execPath = this.sasService.getExecutionPath()
-    let contextname = `&_contextname=${params.contextName}`
-    let admin = `&admin=${params.admin}`
-    let dcPath = `&dcpath=${params.dcPath}`
+    let contextname = `&_contextname=${encodeURIComponent(params.contextName)}`
+    let admin = `&admin=${encodeURIComponent(params.admin)}`
+    let dcPath = `&dcpath=${encodeURIComponent(params.dcPath)}`
     // Debug is ALWAYS enabled for makedata — it runs only once during
     // deployment and the full log is needed to diagnose any issues.
     let debug = '&_debug=131'
@@ -708,22 +649,6 @@ export class AutomaticComponent implements OnInit {
     this.deployService.downloadFile(content, filename, extension)
   }
 
-  public async onJsonFileChange(event: any) {
-    let file = event.target.files[0]
-
-    this.jsonFile = await this.deployService.readFile(file)
-  }
-
-  public recreateDatabaseClicked(event: Event) {
-    ;(<HTMLInputElement>event.target).checked === true
-      ? (this.recreateDatabaseModal = true)
-      : ''
-  }
-
-  public clearUploadInput(event: Event) {
-    this.deployService.clearUploadInput(event)
-  }
-
   public openSasRequestsModal() {
     this.eventService.openRequestsModal()
   }

client/src/app/deploy/sections/sasjs-configurator/sasjs-configurator.component.spec.ts

@@ -0,0 +1,88 @@
+import { ServerType } from '@sasjs/utils/types/serverType'
+import { SasjsConfiguratorComponent } from './sasjs-configurator.component'
+
+/**
+ * The SAS9 / SASjs Server setup screen cannot deploy anything: it builds the
+ * one-off `makedata` URL and opens it, with the admin group and the directory
+ * the control library will live in. These tests pin the URL shape, because a
+ * wrong parameter name or a lost encode still produces a page that opens and
+ * merely fails inside SAS, where nobody is looking.
+ */
+describe('SasjsConfiguratorComponent (SAS9 / SASjs setup)', () => {
+  const build = (serverType: ServerType) => {
+    const sasService: any = {
+      getExecutionPath: () => '/SASStoredProcess/do',
+      sasjsMakedataChecking: () => Promise.resolve(false),
+      logout: () => undefined
+    }
+
+    const component = new SasjsConfiguratorComponent(
+      sasService,
+      { getServerInfo: () => ({ subscribe: () => undefined }) } as any,
+      { replaceState: () => undefined } as any
+    )
+
+    component.sasJsConfig = {
+      serverType,
+      serverUrl: 'https://sas9.example.com',
+      appLoc: '/User Folders/dc'
+    } as any
+    component.dcDirectory = '/tmp/DataController'
+    component.dcAdminGroup = 'DCADMINS'
+
+    const opened: string[] = []
+    spyOn(window, 'open').and.callFake((url?: any) => {
+      opened.push(String(url))
+      return null
+    })
+
+    return { component, opened, sasService }
+  }
+
+  it('opens makedata on SAS9 with the admin group and the directory', () => {
+    const { component, opened } = build(ServerType.Sas9)
+
+    component.makeData()
+
+    expect(opened.length).toBe(1)
+    expect(opened[0]).toContain(
+      '_program=/User Folders/dc/services/admin/makedata'
+    )
+    expect(opened[0]).toContain('admin=DCADMINS')
+    expect(opened[0]).toContain('path=%2Ftmp%2FDataController')
+    expect(opened[0]).toContain('_debug=131')
+  })
+
+  it('stays on the page origin for SAS9 rather than the configured server', () => {
+    const { component, opened } = build(ServerType.Sas9)
+
+    component.makeData()
+
+    // A SAS9 deployment is reached through the URL the user is already on; the
+    // configured serverUrl would point the browser at a host it may not reach.
+    expect(opened[0].startsWith(window.location.origin)).toBe(true)
+    expect(opened[0]).not.toContain('sas9.example.com')
+  })
+
+  it('keeps the directory readable for SASjs Server', () => {
+    const { component, opened } = build(ServerType.Sasjs)
+
+    component.makeData()
+
+    // The SASjs branch does not encode the path, so a plain directory stays as
+    // written; encoding it there would make makedata look for a folder with %2F
+    // in its name.
+    expect(opened[0]).toContain('path=/tmp/DataController')
+  })
+
+  it('checks the SASjs Server deployment before reloading the app', () => {
+    const { component, sasService } = build(ServerType.Sasjs)
+    const spy = spyOn(sasService, 'sasjsMakedataChecking').and.returnValue(
+      Promise.resolve(false)
+    )
+
+    component.makeData()
+
+    expect(spy).toHaveBeenCalled()
+  })
+})

client/src/app/editor/editor.component.ts

@@ -3647,7 +3647,16 @@ export class EditorComponent
     if (clauses.length === 0) return false
 
     try {
-      const res: any = await this.sasStoreService.saveQuery(libds, clauses)
+      // A `va_parmap` entry is a column the report author typed, so this
+      // request can legitimately fail: the table may not have that column, and
+      // validatefilter aborts on an unknown variable (dc/dc#404). Attempting it
+      // is still worth it - when the mapping is right the very first getdata is
+      // already filtered - but the failure must not stop the page. An abort
+      // modal here covers an editor that is about to load fine; the mapping
+      // banner names the bad entry once the real metadata arrives.
+      const res: any = await this.sasStoreService.saveQuery(libds, clauses, {
+        suppressErrorAbortModal: true
+      })
       const id = res?.result?.[0]?.FILTER_RK
       const table = res?.result?.[0]?.FILTER_TABLE
       if (id === undefined || table === undefined) return false

client/src/app/services/deploy.service.ts

@@ -26,22 +26,4 @@ export class DeployService {
       document.body.removeChild(downloadLink)
     }
   }
-
-  public readFile(file: File): Promise<any> {
-    return new Promise((resolve, reject) => {
-      let JsonFileReader = new FileReader()
-
-      JsonFileReader.onload = () => {
-        if (JsonFileReader.result) {
-          resolve(JSON.parse(JsonFileReader.result.toString()))
-        }
-      }
-
-      JsonFileReader.readAsText(file)
-    })
-  }
-
-  public clearUploadInput(event: Event) {
-    ;(<HTMLInputElement>event.target).value = ''
-  }
 }

client/src/app/services/sas-store.service.spec.ts

@@ -116,6 +116,53 @@ describe('SasStoreService', () => {
       expect(seen.length).toBe(0)
     })
   })
+
+  describe('saveQuery', () => {
+    const capture = (adapterResponse: any = { result: [] }) => {
+      const calls: any[][] = []
+      const store = new SasStoreService(
+        {
+          request: (...args: any[]) => {
+            calls.push(args)
+            return Promise.resolve({ adapterResponse })
+          }
+        } as any,
+        {} as any,
+        {} as any
+      )
+      return { store, calls }
+    }
+
+    it('posts the filter to public/validatefilter', async () => {
+      const { store, calls } = capture()
+      await store.saveQuery('DC.MY_TABLE', [
+        { VARIABLE_NM: 'A', OPERATOR_NM: '=', RAW_VALUE: "'x'" } as any
+      ])
+
+      expect(calls[0][0]).toBe('public/validatefilter')
+      expect(calls[0][1].iwant[0].filter_table).toBe('DC.MY_TABLE')
+      expect(calls[0][1].filterquery.length).toBe(1)
+    })
+
+    it('passes wrapper options through, so a rejected filter can stay quiet', async () => {
+      // The VA deferred load attempts a filter built from a va_parmap entry the
+      // table may not have, and must not cover a page that loads fine with an
+      // abort modal (dc/dc#404).
+      const { store, calls } = capture()
+      await store.saveQuery('DC.MY_TABLE', [], {
+        suppressErrorAbortModal: true
+      })
+
+      expect(calls[0][3]).toEqual({ suppressErrorAbortModal: true })
+    })
+
+    it('sends no wrapper options when the caller gives none', async () => {
+      const { store, calls } = capture()
+      await store.saveQuery('DC.MY_TABLE', [])
+
+      expect(calls[0][3]).toBeUndefined()
+    })
+  })
 })
 
 /**

client/src/app/services/sas-store.service.ts

@@ -8,6 +8,7 @@ import { HelperService } from './helper.service'
 import { QueryClause } from '../models/TableData'
 import { globals } from '../_globals'
 import { FilterClause, FilterGroup, FilterQuery } from '../models/FilterQuery'
+import { RequestWrapperOptions } from '../models/request-wrapper/RequestWrapperOptions'
 import {
   $DataFormats,
   EditorsGetDataSASResponse,
@@ -395,13 +396,30 @@ export class SasStoreService {
     ).adapterResponse
   }
 
-  public async saveQuery(libds: string, filterQuery: QueryClause[]) {
+  /**
+   * Persists a filter and returns its FILTER_RK.
+   *
+   * `wrapperOptions` reaches the request wrapper, so a caller that can live with
+   * a rejected filter (the VA deferred load, whose `va_parmap` entry may name a
+   * column the table does not have) can suppress the abort modal that would
+   * otherwise cover a page which is about to load perfectly well.
+   */
+  public async saveQuery(
+    libds: string,
+    filterQuery: QueryClause[],
+    wrapperOptions?: RequestWrapperOptions
+  ) {
     let tables = {
       iwant: [{ filter_table: libds }],
       filterquery: filterQuery
     }
 
-    const res = await this.sasService.request('public/validatefilter', tables)
+    const res = await this.sasService.request(
+      'public/validatefilter',
+      tables,
+      undefined,
+      wrapperOptions
+    )
 
     this.filter.next(res)
 

client/src/app/services/va-filter.service.spec.ts

@@ -131,6 +131,41 @@ describe('VaFilterService', () => {
       expect(build([{ label: 'SOME_DATE', value: '2025/03/31' }])).toHaveSize(1)
     })
 
+    it('parses word dates, in either order and with an optional comma', () => {
+      const expected = "'01JAN2026'd"
+      // The reported shape: a date parameter displayed as WORDDATE.
+      expect(
+        build([{ label: 'SOME_DATE', value: 'January 1, 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: 'Jan 1, 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: 'Jan. 1, 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: '1 January 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: '1-Jan-2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+    })
+
+    it('matches a spelled-out month case-insensitively, and keeps DATE9 working', () => {
+      expect(
+        build([{ label: 'SOME_DATE', value: 'DECEMBER 31, 2025' }])[0].RAW_VALUE
+      ).toBe("'31DEC2025'd")
+      expect(
+        build([{ label: 'SOME_DATE', value: '01AUG1962' }])[0].RAW_VALUE
+      ).toBe("'01AUG1962'd")
+    })
+
+    it('still drops a month-name value that has no day', () => {
+      // A month and year alone is not a date - accepted shapes always carry a
+      // day, so this must not silently become the first of the month.
+      expect(build([{ label: 'SOME_DATE', value: 'January 2026' }])).toEqual([])
+    })
+
     it('resolves ambiguous slash dates day-first (never US), disambiguating by value', () => {
       const iso = build([{ label: 'SOME_DATE', value: '2025-03-31' }])[0]
         .RAW_VALUE

client/src/app/services/va-filter.service.ts

@@ -707,14 +707,42 @@ export class VaFilterService {
     return this.parseMonthNameDate(value) ?? this.parseNumericDate(value, order)
   }
 
-  /** DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62) -> local Date, or null. */
+  /**
+   * Month-name date shapes -> local Date, or null. Three families:
+   *  - DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62);
+   *  - word date with the month first, comma optional: "January 1, 2026",
+   *    "Jan 1 2026", "Jan. 1, 2026";
+   *  - word date with the day first: "1 January 2026", "1-Jan-2026".
+   *
+   * A date parameter's display format decides which of these the report author
+   * sees, and VA sends the formatted text, so a report using WORDDATE (or a
+   * long month name in any order) has to parse here rather than being dropped.
+   * The month name may be abbreviated or spelled out; only its first three
+   * letters are matched.
+   */
   private parseMonthNameDate(value: string): Date | null {
-    const m = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
-    if (!m) return null
-    const [, day, monthAbbr, year] = m
-    const monthIndex = VaFilterService.SAS_MONTHS.indexOf(
-      monthAbbr.toUpperCase()
-    )
+    const monthByName = (name: string): number =>
+      VaFilterService.SAS_MONTHS.indexOf(
+        name.replace(/\./g, '').slice(0, 3).toUpperCase()
+      )
+
+    const date9 = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
+    if (date9) {
+      const [, day, monthAbbr, year] = date9
+      const monthIndex = monthByName(monthAbbr)
+      if (monthIndex < 0) return null
+      return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
+    }
+
+    const monthFirst =
+      /^([A-Za-z]{3,9})\.?[\s-]+(\d{1,2}),?[\s-]+(\d{2}|\d{4})$/.exec(value)
+    const dayFirst =
+      /^(\d{1,2})[\s-]+([A-Za-z]{3,9})\.?,?[\s-]+(\d{2}|\d{4})$/.exec(value)
+    const name = monthFirst ? monthFirst[1] : dayFirst ? dayFirst[2] : null
+    if (!name) return null
+    const day = monthFirst ? monthFirst[2] : dayFirst![1]
+    const year = monthFirst ? monthFirst[3] : dayFirst![3]
+    const monthIndex = monthByName(name)
     if (monthIndex < 0) return null
     return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
   }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "dcfrontend",
-  "version": "7.20.0",
+  "version": "7.21.0",
   "description": "Data Controller",
   "devDependencies": {
     "@nogoo9/gitleaks": "8.30.1-post.2",
@@ -15,7 +15,6 @@
     "got": "11.8.6"
   },
   "scripts": {
-    "install": "cd client && npm i && cd ../sas && npm i",
     "build-frontend": "cd client && npm run build",
     "lint": "cd client && npm run lint",
     "lint:fix": "cd client && npm run lint:fix",

sas/.sasjslint

@@ -12,5 +12,10 @@
   "hasMacroParentheses": true,
   "strictMacroDefinition": true,
   "ignoredMacros": ["webout", "mpeinit", "mpeterm"],
-  "ignoreList":["sasjs/scratch"]
+  "ignoreList":["sasjs/scratch"],
+  "severityLevel": {
+    "noUnusedMacros": "error",
+    "noUndeclaredMacros": "error",
+    "noBannedStrings": "error"
+  }
 }

sas/deploy/makeDataServer.json

@@ -1,7 +1,7 @@
 {
   "fromjs": [
     {
-      "ADMIN": "AllUsers",
+      "ADMIN": "all-users",
       "DCPATH": "/tmp/dcdata",
       "OUTFMT": "JSON"
     }

sas/mocks/sasjs/services/hooks/mpe_derived_postedit.js

@@ -13,10 +13,12 @@
 // The rules mirror services/hooks/mpe_derived_postedit.sas, in its order:
 // normalise the incoming rows, reject a duplicate name and a bad field, then
 // check the dependency graph that the resulting configuration would have (the
-// incoming rows plus the current rows they do not replace).  As on the server,
-// the collision between VAR_NM and a physical column is left to the read-time
-// engine, which runs with the target library assigned - the hook does not have
-// it here either.
+// incoming rows plus the current rows they do not replace).  The field rules
+// are core's mp_derivedvarcheck, which the server hook calls and the read-time
+// engine runs too - this file spells them out a second time, so a change there
+// has to be mirrored here.  As on the server, the collision between VAR_NM and
+// a physical column is left to the read-time engine, which runs with the target
+// library assigned - the hook does not have it here either.
 // ─────────────────────────────────────────────────────────────────────────────
 
 ;(function mpeDerivedPostedit() {
@@ -83,9 +85,45 @@
         seen.add(key)
     }
 
-    // ─── Field level checks, the rules the engine applies at read time ─────────
-    // C0 controls and DEL: what a single-line label cannot carry
+    // ─── Field level checks ────────────────────────────────────────────────────
+    // Two passes, in the order the SAS hook runs them.  The first is the fields
+    // this hook owns - the control table key, which the loader normalises but
+    // does not require, and ACTIVE, the picker pair the EDIT screen offers.
+    for (const row of rows) {
+        if (isDeleted(row)) continue
+        const name = field(row, 'VAR_NM')
+        let reason = ''
+        if (!field(row, 'LIBREF') || !field(row, 'DSN'))
+            reason = 'LIBREF and DSN are required'
+        // ACTIVE is a picker in the EDIT screen (HARDSELECT over the
+        // MPE_SELECTBOX pair) - the same pair the hook enforces
+        else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
+            reason = 'ACTIVE must be 1 or 0'
+        if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
+    }
+
+    // The second pass is the rules for the derived variable itself, which are
+    // core's (mp_derivedvarcheck): the engine runs the same macro at read time,
+    // so a definition is measured against one rule list wherever it is checked.
+
+    // C0 controls and DEL: what a label cannot carry once a line break has been
+    // flattened to a space, which is what the engine does with it
     const CONTROL_CHARS = /[\u0000-\u001F\u007F]/
+    const flatten = (v) => v.replace(/[\r\n]+/g, ' ')
+    const AUTOMATIC_VARS = ['_N_', '_ERROR_', '_IORC_']
+
+    // the capabilities refused by name - a function name cannot be assembled at
+    // run time, so naming them is a control the expression cannot evade
+    const DENY_FUNCS = new RegExp(
+        '\\b(DOSUBL|JAVAOBJ|JAVAMETA|SYMGET|SYMPUTX?|SYSGET|GETOPTION' +
+            '|PATHNAME|FILEEXIST|FOPEN|FREAD|FGET|FCLOSE' +
+            '|DOPEN|DREAD|DNUM|MOPEN|SLEEP' +
+            '|FILENAME|FDELETE|FCOPY|DCREATE|DLCREATE)\\s*\\('
+    )
+
+    // quoted literals are data, not calls
+    const unquoted = (v) =>
+        v.replace(/'([^']|'')*'/g, ' ').replace(/"([^"]|"")*"/g, ' ')
 
     // true when the value carries a character outside the allowed set
     const hasInvalidChar = (value, chars) =>
@@ -93,8 +131,6 @@
 
     for (const row of rows) {
         if (isDeleted(row)) continue
-        const libref = field(row, 'LIBREF')
-        const dsn = field(row, 'DSN')
         const name = field(row, 'VAR_NM')
         const type = field(row, 'VAR_TYPE')
         const len = Number(colVal(row, 'VAR_LEN'))
@@ -103,19 +139,16 @@
         const logic = strip(colVal(row, 'VAR_LOGIC'))
 
         let reason = ''
-        if (!libref || !dsn) reason = 'LIBREF and DSN are required'
-        else if (!name) reason = 'VAR_NM is required'
+        if (!name) reason = 'VAR_NM is required'
         else if (hasInvalidChar(name, NAME_CHARS))
             reason = 'VAR_NM must be a valid SAS name'
         else if ('0123456789'.includes(name[0]))
             reason = 'VAR_NM must not start with a digit'
+        else if (AUTOMATIC_VARS.includes(name))
+            reason = 'VAR_NM is an automatic variable of the data step'
         else if (name === DELETE_MARK) reason = 'VAR_NM is reserved'
         else if (type !== 'C' && type !== 'N')
             reason = 'VAR_TYPE must be C or N'
-        // ACTIVE is a picker in the EDIT screen (HARDSELECT over the
-        // MPE_SELECTBOX pair) - the same pair mpe_derived_check.sas enforces
-        else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
-            reason = 'ACTIVE must be 1 or 0'
         else if (!Number.isInteger(len) || len < 1)
             reason = 'VAR_LEN must be a positive integer'
         else if (type === 'C' && len > 32767)
@@ -132,20 +165,24 @@
             reason = 'VAR_FMT must be a format name, without ; & % or /* */'
         else if (hasInvalidChar(fmt, FMT_CHARS))
             reason = 'VAR_FMT must be a format name'
-        // free text, in any script - what is refused is anything that cannot be
-        // a single-line column label, and a value the engine would cut to 256
-        else if (CONTROL_CHARS.test(label))
-            reason =
-                'VAR_LABEL must be plain text - no line breaks or control characters'
+        // free text, in any script - what is refused is a control character,
+        // and a value the engine would cut to 256
+        else if (CONTROL_CHARS.test(flatten(label)))
+            reason = 'VAR_LABEL must be plain text - no control characters'
         else if ([...label].length > 256)
             reason = 'VAR_LABEL exceeds 256 characters'
         else if (!logic) reason = 'VAR_LOGIC is required'
+        else if (logic.length > 4000)
+            reason = 'VAR_LOGIC exceeds 4000 characters'
         else if (logic.includes(';'))
             reason = 'VAR_LOGIC must be a single expression - remove the ;'
         else if (logic.includes('&') || logic.includes('%'))
             reason = 'VAR_LOGIC must not contain macro triggers'
         else if (logic.includes('/*') || logic.includes('*/'))
             reason = 'VAR_LOGIC must not contain comments'
+        else if (DENY_FUNCS.test(unquoted(logic).toUpperCase()))
+            reason =
+                'VAR_LOGIC may not call a function that submits code or reaches the file system, the environment or the macro symbol table'
 
         if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
     }

sas/mocks/sasjs/services/public/getddl.js

@@ -1,7 +1,20 @@
 let mimeType = `application/octet-stream`
 let filename = `dc-download`
 
-if (libref) filename = `${libref}.ddl`
+/* typeof, not a truthiness test: if the service is called without a ds in scope
+   the read itself throws and the fallback never runs */
+if (
+  typeof libref !== 'undefined' &&
+  libref &&
+  typeof ds !== 'undefined' &&
+  ds
+)
+  filename = `${libref}_${ds}.ddl`
+else if (typeof libref !== 'undefined' && libref) {
+  /* production aborts when ds is missing (getddl.sas:43-45), so the mock does
+     too: a client that dropped ds would otherwise stay green here */
+  throw new Error(`Dataset ${libref}..${ds} was not found`)
+}
 
 fs.writeFile(_SASJS_WEBOUT_HEADERS, `Content-type: ${mimeType} \n Content-Disposition: attachment;filename=\"${filename}\"`, function (err) {
   if (err) throw err;

sas/package-lock.json

@@ -6,8 +6,8 @@
     "": {
       "name": "dc-sas",
       "dependencies": {
-        "@sasjs/cli": "6.2.0",
-        "@sasjs/core": "6.1.0"
+        "@sasjs/cli": "6.5.0",
+        "@sasjs/core": "7.0.3"
       }
     },
     "node_modules/@asamuzakjp/css-color": {
@@ -165,14 +165,15 @@
       }
     },
     "node_modules/@sasjs/cli": {
-      "version": "6.2.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-6.2.0.tgz",
-      "integrity": "sha512-zovAOCmyPqcm+gooAZgwvgGH8n5cas3sY0lA80+hyFoFtmZ8VjW/bh19LM8OVwheT9dBCRCx4fnHwjy5XGpzvQ==",
+      "version": "6.5.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-6.5.0.tgz",
+      "integrity": "sha512-1I7B0iKd+16ByIGPToGfODU3S1bxO/3VCGpmu5alql/HlXdxu75djdpuzCovbsSeWm+M4FrsrBKDZTc9hhBrLA==",
       "license": "ISC",
       "dependencies": {
         "@sasjs/adapter": "^5.1.1",
-        "@sasjs/core": "^6.0.0",
-        "@sasjs/utils": "^4.1.0",
+        "@sasjs/core": "^6.1.1",
+        "@sasjs/lint": "4.9.0",
+        "@sasjs/utils": "^4.1.2",
         "axios": "1.20.0",
         "chalk": "4.1.2",
         "dotenv": "18.0.8",
@@ -194,15 +195,18 @@
       },
       "engines": {
         "node": ">=22"
-      },
-      "peerDependencies": {
-        "@sasjs/lint": "^4.3.0"
       }
     },
+    "node_modules/@sasjs/cli/node_modules/@sasjs/core": {
+      "version": "6.1.4",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.4.tgz",
+      "integrity": "sha512-vIF2LIftxPWg4CvO3i5raEfrHFHY/6S+MqVGpjRR71dgjLRJEYNyPJgGqD6BCZwHv5V6/2nDnBi/BIIMiLXfzA==",
+      "license": "MIT"
+    },
     "node_modules/@sasjs/cli/node_modules/@sasjs/utils": {
-      "version": "4.1.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.1.0.tgz",
-      "integrity": "sha512-aC5pOugg6/2jKQR7ZaYsLtbsFy/iVacsrN3/Kzh6BouJUszuVrr13HMrQUSKcWL+rJzxYof7ymmmCl+vE6YwAw==",
+      "version": "4.1.2",
+      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.1.2.tgz",
+      "integrity": "sha512-5QLBVlCqiiDW/GoxiHUCyT+8GJjJA/S/TX0DS/t4O4HC1Im/QpQzxlS86QkjLAQBBHovYPyh+v4pgdh+IYwaEQ==",
       "license": "ISC",
       "dependencies": {
         "@fast-csv/format": "5.0.7",
@@ -235,62 +239,24 @@
       }
     },
     "node_modules/@sasjs/core": {
-      "version": "6.1.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.0.tgz",
-      "integrity": "sha512-U0Ekq/TXVMquYV+AG6urBQSUgqUvV37yduufy68QQVgDj3PJiLUJDcOXKOGbhWGD+Nv877VuPCviGYqrqbjXvg==",
+      "version": "7.0.3",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-7.0.3.tgz",
+      "integrity": "sha512-x67TJOXCKvdtlYoHGKNnVWz76QFwRpcp/+zpzVqjll6prwHTUAQt8SBjCGmC3/fhadBtylGGj37rgC8Lz8fSww==",
       "license": "MIT"
     },
     "node_modules/@sasjs/lint": {
-      "version": "4.3.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-4.3.0.tgz",
-      "integrity": "sha512-65cjy4IjA4PgSlicOruF0DI9N2HO3LJqtgphsZCg8g7N/StE2FWLJLJbD1N4OHxN6htrUfj5SEawJJiHo5T9Yw==",
+      "version": "4.9.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-4.9.0.tgz",
+      "integrity": "sha512-QEBo19bWUdKIaE62Pk8eb35Q9EjHlifyOmbH+voItX6muCqHTX88SOzLD4MYtFVuagm/cnGplIVcKvUZzO+bSw==",
       "license": "MIT",
-      "peer": true,
       "dependencies": {
-        "@sasjs/utils": "4.0.0",
+        "@sasjs/utils": "^4.0.0",
         "ignore": "7.0.11"
       },
       "engines": {
         "node": ">=22"
       }
     },
-    "node_modules/@sasjs/lint/node_modules/@sasjs/utils": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.0.0.tgz",
-      "integrity": "sha512-lARoCnRM4gSVf8ShoP5elHi7W28K/14MK037dQ0ABgMlP3+r6OuhAZYwK/a70tED47tKxyIoCAKLdP6jrG0ewQ==",
-      "license": "ISC",
-      "peer": true,
-      "dependencies": {
-        "@fast-csv/format": "5.0.7",
-        "@types/fs-extra": "11.0.4",
-        "@types/prompts": "2.4.9",
-        "chalk": "4.1.1",
-        "consola": "3.4.2",
-        "fs-extra": "11.4.1",
-        "jwt-decode": "4.0.0",
-        "prompts": "2.4.2"
-      },
-      "engines": {
-        "node": ">=22"
-      }
-    },
-    "node_modules/@sasjs/lint/node_modules/chalk": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
-      "integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
-      "license": "MIT",
-      "peer": true,
-      "dependencies": {
-        "ansi-styles": "^4.1.0",
-        "supports-color": "^7.1.0"
-      },
-      "engines": {
-        "node": ">=10"
-      },
-      "funding": {
-        "url": "https://github.com/chalk/chalk?sponsor=1"
-      }
-    },
     "node_modules/@sasjs/utils": {
       "version": "4.0.1",
       "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.0.1.tgz",
@@ -825,9 +791,9 @@
       "license": "MIT"
     },
     "node_modules/follow-redirects": {
-      "version": "1.16.1",
-      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.1.tgz",
-      "integrity": "sha512-FNvFGzoMLWmE6Yj9spb/zjd7yiNCHiAW9/Tg9CXrQ8wuu32HtlJOwWO11OJafl5FfY3DxTdQ0vj42zU1kvv5jg==",
+      "version": "1.16.2",
+      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.2.tgz",
+      "integrity": "sha512-/kVP2U8z596Ia0gwFrwyIeo74ta5aVmSeU1FfAUMAmcEP5EP7zydmag3SLiN64NaRKef3lxsq7i/g7ltJQDGow==",
       "funding": [
         {
           "type": "individual",
@@ -1106,7 +1072,6 @@
       "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.11.tgz",
       "integrity": "sha512-YChdK5txDjwGUvgR7oCJcLGkwi3LDh8Zx8tS7ndYciLEg/oOCL26VUBSUEtgS7EjiybcBqFxh5j0rAdDX7/bbg==",
       "license": "MIT",
-      "peer": true,
       "engines": {
         "node": ">= 4"
       }

sas/package.json

@@ -3,6 +3,7 @@
   "description": "dc-sas",
   "scripts": {
     "cpfavicon": "cp sasjs/utils/favicon.ico ../client/dist/favicon.ico",
+    "lint:sas": "sasjs lint",
     "build": "cd ../client && npm run build",
     "deploy:server": "rsync -avhe ssh ../client/dist/* --delete allbow@sas.4gl.io:/opt/sas/sas9/config/Lev1/Web/WebServer/htdocs/allbow/server",
     "sas9": "rsync -avhe ssh ./dist/* --delete ${npm_config_account}@sas.4gl.io:/opt/sas/sas9/config/Lev1/Web/WebServer/htdocs/${npm_config_account}/dc",
@@ -33,8 +34,8 @@
   },
   "private": true,
   "dependencies": {
-    "@sasjs/cli": "6.2.0",
-    "@sasjs/core": "6.1.0"
+    "@sasjs/cli": "6.5.0",
+    "@sasjs/core": "7.0.3"
   },
   "overrides": {
     "nanoid": "3.3.18"

sas/sasjs/db/migrations/20220615_v5.0_release.sas

@@ -5,9 +5,6 @@
   BREAKING CHANGE - deprecate MPE_APPROVALS and MPE_REVIEW (now MPE_SUBMIT does
   both)
 
-  <h4> SAS Macros </h4>
-  @li mm_assigndirectlib.sas
-
 **/
 
 %let dclib=YOURDCLIB;

sas/sasjs/db/migrations/20261003_row_and_cell_limits.sas

@@ -98,7 +98,7 @@ quit;
       where var_name='DC_MAXCELLS_WEBEDIT' and var_active=1 and &now lt tx_to;
 
     /* Why the cell limit exists, kept out of var_desc - MPE_CONFIG.VAR_DESC
-      is varchar(300), and a truncation WARNING aborts the build.  A wide
+      is varchar(300), and a truncation aborts the build.  A wide
       table is expensive long before it is tall - 250 rows of a 1000 column
       table is a quarter of a million cells. */
     insert into &dclib..mpe_config

sas/sasjs/macros/bitemporal_dataloader.sas

@@ -98,6 +98,7 @@ Areas for optimisation
   @li mf_fmtdttm.sas
   @li mf_getattrn.sas
   @li mf_getengine.sas
+  @li mf_getquotedstr.sas
   @li mf_getschema.sas
   @li mf_getuniquefileref.sas
   @li mf_getuniquename.sas
@@ -122,6 +123,8 @@ Areas for optimisation
   @warning multitemporal loads (bitemporal for multiple points in business time)
     are in experimental stage
 
+  @sasjslint {"noBannedStrings": false}
+
 **/
 
 %macro bitemporal_dataloader(

sas/sasjs/macros/bitemporal_dataloader.test.2.sas

@@ -16,6 +16,7 @@
 
 
   <h4> SAS Macros </h4>
+  @li bitemporal_closeouts.sas
   @li bitemporal_dataloader.sas
   @li dc_assignlib.sas
   @li mf_getattrn.sas
@@ -1349,44 +1350,6 @@ run;
   proc sql; insert into results set test="TEST&testnum",result="&result";
 %end;
 
-/* test 30 - testing a load to OLEDB (should use SQL server temp table) */
-%if (&test=ALL or &test=30 ) and 0 %then %do; /* skip until have a db */
-  %let testnum=30;
-
-%assign_direct_lib(libref=XXX);
-  data appendtable&testnum ;
-    set XXX.temp_rk_table
-    (where=(tx_from le %sysfunc(datetime()) lt tx_to));
-    some_num+1;
-    output;
-    some_id=put(ranuni(0)*10000000,8.);output;
-    stop;
-  run;
-  proc sql noprint;
-  select count(*) into:testnobs&testnum from web.temp_rk_table;
-
-  %bitemporal_dataloader(dclib=work,PK=some_rk
-      ,append_dsn=appendtable&testnum.
-      ,ETLSOURCE=bitemporal_tester test&testnum override
-      ,tech_from=tx_from
-      ,tech_to=tx_to
-      ,base_lib=web
-      ,base_dsn=temp_rk_table
-      ,LOG=YES
-      ,rk_underlying=some_id
-      ,LOADTARGET=YES
-      ,LOADTYPE=TXTEMPORAL
-  );
-
-
-  %let result=;
-  proc sql noprint;
-  select count(*) into:testnobs2&testnum from web.temp_rk_table;
-  %if (&&testnobs2&testnum =%eval(&&testnobs&testnum+2)) %then %let result=PASS;
-  %else %let result=FAIL;
-  proc sql; insert into results set test="TEST&testnum",result="&result";
-%end;
-
 /* test 31 - bitemporal closeouts with embedded speechmarks in PK */
 %let testnum=31;
 %if &test=ALL or &test=&testnum %then %do;

sas/sasjs/macros/mpe_accesscheck.test.sas

@@ -5,7 +5,6 @@
 
 
   <h4> SAS Macros </h4>
-  @li mf_getuniquename.sas
   @li mf_getuser.sas
   @li mp_assertdsobs.sas
   @li mpe_getgroups.sas

sas/sasjs/macros/mpe_alerts.sas

@@ -10,7 +10,6 @@
 
   <h4> SAS Macros </h4>
   @li mf_getattrn.sas
-  @li mf_existds.sas
   @li mf_getuser.sas
 
   @version 9.4

sas/sasjs/macros/mpe_assert_abort.sas

@@ -1,38 +1,43 @@
 /**
   @file
   @brief Asserts that a service returned a sasjsAbort object
-  @details Reads the child's response as text and records a PASS when it carries
-  a sasjsAbort object, which is what mp_abort leaves in the webout when a service
-  rejects its input.  When msg is supplied the MSG value inside that object must
-  match it, so the assertion proves the service aborted for the expected reason
-  rather than for any reason at all.
+  @details Reads the child's status row and records a PASS when the child was
+  canceled, which is what happens when a service rejects its input and calls
+  mp_abort.  When msg is supplied the ERRMSG value must match it, so the
+  assertion proves the service aborted for the expected reason rather than for
+  any reason at all.
 
-  The response is scanned a line at a time and only the first MSG that follows
-  the sasjsAbort marker is taken: a single 32767 buffer both truncated longer
-  responses and, with a leading `.*`, returned the last MSG in the text rather
-  than the abort's own.
+  The message comes from the status row rather than from the response text: core
+  publishes STATE and ERRMSG per call, ERRMSG holds the whole message including
+  quotes and anything longer than a single record, and reading it needs no
+  parsing of the response.
 
-  An unassigned outref (a child that died before writing a response) records a
-  FAIL rather than letting the data step fail on an unassigned fileref.
+  Two things are handled here rather than in core:
+
+  - mp_abort appends cats(msg,'\n\nLog Extract:\n',logmsg) to the message, so
+    the tail is cut at the two newlines that precede the marker.  A message that
+    merely mentions "Log Extract:" is not preceded by them and survives.
+  - a platform that reports no job state at all (SAS 9 PROC STP) cannot show the
+    abort this way; the row reads 'unknown' and the assertion fails rather than
+    guessing.
 
   Usage:
 
       %mpe_execute(&_program,
         inputfiles=&f1:sascontroltable,
-        outref=web1,
         outstatus=work.ab1,
         failfast=0
       )
       %mpe_assert_abort(
         desc=An invalid libds aborts the service,
-        outref=web1,
+        outstatus=work.ab1,
         msg=Invalid libds
       )
 
   @param [in] desc The description recorded in the test results
-  @param [in] outref The fileref given to %mx_execute, holding the response
-  @param [in] msg= (0) Expected MSG value in the sasjsAbort object.  0 accepts
-    any abort message.
+  @param [in] outstatus The dataset given to %mpe_execute, holding STATE and
+    ERRMSG for the child call
+  @param [in] msg= (0) Expected ERRMSG value.  0 accepts any abort message.
   @param [out] outds= (work.test_results) Results dataset
 
   <h4> SAS Macros </h4>
@@ -40,53 +45,37 @@
 
 **/
 
-%macro mpe_assert_abort(desc, outref, msg=0, outds=work.test_results);
+%macro mpe_assert_abort(desc, outstatus, msg=0, outds=work.test_results);
 %local ok seen abortseen;
 %let ok=0;
 %let seen=;
 %let abortseen=0;
 
-%if %sysfunc(fileref(&outref))>0 %then %do;
-  %put %str(WARN)ING: &sysmacroname: fileref &outref is not assigned - the child left no response to read;
-%end;
-%else %if %sysfunc(fileexist(%sysfunc(pathname(&outref)))) ne 1 %then %do;
-  /* the fileref exists but nothing was written to it: a child that aborted
-    before it produced a response.  Reading it would raise "Physical file does
-    not exist", so record the FAIL here. */
-  %put %str(WARN)ING: &sysmacroname: &outref has no response written to it - the child produced nothing;
+%if %sysfunc(exist(&outstatus)) ne 1 %then %do;
+  %put %str(WARN)ING: &sysmacroname: &outstatus does not exist - the child left no status row;
 %end;
 %else %do;
-data _null_;
-  length line $32767 got $256;
-  retain prxid 0 inabort 0 found 0 abortseen 0;
-  if _n_=1 then prxid=prxparse('/"MSG" *: *"([^"]*)"/');
-  infile &outref end=eof lrecl=32767;
-  do while(not eof);
-    input;
-    line=_infile_;
-    if index(line,'sasjsAbort')>0 then do;
-      abortseen=1;
-      inabort=1;
-    end;
-    /* first MSG at or after the abort marker - stops at the first found */
-    if inabort=1 and found=0 then do;
-      if prxmatch(prxid,line)>0 then do;
-        got=prxposn(prxid,1,line);
-        found=1;
-      end;
-    end;
-  end;
-  call symputx('abortseen',abortseen,'l');
-  if found=1 then call symputx('seen',got,'l');
-  stop;
-run;
+  data _null_;
+    length got $32767;
+    set &outstatus;
+    if state='canceled' then call symputx('abortseen',1,'l');
+    got=errmsg;
+    /* mp_abort appends cats(msg,'\n\nLog Extract:\n',logmsg).  Those newlines
+      reach here as characters, as an escaped pair, or as a doubled pair - which
+      one depends on the producer - so all three are tried */
+    cut=index(got,'0A0A'x||'Log Extract:');
+    if cut=0 then cut=index(got,'\n\nLog Extract:');
+    if cut=0 then cut=index(got,'\\n\\nLog Extract:');
+    if cut>1 then got=substr(got,1,cut-1);
+    call symputx('seen',got,'l');
+  run;
 %end;
 
 %if &abortseen=1 %then %do;
   %if "%superq(msg)"="0" %then %let ok=1;
-  %else %if "&seen"="%superq(msg)" %then %let ok=1;
+  %else %if "%superq(seen)"="%superq(msg)" %then %let ok=1;
 %end;
-%if "&seen" ne "" %then %put NOTE: &sysmacroname: sasjsAbort MSG=&seen;
+%if "%superq(seen)" ne "" %then %put NOTE: &sysmacroname: sasjsAbort MSG=&seen;
 
 %mp_assert(
   iftrue=(&ok=1),

sas/sasjs/macros/mpe_assert_abort.test.sas

@@ -2,25 +2,18 @@
   @file
   @brief testing the mpe_assert_abort macro
 
-  @details The macro decides PASS/FAIL by scanning the child's response text, so
-  the cases that matter can be built as files instead of services: an abort with
-  the expected message, an abort with a different one, no abort at all, and a
-  fileref the child never wrote to.
+  @details The macro decides PASS/FAIL from the child's status row - STATE says
+  whether the child was canceled, ERRMSG holds the reason - so the cases can be
+  built as datasets instead of services: an abort with the expected message, an
+  abort with a different one, a completed call, a status dataset that does not
+  exist, a message past the old 256 character width, quotes, backslashes, and
+  the log extract that mp_abort appends to the message.
 
   Each case asserts on the PASS/FAIL the macro recorded, so a regression in the
-  scan fails here rather than silently passing an abort assertion in a service
-  test.
-
-  Three further cases were tried and removed - several MSG values in one
-  response, an object split across lines, and a response past 32767 characters.
-  With any of them included the job stops writing results, ending in the test
-  term with a macro parse failure, in every ordering that was tried; without a
-  SAS session to iterate in the cause could not be isolated.  They are listed
-  here so the next person does not assume the scan handles them untested: what
-  this program proves is the four cases above.
+  comparison fails here rather than silently passing an abort assertion in a
+  service test.
 
   <h4> SAS Macros </h4>
-  @li mf_getuniquefileref.sas
   @li mp_assert.sas
   @li mpe_assert_abort.sas
 
@@ -31,96 +24,202 @@
 
 **/
 
-%let rc_any=work.assert_rc1;
-%let rc_other=work.assert_rc2;
-%let rc_none=work.assert_rc3;
-%let rc_first=work.assert_rc4;
-%let rc_split=work.assert_rc5;
-%let rc_long=work.assert_rc6;
-%let rc_noref=work.assert_rc7;
+/** a status row as %mpe_execute leaves it: the job state and the message */
+%macro mk_status(ds, state, msg);
+/* the values come from the symbol table rather than from quoted text: a
+  message holding double quotes would otherwise end the literal early */
+data &ds;
+  length state $20 uri $256 errmsg $32767 platform $10;
+  state=symget('state');
+  uri='';
+  errmsg=symget('msg');
+  platform='SASJS';
+run;
+%mend mk_status;
 
 /**
-  * Test 1 - an abort whose MSG matches is a PASS
+  * Test 1 - an abort whose message matches is a PASS
   */
-%let f1=%mf_getuniquefileref();
-data _null_;
-  file &f1 termstr=crlf;
-  put '{"sasjsAbort": [{"MSG": "Invalid libds", "MAC": "mp_abort"}]}';
-run;
+%mk_status(work.st1, canceled, Invalid libds)
 
 %mpe_assert_abort(
   desc=an abort with the expected MSG,
-  outref=&f1,
+  outstatus=work.st1,
   msg=Invalid libds,
-  outds=&rc_any
+  outds=work.rc1
 )
 
 /**
   * Test 2 - an abort for a different reason is a FAIL
   */
-%let f2=%mf_getuniquefileref();
-data _null_;
-  file &f2 termstr=crlf;
-  put '{"sasjsAbort": [{"MSG": "Invalid libds", "MAC": "mp_abort"}]}';
-run;
+%mk_status(work.st2, canceled, Invalid libds)
 
 %mpe_assert_abort(
   desc=an abort with a different MSG,
-  outref=&f2,
+  outstatus=work.st2,
   msg=Invalid filter,
-  outds=&rc_other
+  outds=work.rc2
 )
 
 /**
-  * Test 3 - a completed call (no sasjsAbort) is a FAIL
+  * Test 3 - a completed call is a FAIL
   */
-%let f3=%mf_getuniquefileref();
-data _null_;
-  file &f3 termstr=crlf;
-  put '{"sasparams": [{"STATUS": "SUCCESS"}]}';
-  put '{"MSG": "Invalid libds"}';
+%mk_status(work.st3, completed, )
+
+%mpe_assert_abort(
+  desc=a completed call,
+  outstatus=work.st3,
+  msg=0,
+  outds=work.rc3
+)
+
+/**
+  * Test 4 - a status dataset that does not exist fails the assertion, not the step
+  */
+%mpe_assert_abort(
+  desc=a status dataset that was never written,
+  outstatus=work.doesnotexist,
+  msg=0,
+  outds=work.rc4
+)
+
+/**
+  * Test 5 - a message longer than the old capture width is compared in full
+  *
+  * The status row carries the whole message, so a caller comparing a long one
+  * gets an answer rather than a silent 256 character cut.
+  */
+%let longmsg=;
+data work.st5;
+  length state $20 errmsg $32767;
+  state='canceled';
+  errmsg=repeat('a clause that runs on ',30);
+  call symputx('longmsg',errmsg,'l');
 run;
 
 %mpe_assert_abort(
-  desc=a response with no sasjsAbort object,
-  outref=&f3,
-  msg=0,
-  outds=&rc_none
+  desc=an abort with a message past the old capture width,
+  outstatus=work.st5,
+  msg=%superq(longmsg),
+  outds=work.rc5
 )
 
 /**
-  * Test 4 - a fileref with no response written records a FAIL instead of aborting the step
+  * Test 6 - quotes in the message survive
   */
-%let f7=%mf_getuniquefileref();
+%mk_status(work.st6, canceled, %str(alpha "quoted" beta))
 
 %mpe_assert_abort(
-  desc=an unassigned outref,
-  outref=&f7,
-  msg=0,
-  outds=&rc_noref
+  desc=a message holding double quotes,
+  outstatus=work.st6,
+  msg=%str(alpha "quoted" beta),
+  outds=work.rc6
 )
 
 /**
-  * Test 5 - the recorded results
+  * Test 7 - the log extract mp_abort appends is not part of the message
+  */
+data work.st7;
+  length state $20 errmsg $32767;
+  state='canceled';
+  errmsg=cats('Invalid libds','0A0A'x,'Log Extract:','0A0A'x,'NOTE: a log line');
+run;
+
+%mpe_assert_abort(
+  desc=an abort with the appended log extract,
+  outstatus=work.st7,
+  msg=Invalid libds,
+  outds=work.rc7
+)
+
+/**
+  * Test 8 - a message that itself mentions the marker is not cut at it
+  *
+  * Only the two newlines before the marker identify the appended extract; a
+  * message that merely says "Log Extract:" keeps its text.
+  */
+data work.st8;
+  length state $20 errmsg $32767;
+  state='canceled';
+  errmsg='check the hook Log Extract: is mentioned here';
+run;
+
+%mpe_assert_abort(
+  desc=a message containing the extract marker,
+  outstatus=work.st8,
+  msg=%str(check the hook Log Extract: is mentioned here),
+  outds=work.rc8
+)
+
+/**
+  * Test 9 - a canceled call with no message fails a comparison it cannot support
+  *
+  * Core fills errmsg on both platforms now, so a blank one means the reason was
+  * never recorded - an expected message cannot be proved, and the assertion
+  * says so rather than passing on the state alone.
+  */
+%mk_status(work.st9, canceled, )
+
+%mpe_assert_abort(
+  desc=a canceled call whose status row carries no message,
+  outstatus=work.st9,
+  msg=Invalid libds,
+  outds=work.rc9
+)
+
+/**
+  * Test 10 - backslashes in the message compare as they were raised
+  */
+%mk_status(work.st10, canceled, %str(C:\temp\share))
+
+%mpe_assert_abort(
+  desc=a message holding backslashes,
+  outstatus=work.st10,
+  msg=%str(C:\temp\share),
+  outds=work.rc10
+)
+
+/**
+  * Test 11 - the doubled separator the escaping producer writes
+  *
+  * mp_abort doubles backslashes, so the appended extract arrives as a literal
+  * doubled pair rather than as characters.
+  */
+data work.st11;
+  length state $20 errmsg $32767;
+  state='canceled';
+  errmsg=cats('Invalid libds','\\n\\nLog Extract:\\n','NOTE: a log line');
+run;
+
+%mpe_assert_abort(
+  desc=an abort whose extract arrives doubled,
+  outstatus=work.st11,
+  msg=Invalid libds,
+  outds=work.rc11
+)
+
+/**
+  * Test 12 - the recorded results
   */
 
+%macro getres(ds, var);
+data _null_;
+  set &ds;
+  call symputx("&var",test_result,'g');
+run;
+%mend getres;
 
-data _null_;
-  set &rc_any;
-  call symputx('res_any',test_result,'g');
-run;
-data _null_;
-  set &rc_other;
-  call symputx('res_other',test_result,'g');
-run;
-data _null_;
-  set &rc_none;
-  call symputx('res_none',test_result,'g');
-run;
-data _null_;
-  set &rc_noref;
-  call symputx('res_noref',test_result,'g');
-run;
+%getres(work.rc1, res_any)
+%getres(work.rc2, res_other)
+%getres(work.rc3, res_none)
+%getres(work.rc4, res_noref)
+%getres(work.rc5, res_long)
+%getres(work.rc6, res_quotes)
+%getres(work.rc7, res_logmark)
+%getres(work.rc8, res_marker)
+%getres(work.rc9, res_nomsg)
+%getres(work.rc10, res_bslash)
+%getres(work.rc11, res_doubled)
 
 %mp_assert(
   iftrue=("&res_any"="PASS"),
@@ -136,15 +235,54 @@ run;
 
 %mp_assert(
   iftrue=("&res_none"="FAIL"),
-  desc=%str(Checking a response without a sasjsAbort object fails),
+  desc=%str(Checking a completed call fails),
   outds=work.test_results
 )
 
-
-
-
 %mp_assert(
   iftrue=("&res_noref"="FAIL"),
-  desc=%str(Checking an unassigned outref fails the assertion, not the step),
+  desc=%str(Checking a missing status dataset fails the assertion, not the step),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_long"="PASS"),
+  desc=%str(Checking an abort message past the old capture width matches in full),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_quotes"="PASS"),
+  desc=%str(Checking a message holding double quotes compares),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_logmark"="PASS"),
+  desc=%str(Checking the appended log extract is stripped),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_marker"="PASS"),
+  desc=%str(Checking a message containing the extract marker is not cut at it),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_nomsg"="FAIL"),
+  desc=%str(Checking a canceled call with no message fails the comparison),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_bslash"="PASS"),
+  desc=%str(Checking a message holding backslashes compares),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_doubled"="PASS"),
+  desc=%str(Checking the doubled extract separator is stripped),
   outds=work.test_results
 )

sas/sasjs/macros/mpe_columnlevelsecurity.sas

@@ -38,6 +38,7 @@
 
 
   <h4> SAS Macros </h4>
+  @li mf_nobs.sas
 
   @author 4GL Apps Ltd
   @copyright 4GL Apps Ltd.  This code may only be used within Data Controller

sas/sasjs/macros/mpe_derived.sas

@@ -49,9 +49,19 @@
   being maintained.  A cycle is rejected.
 
   Semicolons, macro triggers and comments are rejected, so a row can only ever
-  be an expression.  A derived variable may not reuse the name of a physical
-  column, and may not reference itself: an intra-row calculation would imply a
-  retained value, which depends on the order rows happen to arrive in.
+  be an expression, and an expression that would let the row submit code of its
+  own - DOSUBL, or a function reaching the file system, the environment or the
+  macro symbol table - is refused by name.
+
+  The field level rules themselves are core's: mp_derivedvarcheck scans the
+  definitions and mp_derivedvars calls it, which is the same macro the MPE_DERIVED
+  post edit hook runs at submit time (services/hooks/mpe_derived_postedit).  A
+  row is therefore measured against one rule list, whether it is being staged
+  or read.
+
+  A derived variable may not reuse the name of a physical column, and may not
+  reference itself: an intra-row calculation would imply a retained value,
+  which depends on the order rows happen to arrive in.
 
   <h4> SAS Macros </h4>
   @li mp_derivedvars.sas
@@ -87,7 +97,7 @@
 
 /* dc_dttmtfmt is the bitemporal cut-off used in the where= below - callers get
   it from %mpeinit().  Unset, the filter degenerates to "lt tx_to" and the
-  control-table step fails with a syntax error a long way from the cause. */
+  control-table step fails with a syntax fault a long way from the cause. */
 %mp_abort(iftrue= (%length(&libds)=0 or %length(&dclib)=0 or %length(&dc_dttmtfmt)=0)
   ,mac=&sysmacroname
   ,msg=%str(Missing: libds dclib dc_dttmtfmt)

sas/sasjs/macros/mpe_derived_check.sas

@@ -1,75 +0,0 @@
-/**
-  @file
-  @brief Field level checks for a staged MPE_DERIVED submission
-  @details Writes one row per staged row that breaks a rule, with the reason in
-  REASON, so a caller can report the first failure and abort.  Rows marked for
-  deletion are skipped - they are being removed, so their fields are not loaded.
-
-  The rules are the ones %mpe_derived applies at read time, so a submission
-  cannot stage a row the engine would then refuse.
-
-  @param inds the staged rows, already normalised by the caller
-  @param outds the dataset to create: one row per failing row, plus REASON
-
-  @author 4GL Apps Ltd
-  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
-    and may not be re-distributed or re-sold without the express permission of
-    4GL Apps Ltd.
-
-**/
-
-%macro mpe_derived_check(inds=work.staging_ds, outds=work.derived_check);
-data &outds;
-  set &inds;
-  length reason $200;
-  if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
-  call missing(reason);
-
-  if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
-  else if var_nm='' then reason='VAR_NM is required';
-  else if verify(strip(var_nm),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_')>0
-    then reason='VAR_NM must be a valid SAS name';
-  else if verify(substr(var_nm,1,1),'0123456789')=0
-    then reason='VAR_NM must not start with a digit';
-  else if var_nm='_____DELETE__THIS__RECORD_____'
-    then reason='VAR_NM is reserved';
-  else if var_type not in ('C','N')
-    then reason='VAR_TYPE must be C or N';
-  /* ACTIVE is a picker in the EDIT screen (HARDSELECT over the MPE_SELECTBOX
-    pair) - the same pair is enforced here, so a submission that bypasses the
-    grid cannot land a value the screen would never offer. */
-  else if missing(active) or active not in (0,1)
-    then reason='ACTIVE must be 1 or 0';
-  else if missing(var_len) or var_len ne int(var_len) or var_len<1
-    then reason='VAR_LEN must be a positive integer';
-  else if var_type='C' and var_len>32767
-    then reason='VAR_LEN exceeds 32767 for a character variable';
-  else if var_type='N' and var_len>8
-    then reason='VAR_LEN exceeds 8 for a numeric variable';
-  else if index(var_fmt,';')>0 or indexc(var_fmt,'&%')>0
-    or index(var_fmt,'/*')>0 or index(var_fmt,'*/')>0
-    then reason='VAR_FMT must be a format name, without ; & % or /* */';
-  else if verify(strip(var_fmt),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.')>0
-    then reason='VAR_FMT must be a format name';
-  /* VAR_LABEL is free text: any character is allowed, in any script, and the
-    quotes in it are escaped by %mp_derivedvars when the attrib statement is
-    built.  What is refused is anything that cannot be a single-line column
-    label - a control character or a line break - and a value long enough to
-    be cut down silently, since the engine truncates the label to 256. */
-  else if anycntrl(var_label)>0
-    then reason='VAR_LABEL must be plain text - no line breaks or control characters';
-  else if klength(var_label)>256
-    then reason='VAR_LABEL exceeds 256 characters';
-  else if missing(var_logic)
-    then reason='VAR_LOGIC is required';
-  else if index(var_logic,';')>0
-    then reason='VAR_LOGIC must be a single expression - remove the ;';
-  else if indexc(var_logic,'&%')>0
-    then reason='VAR_LOGIC must not contain macro triggers';
-  else if index(var_logic,'/*')>0 or index(var_logic,'*/')>0
-    then reason='VAR_LOGIC must not contain comments';
-
-  if not missing(reason) then output;
-run;
-
-%mend mpe_derived_check;

sas/sasjs/macros/mpe_derived_check.test.sas

@@ -1,194 +0,0 @@
-/**
-  @file
-  @brief testing the mpe_derived_check macro
-
-  @details Checks each field level rule the MPE_DERIVED post edit hook applies
-  to a staged submission.  The macro is exercised directly so the rules can be
-  probed with values a service cannot carry: VAR_LABEL is varchar(256) in the
-  table and a staging dataset declares the same length, so a longer label is
-  truncated before any check would see it.
-
-  <h4> SAS Macros </h4>
-  @li mf_nobs.sas
-  @li mp_assert.sas
-  @li mpe_derived_check.sas
-
-  @author 4GL Apps Ltd
-  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
-    and may not be re-distributed or re-sold without the express permission of
-    4GL Apps Ltd.
-
-**/
-
-/**
-  * Test 1 - one row per failing rule, and only failing rows
-  */
-data work.staging_ds;
-  length libref $8 dsn $32 var_nm $32 var_type $1 var_len 8
-    var_fmt $32 var_label $4000 var_deps $4000 var_logic $4000
-    active 8 _____DELETE__THIS__RECORD_____ $3;
-  libref='DCTEST';
-  dsn='DC_DERIV';
-  /* valid unless a row says otherwise - ACTIVE is mandatory in the screen and
-    the check enforces the same pair here */
-  active=1;
-  _____DELETE__THIS__RECORD_____='No';
-
-  /* meets every rule */
-  var_nm='GOOD';
-  var_type='N';
-  var_len=8;
-  var_fmt='';
-  var_label='a plain label';
-  var_deps='';
-  var_logic='1';
-  output;
-
-  /* delete marked rows are being removed, so their fields are not loaded */
-  var_nm='DELME';
-  var_label='first line'||'0A'x||'second line';
-  _____DELETE__THIS__RECORD_____='Yes';
-  output;
-  _____DELETE__THIS__RECORD_____='No';
-
-  /* free text is allowed, including quotes and punctuation */
-  var_nm='FREETXT';
-  var_label='It''s "quoted", plain text: 100 percent';
-  output;
-
-  /* a line break cannot be a single line column label */
-  var_nm='LBLBREAK';
-  var_label='first line'||'0A'x||'second line';
-  output;
-
-  /* longer than the table holds, so it would be cut down silently */
-  var_nm='LBLLONG';
-  var_label=repeat('x',300);
-  output;
-
-  /* a format name, not an expression */
-  var_nm='BADFMT';
-  var_fmt='DATE9;';
-  var_label='fmt';
-  output;
-
-  /* one expression, no statement separator, no macro triggers */
-  var_nm='BADLOGIC';
-  var_fmt='';
-  var_logic='x; y';
-  output;
-
-  /* C or N */
-  var_nm='BADTYPE';
-  var_logic='1';
-  var_type='X';
-  output;
-
-  /* ACTIVE is a picker in the screen: only 1 or 0 */
-  var_nm='BADACTIVE';
-  var_type='C';
-  var_logic='1';
-  active=2;
-  output;
-
-  stop;
-run;
-
-%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
-
-/**
-  * Test 2 - the reported rows, rule by rule
-  */
-data work.expected;
-  length var_nm $32 reason $200;
-  var_nm='LBLBREAK';
-  reason='VAR_LABEL must be plain text - no line breaks or control characters';
-  output;
-  var_nm='LBLLONG';
-  reason='VAR_LABEL exceeds 256 characters';
-  output;
-  var_nm='BADFMT';
-  reason='VAR_FMT must be a format name, without ; & % or /* */';
-  output;
-  var_nm='BADLOGIC';
-  reason='VAR_LOGIC must be a single expression - remove the ;';
-  output;
-  var_nm='BADTYPE';
-  reason='VAR_TYPE must be C or N';
-  output;
-  var_nm='BADACTIVE';
-  reason='ACTIVE must be 1 or 0';
-  output;
-run;
-
-proc sql;
-create table work.cmp as
-  select e.var_nm
-    ,case when a.reason=e.reason then 1 else 0 end as same
-  from work.expected e
-  left join work.derived_check a
-    on upcase(strip(a.var_nm))=upcase(e.var_nm);
-quit;
-
-data _null_;
-  set work.cmp;
-  call symputx(cats('rule_',var_nm),same,'l');
-run;
-
-%mp_assert(
-  iftrue=(%mf_nobs(work.derived_check)=6),
-  desc=%str(Checking only the six failing rows are reported),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_LBLBREAK=1),
-  desc=%str(Checking a label with a line break is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_LBLLONG=1),
-  desc=%str(Checking a label over 256 characters is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADFMT=1),
-  desc=%str(Checking a format carrying a statement separator is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADLOGIC=1),
-  desc=%str(Checking a logic value with a statement separator is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADTYPE=1),
-  desc=%str(Checking a VAR_TYPE other than C or N is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADACTIVE=1),
-  desc=%str(Checking an ACTIVE outside 1 or 0 is rejected),
-  outds=work.test_results
-)
-
-/**
-  * Test 3 - a row that meets every rule, a free text label, and a delete
-  * marked row are all left alone
-  */
-proc sql noprint;
-select count(*) into: notchecked
-  from work.derived_check
-  where upcase(strip(var_nm)) in ('GOOD','FREETXT','DELME');
-quit;
-
-%mp_assert(
-  iftrue=(&notchecked=0),
-  desc=%str(Checking a valid row, a free text label and a delete marked row are not reported),
-  outds=work.test_results
-)

sas/sasjs/macros/mpe_execute.sas

@@ -129,16 +129,40 @@
   * against whatever the previous call left in outlib.  A missing status table
   * and an empty one are both failures: state would stay blank and the caller
   * would carry on inferring, which is the thing this macro exists to stop. */
+%global _mpe_abortmsg;
+%let _mpe_abortmsg=;
 %let state=;
 %let msg=;
 %if %sysfunc(exist(&a_status))=1 %then %do;
   data _null_;
+    length _prog $32767 _state $32767 _prefix $32767 _childmsg $32767
+      _out $32767 _room 8;
     set &a_status;
     call symputx('state',state,'l');
     call symputx('msg',errmsg,'l');
+    /* the text the caller sees is assembled here rather than with %let: the
+      child's message is arbitrary text, and a %, an & or a quote in it would
+      be macro resolved on the way into a symbol.  symget/symputx hand it over
+      untouched, and a message already at the 32767 limit is cut to the room
+      left rather than overflowed by the join. */
+    if upcase(state) ne 'COMPLETED' then do;
+      _prog=symget('a_pgm');
+      _state=symget('state');
+      _childmsg=symget('msg');
+      _prefix=cat(trim(_prog),' did not complete: state=',trim(_state));
+      _room=32767-length(trim(_prefix))-2;
+      if _room<1 then _room=0;
+      if length(_childmsg)>_room then _childmsg=substr(_childmsg,1,_room);
+      if missing(_childmsg) then _out=trim(_prefix);
+      else _out=trim(_prefix)||' '||_childmsg;
+      call symputx('_mpe_abortmsg',_out,'g');
+    end;
   run;
 %end;
-%else %let state=missing(&a_status);
+%else %do;
+  %let state=missing(&a_status);
+  %let _mpe_abortmsg=&a_pgm did not complete: state=&state;
+%end;
 
 %if %length(&failfast)=0 %then %let failfast=1;
 %if &failfast=1 %then %do;
@@ -152,7 +176,7 @@
     %mp_abort(
       iftrue=(1=1)
       ,mac=&sysmacroname
-      ,msg=%str(&program did not complete: state=&state &msg)
+      ,msg=%superq(_mpe_abortmsg)
     )
   %end;
 %end;

sas/sasjs/macros/mpe_execute.test.sas

@@ -123,3 +123,70 @@ run;
   desc=%str(Checking mdebug=1 keeps the child log),
   outds=work.test_results
 )
+
+/**
+  * Test 3 - the abort text names the program, the state and the child message
+  *
+  * The text is assembled from the status row rather than with %let, so this is
+  * the path that has to hold: a real aborting child, failfast=0, and the text
+  * the caller would see in the abort.
+  */
+%let f3=%mf_getuniquefileref();
+data _null_;
+  file &f3 termstr=crlf;
+  put 'FILTER_TABLE:$41.';
+  put '%sysfunc(coalescec(&dclib..MPE_TABLES,))';
+run;
+
+%let outref3=%mf_getuniquefileref();
+
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f3:iwant,
+  outref=&outref3,
+  outstatus=work.exec_status3,
+  failfast=0,
+  viyaresult=WEBOUT_TXT
+)
+
+%let childmsg3=;
+data _null_;
+  set work.exec_status3 end=last;
+  if last then call symputx('childmsg3',errmsg,'l');
+run;
+
+%mp_assert(
+  iftrue=(%index(%superq(_mpe_abortmsg),%str(did not complete: state=))>0),
+  desc=%str(Checking the abort text names the program state),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(%index(%superq(_mpe_abortmsg),%superq(childmsg3))>0),
+  desc=%str(Checking the abort text carries the child message),
+  outds=work.test_results
+)
+
+/**
+  * Test 4 - the assembled text cannot overflow
+  *
+  * SAS character variables stop at 32767; a child message already at that limit
+  * is cut to the room the prefix leaves, so the join neither errors nor quietly
+  * loses the prefix.
+  */
+%let len_out=0;
+data _null_;
+  call symputx('len_out',length(symget('_mpe_abortmsg')),'l');
+run;
+
+%mp_assert(
+  iftrue=(&len_out<=32767),
+  desc=%str(Checking the abort text cannot overflow),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(%index(%superq(_mpe_abortmsg),%str(did not complete: state=))>0),
+  desc=%str(Checking the prefix survives whatever the child sent),
+  outds=work.test_results
+)

sas/sasjs/macros/mpe_filtermaster.sas

@@ -31,7 +31,6 @@
   @li mf_getuser.sas
   @li mf_getuniquefileref.sas
   @li mf_getuniquename.sas
-  @li mf_nobs.sas
   @li mp_abort.sas
   @li mp_filtergenerate.sas
   @li mpe_getgroups.sas

sas/sasjs/macros/mpe_getlabels.test.sas

@@ -20,8 +20,6 @@
       labels still come from the target.
 
   <h4> SAS Macros </h4>
-  @li mf_getuniquename.sas
-  @li mf_nobs.sas
   @li mp_assert.sas
   @li mp_assertdsobs.sas
   @li mp_assertscope.sas

sas/sasjs/macros/mpe_getversions.test.sas

@@ -9,7 +9,6 @@
   @li mp_assert.sas
   @li mp_assertscope.sas
   @li mpe_getversions.sas
-  @li mpe_targetloader.sas
   @li mx_execute.sas
 
 **/

sas/sasjs/macros/mpe_makedata.sas

@@ -2355,7 +2355,9 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_char="&x bottles of beer on the wall"
       ,some_dropdown='Option 1'
       ,some_num=ranuni(0)
-      ,some_date=round(ranuni(0)*1000,1)
+      /* First of the month, cycling through two years: a randomly spread date
+      gave every row its own day, so a date filter never returned many rows. */
+      ,some_date=intnx('month','01JAN1960'd,mod(&x,24),'S')
       ,some_datetime=round(ranuni(0)*50000,1)
       ,some_time=round(ranuni(0)*100,1)
       ,some_shortnum=6+round(ranuni(0)*94,1)

sas/sasjs/macros/mpe_makedatamodel.test.sas

@@ -22,5 +22,5 @@
 
 %mp_assert(
   iftrue=(&syscc = 0),
-  desc=Checking error condition
+  desc=Checking the abort condition
 )

sas/sasjs/macros/mpe_runhook.sas

@@ -8,7 +8,6 @@
   <h4> SAS Macros </h4>
   @li dc_getservicecode.sas
   @li mf_getapploc.sas
-  @li mp_abort.sas
   @li mp_include.sas
 
   @author 4GL Apps Ltd

sas/sasjs/macros/mpe_targetloader.sas

@@ -14,6 +14,7 @@
   @li mf_existvar.sas
   @li mf_getengine.sas
   @li mf_getuniquename.sas
+  @li mf_getuser.sas
   @li mp_abort.sas
   @li mp_loadformat.sas
   @li mp_lockanytable.sas

sas/sasjs/macros/mpeinit.sas

@@ -8,7 +8,6 @@
   <h4> SAS Macros </h4>
   @li dc_getsettings.sas
   @li mf_fmtdttm.sas
-  @li mf_getuser.sas
   @li mp_abort.sas
   @li mp_init.sas
 

sas/sasjs/macros/mpeterm.sas

@@ -5,8 +5,8 @@
     enables tracking and performance logging
 
   <h4> SAS Macros </h4>
-  @li mp_abort.sas
   @li mf_getplatform.sas
+  @li mf_getuser.sas
 
   @version 9.4
   @author 4GL Apps Ltd

sas/sasjs/services/admin/demodata.sas

@@ -18,12 +18,10 @@
     * CREATE two BASE libraries (HOOKLIB & PUBLIC) in the DC (physical) folder
 
   <h4> SAS Macros </h4>
+  @li mf_getapploc.sas
   @li mpeinit.sas
   @li mf_getengine.sas
-  @li mf_getuser.sas
   @li mf_increment.sas
-  @li mf_nobs.sas
-  @li mf_uid.sas
   @li mp_abort.sas
   @li mp_binarycopy.sas
   @li mp_replace.sas

sas/sasjs/services/admin/exportconfig.sas

@@ -10,8 +10,6 @@
   @li mf_getuser.sas
   @li mf_nobs.sas
   @li mp_abort.sas
-  @li mp_binarycopy.sas
-  @li mp_ds2cards.sas
   @li mp_ds2csv.sas
   @li mp_streamfile.sas
   @li mp_validatecol.sas

sas/sasjs/services/admin/refreshcatalog.test.1.sas

@@ -35,7 +35,7 @@ run;
 
 %mpe_assert_abort(
   desc=Macro content in libref aborts the service,
-  outref=web1,
+  outstatus=work.ab1,
   msg=Invalid libref
 )
 

sas/sasjs/services/admin/refreshcatalog.test.sas

@@ -4,7 +4,6 @@
 
   <h4> SAS Macros </h4>
   @li mp_assert.sas
-  @li mp_assertdsobs.sas
   @li mx_execute.sas
 
 **/

sas/sasjs/services/approvers/rejection.sas

@@ -4,8 +4,8 @@
   @details
 
   <h4> SAS Macros </h4>
+  @li mf_getattrn.sas
   @li mf_getuser.sas
-  @li mf_getvarlist.sas
   @li mf_verifymacvars.sas
   @li mp_abort.sas
   @li mp_lockanytable.sas

sas/sasjs/services/auditors/getauditfile.sas

@@ -11,7 +11,6 @@
   @li mpe_accesscheck.sas
   @li mp_abort.sas
   @li mp_dirlist.sas
-  @li mp_binarycopy.sas
   @li mf_getattrn.sas
   @li mp_streamfile.sas
 

sas/sasjs/services/auditors/getdiffs.sas

@@ -9,7 +9,6 @@
   @li mf_getattrn.sas
   @li mf_getuser.sas
   @li mp_abort.sas
-  @li mp_binarycopy.sas
   @li mp_streamfile.sas
   @li mp_validatecol.sas
 

sas/sasjs/services/auditors/getdiffs.test.sas

@@ -114,7 +114,7 @@ run;
 
 %mpe_assert_abort(
   desc=SQL injection payload in libds aborts the service,
-  outref=web1,
+  outstatus=work.ab1,
   msg=Invalid libds
 )
 
@@ -146,7 +146,7 @@ run;
 
 %mpe_assert_abort(
   desc=Path traversal in table aborts the service,
-  outref=web2,
+  outstatus=work.ab2,
   msg=Invalid table
 )
 
@@ -171,7 +171,7 @@ run;
 
 %mpe_assert_abort(
   desc=Path traversal in stp_diffs_csv aborts the service,
-  outref=web3,
+  outstatus=work.ab3,
   msg=Invalid stp_diffs_csv
 )
 

sas/sasjs/services/auditors/getstagetable.sas

@@ -13,7 +13,6 @@
 
   <h4> SAS Macros </h4>
   @li dc_assignlib.sas
-  @li mf_existds.sas
   @li mp_abort.sas
   @li mp_applyformats.sas
   @li mp_getcols.sas

sas/sasjs/services/auditors/getstagetable.test.sas

@@ -5,7 +5,6 @@
   <h4> SAS Macros </h4>
   @li mx_execute.sas
   @li mp_assert.sas
-  @li mp_assertdsobs.sas
 
 **/
 

sas/sasjs/services/auditors/postdata.sas

@@ -17,17 +17,14 @@
   |SHOW_DIFFS|DC20220208T142124517_124703_1184|"Tue, 08 Feb 2022 14:23:05 GMT"|
 
   <h4> SAS Macros </h4>
-  @li bitemporal_dataloader.sas
   @li dc_assignlib.sas
   @li dc_cassave.sas
-  @li mf_existds.sas
   @li mf_existvar.sas
   @li mf_getattrn.sas
   @li mf_getengine.sas
   @li mf_getquotedstr.sas
   @li mf_getuniquelibref.sas
   @li mf_getuser.sas
-  @li mf_getvarlist.sas
   @li mf_nobs.sas
   @li mf_verifymacvars.sas
   @li mp_abort.sas
@@ -707,7 +704,7 @@ run;
 %postdata()
 
 /* mp_abort cannot exit cleanly from inside a %include, so an abort there
-  (eg in a hook script run via mp_include) only records the error in
+  (eg in a hook script run via mp_include) only records the failure in
   work.mp_abort_errds.  mode=INCLUDE picks that up and aborts properly here. */
 %mp_abort(mode=INCLUDE)
 

sas/sasjs/services/editors/getdynamiccolvals.sas

@@ -166,7 +166,7 @@ run;
 /* execute the dynamic code */
 %mp_include(sascode)
 /* mp_abort cannot exit cleanly from inside a %include, so an abort there
-  (eg in a hook script run via mp_include) only records the error in
+  (eg in a hook script run via mp_include) only records the failure in
   work.mp_abort_errds.  mode=INCLUDE picks that up and aborts properly here. */
 %mp_abort(mode=INCLUDE)
 

sas/sasjs/services/editors/getdynamiccolvals.test.1.sas

@@ -5,7 +5,6 @@
   <h4> SAS Macros </h4>
   @li mp_assertcols.sas
   @li mp_assertcolvals.sas
-  @li mp_assertdsobs.sas
   @li mf_getuniquefileref.sas
   @li mx_execute.sas
 

sas/sasjs/services/editors/getlog.sas

@@ -8,7 +8,6 @@
   @li mf_getuser.sas
   @li mp_abort.sas
   @li mp_dirlist.sas
-  @li mp_binarycopy.sas
   @li mp_streamfile.sas
 
   @version 9.2

sas/sasjs/services/editors/loadfile.sas

@@ -3,6 +3,12 @@
   @brief Loads a file
   @details
 
+  mp_binarycopy is declared for the SAS 9.3 and earlier fallback in the copy
+  step below: the %mp_binarycopy call is commented out on 9.4 and later, so
+  nothing in the file calls it.
+
+  @sasjslint {"noUnusedMacros": false}
+
   <h4> SAS Macros </h4>
   @li mddl_sas_cntlout.sas
   @li mp_abort.sas
@@ -286,14 +292,14 @@ options mprint;
   ,dc_dttmtfmt=&dc_dttmtfmt
 )
 /* mp_abort cannot exit cleanly from inside a %include, so an abort there
-  (eg in a hook script run via mp_include) only records the error in
+  (eg in a hook script run via mp_include) only records the failure in
   work.mp_abort_errds.  mode=INCLUDE picks that up and aborts properly here. */
 %mp_abort(mode=INCLUDE)
 
 %mp_abort(
   iftrue= (%sysfunc(fileexist(%sysfunc(pathname(work))/mf_abort.error)) ne 0)
   ,mac=&_program
-  ,msg=%nrstr(Problem occurred in &sysmacroname (mf_abort.error file found))
+  ,msg=%nrstr(Problem occurred in &sysmacroname (abort marker file found))
 )
 
 %mp_abort(iftrue= (&syscc ne 0)

sas/sasjs/services/editors/restore.sas

@@ -155,7 +155,7 @@ run;
   ,dc_dttmtfmt=&dc_dttmtfmt
 )
 /* mp_abort cannot exit cleanly from inside a %include, so an abort there
-  (eg in a hook script run via mp_include) only records the error in
+  (eg in a hook script run via mp_include) only records the failure in
   work.mp_abort_errds.  mode=INCLUDE picks that up and aborts properly here. */
 %mp_abort(mode=INCLUDE)
 

sas/sasjs/services/editors/restore.test.2.sas

@@ -0,0 +1,172 @@
+/**
+  @file
+  @brief testing restore of a change to a special missing value
+
+  @details A cell holding a special missing (._ .A-.Z) and changed to an
+    ordinary number must be reverted to that special missing. mp_stripdiffs
+    writes the old value into a generated UPDATE statement, and PUT drops the
+    leading period of a special missing, so the statement used to be
+    "set SOME_BESTNUM =_" - which PROC SQL reads as a column name.
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+  @li mx_execute.sas
+
+**/
+
+%let _program=&appLoc/services/editors/restore;
+
+/* seed the base table with a special missing - the audit stores whatever the
+  base holds, so this is the value the revert has to write back */
+proc sql noprint;
+update &dclib..mpe_x_test set some_bestnum=._
+  where primary_key_field=3;
+quit;
+
+/**
+  * load 1 - change that cell to an ordinary number
+  */
+data work.sascontroltable;
+  action='LOAD';
+  message="special missing prep";
+  libds="&dclib..MPE_X_TEST";
+  output;
+  stop;
+run;
+
+data work.jsdata;
+  set &dclib..mpe_x_test(where=(primary_key_field=3)
+    rename=(some_date=dt2 SOME_DATETIME=dttm2 SOME_TIME=tm2)
+  );
+  /* the adapter sends these as strings */
+  some_date=put(dt2,date9.);
+  SOME_DATETIME=put(dttm2,datetime19.);
+  some_time=put(tm2,time.);
+  drop dt2 dttm2 tm2;
+  _____DELETE__THIS__RECORD_____='No';
+  some_bestnum=42;
+run;
+
+%mx_execute(&appLoc/services/editors/stagedata,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.jsdata work.sascontroltable,
+  outlib=web1,
+  mdebug=&sasjs_mdebug
+)
+
+%let status=0;
+data _null_;
+  set web1.sasparams;
+  if status='SUCCESS' then call symputx('status',1);
+  call symputx('dsid',dsid);
+run;
+%mp_assert(
+  iftrue=(&status=1 and &syscc=0),
+  desc=Checking the special missing change was staged,
+  outds=work.test_results
+)
+
+data work.sascontroltable;
+  ACTION='APPROVE_TABLE';
+  TABLE="&dsid";
+  DIFFTIME="%sysfunc(datetime())";
+  output;
+  stop;
+run;
+%mx_execute(&appLoc/services/auditors/postdata,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.sascontroltable,
+  outlib=web3,
+  outref=wb3,
+  mdebug=&sasjs_mdebug
+)
+
+%let status=0;
+data _null_;
+  set web3.apparams;
+  if response='SUCCESS!' then call symputx('status',1);
+run;
+%mp_assert(
+  iftrue=(&status=1 and &syscc=0),
+  desc=Checking the change was approved,
+  outds=work.test_results
+)
+
+/* the load reference to revert */
+%let loadref=0;
+data _null_;
+  set &dclib..mpe_submit end=last;
+  if last then call symputx('loadref',table_id);
+run;
+
+/**
+  * revert it - the old value in the diff is the special missing
+  */
+data work.restore_in;
+  load_ref="&loadref";
+  output;
+  stop;
+run;
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.restore_in,
+  outlib=web1,
+  mdebug=&sasjs_mdebug
+)
+
+%let newref=0;
+data _null_;
+  set web1.restore_out;
+  call symputx('newref',loadref);
+run;
+%mp_assert(
+  iftrue=(&newref ne 0 and &syscc=0),
+  desc=Checking the revert was submitted,
+  outds=work.test_results
+)
+
+/* approve the reversion */
+data work.sascontroltable;
+  ACTION='APPROVE_TABLE';
+  TABLE="&newref";
+  DIFFTIME="%sysfunc(datetime())";
+  output;
+  stop;
+run;
+%mx_execute(&appLoc/services/auditors/postdata,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.sascontroltable,
+  outlib=web3,
+  outref=wb3,
+  mdebug=&sasjs_mdebug
+)
+
+%let status=0;
+data _null_;
+  set web3.apparams;
+  if response='SUCCESS!' then call symputx('status',1);
+run;
+%mp_assert(
+  iftrue=(&status=1 and &syscc=0),
+  desc=Checking the reversion was approved,
+  outds=work.test_results
+)
+
+/* and the cell is back to the special missing, not a number and not blank */
+%let spmcheck=0;
+data _null_;
+  set &dclib..mpe_x_test(where=(primary_key_field=3));
+  if some_bestnum=._ then call symputx('spmcheck',1);
+  putlog some_bestnum=;
+run;
+%mp_assert(
+  iftrue=(&spmcheck=1),
+  desc=Checking the special missing was restored,
+  outds=work.test_results
+)
+
+/* put the fixture back as makedata left it */
+proc sql noprint;
+update &dclib..mpe_x_test set some_bestnum=44
+  where primary_key_field=3;
+quit;

sas/sasjs/services/editors/restore.test.sas

@@ -3,7 +3,6 @@
   @brief testing restore process
 
   <h4> SAS Macros </h4>
-  @li mf_getuniquefileref.sas
   @li mx_execute.sas
   @li mp_assert.sas
 

sas/sasjs/services/editors/stagedata.test.3.sas

@@ -112,7 +112,7 @@ run;
 
 %mpe_assert_abort(
   desc=Checking RLS EDIT rule is rejected for REPLACE loadtype table (#211),
-  outref=wb1
+  outstatus=work.ab1
 )
 
 /**
@@ -179,7 +179,7 @@ run;
 
 %mpe_assert_abort(
   desc=Checking an RLS rule on a derived variable is rejected,
-  outref=wb3
+  outstatus=work.ab3
 )
 
 /* tidy up the derived configuration */
@@ -222,7 +222,7 @@ run;
 
 %mpe_assert_abort(
   desc=Checking REPLACE loadtype is rejected for RLS EDIT-secured table (#211),
-  outref=wb2
+  outstatus=work.ab2
 )
 
 /**

sas/sasjs/services/editors/stagedata.test.4.sas

@@ -6,16 +6,19 @@
       table - the engine reads only active rows, so the dependency can never
       resolve at run time
     2. a row is staged with active=0 while a surviving active row depends on it
+    3. a staged row breaks a field rule of the shared scan (core's
+      mp_derivedvarcheck), or carries an ACTIVE value the EDIT screen's picker
+      cannot produce
 
   They should NOT abort when:
-    3. a new active row is added to a table that also holds an inactive row -
+    4. a new active row is added to a table that also holds an inactive row -
       the inactive row is not part of the dependency graph, so it cannot
       create a cycle or a stale reference
-    4. a delete-marked RLS rule naming a derived variable is submitted - the
+    5. a delete-marked RLS rule naming a derived variable is submitted - the
       rule is being removed, so its variables are not validated
-    5. a delete and re-add of the same variable is submitted together - the
+    6. a delete and re-add of the same variable is submitted together - the
       re-added active row resolves the dependency
-    6. an inactive row is set active again while another active row depends on
+    7. an inactive row is set active again while another active row depends on
       it - the re-activation is what makes the dependency resolvable
 
   The staged loads are aborted by the post edit hook. The assertion fetches the
@@ -100,7 +103,7 @@ run;
 
 %mpe_assert_abort(
   desc=Checking a dependency on an inactive derived row is rejected,
-  outref=wb1
+  outstatus=work.ab1
 )
 
 /**
@@ -166,7 +169,7 @@ run;
 
 %mpe_assert_abort(
   desc=Checking deactivating a relied-on derived row is rejected,
-  outref=wb2
+  outstatus=work.ab2
 )
 
 /**
@@ -379,7 +382,7 @@ run;
 
 %mpe_assert_abort(
   desc=Checking deleting a relied-on derived row is rejected,
-  outref=wb5
+  outstatus=work.ab5
 )
 
 /**
@@ -553,6 +556,115 @@ run;
 )
 
 
+/**
+  * Test 8 - a field rule of the shared scan (mp_derivedvarcheck, core) is
+  * applied at submit time: an expression that could submit code of its own is
+  * refused by name, with the reason the scan gives. This is the same scan the
+  * engine runs at read time, so the case pins that the hook calls it rather
+  * than carrying a rule list of its own.
+  */
+data work.sascontroltable;
+  action='LOAD';
+  message='A refused function in VAR_LOGIC should abort';
+  libds="&dc_libref..MPE_DERIVED";
+  output;
+  stop;
+run;
+
+data work.jsdata;
+  length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
+    VAR_LOGIC $4000 VAR_FMT $32
+    _____DELETE__THIS__RECORD_____ $3;
+  LIBREF='DCTEST';
+  DSN='DC_DERIV';
+  VAR_NM='CALLER';
+  VAR_DEPS='';
+  VAR_TYPE='N';
+  VAR_LEN=8;
+  VAR_FMT='';
+  /* no semicolon in the payload: the single-expression rule is tested before
+    the function-name rule, so a quoted semicolon would abort on that first */
+  VAR_LOGIC='dosubl("data _null_")';
+  ACTIVE=1;
+  _____DELETE__THIS__RECORD_____='No';
+run;
+
+/* the child is expected to abort here, so the failure must be asserted rather
+  than aborting this program - hence failfast=0 */
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.sascontroltable work.jsdata,
+  outlib=web8,
+  outref=wb8,
+  outstatus=work.ab8,
+  failfast=0,
+  mdebug=&sasjs_mdebug
+)
+
+/* the assertion compares the abort's MSG exactly, so the expected value is
+  assembled on one line: a literal wrapped across two source lines carries the
+  newline and the continuation indent with it.
+  The text has no space after either colon because that is how the child builds
+  it - mpe_derived emits &mac.:&VAR_NM.:&reason, and the reason itself is
+  'VAR_LOGIC may not call a function that' || ' reaches the file system, ...'.
+  Spacing here is the child's, not a formatting choice. */
+%let exp8=MPE_DERIVED:CALLER:VAR_LOGIC may not call a function that;
+%let exp8=&exp8 submits code or reaches the file system,;
+%let exp8=&exp8 the environment or the macro symbol table;
+
+%mpe_assert_abort(
+  desc=Checking a refused function in VAR_LOGIC is rejected,
+  outstatus=work.ab8,
+  msg=%superq(exp8)
+)
+
+/**
+  * Test 9 - the fields the hook owns rather than the scan: ACTIVE is a picker
+  * in the EDIT screen (HARDSELECT over the MPE_SELECTBOX pair), so a submission
+  * that bypasses the grid cannot land a value the screen would never offer.
+  */
+data work.sascontroltable;
+  action='LOAD';
+  message='An ACTIVE value the screen cannot offer should abort';
+  libds="&dc_libref..MPE_DERIVED";
+  output;
+  stop;
+run;
+
+data work.jsdata;
+  length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
+    VAR_LOGIC $4000 VAR_FMT $32
+    _____DELETE__THIS__RECORD_____ $3;
+  LIBREF='DCTEST';
+  DSN='DC_DERIV';
+  VAR_NM='BADACT';
+  VAR_DEPS='';
+  VAR_TYPE='N';
+  VAR_LEN=8;
+  VAR_FMT='';
+  VAR_LOGIC='1';
+  ACTIVE=2;
+  _____DELETE__THIS__RECORD_____='No';
+run;
+
+/* this child is expected to abort too - assert the failure rather than
+  cancelling this program */
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.sascontroltable work.jsdata,
+  outlib=web9,
+  outref=wb9,
+  outstatus=work.ab9,
+  failfast=0,
+  mdebug=&sasjs_mdebug
+)
+
+%mpe_assert_abort(
+  desc=Checking an ACTIVE value outside the picker pair is rejected,
+  outstatus=work.ab9,
+  msg=MPE_DERIVED:BADACT:ACTIVE must be 1 or 0
+)
+
 /* tidy up so repeated runs start clean */
 proc sql noprint;
 delete from &dc_libref..mpe_derived

sas/sasjs/services/hooks/mpe_column_level_security_postedit.sas

@@ -13,6 +13,11 @@
   This validation checks the incoming column_level_security settings to ensure
   each individual filter is valid
 
+  Hook scripts run inside the service that includes them, so the macros
+  they call are already in that session. They are listed as ignored rather
+  than declared, so the script keeps no macro list of its own to maintain.
+
+  @sasjslint {"ignoredMacros": ["mp_abort"]}
 
 **/
 

sas/sasjs/services/hooks/mpe_derived_postedit.sas

@@ -34,7 +34,7 @@
   <h4> SAS Macros </h4>
   @li mf_nobs.sas
   @li mp_abort.sas
-  @li mpe_derived_check.sas
+  @li mp_derivedvarcheck.sas
 
   <h4> Related Macros </h4>
   @li mpe_derived.sas
@@ -100,14 +100,43 @@ run;
 )
 
 /**
-  * Field level checks - the same rules %mpe_derived applies at read time
+  * Field level checks.  The rules for the derived variable itself - the name,
+  * the type, length, format, label and the expression - are core's, in
+  * mp_derivedvarcheck, so this hook and the read time engine (%mpe_derived ->
+  * mp_derivedvars) scan a row with the same rules and report the same reasons.
+  * What is added here are the two fields that belong to Data Controller rather
+  * than to the expression: the control table key, which the loader normalises
+  * but does not require, and ACTIVE, which the EDIT screen offers as a picker
+  * over the MPE_SELECTBOX pair - so a submission that bypasses the grid cannot
+  * land a value the screen would never offer.
   */
-%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
+data work.derived_dcfield(keep=name reason);
+  set work.staging_ds;
+  if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
+  length name $32 reason $200;
+  call missing(reason);
+  name=upcase(strip(var_nm));
+  if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
+  else if missing(active) or active not in (0,1)
+    then reason='ACTIVE must be 1 or 0';
+  if not missing(reason) then output;
+run;
+
+%mp_derivedvarcheck(inds=work.staging_ds
+  ,outds=work.derived_field
+  ,reserved=%str(_____DELETE__THIS__RECORD_____)
+)
+
+/* the control table fields are reported before the expression's own rules, so
+  which of the two passes found a row does not change the message it gets */
+data work.derived_check;
+  set work.derived_dcfield work.derived_field;
+run;
 
 %if %mf_nobs(work.derived_check)>0 %then %do;
   data _null_;
     set work.derived_check(obs=1);
-    call symputx('der_errmsg',cats('MPE_DERIVED: ',var_nm,': ',reason));
+    call symputx('der_errmsg',cats('MPE_DERIVED: ',name,': ',reason));
     call symputx('der_errflag',1);
   run;
 %end;

sas/sasjs/services/hooks/mpe_security_postedit.sas

@@ -10,6 +10,11 @@
     @li LIBREF - The library of the dataset being edited (is assigned)
     @li DS - The dataset being edited
 
+  Hook scripts run inside the service that includes them, so the macros
+  they call are already in that session. They are listed as ignored rather
+  than declared, so the script keeps no macro list of its own to maintain.
+
+  @sasjslint {"ignoredMacros": ["mp_abort"]}
 
 **/
 
@@ -35,5 +40,7 @@ run;
 
 %mp_abort(iftrue=(&errval=1)
   ,mac=mpe_security_postedit.sas
-  ,msg=%str(&errmsg)
+  /* %bquote: the submitted value is interpolated into this message, and a
+    comma in it would split this call into extra positional parameters */
+  ,msg=%bquote(&errmsg)
 )

sas/sasjs/services/hooks/mpe_tables_postedit.sas

@@ -14,6 +14,11 @@
   valid.  If a non-default AUDIT_LIBDS is being used, there is also a check
   to ensure that this table already exists.
 
+  Hook scripts run inside the service that includes them, so the macros
+  they call are already in that session. They are listed as ignored rather
+  than declared, so the script keeps no macro list of its own to maintain.
+
+  @sasjslint {"ignoredMacros": ["mp_abort", "dc_assignlib"]}
 
 **/
 

sas/sasjs/services/hooks/mpe_validations_postedit.sas

@@ -19,6 +19,12 @@
   <h4> Related Macros </h4>
   @li mpe_loader.sas
 
+  Hook scripts run inside the service that includes them, so the macros
+  they call are already in that session. They are listed as ignored rather
+  than declared, so the script keeps no macro list of its own to maintain.
+
+  @sasjslint {"ignoredMacros": ["mp_abort"]}
+
 **/
 
 

sas/sasjs/services/hooks/mpe_xlmap_info_postedit.sas

@@ -17,6 +17,12 @@
   @li dc_assignlib.sas
   @li mp_validatecol.sas
 
+  Hook scripts run inside the service that includes them, so the macros
+  they call are already in that session. They are listed as ignored rather
+  than declared, so the script keeps no macro list of its own to maintain.
+
+  @sasjslint {"ignoredMacros": ["mp_abort"]}
+
 **/
 
 data work.staging_ds;

sas/sasjs/services/hooks/sample_xlmap_data_postedit.sas

@@ -11,6 +11,12 @@
     @li LIBREF - The library of the dataset being edited (is assigned)
     @li DS - The target dataset being loaded
 
+  Hook scripts run inside the service that includes them, so the macros
+  they call are already in that session. They are listed as ignored rather
+  than declared, so the script keeps no macro list of its own to maintain.
+
+  @sasjslint {"ignoredMacros": ["mp_abort"]}
+
 **/
 
 %let abort=0;
@@ -35,7 +41,7 @@ data work.staging_ds;
     end;
   end;
 
-  /* publish error message */
+  /* publish abort message */
   if err=1 then do;
     errmsg=catx(' ',xlmap_range_id,':',value_txt,'->',errmsg);
     call symputx('errmsg',errmsg);

sas/sasjs/services/public/getchangeinfo.test.sas

@@ -3,9 +3,9 @@
   @brief testing getchangeinfo service
 
   <h4> SAS Macros </h4>
+  @li mf_nobs.sas
   @li mx_execute.sas
   @li mp_assert.sas
-  @li mp_assertcolvals.sas
   @li mf_getuniquefileref.sas
 
 **/

sas/sasjs/services/public/getcols.test.sas

@@ -42,7 +42,7 @@ run;
 
 %mpe_assert_abort(
   desc=Macro content in libds aborts the service,
-  outref=web1,
+  outstatus=work.ab1,
   msg=Invalid libds
 )
 

sas/sasjs/services/public/getcolvals.test.4.sas

@@ -43,7 +43,7 @@ run;
 
 %mpe_assert_abort(
   desc=Macro content in libds aborts the service,
-  outref=web1,
+  outstatus=work.ab1,
   msg=Invalid libds
 )
 
@@ -68,7 +68,7 @@ run;
 
 %mpe_assert_abort(
   desc=Macro content in col aborts the service,
-  outref=web2,
+  outstatus=work.ab2,
   msg=Invalid col
 )
 

sas/sasjs/services/public/getgroups.test.sas

@@ -3,7 +3,6 @@
   @brief testing getgroups service.
 
   <h4> SAS Macros </h4>
-  @li mf_getuser.sas
   @li mp_assertcols.sas
   @li mx_execute.sas
 

sas/sasjs/services/public/getrawdata.test.1.sas

@@ -43,7 +43,7 @@ run;
 
 %mpe_assert_abort(
   desc=Macro content in table aborts the service,
-  outref=web1,
+  outstatus=work.ab1,
   msg=Invalid table
 )
 
@@ -68,7 +68,7 @@ run;
 
 %mpe_assert_abort(
   desc=SQL injection in filter aborts the service,
-  outref=web2,
+  outstatus=work.ab2,
   msg=Invalid filter
 )
 
@@ -95,7 +95,7 @@ run;
 
 %mpe_assert_abort(
   desc=Path traversal in table aborts the service,
-  outref=web3,
+  outstatus=work.ab3,
   msg=Invalid table
 )
 

sas/sasjs/services/public/getversion.sas

@@ -17,7 +17,7 @@
   The data for a particular version
 
   <h4> SAS Macros </h4>
-  @li mf_getuser.sas
+  @li mp_abort.sas
   @li mpeinit.sas
   @li mpe_getvars.sas
 

sas/sasjs/services/public/validatefilter.test.1.sas

@@ -42,7 +42,7 @@ run;
 
 %mpe_assert_abort(
   desc=Macro content in filter_table aborts the service,
-  outref=web1,
+  outstatus=work.ab1,
   msg=Invalid filter_table
 )
 

sas/sasjs/services/public/viewtables.sas

@@ -34,6 +34,7 @@
   <h4> SAS Macros </h4>
   @li dc_assignlib.sas
   @li mf_getuser.sas
+  @li mp_abort.sas
   @li mpe_getgroups.sas
   @li mpe_getvars.sas
   @li mpeinit.sas

sas/sasjs/services/validations/mpe_alerts.alert_lib.sas

@@ -47,8 +47,7 @@
   |2|CURRENCY_SYMBOL|C||"HKD"|1|
 
   <h4> SAS Macros </h4>
-  @li dc_getlibs.sas
-
+  @li mp_abort.sas
 
 **/
 

sas/sasjs/services/validations/mpe_tables.dsn.sas

@@ -47,10 +47,10 @@
   |2|CURRENCY_SYMBOL|C||"EUR"|
   |2|CURRENCY_SYMBOL|C||"HKD"|
 
-
   <h4> SAS Macros </h4>
-  @li dc_getlibs.sas
-
+  @li dc_assignlib.sas
+  @li mf_getuniquename.sas
+  @li mp_abort.sas
 
 **/
 

sas/sasjs/services/validations/mpe_x_test.some_num.sas

@@ -18,9 +18,6 @@
   |---|
   |44|
 
-  <h4> SAS Macros </h4>
-  @li dc_getlibs.sas
-
 
 **/
 

sas/sasjs/services/validations/tables_all.sas

@@ -45,6 +45,8 @@
 
   <h4> SAS Macros </h4>
   @li dc_assignlib.sas
+  @li mf_getuniquename.sas
+  @li mp_abort.sas
 
 **/
 

sas/sasjs/targets/sas9/services_meta/admin/makedata.sas

@@ -10,6 +10,7 @@
   @warning This STP self destructs! It will delete itself after a successful run
     to avoid being executed twice (and overwriting actual data)
 
+  @sasjslint {"noBannedStrings": false}
 
   <h4> SAS Macros </h4>
   @li mf_getapploc.sas
@@ -20,7 +21,6 @@
   @li mm_createstp.sas
   @li mm_deletedocument.sas
   @li mm_deletestp.sas
-  @li mm_getstpcode.sas
   @li mm_getstpinfo.sas
   @li mp_abort.sas
   @li mp_init.sas