chore(release): 7.19.0 [skip ci]

# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)

### Bug Fixes

* address the review findings on the harness and refresh changes ([36e4e88](36e4e88eb1))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](698dea6d6b))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](0d58be975d))
* **client:** open the SAS log page with noopener ([072a133](072a133702))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](1fb4818d21))
* **cypress:** apply the licence key when the estate has none ([be8f445](be8f445ae1)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](fed43cb07e))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](f5aade5976))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](1e224763ba))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](9b3cf50b9c))
* **derived:** address five review findings ([ccd468b](ccd468b335))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](be50cb5d08))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](9914790777))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](9ef813d439))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](422f766cd4))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](7830b7bcfa))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](2d5de9a324))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](6a1128a463))
* **derived:** eight review findings ([561d7fc](561d7fc3ed))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](1863076bc9)), closes [#366](#366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](d3dec682aa))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](469229aecb))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](7fb981daa9))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](5f5d8681b9))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](5f443b0b16))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](6b5bf3e6be))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](85c89ca2a4))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](f8f65681e1))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](7efbf31ea5))
* **derived:** only the delete marker is a reserved name ([af31c36](af31c36b67))
* **derived:** open the statements file with a long record length ([ee31782](ee31782fbf))
* **derived:** pair the filterds %else with %do/%end ([88a38db](88a38db877))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](27ddcfb575))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](7799a300df))
* **derived:** reject a dependency on a row this submission deletes ([8806504](8806504967))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](9a047ad42e))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](e0da9f1098))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](688550a28d))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](1b5c4b3ecd))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](4d9ad38df1))
* **editor:** a blank truncation note is not a warning ([f182179](f182179403))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](04ce1861e7))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](4bcee706ff))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](45dce7b11b))
* **getdata:** clear the globals it declares before use ([359b570](359b570b85))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](059cdf176a))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](661ee19cfa))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](355c46a7b6))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](d626cb8f2d))
* **mocks:** address the audit-pack review findings ([489819b](489819b051))
* **mocks:** carry the zip flags into the central directory ([c4021a4](c4021a42ff))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](62f55db170))
* **mocks:** copy the physical column list before deriving ([f411b7d](f411b7d269))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](60ec470404))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](74060c0585))
* **mocks:** drop derived variables before staging ([73d25b1](73d25b1220))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](347dbb53b0))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](ee3c8787ef))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](6cad23fce9))
* **mocks:** search the physical columns only in the viewer ([293022d](293022d46b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](fb36379520))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](9c294c0ab5))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](0a52d20c5f))
* **mpe_execute:** fail fast when the child left no status ([a036386](a03638667f))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](220408538b))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](f2b1428ec3))
* name the startupservice launcher button "Show log" ([3445107](3445107d9a))
* offer the launcher on the globvars failure modal too ([ad2da45](ad2da45389))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](46bacdd6d9))
* **refresh:** one audit folder per refresh call ([56b6620](56b6620ef4))
* **sites:** round-trip the committed licence before publishing ([4d462d3](4d462d310f))
* **test:** getdata test ([da30266](da30266278))
* **tests:** enable closeouts logging ([fdcd8fa](fdcd8fa4b3))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](033ffd3025))
* **ui:** one error modal at a time ([81ccf75](81ccf758e9))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](42355f17bc))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](337ea8e5e0))

### Features

* **dc_getlibs:** dump the library list when debug is on ([582523d](582523dc87))
* **derived:** allow a variable to shadow a physical column ([12beeff](12beeff695))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](813bce6856))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](7c9ce9a712))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](87e794f769))
* **derived:** report the dependency cycle path ([cee1a95](cee1a954ee))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](78e88a4112))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](0b010f2c65))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](aa6d50e1ea))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](33d043ea92))
* **editor:** derived variables in the EDIT screen ([b0fd354](b0fd354ecd))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](6513d09f05))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](34691dceec))
* **editor:** show the derived column label in the header tooltip ([cfe6647](cfe6647ca8))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](752d944c20))
* **mocks:** implement derived variables in the JS services ([b4fae88](b4fae88ae1))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](5d183d1350))
* run startupservice in a new window from its error modal ([40f6133](40f61331aa))
* **sites:** mint a licence for the published sites ([7d9df71](7d9df71fdb))

### Reverts

* **derived:** no shadowing, and no self reference ([5af8e6f](5af8e6fc35))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](31aa859153))

.gitea/workflows/build.yaml

@@ -15,6 +15,22 @@ jobs:
         with:
           node-version: ${{ env.NODE_VERSION }}
 
+      - name: Scan for secrets (gitleaks)
+        # The local pre-commit hook is suppressed entirely when .npmrc sets
+        # ignore-scripts=true (the prepare script that would activate the
+        # hooks never runs), so CI is the enforcement point: a leaked
+        # credential must fail the build, not just a commit.
+        # Uses the same @nogoo9/gitleaks binary the pre-commit hook pins - the
+        # gitleaks-action requires a license key for organization repos.
+        # Runs before client/.npmrc is written below, because that file is
+        # created from the NPMRC secret and a later scan would flag the
+        # registry token this job injects itself.
+        # --no-git scans the working tree, which is what a pull request gate
+        # is about.
+        run: |
+          npm ci
+          node_modules/.bin/gitleaks detect --no-git --source . --redact --no-banner -v
+
       - name: Install Google Chrome
         run: |
           apt-get update
@@ -126,6 +142,7 @@ jobs:
           echo NODE_PATH=node >> .env
           echo CORS=enable >> .env
           echo WHITELIST=http://localhost:4200 >> .env
+          echo HTTP_PORT=5000 >> .env
           # The server refuses to run as root unless RUN_AS names an account
           # (a security change in the recent releases). This job runs as root
           # in a throwaway container, so opt in explicitly.
@@ -165,7 +182,7 @@ jobs:
           sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
 
       - name: Prepare and run frontend and cypress
-        timeout-minutes: 35
+        timeout-minutes: 50
         run: |
           cd ./client
           mv ./cypress.env.example.json ./cypress.env.json
@@ -180,9 +197,9 @@ jobs:
           replace-in-files --regex='"hosturl".*' --replacement='hosturl:"http://localhost:4200",' ./cypress.config.ts
           cat ./cypress.config.ts
           # Start frontend and run cypress
-          # timeout 1800: SIGTERM after 30 min so Cypress can flush video/screenshots
+          # timeout 2700: SIGTERM after 45 min so Cypress can flush video/screenshots
           # before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
-          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/hook-programs.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/filter-panel.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts,cypress/e2e/browser-info.cy.ts,cypress/e2e/ddl-export.cy.ts,cypress/e2e/row-cell-limits.cy.ts"
+          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/hook-programs.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/filter-panel.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts,cypress/e2e/browser-info.cy.ts,cypress/e2e/ddl-export.cy.ts,cypress/e2e/row-cell-limits.cy.ts,cypress/e2e/derived-variables.cy.ts,cypress/e2e/startupservice-launcher.cy.ts"
 
       - name: Zip Cypress videos
         if: always()

.gitea/workflows/lighthouse.yaml

@@ -33,6 +33,7 @@ jobs:
           echo NODE_PATH=node >> .env
           echo CORS=enable >> .env
           echo WHITELIST=http://localhost:4200 >> .env
+          echo HTTP_PORT=5000 >> .env
           # The server refuses to run as root unless RUN_AS names an account
           # (a security change in the recent releases). This job runs as root
           # in a throwaway container, so opt in explicitly.

.gitea/workflows/release.yaml

@@ -107,6 +107,16 @@ jobs:
           echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
           npm ci
 
+      - name: Build frontend for web streaming
+        # The server-ci target streams client/dist as the 'clickme' web app, so
+        # the production build has to exist before the mocked services are
+        # deployed. Without it `sasjs cbd` reports
+        # "webSourcePath: .../client/dist ... doesn't exist", streams no
+        # frontend and the deploy step fails.
+        run: |
+          cd client
+          npm run build
+
       - name: Setup and start SASjs server
         run: |
           npm i -g pm2
@@ -117,6 +127,7 @@ jobs:
           echo NODE_PATH=node >> .env
           echo CORS=enable >> .env
           echo WHITELIST=http://localhost:4200 >> .env
+          echo HTTP_PORT=5000 >> .env
           # The server refuses to run as root unless RUN_AS names an account
           # (a security change in the recent releases). This job runs as root
           # in a throwaway container, so opt in explicitly.
@@ -193,11 +204,15 @@ jobs:
     # and its permissions are derived from this block.
     #   contents: write -> push CHANGELOG.md / package.json commit + tag, and
     #                      create the Release via the Gitea API.
+    #   actions: write  -> dispatch the Sites workflow for the new tag (that
+    #                      route is scoped to the actions unit, which contents
+    #                      does not grant).
     # The TSDOC_TOKEN and CODE_DATACONTROLLER_IO secrets continue to be used
     # because they target *different* servers (webdoc.datacontroller.io and
     # code.datacontroller.io), not the Gitea API.
     permissions:
       contents: write
+      actions: write
 
     steps:
       - uses: actions/checkout@v4
@@ -380,8 +395,12 @@ jobs:
           # ahead of the installation footer. The section text lives in a
           # dedicated script below (.gitea/scripts/verify-section.sh) so this
           # YAML scalar stays clean.
+          # `jq -r` is required: without it jq emits the result as a JSON string
+          # literal (quoted, with \n escapes), and the PATCH below then encodes
+          # that literal a second time, so the release body renders as one
+          # escaped line instead of markdown.
           VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
-          NEW_BODY=$(jq -n --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
+          NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
             '$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
           curl -k --fail-with-body -sS -X PATCH \
             -H "$AUTH_HEADER" \
@@ -397,3 +416,27 @@ jobs:
           done
           echo "Uploading SHA256SUMS ..."
           curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
+
+      - name: Publish the demo site for this release
+        # The tag is created by this job's semantic-release step, so it carries
+        # the Actions user as its author - and the forge suppresses ref events
+        # from that user, which means `on: push: tags` in sites.yaml never fires.
+        # Dispatch it instead: the API call is authenticated, and the run it
+        # starts checks out the tag the same way.
+        env:
+          GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
+        run: |
+          set -euo pipefail
+          AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
+          BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
+
+          TAG=$(curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest" | jq -r '.tag_name')
+          echo "Dispatching Sites for ${TAG}"
+
+          # the dispatch endpoint resolves a bare name as a branch; a tag needs
+          # the refs/tags/ prefix
+          curl -k --fail-with-body -sS -X POST \
+            -H "$AUTH_HEADER" \
+            -H 'Content-Type: application/json' \
+            --data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
+            "$BASE/actions/workflows/sites.yaml/dispatches"

.gitea/workflows/sites.yaml

@@ -0,0 +1,183 @@
+name: Sites
+on:
+  push:
+    branches:
+      - 'release/**'
+    tags:
+      - 'v*'
+    paths-ignore:
+      - '**.md'
+  workflow_dispatch:
+
+env:
+  NODE_VERSION: '24.15.0'
+  APP_LOC: /Public/app/dc
+  SURFER_HOST: https://demo.datacontroller.io
+  # A release branch publishes the staging site (a named site on the instance,
+  # served at its alias domain); a release tag publishes the demo, which is the
+  # default site and so needs no site name.
+  STAGING_SITE: staging
+  STAGING_DOMAIN: staging.datacontroller.io
+
+# One deploy per target site: a newer push supersedes the one in flight, and two
+# release branches (which share the staging site) cannot interleave their swaps.
+# The demo and staging deploys are independent, so they do not queue.
+concurrency:
+  group: surfer-${{ startsWith(gitea.ref, 'refs/tags/') && 'demo' || 'staging' }}
+  cancel-in-progress: true
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # The tag is what the client's build stamp describes (src/version.ts
+          # runs git-describe), so the demo build needs the tags in the clone.
+          fetch-depth: 0
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - name: Write .npmrc file
+        run: |
+          touch client/.npmrc
+          echo '${{ secrets.NPMRC}}' > client/.npmrc
+
+      - name: Install client dependencies
+        run: |
+          cd client
+          # Decrypt and Install sheet
+          echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
+          npm ci
+
+      - name: Build frontend
+        # The site serves this build. `sasjs build` rewrites its index.html to
+        # point the adapter at the in-browser backend.
+        run: |
+          cd client
+          npm run postinstall
+          npm run build
+
+      - name: Install the SASjs CLI
+        # Pinned in sas/package.json - the browser build needs CLI >= 6.1.0
+        run: |
+          cd sas
+          npm ci
+
+      - name: Start SASjs server
+        # Needed only to run makedata: the drive the site ships with is captured
+        # from a real server (see the snapshot step below).
+        run: |
+          curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
+          unzip -q linux.zip
+          touch .env
+          echo RUN_TIMES=js >> .env
+          echo NODE_PATH=node >> .env
+          echo HTTP_PORT=5000 >> .env
+          # The server refuses to run as root unless RUN_AS names an account
+          echo RUN_AS=root >> .env
+          nohup ./api-linux > server.log 2>&1 &
+          for i in $(seq 1 60); do
+            if curl -sf -o /dev/null http://localhost:5000/; then
+              echo "SASjs server answered on attempt ${i}"
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "SASjs server did not come up within 120s"
+          cat server.log
+          exit 1
+
+      - name: Deploy the mocked services and seed the database
+        run: |
+          cd sas/mocks/sasjs
+          # Start from a clean appLoc so the snapshot below is this build alone,
+          # not a merge with whatever a previous run left on the drive.
+          curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=$APP_LOC"
+          ../../node_modules/.bin/sasjs cbd -t server-ci
+          # makedata replies with HTML (it is normally called as a URL redirect),
+          # which the CLI reports as "invalid Json string" - that exit is
+          # expected. A real failure must not be published as an empty site, so
+          # the drive is checked before the snapshot is taken.
+          ../../node_modules/.bin/sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
+
+          DATA_DIR="../../../sasjs_root/drive/files$APP_LOC/data"
+          SEEDED=$(find "$DATA_DIR" -name '*.json' 2>/dev/null | wc -l)
+          echo "makedata seeded $SEEDED table(s) under $DATA_DIR"
+          if [ -z "$(find "$DATA_DIR" -name 'mpe_tables.json' 2>/dev/null | head -1)" ] || [ "$SEEDED" -lt 10 ]; then
+            echo "makedata did not seed the drive - refusing to publish a data-less site"
+            find ../../../sasjs_root/drive/files -maxdepth 3 2>/dev/null | head -20
+            exit 1
+          fi
+
+      - name: Build the browser-hosted site
+        run: |
+          # The published sites ship a working licence so they are usable beyond
+          # the free tier. The pair is committed in sas/mocks/mock-licence.json
+          # rather than minted here, so every deploy of both sites carries the
+          # same key; the CI estate stays unlicensed because that file is outside
+          # the deployed sync folder, so the specs still exercise the free tier.
+          # The pair must round trip through the same WebCrypto the app uses,
+          # because a committed pair can drift (one half regenerated, a value
+          # truncated on copy) and nothing else would notice before publishing.
+          # Identical values in the two fields is the HTTP-format marker the app
+          # refuses in a secure context, so refuse to publish one.
+          node -e "
+            const d = require('./sas/mocks/mock-licence.json')
+            const b = (s) => Buffer.from(String(s || ''), 'base64')
+            if (!d.licenceKey || !d.activationKey || d.licenceKey === d.activationKey) {
+              console.error('sas/mocks/mock-licence.json must hold an HTTPS-format key pair')
+              process.exit(1)
+            }
+            crypto.subtle
+              .importKey('pkcs8', b(d.activationKey), { name: 'RSA-OAEP', hash: 'SHA-256' }, false, ['decrypt'])
+              .then((key) => crypto.subtle.decrypt({ name: 'RSA-OAEP', hash: 'SHA-256' }, key, b(d.licenceKey)))
+              .then((plain) => {
+                const data = JSON.parse(new TextDecoder().decode(plain))
+                console.log('licence pair round trips: site', data.site_id, 'until', data.valid_until)
+              })
+              .catch((e) => {
+                console.error('sas/mocks/mock-licence.json does not decrypt:', e.message || e)
+                process.exit(1)
+              })
+          "
+          LICENCE_DIR="sasjs_root/drive/files$APP_LOC/mock-storage"
+          mkdir -p "$LICENCE_DIR"
+          cp sas/mocks/mock-licence.json "$LICENCE_DIR/licence.json"
+
+          # DC's data files exist only after makedata has run, so the drive is
+          # captured first and shipped as the site's starting drive. services/web9
+          # is the streamed frontend, which is already the site itself.
+          node sas/mocks/snapshot.js sasjs_root/drive "$APP_LOC" sas/mocks/sasjsbuild/mock-snapshot.json services/web9
+          cd sas/mocks/sasjs
+          ../../node_modules/.bin/sasjs b -t server-ci
+
+      - name: Publish to Surfer
+        env:
+          SURFER_USER: allan
+          SURFER_PASSWORD: ${{ secrets.DEMO_DATACONTROLLER_IO }}
+          REF: ${{ gitea.ref }}
+          MESSAGE: Build from ${{ gitea.ref_name }} @ ${{ gitea.sha }}
+        run: |
+          # `deploy` uploads the folder as an archive and swaps the site in one
+          # step, so a failed upload leaves the current site serving.
+          if [ -z "$REF" ]; then
+            echo "REF is empty - cannot tell a branch push from a tag push"
+            exit 1
+          fi
+
+          SITE_ARGS=()
+          case "$REF" in
+            refs/tags/*) ;;
+            *) SITE_ARGS=(--site-name "$STAGING_SITE" --domain "$STAGING_DOMAIN") ;;
+          esac
+
+          npx --yes @cloudron/surfer@7.2.0 deploy \
+            --server "$SURFER_HOST" \
+            --username "$SURFER_USER" \
+            --password "$SURFER_PASSWORD" \
+            --message "$MESSAGE" \
+            "${SITE_ARGS[@]}" \
+            sas/mocks/sasjsbuild/mock

.gitleaksignore

@@ -0,0 +1,5 @@
+# sas/mocks/mock-licence.json holds a licence key pair for the mock estate: the
+# activation key is RSA key material the browser demo decrypts with, not a
+# credential for any service.
+sas/mocks/mock-licence.json:generic-api-key:2
+sas/mocks/mock-licence.json:generic-api-key:3

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+# Hooks for the pre-commit framework - https://pre-commit.com
+#
+# The repo also ships native hooks in .git-hooks/, covering the secret scan, a
+# 2MB commit size limit, a prettier check and Conventional Commits. But .npmrc
+# sets ignore-scripts=true, so the `prepare` script that points git at them
+# never runs on `npm i` - a fresh clone commits unchecked until the one-time
+# command in CONTRIBUTING.md is run by hand.
+#
+# This file is the standard route instead: install the framework once with
+# `pre-commit install`, and the secret scan runs on every commit. The hook is
+# pinned to an exact gitleaks release tag.
+repos:
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.30.1
+    hooks:
+      - id: gitleaks

AGENTS.md

@@ -32,6 +32,8 @@ Never consider a change complete until the relevant linters pass on the files yo
 
 - **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
 - **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
+  `sas/.sasjslint` carries `ignoredMacros: ["webout", "mpeinit", "mpeterm"]`. `webout` is the linter's own default - `sasjs compile` injects `%webout` into every service - and an explicit list replaces that default, so it has to be restated there. `mpeinit` and `mpeterm` are listed because every service is compiled with a serviceInit program (`sasjs/utils/serviceinitsas9|viya|server.sas`) that declares both macros in its own header, so a service body may call them without declaring them. Add a macro to that list only when it reaches a file the same way - injected by the compiler, through serviceInit, testSetUp or another implicit provider - never to silence a genuinely missing declaration.
+  `noBannedStrings` flags the literal words `error ` and `warning ` anywhere in a file, comments included. Reword prose; where the text has to stay (a real `%put`, a doxygen `@warning` tag), keep the log output identical with `%put %str(WARN)ING:` rather than a per-file override, and use `@sasjslint {"noBannedStrings": false}` only when the word itself is the content.
 
 ## No external assets
 

CHANGELOG.md

@@ -1,3 +1,177 @@
+# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
+
+
+### Bug Fixes
+
+* address the review findings on the harness and refresh changes ([36e4e88](https://git.datacontroller.io/dc/dc/commit/36e4e88eb134d3b48217d6dd185ea3c03f03fba5))
+* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](https://git.datacontroller.io/dc/dc/commit/698dea6d6b7a8635313f11acfcc7a4f39a02bc99))
+* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](https://git.datacontroller.io/dc/dc/commit/0d58be975d8eb2d7f1ab6a648f760151392c7973))
+* **client:** open the SAS log page with noopener ([072a133](https://git.datacontroller.io/dc/dc/commit/072a1337020970b5b4ff8aa31719cc5ffec7701e))
+* **client:** refuse a truncated table in the multi-table upload ([1fb4818](https://git.datacontroller.io/dc/dc/commit/1fb4818d215f538a0ac51772ab1f261aeb91f7e1))
+* **cypress:** apply the licence key when the estate has none ([be8f445](https://git.datacontroller.io/dc/dc/commit/be8f445ae1051e132fb6eb5a69b45264255326b5)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
+* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](https://git.datacontroller.io/dc/dc/commit/fed43cb07ef7f2afc685612a124dbc16d7618e21))
+* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](https://git.datacontroller.io/dc/dc/commit/f5aade59760cc376b0f96eb4c892c8bca82dde14))
+* **dc_refreshcatalog:** terminate the generated statements ([1e22476](https://git.datacontroller.io/dc/dc/commit/1e224763ba80500a112812a983801968e3751959))
+* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](https://git.datacontroller.io/dc/dc/commit/9b3cf50b9c3889786605f0f5d9032d84c50f7107))
+* **derived:** address five review findings ([ccd468b](https://git.datacontroller.io/dc/dc/commit/ccd468b3358c1418ee6d8e198b088a0cf23b8c24))
+* **derived:** carry the placement sequence in mpe_derived ([be50cb5](https://git.datacontroller.io/dc/dc/commit/be50cb5d082e788c6ae106bc1d02e73a8c7b9646))
+* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](https://git.datacontroller.io/dc/dc/commit/9914790777f92128c74ecce67e6c67ee48824b45))
+* **derived:** close the RLS and search gaps on derived columns ([9ef813d](https://git.datacontroller.io/dc/dc/commit/9ef813d439b48e7fefd35c426e62b99e5943002a))
+* **derived:** close the two gaps in the derived/RLS invariant ([422f766](https://git.datacontroller.io/dc/dc/commit/422f766cd43db4ce740e1890fd5df30790eaa7e3))
+* **derived:** compare the outref as text, prove the file is empty ([7830b7b](https://git.datacontroller.io/dc/dc/commit/7830b7bcfa9806dde840ce8dd3e6a1887d0b38be))
+* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](https://git.datacontroller.io/dc/dc/commit/2d5de9a324a07a36edc0a4f74c8fad09fde07cbf))
+* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](https://git.datacontroller.io/dc/dc/commit/6a1128a46352f77c95dd02df9309f0984a1db2d1))
+* **derived:** eight review findings ([561d7fc](https://git.datacontroller.io/dc/dc/commit/561d7fc3ed2a460835619962cf4dcb29d3dc87c7))
+* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](https://git.datacontroller.io/dc/dc/commit/1863076bc956b4ff94bf7b85b3dde0d6e3b54d7a)), closes [#366](https://git.datacontroller.io/dc/dc/issues/366)
+* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](https://git.datacontroller.io/dc/dc/commit/d3dec682aafeb143fe983d6546337cb6490f5620))
+* **derived:** filtering a format catalog no longer aborts the service ([469229a](https://git.datacontroller.io/dc/dc/commit/469229aecbaad304a835a5f229df111c37ff7c0d))
+* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](https://git.datacontroller.io/dc/dc/commit/7fb981daa91bce29c2d62c246185d01dcad08d6b))
+* **derived:** guard the empty physical column list in the dep check ([5f5d868](https://git.datacontroller.io/dc/dc/commit/5f5d8681b9d78280a5200309917415e135c3ce5b))
+* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](https://git.datacontroller.io/dc/dc/commit/5f443b0b16a2abd3c11d96caa7a043911c29ef01))
+* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](https://git.datacontroller.io/dc/dc/commit/6b5bf3e6be89b22e5a633c34b9aecbc8c7d48ee2))
+* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](https://git.datacontroller.io/dc/dc/commit/85c89ca2a4e02112ba496ac36a222f33675544da))
+* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](https://git.datacontroller.io/dc/dc/commit/f8f65681e1489f4ce94d9127d5e4c0c82798e960))
+* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](https://git.datacontroller.io/dc/dc/commit/7efbf31ea5ac234bd8f924fdf89a95d101b3aff1))
+* **derived:** only the delete marker is a reserved name ([af31c36](https://git.datacontroller.io/dc/dc/commit/af31c36b670ec63a2a4ba056bf6c4b28f6096e08))
+* **derived:** open the statements file with a long record length ([ee31782](https://git.datacontroller.io/dc/dc/commit/ee31782fbff909b540162b1bdfca6f307208331b))
+* **derived:** pair the filterds %else with %do/%end ([88a38db](https://git.datacontroller.io/dc/dc/commit/88a38db877dba9b98ccf50f0cc7242674654586b))
+* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](https://git.datacontroller.io/dc/dc/commit/27ddcfb5759d6eb8f9880adc7c271a408e7741ad))
+* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](https://git.datacontroller.io/dc/dc/commit/7799a300df82339683eea21e4a89ce7a6471c575))
+* **derived:** reject a dependency on a row this submission deletes ([8806504](https://git.datacontroller.io/dc/dc/commit/88065049674c2cc0141eef22704b759f5b708b0f))
+* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](https://git.datacontroller.io/dc/dc/commit/9a047ad42ec86df7d1eaf5058c0edde56016e88f))
+* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](https://git.datacontroller.io/dc/dc/commit/e0da9f1098db63185538398ca478696cc849da4b))
+* **derived:** stop the hooks aborting on their own macro variables ([688550a](https://git.datacontroller.io/dc/dc/commit/688550a28d9794ac6b878926a0e1e2ef8f6e9335))
+* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](https://git.datacontroller.io/dc/dc/commit/1b5c4b3ecd962a28e08fc5671b57dbf7ae0b8d9f))
+* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](https://git.datacontroller.io/dc/dc/commit/4d9ad38df1fa89b0d5a12214d703d9c1fe5295f2))
+* **editor:** a blank truncation note is not a warning ([f182179](https://git.datacontroller.io/dc/dc/commit/f18217940365f27ae388fae087e6126931e1222c))
+* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](https://git.datacontroller.io/dc/dc/commit/04ce1861e7821063b1be42c22f92f62aed509602))
+* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](https://git.datacontroller.io/dc/dc/commit/4bcee706ffaf1929bdd14bb2fb15111174ca3916))
+* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](https://git.datacontroller.io/dc/dc/commit/45dce7b11b7a4399c1d0dc9c3e7a283b1cf59df1))
+* **getdata:** clear the globals it declares before use ([359b570](https://git.datacontroller.io/dc/dc/commit/359b570b85aa2f66507e81ea2977243be4590f83))
+* **getddl.test:** ask for a table, not a whole library ([059cdf1](https://git.datacontroller.io/dc/dc/commit/059cdf176a7ec556b45ea1b21d880c048eba14ca))
+* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](https://git.datacontroller.io/dc/dc/commit/661ee19cfa5f996bf01f2b11817039f38dbb9985))
+* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](https://git.datacontroller.io/dc/dc/commit/355c46a7b64994d0146a88657dc5635593249138))
+* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](https://git.datacontroller.io/dc/dc/commit/d626cb8f2daf94fde7ff21d924b9a2ca4968d983))
+* **mocks:** address the audit-pack review findings ([489819b](https://git.datacontroller.io/dc/dc/commit/489819b05111323ae1e34edda22a5446cdbc2d87))
+* **mocks:** carry the zip flags into the central directory ([c4021a4](https://git.datacontroller.io/dc/dc/commit/c4021a42ffaed2af80c53aaa1455249190b339b8))
+* **mocks:** clear the licence for published sites, not for the estate ([62f55db](https://git.datacontroller.io/dc/dc/commit/62f55db170185900063f7a431cd8f375ebabf930))
+* **mocks:** copy the physical column list before deriving ([f411b7d](https://git.datacontroller.io/dc/dc/commit/f411b7d269381e3f7cab2d85b6d8f7d7d50ea4d1))
+* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](https://git.datacontroller.io/dc/dc/commit/60ec470404e5c279b58ac6a01768dfc83413b197))
+* **mocks:** derive the retained key wherever rows are staged ([74060c0](https://git.datacontroller.io/dc/dc/commit/74060c0585fb8b217cb181ea747ad318f5238868))
+* **mocks:** drop derived variables before staging ([73d25b1](https://git.datacontroller.io/dc/dc/commit/73d25b1220d574ef2547fbd9e6e3cf1ceda375ee))
+* **mocks:** key rk_underlying tables on their business key ([347dbb5](https://git.datacontroller.io/dc/dc/commit/347dbb53b01c47bbd562de14d5ccbae7d96407c6))
+* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](https://git.datacontroller.io/dc/dc/commit/ee3c8787efef9cd716bdd2bf80f6cb6751461db9))
+* **mocks:** scope the derived hook's graph to the submission ([6cad23f](https://git.datacontroller.io/dc/dc/commit/6cad23fce9ac0cf64c247c40f3b7eac875fd4cfb))
+* **mocks:** search the physical columns only in the viewer ([293022d](https://git.datacontroller.io/dc/dc/commit/293022d46b27e096fd514cdda49f688612df2f5b))
+* **mocks:** stop the mock advertising invalid licence keys ([fb36379](https://git.datacontroller.io/dc/dc/commit/fb36379520028baaf4bc3f08131bfc88deab3ac3))
+* **mpe_assert_abort:** drop the unused line variable ([9c294c0](https://git.datacontroller.io/dc/dc/commit/9c294c0ab5ca8fe30a235553a298c1c68a61442f))
+* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](https://git.datacontroller.io/dc/dc/commit/0a52d20c5fc27b4adde5fc8ed6f39a58c48f62ff))
+* **mpe_execute:** fail fast when the child left no status ([a036386](https://git.datacontroller.io/dc/dc/commit/a03638667f1968c5909f496d954eeaa13f316763))
+* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](https://git.datacontroller.io/dc/dc/commit/220408538b26016bda45e48d619cdbc3e1cbcc59))
+* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](https://git.datacontroller.io/dc/dc/commit/f2b1428ec3099b2b6e3c6da6627530ef81fab201))
+* name the startupservice launcher button "Show log" ([3445107](https://git.datacontroller.io/dc/dc/commit/3445107d9a91642b67e114bcfcd81c3ce47f3f0f))
+* offer the launcher on the globvars failure modal too ([ad2da45](https://git.datacontroller.io/dc/dc/commit/ad2da453896a51f2cc7e49cebb6c84f7b47d7395))
+* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](https://git.datacontroller.io/dc/dc/commit/46bacdd6d9ea814bcd02a1ff8aa3f447baf02f64))
+* **refresh:** one audit folder per refresh call ([56b6620](https://git.datacontroller.io/dc/dc/commit/56b6620ef402b288ce78a2a77662b651e310753e))
+* **sites:** round-trip the committed licence before publishing ([4d462d3](https://git.datacontroller.io/dc/dc/commit/4d462d310f7df23e5d5229cb0702721b7ad41bbd))
+* **test:** getdata test ([da30266](https://git.datacontroller.io/dc/dc/commit/da30266278c4fe27a72810cd5ed647cf3bd823af))
+* **tests:** enable closeouts logging ([fdcd8fa](https://git.datacontroller.io/dc/dc/commit/fdcd8fa4b3e7a7a119c662974d70dc9eea80bdae))
+* **tests:** read the work location from _SASJS_WORK ([033ffd3](https://git.datacontroller.io/dc/dc/commit/033ffd302502c7530fc44a314364f5cdb3be6ad2))
+* **ui:** one error modal at a time ([81ccf75](https://git.datacontroller.io/dc/dc/commit/81ccf758e9dab0c7632ab50bf469e88b8c4c4420))
+* **validatefilter:** build targetds without macro triggers in comments ([42355f1](https://git.datacontroller.io/dc/dc/commit/42355f17bc07bf4d44ba858984d83b9ed72cc0ba))
+* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](https://git.datacontroller.io/dc/dc/commit/337ea8e5e021c6f2451fa3ad8396148744118fc9))
+
+
+### Features
+
+* **dc_getlibs:** dump the library list when debug is on ([582523d](https://git.datacontroller.io/dc/dc/commit/582523dc87a253b71824359224070ccb4746a386))
+* **derived:** allow a variable to shadow a physical column ([12beeff](https://git.datacontroller.io/dc/dc/commit/12beeff69518376a042fbbc58332acc40d89d132))
+* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](https://git.datacontroller.io/dc/dc/commit/813bce685691ad252043eeab31eb5460de81b756))
+* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](https://git.datacontroller.io/dc/dc/commit/7c9ce9a712137e9825a5f2ff52796d765423147b))
+* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](https://git.datacontroller.io/dc/dc/commit/87e794f769fe420f214f43b141f0b41cf88eefee))
+* **derived:** report the dependency cycle path ([cee1a95](https://git.datacontroller.io/dc/dc/commit/cee1a954ee97e17f297b7cf43ef0e5cfc6e393a6))
+* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](https://git.datacontroller.io/dc/dc/commit/78e88a41128b047c0276ccf647f128229152604c))
+* **derived:** validate MPE_DERIVED at submit time ([0b010f2](https://git.datacontroller.io/dc/dc/commit/0b010f2c65ddada70d62ea83c706943c6c45756f))
+* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](https://git.datacontroller.io/dc/dc/commit/aa6d50e1ea5fc2fe6be8aa9fa859b92d0a1061c5))
+* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](https://git.datacontroller.io/dc/dc/commit/33d043ea92f007dca70c95d5363329d4447eeba9))
+* **editor:** derived variables in the EDIT screen ([b0fd354](https://git.datacontroller.io/dc/dc/commit/b0fd354ecdbd7bf29196ed983ff3cce691affeed))
+* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](https://git.datacontroller.io/dc/dc/commit/6513d09f059de15c5304f99b3f8d898d46ea1be7))
+* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](https://git.datacontroller.io/dc/dc/commit/34691dceecb63f952b23ad8acc5bc93753c6cfee))
+* **editor:** show the derived column label in the header tooltip ([cfe6647](https://git.datacontroller.io/dc/dc/commit/cfe6647ca870c45a3d8abc2753ce7f070e3fdf4a))
+* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](https://git.datacontroller.io/dc/dc/commit/752d944c2091ef6d2624a15ab2e2e7be35608798))
+* **mocks:** implement derived variables in the JS services ([b4fae88](https://git.datacontroller.io/dc/dc/commit/b4fae88ae13658607e4209449577cd42849d3a99))
+* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](https://git.datacontroller.io/dc/dc/commit/5d183d135085ea023f32e5b1d4992d997394a8fb))
+* run startupservice in a new window from its error modal ([40f6133](https://git.datacontroller.io/dc/dc/commit/40f61331aad6bc8a52c551b924f0d1f1e6426cee))
+* **sites:** mint a licence for the published sites ([7d9df71](https://git.datacontroller.io/dc/dc/commit/7d9df71fdb0ab265e745ed3b708038983e855a87))
+
+
+### Reverts
+
+* **derived:** no shadowing, and no self reference ([5af8e6f](https://git.datacontroller.io/dc/dc/commit/5af8e6fc359221586af74a685159dfd75a40ec45))
+* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](https://git.datacontroller.io/dc/dc/commit/31aa8591538409fc1a3ef97d7474120094ce202d))
+
+# [7.18.0](https://git.datacontroller.io/dc/dc/compare/v7.17.0...v7.18.0) (2026-10-19)
+
+
+### Bug Fixes
+
+* **deps:** bump @sasjs/core to 5.4.0 ([19fb032](https://git.datacontroller.io/dc/dc/commit/19fb032f0dbdd612651ca4ff3b55d1aefa4850d5))
+* **deps:** pin @sasjs/utils and drop the unused jsrsasign ([05d2d1e](https://git.datacontroller.io/dc/dc/commit/05d2d1e08e2c3082f0dd5aa4629d91907c0401fc))
+* **editor:** own the delete-record column client-side ([56025cc](https://git.datacontroller.io/dc/dc/commit/56025cc0524d34d3589992e9f1059c13fb5581ed))
+* **editor:** surface the reserved-column refusal on every getdata path ([b26f557](https://git.datacontroller.io/dc/dc/commit/b26f5571b66b65166c1158817f2df3027ae45cdc))
+* **filters:** do not turn a fractional temporal value into a literal ([f710d87](https://git.datacontroller.io/dc/dc/commit/f710d871fffc0fa451fd8cc4d037a26414725654))
+* **filters:** fix the dynamic where clause deref and map its temporal values ([ece67d3](https://git.datacontroller.io/dc/dc/commit/ece67d37ba2c4e2037dea626b9730a692388f025))
+* **filters:** keep the range field readable and cover a BETWEEN in the rewrite ([be79f3a](https://git.datacontroller.io/dc/dc/commit/be79f3a1ea5e4c4a157396d276f2c6654f615d39))
+* **filters:** map an IN entry to its unformatted value before the literal ([8f98ebe](https://git.datacontroller.io/dc/dc/commit/8f98ebe326978b8da58d8ac895ec946e562b5de4))
+* **filters:** stop the temporal rewrite corrupting a typed value, and bind the range dropdown to the unformatted value ([a5f61d4](https://git.datacontroller.io/dc/dc/commit/a5f61d4a73051237b671695f67122c237770c95a))
+* **filters:** use the unformatted value in every IN branch ([cc01f54](https://git.datacontroller.io/dc/dc/commit/cc01f540c29d3d5f7dd86943b6eed82b332ce868))
+* **makedata:** shorten the DC_MAXCELLS_WEBEDIT description to fit var_desc ([a51d9e5](https://git.datacontroller.io/dc/dc/commit/a51d9e5574383e5d1e1943c95261fbce63e8788d))
+* **multi-dataset:** surface the reserved-column guard, use the name constant ([2b8a2fa](https://git.datacontroller.io/dc/dc/commit/2b8a2fa1039da3ab888af8c3ffae7e1b848b250a))
+* **release:** emit the release body as raw text, not a JSON string ([a05554e](https://git.datacontroller.io/dc/dc/commit/a05554e67a8d44e32e2d9f5e4a959a9d0592be5e))
+
+
+### Features
+
+* **filters:** show dates as date literals in the filter query ([662d000](https://git.datacontroller.io/dc/dc/commit/662d00032c27634bbc2d82c392e7875a2e174eff))
+
+# [7.17.0](https://git.datacontroller.io/dc/dc/compare/v7.16.0...v7.17.0) (2026-10-15)
+
+
+### Bug Fixes
+
+* **client:** do not publish an empty getcols fallback to the picker ([9f8fc6f](https://git.datacontroller.io/dc/dc/commit/9f8fc6f2eb83c78ea9cb220bb74059bab39e66ad))
+* **client:** map the getcols fallback onto the picker vocabulary ([7cecf79](https://git.datacontroller.io/dc/dc/commit/7cecf795153138d6f2ad1f87f7bdd12b5d861341))
+* **client:** pin the dependency entries this release changes ([b034fe5](https://git.datacontroller.io/dc/dc/commit/b034fe5335c41ec0289e459270839ee7caa8e126))
+* **deploy:** apply the same identity-call fix to the automatic deploy ([5693b44](https://git.datacontroller.io/dc/dc/commit/5693b4488b0328022c69dd34f98662d660612578))
+* **deploy:** disable the manual deploy actions without an admin group ([6ffbdba](https://git.datacontroller.io/dc/dc/commit/6ffbdbadf9804e0128694ed1c2bdb8f957743ad9))
+* **deploy:** disable the manual deploy file inputs while admin groups load ([7094bfd](https://git.datacontroller.io/dc/dc/commit/7094bfdf7cdab4c70e5f1a7763a8a8aeb8cc61a7))
+* **deploy:** guard the manual deploy form and Viya identity calls ([6522fcb](https://git.datacontroller.io/dc/dc/commit/6522fcb28b67b910b2784b93251f591bbc79d294))
+* **deploy:** never bake an empty admin group into the deploy script ([b0e47fc](https://git.datacontroller.io/dc/dc/commit/b0e47fc4ed4e36878de6505e678667b14d19c5e6))
+* **deps:** pin shelljs to clear the braces advisory in the sas audit ([6e50938](https://git.datacontroller.io/dc/dc/commit/6e5093858be106a1c3545a16119d246f69aac3da))
+* **editor:** check the EDIT limits once, and skip switched-off options ([f08c4c2](https://git.datacontroller.io/dc/dc/commit/f08c4c231f019979a27fe71fe73798752c015710))
+* **filters:** carry TYPE on the fallback list, and drop the temporal exclusion ([bcb8fde](https://git.datacontroller.io/dc/dc/commit/bcb8fde9868b8e1ded1590e204c34054f62d8250))
+* **filters:** source the picker variables from public/getcols when the table is not returned ([3a6a216](https://git.datacontroller.io/dc/dc/commit/3a6a216b9d4dfc462e05031039c47002a9156130))
+* **getcols:** use %global and %mp_getcols, and merge the base branch ([1385741](https://git.datacontroller.io/dc/dc/commit/1385741f2d303c1432f1f0ba637dd3cb39500a71))
+* **getcols:** widen TYPE before assigning the client vocabulary ([bcad05e](https://git.datacontroller.io/dc/dc/commit/bcad05e03291fc3e2b72a4465629e1e330d70c8a))
+* **getdata:** apply row level security to selectbox source values ([768d33c](https://git.datacontroller.io/dc/dc/commit/768d33c618a7ea453c3c19723f342a170ec576ea)), closes [#340](https://git.datacontroller.io/dc/dc/issues/340)
+* **mocks:** read MPE_CONFIG correctly, and pin the limits to it ([05126ae](https://git.datacontroller.io/dc/dc/commit/05126ae73a1ab2461066eb701b7475674b96f1ec))
+* repair the SAS suite for the row/cell limits and default the manual deploy admin group ([99ff732](https://git.datacontroller.io/dc/dc/commit/99ff732d67425b01a8a8f5e3e7c98b3e6d604323))
+* **requests-modal:** widen the dialog so log lines need less scrolling ([3c10d91](https://git.datacontroller.io/dc/dc/commit/3c10d915a5134247aff0e987dd4d5c83c197e4c5))
+* **sas:** keep format and label in the public getcols response ([9c32b1b](https://git.datacontroller.io/dc/dc/commit/9c32b1b1dae00e4b7bfd4acad94cbed831819d84))
+* **sas:** keep the SAS C/N vocabulary in the public getcols response ([e90bcde](https://git.datacontroller.io/dc/dc/commit/e90bcde51218c9e5f562f7aaa65506761b21d6bd))
+* **sas:** run the row/cell limit migration as a Viya job ([90af6b8](https://git.datacontroller.io/dc/dc/commit/90af6b851542a97f79172b655070cda559386a5c))
+* **startup:** stack the SASjs tag detail so the dump is readable ([1f4c4a4](https://git.datacontroller.io/dc/dc/commit/1f4c4a4d8297568f46bcc16ea208673fedad345b))
+
+
+### Features
+
+* **debug:** keep the debug toggle state across a page refresh ([3ce006d](https://git.datacontroller.io/dc/dc/commit/3ce006d6b469b4ece2bc769f0a32a1792eb1e774))
+* **editor:** raise the row limits and add a cell limit to the EDIT screen ([3de6655](https://git.datacontroller.io/dc/dc/commit/3de6655f26b7e895f60266e9931b46cfc0053975))
+* **mocks:** read the DC_MAX* options from MPE_CONFIG, and mock the DDL export ([edac9a9](https://git.datacontroller.io/dc/dc/commit/edac9a987aad3e4217641eed43f697f61c4bf739))
+* **startup:** call out a missing SASjs tag on the boot screen ([b24eb73](https://git.datacontroller.io/dc/dc/commit/b24eb731ce089acf496bed1e11a5e34208c1bd11))
+* **startup:** dump the SASjs tag contents in the boot screen checks ([75c1dcc](https://git.datacontroller.io/dc/dc/commit/75c1dccfd23557ab321cd7abd1508f20d817c7b8))
+* **system:** export the DC library DDL from the system screen ([05f3f49](https://git.datacontroller.io/dc/dc/commit/05f3f49e0c4d7005fa1f93852d73f2368395ab28)), closes [#179](https://git.datacontroller.io/dc/dc/issues/179)
+
 # [7.16.0](https://git.datacontroller.io/dc/dc/compare/v7.15.0...v7.16.0) (2026-09-28)
 
 

CONTRIBUTING.md

@@ -19,6 +19,8 @@ git config core.hooksPath ./.git-hooks
 
 The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
 
+The standard [pre-commit](https://pre-commit.com) route is available as well, and does not depend on the lifecycle scripts above: install it once with `pre-commit install` and the secret scan in [`.pre-commit-config.yaml`](./.pre-commit-config.yaml) runs on every commit. The same scan also runs in CI, so a leaked credential fails the build regardless of which local hooks are active.
+
 ## Dependencies that requires licences
 
 [SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)
@@ -48,15 +50,12 @@ npm i
 
 [Handsontable](https://www.npmjs.com/package/handsontable)
 
-Licence should be inserted in the `client/src/index.html` file:
+The licence key comes from the backend licence: the decrypted licence carries `hot_license_key`,
+which is published on the `hot_license_key` subject and pushed into every Handsontable instance.
+Until a licence is loaded the client falls back to the non-commercial key
+(`DEFAULT_HOT_LICENCE_KEY` in `client/src/app/services/licence.service.ts`), so there is nothing to
+configure in `index.html` for this.
 
-```
- <sasjs
-    ...
-    hotLicenceKey="LICENCE-GOES-HERE"
->
-</sasjs>
-```
 ## Development
 Update `client/src/index.html` so that it points to your SAS9, SASVIYA or SASJS backend.  
 Be aware that VIYA can be configured in such way that it would not work with cross origin frontend.  
@@ -87,6 +86,7 @@ RUN_TIMES=js
 NODE_PATH=node
 CORS=enable
 WHITELIST=http://localhost:4200
+HTTP_PORT=5000
 ```
 
 `RUN_TIMES=js` means the server executes `.js` files as Stored Programs (no SAS binary needed). `CORS=enable` plus `WHITELIST` allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
@@ -95,7 +95,7 @@ WHITELIST=http://localhost:4200
 ./api-linux    # or api-macos / api-win.exe depending on your platform
 ```
 
-The server runs in desktop mode (no auth) on `http://localhost:5000` by default.
+`HTTP_PORT` is not optional in SASjs Server 3.x: `PORT` and `PROTOCOL` are deprecated, and a server started without `HTTP_PORT` or `HTTPS_PORT` exits with "no listener is enabled". With it set, the server runs in desktop mode (no auth) on `http://localhost:5000`.
 
 ### 2. Deploy the mock services and streaming frontend
 
@@ -134,6 +134,10 @@ There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`)
 
 After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sasjs/`. After changing the frontend, rebuild (`cd client && npm run build`) then re-deploy so the streaming app picks up the new `client/dist/`. If you are using `ng serve` (step 3), only the mock services need re-deploying; the frontend hot-reloads automatically.
 
+### Demo and staging sites
+
+A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs `makedata` on a throwaway SASjs Server first and ships that drive as `sasjsbuild/mock-snapshot.json`; without it the site opens with no data. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
+
 ## GUI Elements
 
 For documentation on the Clarity Design System, including a list of components and example usage, see [our website](https://vmware.github.io/clarity).

client/cypress/clips/special-missings-clip.cy.ts

@@ -40,8 +40,8 @@ context('special missings clip (DEMO_01)', function () {
   this.beforeEach(() => {
     cy.visit(hostUrl + appLocation)
 
-    // The mock estate carries a valid licence key (mock-storage/licence.json),
-    // so the app activates directly with no free-tier banner in shot.
+    // Assumes the estate is already activated (the licensing spec registers a
+    // valid key); otherwise a licensing page would sit in shot.
     cy.get('.nav-tree', { timeout: longerCommandTimeout }).should('exist')
   })
 

client/cypress/e2e/coltype-delete-record.cy.ts

@@ -25,11 +25,13 @@ const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
 //   2. Ordinary columns (e.g. SOME_DROPDOWN) still get their rule via the
 //      new per-row cols[].COLTYPE path — test 2 proves the migration didn't
 //      silently drop real columns' rules either.
-// Column positions below come from COLHEADERS in
-// sas/mocks/sasjs/services/editors/getdata.js:
-// "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,..."
-// — childNodes[0] on a body <tr> is the row-header <th>, so childNodes[N+1]
-// is the Nth (0-indexed) column in that list.
+// Column positions below come from the mock's COLHEADERS
+// (sas/mocks/sasjs/services/editors/getdata.js):
+// "PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,..."
+// — the client-owned delete column is prepended by the editor itself (see
+// deleteRecordColumn.ts), so on screen it is still column 0. childNodes[0] on
+// a body <tr> is the row-header <th>, so childNodes[N+1] is the Nth
+// (0-indexed) column in that list.
 context('coltype vertical-array regression tests: ', function () {
   this.beforeAll(() => {
     cy.visit(`${hostUrl}/SASLogon/logout`)

client/cypress/e2e/derived-variables.cy.ts

@@ -0,0 +1,249 @@
+export {}
+//
+// Derived (computed) variables are configured per table in MPE_DERIVED and
+// created at read time, ahead of the filter, so that they can be filtered on.
+//
+// TESTDATA.DERIVED_DEMO carries two of them, neither of which is a physical
+// column of the table:
+//   REGION      ifc(STATE in ('CA','OR','WA'),'West','East')
+//   LINE_VALUE  QTY * PRICE
+//
+// The mock implements the derivation in getdata.js / getcolvals.js (see
+// mpeDerived in dcMockUtils.js), so the EDIT screen can be exercised without a
+// SAS server. The real evaluation is done by SAS.
+
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+
+context('derived variables: ', function () {
+  this.beforeAll(() => {
+    cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
+    cy.loginAndUpdateValidKey(true)
+  })
+
+  this.beforeEach(() => {
+    cy.viewport(1440, 900)
+    bootApp()
+  })
+
+  it('1 | derived columns are shown in the EDIT grid with computed values', () => {
+    openEditor('TESTDATA.DERIVED_DEMO')
+
+    getCellByHeaderAndRow(0, 'REGION').should('have.text', 'West')
+    getCellByHeaderAndRow(1, 'REGION').should('have.text', 'East')
+    getCellByHeaderAndRow(0, 'LINE_VALUE').should('have.text', '21')
+    getCellByHeaderAndRow(1, 'LINE_VALUE').should('have.text', '99')
+    // REGION_LABEL is built on REGION, so it only resolves if the derived
+    // statements run in VARIABLE_ORDER sequence
+    getCellByHeaderAndRow(0, 'REGION_LABEL').should('have.text', 'West-CA')
+    getCellByHeaderAndRow(1, 'REGION_LABEL').should('have.text', 'East-NY')
+
+    cy.screenshot('derived-1-grid', { capture: 'viewport' })
+  })
+
+  it('2 | filters on a character derived column', () => {
+    openEditor('TESTDATA.DERIVED_DEMO')
+
+    openFilterPopup()
+    setFilterVariable('REGION')
+    setFilterValue('West')
+    submitFilter()
+
+    cy.get('.infoBar-text').should('contain.text', "REGION = 'West'")
+    cy.get('.ht_master .htCore tbody tr', {
+      timeout: longerCommandTimeout
+    }).should('have.length', 2)
+    getCellByHeaderAndRow(1, 'REGION').should('have.text', 'West')
+
+    cy.screenshot('derived-2-filter-region', { capture: 'viewport' })
+  })
+
+  it('3 | filters on a numeric derived column', () => {
+    openEditor('TESTDATA.DERIVED_DEMO')
+
+    openFilterPopup()
+    setFilterVariable('LINE_VALUE')
+    setFilterOperator('>')
+    setFilterValue('50')
+    submitFilter()
+
+    cy.get('.ht_master .htCore tbody tr', {
+      timeout: longerCommandTimeout
+    }).should('have.length', 1)
+    getCellByHeaderAndRow(0, 'LINE_VALUE').should('have.text', '99')
+
+    cy.screenshot('derived-3-filter-value', { capture: 'viewport' })
+  })
+
+  it('4 | a derived column is marked read-only, and opens no editor', () => {
+    openEditor('TESTDATA.DERIVED_DEMO')
+
+    // a derived column is read-only AND computed, so the grid marks it apart
+    // from an ordinary read-only column - its own header/cell class, no
+    // readonlyCell dimming (editor.component.ts, afterGetColHeader /
+    // afterGetCellMeta). STATE is the control: same grid, a physical column.
+    headerCell('REGION').should('have.class', 'dc-derived-header')
+    headerCell('REGION').should('not.have.class', 'readonlyCell')
+    headerCell('STATE')
+      .should('not.have.class', 'dc-derived-header')
+      .and('not.have.class', 'readonlyCell')
+    getCellByHeaderAndRow(0, 'REGION').should('have.class', 'dc-derived-cell')
+    getCellByHeaderAndRow(0, 'STATE').should(
+      'not.have.class',
+      'dc-derived-cell'
+    )
+    // The computed block opens at whichever derived column comes first in the
+    // grid - MPE_DERIVED order, so LINE_VALUE here, not REGION. Assert the
+    // invariant rather than a name: exactly one derived header carries the
+    // marker, and it is the first one.
+    cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
+      const derived = [...$ths].filter((th) =>
+        th.classList.contains('dc-derived-header')
+      )
+      expect(derived.length, 'derived headers').to.be.greaterThan(0)
+      expect(
+        derived[0].classList.contains('dc-derived-first'),
+        'the first derived header opens the block'
+      ).to.be.true
+      derived
+        .slice(1)
+        .forEach(
+          (th) =>
+            expect(
+              th.classList.contains('dc-derived-first'),
+              'only the first derived header carries the marker'
+            ).to.be.false
+        )
+    })
+
+    // the hover text carries the label, which is what says what the column is:
+    // the configured VAR_LABEL where there is one, otherwise the generated
+    // name=expression
+    headerCell('REGION_LABEL')
+      .should('have.attr', 'title')
+      .and('contain', 'Region and state')
+    headerCell('REGION')
+      .should('have.attr', 'title')
+      .and('contain', 'REGION=ifc')
+
+    enterEditMode()
+
+    getCellByHeaderAndRow(0, 'REGION').dblclick({ force: true })
+    cy.get('.handsontableInput').should('not.exist')
+    getCellByHeaderAndRow(0, 'REGION').should('have.text', 'West')
+  })
+
+  /**
+   * The positive control for the case above: same screen, same edit mode, a
+   * physical column - so its silence is about the column, not the screen.
+   */
+  it('5 | a physical column in the same grid does accept an edit', () => {
+    openEditor('TESTDATA.DERIVED_DEMO')
+    enterEditMode()
+
+    getCellByHeaderAndRow(0, 'STATE').dblclick({ force: true })
+    cy.get('.handsontableInput').should('exist')
+  })
+})
+
+/**
+ * Cypress clears cookies between tests, which drops the SASjs Server session
+ * and lands the app back on the evaluation agreement card. Accept it when it is
+ * shown - a no-op when the session survived.
+ */
+const bootApp = () => {
+  cy.visit(hostUrl, { timeout: longerCommandTimeout })
+  cy.get('body').then(($body: any) => {
+    if ($body.find('#TCS input[type="checkbox"]').length) {
+      cy.get('#TCS input[type="checkbox"]').check({ force: true })
+      cy.wait(4000)
+    }
+  })
+}
+
+/** Opens a table straight in the EDIT screen and waits for the grid. */
+const openEditor = (libMem: string) => {
+  cy.visit(`${hostUrl}${appLocation}/#/editor/${libMem}`, {
+    timeout: longerCommandTimeout
+  })
+  cy.get('.ht_clone_top .htCore thead tr th', {
+    timeout: longerCommandTimeout
+  }).should('exist')
+  cy.wait(2500)
+}
+
+/** The EDIT screen opens read-only; this is the button that starts an edit. */
+const enterEditMode = () => {
+  cy.get('.btnCtrl button.btn-primary', { timeout: longerCommandTimeout })
+    .click()
+    .then(() => cy.wait(2500))
+}
+
+/** The grid header cell for a column, by its rendered text. */
+const headerCell = (headerText: string) =>
+  cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
+    const th = [...$ths].find(
+      (candidate) => (candidate as HTMLElement).innerText.trim() === headerText
+    )
+    if (!th) throw new Error(`header not found: ${headerText}`)
+    return cy.wrap(th)
+  })
+
+const openFilterPopup = () => {
+  cy.contains('button.btnView', 'Filter', {
+    timeout: longerCommandTimeout
+  }).click()
+  cy.get('.filter-modal', { timeout: longerCommandTimeout }).should(
+    'be.visible'
+  )
+}
+
+/**
+ * The soft-select inputs drop a transparent .overlay click-catcher over the
+ * modal while their suggestion list is open, so they need force.
+ */
+const setFilterVariable = (column: string) => {
+  cy.get('#vals_var_id0_0').clear({ force: true }).type(column, { force: true })
+  cy.get('#datalist_vals_var_id0_0 option').contains(column).click({
+    force: true
+  })
+  cy.get('#vals_var_id0_0').trigger('keyup', { key: 'Escape', force: true })
+  cy.wait(400)
+}
+
+const setFilterValue = (value: string) => {
+  cy.get('#vals_0_0').clear({ force: true }).type(value, { force: true })
+  cy.get('#vals_0_0').trigger('keyup', { key: 'Escape', force: true })
+  cy.wait(400)
+}
+
+const setFilterOperator = (operator: string) => {
+  cy.get('.filter-modal .operator-col select').first().select(operator)
+  cy.wait(500)
+}
+
+const submitFilter = () => {
+  cy.contains('.filter-modal button', 'Ok').click()
+  cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
+  cy.wait(2500)
+}
+
+const getCellByHeaderAndRow = (rowIndex: number, headerText: string) => {
+  return cy
+    .get('.ht_clone_top .htCore thead tr th')
+    .should(($ths) => {
+      const texts = [...$ths].map((th) => (th as HTMLElement).innerText.trim())
+      expect(texts).to.include(headerText)
+    })
+    .then(($ths) => {
+      const index = [...$ths].findIndex(
+        (th) => (th as HTMLElement).innerText.trim() === headerText
+      )
+
+      return cy
+        .get('.ht_master tbody tr')
+        .then((rows: any) => rows[rowIndex].childNodes[index])
+        .then((cell) => cy.get(cell))
+    })
+}

client/cypress/e2e/excel.cy.ts

@@ -399,6 +399,26 @@ context('excel tests: ', function () {
     })
   })
 
+  it('25 | Keeps decimals when the upload preview becomes a manual edit', () => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    // Entering manual edit mode discards the source file but keeps the values
+    // parsed from it. SOME_NUM holds 0.901045951946194 in the first data row
+    // of the fixture, so a truncated or zeroed cell fails the assertion.
+    attachExcelFile('regular_excel.xlsx', () => {
+      getFirstRowCells().then((previewCells) => {
+        expect(previewCells).to.include(decimalSomeNum)
+
+        cy.get('.btn-upload-preview.btn-primary').click()
+        cy.get('.modal-footer .btn-success-outline').click()
+
+        getFirstRowCells().then((editCells) => {
+          expect(editCells).to.include(decimalSomeNum)
+        })
+      })
+    })
+  })
+
   // Large files break Cypress
 
   // it ('? | Uploads Excel with size of 5MB', (done) => {
@@ -476,6 +496,19 @@ const attachExcelFile = (excelFilename: string, callback?: any) => {
     })
 }
 
+const decimalSomeNum = '0.901045951946194'
+
+// The row's cell texts, without depending on column positions: the preview
+// grid renders row-status and delete cells in an order that varies.
+const getFirstRowCells = () =>
+  cy
+    .get('#hotTable', { timeout: longerCommandTimeout })
+    .find('div.ht_master.handsontable tbody tr')
+    .eq(0)
+    .then(($row) =>
+      Array.from($row[0].children).map((cell: any) => cell.innerText.trim())
+    )
+
 const submitExcel = (callback?: any) => {
   cy.get('.buttonBar button.preview-submit', { timeout: longerCommandTimeout })
     .click()

client/cypress/e2e/filtering.cy.ts

@@ -11,6 +11,18 @@ context('filtering tests: ', function () {
   this.beforeAll(() => {
     cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
     cy.loginAndUpdateValidKey()
+
+    /**
+     * The evaluation agreement is per-origin localStorage, so a fresh browser
+     * profile lands on it and the nav tree never renders.  Guarded, so it is a
+     * no-op once an earlier spec in the run has accepted it.
+     */
+    cy.get('body').then(($body) => {
+      if ($body.find('#TCS input[type="checkbox"]').length) {
+        cy.get('#TCS input[type="checkbox"]').check({ force: true })
+        cy.wait(4000)
+      }
+    })
   })
 
   this.beforeEach(() => {
@@ -51,9 +63,13 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_TIME', '00:00:42', 'time', () => {
-        checkInfoBarIncludes(`(( SOME_TIME = 42 ))`, (includes: boolean) => {
-          if (includes) done()
-        })
+        checkInfoBarIncludes(
+          `(( SOME_TIME = '00:00:42't ))`,
+          (includes: boolean) => {
+            if (!includes) return
+            assertGridRowCount(3, () => done())
+          }
+        )
       })
     })
   })
@@ -63,9 +79,13 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_TIME', '42', 'value', () => {
-        checkInfoBarIncludes(`(( SOME_TIME = 42 ))`, (includes: boolean) => {
-          if (includes) done()
-        })
+        checkInfoBarIncludes(
+          `(( SOME_TIME = '00:00:42't ))`,
+          (includes: boolean) => {
+            if (!includes) return
+            assertGridRowCount(3, () => done())
+          }
+        )
       })
     }, false)
   })
@@ -75,9 +95,13 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATE', '12/02/1960', 'date', () => {
-        checkInfoBarIncludes(`(( SOME_DATE = 42 ))`, (includes: boolean) => {
-          if (includes) done()
-        })
+        checkInfoBarIncludes(
+          `(( SOME_DATE = '12FEB1960'd ))`,
+          (includes: boolean) => {
+            if (!includes) return
+            assertGridRowCount(5, () => done())
+          }
+        )
       })
     })
   })
@@ -87,9 +111,13 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATE', '42', 'value', () => {
-        checkInfoBarIncludes(`(( SOME_DATE = 42 ))`, (includes: boolean) => {
-          if (includes) done()
-        })
+        checkInfoBarIncludes(
+          `(( SOME_DATE = '12FEB1960'd ))`,
+          (includes: boolean) => {
+            if (!includes) return
+            assertGridRowCount(5, () => done())
+          }
+        )
       })
     }, false)
   })
@@ -104,9 +132,10 @@ context('filtering tests: ', function () {
         'datetime',
         () => {
           checkInfoBarIncludes(
-            `(( SOME_DATETIME = 42 ))`,
+            `(( SOME_DATETIME = '01JAN1960:00:00:42'dt ))`,
             (includes: boolean) => {
-              if (includes) done()
+              if (!includes) return
+              assertGridRowCount(5, () => done())
             }
           )
         }
@@ -120,9 +149,10 @@ context('filtering tests: ', function () {
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATETIME', '42', 'value', () => {
         checkInfoBarIncludes(
-          `(( SOME_DATETIME = 42 ))`,
+          `(( SOME_DATETIME = '01JAN1960:00:00:42'dt ))`,
           (includes: boolean) => {
-            if (includes) done()
+            if (!includes) return
+            assertGridRowCount(5, () => done())
           }
         )
       })
@@ -134,9 +164,13 @@ context('filtering tests: ', function () {
 
     openFilterPopup(() => {
       setFilterWithValue('SOME_DATE', '', 'in', () => {
-        checkInfoBarIncludes(`(( SOME_DATE IN (42) ))`, (includes: boolean) => {
-          if (includes) done()
-        })
+        checkInfoBarIncludes(
+          `(( SOME_DATE IN ('12FEB1960'd) ))`,
+          (includes: boolean) => {
+            if (!includes) return
+            assertGridRowCount(5, () => done())
+          }
+        )
       })
     })
   })
@@ -156,6 +190,34 @@ context('filtering tests: ', function () {
   //     })
   //   })
   // })
+
+  /**
+   * With the pickers off, a temporal BETWEEN takes both of its values from a
+   * soft-select over the column's values, so the value is chosen from that list
+   * rather than typed.  The range dropdown binds the formatted label, as the
+   * single-value dropdown's list does, so the field reads the date and the
+   * clause composition maps it back to the internal day / second count before
+   * the literal is built - a display string that reached sasTemporalLiteral as
+   * a non-number would come back null and be passed through to RAW_VALUE
+   * unquoted, which mp_filtercheck rejects on a numeric column and aborts the
+   * save.  The test asserts the applied clause and the row count, so it pins
+   * the mapping end to end.
+   */
+  it('7 | Non picker - filter date field BETWEEN from the value list', (done) => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    openFilterPopup(() => {
+      setTemporalRangeFromValueList('SOME_DATE', '12FEB1960', () => {
+        checkInfoBarIncludes(
+          `BETWEEN '12FEB1960'd AND '12FEB1960'd`,
+          (includes: boolean) => {
+            if (!includes) return
+            assertGridRowCount(5, () => done())
+          }
+        )
+      })
+    }, false)
+  })
 })
 
 const checkInfoBarIncludes = (text: string, callback: any) => {
@@ -166,6 +228,21 @@ const checkInfoBarIncludes = (text: string, callback: any) => {
   })
 }
 
+/**
+ * The temporal filter tests assert the applied filter still selects the same
+ * rows as the internal value did - a readable clause that filtered differently
+ * would be a worse bug than the one being fixed.
+ */
+const assertGridRowCount = (expected: number, callback: any) => {
+  cy.get('#hotTable .ht_master tbody tr', {
+    timeout: longerCommandTimeout
+  })
+    .should('have.length', expected)
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
 const openFilterPopup = (
   callback?: any,
   usePickers: boolean = true,
@@ -331,6 +408,54 @@ const setFilterWithValue = (
   if (callback) callback()
 }
 
+/**
+ * Drives a temporal BETWEEN with the pickers off, choosing both values from the
+ * range soft-select's list rather than typing them - typing bypasses the
+ * binding this test exists to pin.
+ */
+const setTemporalRangeFromValueList = (
+  variableValue: string,
+  formatted: string,
+  callback?: any
+) => {
+  cy.wait(600)
+
+  cy.focused().type(variableValue)
+  cy.wait(100)
+  cy.get('.variable-col .autocomplete-wrapper', { withinSubject: null })
+    .first()
+    .trigger('keydown', { key: 'ArrowDown' })
+  cy.get('.variable-col .autocomplete-wrapper', {
+    withinSubject: null
+  }).trigger('keydown', { key: 'Enter' })
+  cy.focused().tab()
+  cy.wait(100)
+
+  cy.focused().select('BETWEEN').trigger('change')
+  cy.wait(400)
+
+  const chooseFromList = (range: 'start' | 'end') => {
+    const inputId = `vals-${range}_0_0`
+
+    cy.get(`#${inputId}`)
+      .clear({ force: true })
+      .type(formatted, { force: true })
+    cy.get(`#datalist_${inputId} option`)
+      .contains(formatted)
+      .click({ force: true })
+    cy.get(`#${inputId}`).trigger('keyup', { key: 'Escape', force: true })
+    cy.wait(300)
+  }
+
+  chooseFromList('start')
+  chooseFromList('end')
+
+  cy.contains('.filter-modal button', 'Ok').click()
+  cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
+
+  if (callback) callback()
+}
+
 const openTableFromTree = (libNameIncludes: string, tablename: string) => {
   cy.get('.app-loading', { timeout: longerCommandTimeout })
     .should('not.exist')

client/cypress/e2e/row-cell-limits.cy.ts

@@ -1,13 +1,15 @@
-// The EDIT screen refuses a selection that is too large - DC_MAXOBS_WEBEDIT
-// rows, or DC_MAXCELLS_WEBEDIT cells (rows x columns), whichever is reached
-// first - and the VIEW screen caps at DC_MAXOBS_WEBVIEW rows.
+// The EDIT screen is bounded by DC_MAXOBS_WEBEDIT rows and DC_MAXCELLS_WEBEDIT
+// cells (rows x columns), whichever is reached first, and the VIEW screen caps
+// at DC_MAXOBS_WEBVIEW rows.
 //
+// A selection over an EDIT limit is served TRUNCATED - the grid carries the
+// first rows that fit, and a warning triangle against the table name says why.
 // This spec pins those limits against the mock, which mirrors the shipped
 // defaults (250 rows / 200000 cells for EDIT, 2000 rows for VIEW).  It uses
 // two existing fixtures rather than new ones:
 //   MPE_X_SEARCH - 1000 rows x 9 cols, so it trips the ROW limit
-//   MPE_X_WIDE   - 200 rows x 1001 cols (200,400 cells), under the row limit
-//                  but over the CELL limit
+//   MPE_X_WIDE   - 200 rows x 1001 cols (200,200 cells), under the row limit
+//                  but over the CELL limit, so the row cap is 199
 export {}
 
 const hostUrl = Cypress.env('hosturl')
@@ -60,27 +62,51 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
     })
   })
 
-  it('2 | EDIT refuses a table over the row limit', () => {
+  it('2 | EDIT serves a table over the row limit, truncated, with a warning', () => {
     openTableFromTree(libraryToOpenIncludes, 'mpe_x_search')
     openEdit()
 
-    cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
+    // 1000 rows x 9 columns is over the 250 row limit, so the grid carries the
+    // first 250 rows and the header says how many it is showing.
+    cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
       'contain.text',
-      'Table is too big (1000 rows) - please filter and try again!'
+      '250 rows'
     )
+    cy.get('#hotTable .ht_master .htCore tbody tr', {
+      timeout: longerCommandTimeout
+    }).should('have.length.at.least', 1)
+
+    cy.get('.truncation-warning', { timeout: longerCommandTimeout })
+      .should('be.visible')
+      .realHover()
+    cy.get('.editor-title clr-tooltip-content', {
+      timeout: longerCommandTimeout
+    })
+      .should('be.visible')
+      .should('contain.text', 'DC_MAXOBS_WEBEDIT')
+      .should(($el) => expect($el.css('opacity')).to.eq('1'))
   })
 
-  it('3 | EDIT refuses a table over the cell limit', () => {
+  it('3 | EDIT serves a table over the cell limit, truncated, with a warning', () => {
     openTableFromTree(libraryToOpenIncludes, 'mpe_x_wide')
     openEdit()
 
-    // 200 rows x 1002 columns (the 1001 stored columns plus the housekeeping
-    // delete column) = 200,400 cells, over the 200,000 cell limit, while the
-    // row count is under the 250 row limit.
-    cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
+    // 200 rows x 1001 stored columns = 200,200 cells, over the 200,000 cell
+    // limit, while the row count is under the 250 row limit - so the cell
+    // limit sets the cap: floor(200000 / 1001) = 199 rows.
+    cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
       'contain.text',
-      'Selection is too wide (200 rows x 1002 cols) - please filter and try again!'
+      '199 rows'
     )
+    cy.get('.truncation-warning', { timeout: longerCommandTimeout })
+      .should('be.visible')
+      .realHover()
+    cy.get('.editor-title clr-tooltip-content', {
+      timeout: longerCommandTimeout
+    })
+      .should('be.visible')
+      .should('contain.text', 'DC_MAXCELLS_WEBEDIT')
+      .should(($el) => expect($el.css('opacity')).to.eq('1'))
   })
 
   it('4 | the EDIT limits follow MPE_CONFIG, not the shipped defaults', () => {
@@ -90,20 +116,29 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
     // all.  And MPE_CONFIG lives in the control library, so reading it from the
     // table's own library would fail for this table in particular.
     //
-    // The cell limit is the one to move: 7 rows x 18 columns is 126 cells, so
-    // at 110 the TESTDATA.MPE_X_NEW edit must be refused, where a hard-coded
-    // 200000 - or a config read that fails and falls back to it - lets it
-    // through.  110 is still above MPE_CONFIG's own 17 x 6 = 102, so the option
-    // can be put back afterwards.
+    // The cell limit is the one to move: 7 rows x 17 columns is 119 cells, so
+    // at 110 the TESTDATA.MPE_X_NEW edit must be truncated to
+    // floor(110 / 17) = 6 rows, where a hard-coded 200000 - or a config read
+    // that fails and falls back to it - lets it through.  110 is still above
+    // MPE_CONFIG's own 6 x 17 = 102, so the option can be put back afterwards.
     setConfigValue('DC_MAXCELLS_WEBEDIT', '110', () => {
       visitPage('view/data/TESTDATA.MPE_X_NEW')
       closeAbortModalIfPresent()
       openEdit()
 
-      cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
+      cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
         'contain.text',
-        'Selection is too wide (7 rows x 18 cols) - please filter and try again!'
+        '6 rows'
       )
+      cy.get('.truncation-warning', { timeout: longerCommandTimeout })
+        .should('be.visible')
+        .realHover()
+      cy.get('.editor-title clr-tooltip-content', {
+        timeout: longerCommandTimeout
+      })
+        .should('be.visible')
+        .should('contain.text', 'DC_MAXCELLS_WEBEDIT')
+        .should(($el) => expect($el.css('opacity')).to.eq('1'))
     })
 
     // Put the option back, so the rest of the estate sees the shipped default.
@@ -112,28 +147,31 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
       closeAbortModalIfPresent()
       openEdit()
 
-      // 126 cells under a 200000 cell limit: the editor loads, no abort.
-      cy.get('#hotTable .ht_master .htCore tbody tr', {
-        timeout: longerCommandTimeout
-      }).should('exist')
+      // 119 cells under a 200000 cell limit: all 7 rows load, no warning.
+      cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
+        'contain.text',
+        '7 rows'
+      )
+      cy.get('.truncation-warning').should('not.exist')
     })
   })
 
-  it('5 | the filter picker still lists the variables after the abort', () => {
+  it('5 | the warning says to filter, and filtering narrows the grid', () => {
     openTableFromTree(libraryToOpenIncludes, 'mpe_x_search')
     openEdit()
 
-    cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
-      'contain.text',
-      'Table is too big (1000 rows) - please filter and try again!'
-    )
+    cy.get('.truncation-warning', { timeout: longerCommandTimeout })
+      .should('be.visible')
+      .realHover()
+    cy.get('.editor-title clr-tooltip-content', {
+      timeout: longerCommandTimeout
+    })
+      .should('be.visible')
+      .should('contain.text', 'Filter the table to work with the rest')
+      .should(($el) => expect($el.css('opacity')).to.eq('1'))
 
-    closeAbortModal()
-
-    // The abort returns no table, and the picker's variable list normally comes
-    // from the table's own columns - so without the public/getcols fallback the
-    // "please filter and try again" advice has nothing to pick from.  Every
-    // column of MPE_X_SEARCH must be on offer.
+    // The advice has to be actionable: the picker must offer the table's
+    // columns and the filter must reload the grid.
     openFilterPopup()
 
     filterVariables().should(($options) => {
@@ -155,8 +193,20 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
       ])
     })
 
-    // and the suggestion list is usable, not just populated
     setFilterVariable('SITE_NAME')
+    // 100 of the 1000 rows: under the limit, so nothing is truncated.
+    setFilterValue('Nereid Trench')
+    submitFilter()
+
+    cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
+      'contain.text',
+      "SITE_NAME = 'Nereid Trench'"
+    )
+    cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
+      'contain.text',
+      '100 rows'
+    )
+    cy.get('.truncation-warning').should('not.exist')
   })
 })
 
@@ -173,13 +223,6 @@ const openEdit = () => {
   }).click()
 }
 
-/** The abort modal offers Close, and the filter button sits behind it. */
-const closeAbortModal = () => {
-  cy.get('app-info-modal', { timeout: longerCommandTimeout })
-    .contains('button', 'Close')
-    .click()
-}
-
 /**
  * The EDIT screen's Filter button opens the same .filter-modal the viewer uses.
  * It only renders once the editor is in its read-only (non-editing) state,
@@ -207,6 +250,19 @@ const setFilterVariable = (column: string) => {
   cy.get('#vals_var_id0_0').trigger('keyup', { key: 'Escape', force: true })
 }
 
+/** The value input of the same soft-select row. */
+const setFilterValue = (value: string) => {
+  cy.get('#vals_0_0').clear({ force: true }).type(value, { force: true })
+  cy.get('#vals_0_0').trigger('keyup', { key: 'Escape', force: true })
+  cy.wait(400)
+}
+
+const submitFilter = () => {
+  cy.contains('.filter-modal button', 'Ok').click()
+  cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
+  cy.wait(2500)
+}
+
 /**
  * Sets a DC option in MPE_CONFIG through the editor and approves it, then hands
  * back to the callback.  The same route licensing.cy.ts uses for the licence

client/cypress/e2e/startupservice-launcher.cy.ts

@@ -0,0 +1,81 @@
+// Marks this file as an ES module so its top-level consts don't collide with
+// other spec files under the TS type-checker.
+export {}
+//
+// When the startup service fails without a usable response, the error modal
+// offers a button that runs the same service in a new browser window, where the
+// SAS log is returned instead. This spec forces a startup failure against the
+// mock backend and checks the button and the URL it opens.
+const hostUrl = Cypress.env('hosturl')
+
+const STARTUP_PROGRAM = '/Public/app/dc/services/public/startupservice'
+
+/** The body a Viya Compute task returns when the run outlives the
+ * execution-task timeout: a bare string, so the adapter has no webout and no
+ * log to report. */
+const rawJobError = [
+  'Job error',
+  'The Compute service could not execute the task c74a707a-8b88-46a5-8f28-9a7694ad5e13 because the context is not reusable.',
+  'path: /compute/tasks',
+  'correlator: dc1a9fda-8f65-4f32-b4d4-2a70e02179b1;1bcb6a4c-8a16-4053-a790-292544c25a03'
+].join('\n')
+
+context('startupservice launcher: ', function () {
+  it('1 | runs the startupservice in a new window when its response is malformed', () => {
+    // Report an empty admin folder so the app loads the startup service: on a
+    // freshly seeded mock estate makedata is still present, which sends the app
+    // to the deploy screen instead.
+    cy.intercept('GET', '**/drive/folder**', (req) => {
+      // Guarded parse: a lone '%' in the URL would make decodeURIComponent
+      // throw, and an exception here aborts the request.
+      let url = req.url
+      try {
+        url = decodeURIComponent(req.url)
+      } catch (e) {
+        url = req.url
+      }
+
+      if (url.includes('services/admin')) {
+        req.reply({ files: [], folders: [] })
+        return
+      }
+      req.continue()
+    })
+
+    cy.intercept('**/stp/execute/**', (req) => {
+      if (req.url.includes('public/startupservice')) {
+        // The adapter cannot use a bare string as a response payload, so the
+        // startup request fails with nothing to show.
+        req.reply({
+          statusCode: 200,
+          headers: { 'content-type': 'application/json' },
+          body: JSON.stringify(rawJobError)
+        })
+        return
+      }
+      req.continue()
+    })
+
+    // The launcher must not actually open a tab under test.
+    cy.visit(hostUrl, {
+      timeout: 60000,
+      onBeforeLoad(win) {
+        cy.stub(win, 'open').as('windowOpen')
+      }
+    })
+
+    cy.get('.clr-abort-modal', { timeout: 60000 }).should('be.visible')
+    cy.get('.clr-abort-modal').last().should('contain.text', 'startupservice')
+    cy.get('.clr-abort-modal').last().contains('button', 'Show log').click()
+
+    cy.get('@windowOpen').should('have.been.calledOnce')
+    cy.get('@windowOpen').then((stub: any) => {
+      const url = stub.firstCall.args[0]
+
+      expect(url).to.contain(STARTUP_PROGRAM)
+      expect(url).to.contain('_debug=131')
+      // The browser route is what makes the log available: no execution task.
+      expect(url).not.to.contain('_executionTasks')
+    })
+  })
+})

client/cypress/support/commands.ts

@@ -79,14 +79,34 @@ Cypress.Commands.add(
         .then(() => {
           cy.wait(2000)
 
-          if ($body.find('.nav-tree').length > 0) {
+          /**
+           * A key that is already working needs no update.  Ask the loaded
+           * page, not the tree: a free tier estate renders the nav tree as
+           * well, so the tree cannot tell a licensed estate from an
+           * unlicensed one - the licence banner is what does.  Reading the
+           * body again here also matters on an estate that shows a login
+           * form, where the body captured above has no tree yet.
+           */
+          cy.get('body').then(($loaded) => {
+            const licenceProblem = $loaded.find('#demo-banner').length > 0
+
+            if (!licenceProblem && $loaded.find('.nav-tree').length > 0) {
+              return logout(() => {
+                return
+              })
+            }
+
             /**
-             * If licence key is already working, then skip rest of the function
+             * An estate with no key is left as it is unless the caller asked
+             * for a licensed one - the free tier specs are written against
+             * that state and must not have a key planted under them.
              */
-            return logout(() => {
-              return
-            })
-          } else {
+            if (!forceLicenceKey) {
+              return logout(() => {
+                return
+              })
+            }
+
             const keyData = {
               valid_until: moment().add(20, 'day').format('YYYY-MM-DD'),
               number_of_users: 10,
@@ -106,27 +126,21 @@ Cypress.Commands.add(
                     cy.wait(20000)
                   }
 
-                  if (!forceLicenceKey) {
-                    return logout(() => {
-                      return
-                    })
-                  } else {
-                    return updateLicenseKeyQuick(keysGen, () => {
-                      cy.wait(25000)
-                      return acceptTermsIfPresented((result: boolean) => {
-                        if (result) {
-                          cy.wait(20000)
-                        }
-                        return logout(() => {
-                          return
-                        })
+                  return updateLicenseKeyQuick(keysGen, () => {
+                    cy.wait(25000)
+                    return acceptTermsIfPresented((result: boolean) => {
+                      if (result) {
+                        cy.wait(20000)
+                      }
+                      return logout(() => {
+                        return
                       })
                     })
-                  }
+                  })
                 })
               }
             )
-          }
+          })
         })
     })
   }

client/licenseChecker.js

@@ -10,7 +10,7 @@ const check = (cwd) => {
         onlyAllow:
           'AFLv2.1;Apache 2.0;Apache-2.0;Apache*;Artistic-2.0;BlueOak-1.0.0;0BSD;BSD*;BSD-2-Clause;BSD-3-Clause;CC0-1.0;CC-BY-3.0;CC-BY-4.0;ISC;MIT;MPL-2.0;ODC-By-1.0;Python-2.0;Unlicense;',
         excludePackages:
-          '@cds/city@1.1.0;@handsontable/angular-wrapper@16.0.1;@handsontable/angular-wrapper@17.1.0;@handsontable/angular-wrapper@18.0.0;@handsontable/angular-wrapper@18.1.1;handsontable@^16.0.1;handsontable@16.2.0;handsontable@17.1.0;handsontable@18.0.0;handsontable@18.1.1;hyperformula@2.7.1;hyperformula@3.0.0;hyperformula@3.1.0;hyperformula@3.2.0;hyperformula@3.3.0;jackspeak@3.4.3;path-scurry@1.11.1;package-json-from-dist@1.0.1;buffers@0.1.1'
+          '@cds/city@1.1.0;@handsontable/angular-wrapper@16.0.1;@handsontable/angular-wrapper@17.1.0;@handsontable/angular-wrapper@18.0.0;@handsontable/angular-wrapper@18.1.1;@handsontable/angular-wrapper@18.1.2;handsontable@^16.0.1;handsontable@16.2.0;handsontable@17.1.0;handsontable@18.0.0;handsontable@18.1.1;handsontable@18.1.2;hyperformula@2.7.1;hyperformula@3.0.0;hyperformula@3.1.0;hyperformula@3.2.0;hyperformula@3.3.0;jackspeak@3.4.3;path-scurry@1.11.1;package-json-from-dist@1.0.1;buffers@0.1.1'
       },
       (error, json) => {
         if (error) {

client/package.json

@@ -41,22 +41,22 @@
   },
   "private": true,
   "dependencies": {
-    "@angular/animations": "^20.3.33",
+    "@angular/animations": "20.3.34",
     "@angular/cdk": "^20.2.14",
-    "@angular/common": "^20.3.33",
-    "@angular/compiler": "^20.3.33",
-    "@angular/core": "^20.3.33",
-    "@angular/forms": "^20.3.33",
-    "@angular/platform-browser": "^20.3.33",
-    "@angular/platform-browser-dynamic": "^20.3.33",
-    "@angular/router": "^20.3.33",
+    "@angular/common": "20.3.34",
+    "@angular/compiler": "20.3.34",
+    "@angular/core": "20.3.34",
+    "@angular/forms": "20.3.34",
+    "@angular/platform-browser": "20.3.34",
+    "@angular/platform-browser-dynamic": "20.3.34",
+    "@angular/router": "20.3.34",
     "@cds/core": "^6.15.1",
     "@clr/angular": "file:libraries/clr-angular-17.9.0.tgz",
     "@clr/icons": "^13.0.2",
     "@clr/ui": "file:libraries/clr-ui-17.9.0.tgz",
-    "@handsontable/angular-wrapper": "18.1.1",
-    "@sasjs/adapter": "^4.19.1",
-    "@sasjs/utils": "^3.6.0",
+    "@handsontable/angular-wrapper": "18.1.2",
+    "@sasjs/adapter": "5.1.2",
+    "@sasjs/utils": "4.0.0",
     "@sheet/crypto": "file:libraries/sheet-crypto.tgz",
     "@types/d3-graphviz": "^2.6.7",
     "@types/text-encoding": "0.0.35",
@@ -69,12 +69,11 @@
     "d3-graphviz": "^5.0.2",
     "exceljs": "^4.4.0",
     "fs-extra": "^7.0.1",
-    "handsontable": "18.1.1",
+    "handsontable": "18.1.2",
     "https-browserify": "1.0.0",
     "hyperformula": "^2.5.0",
     "iconv-lite": "^0.5.0",
     "jquery-datetimepicker": "^2.5.21",
-    "jsrsasign": "11.1.1",
     "marked": "^5.0.0",
     "moment": "^2.30.1",
     "ngx-clipboard": "^16.0.0",
@@ -99,8 +98,8 @@
     "@angular-eslint/eslint-plugin-template": "19.8.1",
     "@angular-eslint/schematics": "19.8.1",
     "@angular-eslint/template-parser": "19.8.1",
-    "@angular/cli": "^20.3.32",
-    "@angular/compiler-cli": "^20.3.33",
+    "@angular/cli": "20.3.37",
+    "@angular/compiler-cli": "20.3.34",
     "@babel/plugin-proposal-private-methods": "^7.18.6",
     "@compodoc/compodoc": "^2.0.0",
     "@cypress/webpack-preprocessor": "^5.17.1",
@@ -148,7 +147,7 @@
     "ajv": "8.18.0",
     "uuid": "11.1.1",
     "lighthouse": "13.4.0",
-    "fast-uri": "3.1.8",
+    "fast-uri": "4.2.3",
     "readdir-glob": {
       "brace-expansion": "^5.0.9"
     },

client/src/app/app.component.html

@@ -318,6 +318,7 @@
             class="startup-check"
             [class.startup-check--error]="step.status === 'error'"
             [class.startup-check--done]="step.status === 'done'"
+            [class.startup-check--grid]="step.details?.length"
           >
             @switch (step.status) {
               @case ('pending') {
@@ -346,6 +347,18 @@
             @if (step.detail) {
               <span class="startup-check__detail">{{ step.detail }}</span>
             }
+            @if (step.details?.length) {
+              <div class="startup-check__grid">
+                @for (detail of step.details; track detail.name) {
+                  <span class="startup-check__grid-name">{{
+                    detail.name
+                  }}</span>
+                  <span class="startup-check__grid-value">{{
+                    detail.value
+                  }}</span>
+                }
+              </div>
+            }
           </div>
         }
       </div>

client/src/app/app.component.ts

@@ -20,6 +20,8 @@ import { DcAdapterSettings } from './models/DcAdapterSettings'
 import { AppStoreService } from './services/app-store.service'
 import { LicenceService } from './services/licence.service'
 import {
+  getSasjsTagDetails,
+  SASJS_TAG_STEP,
   StartupCheckService,
   StartupStep
 } from './services/startup-check.service'
@@ -190,40 +192,45 @@ export class AppComponent {
    * Parses adapter settings that are found in the <sasjs> tag inside index.html
    */
   private parseDcAdapterSettings() {
+    /* The <sasjs> tag is a mandatory part of index.html - it carries serverUrl,
+      serverType and the rest of the adapter config - so there is no config to
+      fall back on when it is absent. A missing one is reported on the boot
+      screen below rather than handled with an abort modal. */
     const sasjsElement = document.querySelector('sasjs')
 
-    if (!sasjsElement) {
-      this.licenceService.deactivateApp()
-      setTimeout(() => {
-        this.eventService.showAbortModal(
-          null,
-          "Please make sure 'SASJS' tag with config attributes is added to index.html",
-          null,
-          'SASjs Config not found'
-        )
-      })
-      return
-    }
-
     const getAppAttribute = (attribute: string) =>
-      sasjsElement.getAttribute(attribute) || undefined
+      sasjsElement?.getAttribute(attribute) || undefined
 
     const dcAdapterSettings = {
       serverUrl: getAppAttribute('serverUrl') || '',
       appLoc: getAppAttribute('appLoc') || '',
       serverType: getAppAttribute('serverType'),
       loginMechanism: getAppAttribute('loginMechanism') || '',
-      adminGroup: getAppAttribute('adminGroup') || '',
       dcPath: getAppAttribute('dcPath') || '',
       debug: getAppAttribute('debug') === 'true' || false,
       useComputeApi: this.parseComputeApi(getAppAttribute('useComputeApi')),
       runAsTask: getAppAttribute('runAsTask') === 'true' || false,
-      contextName: getAppAttribute('contextName') || '',
-      hotLicenceKey: getAppAttribute('hotLicenceKey') || ''
+      contextName: getAppAttribute('contextName') || ''
     }
 
     this.dcAdapterSettings = dcAdapterSettings as any
     this.appStoreService.setDcAdapterSettings(dcAdapterSettings as any)
+
+    /* Show the <sasjs> tag contents on the boot screen, so the effective
+      configuration (serverType, debug, useComputeApi, ...) is visible without
+      having to read the page source. The tag is mandatory - without it the
+      adapter has no serverUrl or serverType - so a missing one is called out
+      plainly rather than silently leaving the step pending. */
+    if (sasjsElement) {
+      this.startupCheckService.setTagDetails(getSasjsTagDetails(sasjsElement))
+    } else {
+      this.startupCheckService.updateStep(
+        SASJS_TAG_STEP,
+        'error',
+        'SASJS TAG MISSING!!'
+      )
+    }
+
     this.appService.sasServiceInit()
   }
 
@@ -307,9 +314,12 @@ export class AppComponent {
    */
   public subscribeToShowAbortModal() {
     this.eventService.onShowAbortModal.subscribe((sasjsAbort: InfoModal) => {
-      let abortId = this.sasjsAborts.length + 1
-      sasjsAbort.id = abortId
-      this.sasjsAborts.push(sasjsAbort)
+      /* One message at a time.  Pushing to the array stacked a second modal on
+         top of the one already open - two dismissals for one failure, and the
+         lower one was often a duplicate of the upper.  Replacing keeps the
+         newest message, which is the one the user has to act on. */
+      sasjsAbort.id = Date.now()
+      this.sasjsAborts = [sasjsAbort]
       this.cdr.detectChanges() //Changes were not triggered while hot is focused
     })
   }

client/src/app/deploy/deploy-routing.module.ts

@@ -2,10 +2,7 @@ import { NgModule } from '@angular/core'
 import { RouterModule, Routes } from '@angular/router'
 import { DeployComponent } from './deploy.component'
 
-const routes: Routes = [
-  { path: '', component: DeployComponent },
-  { path: 'manualdeploy', component: DeployComponent }
-]
+const routes: Routes = [{ path: '', component: DeployComponent }]
 
 @NgModule({
   imports: [RouterModule.forChild(routes)],

client/src/app/deploy/deploy.component.html

@@ -29,20 +29,6 @@
               ></label
             >
           </clr-checkbox-wrapper>
-          <!-- <hr />
-                <div class="clr-checkbox-wrapper">
-                  <input
-                    [(ngModel)]="autodeploy"
-                    type="checkbox"
-                    id="checkbox2"
-                    class="clr-checkbox"
-                    checked
-                    />
-                  <label for="checkbox2"
-                    >Autodeploy
-                    {{ !jsonFile ? '(json file is not available)' : '' }}</label
-                    >
-                  </div> -->
         </div>
       </div>
     }
@@ -52,29 +38,17 @@
   @if (step > 0) {
     @switch (true) {
       @case (sasJsConfig.serverType === ServerType.SasViya) {
-        @if (autodeploy) {
-          <div class="autodeploy-section card">
-            <app-automatic-deploy
-              [sasJs]="sasJs"
-              [sasJsConfig]="sasJsConfig"
-              [dcAdapterSettings]="dcAdapterSettings"
-              [appLoc]="appLoc"
-              [dcPath]="dcPath"
-              [selectedAdminGroup]="selectedAdminGroup"
-              (onNavigateToHome)="onNavigateToHome()"
-            ></app-automatic-deploy>
-          </div>
-        }
-        @if (!autodeploy) {
-          <div id="mainbody" class="card">
-            <app-manual-deploy
-              [sasJs]="sasJs"
-              [sasJsConfig]="sasJsConfig"
-              [dcAdapterSettings]="dcAdapterSettings"
-              (onNavigateToHome)="onNavigateToHome()"
-            ></app-manual-deploy>
-          </div>
-        }
+        <div class="autodeploy-section card">
+          <app-automatic-deploy
+            [sasJs]="sasJs"
+            [sasJsConfig]="sasJsConfig"
+            [dcAdapterSettings]="dcAdapterSettings"
+            [appLoc]="appLoc"
+            [dcPath]="dcPath"
+            [selectedAdminGroup]="selectedAdminGroup"
+            (onNavigateToHome)="onNavigateToHome()"
+          ></app-automatic-deploy>
+        </div>
       }
       @case (sasJsConfig.serverType === ServerType.Sasjs) {
         <div class="autodeploy-section card">

client/src/app/deploy/deploy.component.ts

@@ -1,8 +1,6 @@
 import { Component, OnInit, ViewEncapsulation } from '@angular/core'
 import { SasService } from '../services/sas.service'
 import { SASjsConfig } from '@sasjs/adapter'
-import { Router } from '@angular/router'
-import { LoggerService } from '../services/logger.service'
 import { ServerType } from '@sasjs/utils/types/serverType'
 import { AppStoreService } from '../services/app-store.service'
 import { DcAdapterSettings } from '../models/DcAdapterSettings'
@@ -19,15 +17,11 @@ import { DcAdapterSettings } from '../models/DcAdapterSettings'
 })
 export class DeployComponent implements OnInit {
   public step: number = 0
-  public adminGroups: any = []
 
   public appLoc: string = ''
   public dcPath: string = ''
   public selectedAdminGroup: string = ''
 
-  public autodeploy: boolean = true
-  public jsonFile: any = null
-
   public sasJs: any
   public sasJsConfig: SASjsConfig = new SASjsConfig()
 
@@ -37,16 +31,10 @@ export class DeployComponent implements OnInit {
 
   constructor(
     private appStoreService: AppStoreService,
-    private sasService: SasService,
-    private loggerService: LoggerService,
-    private router: Router
+    private sasService: SasService
   ) {
     this.dcAdapterSettings = this.appStoreService.getDcAdapterSettings()
 
-    if (this.router.url.includes('manualdeploy')) {
-      this.autodeploy = false
-    }
-
     this.sasJs = this.sasService.getSasjsInstance()
     this.sasJsConfig = this.sasService.getSasjsConfig()
     this.appLoc = this.dcAdapterSettings?.appLoc || ''
@@ -61,54 +49,16 @@ export class DeployComponent implements OnInit {
    */
   public setDeployDefaults() {
     this.dcPath = this.dcAdapterSettings?.dcPath || ''
-    this.selectedAdminGroup = this.dcAdapterSettings?.adminGroup || ''
-    if (!this.selectedAdminGroup) {
-      this.selectedAdminGroup = 'SASAdministrators'
-    }
+    this.selectedAdminGroup = 'SASAdministrators'
   }
 
   /**
    * Accepting terms of service shows next screen
    */
   public termsAgreeChange() {
-    if (!this.autodeploy) {
-      this.getAdminGroups()
-    }
-
     this.step++
   }
 
-  /**
-   * Fetches admin groups from VIYA to be selected for a backend deploy
-   */
-  public getAdminGroups() {
-    fetch(
-      this.sasJsConfig.serverUrl + '/identities/groups?sortBy=name&limit=5000',
-      {
-        headers: {
-          Accept: 'application/json'
-        }
-      }
-    )
-      .then((res: any) => {
-        return res.text()
-      })
-      .then((res: any) => {
-        let jsonRes
-
-        try {
-          jsonRes = JSON.parse(res)
-        } catch (err: any) {
-          console.error(err)
-        }
-
-        if (jsonRes) {
-          this.adminGroups = jsonRes.items
-          this.selectedAdminGroup = this.adminGroups[0].id
-        }
-      })
-  }
-
   public onNavigateToHome() {
     window.open(location.href.split('#')[0], '_blank')
   }

client/src/app/deploy/deploy.module.ts

@@ -2,7 +2,6 @@ import { NgModule } from '@angular/core'
 import { CommonModule } from '@angular/common'
 import { DeployComponent } from './deploy.component'
 import { AutomaticComponent } from './sections/automatic/automatic.component'
-import { ManualComponent } from './sections/manual/manual.component'
 import { SasjsConfiguratorComponent } from './sections/sasjs-configurator/sasjs-configurator.component'
 import { ClarityModule } from '@clr/angular'
 import { FormsModule } from '@angular/forms'
@@ -12,7 +11,6 @@ import { DeployRoutingModule } from './deploy-routing.module'
   declarations: [
     DeployComponent,
     AutomaticComponent,
-    ManualComponent,
     SasjsConfiguratorComponent
   ],
   imports: [CommonModule, FormsModule, ClarityModule, DeployRoutingModule]

client/src/app/deploy/sections/manual/manual.component.html

@@ -1,229 +0,0 @@
-<div class="card-header position-relative">
-  Configurator
-  <p class="d-inline-block ml-10 mb-10 mt-0">App Location: {{ appLoc }}</p>
-  <button
-    (click)="deleteKeys()"
-    class="btn btn-primary clear-memory-button position-absolute"
-  >
-    Clear memory
-  </button>
-</div>
-<div class="card-block">
-  <!-- <div class="card-title">
-  Client Details
-</div> -->
-
-  @if (needsLogin) {
-    <div id="loginForm" class="d-none">
-      <p class="mb-10">Please log in first</p>
-      <label for="username" class="clr-control-label">Username</label>
-      <div class="mb-10 clr-control-container">
-        <div class="clr-input-wrapper">
-          <input type="text" id="username" class="clr-input" />
-        </div>
-      </div>
-      <label for="password" class="clr-control-label">Password</label>
-      <div class="mb-10 clr-control-container">
-        <div class="clr-input-wrapper">
-          <input type="password" id="password" class="clr-input" />
-        </div>
-      </div>
-      <button class="btn btn-primary d-none" id="loginBtn">Log in</button>
-    </div>
-  }
-
-  @if (!needsLogin) {
-    <form>
-      <div class="clr-form-control">
-        <label for="select-full" class="clr-control-label">Admin group</label>
-        <div class="clr-control-container">
-          <div class="clr-select-wrapper">
-            <select
-              [(ngModel)]="selectedAdminGroup"
-              [ngModelOptions]="{ standalone: true }"
-              name="selectedAdminGroup"
-              id="adminGroupsSelect"
-              class="clr-select"
-            >
-              @for (adminGroup of adminGroups; track adminGroup) {
-                <option [value]="adminGroup.id">
-                  {{ adminGroup.name }}
-                </option>
-              }
-            </select>
-          </div>
-        </div>
-      </div>
-      <div class="clr-form-control">
-        <div [class.hidden]="contextsLoading">
-          <label for="select-full" class="clr-control-label">Context</label>
-          <div class="clr-control-container">
-            <div class="clr-select-wrapper">
-              <select
-                [(ngModel)]="selectedContext"
-                [ngModelOptions]="{ standalone: true }"
-                name="selectedContext"
-                class="clr-select"
-              >
-                @for (context of allContexts; track context) {
-                  <option [value]="context.name">
-                    {{ context.name }}
-                    @if ((context.attributes | json) != '{}') {
-                      <span>( {{ context.attributes.sysUserId }} )</span>
-                    }
-                  </option>
-                }
-              </select>
-            </div>
-          </div>
-          <button
-            (click)="executableContext()"
-            type="button"
-            class="btn btn-icon"
-            id="contexts-btn"
-          >
-            <clr-icon shape="play"></clr-icon>
-          </button>
-        </div>
-        <div [class.hidden]="!contextsLoading" class="d-flex">
-          <span class="spinner spinner-inline mr-10">
-            Loading contexts...
-          </span>
-          <span> Loading contexts... </span>
-        </div>
-        <label for="dcloc" class="mt-20 clr-control-label">DC Loc</label>
-        <div class="mb-10 clr-control-container">
-          <div class="clr-input-wrapper">
-            <input
-              [(ngModel)]="dcPath"
-              [ngModelOptions]="{ standalone: true }"
-              (focusout)="saveDcPath()"
-              name="dcPath"
-              type="text"
-              id="dcloc"
-              class="clr-input"
-            />
-          </div>
-        </div>
-        <p class="mt-10">
-          Select JSON file to upload (json build file preloaded):
-        </p>
-        <div class="d-flex flex-column">
-          <input
-            type="file"
-            (click)="clearUploadInput($event)"
-            (change)="onJsonFileChange($event)"
-          />
-          <!-- <button *ngIf="downloadFileBtn" (click)="downloadSasPrecodeFile()" style="width: 40px; min-width: 0;" class="btn btn-sm btn-icon">
-          <clr-icon shape="download"></clr-icon>
-        </button> -->
-        </div>
-        <div class="mt-20 d-flex align-items-center">
-          <button
-            (click)="executeJson()"
-            class="btn btn-primary"
-            [clrLoading]="isSubmittingJson"
-            id=""
-            class="mt-0"
-          >
-            SUBMIT JSON
-          </button>
-          @if (isJsonSubmitted) {
-            <span>JSON Submitted Successfully</span>
-          }
-          <!-- <span *ngIf="executingScript" class="spinner spinner-inline ml-3">
-        Loading...
-      </span> -->
-        </div>
-        <p class="mt-10">Select SAS file to upload:</p>
-        <div class="d-flex flex-column">
-          <input
-            type="file"
-            (click)="clearUploadInput($event)"
-            (change)="onSasFileChange($event); downloadFileBtn = true"
-          />
-          @if (downloadFileBtn) {
-            <button
-              (click)="downloadSasPrecodeFile()"
-              class="btn btn-sm btn-icon min-w-0 w-40"
-            >
-              <clr-icon shape="download"></clr-icon>
-            </button>
-          }
-        </div>
-        <div class="mt-20 d-flex align-items-center">
-          <button
-            (click)="executeSAS()"
-            class="btn btn-primary"
-            [clrLoading]="executingScript"
-            id="deploy"
-            class="mt-0"
-          >
-            SUBMIT
-          </button>
-          <!-- <span *ngIf="executingScript" class="spinner spinner-inline ml-3">
-      Loading...
-    </span> -->
-        </div>
-      </div>
-      @if (jobLog.length > 0) {
-        <p class="mb-0 mt-10">File execute completed</p>
-        <hr />
-        <div>
-          <button
-            (click)="downloadFile(jobLog, 'execute-script-log', 'txt')"
-            class="btn btn-primary mt-0 mr-20"
-          >
-            Download log
-          </button>
-        </div>
-      }
-      <button
-        [clrLoading]="createDatabaseLoading"
-        (click)="createDatabase()"
-        class="btn btn-primary mt-10"
-      >
-        Create Database
-      </button>
-      @if (makeDataResponse.length > 0) {
-        <p class="mb-0 mt-10">Create Database Completed</p>
-        <hr />
-        @if (makeDataResponse.length > 0) {
-          <div class="log-wrapper">
-            {{ makeDataResponse }}
-          </div>
-        }
-        <button (click)="navigateToHome()" class="btn btn-primary mt-15">
-          Let's get started
-        </button>
-        <button
-          (click)="downloadFile(makeDataResponse, 'create-database-log', 'txt')"
-          class="btn btn-primary mt-15"
-        >
-          Download log
-        </button>
-        <button (click)="validateDeploy()" class="btn btn-primary mt-15">
-          Validate
-        </button>
-        @if (validationState !== 'none' || isValidating) {
-          <div class="validation-bar">
-            @if (isValidating) {
-              <span class="spinner spinner-inline mr-10">
-                Validating deploy...
-              </span>
-              <span> Validating deploy... </span>
-            }
-            @if (!isValidating && validationState === 'error') {
-              <clr-icon shape="exclamation-circle" class="is-error"></clr-icon>
-              <span> Validation failed </span>
-            }
-            @if (!isValidating && validationState === 'success') {
-              <clr-icon shape="check-circle" class="is-success"></clr-icon>
-              <span> Validation succeeded </span>
-            }
-          </div>
-        }
-      }
-    </form>
-  }
-</div>

client/src/app/deploy/sections/manual/manual.component.spec.ts

@@ -1,90 +0,0 @@
-import { ManualComponent } from './manual.component'
-
-// validateDeploy() is fire-and-forget (doesn't return its own promise
-// chain), so tests need to wait for it to settle some other way. A
-// macrotask boundary guarantees every already-queued microtask (regardless
-// of how many .then() hops validateDeploy's chain has) has run first.
-const flushPromiseChain = () => new Promise((resolve) => setTimeout(resolve, 0))
-
-/**
- * validateDeploy() only touches sasService.request, eventService.showInfoModal
- * and loggerService.log - these stubs cover exactly that surface, not the
- * full real services (no TestBed/DI needed, same plain-instantiation-with-
- * stubs precedent as app.service.spec.ts).
- */
-const buildDeps = () => {
-  const sasService: any = {
-    request: jasmine.createSpy('request')
-  }
-  const eventService: any = {
-    showInfoModal: jasmine.createSpy('showInfoModal')
-  }
-  const loggerService: any = {
-    log: jasmine.createSpy('log')
-  }
-  const deployService: any = {}
-
-  return { sasService, eventService, loggerService, deployService }
-}
-
-const buildManualComponent = (deps: ReturnType<typeof buildDeps>) =>
-  new ManualComponent(
-    deps.sasService,
-    deps.eventService,
-    deps.loggerService,
-    deps.deployService
-  )
-
-describe('ManualComponent - validateDeploy', () => {
-  it('sets validationState to success when saslibs is present on a normal object response', async () => {
-    const deps = buildDeps()
-    deps.sasService.request.and.resolveTo({
-      adapterResponse: { saslibs: { SOME_LIB: ['SOME_TABLE'] } }
-    })
-
-    const component = buildManualComponent(deps)
-    component.validateDeploy()
-    await flushPromiseChain()
-
-    expect(component.validationState).toBe('success')
-    expect(deps.eventService.showInfoModal).not.toHaveBeenCalled()
-  })
-
-  it('sets validationState to error with no modal when saslibs is missing from an otherwise normal object response', async () => {
-    const deps = buildDeps()
-    deps.sasService.request.and.resolveTo({
-      adapterResponse: { SYSSITE: 'SITE1' }
-      // saslibs deliberately omitted
-    })
-
-    const component = buildManualComponent(deps)
-    component.validateDeploy()
-    await flushPromiseChain()
-
-    expect(component.validationState).toBe('error')
-    expect(deps.eventService.showInfoModal).not.toHaveBeenCalled()
-  })
-
-  // Same underlying bug as app.service.spec.ts's equivalent test: a
-  // misconfigured Viya computeTasks deployment returns a raw "Job error"
-  // text body, which the adapter resolves as adapterResponse verbatim.
-  it('sets validationState to error and shows the real response text when adapterResponse is a raw string, not an object', async () => {
-    const deps = buildDeps()
-    const rawJobError = [
-      'Job error',
-      'The Compute service could not execute the task c74a707a-8b88-46a5-8f28-9a7694ad5e13 because the context 340cd3eb-72ae is not reusable. Please provide a reusable context or supply the ID of an existing session in the task request.',
-      'path: /compute/tasks',
-      'correlator: dc1a9fda-8f65-4f32-b4d4-2a70e02179b1;1bcb6a4c-8a16-4053-a790-292544c25a03'
-    ].join('\n')
-    deps.sasService.request.and.resolveTo({ adapterResponse: rawJobError })
-
-    const component = buildManualComponent(deps)
-    component.validateDeploy()
-    await flushPromiseChain()
-
-    expect(component.validationState).toBe('error')
-    expect(deps.eventService.showInfoModal).toHaveBeenCalledTimes(1)
-    const [, message] = deps.eventService.showInfoModal.calls.mostRecent().args
-    expect(message).toContain(rawJobError)
-  })
-})

client/src/app/deploy/sections/manual/manual.component.ts

@@ -1,346 +0,0 @@
-import {
-  Component,
-  Input,
-  OnInit,
-  Output,
-  EventEmitter,
-  ViewEncapsulation
-} from '@angular/core'
-import SASjs, { SASjsConfig } from '@sasjs/adapter'
-import { DcAdapterSettings } from 'src/app/models/DcAdapterSettings'
-import { RequestWrapperResponse } from 'src/app/models/request-wrapper/RequestWrapperResponse'
-import { DeployService } from 'src/app/services/deploy.service'
-import { EventService } from 'src/app/services/event.service'
-import { LoggerService } from 'src/app/services/logger.service'
-import { SasService } from 'src/app/services/sas.service'
-import { getMalformedAdapterResponseMessage } from 'src/app/shared/utils/get-malformed-adapter-response-message'
-
-@Component({
-  selector: 'app-manual-deploy',
-  templateUrl: './manual.component.html',
-  styleUrls: ['./manual.component.scss'],
-  encapsulation: ViewEncapsulation.None,
-  standalone: false
-})
-export class ManualComponent implements OnInit {
-  @Input() sasJs!: SASjs
-  @Input() sasJsConfig: SASjsConfig = new SASjsConfig()
-  @Input() dcAdapterSettings: DcAdapterSettings | undefined
-
-  @Output() onNavigateToHome: EventEmitter<any> = new EventEmitter<any>()
-
-  public needsLogin: boolean = false
-  public adminGroups: any = []
-  public allContexts: any = []
-
-  public appLoc: string = ''
-  public dcPath: string = ''
-  public selectedAdminGroup: string = ''
-  public selectedContext: string = ''
-  public jobLog: string = ''
-  public makeDataResponse: string = ''
-  public linesOfCode: any = []
-  public fileName: string = ''
-  public preloadedFile: boolean = true
-  public executeSASEnabled: boolean = false
-
-  public contextsLoading: boolean = false
-  public createDatabaseLoading: boolean = false
-  public executingScript: boolean = false
-  public downloadFileBtn: boolean = false
-  public isValidating: boolean = false
-  public jsonFile: any = null
-  public isSubmittingJson: boolean = false
-  public isJsonSubmitted: boolean = false
-  public validationState: string = 'none'
-
-  constructor(
-    private sasService: SasService,
-    private eventService: EventService,
-    private loggerService: LoggerService,
-    private deployService: DeployService
-  ) {}
-
-  ngOnInit(): void {}
-
-  /**
-   * FIXME: Remove
-   */
-  public async executableContext() {
-    // getExecutableContexts now need AuthConfig parameter which we don't have on web
-    // this.contextsLoading = true
-    // let contexts = await this.sasJs.getExecutableContexts()
-    // this.allContexts = contexts
-    // if (contexts.length === 0) {
-    //   this.selectedContext = ''
-    // } else {
-    //   this.selectedContext = this.allContexts[0].name
-    // }
-    // this.contextsLoading = false
-  }
-
-  /**
-   * Removes sas.json file attached to the input
-   */
-  public clearUploadInput(event: Event) {
-    this.deployService.clearUploadInput(event)
-  }
-
-  /**
-   * Reads attached SAS file to be sent to sas for execution (backend deploy)
-   */
-  public onSasFileChange(event: any) {
-    this.preloadedFile = false
-
-    let file = event.target.files[0]
-    this.fileName = file.name
-
-    let fileReader = new FileReader()
-
-    fileReader.onload = () => {
-      if (fileReader.result) {
-        this.linesOfCode = (fileReader.result as string).split('\n')
-        this.linesOfCode = this.linesOfCode.filter((el: string) => {
-          return el !== '' && el !== null
-        })
-
-        this.executeSASEnabled = true
-
-        this.addPrecodeLines()
-      }
-    }
-
-    fileReader.readAsText(file)
-  }
-
-  /**
-   * Reads attached JSON file to be sent to sas for execution (backend deploy)
-   */
-  public async onJsonFileChange(event: any) {
-    let file = event.target.files[0]
-
-    this.jsonFile = await this.deployService.readFile(file)
-  }
-
-  /**
-   * Appending precode lines to the attached sas or json file for backend deploy
-   */
-  public addPrecodeLines() {
-    let headerLines = [
-      `%let context=${this.selectedContext};`,
-      `%let appLoc=${this.appLoc};`,
-      `%let admin=${this.selectedAdminGroup};`,
-      `%let dcpath=${this.dcPath};`
-    ]
-
-    this.linesOfCode.unshift(...headerLines)
-  }
-
-  /**
-   * Downloadng file with precode included
-   */
-  public downloadSasPrecodeFile() {
-    let linesAsText = this.linesOfCode.join('\n')
-    let filename = this.fileName.split('.')[0]
-
-    this.downloadFile(linesAsText, filename, 'sas')
-  }
-
-  /**
-   * Used for downloading log and repsonse as a file
-   */
-  public downloadFile(
-    content: any,
-    filename: string,
-    extension: string = 'txt'
-  ) {
-    this.deployService.downloadFile(content, filename, extension)
-  }
-
-  /**
-   * Saving dcpath to localstorage
-   * FIXME: maybe it'snot necessary
-   */
-  public saveDcPath() {
-    localStorage.setItem('deploy_dc_loc', this.dcPath)
-  }
-
-  /**
-   * Send sas.json to be executed (deploying backend)
-   */
-  public async executeJson() {
-    this.isSubmittingJson = true
-
-    try {
-      let uploadJsonFile = await this.sasJs.deployServicePack(
-        this.jsonFile,
-        this.dcAdapterSettings?.appLoc || '',
-        undefined,
-        undefined,
-        true
-      )
-
-      this.isJsonSubmitted = true
-    } catch (ex: any) {
-      let textEx = ''
-
-      if (typeof ex.message !== 'string') {
-        textEx = JSON.stringify(ex).replace(/\\/gm, '')
-      } else {
-        textEx = ex.message
-      }
-
-      this.eventService.showInfoModal(
-        'Deploy error',
-        `Exception: \n ${textEx !== '' ? textEx : ex}`
-      )
-
-      return
-    }
-
-    this.isSubmittingJson = false
-  }
-
-  /**
-   * Send sas file to be executed (deploying backend)
-   */
-  public async executeSAS() {
-    this.executingScript = true
-    this.jobLog = ''
-    this.makeDataResponse = ''
-
-    try {
-      let executeResponse = await this.sasJs.executeScript({
-        fileName: this.fileName,
-        linesOfCode: this.linesOfCode,
-        contextName: this.selectedContext
-      })
-
-      this.loggerService.log(executeResponse)
-
-      if (typeof executeResponse.log === 'string') {
-        executeResponse.log = JSON.parse(executeResponse.log)
-      }
-
-      if (executeResponse.jobStatus === 'error') {
-        alert('Error!')
-      } else {
-        this.jobLog = executeResponse.log.items
-          ? executeResponse.log.items.map((i: any) => i.line).join('\n')
-          : JSON.stringify(executeResponse.log)
-      }
-
-      this.executingScript = false
-    } catch (err) {
-      this.executingScript = false
-    }
-  }
-
-  /**
-   * Running makedata service
-   * @param newTab open and run in new tab
-   */
-  public createDatabase(newTab: boolean = true) {
-    if (newTab) {
-      // Debug is ALWAYS enabled for makedata — it runs only once during
-      // deployment and the full log is needed to diagnose any issues.
-      const _debug = '&_debug=131'
-      let url =
-        this.sasService.getSasjsConfig().serverUrl +
-          '/SASJobExecution/?_program=' +
-          this.dcAdapterSettings?.appLoc ||
-        '' +
-          '/admin/makedata' +
-          '&ADMIN=' +
-          this.selectedAdminGroup +
-          '&DCPATH=' +
-          this.dcPath +
-          _debug
-
-      window.open(url, '_blank')
-
-      return
-    }
-
-    this.createDatabaseLoading = true
-
-    let data = {
-      fromjs: [
-        {
-          ADMIN: this.selectedAdminGroup,
-          DCPATH: this.dcPath
-        }
-      ]
-    }
-
-    /**
-     * We are overriding default `sasjsConfig` object fields with this object fields.
-     * Here we want to run this request using original WEB method.
-     * contextName: null is the MUST field for it.
-     */
-    let overrideConfig = {
-      useComputeApi: null,
-      contextName: this.sasJsConfig.contextName,
-      debug: true
-    }
-
-    this.sasJs
-      .request(`services/admin/makedata`, data, overrideConfig, () => {
-        this.sasService.shouldLogin.next(true)
-      })
-      .then((res: any) => {
-        try {
-          this.makeDataResponse = JSON.stringify(res)
-        } catch {
-          this.makeDataResponse = res
-        }
-
-        this.createDatabaseLoading = false
-      })
-      .catch((err: any) => {
-        this.createDatabaseLoading = false
-
-        try {
-          this.makeDataResponse = JSON.stringify(err)
-        } catch {
-          this.makeDataResponse = err
-        }
-      })
-  }
-
-  public navigateToHome() {
-    this.onNavigateToHome.emit()
-  }
-
-  public validateDeploy() {
-    this.isValidating = true
-
-    this.sasService
-      .request('public/startupservice', null)
-      .then((res: RequestWrapperResponse) => {
-        this.loggerService.log(res.adapterResponse)
-
-        const malformedMessage = getMalformedAdapterResponseMessage(
-          res.adapterResponse
-        )
-        if (malformedMessage) {
-          this.validationState = 'error'
-          this.eventService.showInfoModal('Error', malformedMessage)
-        } else if (res.adapterResponse.saslibs) {
-          this.validationState = 'success'
-        } else {
-          this.validationState = 'error'
-        }
-
-        this.isValidating = false
-      })
-      .catch((err: any) => {
-        this.isValidating = false
-        this.validationState = 'error'
-      })
-  }
-
-  public deleteKeys() {
-    localStorage.removeItem('deploy_dc_loc')
-  }
-}

client/src/app/editor/editor.component.html

@@ -245,6 +245,25 @@
               }
             </clr-tooltip>
 
+            @if (truncationNote) {
+              <clr-tooltip>
+                <clr-icon
+                  clrTooltipTrigger
+                  shape="warning"
+                  aria-label="The selection was truncated"
+                  class="color-orange truncation-warning"
+                  size="20"
+                ></clr-icon>
+                <clr-tooltip-content
+                  clrPosition="right"
+                  clrSize="lg"
+                  *clrIfOpen
+                >
+                  {{ truncationNote }}
+                </clr-tooltip-content>
+              </clr-tooltip>
+            }
+
             @if (dataSource) {
               @if (!zeroFilterRows) {
                 ({{ dataSource.length | thousandSeparator: ',' }}

client/src/app/editor/editor.component.ts

@@ -89,6 +89,11 @@ import { withoutEditStatus } from './utils/withoutEditStatus'
 import { getEditStatusSymbol } from './utils/getEditStatusSymbol'
 import { normalizeSortConfig } from './utils/normalizeSortConfig'
 import { EDIT_STATUS_COLUMN_NAME } from '../shared/dc-validator/utils/editStatusColumnRule'
+import { DELETE_RECORD_COLUMN_NAME } from '../shared/dc-validator/utils/deleteRecordColumnRule'
+import {
+  prepareDeleteRecordColumn,
+  seedDeleteRecordColumn
+} from '../shared/dc-validator/utils/deleteRecordColumn'
 import {
   errorRenderer,
   noSpinnerRenderer,
@@ -155,6 +160,9 @@ export class EditorComponent
   dsmeta: DSMeta[] = []
   versions: Version[] = []
   dsNote = ''
+  /** Set when getdata served a truncated selection - shown as a warning
+   * against the table name, with the reason in a tooltip. */
+  truncationNote = ''
 
   viewboxes = false
 
@@ -567,6 +575,8 @@ export class EditorComponent
   public hasBaseDropZoneOver = false
   public hasAnotherDropZoneOver = false
   public headerPks: string[] = []
+  /** Derived (computed) columns from MPE_DERIVED - read-only, and marked as such */
+  public derivedVars: string[] = []
   public columnLevelSecurityFlag = false
   public dateTimeHeaders: string[] = []
   public timeHeaders: string[] = []
@@ -1081,6 +1091,20 @@ export class EditorComponent
     return colRules?.readOnly
   }
 
+  /**
+   * A derived (computed) column - configured in MPE_DERIVED, created at read
+   * time. Read-only, and styled apart from an ordinary read-only column so the
+   * user can see that there is nothing to edit rather than that they may not.
+   */
+  isDerivedCol(col: string | number) {
+    const name =
+      typeof col === 'number'
+        ? (this.hotInstance?.colToProp(col) as string) || this.columnHeader[col]
+        : col
+
+    return !!name && this.derivedVars.includes(name.toUpperCase())
+  }
+
   isColHeader(col: string) {
     return this.headerArray.indexOf(col.toUpperCase()) > -1
   }
@@ -1355,8 +1379,14 @@ export class EditorComponent
             colSpecs.TYPE === 'num' &&
             !colSpecs.DDTYPE.includes('TIME') &&
             !colSpecs.DDTYPE.includes('DATE')
-          )
-            row[colKey] = parseInt(row[colKey])
+          ) {
+            const numericValue = parseFloat(row[colKey])
+
+            // A cell the file parser could not read as a number stays as it
+            // was: the editor's numeric validation then flags it as text,
+            // where a NaN would render as an empty cell.
+            if (!Number.isNaN(numericValue)) row[colKey] = numericValue
+          }
         }
       }
     }
@@ -2743,17 +2773,24 @@ export class EditorComponent
             this.disableSubmit = true
             this.submit = false
 
-            const errorText =
-              typeof err.adapterRespnse === 'string'
-                ? err.adapterRespnse
-                : JSON.stringify(err.adapterRespnse)
+            /* sasService has already shown the SAS message when it rejected
+               (the abort, or the adapter's error), so reporting it again here
+               stacked a second modal on top of the first - and read
+               'undefined' because it looked for a property that does not
+               exist.  Only speak up when nothing was shown. */
+            if (!err?.errorShown) {
+              const errorText =
+                typeof err?.adapterResponse?.error === 'string'
+                  ? err.adapterResponse.error
+                  : JSON.stringify(err?.adapterResponse ?? err)
 
-            this.eventService.showAbortModal(
-              'editors/stagedata',
-              `Submit request failed\n${errorText}`,
-              null,
-              'Submit error'
-            )
+              this.eventService.showAbortModal(
+                'editors/stagedata',
+                `Submit request failed\n${errorText}`,
+                null,
+                'Submit error'
+              )
+            }
 
             return false
           })
@@ -3519,14 +3556,24 @@ export class EditorComponent
           // A synchronous throw inside initSetup (called from the .then()
           // above) lands here too, not just a network failure - log it, or
           // the table just renders hidden with no clue why.
-          // eslint-disable-next-line no-console
-          console.error('editors/getdata failed:', err)
-          this.getdataError = true
-          this.tableTrue = true
+          this.handleGetdataFailure(err)
         })
     }
   }
 
+  /**
+   * Puts the page into its getdata error state. Shared by the callService
+   * .catch and initSetup's own retry branch: a throw from the retry runs on a
+   * bare setTimeout, outside that promise chain, so it would otherwise never
+   * reach the .catch and the user would be left on the loading spinner.
+   */
+  private handleGetdataFailure(err: any) {
+    // eslint-disable-next-line no-console
+    console.error('editors/getdata failed:', err)
+    this.getdataError = true
+    this.tableTrue = true
+  }
+
   /**
    * VA mode, initial (unfiltered) route only: briefly wait for VA's buffered
    * first message and, if it already carries a filter, save that filter and
@@ -3875,13 +3922,28 @@ export class EditorComponent
 
     this.hotInstance = this.hotTableComponent?.hotInstance!
     if (!this.hotInstance) {
-      // Retry init, don't permanently abort
+      // Retry init, don't permanently abort. A throw from the retry runs on a
+      // bare setTimeout rather than in the caller's promise chain, so it would
+      // never reach that .catch - surface it through the same handler here.
       if (attempt < 60) {
-        setTimeout(() => this.initSetup(response, attempt + 1), 50)
+        setTimeout(() => {
+          try {
+            this.initSetup(response, attempt + 1)
+          } catch (err) {
+            this.handleGetdataFailure(err)
+          }
+        }, 50)
       }
       return
     }
 
+    // The delete column is client-owned (see deleteRecordColumnRule.ts):
+    // this asserts getdata did not send it, then prepends it to COLHEADERS so
+    // it is the grid's first column. A throw here lands in the caller's
+    // .catch - or, on a retry, the branch above - which sets getdataError and
+    // shows the error state.
+    prepareDeleteRecordColumn(response)
+
     this.cols = response.data.cols
     this.dsmeta = response.data.dsmeta
     this.versions = response.data.versions || []
@@ -3900,6 +3962,14 @@ export class EditorComponent
       this.dsNote = ''
     }
 
+    // getdata serves a selection over DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT
+    // truncated to the first rows within the limit, and carries the reason
+    // here.  Empty means the whole selection is in the grid - and a blank
+    // character arrives as a single blank (mp_jsonout cannot emit a zero-length
+    // string, since SAS has none), so trim it: whitespace is "no note", not a
+    // warning with an empty tooltip.
+    this.truncationNote = (response.data.sasparams[0].TRUNC_MSG || '').trim()
+
     const hot = this.hotInstance
 
     const approvers: Approver[] = response.data.approvers
@@ -3920,6 +3990,12 @@ export class EditorComponent
 
     this.headerColumns = response.data.sasparams[0].COLHEADERS.split(',')
     this.headerPks = response.data.sasparams[0].PK.split(' ')
+    // Derived (computed) columns, named by getdata from MPE_DERIVED - read-only
+    // through a READONLY DQ rule, and marked in the grid so they read as
+    // computed rather than merely non-editable.
+    this.derivedVars = (response.data.sasparams[0].DERIVEDVARS || '')
+      .split(' ')
+      .filter((col: string) => !!col)
     this.columnLevelSecurityFlag = !!response.data.sasparams[0].CLS_FLAG
 
     if (this.columnLevelSecurityFlag)
@@ -3931,9 +4007,9 @@ export class EditorComponent
 
     this.checkEmptyRowWhenFilter()
 
-    if (this.headerColumns.indexOf('_____DELETE__THIS__RECORD_____') !== -1) {
+    if (this.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME) !== -1) {
       this.headerColumns[
-        this.headerColumns.indexOf('_____DELETE__THIS__RECORD_____')
+        this.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME)
       ] = 'Delete?'
     }
 
@@ -3998,6 +4074,10 @@ export class EditorComponent
       row[EDIT_STATUS_COLUMN_NAME] = 'U'
     }
 
+    // The delete flag is client-owned too (see deleteRecordColumnRule.ts) -
+    // getdata no longer sends the column, so every row starts unmarked.
+    seedDeleteRecordColumn(this.dataSource)
+
     // Seed HARDFORMULA/SOFTFORMULA columns with their computed formula
     // string per row. dataSourceUnchanged isn't populated yet this early in
     // a fresh load (only the excel-preview/add-row flows set it) - at this
@@ -4290,9 +4370,37 @@ export class EditorComponent
           // header columns styling - primary keys
           const isPKCol = column && this.isColPk(column)
           const isReadonlyCol = column && this.isReadonlyCol(column)
+          const isDerivedCol = column && this.isDerivedCol(column)
 
           if (isPKCol) th.classList.add('primaryKeyHeaderStyle')
-          if (isReadonlyCol && !isPKCol) th.classList.add('readonlyCell')
+          if (isReadonlyCol && !isPKCol && !isDerivedCol) {
+            th.classList.add('readonlyCell')
+          }
+
+          // a derived column is computed, not merely locked - give it its own
+          // hue and an 'fx' marker rather than the read-only dimming
+          if (isDerivedCol) {
+            th.classList.add('dc-derived-header')
+            // the label says what the column is - for a derived variable that
+            // is the configured VAR_LABEL, or the generated name=expression
+            // when the administrator left it blank.  A label that merely
+            // repeats the name adds nothing, so it falls back to the rule.
+            const derivedLabel = (this.cols || [])
+              .filter((c: any) => c.NAME === column)
+              .map((c: any) => String(c.LABEL || '').trim())
+              .find(
+                (l: string) =>
+                  l.length > 0 &&
+                  l.toUpperCase() !== String(column).toUpperCase()
+              )
+            th.title = derivedLabel
+              ? `${derivedLabel} - derived variable, computed at read time and cannot be edited`
+              : 'Derived variable - computed at read time and cannot be edited'
+            const firstDerivedCol = this.columnHeader.findIndex(
+              (name: string) => this.isDerivedCol(name)
+            )
+            if (col === firstDerivedCol) th.classList.add('dc-derived-first')
+          }
 
           // Remove header arrow from Delete column
           if (col === 0) {
@@ -4308,6 +4416,7 @@ export class EditorComponent
           cellProperties: Handsontable.CellProperties
         ) => {
           const isReadonlyCol = col && this.isReadonlyCol(col)
+          const isDerivedCol = col && this.isDerivedCol(col)
 
           // Check if this cell should be marked as invalid due to duplicate primary key values
           // Only applies to primary key columns (col 1 through readOnlyFields)
@@ -4331,7 +4440,26 @@ export class EditorComponent
           }
 
           // Add readonlyCell class for readonly columns to maintain original styling
-          if (isReadonlyCol && !classes.includes('readonlyCell')) {
+          if (isDerivedCol) {
+            // a derived cell carries its own tint; the read-only dimming would
+            // wash it out, and "greyed out" is a different message from
+            // "computed here"
+            classes = classes.filter((c: string) => c !== 'readonlyCell')
+            if (!classes.includes('dc-derived-cell')) {
+              classes.push('dc-derived-cell')
+            }
+            // derived columns are appended after the physical ones - mark the
+            // start of the block so it reads as one group
+            const firstDerivedCol = this.columnHeader.findIndex(
+              (name: string) => this.isDerivedCol(name)
+            )
+            if (
+              col === firstDerivedCol &&
+              !classes.includes('dc-derived-first')
+            ) {
+              classes.push('dc-derived-first')
+            }
+          } else if (isReadonlyCol && !classes.includes('readonlyCell')) {
             classes.push('readonlyCell')
           }
 

client/src/app/models/DcAdapterSettings.ts

@@ -1,7 +1,5 @@
 import { SASjsConfig } from '@sasjs/adapter'
 
 export interface DcAdapterSettings extends Partial<SASjsConfig> {
-  adminGroup: string
   dcPath: string
-  hotLicenceKey: string
 }

client/src/app/models/TableData.ts

@@ -47,6 +47,15 @@ export interface SASParam {
   PKCNT: number
   RK_FLAG: number
   TMVARS: string
+  /** 1 when getdata cut the selection down to the row / cell limit */
+  TRUNCATED?: number
+  /** Reason shown in the tooltip when TRUNCATED is 1, else empty */
+  TRUNC_MSG?: string
+  /**
+   * Space separated list of the derived (computed) columns from MPE_DERIVED,
+   * upper case. Empty when the table has none.
+   */
+  DERIVEDVARS?: string
 }
 
 export interface ExcelRule {

client/src/app/multi-dataset/multi-dataset.component.ts

@@ -18,6 +18,11 @@ import {
 import { globals } from '../_globals'
 import { EditorsGetDataServiceResponse } from '../models/sas/editors-getdata.model'
 import { DcValidator } from '../shared/dc-validator/dc-validator'
+import {
+  prepareDeleteRecordColumn,
+  seedDeleteRecordColumn
+} from '../shared/dc-validator/utils/deleteRecordColumn'
+import { DELETE_RECORD_COLUMN_NAME } from '../shared/dc-validator/utils/deleteRecordColumnRule'
 import { ExcelRule } from '../models/TableData'
 import { HotTableInterface } from '../models/HotTable.interface'
 import { Col } from '../shared/dc-validator/models/col.model'
@@ -360,7 +365,21 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
 
       this.uploadLoading = false
 
-      const datasetObjects = this.buildDatasetsObjects(datasets)
+      // buildDatasetsObjects throws when a getdata response carries the
+      // reserved delete-record column (see deleteRecordColumn.ts). This chain
+      // has no .catch, so an uncaught throw here would become an unhandled
+      // rejection that aborts the loop with nothing shown. Surface it on
+      // screen, the way the editor flow refuses the same response, rather
+      // than leaving the user on an empty table selection with the file
+      // still attached.
+      let datasetObjects: DatasetsObject[] = []
+      try {
+        datasetObjects = this.buildDatasetsObjects(datasets)
+      } catch (err) {
+        console.warn('Error building dataset objects', err)
+        this.eventService.showInfoModal('Upload error', (err as Error).message)
+        return
+      }
 
       datasetObjects.forEach((datasetObject) => {
         this.parsedDatasets.push({
@@ -849,6 +868,21 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
       return this.sasStoreService
         .callService(myParams, 'SASControlTable', 'editors/getdata', libds)
         .then((res: EditorsGetDataServiceResponse) => {
+          // getdata serves a selection over DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT
+          // truncated, rather than aborting the way it used to. This flow compares
+          // the whole table against the upload, so a partial read would compute
+          // the changes against the wrong starting state - refuse the table
+          // instead of submitting differences nobody asked for.
+          const sasParams = res?.data?.sasparams?.[0]
+
+          if (sasParams && Number(sasParams.TRUNCATED) === 1) {
+            this.eventService.showInfoModal(
+              'Upload error',
+              `${libds} holds more than the EDIT limit allows (${sasParams.TRUNC_MSG}), so it cannot be compared with the upload. Raise DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT, or submit the table without the multi-table upload.`
+            )
+            return undefined
+          }
+
           return res
         })
         .catch((err: any) => {
@@ -1074,6 +1108,12 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
 
     for (let response of responses) {
       if (response.data) {
+        // The delete column is client-owned (see deleteRecordColumnRule.ts):
+        // asserts getdata did not send it, then prepends it to COLHEADERS so
+        // it is index 0 for everything below.
+        prepareDeleteRecordColumn(response)
+        seedDeleteRecordColumn(response.data.sasdata)
+
         const datasetObject: DatasetsObject = {
           ...response,
           hotTable: {
@@ -1097,14 +1137,10 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
         datasetObject.headerPks = response.data.sasparams[0].PK.split(' ')
 
         if (
-          datasetObject.headerColumns.indexOf(
-            '_____DELETE__THIS__RECORD_____'
-          ) !== -1
+          datasetObject.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME) !== -1
         ) {
           datasetObject.headerColumns[
-            datasetObject.headerColumns.indexOf(
-              '_____DELETE__THIS__RECORD_____'
-            )
+            datasetObject.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME)
           ] = 'Delete?'
         }
 

client/src/app/query/query.component.spec.ts

@@ -0,0 +1,110 @@
+import { QueryComponent } from './query.component'
+import { HelperService } from '../services/helper.service'
+
+/**
+ * dynamicWhereClauseCreator builds the prefilter that getcolvals sends when the
+ * dynamic where clause is on, and it composes a temporal value into the same
+ * SAS literal the applied filter carries.  The value it prefilters with comes
+ * from the *other* clauses in the query, so the clause under test sits at index
+ * 1 and index 0 is skipped.
+ */
+describe('QueryComponent - dynamic where clause', () => {
+  const buildComponent = (elements: any[]) => {
+    const component = new QueryComponent(
+      {} as any,
+      { catchResponseError: () => undefined } as any,
+      new HelperService(null as any),
+      { detectChanges: () => undefined } as any
+    )
+
+    component.clauses = {
+      queryObj: [{ clauseLogic: 'AND', elements }],
+      groupLogic: 'AND'
+    } as any
+
+    return component
+  }
+
+  const temporalClause = (over: any = {}) => ({
+    variable: 'SOME_DATE',
+    operator: '=',
+    value: '',
+    values: [],
+    type: 'num',
+    ddtype: 'DATE',
+    valueVariable: false,
+    ...over
+  })
+
+  it('sends an IN entry as the literal for its unformatted value', () => {
+    const component = buildComponent([
+      temporalClause(),
+      temporalClause({
+        operator: 'IN',
+        value: [{ checked: true, val: '12FEB1960' }],
+        values: [{ formatted: '12FEB1960', unformatted: 42 }]
+      })
+    ])
+
+    const table = component.dynamicWhereClauseCreator(0, 0)
+
+    expect(table.length).toBe(1)
+    expect(table[0].RAW_VALUE).toBe("('12FEB1960'd)")
+  })
+
+  it('maps both ends of a BETWEEN picked from the value list', () => {
+    const component = buildComponent([
+      temporalClause(),
+      temporalClause({
+        operator: 'BETWEEN',
+        value: ['12FEB1960', '13FEB1960'],
+        values: [
+          { formatted: '12FEB1960', unformatted: 42 },
+          { formatted: '13FEB1960', unformatted: 43 }
+        ]
+      })
+    ])
+
+    const table = component.dynamicWhereClauseCreator(0, 0)
+
+    expect(table.length).toBe(1)
+    expect(table[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
+  })
+
+  it('leaves an already-unformatted BETWEEN alone', () => {
+    const component = buildComponent([
+      temporalClause(),
+      temporalClause({
+        operator: 'BETWEEN',
+        value: [42, 43],
+        values: [
+          { formatted: '12FEB1960', unformatted: 42 },
+          { formatted: '13FEB1960', unformatted: 43 }
+        ]
+      })
+    ])
+
+    const table = component.dynamicWhereClauseCreator(0, 0)
+
+    expect(table[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
+  })
+
+  it('emits the unformatted value in the non-literal IN fallback', () => {
+    const component = buildComponent([
+      temporalClause(),
+      {
+        variable: 'SOME_CHAR',
+        operator: 'IN',
+        value: [{ checked: true, val: 'ABC ' }],
+        values: [{ formatted: 'ABC ', unformatted: 'ABC' }],
+        type: 'char',
+        ddtype: 'CHAR',
+        valueVariable: false
+      }
+    ])
+
+    const table = component.dynamicWhereClauseCreator(0, 0)
+
+    expect(table[0].RAW_VALUE).toBe("('ABC')")
+  })
+})

client/src/app/query/query.component.ts

@@ -425,35 +425,57 @@ export class QueryComponent
             continue
 
           let rawValue = ''
-          const operator =
-            this.clauses.queryObj[groupInd].elements[clauseInd].operator
-          const variable =
-            this.clauses.queryObj[groupInd].elements[clauseInd].variable
+          const element = this.clauses.queryObj[groupInd].elements[clauseInd]
+          const operator = element.operator
+          const variable = element.variable
+          /* a temporal value is sent as a SAS literal (eg '20APR2042'd), the
+             same way the applied filter carries it */
+          const temporalValue = (value: any) =>
+            element.valueVariable
+              ? null
+              : this.helperService.sasTemporalLiteral(value, element.ddtype)
+          /* the dropdowns and the IN modal hold the *formatted* value (the
+             soft-select option value, and setINValues), so a picked value is
+             mapped back to the unformatted one the filter needs */
+          const unformattedValue = (value: any) =>
+            (element.values || []).find((v: any) => v.formatted === value)
+              ?.unformatted ?? value
 
           if (operator === 'BETWEEN') {
-            rawValue = `${this.clauses.queryObj[groupInd].elements[clauseInd].value[0]} AND ${this.clauses.queryObj[groupInd][clauseInd].value[1]}`
+            const from = unformattedValue(element.value[0])
+            const to = unformattedValue(element.value[1])
+            rawValue = `${temporalValue(from) ?? from} AND ${
+              temporalValue(to) ?? to
+            }`
           } else if (operator === 'IN' || operator === 'NOT IN') {
-            this.clauses.queryObj[groupInd][clauseInd].value.forEach(
-              (element: any) => {
-                if (element.checked) {
-                  let value = element.val
+            element.value.forEach((item: any) => {
+              if (item.checked) {
+                /* map once, and use the unformatted value in every branch - the
+                   literal is built from it, and so are the fallbacks, so a
+                   formatted label can never reach RAW_VALUE */
+                const value = unformattedValue(item.val)
+                const literal = temporalValue(value)
 
-                  if (typeof value === 'string' && !isSpecialMissing(value)) {
-                    if (rawValue.length > 0) {
-                      rawValue += `,'${value}'`
-                    } else {
-                      rawValue = `('${value}'`
-                    }
+                if (literal) {
+                  rawValue += (rawValue.length > 0 ? ',' : '(') + literal
+                } else if (
+                  typeof value === 'string' &&
+                  !isSpecialMissing(value)
+                ) {
+                  if (rawValue.length > 0) {
+                    rawValue += `,'${value}'`
                   } else {
-                    if (rawValue.length > 0) {
-                      rawValue += `,${value}`
-                    } else {
-                      rawValue = `(${value}`
-                    }
+                    rawValue = `('${value}'`
+                  }
+                } else {
+                  if (rawValue.length > 0) {
+                    rawValue += `,${value}`
+                  } else {
+                    rawValue = `(${value}`
                   }
                 }
               }
-            )
+            })
 
             if (rawValue.length > 0) {
               rawValue += ')'
@@ -470,11 +492,12 @@ export class QueryComponent
               rawValue = '.'
             }
           } else {
-            if (
-              this.clauses.queryObj[groupInd].elements[clauseInd].type ===
-              'char'
-            ) {
-              rawValue = `'${this.clauses.queryObj[groupInd].elements[clauseInd].value}'`
+            const literal = temporalValue(element.value)
+
+            if (literal) {
+              rawValue = literal
+            } else if (element.type === 'char') {
+              rawValue = `'${element.value}'`
             }
           }
 

client/src/app/services/app.service.spec.ts

@@ -170,9 +170,11 @@ describe('AppService - startup retry', () => {
 
     expect(deps.sasService.request).toHaveBeenCalledTimes(1)
     expect(deps.eventService.showInfoModal).toHaveBeenCalledTimes(1)
-    const [, message] = deps.eventService.showInfoModal.calls.mostRecent().args
+    const [, message, sasService] =
+      deps.eventService.showInfoModal.calls.mostRecent().args
     expect(message).toContain(rawJobError)
     expect(message).not.toContain('Globvars, Sasdatasets, Saslibs, XLMaps')
+    expect(sasService).toBe('public/startupservice')
     expect(deps.licenceService.isAppActivated.value).toBeFalse()
   })
 })

client/src/app/services/app.service.ts

@@ -107,7 +107,11 @@ export class AppService {
             malformedMessage
           )
           this.startupCheckService.updateStep(2, 'error', 'Malformed response')
-          this.eventService.showInfoModal('Error', malformedMessage)
+          this.eventService.showInfoModal(
+            'Error',
+            malformedMessage,
+            'public/startupservice'
+          )
           this.licenceService.isAppActivated.next(false)
 
           return
@@ -139,7 +143,8 @@ export class AppService {
           )
           this.eventService.showInfoModal(
             'Error',
-            `${missingProps.join(', ')} are not present in the startupservice`
+            `${missingProps.join(', ')} are not present in the startupservice`,
+            'public/startupservice'
           )
           this.licenceService.isAppActivated.next(false)
 
@@ -221,7 +226,8 @@ export class AppService {
         this.startupCheckService.updateStep(2, 'error', 'Request failed')
         this.eventService.showInfoModal(
           'Error',
-          'There is an issue with startupservice response'
+          'There is an issue with startupservice response',
+          'public/startupservice'
         )
         this.licenceService.isAppActivated.next(false)
       })

client/src/app/services/event.service.ts

@@ -42,11 +42,15 @@ export class EventService {
     this.onDemoLimitModalShow.emit(featureName)
   }
 
-  public showInfoModal(modalTitle: string = 'Info', message: string) {
+  public showInfoModal(
+    modalTitle: string = 'Info',
+    message: string,
+    sasService: string | null = null
+  ) {
     let infoModal: InfoModal = {
       modalTitle,
       message,
-      sasService: '',
+      sasService,
       details: null
     }
 

client/src/app/services/helper.service.spec.ts

@@ -46,3 +46,72 @@ describe('HelperService - convertArrayValues', () => {
     expect(result.length).toBe(3)
   })
 })
+
+describe('HelperService - sasTemporalLiteral', () => {
+  let service: HelperService
+
+  beforeEach(() => {
+    service = new HelperService(null as any)
+  })
+
+  it('renders a SAS date value as a date literal', () => {
+    expect(service.sasTemporalLiteral(42, 'DATE')).toBe("'12FEB1960'd")
+    expect(service.sasTemporalLiteral(0, 'DATE')).toBe("'01JAN1960'd")
+    expect(service.sasTemporalLiteral(34322, 'DATE')).toBe("'20DEC2053'd")
+  })
+
+  it('renders a date before 1960', () => {
+    expect(service.sasTemporalLiteral(-1, 'DATE')).toBe("'31DEC1959'd")
+  })
+
+  it('renders a SAS datetime value as a datetime literal', () => {
+    expect(service.sasTemporalLiteral(42, 'DATETIME')).toBe(
+      "'01JAN1960:00:00:42'dt"
+    )
+    expect(service.sasTemporalLiteral(86400, 'DATETIME')).toBe(
+      "'02JAN1960:00:00:00'dt"
+    )
+    expect(service.sasTemporalLiteral(31536001, 'DATETIME')).toBe(
+      "'31DEC1960:00:00:01'dt"
+    )
+  })
+
+  it('rounds double noise onto the second', () => {
+    expect(service.sasTemporalLiteral(1.9999999999999, 'TIME')).toBe(
+      "'00:00:02't"
+    )
+    expect(service.sasTemporalLiteral(42.000000000000004, 'DATETIME')).toBe(
+      "'01JAN1960:00:00:42'dt"
+    )
+  })
+
+  it('returns null rather than round a genuinely fractional value', () => {
+    expect(service.sasTemporalLiteral(42.5, 'DATETIME')).toBeNull()
+    expect(service.sasTemporalLiteral(42.5, 'TIME')).toBeNull()
+    expect(service.sasTemporalLiteral(42.3, 'TIME')).toBeNull()
+  })
+
+  it('renders a SAS time value as a time literal', () => {
+    expect(service.sasTemporalLiteral(0, 'TIME')).toBe("'00:00:00't")
+    expect(service.sasTemporalLiteral(42, 'TIME')).toBe("'00:00:42't")
+    expect(service.sasTemporalLiteral(3661, 'TIME')).toBe("'01:01:01't")
+  })
+
+  it('accepts a value that arrives as a string', () => {
+    expect(service.sasTemporalLiteral('42', 'DATE')).toBe("'12FEB1960'd")
+  })
+
+  it('returns null when the value cannot be a literal', () => {
+    // not temporal
+    expect(service.sasTemporalLiteral(42, 'N')).toBeNull()
+    expect(service.sasTemporalLiteral(42, 'C')).toBeNull()
+    expect(service.sasTemporalLiteral(42, '')).toBeNull()
+    // not a number
+    expect(service.sasTemporalLiteral('12FEB1960', 'DATE')).toBeNull()
+    expect(service.sasTemporalLiteral('', 'DATE')).toBeNull()
+    expect(service.sasTemporalLiteral(null, 'DATE')).toBeNull()
+    expect(service.sasTemporalLiteral(undefined, 'DATE')).toBeNull()
+    // a fractional number of days is not a SAS date
+    expect(service.sasTemporalLiteral(42.5, 'DATE')).toBeNull()
+  })
+})

client/src/app/services/helper.service.ts

@@ -5,6 +5,22 @@ import { SasService } from './sas.service'
 
 const librariesToShow = 50
 
+/** Month names used by a SAS date literal, eg 20APR2042. */
+const sasMonths = [
+  'JAN',
+  'FEB',
+  'MAR',
+  'APR',
+  'MAY',
+  'JUN',
+  'JUL',
+  'AUG',
+  'SEP',
+  'OCT',
+  'NOV',
+  'DEC'
+]
+
 @Injectable({
   providedIn: 'root'
 })
@@ -226,6 +242,70 @@ export class HelperService {
     })
   }
 
+  /**
+   * Formats a SAS internal date / datetime / time value as the SAS literal a
+   * WHERE clause can carry, eg:
+   *   34322 (DATE)     -> '20DEC2053'd
+   *   42    (DATETIME) -> '01JAN1960:00:00:42'dt
+   *   42    (TIME)     -> '00:00:42't
+   *
+   * The value is days since 1960-01-01 for a date, and seconds since
+   * 1960-01-01 (datetime) or since midnight (time).  The date part is built
+   * arithmetically in UTC, so the day cannot shift with the browser timezone.
+   *
+   * SAS temporal values are whole days and whole seconds, so the value is
+   * rounded to seconds first - that kills the noise a double picks up on the
+   * way here, where 1.9999999999999 is simply the second 2.  A value that is
+   * genuinely fractional does not land on a second, and rounding it would
+   * silently widen the filter, so it returns null instead.
+   *
+   * Returns null when the value is not a finite number, is not a whole day or
+   * second, or the ddtype is not temporal - callers then send the raw value.
+   */
+  public sasTemporalLiteral(value: any, ddtype: string): string | null {
+    const kind = (ddtype ?? '').toString().toUpperCase()
+    if (kind !== 'DATE' && kind !== 'DATETIME' && kind !== 'TIME') return null
+    if (value === null || value === undefined || value === '') return null
+
+    const num = typeof value === 'number' ? value : Number(String(value).trim())
+    if (!isFinite(num)) return null
+
+    /* Within the precision of a double, is this a whole second? */
+    const seconds = Math.round(num * 1e8) / 1e8
+    if (!Number.isInteger(seconds)) return null
+
+    if (kind === 'TIME') return `'${this.sasTimeFromSeconds(seconds)}'t`
+
+    const days = kind === 'DATETIME' ? Math.floor(seconds / 86400) : seconds
+    const datePart = this.sasDateFromDays(days)
+
+    if (kind === 'DATE') return `'${datePart}'d`
+
+    return `'${datePart}:${this.sasTimeFromSeconds(seconds - days * 86400)}'dt`
+  }
+
+  /** `ddMMMyyyy` for a SAS date value (days since 1960-01-01). */
+  private sasDateFromDays(days: number): string {
+    const date = new Date(Date.UTC(1960, 0, 1) + days * 86400000)
+
+    return (
+      this.addLeadingZero(date.getUTCDate()) +
+      sasMonths[date.getUTCMonth()] +
+      date.getUTCFullYear()
+    )
+  }
+
+  /** `hh:mm:ss` for a SAS time value (whole seconds since midnight). */
+  private sasTimeFromSeconds(seconds: number): string {
+    return (
+      this.addLeadingZero(Math.floor(seconds / 3600)) +
+      ':' +
+      this.addLeadingZero(Math.floor((seconds % 3600) / 60)) +
+      ':' +
+      this.addLeadingZero(seconds % 60)
+    )
+  }
+
   public addLeadingZero(value: string | number): string {
     if (typeof value !== 'string') value = value.toString()
 

client/src/app/services/licence.service.ts

@@ -8,13 +8,20 @@ import * as encoding from 'text-encoding'
 import { Router } from '@angular/router'
 import { Globvar } from '../models/sas/public-startupservice.model'
 import { freeTierConfig } from '../free-tier.config'
-import { AppStoreService } from './app-store.service'
 import { HelperService } from './helper.service'
 import { LicenceState } from '../models/LicenceState'
 import { decodeLicenceFeatures as decodeFeatures } from './utils/decodeLicenceFeatures'
 import { LoggerService } from './logger.service'
 import { EventService } from './event.service'
 
+/**
+ * Handsontable licence key used until a licence is loaded from the backend.
+ * The decrypted licence carries the customer's own key (`hot_license_key`), so
+ * this is only the pre-licence / free-tier default - it used to be supplied by
+ * the `<sasjs>` tag.
+ */
+export const DEFAULT_HOT_LICENCE_KEY = 'non-commercial-and-evaluation'
+
 @Injectable({
   providedIn: 'root'
 })
@@ -43,7 +50,7 @@ export class LicenceService {
 
   private freeTierLicenceData: LicenseKeyData = {
     demo: true,
-    hot_license_key: this.appStoreService.getDcAdapterSettings()?.hotLicenceKey,
+    hot_license_key: DEFAULT_HOT_LICENCE_KEY,
     users_allowed: this._licenceState.users_allowed || 1,
     valid_until: moment().add(1, 'year').format('YYYY-MM-DD'),
     site_id: '',
@@ -63,7 +70,6 @@ export class LicenceService {
 
   constructor(
     private loggerService: LoggerService,
-    private appStoreService: AppStoreService,
     private eventService: EventService,
     private sasService: SasService,
     private helperService: HelperService,
@@ -79,8 +85,7 @@ export class LicenceService {
   }
 
   public async activation(startupservice: any) {
-    this.freeTierLicenceData.hot_license_key =
-      this.appStoreService.getDcAdapterSettings()?.hotLicenceKey
+    this.freeTierLicenceData.hot_license_key = DEFAULT_HOT_LICENCE_KEY
 
     await this.setStartupserviceRules(startupservice)
 
@@ -137,7 +142,8 @@ export class LicenceService {
       this.isAppActivated.next(false)
       this.eventService.showInfoModal(
         'Error',
-        'Some of the globvars are not present in the startupservice'
+        'Some of the globvars are not present in the startupservice',
+        'public/startupservice'
       )
       // this.router.navigateByUrl('/licensing/key?error=missing')
       return
@@ -225,8 +231,7 @@ export class LicenceService {
         ? licenseData.hot_license_key
         : this.hot_license_key.value
 
-    if (!hotLicenceKey)
-      hotLicenceKey = this.appStoreService.getDcAdapterSettings()?.hotLicenceKey
+    if (!hotLicenceKey) hotLicenceKey = DEFAULT_HOT_LICENCE_KEY
 
     this.hot_license_key.next(hotLicenceKey)
 

client/src/app/services/sas-store.service.spec.ts

@@ -0,0 +1,313 @@
+import { SasStoreService } from './sas-store.service'
+import { HelperService } from './helper.service'
+
+/**
+ * The filter picker keys its operator set, its variable highlight and its value
+ * quoting on Col.TYPE being exactly 'num' or 'char'.  On the normal path
+ * mergeColsRules derives that from $dataFormats; on the public/getcols fallback
+ * (used when the EDIT/VIEW limits abort the load) the service speaks the SAS
+ * vocabulary (TYPE C/N, DDTYPE separating DATE/DATETIME/TIME), so the store has
+ * to map it.  These tests pin both the mapping and the empty-payload guard.
+ */
+describe('SasStoreService', () => {
+  const buildStore = (cols: any[]) =>
+    new SasStoreService(
+      {
+        request: () => Promise.resolve({ adapterResponse: { cols } })
+      } as any,
+      {} as any,
+      {} as any
+    )
+
+  const collectColumns = (store: SasStoreService) => {
+    const seen: any[] = []
+    store.columns.subscribe((value: any) => seen.push(value))
+    return seen
+  }
+
+  describe('getCols', () => {
+    it('maps the SAS C/N TYPE onto the picker vocabulary', async () => {
+      const store = buildStore([
+        { NAME: 'CHAR_COL', TYPE: 'C', DDTYPE: 'C' },
+        { NAME: 'NUM_COL', TYPE: 'N', DDTYPE: 'N' },
+        { NAME: 'DATE_COL', TYPE: 'N', DDTYPE: 'DATE' },
+        { NAME: 'DATETIME_COL', TYPE: 'N', DDTYPE: 'DATETIME' },
+        { NAME: 'TIME_COL', TYPE: 'N', DDTYPE: 'TIME' }
+      ])
+
+      const cols = await store.getCols('DC.MY_TABLE')
+
+      expect(cols.map((col: any) => col.TYPE)).toEqual([
+        'char',
+        'num',
+        'num',
+        'num',
+        'num'
+      ])
+    })
+
+    it('leaves the rest of the column payload untouched', async () => {
+      const store = buildStore([
+        {
+          NAME: 'CHAR_COL',
+          TYPE: 'C',
+          DDTYPE: 'C',
+          LENGTH: 8,
+          LABEL: 'A label'
+        }
+      ])
+
+      const cols = await store.getCols('DC.MY_TABLE')
+
+      expect(cols[0].NAME).toBe('CHAR_COL')
+      expect(cols[0].DDTYPE).toBe('C')
+      expect(cols[0].LENGTH).toBe(8)
+      expect(cols[0].LABEL).toBe('A label')
+    })
+
+    it('returns an empty list rather than throwing when the payload is empty', async () => {
+      const store = buildStore([])
+
+      const cols = await store.getCols('DC.MY_TABLE')
+
+      expect(cols).toEqual([])
+    })
+  })
+
+  describe('setQueryVariablesFromTable', () => {
+    it('publishes the table columns without calling the service', async () => {
+      let called = false
+      const store = new SasStoreService(
+        {
+          request: () => {
+            called = true
+            return Promise.resolve({})
+          }
+        } as any,
+        {} as any,
+        {} as any
+      )
+      const seen = collectColumns(store)
+
+      await store.setQueryVariablesFromTable('DC.MY_TABLE', [{ NAME: 'A' }])
+
+      expect(called).toBe(false)
+      expect(seen.length).toBe(1)
+      expect(seen[0].data.cols.length).toBe(1)
+    })
+
+    it('falls back to getcols and publishes the mapped columns', async () => {
+      const store = buildStore([{ NAME: 'A', TYPE: 'C', DDTYPE: 'C' }])
+      const seen = collectColumns(store)
+
+      await store.setQueryVariablesFromTable('DC.MY_TABLE', [])
+
+      expect(seen.length).toBe(1)
+      expect(seen[0].libds).toBe('DC.MY_TABLE')
+      expect(seen[0].data.cols[0].TYPE).toBe('char')
+    })
+
+    it('does not publish an empty fallback, which the picker cannot render', async () => {
+      const store = buildStore([])
+      const seen = collectColumns(store)
+
+      await store.setQueryVariablesFromTable('DC.MY_TABLE', [])
+
+      expect(seen.length).toBe(0)
+    })
+  })
+})
+
+/**
+ * The clause shown in the filter panel is composed by whereClauseCreator, and
+ * its temporal rewrite turns the internal day / second count into a SAS
+ * literal.  It may only do that when the composed value really is nothing but
+ * those numbers - a value typed by hand is not, and rewriting the digit runs
+ * inside it produced clauses like '13JAN1960'd/'03JAN1960'd/'14MAY1965'd for
+ * a date typed as 12/02/1960.
+ */
+describe('SasStoreService - temporal clause preview', () => {
+  const buildStore = () =>
+    new SasStoreService({} as any, new HelperService(null as any), {} as any)
+
+  const clauseFor = (
+    value: any,
+    operator: string,
+    ddtype: string,
+    values: any[] = []
+  ) => ({
+    queryObj: [
+      {
+        clauseLogic: '',
+        elements: [
+          {
+            variable: 'SOME_DATE',
+            operator,
+            value,
+            type: 'num',
+            ddtype,
+            valueVariable: false,
+            values
+          }
+        ]
+      }
+    ],
+    groupLogic: ''
+  })
+
+  it('leaves a typed temporal value alone when it is not a number', () => {
+    const store = buildStore()
+
+    const result = store.whereClauseCreator(
+      clauseFor('12/02/1960', '=', 'DATE'),
+      '',
+      'lib.ds'
+    )
+
+    expect(result.whereClause).toContain('12/02/1960')
+    expect(result.whereClause).not.toContain("'d")
+  })
+
+  it('still shows an internal day count as a date literal', () => {
+    const store = buildStore()
+
+    const result = store.whereClauseCreator(
+      clauseFor('42', '=', 'DATE'),
+      '',
+      'lib.ds'
+    )
+
+    expect(result.whereClause).toContain("'12FEB1960'd")
+  })
+
+  /**
+   * A BETWEEN pair is composed as `<from> AND <to>`, so the guard that decides
+   * whether the rewrite may run has to see past the separator - its letters are
+   * not part of the numbers the value is built from.
+   */
+  it('shows a BETWEEN pair picked from the value list as literals', () => {
+    const store = buildStore()
+
+    const result = store.whereClauseCreator(
+      clauseFor(['12FEB1960', '13FEB1960'], 'BETWEEN', 'DATE', [
+        { formatted: '12FEB1960', unformatted: '42' },
+        { formatted: '13FEB1960', unformatted: '43' }
+      ]),
+      '',
+      'lib.ds'
+    )
+
+    expect(result.whereClause).toContain(
+      "BETWEEN '12FEB1960'd AND '13FEB1960'd"
+    )
+  })
+})
+
+/**
+ * createFilterQueryTable builds the RAW_VALUE the backend validates and stores.
+ * The range dropdown binds the formatted label, so a BETWEEN has to map each
+ * end back to its unformatted value before the literal is built - otherwise
+ * sasTemporalLiteral returns null and an unquoted date reaches the backend.
+ */
+describe('SasStoreService - temporal RAW_VALUE', () => {
+  const buildStore = () =>
+    new SasStoreService({} as any, new HelperService(null as any), {} as any)
+
+  const betweenQuery = (value: any[], values: any[]) => ({
+    groupLogic: 'AND',
+    filterGroups: [
+      {
+        clauseLogic: 'AND',
+        filterClauses: [
+          {
+            ddtype: 'DATE',
+            invalidClause: false,
+            logic: '',
+            operator: 'BETWEEN',
+            operators: [],
+            type: 'num',
+            value,
+            valueVariable: false,
+            values,
+            variable: 'SOME_DATE'
+          }
+        ]
+      }
+    ]
+  })
+
+  it('maps a BETWEEN picked from the value list back to the internal value', () => {
+    const store = buildStore()
+
+    const clauses = store.createFilterQueryTable(
+      betweenQuery(
+        ['12FEB1960', '13FEB1960'],
+        [
+          { formatted: '12FEB1960', unformatted: 42 },
+          { formatted: '13FEB1960', unformatted: 43 }
+        ]
+      ) as any
+    )
+
+    expect(clauses[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
+  })
+
+  it('still builds literals when the values are already internal counts', () => {
+    const store = buildStore()
+
+    const clauses = store.createFilterQueryTable(
+      betweenQuery(['42', '43'], []) as any
+    )
+
+    expect(clauses[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
+  })
+
+  const inQuery = (value: any[], values: any[]) => ({
+    groupLogic: 'AND',
+    filterGroups: [
+      {
+        clauseLogic: 'AND',
+        filterClauses: [
+          {
+            ddtype: 'DATE',
+            invalidClause: false,
+            logic: '',
+            operator: 'IN',
+            operators: [],
+            type: 'num',
+            value,
+            valueVariable: false,
+            values,
+            variable: 'SOME_DATE'
+          }
+        ]
+      }
+    ]
+  })
+
+  it('maps an IN entry picked from the value list back to the internal value', () => {
+    const store = buildStore()
+
+    const clauses = store.createFilterQueryTable(
+      inQuery(
+        [{ checked: true, val: '12FEB1960' }],
+        [{ formatted: '12FEB1960', unformatted: 42 }]
+      ) as any
+    )
+
+    expect(clauses[0].RAW_VALUE).toBe("('12FEB1960'd)")
+  })
+
+  it('passes an already-unformatted IN entry through unchanged', () => {
+    const store = buildStore()
+
+    const clauses = store.createFilterQueryTable(
+      inQuery(
+        [{ checked: true, val: 42 }],
+        [{ formatted: '12FEB1960', unformatted: 42 }]
+      ) as any
+    )
+
+    expect(clauses[0].RAW_VALUE).toBe("('12FEB1960'd)")
+  })
+})

client/src/app/services/sas-store.service.ts

@@ -301,13 +301,29 @@ export class SasStoreService {
    * The picker's variable list normally comes from the columns that arrived
    * with the table (see setQueryVariablesFromTable). This is the fallback for
    * when there is no table - public/getcols reads the metadata only.
+   *
+   * public/getcols speaks the SAS vocabulary: TYPE is C/N and DDTYPE separates
+   * DATE, DATETIME and TIME. The picker keys its operator set (query.component),
+   * its variable highlight and its value quoting (sas-store) on the client
+   * vocabulary instead - TYPE is exactly 'num' or 'char' - which
+   * mergeColsRules derives from $dataFormats on the normal path. That pass
+   * never runs for this fallback, so map TYPE here, with the same rule: the
+   * date and time types are numeric.
    */
   public async getCols(libds: string) {
     const tables = { iwant: [{ libds: libds }] }
 
     const res = await this.sasService.request('public/getcols', tables)
 
-    return res.adapterResponse.cols || []
+    const cols = res.adapterResponse.cols || []
+
+    return cols.map((col: any) => ({
+      ...col,
+      TYPE:
+        ['DATE', 'DATETIME', 'TIME'].includes(col.DDTYPE) || col.TYPE === 'N'
+          ? 'num'
+          : 'char'
+    }))
   }
 
   /**
@@ -326,7 +342,19 @@ export class SasStoreService {
       return
     }
 
-    this.setQueryVariables(libds, await this.getCols(libds))
+    const fallback = await this.getCols(libds)
+
+    /*
+      public/getcols emits an empty cols table when the dataset has no
+      metadata.  Publishing that would push an empty array into the filter
+      picker, whose subscriber reads cols[0].NAME - the throw is caught, but
+      the picker is left half-updated (this.cols assigned, the filter.cols
+      assignment below the throwing line never run).  There is nothing to
+      offer, so leave the picker as it was.
+    */
+    if (!fallback || fallback.length === 0) return
+
+    this.setQueryVariables(libds, fallback)
   }
 
   public async getChangeInfo(tableId: any) {
@@ -568,6 +596,27 @@ export class SasStoreService {
         }
         oper = this.checkOperator(oper, value, type).operator
 
+        /* A temporal value is shown as a SAS literal (eg '20DEC2053'd) rather
+           than the internal day / second count.  Only rewrite when the composed
+           value is nothing but the numbers the clause is built from - a single
+           value, a BETWEEN pair or an IN list.  The BETWEEN separator is
+           removed before the test because ' AND ' is not part of that character
+           set; a typed value that is not a number, such as a date entered by
+           hand, would otherwise have every digit run inside it rewritten into
+           an unrelated literal. */
+        if (
+          !isValueVariable &&
+          ['DATE', 'DATETIME', 'TIME'].includes(query.ddtype) &&
+          /^[-0-9.,()\s]+$/.test(String(value).replace(/\s+AND\s+/gi, ' '))
+        ) {
+          value = String(value).replace(
+            /-?\d+(?:\.\d+)?/g,
+            (match) =>
+              this.helperService.sasTemporalLiteral(match, query.ddtype) ??
+              match
+          )
+        }
+
         if (type === 'char' && oper !== 'IN' && oper !== 'NOT IN') {
           if (typeof value === 'undefined') {
             value = ''
@@ -634,17 +683,53 @@ export class SasStoreService {
     filterQuery.filterGroups.forEach(
       (filterGroup: FilterGroup, groupIndex: number) => {
         filterGroup.filterClauses.forEach((filterClause: FilterClause) => {
+          /* A temporal value is sent as a SAS literal (eg '20APR2042'd) rather
+             than the internal day / second count, so the resolved WHERE clause
+             - and the FILTER_TEXT shown above the grid - carries a date. */
+          const temporalValue = (value: any) =>
+            filterClause.valueVariable
+              ? null
+              : this.helperService.sasTemporalLiteral(
+                  value,
+                  filterClause.ddtype
+                )
+
+          /* The range dropdown binds the formatted label, as the value list
+             and the IN modal do, so each end of a BETWEEN is mapped back to
+             its unformatted value before the literal is built.  This is the
+             same lookup whereClauseCreator applies for the preview - without
+             it the formatted string reaches sasTemporalLiteral, comes back
+             null, and RAW_VALUE carries an unquoted date that the backend
+             rejects on a numeric column. */
+          const unformattedValue = (value: any) => {
+            const match = (filterClause.values || []).find(
+              (entry: any) => entry.formatted === value
+            )
+            return match ? match.unformatted : value
+          }
+
           let rawValue = ''
           if (filterClause.operator === 'BETWEEN') {
-            rawValue = `${filterClause.value[0]} AND ${filterClause.value[1]}`
+            const from = unformattedValue(filterClause.value[0])
+            const to = unformattedValue(filterClause.value[1])
+            rawValue = `${temporalValue(from) ?? from} AND ${
+              temporalValue(to) ?? to
+            }`
           } else if (
             filterClause.operator === 'IN' ||
             filterClause.operator === 'NOT IN'
           ) {
             filterClause.value.forEach((element: any) => {
               if (element.checked) {
-                const value = element.val
-                if (
+                /* Mapped like the BETWEEN ends - whereClauseCreator normally
+                   rewrites element.val to the unformatted count before submit,
+                   so this is a pass-through for the usual path and a guard for
+                   any path that reaches here without the preview having run. */
+                const value = unformattedValue(element.val)
+                const literal = temporalValue(value)
+                if (literal) {
+                  rawValue += (rawValue.length > 0 ? ',' : '(') + literal
+                } else if (
                   typeof value === 'string' &&
                   (!isSpecialMissing(value) || filterClause.type === 'char')
                 ) {
@@ -672,7 +757,13 @@ export class SasStoreService {
               rawValue = '.'
             }
           } else {
-            if (filterClause.type === 'char' && !filterClause.valueVariable) {
+            const literal = temporalValue(filterClause.value)
+            if (literal) {
+              rawValue = literal
+            } else if (
+              filterClause.type === 'char' &&
+              !filterClause.valueVariable
+            ) {
               rawValue = `'${filterClause.value.replace(/'/g, "''")}'`
             }
           }

client/src/app/services/sas.service.spec.ts

@@ -1,4 +1,4 @@
-import { SasService } from './sas.service'
+import { DEBUG_STATE_KEY, SasService } from './sas.service'
 
 /**
  * SasService's constructor is a plain 8-argument constructor with an empty
@@ -13,7 +13,13 @@ import { SasService } from './sas.service'
 const buildService = () => {
   const sasjsAdapter: any = {
     request: jasmine.createSpy('request').and.resolveTo({}),
-    getSasRequests: () => []
+    getSasRequests: () => [],
+    setDebugState: jasmine.createSpy('setDebugState'),
+    getSasjsConfig: () => ({
+      serverUrl: 'https://sas.example.com',
+      appLoc: '/Public/app/dc',
+      serverType: 'SASVIYA'
+    })
   }
   const loggerService: any = {
     logRequestData: jasmine.createSpy('logRequestData')
@@ -87,3 +93,115 @@ describe('SasService diagnostics payload', () => {
     expect(sentBody(sasjsAdapter).browser_info).toBe(supplied)
   })
 })
+
+/**
+ * The `<sasjs>` tag is re-read on every load, so the debug toggle has to survive
+ * a refresh on its own.
+ */
+describe('SasService debug state', () => {
+  /** Builds a service whose init can actually run (real adapter, stub deps). */
+  const initService = (tagDebug = false) => {
+    const appStoreService: any = {
+      getDcAdapterSettings: () => ({
+        serverUrl: '',
+        appLoc: '/Public/app/dc',
+        serverType: 'SAS9',
+        loginMechanism: '',
+        dcPath: '',
+        debug: tagDebug,
+        useComputeApi: null,
+        runAsTask: true,
+        contextName: ''
+      })
+    }
+
+    const service = new SasService(
+      appStoreService,
+      { user: undefined } as any,
+      { showInfoModal: jasmine.createSpy('showInfoModal') } as any,
+      { setup: jasmine.createSpy('setup') } as any,
+      { setup: jasmine.createSpy('setup') } as any,
+      { logRequestData: jasmine.createSpy('logRequestData') } as any,
+      { updateStep: jasmine.createSpy('updateStep') } as any,
+      { navigateByUrl: jasmine.createSpy('navigateByUrl') } as any
+    )
+
+    service.sasServiceInit()
+
+    return service
+  }
+
+  beforeEach(() => localStorage.removeItem(DEBUG_STATE_KEY))
+  afterEach(() => localStorage.removeItem(DEBUG_STATE_KEY))
+
+  it('stores the state when the toggle changes', () => {
+    const { service, sasjsAdapter } = buildService()
+
+    service.setDebugState(true)
+
+    expect(localStorage.getItem(DEBUG_STATE_KEY)).toBe('true')
+    expect(sasjsAdapter.setDebugState).toHaveBeenCalledWith(true)
+  })
+
+  it('stores a switch back off, rather than clearing it', () => {
+    const { service } = buildService()
+
+    service.setDebugState(true)
+    service.setDebugState(false)
+
+    expect(localStorage.getItem(DEBUG_STATE_KEY)).toBe('false')
+  })
+
+  it('reinstates the stored state on the next load', () => {
+    localStorage.setItem(DEBUG_STATE_KEY, 'true')
+
+    expect(initService().getSasjsConfig().debug).toBe(true)
+  })
+
+  it('leaves the tag value alone when nothing is stored', () => {
+    expect(initService(true).getSasjsConfig().debug).toBe(true)
+  })
+
+  it('reinstates an off state over a tag that turns debug on', () => {
+    localStorage.setItem(DEBUG_STATE_KEY, 'false')
+
+    expect(initService(true).getSasjsConfig().debug).toBe(false)
+  })
+})
+
+/**
+ * The URL shape the deploy configurator already uses for makedata: the service
+ * is run by the JES / STP web app in a browser tab, so the SAS log comes back
+ * as the page body instead of through the adapter.
+ */
+describe('SasService openServiceInNewWindow', () => {
+  it('opens the service URL built from the live adapter config', () => {
+    const { service } = buildService()
+    spyOn(service, 'getExecutionPath').and.returnValue('/SASJobExecution')
+    const openSpy = spyOn(window, 'open')
+
+    service.openServiceInNewWindow('public/startupservice', {
+      _contextname: 'ctx'
+    })
+
+    expect(openSpy).toHaveBeenCalledWith(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_contextname=ctx&_debug=131',
+      '_blank',
+      'noopener'
+    )
+  })
+
+  it('opens without parameters when none are given', () => {
+    const { service } = buildService()
+    spyOn(service, 'getExecutionPath').and.returnValue('/SASJobExecution')
+    const openSpy = spyOn(window, 'open')
+
+    service.openServiceInNewWindow('public/startupservice')
+
+    expect(openSpy).toHaveBeenCalledWith(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131',
+      '_blank',
+      'noopener'
+    )
+  })
+})

client/src/app/services/sas.service.ts

@@ -21,6 +21,7 @@ import { SasViyaService } from './sas-viya.service'
 import { ViyaApiFolder } from '../viya-api-explorer/models/viya-api-folder.model'
 import { ViyaApiFolderMembers } from '../viya-api-explorer/models/viya-api-folder-content.model'
 import { parseUserAgent } from '../shared/utils/parse-user-agent'
+import { buildServiceUrl } from '../shared/utils/service-url'
 import { VERSION } from '../../environments/version'
 
 /**
@@ -41,6 +42,13 @@ const DIAGNOSTICS_SERVICES = [
   'services/auditors/postdata'
 ]
 
+/**
+ * Where the debug toggle state is kept.  The `<sasjs>` tag in index.html is
+ * re-read on every load, so without this a refresh would drop the state the user
+ * set from the UI.
+ */
+export const DEBUG_STATE_KEY = 'dc-debug-state'
+
 @Injectable({
   providedIn: 'root'
 })
@@ -89,6 +97,13 @@ export class SasService {
 
     this.sasjsAdapter = new SASjs(this.dcAdapterSettings)
 
+    /* The tag resets debug on every load, so reinstate the state the user last
+      chose from the UI. */
+    const storedDebugState = localStorage.getItem(DEBUG_STATE_KEY)
+    if (storedDebugState !== null) {
+      this.sasjsAdapter.setDebugState(storedDebugState === 'true')
+    }
+
     switch (this.dcAdapterSettings.serverType) {
       case ServerType.SasViya: {
         this.checkViyaDeploy(this.dcAdapterSettings.appLoc || '')
@@ -271,7 +286,10 @@ export class SasService {
                 adapterResponse: {
                   error: abortMsg
                 },
-                log: sasRequest?.logFile
+                log: sasRequest?.logFile,
+                /* the modal above already told the user: a caller that also
+                   reports the failure would stack a second one on top */
+                errorShown: !wrapperOptions?.suppressSuccessAbortModal
               })
             }
 
@@ -314,7 +332,8 @@ export class SasService {
                 adapterResponse: {
                   error: errorMessage
                 },
-                log: sasRequest?.logFile
+                log: sasRequest?.logFile,
+                errorShown: !wrapperOptions?.suppressErrorAbortModal
               })
             }
 
@@ -803,6 +822,7 @@ export class SasService {
   }
 
   public setDebugState(state: boolean) {
+    localStorage.setItem(DEBUG_STATE_KEY, String(state))
     this.sasjsAdapter.setDebugState(state)
   }
 
@@ -827,6 +847,36 @@ export class SasService {
     return this.sasjsAdapter
   }
 
+  /**
+   * Runs a SAS service in a new browser window, bypassing the adapter.
+   *
+   * The adapter's request path can hide the backend log when a service fails:
+   * on Viya a long startup service may be submitted as an execution task and
+   * time out before it finishes, returning neither webout nor log. A browser
+   * request to the JES / STP web app runs the same `_program` with no task and
+   * returns the SAS log (`_debug=131`), so this is the diagnostic path offered
+   * when a failed request leaves nothing usable to display.
+   */
+  public openServiceInNewWindow(
+    service: string,
+    params: { [name: string]: string | number | null | undefined } = {}
+  ) {
+    const config = this.getSasjsConfig()
+
+    const url = buildServiceUrl({
+      serverUrl: config.serverUrl || '',
+      appLoc: config.appLoc || '',
+      executionPath: this.getExecutionPath() || '',
+      service,
+      params
+    })
+
+    // noopener: the SAS log page is on a different origin (Viya's serverUrl is
+    // not the app's), and without it the opened window keeps a handle on
+    // window.opener and can navigate this tab.
+    window.open(url, '_blank', 'noopener')
+  }
+
   public getServerType(): string {
     const sasjsConfig = this.sasjsAdapter.getSasjsConfig()
 

client/src/app/services/startup-check.service.spec.ts

@@ -0,0 +1,93 @@
+import {
+  getSasjsTagDetails,
+  SASJS_TAG_STEP,
+  StartupCheckService
+} from './startup-check.service'
+
+describe('StartupCheckService', () => {
+  let service: StartupCheckService
+
+  beforeEach(() => {
+    service = new StartupCheckService()
+  })
+
+  it('includes a SASjs tag step after the startup checks', () => {
+    const steps = service.steps$.value
+
+    expect(steps.length).toBe(SASJS_TAG_STEP + 1)
+    expect(steps[SASJS_TAG_STEP].label).toBe('SASjs tag')
+    expect(steps[SASJS_TAG_STEP].status).toBe('pending')
+  })
+
+  it('shows the tag contents against the SASjs tag step', () => {
+    service.setTagDetails([{ name: 'serverType', value: 'SASVIYA' }])
+
+    expect(service.steps$.value[SASJS_TAG_STEP]).toEqual({
+      label: 'SASjs tag',
+      status: 'done',
+      details: [{ name: 'serverType', value: 'SASVIYA' }]
+    })
+  })
+
+  it('flags a missing tag on the SASjs tag step', () => {
+    service.updateStep(SASJS_TAG_STEP, 'error', 'SASJS TAG MISSING!!')
+
+    expect(service.steps$.value[SASJS_TAG_STEP]).toEqual({
+      label: 'SASjs tag',
+      status: 'error',
+      detail: 'SASJS TAG MISSING!!'
+    })
+  })
+})
+
+describe('getSasjsTagDetails', () => {
+  const tagWith = (attributes: Record<string, string>) => {
+    const element = document.createElement('sasjs')
+
+    Object.entries(attributes).forEach(([name, value]) =>
+      element.setAttribute(name, value)
+    )
+
+    return element
+  }
+
+  it('reads the tag attributes as name/value pairs', () => {
+    const details = getSasjsTagDetails(
+      tagWith({ serverType: 'SASVIYA', debug: 'false', useComputeApi: 'true' })
+    )
+
+    expect(details).toEqual([
+      { name: 'serverType', value: 'SASVIYA' },
+      { name: 'debug', value: 'false' },
+      { name: 'useComputeApi', value: 'true' }
+    ])
+  })
+
+  it('keeps attributes the parser does not know about', () => {
+    expect(getSasjsTagDetails(tagWith({ serverTyp: 'SASJS' }))).toEqual([
+      { name: 'servertyp', value: 'SASJS' }
+    ])
+  })
+
+  it('shows every attribute, including the admin group and the licence key', () => {
+    const details = getSasjsTagDetails(
+      tagWith({
+        serverType: 'SASJS',
+        adminGroup: 'SASAdministrators',
+        hotLicenceKey: 'non-commercial-and-evaluation'
+      })
+    )
+
+    expect(details).toEqual([
+      { name: 'serverType', value: 'SASJS' },
+      { name: 'adminGroup', value: 'SASAdministrators' },
+      { name: 'hotLicenceKey', value: 'non-commercial-and-evaluation' }
+    ])
+  })
+
+  it('flags attributes that carry no value', () => {
+    expect(getSasjsTagDetails(tagWith({ runAsTask: '' }))).toEqual([
+      { name: 'runAsTask', value: '(empty)' }
+    ])
+  })
+})

client/src/app/services/startup-check.service.ts

@@ -3,18 +3,64 @@ import { BehaviorSubject } from 'rxjs'
 
 export type StartupStepStatus = 'pending' | 'in_progress' | 'done' | 'error'
 
+/** A name/value pair shown under a startup step. */
+export interface StartupStepDetail {
+  name: string
+  value: string
+}
+
 export interface StartupStep {
   label: string
   status: StartupStepStatus
   detail?: string
+  /** Rendered as a grid under the label. */
+  details?: StartupStepDetail[]
 }
 
+/**
+ * Index of the `<sasjs>` tag dump within the startup steps.  It is appended
+ * after the three startup checks so the indices the startup flow already drives
+ * (sas.service / app.service) are unchanged.
+ */
+export const SASJS_TAG_STEP = 3
+
+/**
+ * Canonical spelling of the `<sasjs>` tag attributes.  HTML lower-cases attribute
+ * names in the DOM, so they are mapped back for display.
+ */
+const TAG_ATTRIBUTE_NAMES: Record<string, string> = {
+  serverurl: 'serverUrl',
+  apploc: 'appLoc',
+  servertype: 'serverType',
+  loginmechanism: 'loginMechanism',
+  admingroup: 'adminGroup',
+  dcpath: 'dcPath',
+  debug: 'debug',
+  usecomputeapi: 'useComputeApi',
+  runastask: 'runAsTask',
+  contextname: 'contextName',
+  hotlicencekey: 'hotLicenceKey'
+}
+
+/**
+ * Reads the attributes of the `<sasjs>` tag for display in the startup checks.
+ * Every attribute is shown - including ones the parser does not know about, so a
+ * misspelt one is visible and a wrong value is exactly the misconfiguration this
+ * dump exists to expose.
+ */
+export const getSasjsTagDetails = (element: Element): StartupStepDetail[] =>
+  Array.from(element.attributes).map((attribute) => ({
+    name: TAG_ATTRIBUTE_NAMES[attribute.name.toLowerCase()] || attribute.name,
+    value: attribute.value || '(empty)'
+  }))
+
 @Injectable({ providedIn: 'root' })
 export class StartupCheckService {
   private steps: StartupStep[] = [
     { label: 'Checking app location', status: 'pending' },
     { label: 'Checking Viya deploy', status: 'pending' },
-    { label: 'Loading startup service', status: 'pending' }
+    { label: 'Loading startup service', status: 'pending' },
+    { label: 'SASjs tag', status: 'pending' }
   ]
 
   public steps$ = new BehaviorSubject<StartupStep[]>([...this.steps])
@@ -30,6 +76,16 @@ export class StartupCheckService {
     }
   }
 
+  /** Shows the `<sasjs>` tag contents on the boot screen. */
+  public setTagDetails(details: StartupStepDetail[]) {
+    this.steps[SASJS_TAG_STEP] = {
+      ...this.steps[SASJS_TAG_STEP],
+      status: 'done',
+      details
+    }
+    this.steps$.next([...this.steps])
+  }
+
   public setSteps(steps: StartupStep[]) {
     this.steps = steps
     this.steps$.next([...this.steps])

client/src/app/services/va-filter.service.spec.ts

@@ -1,27 +1,6 @@
 import { VaFilterService } from './va-filter.service'
 import { VaMessage } from './va-messaging.service'
-
-/**
- * Minimal stand-in for HelperService.convertJsDateToSasDate using the real
- * epoch math, so date/datetime assertions use the actual SAS internal values
- * without pulling in HelperService's dependency graph (SasService etc.).
- */
-class HelperServiceStub {
-  convertJsDateToSasDate(jsDate: Date, unit = 'days'): number {
-    const ms = Date.UTC(
-      jsDate.getFullYear(),
-      jsDate.getMonth(),
-      jsDate.getDate(),
-      jsDate.getHours(),
-      jsDate.getMinutes(),
-      jsDate.getSeconds()
-    )
-    const sasMs = ms + 315619200000
-    return unit === 'days'
-      ? Math.floor(Math.abs(sasMs / 86400000))
-      : sasMs / 1000
-  }
-}
+import { HelperService } from './helper.service'
 
 function msg(parameters: any[]): VaMessage {
   return {
@@ -63,7 +42,7 @@ describe('VaFilterService', () => {
   ]
 
   beforeEach(() => {
-    service = new VaFilterService(new HelperServiceStub() as any)
+    service = new VaFilterService(new HelperService({} as any))
   })
 
   const labelToVar = () =>
@@ -109,28 +88,30 @@ describe('VaFilterService', () => {
       expect(build([{ label: 'SOME_NUM', value: 'abc' }])).toEqual([])
     })
 
-    it('converts temporal values to SAS internal numbers', () => {
+    it('renders temporal values as SAS literals', () => {
       expect(
         build([{ label: 'SOME_TIME', value: '0:00:02' }])[0].RAW_VALUE
-      ).toBe('2')
+      ).toBe("'00:00:02't")
       expect(
         build([{ label: 'SOME_DATE', value: '01JUL1960' }])[0].RAW_VALUE
-      ).toBe('182')
+      ).toBe("'01JUL1960'd")
       expect(
         build([{ label: 'SOME_DATETIME', value: '01JAN1960:00:06:05' }])[0]
           .RAW_VALUE
-      ).toBe('365')
+      ).toBe("'01JAN1960:00:06:05'dt")
     })
 
-    it('passes an already-internal (bare number) temporal value through raw', () => {
+    it('renders an already-internal (bare number) temporal value as a literal', () => {
       // VA sends the raw SAS internal value when the data item has no format.
       expect(build([{ label: 'SOME_DATE', value: '182' }])[0].RAW_VALUE).toBe(
-        '182'
+        "'01JUL1960'd"
       )
       expect(
         build([{ label: 'SOME_DATETIME', value: '365' }])[0].RAW_VALUE
-      ).toBe('365')
-      expect(build([{ label: 'SOME_TIME', value: '2' }])[0].RAW_VALUE).toBe('2')
+      ).toBe("'01JAN1960:00:06:05'dt")
+      expect(build([{ label: 'SOME_TIME', value: '2' }])[0].RAW_VALUE).toBe(
+        "'00:00:02't"
+      )
     })
 
     it('parses ISO date shapes identically to DATE9 (VA sends no format)', () => {
@@ -232,9 +213,9 @@ describe('VaFilterService', () => {
       expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
         ['SOME_CHAR', "'x'"],
         ['SOME_NUM', '10'],
-        ['SOME_TIME', '2'],
-        ['SOME_DATE', '182'],
-        ['SOME_DATETIME', '365']
+        ['SOME_TIME', "'00:00:02't"],
+        ['SOME_DATE', "'01JUL1960'd"],
+        ['SOME_DATETIME', "'01JAN1960:00:06:05'dt"]
       ])
     })
 
@@ -247,9 +228,9 @@ describe('VaFilterService', () => {
         ])
       )
       expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
-        ['SOME_DATE', '182'],
-        ['SOME_DATE2', '182'],
-        ['SOME_DTTM', '365']
+        ['SOME_DATE', "'01JUL1960'd"],
+        ['SOME_DATE2', "'01JUL1960'd"],
+        ['SOME_DTTM', "'01JAN1960:00:06:05'dt"]
       ])
     })
   })

client/src/app/services/va-filter.service.ts

@@ -223,15 +223,18 @@ export class VaFilterService {
 
   /**
    * Formats one VA filter value into a DC RAW_VALUE for the column kind, or null
-   * when invalid (dropped). Matches DC's native filters, which send the NUMERIC
-   * INTERNAL value (not SAS literals — those error out in the backend):
-   *  - numeric  -> finite number, raw (else null — injection guard)
-   *  - time     -> seconds since midnight   (VA: "0:00:02" -> 2)
-   *  - date     -> days since 1960-01-01    (VA: "01JUL1960" -> 182)
-   *  - datetime -> seconds since 1960-01-01 (VA: "01JAN1960:00:06:05" -> 365)
+   * when invalid (dropped). Matches DC's native filters, which send a SAS
+   * literal for a temporal column and the raw value otherwise:
+   *  - numeric  -> finite number, raw (else null - injection guard)
+   *  - time     -> seconds since midnight, as a SAS time literal
+   *                (VA: "0:00:02" -> '00:00:02't)
+   *  - date     -> days since 1960-01-01, as a SAS date literal
+   *                (VA: "01JUL1960" -> '01JUL1960'd)
+   *  - datetime -> seconds since 1960-01-01, as a SAS datetime literal
+   *                (VA: "01JAN1960:00:06:05" -> '01JAN1960:00:06:05'dt)
    *  - char     -> single-quote escaped
    * Temporal strings are regex-parsed (digits/letters/colons only), so a value
-   * either yields a finite number or is dropped — also the injection guard.
+   * either yields a finite number or is dropped - also the injection guard.
    * Date/datetime reuse HelperService.convertJsDateToSasDate so the value is
    * byte-identical to what DC's own pickers produce.
    *
@@ -239,7 +242,7 @@ export class VaFilterService {
    * itself (VA sends it unformatted when the data item carries no format / is
    * used as a measure), so it is passed through raw. This is unambiguous because
    * the only date formats in use (DATE9, ISO) always contain a letter or a
-   * separator — a plain integer is never a formatted date.
+   * separator - a plain integer is never a formatted date.
    */
   private formatValueByKind(
     value: any,
@@ -254,23 +257,32 @@ export class VaFilterService {
         return Number.isFinite(num) ? `${num}` : null
       }
       case 'time': {
-        if (this.isBareNumber(s)) return `${Number(s)}`
-        const secs = this.timeToSeconds(s)
-        return secs === null ? null : `${secs}`
+        const secs = this.isBareNumber(s) ? Number(s) : this.timeToSeconds(s)
+        return secs === null
+          ? null
+          : this.helperService.sasTemporalLiteral(secs, 'TIME')
       }
       case 'date': {
-        if (this.isBareNumber(s)) return `${Number(s)}`
-        const d = this.parseDate(s, order)
-        return d
-          ? `${this.helperService.convertJsDateToSasDate(d, 'days')}`
-          : null
+        let days: number | null = null
+        if (this.isBareNumber(s)) days = Number(s)
+        else {
+          const d = this.parseDate(s, order)
+          if (d) days = this.helperService.convertJsDateToSasDate(d, 'days')
+        }
+        return days === null
+          ? null
+          : this.helperService.sasTemporalLiteral(days, 'DATE')
       }
       case 'datetime': {
-        if (this.isBareNumber(s)) return `${Number(s)}`
-        const d = this.parseDatetime(s, order)
-        return d
-          ? `${this.helperService.convertJsDateToSasDate(d, 'seconds')}`
-          : null
+        let secs: number | null = null
+        if (this.isBareNumber(s)) secs = Number(s)
+        else {
+          const d = this.parseDatetime(s, order)
+          if (d) secs = this.helperService.convertJsDateToSasDate(d, 'seconds')
+        }
+        return secs === null
+          ? null
+          : this.helperService.sasTemporalLiteral(secs, 'DATETIME')
       }
       default:
         return `'${s.replace(/'/g, "''")}'`

client/src/app/shared/abort-modal/info-modal.component.html

@@ -28,6 +28,16 @@
     }
   </div>
   <div class="modal-footer">
+    @if (showServiceLauncher(data.sasService)) {
+      <button
+        type="button"
+        class="btn btn-primary"
+        (click)="openServiceInNewWindow()"
+      >
+        Show log
+      </button>
+    }
+
     @if (showConfiguratorButton(data.sasService)) {
       <button
         type="button"

client/src/app/shared/abort-modal/info-modal.component.spec.ts

@@ -0,0 +1,71 @@
+import { InfoModalComponent } from './info-modal.component'
+
+describe('InfoModalComponent', () => {
+  let component: InfoModalComponent
+  let sasService: any
+
+  const buildComponent = (sasServiceConfig: any = {}) => {
+    sasService = {
+      getSasjsConfig: () => sasServiceConfig,
+      openServiceInNewWindow: jasmine.createSpy('openServiceInNewWindow')
+    }
+
+    component = new InfoModalComponent(
+      {} as any,
+      {} as any,
+      sasService,
+      {} as any
+    )
+    component.data = {
+      modalTitle: 'Error',
+      message: 'startupservice failed',
+      sasService: 'public/startupservice',
+      details: null
+    }
+    component.ngOnInit()
+  }
+
+  describe('showServiceLauncher', () => {
+    beforeEach(() => buildComponent())
+
+    it('is shown for the startupservice, on every server type', () => {
+      expect(component.showServiceLauncher('public/startupservice')).toBeTrue()
+      expect(
+        component.showServiceLauncher('services/public/startupservice')
+      ).toBeTrue()
+    })
+
+    it('is not shown for other services or a missing service', () => {
+      expect(component.showServiceLauncher('public/getdata')).toBeFalse()
+      expect(component.showServiceLauncher('')).toBeFalse()
+      expect(component.showServiceLauncher(null)).toBeFalse()
+    })
+  })
+
+  describe('openServiceInNewWindow', () => {
+    it('runs the startupservice in a new window', () => {
+      buildComponent({ serverType: 'SASVIYA' })
+
+      component.openServiceInNewWindow()
+
+      expect(sasService.openServiceInNewWindow).toHaveBeenCalledWith(
+        'public/startupservice',
+        {}
+      )
+    })
+
+    it('passes the configured Viya context so the log comes from it', () => {
+      buildComponent({
+        serverType: 'SASVIYA',
+        contextName: 'SAS Job Execution compute context'
+      })
+
+      component.openServiceInNewWindow()
+
+      expect(sasService.openServiceInNewWindow).toHaveBeenCalledWith(
+        'public/startupservice',
+        { _contextname: 'SAS Job Execution compute context' }
+      )
+    })
+  })
+})

client/src/app/shared/abort-modal/info-modal.component.ts

@@ -71,6 +71,34 @@ export class InfoModalComponent implements OnInit {
     this.onConfirmModalClick.emit()
   }
 
+  /**
+   * Whether to show the `Show log` button.
+   *
+   * The startupservice failure modal is a dead end otherwise: the adapter
+   * returned no log, so the only way to see the backend reason is to run the
+   * service through the JES / STP web app in a browser tab. Offered on every
+   * server type - SAS 9, Viya and SASjs Server all accept the `_program` URL.
+   */
+  showServiceLauncher(sasService: string | null) {
+    return !!sasService?.includes('startupservice')
+  }
+
+  /**
+   * Runs the startupservice in a new browser window so the SAS log is
+   * available even when the adapter response carried none (e.g. a Viya
+   * execution task that timed out).
+   */
+  openServiceInNewWindow() {
+    const sasjsConfig = this.sasService.getSasjsConfig()
+    const params: { [name: string]: string } = {}
+
+    if (sasjsConfig.contextName) {
+      params['_contextname'] = sasjsConfig.contextName
+    }
+
+    this.sasService.openServiceInNewWindow(this.data.sasService || '', params)
+  }
+
   /**
    * Only on SAS9, opening a backend configurator/deploy page
    */

client/src/app/shared/dc-validator/dc-validator.ts

@@ -61,13 +61,13 @@ export class DcValidator {
     // injected separately — see its own doc comment for why.
     //
     // COLHEADERS is the authoritative grid order and the only safe key to
-    // build against: it is the variable list of the *output* dataset, where
-    // getdata.sas hoists the delete column and the PK to the front ("order
-    // delete var and pk fields at start of table"), whereas cols[].VARNUM
-    // describes the *source* table and cols[] itself can arrive sorted by
-    // NAME. Those orders diverge whenever the PK isn't already the source's
-    // first column, so rules are keyed by name and emitted in COLHEADERS
-    // order — never paired positionally against cols[] or VARNUM.
+    // build against: it is the variable list of the *output* dataset, and the
+    // editor prepends the client-owned delete column to it (see
+    // editor.component.ts's initSetup) before this constructor runs, whereas
+    // cols[].VARNUM describes the *source* table and cols[] itself can arrive
+    // sorted by NAME. Those orders diverge whenever the PK isn't already the
+    // source's first column, so rules are keyed by name and emitted in
+    // COLHEADERS order — never paired positionally against cols[] or VARNUM.
     //
     // A column with no parseable COLTYPE falls back to a plain text rule
     // rather than being dropped: getRules() is consumed positionally against

client/src/app/shared/dc-validator/utils/deleteRecordColumn.spec.ts

@@ -0,0 +1,75 @@
+import {
+  prepareDeleteRecordColumn,
+  seedDeleteRecordColumn
+} from './deleteRecordColumn'
+import { DELETE_RECORD_COLUMN_NAME } from './deleteRecordColumnRule'
+
+const response = (overrides: any = {}) => ({
+  data: {
+    sasparams: [{ COLHEADERS: 'PRIMARY_KEY_FIELD,SOME_CHAR' }],
+    cols: [],
+    $sasdata: { vars: {} },
+    ...overrides
+  }
+})
+
+describe('prepareDeleteRecordColumn', () => {
+  it('prepends the delete column to COLHEADERS', () => {
+    const res = response()
+
+    prepareDeleteRecordColumn(res)
+
+    expect(res.data.sasparams[0].COLHEADERS).toEqual(
+      `${DELETE_RECORD_COLUMN_NAME},PRIMARY_KEY_FIELD,SOME_CHAR`
+    )
+  })
+
+  it('throws when COLHEADERS already carries the delete column', () => {
+    const res = response({
+      sasparams: [{ COLHEADERS: `${DELETE_RECORD_COLUMN_NAME},SOME_CHAR` }]
+    })
+
+    expect(() => prepareDeleteRecordColumn(res)).toThrowError(/reserved column/)
+  })
+
+  it('throws when cols carries the delete column', () => {
+    const res = response({ cols: [{ NAME: DELETE_RECORD_COLUMN_NAME }] })
+
+    expect(() => prepareDeleteRecordColumn(res)).toThrowError(/reserved column/)
+  })
+
+  it('throws when $sasdata vars carries the delete column', () => {
+    const res = response({
+      $sasdata: { vars: { [DELETE_RECORD_COLUMN_NAME]: {} } }
+    })
+
+    expect(() => prepareDeleteRecordColumn(res)).toThrowError(/reserved column/)
+  })
+
+  it('leaves COLHEADERS untouched when it throws', () => {
+    const colHeaders = `${DELETE_RECORD_COLUMN_NAME},SOME_CHAR`
+    const res = response({ sasparams: [{ COLHEADERS: colHeaders }] })
+
+    expect(() => prepareDeleteRecordColumn(res)).toThrowError()
+    expect(res.data.sasparams[0].COLHEADERS).toEqual(colHeaders)
+  })
+})
+
+describe('seedDeleteRecordColumn', () => {
+  it("sets every row's delete flag to 'No'", () => {
+    const rows: any[] = [{ PRIMARY_KEY_FIELD: 1 }, { PRIMARY_KEY_FIELD: 2 }]
+
+    seedDeleteRecordColumn(rows)
+
+    expect(rows[0][DELETE_RECORD_COLUMN_NAME]).toEqual('No')
+    expect(rows[1][DELETE_RECORD_COLUMN_NAME]).toEqual('No')
+  })
+
+  it('overwrites a flag a row already carried', () => {
+    const rows: any[] = [{ [DELETE_RECORD_COLUMN_NAME]: 'Yes' }]
+
+    seedDeleteRecordColumn(rows)
+
+    expect(rows[0][DELETE_RECORD_COLUMN_NAME]).toEqual('No')
+  })
+})

client/src/app/shared/dc-validator/utils/deleteRecordColumn.ts

@@ -0,0 +1,64 @@
+import { Col } from '../models/col.model'
+import { DELETE_RECORD_COLUMN_NAME } from './deleteRecordColumnRule'
+
+/**
+ * The parts of a getdata response this module touches. Declared structurally
+ * so the editor and the multi-dataset loader can both use it without either
+ * owning the other's response type.
+ */
+interface DeleteColumnResponse {
+  data: {
+    sasparams: { COLHEADERS: string }[]
+    cols?: Col[]
+    $sasdata?: { vars?: Record<string, unknown> }
+  }
+}
+
+/**
+ * Asserts that getdata did NOT send the client-owned delete column, then
+ * prepends it to COLHEADERS so the grid's first column is the delete flag.
+ *
+ * `_____DELETE__THIS__RECORD_____` is created by the editor itself (see
+ * deleteRecordColumnRule.ts) and getdata must never emit it. Its presence in a
+ * response can only mean a real table column of that name, or a hook that
+ * re-added it - either way it would shadow the delete flag, so refuse the load
+ * rather than render a table whose delete column cannot be trusted. Callers
+ * run this inside a promise chain whose `.catch` surfaces the error (the
+ * editor sets `getdataError`).
+ *
+ * Call it BEFORE constructing a DcValidator or deriving headerArray: rules,
+ * `columnHeader` and `headerArray` are all keyed off COLHEADERS order, and the
+ * delete column is always index 0.
+ */
+export const prepareDeleteRecordColumn = (response: DeleteColumnResponse) => {
+  const sasparams = response.data.sasparams[0]
+
+  const sentByBackend =
+    sasparams.COLHEADERS.split(',').includes(DELETE_RECORD_COLUMN_NAME) ||
+    (response.data.cols || []).some(
+      (col) => col.NAME === DELETE_RECORD_COLUMN_NAME
+    ) ||
+    Object.prototype.hasOwnProperty.call(
+      response.data.$sasdata?.vars || {},
+      DELETE_RECORD_COLUMN_NAME
+    )
+
+  if (sentByBackend) {
+    throw new Error(
+      `getdata returned the reserved column ${DELETE_RECORD_COLUMN_NAME}. ` +
+        'That name is created by the editor itself and must not exist in the target table.'
+    )
+  }
+
+  sasparams.COLHEADERS = `${DELETE_RECORD_COLUMN_NAME},${sasparams.COLHEADERS}`
+}
+
+/**
+ * Seeds every row's delete flag to 'No' - getdata no longer sends the column,
+ * so the editor owns the value too (see deleteRecordColumnRule.ts).
+ */
+export const seedDeleteRecordColumn = (rows: any[]) => {
+  for (const row of rows) {
+    row[DELETE_RECORD_COLUMN_NAME] = 'No'
+  }
+}

client/src/app/shared/dc-validator/utils/deleteRecordColumnRule.ts

@@ -1,16 +1,21 @@
 import { DcValidation } from '../models/dc-validation.model'
 
 /**
- * `_____DELETE__THIS__RECORD_____` is a client-side UI concept (the delete
- * checkbox column) that getdata.sas synthesizes into vars3/vars4 for display
- * purposes only — it's never a real column of the underlying dataset, so
- * `%mp_getcols` (which builds `cols1`, the left side of the join into
- * `cols`) never has a row for it, and its rule can no longer travel via
- * `cols[].COLTYPE` (see issue #253's vertical-array fix). The rule is fixed
- * and data-independent, so it's injected directly here instead.
+ * `_____DELETE__THIS__RECORD_____` is a purely client-owned column - the
+ * delete checkbox column. getdata.sas does NOT emit it: the editor adds the
+ * name to COLHEADERS itself (see editor.component.ts's initSetup) and seeds
+ * every row to 'No'. A response that carries it is refused, because the name
+ * can then only come from a real table column of that name (or a hook that
+ * re-added it), which would silently shadow the delete flag.
+ *
+ * Its rule can never travel via `cols[].COLTYPE` - `%mp_getcols` (which
+ * builds `cols1`, the left side of the join into `cols`) has no row for it -
+ * so it is injected directly, here and in DcValidator's constructor.
  */
+export const DELETE_RECORD_COLUMN_NAME = '_____DELETE__THIS__RECORD_____'
+
 export const DELETE_RECORD_COLUMN_RULE: DcValidation = {
-  data: '_____DELETE__THIS__RECORD_____',
+  data: DELETE_RECORD_COLUMN_NAME,
   type: 'dropdown',
   source: ['No', 'Yes']
 }

client/src/app/shared/utils/service-url.spec.ts

@@ -0,0 +1,86 @@
+import { buildServiceUrl } from './service-url'
+
+describe('buildServiceUrl', () => {
+  const viyaConfig = {
+    serverUrl: 'https://sas.example.com',
+    appLoc: '/Public/app/dc',
+    executionPath: '/SASJobExecution',
+    service: 'public/startupservice'
+  }
+
+  it('builds the Viya JES URL for a service, with the log enabled', () => {
+    expect(buildServiceUrl(viyaConfig)).toBe(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
+    )
+  })
+
+  it('leaves serverUrl empty on SAS 9 so the browser resolves the origin', () => {
+    expect(
+      buildServiceUrl({
+        ...viyaConfig,
+        serverUrl: '',
+        executionPath: '/SASStoredProcess/do'
+      })
+    ).toBe(
+      '/SASStoredProcess/do/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
+    )
+  })
+
+  it('builds the SASjs Server URL from its execution path', () => {
+    expect(
+      buildServiceUrl({
+        ...viyaConfig,
+        executionPath: '/SASjsApi/stp/execute'
+      })
+    ).toBe(
+      'https://sas.example.com/SASjsApi/stp/execute/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
+    )
+  })
+
+  it('appends encoded parameters before the debug flag', () => {
+    expect(
+      buildServiceUrl({
+        ...viyaConfig,
+        params: { _contextname: 'SAS Job Execution compute context' }
+      })
+    ).toBe(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_contextname=SAS%20Job%20Execution%20compute%20context&_debug=131'
+    )
+  })
+
+  it('skips parameters with no value', () => {
+    expect(
+      buildServiceUrl({
+        ...viyaConfig,
+        params: { _contextname: undefined, admin: null, dcpath: 'sas' }
+      })
+    ).toBe(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&dcpath=sas&_debug=131'
+    )
+  })
+
+  it('omits the debug flag when debug is off', () => {
+    expect(buildServiceUrl({ ...viyaConfig, debug: false })).toBe(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice'
+    )
+  })
+
+  it('accepts a service with a leading slash', () => {
+    expect(
+      buildServiceUrl({ ...viyaConfig, service: '/public/startupservice' })
+    ).toBe(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
+    )
+  })
+
+  it('accepts the services/ prefix the request wrapper reports', () => {
+    expect(
+      buildServiceUrl({
+        ...viyaConfig,
+        service: 'services/public/startupservice'
+      })
+    ).toBe(
+      'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
+    )
+  })
+})

client/src/app/shared/utils/service-url.ts

@@ -0,0 +1,46 @@
+/**
+ * Builds the URL that runs a SAS service directly in a browser window.
+ *
+ * The adapter submits requests from JavaScript, and on Viya that can route the
+ * run through the Compute service (`_executionTasks=true`), where the SAS log
+ * is not returned to the caller and the request can time out before the
+ * service finishes. A browser request to the JES / STP web app runs the same
+ * `_program` without any task involved, and `_debug=131` returns the SAS log
+ * as the page body - which is what makes it useful when a service fails.
+ *
+ * `serverUrl` is left empty on SAS 9 when the app is served from the SAS
+ * server itself; the browser then resolves the path against the current
+ * origin.
+ */
+export interface ServiceUrlOptions {
+  serverUrl: string
+  appLoc: string
+  executionPath: string
+  service: string
+  params?: { [name: string]: string | number | null | undefined }
+  debug?: boolean
+}
+
+export const buildServiceUrl = ({
+  serverUrl,
+  appLoc,
+  executionPath,
+  service,
+  params = {},
+  debug = true
+}: ServiceUrlOptions): string => {
+  // Callers pass the service as the app knows it: `public/getdata` in the
+  // request path, but the request wrapper reports `services/public/getdata`
+  // on a failed request. Both resolve to the same program.
+  const servicePath = service.replace(/^\/+/, '').replace(/^services\//, '')
+  const paramString = Object.entries(params)
+    .filter(([, value]) => value !== null && value !== undefined)
+    .map(
+      ([name, value]) =>
+        `&${encodeURIComponent(name)}=${encodeURIComponent(String(value))}`
+    )
+    .join('')
+  const debugParam = debug ? '&_debug=131' : ''
+
+  return `${serverUrl}${executionPath}/?_program=${appLoc}/services/${servicePath}${paramString}${debugParam}`
+}

client/src/index.html

@@ -42,11 +42,13 @@
     * debug: Whether to enable debug on startup
 
     ## Viya only settings ##
-    * adminGroup: This group will have unrestricted access (Viya Only)
     * dcPath: The physical location in which to deploy the staging area (Viya only)
     * useComputeApi: Connection method (Viya only)
     * contextName: Compute Context (Viya only)
 
+    The admin group is not configured here - it is a DC setting (DC_ADMIN_GROUP),
+    shown on the System screen.
+
   -->
 
   <sasjs
@@ -58,8 +60,6 @@
     useComputeApi="null"
     runAsTask="true"
     contextName="SAS Job Execution compute context"
-    adminGroup="SASAdministrators"
-    hotLicenceKey="non-commercial-and-evaluation"
   >
   </sasjs>
 

client/src/styles.scss

@@ -1664,6 +1664,13 @@ app-requests-modal {
   padding: 20px 10px 5px 10px;
 }
 
+/* The SAS log pane is `white-space: pre`, so it never wraps.  The stock `xl`
+  width is a fixed 57.6rem, which forced left/right scrolling to read a log
+  line. */
+.requests-modal .modal-dialog {
+  width: 90vw;
+}
+
 .work-tables-dropdown button {
   color: var(--clr-nav-link-color, #8c8c8c) !important;
 }
@@ -2289,15 +2296,6 @@ app-alerts {
 }
 // ALERTS.COMPONENT END
 
-// MANUAL-DEPLOY.COMPONENT
-app-manual-deploy {
-  .clear-memory-button {
-    right: 10px;
-    top: 2px;
-  }
-}
-// MANUAL-DEPLOY.COMPONENT END
-
 // LINEAGE.COMPONENT
 app-lineage {
   .toggle-switch input[type='checkbox']:checked + label:before {
@@ -4255,6 +4253,30 @@ body[cds-theme='light'] {
     color: #c21d00;
   }
 
+  /* The SASjs tag dump drops onto its own line and reads as a name/value grid. */
+  &--grid {
+    flex-wrap: wrap;
+    align-items: baseline;
+  }
+
+  &__grid {
+    flex-basis: 100%;
+    display: grid;
+    grid-template-columns: auto 1fr;
+    gap: 1px 12px;
+    margin-top: 4px;
+    font-size: 12px;
+  }
+
+  &__grid-name {
+    color: #999;
+  }
+
+  &__grid-value {
+    color: #ccc;
+    word-break: break-word;
+  }
+
   &__icon--pending {
     color: #999;
   }
@@ -4997,6 +5019,57 @@ body[cds-theme='dark'] {
   border: 1px solid var(--cds-alias-status-warning) !important;
 }
 
+/* Derived (computed) columns: read-only, but computed at read time rather than
+  locked by policy.  Tinted and marked with an 'fx' glyph instead of the
+  read-only dimming, so the two do not read as the same thing.  Blue is the
+  info family - the primary key green and the warning ochre keep their meaning. */
+.handsontable th.dc-derived-header {
+  background: #d3e5f2 !important;
+  box-shadow: inset 0 -2px 0 #1a6ea3;
+}
+
+.handsontable th.dc-derived-header div::after {
+  content: 'fx';
+  margin-left: 4px;
+  font-size: 9px;
+  font-style: italic;
+  opacity: 0.7;
+}
+
+.handsontable td.dc-derived-cell {
+  background: #eef5fa !important;
+  font-style: italic;
+}
+
+/* the first derived column starts the computed block */
+.handsontable th.dc-derived-first {
+  border-left: 2px solid #1a6ea3 !important;
+}
+
+.handsontable td.dc-derived-cell.dc-derived-first {
+  box-shadow: inset 2px 0 0 #1a6ea3;
+}
+
+body[cds-theme='dark'] {
+  .handsontable th.dc-derived-header {
+    background: #2b4a5c !important;
+    box-shadow: inset 0 -2px 0 #4b9cc9;
+  }
+
+  .handsontable td.dc-derived-cell {
+    background: #24343d !important;
+    color: #dbe7ee;
+  }
+
+  .handsontable th.dc-derived-first {
+    border-left-color: #4b9cc9 !important;
+  }
+
+  .handsontable td.dc-derived-cell.dc-derived-first {
+    box-shadow: inset 2px 0 0 #4b9cc9;
+  }
+}
+
 .handsontable .numericListbox {
   text-align: right;
 }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "dcfrontend",
-  "version": "7.16.0",
+  "version": "7.19.0",
   "description": "Data Controller",
   "devDependencies": {
     "@nogoo9/gitleaks": "8.30.1-post.2",

sas/.sasjslint

@@ -11,5 +11,6 @@
   "noNestedMacros": true,
   "hasMacroParentheses": true,
   "strictMacroDefinition": true,
+  "ignoredMacros": ["webout", "mpeinit", "mpeterm"],
   "ignoreList":["sasjs/scratch"]
 }

sas/deploy/makeDataSas719.json

@@ -0,0 +1,10 @@
+{
+  "fromjs": [
+    {
+      "ADMIN": "all-users",
+      "DCPATH": "/tmp/dcdata",
+      "OUTFMT": "JSON",
+      "KEEPME": "Y"
+    }
+  ]
+}

sas/mocks/mock-licence.json

@@ -0,0 +1,4 @@
+{
+  "licenceKey": "G4H/DNTdEMkmCIxOrYkRYTQ1P4sWg0gs8Zk9rakxFNYDZcLyEPMJJo82YJQ/S24P2E/YPxaWsDZXgOrWKfeCDS7NrLcwsoFNa8KRF2WTlRgSMiG4sjBRX8zNEIvszj5DAq4LEgOUUAdfctNfpUcygbxT1dhqy6gU3hBu/xRuXGd36rhxQwfahaKkfaCg/rFplkrvqnjoJ6622BxWP/HG1dxZUE7HNJyT4tNaXzlC9U6u/aaJJ66+V+9511BwjM9ss4nyHuHoogVE1yrzLMlL94P3QoM1hwkGHnEHmFzg3zHbQG1f3pUs9YzK0ytsFKU7AK948gJrGbf37Zqa+Mz7EZf6XzqqXx0zQ4y9zAhLV9s4QG8JdhrByhfG5kefrs4L/BpgKq3ElgYL5EMkrLJUrlqWYI2ouxJuIGCQz41VCWXJ2S7pd39ljifc8Zj+t2Mn5EEkRVsrbdxjCkyoZOQ1ALofa1+auUI26Rm5rwEgrfhEocDmphJpO2F8lZgwDxbepSE6v/vEV/JQWBQTyrX10W+qOEu8OhvZrYJo+eIJMaO7q1pjfdrn3XhoH+BeB3Sy1sevGLXQZC+CzOtcvajetGGCDAabtTLc49ouWPUiUOAWPE3uYymsMoRZgpLIMxaUM86hoK0r9T6wSaP0bddK754sXN1jWwIDAHSGz5WO1ko=",
+  "activationKey": "MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQD2w8Ge6tMI82vaQ7x5ZdlwnTJycn0UFZoSgRwObaM0NC5wp9Y8lwkVzN8FUhg2oYSTpvyJpix+VGRj0AlNBwFAVcJG9VDPYh2LE3Us9ghRfkzFUiM2PqorHnzL5DwpailRtmFHoDQPGwwSmKncTo2QHql/K6tWe7g3Nv9uqQNks283hvDQ6PVGU4TaFTSRbjGX4j+JAR651J6wI6nY+9kNxOBwnL/yKTnJGmL8oqZ+oC45quZzgRW4dLIeTODcB4IZe2ruxZ9tvXpSuAESiOv5bBZyMJEKdKwsRdt6vk+bezkZBCzPq+ly892Ncq7x5u4/8dGcGxRxxp5TnIIx3hFoiXZ7svXJ7st6uHuKkuwYq+ebWh6bcvQAwH2wCP40twMjQgmU8nHHo6+pulxu1wVZBxtoyM4m3Lu0ZncTi6mF0duiPvpkNNAZzKOF98aUq3hxYp5PP28qrtWuPsgSdTNWJwnV2R288yAPx4WLUAGB9h889d/m67HUiKadmfG4oyUnChCU/mqeJ/5nQkK1n76ezRPGsTcMZIv2mmPLShA/j38lGMjR/3pzM2Ha6UFjJKpurpeDiyt5pUdmRBpQRoBUx/PHnbXBsO9PBXa0mjdq3UFbgb5iBD1Fz2YClZ/h0d4YlYUMSeharDrLfR15g3fGnzxrkR03zYeHkxMUi5vYHQIDAQABAoICAFxUg5pcH8lOF1FMHfXIY5MM5mbHKsC8knERCPfnEvVB/wxc6D60fow/I0Vg4blzNuFuTjiMykIcfpGQMiK1l8TEMdSODoYCHq6+gY3ootUnLj94i79MtCw9FJHPnKkcVUiDNqPvr/rk8l/06j2J0NMXffEc3FeYJCXYuLtlluTzdI4wueNY/wkoz+EvpQMzWXAMkhxHpIEAqr2aIn+TQDQ0MYxBQlqwDTSPYHFJxIVFlvYx2VdnecKSO2E1/76kOowhTzH3FL+hcWcI/TG7tJuXU1C3llV0sUVBgC1NjwjMkuEm3EN5/ffqgj/eJNJTR1r6tqSzxzMJD42WVmqlPhqEmmRE1yeaio6Ulm5EJLREYEY4/1EotdIO6PCXF1Xeo+10VJLcwi19vzMlz7vFB6vSJzJNYOaiwm4rgmF12+nNt18cHoFvegGPAoCJurUhXU+iV6LlKYB8J/+kpITleZwm4dEaa6EWxgFf4nZ4N+qRVqk4JfC+rN3DQgbPcII+z7axAb8xutq5rvzskGAmKcTEagMG+pnk9DI+0m1C2J6+mC/YruVlpSlPB7Ecu/OF++vJoSCLuDXVuxMTQ3tQXoicOG/S8d1aa3nq4fvLIgZTOZoffUvNvjCyy/Dlv6vaDVCjgqt2kFZVSZv3C6ttyrNVBSjhucASGTq/7q82JKVrAoIBAQD7e7SYqDM8SAjOfpRAHCbIVVSaHw1tul16B6JAF67OIs1uD//Y+maL4GHlpFxnib0tg6z3+sG10NMUhK7elB6SkJ8voT+Ztg/+on659vmaqwTlkGjgks+8DmIw1b0ZbXuIqhnkOeSsll2V9NWItIrbzNwvhtMEQsOtFvmrLlKQvx9Bo3lGmbfdU1/9XpwW4wiA5ONcFCl1xllX9nItB+ActPoAeQgnuejWAISQkITDTOFcGu0AjMjJ66HUlY/WDbf7wY/s2aMbOiVx7B6YzQnK+lZdjjNfoaL8nfN8Z8ZzJFAgmTQ9ztZCJ8fLTfZDsYD6XZnVEIG401+6de0Ea4MHAoIBAQD7Mlr++keNDxYy3k6kwTXDOSKMDfXIDverFy/tiNorcVU5nIpuhjowl4HIQVAhzlXhATp938LM/PvJIeu9HtgsY9BFNL52fr7iGEWknqZQVE+8Um35tzV3vtPH9NVXHIYmcRCAQQJ5a6oVQRJKAM3E1+htM2AeZKMaQSCC6VHGBXJG9nEltdMKTet09pd26Z5y/YIfTqbIekyjRomGdInyWBkpJLJ/uROVb9OGnsYWadDktn7y9+oH+VZoJxW5QwXV0OL7wMzymlrKhhBNMziJRBk4rVztclYzUShpwE8Xfjt5WYkOoieXtmkdpF7TMvRYy6cdflh39l8s7THq4U67AoIBAQDPShR+kIZK9gVsUCHbBllhYBSwRVBIVgmfAJPYyKyp9T3nrwBw1VWZjoajR2tzXLG2xobYkLUDLPIQQ53wkZ7xqIvOMte917gvZ7mVh+IOf1E1+Ga6VcktdZ2yKzwHrZrZ93wIB0BakkbpyDoKyOZ8bsw/0J43MOEh14JMe45LCX62GA1rCGHp6UOCEW4P32uu8vwFdgkGr0wNmXJbc9ZjUzmPEjSKqk8ltp+OmdEVOlW4GempPmJzLeri07UP95bjyZTc4jfzxWYAFWTTBo8/5EE0NHZcKVuzZm7A6LlHR3NUq3PITqmFNk0VY1nOSo3J0UVvvn9u3axWgcWwkv4ZAoIBAQD5GaB/RfZ6fcGHxEi8vRTxedE2jglz0NPuCFpw9ctUVVksfn4QyG8gDL6YtZwfShyDutIkTSdCXIXbmRXLP7Ejo/+/AWT24Q2t7F/ztZg8BL/lguCoR/RvwP8HRVXiwtiNeoXgiZdIjWnnFdZrKIzlbS6CS/Y/bZEI0hjJeVKsSm5ezLi7qqmTiF28aWUeeOG2KotFPJvMJuuk5qgue5boilC5KjkQy5oOEQz0bLMHOvv6MreFfi/IpIv5UPkL9NjuhezxtuoGkucB+xxjUi34gV15hQGoPkAZN5wZ8qj8JSQZXJ67sjYNDskCqiOW0Orau2oHp2lX3SvYw4dC4xdNAoIBAQDSXF8OnWu08smGDs4S57yfqe59N0ULIBDOZdEQcLXRaDeAoOwQj8szUf0pycXUZLvXxv29iF61FJg70iGGCp3lW+9Oe0P+dGsKvzphRGsQPZy4h/qZDoQ9dE667ejI4aDb/C1GaOZ8TT1QfpQcVKbgLjdC19IGQPhFljGM1wMQojU5lp4/iKmA3J3ahOuV4dvEUr6xkqzq+PhH405XdURXsVc1J6qYX2wQXRoB0oDsrCbqidZi14DaeE8UhZRA6m9haUOb/3Y+OkCs2QIW4MNZ+zg8XtHjiUhMxAaJ6zdEFltWPoVFsVfgoecquAHb0uDJJRz/+RfR0AiG0XG0NW+C"
+}

sas/mocks/sas9/sasjs/services/editors/getdata.js

@@ -14,7 +14,6 @@ function makeRows(n) {
     const rows = []
     for (let i = 0; i < n; i++) {
         rows.push({
-            _____DELETE__THIS__RECORD_____: "No",
             PRIMARY_KEY_FIELD: i,
             SOME_CHAR: "dummy data row " + i,
             SOME_DROPDOWN: dropdowns[i % dropdowns.length],
@@ -241,7 +240,6 @@ const data = {
         { ODS_TABLE: "ENGINEHOST", NAME: "File Size (bytes)", VALUE: "1048576" }
     ],
     maxvarlengths: [
-        { NAME: "_____DELETE__THIS__RECORD_____", MAXLEN: 3 },
         { NAME: "PRIMARY_KEY_FIELD", MAXLEN: 4 },
         { NAME: "some_char", MAXLEN: 591 },
         { NAME: "some_dropdown", MAXLEN: 8 },
@@ -261,7 +259,6 @@ const data = {
     sasdata: makeRows(100),
     $sasdata: {
         vars: {
-            _____DELETE__THIS__RECORD_____: { format: "$3.", label: "_____DELETE__THIS__RECORD_____", length: "3", type: "char" },
             PRIMARY_KEY_FIELD: { format: "best.", label: "PRIMARY_KEY_FIELD", length: "8", type: "num" },
             SOME_CHAR: { format: "$32767.", label: "SOME_CHAR", length: "32767", type: "char" },
             SOME_DROPDOWN: { format: "$128.", label: "SOME_DROPDOWN", length: "128", type: "char" },
@@ -280,14 +277,14 @@ const data = {
     },
     sasparams: [
         {
-            COLHEADERS: "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,SOME_HARDSELECT,SOME_NUM,SOME_DATE,SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM,READONLY_COL,HIDDEN_COL,ROUND_COL,NUMFMT_COL",
+            COLHEADERS: "PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,SOME_HARDSELECT,SOME_NUM,SOME_DATE,SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM,READONLY_COL,HIDDEN_COL,ROUND_COL,NUMFMT_COL",
             FILTER_TEXT: "",
             PKCNT: 1,
             PK: "PRIMARY_KEY_FIELD",
             DTVARS: " SOME_DATE",
             DTTMVARS: " SOME_DATETIME",
             TMVARS: " SOME_TIME",
-            COLTYPE: "{\"data\":\"_____DELETE__THIS__RECORD_____\",\"type\":\"dropdown\",\"source\":[\"No\",\"Yes\"]},{\"data\":\"PRIMARY_KEY_FIELD\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_CHAR\"},{\"data\":\"SOME_DROPDOWN\"},{\"data\":\"SOME_HARDSELECT\"},{\"data\":\"SOME_NUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_DATE\",\"type\":\"date\"},{\"data\":\"SOME_DATETIME\",\"type\":\"datetime\"},{\"data\":\"SOME_TIME\",\"type\":\"time\"},{\"data\":\"SOME_SHORTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_BESTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"READONLY_COL\"},{\"data\":\"HIDDEN_COL\"},{\"data\":\"ROUND_COL\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"NUMFMT_COL\",\"type\":\"numeric\",\"format\":\"0\"}",
+            COLTYPE: "{\"data\":\"PRIMARY_KEY_FIELD\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_CHAR\"},{\"data\":\"SOME_DROPDOWN\"},{\"data\":\"SOME_HARDSELECT\"},{\"data\":\"SOME_NUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_DATE\",\"type\":\"date\"},{\"data\":\"SOME_DATETIME\",\"type\":\"datetime\"},{\"data\":\"SOME_TIME\",\"type\":\"time\"},{\"data\":\"SOME_SHORTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_BESTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"READONLY_COL\"},{\"data\":\"HIDDEN_COL\"},{\"data\":\"ROUND_COL\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"NUMFMT_COL\",\"type\":\"numeric\",\"format\":\"0\"}",
             LOADTYPE: "UPDATE",
             RK_FLAG: 0,
             CLS_FLAG: 0

sas/mocks/sasjs/mock-storage/licence.json

@@ -1,4 +1,4 @@
 {
-  "licenceKey": "invalid",
-  "activationKey": "invalid"
-}
+  "licenceKey": "",
+  "activationKey": ""
+}

sas/mocks/sasjs/sasjsconfig.json

@@ -16,7 +16,9 @@
         "streamWebFolder": "web9",
         "webSourcePath": "../../client/dist",
         "assetPaths": [],
-        "streamServiceName": "clickme"
+        "streamServiceName": "clickme",
+        "mockWeb": true,
+        "mockWebFolder": "sasjsbuild/mock"
       }
     },
     {

sas/mocks/sasjs/services/admin/makedata.js

@@ -281,6 +281,53 @@ const tables = {
       DD_SENSITIVITY: 'Low'
     }
   ],
+  // Derived (computed) variables.  Values are display-time only - the logic is
+  // applied by mpe_derived in the mock's getdata/getcolvals, and must use the
+  // expression subset that compileSasExpression supports (the real evaluation
+  // is done by SAS).
+  MPE_DERIVED: [
+    {
+      tx_from: 0,
+      tx_to: 253717919999,
+      libref: 'TESTDATA',
+      dsn: 'DERIVED_DEMO',
+      var_nm: 'REGION',
+      var_deps: 'STATE',
+      var_type: 'C',
+      var_len: 8,
+      var_fmt: '',
+      var_logic: "ifc(STATE in ('CA','OR','WA'),'West','East')",
+      active: 1
+    },
+    {
+      tx_from: 0,
+      tx_to: 253717919999,
+      libref: 'TESTDATA',
+      dsn: 'DERIVED_DEMO',
+      var_nm: 'LINE_VALUE',
+      var_deps: 'QTY PRICE',
+      var_type: 'N',
+      var_len: 8,
+      var_fmt: 'dollar12.2',
+      var_logic: 'QTY * PRICE',
+      active: 1
+    },
+    {
+      // built on REGION, so it proves the dependency order is applied
+      tx_from: 0,
+      tx_to: 253717919999,
+      libref: 'TESTDATA',
+      dsn: 'DERIVED_DEMO',
+      var_nm: 'REGION_LABEL',
+      var_deps: 'REGION STATE',
+      var_type: 'C',
+      var_len: 14,
+      var_fmt: '',
+      var_label: 'Region and state',
+      var_logic: "cats(REGION,'-',STATE)",
+      active: 1
+    }
+  ],
   MPE_DATALOADS: [],
   MPE_EMAILS: [],
   MPE_EXCEL_CONFIG: [],
@@ -1112,6 +1159,26 @@ const tables = {
       selectbox_value: 'FORMULA',
       selectbox_order: 1,
       ver_to_dttm: 127490111999
+    },
+    {
+      selectbox_rk: 73,
+      ver_from_dttm: 0,
+      select_lib: dcLibref,
+      select_ds: 'MPE_DERIVED',
+      base_column: 'VAR_TYPE',
+      selectbox_value: 'C',
+      selectbox_order: 1,
+      ver_to_dttm: 127490111999
+    },
+    {
+      selectbox_rk: 74,
+      ver_from_dttm: 0,
+      select_lib: dcLibref,
+      select_ds: 'MPE_DERIVED',
+      base_column: 'VAR_TYPE',
+      selectbox_value: 'N',
+      selectbox_order: 2,
+      ver_to_dttm: 127490111999
     }
   ],
   MPE_SUBMIT: [],
@@ -1324,6 +1391,19 @@ const tables = {
         'RLS_SCOPE RLS_GROUP RLS_LIBREF RLS_TABLE RLS_GROUP_LOGIC RLS_SUBGROUP_LOGIC RLS_SUBGROUP_ID RLS_VARIABLE_NM RLS_OPERATOR_NM RLS_RAW_VALUE',
       post_edit_hook: 'services/hooks/mpe_row_level_security_postedit'
     },
+    {
+      tx_from: 0,
+      tx_to: 127490111999,
+      libref: dcLibref,
+      dsn: 'MPE_DERIVED',
+      num_of_approvals_required: 1,
+      loadtype: 'TXTEMPORAL',
+      buskey: 'LIBREF DSN VAR_NM',
+      notes: 'Derived (computed) variables for the EDIT screen',
+      var_txfrom: 'TX_FROM',
+      var_txto: 'TX_TO',
+      post_edit_hook: 'services/hooks/mpe_derived_postedit'
+    },
     {
       tx_from: 0,
       tx_to: 127490111999,
@@ -1955,6 +2035,206 @@ const tables = {
       rule_value: 'services/validations/mpe_x_test.some_num',
       rule_active: 1,
       tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'LIBREF',
+      rule_type: 'NOTNULL',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'LIBREF',
+      rule_type: 'CASE',
+      rule_value: 'UPCASE',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'DSN',
+      rule_type: 'NOTNULL',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'DSN',
+      rule_type: 'CASE',
+      rule_value: 'UPCASE',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_NM',
+      rule_type: 'NOTNULL',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_NM',
+      rule_type: 'CASE',
+      rule_value: 'UPCASE',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_NM',
+      rule_type: 'HARDREGEX',
+      rule_value: '^[A-Za-z_][A-Za-z0-9_]*$',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_DEPS',
+      rule_type: 'CASE',
+      rule_value: 'UPCASE',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_DEPS',
+      rule_type: 'HARDREGEX',
+      rule_value: '^[A-Za-z0-9_ ]*$',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_TYPE',
+      rule_type: 'NOTNULL',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_TYPE',
+      rule_type: 'CASE',
+      rule_value: 'UPCASE',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_TYPE',
+      rule_type: 'HARDSELECT',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_LEN',
+      rule_type: 'NOTNULL',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_LEN',
+      rule_type: 'MINVAL',
+      rule_value: '1',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_LEN',
+      rule_type: 'MAXVAL',
+      rule_value: '32767',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_FMT',
+      rule_type: 'CASE',
+      rule_value: 'UPCASE',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_FMT',
+      rule_type: 'HARDREGEX',
+      rule_value: '^(\$?[A-Za-z][A-Za-z0-9_]*(\.[0-9]+)?\.?|\$?[0-9]+(\.[0-9]+)?\.?)?$',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_LOGIC',
+      rule_type: 'NOTNULL',
+      rule_value: ' ',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_LABEL',
+      rule_type: 'HARDREGEX',
+      rule_value: '^[^\\x00-\\x1F\\x7F]*$',
+      rule_active: 1,
+      tx_to: 127490111999
+    },
+    {
+      tx_from: 0,
+      base_lib: dcLibref,
+      base_ds: 'MPE_DERIVED',
+      base_col: 'VAR_LOGIC',
+      rule_type: 'HARDREGEX',
+      rule_value: '^(?!.*(/\\*|\\*/))[^;&%]*$',
+      rule_active: 1,
+      tx_to: 127490111999
     }
   ],
   MPE_X_TEST: [
@@ -2226,7 +2506,7 @@ tables.MPE_TABLES.push({
 
 // ─── MPE_X_WIDE: wide table for the EDIT cell limit ─────────────────────────
 // 200 rows x 1001 columns (ROW_ID + 500 single-character + 500 numeric) is
-// 200,400 cells - over DC_MAXCELLS_WEBEDIT (200000) while staying under
+// 200,200 cells - over DC_MAXCELLS_WEBEDIT (200000) while staying under
 // DC_MAXOBS_WEBEDIT (250), so it exercises the CELL limit on its own and the
 // row limit spec can use MPE_X_SEARCH (1000 rows) instead.  Values are one
 // character / one digit on purpose: the point is width, and small values keep
@@ -3262,6 +3542,20 @@ const schema = {
       notnull: false
     }
   ],
+  MPE_DERIVED: [
+    { name: 'TX_FROM', type: 'N', length: 8, format: 'datetime19.3', label: '', notnull: true },
+    { name: 'LIBREF', type: 'C', length: 8, format: '', label: '', notnull: false },
+    { name: 'DSN', type: 'C', length: 32, format: '', label: '', notnull: false },
+    { name: 'VAR_NM', type: 'C', length: 32, format: '', label: '', notnull: false },
+    { name: 'VAR_DEPS', type: 'C', length: 4000, format: '', label: '', notnull: false },
+    { name: 'VAR_TYPE', type: 'C', length: 1, format: '', label: '', notnull: false },
+    { name: 'VAR_LEN', type: 'N', length: 8, format: '', label: '', notnull: false },
+    { name: 'VAR_FMT', type: 'C', length: 32, format: '', label: '', notnull: false },
+    { name: 'VAR_LABEL', type: 'C', length: 256, format: '', label: '', notnull: false },
+    { name: 'VAR_LOGIC', type: 'C', length: 4000, format: '', label: '', notnull: false },
+    { name: 'ACTIVE', type: 'N', length: 8, format: '', label: '', notnull: false },
+    { name: 'TX_TO', type: 'N', length: 8, format: 'datetime19.3', label: '', notnull: true }
+  ],
   MPE_DATALOADS: [
     {
       name: 'LIBREF',
@@ -5349,7 +5643,15 @@ const testTables = {
     { ORDER_ID: 1003, CUSTOMER: 'Cobalt GmbH', REGION: 'EU', AMOUNT: 620, STATUS: 'CLOSED' },
     { ORDER_ID: 1004, CUSTOMER: 'Delta Systems', REGION: 'US', AMOUNT: 4500, STATUS: 'NEW' }
   ],
-  DEMO_MIRROR: []
+  DEMO_MIRROR: [],
+  // Derived variable demo - REGION and LINE_VALUE are not columns of this
+  // table, they are computed at read time from MPE_DERIVED rows.
+  DERIVED_DEMO: [
+    { ID: 1, STATE: 'CA', QTY: 2, PRICE: 10.5 },
+    { ID: 2, STATE: 'NY', QTY: 1, PRICE: 99 },
+    { ID: 3, STATE: 'OR', QTY: 5, PRICE: 3 },
+    { ID: 4, STATE: 'TX', QTY: 3, PRICE: 7.25 }
+  ]
 }
 
 // Schemas for the TESTDATA tables, matching the old inline getdata.js payload
@@ -5691,6 +5993,12 @@ const testSchema = {
       label: 'Last reviewed',
       notnull: false
     }
+  ],
+  DERIVED_DEMO: [
+    { name: 'ID', type: 'N', length: 8, format: '', label: 'ID', notnull: true },
+    { name: 'STATE', type: 'C', length: 2, format: '', label: 'State', notnull: false },
+    { name: 'QTY', type: 'N', length: 8, format: '', label: 'Quantity', notnull: false },
+    { name: 'PRICE', type: 'N', length: 8, format: 'dollar12.2', label: 'Price', notnull: false }
   ]
 }
 
@@ -5739,6 +6047,16 @@ tables.MPE_TABLES.push(
     buskey: 'ID',
     notes: 'Compact rule demo table (eight columns)'
   },
+  {
+    tx_from: 0,
+    tx_to: 127490111999,
+    libref: testDataLibref,
+    dsn: 'DERIVED_DEMO',
+    num_of_approvals_required: 1,
+    loadtype: 'UPDATE',
+    buskey: 'ID',
+    notes: 'Derived variable demo table'
+  },
   {
     tx_from: 0,
     tx_to: 127490111999,

sas/mocks/sasjs/services/auditors/getauditfile.js

@@ -0,0 +1,104 @@
+const nodePath = require('path')
+
+let appLoc = nodePath.join(..._program.split('services')[0].split('/'))
+const sasjsRoot = nodePath.resolve(weboutPath, '..', '..', '..')
+const driveRoot = nodePath.resolve(sasjsRoot, 'drive')
+
+// Load shared DC mock utilities
+eval(
+  fs.readFileSync(
+    nodePath.resolve(driveRoot, 'files', appLoc, 'services', 'dcMockUtils.js'),
+    'utf8'
+  )
+)
+
+const dcConf = loadDcSettings(driveRoot, appLoc)
+const mpeDataDir = dcConf.dataDir
+const stagingDir = nodePath.resolve(mpeDataDir, 'staging')
+const mpeLoadTableData = makeTableLoader(mpeDataDir)
+
+// ─── Parse input ──────────────────────────────────────────────────────────────
+// The submission id arrives as the `table` URL param, as in getauditfile.sas.
+
+let tableId = ''
+if (typeof table !== 'undefined' && table) tableId = String(table).trim()
+
+// libds comes from MPE_SUBMIT, the same lookup the SAS service makes before it
+// decides the request is valid.
+let libds = ''
+if (tableId) {
+  const submit = mpeLoadTableData('MPE_SUBMIT')
+  const row =
+    submit && submit.rows
+      ? submit.rows.find((r) => r.TABLE_ID === tableId)
+      : null
+  if (row) {
+    libds = (String(row.BASE_LIB || '') + '.' + String(row.BASE_DS || '')).toUpperCase()
+  }
+}
+
+// ─── Respond ──────────────────────────────────────────────────────────────────
+
+if (!tableId || !libds) {
+  // getauditfile.sas aborts with the same message when either is missing.
+  webOutOpen()
+  webOutObj([{ MSG: 'Missing: libds table', MAC: 'getauditfile' }], 'sasjsAbort')
+  webOutObj([{ STATUS: 'ERROR', DSID: '' }], 'sasparams')
+  webOutClose()
+} else {
+  // The SAS service also checks EDIT or APPROVE access (mpe_accesscheck). The
+  // mock seeds MPE_SECURITY but no mock service enforces it, so access is not
+  // emulated here.
+
+  // The approval folder is zipped as it stands. getauditfile.sas lists it with
+  // mp_dirlist and adds each file to the package under contents/.
+  const stageFolder = nodePath.resolve(stagingDir, tableId)
+  const names = []
+  if (fs.existsSync(stageFolder)) {
+    for (const name of fs.readdirSync(stageFolder)) {
+      const absolute = nodePath.resolve(stageFolder, name)
+      let stat
+      try {
+        stat = fs.statSync(absolute)
+      } catch (err) {
+        continue
+      }
+      if (stat.isFile()) names.push(name)
+    }
+  }
+  names.sort()
+
+  const entries = names.map((name) => ({
+    name: 'contents/' + name,
+    content: fs
+      .readFileSync(nodePath.resolve(stageFolder, name), { encoding: 'utf8' })
+      .toString()
+  }))
+
+  const archive = mpZipPackage(entries)
+  const streamed = mpStreamfile({
+    filename: tableId + '.zip',
+    contentType: 'application/zip',
+    bytes: archive
+  })
+
+  if (!streamed) {
+    // The browser host returns a text body and ignores the streamed headers, so
+    // the archive cannot reach the browser as a download. The pack is described
+    // instead, which is what a reviewer opening the tab gets there.
+    _webout = JSON.stringify(
+      {
+        table: tableId,
+        libds: libds,
+        archive: tableId + '.zip',
+        note: 'The browser mock host streams text only, so this lists the archive contents.',
+        contents: entries.map((entry) => ({
+          name: entry.name,
+          bytes: entry.content.length
+        }))
+      },
+      null,
+      2
+    )
+  }
+}

sas/mocks/sasjs/services/dcMockUtils.js

@@ -460,6 +460,60 @@ function sasVarsEntry(col) {
     return { format: fmt, label: col.label || col.name, length: String(col.length), type: col.type === 'N' ? 'num' : 'char' }
 }
 
+/**
+ * Assigns the retained key (RK) to staged rows for a table registered with
+ * rk_underlying, mirroring %mp_retainedkey as %mpe_loader invokes it: any RK
+ * column in the input is dropped, and every row's RK is derived from its
+ * business key - the value that key already holds on the base table, or the
+ * next key value for a new one. Business keys are matched the way SAS compares
+ * them: case-sensitive, trailing blanks ignored.
+ *
+ * @param {Object} opts - { rows, tableReg, baseData }
+ *   tableReg - the table's MPE_TABLES row (needs rk_underlying and buskey)
+ *   baseData - { columns, rows } of the base table, for the key lookup
+ * @returns {String} the RK column name when the assignment ran, else ''
+ */
+function mpeAssignRetainedKey(opts) {
+    const { rows, tableReg, baseData } = opts
+    const rkUnderlying = tableReg && tableReg.rk_underlying
+        ? String(tableReg.rk_underlying).trim()
+        : ''
+    const rkCol = tableReg && tableReg.buskey ? String(tableReg.buskey).trim() : ''
+    if (!rkUnderlying || !rkCol || !rows || rows.length === 0) return ''
+
+    const base = baseData || { columns: [], rows: [] }
+    const submitted = Object.keys(rows[0] || {}).find(
+        (k) => k.toUpperCase() === rkCol.toUpperCase()
+    )
+    const rkName = submitted ||
+        (base.columns.find((c) => c.name.toUpperCase() === rkCol.toUpperCase()) || {}).name ||
+        rkCol
+    const keys = rkUnderlying.split(/\s+/)
+    const keyValue = (value) => {
+        if (value === null || value === undefined) return ''
+        return typeof value === 'number' ? String(value) : String(value).replace(/\s+$/, '')
+    }
+    const businessKey = (row) => keys.map((k) => keyValue(colVal(row, k))).join('|')
+
+    const baseByKey = new Map()
+    let nextRk = 0
+    for (const row of base.rows) {
+        const rk = colVal(row, rkCol)
+        baseByKey.set(businessKey(row), rk)
+        const num = Number(rk)
+        if (isFinite(num) && num > nextRk) nextRk = num
+    }
+
+    for (const row of rows) {
+        Object.keys(row).forEach((k) => {
+            if (k.toUpperCase() === rkCol.toUpperCase()) delete row[k]
+        })
+        const existing = baseByKey.get(businessKey(row))
+        row[rkName] = existing === undefined || existing === null ? ++nextRk : existing
+    }
+    return rkName
+}
+
 /**
  * Stages a submission: writes staged.json under staging/<tableId>/, copies the
  * session audit items, and inserts the MPE_SUBMIT row. Shared by the
@@ -493,6 +547,12 @@ function stageSubmission(opts) {
     // keep audit items from this Stored Program run alongside the staged data
     copySessionAuditItems(stageFolder, 'submit')
 
+    // Files the SAS loader leaves in the approval folder, so the audit pack
+    // holds the same artefacts: the cleaned CSV and the uploaded file itself.
+    for (const file of opts.auditFiles || []) {
+        fs.writeFileSync(nodePath.resolve(stageFolder, file.name), file.content)
+    }
+
     // Insert a record into MPE_SUBMIT (always in the control library /
     // mpelib, regardless of which library the base table lives in)
     const mpeDataDir = libDataDir('DC_JSLIB')
@@ -549,6 +609,146 @@ function copySessionAuditItems(stageFolder, prefix) {
     }
 }
 
+// ─── %mp_streamfile + ODS package (zip) ───────────────────────────────────────
+
+const ZIP_CRC_TABLE = (() => {
+    const table = new Array(256)
+    for (let i = 0; i < 256; i++) {
+        let c = i
+        for (let k = 0; k < 8; k++) c = (c & 1) ? (0xedb88320 ^ (c >>> 1)) : (c >>> 1)
+        table[i] = c >>> 0
+    }
+    return table
+})()
+
+function zipCrc32(bytes) {
+    let crc = -1
+    for (let i = 0; i < bytes.length; i++)
+        crc = (crc >>> 8) ^ ZIP_CRC_TABLE[(crc ^ bytes[i]) & 0xff]
+    return (crc ^ -1) >>> 0
+}
+
+function zipUtf8(value) {
+    const text = String(value)
+    if (typeof TextEncoder !== 'undefined') return new TextEncoder().encode(text)
+    const out = []
+    for (let i = 0; i < text.length; i++) {
+        const code = text.charCodeAt(i)
+        if (code < 0x80) out.push(code)
+        else if (code < 0x800) out.push(0xc0 | (code >> 6), 0x80 | (code & 0x3f))
+        else out.push(0xe0 | (code >> 12), 0x80 | ((code >> 6) & 0x3f), 0x80 | (code & 0x3f))
+    }
+    return Uint8Array.from(out)
+}
+
+/**
+ * Renders rows as CSV, the counterpart of %mp_ds2csv: the header row, then one
+ * line per row, with values quoted only when they carry a delimiter, quote or
+ * line break.
+ *
+ * @param {string[]} headers column names, in order
+ * @param {Array<Object>} rows objects keyed by column name
+ * @returns {string} CSV text with a trailing newline
+ */
+function toCsv(headers, rows) {
+    const cell = (value) => {
+        const text = value === undefined || value === null ? '' : String(value)
+        return /[",\n\r]/.test(text) ? '"' + text.replace(/"/g, '""') + '"' : text
+    }
+    const lines = [headers.map(cell).join(',')]
+    for (const row of rows) {
+        lines.push(headers.map((header) => cell(row[header])).join(','))
+    }
+    return lines.join('\n') + '\n'
+}
+
+/**
+ * Builds a zip archive in memory, entries stored rather than deflated - the JS
+ * counterpart of the ODS package getauditfile.sas publishes. The zip format
+ * leaves compression to the writer, so a stored archive opens anywhere.
+ *
+ * @param {Array<{name: string, content: string}>} entries archive entries
+ * @returns {Uint8Array} the archive
+ */
+function mpZipPackage(entries) {
+    const u16 = (n) => [n & 0xff, (n >>> 8) & 0xff]
+    const u32 = (n) => [n & 0xff, (n >>> 8) & 0xff, (n >>> 16) & 0xff, (n >>> 24) & 0xff]
+
+    const local = []
+    const central = []
+    let offset = 0
+
+    for (const entry of entries) {
+        const name = zipUtf8(entry.name)
+        const data = zipUtf8(entry.content)
+        const crc = zipCrc32(data)
+        // Bit 11 marks the name as UTF-8. Without it an extractor decodes a
+        // non-ASCII name as CP437, so the entry cannot be read back by name.
+        const flags = name.some((byte) => byte >= 0x80) ? 0x0800 : 0
+
+        // 1980-01-01 00:00, the earliest DOS timestamp: a stable value keeps
+        // the archive byte-identical for identical inputs.
+        const header = [].concat(
+            u32(0x04034b50), u16(20), u16(flags), u16(0), u16(0), u16(0x21),
+            u32(crc), u32(data.length), u32(data.length), u16(name.length), u16(0)
+        )
+
+        local.push(Uint8Array.from(header), name, data)
+        central.push({ name, crc, size: data.length, offset, flags })
+        offset += header.length + name.length + data.length
+    }
+
+    const directory = []
+    let directorySize = 0
+    for (const entry of central) {
+        const header = [].concat(
+            u32(0x02014b50), u16(20), u16(20), u16(entry.flags), u16(0), u16(0), u16(0x21),
+            u32(entry.crc), u32(entry.size), u32(entry.size),
+            u16(entry.name.length), u16(0), u16(0), u16(0), u16(0), u32(0),
+            u32(entry.offset)
+        )
+        directory.push(Uint8Array.from(header), entry.name)
+        directorySize += header.length + entry.name.length
+    }
+
+    const end = Uint8Array.from([].concat(
+        u32(0x06054b50), u16(0), u16(0),
+        u16(central.length), u16(central.length),
+        u32(directorySize), u32(offset), u16(0)
+    ))
+
+    const chunks = local.concat(directory, [end])
+    const total = chunks.reduce((sum, chunk) => sum + chunk.length, 0)
+    const archive = new Uint8Array(total)
+    let at = 0
+    for (const chunk of chunks) {
+        archive.set(chunk, at)
+        at += chunk.length
+    }
+    return archive
+}
+
+/**
+ * Streams a file back as an attachment, the JS counterpart of %mp_streamfile on
+ * the SASJS platform: the headers name the content type and the body is the file
+ * itself. Returns false where that is not possible - the browser mock host has
+ * a text-only webout - so the caller can fall back to something textual.
+ *
+ * @param {{filename: string, contentType: string, bytes: Uint8Array}} opts
+ * @returns {boolean} whether the file was streamed
+ */
+function mpStreamfile(opts) {
+    if (typeof Buffer === 'undefined') return false
+    // A content-type header is what tells the SASjs Server to send webout.txt as
+    // binary rather than a string (api/src/controllers/internal/Execution.ts).
+    fs.writeFileSync(
+        _SASJS_WEBOUT_HEADERS,
+        `Content-type: ${opts.contentType} \n Content-Disposition: attachment;filename="${opts.filename}"`
+    )
+    fs.writeFileSync(weboutPath, Buffer.from(opts.bytes))
+    return true
+}
+
 /**
  * Generates a DC table_id in the same format as the real backend
  * (DC<yyyymmdd-hhmmss.sss>_<rand6>_<pid4>).
@@ -580,6 +780,7 @@ function makeTableId() {
  *   ('A','B')   -> ["A","B"]        (IN list)
  *   (1,2,3)     -> [1,2,3]          (IN list, numeric)
  *   1 AND 5     -> [1, 5]           (BETWEEN)
+ *   '20DEC2053'd-> 34322            (date / datetime / time literal)
  */
 function parseFilterRawValue(raw, operator) {
     if (raw === undefined || raw === null || raw === '') return null
@@ -598,8 +799,40 @@ function parseFilterRawValue(raw, operator) {
     return parseFilterScalar(s)
 }
 
+/**
+ * SAS date / datetime / time literal -> the SAS internal value, or null.
+ *   34322 <- '20DEC2053'd
+ *   42    <- '01JAN1960:00:00:42'dt
+ *   42    <- '00:00:42't
+ * The client builds these in canonical ddMMMyyyy form (HelperService
+ * .sasTemporalLiteral), so only that shape needs parsing here.
+ */
+const TEMPORAL_LITERAL_RE = /^'(\d{2})([A-Z]{3})(\d{4})(?::(\d{2}):(\d{2}):(\d{2})(\.\d+)?)?'(dt|d|t)$/i
+/* A time literal carries no date part, so it is matched separately */
+const TIME_LITERAL_RE = /^'(\d{2}):(\d{2}):(\d{2})(\.\d+)?'t$/i
+
+function parseTemporalLiteral(s) {
+    const t = TIME_LITERAL_RE.exec(s)
+    if (t) {
+        return Number(t[1]) * 3600 + Number(t[2]) * 60 + Number(t[3]) +
+            (t[4] ? Number(t[4]) : 0)
+    }
+    const m = TEMPORAL_LITERAL_RE.exec(s)
+    if (!m) return null
+    const month = MONTHS_ABBR.indexOf(m[2].toUpperCase())
+    if (month < 0) return null
+    const days = (Date.UTC(Number(m[3]), month, Number(m[1])) - SAS_EPOCH_MS) / 86400000
+    const suffix = m[8].toLowerCase()
+    if (suffix === 'd') return days
+    const secs = Number(m[4]) * 3600 + Number(m[5]) * 60 + Number(m[6]) +
+        (m[7] ? Number(m[7]) : 0)
+    return suffix === 'dt' ? days * 86400 + secs : secs
+}
+
 function parseFilterScalar(s) {
     s = s.trim()
+    const temporal = parseTemporalLiteral(s)
+    if (temporal !== null) return temporal
     if (s.startsWith("'") && s.endsWith("'")) {
         return s.slice(1, -1).replace(/''/g, "'")
     }
@@ -1121,6 +1354,289 @@ function mpFilterStore(opts) {
     }
 }
 
+// ─── mpe_derived.sas ────────────────────────────────────────────────────────
+
+/**
+ * Compiles a SAS expression into a JS function - for the mock only.
+ *
+ * The real evaluation is done by SAS. This covers the subset used by the mock
+ * and demo configs, so the EDIT screen can be exercised (display, filtering,
+ * submission) without a SAS server:
+ *
+ *   - column names, numbers, quoted strings
+ *   - + - * / and comparisons  (= ne < > <= >=)
+ *   - AND OR, and `X IN (a, b, c)`
+ *   - ifc(), cats(), catx(), upcase(), lowcase(), strip(), trim(), length(),
+ *     substr(), index()
+ *
+ * Anything outside that subset throws when a row is derived, and the column is
+ * left missing for that row - the same way a failing SAS expression would.
+ */
+function compileSasExpression(expr, columnNames) {
+    // Tokenise first, so operators are only rewritten outside quoted strings
+    // (a literal such as 'OR' must never be read as an operator).
+    const tokens = []
+    const s = String(expr)
+    let i = 0
+    while (i < s.length) {
+        const c = s[i]
+        if (/\s/.test(c)) { i++; continue }
+        if (c === "'" || c === '"') {
+            const q = c
+            let j = i + 1
+            let val = ''
+            while (j < s.length) {
+                if (s[j] === q) {
+                    if (s[j + 1] === q) { val += q; j += 2; continue }
+                    break
+                }
+                val += s[j]
+                j++
+            }
+            tokens.push({ type: 'str', value: val })
+            i = j + 1
+            continue
+        }
+        if (/[0-9]/.test(c)) {
+            let j = i
+            while (j < s.length && /[0-9.]/.test(s[j])) j++
+            tokens.push({ type: 'num', value: s.slice(i, j) })
+            i = j
+            continue
+        }
+        if (/[A-Za-z_]/.test(c)) {
+            let j = i
+            while (j < s.length && /[A-Za-z0-9_]/.test(s[j])) j++
+            tokens.push({ type: 'word', value: s.slice(i, j) })
+            i = j
+            continue
+        }
+        const two = s.slice(i, i + 2)
+        if (['<=', '>=', '!=', '<>'].includes(two)) {
+            tokens.push({ type: 'punct', value: two })
+            i += 2
+            continue
+        }
+        // only the operators in the documented subset may reach the generated
+        // function - anything else is rejected rather than executed
+        if (!'+-*/(),<>='.includes(c)) {
+            throw new Error('Unsupported character in derived expression: ' + c)
+        }
+        tokens.push({ type: 'punct', value: c })
+        i++
+    }
+
+    // X IN (a, b, c)  ->  __in([a, b, c], X)
+    const out = []
+    for (let t = 0; t < tokens.length; t++) {
+        const tok = tokens[t]
+        if (
+            tok.type === 'word' &&
+            tok.value.toLowerCase() === 'in' &&
+            out.length > 0 &&
+            tokens[t + 1] &&
+            tokens[t + 1].value === '('
+        ) {
+            let depth = 0
+            let j = t + 1
+            for (; j < tokens.length; j++) {
+                if (tokens[j].value === '(') depth++
+                else if (tokens[j].value === ')') {
+                    depth--
+                    if (depth === 0) break
+                }
+            }
+            if (j < tokens.length) {
+                const left = out.pop()
+                const inner = tokens.slice(t + 2, j)
+                out.push(
+                    { type: 'word', value: '__in' },
+                    { type: 'punct', value: '(' },
+                    { type: 'punct', value: '[' },
+                    ...inner,
+                    { type: 'punct', value: ']' },
+                    { type: 'punct', value: ',' },
+                    left,
+                    { type: 'punct', value: ')' }
+                )
+                t = j
+                continue
+            }
+        }
+        out.push(tok)
+    }
+
+    const OPS = {
+        and: '&&', or: '||',
+        eq: '==', ne: '!=', lt: '<', gt: '>', le: '<=', ge: '>='
+    }
+    const js = out
+        .map(tok => {
+            if (tok.type === 'str') return JSON.stringify(tok.value)
+            if (tok.type === 'num') return tok.value
+            if (tok.type === 'word') return OPS[tok.value.toLowerCase()] || tok.value
+            if (tok.value === '=') return '=='
+            if (tok.value === '<>') return '!='
+            return tok.value
+        })
+        .join(' ')
+
+    const sasTrue = v => {
+        if (typeof v === 'boolean') return v
+        if (typeof v === 'number') return v !== 0 && !isNaN(v)
+        if (v === null || v === undefined) return false
+        const str = String(v).trim()
+        return str !== '' && str !== '0'
+    }
+    const toStr = v => {
+        if (v === null || v === undefined) return ''
+        if (typeof v === 'number') return isNaN(v) ? '' : String(v)
+        return String(v).trim()
+    }
+    const fns = {
+        ifc: (c, a, b) => (sasTrue(c) ? a : b),
+        cats: (...args) => args.map(toStr).join(''),
+        catx: (sep, ...args) => args.map(toStr).filter(v => v !== '').join(toStr(sep)),
+        upcase: v => toStr(v).toUpperCase(),
+        lowcase: v => toStr(v).toLowerCase(),
+        strip: v => toStr(v),
+        trim: v => toStr(v),
+        length: v => toStr(v).length,
+        substr: (v, p, n) => {
+            const str = toStr(v)
+            const start = Number(p) - 1
+            return n === undefined
+                ? str.substring(start)
+                : str.substring(start, start + Number(n))
+        },
+        index: (v, sub) => toStr(v).indexOf(toStr(sub)) + 1,
+        __in: (list, v) => list.some(item => String(item) === String(v) || Number(item) === Number(v))
+    }
+
+    const names = Object.keys(fns)
+    const fn = new Function(...columnNames, ...names, 'return (' + js + ');')
+    return row => fn(...columnNames.map(n => row[n]), ...names.map(n => fns[n]))
+}
+
+/**
+ * Reads MPE_DERIVED for a table, mirroring %mpe_derived.  Returns the active
+ * variables with a compiled row function, in dependency order (ties by name),
+ * the same sequence the real macro emits.
+ *
+ * @param {Object} opts
+ * @param {string} opts.libref - target libref
+ * @param {string} opts.table - target dataset
+ * @param {string} opts.dataDir - data directory of the DC control library
+ * @param {Array<Object>} opts.columns - physical columns of the target table
+ */
+function mpeDerived(opts) {
+    const { libref, table, dataDir, columns } = opts
+    const loadTableData = makeTableLoader(dataDir)
+    const cfg = loadTableData('MPE_DERIVED')
+    if (!cfg || !Array.isArray(cfg.rows) || cfg.rows.length === 0) return []
+
+    const now = Date.now() / 1000
+    const lib = String(libref || '').toUpperCase()
+    const dsn = String(table || '').toUpperCase()
+    const columnNames = (columns || []).map(c => c.name)
+
+    const vars = cfg.rows
+        .filter(r =>
+            String(colVal(r, 'LIBREF') || '').toUpperCase() === lib &&
+            String(colVal(r, 'DSN') || '').toUpperCase() === dsn &&
+            Number(colVal(r, 'ACTIVE')) === 1 &&
+            (!Number(colVal(r, 'TX_TO')) || Number(colVal(r, 'TX_TO')) > now)
+        )
+        .map(r => {
+            const logic = String(colVal(r, 'VAR_LOGIC') || '').trim()
+            const nm = String(colVal(r, 'VAR_NM') || '').toUpperCase()
+            return {
+                name: nm,
+                // every input the expression uses - columns and derived vars
+                deps: String(colVal(r, 'VAR_DEPS') || '')
+                    .toUpperCase()
+                    .split(/\s+/)
+                    .filter(Boolean),
+                type: String(colVal(r, 'VAR_TYPE') || '').toUpperCase(),
+                length: Number(colVal(r, 'VAR_LEN')) || 8,
+                format: String(colVal(r, 'VAR_FMT') || ''),
+                logic,
+                label: String(colVal(r, 'VAR_LABEL') || '').trim() || nm + '=' + logic
+            }
+        })
+        .filter(d => d.name && (d.type === 'C' || d.type === 'N') && d.logic)
+
+    // dependency order: repeatedly take the variables whose derived inputs are
+    // already placed, alphabetically within a round
+    const byName = new Map(vars.map(v => [v.name, v]))
+    const placed = []
+    const done = new Set()
+    while (placed.length < vars.length) {
+        const ready = vars
+            .filter(v => !done.has(v.name))
+            .filter(v => v.deps.every(d => !byName.has(d) || done.has(d)))
+            .sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0))
+        if (ready.length === 0) {
+            console.log('derived variables for ' + lib + '.' + dsn + ' contain a cycle')
+            return []
+        }
+        for (const v of ready) {
+            placed.push(v)
+            done.add(v.name)
+        }
+    }
+    vars.length = 0
+    vars.push.apply(vars, placed)
+
+    // compile each against the physical columns plus the variables defined
+    // earlier, so a derived variable can build on another one
+    const compileCols = columnNames.slice()
+    for (const d of vars) {
+        try {
+            // slice(): the compiled function captures the column list it was
+            // built with, not the array as it grows
+            d.fn = compileSasExpression(d.logic, compileCols.slice())
+        } catch (err) {
+            console.log('derived variable ' + d.name + ' could not be compiled: ' + err.message)
+            d.fn = () => (d.type === 'N' ? null : '')
+        }
+        compileCols.push(d.name)
+    }
+    return vars
+}
+
+/**
+ * Adds the derived variables to every row and to the column list, so that the
+ * filter (which runs afterwards) and the grid both see them.
+ */
+function mpeDeriveRows(tableData, derived) {
+    if (!derived || derived.length === 0) return tableData
+    const rows = tableData.rows.map(row => {
+        const copy = Object.assign({}, row)
+        for (const d of derived) {
+            try {
+                copy[d.name] = d.fn(copy)
+            } catch (err) {
+                copy[d.name] = d.type === 'N' ? null : ''
+            }
+        }
+        return copy
+    })
+    const derivedCols = derived.map(d => ({
+        name: d.name,
+        type: d.type,
+        length: d.length,
+        format: d.format,
+        label: d.label,
+        notnull: false
+    }))
+    return {
+        name: tableData.name,
+        columns: (tableData.columns || []).concat(derivedCols),
+        rows
+    }
+}
+
 // ─── DC settings (services/settings.js) ──────────────────────────────────────
 
 /**

sas/mocks/sasjs/services/editors/getdata.js

@@ -77,6 +77,42 @@ if (mpeTablesData && mpeTablesData.rows) {
     tableReg = mpeTablesData.rows.find(r => r.libref === requestedLibref && r.dsn === requestedTable)
 }
 
+// The variables exposed to the editor as the primary key, matching getdata.sas:
+// pk = coalesce(rk_underlying, buskey). A table registered with rk_underlying
+// is keyed on its business key in the editor - the RK itself is assigned by the
+// backend loader, so it is neither shown nor sent by the user.
+const rkUnderlying = tableReg && tableReg.rk_underlying ? tableReg.rk_underlying.trim() : ''
+const buskey = tableReg && tableReg.buskey ? tableReg.buskey.trim() : ''
+const pkFields = rkUnderlying || buskey
+const pkColumns = pkFields ? pkFields.split(/\s+/) : []
+
+// Order columns with the PK variables first, in PK order, then the remaining
+// columns in schema order. getdata.sas does the same ("order delete var and pk
+// fields at start of table") and the client reads the first PKCNT columns
+// positionally when it detects duplicate keys.
+function pkFirst(columns, keys) {
+    const rank = new Map(keys.map((name, i) => [name.toUpperCase(), i]))
+    const keyColumns = columns
+        .filter(col => rank.has(col.name.toUpperCase()))
+        .sort((a, b) => rank.get(a.name.toUpperCase()) - rank.get(b.name.toUpperCase()))
+    return keyColumns.concat(columns.filter(col => !rank.has(col.name.toUpperCase())))
+}
+
+// ─── Derived (computed) variables (mpe_derived.sas) ─────────────────────────
+// Applied before the filter, so a filter can reference them.  The derived
+// columns are added to the schema too, which is what puts them in COLS and in
+// the grid; they are held read-only by a READONLY rule below.
+
+const derived = mpeDerived({
+    libref: requestedLibref,
+    table: requestedTable,
+    dataDir: mpeDataDir,
+    columns: tableData ? tableData.columns : []
+})
+if (tableData && derived.length > 0) {
+    tableData = mpeDeriveRows(tableData, derived)
+}
+
 // ─── DQ rules (from MPE_VALIDATIONS + NOTNULL from schema) ──────────────────
 // Mirrors getdata.sas: dqrules = MPE_VALIDATIONS rows UNION dictionary.columns
 // NOTNULL constraints. MPE_VALIDATIONS lives in the control library.
@@ -85,10 +121,7 @@ const validationsData = mpeLoadTableData('MPE_VALIDATIONS')
 let dqrules = []
 let dqdata = []
 if (validationsData && validationsData.rows) {
-    const pkCols = new Set()
-    if (tableReg && tableReg.buskey) {
-        tableReg.buskey.trim().split(/\s+/).forEach(c => pkCols.add(c.toUpperCase()))
-    }
+    const pkCols = new Set(pkColumns.map(c => c.toUpperCase()))
     dqrules = validationsData.rows
         .filter(r => r.base_lib === requestedLibref && r.base_ds === requestedTable)
         .map(r => ({
@@ -127,6 +160,16 @@ if (tableData && tableData.columns) {
     }
 }
 
+// A derived variable is computed at read time, so the grid must not accept a
+// value for it.  COLS carries READ for the same reason (a derived column has no
+// CLS rule row), but the client only applies COLS.CLS_RULE when CLS_FLAG is
+// set, which column-level security drives - so the rule is declared here too,
+// as getdata.sas does.  dqrules is the request-local array the filter/map above
+// built, so appending to it leaves the loaded MPE_VALIDATIONS rows untouched.
+for (const v of derived) {
+    dqrules.push({ BASE_COL: v.name, RULE_TYPE: 'READONLY', RULE_VALUE: '' })
+}
+
 // ─── DQ dropdown data (from MPE_SELECTBOX + SOFTSELECT from table data) ──────
 // MPE_SELECTBOX lives in the control library.
 
@@ -191,9 +234,8 @@ if (tableData && tableData.rows && tableData.columns) {
 // ─── Build response ──────────────────────────────────────────────────────────
 
 let response
-// Set when the row or cell limit is exceeded - mirrors the %mp_abort in
-// getdata.sas, and is emitted as sasjsAbort below.
-let abortMsg = ''
+// Set when the row or cell limit cut the selection down - mirrors the
+// TRUNCATED / TRUNC_MSG in sasparams in getdata.sas.
 
 if (tableData && tableData.columns && tableData.rows) {
     // Exclude temporal columns (TX_FROM/TX_TO etc.) from the editor view
@@ -206,8 +248,17 @@ if (tableData && tableData.columns && tableData.rows) {
         if (tableReg.var_busfrom) excludeCols.add(tableReg.var_busfrom.toUpperCase())
         if (tableReg.var_busto) excludeCols.add(tableReg.var_busto.toUpperCase())
     }
+    // getdata.sas drops the RK field from the editor view when the table is
+    // registered with rk_underlying (drop_rk=&buskey), and exposes the business
+    // key instead.
+    if (rkUnderlying && buskey) {
+        buskey.split(/\s+/).forEach(c => excludeCols.add(c.toUpperCase()))
+    }
 
-    let visibleColumns = tableData.columns.filter(c => !excludeCols.has(c.name.toUpperCase()))
+    let visibleColumns = pkFirst(
+        tableData.columns.filter(c => !excludeCols.has(c.name.toUpperCase())),
+        pkColumns
+    )
     let visibleRows = tableData.rows
         .filter(r => filterPredicate(r))
         .map(r => {
@@ -236,15 +287,15 @@ if (tableData && tableData.columns && tableData.rows) {
     }
 
     // ─── Row and cell limits ─────────────────────────────────────────────────
-    // Mirrors getdata.sas: DC_MAXOBS_WEBEDIT rows, or DC_MAXCELLS_WEBEDIT
-    // cells (rows x columns), whichever is reached first.  Both are read from
-    // MPE_CONFIG, the way the real service picks them up from the settings
-    // program it generates - so editing the option in DC changes what the mock
-    // allows, and an option that is absent (or switched off, var_active=0)
-    // falls back to the shipped default.  The check runs after the
-    // PRE_EDIT_HOOK, as the single check in getdata.sas does - a hook may
-    // replace the data with a larger table.  The cell count includes the
-    // housekeeping delete column, which is what getdata.sas counts.
+    // Mirrors getdata.sas: DC_MAXOBS_WEBEDIT rows or DC_MAXCELLS_WEBEDIT
+    // cells (rows x columns), whichever is reached first - the selection is
+    // cut to min(rows, cells/columns) and flagged in sasparams.
+    // Both options are read from MPE_CONFIG, the way the real service picks
+    // them up from the settings program it generates - so editing the option
+    // in DC changes what the mock serves, and an option that is absent (or
+    // switched off, var_active=0) falls back to the shipped default.  The
+    // check runs after the PRE_EDIT_HOOK, as the single check in getdata.sas
+    // does - a hook may replace the data with a larger table.
     // MPE_CONFIG lives in the control library, not in the table's own library,
     // so this reads mpeDataDir - the same directory the MPE_TABLES lookup above
     // uses.  Passing the requested table's dataDir made the read fail for every
@@ -252,20 +303,32 @@ if (tableData && tableData.columns && tableData.rows) {
     const dcCfg = loadDcConfig(driveRoot, appLoc, mpeDataDir)
     const maxObsWebEdit = dcConfigNumber(dcCfg, 'DC_MAXOBS_WEBEDIT', 250)
     const maxCellsWebEdit = dcConfigNumber(dcCfg, 'DC_MAXCELLS_WEBEDIT', 200000)
-    const editColCount = visibleColumns.length + 1
-    const cellCount = visibleRows.length * editColCount
-    if (visibleRows.length > maxObsWebEdit) {
-        abortMsg =
-            'Table is too big (' +
-            visibleRows.length +
-            ' rows) - please filter and try again!'
-    } else if (cellCount > maxCellsWebEdit) {
-        abortMsg =
-            'Selection is too wide (' +
-            visibleRows.length +
-            ' rows x ' +
-            editColCount +
-            ' cols) - please filter and try again!'
+    // getdata.sas counts the columns of work.out, i.e. the table's own
+    // columns - the delete column is client-owned and not part of it.
+    const editColCount = visibleColumns.length
+    let maxRows = maxObsWebEdit
+    let limitVar = 'DC_MAXOBS_WEBEDIT'
+    if (editColCount > 0) {
+        const cellRows = Math.floor(maxCellsWebEdit / editColCount)
+        if (cellRows < maxRows) {
+            maxRows = cellRows
+            limitVar = 'DC_MAXCELLS_WEBEDIT'
+        }
+    }
+    if (maxRows < 1) maxRows = 1
+    let truncated = 0
+    let truncMsg = ''
+    if (visibleRows.length > maxRows) {
+        truncated = 1
+        // Same wording as the macro getdata.sas sets, so the spec can assert
+        // on the text the real service would send.
+        truncMsg =
+            'Only the first ' +
+            maxRows +
+            ' rows are loaded - the ' +
+            limitVar +
+            ' limit has been reached.  Filter the table to work with the rest.'
+        visibleRows = visibleRows.slice(0, maxRows)
     }
 
     // Augment rows: add delete flag, normalise keys, convert temporal values to ISO
@@ -280,16 +343,16 @@ if (tableData && tableData.columns && tableData.rows) {
             const targetKey = info ? info.name : key
             normalised[targetKey] = info ? formatCellValue(r[key], info.ddtype) : r[key]
         }
-        return { _____DELETE__THIS__RECORD_____: 'No', ...normalised }
+        return { ...normalised }
     })
 
     // $sasdata vars: temporal columns are char ($200.), others keep their SAS type
-    const vars = { _____DELETE__THIS__RECORD_____: { format: '$3.', label: '_____DELETE__THIS__RECORD_____', length: '3', type: 'char' } }
+    const vars = {}
     for (const col of visibleColumns) {
         vars[col.name] = sasVarsEntry(col)
     }
 
-    const colHeaders = ['_____DELETE__THIS__RECORD_____', ...visibleColumns.map(c => c.name)]
+    const colHeaders = visibleColumns.map(c => c.name)
 
     // Determine DTVARS, DTTMVARS, TMVARS from column formats
     let dtVars = [], dttmVars = [], tmVars = []
@@ -300,11 +363,14 @@ if (tableData && tableData.columns && tableData.rows) {
         else if (ddtype === 'TIME') tmVars.push(col.name)
     }
 
-    // Build cols array.  mpGetcols is the JS counterpart of %mp_getcols.sas -
-    // the macro getdata.sas itself calls to type these columns - so the editor
-    // sees the same DDTYPE the service would give it.
-    const cols = mpGetcols(visibleColumns).map((c, i) => {
-        const raw = visibleColumns[i]
+    // Build cols array.  Every column of the table, as getdata.sas emits them:
+    // the temporal columns are dropped from the data (the rows below) but not
+    // from the cols payload, so a filter can still be built on them.  mpGetcols
+    // is the JS counterpart of %mp_getcols.sas - the macro getdata.sas itself
+    // calls to type these columns - so the editor sees the same DDTYPE the
+    // service would give it.
+    const cols = mpGetcols(tableData.columns).map((c, i) => {
+        const raw = tableData.columns[i]
         const ddtype = c.DDTYPE === 'CHARACTER' ? 'C' : c.DDTYPE === 'NUMERIC' ? 'N' : c.DDTYPE
         let coltype
         if (ddtype === 'DATE') coltype = `{"data":"${raw.name}","type":"date"}`
@@ -327,7 +393,6 @@ if (tableData && tableData.columns && tableData.rows) {
 
     // Build maxvarlengths
     const maxvarlengths = [
-        { NAME: '_____DELETE__THIS__RECORD_____', MAXLEN: 3 },
         ...visibleColumns.map(col => {
             let maxLen = 0
             const lcName = col.name.toLowerCase()
@@ -361,15 +426,21 @@ if (tableData && tableData.columns && tableData.rows) {
         sasparams: [{
             COLHEADERS: colHeaders.join(','),
             FILTER_TEXT: filterText === '' ? ' ' : filterText,
-            PKCNT: tableReg && tableReg.buskey ? tableReg.buskey.trim().split(/\s+/).length : 0,
-            PK: tableReg ? tableReg.buskey || '' : '',
+            PKCNT: pkColumns.length,
+            PK: pkFields,
             DTVARS: dtVars.length ? ' ' + dtVars.join(' ') : '',
             DTTMVARS: dttmVars.length ? ' ' + dttmVars.join(' ') : '',
             TMVARS: tmVars.length ? ' ' + tmVars.join(' ') : '',
             LOADTYPE: tableReg ? tableReg.loadtype || 'UPDATE' : 'UPDATE',
             RK_FLAG: tableReg && tableReg.rk_underlying ? 1 : 0,
             CLS_FLAG: 0,
-            ISMAP: 0
+            ISMAP: 0,
+            TRUNCATED: truncated,
+            TRUNC_MSG: truncMsg,
+            // names the derived columns for the grid, as getdata.sas does - the
+            // READONLY rule above is why they cannot be edited, this is how the
+            // EDIT screen knows they are computed rather than locked
+            DERIVEDVARS: derived.map(d => d.name).join(' ')
         }],
         versions: [],
         xl_rules: []
@@ -384,26 +455,21 @@ if (tableData && tableData.columns && tableData.rows) {
         ],
         maxvarlengths: [], query: [], sasdata: [],
         $sasdata: { vars: {} },
-        sasparams: [{ COLHEADERS: '', FILTER_TEXT: ' ', PKCNT: 0, PK: '', DTVARS: '', DTTMVARS: '', TMVARS: '', LOADTYPE: 'UPDATE', RK_FLAG: 0, CLS_FLAG: 0, ISMAP: 0 }],
+        sasparams: [{ COLHEADERS: '', FILTER_TEXT: ' ', PKCNT: 0, PK: '', DTVARS: '', DTTMVARS: '', TMVARS: '', LOADTYPE: 'UPDATE', RK_FLAG: 0, CLS_FLAG: 0, ISMAP: 0, TRUNCATED: 0, TRUNC_MSG: '', DERIVEDVARS: '' }],
         versions: [], xl_rules: []
     }
 }
 
 webOutOpen()
-if (abortMsg) {
-    webOutObj([{ MSG: abortMsg, MAC: 'getdata' }], 'sasjsAbort')
-    webOutObj([{ STATUS: 'ERROR' }], 'sasparams')
-} else {
-    webOutObj(response.approvers, 'approvers')
-    webOutObj(response.cols, 'cols')
-    webOutObj(response.dqdata, 'dqdata')
-    webOutObj(response.dqrules, 'dqrules')
-    webOutObj(response.dsmeta, 'dsmeta')
-    webOutObj(response.maxvarlengths, 'maxvarlengths')
-    webOutObj(response.query, 'query')
-    webOutObj(response.sasdata, 'sasdata', response.$sasdata)
-    webOutObj(response.sasparams, 'sasparams')
-    webOutObj(response.versions, 'versions')
-    webOutObj(response.xl_rules, 'xl_rules')
-}
+webOutObj(response.approvers, 'approvers')
+webOutObj(response.cols, 'cols')
+webOutObj(response.dqdata, 'dqdata')
+webOutObj(response.dqrules, 'dqrules')
+webOutObj(response.dsmeta, 'dsmeta')
+webOutObj(response.maxvarlengths, 'maxvarlengths')
+webOutObj(response.query, 'query')
+webOutObj(response.sasdata, 'sasdata', response.$sasdata)
+webOutObj(response.sasparams, 'sasparams')
+webOutObj(response.versions, 'versions')
+webOutObj(response.xl_rules, 'xl_rules')
 webOutClose()

sas/mocks/sasjs/services/editors/loadfile.js

@@ -59,6 +59,9 @@ for (let i = 1; i <= 10; i++) {
 
 let libds = ''
 if (typeof table !== 'undefined' && table) libds = String(table).toUpperCase()
+// captured before any hook can re-point the submission: the cleaned CSV is
+// named after the original table, as loadfile.sas names it
+const origLibds = libds
 let libref = libds.split('.')[0] || dcLibref
 let dsn = libds.split('.')[1] || ''
 
@@ -147,6 +150,12 @@ if (!msg) {
                     if (tableEntry.var_txfrom) computedCols.push(tableEntry.var_txfrom.toUpperCase())
                     if (tableEntry.var_txto) computedCols.push(tableEntry.var_txto.toUpperCase())
                     if (tableEntry.var_processed) computedCols.push(tableEntry.var_processed.toUpperCase())
+                    // loadfile.sas drops the RK from the comparison for a table
+                    // with rk_underlying (the loader derives it), so an upload
+                    // for one legitimately omits it.
+                    if (tableEntry.rk_underlying && tableEntry.buskey) {
+                        computedCols.push(String(tableEntry.buskey).toUpperCase())
+                    }
                 }
             }
             const baseCols = schema.map((c) => c.name.toUpperCase())
@@ -172,6 +181,9 @@ if (!msg) {
         // written, while LIBREF / DS are still ordinary macro variables, so it
         // can re-point the changeset at a different table.  The mock eval()s the
         // hook's .js counterpart in this scope so it can reassign libref / dsn.
+        // A validating hook rejects the submission by setting hookErrmsg (see
+        // hooks/mpe_derived_postedit.js), which stops the staging below.
+        let hookErrmsg = ''
         const hookTables = mpeLoadTableData('MPE_TABLES')
         const hookReg = (hookTables && hookTables.rows)
             ? hookTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -179,19 +191,41 @@ if (!msg) {
         if (hookReg && hookReg.post_edit_hook) {
             const hookSrc = mockHookSource(hookReg.post_edit_hook)
             if (hookSrc) {
+                // a hook reads and normalises the submitted rows as jsDataRows
+                // (stagedata.js names them that); here they are `rows`
+                const jsDataRows = rows
                 try {
                     eval(hookSrc)
                 } catch (err) {
-                    console.log(
+                    // a hook that throws has not validated anything, so the
+                    // submission is refused rather than staged unchecked
+                    hookErrmsg =
                         'post_edit_hook ' + hookReg.post_edit_hook + ' failed: ' + err.message
-                    )
                 }
                 libds = libref + '.' + dsn
             }
         }
+        if (hookErrmsg) msg = hookErrmsg
 
         tableId = makeTableId()
-        stageSubmission({
+        // ─── RK assignment (mpe_loader) ──────────────────────────────────────
+        // The uploaded file carries the business key and no RK (see the
+        // expected-columns check above), so the loader derives it before the
+        // rows are staged.
+        const finalTables = mpeLoadTableData('MPE_TABLES')
+        const finalReg =
+            finalTables && finalTables.rows
+                ? finalTables.rows.find((r) => r.libref === libref && r.dsn === dsn)
+                : null
+        const rkName = mpeAssignRetainedKey({
+            rows,
+            tableReg: finalReg,
+            baseData: makeTableLoader(libDataDir(libref))(dsn)
+        })
+        if (rkName && !headers.some((h) => h.toUpperCase() === rkName.toUpperCase())) {
+            headers.push(rkName)
+        }
+        if (!msg) stageSubmission({
             dataDir: libDataDir(libref),
             stagingDir,
             tableId,
@@ -200,7 +234,12 @@ if (!msg) {
             dsn,
             message: 'File upload: ' + csvName,
             headers,
-            rows
+            rows,
+            // loadfile.sas leaves the cleaned CSV (%mp_cleancsv) in the approval
+            // folder, named after the table as the request found it. It copies
+            // the upload as BKP_<name> only when a second, non-CSV fileref came
+            // with the request, which the mock does not accept - so no BKP here.
+            auditFiles: [{ name: origLibds + '.csv', content: csvText }]
         })
     }
 }

sas/mocks/sasjs/services/editors/stagedata.js

@@ -22,6 +22,9 @@ let message = _sctRow.MESSAGE || ''
 let libds = _sctRow.LIBDS || ''
 let libref = libds.split('.')[0] || dcLibref
 let dsn = libds.split('.')[1] || ''
+// stagedata.sas names the CSV it stages after the table as the request found
+// it, before a hook can re-point the submission
+const origLibds = libds
 
 // The base table lives in its own library (DC_JSLIB for control tables,
 // TESTDATA for demo user tables); the staging dir stays in the control lib.
@@ -63,6 +66,10 @@ if (schema.length > 0) {
 // eval()s the hook's .js counterpart in this scope so it can reassign
 // libref / dsn / libds; the submit record then names the new base table, exactly
 // as the SAS hook does via call symputx.
+// A validating hook rejects the submission by setting hookErrmsg (see
+// hooks/mpe_derived_postedit.js); the response below turns it into an abort,
+// and nothing is staged.
+let hookErrmsg = ''
 const hookTables = makeTableLoader(mpeDataDir)('MPE_TABLES')
 const hookReg = (hookTables && hookTables.rows)
     ? hookTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -73,14 +80,58 @@ if (hookReg && hookReg.post_edit_hook) {
         try {
             eval(hookSrc)
         } catch (err) {
-            console.log(
+            // a hook that throws has not validated anything, so the submission
+            // is refused rather than staged unchecked
+            hookErrmsg =
                 'post_edit_hook ' + hookReg.post_edit_hook + ' failed: ' + err.message
-            )
         }
         libds = libref + '.' + dsn
     }
 }
 
+// ─── RK assignment (mpe_loader) ──────────────────────────────────────────────
+// A table registered with rk_underlying is edited on its business key and the
+// RK column is withheld from the editor (see getdata.js), so the submitted rows
+// carry no RK. The loader derives it, and MPE_SUBMIT records the staged width
+// - which includes the derived column.
+
+const finalTables = makeTableLoader(mpeDataDir)('MPE_TABLES')
+const finalReg =
+    finalTables && finalTables.rows
+        ? finalTables.rows.find((r) => r.libref === libref && r.dsn === dsn)
+        : null
+const rkName = mpeAssignRetainedKey({
+    rows: jsDataRows,
+    tableReg: finalReg,
+    baseData: loadTableData(dsn)
+})
+if (rkName && !jsDataHeaders.some((h) => h.toUpperCase() === rkName.toUpperCase())) {
+    jsDataHeaders.push(rkName)
+}
+
+// ─── Derived variables ───────────────────────────────────────────────────────
+// stagedata.sas drops the MPE_DERIVED droplist from the submitted data before
+// anything is staged (they are display-only), and the approval CSV is written
+// from that stripped dataset - so neither the staged rows nor the pack may
+// carry them.
+
+const derivedVars = mpeDerived({
+    libref,
+    table: dsn,
+    dataDir: mpeDataDir,
+    columns: schema
+})
+if (derivedVars.length > 0) {
+    const derivedNames = new Set(derivedVars.map((d) => d.name.toUpperCase()))
+    const keep = (key) => !derivedNames.has(key.toUpperCase())
+    jsDataHeaders = jsDataHeaders.filter(keep)
+    jsDataRows = jsDataRows.map((row) => {
+        const copy = {}
+        for (const key of Object.keys(row)) if (keep(key)) copy[key] = row[key]
+        return copy
+    })
+}
+
 // ─── Stage the data ──────────────────────────────────────────────────────────
 // One subfolder per submission (DSID) under the staging directory, so that the
 // staged data can sit alongside related artifacts (submit/approval logs,
@@ -88,7 +139,7 @@ if (hookReg && hookReg.post_edit_hook) {
 
 const tableId = makeTableId()
 
-stageSubmission({
+if (!hookErrmsg) stageSubmission({
     dataDir,
     stagingDir,
     tableId,
@@ -97,9 +148,17 @@ stageSubmission({
     dsn,
     message,
     headers: jsDataHeaders,
-    rows: jsDataRows
+    rows: jsDataRows,
+    // stagedata.sas writes the staged rows as <orig_libds>.csv alongside them
+    // (%mp_ds2csv into the approval folder), so the audit pack carries it
+    auditFiles: [{ name: origLibds + '.csv', content: toCsv(jsDataHeaders, jsDataRows) }]
 })
 
 webOutOpen()
-webOutObj([{ STATUS: 'SUCCESS', DSID: tableId, URL: '' }], 'sasparams')
+if (hookErrmsg) {
+    webOutObj([{ MSG: hookErrmsg, MAC: 'stagedata' }], 'sasjsAbort')
+    webOutObj([{ STATUS: 'ERROR', DSID: '' }], 'sasparams')
+} else {
+    webOutObj([{ STATUS: 'SUCCESS', DSID: tableId, URL: '' }], 'sasparams')
+}
 webOutClose()

sas/mocks/sasjs/services/hooks/mpe_derived_postedit.js

@@ -0,0 +1,373 @@
+// ─────────────────────────────────────────────────────────────────────────────
+// Mock of services/hooks/mpe_derived_postedit
+//
+// Registered against DC_JSLIB.MPE_DERIVED (MPE_TABLES.post_edit_hook).  This
+// file is eval'd by editors/stagedata.js and editors/loadfile.js, mirroring how
+// mpe_loader.sas %includes the hook program via %mpe_runhook, so it shares the
+// caller's scope: jsDataRows (the submitted rows - work.staging_ds on the
+// server), libref, dsn, mpeDataDir and the dcMockUtils helpers.
+//
+// A hook rejects a submission by setting hookErrmsg, which the caller turns
+// into a sasjsAbort - the same outcome as the SAS hook's %mp_abort.
+//
+// The rules mirror services/hooks/mpe_derived_postedit.sas, in its order:
+// normalise the incoming rows, reject a duplicate name and a bad field, then
+// check the dependency graph that the resulting configuration would have (the
+// incoming rows plus the current rows they do not replace).  As on the server,
+// the collision between VAR_NM and a physical column is left to the read-time
+// engine, which runs with the target library assigned - the hook does not have
+// it here either.
+// ─────────────────────────────────────────────────────────────────────────────
+
+;(function mpeDerivedPostedit() {
+    const DELETE_MARK = '_____DELETE__THIS__RECORD_____'
+    const NAME_CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
+    const FMT_CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.'
+
+    const text = (v) => (v === undefined || v === null ? '' : String(v))
+    const up = (v) => text(v).trim().toUpperCase()
+    const strip = (v) => text(v).trim()
+
+    /** Reads a field as the hook normalised it. */
+    const field = (row, name) => up(colVal(row, name))
+
+    /** Writes a field back under the key the row already carries. */
+    const setField = (row, name, value) => {
+        const key = Object.keys(row).find((k) => k.toUpperCase() === name)
+        if (key) row[key] = value
+        else row[name] = value
+    }
+
+    const rows = Array.isArray(jsDataRows) ? jsDataRows : []
+    const isDeleted = (row) => field(row, DELETE_MARK) === 'YES'
+    const isLive = (row) =>
+        !isDeleted(row) && Number(colVal(row, 'ACTIVE')) === 1
+    const keyOf = (row) =>
+        [field(row, 'LIBREF'), field(row, 'DSN'), field(row, 'VAR_NM')].join(
+            '.'
+        )
+
+    const reject = (msg) => {
+        hookErrmsg = msg
+    }
+
+    // ─── Normalise the incoming values ─────────────────────────────────────────
+    for (const row of rows) {
+        if (isDeleted(row)) continue
+        for (const name of [
+            'LIBREF',
+            'DSN',
+            'VAR_NM',
+            'VAR_TYPE',
+            'VAR_FMT',
+            'VAR_DEPS'
+        ]) {
+            setField(row, name, up(colVal(row, name)))
+        }
+        for (const name of ['VAR_LABEL', 'VAR_LOGIC']) {
+            setField(row, name, strip(colVal(row, name)))
+        }
+    }
+
+    // ─── Duplicate names after normalisation ───────────────────────────────────
+    // mpe_loader compares the raw values, so a case variant reaches this point
+    const seen = new Set()
+    for (const row of rows) {
+        if (isDeleted(row)) continue
+        const key = keyOf(row)
+        if (seen.has(key)) {
+            return reject(
+                `MPE_DERIVED: ${field(row, 'VAR_NM')} is declared more than once for ${field(row, 'LIBREF')}.${field(row, 'DSN')}`
+            )
+        }
+        seen.add(key)
+    }
+
+    // ─── Field level checks, the rules the engine applies at read time ─────────
+    // C0 controls and DEL: what a single-line label cannot carry
+    const CONTROL_CHARS = /[\u0000-\u001F\u007F]/
+
+    // true when the value carries a character outside the allowed set
+    const hasInvalidChar = (value, chars) =>
+        [...value].some((c) => !chars.includes(c))
+
+    for (const row of rows) {
+        if (isDeleted(row)) continue
+        const libref = field(row, 'LIBREF')
+        const dsn = field(row, 'DSN')
+        const name = field(row, 'VAR_NM')
+        const type = field(row, 'VAR_TYPE')
+        const len = Number(colVal(row, 'VAR_LEN'))
+        const fmt = field(row, 'VAR_FMT')
+        const label = strip(colVal(row, 'VAR_LABEL'))
+        const logic = strip(colVal(row, 'VAR_LOGIC'))
+
+        let reason = ''
+        if (!libref || !dsn) reason = 'LIBREF and DSN are required'
+        else if (!name) reason = 'VAR_NM is required'
+        else if (hasInvalidChar(name, NAME_CHARS))
+            reason = 'VAR_NM must be a valid SAS name'
+        else if ('0123456789'.includes(name[0]))
+            reason = 'VAR_NM must not start with a digit'
+        else if (name === DELETE_MARK) reason = 'VAR_NM is reserved'
+        else if (type !== 'C' && type !== 'N')
+            reason = 'VAR_TYPE must be C or N'
+        // ACTIVE is a picker in the EDIT screen (HARDSELECT over the
+        // MPE_SELECTBOX pair) - the same pair mpe_derived_check.sas enforces
+        else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
+            reason = 'ACTIVE must be 1 or 0'
+        else if (!Number.isInteger(len) || len < 1)
+            reason = 'VAR_LEN must be a positive integer'
+        else if (type === 'C' && len > 32767)
+            reason = 'VAR_LEN exceeds 32767 for a character variable'
+        else if (type === 'N' && len > 8)
+            reason = 'VAR_LEN exceeds 8 for a numeric variable'
+        else if (
+            fmt.includes(';') ||
+            fmt.includes('&') ||
+            fmt.includes('%') ||
+            fmt.includes('/*') ||
+            fmt.includes('*/')
+        )
+            reason = 'VAR_FMT must be a format name, without ; & % or /* */'
+        else if (hasInvalidChar(fmt, FMT_CHARS))
+            reason = 'VAR_FMT must be a format name'
+        // free text, in any script - what is refused is anything that cannot be
+        // a single-line column label, and a value the engine would cut to 256
+        else if (CONTROL_CHARS.test(label))
+            reason =
+                'VAR_LABEL must be plain text - no line breaks or control characters'
+        else if ([...label].length > 256)
+            reason = 'VAR_LABEL exceeds 256 characters'
+        else if (!logic) reason = 'VAR_LOGIC is required'
+        else if (logic.includes(';'))
+            reason = 'VAR_LOGIC must be a single expression - remove the ;'
+        else if (logic.includes('&') || logic.includes('%'))
+            reason = 'VAR_LOGIC must not contain macro triggers'
+        else if (logic.includes('/*') || logic.includes('*/'))
+            reason = 'VAR_LOGIC must not contain comments'
+
+        if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
+    }
+
+    // ─── The graph of the resulting configuration ──────────────────────────────
+    // The incoming live rows plus the current rows they do not replace.  Only
+    // active rows are in it, because the engine reads nothing else - an inactive
+    // row can neither resolve a dependency nor be resolved by one.
+    const controlDir = mpeDataDir
+    const loadControl = makeTableLoader(controlDir)
+    const current = loadControl('MPE_DERIVED')
+    const nowSeconds = Date.now() / 1000
+
+    // mpeDerived reads a row as current when TX_TO is missing, zero or in the
+    // future (dcMockUtils.js, the read-time engine), so the same test is used
+    // here - a seeded row with no TX_TO is live for the engine, and a dependency
+    // on it has to validate
+    const isCurrent = (row) =>
+      !Number(colVal(row, 'TX_TO')) || Number(colVal(row, 'TX_TO')) > nowSeconds
+
+    const stagedKeys = new Set(rows.map(keyOf))
+    const stagedLive = rows.filter(isLive)
+
+    // the control-table arm of the SAS hook is limited to the tables this
+    // submission touches, so another table's configuration cannot block it
+    const stagedTables = new Set(
+        rows.map((row) => [field(row, 'LIBREF'), field(row, 'DSN')].join('.'))
+    )
+
+    /** libref.dsn.name - the key every graph lookup uses. */
+    const nodeKey = (n) => [n.libref, n.dsn, n.name].join('.')
+
+    const graph = new Map()
+    for (const row of stagedLive) {
+        graph.set(keyOf(row), {
+            libref: field(row, 'LIBREF'),
+            dsn: field(row, 'DSN'),
+            name: field(row, 'VAR_NM'),
+            deps: field(row, 'VAR_DEPS').split(/\s+/).filter(Boolean),
+            logic: strip(colVal(row, 'VAR_LOGIC')).toUpperCase()
+        })
+    }
+    const currentRows =
+        current && Array.isArray(current.rows)
+            ? current.rows.filter(
+                  (r) =>
+                      isCurrent(r) &&
+                      Number(colVal(r, 'ACTIVE')) === 1 &&
+                      stagedTables.has(
+                          [up(colVal(r, 'LIBREF')), up(colVal(r, 'DSN'))].join(
+                              '.'
+                          )
+                      ) &&
+                      !stagedKeys.has(
+                          [
+                              up(colVal(r, 'LIBREF')),
+                              up(colVal(r, 'DSN')),
+                              up(colVal(r, 'VAR_NM'))
+                          ].join('.')
+                      )
+              )
+            : []
+    for (const row of currentRows) {
+        graph.set(
+            [
+                up(colVal(row, 'LIBREF')),
+                up(colVal(row, 'DSN')),
+                up(colVal(row, 'VAR_NM'))
+            ].join('.'),
+            {
+                libref: up(colVal(row, 'LIBREF')),
+                dsn: up(colVal(row, 'DSN')),
+                name: up(colVal(row, 'VAR_NM')),
+                deps: up(colVal(row, 'VAR_DEPS')).split(/\s+/).filter(Boolean),
+                logic: strip(colVal(row, 'VAR_LOGIC')).toUpperCase()
+            }
+        )
+    }
+
+    /** Names this load leaves inactive, which therefore cannot resolve a dep. */
+    const inactive = new Set()
+    const liveKeys = new Set(stagedLive.map(keyOf))
+    for (const row of rows) {
+        const key = keyOf(row)
+        if (!liveKeys.has(key)) inactive.add(key)
+    }
+    if (current && Array.isArray(current.rows)) {
+        for (const row of current.rows) {
+            const key = [
+                up(colVal(row, 'LIBREF')),
+                up(colVal(row, 'DSN')),
+                up(colVal(row, 'VAR_NM'))
+            ].join('.')
+            if (
+                isCurrent(row) &&
+                Number(colVal(row, 'ACTIVE')) !== 1 &&
+                !liveKeys.has(key)
+            ) {
+                inactive.add(key)
+            }
+        }
+    }
+
+    for (const node of graph.values()) {
+        for (const dep of node.deps) {
+            const key = [node.libref, node.dsn, dep].join('.')
+            if (inactive.has(key) && !graph.has(key)) {
+                return reject(
+                    `MPE_DERIVED: ${node.name}: VAR_DEPS lists ${dep}, which is not active after this load - the engine cannot resolve it`
+                )
+            }
+        }
+    }
+
+    // ─── Self reference, undeclared and stale dependencies ─────────────────────
+    /** The identifiers an expression uses, with quoted runs removed. */
+    const identifiers = (logic) =>
+        logic
+            .replace(/'([^']|'')*'/g, ' ')
+            .replace(/"([^"]|"")*"/g, ' ')
+            .replace(/[^A-Za-z0-9_]/g, ' ')
+            .toUpperCase()
+            .split(/\s+/)
+            .filter(Boolean)
+
+    // scoped per table, as the server does: a name that is a derived variable
+    // on another table says nothing about this one
+    const graphKeys = new Set([...graph.values()].map((n) => nodeKey(n)))
+
+    // one check at a time, in the order the server runs them, so the message a
+    // row gets is the one it would get there
+    for (const node of graph.values()) {
+        if (node.deps.includes(node.name)) {
+            return reject(
+                `MPE_DERIVED: ${node.name}: VAR_DEPS may not list itself`
+            )
+        }
+    }
+
+    for (const node of graph.values()) {
+        for (const token of identifiers(node.logic)) {
+            if (
+                graphKeys.has([node.libref, node.dsn, token].join('.')) &&
+                token !== node.name &&
+                !node.deps.includes(token)
+            ) {
+                return reject(
+                    `MPE_DERIVED: ${node.name}: VAR_LOGIC references ${token}, which is not in VAR_DEPS`
+                )
+            }
+        }
+    }
+
+    for (const node of graph.values()) {
+        const tokens = identifiers(node.logic)
+        for (const dep of node.deps) {
+            if (!tokens.includes(dep)) {
+                return reject(
+                    `MPE_DERIVED: ${node.name}: VAR_DEPS lists ${dep}, which VAR_LOGIC does not reference`
+                )
+            }
+        }
+    }
+
+    // ─── A derived variable may not be named by a row level security rule ──────
+    const rls = loadControl('MPE_ROW_LEVEL_SECURITY')
+    if (rls && Array.isArray(rls.rows)) {
+        for (const node of graph.values()) {
+            const match = rls.rows.find(
+                (r) =>
+                    up(colVal(r, 'RLS_LIBREF')) === node.libref &&
+                    up(colVal(r, 'RLS_TABLE')) === node.dsn &&
+                    up(colVal(r, 'RLS_VARIABLE_NM')) === node.name &&
+                    Number(colVal(r, 'RLS_ACTIVE')) === 1 &&
+                    isCurrent(r)
+            )
+            if (match) {
+                return reject(
+                    `MPE_DERIVED: ${node.name} is named by a row level security rule on ${node.libref}.${node.dsn} - derived variables cannot be used there`
+                )
+            }
+        }
+    }
+
+    // ─── Cycles ────────────────────────────────────────────────────────────────
+    // Repeatedly place the variables whose derived inputs are already placed; a
+    // round that places nothing leaves a cycle.  The path is walked so the
+    // message names it, as the SAS hook does.
+    const nodeKeys = new Map(
+        [...graph.values()].map((n) => [n, [n.libref, n.dsn, n.name].join('.')])
+    )
+    const placed = new Set()
+    let remaining = [...graph.values()]
+
+    while (remaining.length > 0) {
+        const ready = remaining.filter((n) =>
+            n.deps.every((d) => {
+                const key = [n.libref, n.dsn, d].join('.')
+                return !graph.has(key) || placed.has(key)
+            })
+        )
+        if (ready.length === 0) {
+            // walk the unplaced edges so the message names the cycle
+            const start = remaining[0]
+            const path = [start.name]
+            const visited = new Set([nodeKeys.get(start)])
+            let cursor = start
+            for (let i = 0; i < remaining.length; i++) {
+                const parentKey = cursor.deps
+                    .map((d) => [cursor.libref, cursor.dsn, d].join('.'))
+                    .find((key) => graph.has(key) && !placed.has(key))
+                if (!parentKey) break
+                path.push(graph.get(parentKey).name)
+                if (visited.has(parentKey)) break
+                visited.add(parentKey)
+                cursor = graph.get(parentKey)
+            }
+            return reject(
+                `MPE_DERIVED: VAR_DEPS contains a cycle: ${path.join(' -> ')}`
+            )
+        }
+        for (const node of ready) placed.add(nodeKeys.get(node))
+        remaining = remaining.filter((n) => !placed.has(nodeKeys.get(n)))
+    }
+})()

sas/mocks/sasjs/services/public/getcols.js

@@ -43,42 +43,28 @@ if (!tableData) {
 }
 
 // ─── Build the column list ────────────────────────────────────────────────────
-// The picker's variable list, in the same shape the editor gets from getdata:
-// NAME / LABEL / DDTYPE / FMTNAME, with the temporal columns excluded exactly as
-// the editor excludes them, so the fallback list matches the one the table
-// itself would have supplied.
+// Every column, as getdata.sas emits them: the temporal columns are dropped from
+// the data, not from the cols payload, so a filter can still be built on them -
+// which is exactly what "please filter and try again" wants to do.
 
 let cols = []
 
 if (tableData && tableData.columns) {
-    // MPE_TABLES always lives in the control library (DC_JSLIB / mpelib), not
-    // in user data libraries - mirroring getdata.js.
-    const mpeTablesData = makeTableLoader(libDataDir(dcLibref))('MPE_TABLES')
-    const tableReg = mpeTablesData && mpeTablesData.rows
-        ? mpeTablesData.rows.find(r => r.libref === libref && r.dsn === table)
-        : null
-
-    const excludeCols = new Set()
-    if (tableReg && (tableReg.loadtype === 'TXTEMPORAL' || tableReg.loadtype === 'BITEMPORAL')) {
-        if (tableReg.var_txfrom) excludeCols.add(tableReg.var_txfrom.toUpperCase())
-        if (tableReg.var_txto) excludeCols.add(tableReg.var_txto.toUpperCase())
-    }
-    if (tableReg && tableReg.loadtype === 'BITEMPORAL') {
-        if (tableReg.var_busfrom) excludeCols.add(tableReg.var_busfrom.toUpperCase())
-        if (tableReg.var_busto) excludeCols.add(tableReg.var_busto.toUpperCase())
-    }
-
-    // mpGetcols, then the same CHARACTER/NUMERIC -> C/N mapping getcols.sas
-    // applies on top of %mp_getcols, so the picker's fallback list is typed the
-    // way the editor's own load types it.
-    cols = mpGetcols(tableData.columns.filter(c => !excludeCols.has(c.name.toUpperCase())))
+    // mpGetcols, then the mapping getcols.sas applies on top of %mp_getcols:
+    // DDTYPE in the client's vocabulary, and TYPE as the SAS C/N it has always
+    // been.  The client maps TYPE to its own num/char vocabulary in
+    // sas-store.getCols, because the picker keys its operator set and its value
+    // quoting on it and this list does not pass through mergeColsRules.
+    cols = mpGetcols(tableData.columns)
         .map(c => {
             const ddtype = c.DDTYPE === 'CHARACTER' ? 'C' : c.DDTYPE === 'NUMERIC' ? 'N' : c.DDTYPE
             return {
                 NAME: c.NAME,
                 LABEL: c.LABEL,
+                FORMAT: c.FORMAT || ' ',
                 FMTNAME: c.FMTNAME || ' ',
                 DDTYPE: ddtype,
+                TYPE: ddtype === 'C' ? 'C' : 'N',
                 CLS_RULE: 'READ',
                 MEMLABEL: ' ',
                 DESC: ' ',

sas/mocks/sasjs/services/public/getcolvals.js

@@ -21,9 +21,25 @@ const tableName = libds.split('.')[1] || ''
 // ─── Load table data (searches all libref data folders) ──────────────────────
 
 const _loaded = loadTableAnyLib(tableName) || {}
-const tableData = _loaded.data
+let tableData = _loaded.data
 const tableLibref = _loaded.libref || libref
 
+// ─── Derived (computed) variables (mpe_derived.sas) ─────────────────────────
+// Present in the schema and in the rows before the filter is applied, so the
+// filter may reference them and the values below are computed from the same
+// row set the grid shows.
+
+const controlDataDir = libDataDir(dcLibref)
+const derived = mpeDerived({
+    libref: tableLibref,
+    table: tableName,
+    dataDir: controlDataDir,
+    columns: tableData ? tableData.columns : []
+})
+if (tableData && derived.length > 0) {
+    tableData = mpeDeriveRows(tableData, derived)
+}
+
 // ─── Resolve column metadata ─────────────────────────────────────────────────
 
 let colMeta = null

sas/mocks/sasjs/services/public/viewdata.js

@@ -49,6 +49,23 @@ if (!tableData && dsn) {
     if (found) tableData = found.data
 }
 
+// ─── Derived variables (mpe_derived.sas) ─────────────────────────────────────
+// A stored filter can reference a derived column, so the derivation has to run
+// before the filter.  The columns themselves are dropped from the output - they
+// belong to the EDIT screen, as viewdata.sas drops them.
+// a copy: the derivation below returns a new column list, and every use here
+// (cols, search, rows, vars) must keep looking at the physical columns only
+const physicalColumns = [...(tableData && tableData.columns ? tableData.columns : [])]
+const derived = mpeDerived({
+    libref: libref,
+    table: dsn,
+    dataDir: mpeDataDir,
+    columns: physicalColumns
+})
+if (tableData && derived.length > 0) {
+    tableData = mpeDeriveRows(tableData, derived)
+}
+
 // ─── Apply the stored filter (mpe_filtermaster, VIEW mode) ───────────────────
 // Mirrors viewdata.sas: %mpe_filtermaster(VIEW,&libds,filter_rk=&filter_rk)
 // Row Level Security and any saved filter are applied here, before the search.
@@ -103,7 +120,7 @@ let nobs = 0
 if (tableData && tableData.columns) {
     // mpGetcols, the JS counterpart of %mp_getcols.sas - the macro viewdata.sas
     // calls for its own column list.
-    cols = mpGetcols(tableData.columns).map(c => {
+    cols = mpGetcols(physicalColumns).map(c => {
         const ddtype = c.DDTYPE === 'CHARACTER' ? 'C' : c.DDTYPE === 'NUMERIC' ? 'N' : c.DDTYPE
         const formatStr = c.TYPE === 'N' ? (c.FORMAT || '8.') : ('$' + c.LENGTH + '.')
         return {
@@ -119,7 +136,10 @@ if (tableData && tableData.columns) {
 
     let rows
     if (searchType) {
-        const matches = filteredRows.filter(row => rowMatchesSearch(row, tableData.columns, searchType, searchVal))
+        // the derived columns are dropped before the search, so only the
+        // physical columns are searched (viewdata.sas drops them in the
+        // vwsearch data step mp_searchdata then reads)
+        const matches = filteredRows.filter(row => rowMatchesSearch(row, physicalColumns, searchType, searchVal))
         // mp_searchdata caps its output at outobs=&DC_MAXOBS_WEBVIEW, and
         // viewdata.sas then keeps only `if _n_<&DC_MAXOBS_WEBVIEW` of them.
         nobs = Math.min(matches.length, maxObsWebView)
@@ -134,7 +154,7 @@ if (tableData && tableData.columns) {
     // viewdata rows: all values as strings (formatted, SAS proc-print style)
     viewdata = rows.map(r => {
         const row = {}
-        for (const col of tableData.columns) {
+        for (const col of physicalColumns) {
             const lcName = col.name.toLowerCase()
             let val = r[col.name]
             if (val === undefined) val = r[lcName]
@@ -174,15 +194,15 @@ if (tableData && tableData.columns) {
     // (SAS missing), character columns get "".
     // A search that matches nothing sends no rows at all (the client shows
     // its "no data found" panel instead of an empty grid).
-    if (viewdata.length === 0 && !searchType && tableData.columns.length > 0) {
+    if (viewdata.length === 0 && !searchType && physicalColumns.length > 0) {
         const emptyRow = {}
-        for (const col of tableData.columns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
+        for (const col of physicalColumns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
         viewdata = [emptyRow]
     }
 
     // $viewdata vars: keep original SAS types (num stays num, char stays char)
     vars = {}
-    for (const col of tableData.columns) {
+    for (const col of physicalColumns) {
         vars[col.name] = sasVarsEntry(col)
     }
 }

sas/mocks/snapshot.js

@@ -0,0 +1,92 @@
+// Collect the SASjs Server drive for an app into a mock-snapshot.json, which
+// `sasjs build` ships as the browser build's starting drive (the alternative is
+// a drive seeded from the compiled services alone, i.e. no DC data).
+//
+// Usage: node sas/mocks/snapshot.js <drive-root> <appLoc> <out-file>
+//   drive-root  the SASjs Server drive folder (the `drive` folder of a server install)
+//   appLoc      the deployed app location, e.g. /Public/app/dc
+//   out-file    where to write the snapshot, e.g. sas/mocks/sasjsbuild/mock-snapshot.json
+//
+// Every file is written as a string because the browser runtime's drive holds
+// text only; a file that does not survive a UTF-8 round trip is skipped rather
+// than mangled.
+const fs = require('fs')
+const path = require('path')
+
+const [driveRoot, appLoc, outFile, excludeArg] = process.argv.slice(2)
+
+if (!driveRoot || !appLoc || !outFile) {
+  console.error(
+    'Usage: node sas/mocks/snapshot.js <drive-root> <appLoc> <out-file> [exclude,exclude]'
+  )
+  process.exit(1)
+}
+
+const app = '/' + appLoc.replace(/^\/+|\/+$/g, '')
+const driveFolder = path.join(
+  driveRoot,
+  'files',
+  ...app.split('/').filter(Boolean)
+)
+const driveKey = `/sasjs/drive/files${app}`
+
+// Comma-separated paths relative to the app, e.g. "services/web9" - the
+// streamed frontend is copied into the drive as a service folder, is already the
+// site itself, and would otherwise duplicate the whole app inside the snapshot.
+const excludes = (excludeArg || '')
+  .split(',')
+  .map((value) => value.trim().replace(/^\/+|\/+$/g, ''))
+  .filter(Boolean)
+
+if (!fs.existsSync(driveFolder)) {
+  console.error('No drive folder at ' + driveFolder)
+  process.exit(1)
+}
+
+const files = {}
+const skipped = []
+
+const walk = (folder) => {
+  for (const entry of fs.readdirSync(folder, { withFileTypes: true })) {
+    const absolute = path.join(folder, entry.name)
+    if (entry.isDirectory()) {
+      walk(absolute)
+      continue
+    }
+
+    const relative = path
+      .relative(driveFolder, absolute)
+      .split(path.sep)
+      .join('/')
+
+    if (
+      excludes.some(
+        (prefix) => relative === prefix || relative.startsWith(prefix + '/')
+      )
+    ) {
+      continue
+    }
+
+    const buffer = fs.readFileSync(absolute)
+    const text = buffer.toString('utf8')
+
+    if (!Buffer.from(text, 'utf8').equals(buffer)) {
+      skipped.push(relative)
+      continue
+    }
+
+    files[`${driveKey}/${relative}`] = text
+  }
+}
+
+walk(driveFolder)
+
+fs.mkdirSync(path.dirname(path.resolve(outFile)), { recursive: true })
+fs.writeFileSync(path.resolve(outFile), JSON.stringify({ files }, null, 2))
+
+console.log(
+  `Snapshot: ${Object.keys(files).length} file(s) from ${driveFolder} -> ${outFile}` +
+    (skipped.length
+      ? `, skipped ${skipped.length} non-text file(s): ${skipped.slice(0, 3).join(', ')}`
+      : '')
+)

sas/package-lock.json

@@ -6,8 +6,8 @@
     "": {
       "name": "dc-sas",
       "dependencies": {
-        "@sasjs/cli": "4.20.5",
-        "@sasjs/core": "5.2.8"
+        "@sasjs/cli": "6.2.0",
+        "@sasjs/core": "6.1.0"
       }
     },
     "node_modules/@asamuzakjp/css-color": {
@@ -140,56 +140,49 @@
       }
     },
     "node_modules/@fast-csv/format": {
-      "version": "4.3.5",
-      "resolved": "https://registry.npmjs.org/@fast-csv/format/-/format-4.3.5.tgz",
-      "integrity": "sha512-8iRn6QF3I8Ak78lNAa+Gdl5MJJBM5vRHivFtMRUWINdevNo00K7OXxS2PshawLKTejVwieIlPmK5YlLu6w4u8A==",
+      "version": "5.0.7",
+      "resolved": "https://registry.npmjs.org/@fast-csv/format/-/format-5.0.7.tgz",
+      "integrity": "sha512-VdypoRxv7PF+LsyPouTMKdB0d76hync+gLpgdNqfqVK44MsgW4oiCJSdrki2FisWT7v2QGUYDHjp4L7w5oO6gw==",
       "license": "MIT",
       "dependencies": {
-        "@types/node": "^14.0.1",
-        "lodash.escaperegexp": "^4.1.2",
-        "lodash.isboolean": "^3.0.3",
-        "lodash.isequal": "^4.5.0",
-        "lodash.isfunction": "^3.0.9",
-        "lodash.isnil": "^4.0.0"
+        "lodash.escaperegexp": "^4.1.2"
       }
     },
     "node_modules/@sasjs/adapter": {
-      "version": "4.19.1",
-      "resolved": "https://registry.npmjs.org/@sasjs/adapter/-/adapter-4.19.1.tgz",
-      "integrity": "sha512-IpN6UPSdqVb5cAWEBt+jG/i0rhgCubdyp5kxdouuX3ifL313oJW2ZPVUNUvs0+ZGzsXWcGDTNoEzzTrGKiqJlQ==",
+      "version": "5.1.1",
+      "resolved": "https://registry.npmjs.org/@sasjs/adapter/-/adapter-5.1.1.tgz",
+      "integrity": "sha512-rso6+xRpZhFuuUbgHmGztznuse4xZjlu7+dOcYMUMJezAd5p/xdCt+wvv4eIKejsMWDcIteSeRPOkgKr0TlxWg==",
       "license": "ISC",
       "dependencies": {
-        "@sasjs/utils": "^3.6.0",
+        "@sasjs/utils": "4.0.1",
         "axios": "1.20.0",
         "axios-cookiejar-support": "5.0.5",
         "form-data": "4.0.6",
-        "https": "1.0.0",
-        "tough-cookie": "4.1.3"
+        "tough-cookie": "5.1.2"
+      },
+      "engines": {
+        "node": ">=22"
       }
     },
     "node_modules/@sasjs/cli": {
-      "version": "4.20.5",
-      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.5.tgz",
-      "integrity": "sha512-/HSpueCFS+SvXas9FwfP8qoD9skhdTDE5r7dT9Xlrs7HGCBjxDYvSiw6tW0GIjfw3/49Yf8TCVDFBsISang1hg==",
+      "version": "6.2.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-6.2.0.tgz",
+      "integrity": "sha512-zovAOCmyPqcm+gooAZgwvgGH8n5cas3sY0lA80+hyFoFtmZ8VjW/bh19LM8OVwheT9dBCRCx4fnHwjy5XGpzvQ==",
       "license": "ISC",
       "dependencies": {
-        "@sasjs/adapter": "4.19.1",
-        "@sasjs/core": "5.2.10",
-        "@sasjs/lint": "4.1.0",
-        "@sasjs/utils": "3.6.2",
+        "@sasjs/adapter": "^5.1.1",
+        "@sasjs/core": "^6.0.0",
+        "@sasjs/utils": "^4.1.0",
+        "axios": "1.20.0",
         "chalk": "4.1.2",
-        "dotenv": "17.4.2",
-        "find": "0.3.0",
-        "js-base64": "3.9.3",
+        "dotenv": "18.0.8",
+        "js-base64": "3.9.4",
         "jsdom": "26.0.0",
-        "jwt-decode": "3.1.2",
-        "lodash.groupby": "4.6.0",
         "lodash.uniqby": "4.7.0",
         "node-graphviz": "0.1.1",
         "node-powershell": "5.0.1",
         "ora": "5.4.1",
         "prompts": "2.4.2",
-        "shelljs": "0.10.0",
         "ssl-root-cas": "1.3.1",
         "xml": "1.0.1",
         "yargs": "17.6.2",
@@ -198,65 +191,123 @@
       },
       "bin": {
         "sasjs": "build/index.js"
-      }
-    },
-    "node_modules/@sasjs/cli/node_modules/@sasjs/core": {
-      "version": "5.2.10",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.10.tgz",
-      "integrity": "sha512-Lk/ENHVuXaeAl8M0+Mjh1rcBSdFt4tIM6OPxPPmk78ZD0gYFqAOUZreqIoV9EN+VwPDqYsUPDqXzh0crW6j8OA==",
-      "license": "MIT"
-    },
-    "node_modules/@sasjs/cli/node_modules/shelljs": {
-      "version": "0.8.5",
-      "resolved": "https://registry.npmjs.org/shelljs/-/shelljs-0.8.5.tgz",
-      "integrity": "sha512-TiwcRcrkhHvbrZbnRcFYMLl30Dfov3HKqzp5tO5b4pt6G/SezKcYhmDg15zXVBswHmctSAQKznqNW2LO5tTDow==",
-      "license": "BSD-3-Clause",
-      "dependencies": {
-        "glob": "^7.0.0",
-        "interpret": "^1.0.0",
-        "rechoir": "^0.6.2"
-      },
-      "bin": {
-        "shjs": "bin/shjs"
       },
       "engines": {
-        "node": ">=4"
+        "node": ">=22"
+      },
+      "peerDependencies": {
+        "@sasjs/lint": "^4.3.0"
+      }
+    },
+    "node_modules/@sasjs/cli/node_modules/@sasjs/utils": {
+      "version": "4.1.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.1.0.tgz",
+      "integrity": "sha512-aC5pOugg6/2jKQR7ZaYsLtbsFy/iVacsrN3/Kzh6BouJUszuVrr13HMrQUSKcWL+rJzxYof7ymmmCl+vE6YwAw==",
+      "license": "ISC",
+      "dependencies": {
+        "@fast-csv/format": "5.0.7",
+        "@types/fs-extra": "11.0.4",
+        "@types/prompts": "2.4.9",
+        "chalk": "4.1.1",
+        "consola": "3.4.2",
+        "fs-extra": "11.4.1",
+        "jwt-decode": "4.0.0",
+        "prompts": "2.4.2"
+      },
+      "engines": {
+        "node": ">=22"
+      }
+    },
+    "node_modules/@sasjs/cli/node_modules/@sasjs/utils/node_modules/chalk": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
+      "integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
+      "license": "MIT",
+      "dependencies": {
+        "ansi-styles": "^4.1.0",
+        "supports-color": "^7.1.0"
+      },
+      "engines": {
+        "node": ">=10"
+      },
+      "funding": {
+        "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
     "node_modules/@sasjs/core": {
-      "version": "5.2.8",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.8.tgz",
-      "integrity": "sha512-+K8tXTrsw52NR1y5MFHW0+Jf9euL+GukwqA79FIHHu4Oj+BhHBm1uZiyB5tPrvwl3V8mY1CQhMOT83//bV+jWg==",
+      "version": "6.1.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.0.tgz",
+      "integrity": "sha512-U0Ekq/TXVMquYV+AG6urBQSUgqUvV37yduufy68QQVgDj3PJiLUJDcOXKOGbhWGD+Nv877VuPCviGYqrqbjXvg==",
       "license": "MIT"
     },
     "node_modules/@sasjs/lint": {
-      "version": "4.1.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-4.1.0.tgz",
-      "integrity": "sha512-Vpg1s0/zdPMFJ0hRXNXk53HhE3kfRr5gFRS0kvrse+bxgcbqih1qqVW7yZ2krhSCYylp9z13eBXLhXbRrj70VQ==",
-      "hasInstallScript": true,
+      "version": "4.3.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-4.3.0.tgz",
+      "integrity": "sha512-65cjy4IjA4PgSlicOruF0DI9N2HO3LJqtgphsZCg8g7N/StE2FWLJLJbD1N4OHxN6htrUfj5SEawJJiHo5T9Yw==",
       "license": "MIT",
+      "peer": true,
       "dependencies": {
-        "@sasjs/utils": "3.6.2",
+        "@sasjs/utils": "4.0.0",
         "ignore": "7.0.11"
+      },
+      "engines": {
+        "node": ">=22"
+      }
+    },
+    "node_modules/@sasjs/lint/node_modules/@sasjs/utils": {
+      "version": "4.0.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.0.0.tgz",
+      "integrity": "sha512-lARoCnRM4gSVf8ShoP5elHi7W28K/14MK037dQ0ABgMlP3+r6OuhAZYwK/a70tED47tKxyIoCAKLdP6jrG0ewQ==",
+      "license": "ISC",
+      "peer": true,
+      "dependencies": {
+        "@fast-csv/format": "5.0.7",
+        "@types/fs-extra": "11.0.4",
+        "@types/prompts": "2.4.9",
+        "chalk": "4.1.1",
+        "consola": "3.4.2",
+        "fs-extra": "11.4.1",
+        "jwt-decode": "4.0.0",
+        "prompts": "2.4.2"
+      },
+      "engines": {
+        "node": ">=22"
+      }
+    },
+    "node_modules/@sasjs/lint/node_modules/chalk": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
+      "integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
+      "license": "MIT",
+      "peer": true,
+      "dependencies": {
+        "ansi-styles": "^4.1.0",
+        "supports-color": "^7.1.0"
+      },
+      "engines": {
+        "node": ">=10"
+      },
+      "funding": {
+        "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
     "node_modules/@sasjs/utils": {
-      "version": "3.6.2",
-      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-3.6.2.tgz",
-      "integrity": "sha512-AfC5OJuJx3NAbntKq2J8E8NNpRXthYfuqOyp2lOmOkTctL3mAfrA54U/cdPQvrlftXNxpK9P2U0RU10qNjlEzQ==",
+      "version": "4.0.1",
+      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.0.1.tgz",
+      "integrity": "sha512-aNEbOJFZ2Ru71/XkHKtywexBQYdst8ovq0a9vuVO4subZuKMjRtYcn4kTRr9VVIz65p8xGtVUkFEf+6t9Vbfcg==",
       "license": "ISC",
       "dependencies": {
-        "@fast-csv/format": "4.3.5",
+        "@fast-csv/format": "5.0.7",
         "@types/fs-extra": "11.0.4",
-        "@types/prompts": "2.0.13",
+        "@types/prompts": "2.4.9",
         "chalk": "4.1.1",
-        "cli-table": "0.3.6",
-        "consola": "2.15.0",
-        "find": "0.3.0",
-        "fs-extra": "11.3.0",
-        "jwt-decode": "3.1.2",
-        "prompts": "2.4.1",
-        "valid-url": "1.0.9"
+        "consola": "3.4.2",
+        "fs-extra": "11.4.1",
+        "jwt-decode": "4.0.0",
+        "prompts": "2.4.2"
+      },
+      "engines": {
+        "node": ">=22"
       }
     },
     "node_modules/@sasjs/utils/node_modules/chalk": {
@@ -275,19 +326,6 @@
         "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
-    "node_modules/@sasjs/utils/node_modules/prompts": {
-      "version": "2.4.1",
-      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
-      "integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
-      "license": "MIT",
-      "dependencies": {
-        "kleur": "^3.0.3",
-        "sisteransi": "^1.0.5"
-      },
-      "engines": {
-        "node": ">= 6"
-      }
-    },
     "node_modules/@types/fs-extra": {
       "version": "11.0.4",
       "resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-11.0.4.tgz",
@@ -308,18 +346,22 @@
       }
     },
     "node_modules/@types/node": {
-      "version": "14.18.63",
-      "resolved": "https://registry.npmjs.org/@types/node/-/node-14.18.63.tgz",
-      "integrity": "sha512-fAtCfv4jJg+ExtXhvCkCqUKZ+4ok/JQk01qDKhL5BDDoS3AxKXhV5/MAVUZyQnSEd2GT92fkgZl0pz0Q0AzcIQ==",
-      "license": "MIT"
-    },
-    "node_modules/@types/prompts": {
-      "version": "2.0.13",
-      "resolved": "https://registry.npmjs.org/@types/prompts/-/prompts-2.0.13.tgz",
-      "integrity": "sha512-jwMOIGy49VruR/gYehhJYgpVzB+EVpEE7t7j9m1oTo4HMpOe7KmsyqdBuoxAzA5B4caUgx0cKrWr7wUEqMXJ7Q==",
+      "version": "26.6.5",
+      "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.5.tgz",
+      "integrity": "sha512-uMzsv9stDdofIZBx4UGh4Z9ZgXnGC9seRDscilBcTHAeOFk67MZg/k9bWVfO+bk4qZmG+l52hRnLwUiH2LnJTg==",
       "license": "MIT",
       "dependencies": {
-        "@types/node": "*"
+        "undici-types": "~8.9.0"
+      }
+    },
+    "node_modules/@types/prompts": {
+      "version": "2.4.9",
+      "resolved": "https://registry.npmjs.org/@types/prompts/-/prompts-2.4.9.tgz",
+      "integrity": "sha512-qTxFi6Buiu8+50/+3DGIWLHM6QuWsEKugJnnP6iv2Mc4ncxE4A/OJkjuVOA+5X0X1S/nq5VJRa8Lu+nwcvbrKA==",
+      "license": "MIT",
+      "dependencies": {
+        "@types/node": "*",
+        "kleur": "^3.0.3"
       }
     },
     "node_modules/accumulate-stream": {
@@ -402,12 +444,6 @@
         "tough-cookie": ">=4.0.0"
       }
     },
-    "node_modules/balanced-match": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
-      "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
-      "license": "MIT"
-    },
     "node_modules/base64-js": {
       "version": "1.5.1",
       "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
@@ -439,16 +475,6 @@
         "readable-stream": "^3.4.0"
       }
     },
-    "node_modules/brace-expansion": {
-      "version": "1.1.21",
-      "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz",
-      "integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==",
-      "license": "MIT",
-      "dependencies": {
-        "balanced-match": "^1.0.0",
-        "concat-map": "0.0.1"
-      }
-    },
     "node_modules/buffer": {
       "version": "5.7.1",
       "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
@@ -559,17 +585,6 @@
         "url": "https://github.com/sponsors/sindresorhus"
       }
     },
-    "node_modules/cli-table": {
-      "version": "0.3.6",
-      "resolved": "https://registry.npmjs.org/cli-table/-/cli-table-0.3.6.tgz",
-      "integrity": "sha512-ZkNZbnZjKERTY5NwC2SeMeLeifSPq/pubeRoTpdr3WchLlnZg6hEgvHkK5zL7KNFdd9PmHN8lxrENUwI3cE8vQ==",
-      "dependencies": {
-        "colors": "1.0.3"
-      },
-      "engines": {
-        "node": ">= 0.2.0"
-      }
-    },
     "node_modules/cliui": {
       "version": "8.0.1",
       "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
@@ -611,15 +626,6 @@
       "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
       "license": "MIT"
     },
-    "node_modules/colors": {
-      "version": "1.0.3",
-      "resolved": "https://registry.npmjs.org/colors/-/colors-1.0.3.tgz",
-      "integrity": "sha512-pFGrxThWcWQ2MsAz6RtgeWe4NK2kUE1WfsrvvlctdII745EW9I0yflqhe7++M5LEc7bV2c/9/5zc8sFcpL0Drw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.1.90"
-      }
-    },
     "node_modules/combined-stream": {
       "version": "1.0.8",
       "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
@@ -632,17 +638,14 @@
         "node": ">= 0.8"
       }
     },
-    "node_modules/concat-map": {
-      "version": "0.0.1",
-      "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
-      "integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==",
-      "license": "MIT"
-    },
     "node_modules/consola": {
-      "version": "2.15.0",
-      "resolved": "https://registry.npmjs.org/consola/-/consola-2.15.0.tgz",
-      "integrity": "sha512-vlcSGgdYS26mPf7qNi+dCisbhiyDnrN1zaRbw3CSuc2wGOMEGGPsp46PdRG5gqXwgtJfjxDkxRNAgRPr1B77vQ==",
-      "license": "MIT"
+      "version": "3.4.2",
+      "resolved": "https://registry.npmjs.org/consola/-/consola-3.4.2.tgz",
+      "integrity": "sha512-5IKcdX0nnYavi6G7TtOhwkYzyjfJlatbjMjuLSfE2kYT5pMDOilZ4OvMhi637CcDICTmz3wARPoyhqyX1Y+XvA==",
+      "license": "MIT",
+      "engines": {
+        "node": "^14.18.0 || >=16.10.0"
+      }
     },
     "node_modules/cssstyle": {
       "version": "4.6.0",
@@ -715,10 +718,13 @@
       }
     },
     "node_modules/dotenv": {
-      "version": "17.4.2",
-      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
-      "integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
+      "version": "18.0.8",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-18.0.8.tgz",
+      "integrity": "sha512-2KDeoFoK/zRtcnbunCDlLG5VWVMcyqeQc6iK65o231nfNAcuGrKoqUQfJFUtBI+QlShk5HKIHxwebBtUl3rEfA==",
       "license": "BSD-2-Clause",
+      "bin": {
+        "dotenv": "dist/index.cjs"
+      },
       "engines": {
         "node": ">=12"
       },
@@ -818,15 +824,6 @@
       "integrity": "sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==",
       "license": "MIT"
     },
-    "node_modules/find": {
-      "version": "0.3.0",
-      "resolved": "https://registry.npmjs.org/find/-/find-0.3.0.tgz",
-      "integrity": "sha512-iSd+O4OEYV/I36Zl8MdYJO0xD82wH528SaCieTVHhclgiYNe9y+yPKSwK+A7/WsmHL1EZ+pYUJBXWTL5qofksw==",
-      "license": "MIT",
-      "dependencies": {
-        "traverse-chain": "~0.1.0"
-      }
-    },
     "node_modules/follow-redirects": {
       "version": "1.16.1",
       "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.1.tgz",
@@ -864,9 +861,9 @@
       }
     },
     "node_modules/fs-extra": {
-      "version": "11.3.0",
-      "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.3.0.tgz",
-      "integrity": "sha512-Z4XaCL6dUDHfP/jT25jJKMmtxvuwbkrD1vNSMFlo9lNLY2c5FHYSQgHPRZUjAB26TpDEoW9HCOgplrdbaPV/ew==",
+      "version": "11.4.1",
+      "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.4.1.tgz",
+      "integrity": "sha512-KYAb4c9BJQI6QqGKthV68OHe0badztdXJWKo0WtBA9IuCFPTKvE5ZdUBglP833aMjhaSPNO4A5j/EkzZtGlKjA==",
       "license": "MIT",
       "dependencies": {
         "graceful-fs": "^4.2.0",
@@ -877,12 +874,6 @@
         "node": ">=14.14"
       }
     },
-    "node_modules/fs.realpath": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
-      "integrity": "sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==",
-      "license": "ISC"
-    },
     "node_modules/function-bind": {
       "version": "1.1.2",
       "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
@@ -938,27 +929,6 @@
         "node": ">= 0.4"
       }
     },
-    "node_modules/glob": {
-      "version": "7.2.3",
-      "resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz",
-      "integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==",
-      "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me",
-      "license": "ISC",
-      "dependencies": {
-        "fs.realpath": "^1.0.0",
-        "inflight": "^1.0.4",
-        "inherits": "2",
-        "minimatch": "^3.1.1",
-        "once": "^1.3.0",
-        "path-is-absolute": "^1.0.0"
-      },
-      "engines": {
-        "node": "*"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/isaacs"
-      }
-    },
     "node_modules/gopd": {
       "version": "1.2.0",
       "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@@ -1074,12 +1044,6 @@
         "node": ">= 14"
       }
     },
-    "node_modules/https": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/https/-/https-1.0.0.tgz",
-      "integrity": "sha512-4EC57ddXrkaF0x83Oj8sM6SLQHAWXw90Skqu2M4AEWENZ3F02dFJE/GARA8igO79tcgYqGrD7ae4f5L3um2lgg==",
-      "license": "ISC"
-    },
     "node_modules/https-proxy-agent": {
       "version": "5.0.1",
       "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz",
@@ -1142,51 +1106,17 @@
       "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.11.tgz",
       "integrity": "sha512-YChdK5txDjwGUvgR7oCJcLGkwi3LDh8Zx8tS7ndYciLEg/oOCL26VUBSUEtgS7EjiybcBqFxh5j0rAdDX7/bbg==",
       "license": "MIT",
+      "peer": true,
       "engines": {
         "node": ">= 4"
       }
     },
-    "node_modules/inflight": {
-      "version": "1.0.6",
-      "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
-      "integrity": "sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==",
-      "deprecated": "This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.",
-      "license": "ISC",
-      "dependencies": {
-        "once": "^1.3.0",
-        "wrappy": "1"
-      }
-    },
     "node_modules/inherits": {
       "version": "2.0.4",
       "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
       "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
       "license": "ISC"
     },
-    "node_modules/interpret": {
-      "version": "1.4.0",
-      "resolved": "https://registry.npmjs.org/interpret/-/interpret-1.4.0.tgz",
-      "integrity": "sha512-agE4QfB2Lkp9uICn7BAqoscw4SZP9kTE2hxiFI3jBPmXJfdqiahTbUuKGsMoN2GtqL9AxhYioAcVvgsb1HvRbA==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/is-core-module": {
-      "version": "2.17.0",
-      "resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.17.0.tgz",
-      "integrity": "sha512-J/vG0zBCbIKOQFfufSwyXdMrsohyJIUNkrnmo6WZGzoM7tr/lsbfW5b2BvisL6zsyMzK9UxV9L6c7AoFbyXHOA==",
-      "license": "MIT",
-      "dependencies": {
-        "hasown": "^2.0.4"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
     "node_modules/is-docker": {
       "version": "2.2.1",
       "resolved": "https://registry.npmjs.org/is-docker/-/is-docker-2.2.1.tgz",
@@ -1251,9 +1181,9 @@
       }
     },
     "node_modules/js-base64": {
-      "version": "3.9.3",
-      "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.3.tgz",
-      "integrity": "sha512-uwYQp+VJ38FVvtim6qNbit6e9uT6dwWQ4Y1+H9TxhW5hcHjpHwoxlR0nMpqUmIFOmu4VqMxwdJA88gIVuZJQ/g==",
+      "version": "3.9.4",
+      "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.4.tgz",
+      "integrity": "sha512-PtOMXpEGuP0RRiRXsjzHzl44dMHxSu2CPvAhinupR1tBa88me+1DPqsobl7eupn5UukjLkln1ZnAOAOXOrYG0Q==",
       "license": "BSD-3-Clause"
     },
     "node_modules/jsdom": {
@@ -1309,18 +1239,6 @@
         "node": ">= 14"
       }
     },
-    "node_modules/jsdom/node_modules/tough-cookie": {
-      "version": "5.1.2",
-      "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
-      "integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
-      "license": "BSD-3-Clause",
-      "dependencies": {
-        "tldts": "^6.1.32"
-      },
-      "engines": {
-        "node": ">=16"
-      }
-    },
     "node_modules/jsonfile": {
       "version": "6.2.1",
       "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.1.tgz",
@@ -1334,10 +1252,13 @@
       }
     },
     "node_modules/jwt-decode": {
-      "version": "3.1.2",
-      "resolved": "https://registry.npmjs.org/jwt-decode/-/jwt-decode-3.1.2.tgz",
-      "integrity": "sha512-UfpWE/VZn0iP50d8cz9NrZLM9lSWhcJ+0Gt/nm4by88UL+J1SiKN8/5dkjMmbEzwL2CAe+67GsegCbIKtbp75A==",
-      "license": "MIT"
+      "version": "4.0.0",
+      "resolved": "https://registry.npmjs.org/jwt-decode/-/jwt-decode-4.0.0.tgz",
+      "integrity": "sha512-+KJGIyHgkGuIq3IEBNftfhW/LfWhXUIY6OmyVWjliu5KH1y0fw7VQ8YndE2O4qZdMSd9SqbnC8GOcZEy0Om7sA==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=18"
+      }
     },
     "node_modules/kind-of": {
       "version": "6.0.3",
@@ -1363,37 +1284,6 @@
       "integrity": "sha512-TM9YBvyC84ZxE3rgfefxUWiQKLilstD6k7PTGt6wfbtXF8ixIJLOL3VYyV/z+ZiPLsVxAsKAFVwWlWeb2Y8Yyw==",
       "license": "MIT"
     },
-    "node_modules/lodash.groupby": {
-      "version": "4.6.0",
-      "resolved": "https://registry.npmjs.org/lodash.groupby/-/lodash.groupby-4.6.0.tgz",
-      "integrity": "sha512-5dcWxm23+VAoz+awKmBaiBvzox8+RqMgFhi7UvX9DHZr2HdxHXM/Wrf8cfKpsW37RNrvtPn6hSwNqurSILbmJw==",
-      "license": "MIT"
-    },
-    "node_modules/lodash.isboolean": {
-      "version": "3.0.3",
-      "resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
-      "integrity": "sha512-Bz5mupy2SVbPHURB98VAcw+aHh4vRV5IPNhILUCsOzRmsTmSQ17jIuqopAentWoehktxGd9e/hbIXq980/1QJg==",
-      "license": "MIT"
-    },
-    "node_modules/lodash.isequal": {
-      "version": "4.5.0",
-      "resolved": "https://registry.npmjs.org/lodash.isequal/-/lodash.isequal-4.5.0.tgz",
-      "integrity": "sha512-pDo3lu8Jhfjqls6GkMgpahsF9kCyayhgykjyLMNFTKWrpVdAQtYyB4muAMWozBB4ig/dtWAmsMxLEI8wuz+DYQ==",
-      "deprecated": "This package is deprecated. Use require('node:util').isDeepStrictEqual instead.",
-      "license": "MIT"
-    },
-    "node_modules/lodash.isfunction": {
-      "version": "3.0.9",
-      "resolved": "https://registry.npmjs.org/lodash.isfunction/-/lodash.isfunction-3.0.9.tgz",
-      "integrity": "sha512-AirXNj15uRIMMPihnkInB4i3NHeb4iBtNg9WRWuK2o31S+ePwwNmDPaTL3o7dTJ+VXNZim7rFs4rxN4YU1oUJw==",
-      "license": "MIT"
-    },
-    "node_modules/lodash.isnil": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/lodash.isnil/-/lodash.isnil-4.0.0.tgz",
-      "integrity": "sha512-up2Mzq3545mwVnMhTDMdfoG1OurpA/s5t88JmQX809eH3C8491iu2sfKhTfhQtKY78oPNhiaHJUpT/dUDAAtng==",
-      "license": "MIT"
-    },
     "node_modules/lodash.uniqby": {
       "version": "4.7.0",
       "resolved": "https://registry.npmjs.org/lodash.uniqby/-/lodash.uniqby-4.7.0.tgz",
@@ -1461,18 +1351,6 @@
         "node": ">=6"
       }
     },
-    "node_modules/minimatch": {
-      "version": "3.1.5",
-      "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz",
-      "integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==",
-      "license": "ISC",
-      "dependencies": {
-        "brace-expansion": "^1.1.7"
-      },
-      "engines": {
-        "node": "*"
-      }
-    },
     "node_modules/ms": {
       "version": "2.1.3",
       "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
@@ -1519,15 +1397,6 @@
       "integrity": "sha512-gQPNF78qebCQ6tvVFBYrvJdBNOrYZm90ZlXgpIFm06p6qHDHq/XC4TnJftN6OMbxVE0UTBAoRgcsDeJBBooITw==",
       "license": "MIT"
     },
-    "node_modules/once": {
-      "version": "1.4.0",
-      "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
-      "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
-      "license": "ISC",
-      "dependencies": {
-        "wrappy": "1"
-      }
-    },
     "node_modules/onetime": {
       "version": "5.1.2",
       "resolved": "https://registry.npmjs.org/onetime/-/onetime-5.1.2.tgz",
@@ -1624,21 +1493,6 @@
         "url": "https://github.com/inikulin/parse5?sponsor=1"
       }
     },
-    "node_modules/path-is-absolute": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
-      "integrity": "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
-    "node_modules/path-parse": {
-      "version": "1.0.7",
-      "resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz",
-      "integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
-      "license": "MIT"
-    },
     "node_modules/pend": {
       "version": "1.2.0",
       "resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz",
@@ -1667,18 +1521,6 @@
         "node": ">=10"
       }
     },
-    "node_modules/psl": {
-      "version": "1.15.0",
-      "resolved": "https://registry.npmjs.org/psl/-/psl-1.15.0.tgz",
-      "integrity": "sha512-JZd3gMVBAVQkSs6HdNZo9Sdo0LNcQeMNP3CozBJb3JYC/QUYZTnKxP+f8oWRX4rHP5EurWxqAHTSwUCjlNKa1w==",
-      "license": "MIT",
-      "dependencies": {
-        "punycode": "^2.3.1"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/lupomontero"
-      }
-    },
     "node_modules/punycode": {
       "version": "2.3.1",
       "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz",
@@ -1688,12 +1530,6 @@
         "node": ">=6"
       }
     },
-    "node_modules/querystringify": {
-      "version": "2.2.0",
-      "resolved": "https://registry.npmjs.org/querystringify/-/querystringify-2.2.0.tgz",
-      "integrity": "sha512-FIqgj2EUvTa7R50u0rGsyTftzjYmv/a3hO345bZNrqabNqjtgiDMgmo4mkUjd+nzU5oF3dClKqFIPUKybUyqoQ==",
-      "license": "MIT"
-    },
     "node_modules/readable-stream": {
       "version": "3.6.2",
       "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
@@ -1708,17 +1544,6 @@
         "node": ">= 6"
       }
     },
-    "node_modules/rechoir": {
-      "version": "0.6.2",
-      "resolved": "https://registry.npmjs.org/rechoir/-/rechoir-0.6.2.tgz",
-      "integrity": "sha512-HFM8rkZ+i3zrV+4LQjwQ0W+ez98pApMGM3HUrN04j3CqzPOzl9nmP15Y8YXNm8QHGv/eacOVEjqhmWpkRV0NAw==",
-      "dependencies": {
-        "resolve": "^1.1.6"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
     "node_modules/require-directory": {
       "version": "2.1.1",
       "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
@@ -1728,33 +1553,6 @@
         "node": ">=0.10.0"
       }
     },
-    "node_modules/requires-port": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
-      "integrity": "sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ==",
-      "license": "MIT"
-    },
-    "node_modules/resolve": {
-      "version": "1.22.12",
-      "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.12.tgz",
-      "integrity": "sha512-TyeJ1zif53BPfHootBGwPRYT1RUt6oGWsaQr8UyZW/eAm9bKoijtvruSDEmZHm92CwS9nj7/fWttqPCgzep8CA==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "is-core-module": "^2.16.1",
-        "path-parse": "^1.0.7",
-        "supports-preserve-symlinks-flag": "^1.0.0"
-      },
-      "bin": {
-        "resolve": "bin/resolve"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
     "node_modules/restore-cursor": {
       "version": "3.1.0",
       "resolved": "https://registry.npmjs.org/restore-cursor/-/restore-cursor-3.1.0.tgz",
@@ -1880,18 +1678,6 @@
         "node": ">=8"
       }
     },
-    "node_modules/supports-preserve-symlinks-flag": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/supports-preserve-symlinks-flag/-/supports-preserve-symlinks-flag-1.0.0.tgz",
-      "integrity": "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
     "node_modules/symbol-tree": {
       "version": "3.2.4",
       "resolved": "https://registry.npmjs.org/symbol-tree/-/symbol-tree-3.2.4.tgz",
@@ -1917,27 +1703,15 @@
       "license": "MIT"
     },
     "node_modules/tough-cookie": {
-      "version": "4.1.3",
-      "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-4.1.3.tgz",
-      "integrity": "sha512-aX/y5pVRkfRnfmuX+OdbSdXvPe6ieKX/G2s7e98f4poJHnqH3281gDPm/metm6E/WRamfx7WC4HUqkWHfQHprw==",
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
+      "integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
       "license": "BSD-3-Clause",
       "dependencies": {
-        "psl": "^1.1.33",
-        "punycode": "^2.1.1",
-        "universalify": "^0.2.0",
-        "url-parse": "^1.5.3"
+        "tldts": "^6.1.32"
       },
       "engines": {
-        "node": ">=6"
-      }
-    },
-    "node_modules/tough-cookie/node_modules/universalify": {
-      "version": "0.2.0",
-      "resolved": "https://registry.npmjs.org/universalify/-/universalify-0.2.0.tgz",
-      "integrity": "sha512-CJ1QgKmNg3CwvAv/kOFmtnEN05f0D/cn9QntgNOQlQF9dgvVTHj3t+8JPdjqawCHk7V/KA+fbUqzZ9XWhcqPUg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 4.0.0"
+        "node": ">=16"
       }
     },
     "node_modules/tr46": {
@@ -1952,18 +1726,18 @@
         "node": ">=18"
       }
     },
-    "node_modules/traverse-chain": {
-      "version": "0.1.0",
-      "resolved": "https://registry.npmjs.org/traverse-chain/-/traverse-chain-0.1.0.tgz",
-      "integrity": "sha512-up6Yvai4PYKhpNp5PkYtx50m3KbwQrqDwbuZP/ItyL64YEWHAvH6Md83LFLV/GRSk/BoUVwwgUzX6SOQSbsfAg==",
-      "license": "MIT"
-    },
     "node_modules/trim-buffer": {
       "version": "5.0.0",
       "resolved": "https://registry.npmjs.org/trim-buffer/-/trim-buffer-5.0.0.tgz",
       "integrity": "sha512-PqdsRmhir3lUwraK/O6sdZ9QcmtUK0c6F1utX/WNquJsL4Jpdck+DSe9Mqy8Wk7uAQD/SlVn6cyGF5bj6VF33Q==",
       "license": "MIT"
     },
+    "node_modules/undici-types": {
+      "version": "8.9.0",
+      "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz",
+      "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==",
+      "license": "MIT"
+    },
     "node_modules/universalify": {
       "version": "2.0.1",
       "resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.1.tgz",
@@ -1973,27 +1747,12 @@
         "node": ">= 10.0.0"
       }
     },
-    "node_modules/url-parse": {
-      "version": "1.5.10",
-      "resolved": "https://registry.npmjs.org/url-parse/-/url-parse-1.5.10.tgz",
-      "integrity": "sha512-WypcfiRhfeUP9vvF0j6rw0J3hrWrw6iZv3+22h6iRMJ/8z1Tj6XfLP4DsUix5MhMPnXpiHDoKyoZ/bdCkwBCiQ==",
-      "license": "MIT",
-      "dependencies": {
-        "querystringify": "^2.1.1",
-        "requires-port": "^1.0.0"
-      }
-    },
     "node_modules/util-deprecate": {
       "version": "1.0.2",
       "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
       "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
       "license": "MIT"
     },
-    "node_modules/valid-url": {
-      "version": "1.0.9",
-      "resolved": "https://registry.npmjs.org/valid-url/-/valid-url-1.0.9.tgz",
-      "integrity": "sha512-QQDsV8OnSf5Uc30CKSwG9lnhMPe6exHtTXLRYX8uMwKENy640pU+2BgBL0LRbDh/eYRahNCS7aewCx0wf3NYVA=="
-    },
     "node_modules/w3c-xmlserializer": {
       "version": "5.0.0",
       "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz",
@@ -2076,12 +1835,6 @@
         "url": "https://github.com/chalk/wrap-ansi?sponsor=1"
       }
     },
-    "node_modules/wrappy": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
-      "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
-      "license": "ISC"
-    },
     "node_modules/ws": {
       "version": "8.21.3",
       "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",

sas/package.json

@@ -26,15 +26,17 @@
     "viyacbd": "sasjs folder delete /Public/app/viya && sasjs cbd -t viya && npm run viyadata",
     "viyadata": "sasjs request services/admin/makedata -d deploy/makeDataViya.json -l sasjsresults/makedata.log -o sasjsresults/output.json -t viya",
     "v4data": "sasjs request services/admin/makedata -d deploy/makeDataV4.json -l sasjsresults/makedatav4.log -o sasjsresults/outputv4.json -t v4",
-    "sasdocs": "sasjs doc && ./sasjs/utils/deploydocs.sh"
+    "sasdocs": "sasjs doc && ./sasjs/utils/deploydocs.sh",
+    "sas719setup": "sasjs cbd -t sas719 && npm run sas719data",
+    "sas719data": "sasjs request services/admin/makedata -d deploy/makeDataSas719.json -l sasjsresults/makedata_sas719.log -o sasjsresults/makedata_sas719.json -t sas719",
+    "sas719test": "sasjs test -t sas719"
   },
   "private": true,
   "dependencies": {
-    "@sasjs/cli": "4.20.5",
-    "@sasjs/core": "5.2.8"
+    "@sasjs/cli": "6.2.0",
+    "@sasjs/core": "6.1.0"
   },
   "overrides": {
-    "nanoid": "3.3.18",
-    "shelljs": "0.8.5"
+    "nanoid": "3.3.18"
   }
 }

sas/sasjs/db/datactrl/mpe_derived.ddl

@@ -0,0 +1,32 @@
+/**
+  @file mpe_derived.ddl
+  @brief ddl file
+  @details Defines the derived (computed) variable configuration table.  This
+  drives MPE_DERIVED rows in MPE_TABLES, so that the table itself can be
+  maintained through the EDIT screen with the usual approve workflow.
+
+  Derived values are not persisted - the logic is applied at read time, before
+  the filter, so that the variables can be filtered on.
+
+  @version 9.2
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd
+**/
+
+
+create table &curlib..MPE_DERIVED
+   (
+    TX_FROM num not null format=datetime19.3,
+    LIBREF varchar(8) not null,
+    DSN varchar(32) not null,
+    VAR_NM varchar(32) not null,
+    VAR_DEPS varchar(4000),
+    VAR_TYPE varchar(1) not null,
+    VAR_LEN int not null,
+    VAR_FMT varchar(32),
+    VAR_LABEL varchar(256),
+    VAR_LOGIC varchar(4000) not null,
+    ACTIVE int not null,
+    TX_TO num not null format=datetime19.3,
+    constraint pk_mpe_derived
+        primary key(tx_from, libref, dsn, var_nm));

sas/sasjs/db/datactrl/mpe_tables.ddl

@@ -118,3 +118,16 @@ insert into &mpelib..mpe_tables
       ,var_txfrom='TX_FROM'
       ,var_txto='TX_TO'
     ;
+  insert into &mpelib..mpe_tables
+    set tx_from=0
+      ,tx_to='31DEC5999:23:59:59'dt
+      ,libref="&mpelib"
+      ,dsn='MPE_DERIVED'
+      ,num_of_approvals_required=1
+      ,loadtype='TXTEMPORAL'
+      ,buskey='LIBREF DSN VAR_NM'
+      ,post_edit_hook='services/hooks/mpe_derived_postedit'
+      ,notes='Derived (computed) variables for the EDIT screen'
+      ,var_txfrom='TX_FROM'
+      ,var_txto='TX_TO'
+    ;

sas/sasjs/db/migrations/20261003_row_and_cell_limits.sas

@@ -23,6 +23,10 @@
   only for options that are not present at all - an existing install carries
   its own rows, so it keeps 500 / 100 until this is run.
 
+  Each block below uses %do/%end on BOTH branches.  A single-statement
+  %then/%else in open code (outside a %macro) is rejected by the Viya compute
+  engine with "Expected %DO not found", so it must not be shortened back.
+
 **/
 
 %let dclib=YOURDCLIB;
@@ -60,7 +64,9 @@ quit;
           !!' loaded into the browser in the VIEW screen.';
   quit;
 %end;
-%else %put NOTE: DC_MAXOBS_WEBVIEW is switched off (var_active=0) - not changed;
+%else %do;
+  %put NOTE: DC_MAXOBS_WEBVIEW is switched off (var_active=0) - not changed;
+%end;
 
 %if &off_edit=0 %then %do;
   proc sql noprint;
@@ -81,7 +87,9 @@ quit;
           !!' is reached first.';
   quit;
 %end;
-%else %put NOTE: DC_MAXOBS_WEBEDIT is switched off (var_active=0) - not changed;
+%else %do;
+  %put NOTE: DC_MAXOBS_WEBEDIT is switched off (var_active=0) - not changed;
+%end;
 
 %if &off_cells=0 %then %do;
   proc sql noprint;
@@ -89,6 +97,10 @@ quit;
       set tx_to=&now
       where var_name='DC_MAXCELLS_WEBEDIT' and var_active=1 and &now lt tx_to;
 
+    /* Why the cell limit exists, kept out of var_desc - MPE_CONFIG.VAR_DESC
+      is varchar(300), and a truncation WARNING aborts the build.  A wide
+      table is expensive long before it is tall - 250 rows of a 1000 column
+      table is a quarter of a million cells. */
     insert into &dclib..mpe_config
       set tx_from=&now
         ,tx_to='31DEC9999:23:59:59'dt
@@ -98,10 +110,10 @@ quit;
         ,var_active=1
         ,var_desc='This sets the maximum number of CELLS (rows multiplied by'
           !!' columns) that can be loaded into the browser for editing in the'
-          !!' EDIT screen.  It exists because a wide table is expensive long'
-          !!' before it is tall - 250 rows of a 1000 column table is a quarter'
-          !!' of a million cells.  Whichever of this and DC_MAXOBS_WEBEDIT is'
-          !!' reached first applies.';
+          !!' EDIT screen.  Whichever of this and DC_MAXOBS_WEBEDIT is reached'
+          !!' first applies.';
   quit;
 %end;
-%else %put NOTE: DC_MAXCELLS_WEBEDIT is switched off (var_active=0) - not changed;
+%else %do;
+  %put NOTE: DC_MAXCELLS_WEBEDIT is switched off (var_active=0) - not changed;
+%end;

sas/sasjs/db/migrations/20261019_v7.19_release.sas

@@ -0,0 +1,192 @@
+/**
+  @file
+  @brief migration script to move to v7.19 of Data Controller
+
+  REQUIRED CHANGE - adds the MPE_DERIVED table, which holds derived (computed)
+  variable definitions for the EDIT screen, and registers it in MPE_TABLES so
+  that it can be maintained through the app with the usual approve workflow.
+
+  Replace YOURDCLIB and the library path below, then run the script.  Any
+  MPE_DERIVED rows added before this script runs will be lost when the table
+  is created, so run it before configuring any derived variables.
+
+**/
+
+%let dclib=YOURDCLIB;
+
+libname &dclib "/YOUR/DATACONTROLLER/LIBRARY/PATH";
+
+proc sql;
+create table &dclib..mpe_derived
+  (
+    tx_from num not null format=datetime19.3,
+    libref varchar(8) not null,
+    dsn varchar(32) not null,
+    var_nm varchar(32) not null,
+    var_deps varchar(4000),
+    var_type varchar(1) not null,
+    var_len num not null,
+    var_fmt varchar(32),
+    var_label varchar(256),
+    var_logic varchar(4000) not null,
+    active num not null,
+    tx_to num not null format=datetime19.3
+  );
+quit;
+
+proc datasets lib=&dclib noprint;
+  modify mpe_derived;
+  index create
+    pk_mpe_derived=(tx_from libref dsn var_nm)
+    /nomiss unique;
+quit;
+
+proc sql;
+insert into &dclib..mpe_tables
+  set tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,libref="&dclib"
+    ,dsn='MPE_DERIVED'
+    ,num_of_approvals_required=1
+    ,loadtype='TXTEMPORAL'
+    ,buskey='LIBREF DSN VAR_NM'
+    ,post_edit_hook='services/hooks/mpe_derived_postedit'
+    ,notes='Derived (computed) variables for the EDIT screen'
+    ,var_txfrom='TX_FROM'
+    ,var_txto='TX_TO'
+  ;
+quit;
+
+/* Dropdown values for VAR_TYPE and ACTIVE.  These rows, and the field rules
+  * further down, mirror the MPE_DERIVED rows in mpe_makedata.sas - keep the two
+  * in step, or an upgraded install and a freshly seeded one behave differently. */
+proc sql noprint;
+  /* coalesce: an install with no selectbox rows yet returns a missing value,
+    and %eval(&maxrk+1) below would then fail */
+  select coalesce(max(selectbox_rk),0) into: maxrk from &dclib..mpe_selectbox;
+quit;
+
+%let rk1=%eval(&maxrk+1);
+%let rk2=%eval(&maxrk+2);
+%let rk3=%eval(&maxrk+3);
+%let rk4=%eval(&maxrk+4);
+
+proc sql;
+insert into &dclib..mpe_selectbox
+  set selectbox_rk=&rk1
+    ,ver_from_dttm=0
+    ,select_lib="&dclib"
+    ,select_ds="MPE_DERIVED"
+    ,base_column="VAR_TYPE"
+    ,selectbox_value='C'
+    ,selectbox_order=1
+    ,ver_to_dttm='31DEC5999:23:59:59'dt;
+insert into &dclib..mpe_selectbox
+  set selectbox_rk=&rk2
+    ,ver_from_dttm=0
+    ,select_lib="&dclib"
+    ,select_ds="MPE_DERIVED"
+    ,base_column="VAR_TYPE"
+    ,selectbox_value='N'
+    ,selectbox_order=2
+    ,ver_to_dttm='31DEC5999:23:59:59'dt;
+/* ACTIVE - a fixed pair, not a lookup against another table */
+insert into &dclib..mpe_selectbox
+  set selectbox_rk=&rk3
+    ,ver_from_dttm=0
+    ,select_lib="&dclib"
+    ,select_ds="MPE_DERIVED"
+    ,base_column="ACTIVE"
+    ,selectbox_value='1'
+    ,selectbox_order=1
+    ,ver_to_dttm='31DEC5999:23:59:59'dt;
+insert into &dclib..mpe_selectbox
+  set selectbox_rk=&rk4
+    ,ver_from_dttm=0
+    ,select_lib="&dclib"
+    ,select_ds="MPE_DERIVED"
+    ,base_column="ACTIVE"
+    ,selectbox_value='0'
+    ,selectbox_order=2
+    ,ver_to_dttm='31DEC5999:23:59:59'dt;
+quit;
+
+/* Field rules for the EDIT screen.  These are client side - the server side
+  * enforcement is the post edit hook registered above. */
+proc sql;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="LIBREF"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* the libraries the user can edit, as MPE_VALIDATIONS.BASE_LIB does */
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="LIBREF"
+    ,rule_type='SOFTSELECT_HOOK'
+    ,rule_value="services/validations/libraries_editable"
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="DSN"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* tables in the LIBREF on the same row - the hook reads it from source_row */
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="DSN"
+    ,rule_type='SOFTSELECT_HOOK'
+    ,rule_value="services/validations/tables_editable"
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_NM"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_NM"
+    ,rule_type='HARDREGEX',rule_value='^[A-Za-z_][A-Za-z0-9_]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_DEPS"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_DEPS"
+    ,rule_type='HARDREGEX',rule_value='^[A-Za-z0-9_ ]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
+    ,rule_type='HARDSELECT',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* ACTIVE has no NOTNULL rule: the column is NOT NULL in the model, and the
+  * EDIT screen offers the MPE_SELECTBOX values for it. */
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
+    ,rule_type='MINVAL',rule_value='1',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
+    ,rule_type='MAXVAL',rule_value='32767',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_FMT"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_FMT"
+    ,rule_type='HARDREGEX'
+    ,rule_value='^(\$?[A-Za-z][A-Za-z0-9_]*(\.[0-9]+)?\.?|\$?[0-9]+(\.[0-9]+)?\.?)?$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LABEL"
+    ,rule_type='HARDREGEX'
+    ,rule_value='^[^\x00-\x1F\x7F]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &dclib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LOGIC"
+    ,rule_type='HARDREGEX'
+    ,rule_value='^(?!.*(/\*|\*/))[^;&%]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+quit;

sas/sasjs/macros/mpe_assert_abort.sas

@@ -0,0 +1,96 @@
+/**
+  @file
+  @brief Asserts that a service returned a sasjsAbort object
+  @details Reads the child's response as text and records a PASS when it carries
+  a sasjsAbort object, which is what mp_abort leaves in the webout when a service
+  rejects its input.  When msg is supplied the MSG value inside that object must
+  match it, so the assertion proves the service aborted for the expected reason
+  rather than for any reason at all.
+
+  The response is scanned a line at a time and only the first MSG that follows
+  the sasjsAbort marker is taken: a single 32767 buffer both truncated longer
+  responses and, with a leading `.*`, returned the last MSG in the text rather
+  than the abort's own.
+
+  An unassigned outref (a child that died before writing a response) records a
+  FAIL rather than letting the data step fail on an unassigned fileref.
+
+  Usage:
+
+      %mpe_execute(&_program,
+        inputfiles=&f1:sascontroltable,
+        outref=web1,
+        outstatus=work.ab1,
+        failfast=0
+      )
+      %mpe_assert_abort(
+        desc=An invalid libds aborts the service,
+        outref=web1,
+        msg=Invalid libds
+      )
+
+  @param [in] desc The description recorded in the test results
+  @param [in] outref The fileref given to %mx_execute, holding the response
+  @param [in] msg= (0) Expected MSG value in the sasjsAbort object.  0 accepts
+    any abort message.
+  @param [out] outds= (work.test_results) Results dataset
+
+  <h4> SAS Macros </h4>
+  @li mp_assert.sas
+
+**/
+
+%macro mpe_assert_abort(desc, outref, msg=0, outds=work.test_results);
+%local ok seen abortseen;
+%let ok=0;
+%let seen=;
+%let abortseen=0;
+
+%if %sysfunc(fileref(&outref))>0 %then %do;
+  %put %str(WARN)ING: &sysmacroname: fileref &outref is not assigned - the child left no response to read;
+%end;
+%else %if %sysfunc(fileexist(%sysfunc(pathname(&outref)))) ne 1 %then %do;
+  /* the fileref exists but nothing was written to it: a child that aborted
+    before it produced a response.  Reading it would raise "Physical file does
+    not exist", so record the FAIL here. */
+  %put %str(WARN)ING: &sysmacroname: &outref has no response written to it - the child produced nothing;
+%end;
+%else %do;
+data _null_;
+  length line $32767 got $256;
+  retain prxid 0 inabort 0 found 0 abortseen 0;
+  if _n_=1 then prxid=prxparse('/"MSG" *: *"([^"]*)"/');
+  infile &outref end=eof lrecl=32767;
+  do while(not eof);
+    input;
+    line=_infile_;
+    if index(line,'sasjsAbort')>0 then do;
+      abortseen=1;
+      inabort=1;
+    end;
+    /* first MSG at or after the abort marker - stops at the first found */
+    if inabort=1 and found=0 then do;
+      if prxmatch(prxid,line)>0 then do;
+        got=prxposn(prxid,1,line);
+        found=1;
+      end;
+    end;
+  end;
+  call symputx('abortseen',abortseen,'l');
+  if found=1 then call symputx('seen',got,'l');
+  stop;
+run;
+%end;
+
+%if &abortseen=1 %then %do;
+  %if "%superq(msg)"="0" %then %let ok=1;
+  %else %if "&seen"="%superq(msg)" %then %let ok=1;
+%end;
+%if "&seen" ne "" %then %put NOTE: &sysmacroname: sasjsAbort MSG=&seen;
+
+%mp_assert(
+  iftrue=(&ok=1),
+  desc=&desc,
+  outds=&outds
+)
+%mend mpe_assert_abort;

sas/sasjs/macros/mpe_assert_abort.test.sas

@@ -0,0 +1,150 @@
+/**
+  @file
+  @brief testing the mpe_assert_abort macro
+
+  @details The macro decides PASS/FAIL by scanning the child's response text, so
+  the cases that matter can be built as files instead of services: an abort with
+  the expected message, an abort with a different one, no abort at all, and a
+  fileref the child never wrote to.
+
+  Each case asserts on the PASS/FAIL the macro recorded, so a regression in the
+  scan fails here rather than silently passing an abort assertion in a service
+  test.
+
+  Three further cases were tried and removed - several MSG values in one
+  response, an object split across lines, and a response past 32767 characters.
+  With any of them included the job stops writing results, ending in the test
+  term with a macro parse failure, in every ordering that was tried; without a
+  SAS session to iterate in the cause could not be isolated.  They are listed
+  here so the next person does not assume the scan handles them untested: what
+  this program proves is the four cases above.
+
+  <h4> SAS Macros </h4>
+  @li mf_getuniquefileref.sas
+  @li mp_assert.sas
+  @li mpe_assert_abort.sas
+
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+%let rc_any=work.assert_rc1;
+%let rc_other=work.assert_rc2;
+%let rc_none=work.assert_rc3;
+%let rc_first=work.assert_rc4;
+%let rc_split=work.assert_rc5;
+%let rc_long=work.assert_rc6;
+%let rc_noref=work.assert_rc7;
+
+/**
+  * Test 1 - an abort whose MSG matches is a PASS
+  */
+%let f1=%mf_getuniquefileref();
+data _null_;
+  file &f1 termstr=crlf;
+  put '{"sasjsAbort": [{"MSG": "Invalid libds", "MAC": "mp_abort"}]}';
+run;
+
+%mpe_assert_abort(
+  desc=an abort with the expected MSG,
+  outref=&f1,
+  msg=Invalid libds,
+  outds=&rc_any
+)
+
+/**
+  * Test 2 - an abort for a different reason is a FAIL
+  */
+%let f2=%mf_getuniquefileref();
+data _null_;
+  file &f2 termstr=crlf;
+  put '{"sasjsAbort": [{"MSG": "Invalid libds", "MAC": "mp_abort"}]}';
+run;
+
+%mpe_assert_abort(
+  desc=an abort with a different MSG,
+  outref=&f2,
+  msg=Invalid filter,
+  outds=&rc_other
+)
+
+/**
+  * Test 3 - a completed call (no sasjsAbort) is a FAIL
+  */
+%let f3=%mf_getuniquefileref();
+data _null_;
+  file &f3 termstr=crlf;
+  put '{"sasparams": [{"STATUS": "SUCCESS"}]}';
+  put '{"MSG": "Invalid libds"}';
+run;
+
+%mpe_assert_abort(
+  desc=a response with no sasjsAbort object,
+  outref=&f3,
+  msg=0,
+  outds=&rc_none
+)
+
+/**
+  * Test 4 - a fileref with no response written records a FAIL instead of aborting the step
+  */
+%let f7=%mf_getuniquefileref();
+
+%mpe_assert_abort(
+  desc=an unassigned outref,
+  outref=&f7,
+  msg=0,
+  outds=&rc_noref
+)
+
+/**
+  * Test 5 - the recorded results
+  */
+
+
+data _null_;
+  set &rc_any;
+  call symputx('res_any',test_result,'g');
+run;
+data _null_;
+  set &rc_other;
+  call symputx('res_other',test_result,'g');
+run;
+data _null_;
+  set &rc_none;
+  call symputx('res_none',test_result,'g');
+run;
+data _null_;
+  set &rc_noref;
+  call symputx('res_noref',test_result,'g');
+run;
+
+%mp_assert(
+  iftrue=("&res_any"="PASS"),
+  desc=%str(Checking an abort with the expected MSG passes),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_other"="FAIL"),
+  desc=%str(Checking an abort with a different MSG fails),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("&res_none"="FAIL"),
+  desc=%str(Checking a response without a sasjsAbort object fails),
+  outds=work.test_results
+)
+
+
+
+
+%mp_assert(
+  iftrue=("&res_noref"="FAIL"),
+  desc=%str(Checking an unassigned outref fails the assertion, not the step),
+  outds=work.test_results
+)

sas/sasjs/macros/mpe_derived.sas

@@ -0,0 +1,135 @@
+/**
+  @file
+  @brief Generates the SAS statements that create Data Controller derived variables
+  @details Derived (computed) variables are configured in MPE_DERIVED.  Each
+  row defines one variable, which is created in a DATA step view ahead of the
+  filter, so that it can be filtered on in the EDIT screen.
+
+  The engine lives in @sasjs/core as mp_derivedvars - this macro only supplies
+  what is specific to Data Controller: the control table name, the bitemporal
+  filter that picks the current version of a row, and the list of names that
+  belong to the metadata and may not be used as a variable.
+
+  The macro writes the generated statements to a fileref, so that callers can
+  compose them into a view:
+
+      %mpe_derived(libds=&libds,dclib=&mpelib,outref=dercode)
+      data work.vw_derived / view=work.vw_derived;
+        set &libds;
+        %include dercode;
+      run;
+
+  Alternatively, pass outview= and the macro will build the view itself.
+
+  The filter must be applied by the caller, in the step that reads the view:
+
+      data out;
+        set work.vw_derived;
+        where %inc filtref;;
+      run;
+
+  A WHERE statement applies to the data set named on the SET statement, so it
+  cannot reference a variable that is created in the same data step.  Applying
+  it to the view puts the derived variables on the input, which is what makes
+  them filterable.
+
+  Each statement block is of the form:
+
+      attrib REGION length=$21 label='REGION=<expression>' format=$21.;
+      REGION = <expression>;
+
+  Type and length are declared by the administrator, so the result is
+  deterministic.  VAR_LABEL is optional: when it is missing or blank the label
+  is the variable name, an equals sign and the expression, otherwise the
+  configured label is used as given.
+
+  VAR_DEPS lists every input the expression uses - physical columns and
+  derived variables alike.  The statements are generated in dependency order,
+  so a derived variable may build on another one without a sequence number
+  being maintained.  A cycle is rejected.
+
+  Semicolons, macro triggers and comments are rejected, so a row can only ever
+  be an expression.  A derived variable may not reuse the name of a physical
+  column, and may not reference itself: an intra-row calculation would imply a
+  retained value, which depends on the order rows happen to arrive in.
+
+  <h4> SAS Macros </h4>
+  @li mp_derivedvars.sas
+  @li mp_abort.sas
+
+  @param [in] libds= The target table (libref.dataset, lib must be assigned)
+  @param [in] dclib= The Data Controller control library
+  @param [out] outref= (0) Fileref to receive the generated statements
+  @param [out] outvars= (work.derived_vars) Dataset of derived variable names
+  @param [in] outview= (0) Optional two-level name for the derived view
+
+  @version 9.2
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+%macro mpe_derived(libds=,
+  dclib=,
+  outref=0,
+  outvars=work.derived_vars,
+  outview=0
+);
+
+%local reserved;
+
+/* the only name that has to be refused outright - the metadata columns of the
+  target table are physical columns, so the engine's collision check covers
+  them whatever MPE_TABLES calls them */
+%let reserved=%str(_____DELETE__THIS__RECORD_____);
+
+/* dc_dttmtfmt is the bitemporal cut-off used in the where= below - callers get
+  it from %mpeinit().  Unset, the filter degenerates to "lt tx_to" and the
+  control-table step fails with a syntax error a long way from the cause. */
+%mp_abort(iftrue= (%length(&libds)=0 or %length(&dclib)=0 or %length(&dc_dttmtfmt)=0)
+  ,mac=&sysmacroname
+  ,msg=%str(Missing: libds dclib dc_dttmtfmt)
+)
+
+/* A format catalog is addressed as LIBREF.CATALOGNAME-FC (the form
+  mpe_validatecol accepts, and the one viewtables returns).  A catalog has no
+  observations, so it can have no derived variables - and it is not a dataset,
+  so the engine's existence check would abort the caller.  Return an empty
+  variable list and an empty statement file instead, and leave outview alone:
+  a caller only reads the view when the list is not empty. */
+%if %length(&libds)>3 and "%upcase(%substr(&libds,%length(&libds)-2,3))"="-FC" %then %do;
+  /* the file has to exist even though it stays empty - callers %include it
+    unconditionally, and a fileref with no file behind it fails there */
+  %if "%upcase(&outref)" ne "0" %then %do;
+    filename &outref temp lrecl=32767;
+    data _null_;
+      file &outref;
+    run;
+  %end;
+  /* The shape is the engine's own: mp_derivedvars builds
+    `data &outvars(keep=name); length name $32; name=''; if 0; run;` before it
+    returns (empty case, and again from the sorted list in the populated case),
+    so NAME is the only column either path can produce.  Every caller reads
+    NAME alone - getdata's `where a.NAME in (select NAME from
+    work.derived_vars)` is the only projection of this dataset in the SAS tree. */
+  data &outvars(keep=name);
+    length name $32;
+    name='';
+    if 0;
+  run;
+  %return;
+%end;
+
+%mp_derivedvars(
+  libds=&libds
+  ,control=&dclib..mpe_derived
+  ,where=%str(&dc_dttmtfmt lt tx_to and active=1)
+  ,outref=&outref
+  ,outvars=&outvars
+  ,outview=&outview
+  ,reserved=&reserved
+)
+
+%mend mpe_derived;

sas/sasjs/macros/mpe_derived.test.sas

@@ -0,0 +1,343 @@
+/**
+  @file
+  @brief testing the mpe_derived macro
+
+  @details Covers the generated statements, the view, the metadata the callers
+  consume, and the dependency rules.
+
+  The configuration checks abort the run when they fail, so they are not
+  exercised here - this suite covers the behaviour that callers depend on.
+
+  <h4> SAS Macros </h4>
+  @li mf_existds.sas
+  @li mp_assert.sas
+  @li mp_assertcols.sas
+  @li mp_assertdsobs.sas
+  @li mp_assertscope.sas
+  @li mpe_derived.sas
+
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+
+/* a source table, and a table with no derived variables */
+data work.class;
+  set sashelp.class;
+run;
+data work.noconfig;
+  set sashelp.class;
+run;
+
+/* clean up any config left behind by an earlier run */
+proc sql;
+delete from &dclib..mpe_derived where upcase(libref)='WORK';
+quit;
+
+/**
+  * Test 1 - statements, view, chaining
+  */
+
+proc sql;
+insert into &dclib..mpe_derived set
+    tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,libref='WORK'
+    ,dsn='CLASS'
+    ,var_nm='BAND'
+    ,var_deps='AGE'
+    ,var_type='C'
+    ,var_len=8
+    ,var_fmt=''
+    ,var_label='Age band'
+    ,var_logic="ifc(AGE>13,'TEEN','JUNIOR')"
+    ,active=1;
+insert into &dclib..mpe_derived set
+    tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,libref='WORK'
+    ,dsn='CLASS'
+    ,var_nm='BAND_AGE'
+    ,var_deps='BAND AGE'
+    ,var_type='C'
+    ,var_len=16
+    ,var_fmt=''
+    ,var_logic="cats(BAND,'-',AGE)"
+    ,active=1;
+insert into &dclib..mpe_derived set
+    tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,libref='WORK'
+    ,dsn='CLASS'
+    ,var_nm='DOUBLE_AGE'
+    ,var_deps='AGE'
+    ,var_type='N'
+    ,var_len=8
+    ,var_fmt=''
+    ,var_logic='AGE * 2'
+    ,active=1;
+quit;
+
+/* lower case libds on purpose - the config match must be case insensitive */
+/* the setup SQL above creates the SQL automatic macro variables, so the
+  snapshot is taken here - the check is for the macro's own leakage */
+%mp_assertscope(SNAPSHOT)
+
+%mpe_derived(libds=work.class,
+  dclib=&dclib,
+  outref=derref1,
+  outview=work.vw_derived,
+  outvars=work.derived_vars
+)
+
+%mp_assertdsobs(work.derived_vars,
+  desc=Test1 - one row per derived variable,
+  test=EQUALS 3,
+  outds=work.test_results
+)
+
+%mp_assertcols(work.vw_derived,
+  desc=Test1 - the view carries the derived columns,
+  cols=BAND BAND_AGE DOUBLE_AGE,
+  test=ALL,
+  outds=work.test_results
+)
+
+data work.chk1;
+  set work.vw_derived;
+  where BAND='TEEN';
+run;
+%mp_assertdsobs(work.chk1,
+  desc=Test1 - the filter on a derived column selects rows,
+  test=HASOBS,
+  outds=work.test_results
+)
+
+/* the chained value only resolves if the statements run in order */
+data work.chk1b;
+  set work.vw_derived;
+  where BAND='TEEN';
+  if BAND_AGE ne cats(BAND,'-',AGE) then output;
+run;
+%mp_assertdsobs(work.chk1b,
+  desc=Test1 - a derived variable can reference an earlier one,
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+data work.chk1c;
+  set work.vw_derived;
+  if DOUBLE_AGE ne AGE*2 then output;
+run;
+%mp_assertdsobs(work.chk1c,
+  desc=Test1 - the numeric derived value is correct,
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+/* proc contents on a view reports a format name without its width, so the
+  format is compared on the name only */
+proc contents noprint data=work.vw_derived
+  out=work.vw_cols(keep=name label format);
+run;
+data work.chk1d;
+  set work.vw_cols;
+  where name='BAND' and label='Age band' and format=:'$';
+run;
+%mp_assertdsobs(work.chk1d,
+  desc=Test1 - the label and default format are applied,
+  test=EQUALS 1,
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - a configured format is used as-is
+  */
+
+proc sql;
+insert into &dclib..mpe_derived set
+    tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,libref='WORK'
+    ,dsn='CLASS'
+    ,var_nm='FLAG_DT'
+    ,var_deps=''
+    ,var_type='N'
+    ,var_len=8
+    ,var_fmt='DATE9.'
+    ,var_logic='mdy(1,1,2020)'
+    ,active=1;
+quit;
+
+%mpe_derived(libds=WORK.CLASS,
+  dclib=&dclib,
+  outref=derref2,
+  outview=work.vw_derived2,
+  outvars=work.derived_vars2
+)
+
+/* asserted on the emitted statement - proc contents on a view does not
+  report the format in full */
+data work.chk2;
+  infile derref2 truncover;
+  input line $4000.;
+  if index(line,'format=DATE9.')>0 then output;
+run;
+%mp_assertdsobs(work.chk2,
+  desc=Test2 - the configured format is applied,
+  test=EQUALS 1,
+  outds=work.test_results
+)
+
+data work.chk2b;
+  set work.vw_derived2;
+  if flag_dt ne mdy(1,1,2020) then output;
+  stop;
+run;
+%mp_assertdsobs(work.chk2b,
+  desc=Test2 - the numeric value is correct,
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+/**
+  * Test 3 - no configuration
+  */
+
+%mpe_derived(libds=WORK.NOCONFIG,
+  dclib=&dclib,
+  outref=derref3,
+  outview=work.vw_derived3,
+  outvars=work.derived_vars3
+)
+
+%mp_assertdsobs(work.derived_vars3,
+  desc=Test3 - no derived variables returned,
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+%mp_assert(iftrue=(%mf_existds(libds=work.vw_derived3)=0),
+  desc=Test3 - no view is built when there is no config,
+  outds=work.test_results
+)
+
+data _null_;
+  infile derref3 end=eof;
+  input;
+  if not eof or cats(_infile_) ne '' then do;
+    putlog 'ERR' 'OR: derived statement file is not empty';
+    call symputx('emptycheck','0','l');
+  end;
+run;
+%mp_assert(iftrue=(%symexist(emptycheck)=0),
+  desc=Test3 - no statements are generated,
+  outds=work.test_results
+)
+
+/**
+  * Test 4 - the wrapper scopes the config: only the current, active row of a
+  * table is used
+  */
+
+proc sql;
+insert into &dclib..mpe_derived set
+    tx_from=0
+    ,tx_to='01JAN2020:00:00:00'dt
+    ,libref='WORK'
+    ,dsn='CLASS'
+    ,var_nm='CLOSED_OFF'
+    ,var_deps='AGE'
+    ,var_type='N'
+    ,var_len=8
+    ,var_fmt=''
+    ,var_logic='AGE + 1'
+    ,active=1;
+insert into &dclib..mpe_derived set
+    tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,libref='WORK'
+    ,dsn='CLASS'
+    ,var_nm='NOT_ACTIVE'
+    ,var_deps='AGE'
+    ,var_type='N'
+    ,var_len=8
+    ,var_fmt=''
+    ,var_logic='AGE + 2'
+    ,active=0;
+quit;
+
+%mpe_derived(libds=WORK.CLASS,
+  dclib=&dclib,
+  outref=derref4,
+  outview=work.vw_derived4,
+  outvars=work.derived_vars4
+)
+
+data work.chk4;
+  set work.derived_vars4;
+  where name in ('CLOSED_OFF','NOT_ACTIVE');
+run;
+
+%mp_assertdsobs(work.chk4,
+  desc=Test4 - a closed version and an inactive row are ignored,
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+/* tidy up so repeated runs start clean */
+proc sql;
+delete from &dclib..mpe_derived where upcase(libref)='WORK';
+quit;
+
+%mp_assertscope(COMPARE,ignorelist=_SASJS_WEBOUT_HEADERS,outds=work.test_results)
+
+/* Test 5 - a format catalog is not a dataset: no derived variables, no abort.
+  Callers pass LIBREF.CATALOGNAME-FC (validatefilter, getrawdata, viewdata), and
+  the engine's existence check would otherwise end the service. */
+proc format library=work.dcfmts;
+  value $band 'A'='Alpha';
+run;
+
+%mpe_derived(libds=work.DCFMTS-FC,
+  dclib=&dclib,
+  outref=derref5,
+  outvars=work.derived_vars5
+)
+
+%mp_assertdsobs(work.derived_vars5,
+  desc=Test5 - a format catalog yields no derived variables,
+  test=EQUALS 0,
+  outds=work.test_results
+)
+
+/* The branch has to create the statement file even though it writes nothing,
+  because callers %include it unconditionally.  Two things to prove: the branch created the file, and it left it empty.
+  fopen answers the first (0 when the fileref cannot be opened), the loop the
+  second - a single INPUT would stop the step at once on an empty file and never
+  reach the symputx. */
+%let deropen=0;
+%let derlines=-1;
+data _null_;
+  did=fopen('derref5');
+  call symputx('deropen',ifn(did>0,1,0),'l');
+  if did>0 then rc=fclose(did);
+run;
+data _null_;
+  infile derref5 end=eof;
+  n=0;
+  do while(not eof);
+    input;
+    n+1;
+  end;
+  call symputx('derlines',n,'l');
+run;
+
+%mp_assert(
+  iftrue=(&deropen=1 and &derlines=0),
+  desc=Test5 - a format catalog leaves an empty statement file,
+  outds=work.test_results
+)

sas/sasjs/macros/mpe_derived_check.sas

@@ -0,0 +1,75 @@
+/**
+  @file
+  @brief Field level checks for a staged MPE_DERIVED submission
+  @details Writes one row per staged row that breaks a rule, with the reason in
+  REASON, so a caller can report the first failure and abort.  Rows marked for
+  deletion are skipped - they are being removed, so their fields are not loaded.
+
+  The rules are the ones %mpe_derived applies at read time, so a submission
+  cannot stage a row the engine would then refuse.
+
+  @param inds the staged rows, already normalised by the caller
+  @param outds the dataset to create: one row per failing row, plus REASON
+
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+%macro mpe_derived_check(inds=work.staging_ds, outds=work.derived_check);
+data &outds;
+  set &inds;
+  length reason $200;
+  if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
+  call missing(reason);
+
+  if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
+  else if var_nm='' then reason='VAR_NM is required';
+  else if verify(strip(var_nm),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_')>0
+    then reason='VAR_NM must be a valid SAS name';
+  else if verify(substr(var_nm,1,1),'0123456789')=0
+    then reason='VAR_NM must not start with a digit';
+  else if var_nm='_____DELETE__THIS__RECORD_____'
+    then reason='VAR_NM is reserved';
+  else if var_type not in ('C','N')
+    then reason='VAR_TYPE must be C or N';
+  /* ACTIVE is a picker in the EDIT screen (HARDSELECT over the MPE_SELECTBOX
+    pair) - the same pair is enforced here, so a submission that bypasses the
+    grid cannot land a value the screen would never offer. */
+  else if missing(active) or active not in (0,1)
+    then reason='ACTIVE must be 1 or 0';
+  else if missing(var_len) or var_len ne int(var_len) or var_len<1
+    then reason='VAR_LEN must be a positive integer';
+  else if var_type='C' and var_len>32767
+    then reason='VAR_LEN exceeds 32767 for a character variable';
+  else if var_type='N' and var_len>8
+    then reason='VAR_LEN exceeds 8 for a numeric variable';
+  else if index(var_fmt,';')>0 or indexc(var_fmt,'&%')>0
+    or index(var_fmt,'/*')>0 or index(var_fmt,'*/')>0
+    then reason='VAR_FMT must be a format name, without ; & % or /* */';
+  else if verify(strip(var_fmt),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.')>0
+    then reason='VAR_FMT must be a format name';
+  /* VAR_LABEL is free text: any character is allowed, in any script, and the
+    quotes in it are escaped by %mp_derivedvars when the attrib statement is
+    built.  What is refused is anything that cannot be a single-line column
+    label - a control character or a line break - and a value long enough to
+    be cut down silently, since the engine truncates the label to 256. */
+  else if anycntrl(var_label)>0
+    then reason='VAR_LABEL must be plain text - no line breaks or control characters';
+  else if klength(var_label)>256
+    then reason='VAR_LABEL exceeds 256 characters';
+  else if missing(var_logic)
+    then reason='VAR_LOGIC is required';
+  else if index(var_logic,';')>0
+    then reason='VAR_LOGIC must be a single expression - remove the ;';
+  else if indexc(var_logic,'&%')>0
+    then reason='VAR_LOGIC must not contain macro triggers';
+  else if index(var_logic,'/*')>0 or index(var_logic,'*/')>0
+    then reason='VAR_LOGIC must not contain comments';
+
+  if not missing(reason) then output;
+run;
+
+%mend mpe_derived_check;

sas/sasjs/macros/mpe_derived_check.test.sas

@@ -0,0 +1,194 @@
+/**
+  @file
+  @brief testing the mpe_derived_check macro
+
+  @details Checks each field level rule the MPE_DERIVED post edit hook applies
+  to a staged submission.  The macro is exercised directly so the rules can be
+  probed with values a service cannot carry: VAR_LABEL is varchar(256) in the
+  table and a staging dataset declares the same length, so a longer label is
+  truncated before any check would see it.
+
+  <h4> SAS Macros </h4>
+  @li mf_nobs.sas
+  @li mp_assert.sas
+  @li mpe_derived_check.sas
+
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+/**
+  * Test 1 - one row per failing rule, and only failing rows
+  */
+data work.staging_ds;
+  length libref $8 dsn $32 var_nm $32 var_type $1 var_len 8
+    var_fmt $32 var_label $4000 var_deps $4000 var_logic $4000
+    active 8 _____DELETE__THIS__RECORD_____ $3;
+  libref='DCTEST';
+  dsn='DC_DERIV';
+  /* valid unless a row says otherwise - ACTIVE is mandatory in the screen and
+    the check enforces the same pair here */
+  active=1;
+  _____DELETE__THIS__RECORD_____='No';
+
+  /* meets every rule */
+  var_nm='GOOD';
+  var_type='N';
+  var_len=8;
+  var_fmt='';
+  var_label='a plain label';
+  var_deps='';
+  var_logic='1';
+  output;
+
+  /* delete marked rows are being removed, so their fields are not loaded */
+  var_nm='DELME';
+  var_label='first line'||'0A'x||'second line';
+  _____DELETE__THIS__RECORD_____='Yes';
+  output;
+  _____DELETE__THIS__RECORD_____='No';
+
+  /* free text is allowed, including quotes and punctuation */
+  var_nm='FREETXT';
+  var_label='It''s "quoted", plain text: 100 percent';
+  output;
+
+  /* a line break cannot be a single line column label */
+  var_nm='LBLBREAK';
+  var_label='first line'||'0A'x||'second line';
+  output;
+
+  /* longer than the table holds, so it would be cut down silently */
+  var_nm='LBLLONG';
+  var_label=repeat('x',300);
+  output;
+
+  /* a format name, not an expression */
+  var_nm='BADFMT';
+  var_fmt='DATE9;';
+  var_label='fmt';
+  output;
+
+  /* one expression, no statement separator, no macro triggers */
+  var_nm='BADLOGIC';
+  var_fmt='';
+  var_logic='x; y';
+  output;
+
+  /* C or N */
+  var_nm='BADTYPE';
+  var_logic='1';
+  var_type='X';
+  output;
+
+  /* ACTIVE is a picker in the screen: only 1 or 0 */
+  var_nm='BADACTIVE';
+  var_type='C';
+  var_logic='1';
+  active=2;
+  output;
+
+  stop;
+run;
+
+%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
+
+/**
+  * Test 2 - the reported rows, rule by rule
+  */
+data work.expected;
+  length var_nm $32 reason $200;
+  var_nm='LBLBREAK';
+  reason='VAR_LABEL must be plain text - no line breaks or control characters';
+  output;
+  var_nm='LBLLONG';
+  reason='VAR_LABEL exceeds 256 characters';
+  output;
+  var_nm='BADFMT';
+  reason='VAR_FMT must be a format name, without ; & % or /* */';
+  output;
+  var_nm='BADLOGIC';
+  reason='VAR_LOGIC must be a single expression - remove the ;';
+  output;
+  var_nm='BADTYPE';
+  reason='VAR_TYPE must be C or N';
+  output;
+  var_nm='BADACTIVE';
+  reason='ACTIVE must be 1 or 0';
+  output;
+run;
+
+proc sql;
+create table work.cmp as
+  select e.var_nm
+    ,case when a.reason=e.reason then 1 else 0 end as same
+  from work.expected e
+  left join work.derived_check a
+    on upcase(strip(a.var_nm))=upcase(e.var_nm);
+quit;
+
+data _null_;
+  set work.cmp;
+  call symputx(cats('rule_',var_nm),same,'l');
+run;
+
+%mp_assert(
+  iftrue=(%mf_nobs(work.derived_check)=6),
+  desc=%str(Checking only the six failing rows are reported),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(&rule_LBLBREAK=1),
+  desc=%str(Checking a label with a line break is rejected),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(&rule_LBLLONG=1),
+  desc=%str(Checking a label over 256 characters is rejected),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(&rule_BADFMT=1),
+  desc=%str(Checking a format carrying a statement separator is rejected),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(&rule_BADLOGIC=1),
+  desc=%str(Checking a logic value with a statement separator is rejected),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(&rule_BADTYPE=1),
+  desc=%str(Checking a VAR_TYPE other than C or N is rejected),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=(&rule_BADACTIVE=1),
+  desc=%str(Checking an ACTIVE outside 1 or 0 is rejected),
+  outds=work.test_results
+)
+
+/**
+  * Test 3 - a row that meets every rule, a free text label, and a delete
+  * marked row are all left alone
+  */
+proc sql noprint;
+select count(*) into: notchecked
+  from work.derived_check
+  where upcase(strip(var_nm)) in ('GOOD','FREETXT','DELME');
+quit;
+
+%mp_assert(
+  iftrue=(&notchecked=0),
+  desc=%str(Checking a valid row, a free text label and a delete marked row are not reported),
+  outds=work.test_results
+)

sas/sasjs/macros/mpe_execute.sas

@@ -0,0 +1,160 @@
+/**
+  @file
+  @brief Runs a DC service the way a test needs it run
+  @details Wraps mx_execute so that every call is measurable, using only
+  documented mx_execute outputs:
+
+  @li the child's log is kept when mdebug is on, through outlogds, and printed
+    as CHILD: lines.  A normal run asks for no log, so it stays quiet - flip
+    sasjs_mdebug for an investigation run instead of editing tests;
+  @li the child's outcome is written to outstatus (STATE, URI, ERRMSG,
+    PLATFORM).  state=canceled means the child aborted, so a test can assert
+    what happened rather than inferring it from an outlib that may still hold
+    the previous call's tables;
+  @li failfast aborts the caller as soon as the child is not completed, so the
+    first failure is the real one and the log names it.  Tests that expect an
+    abort pass failfast=0 and check the status row.  SAS 9 reports no job state
+    at all (PROC STP), which core records as state=unknown; that is treated as
+    unmeasured rather than failed, and noted in the log.
+
+  Usage:
+
+      %mpe_execute(&appLoc/services/editors/getdata,
+        viyacontext=&defaultcontext,
+        inputfiles=&f1:sascontroltable,
+        outlib=web1
+      )
+      %put reaching here means the child completed;
+      data _null_;
+        set work.dc_childstatus;
+        putlog "child said: " state= errmsg=;
+      run;
+
+  @param [in] program The _PROGRAM endpoint to run
+  @param [in] inputfiles=(0) fileref:filename pairs, as mx_execute
+  @param [in] inputdatasets=(0) datasets to send as CSV, as mx_execute
+  @param [in] inputparams=(0) name/value dataset, as mx_execute
+  @param [out] outlib=(0) Output libref for the response JSON
+  @param [out] outref=(0) Output fileref for the raw response text
+  @param [in] debug= (131) The _debug value sent to the child.  Numeric, as the
+    server requires; 131 asks for the log.
+  @param [in] mdebug= (&sasjs_mdebug) 1 keeps the child's log and prints it
+  @param [in] viyacontext= (&defaultcontext) Viya compute context
+  @param [out] outstatus=(work.dc_childstatus) One row per child run
+  @param [in] failfast= (1) Abort the caller when the child did not complete
+
+  <h4> SAS Macros </h4>
+  @li mp_abort.sas
+  @li mx_execute.sas
+
+**/
+
+%macro mpe_execute(program
+    ,inputfiles=0
+    ,inputdatasets=0
+    ,inputparams=0
+    ,outlib=0
+    ,outref=0
+    ,debug=
+    ,mdebug=&sasjs_mdebug
+    ,viyacontext=&defaultcontext
+    ,outstatus=work.dc_childstatus
+    ,viyaresult=WEBOUT_JSON
+    ,failfast=1
+  );
+%local logds state msg;
+/* Aliases: passing &program / &inputfiles ... straight into another macro that
+  * declares the same parameter names makes SAS resolve them against the callee
+  * and null them ("recursive reference to the macro variable").  Unique names
+  * on this side keep the values intact. */
+%local a_viyaresult a_pgm a_inds a_indatasets a_inparams a_dbg a_mdbg
+  a_outlib a_outref a_logds a_status a_ctx;
+/* numeric: the server validates _debug as a number, so anything else never asks
+  * for the log and outlogds comes back empty */
+%if %length(&debug)=0 %then %let debug=131;
+%if %length(&mdebug)=0 %then %let mdebug=0;
+%let a_pgm=&program;
+%let a_viyaresult=&viyaresult;
+%let a_inds=&inputfiles;
+%let a_indatasets=&inputdatasets;
+%let a_inparams=&inputparams;
+%let a_dbg=&debug;
+%let a_mdbg=&mdebug;
+%let a_outlib=&outlib;
+%let a_outref=&outref;
+%let a_status=&outstatus;
+%let a_ctx=&viyacontext;
+
+%if &mdebug>0 %then %let logds=work.dc_childlog;
+%else %let logds=_null_;
+%let a_logds=&logds;
+
+/* a status row from an earlier call must not be read as this call's outcome:
+  a child that dies before core writes one would otherwise look completed */
+%if %index(&a_status,.)>0 and %sysfunc(exist(&a_status)) %then %do;
+  proc datasets library=%scan(&a_status,1,.) nolist;
+    delete %scan(&a_status,2,.);
+  quit;
+%end;
+
+%mx_execute(&a_pgm
+  ,inputfiles=&a_inds
+  ,inputdatasets=&a_indatasets
+  ,inputparams=&a_inparams
+  ,debug=&a_dbg
+  ,mdebug=&a_mdbg
+  ,viyacontext=&a_ctx
+  ,outlib=&a_outlib
+  ,outref=&a_outref
+  ,outlogds=&a_logds
+  ,outstatus=&a_status
+  ,viyaresult=&a_viyaresult
+)
+
+/* the child's own log, for an investigation run */
+%if &mdebug>0 %then %do;
+  data _null_;
+    set &a_status;
+    putlog "DC_EXECUTE: " state= errmsg=;
+  run;
+  %if %sysfunc(exist(work.dc_childlog)) %then %do;
+    data _null_;
+      set work.dc_childlog;
+      putlog "CHILD: " line;
+    run;
+  %end;
+%end;
+
+/* fail fast - a child that did not complete must not leave the test running
+  * against whatever the previous call left in outlib.  A missing status table
+  * and an empty one are both failures: state would stay blank and the caller
+  * would carry on inferring, which is the thing this macro exists to stop. */
+%let state=;
+%let msg=;
+%if %sysfunc(exist(&a_status))=1 %then %do;
+  data _null_;
+    set &a_status;
+    call symputx('state',state,'l');
+    call symputx('msg',errmsg,'l');
+  run;
+%end;
+%else %let state=missing(&a_status);
+
+%if %length(&failfast)=0 %then %let failfast=1;
+%if &failfast=1 %then %do;
+  /* SAS 9 (PROC STP) reports no job status at all: core writes state='unknown'
+  * with that explanation on a call that worked, so it cannot be read as a
+  * failure - only as unmeasured.  Everything else that is not completed is a
+  * failure, a blank state included. */
+  %if "%upcase(&state)"="UNKNOWN" %then
+    %put NOTE: &sysmacroname: &program returned no job status (SAS 9) - the outcome was not measured;
+  %else %if "%upcase(&state)" ne "COMPLETED" %then %do;
+    %mp_abort(
+      iftrue=(1=1)
+      ,mac=&sysmacroname
+      ,msg=%str(&program did not complete: state=&state &msg)
+    )
+  %end;
+%end;
+
+%mend mpe_execute;

sas/sasjs/macros/mpe_execute.test.sas

@@ -0,0 +1,125 @@
+/**
+  @file
+  @brief testing the mpe_execute macro
+
+  @details The wrapper exists so that a test can assert what a child call did
+  instead of inferring it from an outlib that still holds the previous call's
+  tables.  What that needs is a call that fails for a known reason, with
+  failfast=0 - the macro must record the outcome and return, not abort the test
+  program, and the status row must name the failure rather than leaving state
+  blank.
+
+  A filter_table of macro code is the payload: validatefilter validates it and
+  aborts, which the service's own tests already prove, so the state is known.
+
+  <h4> SAS Macros </h4>
+  @li mf_getuniquefileref.sas
+  @li mf_nobs.sas
+  @li mp_assert.sas
+  @li mpe_execute.sas
+
+  @author 4GL Apps Ltd
+  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
+    and may not be re-distributed or re-sold without the express permission of
+    4GL Apps Ltd.
+
+**/
+
+/**
+  * Test 1 - a child that aborts is recorded in outstatus, and the caller
+  * continues (failfast=0)
+  */
+%let _program=&appLoc/services/public/validatefilter;
+
+%let f1=%mf_getuniquefileref();
+data _null_;
+  file &f1 termstr=crlf;
+  put 'FILTER_TABLE:$41.';
+  put '%sysfunc(coalescec(&dclib..MPE_TABLES,))';
+run;
+
+%let outref1=%mf_getuniquefileref();
+
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:iwant,
+  outref=&outref1,
+  outstatus=work.exec_status,
+  failfast=0,
+  viyaresult=WEBOUT_TXT
+)
+
+/* The macro reads the LAST row of outstatus - a row is written per HTTP
+  * attempt, so a retried or aborted call leaves more than one - which is what
+  * the assertions below check.  Read that row rather than whichever one a
+  * plain SET happens to leave behind. */
+%let state=;
+%let uri=;
+%let platform=;
+%let errmsg=;
+data _null_;
+  set work.exec_status nobs=nobs;
+  if nobs>0 then do;
+    set work.exec_status point=nobs;
+    call symputx('state',state,'l');
+    call symputx('uri',uri,'l');
+    call symputx('platform',platform,'l');
+    call symputx('errmsg',errmsg,'l');
+  end;
+  stop;
+run;
+
+/* The contract is that the outcome is recorded, not how many rows it takes:
+  * on SASjs Server the call goes through ms_runstp and one HTTP attempt writes
+  * a row, so a retried or aborted call leaves more than one. */
+%mp_assert(
+  iftrue=(%mf_nobs(work.exec_status)>0),
+  desc=%str(Checking the status table records the call),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("%upcase(&state)" ne ""),
+  desc=%str(Checking the status row names the child state),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("%upcase(&state)" ne "COMPLETED"),
+  desc=%str(Checking an aborted child is not recorded as completed),
+  outds=work.test_results
+)
+
+%mp_assert(
+  iftrue=("%upcase(&platform)" ne ""),
+  desc=%str(Checking the status row names the platform),
+  outds=work.test_results
+)
+
+/**
+  * Test 2 - the child's log is only kept when mdebug asks for it
+  */
+%let f2=%mf_getuniquefileref();
+data _null_;
+  file &f2 termstr=crlf;
+  put 'FILTER_TABLE:$41.';
+  put '%sysfunc(coalescec(&dclib..MPE_TABLES,))';
+run;
+
+%let outref2=%mf_getuniquefileref();
+
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f2:iwant,
+  outref=&outref2,
+  outstatus=work.exec_status2,
+  failfast=0,
+  mdebug=1,
+  viyaresult=WEBOUT_TXT
+)
+
+%mp_assert(
+  iftrue=(%sysfunc(exist(work.dc_childlog))=1),
+  desc=%str(Checking mdebug=1 keeps the child log),
+  outds=work.test_results
+)

sas/sasjs/macros/mpe_makedata.sas

@@ -79,6 +79,10 @@ insert into &lib..mpe_config set
       !!' will require a decent browser (ie, not IE) and more memory on the'
       !!' client side.  The DC_MAXCELLS_WEBEDIT limit is applied at the same'
       !!' time, whichever is reached first.';
+/* Why the cell limit exists, kept out of var_desc - MPE_CONFIG.VAR_DESC is
+  varchar(300), and a truncated value raises a diagnostic that aborts this
+  whole build.  A wide table is expensive long before it is tall - 250 rows of
+  a 1000 column table is a quarter of a million cells. */
 insert into &lib..mpe_config set
     tx_from=0
     ,tx_to='31DEC9999:23:59:59'dt
@@ -88,9 +92,7 @@ insert into &lib..mpe_config set
     ,var_active=1
     ,var_desc='This sets the maximum number of CELLS (rows multiplied by'
       !!' columns) that can be loaded into the browser for editing in the EDIT'
-      !!' screen.  It exists because a wide table is expensive long before it'
-      !!' is tall - 250 rows of a 1000 column table is a quarter of a million'
-      !!' cells.  Whichever of this and DC_MAXOBS_WEBEDIT is reached first'
+      !!' screen.  Whichever of this and DC_MAXOBS_WEBEDIT is reached first'
       !!' applies.';
 insert into &lib..mpe_config set
     tx_from=0
@@ -677,6 +679,42 @@ insert into &lib..mpe_security set
       ,selectbox_value='UNLOCKED'
       ,selectbox_order=2
       ,ver_to_dttm='31DEC5999:23:59:59'dt;
+  insert into &lib..mpe_selectbox set
+      selectbox_rk=%mf_increment(rk)
+      ,ver_from_dttm=0
+      ,select_lib="&lib"
+      ,select_ds="MPE_DERIVED"
+      ,base_column="VAR_TYPE"
+      ,selectbox_value='C'
+      ,selectbox_order=1
+      ,ver_to_dttm='31DEC5999:23:59:59'dt;
+  insert into &lib..mpe_selectbox set
+      selectbox_rk=%mf_increment(rk)
+      ,ver_from_dttm=0
+      ,select_lib="&lib"
+      ,select_ds="MPE_DERIVED"
+      ,base_column="VAR_TYPE"
+      ,selectbox_value='N'
+      ,selectbox_order=2
+      ,ver_to_dttm='31DEC5999:23:59:59'dt;
+  insert into &lib..mpe_selectbox set
+      selectbox_rk=%mf_increment(rk)
+      ,ver_from_dttm=0
+      ,select_lib="&lib"
+      ,select_ds="MPE_DERIVED"
+      ,base_column="ACTIVE"
+      ,selectbox_value='1'
+      ,selectbox_order=1
+      ,ver_to_dttm='31DEC5999:23:59:59'dt;
+  insert into &lib..mpe_selectbox set
+      selectbox_rk=%mf_increment(rk)
+      ,ver_from_dttm=0
+      ,select_lib="&lib"
+      ,select_ds="MPE_DERIVED"
+      ,base_column="ACTIVE"
+      ,selectbox_value='0'
+      ,selectbox_order=2
+      ,ver_to_dttm='31DEC5999:23:59:59'dt;
   insert into &lib..mpe_selectbox set
       selectbox_rk=%mf_increment(rk)
       ,ver_from_dttm=0
@@ -1493,6 +1531,19 @@ insert into &lib..mpe_selectbox set
       ,var_txfrom='TX_FROM'
       ,var_txto='TX_TO'
     ;
+  insert into &lib..mpe_tables
+    set tx_from=0
+      ,tx_to='31DEC5999:23:59:59'dt
+      ,libref="&lib"
+      ,dsn='MPE_DERIVED'
+      ,num_of_approvals_required=1
+      ,loadtype='TXTEMPORAL'
+      ,buskey='LIBREF DSN VAR_NM'
+      ,post_edit_hook='services/hooks/mpe_derived_postedit'
+      ,notes='Derived (computed) variables for the EDIT screen'
+      ,var_txfrom='TX_FROM'
+      ,var_txto='TX_TO'
+    ;
   insert into &lib..mpe_tables
     set tx_from=0
       ,tx_to='31DEC5999:23:59:59'dt
@@ -1942,6 +1993,115 @@ insert into &lib..MPE_VALIDATIONS set
     ,rule_value="services/validations/columns_in_libds"
     ,rule_active=1
     ,tx_to='31DEC5999:23:59:59'dt;
+
+/* MPE_DERIVED - the field rules are also enforced server-side, in
+  * services/hooks/mpe_derived_postedit.sas.  These rows give the user
+  * immediate feedback in the EDIT screen. */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="LIBREF"
+    ,rule_type='NOTNULL',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="LIBREF"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* the libraries the user can edit, as MPE_VALIDATIONS.BASE_LIB does */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="LIBREF"
+    ,rule_type='SOFTSELECT_HOOK'
+    ,rule_value="services/validations/libraries_editable"
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="DSN"
+    ,rule_type='NOTNULL',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="DSN"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* tables in the LIBREF on the same row - the hook reads it from source_row */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="DSN"
+    ,rule_type='SOFTSELECT_HOOK'
+    ,rule_value="services/validations/tables_editable"
+    ,rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_NM"
+    ,rule_type='NOTNULL',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_NM"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_NM"
+    ,rule_type='HARDREGEX',rule_value='^[A-Za-z_][A-Za-z0-9_]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_DEPS"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_DEPS"
+    ,rule_type='HARDREGEX',rule_value='^[A-Za-z0-9_ ]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
+    ,rule_type='NOTNULL',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
+    ,rule_type='HARDSELECT',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+/* ACTIVE is mandatory, and its NOTNULL rule carries '1' as its value: the EDIT
+  * screen pre-fills a new row from the value of the NOTNULL / READONLY / HIDDEN
+  * rule (getColumnDefault), so a derived variable starts active unless the user
+  * says otherwise.  The values themselves come from MPE_SELECTBOX. */
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="ACTIVE"
+    ,rule_type='NOTNULL',rule_value='1',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
+    ,rule_type='NOTNULL',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
+    ,rule_type='MINVAL',rule_value='1',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
+    ,rule_type='MAXVAL',rule_value='32767',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_FMT"
+    ,rule_type='CASE',rule_value='UPCASE',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_FMT"
+    ,rule_type='HARDREGEX'
+    ,rule_value='^(\$?[A-Za-z][A-Za-z0-9_]*(\.[0-9]+)?\.?|\$?[0-9]+(\.[0-9]+)?\.?)?$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_LOGIC"
+    ,rule_type='NOTNULL',rule_value=' ',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_LABEL"
+    ,rule_type='HARDREGEX'
+    ,rule_value='^[^\x00-\x1F\x7F]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+insert into &lib..MPE_VALIDATIONS set
+    tx_from=0,base_lib="&lib",base_ds="MPE_DERIVED",base_col="VAR_LOGIC"
+    ,rule_type='HARDREGEX'
+    ,rule_value='^(?!.*(/\*|\*/))[^;&%]*$',rule_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
 insert into &lib..MPE_VALIDATIONS set
     tx_from=0
     ,base_lib="&lib"
@@ -2054,8 +2214,8 @@ insert into &lib..MPE_VALIDATIONS set
     ,rule_active=1
     ,tx_to='31DEC5999:23:59:59'dt;
 /* test soft regex - PRIMARY_KEY_FIELD should be an integer (yellow if it
-contains a decimal point).  All generated keys are integers, so no
-warning shows until a demo user enters a decimal. */
+contains a decimal point).  All generated keys are integers, so nothing is
+flagged until a demo user enters a decimal. */
 insert into &lib..MPE_VALIDATIONS set
     tx_from=0
     ,base_lib="&lib"

sas/sasjs/macros/mpe_makedatamodel.sas

@@ -692,6 +692,29 @@ proc datasets lib=&lib noprint;
     pk_mpe_validations=(tx_from base_lib base_ds base_col rule_type)
     /nomiss unique;
 quit;
+/* mpe_derived - computed variables for the EDIT screen */
+proc sql;
+create table &lib..MPE_DERIVED
+  (
+    TX_FROM num &notnull format=datetime19.3,
+    LIBREF varchar(8),
+    DSN varchar(32),
+    VAR_NM varchar(32),
+    VAR_DEPS varchar(4000),
+    VAR_TYPE varchar(1),
+    VAR_LEN num,
+    VAR_FMT varchar(32),
+    VAR_LABEL varchar(256),
+    VAR_LOGIC varchar(4000),
+    ACTIVE num,
+    TX_TO num &notnull format=datetime19.3
+);quit;
+proc datasets lib=&lib noprint;
+  modify mpe_derived;
+  index create
+    pk_mpe_derived=(tx_from libref dsn var_nm)
+    /nomiss unique;
+quit;
 proc sql;
 create table &lib..mpe_x_test(
           primary_key_field num &notnull,

sas/sasjs/macros/mpe_refreshcatalogs.sas

@@ -2,11 +2,17 @@
   @file
   @brief Refreshes the data catalog
   @details Assumes library is already assigned.
+
+  @param [in] etlsource= (0) Defaults to %mf_getuniquename(prefix=refreshcatalogs); it
+    identifies this load in the MPE_DATALOADS audit row.
+
   Usage:
 
       %mpe_refreshcatalogs(sashelp)
 
   <h4> SAS Macros </h4>
+  @li mf_getuniquename.sas
+  @li mp_abort.sas
   @li bitemporal_dataloader.sas
 
   @version 9.3
@@ -14,9 +20,12 @@
 
 **/
 
-%macro mpe_refreshcatalogs(lib,cat=#all);
+%macro mpe_refreshcatalogs(lib,cat=#all
+  ,etlsource=0
+);
 %let lib=%upcase(&lib);
 %let cat=%upcase(&cat);
+%if "&etlsource"="0" %then %let etlsource=%mf_getuniquename(prefix=refreshcatalogs);
 
 %put running &sysmacroname &lib for &cat;
 proc sql;
@@ -50,7 +59,7 @@ create table datacats as select distinct libref,memname from catdata;
   ,base_dsn=mpe_datacatalog_CATS
   ,append_dsn=datacats
   ,PK=LIBREF MEMNAME
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO
@@ -76,7 +85,7 @@ quit;
   ,base_dsn=mpe_datacatalog_OBJS
   ,append_dsn=dataobjs
   ,PK=LIBREF MEMNAME OBJNAME OBJTYPE
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO
@@ -102,7 +111,7 @@ create table statusobjs as
   ,base_dsn=mpe_datastatus_OBJS
   ,append_dsn=statusobjs
   ,PK=LIBREF MEMNAME OBJNAME OBJTYPE
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO
@@ -127,7 +136,7 @@ create table statuscats as
   ,base_dsn=mpe_datastatus_cats
   ,append_dsn=statuscats
   ,PK=LIBREF MEMNAME
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO

sas/sasjs/macros/mpe_refreshlibs.sas

@@ -3,7 +3,12 @@
   @brief Refreshes the library data catalog
   @details
 
+  @param [in] etlsource= (0) Defaults to %mf_getuniquename(prefix=refreshlibs), identifies this load
+    in the MPE_DATALOADS audit row.  That table is keyed on etlsource together
+    with processed_dttm, libref and dsn, so the uid keeps each load distinct.
+
   <h4> SAS Macros </h4>
+  @li mf_getuniquename.sas
   @li dc_getlibs.sas
   @li mf_getplatform.sas
   @li bitemporal_dataloader.sas
@@ -16,7 +21,14 @@
 
 **/
 
-%macro mpe_refreshlibs(lib=0);
+%macro mpe_refreshlibs(lib=0
+  ,etlsource=0
+);
+/* a default of 0 rather than a %mf_uid() expression: a default is
+  resolved when the macro is compiled, so it would be one value for the
+  whole session and the audit folder below it would collide on the
+  second call - as it did between the refresh steps of one run */
+%if "&etlsource"="0" %then %let etlsource=%mf_getuniquename(prefix=refreshlibs);
 
 %dc_getlibs(outds=work.mm_getLibs)
 proc sort data=mm_getlibs;
@@ -119,7 +131,7 @@ create table libs5 as
   ,base_dsn=MPE_DATACATALOG_LIBS
   ,append_dsn=libs5
   ,PK=LIBREF
-  ,etlsource=&_program
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO

sas/sasjs/macros/mpe_refreshtables.sas

@@ -1,12 +1,18 @@
 /**
   @file mpe_refreshtables.sas
   @brief Refreshes the data catalog
-  @details Assumes library is already assigned.
+  @details
+
+  @param [in] etlsource= (0) Defaults to %mf_getuniquename(prefix=refreshtables), identifies this load
+    in the MPE_DATALOADS audit row.  That table is keyed on etlsource together
+    with processed_dttm, libref and dsn, so the uid keeps each load distinct.
   Usage:
 
       %mpe_refreshtables(sashelp)
 
   <h4> SAS Macros </h4>
+  @li mf_getuniquename.sas
+  @li mp_abort.sas
   @li bitemporal_dataloader.sas
   @li mf_existfeature.sas
   @li mf_getengine.sas
@@ -17,7 +23,14 @@
   @author 4GL Apps Ltd
 **/
 
-%macro mpe_refreshtables(lib,ds=#all);
+%macro mpe_refreshtables(lib,ds=#all
+  ,etlsource=0
+);
+/* a default of 0 rather than a %mf_uid() expression: a default is
+  resolved when the macro is compiled, so it would be one value for the
+  whole session and the audit folder below it would collide on the
+  second call - as it did between the refresh steps of one run */
+%if "&etlsource"="0" %then %let etlsource=%mf_getuniquename(prefix=refreshtables);
 %let lib=%upcase(&lib);
 %let ds=%upcase(&ds);
 %local engine; %let engine=%mf_getengine(&lib);
@@ -180,7 +193,7 @@ create table cols as
   ,base_dsn=mpe_datacatalog_vars
   ,append_dsn=finalcols
   ,PK=LIBREF DSN NAME
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO
@@ -242,7 +255,7 @@ create table work.finaltabs as
   ,base_dsn=mpe_datacatalog_tabs
   ,append_dsn=finaltabs
   ,PK=LIBREF DSN
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO
@@ -302,7 +315,7 @@ proc sql;
   ,base_dsn=mpe_datastatus_tabs
   ,append_dsn=statustabs
   ,PK=LIBREF DSN
-  ,etlsource=&sysmacroname
+  ,etlsource=&etlsource
   ,loadtype=TXTEMPORAL
   ,tech_from=TX_FROM
   ,tech_to=TX_TO
@@ -345,7 +358,7 @@ proc sql;
     ,base_dsn=mpe_datastatus_libs
     ,append_dsn=statuslibs
     ,PK=LIBREF
-    ,etlsource=&sysmacroname
+    ,etlsource=&etlsource
     ,loadtype=TXTEMPORAL
     ,tech_from=TX_FROM
     ,tech_to=TX_TO

sas/sasjs/macros/mpe_validatecol.test.sas

@@ -60,7 +60,7 @@ data work.mismatch;
 run;
 
 %mp_assertdsobs(work.mismatch,
-  desc=Every LIBDS value validated as expected (catalog form permitted, nothing smuggled past the dash),
+  desc=%str(Every LIBDS value validated as expected (catalog form permitted, nothing smuggled past the dash)),
   test=EQUALS 0,
   outds=work.test_results
 )