Compare commits
68
Commits
424b111737
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
428e70a4c1 | ||
|
|
94e040c006 | ||
|
|
cdc0ab2a91 | ||
|
|
8449ab3987 | ||
|
|
405b28dccc | ||
|
|
559221fad4 | ||
|
|
999e16b8fc | ||
|
|
ce5b1aa961 | ||
|
|
f74100a818 | ||
|
|
a4a0f9cb2f | ||
|
|
e6a5cce475 | ||
|
|
97b4cd4d61 | ||
|
|
eeffa92f9e | ||
|
|
dd25be8e75 | ||
|
|
2e40749052 | ||
|
|
fc687f8643 | ||
|
|
c97e8866db | ||
|
|
ad061946c3 | ||
|
|
914a1606bb | ||
|
|
c803773d3c | ||
|
|
b8eea5063c | ||
|
|
0a172b0c37 | ||
|
|
888873421c | ||
|
|
af1813f829 | ||
|
|
27686d3aad | ||
|
|
d84a1d50f4 | ||
|
|
cf7b21a8b6 | ||
|
|
60589c78b6 | ||
|
|
8e2cce4150 | ||
|
|
f7e753863b | ||
|
|
febfea3e5e | ||
|
|
3580d6e9eb | ||
|
|
bc9e3ae50f | ||
|
|
330448c496 | ||
|
|
00257f9160 | ||
|
|
e51953ca8a | ||
|
|
aa91daf9f7 | ||
|
|
00df87f189 | ||
|
|
65699bdd39 | ||
|
|
459ff2da37 | ||
|
|
c0ea810766 | ||
|
|
62e43034e4 | ||
|
|
80e8911af8 | ||
|
|
bd3c220af2 | ||
|
|
5e8910f238 | ||
|
|
221dd8a15e | ||
|
|
e9216b0895 | ||
|
|
37d81c902a | ||
|
|
689b5b6709 | ||
|
|
10157c1159 | ||
|
|
541a61ba08 | ||
|
|
ad50acf647 | ||
|
|
cd1ce4a7e2 | ||
|
|
7e3045a21b | ||
|
|
88010c3980 | ||
|
|
c5808a16eb | ||
|
|
0011f7e933 | ||
|
|
4ed1e0684d | ||
|
|
2623a3fb4b | ||
|
|
388f418bf4 | ||
|
|
f83dc359e2 | ||
|
|
411f6c3354 | ||
|
|
f042ca82ea | ||
|
|
15a1f34d52 | ||
|
|
6486c80ed4 | ||
|
|
dd482c4069 | ||
|
|
f58925b299 | ||
|
|
25ffa71add |
No files matched your search
@@ -1,10 +1,11 @@
|
||||
# Testing
|
||||
|
||||
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
|
||||
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name.
|
||||
|
||||
## Commands
|
||||
|
||||
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
|
||||
- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`.
|
||||
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
|
||||
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
|
||||
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: dc-estate-testing
|
||||
description: >
|
||||
Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS
|
||||
test suite against it. Covers the target and credential setup, the makedata seed the suite needs on
|
||||
a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend
|
||||
so the streamed app is the build you just made.
|
||||
---
|
||||
|
||||
# Deploying and testing on the shared SASjs Server estate
|
||||
|
||||
The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong.
|
||||
|
||||
cd sas && npm ci # macro dependencies
|
||||
cd ../client && npm ci && npm run build # frontend only - see "Deploying the frontend"
|
||||
cd ../sas && npm run server # cpfavicon + sasjs cbd -t server + makedata
|
||||
npx sasjs test -t server
|
||||
|
||||
`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `<appLoc>/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed.
|
||||
|
||||
## The targets
|
||||
|
||||
`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc` `/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair.
|
||||
|
||||
A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered.
|
||||
|
||||
To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app/<you>/dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at.
|
||||
|
||||
## Credentials
|
||||
|
||||
The CLI logs in with a registered client, not a username and password: `.env.<target>` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless.
|
||||
|
||||
Mint the pair from a local login, with the password read from a file rather than argv:
|
||||
|
||||
1. `GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header.
|
||||
2. `POST /SASLogon/login` with `{"username","password"}`.
|
||||
3. `GET /SASjsApi/client` on the same session - the registered client ids.
|
||||
4. `POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`.
|
||||
5. `POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`.
|
||||
|
||||
An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over.
|
||||
|
||||
## The makedata seed
|
||||
|
||||
`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
|
||||
|
||||
- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
|
||||
- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
|
||||
- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
|
||||
- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.
|
||||
|
||||
## Reading the results
|
||||
|
||||
Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON.
|
||||
|
||||
A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete.
|
||||
|
||||
`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for.
|
||||
|
||||
## Deploying the frontend
|
||||
|
||||
The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists:
|
||||
|
||||
1. `client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`.
|
||||
2. `client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines.
|
||||
3. `cd client && npm ci && npm run build`.
|
||||
4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc.
|
||||
5. `cd ../sas && npx sasjs cbd -t <target>` - with `client/dist` present the log says `Copying web app files` and `Web app is available at <url>`.
|
||||
|
||||
Without step 1 the deploy prints `ERROR An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main.<hash>.js` the page names.
|
||||
|
||||
Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check.
|
||||
|
||||
## Reading a failed run
|
||||
|
||||
- `ERROR An error occurred while executing the request. Job location: <appLoc>/tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/<suite>.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise.
|
||||
- `WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem.
|
||||
- `sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree.
|
||||
- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code.
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env bash
|
||||
# The Cypress specs CI runs, and the order they run in.
|
||||
#
|
||||
# ORDER MATTERS. These specs share one mock estate that is seeded once per job,
|
||||
# so several of them depend on the state they find:
|
||||
#
|
||||
# - csv-limited asserts the free-tier "Locked Feature" modal, which an earlier
|
||||
# spec can remove by planting a valid licence. It has always run first.
|
||||
# - hook-programs asserts that an edit raises an approval row against its
|
||||
# target table; it runs early, before other specs have added rows to the
|
||||
# approval tables.
|
||||
#
|
||||
# Specs added later are appended rather than inserted, so the sequence that was
|
||||
# known green keeps its shape. metanav is deliberately absent - see
|
||||
# https://git.datacontroller.io/dc/dc/issues/395.
|
||||
#
|
||||
# download is held out for a different reason - its download button is covered by
|
||||
# a fixed-position element, see
|
||||
# https://git.datacontroller.io/dc/dc/issues/398.
|
||||
set -euo pipefail
|
||||
|
||||
SPECS="csv-limited hook-programs liveness editor excel-multi-load excel csv filtering filter-panel licensing viewer-labels full-table-search viewbox stage browser-info ddl-export row-cell-limits derived-variables startupservice-launcher coltype-delete-record submitted-details embed-va"
|
||||
|
||||
# Specs that exist but are held out of CI, each with the reason. Listed here so
|
||||
# the coverage check below can tell a deliberate exclusion from a forgotten file.
|
||||
EXCLUDED="metanav download"
|
||||
|
||||
case "${1:-}" in
|
||||
all)
|
||||
# comma separated, the form `cypress run --spec` takes
|
||||
for name in $SPECS; do printf 'cypress/e2e/%s.cy.ts,' "$name"; done | sed 's/,$//'
|
||||
;;
|
||||
coverage)
|
||||
on_disk=$(cd "$2/client" && ls cypress/e2e/*.cy.ts | xargs -n1 basename | sed 's/\.cy\.ts$//' | sort)
|
||||
listed=$(printf '%s\n' $SPECS $EXCLUDED | sort)
|
||||
missing=$(comm -23 <(echo "$on_disk") <(echo "$listed"))
|
||||
if [ -n "$missing" ]; then
|
||||
echo "Specs that are neither run nor explicitly excluded:"
|
||||
echo "$missing"
|
||||
echo "Add them to SPECS (or EXCLUDED, with a reason) in .gitea/scripts/cypress-specs.sh"
|
||||
exit 1
|
||||
fi
|
||||
echo "All $(echo "$on_disk" | wc -l) specs are accounted for: $(echo $SPECS | wc -w) run, $(echo $EXCLUDED | wc -w) excluded."
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 {all|coverage <repo-root>}" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
@@ -75,11 +75,33 @@ jobs:
|
||||
cd client
|
||||
npm run test:headless
|
||||
|
||||
- name: Angular Tests in other timezones
|
||||
# The VA filter clauses are built from local calendar dates, so a
|
||||
# UTC-only run cannot see a date that slips a day west of UTC or east of
|
||||
# it. Only the VA specs carry that property, so both scripts run that
|
||||
# subset rather than the whole suite a second and third time.
|
||||
run: |
|
||||
cd client
|
||||
npm run test:tz-west
|
||||
npm run test:tz-east
|
||||
|
||||
- name: Every Cypress spec is in the CI list
|
||||
# The spec list used to be spelled out inline in --spec, and five files
|
||||
# quietly fell out of it - among them the VA embed suite. Fails when a
|
||||
# spec on disk is not listed.
|
||||
run: bash .gitea/scripts/cypress-specs.sh coverage .
|
||||
|
||||
- name: Production Build
|
||||
run: |
|
||||
cd client
|
||||
npm run build
|
||||
|
||||
|
||||
# Cypress shard 1 of 4. The setup repeats in every shard because each
|
||||
# one gets its own runner and its own SASjs server; that fixed cost is what
|
||||
# limits how far sharding helps. The specs it runs come from
|
||||
# .gitea/scripts/cypress-specs.sh.
|
||||
|
||||
Build-and-test-development:
|
||||
runs-on: ubuntu-latest
|
||||
needs: Build-and-ng-test
|
||||
@@ -182,9 +204,20 @@ jobs:
|
||||
sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
|
||||
|
||||
- name: Prepare and run frontend and cypress
|
||||
# One job, one SASjs server, one ng serve, one Chrome: four of each on one
|
||||
# runner starved the app under test and every failure in that run was a
|
||||
# timeout, not an assertion.
|
||||
timeout-minutes: 50
|
||||
run: |
|
||||
cd ./client
|
||||
# The spec list lives in one place, and an empty list is a broken
|
||||
# workflow rather than a green run with no tests.
|
||||
SPECS="$(bash ../.gitea/scripts/cypress-specs.sh all)"
|
||||
if [ -z "$SPECS" ]; then
|
||||
echo "the spec list resolved to nothing"
|
||||
exit 1
|
||||
fi
|
||||
echo "specs: $SPECS"
|
||||
mv ./cypress.env.example.json ./cypress.env.json
|
||||
replace-in-files --regex='"username".*' --replacement='"username":"'${{ secrets.CYPRESS_USERNAME_SASJS }}'",' ./cypress.env.json
|
||||
replace-in-files --regex='"password".*' --replacement='"password":"'${{ secrets.CYPRESS_PWD_SASJS }}'" ' ./cypress.env.json
|
||||
@@ -199,16 +232,22 @@ jobs:
|
||||
# Start frontend and run cypress
|
||||
# timeout 2700: SIGTERM after 45 min so Cypress can flush video/screenshots
|
||||
# before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
|
||||
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/hook-programs.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/filter-panel.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts,cypress/e2e/browser-info.cy.ts,cypress/e2e/ddl-export.cy.ts,cypress/e2e/row-cell-limits.cy.ts,cypress/e2e/derived-variables.cy.ts,cypress/e2e/startupservice-launcher.cy.ts"
|
||||
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --spec "$SPECS"
|
||||
|
||||
- name: Zip Cypress videos
|
||||
if: always()
|
||||
# Only on a failure: a green run has nothing to watch, and zipping and
|
||||
# uploading every run's videos is the part of the video cost that lands
|
||||
# on the clock.
|
||||
if: failure()
|
||||
run: |
|
||||
mkdir -p ./client/cypress/videos
|
||||
zip -r cypress-videos ./client/cypress/videos
|
||||
|
||||
- name: Add cypress videos artifacts
|
||||
if: always()
|
||||
if: failure()
|
||||
# v3, not v4: this runner reports "GHESNotSupportedError:
|
||||
# @actions/artifact v2.0.0+, upload-artifact@v4+ ... not supported on
|
||||
# GHES", so the current major cannot upload here at all.
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: cypress-videos.zip
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
name: Publish code documentation
|
||||
run-name: Publishing typedoc and SAS docs
|
||||
|
||||
# The code documentation sites are published here rather than inline in the
|
||||
# release job, so that they can be republished on their own - after a secret
|
||||
# rotation, or for a release whose artefact upload failed (the release job
|
||||
# cannot simply be re-run: semantic-release refuses once the tag exists).
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
NODE_VERSION: '24.5.0'
|
||||
|
||||
jobs:
|
||||
typedoc:
|
||||
runs-on: ubuntu-latest
|
||||
# One deploy per site at a time: `surfer deploy` swaps the site atomically,
|
||||
# so two runs racing is not corruption, but it is a wasted build and the
|
||||
# last swap wins.
|
||||
concurrency:
|
||||
group: surfer-webdoc
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: Write .npmrc file
|
||||
run: |
|
||||
printf '%s\n' "$NPMRC" >> client/.npmrc
|
||||
echo "legacy-peer-deps=true" >> client/.npmrc
|
||||
shell: bash
|
||||
env:
|
||||
NPMRC: ${{ secrets.NPMRC }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
cd client
|
||||
# Decrypt and Install sheet
|
||||
echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
|
||||
npm ci
|
||||
shell: bash
|
||||
env:
|
||||
SHEET_PWD: ${{ secrets.SHEET_PWD }}
|
||||
|
||||
- name: Build typedoc
|
||||
run: |
|
||||
cd client
|
||||
npm run compodoc:build
|
||||
|
||||
# Deploy tooling is pinned to an exact version: an unpinned `latest` would
|
||||
# let a breaking or compromised upstream release publish straight to the
|
||||
# live site. `deploy` (v7) replaces the site atomically, and it takes the
|
||||
# Cloudron username plus an app password - the v6 `put --token` flow this
|
||||
# replaces is gone.
|
||||
#
|
||||
# webdoc is a NAMED site on the app whose default site is the marketing
|
||||
# site, so the name and its alias domain are both passed. Surfer picks
|
||||
# the target only from `--site-name` (it travels as the Surfer-Deployment
|
||||
# header); without it `deploy` replaces the app's DEFAULT site, which is
|
||||
# what serves datacontroller.io - so an unnamed deploy published the
|
||||
# Typedoc over the top of the marketing site instead of to webdoc.
|
||||
- name: Deploy typedoc
|
||||
env:
|
||||
SURFER_PASSWORD: ${{ secrets.WEBDOC_DATACONTROLLER_IO }}
|
||||
REF_NAME: ${{ gitea.ref_name }}
|
||||
COMMIT_SHA: ${{ gitea.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
npx --yes @cloudron/surfer@7.2.0 deploy client/documentation \
|
||||
--server webdoc.datacontroller.io \
|
||||
--site-name webdoc \
|
||||
--domain webdoc.datacontroller.io \
|
||||
--username allan \
|
||||
--password "$SURFER_PASSWORD" \
|
||||
--message "Typedoc from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
|
||||
|
||||
sas-docs:
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: surfer-code-datacontroller
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
# `sasjs doc` shells out to doxygen
|
||||
apt-get update
|
||||
apt-get install -y doxygen
|
||||
cd sas
|
||||
npm ci
|
||||
CLI_VERSION=$(node -p "require('./package.json').dependencies['@sasjs/cli']")
|
||||
npm i -g "@sasjs/cli@${CLI_VERSION}"
|
||||
|
||||
- name: Build SAS docs
|
||||
run: |
|
||||
cd sas
|
||||
sasjs doc
|
||||
|
||||
# No `--site-name` here on purpose: code.datacontroller.io is its own app
|
||||
# and the SAS docs are its DEFAULT site, so naming one would create a
|
||||
# separate site and leave the served copy stale. Do not copy the named
|
||||
# form from the typedoc job above - that app's default site is the
|
||||
# marketing site, which is why it must not be the target.
|
||||
- name: Deploy SAS docs
|
||||
env:
|
||||
SURFER_PASSWORD: ${{ secrets.CODE_DATACONTROLLER_IO }}
|
||||
REF_NAME: ${{ gitea.ref_name }}
|
||||
COMMIT_SHA: ${{ gitea.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
npx --yes @cloudron/surfer@7.2.0 deploy sas/sasjsbuild/sasdocs \
|
||||
--server code.datacontroller.io \
|
||||
--username allan \
|
||||
--password "$SURFER_PASSWORD" \
|
||||
--message "SAS docs from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
|
||||
@@ -0,0 +1,209 @@
|
||||
name: Release assets
|
||||
run-name: Building and uploading the release assets
|
||||
|
||||
# Builds the downloadable artefacts for an existing release and uploads them.
|
||||
#
|
||||
# The release job cuts the tag before it builds, so a failure anywhere between
|
||||
# the semantic-release step and the asset upload leaves a published release with
|
||||
# no files - and that job cannot simply be re-run, because semantic-release
|
||||
# refuses once the tag exists. This workflow rebuilds the artefacts from the tag
|
||||
# and attaches them, so the release can be completed without cutting a new one.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tag:
|
||||
description: 'Release tag to build and upload (blank = the latest release)'
|
||||
required: false
|
||||
default: ''
|
||||
|
||||
env:
|
||||
NODE_VERSION: '24.5.0'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
assets:
|
||||
runs-on: ubuntu-latest
|
||||
# Two runs uploading the same asset names would race the delete-then-upload
|
||||
# below.
|
||||
concurrency:
|
||||
group: release-assets
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: Write .npmrc file
|
||||
run: |
|
||||
printf '%s\n' "$NPMRC" >> client/.npmrc
|
||||
echo "legacy-peer-deps=true" >> client/.npmrc
|
||||
shell: bash
|
||||
env:
|
||||
NPMRC: ${{ secrets.NPMRC }}
|
||||
|
||||
- name: Install system packages
|
||||
# jq parses the release JSON below, so it is installed before the tag
|
||||
# resolve.
|
||||
run: |
|
||||
apt-get update
|
||||
apt-get install -y zip jq
|
||||
|
||||
- name: Resolve the release tag
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
REQUESTED_TAG: ${{ inputs.tag }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
|
||||
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
|
||||
|
||||
if [ -n "${REQUESTED_TAG}" ]; then
|
||||
TAG="${REQUESTED_TAG}"
|
||||
# Fail here rather than after a ten-minute build if the tag is not
|
||||
# cut yet or has no release.
|
||||
curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${TAG}" > /dev/null
|
||||
else
|
||||
TAG=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest" | jq -r '.tag_name')
|
||||
fi
|
||||
echo "Building assets for ${TAG}"
|
||||
echo "TAG=${TAG}" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Check out the tag
|
||||
run: |
|
||||
set -euo pipefail
|
||||
git fetch --force --tags origin "+refs/tags/${TAG}:refs/tags/${TAG}"
|
||||
git checkout --force "refs/tags/${TAG}"
|
||||
|
||||
- name: Install the SASjs CLI the tag pins
|
||||
run: |
|
||||
cd sas
|
||||
CLI_VERSION=$(node -p "require('./package.json').dependencies['@sasjs/cli']")
|
||||
npm i -g "@sasjs/cli@${CLI_VERSION}"
|
||||
|
||||
- name: Frontend build
|
||||
run: |
|
||||
cd client
|
||||
# Decrypt and Install sheet
|
||||
echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
|
||||
npm ci
|
||||
npm i webpack
|
||||
npm run build
|
||||
shell: bash
|
||||
env:
|
||||
SHEET_PWD: ${{ secrets.SHEET_PWD }}
|
||||
|
||||
- name: Build SAS9 EBI Release
|
||||
run: |
|
||||
cd sas
|
||||
npm i
|
||||
sasjs c -t sas9
|
||||
rm -rf sasjsbuild/tests
|
||||
sasjs b -t sas9
|
||||
cp sasjsbuild/mysas9deploy.sas ./demostream_sas9.sas
|
||||
# remove streamed component and rebuild SAS 9 services
|
||||
rm -rf sasjsbuild/services/web9
|
||||
rm sasjsbuild/services/clickme.sas
|
||||
sasjs b -t sas9
|
||||
cp sasjsbuild/mysas9deploy.sas ./sas9.sas
|
||||
|
||||
- name: Build SASjs Server Release
|
||||
run: |
|
||||
cd sas
|
||||
cp sasjs/utils/favicon.ico ../client/dist/favicon.ico
|
||||
sasjs c -t server
|
||||
rm -rf sasjsbuild/tests
|
||||
server_apploc="/Public/app/dc"
|
||||
sed -i "s|apploc=\"[^\"]*\"|apploc=\"${server_apploc}\"|g" sasjsbuild/services/web/index.html
|
||||
sasjs b -t server
|
||||
cp sasjsbuild/server.json.zip ./sasjs_server.json.zip
|
||||
|
||||
- name: Build Viya Release
|
||||
run: |
|
||||
cd sas
|
||||
sasjs c -t viya
|
||||
rm -rf sasjsbuild/tests
|
||||
sed -i -e 's/servertype="SASJS"/servertype="SASVIYA"/g' sasjsbuild/services/DC.html
|
||||
sasjs b -t viya
|
||||
cp sasjsbuild/viya.sas ./viya.sas
|
||||
# compile Viya Full deploy (without web)
|
||||
rm -rf sasjsbuild/services/web
|
||||
rm sasjsbuild/services/DC.html
|
||||
sasjs b -t viya
|
||||
cp sasjsbuild/viya.sas ./viya_noweb.sas
|
||||
cp sasjsbuild/viya.json ./viya_noweb.json
|
||||
|
||||
- name: Zip Frontend (including viya.json for full viya deploy)
|
||||
run: |
|
||||
cd sas
|
||||
cp sasjsbuild/viya.json ../client/dist/viya.json
|
||||
# Zip from *inside* dist so the archive holds the frontend files at its
|
||||
# root rather than under a client/dist/ prefix (see #147).
|
||||
cd ../client/dist
|
||||
zip -r ../../frontend.zip .
|
||||
|
||||
- name: Upload assets to the release
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
|
||||
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
|
||||
|
||||
RELEASE_JSON=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${TAG}")
|
||||
RELEASE_ID=$(echo "$RELEASE_JSON" | jq -r '.id')
|
||||
RELEASE_BODY=$(echo "$RELEASE_JSON" | jq -r '.body')
|
||||
|
||||
ASSETS=(frontend.zip
|
||||
sas/demostream_sas9.sas
|
||||
sas/viya.sas
|
||||
sas/sasjs_server.json.zip
|
||||
sas/sas9.sas
|
||||
sas/viya_noweb.sas
|
||||
sas/viya_noweb.json)
|
||||
|
||||
: > SHA256SUMS
|
||||
for f in "${ASSETS[@]}"; do
|
||||
sum=$(sha256sum "$f" | cut -d' ' -f1)
|
||||
printf '%s %s\n' "$sum" "${f##*/}" >> SHA256SUMS
|
||||
done
|
||||
cat SHA256SUMS
|
||||
|
||||
# Replacing the assets means removing the ones this workflow uploads,
|
||||
# and a second run must not append the verification section twice. An
|
||||
# asset the release carries from another source is left alone.
|
||||
OWNED=$(printf '%s\n' "${ASSETS[@]##*/}" SHA256SUMS)
|
||||
echo "$RELEASE_JSON" | jq -r '.assets[]? | "\(.id) \(.name)"' |
|
||||
while read -r asset_id asset_name; do
|
||||
if printf '%s\n' "$OWNED" | grep -qxF -- "$asset_name"; then
|
||||
echo "Removing previous ${asset_name}"
|
||||
curl --fail-with-body -sS -X DELETE -H "$AUTH_HEADER" "$BASE/releases/${RELEASE_ID}/assets/${asset_id}"
|
||||
else
|
||||
echo "Leaving ${asset_name} in place: this workflow did not upload it"
|
||||
fi
|
||||
done
|
||||
|
||||
if echo "$RELEASE_BODY" | grep -q "## Verifying the download"; then
|
||||
echo "Release body already carries the verification section"
|
||||
else
|
||||
VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
|
||||
NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
|
||||
'$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
|
||||
curl --fail-with-body -sS -X PATCH \
|
||||
-H "$AUTH_HEADER" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data "$(jq -n --arg body "$NEW_BODY" '{draft:false, body:$body}')" \
|
||||
"$BASE/releases/$RELEASE_ID"
|
||||
fi
|
||||
|
||||
URL="$BASE/releases/${RELEASE_ID}/assets"
|
||||
for f in "${ASSETS[@]}"; do
|
||||
echo "Uploading $f ..."
|
||||
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
|
||||
done
|
||||
echo "Uploading SHA256SUMS ..."
|
||||
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
|
||||
@@ -281,6 +281,11 @@ jobs:
|
||||
|
||||
semantic-release
|
||||
|
||||
# The tag this job cuts, for the steps that publish against it: a
|
||||
# `releases/latest` lookup would pick whichever release landed last
|
||||
# if two overlap.
|
||||
echo "RELEASE_TAG=v$(node -p "require('./package.json').version")" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Frontend Build
|
||||
description: Must be created AFTER the release as the version (git tag) is used in the interface
|
||||
run: |
|
||||
@@ -341,19 +346,6 @@ jobs:
|
||||
cd ../client/dist
|
||||
zip -r ../../frontend.zip .
|
||||
|
||||
- name: Release Typedoc
|
||||
run: |
|
||||
cd client
|
||||
npm -g install cloudron-surfer
|
||||
npm run compodoc:build
|
||||
surfer put --token ${{ secrets.TSDOC_TOKEN }} --server webdoc.datacontroller.io documentation/* /
|
||||
|
||||
- name: Release code.datacontroller.io
|
||||
run: |
|
||||
cd sas
|
||||
sasjs doc
|
||||
surfer put --token ${{ secrets.CODE_DATACONTROLLER_IO }} --server code.datacontroller.io sasjsbuild/sasdocs/* /
|
||||
|
||||
- name: Upload assets to release
|
||||
env:
|
||||
# Use the same ephemeral per-job Gitea Actions token that the
|
||||
@@ -368,7 +360,7 @@ jobs:
|
||||
AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
|
||||
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
|
||||
|
||||
RELEASE_JSON=$(curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest")
|
||||
RELEASE_JSON=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${RELEASE_TAG}")
|
||||
RELEASE_ID=$(echo "$RELEASE_JSON" | jq -r '.id')
|
||||
RELEASE_BODY=$(echo "$RELEASE_JSON" | jq -r '.body')
|
||||
|
||||
@@ -402,7 +394,7 @@ jobs:
|
||||
VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
|
||||
NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
|
||||
'$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
|
||||
curl -k --fail-with-body -sS -X PATCH \
|
||||
curl --fail-with-body -sS -X PATCH \
|
||||
-H "$AUTH_HEADER" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data "$(jq -n --arg body "$NEW_BODY" '{draft:false, body:$body}')" \
|
||||
@@ -412,10 +404,37 @@ jobs:
|
||||
URL="$BASE/releases/$RELEASE_ID/assets"
|
||||
for f in "${ASSETS[@]}"; do
|
||||
echo "Uploading $f ..."
|
||||
curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
|
||||
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
|
||||
done
|
||||
echo "Uploading SHA256SUMS ..."
|
||||
curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
|
||||
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
|
||||
|
||||
- name: Publish the code documentation
|
||||
# Typedoc (webdoc) and the SAS docs (code) are published by docs.yaml,
|
||||
# not inline here. Two reasons: the v6 `surfer put --token` flow is gone
|
||||
# in Surfer 7, which takes a Cloudron username plus an app password; and
|
||||
# a dispatch means a stale or failed docs deploy can be republished
|
||||
# without re-running this job, which semantic-release refuses once the
|
||||
# tag exists.
|
||||
#
|
||||
# This runs AFTER the release assets are uploaded: the documentation is
|
||||
# a side channel, and a docs problem must not cost a cut release its
|
||||
# artefacts.
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
|
||||
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
|
||||
|
||||
TAG="${RELEASE_TAG}"
|
||||
echo "Dispatching code documentation for ${TAG}"
|
||||
|
||||
curl --fail-with-body -sS -X POST \
|
||||
-H "$AUTH_HEADER" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
|
||||
"$BASE/actions/workflows/docs.yaml/dispatches"
|
||||
|
||||
- name: Publish the demo site for this release
|
||||
# The tag is created by this job's semantic-release step, so it carries
|
||||
@@ -430,12 +449,12 @@ jobs:
|
||||
AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
|
||||
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
|
||||
|
||||
TAG=$(curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest" | jq -r '.tag_name')
|
||||
TAG="${RELEASE_TAG}"
|
||||
echo "Dispatching Sites for ${TAG}"
|
||||
|
||||
# the dispatch endpoint resolves a bare name as a branch; a tag needs
|
||||
# the refs/tags/ prefix
|
||||
curl -k --fail-with-body -sS -X POST \
|
||||
curl --fail-with-body -sS -X POST \
|
||||
-H "$AUTH_HEADER" \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
|
||||
|
||||
@@ -43,7 +43,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
|
||||
|
||||
Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
|
||||
|
||||
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
|
||||
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
|
||||
|
||||
## Code comments and test names
|
||||
|
||||
|
||||
@@ -1,3 +1,55 @@
|
||||
# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
|
||||
* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
|
||||
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
|
||||
|
||||
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **ci:** install doxygen for the SAS docs job ([f58925b](https://git.datacontroller.io/dc/dc/commit/f58925b29956f1f753f0a367fdcd0afe7f3f51a0))
|
||||
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](https://git.datacontroller.io/dc/dc/commit/27686d3aad5e34ace59ca1b735a17cca1a4025e1))
|
||||
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](https://git.datacontroller.io/dc/dc/commit/888873421c3aeaf312d5ec45e3c805317c66976a))
|
||||
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](https://git.datacontroller.io/dc/dc/commit/dd482c4069202a0c6b9983129c19efc09bd160a7))
|
||||
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](https://git.datacontroller.io/dc/dc/commit/25ffa71adddba5e334f4a8c166f284d1be498fcd))
|
||||
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](https://git.datacontroller.io/dc/dc/commit/c5808a16eba0d29562fcc76f13ac4a33ab8da0b7))
|
||||
* **ci:** rebuild and upload release assets on demand ([6486c80](https://git.datacontroller.io/dc/dc/commit/6486c80ed4f368bbf116c0ecdecc0e5e5f02075f))
|
||||
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](https://git.datacontroller.io/dc/dc/commit/af1813f829861df41c89b512c824e8d3ebac7a4b))
|
||||
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](https://git.datacontroller.io/dc/dc/commit/cf7b21a8b61ec3bf7ebe21968ccf13ade264bc64))
|
||||
* **cypress:** hold download out too, and retry once in CI ([c97e886](https://git.datacontroller.io/dc/dc/commit/c97e8866db9ee404b2118d24baf53fc7058debef)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395) [#398](https://git.datacontroller.io/dc/dc/issues/398)
|
||||
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](https://git.datacontroller.io/dc/dc/commit/c803773d3c36476d8263f01fc3b8344aa3503d44)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395)
|
||||
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](https://git.datacontroller.io/dc/dc/commit/80e8911af8857fa2c417f028cd59b64a87233b9e))
|
||||
* **deploy:** keep the app query when opening the standalone URL ([e51953c](https://git.datacontroller.io/dc/dc/commit/e51953ca8a6734e1649bad68cf4ad2057d677334))
|
||||
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](https://git.datacontroller.io/dc/dc/commit/459ff2da371375e996839d69deb40f681c356f1d)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
|
||||
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](https://git.datacontroller.io/dc/dc/commit/f7e753863b61d8545bd0777d80267661b3c58670))
|
||||
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](https://git.datacontroller.io/dc/dc/commit/7e3045a21b67053b76a81bfe836e31571ccbffbf))
|
||||
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](https://git.datacontroller.io/dc/dc/commit/2623a3fb4ba3bf19c00636b1717b53d5a2d54f3a))
|
||||
* **embed:** log a parse-time mapping problem once ([0011f7e](https://git.datacontroller.io/dc/dc/commit/0011f7e933d85be8e33debe3a6ecb9000edaabe3))
|
||||
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](https://git.datacontroller.io/dc/dc/commit/541a61ba08bc7256a911c50de7078859cc46c84e))
|
||||
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](https://git.datacontroller.io/dc/dc/commit/e9216b08956d5599662259a2b05cc1524c458df5))
|
||||
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](https://git.datacontroller.io/dc/dc/commit/bd3c220af2678a2a3f8394fe3084ee03c23d0b15))
|
||||
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](https://git.datacontroller.io/dc/dc/commit/f83dc359e258f5d1cb9fc49f0eaab9748995d823))
|
||||
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](https://git.datacontroller.io/dc/dc/commit/aa91daf9f78c00cea486d0f885f8e634366bd1a0))
|
||||
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](https://git.datacontroller.io/dc/dc/commit/10157c11595685efdc206fe7b16a5cafa098f26d))
|
||||
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](https://git.datacontroller.io/dc/dc/commit/411f6c3354b7fbfec11fef3b174ae1f8066c92ba))
|
||||
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](https://git.datacontroller.io/dc/dc/commit/c0ea810766ebfda5ae5358ab14146fbcc07fcc8e))
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](https://git.datacontroller.io/dc/dc/commit/15a1f34d5263503cac83b4fe4e52482777dc2f1f))
|
||||
* **embed:** resolve va_parmap values through named date anchors ([88010c3](https://git.datacontroller.io/dc/dc/commit/88010c3980c2ea1c39bbb024ed32d63e495e61f0))
|
||||
|
||||
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output
|
||||
|
||||
- `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
|
||||
- `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
|
||||
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
|
||||
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).
|
||||
|
||||
## Roles
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import fs from 'fs'
|
||||
import { defineConfig } from 'cypress'
|
||||
|
||||
export default defineConfig({
|
||||
@@ -34,7 +35,28 @@ export default defineConfig({
|
||||
|
||||
e2e: {
|
||||
video: true,
|
||||
// One retry in a CI run, none interactively. `editor.cy.ts` test 28
|
||||
// intermittently fails on "app-loading ... continuously found" - a slow load,
|
||||
// not an assertion - and a single retry costs a few seconds where a red
|
||||
// pipeline costs forty minutes. Failures that are real still fail twice, and
|
||||
// the retried attempt's output is in the run log.
|
||||
retries: { runMode: 1, openMode: 0 },
|
||||
setupNodeEvents(on, config) {
|
||||
// `download.cy.ts` clears the downloads folder before each test through
|
||||
// `cy.task('deleteFolder')`. That task was registered in
|
||||
// cypress/plugins/index.js, which Cypress 10+ never loads unless it is
|
||||
// required from here - so the task had no handler and the spec failed on
|
||||
// its first hook.
|
||||
on('task', {
|
||||
deleteFolder(folderName: string) {
|
||||
return new Promise((resolve, reject) => {
|
||||
fs.rm(folderName, { recursive: true, force: true }, (err) =>
|
||||
err ? reject(err) : resolve(null)
|
||||
)
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
// Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
|
||||
// render deterministically regardless of the runner's system locale.
|
||||
on('before:browser:launch', (browser, launchOptions) => {
|
||||
|
||||
@@ -17,9 +17,20 @@ context('download files test: ', function () {
|
||||
|
||||
this.beforeEach(() => {
|
||||
cy.visit(hostUrl + appLocation)
|
||||
|
||||
// beforeAll already logged in. Typing into the login form unconditionally
|
||||
// failed with "the center of this element is hidden from view" whenever the
|
||||
// session was live and the form was not on screen, so only fill it when it
|
||||
// is actually shown.
|
||||
cy.get('body').then(($body) => {
|
||||
const usernameInput = $body.find('input.username')[0]
|
||||
|
||||
if (usernameInput && !Cypress.dom.isHidden(usernameInput)) {
|
||||
cy.get('input.username').type(username)
|
||||
cy.get('input.password').type(password)
|
||||
cy.get('.login-group button').click()
|
||||
}
|
||||
})
|
||||
|
||||
visitPage('home')
|
||||
})
|
||||
|
||||
@@ -6,44 +6,90 @@ const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
|
||||
|
||||
// SAS Visual Analytics data-driven content embed mode (`?embed=va`):
|
||||
// - chrome is hidden (same as embed=true),
|
||||
// - the editor opens in edit mode immediately (not read-only),
|
||||
// - the only action button is a single Submit below the grid (stubbed in v1),
|
||||
// - the top button bar and Add Record are hidden.
|
||||
// - the editor opens READ-ONLY (the user clicks Edit, as in normal use),
|
||||
// - the Filter button is hidden - VA filters replace the filter builder,
|
||||
// - the Add Row / Add Record buttons are hidden.
|
||||
//
|
||||
// The detailed PK+label data-merge logic is covered by the VaMessagingService
|
||||
// unit spec; here we only assert the deterministic UI layout and that pushing a
|
||||
// VA postMessage does not break the page.
|
||||
// unit spec; here we assert the deterministic UI layout, that pushing a VA
|
||||
// postMessage does not break the page, and (case 3) that a `va_parmap` URL
|
||||
// mapping filters the column it names end to end.
|
||||
/**
|
||||
* Assert on the row count the editor shows. The assertion runs inside a Cy
|
||||
* `should`, so it retries until the grid has loaded: reading the text once would
|
||||
* capture the count from before a filter is applied.
|
||||
*/
|
||||
const parseRows = (text: string): number => {
|
||||
const match = /\(([\d,]+) rows?/.exec(text)
|
||||
return match ? Number(match[1].replace(/,/g, '')) : NaN
|
||||
}
|
||||
|
||||
const expectRows = (check: (rows: number) => void) =>
|
||||
cy.get('h4', { timeout: longerCommandTimeout }).should(($h4) => {
|
||||
check(parseRows($h4.text()))
|
||||
})
|
||||
|
||||
/** Assert the grid shrank, without pinning a fixture-specific number. */
|
||||
const expectFewerRowsThan = (baseline: number) =>
|
||||
expectRows((rows) => {
|
||||
expect(rows, 'filtered row count').to.be.lessThan(baseline)
|
||||
})
|
||||
|
||||
/** Read the unfiltered row count from the app itself, retrying until it loads. */
|
||||
const readRowCount = (): Cypress.Chainable<number> =>
|
||||
expectRows((rows) => {
|
||||
expect(rows, 'row count').to.be.greaterThan(0)
|
||||
})
|
||||
.get('h4')
|
||||
.invoke('text')
|
||||
.then(parseRows)
|
||||
|
||||
context('embed=va (VA data-driven content) tests: ', function () {
|
||||
// How many rows the table has unfiltered. Captured once, from the app itself.
|
||||
let unfilteredRows = 0
|
||||
|
||||
this.beforeAll(() => {
|
||||
cy.visit(`${hostUrl}/SASLogon/logout`)
|
||||
cy.loginAndUpdateValidKey()
|
||||
// Baseline for the relative row-count assertions: open the table with no VA
|
||||
// message and record what the editor itself reports.
|
||||
cy.visit(`${hostUrl}${appLocation}/#/editor/DC_JSLIB.MPE_X_TEST`, {
|
||||
timeout: 60000
|
||||
})
|
||||
readRowCount().then((count) => {
|
||||
unfilteredRows = count
|
||||
})
|
||||
})
|
||||
|
||||
this.beforeEach(() => {
|
||||
cy.visit(hostUrl + appLocation)
|
||||
// A cy.visit that differs from the current URL only in the hash is a
|
||||
// same-document navigation, so the app is not re-bootstrapped. Reload so
|
||||
// each test starts from a fresh document (the previous test may have left
|
||||
// ?embed=va in the URL, which the app reads once at init).
|
||||
cy.reload()
|
||||
visitPage('home')
|
||||
})
|
||||
|
||||
it('1 | opens the editor in VA mode: chrome hidden, editable, single bottom Submit', () => {
|
||||
it('1 | opens the editor in VA mode: chrome hidden, read-only, no Filter button', () => {
|
||||
openEditorInVaMode('mpe_x_test', () => {
|
||||
// Chrome hidden (same as embed=true)
|
||||
cy.get('header.app-header').should('not.exist')
|
||||
|
||||
// Grid present and in edit mode (read-only Filter/Edit/Upload bar absent)
|
||||
// Grid present, read-only: the user enters edit mode with Edit, as usual
|
||||
cy.get('#hotTable', { timeout: longerCommandTimeout }).should('exist')
|
||||
cy.get('.btnCtrl button').contains('Edit').should('not.exist')
|
||||
cy.get('.btnCtrl button').contains('Edit').should('exist')
|
||||
|
||||
// The filter builder is replaced by VA filters
|
||||
cy.get('.btnCtrl button').contains('Filter').should('not.exist')
|
||||
|
||||
// Top edit-mode buttons hidden in VA mode
|
||||
// Not in edit mode, so no edit-mode controls
|
||||
cy.get('.btnCtrl button.btn-outline-danger').should('not.exist') // Cancel
|
||||
cy.get('.btnCtrl').contains('Add Row').should('not.exist')
|
||||
cy.get('.btnCtrl').contains('Submit').should('not.exist')
|
||||
|
||||
// Bottom Add Record hidden
|
||||
// Bottom Add Record hidden in VA mode
|
||||
cy.contains('button', 'Add Record').should('not.exist')
|
||||
|
||||
// Exactly one Submit button, below the grid, disabled in v1
|
||||
cy.get('button').filter(':contains("Submit")').should('have.length', 1)
|
||||
cy.get('button').contains('Submit').should('be.disabled')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -69,14 +115,382 @@ context('embed=va (VA data-driven content) tests: ', function () {
|
||||
// Editor still loads in VA mode after the message (chrome hidden, grid up)
|
||||
cy.get('#hotTable', { timeout: longerCommandTimeout }).should('exist')
|
||||
cy.get('header.app-header').should('not.exist')
|
||||
cy.get('button').contains('Submit').should('exist')
|
||||
cy.get('.btnCtrl button').contains('Edit').should('exist')
|
||||
})
|
||||
})
|
||||
it('3 | va_parmap filters a column from a parameter that is not its name (tab 1)', () => {
|
||||
// Two DC instances over one table (one per report tab), each filtering the
|
||||
// same DC column from its own report parameter. Without the URL mapping the
|
||||
// parameter label is not a column name, so nothing would filter at all. The
|
||||
// mapping here is written in lower case against an upper-case label and
|
||||
// column, which pins the case-insensitivity of both sides end to end.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
})
|
||||
// The applied filter names the MAPPED column, and the grid shrinks to
|
||||
// the rows matching that value (Option 2 occurs twice in this table).
|
||||
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'SOME_DROPDOWN'
|
||||
)
|
||||
cy.get('.infoBar-text').should('contain.text', "'Option 2'")
|
||||
expectFewerRowsThan(unfilteredRows)
|
||||
},
|
||||
'va_parmap=dropdown_tab1-some_dropdown'
|
||||
)
|
||||
})
|
||||
|
||||
it('4 | the other tab maps its own parameter to the same column', () => {
|
||||
// Same table, same column, a different parameter label and value (Option 1
|
||||
// occurs four times) - the per-instance URL mapping is what keeps the two
|
||||
// tabs independent.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB2',
|
||||
dataType: 'string',
|
||||
value: 'Option 1'
|
||||
})
|
||||
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'SOME_DROPDOWN'
|
||||
)
|
||||
cy.get('.infoBar-text').should('contain.text', "'Option 1'")
|
||||
expectFewerRowsThan(unfilteredRows)
|
||||
},
|
||||
'va_parmap=DROPDOWN_TAB2-SOME_DROPDOWN'
|
||||
)
|
||||
})
|
||||
|
||||
it('5 | warns in the grid when a mapping names a column the table lacks', () => {
|
||||
// Written by hand in the report's URL, so a typo must be visible to the
|
||||
// report author: the parameter is skipped, and the reason is shown rather
|
||||
// than leaving an unfiltered grid that looks like a working filter.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
})
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'VA parameter mapping problem'
|
||||
)
|
||||
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
|
||||
cy.get('.va-parmap-warning').should('contain.text', 'NOT_A_COLUMN')
|
||||
// VA mode hides the app header, so the banner must paint inside the
|
||||
// viewport on its own - assert the real box, not just its presence.
|
||||
cy.get('.va-parmap-warning').should(($alert) => {
|
||||
const rect = $alert[0].getBoundingClientRect()
|
||||
expect(rect.width, 'banner width').to.be.greaterThan(100)
|
||||
expect(rect.height, 'banner height').to.be.greaterThan(20)
|
||||
expect(rect.top, 'banner top').to.be.at.least(0)
|
||||
expect(rect.bottom, 'banner bottom').to.be.at.most(
|
||||
Cypress.config('viewportHeight')
|
||||
)
|
||||
})
|
||||
cy.screenshot('va-parmap-banner', { capture: 'viewport' })
|
||||
// Nothing was filtered: the row count matches the unfiltered baseline.
|
||||
expectRows((rows) => expect(rows).to.equal(unfilteredRows))
|
||||
},
|
||||
'va_parmap=DROPDOWN_TAB1-NOT_A_COLUMN'
|
||||
)
|
||||
})
|
||||
|
||||
it('6 | warns straight away when a mapping entry cannot be parsed', () => {
|
||||
// No separator, so the entry never becomes a mapping - reported before any
|
||||
// VA message arrives.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'it needs a "-" between the VA parameter and the column name'
|
||||
)
|
||||
},
|
||||
'va_parmap=DROPDOWN_TAB1'
|
||||
)
|
||||
})
|
||||
|
||||
it('7 | warns when a mapping names a parameter VA is not sending', () => {
|
||||
// Parameter-side typo: the mapping never applies, so without the banner this
|
||||
// is indistinguishable from a working filter over every row.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
})
|
||||
// One message only tells us this object's parameter set is still being
|
||||
// discovered; the warning waits for a message that introduces no new name.
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'not.exist'
|
||||
)
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
})
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'names VA parameter "dropdown_tab_1", which VA is not sending'
|
||||
)
|
||||
expectRows((rows) => expect(rows).to.equal(unfilteredRows))
|
||||
},
|
||||
'va_parmap=DROPDOWN_TAB_1-SOME_DROPDOWN'
|
||||
)
|
||||
})
|
||||
|
||||
it('8 | applies a mapping from the buffered message on the very first load', () => {
|
||||
// VA posts once on iframe load, before the app bootstraps, so the first
|
||||
// getdata is already filtered from that buffered message - with no column
|
||||
// metadata available, the mapping is what names the column.
|
||||
visitWithBufferedMessage(
|
||||
'va_parmap=DROPDOWN_TAB1-SOME_DROPDOWN',
|
||||
{
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
},
|
||||
() => {
|
||||
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
"SOME_DROPDOWN = 'Option 2'"
|
||||
)
|
||||
expectFewerRowsThan(unfilteredRows)
|
||||
cy.url().should('match', /\/editor\/[^/]+\/\d+/)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
it('9 | a bad mapping on the first load still ends at the banner', () => {
|
||||
// The deferred first load cannot check the column against metadata, so its
|
||||
// saveQuery may fail and fall back to an unfiltered load - the banner is
|
||||
// what stops that reading as a working filter.
|
||||
visitWithBufferedMessage(
|
||||
'va_parmap=DROPDOWN_TAB1-NOT_A_COLUMN',
|
||||
{
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
},
|
||||
() => {
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'NOT_A_COLUMN'
|
||||
)
|
||||
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
|
||||
// Deliberately no row-count assertion: this path cannot validate the
|
||||
// column (no metadata yet), so the outcome depends on the platform - on
|
||||
// SAS, mp_filtercheck aborts the save and the load falls back to
|
||||
// unfiltered; the JS mock stores the filter and returns 0 rows. The
|
||||
// banner is the part that must hold either way, because both outcomes
|
||||
// are otherwise indistinguishable from a working filter.
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
it('10 | warns when two parameters filter the same column (an empty grid)', () => {
|
||||
// Clauses are AND-ed, so two different values on one column produce no rows
|
||||
// at all - the last silent-empty-grid path. Reported rather than rewritten,
|
||||
// so the author decides whether to use one parameter carrying both values.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage(
|
||||
{
|
||||
name: 'pr1',
|
||||
label: 'DROPDOWN_TAB1',
|
||||
dataType: 'string',
|
||||
value: 'Option 1'
|
||||
},
|
||||
{
|
||||
name: 'pr2',
|
||||
label: 'DROPDOWN_TAB2',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
}
|
||||
)
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'combined with AND'
|
||||
)
|
||||
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
|
||||
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB2')
|
||||
// The consequence: rows must match both clauses, and the two values
|
||||
// differ, so none can match.
|
||||
expectRows((rows) => expect(rows, 'matching rows').to.equal(0))
|
||||
},
|
||||
'va_parmap=DROPDOWN_TAB1-SOME_DROPDOWN&va_parmap=DROPDOWN_TAB2-SOME_DROPDOWN'
|
||||
)
|
||||
})
|
||||
|
||||
it('12 | a transform adds the previous quarter to the filter', () => {
|
||||
// ADD(QUARTER_START,-1) is the reported requirement: one clause carrying the
|
||||
// selected date and the same point one quarter earlier. 2026-10-01 = 24380,
|
||||
// 2026-07-01 = 24288 in SAS days; nothing in the fixture matches either, so
|
||||
// the assertion is on the clause rather than the row count.
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'SELECTED_DATE',
|
||||
dataType: 'string',
|
||||
value: '01OCT2026'
|
||||
})
|
||||
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'SOME_DATE IN ('
|
||||
)
|
||||
cy.get('.infoBar-text').should('contain.text', '24380')
|
||||
cy.get('.infoBar-text').should('contain.text', '24288')
|
||||
cy.get('h4').should('contain.text', '(0 rows')
|
||||
},
|
||||
'va_parmap=SELECTED_DATE-SOME_DATE:ADD(QUARTER_START,-1)'
|
||||
)
|
||||
})
|
||||
|
||||
it('13 | warns when a transform lands on a column it cannot apply to', () => {
|
||||
openEditorInVaMode(
|
||||
'mpe_x_test',
|
||||
() => {
|
||||
pushVaMessage({
|
||||
name: 'pr1',
|
||||
label: 'SELECTED_DATE',
|
||||
dataType: 'string',
|
||||
value: 'x'
|
||||
})
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'is not a date or datetime column'
|
||||
)
|
||||
cy.get('h4').should('contain.text', '(7 rows')
|
||||
},
|
||||
'va_parmap=SELECTED_DATE-SOME_CHAR:ADD(QUARTER_START,-1)'
|
||||
)
|
||||
})
|
||||
|
||||
it('11 | logs a parse-time problem once, however many messages arrive', () => {
|
||||
// The parse problem is logged at app init; the editor's own loop must not log
|
||||
// it again. The second entry is valid-but-missing, so the loop definitely runs
|
||||
// and logs its own line - which is what makes this a real assertion.
|
||||
cy.visit(
|
||||
`${hostUrl}${appLocation}/index.html#/editor/DC_JSLIB.MPE_X_TEST?embed=va&va_parmap=DROPDOWN_TAB1&va_parmap=DROPDOWN_TAB2-NOT_A_COLUMN`,
|
||||
{
|
||||
onBeforeLoad(win: any) {
|
||||
win.__warns = []
|
||||
const original = win.console.warn.bind(win.console)
|
||||
win.console.warn = (...args: any[]) => {
|
||||
win.__warns.push(args.join(' '))
|
||||
original(...args)
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
|
||||
'not.exist'
|
||||
)
|
||||
pushVaMessage({
|
||||
name: 'pr2',
|
||||
label: 'DROPDOWN_TAB2',
|
||||
dataType: 'string',
|
||||
value: 'Option 2'
|
||||
})
|
||||
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
|
||||
'contain.text',
|
||||
'NOT_A_COLUMN'
|
||||
)
|
||||
cy.window().then((win: any) => {
|
||||
const warns: string[] = win.__warns || []
|
||||
const parseLines = warns.filter((w) =>
|
||||
w.includes('va_parmap "DROPDOWN_TAB1"')
|
||||
)
|
||||
const runtimeLines = warns.filter((w) => w.includes('NOT_A_COLUMN'))
|
||||
expect(parseLines.length, warns.join(' | ')).to.eq(1)
|
||||
expect(runtimeLines.length, warns.join(' | ')).to.eq(1)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* Loads the editor with a VA message already in the pre-bootstrap buffer, the
|
||||
* way VA's own postMessage arrives before Angular starts. A distinct pathname is
|
||||
* used so this is a real document load: a cy.visit that differs only in the hash
|
||||
* is a same-document navigation and would not re-bootstrap the app.
|
||||
*/
|
||||
const visitWithBufferedMessage = (
|
||||
extraParams: string,
|
||||
parameter: any,
|
||||
callback?: any
|
||||
) => {
|
||||
cy.visit(
|
||||
`${hostUrl}${appLocation}/index.html#/editor/DC_JSLIB.MPE_X_TEST?embed=va&${extraParams}`,
|
||||
{
|
||||
onBeforeLoad(win: any) {
|
||||
win.__vaLastMessage = {
|
||||
t: Date.now(),
|
||||
origin: win.location.origin,
|
||||
data: {
|
||||
version: '1',
|
||||
resultName: 'dd1',
|
||||
rowCount: 0,
|
||||
availableRowCount: 0,
|
||||
data: [],
|
||||
columns: [],
|
||||
parameters: [parameter]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
|
||||
if (callback) callback()
|
||||
}
|
||||
|
||||
/** Pushes one VA data-driven content message carrying the given parameters. */
|
||||
const pushVaMessage = (...parameters: any[]) => {
|
||||
cy.window().then((win) => {
|
||||
win.postMessage(
|
||||
{
|
||||
version: '1',
|
||||
resultName: 'dd1',
|
||||
rowCount: 0,
|
||||
availableRowCount: 0,
|
||||
data: [],
|
||||
columns: [],
|
||||
parameters
|
||||
},
|
||||
'*'
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
// Opens a table from the tree (which routes to #/editor/LIB.TABLE), then
|
||||
// re-visits the same route with ?embed=va so the app parses VA embed mode.
|
||||
const openEditorInVaMode = (tablename: string, callback?: any) => {
|
||||
// `extraParams` is appended to that query string (e.g. va_parmap entries).
|
||||
const openEditorInVaMode = (
|
||||
tablename: string,
|
||||
callback?: any,
|
||||
extraParams?: string
|
||||
) => {
|
||||
openTableFromTree(libraryToOpenIncludes, tablename)
|
||||
|
||||
cy.get('#hotTable', { timeout: longerCommandTimeout })
|
||||
@@ -84,7 +498,13 @@ const openEditorInVaMode = (tablename: string, callback?: any) => {
|
||||
.then(() => {
|
||||
cy.url().then((url) => {
|
||||
const separator = url.includes('?') ? '&' : '?'
|
||||
cy.visit(`${url}${separator}embed=va`)
|
||||
const extra = extraParams ? `&${extraParams}` : ''
|
||||
cy.visit(`${url}${separator}embed=va${extra}`)
|
||||
// A cy.visit that differs from the current URL only in the hash is a
|
||||
// same-document navigation, so the app is not re-bootstrapped and the
|
||||
// embed parameter (read once at app init) never takes effect. Reload to
|
||||
// bootstrap it, as the VA iframe load does.
|
||||
cy.reload()
|
||||
cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
|
||||
'not.exist'
|
||||
)
|
||||
|
||||
@@ -14,11 +14,10 @@ context('metanav tests: ', function () {
|
||||
})
|
||||
|
||||
this.beforeEach(() => {
|
||||
cy.visit(hostUrl + appLocation)
|
||||
cy.get('input.username').type(username)
|
||||
cy.get('input.password').type(password)
|
||||
cy.get('.login-group button').click()
|
||||
|
||||
// beforeAll has already logged in. Typing into the login form here failed
|
||||
// once the session existed - the form is not on screen, so Cypress reported
|
||||
// "the center of this element is hidden from view" and the whole suite was
|
||||
// skipped behind that hook.
|
||||
visitPage('view/metadata')
|
||||
})
|
||||
|
||||
|
||||
+3
-1
@@ -37,7 +37,9 @@
|
||||
"compodoc:build-and-serve": "compodoc -p tsconfig.doc.json -s --name 'Data Controller Client'",
|
||||
"compodoc:serve": "compodoc -s --name 'Data Controller Client'",
|
||||
"lighthouse": "lhci autorun",
|
||||
"ng": "ng"
|
||||
"ng": "ng",
|
||||
"test:tz-west": "TZ=America/New_York npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'",
|
||||
"test:tz-east": "TZ=Pacific/Auckland npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'"
|
||||
},
|
||||
"private": true,
|
||||
"dependencies": {
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { QueryClause } from './models/TableData'
|
||||
import { VaParmapTarget } from './shared/utils/parse-va-parmap'
|
||||
|
||||
/**
|
||||
* Filtering cache info, to be reused when filtering modal is re-open
|
||||
@@ -61,6 +62,21 @@ export const globals: {
|
||||
* Defaults to 'live'; toggled by the Auto-apply checkbox in the editor.
|
||||
*/
|
||||
vaApplyMode: 'live' | 'confirm'
|
||||
/**
|
||||
* Explicit VA parameter -> DC column mappings, parsed from the `va_parmap`
|
||||
* URL parameter (see parseVaParmap). Empty unless configured; entries override
|
||||
* VA's default same-name (parameter label -> column) resolution, so two DC
|
||||
* instances over one table can filter the same column from differently-named
|
||||
* VA parameters.
|
||||
*/
|
||||
vaParamMap: Map<string, VaParmapTarget>
|
||||
/**
|
||||
* `va_parmap` entries that could not be used at all (not a plain SAS column
|
||||
* name, or missing a side of the `-`). Surfaced to the report author, because
|
||||
* these are hand-written in the report's URL and a typo otherwise looks like
|
||||
* a working, unfiltered view.
|
||||
*/
|
||||
vaParmapProblems: string[]
|
||||
rootParam: string
|
||||
dcLib: string
|
||||
xlmaps: XLMapListItem[]
|
||||
@@ -77,6 +93,8 @@ export const globals: {
|
||||
} = {
|
||||
embed: false,
|
||||
vaApplyMode: 'live',
|
||||
vaParamMap: new Map(),
|
||||
vaParmapProblems: [],
|
||||
rootParam: <string>'',
|
||||
dcLib: '',
|
||||
xlmaps: [],
|
||||
|
||||
@@ -12,6 +12,7 @@ import '@clr/icons'
|
||||
import '@clr/icons/shapes/all-shapes'
|
||||
import { globals } from './_globals'
|
||||
import { parseEmbedParam } from './shared/utils/parse-embed-param'
|
||||
import { parseVaParmap } from './shared/utils/parse-va-parmap'
|
||||
import moment from 'moment'
|
||||
import { EventService } from './services/event.service'
|
||||
import { AppService } from './services/app.service'
|
||||
@@ -163,6 +164,21 @@ export class AppComponent {
|
||||
globals.embed = embed
|
||||
this.embed = embed
|
||||
}
|
||||
// VA parameter -> DC column mappings for this instance, held on globals
|
||||
// (like `embed` and `vaApplyMode`) so in-app navigation that rebuilds the
|
||||
// editor keeps them - e.g. returning from the stage / approve pages.
|
||||
if (globals.embed === 'va') {
|
||||
const parmap = parseVaParmap(hashQuery ?? '')
|
||||
globals.vaParamMap = parmap.map
|
||||
globals.vaParmapProblems = parmap.problems
|
||||
// Visible in the browser console as well as the editor banner: the entry
|
||||
// is hand-written in the report's URL, so a typo has to be findable when
|
||||
// someone is looking at the report rather than at DC's UI.
|
||||
for (const problem of parmap.problems) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn('Data Controller: ' + problem)
|
||||
}
|
||||
}
|
||||
|
||||
this.subscribeToShowAbortModal()
|
||||
this.subscribeToRequestsModal()
|
||||
|
||||
@@ -1,7 +1,36 @@
|
||||
<main class="content-area position-relative">
|
||||
<!-- An embedded instance (a report object) has no report author who can deploy:
|
||||
showing the wizard there only produces a screen whose buttons cannot work,
|
||||
because the deployment needs an authenticated, non-embedded session. Name
|
||||
the state instead, and keep the wizard for standalone use. -->
|
||||
@if (isEmbedded) {
|
||||
<div class="clr-row">
|
||||
<div id="not-configured" class="card">
|
||||
<div class="card-header">Data Controller is not configured</div>
|
||||
<div class="card-block">
|
||||
<div class="card-text">
|
||||
<p class="mt-0">
|
||||
This Data Controller instance has not been deployed yet, so it
|
||||
cannot be shown inside a report.
|
||||
</p>
|
||||
<p>
|
||||
Deployment has to be done once, in its own browser tab, by someone
|
||||
who can write to the application location:
|
||||
<a [href]="deployUrl" target="_blank">{{ deployUrl }}</a>
|
||||
</p>
|
||||
<p>
|
||||
When it is configured, reload the report and this object will show
|
||||
the editor.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
|
||||
<div class="clr-row">
|
||||
<!-- T&C section -->
|
||||
@if (step === 0) {
|
||||
@if (!isEmbedded && step === 0) {
|
||||
<div id="TCS" class="card">
|
||||
<div class="card-header">Terms and Conditions</div>
|
||||
<div class="card-block">
|
||||
@@ -35,7 +64,7 @@
|
||||
</div>
|
||||
<!-- T&C section end -->
|
||||
|
||||
@if (step > 0) {
|
||||
@if (!isEmbedded && step > 0) {
|
||||
@switch (true) {
|
||||
@case (sasJsConfig.serverType === ServerType.SasViya) {
|
||||
<div class="autodeploy-section card">
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
import { NO_ERRORS_SCHEMA } from '@angular/core'
|
||||
import { ComponentFixture, TestBed } from '@angular/core/testing'
|
||||
import { ServerType } from '@sasjs/utils/types/serverType'
|
||||
import { DeployComponent } from './deploy.component'
|
||||
import { standaloneUrl } from './standalone-url'
|
||||
import { globals } from '../_globals'
|
||||
import { AppStoreService } from '../services/app-store.service'
|
||||
import { SasService } from '../services/sas.service'
|
||||
|
||||
/**
|
||||
* An embedded instance cannot deploy itself: the deployment needs an
|
||||
* authenticated, standalone session, and the person looking at a report object
|
||||
* is not the person who installs the app. The component therefore withholds the
|
||||
* wizard and names the state instead.
|
||||
*
|
||||
* These cases render the template, so they fail if the `!isEmbedded` guards in
|
||||
* deploy.component.html are dropped - asserting the getter alone would not.
|
||||
*/
|
||||
describe('DeployComponent embed guard', () => {
|
||||
let fixture: ComponentFixture<DeployComponent>
|
||||
let component: DeployComponent
|
||||
|
||||
const originalEmbed = globals.embed
|
||||
|
||||
const configure = () =>
|
||||
TestBed.configureTestingModule({
|
||||
declarations: [DeployComponent],
|
||||
providers: [
|
||||
{
|
||||
provide: AppStoreService,
|
||||
useValue: {
|
||||
getDcAdapterSettings: () => ({
|
||||
appLoc: '/Public/app/dc',
|
||||
dcPath: '/tmp/dc'
|
||||
})
|
||||
}
|
||||
},
|
||||
{
|
||||
provide: SasService,
|
||||
useValue: {
|
||||
getSasjsInstance: () => ({}),
|
||||
getSasjsConfig: () => ({ serverType: ServerType.Sasjs })
|
||||
}
|
||||
}
|
||||
],
|
||||
// The wizard renders child components this spec does not exercise.
|
||||
schemas: [NO_ERRORS_SCHEMA]
|
||||
}).compileComponents()
|
||||
|
||||
const render = () => {
|
||||
fixture = TestBed.createComponent(DeployComponent)
|
||||
component = fixture.componentInstance
|
||||
fixture.detectChanges()
|
||||
}
|
||||
|
||||
const tcsSection = () =>
|
||||
fixture.nativeElement.querySelector('#TCS') as HTMLElement | null
|
||||
const notConfigured = () =>
|
||||
fixture.nativeElement.querySelector('#not-configured') as HTMLElement | null
|
||||
const deploySection = () =>
|
||||
fixture.nativeElement.querySelector(
|
||||
'.autodeploy-section'
|
||||
) as HTMLElement | null
|
||||
|
||||
afterEach(() => {
|
||||
globals.embed = originalEmbed
|
||||
TestBed.resetTestingModule()
|
||||
})
|
||||
|
||||
it('offers the wizard when running in its own tab', () => {
|
||||
globals.embed = false
|
||||
configure()
|
||||
render()
|
||||
|
||||
expect(tcsSection()).toBeTruthy()
|
||||
expect(notConfigured()).toBeNull()
|
||||
expect(component.isEmbedded).toBeFalse()
|
||||
})
|
||||
|
||||
it('replaces the wizard with the notice for a VA embed', () => {
|
||||
globals.embed = 'va'
|
||||
configure()
|
||||
render()
|
||||
|
||||
expect(tcsSection()).toBeNull()
|
||||
expect(notConfigured()).toBeTruthy()
|
||||
})
|
||||
|
||||
it('replaces the wizard with the notice for a plain embed too', () => {
|
||||
globals.embed = true
|
||||
configure()
|
||||
render()
|
||||
|
||||
expect(tcsSection()).toBeNull()
|
||||
expect(notConfigured()).toBeTruthy()
|
||||
})
|
||||
|
||||
it('names the standalone URL in the embedded notice', () => {
|
||||
globals.embed = 'va'
|
||||
configure()
|
||||
render()
|
||||
|
||||
const link = notConfigured()!.querySelector('a') as HTMLAnchorElement
|
||||
|
||||
expect(link.getAttribute('href')).toBe(component.deployUrl)
|
||||
expect(link.textContent).toBe(component.deployUrl)
|
||||
})
|
||||
|
||||
it('withholds the deploy section after the terms are accepted, when embedded', () => {
|
||||
globals.embed = 'va'
|
||||
configure()
|
||||
render()
|
||||
|
||||
component.step = 1
|
||||
fixture.detectChanges()
|
||||
|
||||
expect(deploySection()).toBeNull()
|
||||
})
|
||||
|
||||
it('shows the deploy section after the terms are accepted, standalone', () => {
|
||||
globals.embed = false
|
||||
configure()
|
||||
render()
|
||||
|
||||
component.step = 1
|
||||
fixture.detectChanges()
|
||||
|
||||
expect(deploySection()).toBeTruthy()
|
||||
})
|
||||
})
|
||||
|
||||
describe('DeployComponent onNavigateToHome', () => {
|
||||
const originalEmbed = globals.embed
|
||||
const originalHref = window.location.href
|
||||
|
||||
afterEach(() => {
|
||||
globals.embed = originalEmbed
|
||||
history.replaceState(null, '', originalHref)
|
||||
TestBed.resetTestingModule()
|
||||
})
|
||||
|
||||
it('opens the standalone URL, merging an embed tab into the app URL', () => {
|
||||
// Karma serves the runner with its own query, so the app URL is set here
|
||||
// rather than read from window.location: the assertion has to see the
|
||||
// app's query survive the embed parameters being dropped. The shape is
|
||||
// the real SAS 9 one - the app's `_program` query in the search part,
|
||||
// the embed switches in the hash query.
|
||||
history.replaceState(
|
||||
null,
|
||||
'',
|
||||
'/deploy-tc?_program=/Public/app/dc&_debug=131#/editor?embed=va&va_parmap=A-B'
|
||||
)
|
||||
const open = spyOn(window, 'open')
|
||||
TestBed.configureTestingModule({
|
||||
declarations: [DeployComponent],
|
||||
providers: [
|
||||
{
|
||||
provide: AppStoreService,
|
||||
useValue: { getDcAdapterSettings: () => ({ appLoc: '', dcPath: '' }) }
|
||||
},
|
||||
{
|
||||
provide: SasService,
|
||||
useValue: {
|
||||
getSasjsInstance: () => ({}),
|
||||
getSasjsConfig: () => ({ serverType: ServerType.Sasjs })
|
||||
}
|
||||
}
|
||||
],
|
||||
schemas: [NO_ERRORS_SCHEMA]
|
||||
})
|
||||
const fixture = TestBed.createComponent(DeployComponent)
|
||||
|
||||
fixture.componentInstance.onNavigateToHome()
|
||||
|
||||
const opened = open.calls.mostRecent().args[0] as string
|
||||
expect(opened).toBe(
|
||||
window.location.origin + '/deploy-tc?_program=/Public/app/dc&_debug=131'
|
||||
)
|
||||
expect(opened).not.toContain('#')
|
||||
expect(opened).not.toContain('embed=')
|
||||
})
|
||||
})
|
||||
|
||||
describe('standaloneUrl', () => {
|
||||
const origin = window.location.origin
|
||||
|
||||
it('keeps the path of a plain URL', () => {
|
||||
expect(standaloneUrl(`${origin}/SASJobExecution/dc`)).toBe(
|
||||
`${origin}/SASJobExecution/dc`
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps a query the app needs, and drops the hash route', () => {
|
||||
expect(standaloneUrl(`${origin}/path?x=1#/deploy`)).toBe(
|
||||
`${origin}/path?x=1`
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps `_program=`, which is how SAS 9 and Viya address the app', () => {
|
||||
// The whole app URL is the query here: /SASJobExecution/?_program=... is
|
||||
// the standalone app, so dropping the query would open a blank executor.
|
||||
const sas9 =
|
||||
`${origin}/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice` +
|
||||
'&_debug=131'
|
||||
expect(standaloneUrl(sas9)).toBe(sas9)
|
||||
})
|
||||
|
||||
it('drops a bare embed query', () => {
|
||||
expect(standaloneUrl(`${origin}/app?embed=va`)).toBe(`${origin}/app`)
|
||||
})
|
||||
|
||||
it('drops the va_parmap entries along with the embed switch', () => {
|
||||
expect(
|
||||
standaloneUrl(
|
||||
`${origin}/app?embed=va&va_parmap=A-B&va_parmap=C-D#/editor/x`
|
||||
)
|
||||
).toBe(`${origin}/app`)
|
||||
})
|
||||
|
||||
it('keeps a query parameter that merely starts with va_', () => {
|
||||
expect(standaloneUrl(`${origin}/app?embed=va&va_rowlimit=100`)).toBe(
|
||||
`${origin}/app?va_rowlimit=100`
|
||||
)
|
||||
})
|
||||
|
||||
it('resolves a relative URL against the current origin', () => {
|
||||
expect(standaloneUrl('/app/#/deploy')).toBe(`${origin}/app/`)
|
||||
})
|
||||
})
|
||||
@@ -4,6 +4,8 @@ import { SASjsConfig } from '@sasjs/adapter'
|
||||
import { ServerType } from '@sasjs/utils/types/serverType'
|
||||
import { AppStoreService } from '../services/app-store.service'
|
||||
import { DcAdapterSettings } from '../models/DcAdapterSettings'
|
||||
import { globals } from '../_globals'
|
||||
import { standaloneUrl } from './standalone-url'
|
||||
|
||||
@Component({
|
||||
selector: 'app-deploy',
|
||||
@@ -44,6 +46,20 @@ export class DeployComponent implements OnInit {
|
||||
this.setDeployDefaults()
|
||||
}
|
||||
|
||||
/**
|
||||
* True when DC is running inside a report rather than in its own tab. The
|
||||
* deployment needs an authenticated, standalone session, so the wizard is not
|
||||
* offered here - a report object cannot deploy the app that it is displaying.
|
||||
*/
|
||||
get isEmbedded(): boolean {
|
||||
return !!globals.embed
|
||||
}
|
||||
|
||||
/** The standalone URL an administrator deploys from, for the embedded notice. */
|
||||
get deployUrl(): string {
|
||||
return standaloneUrl(window.location.href)
|
||||
}
|
||||
|
||||
/**
|
||||
* Setting default values used for deploy request
|
||||
*/
|
||||
@@ -60,6 +76,8 @@ export class DeployComponent implements OnInit {
|
||||
}
|
||||
|
||||
public onNavigateToHome() {
|
||||
window.open(location.href.split('#')[0], '_blank')
|
||||
// Same helper as deployUrl: splitting on '#' left the query string in
|
||||
// place, so a tab opened from `?embed=va#/deploy` stayed embedded.
|
||||
window.open(standaloneUrl(window.location.href), '_blank')
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* The URL a Data Controller instance runs at when it is not embedded: the same
|
||||
* URL, without the parts that make it an embed - the `embed` switch, the
|
||||
* `va_parmap` entries, and the hash route.
|
||||
*
|
||||
* The rest of the query is kept. On SAS 9 and Viya the app is addressed
|
||||
* entirely through it (`/SASJobExecution/?_program=/.../startupservice`), so
|
||||
* dropping the query would open a blank executor rather than the app, and any
|
||||
* other parameter the deployment relies on would go the same way.
|
||||
*
|
||||
* Only these two parameters are embed parameters. A later parameter that
|
||||
* happens to start with `va_` belongs to the app, not the embed, so it stays.
|
||||
*
|
||||
* Parsing with URL also handles the relative hrefs the adapter can hand back,
|
||||
* which plain string splicing cannot.
|
||||
*/
|
||||
const EMBED_PARAMS = ['embed', 'va_parmap']
|
||||
|
||||
export const standaloneUrl = (href: string): string => {
|
||||
const url = new URL(href, window.location.origin)
|
||||
url.hash = ''
|
||||
for (const key of Array.from(url.searchParams.keys())) {
|
||||
if (EMBED_PARAMS.includes(key.toLowerCase())) {
|
||||
url.searchParams.delete(key)
|
||||
}
|
||||
}
|
||||
|
||||
return url.toString()
|
||||
}
|
||||
@@ -459,6 +459,43 @@
|
||||
<!--<hr>-->
|
||||
<!--{{ validationDone }}-->
|
||||
|
||||
<!-- VA embed: something about a `va_parmap` entry in this report
|
||||
object's URL needs the report author's attention - it could not be
|
||||
used, it was applied but AND-ed against another parameter, or it
|
||||
names a parameter VA never sends. Shown as a banner, not just a
|
||||
console warning, because
|
||||
the entries are written by the report author and a typo otherwise
|
||||
looks like a working filter over the whole table. -->
|
||||
@if (isVaEmbed && vaMappingWarnings.length) {
|
||||
<clr-alert
|
||||
class="va-parmap-warning"
|
||||
[clrAlertType]="'warning'"
|
||||
[clrAlertClosable]="false"
|
||||
>
|
||||
<clr-alert-item>
|
||||
<div class="alert-text">
|
||||
<strong>VA parameter mapping problem</strong>
|
||||
@for (warning of vaMappingWarnings; track warning) {
|
||||
<div>{{ warning }}</div>
|
||||
}
|
||||
<!-- The reader is the report author, who can only fix this in
|
||||
the report: name the parameter they edit, and say what a
|
||||
correct entry looks like, so the message is actionable
|
||||
without the docs. -->
|
||||
<div>
|
||||
Fix: edit the <code>va_parmap</code> entries in the URL of
|
||||
this Data Controller object in the report. An entry is
|
||||
<code>parameter-column</code>, optionally
|
||||
<code>:ADD(anchor,offset)</code> or
|
||||
<code>:RANGE(period,offset)</code> - for example
|
||||
<code>va_parmap=SEL_DATE-D_DATE:ADD(QUARTER_START,-1)</code
|
||||
>.
|
||||
</div>
|
||||
</div>
|
||||
</clr-alert-item>
|
||||
</clr-alert>
|
||||
}
|
||||
|
||||
<div class="hot-wrapper clr-flex-1">
|
||||
<hot-table
|
||||
#hotInstance
|
||||
|
||||
@@ -27,3 +27,15 @@
|
||||
font-style: italic;
|
||||
opacity: 0.7;
|
||||
}
|
||||
|
||||
/* VA embed: `va_parmap` diagnostic banner. Clarity's alert lays `.alert-title`
|
||||
and `.alert-text` out on one row, which pushes a long message off the right
|
||||
edge; this keeps the whole message in the flow and wraps it. */
|
||||
.va-parmap-warning {
|
||||
width: 100%;
|
||||
|
||||
.alert-text {
|
||||
white-space: normal;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
}
|
||||
@@ -73,6 +73,7 @@ import {
|
||||
} from '../shared/dc-validator/utils/autoEscapedCellTracker'
|
||||
import { HyperFormula } from 'hyperformula'
|
||||
import { parseLabelsParam } from '../shared/utils/parse-labels-param'
|
||||
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
|
||||
import { getDisplayColHeaders } from '../shared/utils/display-col-headers'
|
||||
import { buildColInfoHtml } from '../shared/utils/col-info-html'
|
||||
import { globals } from '../_globals'
|
||||
@@ -485,6 +486,46 @@ export class EditorComponent
|
||||
get isVaEmbed() {
|
||||
return globals.embed === 'va'
|
||||
}
|
||||
/**
|
||||
* Explicit VA parameter -> DC column mappings (`va_parmap`) for this instance.
|
||||
* Parsed once at app init (see parseVaParmap) and held on globals so the
|
||||
* editor reloads a filter apply performs keep using the same mapping.
|
||||
*/
|
||||
get vaParamMap(): Map<string, VaParmapTarget> {
|
||||
return globals.vaParamMap
|
||||
}
|
||||
/** `va_parmap` entries that name a column this table does not have. */
|
||||
public vaUnresolvedMappings: string[] = []
|
||||
/** `va_parmap` keys that no VA message has named (a parameter-side typo). */
|
||||
public vaUnmatchedMappings: string[] = []
|
||||
/** Parameters in this message that filter the same column (an AND-ed pair). */
|
||||
public vaSameColumnMappings: string[] = []
|
||||
/** Transform mappings this message cannot apply (non-temporal column or value). */
|
||||
public vaTransformProblems: string[] = []
|
||||
/** The warning text last written to the console, so repeated VA messages log once. */
|
||||
private vaWarningsReported = ''
|
||||
/**
|
||||
* Everything wrong with this instance's `va_parmap` entries, in the report
|
||||
* author's terms: entries that could not be parsed at all, plus entries that
|
||||
* name a column the open table does not have. Shown as a banner in the editor
|
||||
* - a report developer is usually not going to read the browser console.
|
||||
*/
|
||||
get vaMappingWarnings(): string[] {
|
||||
return [...globals.vaParmapProblems, ...this.vaRuntimeWarnings]
|
||||
}
|
||||
/**
|
||||
* The warnings this instance discovers for itself, i.e. everything except the
|
||||
* parse-time problems - those are logged once at app init (`app.component`),
|
||||
* so logging them again here would double every line in the console.
|
||||
*/
|
||||
get vaRuntimeWarnings(): string[] {
|
||||
return [
|
||||
...this.vaUnresolvedMappings,
|
||||
...this.vaUnmatchedMappings,
|
||||
...this.vaSameColumnMappings,
|
||||
...this.vaTransformProblems
|
||||
]
|
||||
}
|
||||
/**
|
||||
* VA filter mode. Live = auto-apply each VA change; confirm = stage the change
|
||||
* and wait for the user to click Apply. Backed by globals (toggled by the
|
||||
@@ -3602,7 +3643,7 @@ export class EditorComponent
|
||||
const msg = this.vaMessaging.latestMessage()
|
||||
if (!msg) return false
|
||||
|
||||
const clauses = this.vaFilter.buildInitialClauses(msg)
|
||||
const clauses = this.vaFilter.buildInitialClauses(msg, this.vaParamMap)
|
||||
if (clauses.length === 0) return false
|
||||
|
||||
try {
|
||||
@@ -5048,7 +5089,53 @@ export class EditorComponent
|
||||
|
||||
this.updateVaFilterText(msg)
|
||||
|
||||
const clauses = this.vaFilter.buildClauses(msg, labelToVar, this.cols)
|
||||
// A `va_parmap` entry naming a column this table does not have is skipped by
|
||||
// buildClauses. Report it (banner + console, once per distinct set) so the
|
||||
// report author sees a misconfiguration instead of an unfiltered grid.
|
||||
this.vaUnresolvedMappings = this.vaFilter.unresolvedMappings(
|
||||
msg,
|
||||
this.vaParamMap,
|
||||
this.cols,
|
||||
this.libds
|
||||
)
|
||||
// A key naming no parameter VA has ever sent is a typo on the parameter
|
||||
// side: the mapping never applies. Only judged once parameters are actually
|
||||
// arriving, since VA sends just the parameters its query uses.
|
||||
this.vaUnmatchedMappings = this.vaFilter.unmatchedMappings(
|
||||
this.vaParamMap,
|
||||
this.vaMessaging.seenParamKeys,
|
||||
this.vaMessaging.lastMessageParamsStable
|
||||
)
|
||||
// Two parameters filtering one column are AND-ed, so differing values leave
|
||||
// an empty grid. Reported, not rewritten: turning that AND into an OR is a
|
||||
// filter-semantics decision, taken in buildClauses and not in a diagnostic.
|
||||
this.vaSameColumnMappings = this.vaFilter.sameColumnMappings(
|
||||
msg,
|
||||
this.vaParamMap,
|
||||
this.cols
|
||||
)
|
||||
// A transform that cannot run here (char column, value that is not a date) is
|
||||
// skipped by the clause builder, so the banner has to say why.
|
||||
this.vaTransformProblems = this.vaFilter.unsupportedTransforms(
|
||||
msg,
|
||||
this.vaParamMap,
|
||||
this.cols
|
||||
)
|
||||
const warnings = this.vaRuntimeWarnings.join(' | ')
|
||||
if (warnings !== this.vaWarningsReported) {
|
||||
this.vaWarningsReported = warnings
|
||||
for (const warning of this.vaRuntimeWarnings) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn('Data Controller: ' + warning)
|
||||
}
|
||||
}
|
||||
|
||||
const clauses = this.vaFilter.buildClauses(
|
||||
msg,
|
||||
labelToVar,
|
||||
this.cols,
|
||||
this.vaParamMap
|
||||
)
|
||||
const signature = this.vaFilter.signature(clauses)
|
||||
|
||||
if (signature === this.vaMessaging.filterSignature) {
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { VaFilterService } from './va-filter.service'
|
||||
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
|
||||
import { VaMessage } from './va-messaging.service'
|
||||
import { HelperService } from './helper.service'
|
||||
|
||||
@@ -14,6 +15,7 @@ function msg(parameters: any[]): VaMessage {
|
||||
|
||||
describe('VaFilterService', () => {
|
||||
let service: VaFilterService
|
||||
let helperService: HelperService
|
||||
|
||||
// DC column metadata used by the metadata path. cellValidation[0] is the
|
||||
// delete column and must be skipped by buildLabelToVarMap.
|
||||
@@ -42,7 +44,8 @@ describe('VaFilterService', () => {
|
||||
]
|
||||
|
||||
beforeEach(() => {
|
||||
service = new VaFilterService(new HelperService({} as any))
|
||||
helperService = new HelperService({} as any)
|
||||
service = new VaFilterService(helperService)
|
||||
})
|
||||
|
||||
const labelToVar = () =>
|
||||
@@ -128,6 +131,41 @@ describe('VaFilterService', () => {
|
||||
expect(build([{ label: 'SOME_DATE', value: '2025/03/31' }])).toHaveSize(1)
|
||||
})
|
||||
|
||||
it('parses word dates, in either order and with an optional comma', () => {
|
||||
const expected = "'01JAN2026'd"
|
||||
// The reported shape: a date parameter displayed as WORDDATE.
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: 'January 1, 2026' }])[0].RAW_VALUE
|
||||
).toBe(expected)
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: 'Jan 1, 2026' }])[0].RAW_VALUE
|
||||
).toBe(expected)
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: 'Jan. 1, 2026' }])[0].RAW_VALUE
|
||||
).toBe(expected)
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: '1 January 2026' }])[0].RAW_VALUE
|
||||
).toBe(expected)
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: '1-Jan-2026' }])[0].RAW_VALUE
|
||||
).toBe(expected)
|
||||
})
|
||||
|
||||
it('matches a spelled-out month case-insensitively, and keeps DATE9 working', () => {
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: 'DECEMBER 31, 2025' }])[0].RAW_VALUE
|
||||
).toBe("'31DEC2025'd")
|
||||
expect(
|
||||
build([{ label: 'SOME_DATE', value: '01AUG1962' }])[0].RAW_VALUE
|
||||
).toBe("'01AUG1962'd")
|
||||
})
|
||||
|
||||
it('still drops a month-name value that has no day', () => {
|
||||
// A month and year alone is not a date - accepted shapes always carry a
|
||||
// day, so this must not silently become the first of the month.
|
||||
expect(build([{ label: 'SOME_DATE', value: 'January 2026' }])).toEqual([])
|
||||
})
|
||||
|
||||
it('resolves ambiguous slash dates day-first (never US), disambiguating by value', () => {
|
||||
const iso = build([{ label: 'SOME_DATE', value: '2025-03-31' }])[0]
|
||||
.RAW_VALUE
|
||||
@@ -199,7 +237,694 @@ describe('VaFilterService', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildClauses with an explicit va_parmap', () => {
|
||||
const build = (parameters: any[], paramMap: Map<string, VaParmapTarget>) =>
|
||||
service.buildClauses(msg(parameters), labelToVar(), cols, paramMap)
|
||||
|
||||
it('overrides the same-name match, including the value formatting', () => {
|
||||
// The VA label IS a DC column here (SOME_CHAR), but the mapping points at
|
||||
// a different one - the numeric column - so the value is formatted numeric.
|
||||
const map = new Map([['some_char', { column: 'SOME_NUM' }]])
|
||||
expect(build([{ label: 'SOME_CHAR', value: '10' }], map)).toEqual([
|
||||
{
|
||||
GROUP_LOGIC: 'AND',
|
||||
SUBGROUP_LOGIC: 'AND',
|
||||
SUBGROUP_ID: 0,
|
||||
VARIABLE_NM: 'SOME_NUM',
|
||||
OPERATOR_NM: '=',
|
||||
RAW_VALUE: '10'
|
||||
}
|
||||
])
|
||||
})
|
||||
|
||||
it('lets two differently-named params filter the same column', () => {
|
||||
// The multi-tab case: one DC instance per report tab, each with its own
|
||||
// VA parameter, both mapping to the same DC column.
|
||||
const map = new Map([
|
||||
['region_tab1', { column: 'SOME_CHAR' }],
|
||||
['region_tab2', { column: 'SOME_CHAR' }]
|
||||
])
|
||||
const clauses = build(
|
||||
[
|
||||
{ label: 'REGION_TAB1', value: 'north' },
|
||||
{ label: 'REGION_TAB2', value: 'south' }
|
||||
],
|
||||
map
|
||||
)
|
||||
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
|
||||
['SOME_CHAR', "'north'"],
|
||||
['SOME_CHAR', "'south'"]
|
||||
])
|
||||
})
|
||||
|
||||
it('matches the parameter label only, not its internal name', () => {
|
||||
// The internal name is the report's own bookkeeping. Matching it as well
|
||||
// meant one parameter answered to two keys, so an entry written against the
|
||||
// name and one against the label silently fought over it.
|
||||
const map = new Map([['pr273', { column: 'SOME_NUM' }]])
|
||||
expect(
|
||||
build([{ name: 'pr273', label: 'Tab_1_Value', value: '7' }], map)
|
||||
).toEqual([])
|
||||
// The label still matches, case-insensitively.
|
||||
expect(
|
||||
build(
|
||||
[{ name: 'pr273', label: 'Tab_1_Value', value: 'x' }],
|
||||
new Map([['tab_1_value', { column: 'SOME_CHAR' }]])
|
||||
)[0].VARIABLE_NM
|
||||
).toBe('SOME_CHAR')
|
||||
})
|
||||
|
||||
it('uses the column case from the loaded metadata', () => {
|
||||
const map = new Map([['region_tab1', { column: 'some_char' }]])
|
||||
expect(
|
||||
build([{ label: 'REGION_TAB1', value: 'x' }], map)[0].VARIABLE_NM
|
||||
).toBe('SOME_CHAR')
|
||||
})
|
||||
|
||||
it('leaves unmapped params on the same-name path', () => {
|
||||
const map = new Map([['region_tab1', { column: 'SOME_NUM' }]])
|
||||
const clauses = build(
|
||||
[
|
||||
{ label: 'REGION_TAB1', value: '10' },
|
||||
{ label: 'SOME_CHAR', value: 'x' }
|
||||
],
|
||||
map
|
||||
)
|
||||
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
|
||||
['SOME_NUM', '10'],
|
||||
['SOME_CHAR', "'x'"]
|
||||
])
|
||||
})
|
||||
|
||||
it('skips a mapping whose column is not in this table (no fallback)', () => {
|
||||
// Falling back to the same-name match would silently apply the filter the
|
||||
// mapping exists to override, so the param is dropped instead.
|
||||
const map = new Map([['some_char', { column: 'NOT_IN_THIS_TABLE' }]])
|
||||
expect(build([{ label: 'SOME_CHAR', value: 'x' }], map)).toEqual([])
|
||||
})
|
||||
|
||||
it('ANDs two parameters that map to the same column - it does not union them', () => {
|
||||
// What a report gets when one column's two values arrive in two
|
||||
// parameters: every clause is AND-ed, so the result is the INTERSECTION.
|
||||
// Two different dates therefore match no rows at all rather than both
|
||||
// dates' rows - a single parameter carrying both values is the shape that
|
||||
// ORs, via `IN (...)`.
|
||||
const map = new Map([
|
||||
['param_a', { column: 'SOME_DATE' }],
|
||||
['param_b', { column: 'SOME_DATE' }]
|
||||
])
|
||||
const one = build([{ label: 'PARAM_A', value: '01JUL1960' }], map)[0]
|
||||
const other = build([{ label: 'PARAM_B', value: '01JAN1961' }], map)[0]
|
||||
const clauses = build(
|
||||
[
|
||||
{ label: 'PARAM_A', value: '01JUL1960' },
|
||||
{ label: 'PARAM_B', value: '01JAN1961' }
|
||||
],
|
||||
map
|
||||
)
|
||||
expect(one.RAW_VALUE).not.toBe(other.RAW_VALUE)
|
||||
expect(clauses).toHaveSize(2)
|
||||
expect(clauses.map((c) => c.RAW_VALUE)).toEqual([
|
||||
one.RAW_VALUE,
|
||||
other.RAW_VALUE
|
||||
])
|
||||
for (const clause of clauses) {
|
||||
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
|
||||
expect(clause.OPERATOR_NM).toBe('=')
|
||||
expect(clause.GROUP_LOGIC).toBe('AND')
|
||||
expect(clause.SUBGROUP_LOGIC).toBe('AND')
|
||||
expect(clause.SUBGROUP_ID).toBe(0)
|
||||
}
|
||||
})
|
||||
|
||||
it('applies a transform to the clause it builds', () => {
|
||||
// ADD(QUARTER_START,-1) is the reported requirement: the selected row plus
|
||||
// the row for the previous quarter, as one IN clause so the AND rule is
|
||||
// untouched. 2026-10-01 = 24380, 2026-07-01 = 24288 in SAS days.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const [clause] = build([{ label: 'SELECTED', value: '01OCT2026' }], map)
|
||||
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
|
||||
expect(clause.OPERATOR_NM).toBe('IN')
|
||||
expect(clause.RAW_VALUE).toBe('(24380,24288)')
|
||||
})
|
||||
|
||||
it('applies a RANGE as BETWEEN', () => {
|
||||
// RANGE(QUARTER,-1) for a mid-quarter value: 2026-04-01..2026-06-30
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'RANGE' as const,
|
||||
period: 'QUARTER' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const [clause] = build([{ label: 'SELECTED', value: '1962-08-15' }], map)
|
||||
expect(clause.OPERATOR_NM).toBe('BETWEEN')
|
||||
expect(clause.RAW_VALUE).toContain(' AND ')
|
||||
expect(clause.RAW_VALUE.split(' AND ')).toHaveSize(2)
|
||||
})
|
||||
|
||||
it('keeps equality when the value already sits on the anchor', () => {
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: 0
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const [clause] = build([{ label: 'SELECTED', value: '01OCT2026' }], map)
|
||||
expect(clause.OPERATOR_NM).toBe('=')
|
||||
expect(clause.RAW_VALUE).toBe('24380')
|
||||
})
|
||||
|
||||
it('drops a transform on a column it cannot apply to', () => {
|
||||
// A character column has no dates to shift; the parameter is skipped rather
|
||||
// than filtered on the raw value the author did not see.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_CHAR',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
expect(build([{ label: 'SELECTED', value: 'x' }], map)).toEqual([])
|
||||
})
|
||||
|
||||
it('adds nothing when the value is unparseable as a date', () => {
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
expect(build([{ label: 'SELECTED', value: 'not a date' }], map)).toEqual(
|
||||
[]
|
||||
)
|
||||
})
|
||||
|
||||
it('round-trips a bare SAS day number unchanged', () => {
|
||||
// VA may send the SAS internal value rather than a formatted string. The
|
||||
// Date a transform works from is local, so the number must come back out of
|
||||
// the clause identical - a UTC-built Date shifts a day west of UTC.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const [clause] = build([{ label: 'SELECTED', value: 24380 }], map)
|
||||
expect(clause.RAW_VALUE).toBe('(24380,24288)')
|
||||
})
|
||||
|
||||
it('reports a range mapping fed more than one value', () => {
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'RANGE' as const,
|
||||
period: 'QUARTER' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const problems = service.unsupportedTransforms(
|
||||
msg([{ label: 'SELECTED', value: ['01APR2026', '01JUL2026'] }]),
|
||||
map,
|
||||
[{ NAME: 'SOME_DATE', DDTYPE: 'DATE', FMTNAME: 'DATE9' }]
|
||||
)
|
||||
expect(problems).toHaveSize(1)
|
||||
expect(problems[0]).toContain('is a range')
|
||||
expect(problems[0]).toContain('2 values')
|
||||
})
|
||||
|
||||
it('refuses a range fed more than one value, rather than picking one', () => {
|
||||
// Two values, one of which does not parse: building a BETWEEN from the
|
||||
// survivor would silently filter one period when VA selected two.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'RANGE' as const,
|
||||
period: 'QUARTER' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
expect(
|
||||
build([{ label: 'SELECTED', value: ['01APR2026', 'not a date'] }], map)
|
||||
).toEqual([])
|
||||
})
|
||||
|
||||
it('never lets a non-finite date reach the clause', () => {
|
||||
// convertJsDateToSasDate returning NaN would otherwise put a literal NaN in
|
||||
// the WHERE clause.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
spyOn(helperService, 'convertJsDateToSasDate').and.returnValue(NaN as any)
|
||||
expect(build([{ label: 'SELECTED', value: '01OCT2026' }], map)).toEqual(
|
||||
[]
|
||||
)
|
||||
})
|
||||
|
||||
it('reports a partly unparseable value set instead of covering the rest', () => {
|
||||
// One good value and one that is not a date: the clause still filters on
|
||||
// the good one, so the author has to be told the other was dropped.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const problems = service.unsupportedTransforms(
|
||||
msg([{ label: 'SELECTED', value: ['01OCT2026', 'not a date'] }]),
|
||||
map,
|
||||
cols
|
||||
)
|
||||
expect(problems).toHaveSize(1)
|
||||
expect(problems[0]).toContain('1 of its 2 values was not a date')
|
||||
})
|
||||
|
||||
it('reports a range holding a value that does not parse', () => {
|
||||
// Counted on what VA sent, not on what parsed: otherwise the second value
|
||||
// vanishes and the range looks like a clean single-value case.
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'RANGE' as const,
|
||||
period: 'QUARTER' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const problems = service.unsupportedTransforms(
|
||||
msg([{ label: 'SELECTED', value: ['01APR2026', 'not a date'] }]),
|
||||
map,
|
||||
cols
|
||||
)
|
||||
expect(problems).toHaveSize(1)
|
||||
expect(problems[0]).toContain('is a range')
|
||||
expect(problems[0]).toContain('2 values')
|
||||
})
|
||||
|
||||
it('applies a transform on the deferred initial load too', () => {
|
||||
const map = new Map([
|
||||
[
|
||||
'selected',
|
||||
{
|
||||
column: 'SOME_DATE',
|
||||
transform: {
|
||||
op: 'ADD' as const,
|
||||
anchor: 'QUARTER_START' as const,
|
||||
offset: -1
|
||||
}
|
||||
}
|
||||
]
|
||||
])
|
||||
const [clause] = service.buildInitialClauses(
|
||||
msg([{ name: 'pr1', label: 'SELECTED', value: '01OCT2026' }]),
|
||||
map
|
||||
)
|
||||
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
|
||||
expect(clause.RAW_VALUE).toBe('(24380,24288)')
|
||||
})
|
||||
|
||||
it('behaves exactly as before with an empty map', () => {
|
||||
expect(
|
||||
service.buildClauses(
|
||||
msg([{ label: 'SOME_CHAR', value: 'x' }]),
|
||||
labelToVar(),
|
||||
cols
|
||||
)
|
||||
).toEqual(build([{ label: 'SOME_CHAR', value: 'x' }], new Map()))
|
||||
})
|
||||
})
|
||||
|
||||
describe('unresolvedMappings (what the report author is warned about)', () => {
|
||||
const report = (
|
||||
parameters: any[],
|
||||
map: Map<string, VaParmapTarget>,
|
||||
libds?: string
|
||||
) => service.unresolvedMappings(msg(parameters), map, cols, libds)
|
||||
|
||||
it('reports a mapping whose column is not in the loaded table', () => {
|
||||
expect(
|
||||
report(
|
||||
[{ label: 'REGION_TAB1', value: 'x' }],
|
||||
new Map([['region_tab1', { column: 'NOT_A_COLUMN' }]]),
|
||||
'DC_JSLIB.MPE_X_TEST'
|
||||
)
|
||||
).toEqual([
|
||||
'VA parameter "REGION_TAB1" is mapped to NOT_A_COLUMN, which is not a column of DC_JSLIB.MPE_X_TEST - the filter was ignored.'
|
||||
])
|
||||
})
|
||||
|
||||
it('says "this table" when the table name is not supplied', () => {
|
||||
expect(
|
||||
report(
|
||||
[{ label: 'REGION_TAB1', value: 'x' }],
|
||||
new Map([['region_tab1', { column: 'NOPE' }]])
|
||||
)[0]
|
||||
).toContain('is not a column of this table')
|
||||
})
|
||||
|
||||
it('reports nothing when the mapping resolves, or the param is unmapped', () => {
|
||||
expect(
|
||||
report(
|
||||
[{ label: 'REGION_TAB1', value: 'x' }],
|
||||
new Map([['region_tab1', { column: 'some_char' }]])
|
||||
)
|
||||
).toEqual([])
|
||||
expect(report([{ label: 'SOME_CHAR', value: 'x' }], new Map())).toEqual(
|
||||
[]
|
||||
)
|
||||
// A mapping whose parameter this message does not carry is not an error.
|
||||
expect(
|
||||
report([], new Map([['region_tab1', { column: 'NOPE' }]]))
|
||||
).toEqual([])
|
||||
})
|
||||
|
||||
it('reports each mapping once, not once per message parameter', () => {
|
||||
const problems = report(
|
||||
[
|
||||
{ label: 'REGION_TAB1', value: '1' },
|
||||
{ name: 'pr99', label: 'REGION_TAB1', value: '2' }
|
||||
],
|
||||
new Map([['region_tab1', { column: 'NOPE' }]])
|
||||
)
|
||||
expect(problems).toHaveSize(1)
|
||||
})
|
||||
|
||||
it('treats an unloaded column list as "not known yet", not as a bad mapping', () => {
|
||||
// While the grid is still loading there is no metadata to judge against, so
|
||||
// reporting would flash the banner on every load.
|
||||
expect(
|
||||
service.unresolvedMappings(
|
||||
msg([{ label: 'REGION_TAB1', value: 'x' }]),
|
||||
new Map([['region_tab1', { column: 'NOPE' }]]),
|
||||
[],
|
||||
'DC_JSLIB.MPE_X_TEST'
|
||||
)
|
||||
).toEqual([])
|
||||
expect(
|
||||
service.unresolvedMappings(
|
||||
msg([{ label: 'REGION_TAB1', value: 'x' }]),
|
||||
new Map([['region_tab1', { column: 'NOPE' }]]),
|
||||
undefined as any,
|
||||
'DC_JSLIB.MPE_X_TEST'
|
||||
)
|
||||
).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('sameColumnMappings (two parameters, one column)', () => {
|
||||
it('reports an AND-ed pair, naming both parameters and the column', () => {
|
||||
const map = new Map([
|
||||
['dropdown_tab1', { column: 'SOME_DROPDOWN' }],
|
||||
['dropdown_tab2', { column: 'SOME_DROPDOWN' }]
|
||||
])
|
||||
const problems = service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'DROPDOWN_TAB1', value: 'Option 1' },
|
||||
{ label: 'DROPDOWN_TAB2', value: 'Option 2' }
|
||||
]),
|
||||
map
|
||||
)
|
||||
expect(problems).toHaveSize(1)
|
||||
expect(problems[0]).toContain('"DROPDOWN_TAB1" and "DROPDOWN_TAB2"')
|
||||
expect(problems[0]).toContain('all filter SOME_DROPDOWN')
|
||||
expect(problems[0]).toContain('combined with AND')
|
||||
})
|
||||
|
||||
it('says nothing for one parameter per column', () => {
|
||||
const map = new Map([
|
||||
['dropdown_tab1', { column: 'SOME_DROPDOWN' }],
|
||||
['num_tab1', { column: 'SOME_NUM' }]
|
||||
])
|
||||
expect(
|
||||
service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'DROPDOWN_TAB1', value: 'Option 1' },
|
||||
{ label: 'NUM_TAB1', value: '1' }
|
||||
]),
|
||||
map
|
||||
)
|
||||
).toEqual([])
|
||||
expect(
|
||||
service.sameColumnMappings(
|
||||
msg([{ label: 'DROPDOWN_TAB1', value: 'Option 1' }]),
|
||||
map
|
||||
)
|
||||
).toEqual([])
|
||||
expect(service.sameColumnMappings(msg([]), map)).toEqual([])
|
||||
})
|
||||
|
||||
it('says nothing when the AND-ed parameters carry the same value', () => {
|
||||
// col = 'x' AND col = 'x' filters exactly as one clause does, so a banner
|
||||
// claiming "different values will show no rows" would be wrong.
|
||||
const map = new Map([
|
||||
['dropdown_tab1', { column: 'SOME_CHAR' }],
|
||||
['dropdown_tab2', { column: 'SOME_CHAR' }]
|
||||
])
|
||||
expect(
|
||||
service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'DROPDOWN_TAB1', value: 'x' },
|
||||
{ label: 'DROPDOWN_TAB2', value: 'x' }
|
||||
]),
|
||||
map,
|
||||
cols
|
||||
)
|
||||
).toEqual([])
|
||||
// One differing value and the intersection can be empty: reported.
|
||||
const problems = service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'DROPDOWN_TAB1', value: 'x' },
|
||||
{ label: 'DROPDOWN_TAB2', value: 'y' }
|
||||
]),
|
||||
map,
|
||||
cols
|
||||
)
|
||||
expect(problems).toHaveSize(1)
|
||||
expect(problems[0]).toContain('"DROPDOWN_TAB1" and "DROPDOWN_TAB2"')
|
||||
})
|
||||
|
||||
it('leaves a column the table does not have to unresolvedMappings', () => {
|
||||
// One cause, one line: the column is missing, so the AND story is moot.
|
||||
const map = new Map([
|
||||
['a', { column: 'NOPE' }],
|
||||
['b', { column: 'NOPE' }]
|
||||
])
|
||||
const cols = [
|
||||
{ LIBREF: 'DC_JSLIB', TABLE: 'MPE_X_TEST', VARIABLE: 'SOME_CHAR' }
|
||||
]
|
||||
expect(
|
||||
service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'A', value: '1' },
|
||||
{ label: 'B', value: '2' }
|
||||
]),
|
||||
map,
|
||||
cols
|
||||
)
|
||||
).toEqual([])
|
||||
// Without metadata the pair is still reported: the AND semantics hold
|
||||
// whatever the table turns out to be.
|
||||
expect(
|
||||
service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'A', value: '1' },
|
||||
{ label: 'B', value: '2' }
|
||||
]),
|
||||
map
|
||||
)
|
||||
).toHaveSize(1)
|
||||
})
|
||||
|
||||
it('reports one line per contended column', () => {
|
||||
const map = new Map([
|
||||
['a', { column: 'SOME_DROPDOWN' }],
|
||||
['b', { column: 'SOME_DROPDOWN' }],
|
||||
['c', { column: 'SOME_NUM' }],
|
||||
['d', { column: 'SOME_NUM' }]
|
||||
])
|
||||
const problems = service.sameColumnMappings(
|
||||
msg([
|
||||
{ label: 'A', value: '1' },
|
||||
{ label: 'B', value: '2' },
|
||||
{ label: 'C', value: '3' },
|
||||
{ label: 'D', value: '4' }
|
||||
]),
|
||||
map
|
||||
)
|
||||
expect(problems).toHaveSize(2)
|
||||
})
|
||||
})
|
||||
|
||||
describe('unmatchedMappings (parameter-side typo)', () => {
|
||||
const seen = (...keys: string[]) => new Set(keys)
|
||||
|
||||
it('reports a key that no message has named', () => {
|
||||
expect(
|
||||
service.unmatchedMappings(
|
||||
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
|
||||
seen('dropdown_tab_1', 'pr1'),
|
||||
true
|
||||
)
|
||||
).toEqual([
|
||||
'This object\'s va_parmap names VA parameter "dropdown_tab1", which VA is not sending - check the spelling.'
|
||||
])
|
||||
})
|
||||
|
||||
it('says nothing until a parameter has actually been seen', () => {
|
||||
// Without evidence that this object receives parameters at all, an absent
|
||||
// key proves nothing - VA sends only the parameters its query uses.
|
||||
expect(
|
||||
service.unmatchedMappings(
|
||||
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
|
||||
seen()
|
||||
)
|
||||
).toEqual([])
|
||||
expect(
|
||||
service.unmatchedMappings(
|
||||
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
|
||||
undefined
|
||||
)
|
||||
).toEqual([])
|
||||
})
|
||||
|
||||
it('says nothing when every key has been seen', () => {
|
||||
expect(
|
||||
service.unmatchedMappings(
|
||||
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
|
||||
seen('dropdown_tab1'),
|
||||
true
|
||||
)
|
||||
).toEqual([])
|
||||
expect(
|
||||
service.unmatchedMappings(new Map(), seen('anything'), true)
|
||||
).toEqual([])
|
||||
})
|
||||
|
||||
it('says nothing while the parameter set is still growing', () => {
|
||||
// A message that introduces a new parameter name means this object's set
|
||||
// has not settled, so an absent key may arrive in the next message.
|
||||
expect(
|
||||
service.unmatchedMappings(
|
||||
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
|
||||
seen('pr1', 'pr2'),
|
||||
false
|
||||
)
|
||||
).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildInitialClauses (no metadata, infer kind from value)', () => {
|
||||
it('uses an explicit mapping instead of the label', () => {
|
||||
// The deferred first load has no column metadata, so a mapped column is
|
||||
// used as given and the SAS kind still comes from the value shape.
|
||||
const map = new Map([
|
||||
['region_tab1', { column: 'REGION' }],
|
||||
['amt_tab1', { column: 'AMOUNT' }]
|
||||
])
|
||||
const clauses = service.buildInitialClauses(
|
||||
msg([
|
||||
{ name: 'pr1', label: 'REGION_TAB1', dataType: 'string', value: 'n' },
|
||||
{ name: 'pr2', label: 'AMT_TAB1', dataType: 'number', value: '10' }
|
||||
]),
|
||||
map
|
||||
)
|
||||
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
|
||||
['REGION', "'n'"],
|
||||
['AMOUNT', '10']
|
||||
])
|
||||
})
|
||||
|
||||
it('falls back to the label for params the map does not name', () => {
|
||||
const map = new Map([['region_tab2', { column: 'REGION' }]])
|
||||
const clauses = service.buildInitialClauses(
|
||||
msg([
|
||||
{ name: 'pr1', label: 'REGION_TAB1', value: 'n' },
|
||||
{ name: 'pr2', label: 'REGION_TAB2', value: 's' }
|
||||
]),
|
||||
map
|
||||
)
|
||||
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
|
||||
['REGION_TAB1', "'n'"],
|
||||
['REGION', "'s'"]
|
||||
])
|
||||
})
|
||||
|
||||
it('uses the VA label as the column and infers kinds by value shape', () => {
|
||||
const clauses = service.buildInitialClauses(
|
||||
msg([
|
||||
|
||||
@@ -2,6 +2,12 @@ import { Injectable } from '@angular/core'
|
||||
import { HelperService } from './helper.service'
|
||||
import { VaMessage } from './va-messaging.service'
|
||||
import { QueryClause } from '../models/TableData'
|
||||
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
|
||||
import {
|
||||
applyVaTransform,
|
||||
isRangeTransform,
|
||||
VaTransform
|
||||
} from '../shared/utils/va-parmap-transform'
|
||||
|
||||
/** SAS data-type kind used to format a VA filter value into a DC RAW_VALUE. */
|
||||
type VaColumnKind = 'char' | 'numeric' | 'time' | 'date' | 'datetime'
|
||||
@@ -54,11 +60,19 @@ export class VaFilterService {
|
||||
* Translates VA `parameters` into DC filter clauses. Each parameter is matched
|
||||
* to a DC column by label; a single value becomes `col = value`, multiple
|
||||
* values become `col IN (...)`. Values are formatted for the column's SAS kind.
|
||||
*
|
||||
* `paramMap` (from the URL `va_parmap` parameter, see parseVaParmap) overrides
|
||||
* that same-name resolution for the parameters it names, so two DC instances
|
||||
* over one table can filter the same column from differently-named VA
|
||||
* parameters. A mapped column must exist in `cols` - if it does not the
|
||||
* parameter is skipped rather than falling back to the same-name match, which
|
||||
* is exactly the behaviour the mapping exists to override.
|
||||
*/
|
||||
buildClauses(
|
||||
msg: VaMessage,
|
||||
labelToVar: Map<string, string>,
|
||||
cols: any[]
|
||||
cols: any[],
|
||||
paramMap?: Map<string, VaParmapTarget>
|
||||
): QueryClause[] {
|
||||
const clauses: QueryClause[] = []
|
||||
for (const param of msg.parameters || []) {
|
||||
@@ -67,60 +81,194 @@ export class VaFilterService {
|
||||
.toString()
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
// A VA param whose label isn't a DC column (beyond case, which is handled)
|
||||
// simply doesn't map — skip it.
|
||||
const varName = labelToVar.get(key)
|
||||
// An explicit URL mapping wins over the label->column lookup. The mapped
|
||||
// column has to exist in THIS table, else the entry does not apply here
|
||||
// (and the same-name fallback would be the behaviour it exists to
|
||||
// override).
|
||||
const mapped = this.mappedColumn(param, paramMap)
|
||||
const mappedCol = mapped ? this.findCol(mapped, cols) : undefined
|
||||
if (mapped && !mappedCol) continue
|
||||
// The table's own case, so the clause matches what the metadata build
|
||||
// produces for a same-name match.
|
||||
const varName: string = mappedCol
|
||||
? mappedCol.NAME
|
||||
: (labelToVar.get(key) ?? '')
|
||||
if (!varName) continue
|
||||
|
||||
const col = this.findCol(varName, cols)
|
||||
const col = mappedCol ?? this.findCol(varName, cols)
|
||||
const kind = this.columnKind(col)
|
||||
const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
|
||||
const rawValues = Array.isArray(param.value) ? param.value : [param.value]
|
||||
// Format/validate every value for the column's SAS type. Numeric columns
|
||||
// must receive a finite number (untrusted postMessage input is
|
||||
// interpolated into a SAS WHERE clause, so a non-numeric value here would
|
||||
// be an injection vector); temporal -> SAS internal number; char ->
|
||||
// escaped. Invalid values are dropped.
|
||||
const formatted = rawValues
|
||||
.filter((v) => v !== undefined && v !== null && v !== '')
|
||||
.map((v) => this.formatValueByKind(v, kind, order))
|
||||
.filter((v): v is string => v !== null)
|
||||
if (formatted.length === 0) continue
|
||||
clauses.push(this.clause(varName, formatted))
|
||||
const built = this.buildClauseValueSet(
|
||||
param,
|
||||
kind,
|
||||
order,
|
||||
this.mappedTarget(param, paramMap)?.transform
|
||||
)
|
||||
if (!built) continue
|
||||
clauses.push({
|
||||
GROUP_LOGIC: 'AND',
|
||||
SUBGROUP_LOGIC: 'AND',
|
||||
SUBGROUP_ID: 0,
|
||||
VARIABLE_NM: varName,
|
||||
OPERATOR_NM: built.operator,
|
||||
RAW_VALUE: built.rawValue
|
||||
})
|
||||
}
|
||||
return clauses
|
||||
}
|
||||
|
||||
/**
|
||||
* The operator and raw value for one parameter, after any transform.
|
||||
*
|
||||
* Values are formatted for the column's SAS kind: numeric columns must receive
|
||||
* a finite number (untrusted postMessage input is interpolated into a WHERE
|
||||
* clause, so anything else would be an injection vector); temporal values
|
||||
* become SAS internal numbers; char values are escaped. Invalid values are
|
||||
* dropped, and a transform that cannot be applied to the column's kind yields
|
||||
* nothing at all - the caller reports that separately rather than filtering on
|
||||
* a value the author did not choose.
|
||||
*/
|
||||
private buildClauseValueSet(
|
||||
param: any,
|
||||
kind: VaColumnKind,
|
||||
order: 'YMD' | 'DMY' | 'MDY' | undefined,
|
||||
transform?: VaTransform
|
||||
): { operator: '=' | 'IN' | 'BETWEEN'; rawValue: string } | null {
|
||||
const rawValues = (
|
||||
Array.isArray(param.value) ? param.value : [param.value]
|
||||
).filter((v: any) => v !== undefined && v !== null && v !== '')
|
||||
if (rawValues.length === 0) return null
|
||||
|
||||
if (!transform) {
|
||||
const formatted = rawValues
|
||||
.map((v: any) => this.formatValueByKind(v, kind, order))
|
||||
.filter((v: string | null): v is string => v !== null)
|
||||
if (formatted.length === 0) return null
|
||||
return formatted.length === 1
|
||||
? { operator: '=', rawValue: formatted[0] }
|
||||
: { operator: 'IN', rawValue: `(${formatted.join(',')})` }
|
||||
}
|
||||
|
||||
// Transforms resolve to dates, so only temporal columns can take one.
|
||||
if (kind !== 'date' && kind !== 'datetime') return null
|
||||
const unit = kind === 'datetime' ? 'seconds' : 'days'
|
||||
const values = new Set<string>()
|
||||
for (const raw of rawValues) {
|
||||
const resolved = this.valueAsDate(raw, kind, order)
|
||||
if (!resolved) continue
|
||||
for (const date of applyVaTransform(
|
||||
resolved,
|
||||
transform,
|
||||
kind === 'datetime'
|
||||
)) {
|
||||
// A non-finite result (an Invalid Date reaching the helper) must never be
|
||||
// interpolated into the clause: `NaN` is not a SAS value. Dropped here,
|
||||
// and reported by unsupportedTransforms so the parameter is not silently
|
||||
// filtered on nothing.
|
||||
const sasDate = this.helperService.convertJsDateToSasDate(date, unit)
|
||||
if (!Number.isFinite(Number(sasDate))) continue
|
||||
values.add(`${sasDate}`)
|
||||
}
|
||||
}
|
||||
const formatted = Array.from(values)
|
||||
if (formatted.length === 0) return null
|
||||
if (isRangeTransform(transform)) {
|
||||
// One BETWEEN carries one period, so more than one value has no clause -
|
||||
// dropping it here (rather than building a BETWEEN from whichever value
|
||||
// parsed) is what unsupportedTransforms' count warning then describes.
|
||||
if (rawValues.length > 1) return null
|
||||
if (formatted.length !== 2) return null
|
||||
return { operator: 'BETWEEN', rawValue: formatted.join(' AND ') }
|
||||
}
|
||||
return formatted.length === 1
|
||||
? { operator: '=', rawValue: formatted[0] }
|
||||
: { operator: 'IN', rawValue: `(${formatted.join(',')})` }
|
||||
}
|
||||
|
||||
/**
|
||||
* A VA value as a local Date, for a transform to work from. Formatted strings
|
||||
* are parsed by shape; a bare number is already the SAS internal value, so it
|
||||
* is converted back from days (date) or seconds (datetime) since 1960-01-01.
|
||||
*
|
||||
* Built with local calendar construction, matching `parseDate` and
|
||||
* `HelperService.convertJsDateToSasDate`: a UTC instant read back through local
|
||||
* accessors lands on the previous day west of UTC, and the round trip to a SAS
|
||||
* day number would be one out.
|
||||
*/
|
||||
private valueAsDate(
|
||||
value: any,
|
||||
kind: VaColumnKind,
|
||||
order?: 'YMD' | 'DMY' | 'MDY'
|
||||
): Date | null {
|
||||
const s = String(value).trim()
|
||||
if (s === '') return null
|
||||
if (this.isBareNumber(s)) {
|
||||
const internal = Number(s)
|
||||
if (kind !== 'datetime') return new Date(1960, 0, 1 + internal)
|
||||
// Seconds since midnight 1960-01-01, split into whole days and a remainder
|
||||
// so the constructor normalises the clock time locally.
|
||||
const days = Math.floor(internal / 86400)
|
||||
const seconds = internal - days * 86400
|
||||
return new Date(1960, 0, 1 + days, 0, 0, seconds)
|
||||
}
|
||||
return kind === 'datetime'
|
||||
? this.parseDatetime(s, order)
|
||||
: this.parseDate(s, order)
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-getdata variant used on the deferred initial load, when DC column
|
||||
* metadata isn't available. Uses the VA `label` as the column name and infers
|
||||
* the SAS kind from the value shape, so the clause matches what the metadata
|
||||
* build later produces (no reconcile reload).
|
||||
*
|
||||
* `paramMap` overrides the label-as-column rule for the parameters it names,
|
||||
* so the very first (already filtered) load also honours an explicit mapping.
|
||||
*/
|
||||
buildInitialClauses(msg: VaMessage): QueryClause[] {
|
||||
buildInitialClauses(
|
||||
msg: VaMessage,
|
||||
paramMap?: Map<string, VaParmapTarget>
|
||||
): QueryClause[] {
|
||||
const clauses: QueryClause[] = []
|
||||
for (const param of msg.parameters || []) {
|
||||
if (!param || param.value === undefined || param.value === null) continue
|
||||
// The DC column name is the VA `label`; `name` is VA's internal id (pr###).
|
||||
const varName = (param.label ?? param.name ?? '').toString().trim()
|
||||
const target = this.mappedTarget(param, paramMap)
|
||||
// Without a mapping the DC column name is the VA `label`; `name` is VA's
|
||||
// internal id (pr###). A `va_parmap` entry wins over both.
|
||||
const varName = (target?.column ?? param.label ?? param.name ?? '')
|
||||
.toString()
|
||||
.trim()
|
||||
if (!varName) continue
|
||||
|
||||
const rawValues = Array.isArray(param.value) ? param.value : [param.value]
|
||||
const rawValues = (
|
||||
Array.isArray(param.value) ? param.value : [param.value]
|
||||
) as any[]
|
||||
const nonEmpty = rawValues.filter(
|
||||
(v) => v !== undefined && v !== null && v !== ''
|
||||
)
|
||||
if (nonEmpty.length === 0) continue
|
||||
|
||||
const formatted = nonEmpty
|
||||
.map((v) =>
|
||||
this.formatValueByKind(
|
||||
v,
|
||||
this.inferKindFromValue(String(v).trim(), param.dataType)
|
||||
// A transform only ever produces dates, so it fixes the kind: inferring it
|
||||
// from the value would see a formatted date either way, but with the
|
||||
// transform the column is known to be temporal.
|
||||
const kind = target?.transform
|
||||
? ('date' as VaColumnKind)
|
||||
: this.inferKindFromValue(String(nonEmpty[0]).trim(), param.dataType)
|
||||
const built = this.buildClauseValueSet(
|
||||
param,
|
||||
kind,
|
||||
undefined,
|
||||
target?.transform
|
||||
)
|
||||
)
|
||||
.filter((v): v is string => v !== null)
|
||||
if (formatted.length === 0) continue
|
||||
clauses.push(this.clause(varName, formatted))
|
||||
if (!built) continue
|
||||
clauses.push({
|
||||
GROUP_LOGIC: 'AND',
|
||||
SUBGROUP_LOGIC: 'AND',
|
||||
SUBGROUP_ID: 0,
|
||||
VARIABLE_NM: varName,
|
||||
OPERATOR_NM: built.operator,
|
||||
RAW_VALUE: built.rawValue
|
||||
})
|
||||
}
|
||||
return clauses
|
||||
}
|
||||
@@ -141,16 +289,254 @@ export class VaFilterService {
|
||||
)
|
||||
}
|
||||
|
||||
private clause(varName: string, formatted: string[]): QueryClause {
|
||||
return {
|
||||
GROUP_LOGIC: 'AND',
|
||||
SUBGROUP_LOGIC: 'AND',
|
||||
SUBGROUP_ID: 0,
|
||||
VARIABLE_NM: varName,
|
||||
OPERATOR_NM: formatted.length === 1 ? '=' : 'IN',
|
||||
RAW_VALUE:
|
||||
formatted.length === 1 ? formatted[0] : `(${formatted.join(',')})`
|
||||
/**
|
||||
* Transform mappings in this message that cannot be applied, one plain-English
|
||||
* line each: the column is not temporal, or the value is not a date the browser
|
||||
* can parse. `buildClauseValueSet` skips those parameters, so this is what stops
|
||||
* the misconfiguration reading as a filter that simply matched nothing.
|
||||
*
|
||||
* A missing column is left to `unresolvedMappings`, which already reports it.
|
||||
*/
|
||||
unsupportedTransforms(
|
||||
msg: VaMessage,
|
||||
paramMap: Map<string, VaParmapTarget> | undefined,
|
||||
cols: any[]
|
||||
): string[] {
|
||||
if (!paramMap || paramMap.size === 0) return []
|
||||
if (!cols || cols.length === 0) return []
|
||||
const problems = new Set<string>()
|
||||
for (const param of msg.parameters || []) {
|
||||
const target = this.mappedTarget(param, paramMap)
|
||||
if (!target?.transform) continue
|
||||
const transform = target.transform
|
||||
const col = this.findCol(target.column, cols)
|
||||
if (!col) continue
|
||||
const kind = this.columnKind(col)
|
||||
const paramName = (param.label ?? param.name ?? '').toString().trim()
|
||||
const spec =
|
||||
target.transform.op === 'ADD'
|
||||
? `ADD(${target.transform.anchor},${target.transform.offset})`
|
||||
: `RANGE(${target.transform.period},${target.transform.offset})`
|
||||
if (kind !== 'date' && kind !== 'datetime') {
|
||||
problems.add(
|
||||
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${target.column} is not a date or datetime column - the filter was ignored.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
|
||||
const sent = (
|
||||
Array.isArray(param.value) ? param.value : [param.value]
|
||||
).filter((v: any) => v !== undefined && v !== null && v !== '')
|
||||
const usable = sent.filter((v: any) => {
|
||||
const date = this.valueAsDate(v, kind, order)
|
||||
if (!date) return false
|
||||
// Same test the clause builder applies, so this count matches what it uses.
|
||||
return applyVaTransform(date, transform, kind === 'datetime').every(
|
||||
(resolved) => {
|
||||
const sasDate = this.helperService.convertJsDateToSasDate(
|
||||
resolved,
|
||||
kind === 'datetime' ? 'seconds' : 'days'
|
||||
)
|
||||
return Number.isFinite(Number(sasDate))
|
||||
}
|
||||
)
|
||||
})
|
||||
if (usable.length === 0) {
|
||||
problems.add(
|
||||
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but its value was not a date - the filter was ignored.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
// One BETWEEN can only express one period. A multi-select would have to
|
||||
// become an OR of ranges, which the clause model does not carry, so say so
|
||||
// rather than building a BETWEEN from whichever value happened to parse.
|
||||
// Counted on what VA sent: a second value that does not parse must not make
|
||||
// the single-value case look clean.
|
||||
if (isRangeTransform(target.transform) && sent.length > 1) {
|
||||
problems.add(
|
||||
`VA parameter "${paramName}" maps to ${target.column}:${spec}, which is a range, but the parameter has ${sent.length} values - a range needs one value, so the filter was ignored.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
if (usable.length < sent.length) {
|
||||
problems.add(
|
||||
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${sent.length - usable.length} of its ${sent.length} values was not a date - the filter covers only the rest.`
|
||||
)
|
||||
}
|
||||
}
|
||||
return Array.from(problems)
|
||||
}
|
||||
|
||||
/**
|
||||
* `va_parmap` entries that this message actually uses but that name a column
|
||||
* the loaded table does not have, one plain-English line each. `buildClauses`
|
||||
* skips those parameters (deliberately: falling back to the same-name match
|
||||
* would silently apply the filter the mapping exists to override), so this is
|
||||
* what makes the misconfiguration visible to the report author - otherwise the
|
||||
* grid simply shows every row, which looks like a working filter.
|
||||
*/
|
||||
unresolvedMappings(
|
||||
msg: VaMessage,
|
||||
paramMap: Map<string, VaParmapTarget> | undefined,
|
||||
cols: any[],
|
||||
libds?: string
|
||||
): string[] {
|
||||
const problems = new Set<string>()
|
||||
if (!paramMap || paramMap.size === 0) return []
|
||||
// No metadata yet is not the same as "the table lacks this column": an empty
|
||||
// `cols` means the grid is still loading, so there is nothing to judge yet.
|
||||
// Reporting here would flash a misconfiguration banner during every load.
|
||||
if (!cols || cols.length === 0) return []
|
||||
for (const param of msg.parameters || []) {
|
||||
const mapped = this.mappedColumn(param, paramMap)
|
||||
if (!mapped || this.findCol(mapped, cols)) continue
|
||||
const paramName = (param.label ?? param.name ?? '').toString().trim()
|
||||
problems.add(
|
||||
`VA parameter "${paramName}" is mapped to ${mapped}, which is not a column of ${
|
||||
libds || 'this table'
|
||||
} - the filter was ignored.`
|
||||
)
|
||||
}
|
||||
return Array.from(problems)
|
||||
}
|
||||
|
||||
/**
|
||||
* `va_parmap` keys that no VA message has named, one plain-English line each.
|
||||
* A key that matches nothing is a parameter-side typo: the mapping never
|
||||
* applies and the grid stays unfiltered, which looks the same as a working
|
||||
* filter.
|
||||
*
|
||||
* `seenKeys` must be every key observed across messages so far. `paramsStable`
|
||||
* says the latest message introduced no new parameter name, i.e. this object's
|
||||
* parameter set has settled - until then a missing name may simply not have
|
||||
* arrived, because VA sends only the parameters its query uses.
|
||||
*/
|
||||
unmatchedMappings(
|
||||
paramMap: Map<string, VaParmapTarget> | undefined,
|
||||
seenKeys: Set<string> | undefined,
|
||||
paramsStable = false
|
||||
): string[] {
|
||||
if (!paramMap || paramMap.size === 0) return []
|
||||
if (!seenKeys || seenKeys.size === 0) return []
|
||||
if (!paramsStable) return []
|
||||
const problems: string[] = []
|
||||
for (const key of Array.from(paramMap.keys())) {
|
||||
if (!seenKeys.has(key)) {
|
||||
problems.push(
|
||||
`This object's va_parmap names VA parameter "${key}", which VA is not sending - check the spelling.`
|
||||
)
|
||||
}
|
||||
}
|
||||
return problems
|
||||
}
|
||||
|
||||
/**
|
||||
* Parameters in this message that filter the same column with *different*
|
||||
* values, one plain-English line per column. Clauses are AND-ed, so two
|
||||
* parameters on one column are an intersection: with different values the grid
|
||||
* comes back empty, which looks like a working filter. Two parameters carrying
|
||||
* the same value intersect cleanly and filter correctly, so they are not
|
||||
* reported - a banner there would call a working configuration broken.
|
||||
*
|
||||
* Reported rather than rewritten - folding them into one `IN (...)` would
|
||||
* silently turn an AND into an OR, which is a semantics decision, not a
|
||||
* diagnostic.
|
||||
*/
|
||||
sameColumnMappings(
|
||||
msg: VaMessage,
|
||||
paramMap: Map<string, VaParmapTarget> | undefined,
|
||||
cols?: any[]
|
||||
): string[] {
|
||||
if (!paramMap || paramMap.size === 0) return []
|
||||
// column -> the parameters on it, each with the clause value it produces
|
||||
const byColumn = new Map<string, { name: string; value: string }[]>()
|
||||
for (const param of msg.parameters || []) {
|
||||
const mapped = this.mappedColumn(param, paramMap)
|
||||
if (!mapped) continue
|
||||
// A column the table lacks is unresolvedMappings' finding; counting it here
|
||||
// would report one cause as two symptoms.
|
||||
const col =
|
||||
cols && cols.length > 0 ? this.findCol(mapped, cols) : undefined
|
||||
if (cols && cols.length > 0 && !col) continue
|
||||
// A parameter that produces no clause cannot conflict with anything.
|
||||
const value = this.valueSignature(
|
||||
param,
|
||||
col,
|
||||
this.mappedTarget(param, paramMap)?.transform
|
||||
)
|
||||
if (value === null) continue
|
||||
const column = mapped.toUpperCase()
|
||||
const params = byColumn.get(column) || []
|
||||
const paramName = (param.label ?? '').toString().trim()
|
||||
if (paramName && !params.some((p) => p.name === paramName)) {
|
||||
params.push({ name: paramName, value })
|
||||
}
|
||||
byColumn.set(column, params)
|
||||
}
|
||||
const problems: string[] = []
|
||||
for (const [column, params] of Array.from(byColumn.entries())) {
|
||||
if (params.length < 2) continue
|
||||
// Same value on every parameter: the AND is a no-op, not an empty grid.
|
||||
const distinct = new Set(params.map((p) => p.value))
|
||||
if (distinct.size === 1) continue
|
||||
problems.push(
|
||||
`VA parameters ${params
|
||||
.map((p) => `"${p.name}"`)
|
||||
.join(
|
||||
' and '
|
||||
)} all filter ${column}. Parameters are combined with AND, so rows must match all of them - different values will show no rows. Use one parameter carrying both values to match either.`
|
||||
)
|
||||
}
|
||||
return problems
|
||||
}
|
||||
|
||||
/**
|
||||
* The DC column an explicit URL mapping (`va_parmap`) assigns to this
|
||||
* parameter, or undefined when the parameter is not mapped. The key is matched
|
||||
* case-insensitively against the parameter's `label`.
|
||||
*/
|
||||
private mappedTarget(
|
||||
param: any,
|
||||
paramMap?: Map<string, VaParmapTarget>
|
||||
): VaParmapTarget | undefined {
|
||||
if (!paramMap || paramMap.size === 0) return undefined
|
||||
// The label only. The internal name (pr273) is an implementation detail of
|
||||
// the report, and matching both made one parameter answer to two keys: a
|
||||
// mapping written against the name could never be distinguished from a typo
|
||||
// once the label was also mapped, and the second entry silently lost.
|
||||
const key = (param.label ?? '').toString().trim().toLowerCase()
|
||||
if (!key) return undefined
|
||||
return paramMap.get(key)
|
||||
}
|
||||
|
||||
/**
|
||||
* The clause value a parameter would filter its column with, as a comparable
|
||||
* string, or null when it produces no clause at all. Used to tell an AND of two
|
||||
* identical values (harmless) from an AND of two different ones (empty grid).
|
||||
*
|
||||
* Delegates to the clause builder so a transform's derived dates take part in
|
||||
* the comparison: two parameters adding the same quarter are the same clause.
|
||||
*/
|
||||
private valueSignature(
|
||||
param: any,
|
||||
col: any,
|
||||
transform?: VaTransform
|
||||
): string | null {
|
||||
const built = this.buildClauseValueSet(
|
||||
param,
|
||||
this.columnKind(col),
|
||||
this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT),
|
||||
transform
|
||||
)
|
||||
return built ? built.rawValue : null
|
||||
}
|
||||
|
||||
/** The mapped column name, for callers that do not need the transform. */
|
||||
private mappedColumn(
|
||||
param: any,
|
||||
paramMap?: Map<string, VaParmapTarget>
|
||||
): string | undefined {
|
||||
return this.mappedTarget(param, paramMap)?.column
|
||||
}
|
||||
|
||||
/** The loaded column spec whose NAME matches varName (case-insensitive). */
|
||||
@@ -321,14 +707,42 @@ export class VaFilterService {
|
||||
return this.parseMonthNameDate(value) ?? this.parseNumericDate(value, order)
|
||||
}
|
||||
|
||||
/** DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62) -> local Date, or null. */
|
||||
/**
|
||||
* Month-name date shapes -> local Date, or null. Three families:
|
||||
* - DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62);
|
||||
* - word date with the month first, comma optional: "January 1, 2026",
|
||||
* "Jan 1 2026", "Jan. 1, 2026";
|
||||
* - word date with the day first: "1 January 2026", "1-Jan-2026".
|
||||
*
|
||||
* A date parameter's display format decides which of these the report author
|
||||
* sees, and VA sends the formatted text, so a report using WORDDATE (or a
|
||||
* long month name in any order) has to parse here rather than being dropped.
|
||||
* The month name may be abbreviated or spelled out; only its first three
|
||||
* letters are matched.
|
||||
*/
|
||||
private parseMonthNameDate(value: string): Date | null {
|
||||
const m = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
|
||||
if (!m) return null
|
||||
const [, day, monthAbbr, year] = m
|
||||
const monthIndex = VaFilterService.SAS_MONTHS.indexOf(
|
||||
monthAbbr.toUpperCase()
|
||||
const monthByName = (name: string): number =>
|
||||
VaFilterService.SAS_MONTHS.indexOf(
|
||||
name.replace(/\./g, '').slice(0, 3).toUpperCase()
|
||||
)
|
||||
|
||||
const date9 = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
|
||||
if (date9) {
|
||||
const [, day, monthAbbr, year] = date9
|
||||
const monthIndex = monthByName(monthAbbr)
|
||||
if (monthIndex < 0) return null
|
||||
return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
|
||||
}
|
||||
|
||||
const monthFirst =
|
||||
/^([A-Za-z]{3,9})\.?[\s-]+(\d{1,2}),?[\s-]+(\d{2}|\d{4})$/.exec(value)
|
||||
const dayFirst =
|
||||
/^(\d{1,2})[\s-]+([A-Za-z]{3,9})\.?,?[\s-]+(\d{2}|\d{4})$/.exec(value)
|
||||
const name = monthFirst ? monthFirst[1] : dayFirst ? dayFirst[2] : null
|
||||
if (!name) return null
|
||||
const day = monthFirst ? monthFirst[2] : dayFirst![1]
|
||||
const year = monthFirst ? monthFirst[3] : dayFirst![3]
|
||||
const monthIndex = monthByName(name)
|
||||
if (monthIndex < 0) return null
|
||||
return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
|
||||
}
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { VaMessagingService, VaMessage } from './va-messaging.service'
|
||||
import { VaFilterService } from './va-filter.service'
|
||||
import { HelperService } from './helper.service'
|
||||
|
||||
// Sample message taken from the official SAS DDC docs (brush column included).
|
||||
const SAMPLE = {
|
||||
@@ -51,6 +53,57 @@ describe('VaMessagingService', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('parameter keys (what the va_parmap typo diagnostic is judged on)', () => {
|
||||
const PARAMS = [
|
||||
{ name: 'pr273', label: 'Tab_1_Date', value: '2025-03-31' },
|
||||
{ name: 'pr274', label: 'Tab 2 Region', value: 'north' }
|
||||
]
|
||||
const withParams = () => ({ ...SAMPLE, parameters: PARAMS })
|
||||
|
||||
it('records each parameter label once, across messages', () => {
|
||||
service.parseMessage({ data: withParams() } as MessageEvent)
|
||||
service.parseMessage({ data: withParams() } as MessageEvent)
|
||||
|
||||
expect(Array.from(service.seenParamKeys)).toEqual([
|
||||
'tab_1_date',
|
||||
'tab 2 region'
|
||||
])
|
||||
// The second message introduced no new label, so the set has settled.
|
||||
expect(service.lastMessageParamsStable).toBeTrue()
|
||||
})
|
||||
|
||||
it('records the label only, so a mapping written against the internal name is reported', () => {
|
||||
// `pr273` is VA's own bookkeeping. A va_parmap entry written against it
|
||||
// never matches a parameter, and recording the name here would make the
|
||||
// typo diagnostic believe VA is sending it - leaving the author with an
|
||||
// unfiltered grid and no warning at all.
|
||||
service.parseMessage({ data: withParams() } as MessageEvent)
|
||||
service.parseMessage({ data: withParams() } as MessageEvent)
|
||||
|
||||
const filter = new VaFilterService(new HelperService({} as any))
|
||||
const seen = service.seenParamKeys
|
||||
const stable = service.lastMessageParamsStable
|
||||
|
||||
expect(
|
||||
filter.unmatchedMappings(
|
||||
new Map([['pr273', { column: 'D_DATE' }]]),
|
||||
seen,
|
||||
stable
|
||||
)
|
||||
).toEqual([
|
||||
'This object\'s va_parmap names VA parameter "pr273", which VA is not sending - check the spelling.'
|
||||
])
|
||||
// The label is a real key, so the same map written that way is silent.
|
||||
expect(
|
||||
filter.unmatchedMappings(
|
||||
new Map([['tab_1_date', { column: 'D_DATE' }]]),
|
||||
seen,
|
||||
stable
|
||||
)
|
||||
).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('dataColumns', () => {
|
||||
it('excludes brush columns and keeps original indices', () => {
|
||||
const msg = service.parseMessage({ data: SAMPLE } as MessageEvent)!
|
||||
|
||||
@@ -82,6 +82,26 @@ export class VaMessagingService {
|
||||
/** DC variable names that should be visible (the rest are hidden). */
|
||||
visibleColumns: string[] | null = null
|
||||
|
||||
/**
|
||||
* Lower-cased label of every parameter any VA message has carried so far, kept
|
||||
* across editor reloads. Lets the editor tell a `va_parmap` key that names no
|
||||
* parameter in the report (a typo, so the filter would silently never apply)
|
||||
* from one that is merely absent from the current message - VA only sends the
|
||||
* parameters its query uses.
|
||||
*
|
||||
* Labels only: that is the key a mapping is matched on, and recording the
|
||||
* internal name as well would make a name-keyed entry - which never applies -
|
||||
* look like a parameter VA is sending.
|
||||
*/
|
||||
seenParamKeys = new Set<string>()
|
||||
/**
|
||||
* True when every parameter name in the latest message had already been seen.
|
||||
* VA sends only the parameters a report object's query uses, and sends them per
|
||||
* object, so a name that is absent from the first messages may simply not have
|
||||
* arrived yet. A stable parameter set is what makes absence meaningful.
|
||||
*/
|
||||
lastMessageParamsStable = false
|
||||
|
||||
/** True once the pre-bootstrap early buffer has been drained (drain-once). */
|
||||
private earlyDrained = false
|
||||
|
||||
@@ -187,6 +207,22 @@ export class VaMessagingService {
|
||||
if (typeof data.resultName !== 'string') return null
|
||||
if (!Array.isArray(data.columns) || !Array.isArray(data.data)) return null
|
||||
|
||||
const parameters = Array.isArray(data.parameters) ? data.parameters : []
|
||||
const keys: string[] = []
|
||||
for (const param of parameters) {
|
||||
// The label only, which is the key a `va_parmap` entry is matched on. The
|
||||
// internal name (`pr273`) is the report's own bookkeeping: recording it
|
||||
// here would let the typo diagnostic below believe a name-keyed entry is
|
||||
// being sent when no code path can ever apply it.
|
||||
const key = (param?.label ?? '').toString().trim().toLowerCase()
|
||||
if (key && !keys.includes(key)) keys.push(key)
|
||||
}
|
||||
// Judged before the keys are recorded: a message that introduces a new name
|
||||
// means the set is still growing, so a missing name proves nothing yet.
|
||||
this.lastMessageParamsStable =
|
||||
keys.length > 0 && keys.every((key) => this.seenParamKeys.has(key))
|
||||
for (const key of keys) this.seenParamKeys.add(key)
|
||||
|
||||
return {
|
||||
version: data.version,
|
||||
resultName: data.resultName,
|
||||
@@ -194,7 +230,7 @@ export class VaMessagingService {
|
||||
availableRowCount: data.availableRowCount,
|
||||
data: data.data,
|
||||
columns: data.columns,
|
||||
parameters: Array.isArray(data.parameters) ? data.parameters : []
|
||||
parameters
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
import { parseVaParmap } from './parse-va-parmap'
|
||||
|
||||
describe('parseVaParmap', () => {
|
||||
it('returns an empty map and no problems when the parameter is absent', () => {
|
||||
for (const query of ['', 'embed=va', 'embed=va&labels=true']) {
|
||||
const parsed = parseVaParmap(query)
|
||||
expect(parsed.map.size).toBe(0)
|
||||
expect(parsed.problems).toEqual([])
|
||||
}
|
||||
})
|
||||
|
||||
it('maps a VA parameter label to a DC column', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'embed=va&va_parmap=REGION_TAB1-REGION'
|
||||
)
|
||||
expect(map.get('region_tab1')?.column).toBe('REGION')
|
||||
expect(map.size).toBe(1)
|
||||
expect(problems).toEqual([])
|
||||
})
|
||||
|
||||
it('collects repeated entries, so two parameters can map to one column', () => {
|
||||
// The multi-tab case: each DC instance filters the same DC column from a
|
||||
// differently-named VA parameter.
|
||||
const { map } = parseVaParmap(
|
||||
'embed=va&va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION'
|
||||
)
|
||||
expect(map.get('region_tab1')?.column).toBe('REGION')
|
||||
expect(map.get('region_tab2')?.column).toBe('REGION')
|
||||
expect(map.size).toBe(2)
|
||||
})
|
||||
|
||||
it('is case-insensitive on the key and trims both sides', () => {
|
||||
const { map } = parseVaParmap('va_parmap=%20Region_Tab1%20-%20region%20')
|
||||
expect(map.get('region_tab1')?.column).toBe('region')
|
||||
})
|
||||
|
||||
it('decodes a key containing a space (VA labels are free text)', () => {
|
||||
expect(
|
||||
parseVaParmap('va_parmap=My+Region-DEPT').map.get('my region')?.column
|
||||
).toBe('DEPT')
|
||||
})
|
||||
|
||||
it('splits at the LAST hyphen, so a label may contain hyphens', () => {
|
||||
expect(
|
||||
parseVaParmap('va_parmap=Sales-2024-DEPT').map.get('sales-2024')?.column
|
||||
).toBe('DEPT')
|
||||
})
|
||||
|
||||
describe('entries that cannot be used', () => {
|
||||
it('reports an entry with no separator', () => {
|
||||
const { map, problems } = parseVaParmap('va_parmap=DEPT')
|
||||
expect(map.size).toBe(0)
|
||||
expect(problems).toEqual([
|
||||
'va_parmap "DEPT" was ignored: it needs a "-" between the VA parameter and the column name.'
|
||||
])
|
||||
})
|
||||
|
||||
it('reports an entry with no parameter name', () => {
|
||||
const { map, problems } = parseVaParmap('va_parmap=-DEPT')
|
||||
expect(map.size).toBe(0)
|
||||
expect(problems).toEqual([
|
||||
'va_parmap "-DEPT" was ignored: the VA parameter name is missing.'
|
||||
])
|
||||
})
|
||||
|
||||
it('reports an entry with no column name', () => {
|
||||
const { map, problems } = parseVaParmap('va_parmap=REGION_TAB1-')
|
||||
expect(map.size).toBe(0)
|
||||
expect(problems).toEqual([
|
||||
'va_parmap "REGION_TAB1-" was ignored: "" is not a valid SAS column name.'
|
||||
])
|
||||
})
|
||||
|
||||
it('reports a column that is not a plain SAS name (injection guard)', () => {
|
||||
// The column is interpolated into a SAS WHERE clause, so anything beyond a
|
||||
// SAS name is refused at parse time rather than carried into a clause.
|
||||
for (const entry of [
|
||||
'A-x;drop',
|
||||
'A-REGION%20OR%201=1',
|
||||
"A-RE'GION",
|
||||
'A-1ABC'
|
||||
]) {
|
||||
const { map, problems } = parseVaParmap(`va_parmap=${entry}`)
|
||||
expect(map.size).toBe(0)
|
||||
expect(problems.length).toBe(1)
|
||||
expect(problems[0]).toContain('is not a valid SAS column name')
|
||||
}
|
||||
expect(parseVaParmap('va_parmap=A-A_B9').map.get('a')?.column).toBe(
|
||||
'A_B9'
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps the usable entries and reports only the bad one', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=GOOD-REGION&va_parmap=BROKEN'
|
||||
)
|
||||
expect(map.get('good')?.column).toBe('REGION')
|
||||
expect(problems.length).toBe(1)
|
||||
expect(problems[0]).toContain('va_parmap "BROKEN"')
|
||||
})
|
||||
})
|
||||
|
||||
describe('transforms', () => {
|
||||
it('parses an ADD anchor with an offset', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=SELECTED-D_DATE:ADD(QUARTER_START,-1)'
|
||||
)
|
||||
expect(problems).toEqual([])
|
||||
expect(map.get('selected')?.column).toBe('D_DATE')
|
||||
expect(map.get('selected')?.transform).toEqual({
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: -1
|
||||
})
|
||||
})
|
||||
|
||||
it('defaults the offset to zero and accepts lower case', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=sel-d_date:add(quarter_start)'
|
||||
)
|
||||
expect(problems).toEqual([])
|
||||
expect(map.get('sel')?.transform).toEqual({
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: 0
|
||||
})
|
||||
})
|
||||
|
||||
it('parses a RANGE period', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=sel-D_DATE:RANGE(quarter,-1)'
|
||||
)
|
||||
expect(problems).toEqual([])
|
||||
expect(map.get('sel')?.transform).toEqual({
|
||||
op: 'RANGE',
|
||||
period: 'QUARTER',
|
||||
offset: -1
|
||||
})
|
||||
})
|
||||
|
||||
it('leaves a mapping without a transform as a plain column', () => {
|
||||
const { map } = parseVaParmap('va_parmap=sel-D_DATE')
|
||||
expect(map.get('sel')?.column).toBe('D_DATE')
|
||||
expect(map.get('sel')?.transform).toBeUndefined()
|
||||
})
|
||||
|
||||
it('reports an unknown operation, anchor, period or offset', () => {
|
||||
const cases: [string, string][] = [
|
||||
['sel-D_DATE:SHIFT(QUARTER_START,-1)', 'is not a known transform'],
|
||||
['sel-D_DATE:ADD(FORTNIGHT_START,-1)', 'is not a known anchor'],
|
||||
['sel-D_DATE:RANGE(DECADE,-1)', 'is not a known period'],
|
||||
['sel-D_DATE:ADD(QUARTER_START,3.5)', 'is not a whole number'],
|
||||
['sel-D_DATE:ADD', 'is not a transform']
|
||||
]
|
||||
for (const [entry, expected] of cases) {
|
||||
const { map, problems } = parseVaParmap(`va_parmap=${entry}`)
|
||||
expect(`${entry}: ${map.size}`).toBe(`${entry}: 0`)
|
||||
expect(`${entry}: ${problems.length}`).toBe(`${entry}: 1`)
|
||||
expect(problems[0]).toContain(expected)
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps the parameter label free to contain a colon', () => {
|
||||
// The key side is untouched by the transform grammar: only the column side
|
||||
// is split, so a label with a colon still maps.
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=Sales:2026-D_DATE:ADD(QUARTER_START,-1)'
|
||||
)
|
||||
expect(problems).toEqual([])
|
||||
expect(map.get('sales:2026')?.column).toBe('D_DATE')
|
||||
expect(map.get('sales:2026')?.transform).toEqual({
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: -1
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('one parameter, one column', () => {
|
||||
it('keeps the first mapping and reports a conflicting duplicate', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER'
|
||||
)
|
||||
expect(map.get('dept')?.column).toBe('REGION')
|
||||
expect(problems).toEqual([
|
||||
'va_parmap "DEPT-OTHER" was ignored: VA parameter "dept" already maps to REGION, and a parameter can filter only one column.'
|
||||
])
|
||||
})
|
||||
|
||||
it('treats a duplicate key as a duplicate across case', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=Dept-REGION&va_parmap=DEPT-SOMETHING_ELSE'
|
||||
)
|
||||
expect(map.get('dept')?.column).toBe('REGION')
|
||||
expect(problems.length).toBe(1)
|
||||
})
|
||||
|
||||
it('accepts the same mapping written twice (no ambiguity)', () => {
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=DEPT-REGION&va_parmap=dept-region'
|
||||
)
|
||||
expect(map.get('dept')?.column).toBe('REGION')
|
||||
expect(problems).toEqual([])
|
||||
})
|
||||
|
||||
it('still allows two parameters to map to one column', () => {
|
||||
// The multi-object case: different parameters, same DC column.
|
||||
const { map, problems } = parseVaParmap(
|
||||
'va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION'
|
||||
)
|
||||
expect(map.size).toBe(2)
|
||||
expect(problems).toEqual([])
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,192 @@
|
||||
import {
|
||||
VA_ANCHORS,
|
||||
VA_PERIODS,
|
||||
VaAnchor,
|
||||
VaPeriod,
|
||||
VaTransform
|
||||
} from './va-parmap-transform'
|
||||
|
||||
/**
|
||||
* What a VA parameter's mapping resolves to: the DC column, and optionally a
|
||||
* transform that adds derived dates to the filter (e.g. the previous quarter's
|
||||
* row alongside the selected one).
|
||||
*/
|
||||
export interface VaParmapTarget {
|
||||
column: string
|
||||
transform?: VaTransform
|
||||
}
|
||||
|
||||
/** Result of `parseVaParmap` - the usable mappings, and what was ignored. */
|
||||
export interface VaParmap {
|
||||
/** Lower-cased VA parameter key -> DC column (and transform, if any). */
|
||||
map: Map<string, VaParmapTarget>
|
||||
/** One plain-English line per `va_parmap` entry that was ignored. */
|
||||
problems: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses the `va_parmap` entries out of a hash query string (the part after `?`
|
||||
* in `window.location.hash`).
|
||||
*
|
||||
* Format - one mapping per occurrence, each `<VA parameter key>-<DC column>`:
|
||||
*
|
||||
* ?embed=va&va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION
|
||||
*
|
||||
* Repeating the parameter (rather than packing a comma-separated list into one
|
||||
* value) keeps the grammar unambiguous: VA parameter labels are free text, so
|
||||
* any list separator we picked could also appear inside a label.
|
||||
*
|
||||
* Each entry is split at its LAST `-`. A SAS column name can never contain a
|
||||
* hyphen, so the final one always delimits the column, and a label that does
|
||||
* contain hyphens (e.g. `Sales-2024`) still maps correctly.
|
||||
*
|
||||
* The key is matched case-insensitively, and trailing/leading blanks are
|
||||
* trimmed, against a VA parameter's `label`, which is the same key the
|
||||
* label->column match uses. The internal `name` (e.g. `pr273`) is not a key: it
|
||||
* is the report's own bookkeeping, so an entry written against it never applies
|
||||
* and is reported by `unmatchedMappings`.
|
||||
*
|
||||
* An entry that cannot be used is reported in `problems` rather than dropped in
|
||||
* silence: these entries are hand-written in the report's DDC URL, usually by
|
||||
* the report author, and a typo would otherwise degrade to "no filter" - which
|
||||
* looks exactly like a working filter over the whole table.
|
||||
*
|
||||
* Returns an empty map when the parameter is absent, which leaves the default
|
||||
* same-name resolution untouched (existing embeds are unaffected).
|
||||
*
|
||||
* Modeled on `parseEmbedParam` / `parseLabelsParam`, which read the same hash
|
||||
* query.
|
||||
*/
|
||||
export function parseVaParmap(hashQuery: string): VaParmap {
|
||||
const map = new Map<string, VaParmapTarget>()
|
||||
const problems: string[] = []
|
||||
if (!hashQuery) return { map, problems }
|
||||
|
||||
const entries = new URLSearchParams(hashQuery).getAll('va_parmap')
|
||||
for (const entry of entries) {
|
||||
// A transform sits at the end, in parentheses: `<param>-<column>:OP(args)`.
|
||||
// It is lifted off FIRST, because its arguments may contain a hyphen (a
|
||||
// negative offset) and the key/column split below uses the hyphen.
|
||||
let body = entry ?? ''
|
||||
let transformSpec = ''
|
||||
if (body.trim().endsWith(')')) {
|
||||
const open = body.lastIndexOf('(')
|
||||
const colon = open > 0 ? body.lastIndexOf(':', open) : -1
|
||||
if (colon > 0) {
|
||||
transformSpec = body.slice(colon + 1).trim()
|
||||
body = body.slice(0, colon)
|
||||
}
|
||||
}
|
||||
// Last hyphen delimits the DC column (see above).
|
||||
const separator = body.lastIndexOf('-')
|
||||
if (separator < 0) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: it needs a "-" between the VA parameter and the column name.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
const key = body.slice(0, separator).trim().toLowerCase()
|
||||
const column = body.slice(separator + 1).trim()
|
||||
if (!key) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: the VA parameter name is missing.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
if (column.includes(':')) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: "${column}" is not a transform - expected e.g. ADD(QUARTER_START,-1).`
|
||||
)
|
||||
continue
|
||||
}
|
||||
// The column name is interpolated into a SAS WHERE clause, so only a plain
|
||||
// SAS name is accepted - anything else is refused rather than mapped.
|
||||
if (!/^[A-Za-z_][A-Za-z0-9_]{0,31}$/.test(column)) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: "${column}" is not a valid SAS column name.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
const transform = transformSpec
|
||||
? parseTransform(transformSpec, entry, problems)
|
||||
: undefined
|
||||
if (transformSpec && !transform) continue
|
||||
// One VA parameter filters one column. A second entry for the same parameter
|
||||
// is ambiguous, so the FIRST one wins and the conflict is reported - the
|
||||
// alternative (last one wins, silently) would apply a filter the author
|
||||
// never sees the origin of. Two parameters mapping to the same column is
|
||||
// fine and is how two report objects filter one column from their own
|
||||
// parameters.
|
||||
const existing = map.get(key)
|
||||
if (existing) {
|
||||
if (existing.column.toUpperCase() !== column.toUpperCase()) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: VA parameter "${key}" already maps to ${existing.column}, and a parameter can filter only one column.`
|
||||
)
|
||||
}
|
||||
continue
|
||||
}
|
||||
map.set(key, { column, transform })
|
||||
}
|
||||
|
||||
return { map, problems }
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses `OP(arg[, arg])`, returning undefined (and pushing a problem line) when
|
||||
* the operation or one of its arguments is not in the allow-list.
|
||||
*
|
||||
* ADD takes an anchor and an optional offset; RANGE takes a period and an
|
||||
* optional offset. The offset is whole periods, negative for earlier.
|
||||
*/
|
||||
function parseTransform(
|
||||
spec: string,
|
||||
entry: string,
|
||||
problems: string[]
|
||||
): VaTransform | undefined {
|
||||
const match = /^([A-Za-z]+)\s*\(([^)]*)\)$/.exec(spec)
|
||||
if (!match) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: "${spec}" is not a transform - expected e.g. ADD(QUARTER_START,-1).`
|
||||
)
|
||||
return undefined
|
||||
}
|
||||
const op = match[1].toUpperCase()
|
||||
const args = match[2]
|
||||
.split(',')
|
||||
.map((arg) => arg.trim())
|
||||
.filter((arg) => arg.length > 0)
|
||||
const offsetArg = args[1] === undefined ? '0' : args[1]
|
||||
if (!/^-?\d+$/.test(offsetArg)) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: the offset "${offsetArg}" is not a whole number of periods.`
|
||||
)
|
||||
return undefined
|
||||
}
|
||||
const offset = Number(offsetArg)
|
||||
|
||||
if (op === 'ADD') {
|
||||
const anchor = (args[0] || '').toUpperCase()
|
||||
if (!VA_ANCHORS.includes(anchor as VaAnchor)) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: "${args[0] || ''}" is not a known anchor (${VA_ANCHORS.join(', ')}).`
|
||||
)
|
||||
return undefined
|
||||
}
|
||||
return { op: 'ADD', anchor: anchor as VaAnchor, offset }
|
||||
}
|
||||
if (op === 'RANGE') {
|
||||
const period = (args[0] || '').toUpperCase()
|
||||
if (!VA_PERIODS.includes(period as VaPeriod)) {
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: "${args[0] || ''}" is not a known period (${VA_PERIODS.join(', ')}).`
|
||||
)
|
||||
return undefined
|
||||
}
|
||||
return { op: 'RANGE', period: period as VaPeriod, offset }
|
||||
}
|
||||
problems.push(
|
||||
`va_parmap "${entry}" was ignored: "${match[1]}" is not a known transform (ADD, RANGE).`
|
||||
)
|
||||
return undefined
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
import {
|
||||
applyVaTransform,
|
||||
isRangeTransform,
|
||||
VA_ANCHORS,
|
||||
VA_PERIODS,
|
||||
VaTransform
|
||||
} from './va-parmap-transform'
|
||||
|
||||
/** Local y-m-d, so assertions do not depend on the runner's timezone offset. */
|
||||
const ymd = (d: Date) =>
|
||||
`${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(
|
||||
d.getDate()
|
||||
).padStart(2, '0')}`
|
||||
|
||||
const date = (y: number, m: number, d: number) => new Date(y, m - 1, d)
|
||||
|
||||
describe('va_parmap transforms', () => {
|
||||
it('exposes only allow-listed anchors and periods', () => {
|
||||
expect(VA_ANCHORS).toContain('QUARTER_START')
|
||||
expect(VA_PERIODS).toEqual(['QUARTER', 'MONTH', 'YEAR'])
|
||||
})
|
||||
|
||||
describe('ADD - the value plus one anchor beside it', () => {
|
||||
it('adds the previous quarter start, which is the reported requirement', () => {
|
||||
// intnx('qtr', dt, -1, 'b') for a value on a quarter boundary
|
||||
const dates = applyVaTransform(date(2026, 10, 1), {
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: -1
|
||||
})
|
||||
expect(dates.map(ymd)).toEqual(['2026-10-01', '2026-07-01'])
|
||||
})
|
||||
|
||||
it('resolves a mid-quarter value to its own value plus the previous quarter start', () => {
|
||||
const dates = applyVaTransform(date(2026, 8, 15), {
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: -1
|
||||
})
|
||||
expect(dates.map(ymd)).toEqual(['2026-08-15', '2026-04-01'])
|
||||
})
|
||||
|
||||
it('de-duplicates when the value already sits on the anchor', () => {
|
||||
// 2026-01-01 is a quarter start, so "0 quarters back" adds nothing
|
||||
const dates = applyVaTransform(date(2026, 1, 1), {
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: 0
|
||||
})
|
||||
expect(dates.map(ymd)).toEqual(['2026-01-01'])
|
||||
})
|
||||
|
||||
it('resolves the other anchors at their boundaries', () => {
|
||||
const at = date(2026, 8, 15)
|
||||
expect(
|
||||
ymd(
|
||||
applyVaTransform(at, {
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_END',
|
||||
offset: 0
|
||||
})[1]
|
||||
)
|
||||
).toBe('2026-09-30')
|
||||
expect(
|
||||
ymd(
|
||||
applyVaTransform(at, { op: 'ADD', anchor: 'MONTH_END', offset: 0 })[1]
|
||||
)
|
||||
).toBe('2026-08-31')
|
||||
expect(
|
||||
ymd(
|
||||
applyVaTransform(at, {
|
||||
op: 'ADD',
|
||||
anchor: 'YEAR_START',
|
||||
offset: -1
|
||||
})[1]
|
||||
)
|
||||
).toBe('2025-01-01')
|
||||
expect(
|
||||
ymd(
|
||||
applyVaTransform(at, { op: 'ADD', anchor: 'YEAR_END', offset: 0 })[1]
|
||||
)
|
||||
).toBe('2026-12-31')
|
||||
})
|
||||
|
||||
it('handles February and leap years at month end', () => {
|
||||
expect(
|
||||
ymd(
|
||||
applyVaTransform(date(2026, 3, 1), {
|
||||
op: 'ADD',
|
||||
anchor: 'MONTH_END',
|
||||
offset: -1
|
||||
})[1]
|
||||
)
|
||||
).toBe('2026-02-28')
|
||||
expect(
|
||||
ymd(
|
||||
applyVaTransform(date(2024, 3, 15), {
|
||||
op: 'ADD',
|
||||
anchor: 'MONTH_END',
|
||||
offset: -1
|
||||
})[1]
|
||||
)
|
||||
).toBe('2024-02-29')
|
||||
})
|
||||
|
||||
it('crosses a year boundary when the offset takes it out of the year', () => {
|
||||
const dates = applyVaTransform(date(2027, 1, 1), {
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: -1
|
||||
})
|
||||
expect(dates.map(ymd)).toEqual(['2027-01-01', '2026-10-01'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('RANGE - a closed period', () => {
|
||||
it('covers the previous quarter', () => {
|
||||
const dates = applyVaTransform(date(2026, 8, 15), {
|
||||
op: 'RANGE',
|
||||
period: 'QUARTER',
|
||||
offset: -1
|
||||
})
|
||||
expect(dates.map(ymd)).toEqual(['2026-04-01', '2026-06-30'])
|
||||
})
|
||||
|
||||
it('covers the current month and the previous year', () => {
|
||||
expect(
|
||||
applyVaTransform(date(2026, 8, 15), {
|
||||
op: 'RANGE',
|
||||
period: 'MONTH',
|
||||
offset: 0
|
||||
}).map(ymd)
|
||||
).toEqual(['2026-08-01', '2026-08-31'])
|
||||
expect(
|
||||
applyVaTransform(date(2026, 8, 15), {
|
||||
op: 'RANGE',
|
||||
period: 'YEAR',
|
||||
offset: -1
|
||||
}).map(ymd)
|
||||
).toEqual(['2025-01-01', '2025-12-31'])
|
||||
})
|
||||
|
||||
it('reports itself as a range so the caller picks BETWEEN', () => {
|
||||
expect(
|
||||
isRangeTransform({ op: 'RANGE', period: 'QUARTER', offset: 0 })
|
||||
).toBe(true)
|
||||
expect(
|
||||
isRangeTransform({
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: 0
|
||||
} as VaTransform)
|
||||
).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('time of day', () => {
|
||||
it('gives a datetime range whole days, not the value clock time', () => {
|
||||
// A BETWEEN that starts at 13:45 on day one loses the earlier part of that
|
||||
// day, and one that ends at 13:45 on the last day loses the rest of it.
|
||||
const value = new Date(2026, 7, 15, 13, 45, 30)
|
||||
const [start, end] = applyVaTransform(
|
||||
value,
|
||||
{ op: 'RANGE', period: 'QUARTER', offset: 0 },
|
||||
true
|
||||
)
|
||||
expect([
|
||||
start.getHours(),
|
||||
start.getMinutes(),
|
||||
start.getSeconds()
|
||||
]).toEqual([0, 0, 0])
|
||||
expect([end.getHours(), end.getMinutes(), end.getSeconds()]).toEqual([
|
||||
23, 59, 59
|
||||
])
|
||||
expect(ymd(start)).toBe('2026-07-01')
|
||||
expect(ymd(end)).toBe('2026-09-30')
|
||||
})
|
||||
|
||||
it('keeps the time for datetime columns and drops it otherwise', () => {
|
||||
const value = new Date(2026, 9, 1, 13, 45, 30)
|
||||
const add: VaTransform = {
|
||||
op: 'ADD',
|
||||
anchor: 'QUARTER_START',
|
||||
offset: -1
|
||||
}
|
||||
const kept = applyVaTransform(value, add, true)[1]
|
||||
expect([kept.getHours(), kept.getMinutes(), kept.getSeconds()]).toEqual([
|
||||
13, 45, 30
|
||||
])
|
||||
const dropped = applyVaTransform(value, add, false)[1]
|
||||
expect([dropped.getHours(), dropped.getMinutes()]).toEqual([0, 0])
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* Anchors and periods a `va_parmap` transform can resolve a report value to.
|
||||
*
|
||||
* The names describe the result rather than mirroring SAS `intnx`, because the
|
||||
* arithmetic runs in the browser: emulating `intnx`'s alignment letters (`b`,
|
||||
* `e`, `s`, `m`, never mind week alignment and fiscal quarters) would invite
|
||||
* assumptions we could not honour. Each anchor below is one small function, so
|
||||
* adding one is a function plus a test row.
|
||||
*/
|
||||
export const VA_ANCHORS = [
|
||||
'QUARTER_START',
|
||||
'QUARTER_END',
|
||||
'MONTH_START',
|
||||
'MONTH_END',
|
||||
'YEAR_START',
|
||||
'YEAR_END'
|
||||
] as const
|
||||
export type VaAnchor = (typeof VA_ANCHORS)[number]
|
||||
|
||||
export const VA_PERIODS = ['QUARTER', 'MONTH', 'YEAR'] as const
|
||||
export type VaPeriod = (typeof VA_PERIODS)[number]
|
||||
|
||||
/**
|
||||
* A parsed transform: the operation plus its validated arguments.
|
||||
*
|
||||
* `ADD` yields the value and one anchor beside it (a list of dates), which is
|
||||
* how "this period's row plus the previous quarter's" is expressed. `RANGE`
|
||||
* yields a closed period (two dates) for a BETWEEN.
|
||||
*/
|
||||
export type VaTransform =
|
||||
| { op: 'ADD'; anchor: VaAnchor; offset: number }
|
||||
| { op: 'RANGE'; period: VaPeriod; offset: number }
|
||||
|
||||
/** True when the transform produces a closed range rather than a list. */
|
||||
export function isRangeTransform(transform: VaTransform): boolean {
|
||||
return transform.op === 'RANGE'
|
||||
}
|
||||
|
||||
/** Local midnight on the first day of the month `offset` months from `date`. */
|
||||
function monthStart(date: Date, offset: number): Date {
|
||||
return new Date(date.getFullYear(), date.getMonth() + offset, 1)
|
||||
}
|
||||
|
||||
/** Local midnight on the last day of the month `offset` months from `date`. */
|
||||
function monthEnd(date: Date, offset: number): Date {
|
||||
return new Date(date.getFullYear(), date.getMonth() + offset + 1, 0)
|
||||
}
|
||||
|
||||
/** First day of the quarter `offset` quarters from the one containing `date`. */
|
||||
function quarterStart(date: Date, offset: number): Date {
|
||||
const firstMonthOfQuarter = 3 * Math.floor(date.getMonth() / 3)
|
||||
return new Date(date.getFullYear(), firstMonthOfQuarter + offset * 3, 1)
|
||||
}
|
||||
|
||||
/** Last day of the quarter `offset` quarters from the one containing `date`. */
|
||||
function quarterEnd(date: Date, offset: number): Date {
|
||||
// Day 0 of a month is the last day of the month before it: calendar
|
||||
// construction, so a daylight-saving transition inside the quarter cannot
|
||||
// shift the result by an hour the way subtracting 24 hours would.
|
||||
const nextStart = quarterStart(date, offset + 1)
|
||||
return new Date(nextStart.getFullYear(), nextStart.getMonth(), 0)
|
||||
}
|
||||
|
||||
/** First day of the year `offset` years from the one containing `date`. */
|
||||
function yearStart(date: Date, offset: number): Date {
|
||||
return new Date(date.getFullYear() + offset, 0, 1)
|
||||
}
|
||||
|
||||
/** Last day of the year `offset` years from the one containing `date`. */
|
||||
function yearEnd(date: Date, offset: number): Date {
|
||||
return new Date(date.getFullYear() + offset, 11, 31)
|
||||
}
|
||||
|
||||
/** Resolve one anchor, keeping the time of day of `date` for datetime columns. */
|
||||
function resolveAnchor(
|
||||
anchor: VaAnchor,
|
||||
date: Date,
|
||||
offset: number,
|
||||
keepTime: boolean
|
||||
): Date {
|
||||
let resolved: Date
|
||||
switch (anchor) {
|
||||
case 'QUARTER_START':
|
||||
resolved = quarterStart(date, offset)
|
||||
break
|
||||
case 'QUARTER_END':
|
||||
resolved = quarterEnd(date, offset)
|
||||
break
|
||||
case 'MONTH_START':
|
||||
resolved = monthStart(date, offset)
|
||||
break
|
||||
case 'MONTH_END':
|
||||
resolved = monthEnd(date, offset)
|
||||
break
|
||||
case 'YEAR_START':
|
||||
resolved = yearStart(date, offset)
|
||||
break
|
||||
default:
|
||||
resolved = yearEnd(date, offset)
|
||||
break
|
||||
}
|
||||
if (keepTime) {
|
||||
resolved.setHours(date.getHours(), date.getMinutes(), date.getSeconds(), 0)
|
||||
}
|
||||
return resolved
|
||||
}
|
||||
|
||||
/**
|
||||
* The dates a transform adds to a mapping, in order.
|
||||
*
|
||||
* `ADD` returns the report's own value plus the anchor beside it, de-duplicated -
|
||||
* so a value that already sits on the anchor yields one date, and the clause
|
||||
* stays an equality rather than an `IN` with a repeat. `RANGE` returns the
|
||||
* period's first and last day.
|
||||
*
|
||||
* `keepTime` carries the time of day across for datetime columns, since a
|
||||
* DATETIME filter compares to the second; date columns use midnight.
|
||||
*/
|
||||
export function applyVaTransform(
|
||||
value: Date,
|
||||
transform: VaTransform,
|
||||
keepTime = false
|
||||
): Date[] {
|
||||
if (transform.op === 'RANGE') {
|
||||
// A range covers whole days: the bounds are the period's own, never the
|
||||
// value's clock time. Reusing keepTime here would start the range mid-morning
|
||||
// on day one and end it mid-morning on the last day, dropping the rest of
|
||||
// both boundary days from a BETWEEN.
|
||||
const start = resolveAnchor(
|
||||
(transform.period + '_START') as VaAnchor,
|
||||
value,
|
||||
transform.offset,
|
||||
false
|
||||
)
|
||||
const end = resolveAnchor(
|
||||
(transform.period + '_END') as VaAnchor,
|
||||
value,
|
||||
transform.offset,
|
||||
false
|
||||
)
|
||||
if (keepTime) end.setHours(23, 59, 59, 0)
|
||||
return [start, end]
|
||||
}
|
||||
const anchor = resolveAnchor(
|
||||
transform.anchor,
|
||||
value,
|
||||
transform.offset,
|
||||
keepTime
|
||||
)
|
||||
return anchor.getTime() === value.getTime() ? [value] : [value, anchor]
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "dcfrontend",
|
||||
"version": "7.19.0",
|
||||
"version": "7.21.0",
|
||||
"description": "Data Controller",
|
||||
"devDependencies": {
|
||||
"@nogoo9/gitleaks": "8.30.1-post.2",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"fromjs": [
|
||||
{
|
||||
"ADMIN": "AllUsers",
|
||||
"ADMIN": "all-users",
|
||||
"DCPATH": "/tmp/dcdata",
|
||||
"OUTFMT": "JSON"
|
||||
}
|
||||
|
||||
@@ -13,10 +13,12 @@
|
||||
// The rules mirror services/hooks/mpe_derived_postedit.sas, in its order:
|
||||
// normalise the incoming rows, reject a duplicate name and a bad field, then
|
||||
// check the dependency graph that the resulting configuration would have (the
|
||||
// incoming rows plus the current rows they do not replace). As on the server,
|
||||
// the collision between VAR_NM and a physical column is left to the read-time
|
||||
// engine, which runs with the target library assigned - the hook does not have
|
||||
// it here either.
|
||||
// incoming rows plus the current rows they do not replace). The field rules
|
||||
// are core's mp_derivedvarcheck, which the server hook calls and the read-time
|
||||
// engine runs too - this file spells them out a second time, so a change there
|
||||
// has to be mirrored here. As on the server, the collision between VAR_NM and
|
||||
// a physical column is left to the read-time engine, which runs with the target
|
||||
// library assigned - the hook does not have it here either.
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
;(function mpeDerivedPostedit() {
|
||||
@@ -83,9 +85,45 @@
|
||||
seen.add(key)
|
||||
}
|
||||
|
||||
// ─── Field level checks, the rules the engine applies at read time ─────────
|
||||
// C0 controls and DEL: what a single-line label cannot carry
|
||||
// ─── Field level checks ────────────────────────────────────────────────────
|
||||
// Two passes, in the order the SAS hook runs them. The first is the fields
|
||||
// this hook owns - the control table key, which the loader normalises but
|
||||
// does not require, and ACTIVE, the picker pair the EDIT screen offers.
|
||||
for (const row of rows) {
|
||||
if (isDeleted(row)) continue
|
||||
const name = field(row, 'VAR_NM')
|
||||
let reason = ''
|
||||
if (!field(row, 'LIBREF') || !field(row, 'DSN'))
|
||||
reason = 'LIBREF and DSN are required'
|
||||
// ACTIVE is a picker in the EDIT screen (HARDSELECT over the
|
||||
// MPE_SELECTBOX pair) - the same pair the hook enforces
|
||||
else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
|
||||
reason = 'ACTIVE must be 1 or 0'
|
||||
if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
|
||||
}
|
||||
|
||||
// The second pass is the rules for the derived variable itself, which are
|
||||
// core's (mp_derivedvarcheck): the engine runs the same macro at read time,
|
||||
// so a definition is measured against one rule list wherever it is checked.
|
||||
|
||||
// C0 controls and DEL: what a label cannot carry once a line break has been
|
||||
// flattened to a space, which is what the engine does with it
|
||||
const CONTROL_CHARS = /[\u0000-\u001F\u007F]/
|
||||
const flatten = (v) => v.replace(/[\r\n]+/g, ' ')
|
||||
const AUTOMATIC_VARS = ['_N_', '_ERROR_', '_IORC_']
|
||||
|
||||
// the capabilities refused by name - a function name cannot be assembled at
|
||||
// run time, so naming them is a control the expression cannot evade
|
||||
const DENY_FUNCS = new RegExp(
|
||||
'\\b(DOSUBL|JAVAOBJ|JAVAMETA|SYMGET|SYMPUTX?|SYSGET|GETOPTION' +
|
||||
'|PATHNAME|FILEEXIST|FOPEN|FREAD|FGET|FCLOSE' +
|
||||
'|DOPEN|DREAD|DNUM|MOPEN|SLEEP' +
|
||||
'|FILENAME|FDELETE|FCOPY|DCREATE|DLCREATE)\\s*\\('
|
||||
)
|
||||
|
||||
// quoted literals are data, not calls
|
||||
const unquoted = (v) =>
|
||||
v.replace(/'([^']|'')*'/g, ' ').replace(/"([^"]|"")*"/g, ' ')
|
||||
|
||||
// true when the value carries a character outside the allowed set
|
||||
const hasInvalidChar = (value, chars) =>
|
||||
@@ -93,8 +131,6 @@
|
||||
|
||||
for (const row of rows) {
|
||||
if (isDeleted(row)) continue
|
||||
const libref = field(row, 'LIBREF')
|
||||
const dsn = field(row, 'DSN')
|
||||
const name = field(row, 'VAR_NM')
|
||||
const type = field(row, 'VAR_TYPE')
|
||||
const len = Number(colVal(row, 'VAR_LEN'))
|
||||
@@ -103,19 +139,16 @@
|
||||
const logic = strip(colVal(row, 'VAR_LOGIC'))
|
||||
|
||||
let reason = ''
|
||||
if (!libref || !dsn) reason = 'LIBREF and DSN are required'
|
||||
else if (!name) reason = 'VAR_NM is required'
|
||||
if (!name) reason = 'VAR_NM is required'
|
||||
else if (hasInvalidChar(name, NAME_CHARS))
|
||||
reason = 'VAR_NM must be a valid SAS name'
|
||||
else if ('0123456789'.includes(name[0]))
|
||||
reason = 'VAR_NM must not start with a digit'
|
||||
else if (AUTOMATIC_VARS.includes(name))
|
||||
reason = 'VAR_NM is an automatic variable of the data step'
|
||||
else if (name === DELETE_MARK) reason = 'VAR_NM is reserved'
|
||||
else if (type !== 'C' && type !== 'N')
|
||||
reason = 'VAR_TYPE must be C or N'
|
||||
// ACTIVE is a picker in the EDIT screen (HARDSELECT over the
|
||||
// MPE_SELECTBOX pair) - the same pair mpe_derived_check.sas enforces
|
||||
else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
|
||||
reason = 'ACTIVE must be 1 or 0'
|
||||
else if (!Number.isInteger(len) || len < 1)
|
||||
reason = 'VAR_LEN must be a positive integer'
|
||||
else if (type === 'C' && len > 32767)
|
||||
@@ -132,20 +165,24 @@
|
||||
reason = 'VAR_FMT must be a format name, without ; & % or /* */'
|
||||
else if (hasInvalidChar(fmt, FMT_CHARS))
|
||||
reason = 'VAR_FMT must be a format name'
|
||||
// free text, in any script - what is refused is anything that cannot be
|
||||
// a single-line column label, and a value the engine would cut to 256
|
||||
else if (CONTROL_CHARS.test(label))
|
||||
reason =
|
||||
'VAR_LABEL must be plain text - no line breaks or control characters'
|
||||
// free text, in any script - what is refused is a control character,
|
||||
// and a value the engine would cut to 256
|
||||
else if (CONTROL_CHARS.test(flatten(label)))
|
||||
reason = 'VAR_LABEL must be plain text - no control characters'
|
||||
else if ([...label].length > 256)
|
||||
reason = 'VAR_LABEL exceeds 256 characters'
|
||||
else if (!logic) reason = 'VAR_LOGIC is required'
|
||||
else if (logic.length > 4000)
|
||||
reason = 'VAR_LOGIC exceeds 4000 characters'
|
||||
else if (logic.includes(';'))
|
||||
reason = 'VAR_LOGIC must be a single expression - remove the ;'
|
||||
else if (logic.includes('&') || logic.includes('%'))
|
||||
reason = 'VAR_LOGIC must not contain macro triggers'
|
||||
else if (logic.includes('/*') || logic.includes('*/'))
|
||||
reason = 'VAR_LOGIC must not contain comments'
|
||||
else if (DENY_FUNCS.test(unquoted(logic).toUpperCase()))
|
||||
reason =
|
||||
'VAR_LOGIC may not call a function that submits code or reaches the file system, the environment or the macro symbol table'
|
||||
|
||||
if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
|
||||
}
|
||||
|
||||
Generated
+4
-4
@@ -7,7 +7,7 @@
|
||||
"name": "dc-sas",
|
||||
"dependencies": {
|
||||
"@sasjs/cli": "6.2.0",
|
||||
"@sasjs/core": "6.1.0"
|
||||
"@sasjs/core": "6.1.4"
|
||||
}
|
||||
},
|
||||
"node_modules/@asamuzakjp/css-color": {
|
||||
@@ -235,9 +235,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@sasjs/core": {
|
||||
"version": "6.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.0.tgz",
|
||||
"integrity": "sha512-U0Ekq/TXVMquYV+AG6urBQSUgqUvV37yduufy68QQVgDj3PJiLUJDcOXKOGbhWGD+Nv877VuPCviGYqrqbjXvg==",
|
||||
"version": "6.1.4",
|
||||
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.4.tgz",
|
||||
"integrity": "sha512-vIF2LIftxPWg4CvO3i5raEfrHFHY/6S+MqVGpjRR71dgjLRJEYNyPJgGqD6BCZwHv5V6/2nDnBi/BIIMiLXfzA==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@sasjs/lint": {
|
||||
|
||||
+1
-1
@@ -34,7 +34,7 @@
|
||||
"private": true,
|
||||
"dependencies": {
|
||||
"@sasjs/cli": "6.2.0",
|
||||
"@sasjs/core": "6.1.0"
|
||||
"@sasjs/core": "6.1.4"
|
||||
},
|
||||
"overrides": {
|
||||
"nanoid": "3.3.18"
|
||||
|
||||
@@ -49,9 +49,19 @@
|
||||
being maintained. A cycle is rejected.
|
||||
|
||||
Semicolons, macro triggers and comments are rejected, so a row can only ever
|
||||
be an expression. A derived variable may not reuse the name of a physical
|
||||
column, and may not reference itself: an intra-row calculation would imply a
|
||||
retained value, which depends on the order rows happen to arrive in.
|
||||
be an expression, and an expression that would let the row submit code of its
|
||||
own - DOSUBL, or a function reaching the file system, the environment or the
|
||||
macro symbol table - is refused by name.
|
||||
|
||||
The field level rules themselves are core's: mp_derivedvarcheck scans the
|
||||
definitions and mp_derivedvars calls it, which is the same macro the MPE_DERIVED
|
||||
post edit hook runs at submit time (services/hooks/mpe_derived_postedit). A
|
||||
row is therefore measured against one rule list, whether it is being staged
|
||||
or read.
|
||||
|
||||
A derived variable may not reuse the name of a physical column, and may not
|
||||
reference itself: an intra-row calculation would imply a retained value,
|
||||
which depends on the order rows happen to arrive in.
|
||||
|
||||
<h4> SAS Macros </h4>
|
||||
@li mp_derivedvars.sas
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
/**
|
||||
@file
|
||||
@brief Field level checks for a staged MPE_DERIVED submission
|
||||
@details Writes one row per staged row that breaks a rule, with the reason in
|
||||
REASON, so a caller can report the first failure and abort. Rows marked for
|
||||
deletion are skipped - they are being removed, so their fields are not loaded.
|
||||
|
||||
The rules are the ones %mpe_derived applies at read time, so a submission
|
||||
cannot stage a row the engine would then refuse.
|
||||
|
||||
@param inds the staged rows, already normalised by the caller
|
||||
@param outds the dataset to create: one row per failing row, plus REASON
|
||||
|
||||
@author 4GL Apps Ltd
|
||||
@copyright 4GL Apps Ltd. This code may only be used within Data Controller
|
||||
and may not be re-distributed or re-sold without the express permission of
|
||||
4GL Apps Ltd.
|
||||
|
||||
**/
|
||||
|
||||
%macro mpe_derived_check(inds=work.staging_ds, outds=work.derived_check);
|
||||
data &outds;
|
||||
set &inds;
|
||||
length reason $200;
|
||||
if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
|
||||
call missing(reason);
|
||||
|
||||
if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
|
||||
else if var_nm='' then reason='VAR_NM is required';
|
||||
else if verify(strip(var_nm),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_')>0
|
||||
then reason='VAR_NM must be a valid SAS name';
|
||||
else if verify(substr(var_nm,1,1),'0123456789')=0
|
||||
then reason='VAR_NM must not start with a digit';
|
||||
else if var_nm='_____DELETE__THIS__RECORD_____'
|
||||
then reason='VAR_NM is reserved';
|
||||
else if var_type not in ('C','N')
|
||||
then reason='VAR_TYPE must be C or N';
|
||||
/* ACTIVE is a picker in the EDIT screen (HARDSELECT over the MPE_SELECTBOX
|
||||
pair) - the same pair is enforced here, so a submission that bypasses the
|
||||
grid cannot land a value the screen would never offer. */
|
||||
else if missing(active) or active not in (0,1)
|
||||
then reason='ACTIVE must be 1 or 0';
|
||||
else if missing(var_len) or var_len ne int(var_len) or var_len<1
|
||||
then reason='VAR_LEN must be a positive integer';
|
||||
else if var_type='C' and var_len>32767
|
||||
then reason='VAR_LEN exceeds 32767 for a character variable';
|
||||
else if var_type='N' and var_len>8
|
||||
then reason='VAR_LEN exceeds 8 for a numeric variable';
|
||||
else if index(var_fmt,';')>0 or indexc(var_fmt,'&%')>0
|
||||
or index(var_fmt,'/*')>0 or index(var_fmt,'*/')>0
|
||||
then reason='VAR_FMT must be a format name, without ; & % or /* */';
|
||||
else if verify(strip(var_fmt),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.')>0
|
||||
then reason='VAR_FMT must be a format name';
|
||||
/* VAR_LABEL is free text: any character is allowed, in any script, and the
|
||||
quotes in it are escaped by %mp_derivedvars when the attrib statement is
|
||||
built. What is refused is anything that cannot be a single-line column
|
||||
label - a control character or a line break - and a value long enough to
|
||||
be cut down silently, since the engine truncates the label to 256. */
|
||||
else if anycntrl(var_label)>0
|
||||
then reason='VAR_LABEL must be plain text - no line breaks or control characters';
|
||||
else if klength(var_label)>256
|
||||
then reason='VAR_LABEL exceeds 256 characters';
|
||||
else if missing(var_logic)
|
||||
then reason='VAR_LOGIC is required';
|
||||
else if index(var_logic,';')>0
|
||||
then reason='VAR_LOGIC must be a single expression - remove the ;';
|
||||
else if indexc(var_logic,'&%')>0
|
||||
then reason='VAR_LOGIC must not contain macro triggers';
|
||||
else if index(var_logic,'/*')>0 or index(var_logic,'*/')>0
|
||||
then reason='VAR_LOGIC must not contain comments';
|
||||
|
||||
if not missing(reason) then output;
|
||||
run;
|
||||
|
||||
%mend mpe_derived_check;
|
||||
@@ -1,194 +0,0 @@
|
||||
/**
|
||||
@file
|
||||
@brief testing the mpe_derived_check macro
|
||||
|
||||
@details Checks each field level rule the MPE_DERIVED post edit hook applies
|
||||
to a staged submission. The macro is exercised directly so the rules can be
|
||||
probed with values a service cannot carry: VAR_LABEL is varchar(256) in the
|
||||
table and a staging dataset declares the same length, so a longer label is
|
||||
truncated before any check would see it.
|
||||
|
||||
<h4> SAS Macros </h4>
|
||||
@li mf_nobs.sas
|
||||
@li mp_assert.sas
|
||||
@li mpe_derived_check.sas
|
||||
|
||||
@author 4GL Apps Ltd
|
||||
@copyright 4GL Apps Ltd. This code may only be used within Data Controller
|
||||
and may not be re-distributed or re-sold without the express permission of
|
||||
4GL Apps Ltd.
|
||||
|
||||
**/
|
||||
|
||||
/**
|
||||
* Test 1 - one row per failing rule, and only failing rows
|
||||
*/
|
||||
data work.staging_ds;
|
||||
length libref $8 dsn $32 var_nm $32 var_type $1 var_len 8
|
||||
var_fmt $32 var_label $4000 var_deps $4000 var_logic $4000
|
||||
active 8 _____DELETE__THIS__RECORD_____ $3;
|
||||
libref='DCTEST';
|
||||
dsn='DC_DERIV';
|
||||
/* valid unless a row says otherwise - ACTIVE is mandatory in the screen and
|
||||
the check enforces the same pair here */
|
||||
active=1;
|
||||
_____DELETE__THIS__RECORD_____='No';
|
||||
|
||||
/* meets every rule */
|
||||
var_nm='GOOD';
|
||||
var_type='N';
|
||||
var_len=8;
|
||||
var_fmt='';
|
||||
var_label='a plain label';
|
||||
var_deps='';
|
||||
var_logic='1';
|
||||
output;
|
||||
|
||||
/* delete marked rows are being removed, so their fields are not loaded */
|
||||
var_nm='DELME';
|
||||
var_label='first line'||'0A'x||'second line';
|
||||
_____DELETE__THIS__RECORD_____='Yes';
|
||||
output;
|
||||
_____DELETE__THIS__RECORD_____='No';
|
||||
|
||||
/* free text is allowed, including quotes and punctuation */
|
||||
var_nm='FREETXT';
|
||||
var_label='It''s "quoted", plain text: 100 percent';
|
||||
output;
|
||||
|
||||
/* a line break cannot be a single line column label */
|
||||
var_nm='LBLBREAK';
|
||||
var_label='first line'||'0A'x||'second line';
|
||||
output;
|
||||
|
||||
/* longer than the table holds, so it would be cut down silently */
|
||||
var_nm='LBLLONG';
|
||||
var_label=repeat('x',300);
|
||||
output;
|
||||
|
||||
/* a format name, not an expression */
|
||||
var_nm='BADFMT';
|
||||
var_fmt='DATE9;';
|
||||
var_label='fmt';
|
||||
output;
|
||||
|
||||
/* one expression, no statement separator, no macro triggers */
|
||||
var_nm='BADLOGIC';
|
||||
var_fmt='';
|
||||
var_logic='x; y';
|
||||
output;
|
||||
|
||||
/* C or N */
|
||||
var_nm='BADTYPE';
|
||||
var_logic='1';
|
||||
var_type='X';
|
||||
output;
|
||||
|
||||
/* ACTIVE is a picker in the screen: only 1 or 0 */
|
||||
var_nm='BADACTIVE';
|
||||
var_type='C';
|
||||
var_logic='1';
|
||||
active=2;
|
||||
output;
|
||||
|
||||
stop;
|
||||
run;
|
||||
|
||||
%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
|
||||
|
||||
/**
|
||||
* Test 2 - the reported rows, rule by rule
|
||||
*/
|
||||
data work.expected;
|
||||
length var_nm $32 reason $200;
|
||||
var_nm='LBLBREAK';
|
||||
reason='VAR_LABEL must be plain text - no line breaks or control characters';
|
||||
output;
|
||||
var_nm='LBLLONG';
|
||||
reason='VAR_LABEL exceeds 256 characters';
|
||||
output;
|
||||
var_nm='BADFMT';
|
||||
reason='VAR_FMT must be a format name, without ; & % or /* */';
|
||||
output;
|
||||
var_nm='BADLOGIC';
|
||||
reason='VAR_LOGIC must be a single expression - remove the ;';
|
||||
output;
|
||||
var_nm='BADTYPE';
|
||||
reason='VAR_TYPE must be C or N';
|
||||
output;
|
||||
var_nm='BADACTIVE';
|
||||
reason='ACTIVE must be 1 or 0';
|
||||
output;
|
||||
run;
|
||||
|
||||
proc sql;
|
||||
create table work.cmp as
|
||||
select e.var_nm
|
||||
,case when a.reason=e.reason then 1 else 0 end as same
|
||||
from work.expected e
|
||||
left join work.derived_check a
|
||||
on upcase(strip(a.var_nm))=upcase(e.var_nm);
|
||||
quit;
|
||||
|
||||
data _null_;
|
||||
set work.cmp;
|
||||
call symputx(cats('rule_',var_nm),same,'l');
|
||||
run;
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(%mf_nobs(work.derived_check)=6),
|
||||
desc=%str(Checking only the six failing rows are reported),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(&rule_LBLBREAK=1),
|
||||
desc=%str(Checking a label with a line break is rejected),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(&rule_LBLLONG=1),
|
||||
desc=%str(Checking a label over 256 characters is rejected),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(&rule_BADFMT=1),
|
||||
desc=%str(Checking a format carrying a statement separator is rejected),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(&rule_BADLOGIC=1),
|
||||
desc=%str(Checking a logic value with a statement separator is rejected),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(&rule_BADTYPE=1),
|
||||
desc=%str(Checking a VAR_TYPE other than C or N is rejected),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(&rule_BADACTIVE=1),
|
||||
desc=%str(Checking an ACTIVE outside 1 or 0 is rejected),
|
||||
outds=work.test_results
|
||||
)
|
||||
|
||||
/**
|
||||
* Test 3 - a row that meets every rule, a free text label, and a delete
|
||||
* marked row are all left alone
|
||||
*/
|
||||
proc sql noprint;
|
||||
select count(*) into: notchecked
|
||||
from work.derived_check
|
||||
where upcase(strip(var_nm)) in ('GOOD','FREETXT','DELME');
|
||||
quit;
|
||||
|
||||
%mp_assert(
|
||||
iftrue=(¬checked=0),
|
||||
desc=%str(Checking a valid row, a free text label and a delete marked row are not reported),
|
||||
outds=work.test_results
|
||||
)
|
||||
@@ -2355,7 +2355,9 @@ insert into &lib..MPE_VALIDATIONS set
|
||||
,some_char="&x bottles of beer on the wall"
|
||||
,some_dropdown='Option 1'
|
||||
,some_num=ranuni(0)
|
||||
,some_date=round(ranuni(0)*1000,1)
|
||||
/* First of the month, cycling through two years: a randomly spread date
|
||||
gave every row its own day, so a date filter never returned many rows. */
|
||||
,some_date=intnx('month','01JAN1960'd,mod(&x,24),'S')
|
||||
,some_datetime=round(ranuni(0)*50000,1)
|
||||
,some_time=round(ranuni(0)*100,1)
|
||||
,some_shortnum=6+round(ranuni(0)*94,1)
|
||||
|
||||
@@ -6,16 +6,19 @@
|
||||
table - the engine reads only active rows, so the dependency can never
|
||||
resolve at run time
|
||||
2. a row is staged with active=0 while a surviving active row depends on it
|
||||
3. a staged row breaks a field rule of the shared scan (core's
|
||||
mp_derivedvarcheck), or carries an ACTIVE value the EDIT screen's picker
|
||||
cannot produce
|
||||
|
||||
They should NOT abort when:
|
||||
3. a new active row is added to a table that also holds an inactive row -
|
||||
4. a new active row is added to a table that also holds an inactive row -
|
||||
the inactive row is not part of the dependency graph, so it cannot
|
||||
create a cycle or a stale reference
|
||||
4. a delete-marked RLS rule naming a derived variable is submitted - the
|
||||
5. a delete-marked RLS rule naming a derived variable is submitted - the
|
||||
rule is being removed, so its variables are not validated
|
||||
5. a delete and re-add of the same variable is submitted together - the
|
||||
6. a delete and re-add of the same variable is submitted together - the
|
||||
re-added active row resolves the dependency
|
||||
6. an inactive row is set active again while another active row depends on
|
||||
7. an inactive row is set active again while another active row depends on
|
||||
it - the re-activation is what makes the dependency resolvable
|
||||
|
||||
The staged loads are aborted by the post edit hook. The assertion fetches the
|
||||
@@ -553,6 +556,103 @@ run;
|
||||
)
|
||||
|
||||
|
||||
/**
|
||||
* Test 8 - a field rule of the shared scan (mp_derivedvarcheck, core) is
|
||||
* applied at submit time: an expression that could submit code of its own is
|
||||
* refused by name, with the reason the scan gives. This is the same scan the
|
||||
* engine runs at read time, so the case pins that the hook calls it rather
|
||||
* than carrying a rule list of its own.
|
||||
*/
|
||||
data work.sascontroltable;
|
||||
action='LOAD';
|
||||
message='A refused function in VAR_LOGIC should abort';
|
||||
libds="&dc_libref..MPE_DERIVED";
|
||||
output;
|
||||
stop;
|
||||
run;
|
||||
|
||||
data work.jsdata;
|
||||
length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
|
||||
VAR_LOGIC $4000 VAR_FMT $32
|
||||
_____DELETE__THIS__RECORD_____ $3;
|
||||
LIBREF='DCTEST';
|
||||
DSN='DC_DERIV';
|
||||
VAR_NM='CALLER';
|
||||
VAR_DEPS='';
|
||||
VAR_TYPE='N';
|
||||
VAR_LEN=8;
|
||||
VAR_FMT='';
|
||||
/* no semicolon in the payload: the single-expression rule is tested before
|
||||
the function-name rule, so a quoted semicolon would abort on that first */
|
||||
VAR_LOGIC='dosubl("data _null_")';
|
||||
ACTIVE=1;
|
||||
_____DELETE__THIS__RECORD_____='No';
|
||||
run;
|
||||
|
||||
%mpe_execute(&_program,
|
||||
viyacontext=&defaultcontext,
|
||||
inputdatasets=work.sascontroltable work.jsdata,
|
||||
outlib=web8,
|
||||
outref=wb8,
|
||||
mdebug=&sasjs_mdebug
|
||||
)
|
||||
|
||||
/* the assertion compares the abort's MSG exactly, so the expected value is
|
||||
assembled on one line: a literal wrapped across two source lines carries the
|
||||
newline and the continuation indent with it */
|
||||
%let exp8=MPE_DERIVED: CALLER: VAR_LOGIC may not call a function that;
|
||||
%let exp8=&exp8 submits code or reaches the file system,;
|
||||
%let exp8=&exp8 the environment or the macro symbol table;
|
||||
|
||||
%mpe_assert_abort(
|
||||
desc=Checking a refused function in VAR_LOGIC is rejected,
|
||||
outref=wb8,
|
||||
msg=&exp8
|
||||
)
|
||||
|
||||
/**
|
||||
* Test 9 - the fields the hook owns rather than the scan: ACTIVE is a picker
|
||||
* in the EDIT screen (HARDSELECT over the MPE_SELECTBOX pair), so a submission
|
||||
* that bypasses the grid cannot land a value the screen would never offer.
|
||||
*/
|
||||
data work.sascontroltable;
|
||||
action='LOAD';
|
||||
message='An ACTIVE value the screen cannot offer should abort';
|
||||
libds="&dc_libref..MPE_DERIVED";
|
||||
output;
|
||||
stop;
|
||||
run;
|
||||
|
||||
data work.jsdata;
|
||||
length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
|
||||
VAR_LOGIC $4000 VAR_FMT $32
|
||||
_____DELETE__THIS__RECORD_____ $3;
|
||||
LIBREF='DCTEST';
|
||||
DSN='DC_DERIV';
|
||||
VAR_NM='BADACT';
|
||||
VAR_DEPS='';
|
||||
VAR_TYPE='N';
|
||||
VAR_LEN=8;
|
||||
VAR_FMT='';
|
||||
VAR_LOGIC='1';
|
||||
ACTIVE=2;
|
||||
_____DELETE__THIS__RECORD_____='No';
|
||||
run;
|
||||
|
||||
%mpe_execute(&_program,
|
||||
viyacontext=&defaultcontext,
|
||||
inputdatasets=work.sascontroltable work.jsdata,
|
||||
outlib=web9,
|
||||
outref=wb9,
|
||||
mdebug=&sasjs_mdebug
|
||||
)
|
||||
|
||||
%mpe_assert_abort(
|
||||
desc=Checking an ACTIVE value outside the picker pair is rejected,
|
||||
outref=wb9,
|
||||
msg=MPE_DERIVED: BADACT: ACTIVE must be 1 or 0
|
||||
)
|
||||
|
||||
/* tidy up so repeated runs start clean */
|
||||
proc sql noprint;
|
||||
delete from &dc_libref..mpe_derived
|
||||
|
||||
@@ -34,7 +34,7 @@
|
||||
<h4> SAS Macros </h4>
|
||||
@li mf_nobs.sas
|
||||
@li mp_abort.sas
|
||||
@li mpe_derived_check.sas
|
||||
@li mp_derivedvarcheck.sas
|
||||
|
||||
<h4> Related Macros </h4>
|
||||
@li mpe_derived.sas
|
||||
@@ -100,14 +100,43 @@ run;
|
||||
)
|
||||
|
||||
/**
|
||||
* Field level checks - the same rules %mpe_derived applies at read time
|
||||
* Field level checks. The rules for the derived variable itself - the name,
|
||||
* the type, length, format, label and the expression - are core's, in
|
||||
* mp_derivedvarcheck, so this hook and the read time engine (%mpe_derived ->
|
||||
* mp_derivedvars) scan a row with the same rules and report the same reasons.
|
||||
* What is added here are the two fields that belong to Data Controller rather
|
||||
* than to the expression: the control table key, which the loader normalises
|
||||
* but does not require, and ACTIVE, which the EDIT screen offers as a picker
|
||||
* over the MPE_SELECTBOX pair - so a submission that bypasses the grid cannot
|
||||
* land a value the screen would never offer.
|
||||
*/
|
||||
%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
|
||||
data work.derived_dcfield(keep=name reason);
|
||||
set work.staging_ds;
|
||||
if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
|
||||
length name $32 reason $200;
|
||||
call missing(reason);
|
||||
name=upcase(strip(var_nm));
|
||||
if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
|
||||
else if missing(active) or active not in (0,1)
|
||||
then reason='ACTIVE must be 1 or 0';
|
||||
if not missing(reason) then output;
|
||||
run;
|
||||
|
||||
%mp_derivedvarcheck(inds=work.staging_ds
|
||||
,outds=work.derived_field
|
||||
,reserved=%str(_____DELETE__THIS__RECORD_____)
|
||||
)
|
||||
|
||||
/* the control table fields are reported before the expression's own rules, so
|
||||
which of the two passes found a row does not change the message it gets */
|
||||
data work.derived_check;
|
||||
set work.derived_dcfield work.derived_field;
|
||||
run;
|
||||
|
||||
%if %mf_nobs(work.derived_check)>0 %then %do;
|
||||
data _null_;
|
||||
set work.derived_check(obs=1);
|
||||
call symputx('der_errmsg',cats('MPE_DERIVED: ',var_nm,': ',reason));
|
||||
call symputx('der_errmsg',cats('MPE_DERIVED: ',name,': ',reason));
|
||||
call symputx('der_errflag',1);
|
||||
run;
|
||||
%end;
|
||||
|
||||
@@ -217,9 +217,11 @@ run;
|
||||
* caller asked to keep them (KEEPME in the input data). Keeping them is for
|
||||
* test harnesses, which reseed before every run and cannot redeploy first. */
|
||||
%if &keepme ne 1 %then %do;
|
||||
%ms_deletefile(&root/services/admin/configurator.sas)
|
||||
%ms_deletefile(&_program..sas)
|
||||
%ms_deletefile(&root/services/admin/configurator.sas);
|
||||
%ms_deletefile(&_program..sas);
|
||||
%end;
|
||||
%else %do;
|
||||
%put NOTE: &_program retained, as KEEPME was requested;
|
||||
%end;
|
||||
%else %put NOTE: &_program retained, as KEEPME was requested;
|
||||
|
||||
|
||||
@@ -298,6 +298,8 @@ run;
|
||||
/* keep the program when KEEPME was requested, so a reseed does not need a
|
||||
* redeploy first */
|
||||
%if &keepme ne 1 %then %do;
|
||||
%mv_deletejes(path=&root/admin, name=makedata)
|
||||
%mv_deletejes(path=&root/admin, name=makedata);
|
||||
%end;
|
||||
%else %do;
|
||||
%put NOTE: &_program retained, as KEEPME was requested;
|
||||
%end;
|
||||
%else %put NOTE: &_program retained, as KEEPME was requested;
|
||||
Reference in new issue
Block a user