From ad061946c3105872b8c3df3dc38d5b921efd4c4b Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 28 Oct 2026 22:03:34 +0000 Subject: [PATCH] docs(skills): add the estate deploy-and-test procedure The shared-estate cycle is two npm scripts and sasjs test, but the seed in npm run server is load-bearing: without it testsetup 404s settings.sas and 59 of the 64 test programs fail to complete, which reads like a broken deployment. Record the targets, the token-pair credentials, the ADMIN group the estate actually uses, how to read the results, and the frontend passphrase steps as a repo-local skill, and point the testing doc at it. --- .agents/docs/testing.md | 3 +- .agents/skills/dc-estate-testing/SKILL.md | 79 +++++++++++++++++++++++ AGENTS.md | 2 +- CONTEXT.md | 2 +- 4 files changed, 83 insertions(+), 3 deletions(-) create mode 100644 .agents/skills/dc-estate-testing/SKILL.md diff --git a/.agents/docs/testing.md b/.agents/docs/testing.md index f2e4dc7..7f15ff5 100644 --- a/.agents/docs/testing.md +++ b/.agents/docs/testing.md @@ -1,10 +1,11 @@ # Testing -Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. +Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name. ## Commands - Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`. +- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`. - Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`. - Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`). diff --git a/.agents/skills/dc-estate-testing/SKILL.md b/.agents/skills/dc-estate-testing/SKILL.md new file mode 100644 index 0000000..9a60c08 --- /dev/null +++ b/.agents/skills/dc-estate-testing/SKILL.md @@ -0,0 +1,79 @@ +--- +name: dc-estate-testing +description: > + Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS + test suite against it. Covers the target and credential setup, the makedata seed the suite needs on + a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend + so the streamed app is the build you just made. +--- + +# Deploying and testing on the shared SASjs Server estate + +The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong. + + cd sas && npm ci # macro dependencies + cd ../client && npm ci && npm run build # frontend only - see "Deploying the frontend" + cd ../sas && npm run server # cpfavicon + sasjs cbd -t server + makedata + npx sasjs test -t server + +`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed. + +## The targets + +`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc` `/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair. + +A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered. + +To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app//dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at. + +## Credentials + +The CLI logs in with a registered client, not a username and password: `.env.` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless. + +Mint the pair from a local login, with the password read from a file rather than argv: + +1. `GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header. +2. `POST /SASLogon/login` with `{"username","password"}`. +3. `GET /SASjsApi/client` on the same session - the registered client ids. +4. `POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`. +5. `POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`. + +An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over. + +## The makedata seed + +`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result. + +- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `AllUsers`, which is not a group on the shared estate - the account belongs to `all-users` (check with a call to `services/public/getgroups`, or read the `MF_GETUSER` the response echoes back). With a group the account is not in, the seed still answers `SUCCESS` and every later `getdata`/`stagedata` call aborts with ` is not authorised to edit . in the .MPE_SECURITY table`. Override `ADMIN` to `all-users` for the estate rather than editing the shipped JSON. +- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset . does not exist` - take the current one from the response. +- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row. +- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order. + +## Reading the results + +Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON. + +A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete. + +`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for. + +## Deploying the frontend + +The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists: + +1. `client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`. +2. `client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines. +3. `cd client && npm ci && npm run build`. +4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc. +5. `cd ../sas && npx sasjs cbd -t ` - with `client/dist` present the log says `Copying web app files` and `Web app is available at `. + +Without step 1 the deploy prints `ERROR An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main..js` the page names. + +Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check. + +## Reading a failed run + +- `ERROR An error occurred while executing the request. Job location: /tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise. +- `WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem. +- `sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree. +- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code. diff --git a/AGENTS.md b/AGENTS.md index bd8d34a..a8fc0ab 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`. -Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead. +Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead. ## Code comments and test names diff --git a/CONTEXT.md b/CONTEXT.md index 10e551f..10cd126 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output - `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`. - `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`. -- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills). +- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills). ## Roles -- 2.53.0