Merge pull request 'docs(skills): add the estate deploy-and-test procedure' (#397) from docs/estate-testing-skill into release/7.20

Reviewed-on: #397

.agents/docs/testing.md

@@ -1,10 +1,11 @@
 # Testing
 
-Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
+Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name.
 
 ## Commands
 
 - Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
+- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`.
 - Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
 - Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
 

.agents/skills/dc-estate-testing/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: dc-estate-testing
+description: >
+  Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS
+  test suite against it. Covers the target and credential setup, the makedata seed the suite needs on
+  a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend
+  so the streamed app is the build you just made.
+---
+
+# Deploying and testing on the shared SASjs Server estate
+
+The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong.
+
+    cd sas && npm ci                             # macro dependencies
+    cd ../client && npm ci && npm run build      # frontend only - see "Deploying the frontend"
+    cd ../sas && npm run server                  # cpfavicon + sasjs cbd -t server + makedata
+    npx sasjs test -t server
+
+`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `<appLoc>/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed.
+
+## The targets
+
+`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc` `/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair.
+
+A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered.
+
+To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app/<you>/dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at.
+
+## Credentials
+
+The CLI logs in with a registered client, not a username and password: `.env.<target>` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless.
+
+Mint the pair from a local login, with the password read from a file rather than argv:
+
+1. `GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header.
+2. `POST /SASLogon/login` with `{"username","password"}`.
+3. `GET /SASjsApi/client` on the same session - the registered client ids.
+4. `POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`.
+5. `POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`.
+
+An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over.
+
+## The makedata seed
+
+`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
+
+- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `AllUsers`, which is not a group on the shared estate - the account belongs to `all-users` (check with a call to `services/public/getgroups`, or read the `MF_GETUSER` the response echoes back). With a group the account is not in, the seed still answers `SUCCESS` and every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Override `ADMIN` to `all-users` for the estate rather than editing the shipped JSON.
+- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
+- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
+- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.
+
+## Reading the results
+
+Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON.
+
+A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete.
+
+`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for.
+
+## Deploying the frontend
+
+The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists:
+
+1. `client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`.
+2. `client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines.
+3. `cd client && npm ci && npm run build`.
+4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc.
+5. `cd ../sas && npx sasjs cbd -t <target>` - with `client/dist` present the log says `Copying web app files` and `Web app is available at <url>`.
+
+Without step 1 the deploy prints `ERROR  An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main.<hash>.js` the page names.
+
+Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check.
+
+## Reading a failed run
+
+- `ERROR An error occurred while executing the request. Job location: <appLoc>/tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/<suite>.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise.
+- `WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem.
+- `sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree.
+- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code.

AGENTS.md

@@ -43,7 +43,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
 
 Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
 
-Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
+Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
 
 ## Code comments and test names
 

CONTEXT.md

@@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output
 
 - `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
 - `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
-- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
+- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).
 
 ## Roles