dc/sas/mocks/test/parity

SAS parity harness

Records what a deployed SAS backend answers, then replays it against the JS mock estate and reports the differences. The mock services are a second implementation; this is the only check that compares behaviour rather than source.

The privacy rule

A golden is a projection, never a copy. project.js keeps a fixed set of members per scenario, drops the whole envelope (SYSHOSTNAME, SYSSITE, SYSUSERID, AUTOEXEC, MF_GETUSER, job and process ids, timings, _PROGRAM), replaces the identifiers the comparison needs (DC###### and DC_JSLIB to <LIBREF>, absolute paths, URLs, emails, staging ids, ISO timestamps, secrets by key name), and then refuses to publish if:

• any identifier class survives (FQDN, IPv4, absolute path, libref, email, eight-digit DC id), or
• any literal the envelope just discarded still appears anywhere in the projection.

That second check is what catches a hostname echoed inside an error message or a home path inside a DCPATH value, which no regex would find. record.js runs both before writing; parity-golden.test.js runs the same scan over every committed golden, so a hand-edited file cannot slip through. One envelope member is dropped but not denied - _PROGRAM, the service path of the request that was just made. It is not uniform across deployments (it carries the target's appLoc), but it is in the URL of the request itself, so publishing it adds nothing a reader does not already have. An abort answers it as MAC, so the value lands in the golden as it arrived; the diff comparing it is a consequence of that, not a reason to keep it.

Record only from the organisation's synthetic estate - one seeded by makedata, plus createdemodata when a scenario needs the fixtures. A customer site would put their rows into a file that looks anonymous and is not: row values are data, not identifiers, and no scan can tell a customer's rows from a demo's. The recorder records estate: synthetic in _meta as a claim a reviewer can check.

Use

# record every scenario from the server target (needs sas/.env.server)
node sas/mocks/test/parity/record.js --target server
# a private estate with its own certificate
node sas/mocks/test/parity/record.js --target server --ca /path/to/estate.pem
# throwaway host only - warns, and _meta records tls: unverified
node sas/mocks/test/parity/record.js --target server --insecure

# record one (an id on its own matches every service that uses it)
node sas/mocks/test/parity/record.js --target server --scenario default
node sas/mocks/test/parity/record.js --target server \
  --scenario editors/getdata/mpe_tables --libref DC634301

# replay against a mock estate and diff (no server, no credentials)
node sas/mocks/test/parity/check.js
node sas/mocks/test/parity/check.js --advisory      # report, exit 0

TLS is verified by default: the response becomes the reference the mock is measured against, and the tool runs with estate credentials. A target that is not https:// is refused outright - the request carries a bearer token and would send it in the clear. --insecure is opt-in, says so on stderr, and is recorded in _meta.tls, so a golden recorded over unverified transport can never be mistaken for one that was not.

Raw responses land in .captures/ (gitignored) so a refusal can be diagnosed without hitting the estate again. Only the projection is committed, under golden/<service>/<scenario>.json.

Scenarios

scenarios.json names, per scenario: the service, the URL parameters, the members to keep, and why the scenario exists. Everything not named in members is dropped by the projection, so a member added upstream is excluded until someone names it - the default is to publish less.

A service that reads a table input rather than URL parameters takes an input block:

"input":     { "table": "SASControlTable", "rows": [ { "LIBDS": "{{LIBREF}}.MPE_TABLES", "FILTER_RK": -1 } ] },
"mockInput": { "table": "SASControlTable", "rows": [ { "LIBDS": "DC_JSLIB.MPE_TABLES",   "FILTER_RK": -1 } ] }

input is posted to the estate and mockInput is handed to the mock at replay, because the rows name the control library and the two sides do not share one - only the values differ, and the shape is the point of the comparison. {{LIBREF}} is filled from --libref, so the estate's library name stays out of the repository.

The uploaded file is not a plain CSV: the server reads its first line as the SAS input statement and the rows from line two, so record.js writes informats (LIBDS :$32. FILTER_RK :best32.) and the runtime rewrites :$N. to :$char. Get that wrong and the control table arrives with zero observations, which the service reports as syscc=1012 prior to dc_getsettings rather than as a bad upload.

A scenario recorded before the mock has been aligned carries "gated": false. The gate then prints its difference count and does not fail, and check.js labels it ADVISORY. The flag is a to-do with a number attached - remove it when the mock matches, at which point the scenario is enforced like the others.

Refreshing the goldens

The build pipeline cannot reach a SAS estate, so re-recording is an ad-hoc job run by an agent or a maintainer holding the estate credentials - not a scheduled CI job. That splits the harness in two:

• The gate is offline and stays in CI. parity-golden.test.js scans the committed goldens, and parity-check.test.js replays every scenario through parity/replay.js against a mock estate built by the test harness - no server, no credentials. Both run in node --test sas/mocks/test/*.test.js, so a mock change that alters a payload fails the pull request. A scenario with no golden fails rather than skipping.
• Differences are declared, not ignored. A scenario's accepted list in scenarios.json names the JSON path of each difference that is expected (the estate's own admin group, its timezone, SAS's space padding of a blank character value) with the reason. Anything else fails, and so does an accepted entry that has stopped differing - the list shrinks to the real residue instead of hiding a fix.
• The refresh needs the estate. Where sas/.env.server exists: record.js --target server, then check.js, then the suite; commit the changed projections. A non-empty diff names the member that moved, so the refresh is a reviewed change rather than a silent overwrite - and when only SAS moved (a version bump, an upstream service change), the diff in the pull request is the whole report.

Runbook

Run from the repository root, with the estate reachable and sas/.env.server holding an ACCESS_TOKEN (what sasjs auth login -t server writes). Deploy the release you are recording from first: a golden describes the build it was taken from, and recording against the previous one silently pins the old behaviour.

1. Find the control library. It is the estate's own DC######, not the mock's DC_JSLIB. node sas/mocks/test/parity/record.js --target server --scenario public/startupservice/default answers it, but the committed golden masks it (DCLIB projects to <LIBREF>), so read the value from that scenario's raw response, .captures/public/startupservice/default.raw.json - globvars[0].DCLIB. That run also refreshes the startup golden.
2. node sas/mocks/test/parity/record.js --target server --libref DC###### re-records every scenario; add --scenario <service>/<id> to scope it.
3. node sas/mocks/test/parity/check.js replays the new goldens against the mock estate and prints each difference. A DIFF line means the mock must change; an ADVISORY line means a scenario still carries "gated": false.
4. node --test sas/mocks/test/*.test.js runs the gate the pull request will run. A scenario whose members changed fails on the recorded list, not on a value - re-record it rather than editing the golden.
5. Commit the projections, and in the same commit delete any accepted entry that has stopped differing and add one for a difference that is genuinely estate state, with a reason a reviewer can weigh.

Two refusals are worth recognising. REFUSED to publish names a literal the projection kept - the raw response is in .captures/<service>/<id>.raw.json, gitignored, so the refusal can be diagnosed without hitting the estate again; fix the projection or the service, never the golden. A missing sas/.env.server means no credentials on this machine: the refresh is an agent or maintainer job, never a CI one, and the target must stay inside SYNTHETIC_TARGETS.

The case a column name is reported in

A table keeps the case SAS created it with - proc sql create table with alert_event leaves alert_event in dictionary.columns, and the DDL reports it that way - while every payload the client reads (getdata, getcols, viewdata, the variables in $sasdata/$viewdata) reports the name upper case. So the mock's table files store the creation case, which is what the DDL generator prints, and every service builds its payload keys through payloadKey() in dcMockUtils.js. A service that reads a row by col.name instead of payloadKey(col) gets undefined, which shows up as an empty grid rather than as an error.

Fixture divergences that are deliberate

createdemodata owns two differences from a SAS estate, both driven by existing specs rather than by the product:

• MPE_X_TEST loses one of the rules mpe_makedata.sas seeds for it: the HARDREGEX on SOME_SHORTNUM, which rejects 1-5. regular_excel.xlsx - the file the specs upload to dc_jslib - holds five rows where SOME_SHORTNUM is 5, so with the rule in place that upload is refused before it reaches the approval screen. SAS's other rules for the table are kept, because they are the ones the mock has always had and editor.cy.ts exercises them (the dropdown on a cell comes from the HARDSELECT_HOOK on SOME_NUM).
• MPE_X_TEST's SOME_CHAR/SOME_DATE/SOME_DATETIME/SOME_TIME carry LABELs that differ from their names, which viewer-labels.cy.ts asserts, while a SAS estate has no labels on that table.

Both are asserted in test/createdemodata.test.js, so a change to either side is a failing test rather than a broken Cypress run. A future scenario covering editors/getdata on MPE_X_TEST would show them as accepted differences; today the startup scenario does not touch that table.

Recording is always against the synthetic estate, and that is enforced rather than asserted: record.js carries the list of target names that point at it (SYNTHETIC_TARGETS, currently server) and refuses every other target, including the ones that point at a customer site. It cannot be detected from the response - makedata is the installer, so a customer's control library holds the same rows. An agent refreshing goldens against a customer site would put their rows into a file that looks anonymous and is not, and no scan can tell the two apart afterwards.

Status

Every scenario in scenarios.json is gated: parity-check.test.js replays it against the mock estate and fails on any difference that is not declared in accepted with a reason. A scenario recorded before the mock has been aligned carries "gated": false instead, which makes it advisory - the gate prints its difference count and does not fail - and the flag comes off once the mock matches. The scenario's own note says what it pins; the per-scenario diff counts live in the gate's output, not here, because a count changes with any mock edit.

What cannot be goldened

Some answers are estate state rather than service behaviour, and a golden for them would be a recording of one installation's setup. They are excluded instead of accepted difference by difference, because the difference is in the rows, not in a path:

• public/getgroups lists the SASjs Server's own groups (all-users, public, and every group created on that server). A fresh install and a test estate answer different lists, so the mock derives its groups from MPE_GROUPS and the startup scenario accepts the admin group's name as estate state.

Next, and why the obvious recording does not work:

• A table the caller may not see. %mpe_accesscheck lets an admin through, and the synthetic estate's user is in DC_ADMIN_GROUP, so every table is editable there and the "not authorised to edit" abort cannot be produced without changing the estate's security rows - which would make the golden a recording of that change. It needs an estate with a non-admin login before it can be recorded.
• The TSQL and PGSQL flavours of public/getddl. The SAS flavour is generated from the table's own metadata and recorded (public/getddl/mpe_tables), byte for byte. The other two dialects are not implemented: the mock answers the SAS text for them, which the download menu labels by flavour, so the gap is visible rather than silent. Each needs its own recording before it is generated - the shape is not guessable from the macro alone.
• public/getcols on a libref that is not assigned. The estate aborts Invalid libds; the mock falls back to searching every libref and answers. Tightening it is a behaviour change for the specs that pass a libref the mock does not hold a folder for, so it needs a survey of those callers before the golden.

See the parity section of the data-controller skill for the surrounding workflow.

Provenance of a golden

Every golden is written by record.js from a live response, never from the mock - a file that the mock produced would make the comparison circular, so the recorder makes the source legible in _meta:

• sas is read from the response envelope's SYSVLONG (family only, e.g. 9.4M9). Only a SAS runtime emits that, so a golden carrying it cannot have come from the mock. When a scenario answers with something other than a SAS JSON envelope - public/getrawdata returns the CSV text itself - the recorder writes unknown, which is the accurate answer rather than a guess. The accepted differences for such a scenario are still read from a live estate response.
• estate is synthetic because the target in sasjsconfig.json declares it, and record.js refuses a target that does not: the flag is the recorder's evidence, not a label typed into the golden.
• tls records whether the conversation was verified or the opt-in --insecure was used, so a reader knows how the answer was obtained.
• dc is the version in package.json at record time. The estate does not report its own DC version through any service, so this is the repository's version and not the deployment's - the one field of _meta that is not read from the response, and it is named here so nobody reads it as the estate's.

A golden that is stale - recorded against an older dc or a different sas family - still compares, because the body is what the gate reads; the _meta block is where a human sees that the answer predates their change.