refactor(mocks): build the control library makedata.sas builds #487

Merged

allan merged 4 commits from feat/mocks-makedata-parity into release/7.23 2026-11-06 14:43:41 +00:00

Conversation 4 Commits 4 Files Changed 14

+3726 -2773

hermes commented 2026-11-06 13:13:30 +00:00

Collaborator

Copy link

Copy source

Why

The JS mock's admin/makedata was doing two jobs: building the control library, and creating the tables the Cypress suite needs. That meant the mock estate was never the estate a SAS deployment builds, and the two had drifted - MPE_MAXKEYVALUES and MPE_SIGNOFFS were not created at all, and MPE_CONFIG (17/23), MPE_VALIDATIONS, MPE_TABLES, MPE_SELECTBOX and MPE_XLMAP_RULES (10/26) were missing seed rows.

What changed

• admin/makedata.js now transcribes mpe_makedatamodel.sas + mpe_makedata.sas: all 40 model tables with their column metadata, and every seed row the macro writes.
• admin/createdemodata.js (new) owns the Cypress fixtures that used to live in makedata (TESTDATA, MPE_X_SEARCH, MPE_X_WIDE, MPE_DERIVED, DC996664). It strips its rows before re-adding them and refuses to run before makedata.
• CI workflows, the Sites demo seed and CONTRIBUTING call createdemodata after makedata.
• harness.seededEstate() builds the estate the way CI does; { demo: false } gives the control library alone.

Verified against real SAS output

Fresh deploy of the server target to the shared estate, makedata with a clean path, then public/getrawdata (type=CSV) for every control table and a dictionary.columns probe uploaded with sasjs run.

• Schema: 40/40 tables, 339/339 columns match the real dataset on name, order, type, length, format and label.
• Rows match row for row for MPE_TABLES (19), MPE_SELECTBOX (76), MPE_VALIDATIONS (86), MPE_XLMAP_RULES (26), MPE_XLMAP_INFO (1), MPE_COLUMN_LEVEL_SECURITY (1), and the empty tables. Remaining differences are environmental or deliberate: the libref (DC###### vs the mock's own), MPE_DATADICTIONARY owner (sysuserid vs sasdemo), the SAS metadata group names in MPE_GROUPS/MPE_ROW_LEVEL_SECURITY/MPE_SECURITY, the resolved DC_MACROS path, and MPE_X_TEST's 491 ranuni() sample rows (the mock keeps the seven deterministic ones; formats are applied at display time).

That run found six defects a source-only comparison could not, all fixed here:

1. int columns are 8 bytes in real SAS, not 4 (5 columns: MPE_TABLES, MPE_DATALOADS x3, MPE_LINEAGE_COLS).
2. MPE_AUDIT.IS_DIFF's label had lost a doubled space.
3. DC_MACROS held the SAS expression cats(symget('path'),'/dc_macros') instead of a path.
4. The three email templates held a literal \u0001 instead of newlines.
5. 75 MPE_SELECTBOX rows held %mf_increment(rk) instead of their key.
6. One MPE_VALIDATIONS row was truncated at a ; inside its regex.

Tests

• makedata-parity.test.js: parses both macros and compares them with the seeded estate (table set, column list and order, seeded row counts and fields), plus a new check that every value fits its column type - that one fails on the previous file (escaped newline, then %mf_increment(rk) in a numeric column).
• createdemodata.test.js: fixtures appear, a second run does not accumulate, refuses before makedata.

Verification run here

• node --test sas/mocks/test/*.test.js - 73 pass, 0 fail.
• makedata-parity.test.js against the pre-change makedata.js - fails (missing MPE_MAXKEYVALUES, MPE_CONFIG 17 vs 23, demo libraries present); against the pre-fix version of this branch - fails on the value-type guard.
• bash .gitea/scripts/tests/cypress-changes.test.sh - all cases passed.
• sasjs compile -t server-ci (mocks) and sasjs cbd -t server (estate deploy) - both succeeded.
• Not run here: the Cypress suite itself, and no SAS 9 or Viya branch was exercised.

## Why The JS mock's `admin/makedata` was doing two jobs: building the control library, and creating the tables the Cypress suite needs. That meant the mock estate was never the estate a SAS deployment builds, and the two had drifted - **MPE_MAXKEYVALUES** and **MPE_SIGNOFFS** were not created at all, and **MPE_CONFIG** (17/23), **MPE_VALIDATIONS**, **MPE_TABLES**, **MPE_SELECTBOX** and **MPE_XLMAP_RULES** (10/26) were missing seed rows. ## What changed - `admin/makedata.js` now transcribes `mpe_makedatamodel.sas` + `mpe_makedata.sas`: all 40 model tables with their column metadata, and every seed row the macro writes. - `admin/createdemodata.js` (new) owns the Cypress fixtures that used to live in makedata (TESTDATA, MPE_X_SEARCH, MPE_X_WIDE, MPE_DERIVED, DC996664). It strips its rows before re-adding them and refuses to run before makedata. - CI workflows, the Sites demo seed and CONTRIBUTING call `createdemodata` after `makedata`. - `harness.seededEstate()` builds the estate the way CI does; `{ demo: false }` gives the control library alone. ## Verified against real SAS output Fresh deploy of the `server` target to the shared estate, `makedata` with a clean path, then `public/getrawdata` (`type=CSV`) for every control table and a `dictionary.columns` probe uploaded with `sasjs run`. - **Schema: 40/40 tables, 339/339 columns match** the real dataset on name, order, type, length, format and label. - **Rows match row for row** for MPE_TABLES (19), MPE_SELECTBOX (76), MPE_VALIDATIONS (86), MPE_XLMAP_RULES (26), MPE_XLMAP_INFO (1), MPE_COLUMN_LEVEL_SECURITY (1), and the empty tables. Remaining differences are environmental or deliberate: the libref (`DC######` vs the mock's own), `MPE_DATADICTIONARY` owner (sysuserid vs `sasdemo`), the SAS metadata group names in MPE_GROUPS/MPE_ROW_LEVEL_SECURITY/MPE_SECURITY, the resolved `DC_MACROS` path, and MPE_X_TEST's 491 `ranuni()` sample rows (the mock keeps the seven deterministic ones; formats are applied at display time). That run found six defects a source-only comparison could not, all fixed here: 1. `int` columns are 8 bytes in real SAS, not 4 (5 columns: MPE_TABLES, MPE_DATALOADS x3, MPE_LINEAGE_COLS). 2. MPE_AUDIT.IS_DIFF's label had lost a doubled space. 3. `DC_MACROS` held the SAS expression `cats(symget('path'),'/dc_macros')` instead of a path. 4. The three email templates held a literal `\u0001` instead of newlines. 5. 75 MPE_SELECTBOX rows held `%mf_increment(rk)` instead of their key. 6. One MPE_VALIDATIONS row was truncated at a `;` inside its regex. ## Tests - `makedata-parity.test.js`: parses both macros and compares them with the seeded estate (table set, column list and order, seeded row counts and fields), plus a new check that every value fits its column type - that one fails on the previous file (escaped newline, then `%mf_increment(rk)` in a numeric column). - `createdemodata.test.js`: fixtures appear, a second run does not accumulate, refuses before makedata. ## Verification run here - `node --test sas/mocks/test/*.test.js` - 73 pass, 0 fail. - `makedata-parity.test.js` against the pre-change makedata.js - fails (missing MPE_MAXKEYVALUES, MPE_CONFIG 17 vs 23, demo libraries present); against the pre-fix version of this branch - fails on the value-type guard. - `bash .gitea/scripts/tests/cypress-changes.test.sh` - all cases passed. - `sasjs compile -t server-ci` (mocks) and `sasjs cbd -t server` (estate deploy) - both succeeded. - **Not run here:** the Cypress suite itself, and no SAS 9 or Viya branch was exercised.

hermes added 1 commit 2026-11-06 13:29:16 +00:00

refactor(mocks): build the control library makedata.sas builds ... a6cfb12579

makedata seeded two things at once: the control library, and the tables the
Cypress suite needs. The estate it produced was therefore not the estate a
SAS deployment produces - and it had drifted: MPE_MAXKEYVALUES and
MPE_SIGNOFFS were never created, and MPE_CONFIG, MPE_VALIDATIONS, MPE_TABLES,
MPE_SELECTBOX and MPE_XLMAP_RULES were missing seed rows that mpe_makedata.sas
inserts.

makedata now transcribes mpe_makedatamodel.sas and mpe_makedata.sas: every
model table with its columns, and every seed row the macro writes. The
Cypress fixtures move to services/admin/createdemodata.js, which the test
estate runs after makedata; it strips and re-adds its rows so a reseed does
not accumulate.

Checked against real SAS output on the shared estate (fresh deploy of the
server target, makedata, then getrawdata for each control table and a
dictionary.columns probe): all 40 tables and 339 columns match on name, type,
length, format and label, and the seeded rows match row for row apart from
the environment (libref, user id, group names) and the sample rows. That run
found six defects a source-only comparison could not: INT columns are 8 bytes
not 4, one label had lost a double space, the DC_MACROS setting held the SAS
expression instead of a path, the email templates held an escaped newline,
75 selectbox rows held %mf_increment(rk) instead of their key, and one
validation row was truncated at a ';' inside its regex.

makedata-parity.test.js parses both macros and compares them with the seeded
estate, and fails on a value that does not fit its column type, so the next
drift fails a test instead of shipping. The CI workflows, the Sites demo seed
and CONTRIBUTING call createdemodata after makedata.

hermes force-pushed feat/mocks-makedata-parity from 75b509c6d8 to a6cfb12579 2026-11-06 13:29:16 +00:00 Compare

hermes referenced this issue from a commit 2026-11-06 13:36:27 +00:00

fix(mocks): answer the review on #487

hermes added 1 commit 2026-11-06 13:36:27 +00:00

fix(mocks): answer the review on #487 ... 5f89314aaf

- createdemodata drops the unused servicesDir binding.
- the DC996664 copy no longer swallows a failure: a broken multi-load fixture
  now fails the seed where it happened instead of surfacing in a Cypress spec.
- makedata answers OUTFMT=JSON with a makedata_response member (as the SAS
  service does) and the CLI payload asks for it, so the `|| true` comes off
  makedata in build.yaml, release.yaml and sites.yaml - a failed seed names the
  service that failed instead of reappearing as "createdemodata needs the
  control library".
- the makedata comment points at sas/mocks/test/makedata-parity.test.js, not a
  macro that does not exist.

hermes commented 2026-11-06 13:36:33 +00:00

Author

Collaborator

Copy link

Copy source

Thanks - all four taken, each one agreed as stated. Head is now 5f89314a.

1. agreed - unused servicesDir. Removed; the service resolves dataDir and nothing else.

2. agreed - the copy loop hid a broken fixture. The try/catch and the warning are gone: MPE_X_TEST/MPE_TABLES are read and written unconditionally, so a failed copy aborts the seed where it happened instead of showing up later in a spec.

3. agreed, and fixed at the cause rather than by dropping the guard. The || true existed because sasjs request cannot parse the HTML body makedata returns for the deploy screen. makedata now honours OUTFMT the way the SAS service does: OUTFMT=JSON returns a makedata_response member, anything else returns the HTML page. The CLI payload (sas/mocks/deploy/makedata.json) now asks for JSON, so || true is gone from makedata in build.yaml, release.yaml and sites.yaml - a failed seed fails the job and names the service, instead of reappearing as "createdemodata needs the control library". New test: MAKEDATA answers the CLI seed with JSON, and the screen with HTML. The stale note in .agents/skills/dc-cypress/SKILL.md that told readers to expect invalid Json string is updated too.

4. agreed - wrong file named. The comment points at sas/mocks/test/makedata-parity.test.js.

Verification: node --test sas/mocks/test/*.test.js - 74 pass, 0 fail; bash .gitea/scripts/tests/cypress-changes.test.sh - all cases passed.

Thanks - all four taken, each one agreed as stated. Head is now `5f89314a`. 1. **agreed - unused `servicesDir`.** Removed; the service resolves `dataDir` and nothing else. 2. **agreed - the copy loop hid a broken fixture.** The `try/catch` and the warning are gone: `MPE_X_TEST`/`MPE_TABLES` are read and written unconditionally, so a failed copy aborts the seed where it happened instead of showing up later in a spec. 3. **agreed, and fixed at the cause rather than by dropping the guard.** The `|| true` existed because `sasjs request` cannot parse the HTML body makedata returns for the deploy screen. `makedata` now honours `OUTFMT` the way the SAS service does: `OUTFMT=JSON` returns a `makedata_response` member, anything else returns the HTML page. The CLI payload (`sas/mocks/deploy/makedata.json`) now asks for JSON, so `|| true` is gone from makedata in `build.yaml`, `release.yaml` and `sites.yaml` - a failed seed fails the job and names the service, instead of reappearing as "createdemodata needs the control library". New test: `MAKEDATA answers the CLI seed with JSON, and the screen with HTML`. The stale note in `.agents/skills/dc-cypress/SKILL.md` that told readers to expect `invalid Json string` is updated too. 4. **agreed - wrong file named.** The comment points at `sas/mocks/test/makedata-parity.test.js`. Verification: `node --test sas/mocks/test/*.test.js` - 74 pass, 0 fail; `bash .gitea/scripts/tests/cypress-changes.test.sh` - all cases passed.

hermes added 1 commit 2026-11-06 14:02:35 +00:00

fix(mocks): keep the derived rows makedata owns ... 112e8e4b8b

createdemodata assigned tables.MPE_DERIVED outright, so any row makedata wrote
there would be wiped by a fixture run - and the idempotency test did not count
the table, so nothing would have said so. Filter the TESTDATA rows out and
append, in the same shape as the tables above, and count MPE_DERIVED in the
idempotency test.

New case: a non-TESTDATA derived row survives a createdemodata run exactly
once. It fails against the previous service and passes now.

hermes commented 2026-11-06 14:02:40 +00:00

Author

Collaborator

Copy link

Copy source

agreed - assigning tables.MPE_DERIVED replaced the array instead of merging into it, and the idempotency test did not count the table, so the wipe would have been silent exactly as described.

Fixed at 112e8e4b:

• the three DERIVED_DEMO rows are now a testDerived list; the control table is filtered on libref !== TESTDATA and the rows pushed, the same shape as MPE_TABLES / MPE_VALIDATIONS / MPE_SELECTBOX above it;
• MPE_DERIVED is added to the list the idempotency test counts;
• new case, a derived row makedata owns survives the fixture rows: it writes a DC_JSLIB derived row, runs createdemodata, and asserts the row is still there exactly once with the TESTDATA rows added beside it. Against the previous service it fails on the makedata-owned row is kept, so the guard discriminates.

node --test sas/mocks/test/*.test.js - 75 pass, 0 fail.

**agreed** - assigning `tables.MPE_DERIVED` replaced the array instead of merging into it, and the idempotency test did not count the table, so the wipe would have been silent exactly as described. Fixed at `112e8e4b`: - the three DERIVED_DEMO rows are now a `testDerived` list; the control table is filtered on `libref !== TESTDATA` and the rows pushed, the same shape as MPE_TABLES / MPE_VALIDATIONS / MPE_SELECTBOX above it; - `MPE_DERIVED` is added to the list the idempotency test counts; - new case, `a derived row makedata owns survives the fixture rows`: it writes a `DC_JSLIB` derived row, runs createdemodata, and asserts the row is still there exactly once with the TESTDATA rows added beside it. Against the previous service it fails on `the makedata-owned row is kept`, so the guard discriminates. `node --test sas/mocks/test/*.test.js` - 75 pass, 0 fail.

hermes reviewed 2026-11-06 14:20:41 +00:00

hermes left a comment

Author

Collaborator

Copy link

Copy source

REVIEWED ON 2026-11-06 14:20 UTC

Reviewed the full base...head diff: the makedata -> makedata + createdemodata split (sas/mocks/sasjs/services/admin/createdemodata.js, makedata.js, deploy/makedata.json, the three workflows, CONTRIBUTING, the cypress spec that references the fixture).

• sas/mocks/sasjs/services/admin/createdemodata.js:1401 - saveTable(tableName, rows, schema[tableName]) writes MPE_X_SEARCH with no column metadata. That table's column list was deleted from makedata.js's schema object and was not carried across: createdemodata's schema is empty (:65) apart from MPE_X_WIDE (:287), so schema.MPE_X_SEARCH is undefined and saveTable falls back to existing.columns (:69). A reseed over an existing estate is fine, but a fresh estate has no existing file - and the CI jobs delete appLoc before seeding - so mpe_x_search.json is written with columns: [], and the demo column order the full-table-search spec derives its counts from no longer exists. Add the MPE_X_SEARCH entry (PRIMARY_KEY_FIELD, SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE, with their labels and lengths) to createdemodata's schema.

1 finding above for review.

REVIEWED ON 2026-11-06 14:20 UTC Reviewed the full base...head diff: the makedata -> makedata + createdemodata split (sas/mocks/sasjs/services/admin/createdemodata.js, makedata.js, deploy/makedata.json, the three workflows, CONTRIBUTING, the cypress spec that references the fixture). - sas/mocks/sasjs/services/admin/createdemodata.js:1401 - saveTable(tableName, rows, schema[tableName]) writes MPE_X_SEARCH with no column metadata. That table's column list was deleted from makedata.js's schema object and was not carried across: createdemodata's schema is empty (:65) apart from MPE_X_WIDE (:287), so schema.MPE_X_SEARCH is undefined and saveTable falls back to existing.columns (:69). A reseed over an existing estate is fine, but a fresh estate has no existing file - and the CI jobs delete appLoc before seeding - so mpe_x_search.json is written with columns: [], and the demo column order the full-table-search spec derives its counts from no longer exists. Add the MPE_X_SEARCH entry (PRIMARY_KEY_FIELD, SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE, with their labels and lengths) to createdemodata's schema. 1 finding above for review.

hermes added 1 commit 2026-11-06 14:42:57 +00:00

fix(mocks): give MPE_X_SEARCH its column metadata back ... 6e0866d876

The fixture's schema entry lived in makedata's schema object (as a second,
later key in the same literal) and did not survive the move to createdemodata.
saveTable falls back to the file already on the drive, so a reseed looked
fine - but CI deletes the appLoc before seeding, and a fresh estate wrote
mpe_x_search.json with columns: [], losing the column order the
full-table-search spec derives its counts from.

Carry the nine columns (name/type/length/format/label/notnull) in
createdemodata's schema, and add a case that every fixture table is written
with column metadata covering its rows, plus the MPE_X_SEARCH order. It fails
against the previous service.

hermes commented 2026-11-06 14:43:02 +00:00

Author

Collaborator

Copy link

Copy source

agreed - MPE_X_SEARCH's schema entry lived in makedata's schema literal as a second, later key (MPE_X_SEARCH: [] earlier, the real list further down), and it did not survive the move. saveTable's fallback to existing.columns hid it on a reseed, exactly as described; CI deletes the appLoc first, so the winning case was the broken one.

Fixed at 6e0866d8:

• createdemodata carries the nine columns (PRIMARY_KEY_FIELD, SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE, EXPEDITION_ID, NOTES) with their labels, lengths, formats and notnull, beside the MPE_X_WIDE schema it already had;
• new case, every fixture table is written with its column metadata: for MPE_X_SEARCH, MPE_X_WIDE, two TESTDATA tables and the DC996664 copy it asserts columns is non-empty, that every field of the first row is described, and that MPE_X_SEARCH's column order is the one the search spec counts from. Against the previous service it fails with DC_JSLIB.MPE_X_SEARCH has no columns.

node --test sas/mocks/test/*.test.js - 76 pass, 0 fail.

**agreed** - MPE_X_SEARCH's schema entry lived in makedata's `schema` literal as a second, later key (`MPE_X_SEARCH: []` earlier, the real list further down), and it did not survive the move. `saveTable`'s fallback to `existing.columns` hid it on a reseed, exactly as described; CI deletes the appLoc first, so the winning case was the broken one. Fixed at `6e0866d8`: - createdemodata carries the nine columns (PRIMARY_KEY_FIELD, SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE, EXPEDITION_ID, NOTES) with their labels, lengths, formats and notnull, beside the MPE_X_WIDE schema it already had; - new case, `every fixture table is written with its column metadata`: for MPE_X_SEARCH, MPE_X_WIDE, two TESTDATA tables and the DC996664 copy it asserts `columns` is non-empty, that every field of the first row is described, and that MPE_X_SEARCH's column order is the one the search spec counts from. Against the previous service it fails with `DC_JSLIB.MPE_X_SEARCH has no columns`. `node --test sas/mocks/test/*.test.js` - 76 pass, 0 fail.

allan merged commit 78b7de7120 into release/7.23 2026-11-06 14:43:41 +00:00

allan deleted branch feat/mocks-makedata-parity 2026-11-06 14:43:41 +00:00

hermes referenced this pull request 2026-11-06 21:27:48 +00:00

feat(licensing): make the app the entry point, licensing via the banner #490

hermes referenced this pull request 2026-11-07 17:41:32 +00:00

feat(release): 7.23 - licence state as a banner, an 849 kB initial load, numbered logs #481

Sign in to join this conversation.

Reviewers

No Reviewers

hermes

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

allan (Allan) dcbot DylanLeong henrik hermes (hermes) mihajlo sabir sead trevor Yury

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dc/dc#487