c407f85000cd412cf70785a53fd7dfea9d49aa6d
34
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dc
|
39dfdf1230 |
fix(mocks): getrawdata reads the library it was asked for, and aborts when it cannot
Stacks on #517, which fixed the download headers of this service. The mock loaded every table from the control library's folder, whatever LIBDS the request named, and answered an unresolvable table with a 200 whose body was the headers and nothing else. getrawdata.sas assigns the library it was given (%dc_assignlib(READ,&lib)) and reads the data from there; the control library supplies only the registration row that names the validity columns to drop. So: - the table now loads from data/<LIBREF>/, which is where makedata and createdemodata write each library's files, and the LIBDS is upcased first as `call symputx('table',upcase(_table))` does - a table that cannot be resolved aborts before anything is written, with the message and MAC the estate answers - recorded, not inferred: NOLIB.MPE_TABLES -> Target table not found: NOLIB.MPE_TABLES MP_ABORT DC634301.NO_SUCH_TABLE -> Target table not found: DC634301.NO_SUCH_TABLE WORK.NOPE -> Target table not found: WORK.NOPE The abort is mp_derivedvars', so the MAC is that macro's own default, and the message carries the upcased LIBDS rather than the service name. Recorded: new parity scenario public/getrawdata/missing_table, golden taken from the synthetic estate (one accepted difference, SYSCC, as for the getddl abort). Tests: download-headers.test.js gains two - a download of TESTDATA.DEMO_ORDERS, which lives outside the control library and used to answer an empty file, and the three unresolvable-table cases above. Both go red against the old code: loading from the control folder fails the first, and removing the abort fails the second. Whole mock suite: 146 tests, 146 pass; parity/check.js ok on all 23 scenarios. |
||
|
dc
|
8ed51607a8 |
fix(mocks): answer getddl the way the estate does, headers and all
Six findings from the review of #510, each checked against the synthetic estate's own public/getddl response before being changed: - the download is an application/text attachment, not octet-stream, and the filename is unquoted: the estate sends `content-type: application/text` and `content-disposition: attachment; filename=<libref>_<ds>.ddl` - the DDL header timestamp is the session's local time, as %sysfunc(datetime(), datetime19.) is, not UTC - the estate answers 07NOV2025:22:27 where UTC getters give 21:27, and the projection masks the comment, so nothing else could see the drift - the headers are written after the DDL is generated, as %mp_streamfile runs after %mp_ds2ddl, so nothing is sent before a request that cannot produce a file has aborted - the refusal of an unknown flavour was invented: getddl.sas passes it to mp_ds2ddl, which has branches for SAS, TSQL and PGSQL only and leaves the file holding its opening comment. The estate answers that as a 200 download, and the mock now does too; a new parity scenario, unknown_flavour, pins it - the "all forty control tables" claim in the DDL comment overstated what the committed goldens show: MPE_TABLES is the table recorded, in three flavours - getcols: drop the dead `libref = knownLib` assignment Tests: node --test sas/mocks/test/*.test.js - 133 pass, 0 fail, and parity/check.js reports 0 differences for every getddl scenario. |
||
|
dc
|
d4335e83c6 |
docs(mocks): the getcols libref work is in this branch now
The README still listed a LIBDS naming a library the estate does not have as outstanding work - the mock falls back to searching every libref and answers. That is done: the library is validated first and the fallback is gone, and the entry would have contradicted the code whichever of the two pull requests landed second. Folds in #515 (the two getcols aborts, the NOLIB scenario and its unit test) so the mock parity work is one branch rather than one per finding. |
||
|
dc
|
4be34a760e |
fix(mocks): tell a malformed LIBDS from one getcols cannot resolve
getcols.sas answers two different things and the previous commit collapsed them. "Invalid libds" comes from the shape check (%mp_validatecol - a libref of eight characters or fewer, a member of 32 or fewer), and only from there. A LIBDS that passes the shape and cannot be resolved reaches the second abort, msg=syscc=&syscc. Probed on the synthetic estate, that second abort says syscc=3000 with a SAS syntax error behind it for a well-formed unknown library, for a missing member in a known library, and for WORK.NO_SUCH alike - a defect in getcols.sas rather than a designed message. The mock reproduces it as it is, because that is what the client renders in production today, and the comment says so rather than pretending it is designed. The scenario is re-recorded with NOLIB.MPE_TABLES, which passes the shape check; the old value, NO_SUCH_LIB, is eleven characters and only ever exercised the shape abort, so its golden proved less than its name claimed. The unit test now covers the malformed, the unresolvable and the answerable cases separately. |
||
|
dc
|
a3f76b45b2 |
Merge release/7.23 into the mock parity branch
# Conflicts: # sas/mocks/test/harness.js # sas/mocks/test/parity/scenarios.json |
||
|
dc
|
f851bfafb4 |
fix(mocks): refuse a library getcols does not have, as the estate does
getcols.sas validates LIBDS against the libraries it can assign before it reads anything (%mpe_validatecol, then dc_assignlib), so a LIBDS naming a library the estate does not have aborts "Invalid libds". The mock searched every libref instead, so NO_SUCH_LIB.MPE_TABLES answered with the columns of the copy that does exist in the control library - a request the estate refuses, answered with data from somewhere else. The named library is resolved the way the estate assigns it, and a missing or malformed LIBDS reaches the same abort. A member missing from a library that does exist is left alone: the estate fails inside its query there, and there is no message worth copying. Recorded as public/getcols/wrong_libref. Also fixes an intermittent suite failure this work exposed: the harness pruned earlier session directories, so a service whose headers are written with the async fs.writeFile could land after its directory was removed, and the ENOENT took the test process down. Each run still gets its own directory - that is what stops a stale body being served - and the estate is removed when the process exits. |
||
|
dc
|
98c93127e2 |
feat(mocks): generate the TSQL and PGSQL flavours of the DDL download too
public/getddl answered the SAS text for both remaining flavours, which the download menu labels by name. Each is generated from the same table metadata now, and each was checked against the estate's own output before it went in: all forty control tables, byte for byte, in all three dialects. The dialects differ in more than type names. Postgres quotes with double quotes and declares its key as a second statement after the table; TSQL brackets the identifiers, carries the key inside the create table as a UNIQUE NONCLUSTERED index, and turns every column LABEL into a sys.sp_addextendedproperty block after the GO. Only a format whose name starts with DATETIME becomes a timestamp in either - a DATE9. or E8601DT26.6 column stays DOUBLE PRECISION or decimal(18,2), which is what the estate does and is not what a format list would suggest. Recorded as public/getddl/mpe_tables_pgsql and public/getddl/mpe_tables_tsql, so each dialect is a gate rather than something re-checked by hand. |
||
|
dc
|
96f70d53b6 |
feat(mocks): generate the DDL download from the table, not a sample
public/getddl answered one fixed Postgres-flavoured body for every request - DC_JSLIB.MPE_X_TEST whatever dataset was asked for - which is the last read path the client calls where the mock invented its answer. It now builds the SAS-flavour DDL from the table file, the way mp_ds2ddl builds it from dictionary.columns and dictionary.indexes: the header, the create table with each column's type, format, NOT NULL and label, and the unique index. The list-output spacing was derived from the estate's own answers and reproduces them byte for byte for all forty control tables. Recorded as public/getddl/mpe_tables, so a mock change that alters the text fails the gate. The header credits a user and a timestamp, so the projection masks both - a user id is not something to publish, and the comparison is about the DDL. TSQL and PGSQL are not generated yet: a caller asking for one gets the SAS text, and the README says so rather than inventing a second dialect without a recording to check it against. |
||
|
dc
|
9b225e9208 |
docs(parity): say what is true about the exemption and the decoration
Two comments outran their evidence. The getcolvals docstring said no fixture reached a decorating format, while the same commit adds a dollar8.2 one that does - the padding and decimals are pinned, the symbol is not. COMMA and SIZEKMG are the formats with no fixture, and it says that now. The _PROGRAM exemption was justified by the path being uniform across deployments. It is not: it carries the target's appLoc. The reason that holds is that the path is the route of the request that was just made and is already in that request's URL, so publishing it adds nothing. Both the projection comment and the README say that instead. |
||
|
dc
|
890f931507 |
docs(parity): _PROGRAM is exempt because it is not sensitive
The comment and the README explained the exemption by what it buys - a MAC compared rather than masked - which was not the reason. A service route is not sensitive and is the same on every deployment, so it is dropped with the rest of the envelope but not denied; that the abort's MAC therefore arrives as it is is a consequence, and the diff comparing it is incidental. |
||
|
dc
|
a6a87e63a6 |
fix(mocks): format a dropdown at the column's format, and check the library
Four findings from the review of this branch, all of them comparisons against a live estate response. `public/getcolvals` hardcoded best12 for every numeric column. The estate formats `formatted` with put() at the column's own format - mf_getVarFormat with force=1 - so a BEST32. column reports $32./32 and pads the value to 32, and the metadata's unformatted copy keeps the column's label rather than the variable name. A widthless format now gets its own default width: the estate answers $16./16 for a DATETIME. column, where the fallback was 200. The same service also searched every libref, so a library the estate does not have still answered for a table that exists elsewhere. The named libref is resolved first and refused as "Invalid libds" - the message %mpe_validatecol gives, before the member is looked for; a missing member in a known library stays "Target table not found". `public/getcols` sorted the column list with localeCompare, the ordering this branch already fixed in viewtables. SAS compares the character value, where '_' sorts after every letter, so A_B comes after AB; the list now sorts that way, with a fixture whose names differ only in underscore placement. An empty getcolvals result answered empty strings where the estate sends one row of SAS missings - a blank for the character and null for the number. Recorded from the estate: a formatted number (MPE_AUDIT.OLDVAL_NUM), a widthless temporal (MPE_DATASTATUS_CATS.CREATED) and a wrong library that names a real table. The header-file test now polls for the line instead of waiting a fixed 20ms. |
||
|
dc
|
38949b4857 |
test(parity): compare an abort's MAC instead of masking it
The `<SERVICE>` placeholder this branch added to publish the getddl abort also projected the mock's MAC, so a mock reporting a different program would have compared equal to the estate. The service path is the app's own route, identical on every deployment, so it is not an identifier to protect: `_PROGRAM` is now dropped from the response but exempt from the deny list, and the MAC is compared as it arrives. The golden is re-recorded with the literal path. The FQDN class grows a lookbehind so a path ending in `.sas` is not read as a hostname - the abort's MAC ends that way. The README says which envelope member is dropped but not denied, and its runbook step 1 now names the raw capture that holds the control library: the committed startup golden masks `DCLIB` to `<LIBREF>`, so the step could not be followed from it. |
||
|
dc
|
6bb0316f2d |
test(mocks): record two fault paths, and write the refresh runbook
`public/getddl` for a dataset that is not there and `public/getcolvals` for a table that is not there - the two aborts the mock used to answer with a body. Both carry one accepted difference, `SYSCC`, for the reason the existing abort scenarios already record: the estate's wrapper adds it to an aborted response, and a JS program's envelope is finalised at CLOSE. The README gains the refresh runbook the status section kept promising: deploy first, how to find the estate's control library, what each gate line means, and how to read the two refusals. It also names what cannot be recorded yet and why. The "not authorised" abort needs a non-admin login, because %mpe_accesscheck lets the admin group through and the synthetic estate's user is in it. The getddl body needs index metadata the mock's table files do not carry. And getcols' libref tolerance is a behaviour change that needs a survey of its callers first. |
||
|
dc
|
b39bdd8ad7 |
fix(mocks): abort where the SAS services abort
Two requests that the estate refuses were answered as if they had worked. `public/getddl` streamed its fixed sample body for any request at all - a dataset it has never heard of, an unassigned library - and the message it did throw for a missing `ds` carried two dots where SAS writes one. It now resolves the member through the mock's own table files and aborts with SAS's message, "Dataset <libref>.<ds> was not found", reported against the program because that is the `mac` the service passes to `%mp_abort`. `public/getcolvals` answered an empty dropdown for a table that is not there. The estate aborts from `%mp_derivedvars` before any values are selected, so the mock does too, with that macro's own MAC. The projection needed one rule to publish either golden: `%mp_abort(mac= &_program)` puts the service's own path in the payload, and `_PROGRAM` is an envelope literal the deny list refuses. A service path is now `<SERVICE>`, which is the app's own route rather than anything about the estate. Both rules are asserted in test/fault-paths.test.js. |
||
|
dc
|
ac8a012827 |
test(mocks): record three more read paths, and gate the download
Recorded from the synthetic estate: the library browse (public/viewtables), a dropdown's values for a character column and for a numeric one (public/getcolvals), and an Excel map (editors/getxlmaps). The first four scenarios recorded in the earlier pass now gate the download that already matched - public/getrawdata had no differences left but still carried "gated": false, so a regression there would have been reported and not failed. public/getgroups is deliberately not recorded. Its answer is the SASjs Server's own group list - all-users, public, and every group created on that install - so a golden would be a copy of one estate's setup, and the difference is in the rows rather than in a path that `accepted` can name. The mock derives its groups from MPE_GROUPS, and the startup scenario already accepts the admin group's name as estate state. The README now says so, and names the two paths still to align: the security abort a table the caller may not see produces, and getddl's hardcoded body. |
||
|
dc
|
1b26cc57e3 |
docs(mocks): say what parseCsvTable does now that it keeps cells verbatim
Its docstring still described the trimmed comparison the previous commit removed, which is exactly the kind of comment a later reader follows back into the bug. It now states that cells are kept as they arrive, padding included, that a leading or trailing blank is part of a SAS character value, and that only the framing - the quotes and the line endings the state machine consumes - is removed. Rebased onto release/7.23 after #492 and #494 merged: the scenario list carried two entries with the id mpe_tables (public/viewdata from #492, public/getcols from this PR), which the merge made a conflict; both are kept, keyed by service and id together, which is how the golden path and the checker already treat them. 114 pass, 0 fail, eight scenarios ok. |
||
|
dc
|
ab2d9131df |
docs(mocks): say how a golden was obtained, and stop trimming CSV cells
Three findings from review. - the abort envelope: no service depended on the CLOSE-time assignment. Members are on the session's webout file as they are added, and the server serves that file whether or not the program finished (api/src/controllers/session.ts sessionOutput returns webout.txt, catching to ''), so a caller receives what was flushed; every abort writes sasjsAbort first, so an abort is never an empty body. The comment now says that, says that dropping CLOSE is estate parity (a SAS soft abort stops before the envelope is finalised), and keeps the note that the estate's wrapper is what adds SYSCC to its own aborts. - parseCsvTable trimmed every cell, quoted content included, so the download comparison could not see a leading or trailing blank - the very class of difference the viewer work just established. Cells are kept as they arrive, a quote opens a field only at its start, and a test pins all three cases. Removing the trim changed no scenario: all seven still ok. - provenance: parity/README.md now has a section saying where each _meta field comes from. sas is the envelope's SYSVLONG, which only a SAS runtime emits, so a golden carrying it cannot have come from the mock; a scenario that answers with something other than a SAS JSON envelope (the getrawdata CSV) records unknown, and that is why that one golden says so while its accepted differences still come from a live response. estate is the target's declared flag, which record.js refuses to do without; dc is the repository version at record time and is named as the one field not read from the response. |
||
|
dc
|
d21152d4e2 |
fix(mocks): gate the abort on the registry, guard the placeholder, cover the path check
Four findings from review.
- stagedata: the abort needs a registry that loaded and does not hold the row; a
drive where MPE_TABLES itself is absent is a different failure and the message
would be a lie.
- record.js checks for {{LIBREF}} before the first substitution rather than only
before the POST, so a scenario that names its table in the query string fails
with the same instruction instead of recording against an empty libref.
- makedata_actions: the macro triggers are asserted one at a time ('%', '&', and
a control character), so removing one from brCheckPath fails the test rather
than being masked by the other; the path-shape case is its own test.
- the getcols drop is parity, not a regression: the estate's getcols returns
NAME, LENGTH, VARNUM, LABEL, FMTNAME, FORMAT, TYPE, DDTYPE and nothing else,
and the picker's column-level security comes from getdata's dqrules/READONLY.
A fallback that needs CLS_RULE is a change to getcols.sas, not to its mock.
The abort shape change (no SYSCC/SYSENCODING on a soft abort) is estate-wide and
deliberate - it is what an aborted SAS response looks like.
|
||
|
dc
|
d84331659d |
fix(mocks): gate the stagedata abort on registration, and stop claiming SYSCC on an abort
Four findings from review. - stagedata aborted on the target's data file existing, which is the state of a first load into a newly registered table - staging there has to work. The gate is the registration now (finalReg, already computed above), and the comment says why: a table registered nowhere is the case a user hits when it is dropped between loading the grid and saving. - getcols rebuilt a character column's FORMAT as '$<length>.' whatever the column declared, so a $CHAR10. or $UPCASE. column was described with a format it does not have. mpGetcols' own FORMAT wins when there is one, the length-derived default applies only when there is none, and an undeclared length no longer produces '$0.'. - getrawdata pushed the CSV header row for every requested type, so the EXCEL, MARKDOWN and PDF bodies - which stream the same lines under their own content type - began with a CSV header. Only type=CSV gets one now. - the harness put SYSCC on the envelope at OPEN, claiming the runtime does that for a program that never closes. It does not: sasjs/server's JS runtime finalises the envelope at CLOSE (createJSProgram.ts -> weboutFinaliseStatement), which is why a JS abort carries neither SYSCC nor SYSENCODING. The stub is back to adding both at CLOSE, and the two abort scenarios declare the difference they now show: the SAS estate's wrapper adds SYSCC to an abort, a JS program stops before the envelope is finalised. The message and macro name the client renders are unchanged. check.js - seven scenarios, five at zero accepted differences. Suite: 99 pass, 0 fail. |
||
|
dc
|
bfadbf24fd |
feat(mocks): record the edit submission and the download, and stop serving stale bodies
Two more scenarios, and both needed recorder support that did not exist.
editors/stagedata on a table that is registered nowhere (0 accepted
differences). An edit submission sends *two* table inputs - the control table
and the edited rows as jsdata - so the recorder now posts several parts and the
harness declares several inputs, as the runtime does. The mock answered a
completed body where the service aborts with "Target table not found"; it aborts
that way now, and mpAbort no longer closes the body, because a SAS soft abort
does not reach the state that carries SYSENCODING.
public/getrawdata (0 accepted differences). A download answers CSV rather than
JSON, so scenarios can now declare format: csv and the body is parsed to
{header, rows} on both sides before the diff - SAS quotes and pads each cell to
the variable's width, so a text diff would fail on spacing that says nothing.
The mock was missing the header row, was exporting the registration's validity
columns (getrawdata.sas drops them: data staged(drop=&txfrom &txto)), and broke
any value containing a newline across two records because it did not quote it.
The harness now clears the response file before each run. Without that, a
service that only assigns _webout was handed the *previous* service's body -
which was hiding a stale-result bug in a DIRLIST test: the mock accepted
/tmp/%sysfunc(system(id)) as a parent, and the test passed because it was
reading the abort from the run before it. brCheckPath refuses '&' and '%' now,
which is the guard the test was written for.
check.js - seven scenarios, five of them at zero accepted differences:
ok public/startupservice/default (8 accepted difference(s))
ok editors/getdata/mpe_tables (1 accepted difference(s))
ok public/viewlibs/default (1 accepted difference(s))
ok public/getcols/mpe_tables (0 accepted difference(s))
ok editors/getdata/missing_table (0 accepted difference(s))
ok editors/stagedata/target_missing (0 accepted difference(s))
ok public/getrawdata/selectbox (0 accepted difference(s))
read-paths.test.js covers the two-input abort and the download's header/validity
columns/quoted newline. Suite: 99 pass, 0 fail.
|
||
|
dc
|
2903d1421c |
feat(mocks): record the column list and the abort path
Two more scenarios, both read-only, and both silent in the mock until now - neither of them broke a Cypress spec, they just tested less than they claimed. public/getcols (0 accepted differences). The filter panel builds its operator set and value quoting from this list, and the mock answered the *getdata* column shape: CLS_RULE, MEMLABEL, DESC, LONGDESC and COLTYPE, with FORMAT blank for every column. The estate answers NAME, LENGTH, VARNUM, LABEL, FMTNAME, FORMAT, TYPE, DDTYPE - VARNUM being the column's position in the table rather than in the sorted list - so the mock now emits exactly that. editors/getdata on a table that is registered nowhere (0 accepted differences). getdata.sas aborts, and the client renders the message; the mock answered an empty grid with an empty dsmeta instead, which is a different answer rather than a quieter one. It now aborts with the same message and macro name, and - like mp_abort - stops before CLOSE. That last point needed the harness to be more faithful: the runtime puts SYSCC on the envelope whether or not the program closed it, while SYSENCODING only appears on a completed body. The stub adds SYSCC at OPEN now and SYSENCODING at CLOSE, so an aborted response matches the estate's shape instead of being accepted as a difference. record.js also cuts the response at SASJS_LOGS_SEPARATOR before parsing: an aborted call comes back with the log appended, the same way a _debug response does, and the recorder used to fail on the trailing text. check.js: ok public/startupservice/default (8 accepted difference(s)) ok editors/getdata/mpe_tables (1 accepted difference(s)) ok public/viewlibs/default (1 accepted difference(s)) ok public/getcols/mpe_tables (0 accepted difference(s)) ok editors/getdata/missing_table (0 accepted difference(s)) read-paths.test.js pins both fixes; the suite is 97 pass, 0 fail. |
||
|
dc
|
b6b6a3fefd |
feat(mocks): record public/viewdata and bring the viewer's payload to parity
The viewer was the third read path the harness could not see. Recorded from the
seeded estate it showed 364 differences; all of them were the mock's, and none
remain.
- cols: sorted by column name, and a column LABEL comes from the
MPE_DATADICTIONARY row for the lib.ds.column when there is one - viewdata.sas
augments mp_getcols the same way getdata does. The format name is upper case,
as SAS reports it.
- rows: a temporal value is rendered the way the estate renders it - two-digit
year and a two-decimal fraction ("01JAN60:00:00:00.00") - rather than by the
format label on the column, which reads DATETIME19.3. A character value is
trimmed and an all-blank one is a space, matching the missing-value rule the
data rows and the synthetic no-data row already share.
- $viewdata: the column's own metadata (num, 8 bytes, DATETIME19.3). The mock
reused the entry built for the edit grid, where the temporal columns have been
converted to character, so the viewer was told $200 char.
- sasparams.PK_FIELDS: the validity column ahead of the business key, which is
what viewdata answers ("TX_TO LIBREF DSN").
sas/mocks/test/viewdata.test.js pins the datetime text and the empty-table row.
|
||
|
dc
|
f32027fe59 |
fix(mocks): read the control rows as they are stored, and say what is gated
Nine findings from review, all taken. Row reads (getdata): - approversForTable reads every column through field(): the security and group rows are seeded with UPPERCASE keys and an edited row comes back lower-cased, so a direct property read compared undefined with 'UNDEFINED' and found no grant - silently falling back to the admin group for every table. - tx_to is handled by rowInForce(): epoch seconds, which is what Date.now()/1000 assumes, with an ISO string accepted; a blank, absent or unparseable end date is open-ended rather than expired, so a rule or a grant that omits it is no longer dropped as NaN. The dqrules filter reads rule_active/base_lib/base_ds through field() for the same reason. Rows and the text that describes them (getdata): - the current-rows clause in FILTER_TEXT is now applied to the selection: the grid no longer shows expired rows while telling the user they were excluded. - the primary-key sort resolves each key to the column's own casing, so a lowercase DDL sorts on real values instead of comparing undefined. Settings (getdata, viewlibs): - dcLibref falls back to 'DC_JSLIB' like dcAdminGroup already did: a deploy that has not run makedata has no settings.js, and viewlibs must still list the control library. Wording that had gone stale: - the parity README's Status paragraph now states the invariant (every scenario is gated; gated:false is advisory) instead of counts and a list of differences that were fixed, and the scenarios.json notes say what each scenario pins now. - viewdata's comment states the contract rather than the behaviour it replaced, and the synthetic no-data row writes the same space for a blank character cell as the data rows do. Tests: getdata-rules.test.js covers an uppercase-keyed APPROVE grant and an open-ended rule (both fail on the previous code); viewdata.test.js covers the numeric missing and the empty-table row. |
||
|
dc
|
3bb9fb52e3 |
fix(mocks): make getdata and viewlibs answer what the estate answers
The two scenarios recorded in the previous commit are now gated, with one declared difference each instead of 388 and 4. The diffs were the mock's, and they are fixed rather than accepted: - cols: dropped the DESC key SAS never emits, and took LABEL/LONGDESC from the MPE_DATADICTIONARY rows for this lib.ds.column, as getdata.sas augments mp_getcols. COLTYPE is blank for DATE/DATETIME/TIME, which is what the estate answers. - sasdata: sorted by the business key; character values trimmed the way the JSON writer writes them, with an all-blank value as a single space. - dqrules: the active MPE_VALIDATIONS rows unioned with the schema's NOTNULL (limited to the columns left in the selection, which is why a NOT NULL on tx_from never reached the grid) and the derived columns' READONLY, then de-duplicated and sorted by column and rule type with X left at the missing value - getdata.sas's union/sort/by-group logic. - sasparams: the current-rows clause mpe_filtermaster appends, and blank character values as the single space SAS sends. - maxvarlengths: the value as stored (trimmed), and the key columns in upper case, which is how the selection spells them. - approvers: computed from MPE_SECURITY's APPROVE rows with the admin group as the fallback, instead of a hardcoded sasdemo. - viewlibs: the shape %mp_getlibs answers - engine in LIBRARYID, blank ENGINE, libref as LIBRARYNAME. The control library and the admin group now come from services/settings.js through dcInit(), which is the mock's %mpe_init(): makedata already writes dc_libref, dc_admin_group and dcpath there, so a deployment with a different DCPATH no longer needs a service change. test/mock-settings.test.js proves it by renaming the library and moving the data. The remaining services still carry const dcLibref = 'DC_JSLIB' and compute paths from it at file scope, so each is a separate change. The sas9 mock keeps its own getdata/viewlibs variant, so it is untouched: the goldens are recorded from the SASjs Server estate. |
||
|
dc
|
49a9bbf022 |
feat(mocks): record getdata and viewlibs, and show what the mock gets wrong
Two more scenarios, and the recorder could not take either of them as it stood. - editors/getdata reads a table input, not URL parameters, so record.js can now post one: it writes the informational header the server expects (the first line is the SAS input statement, not a list of column names), posts it as multipart/form-data, and takes --libref so the estate's control library name stays out of scenarios.json. replay.js hands the mock its own rows through mockInput, since the two sides do not share a libref. - the test harness's webout() stub now honours the showmeta option and writes the body as members are added. Without them the mocks' own requireMemberMetadata() guard throws - it reads the body before CLOSE - and a service that stopped sending formats and labels would have passed here. - --scenario accepts service/id, because two screens legitimately call their first service 'default'. The two goldens are recorded but marked gated: false: the gate prints their difference counts and does not fail, and check.js labels them ADVISORY. They are a work list, not an acceptance. editors/getdata/mpe_tables - 388 differences, all of them the mock's: cols carries a DESC key SAS never emits and misses the LABEL/LONGDESC/COLTYPE SAS reads from MPE_DATADICTIONARY; sasdata leaves blank character values out instead of padding them and is not ordered by the business key; dqrules differ in content and order; approvers is hardcoded to one user where SAS computes it and returns none. public/viewlibs - 4 differences: SAS answers LIBRARYID='V9' with a blank ENGINE and LIBRARYNAME equal to the libref, and lists SASHELP; the mock answers LIBRARYID=<libref>, ENGINE='V9', LIBRARYNAME='Data Controller(<libref>)' and has no SASHELP. dsmeta is excluded on purpose: it is the engine's report on the dataset (owner, host, encoding, paths, created dates), estate state rather than behaviour, and the projection refused the recorded response until it was dropped - the site id and the service account name were in it. |
||
|
dc
|
909580dbda |
docs(mocks): cite the fixture that actually hits the SOME_SHORTNUM rule
Two documentation findings from review:
- The comment in createdemodata.js and the parity README said the excel
fixtures hold "3, 18, 34, 79", which points at values the dropped rule never
saw. The uploads that the rule applies to go to dc_jslib, and
regular_excel.xlsx has five rows where SOME_SHORTNUM is 5 - that is the value
the 1-5 HARDREGEX rejects. The 3s belong to a multi-load fixture whose sheet
targets DC996664.MPE_X_TEST, a libref that carries no seeded validations
because getdata filters MPE_VALIDATIONS on base_lib === requestedLibref and
createdemodata copies only data files into it. Both places now name
regular_excel.xlsx and the value 5.
- README:9 promised the scan refuses an "eight-digit id" while the added class
is \bDC\d{8}\b. It now says "eight-digit DC id", which is what the scan does
and what the date case in parity-golden.test.js relies on.
Checked the fixture rather than the diff: column H (SOME_SHORTNUM) of
regular_excel.xlsx holds 18, 34, 79, 74, 98 ... and exactly five rows of 5.
|
||
|
dc
|
038c40499b |
refactor(mocks): name the synthetic estate by target name, not a sasjsconfig key
The previous commit added "synthetic": true to the server target to stop the recorder claiming an estate it had not checked. That put a harness-only key into the shared build config, where nothing in sasjs reads it and a maintainer could reasonably read it as a sasjs feature. The recorder now carries the list of target names that point at the estate makedata builds (SYNTHETIC_TARGETS = ['server'], which is sas.4gl.io locally) and refuses anything else, so sasjsconfig.json is untouched. The declaration is the same one - it cannot be derived from a response, because makedata is the installer and a customer's control library holds the same rows - it just lives with the tool that acts on it. --target server passes the guard (and needs a token, as before); --target server-mihajlo is refused with a message naming the allowed target. |
||
|
dc
|
7815dcccb8 |
fix(mocks): drop only the MPE_X_TEST rule the excel fixtures break
The previous commit removed every validation rule SAS seeds for MPE_X_TEST, which fixed the excel specs and broke editor 3: the cell dropdown they click for comes from the HARDSELECT_HOOK on SOME_NUM, and that hook had gone with the rest. Only one of SAS's rules actually conflicts with the fixtures - the HARDREGEX on SOME_SHORTNUM, which rejects 1-5 while duplicate_row_excel.xlsx and friends hold 3, 18, 34, 79. Everything else stays, so the mock's rule set for the table is again exactly what it was before the seed work: SOFTSELECT on SOME_BESTNUM, HARDREGEX and SOFTREGEX on SOME_CHAR, SOFTREGEX on PRIMARY_KEY_FIELD and the SOME_NUM hook. The parity README now says this, rather than claiming the table is rule-free, and createdemodata.test.js asserts both halves - the hook is present and the SOME_SHORTNUM rule is not. |
||
|
dc
|
dd2a777438 |
fix(mocks): restore the MPE_X_TEST fixtures and pin the replay's timezone
The parity gate's Cypress job went red after the seed work landed, and the two
causes are the same change seen from the fixture side:
- makedata now carries SAS's demo validation rules for MPE_X_TEST, including the
HARDREGEX on SOME_CHAR that blocks any text but 'the'/'data'. The excel
fixtures upload arbitrary values into that table and expect the submission to
reach the approval screen, so ten excel specs timed out with no Approve
button. createdemodata drops MPE_X_TEST's rules, which is what editor.cy.ts
already documents ("MPE_X_TEST stays clean to match the excel fixtures").
- the LABELs the viewer-labels spec asserts went with the hand-written schema.
createdemodata restores them, because a SAS estate has none and the control
library should not claim otherwise.
Both are declared as fixture divergences in the parity README and asserted in
createdemodata.test.js, so the next change to either side fails a test instead
of a Cypress run.
Review findings:
- the SYSTIMEZONEOFFSET acceptance depended on the host's zone; both entry
points pin TZ=UTC before the harness loads, so the accepted list is the same
everywhere.
- the sas9 mock still published '+0100'; it now answers seconds east of UTC
like the standard estate.
- the README promised an eight-digit id scan that IDENTIFIER_CLASSES no longer
had. A DC-prefixed eight-digit class is back (a plain digit class would fire
on a yyyymmdd date), with a case proving a date is not refused.
|
||
|
dc
|
94cd4912f9 |
fix(mocks): make the golden's claims evidence, not labels
Six findings from review: - estate: 'synthetic' was a constant. The target now declares "synthetic": true in sasjsconfig.json and record.js refuses any other target, so the claim in the file is one the tool had to check before it wrote anything. - project() dropped the key when scrubbing a value, so a top-level member named like a secret was published unredacted. The key is passed through, and checkPublishable now runs secretViolations on the projection, so the recorder refuses before writing rather than leaving it to the next pull request. - denyList restated a subset of ENVELOPE_NEVER. It is built from that list now, so an error text or a hostname in a discarded field is caught too (values shorter than four characters are skipped - they would match half the file). - replay() projected with golden._meta.members while the test validated scenario.members, so editing the scenario had no effect until a re-record. It projects with the scenario's list and reports a member-list mismatch. - replay() deleted SYSCC/SYSENCODING from the expected side when the harness omitted them, which would hide a service that stopped emitting them. The harness now adds the two members the way the server does, and they are compared. - the eight-digit siteId scan class matched any eight-digit payload value (a date, an amount) and refused clean goldens. Removed; the site id is caught by the deny list, which now covers it. |
||
|
dc
|
a57557c60e |
fix(mocks): wire the replay into CI and close the review gaps
Four findings:
- the replay was never run by CI, so the README's "a mock change fails the
pull request" was not true. parity-check.test.js now replays every scenario
through a shared parity/replay.js (which check.js also uses), fails on a
difference that is not declared and on a declared one that no longer differs,
and fails a scenario that has no golden. The differences that remain are
declared per scenario with a reason: the estate's admin group, its
registration state, its timezone and SAS's space padding of blank characters.
- recording stamped tls: verified for a non-https target while sending the
bearer token in the clear. A non-https target is refused outright now, and
--insecure still records tls: unverified.
- the committed-golden gate never applied the secret-key rule. secretViolations
now walks a golden (and a fresh projection) for secret-named fields whose
value is not <REDACTED>, with a case proving a hand-typed licence key fails.
- arg() returned argv[index + 1] unconditionally, so a trailing --ca reached
readFileSync(undefined). Flags without a value are a usage error now.
The replay also found a real mock defect, fixed here: SYSTIMEZONEOFFSET was
emitted as +HHMM while SAS answers seconds east of UTC ('3600'), and the mock's
own test asserted the wrong form - exactly the drift this harness exists for.
The sasdatasets sort used localeCompare, which puts MPE_X_CATALOG-FC before
MPE_XLMAP_RULES; SAS sorts on the byte value, so the mock now does too.
|
||
|
dc
|
6c831bb230 |
fix(mocks): verify TLS when recording, and project staging ids whole
Two findings from review: - record.js turned certificate verification off unconditionally, on the request whose response becomes the golden. Verification is now the default; --ca trusts a private estate's own certificate and --insecure is opt-in, warns on stderr and is recorded as _meta.tls so a golden taken over unverified transport cannot be mistaken for one that was not. The estate's certificate verifies, so the committed golden is tls: verified. - the libref placeholder ran before the staging-id one, so a staging id came out as <LIBREF>-1_2_3. The staging rule now runs first and matches both minted shapes (DC<date>-<time>_<n>_<n>, and the mock's millisecond form); a project() case pins the ordering. |
||
|
dc
|
b98fde1c74 |
docs(mocks): record that the parity refresh is an ad-hoc agent job
The pipeline cannot reach a SAS estate, so the goldens are refreshed where the credentials are - an agent or a maintainer running record.js - and the CI half stays offline: scan the committed goldens, replay them against a mock estate. Say that in the harness README so the next reader does not go looking for a scheduled refresh job that cannot exist. |
||
|
dc
|
6262843f3f |
feat(mocks): check the mock against recorded SAS responses
The mock services are a second implementation of the SAS backend, and the only thing comparing them was a human reading two files. This adds the harness that compares their output instead. A golden is a projection of a SAS response, never a copy: project.js keeps a fixed member list per scenario, drops the whole envelope (hostname, site id, user, home paths, job ids, _PROGRAM), substitutes the identifiers the diff needs (<LIBREF>, <PATH>, <EMAIL>, <STAGING_ID>, <DATETIME>), and redacts secrets by key name. The recorder then refuses to publish if any identifier class survives, or if any literal the envelope just discarded still appears in the projection - which is what catches a hostname echoed inside an error message. record.js records from a live estate into .captures/ (gitignored) and writes only the projection to golden/. check.js replays a golden against a mock estate built by the test harness and prints field-level differences; it needs no server and no credentials. parity-golden.test.js re-runs the identifier scan over every committed golden, so a hand-edited file cannot slip through either. Phase 0 ships one scenario, public/startupservice. The replay is advisory and already reports twelve differences worth a look (admin group casing, the registration counters, timezone representation, the sasdatasets order, blank versus empty Excel map descriptions); the corpus, the merge gate and the nightly drift job follow. Stacked on the mocks context in the parent branch: it seeds the mock estate with seededEstate(), which that branch adds. |