b87649013f9f29afa57b32d1dd5e948a3ce834a0
632
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 allan
|
c2fcce476e |
Merge pull request 'fix: address the release review - temporal formats, settings, version fallback, Lighthouse' (#506) from fix/release-723-review into release/7.23
Reviewed-on: #506 Reviewed-by: Allan <allan@4gl.io> |
||
 dc
|
2cbaae1a20 |
docs(mocks): say which ISO formats the viewer maps
Only E8601DT, E8601DA and E8601TM are printed as ISO. The zoned E8601DZ and E8601TZ, and the interval forms E8601DTM/DN/MN, fall back to the SAS shape - they would need their offset or interval suffix printed too, and no table on the estate carries one. The comment now says that instead of implying the whole family is covered. |
||
|
dc
|
14a973c61c |
test(mocks): seed from the settings the deployment writes
The case added for the settings read deleted services/settings.js, which the reader tolerates on its own, so it said nothing about reading a moved library. It is replaced with the two cases that do: - a drive whose control library is TESTFIX, with settings.js naming it, seeds the fixtures into TESTFIX and creates no DC_JSLIB at all; - a drive with no compiled dcSettings.js, the parser the readers load, still seeds into the default rather than dying at load. The settings guard also stops accepting a bare dcReadSettings( call: that function only exists once dcSettings.js has been loaded, so the call now counts as a read only when the same file loads the parser. dcInit/loadDcSettings are accepted as before. |
||
|
dc
|
b78a94f9e2 |
fix(mocks): guard the settings load, and carry a rounded fraction
Two findings on #506. createdemodata eval'd the compiled settings program unconditionally, so a drive that has the fixtures but not that file went down at load with an ENOENT before it could report anything. The read is guarded: an absent program evals a stub that answers the same default dcReadSettings gives for a missing file, and the service settles on the control library makedata creates instead of failing. A settings program that exists and does not parse is still a broken deployment, and is left to fail loudly. viewIsoDateTime and viewIsoTime clamped a rounded fraction at scale-1 instead of carrying it, so a value 1e-7 under the next second printed .999999 of the second before it where SAS prints the next second, .000000. Both now round to the printed precision first and split the result, which carries into the second. Both are pinned: createdemodata seeds with settings.js deleted, and an E8601DT26.6 value one rounding error short of the next minute prints 03:05:00. |
||
|
dc
|
0d58dea01b |
fix(mocks): read the control library from the settings in createdemodata
createdemodata was the one service the settings migration left naming DC_JSLIB, with a comment asking the reader to keep two constants in step. On a deployment that moves the library it fails with "run services/admin/makedata first" whatever makedata did. It reads dcReadSettings now, the way getrawdata does - loading dcSettings.js alone, since this service's own table helpers would collide with dcMockUtils. The test that exists to catch exactly this exempted any file whose text mentions settings.js, and createdemodata's comment mentioned it, so the constant passed. The exemption now requires a call to a parser (dcReadSettings/dcInit/loadDcSettings), which is what "reads the settings" means. EOF2 fix(build): guard the version fallback, and page the Lighthouse comment Three findings from the release review. src/version.ts read the root package.json unguarded on the no-tag path: a build layout without that file threw at module load, taking the app down to report a version. It falls back to v0.0.0 now. lighthouse-report.mjs looked for its marker in the first fifty comments, so on a pull request past that count the "updated in place" contract silently became a new comment on every push. It pages until the marker is found, or until a page comes back short. lighthouserc.js justified the 0.85 accessibility floor as sitting "just below the observed minimum" while the assertion takes the median - a route 0.09 under the floor passes when the median clears it. The comment now says what is measured, that this is the trade the threshold makes, and that aggregationMethod 'pessimistic' is the lever if the floor must hold for every route (accessibility would pass that way today, performance would not). |
||
|
dc
|
af532bc421 |
fix(mocks): render a temporal column at its own format
The viewer printed one shape for every datetime - a two-digit year and a two-decimal fraction - whatever format the column carried, while the same payload published that format in `$viewdata.vars`. A column formatted E8601DT26.6 therefore answered 02JAN26:03:04:05.00 where the estate prints 2026-01-02T03:04:05.000000. The E8601 family is now printed as ISO 8601, at the precision its format names (a width with no fraction prints none), and every other format keeps the shape the recorded response pins, which is what the DATETIME19.3 columns already matched. |
||
|
dc
|
9b225e9208 |
docs(parity): say what is true about the exemption and the decoration
Two comments outran their evidence. The getcolvals docstring said no fixture reached a decorating format, while the same commit adds a dollar8.2 one that does - the padding and decimals are pinned, the symbol is not. COMMA and SIZEKMG are the formats with no fixture, and it says that now. The _PROGRAM exemption was justified by the path being uniform across deployments. It is not: it carries the target's appLoc. The reason that holds is that the path is the route of the request that was just made and is already in that request's URL, so publishing it adds nothing. Both the projection comment and the README say that instead. |
||
|
dc
|
25bde80bf4 |
fix(mocks): take a numeric format's width from its own name
Three findings, all of them about the same field. `sasNumericFormatted` only parsed `BEST<w>.<d>`, so a DOLLAR8.2 column fell back to a fixed width of 12 while `$vals` declared $8./8 - the value contradicted the metadata a caller sizes the column from. Any `NAME<w>.<d>` now yields the width and decimals, and the declared width is the fallback; a decorating format is padded and rounded but still arrives without its symbol, which the comment says rather than an implementation it does not have. The `formattedWidth` ternary had a temporal arm that nothing could reach - every temporal column is numeric, so the numeric arm answered it - and the docstring justified a 200 fallback no call site asked for. The width is now computed once, before the values are read, from the numeric arm or the character length, and the fallback is the caller's. The formatting itself had no test with a value in it: the MPE_AUDIT fixture is empty, so only metadata was asserted. A mock table with rows and a formatted numeric now pins the padding and the decimals, for best12.2 and for dollar8.2. |
||
|
dc
|
890f931507 |
docs(parity): _PROGRAM is exempt because it is not sensitive
The comment and the README explained the exemption by what it buys - a MAC compared rather than masked - which was not the reason. A service route is not sensitive and is the same on every deployment, so it is dropped with the rest of the envelope but not denied; that the abort's MAC therefore arrives as it is is a consequence, and the diff comparing it is incidental. |
||
|
dc
|
a6a87e63a6 |
fix(mocks): format a dropdown at the column's format, and check the library
Four findings from the review of this branch, all of them comparisons against a live estate response. `public/getcolvals` hardcoded best12 for every numeric column. The estate formats `formatted` with put() at the column's own format - mf_getVarFormat with force=1 - so a BEST32. column reports $32./32 and pads the value to 32, and the metadata's unformatted copy keeps the column's label rather than the variable name. A widthless format now gets its own default width: the estate answers $16./16 for a DATETIME. column, where the fallback was 200. The same service also searched every libref, so a library the estate does not have still answered for a table that exists elsewhere. The named libref is resolved first and refused as "Invalid libds" - the message %mpe_validatecol gives, before the member is looked for; a missing member in a known library stays "Target table not found". `public/getcols` sorted the column list with localeCompare, the ordering this branch already fixed in viewtables. SAS compares the character value, where '_' sorts after every letter, so A_B comes after AB; the list now sorts that way, with a fixture whose names differ only in underscore placement. An empty getcolvals result answered empty strings where the estate sends one row of SAS missings - a blank for the character and null for the number. Recorded from the estate: a formatted number (MPE_AUDIT.OLDVAL_NUM), a widthless temporal (MPE_DATASTATUS_CATS.CREATED) and a wrong library that names a real table. The header-file test now polls for the line instead of waiting a fixed 20ms. |
||
|
dc
|
38949b4857 |
test(parity): compare an abort's MAC instead of masking it
The `<SERVICE>` placeholder this branch added to publish the getddl abort also projected the mock's MAC, so a mock reporting a different program would have compared equal to the estate. The service path is the app's own route, identical on every deployment, so it is not an identifier to protect: `_PROGRAM` is now dropped from the response but exempt from the deny list, and the MAC is compared as it arrives. The golden is re-recorded with the literal path. The FQDN class grows a lookbehind so a path ending in `.sas` is not read as a hostname - the abort's MAC ends that way. The README says which envelope member is dropped but not denied, and its runbook step 1 now names the raw capture that holds the control library: the committed startup golden masks `DCLIB` to `<LIBREF>`, so the step could not be followed from it. |
||
|
dc
|
6bb0316f2d |
test(mocks): record two fault paths, and write the refresh runbook
`public/getddl` for a dataset that is not there and `public/getcolvals` for a table that is not there - the two aborts the mock used to answer with a body. Both carry one accepted difference, `SYSCC`, for the reason the existing abort scenarios already record: the estate's wrapper adds it to an aborted response, and a JS program's envelope is finalised at CLOSE. The README gains the refresh runbook the status section kept promising: deploy first, how to find the estate's control library, what each gate line means, and how to read the two refusals. It also names what cannot be recorded yet and why. The "not authorised" abort needs a non-admin login, because %mpe_accesscheck lets the admin group through and the synthetic estate's user is in it. The getddl body needs index metadata the mock's table files do not carry. And getcols' libref tolerance is a behaviour change that needs a survey of its callers first. |
||
|
dc
|
b39bdd8ad7 |
fix(mocks): abort where the SAS services abort
Two requests that the estate refuses were answered as if they had worked. `public/getddl` streamed its fixed sample body for any request at all - a dataset it has never heard of, an unassigned library - and the message it did throw for a missing `ds` carried two dots where SAS writes one. It now resolves the member through the mock's own table files and aborts with SAS's message, "Dataset <libref>.<ds> was not found", reported against the program because that is the `mac` the service passes to `%mp_abort`. `public/getcolvals` answered an empty dropdown for a table that is not there. The estate aborts from `%mp_derivedvars` before any values are selected, so the mock does too, with that macro's own MAC. The projection needed one rule to publish either golden: `%mp_abort(mac= &_program)` puts the service's own path in the payload, and `_PROGRAM` is an envelope literal the deny list refuses. A service path is now `<SERVICE>`, which is the app's own route rather than anything about the estate. Both rules are asserted in test/fault-paths.test.js. |
||
|
dc
|
ac8a012827 |
test(mocks): record three more read paths, and gate the download
Recorded from the synthetic estate: the library browse (public/viewtables), a dropdown's values for a character column and for a numeric one (public/getcolvals), and an Excel map (editors/getxlmaps). The first four scenarios recorded in the earlier pass now gate the download that already matched - public/getrawdata had no differences left but still carried "gated": false, so a regression there would have been reported and not failed. public/getgroups is deliberately not recorded. Its answer is the SASjs Server's own group list - all-users, public, and every group created on that install - so a golden would be a copy of one estate's setup, and the difference is in the rows rather than in a path that `accepted` can name. The mock derives its groups from MPE_GROUPS, and the startup scenario already accepts the admin group's name as estate state. The README now says so, and names the two paths still to align: the security abort a table the caller may not see produces, and getddl's hardcoded body. |
||
|
dc
|
1926dd0b6b |
fix(mocks): answer a library browse and a dropdown the way SAS does
The parity recorder found two payload differences that no test covered. `public/viewtables` listed the mock's own file name for the format catalog - MPE_X_CATALOG, a dataset the estate does not have - where SAS lists a catalog as <name>-FC. A member with no variables is the catalog, so it is named that way, and the list is now sorted in SAS byte order: '_' is 0x5F and sorts after 'L', so MPE_XLMAP_RULES precedes MPE_X_CATALOG-FC, which localeCompare got backwards. `public/getcolvals` reported the metadata of a column it had not looked at: $32767. and a length of 32767 for the formatted value whatever the column, an empty SASFORMAT for a column whose format is blank (SAS answers a space), and a bare number where put(x,best.) right-justifies into best12. The metadata now describes the two variables the service creates - $<len>. at the column's own length, best12 for a number, the format's width for a temporal - and the value is formatted to match. Each rule is asserted in test/read-paths-payload.test.js so a regression names the rule rather than a JSON path. |
||
|
dc
|
1b26cc57e3 |
docs(mocks): say what parseCsvTable does now that it keeps cells verbatim
Its docstring still described the trimmed comparison the previous commit removed, which is exactly the kind of comment a later reader follows back into the bug. It now states that cells are kept as they arrive, padding included, that a leading or trailing blank is part of a SAS character value, and that only the framing - the quotes and the line endings the state machine consumes - is removed. Rebased onto release/7.23 after #492 and #494 merged: the scenario list carried two entries with the id mpe_tables (public/viewdata from #492, public/getcols from this PR), which the merge made a conflict; both are kept, keyed by service and id together, which is how the golden path and the checker already treat them. 114 pass, 0 fail, eight scenarios ok. |
||
|
dc
|
ab2d9131df |
docs(mocks): say how a golden was obtained, and stop trimming CSV cells
Three findings from review. - the abort envelope: no service depended on the CLOSE-time assignment. Members are on the session's webout file as they are added, and the server serves that file whether or not the program finished (api/src/controllers/session.ts sessionOutput returns webout.txt, catching to ''), so a caller receives what was flushed; every abort writes sasjsAbort first, so an abort is never an empty body. The comment now says that, says that dropping CLOSE is estate parity (a SAS soft abort stops before the envelope is finalised), and keeps the note that the estate's wrapper is what adds SYSCC to its own aborts. - parseCsvTable trimmed every cell, quoted content included, so the download comparison could not see a leading or trailing blank - the very class of difference the viewer work just established. Cells are kept as they arrive, a quote opens a field only at its start, and a test pins all three cases. Removing the trim changed no scenario: all seven still ok. - provenance: parity/README.md now has a section saying where each _meta field comes from. sas is the envelope's SYSVLONG, which only a SAS runtime emits, so a golden carrying it cannot have come from the mock; a scenario that answers with something other than a SAS JSON envelope (the getrawdata CSV) records unknown, and that is why that one golden says so while its accepted differences still come from a live response. estate is the target's declared flag, which record.js refuses to do without; dc is the repository version at record time and is named as the one field not read from the response. |
||
|
dc
|
e8fb891883 |
docs(mocks): make the abort comments agree, and stop leaving session directories behind
Four findings from review. - mpAbort's comment still said the runtime puts SYSCC on an aborted envelope, which is what I believed before checking: a JS program stops before the envelope is finalised, so an abort carries neither SYSCC nor SYSENCODING. The comment now says that, and says that the SAS estate's wrapper is what adds SYSCC to an abort - the difference the abort scenarios declare. - runService removes the previous run's session directory when it creates the next one: a directory is only read back through estate.weboutPath, which has moved by then, and a test file that runs many services was leaving one behind per call. - brCheckPath: both callers are macro-bound (a DIRLIST parent and a WRITECHECK target that is written into a filename statement), so rejecting % and & at every call is right; the comment says so now instead of leaving a reader to check. - getdata's abort message is the recorded text - 'dataset X.Y does not exist!!' with MAC mpestp_getdata.sas, from the estate's own response - so the mock keeps it rather than inventing a diagnosis. The control-library mismatch that could put a misleading name in that sentence is the hardcoded DC_JSLIB, and #494 removes it. |
||
|
dc
|
d21152d4e2 |
fix(mocks): gate the abort on the registry, guard the placeholder, cover the path check
Four findings from review.
- stagedata: the abort needs a registry that loaded and does not hold the row; a
drive where MPE_TABLES itself is absent is a different failure and the message
would be a lie.
- record.js checks for {{LIBREF}} before the first substitution rather than only
before the POST, so a scenario that names its table in the query string fails
with the same instruction instead of recording against an empty libref.
- makedata_actions: the macro triggers are asserted one at a time ('%', '&', and
a control character), so removing one from brCheckPath fails the test rather
than being masked by the other; the path-shape case is its own test.
- the getcols drop is parity, not a regression: the estate's getcols returns
NAME, LENGTH, VARNUM, LABEL, FMTNAME, FORMAT, TYPE, DDTYPE and nothing else,
and the picker's column-level security comes from getdata's dqrules/READONLY.
A fallback that needs CLS_RULE is a change to getcols.sas, not to its mock.
The abort shape change (no SYSCC/SYSENCODING on a soft abort) is estate-wide and
deliberate - it is what an aborted SAS response looks like.
|
||
|
dc
|
d84331659d |
fix(mocks): gate the stagedata abort on registration, and stop claiming SYSCC on an abort
Four findings from review. - stagedata aborted on the target's data file existing, which is the state of a first load into a newly registered table - staging there has to work. The gate is the registration now (finalReg, already computed above), and the comment says why: a table registered nowhere is the case a user hits when it is dropped between loading the grid and saving. - getcols rebuilt a character column's FORMAT as '$<length>.' whatever the column declared, so a $CHAR10. or $UPCASE. column was described with a format it does not have. mpGetcols' own FORMAT wins when there is one, the length-derived default applies only when there is none, and an undeclared length no longer produces '$0.'. - getrawdata pushed the CSV header row for every requested type, so the EXCEL, MARKDOWN and PDF bodies - which stream the same lines under their own content type - began with a CSV header. Only type=CSV gets one now. - the harness put SYSCC on the envelope at OPEN, claiming the runtime does that for a program that never closes. It does not: sasjs/server's JS runtime finalises the envelope at CLOSE (createJSProgram.ts -> weboutFinaliseStatement), which is why a JS abort carries neither SYSCC nor SYSENCODING. The stub is back to adding both at CLOSE, and the two abort scenarios declare the difference they now show: the SAS estate's wrapper adds SYSCC to an abort, a JS program stops before the envelope is finalised. The message and macro name the client renders are unchanged. check.js - seven scenarios, five at zero accepted differences. Suite: 99 pass, 0 fail. |
||
|
dc
|
469e98b2e6 |
test(mocks): give every harness run its own session directory
The runtime creates a session directory per execution - <root>/sessions/<id> with webout.txt and stpsrv_header.txt beside it - so a program that only assigns _webout cannot be served the previous request's body. The harness used one directory for the whole estate, which is how a stagedata abort once answered for a download. runService now creates <root>/sessions/s<n> per call and points weboutPath and _SASJS_WEBOUT_HEADERS at it, as the runtime does, and estate.weboutPath follows the most recent run so a caller can still inspect a response afterwards. The hooks test writes beside weboutPath and reads it back from the session directory of that run rather than from a hardcoded s1. |
||
|
dc
|
bfadbf24fd |
feat(mocks): record the edit submission and the download, and stop serving stale bodies
Two more scenarios, and both needed recorder support that did not exist.
editors/stagedata on a table that is registered nowhere (0 accepted
differences). An edit submission sends *two* table inputs - the control table
and the edited rows as jsdata - so the recorder now posts several parts and the
harness declares several inputs, as the runtime does. The mock answered a
completed body where the service aborts with "Target table not found"; it aborts
that way now, and mpAbort no longer closes the body, because a SAS soft abort
does not reach the state that carries SYSENCODING.
public/getrawdata (0 accepted differences). A download answers CSV rather than
JSON, so scenarios can now declare format: csv and the body is parsed to
{header, rows} on both sides before the diff - SAS quotes and pads each cell to
the variable's width, so a text diff would fail on spacing that says nothing.
The mock was missing the header row, was exporting the registration's validity
columns (getrawdata.sas drops them: data staged(drop=&txfrom &txto)), and broke
any value containing a newline across two records because it did not quote it.
The harness now clears the response file before each run. Without that, a
service that only assigns _webout was handed the *previous* service's body -
which was hiding a stale-result bug in a DIRLIST test: the mock accepted
/tmp/%sysfunc(system(id)) as a parent, and the test passed because it was
reading the abort from the run before it. brCheckPath refuses '&' and '%' now,
which is the guard the test was written for.
check.js - seven scenarios, five of them at zero accepted differences:
ok public/startupservice/default (8 accepted difference(s))
ok editors/getdata/mpe_tables (1 accepted difference(s))
ok public/viewlibs/default (1 accepted difference(s))
ok public/getcols/mpe_tables (0 accepted difference(s))
ok editors/getdata/missing_table (0 accepted difference(s))
ok editors/stagedata/target_missing (0 accepted difference(s))
ok public/getrawdata/selectbox (0 accepted difference(s))
read-paths.test.js covers the two-input abort and the download's header/validity
columns/quoted newline. Suite: 99 pass, 0 fail.
|
||
|
dc
|
2903d1421c |
feat(mocks): record the column list and the abort path
Two more scenarios, both read-only, and both silent in the mock until now - neither of them broke a Cypress spec, they just tested less than they claimed. public/getcols (0 accepted differences). The filter panel builds its operator set and value quoting from this list, and the mock answered the *getdata* column shape: CLS_RULE, MEMLABEL, DESC, LONGDESC and COLTYPE, with FORMAT blank for every column. The estate answers NAME, LENGTH, VARNUM, LABEL, FMTNAME, FORMAT, TYPE, DDTYPE - VARNUM being the column's position in the table rather than in the sorted list - so the mock now emits exactly that. editors/getdata on a table that is registered nowhere (0 accepted differences). getdata.sas aborts, and the client renders the message; the mock answered an empty grid with an empty dsmeta instead, which is a different answer rather than a quieter one. It now aborts with the same message and macro name, and - like mp_abort - stops before CLOSE. That last point needed the harness to be more faithful: the runtime puts SYSCC on the envelope whether or not the program closed it, while SYSENCODING only appears on a completed body. The stub adds SYSCC at OPEN now and SYSENCODING at CLOSE, so an aborted response matches the estate's shape instead of being accepted as a difference. record.js also cuts the response at SASJS_LOGS_SEPARATOR before parsing: an aborted call comes back with the log appended, the same way a _debug response does, and the recorder used to fail on the trailing text. check.js: ok public/startupservice/default (8 accepted difference(s)) ok editors/getdata/mpe_tables (1 accepted difference(s)) ok public/viewlibs/default (1 accepted difference(s)) ok public/getcols/mpe_tables (0 accepted difference(s)) ok editors/getdata/missing_table (0 accepted difference(s)) read-paths.test.js pins both fixes; the suite is 97 pass, 0 fail. |
||
|
allan
|
15eb7c1121 |
Merge pull request 'feat(mocks): record public/viewdata and bring the viewer payload to parity' (#492) from feat/mock-viewdata-parity into release/7.23
Reviewed-on: #492 |
||
|
dc
|
2144ee96b0 |
fix(mocks): strip blanks, not whitespace, and cite the golden's provenance
Three findings from review. - the trailing-blank strip now matches only blanks. SAS pads with blanks, so a value that ends in a tab is not padding and is not silently altered while the same character mid-string survives. - the `typeof val === 'number'` arm was unreachable: both branches above return text (viewCell and formatSasValue), so the ternary is gone and a comment says why a string is all that arrives. - provenance: the docstrings cited "the recorded response" as the authority, and the golden's `_meta.estate` says synthetic, which invites exactly the doubt you raised. They now name the file and the field that answers it - `_meta.sas` is 9.4M9, read from the live envelope's SYSVLONG, which only a SAS runtime emits and which the mock cannot produce - and point at the Provenance section added to parity/README.md in #493. |
||
|
dc
|
bcfd81207c |
refactor(mocks): one settings parser for the whole estate
Three findings from review. - getrawdata no longer parses settings.js itself. The reader moved into services/dcSettings.js, which dcMockUtils.loadDcSettings calls and which a service that cannot eval the whole utility file (its own formatSasValue and SAS_EPOCH_MS would collide) can load on its own. One file, one parser: the greedy object match now parses JSON where it can and falls back to a quote-agnostic pattern, and the ENOENT-only catch is in that one place. - a behavioural test moves the control library and its data and asserts a getrawdata export still comes from it. This was the least-covered path in the change - static tests cannot follow a reader that does not load the utilities. - viewlibs' comment still said a `|| 'DC_JSLIB'` fallback mattered; it lives in loadDcSettings (which defaults dcLibref to DC_JSLIB), so the comment says that and the dead `||` stays gone. |
||
|
dc
|
4b71cf5b03 |
fix(mocks): keep a leading blank, and round a date the way the text branch does
Three findings from review. - viewdata trimmed both ends of a character cell. A leading blank is part of a SAS character value and the viewer renders it; only the trailing blanks SAS pads to are noise, so the cell is trimmed at the end and an all-blank cell is still SAS missing. viewdata-cells.test.js fails on the previous version. - viewDate floored a stored day count while the ISO branch rounded, so the same instant could land on different days depending on how the row carried it. Both round now. - the two label members stay different. The finding was that cols[].LABEL uses the data dictionary business label while $viewdata.vars[.].label uses the physical one; the recorded estate response does exactly that - cols DSN is "Dataset Name to be edited", $viewdata.vars.DSN.label is "DSN" - so unifying them would break parity. The comment says so, and a test asserts the split against the recorded answer so the next reader does not tidy it away. |
||
|
dc
|
8fabd582f1 |
fix(mocks): read the settings object, not its spelling
Five findings from review. - getrawdata's settings reader was a second parser that only matched a double-quoted key and value, so a hand-edited settings.js would leave this one service on DC_JSLIB while the estate used the configured library. The object is parsed as JSON first (what makedata writes) and a quote-agnostic pattern is the fallback, so both a written and an edited file are read the same way. - the bare catch treated every read failure as "makedata has not run": an EACCES or an encoding failure now surfaces instead of silently defaulting. - `dcInit(...).dcLibref || 'DC_JSLIB'` was unreachable - loadDcSettings already defaults dcLibref to DC_JSLIB - so the `||` is gone from all fourteen services; it would have masked a changed settings shape as a silent default. - the "Load shared DC mock utilities" comment had ended up below the eval it describes in getdynamiccolvals, getcols and getcolvals; it describes the eval, so it sits above it, and the note about which tables live in the control library no longer names DC_JSLIB. - merge order with #493 (getcols, getrawdata, stagedata): whichever lands second is rebased; the getrawdata hunks are separate from that PR's CSV changes. The settings guard now also accepts a `let` declaration and a service that keeps the literal as a default while reading settings.js (getrawdata does). |
||
|
dc
|
40550ca56b |
fix(mocks): pad a date's year, read a stored time as seconds since midnight
Three findings from review, and tests that hold them.
- a DATE below the year 1000 printed three digits ('999'), so DATE9.'s
four-digit year is now zero-padded
- a stored ISO text reached a TIME column through the datetime branch, giving
seconds since 1960 rather than seconds since midnight; a time's day is
discarded, and a text without a zone is read as UTC rather than in the host's
zone (the control box sits east of UTC, which shifted every value by an hour)
- the datetime fraction is clamped at 99 as the time branch already was, so a
fraction that rounds up cannot print three digits
viewdata-temporal.test.js builds a table with DATE, TIME and DATETIME columns and
covers each renderer plus a stored ISO string; the year and the ISO cases fail on
the previous version of the service.
|
||
|
dc
|
38be2382cd |
fix(mocks): load the utilities before using them, and read the settings in getrawdata
Three findings from review. - validatefilter used fetchTable/colVal above the eval that defines them, so it threw on every call - the conversion moved the eval down past the parse. The eval and the dcInit read are back above the parse, with a comment saying why they have to be. Reproduced and checked: the service answers now. - a source-level test walks sas/mocks/sasjs/services/**/*.js and asserts the invariant the copy-paste is supposed to keep: a service that loads the utilities reads the settings after them, calls no utility before them, and does not name the control library; makedata is exempt (it creates it) and a hook is exempt (its body is eval'd in the calling service's scope). This is the guard that would have caught the validatefilter slip, and it costs no estate. - getrawdata reads dc_libref out of services/settings.js rather than naming DC_JSLIB, so the documented CSV row-diff path works on a deployment with a different DCPATH. It cannot eval dcMockUtils - its own formatSasValue and SAS_EPOCH_MS would collide - so it reads the one value it needs; createdemodata stays as it is, because it creates the library. |
||
|
dc
|
4cfc5b1e3d |
fix(mocks): round a time before printing its fraction, and default a temporal format by type
Two findings from review. - viewTime was reached with the unrounded value (viewCell passed the number straight through for TIME), so a stored 3661.9999 produced a fraction of 100 and printed "01:01:01.100". The value is rounded to two decimals on entry, as the DATETIME branch already did, and the fraction is clamped as well so a rounding edge cannot reach three digits. - a temporal column with no format fell back on DATETIME19.3 whatever its ddtype, so DATE and TIME columns were labelled as datetimes. The default follows the ddtype now: DATE9., TIME8., DATETIME19.3. viewdata's scenario is still at zero accepted differences, and the suite is 96 pass, 0 fail. |
||
|
dc
|
f831c0eecc |
refactor(mocks): every service that loads dcMockUtils reads the control library from the settings file
makedata writes services/settings.js - the JS twin of the settings program the backend generates in %mpe_init - with dc_libref, dc_admin_group, dc_staging_area and dcpath. dcInit() reads it. getdata and viewlibs used it; the other thirteen services that load dcMockUtils still carried const dcLibref = 'DC_JSLIB' and built their paths from it, so a deployment with a different DCPATH kept working in two services and broke in the rest. Each of those services now loads dcMockUtils first and takes its libref from dcInit(driveRoot, appLoc).dcLibref, with 'DC_JSLIB' only as the default for a drive makedata has not seeded. Services that computed paths from the constant (stagedata's staging directory, getapprovals' data directory, gethistory's) have the eval above that line, so the derived paths come from the settings too. Two services do not load dcMockUtils at all - createdemodata and getrawdata carry their own table and format helpers - so they keep naming the constant, and say why in a comment: adding the eval just for the settings would collide with their own helper names. mock-settings.test.js gains a case that would fail without this: it renames the control library, moves the data, rewrites the settings file, and asserts a row written into MPE_SUBMIT comes back from approvers/getapprovals. With the hardcoded service it is empty, which is the failure mode this removes. |
||
|
dc
|
b6b6a3fefd |
feat(mocks): record public/viewdata and bring the viewer's payload to parity
The viewer was the third read path the harness could not see. Recorded from the
seeded estate it showed 364 differences; all of them were the mock's, and none
remain.
- cols: sorted by column name, and a column LABEL comes from the
MPE_DATADICTIONARY row for the lib.ds.column when there is one - viewdata.sas
augments mp_getcols the same way getdata does. The format name is upper case,
as SAS reports it.
- rows: a temporal value is rendered the way the estate renders it - two-digit
year and a two-decimal fraction ("01JAN60:00:00:00.00") - rather than by the
format label on the column, which reads DATETIME19.3. A character value is
trimmed and an all-blank one is a space, matching the missing-value rule the
data rows and the synthetic no-data row already share.
- $viewdata: the column's own metadata (num, 8 bytes, DATETIME19.3). The mock
reused the entry built for the edit grid, where the temporal columns have been
converted to character, so the viewer was told $200 char.
- sasparams.PK_FIELDS: the validity column ahead of the business key, which is
what viewdata answers ("TX_TO LIBREF DSN").
sas/mocks/test/viewdata.test.js pins the datetime text and the empty-table row.
|
||
|
dc
|
4c5b38810b |
fix(mocks): pad only a character cell the row does not carry
getdata padded every blank character cell to a space, including one the row holds as an explicit ''. That broke nine editor specs: the grid classifies a row by comparing what it holds against the payload it was given (classifyRow), so a payload whose empty strings had become spaces made every row read as modified on load - the row header showed +/- and DC.ROW_STATUS answered 'M' where 'U' is correct. The rule is now the one SAS expresses: a cell the row does not carry is missing and reaches the client as one space, and a value the row does carry is passed through. MPE_TABLES rows omit their blank columns, so the recorded golden still matches; the TESTDATA formula fixture stores '', and keeps it. getdata-blanks.test.js pins both sides. |
||
|
dc
|
f32027fe59 |
fix(mocks): read the control rows as they are stored, and say what is gated
Nine findings from review, all taken. Row reads (getdata): - approversForTable reads every column through field(): the security and group rows are seeded with UPPERCASE keys and an edited row comes back lower-cased, so a direct property read compared undefined with 'UNDEFINED' and found no grant - silently falling back to the admin group for every table. - tx_to is handled by rowInForce(): epoch seconds, which is what Date.now()/1000 assumes, with an ISO string accepted; a blank, absent or unparseable end date is open-ended rather than expired, so a rule or a grant that omits it is no longer dropped as NaN. The dqrules filter reads rule_active/base_lib/base_ds through field() for the same reason. Rows and the text that describes them (getdata): - the current-rows clause in FILTER_TEXT is now applied to the selection: the grid no longer shows expired rows while telling the user they were excluded. - the primary-key sort resolves each key to the column's own casing, so a lowercase DDL sorts on real values instead of comparing undefined. Settings (getdata, viewlibs): - dcLibref falls back to 'DC_JSLIB' like dcAdminGroup already did: a deploy that has not run makedata has no settings.js, and viewlibs must still list the control library. Wording that had gone stale: - the parity README's Status paragraph now states the invariant (every scenario is gated; gated:false is advisory) instead of counts and a list of differences that were fixed, and the scenarios.json notes say what each scenario pins now. - viewdata's comment states the contract rather than the behaviour it replaced, and the synthetic no-data row writes the same space for a blank character cell as the data rows do. Tests: getdata-rules.test.js covers an uppercase-keyed APPROVE grant and an open-ended rule (both fail on the previous code); viewdata.test.js covers the numeric missing and the empty-table row. |
||
|
dc
|
b301399b5c |
fix(mocks): viewdata sends a blank cell as a space, not the word undefined
String(undefined) put the literal text "undefined" into every cell of a column the row does not carry - the ordinary case in MPE_TABLES, where nine columns are blank on every row - so the viewer rendered the word instead of an empty cell. A value the row does not carry is SAS's missing, so a character column gets a space and a numeric one '.', which is what the empty-table branch beside it already did. sas/mocks/test/viewdata.test.js fails without the change and passes with it. |
||
|
dc
|
3bb9fb52e3 |
fix(mocks): make getdata and viewlibs answer what the estate answers
The two scenarios recorded in the previous commit are now gated, with one declared difference each instead of 388 and 4. The diffs were the mock's, and they are fixed rather than accepted: - cols: dropped the DESC key SAS never emits, and took LABEL/LONGDESC from the MPE_DATADICTIONARY rows for this lib.ds.column, as getdata.sas augments mp_getcols. COLTYPE is blank for DATE/DATETIME/TIME, which is what the estate answers. - sasdata: sorted by the business key; character values trimmed the way the JSON writer writes them, with an all-blank value as a single space. - dqrules: the active MPE_VALIDATIONS rows unioned with the schema's NOTNULL (limited to the columns left in the selection, which is why a NOT NULL on tx_from never reached the grid) and the derived columns' READONLY, then de-duplicated and sorted by column and rule type with X left at the missing value - getdata.sas's union/sort/by-group logic. - sasparams: the current-rows clause mpe_filtermaster appends, and blank character values as the single space SAS sends. - maxvarlengths: the value as stored (trimmed), and the key columns in upper case, which is how the selection spells them. - approvers: computed from MPE_SECURITY's APPROVE rows with the admin group as the fallback, instead of a hardcoded sasdemo. - viewlibs: the shape %mp_getlibs answers - engine in LIBRARYID, blank ENGINE, libref as LIBRARYNAME. The control library and the admin group now come from services/settings.js through dcInit(), which is the mock's %mpe_init(): makedata already writes dc_libref, dc_admin_group and dcpath there, so a deployment with a different DCPATH no longer needs a service change. test/mock-settings.test.js proves it by renaming the library and moving the data. The remaining services still carry const dcLibref = 'DC_JSLIB' and compute paths from it at file scope, so each is a separate change. The sas9 mock keeps its own getdata/viewlibs variant, so it is untouched: the goldens are recorded from the SASjs Server estate. |
||
|
dc
|
49a9bbf022 |
feat(mocks): record getdata and viewlibs, and show what the mock gets wrong
Two more scenarios, and the recorder could not take either of them as it stood. - editors/getdata reads a table input, not URL parameters, so record.js can now post one: it writes the informational header the server expects (the first line is the SAS input statement, not a list of column names), posts it as multipart/form-data, and takes --libref so the estate's control library name stays out of scenarios.json. replay.js hands the mock its own rows through mockInput, since the two sides do not share a libref. - the test harness's webout() stub now honours the showmeta option and writes the body as members are added. Without them the mocks' own requireMemberMetadata() guard throws - it reads the body before CLOSE - and a service that stopped sending formats and labels would have passed here. - --scenario accepts service/id, because two screens legitimately call their first service 'default'. The two goldens are recorded but marked gated: false: the gate prints their difference counts and does not fail, and check.js labels them ADVISORY. They are a work list, not an acceptance. editors/getdata/mpe_tables - 388 differences, all of them the mock's: cols carries a DESC key SAS never emits and misses the LABEL/LONGDESC/COLTYPE SAS reads from MPE_DATADICTIONARY; sasdata leaves blank character values out instead of padding them and is not ordered by the business key; dqrules differ in content and order; approvers is hardcoded to one user where SAS computes it and returns none. public/viewlibs - 4 differences: SAS answers LIBRARYID='V9' with a blank ENGINE and LIBRARYNAME equal to the libref, and lists SASHELP; the mock answers LIBRARYID=<libref>, ENGINE='V9', LIBRARYNAME='Data Controller(<libref>)' and has no SASHELP. dsmeta is excluded on purpose: it is the engine's report on the dataset (owner, host, encoding, paths, created dates), estate state rather than behaviour, and the projection refused the recorded response until it was dropped - the site id and the service account name were in it. |
||
|
dc
|
909580dbda |
docs(mocks): cite the fixture that actually hits the SOME_SHORTNUM rule
Two documentation findings from review:
- The comment in createdemodata.js and the parity README said the excel
fixtures hold "3, 18, 34, 79", which points at values the dropped rule never
saw. The uploads that the rule applies to go to dc_jslib, and
regular_excel.xlsx has five rows where SOME_SHORTNUM is 5 - that is the value
the 1-5 HARDREGEX rejects. The 3s belong to a multi-load fixture whose sheet
targets DC996664.MPE_X_TEST, a libref that carries no seeded validations
because getdata filters MPE_VALIDATIONS on base_lib === requestedLibref and
createdemodata copies only data files into it. Both places now name
regular_excel.xlsx and the value 5.
- README:9 promised the scan refuses an "eight-digit id" while the added class
is \bDC\d{8}\b. It now says "eight-digit DC id", which is what the scan does
and what the date case in parity-golden.test.js relies on.
Checked the fixture rather than the diff: column H (SOME_SHORTNUM) of
regular_excel.xlsx holds 18, 34, 79, 74, 98 ... and exactly five rows of 5.
|
||
|
dc
|
038c40499b |
refactor(mocks): name the synthetic estate by target name, not a sasjsconfig key
The previous commit added "synthetic": true to the server target to stop the recorder claiming an estate it had not checked. That put a harness-only key into the shared build config, where nothing in sasjs reads it and a maintainer could reasonably read it as a sasjs feature. The recorder now carries the list of target names that point at the estate makedata builds (SYNTHETIC_TARGETS = ['server'], which is sas.4gl.io locally) and refuses anything else, so sasjsconfig.json is untouched. The declaration is the same one - it cannot be derived from a response, because makedata is the installer and a customer's control library holds the same rows - it just lives with the tool that acts on it. --target server passes the guard (and needs a token, as before); --target server-mihajlo is refused with a message naming the allowed target. |
||
|
dc
|
7815dcccb8 |
fix(mocks): drop only the MPE_X_TEST rule the excel fixtures break
The previous commit removed every validation rule SAS seeds for MPE_X_TEST, which fixed the excel specs and broke editor 3: the cell dropdown they click for comes from the HARDSELECT_HOOK on SOME_NUM, and that hook had gone with the rest. Only one of SAS's rules actually conflicts with the fixtures - the HARDREGEX on SOME_SHORTNUM, which rejects 1-5 while duplicate_row_excel.xlsx and friends hold 3, 18, 34, 79. Everything else stays, so the mock's rule set for the table is again exactly what it was before the seed work: SOFTSELECT on SOME_BESTNUM, HARDREGEX and SOFTREGEX on SOME_CHAR, SOFTREGEX on PRIMARY_KEY_FIELD and the SOME_NUM hook. The parity README now says this, rather than claiming the table is rule-free, and createdemodata.test.js asserts both halves - the hook is present and the SOME_SHORTNUM rule is not. |
||
|
dc
|
dd2a777438 |
fix(mocks): restore the MPE_X_TEST fixtures and pin the replay's timezone
The parity gate's Cypress job went red after the seed work landed, and the two
causes are the same change seen from the fixture side:
- makedata now carries SAS's demo validation rules for MPE_X_TEST, including the
HARDREGEX on SOME_CHAR that blocks any text but 'the'/'data'. The excel
fixtures upload arbitrary values into that table and expect the submission to
reach the approval screen, so ten excel specs timed out with no Approve
button. createdemodata drops MPE_X_TEST's rules, which is what editor.cy.ts
already documents ("MPE_X_TEST stays clean to match the excel fixtures").
- the LABELs the viewer-labels spec asserts went with the hand-written schema.
createdemodata restores them, because a SAS estate has none and the control
library should not claim otherwise.
Both are declared as fixture divergences in the parity README and asserted in
createdemodata.test.js, so the next change to either side fails a test instead
of a Cypress run.
Review findings:
- the SYSTIMEZONEOFFSET acceptance depended on the host's zone; both entry
points pin TZ=UTC before the harness loads, so the accepted list is the same
everywhere.
- the sas9 mock still published '+0100'; it now answers seconds east of UTC
like the standard estate.
- the README promised an eight-digit id scan that IDENTIFIER_CLASSES no longer
had. A DC-prefixed eight-digit class is back (a plain digit class would fire
on a yyyymmdd date), with a case proving a date is not refused.
|
||
|
dc
|
94cd4912f9 |
fix(mocks): make the golden's claims evidence, not labels
Six findings from review: - estate: 'synthetic' was a constant. The target now declares "synthetic": true in sasjsconfig.json and record.js refuses any other target, so the claim in the file is one the tool had to check before it wrote anything. - project() dropped the key when scrubbing a value, so a top-level member named like a secret was published unredacted. The key is passed through, and checkPublishable now runs secretViolations on the projection, so the recorder refuses before writing rather than leaving it to the next pull request. - denyList restated a subset of ENVELOPE_NEVER. It is built from that list now, so an error text or a hostname in a discarded field is caught too (values shorter than four characters are skipped - they would match half the file). - replay() projected with golden._meta.members while the test validated scenario.members, so editing the scenario had no effect until a re-record. It projects with the scenario's list and reports a member-list mismatch. - replay() deleted SYSCC/SYSENCODING from the expected side when the harness omitted them, which would hide a service that stopped emitting them. The harness now adds the two members the way the server does, and they are compared. - the eight-digit siteId scan class matched any eight-digit payload value (a date, an amount) and refused clean goldens. Removed; the site id is caught by the deny list, which now covers it. |
||
|
dc
|
a57557c60e |
fix(mocks): wire the replay into CI and close the review gaps
Four findings:
- the replay was never run by CI, so the README's "a mock change fails the
pull request" was not true. parity-check.test.js now replays every scenario
through a shared parity/replay.js (which check.js also uses), fails on a
difference that is not declared and on a declared one that no longer differs,
and fails a scenario that has no golden. The differences that remain are
declared per scenario with a reason: the estate's admin group, its
registration state, its timezone and SAS's space padding of blank characters.
- recording stamped tls: verified for a non-https target while sending the
bearer token in the clear. A non-https target is refused outright now, and
--insecure still records tls: unverified.
- the committed-golden gate never applied the secret-key rule. secretViolations
now walks a golden (and a fresh projection) for secret-named fields whose
value is not <REDACTED>, with a case proving a hand-typed licence key fails.
- arg() returned argv[index + 1] unconditionally, so a trailing --ca reached
readFileSync(undefined). Flags without a value are a usage error now.
The replay also found a real mock defect, fixed here: SYSTIMEZONEOFFSET was
emitted as +HHMM while SAS answers seconds east of UTC ('3600'), and the mock's
own test asserted the wrong form - exactly the drift this harness exists for.
The sasdatasets sort used localeCompare, which puts MPE_X_CATALOG-FC before
MPE_XLMAP_RULES; SAS sorts on the byte value, so the mock now does too.
|
||
|
dc
|
6c831bb230 |
fix(mocks): verify TLS when recording, and project staging ids whole
Two findings from review: - record.js turned certificate verification off unconditionally, on the request whose response becomes the golden. Verification is now the default; --ca trusts a private estate's own certificate and --insecure is opt-in, warns on stderr and is recorded as _meta.tls so a golden taken over unverified transport cannot be mistaken for one that was not. The estate's certificate verifies, so the committed golden is tls: verified. - the libref placeholder ran before the staging-id one, so a staging id came out as <LIBREF>-1_2_3. The staging rule now runs first and matches both minted shapes (DC<date>-<time>_<n>_<n>, and the mock's millisecond form); a project() case pins the ordering. |
||
|
dc
|
b98fde1c74 |
docs(mocks): record that the parity refresh is an ad-hoc agent job
The pipeline cannot reach a SAS estate, so the goldens are refreshed where the credentials are - an agent or a maintainer running record.js - and the CI half stays offline: scan the committed goldens, replay them against a mock estate. Say that in the harness README so the next reader does not go looking for a scheduled refresh job that cannot exist. |
||
|
dc
|
6262843f3f |
feat(mocks): check the mock against recorded SAS responses
The mock services are a second implementation of the SAS backend, and the only thing comparing them was a human reading two files. This adds the harness that compares their output instead. A golden is a projection of a SAS response, never a copy: project.js keeps a fixed member list per scenario, drops the whole envelope (hostname, site id, user, home paths, job ids, _PROGRAM), substitutes the identifiers the diff needs (<LIBREF>, <PATH>, <EMAIL>, <STAGING_ID>, <DATETIME>), and redacts secrets by key name. The recorder then refuses to publish if any identifier class survives, or if any literal the envelope just discarded still appears in the projection - which is what catches a hostname echoed inside an error message. record.js records from a live estate into .captures/ (gitignored) and writes only the projection to golden/. check.js replays a golden against a mock estate built by the test harness and prints field-level differences; it needs no server and no credentials. parity-golden.test.js re-runs the identifier scan over every committed golden, so a hand-edited file cannot slip through either. Phase 0 ships one scenario, public/startupservice. The replay is advisory and already reports twelve differences worth a look (admin group casing, the registration counters, timezone representation, the sasdatasets order, blank versus empty Excel map descriptions); the corpus, the merge gate and the nightly drift job follow. Stacked on the mocks context in the parent branch: it seeds the mock estate with seededEstate(), which that branch adds. |
||
|
dc
|
6e0866d876 |
fix(mocks): give MPE_X_SEARCH its column metadata back
The fixture's schema entry lived in makedata's schema object (as a second, later key in the same literal) and did not survive the move to createdemodata. saveTable falls back to the file already on the drive, so a reseed looked fine - but CI deletes the appLoc before seeding, and a fresh estate wrote mpe_x_search.json with columns: [], losing the column order the full-table-search spec derives its counts from. Carry the nine columns (name/type/length/format/label/notnull) in createdemodata's schema, and add a case that every fixture table is written with column metadata covering its rows, plus the MPE_X_SEARCH order. It fails against the previous service. |
||
|
dc
|
112e8e4b8b |
fix(mocks): keep the derived rows makedata owns
createdemodata assigned tables.MPE_DERIVED outright, so any row makedata wrote there would be wiped by a fixture run - and the idempotency test did not count the table, so nothing would have said so. Filter the TESTDATA rows out and append, in the same shape as the tables above, and count MPE_DERIVED in the idempotency test. New case: a non-TESTDATA derived row survives a createdemodata run exactly once. It fails against the previous service and passes now. |
||
|
dc
|
5f89314aaf |
fix(mocks): answer the review on #487
- createdemodata drops the unused servicesDir binding. - the DC996664 copy no longer swallows a failure: a broken multi-load fixture now fails the seed where it happened instead of surfacing in a Cypress spec. - makedata answers OUTFMT=JSON with a makedata_response member (as the SAS service does) and the CLI payload asks for it, so the `|| true` comes off makedata in build.yaml, release.yaml and sites.yaml - a failed seed names the service that failed instead of reappearing as "createdemodata needs the control library". - the makedata comment points at sas/mocks/test/makedata-parity.test.js, not a macro that does not exist. |