public/getddl answered one fixed Postgres-flavoured body for every request -
DC_JSLIB.MPE_X_TEST whatever dataset was asked for - which is the last read path
the client calls where the mock invented its answer.
It now builds the SAS-flavour DDL from the table file, the way mp_ds2ddl builds
it from dictionary.columns and dictionary.indexes: the header, the create table
with each column's type, format, NOT NULL and label, and the unique index. The
list-output spacing was derived from the estate's own answers and reproduces
them byte for byte for all forty control tables.
Recorded as public/getddl/mpe_tables, so a mock change that alters the text
fails the gate. The header credits a user and a timestamp, so the projection
masks both - a user id is not something to publish, and the comparison is about
the DDL.
TSQL and PGSQL are not generated yet: a caller asking for one gets the SAS text,
and the README says so rather than inventing a second dialect without a
recording to check it against.
Only E8601DT, E8601DA and E8601TM are printed as ISO. The zoned E8601DZ and
E8601TZ, and the interval forms E8601DTM/DN/MN, fall back to the SAS shape -
they would need their offset or interval suffix printed too, and no table on the
estate carries one. The comment now says that instead of implying the whole
family is covered.
Two findings on #506.
createdemodata eval'd the compiled settings program unconditionally, so a drive
that has the fixtures but not that file went down at load with an ENOENT before
it could report anything. The read is guarded: an absent program evals a stub
that answers the same default dcReadSettings gives for a missing file, and the
service settles on the control library makedata creates instead of failing. A
settings program that exists and does not parse is still a broken deployment,
and is left to fail loudly.
viewIsoDateTime and viewIsoTime clamped a rounded fraction at scale-1 instead of
carrying it, so a value 1e-7 under the next second printed .999999 of the second
before it where SAS prints the next second, .000000. Both now round to the
printed precision first and split the result, which carries into the second.
Both are pinned: createdemodata seeds with settings.js deleted, and an
E8601DT26.6 value one rounding error short of the next minute prints 03:05:00.
The viewer printed one shape for every datetime - a two-digit year and a
two-decimal fraction - whatever format the column carried, while the same
payload published that format in `$viewdata.vars`. A column formatted
E8601DT26.6 therefore answered 02JAN26:03:04:05.00 where the estate prints
2026-01-02T03:04:05.000000.
The E8601 family is now printed as ISO 8601, at the precision its format names
(a width with no fraction prints none), and every other format keeps the shape
the recorded response pins, which is what the DATETIME19.3 columns already
matched.
Three findings from review.
- the trailing-blank strip now matches only blanks. SAS pads with blanks, so a
value that ends in a tab is not padding and is not silently altered while the
same character mid-string survives.
- the `typeof val === 'number'` arm was unreachable: both branches above return
text (viewCell and formatSasValue), so the ternary is gone and a comment says
why a string is all that arrives.
- provenance: the docstrings cited "the recorded response" as the authority, and
the golden's `_meta.estate` says synthetic, which invites exactly the doubt you
raised. They now name the file and the field that answers it - `_meta.sas` is
9.4M9, read from the live envelope's SYSVLONG, which only a SAS runtime emits
and which the mock cannot produce - and point at the Provenance section added
to parity/README.md in #493.
Three findings from review.
- viewdata trimmed both ends of a character cell. A leading blank is part of a
SAS character value and the viewer renders it; only the trailing blanks SAS
pads to are noise, so the cell is trimmed at the end and an all-blank cell is
still SAS missing. viewdata-cells.test.js fails on the previous version.
- viewDate floored a stored day count while the ISO branch rounded, so the same
instant could land on different days depending on how the row carried it. Both
round now.
- the two label members stay different. The finding was that cols[].LABEL uses
the data dictionary business label while $viewdata.vars[.].label uses the
physical one; the recorded estate response does exactly that - cols DSN is
"Dataset Name to be edited", $viewdata.vars.DSN.label is "DSN" - so unifying
them would break parity. The comment says so, and a test asserts the split
against the recorded answer so the next reader does not tidy it away.
Five findings from review.
- getrawdata's settings reader was a second parser that only matched a
double-quoted key and value, so a hand-edited settings.js would leave this one
service on DC_JSLIB while the estate used the configured library. The object is
parsed as JSON first (what makedata writes) and a quote-agnostic pattern is the
fallback, so both a written and an edited file are read the same way.
- the bare catch treated every read failure as "makedata has not run": an
EACCES or an encoding failure now surfaces instead of silently defaulting.
- `dcInit(...).dcLibref || 'DC_JSLIB'` was unreachable - loadDcSettings already
defaults dcLibref to DC_JSLIB - so the `||` is gone from all fourteen services;
it would have masked a changed settings shape as a silent default.
- the "Load shared DC mock utilities" comment had ended up below the eval it
describes in getdynamiccolvals, getcols and getcolvals; it describes the eval,
so it sits above it, and the note about which tables live in the control
library no longer names DC_JSLIB.
- merge order with #493 (getcols, getrawdata, stagedata): whichever lands second
is rebased; the getrawdata hunks are separate from that PR's CSV changes.
The settings guard now also accepts a `let` declaration and a service that keeps
the literal as a default while reading settings.js (getrawdata does).
Three findings from review, and tests that hold them.
- a DATE below the year 1000 printed three digits ('999'), so DATE9.'s
four-digit year is now zero-padded
- a stored ISO text reached a TIME column through the datetime branch, giving
seconds since 1960 rather than seconds since midnight; a time's day is
discarded, and a text without a zone is read as UTC rather than in the host's
zone (the control box sits east of UTC, which shifted every value by an hour)
- the datetime fraction is clamped at 99 as the time branch already was, so a
fraction that rounds up cannot print three digits
viewdata-temporal.test.js builds a table with DATE, TIME and DATETIME columns and
covers each renderer plus a stored ISO string; the year and the ISO cases fail on
the previous version of the service.
Two findings from review.
- viewTime was reached with the unrounded value (viewCell passed the number
straight through for TIME), so a stored 3661.9999 produced a fraction of 100
and printed "01:01:01.100". The value is rounded to two decimals on entry, as
the DATETIME branch already did, and the fraction is clamped as well so a
rounding edge cannot reach three digits.
- a temporal column with no format fell back on DATETIME19.3 whatever its
ddtype, so DATE and TIME columns were labelled as datetimes. The default
follows the ddtype now: DATE9., TIME8., DATETIME19.3.
viewdata's scenario is still at zero accepted differences, and the suite is 96
pass, 0 fail.
makedata writes services/settings.js - the JS twin of the settings program the
backend generates in %mpe_init - with dc_libref, dc_admin_group, dc_staging_area
and dcpath. dcInit() reads it. getdata and viewlibs used it; the other thirteen
services that load dcMockUtils still carried const dcLibref = 'DC_JSLIB' and
built their paths from it, so a deployment with a different DCPATH kept working
in two services and broke in the rest.
Each of those services now loads dcMockUtils first and takes its libref from
dcInit(driveRoot, appLoc).dcLibref, with 'DC_JSLIB' only as the default for a
drive makedata has not seeded. Services that computed paths from the constant
(stagedata's staging directory, getapprovals' data directory, gethistory's) have
the eval above that line, so the derived paths come from the settings too.
Two services do not load dcMockUtils at all - createdemodata and getrawdata
carry their own table and format helpers - so they keep naming the constant, and
say why in a comment: adding the eval just for the settings would collide with
their own helper names.
mock-settings.test.js gains a case that would fail without this: it renames the
control library, moves the data, rewrites the settings file, and asserts a row
written into MPE_SUBMIT comes back from approvers/getapprovals. With the
hardcoded service it is empty, which is the failure mode this removes.
The viewer was the third read path the harness could not see. Recorded from the
seeded estate it showed 364 differences; all of them were the mock's, and none
remain.
- cols: sorted by column name, and a column LABEL comes from the
MPE_DATADICTIONARY row for the lib.ds.column when there is one - viewdata.sas
augments mp_getcols the same way getdata does. The format name is upper case,
as SAS reports it.
- rows: a temporal value is rendered the way the estate renders it - two-digit
year and a two-decimal fraction ("01JAN60:00:00:00.00") - rather than by the
format label on the column, which reads DATETIME19.3. A character value is
trimmed and an all-blank one is a space, matching the missing-value rule the
data rows and the synthetic no-data row already share.
- $viewdata: the column's own metadata (num, 8 bytes, DATETIME19.3). The mock
reused the entry built for the edit grid, where the temporal columns have been
converted to character, so the viewer was told $200 char.
- sasparams.PK_FIELDS: the validity column ahead of the business key, which is
what viewdata answers ("TX_TO LIBREF DSN").
sas/mocks/test/viewdata.test.js pins the datetime text and the empty-table row.
Nine findings from review, all taken.
Row reads (getdata):
- approversForTable reads every column through field(): the security and group
rows are seeded with UPPERCASE keys and an edited row comes back lower-cased,
so a direct property read compared undefined with 'UNDEFINED' and found no
grant - silently falling back to the admin group for every table.
- tx_to is handled by rowInForce(): epoch seconds, which is what Date.now()/1000
assumes, with an ISO string accepted; a blank, absent or unparseable end date
is open-ended rather than expired, so a rule or a grant that omits it is no
longer dropped as NaN. The dqrules filter reads rule_active/base_lib/base_ds
through field() for the same reason.
Rows and the text that describes them (getdata):
- the current-rows clause in FILTER_TEXT is now applied to the selection: the
grid no longer shows expired rows while telling the user they were excluded.
- the primary-key sort resolves each key to the column's own casing, so a
lowercase DDL sorts on real values instead of comparing undefined.
Settings (getdata, viewlibs):
- dcLibref falls back to 'DC_JSLIB' like dcAdminGroup already did: a deploy that
has not run makedata has no settings.js, and viewlibs must still list the
control library.
Wording that had gone stale:
- the parity README's Status paragraph now states the invariant (every scenario
is gated; gated:false is advisory) instead of counts and a list of differences
that were fixed, and the scenarios.json notes say what each scenario pins now.
- viewdata's comment states the contract rather than the behaviour it replaced,
and the synthetic no-data row writes the same space for a blank character cell
as the data rows do.
Tests: getdata-rules.test.js covers an uppercase-keyed APPROVE grant and an
open-ended rule (both fail on the previous code); viewdata.test.js covers the
numeric missing and the empty-table row.
String(undefined) put the literal text "undefined" into every cell of a column
the row does not carry - the ordinary case in MPE_TABLES, where nine columns are
blank on every row - so the viewer rendered the word instead of an empty cell.
A value the row does not carry is SAS's missing, so a character column gets a
space and a numeric one '.', which is what the empty-table branch beside it
already did.
sas/mocks/test/viewdata.test.js fails without the change and passes with it.
The guard that keeps a runtime without webout()'s options argument from serving a
thinner response tested webout.length >= 4. Function.length stops counting at the
first default or rest parameter, so a runtime that does carry the option but
declares it `options = {}` or gathers the tail with `...args` reports three, and
every service that calls the guard refuses to start on a working runtime.
It now asserts the outcome instead: the member the option produces is looked for
in the response body the program is writing. `$sasdata`, `$viewdata`, `$vals` are
the members the client reads for column formats and labels, and their presence is
the thing that actually has to hold.
Where it runs matters too, and the first attempt got that wrong: called after
webout('CLOSE') the throw leaves a complete body behind, and the server serves it
- the caller sees a normal response with SYSCC 0 and no metadata, which is exactly
the silent failure this exists to prevent. It is called before the envelope is
closed, so an unhonoured option fails the run.
The three services that pass { showmeta } - getdata, viewdata, getcolvals - rely
on webout()'s fourth argument, which arrived in @sasjs/utils 4.6.0. @sasjs/server
3.12 declares ^4.4.0, so an install can resolve 4.5.x, and there the argument is
dropped silently: the response still looks complete but $sasdata, $viewdata and
$vals are absent, so the client's format and label paths stop being exercised
while the mock estate reports success.
requireWeboutOptions() checks that the generated webout() takes an options
argument (arity, because the option names live outside the callable function) and
throws naming the member, the cause and both fixes. A 4.6 runtime is unchanged.
The last eleven services still built the response envelope through
dcMockUtils' webOutOpen/webOutObj/webOutClose. They now use the webout() the
SASjs Server emits into every JS program, so dcMockUtils no longer emulates
%webout at all - the helper existed only for these callers and is deleted
with them.
That needed the options object added to webout() in @sasjs/utils 4.6
(showmeta, maxobs), which is what a clean install of @sasjs/server@3.12.2
resolves to (^4.4.0 -> 4.6.1, checked). A server whose install still holds
4.4/4.5 refuses the option by name rather than ignoring it, so the failure is
loud rather than a response that is quietly missing its column metadata.
getdata, viewdata and getcolvals send their metadata as
`webout('OBJ', name, rows, { showmeta })`, and every other call keeps its
member name and rows unchanged.
One call was positional all along: startupservice's xlmaps is an array of
arrays, which the old helper serialised as-is and webout() refuses for OBJ.
It is now webout('ARR', 'xlmaps', xlmaps) - caught by the comparison run
below, not by the rewrite.
physicalColumns held the same array the derive helper reads, so every later
use in the file - cols, the search, the row loop, the empty-row fallback and
vars - depended on that helper returning a new list rather than extending the
one it was given. Taking a copy makes the invariant local to this file.
viewdata.sas drops the derived columns in the vwsearch data step that
mp_searchdata then reads, so a search matches on the columns the grid shows.
The mock handed the full column list, so a search could match a row on a
derived column that is never displayed.
viewdata.sas builds the derived view and filters against it, so a filter stored
on a derived column resolves in the VIEW screen; the mock served the physical
table only, so the same filter matched nothing and the viewer showed its
no-data row. The mock now derives ahead of the filter and reports the physical
columns only, as viewdata.sas drops them from the output.
The mock typed columns from its own format list while the services typed them
from the macro's, so the two could disagree - the review of the getcols.sas
change found exactly that, with the mock covering YYMMDD/MMDDYY/DDMMYY/E8601DT
and the service only the DATETIME/DATE/TIME prefixes.
mpGetcols() now mirrors the macro: NAME (upcased), TYPE (C/N), LENGTH, LABEL,
VARNUM, FORMAT, FMTNAME and DDTYPE (CHARACTER/NUMERIC/DATETIME/DATE/TIME), with
the macro's date list. editors/getdata.js, public/viewdata.js and
public/getcols.js use it, the last mapping CHARACTER/NUMERIC to C/N the way the
service does. getDdType() delegates to it, so there is one inference, not two.
Verified by diffing the column list every fixture table returns from all three
services before and after: 49 tables x 3 services, 9 differences, all of them
FMTNAME on E8601DT columns - the old code rewrote it to DATETIME, the new code
leaves E8601DT, which is what %mp_getcols puts in fmtname. FMTNAME is not read
at runtime. Every DDTYPE, TYPE, LENGTH, FORMAT and LABEL is identical.
The mocked estate could not exercise what 7.17 changed.
- getdata.js and viewdata.js hard-coded their limits, so editing
DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT / DC_MAXOBS_WEBVIEW in DC made
no difference. They now read them from MPE_CONFIG via a new
loadDcConfig/dcConfigNumber pair in dcMockUtils, which mirrors the loop
in the generated settings.sas: DC-scope rows only, var_active=1 only,
current rows only (now < tx_to), latest tx_from wins. An option that is
absent or switched off falls back to the service's own default, which is
what %symexist does in the real service.
- There was no mock for services/admin/exportdb, so the System screen's
Export DC Library DDL action hit a service that did not exist. The new
mock renders every table in the DC libref as basic DDL - SAS, PGSQL and
TSQL flavours, schema defaulting to the libref, a unique index from the
MPE_TABLES buskey, and the download headers mp_streamfile produces. DDL
only: the real service appends inserts for SAS and PGSQL, but
serialising the library's rows in a mock buys no test value.
ddl-export.cy.ts gains two cases that fetch the URL the action opens and
assert the DDL, so the spec covers the service and not just the URL.
VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.
The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.
The viewer's search box searches every column of a table - character columns
by case-sensitive CONTAINS and numeric columns by exact equality - but against
the JS mock backend the box did nothing, because the viewdata mock ignored
SEARCHTYPE/SEARCHVAL (and FILTER_RK) entirely.
Mock changes (sas/mocks/sasjs/services/public/viewdata.js):
- apply the stored filter first via mpeFilterMaster, mirroring
%mpe_filtermaster(VIEW,...) in viewdata.sas, and return its WHERE text as
sasparams.FILTER_TEXT (previously hard-coded blank, so the viewer's info bar
never showed a clause and the filter was never applied to the rows)
- mirror %mp_searchdata: CHAR = case-sensitive CONTAINS across every character
column, NUM = exact equality across every numeric column, only CHAR/NUM
trigger a search, search values stripped of % & ; " like the SAS service
- cap the rows the way the real service does (MAXROWS 500; NOBS is the uncapped
filtered count for a normal view, the capped match count for a search)
- a search with no matches now returns no rows, so the client shows its
"No data found with given conditions" panel; the single-empty-row fallback
stays for the normal-view-empty-table case
Mock data (sas/mocks/sasjs/services/admin/makedata.js):
- new MPE_X_SEARCH demo table: 1000 deterministic, obviously fictional
deep-sea survey rows, registered in MPE_TABLES. Seeded so a partial search
hits a value buried in the middle of a long text column, a value spanning two
columns, a repeated exact numeric, and rows that deliberately do not match.
New spec (client/cypress/e2e/full-table-search.cy.ts):
- seven tests covering open/full view, partial character search, case
sensitivity, numeric exact match, no match, search within a filter, and
clearing the search. No artificial waits - state assertions only.
- added to the Cypress spec list in .gitea/workflows/build.yaml
Also adds the dc-cypress skill (.agents/skills/dc-cypress) covering the mock
backend setup, the spec conventions, the selectors that actually work in the
viewer filter modal, and the fixed 1280x720 video capture.
viewdata.js hardcoded PK_FIELDS as empty, so the viewbox grid treated
all columns as non-PK. removeAllColumns() could delete every column
including the ones the tests expect to be protected.
Now reads MPE_TABLES buskey (mirrors mp_getpk) and falls back to
LOAD_REF LIBREF DSN KEY_HASH TGTVAR_NM for MPE_AUDIT (which has a
known index-based PK in real SAS but isn't registered in MPE_TABLES).
All 12 viewbox.cy.ts tests now pass.
startupservice.js only returned LIBREF in saslibs, but the dc-tree
component renders LIBRARYNAME — without it the nav tree node text was
empty, so libraryToOpenIncludes could not match. Added LIBRARYNAME,
LIBRARYID, and ENGINE to match the viewlibs service format.
Changed libraryToOpenIncludes_SASJS from 'dc' to 'dc_jslib' to avoid
matching multiple libraries (DC_JSLIB and DC996664 both contain 'dc').
viewdata.js: removed alphabetical .sort() on cols (SAS preserves
physical column order via mp_getcols); added empty row with correct
SAS missing values ('.' for numeric, '' for character) when a table
has no data, so the viewbox grid can derive column headers.
Adds a URL-driven toggle (?labels=true, mirroring embed=va) that
swaps grid column headers between NAME and LABEL in both the Viewer
and Editor, with a "Show labels"/"Show names" item in each grid's
right-click context menu. The header-info popup now shows NAME first
regardless of display mode. Sorting/filtering/data binding stay
NAME-based throughout — only the displayed header text changes.
New shared, unit-tested utils (parse-labels-param, display-col-headers,
col-info-html) back both pages. Mock data (viewdata.js, getdata.js)
gains a couple of columns with LABEL != NAME plus a blank-LABEL case,
covered by a new viewer-labels.cy.ts e2e spec now wired into CI.
Also fixes a pre-existing gap in cypress/support/commands.ts: the
isLoggedIn/loginAndUpdateValidKey custom commands had no Cypress
Chainable type augmentation, and moment was imported as a namespace
import despite its `export =` typing making it uncallable that way.
dc
allan
sead