Three findings from review.
- viewdata trimmed both ends of a character cell. A leading blank is part of a
SAS character value and the viewer renders it; only the trailing blanks SAS
pads to are noise, so the cell is trimmed at the end and an all-blank cell is
still SAS missing. viewdata-cells.test.js fails on the previous version.
- viewDate floored a stored day count while the ISO branch rounded, so the same
instant could land on different days depending on how the row carried it. Both
round now.
- the two label members stay different. The finding was that cols[].LABEL uses
the data dictionary business label while $viewdata.vars[.].label uses the
physical one; the recorded estate response does exactly that - cols DSN is
"Dataset Name to be edited", $viewdata.vars.DSN.label is "DSN" - so unifying
them would break parity. The comment says so, and a test asserts the split
against the recorded answer so the next reader does not tidy it away.
Five findings from review.
- getrawdata's settings reader was a second parser that only matched a
double-quoted key and value, so a hand-edited settings.js would leave this one
service on DC_JSLIB while the estate used the configured library. The object is
parsed as JSON first (what makedata writes) and a quote-agnostic pattern is the
fallback, so both a written and an edited file are read the same way.
- the bare catch treated every read failure as "makedata has not run": an
EACCES or an encoding failure now surfaces instead of silently defaulting.
- `dcInit(...).dcLibref || 'DC_JSLIB'` was unreachable - loadDcSettings already
defaults dcLibref to DC_JSLIB - so the `||` is gone from all fourteen services;
it would have masked a changed settings shape as a silent default.
- the "Load shared DC mock utilities" comment had ended up below the eval it
describes in getdynamiccolvals, getcols and getcolvals; it describes the eval,
so it sits above it, and the note about which tables live in the control
library no longer names DC_JSLIB.
- merge order with #493 (getcols, getrawdata, stagedata): whichever lands second
is rebased; the getrawdata hunks are separate from that PR's CSV changes.
The settings guard now also accepts a `let` declaration and a service that keeps
the literal as a default while reading settings.js (getrawdata does).
Three findings from review, and tests that hold them.
- a DATE below the year 1000 printed three digits ('999'), so DATE9.'s
four-digit year is now zero-padded
- a stored ISO text reached a TIME column through the datetime branch, giving
seconds since 1960 rather than seconds since midnight; a time's day is
discarded, and a text without a zone is read as UTC rather than in the host's
zone (the control box sits east of UTC, which shifted every value by an hour)
- the datetime fraction is clamped at 99 as the time branch already was, so a
fraction that rounds up cannot print three digits
viewdata-temporal.test.js builds a table with DATE, TIME and DATETIME columns and
covers each renderer plus a stored ISO string; the year and the ISO cases fail on
the previous version of the service.
Three findings from review.
- validatefilter used fetchTable/colVal above the eval that defines them, so it
threw on every call - the conversion moved the eval down past the parse. The
eval and the dcInit read are back above the parse, with a comment saying why
they have to be. Reproduced and checked: the service answers now.
- a source-level test walks sas/mocks/sasjs/services/**/*.js and asserts the
invariant the copy-paste is supposed to keep: a service that loads the
utilities reads the settings after them, calls no utility before them, and
does not name the control library; makedata is exempt (it creates it) and a
hook is exempt (its body is eval'd in the calling service's scope). This is the
guard that would have caught the validatefilter slip, and it costs no estate.
- getrawdata reads dc_libref out of services/settings.js rather than naming
DC_JSLIB, so the documented CSV row-diff path works on a deployment with a
different DCPATH. It cannot eval dcMockUtils - its own formatSasValue and
SAS_EPOCH_MS would collide - so it reads the one value it needs; createdemodata
stays as it is, because it creates the library.
Two findings from review.
- viewTime was reached with the unrounded value (viewCell passed the number
straight through for TIME), so a stored 3661.9999 produced a fraction of 100
and printed "01:01:01.100". The value is rounded to two decimals on entry, as
the DATETIME branch already did, and the fraction is clamped as well so a
rounding edge cannot reach three digits.
- a temporal column with no format fell back on DATETIME19.3 whatever its
ddtype, so DATE and TIME columns were labelled as datetimes. The default
follows the ddtype now: DATE9., TIME8., DATETIME19.3.
viewdata's scenario is still at zero accepted differences, and the suite is 96
pass, 0 fail.
makedata writes services/settings.js - the JS twin of the settings program the
backend generates in %mpe_init - with dc_libref, dc_admin_group, dc_staging_area
and dcpath. dcInit() reads it. getdata and viewlibs used it; the other thirteen
services that load dcMockUtils still carried const dcLibref = 'DC_JSLIB' and
built their paths from it, so a deployment with a different DCPATH kept working
in two services and broke in the rest.
Each of those services now loads dcMockUtils first and takes its libref from
dcInit(driveRoot, appLoc).dcLibref, with 'DC_JSLIB' only as the default for a
drive makedata has not seeded. Services that computed paths from the constant
(stagedata's staging directory, getapprovals' data directory, gethistory's) have
the eval above that line, so the derived paths come from the settings too.
Two services do not load dcMockUtils at all - createdemodata and getrawdata
carry their own table and format helpers - so they keep naming the constant, and
say why in a comment: adding the eval just for the settings would collide with
their own helper names.
mock-settings.test.js gains a case that would fail without this: it renames the
control library, moves the data, rewrites the settings file, and asserts a row
written into MPE_SUBMIT comes back from approvers/getapprovals. With the
hardcoded service it is empty, which is the failure mode this removes.
The viewer was the third read path the harness could not see. Recorded from the
seeded estate it showed 364 differences; all of them were the mock's, and none
remain.
- cols: sorted by column name, and a column LABEL comes from the
MPE_DATADICTIONARY row for the lib.ds.column when there is one - viewdata.sas
augments mp_getcols the same way getdata does. The format name is upper case,
as SAS reports it.
- rows: a temporal value is rendered the way the estate renders it - two-digit
year and a two-decimal fraction ("01JAN60:00:00:00.00") - rather than by the
format label on the column, which reads DATETIME19.3. A character value is
trimmed and an all-blank one is a space, matching the missing-value rule the
data rows and the synthetic no-data row already share.
- $viewdata: the column's own metadata (num, 8 bytes, DATETIME19.3). The mock
reused the entry built for the edit grid, where the temporal columns have been
converted to character, so the viewer was told $200 char.
- sasparams.PK_FIELDS: the validity column ahead of the business key, which is
what viewdata answers ("TX_TO LIBREF DSN").
sas/mocks/test/viewdata.test.js pins the datetime text and the empty-table row.
getdata padded every blank character cell to a space, including one the row
holds as an explicit ''. That broke nine editor specs: the grid classifies a row
by comparing what it holds against the payload it was given (classifyRow), so a
payload whose empty strings had become spaces made every row read as modified on
load - the row header showed +/- and DC.ROW_STATUS answered 'M' where 'U' is
correct.
The rule is now the one SAS expresses: a cell the row does not carry is missing
and reaches the client as one space, and a value the row does carry is passed
through. MPE_TABLES rows omit their blank columns, so the recorded golden still
matches; the TESTDATA formula fixture stores '', and keeps it.
getdata-blanks.test.js pins both sides.
Every existing case set one state and left the rest empty, so the order the
component relies on was untested: a licence problem and an over-capacity state
were never rendered for a user who is also unregistered. Two cases now do that
and assert the problem/limit message with the registration text absent.
Making the problem state lose to the unregistered state turns the first one
red.
The banner is the only route into the licensing pages now, so it needs a state
for every way the licence can block the user, and each state needs to point at
the page that can fix it:
- a licence problem with no free tier (the startupservice-without-globvars
case, which previously fell through to 'not registered') reports the problem
and opens its encoded page;
- free tier keeps its update-key and problem links;
- over capacity, and an unregistered user who cannot register, point at
licensing/limit, which had no entry point once the forced navigation went;
- an unregistered user points at licensing/register when they can register;
- the expiry banner keeps its update link.
Moving the banner out of app.component means its states can be tested: ten
TestBed cases assert the message and the rendered link for each, including that
nothing renders before the startup data lands or when the licence is in order.
currentUserCanRegister finally has a reader - it decides between the register
and limit links - and the Cypress limit case follows the banner instead of
expecting a redirect.
A startupservice failure sets isAppActivated false but never emits
startupDataLoaded, so the banner would have claimed the user is not registered
when the app had not started at all. The banner now waits for the startup data,
which is the same flag that removes the loading page.
The incorrect-site-id subscription deactivated the app and then navigated to
the licensing key page. With the banner as the entry point the app stays where
it is: licenceProblem already carries the URL the banner's More details link
opens, and deactivateApp's redirect argument had no other caller.
Startup no longer moves the user: an unactivated app stays on the page it
loaded instead of being sent to licensing/key, and an unregistered user is no
longer sent to licensing/register. The banner in app.component reports the
licence state and links to the licensing page - Register while the current user
is not registered, Update key for free tier, and More details for a licence
problem - so the licence page is somewhere the user goes rather than somewhere
they are put.
With nothing forcing the user onto licensing/register there is nothing to lock
them onto it either, so LicensingGuard (canDeactivate) and its route hook go.
A licence problem found while the user is already on a licensing page still
opens the page that explains it (navigateWithinLicensing), which keeps the
apply-key error flow - and its specs - unchanged.
The licensing spec follows the new route through the banner for the expired,
invalid and wrong-organisation cases, and updateLicenseKeyQuick waits for the
reboot that applying keys triggers before the caller continues.
Startup navigation is pinned by app.service.spec: an unactivated app on any
page, licensing included, is left where it is.
Nine findings from review, all taken.
Row reads (getdata):
- approversForTable reads every column through field(): the security and group
rows are seeded with UPPERCASE keys and an edited row comes back lower-cased,
so a direct property read compared undefined with 'UNDEFINED' and found no
grant - silently falling back to the admin group for every table.
- tx_to is handled by rowInForce(): epoch seconds, which is what Date.now()/1000
assumes, with an ISO string accepted; a blank, absent or unparseable end date
is open-ended rather than expired, so a rule or a grant that omits it is no
longer dropped as NaN. The dqrules filter reads rule_active/base_lib/base_ds
through field() for the same reason.
Rows and the text that describes them (getdata):
- the current-rows clause in FILTER_TEXT is now applied to the selection: the
grid no longer shows expired rows while telling the user they were excluded.
- the primary-key sort resolves each key to the column's own casing, so a
lowercase DDL sorts on real values instead of comparing undefined.
Settings (getdata, viewlibs):
- dcLibref falls back to 'DC_JSLIB' like dcAdminGroup already did: a deploy that
has not run makedata has no settings.js, and viewlibs must still list the
control library.
Wording that had gone stale:
- the parity README's Status paragraph now states the invariant (every scenario
is gated; gated:false is advisory) instead of counts and a list of differences
that were fixed, and the scenarios.json notes say what each scenario pins now.
- viewdata's comment states the contract rather than the behaviour it replaced,
and the synthetic no-data row writes the same space for a blank character cell
as the data rows do.
Tests: getdata-rules.test.js covers an uppercase-keyed APPROVE grant and an
open-ended rule (both fail on the previous code); viewdata.test.js covers the
numeric missing and the empty-table row.
String(undefined) put the literal text "undefined" into every cell of a column
the row does not carry - the ordinary case in MPE_TABLES, where nine columns are
blank on every row - so the viewer rendered the word instead of an empty cell.
A value the row does not carry is SAS's missing, so a character column gets a
space and a numeric one '.', which is what the empty-table branch beside it
already did.
sas/mocks/test/viewdata.test.js fails without the change and passes with it.
The two scenarios recorded in the previous commit are now gated, with one
declared difference each instead of 388 and 4. The diffs were the mock's, and
they are fixed rather than accepted:
- cols: dropped the DESC key SAS never emits, and took LABEL/LONGDESC from the
MPE_DATADICTIONARY rows for this lib.ds.column, as getdata.sas augments
mp_getcols. COLTYPE is blank for DATE/DATETIME/TIME, which is what the estate
answers.
- sasdata: sorted by the business key; character values trimmed the way the
JSON writer writes them, with an all-blank value as a single space.
- dqrules: the active MPE_VALIDATIONS rows unioned with the schema's NOTNULL
(limited to the columns left in the selection, which is why a NOT NULL on
tx_from never reached the grid) and the derived columns' READONLY, then
de-duplicated and sorted by column and rule type with X left at the missing
value - getdata.sas's union/sort/by-group logic.
- sasparams: the current-rows clause mpe_filtermaster appends, and blank
character values as the single space SAS sends.
- maxvarlengths: the value as stored (trimmed), and the key columns in upper
case, which is how the selection spells them.
- approvers: computed from MPE_SECURITY's APPROVE rows with the admin group as
the fallback, instead of a hardcoded sasdemo.
- viewlibs: the shape %mp_getlibs answers - engine in LIBRARYID, blank ENGINE,
libref as LIBRARYNAME.
The control library and the admin group now come from services/settings.js
through dcInit(), which is the mock's %mpe_init(): makedata already writes
dc_libref, dc_admin_group and dcpath there, so a deployment with a different
DCPATH no longer needs a service change. test/mock-settings.test.js proves it by
renaming the library and moving the data. The remaining services still carry
const dcLibref = 'DC_JSLIB' and compute paths from it at file scope, so each is
a separate change.
The sas9 mock keeps its own getdata/viewlibs variant, so it is untouched: the
goldens are recorded from the SASjs Server estate.
appLocked is constructed as false and nothing sets it: the two intended
writers in licence.service.ts were commented out, and the only reader was
canDeactivate. The branch could not fire in the app, so the guard's tested
contract was not the one that ships. Remove the branch, its spec case and the
now-unused subject.
The route-params handler and the isAppActivated subscription both called the
same redirect check, so a load ran it twice. They now feed one combineLatest
subscription that assigns the action and decides the redirects, which still
covers a licence state arriving after the page loads and a switch between
licensing actions without a reload.
The same subscription sends an unknown :action home. Only the values in
LicenseActions have a page, so a URL like licensing/whatever previously
rendered the component with no action and left an empty page.
Licensing spec: 12/12 locally in 49s, including the new unknown-action case.
gotoLicensingUpdate() deferred visitPage('licensing/update') into a
cy.location().then() callback, so callers that queued commands after the bare
call had those commands enqueued ahead of the visit and ran them against
whichever page the app was on. It now returns the chainable and the callers
chain on it, so nothing is queued before the navigation.
waitForAppSettled() also gains .card-header: the key, update and limit pages
render the licensing card even when there is no error to show, and an install
that is not activated yet has no nav tree, no demo banner and no username
field, so the previous set could time out on a page the spec was already on.
Licensing spec: 11/11 locally in 43s.
Dropping canActivate left licensing/register reachable by URL on an install
that is already registered. Nothing there can succeed: accepting the EULA posts
public/registeruser, which aborts for an already-registered user, so a
bookmark, a reload or a back navigation into the page ended in an error.
The component now sends an activated install home whenever the register action
is reached, watching isAppActivated so the decision also covers a licence state
that lands after the page loads. That is the part of the old canActivate
behaviour the register page still needs, without closing the other licensing
pages.
The licensing spec's terms helper now waits for the EULA card to disappear:
accepting it reloads the app, so the card is the marker that the reload
happened, and a refused registration fails there rather than in the selector
wait that follows.
The guard now holds the registration page only while isAppActivated is
false. A null state means startup data has not landed, so no registration has
begun and there is nothing to protect; a failed startupservice does not leave
it null either - app.service.ts sets false on the malformed, missing-property
and request-failure paths - but a route loaded before that point should not
depend on it.
The spec case for the unknown state flips to expect leaving; reverting the
condition makes it red.
An activated install that reaches licensing/register (stale bookmark,
hand-typed URL) had nothing to register: accepting the EULA posted
public/registeruser, which aborts for an already-registered user, and every
navigation out was then refused, trapping the tab until a hard reload. The
lockout now applies only while isAppActivated is not true, which is the state
it was written for.
The force query parameter was read only by the deleted canActivate, so it is
dropped from the five licensing URLs and from the AppService spec rather than
left as a dead parameter pinned by a test.
Adds unit coverage for the guard's canDeactivate branches.
52 fixed waits (350s declared) replaced by explicit conditions: the app
settling out of .app-loading, and the licensing page's own markers. With the
licensing routes rendered normally these waits observe real state, so the
spec runs in the same wall-clock (1:32 against 1:33) without arbitrary
timeouts to tune - three consecutive local runs at 10/10.
An activated app that loaded a licensing URL (bookmark, browser reload, or
cy.visit in a spec) was redirected to /home as soon as the startupservice
response landed, and LicensingGuard redirected there too while
isAppActivated was still null. Both redirects meant the page only existed
for the window between the router rendering it and startup completing - so
the same link worked or failed depending on whether a key was installed.
Drop both: the guard keeps only canDeactivate, which still holds a user on
the registration page and blocks leaving a locked app. Unactivated apps are
still sent to licensing/key from any other page.
Two more scenarios, and the recorder could not take either of them as it stood.
- editors/getdata reads a table input, not URL parameters, so record.js can now
post one: it writes the informational header the server expects (the first
line is the SAS input statement, not a list of column names), posts it as
multipart/form-data, and takes --libref so the estate's control library name
stays out of scenarios.json. replay.js hands the mock its own rows through
mockInput, since the two sides do not share a libref.
- the test harness's webout() stub now honours the showmeta option and writes
the body as members are added. Without them the mocks' own
requireMemberMetadata() guard throws - it reads the body before CLOSE - and a
service that stopped sending formats and labels would have passed here.
- --scenario accepts service/id, because two screens legitimately call their
first service 'default'.
The two goldens are recorded but marked gated: false: the gate prints their
difference counts and does not fail, and check.js labels them ADVISORY. They are
a work list, not an acceptance.
editors/getdata/mpe_tables - 388 differences, all of them the mock's:
cols carries a DESC key SAS never emits and misses the LABEL/LONGDESC/COLTYPE
SAS reads from MPE_DATADICTIONARY; sasdata leaves blank character values out
instead of padding them and is not ordered by the business key; dqrules differ
in content and order; approvers is hardcoded to one user where SAS computes it
and returns none.
public/viewlibs - 4 differences: SAS answers LIBRARYID='V9' with a blank ENGINE
and LIBRARYNAME equal to the libref, and lists SASHELP; the mock answers
LIBRARYID=<libref>, ENGINE='V9', LIBRARYNAME='Data Controller(<libref>)' and has
no SASHELP.
dsmeta is excluded on purpose: it is the engine's report on the dataset (owner,
host, encoding, paths, created dates), estate state rather than behaviour, and
the projection refused the recorded response until it was dropped - the site id
and the service account name were in it.
Two documentation findings from review:
- The comment in createdemodata.js and the parity README said the excel
fixtures hold "3, 18, 34, 79", which points at values the dropped rule never
saw. The uploads that the rule applies to go to dc_jslib, and
regular_excel.xlsx has five rows where SOME_SHORTNUM is 5 - that is the value
the 1-5 HARDREGEX rejects. The 3s belong to a multi-load fixture whose sheet
targets DC996664.MPE_X_TEST, a libref that carries no seeded validations
because getdata filters MPE_VALIDATIONS on base_lib === requestedLibref and
createdemodata copies only data files into it. Both places now name
regular_excel.xlsx and the value 5.
- README:9 promised the scan refuses an "eight-digit id" while the added class
is \bDC\d{8}\b. It now says "eight-digit DC id", which is what the scan does
and what the date case in parity-golden.test.js relies on.
Checked the fixture rather than the diff: column H (SOME_SHORTNUM) of
regular_excel.xlsx holds 18, 34, 79, 74, 98 ... and exactly five rows of 5.
The previous commit added "synthetic": true to the server target to stop the
recorder claiming an estate it had not checked. That put a harness-only key
into the shared build config, where nothing in sasjs reads it and a maintainer
could reasonably read it as a sasjs feature.
The recorder now carries the list of target names that point at the estate
makedata builds (SYNTHETIC_TARGETS = ['server'], which is sas.4gl.io locally)
and refuses anything else, so sasjsconfig.json is untouched. The declaration is
the same one - it cannot be derived from a response, because makedata is the
installer and a customer's control library holds the same rows - it just lives
with the tool that acts on it.
--target server passes the guard (and needs a token, as before); --target
server-mihajlo is refused with a message naming the allowed target.
The previous commit removed every validation rule SAS seeds for MPE_X_TEST,
which fixed the excel specs and broke editor 3: the cell dropdown they click for
comes from the HARDSELECT_HOOK on SOME_NUM, and that hook had gone with the rest.
Only one of SAS's rules actually conflicts with the fixtures - the HARDREGEX on
SOME_SHORTNUM, which rejects 1-5 while duplicate_row_excel.xlsx and friends hold
3, 18, 34, 79. Everything else stays, so the mock's rule set for the table is
again exactly what it was before the seed work: SOFTSELECT on SOME_BESTNUM,
HARDREGEX and SOFTREGEX on SOME_CHAR, SOFTREGEX on PRIMARY_KEY_FIELD and the
SOME_NUM hook.
The parity README now says this, rather than claiming the table is rule-free,
and createdemodata.test.js asserts both halves - the hook is present and the
SOME_SHORTNUM rule is not.
The parity gate's Cypress job went red after the seed work landed, and the two
causes are the same change seen from the fixture side:
- makedata now carries SAS's demo validation rules for MPE_X_TEST, including the
HARDREGEX on SOME_CHAR that blocks any text but 'the'/'data'. The excel
fixtures upload arbitrary values into that table and expect the submission to
reach the approval screen, so ten excel specs timed out with no Approve
button. createdemodata drops MPE_X_TEST's rules, which is what editor.cy.ts
already documents ("MPE_X_TEST stays clean to match the excel fixtures").
- the LABELs the viewer-labels spec asserts went with the hand-written schema.
createdemodata restores them, because a SAS estate has none and the control
library should not claim otherwise.
Both are declared as fixture divergences in the parity README and asserted in
createdemodata.test.js, so the next change to either side fails a test instead
of a Cypress run.
Review findings:
- the SYSTIMEZONEOFFSET acceptance depended on the host's zone; both entry
points pin TZ=UTC before the harness loads, so the accepted list is the same
everywhere.
- the sas9 mock still published '+0100'; it now answers seconds east of UTC
like the standard estate.
- the README promised an eight-digit id scan that IDENTIFIER_CLASSES no longer
had. A DC-prefixed eight-digit class is back (a plain digit class would fire
on a yyyymmdd date), with a case proving a date is not refused.
Six findings from review:
- estate: 'synthetic' was a constant. The target now declares "synthetic": true
in sasjsconfig.json and record.js refuses any other target, so the claim in
the file is one the tool had to check before it wrote anything.
- project() dropped the key when scrubbing a value, so a top-level member named
like a secret was published unredacted. The key is passed through, and
checkPublishable now runs secretViolations on the projection, so the recorder
refuses before writing rather than leaving it to the next pull request.
- denyList restated a subset of ENVELOPE_NEVER. It is built from that list now,
so an error text or a hostname in a discarded field is caught too (values
shorter than four characters are skipped - they would match half the file).
- replay() projected with golden._meta.members while the test validated
scenario.members, so editing the scenario had no effect until a re-record.
It projects with the scenario's list and reports a member-list mismatch.
- replay() deleted SYSCC/SYSENCODING from the expected side when the harness
omitted them, which would hide a service that stopped emitting them. The
harness now adds the two members the way the server does, and they are
compared.
- the eight-digit siteId scan class matched any eight-digit payload value (a
date, an amount) and refused clean goldens. Removed; the site id is caught by
the deny list, which now covers it.
Four findings:
- the replay was never run by CI, so the README's "a mock change fails the
pull request" was not true. parity-check.test.js now replays every scenario
through a shared parity/replay.js (which check.js also uses), fails on a
difference that is not declared and on a declared one that no longer differs,
and fails a scenario that has no golden. The differences that remain are
declared per scenario with a reason: the estate's admin group, its
registration state, its timezone and SAS's space padding of blank characters.
- recording stamped tls: verified for a non-https target while sending the
bearer token in the clear. A non-https target is refused outright now, and
--insecure still records tls: unverified.
- the committed-golden gate never applied the secret-key rule. secretViolations
now walks a golden (and a fresh projection) for secret-named fields whose
value is not <REDACTED>, with a case proving a hand-typed licence key fails.
- arg() returned argv[index + 1] unconditionally, so a trailing --ca reached
readFileSync(undefined). Flags without a value are a usage error now.
The replay also found a real mock defect, fixed here: SYSTIMEZONEOFFSET was
emitted as +HHMM while SAS answers seconds east of UTC ('3600'), and the mock's
own test asserted the wrong form - exactly the drift this harness exists for.
The sasdatasets sort used localeCompare, which puts MPE_X_CATALOG-FC before
MPE_XLMAP_RULES; SAS sorts on the byte value, so the mock now does too.
Two findings from review:
- record.js turned certificate verification off unconditionally, on the request
whose response becomes the golden. Verification is now the default; --ca
trusts a private estate's own certificate and --insecure is opt-in, warns on
stderr and is recorded as _meta.tls so a golden taken over unverified
transport cannot be mistaken for one that was not. The estate's certificate
verifies, so the committed golden is tls: verified.
- the libref placeholder ran before the staging-id one, so a staging id came out
as <LIBREF>-1_2_3. The staging rule now runs first and matches both minted
shapes (DC<date>-<time>_<n>_<n>, and the mock's millisecond form); a project()
case pins the ordering.
The pipeline cannot reach a SAS estate, so the goldens are refreshed where the
credentials are - an agent or a maintainer running record.js - and the CI half
stays offline: scan the committed goldens, replay them against a mock estate.
Say that in the harness README so the next reader does not go looking for a
scheduled refresh job that cannot exist.
The mock services are a second implementation of the SAS backend, and the only
thing comparing them was a human reading two files. This adds the harness that
compares their output instead.
A golden is a projection of a SAS response, never a copy: project.js keeps a
fixed member list per scenario, drops the whole envelope (hostname, site id,
user, home paths, job ids, _PROGRAM), substitutes the identifiers the diff
needs (<LIBREF>, <PATH>, <EMAIL>, <STAGING_ID>, <DATETIME>), and redacts secrets
by key name. The recorder then refuses to publish if any identifier class
survives, or if any literal the envelope just discarded still appears in the
projection - which is what catches a hostname echoed inside an error message.
record.js records from a live estate into .captures/ (gitignored) and writes
only the projection to golden/. check.js replays a golden against a mock estate
built by the test harness and prints field-level differences; it needs no
server and no credentials. parity-golden.test.js re-runs the identifier scan
over every committed golden, so a hand-edited file cannot slip through either.
Phase 0 ships one scenario, public/startupservice. The replay is advisory and
already reports twelve differences worth a look (admin group casing, the
registration counters, timezone representation, the sasdatasets order, blank
versus empty Excel map descriptions); the corpus, the merge gate and the
nightly drift job follow.
Stacked on the mocks context in the parent branch: it seeds the mock estate with
seededEstate(), which that branch adds.
The fixture's schema entry lived in makedata's schema object (as a second,
later key in the same literal) and did not survive the move to createdemodata.
saveTable falls back to the file already on the drive, so a reseed looked
fine - but CI deletes the appLoc before seeding, and a fresh estate wrote
mpe_x_search.json with columns: [], losing the column order the
full-table-search spec derives its counts from.
Carry the nine columns (name/type/length/format/label/notnull) in
createdemodata's schema, and add a case that every fixture table is written
with column metadata covering its rows, plus the MPE_X_SEARCH order. It fails
against the previous service.
The panel heading reads 'Configuration (#2 DC_JSLIB.MPE_ALERTS)', so the check
parses lib.dataset out of it and compares it whole. A substring test accepted
another library's table of the same name - this estate seeds MPE_X_TEST under
two libraries - and any name that merely contains the target, which is the
mix-up the check exists to catch.
Verified on the local stack: 12 passing in 2:24 and 2:22.
createdemodata assigned tables.MPE_DERIVED outright, so any row makedata wrote
there would be wiped by a fixture run - and the idempotency test did not count
the table, so nothing would have said so. Filter the TESTDATA rows out and
append, in the same shape as the tables above, and count MPE_DERIVED in the
idempotency test.
New case: a non-TESTDATA derived row survives a createdemodata run exactly
once. It fails against the previous service and passes now.
The table link is looked up inside the matched library node rather than across
the whole picker, so a table of the same name under another expanded library
cannot be clicked instead, and the library fragment is escaped before it
becomes a RegExp - a dot or a bracket in a library name no longer matches more
than the intended node.
The configuration panel is checked after the ~600-800ms window in which
loadData() re-selects the viewbox it just loaded, rather than 400ms into it,
so a moved panel is caught at the click instead of in the column list that
follows.
Verified on the local stack: 12 passing in 2:24 and 2:22.
- createdemodata drops the unused servicesDir binding.
- the DC996664 copy no longer swallows a failure: a broken multi-load fixture
now fails the seed where it happened instead of surfacing in a Cypress spec.
- makedata answers OUTFMT=JSON with a makedata_response member (as the SAS
service does) and the CLI payload asks for it, so the `|| true` comes off
makedata in build.yaml, release.yaml and sites.yaml - a failed seed names the
service that failed instead of reappearing as "createdemodata needs the
control library".
- the makedata comment points at sas/mocks/test/makedata-parity.test.js, not a
macro that does not exist.
makedata seeded two things at once: the control library, and the tables the
Cypress suite needs. The estate it produced was therefore not the estate a
SAS deployment produces - and it had drifted: MPE_MAXKEYVALUES and
MPE_SIGNOFFS were never created, and MPE_CONFIG, MPE_VALIDATIONS, MPE_TABLES,
MPE_SELECTBOX and MPE_XLMAP_RULES were missing seed rows that mpe_makedata.sas
inserts.
makedata now transcribes mpe_makedatamodel.sas and mpe_makedata.sas: every
model table with its columns, and every seed row the macro writes. The
Cypress fixtures move to services/admin/createdemodata.js, which the test
estate runs after makedata; it strips and re-adds its rows so a reseed does
not accumulate.
Checked against real SAS output on the shared estate (fresh deploy of the
server target, makedata, then getrawdata for each control table and a
dictionary.columns probe): all 40 tables and 339 columns match on name, type,
length, format and label, and the seeded rows match row for row apart from
the environment (libref, user id, group names) and the sample rows. That run
found six defects a source-only comparison could not: INT columns are 8 bytes
not 4, one label had lost a double space, the DC_MACROS setting held the SAS
expression instead of a path, the email templates held an escaped newline,
75 selectbox rows held %mf_increment(rk) instead of their key, and one
validation row was truncated at a ';' inside its regex.
makedata-parity.test.js parses both macros and compares them with the seeded
estate, and fails on a value that does not fit its column type, so the next
drift fails a test instead of shipping. The CI workflows, the Sites demo seed
and CONTRIBUTING call createdemodata after makedata.
loadData() re-selects a viewbox about 600ms after its data lands, and that can
land after the row click that opens its configuration - leaving the panel on
the other viewbox and the test looking for a column that table does not have.
The panel's heading names the viewbox it is showing, so the click is repeated
until the heading matches rather than trusting the click once.
The row is only clicked when the panel is not already on that viewbox, since
selectViewbox() toggles and a second click would deselect it.
Verified on the local stack: 12 passing in 2:30 and 2:20, against 5:22 for the
same spec before the change.
openTableFromViewboxTree paused 300ms before looking at each rendered node,
which on a library of this size cost ~13.4s of the ~20s a viewbox test takes.
The helper now expands the library and clicks the table's link, retrying the
expansion when the picker re-renders underneath it.
It also waits for the picker's own state before returning: every requested
table listed, no row spinner, and the ~600ms window in which loadData()
re-selects the box that just loaded. Without those, the next step read the
other viewbox's column list and failed on a column it does not have.
Measured on the local stack: the spec runs 2:18-3:15 against 5:22 before, with
all 12 tests passing in both.
The SAS Logs modal rendered the log, the source code and the generated code as one text block each, so no line could be referred to and the error, warning and note links had to locate a line by arithmetic. Render one row per line instead: every line carries its number, a wrapped line hangs under the text rather than under the number, and the numbers stay in view when a long log line is scrolled sideways.
The links now scroll to the row itself. The old offset was the measured line height multiplied by the line index, and two things were wrong with it: clientHeight rounds the height to a whole pixel, and the wrapper's padding was not counted. In a 168-line log at the stylesheet's own metrics the early links landed within a few pixels of their line while the later ones did not; at an 18.5px line height, which is what 16px monospace at 1.15 produces, the scroll for line 43 was 10.5px out and for line 127 was 52.5px out, so the highlight sat above the line it named and the drift grew with the line number. A row's own position cannot drift: every link lands its line at the top of the log, less the wrapper's 1px border, and flashes the right row.
The flash is a class on the rows rather than an absolutely positioned overlay, so a message SAS wrapped over several lines flashes every line it runs to.
One row per line costs about 0.25ms in Angular - a binding plus a sanitizer pass each - so rendering a whole log would freeze the modal: 20,000 lines took 5.3s to open. Each pane renders a window of 400 lines with a "Show more" line underneath, a jump renders through to the line it was given before scrolling, and opening is now flat at 165-280ms for 1,000, 5,000 and 20,000 lines.
dc
allan