7.1.7 closes the outstatus step with run, stops truncating the flow table's
state, and strips the response envelope before the JSON libname; 7.1.8 lets
mp_replace delete, which is how the envelope is stripped. Without them a Viya
run cannot read an aborted child's result: ten of Data Controller's SAS suites
report no result at all.
The payloads are the SAS 9 recordings, separators included - the front end splits
metadata URIs on the backslash, so a value that arrives without one exercises the
wrong branch. The envelope is not SAS 9: _PROGRAM, SYSUSERID, SYSVLONG and
SYSDATE/SYSTIME come from whatever runtime serves the request, so this variant
must not be read for SAS 9 envelope values.
The guard is the tree's one shared rule, but each of the three services that
needs it repeated the path derivation and then eval'd a file read out of the
SASjs Drive at run time - which is what the file's own header said this tree does
not need. weboutGuard.js now exports the function and the services require it by
absolute path: one copy of the function, no eval, and a load failure is a
require error naming the file rather than a silent no-op.
Being a module also means the guard has to carry its own dependencies: as an
eval'd snippet `fs` came from the program's scope, and the first run as a
required module failed with "fs is not defined". It now requires fs itself.
dcMockUtils.requireMemberMetadata remains the source of truth for the rule and
both files say so, since a site may run both trees and an operator reads
whichever one threw.
Two review findings on the approval hooks.
mpeRunhook read only the lower-case column name, while a row written
through the editor carries the schema's upper-case names (PRE_APPROVE_HOOK)
and the buskey/loadtype reads beside it already try both. A hook
registered that way resolved to nothing, so the approval ran without it
and said nothing.
postdata ran the pre-approve hook before the duplicate-approval guard.
postdata.sas checks prev_upload_check first and returns ALREADY_UPDATED
without reaching the hook, so a second approval from the same user was
aborting with the hook's message instead. The guard is now computed
before the hook and the hook is skipped when it fires. The hook still
runs for SHOW_DIFFS, which is what the SAS service does too - the macro
call sits outside both action branches.
Both cases are pinned by a test, and each fails when its fix is removed.
postdata.sas runs PRE_APPROVE_HOOK before the diff and before any write,
and POST_APPROVE_HOOK after the load is applied. The mock ran neither,
so an approve hook registered on MPE_TABLES did nothing on the
browser-hosted estate - while the post-edit hooks (which live in
stagedata/loadfile, mirroring mpe_loader) did run.
mpe_runhook(tableReg, hookvar) in dcMockUtils picks the hook out of the
MPE_TABLES row by column name and returns its source; the service
eval()s it in its own scope, which is the JS equivalent of the macro's
%include sharing the caller's macro variables. The three call sites that
already resolved a hook by hand (getdata pre_edit, stagedata/loadfile
post_edit) now go through it, so the next hook column costs one line.
postdata runs pre-approve before the staged data is diffed and aborts on
failure (nothing is written), and post-approve after the base table and
control tables are updated - so its failure is reported but cannot undo
the load, which is what postdata.sas does with its SYSCC= response.
A hook refuses by throwing or by setting hookErrmsg, the convention the
post-edit hooks already use.
sas/mocks/test/hooks.test.js covers the ordering: pre-approve refuses
with the base table untouched and the submission still SUBMITTED;
post-approve aborts with the row already in the base table and
MPE_SUBMIT APPROVED; and one hook proves it ran by writing a file, not
by the service reporting success. 30 tests pass; 4 of the 5 hook cases
fail against the base service.
Three more places where the mock answered differently from the SAS service.
mp_abort (dcMockUtils): every mock refusal is now a sasjsAbort table
(MSG + MAC) written where the macro writes it, so the client shows the
abort modal and rejects the promise. A thrown error had become an HTTP
500 "Job execution failed" - the browser host discards the webout when a
program throws - which is unreadable to the user and hides the message in
a stack. mpAbort also stops the program, so nothing is written after a
refusal; each service that can abort catches that stop at its top level.
stagedata, loadfile, getauditfile and makedata now go through it, and the
hand-rolled sasjsAbort blocks are gone.
mp_lockanytable (dcMockUtils): a lock is a row in MPE_LOCKANYTABLE marked
LOCKED carrying the holder's job id, user and reference, exactly as the
macro records it, and a lock held by another session refuses the write
with the macro's message. postdata and rejection now lock MPE_REVIEW and
MPE_SUBMIT around each write, which is where postdata.sas and
rejection.sas take them. The macro's retry loop is collapsed to a single
attempt: the runtime is single threaded, so nothing can release a lock
while the request waits.
startupservice: saslibs is the distinct LIBREF of the permitted datasets,
which is what "create table saslibs as select distinct libref from
&syslast" produces - it was a four-column row describing the control
library, with fields the client does not read.
Tests: abort.test.js and locks.test.js join the harness suites; 25 pass,
and 9 of the 13 relevant cases fail against the base services.
Every screen reads public/startupservice, so a divergence there is a
divergence in the whole app. Three of its answers were fixed values:
xlmaps a hardcoded list of three maps, so editing MPE_XLMAP_RULES
or MPE_XLMAP_INFO changed nothing on the site
sasdatasets included the map target dataset, which the SAS service
deletes - the demo offered MPE_XLMAP_DATA as an editable
table while SAS refuses it
timezone TIMEZONE and SYSTIMEZONEIDENT pinned to Europe/London and
the offset to +0100, so a GMT session reported an hour of
DST it did not have
They now come from the same sources the SAS service reads: the current
MPE_XLMAP_RULES rows joined to MPE_XLMAP_INFO (coalescec to
<mpelib>.MPE_XLMAP_DATA), the target datasets removed from sasdatasets,
and the session zone and current offset taken from the runtime, with
TIMEZONE left blank as an unset option is.
The shared harness (sas/mocks/test/harness.js) makes the service tests
one-liners: it builds a temporary estate with every service on its drive
and runs a service file the way the runtime does, webout() included.
sas/mocks/test/startupservice.test.js covers the joins, the target
removal, closed rules and the offset.
Staging now opens on the deploy screen, and its folder picker asks
makedata for a listing. The mock ignored ACTION entirely: every request
seeded the control library, deleted makedata from the drive and replied
with the HTML success page. So the first BROWSE deployed the site and
closed the wizard the visitor was standing on, behind an "invalid Json
string" error from the adapter, and VALIDATE and SUBMIT then answered
500 "Program not found in drive".
The mock now dispatches on ACTION the way the SAS service does:
DIRLIST the folders below PARENT, one level at a time, with the
dirinfo/dirlist shape mpe_dirlist emits
WRITECHECK the checkwrite row mpe_writecheck emits, probed by creating
and removing a file
MAKEDATA the existing seeding path, unchanged
ACTION is required and refused when absent or unknown, and the two browse
actions are refused once the settings file exists, mirroring browse_gate.
Both input routes are accepted - the screen's fromjs table and a CLI
seed's SASControlTable - because %webout(FETCH) does not care which one
arrived; deploy/makedata.json now carries ACTION=MAKEDATA as the real
service requires.
sas/mocks/test/makedata-actions.test.js runs the service file itself
against a temporary estate and is wired into the build workflow.
Three review follow-ups on the guard the sas9 tree loads.
- A body that cannot be read is now its own error, naming the path and the
filesystem message. Reporting it as missing metadata sent the reader to a
@sasjs/utils reinstall that cannot fix a path.
- One statement of the dependency fact: showmeta arrived in @sasjs/utils
4.6.0 and @sasjs/server declares that floor from 3.12.3 (3.12.0 and
3.12.2 declare ^4.4.0, checked with npm view), so the message names the
server version to install instead of "reinstall the server".
- CONTRIBUTING lists _program and weboutPath among the predeclared globals,
which the guarded services use to locate the helper.
The sasjs tree's copy in dcMockUtils.js had the same two defects, so both
now say one thing rather than one tree being clearer than the other.
The sasjs tree has emitted its responses through the runtime's webout()
since #461; the SAS 9 tree under sas/mocks/sas9/ still assigned every
captured payload to _webout, so its envelope was frozen at the 2022
recording and its values carried the separators a template literal eats.
All 35 services now call webout('OPEN') / webout('OBJ'|'ARR', name, rows)
/ webout('CLOSE'), with the three services that carry column metadata
passing { showmeta } and loading the guard from the new weboutGuard.js.
The payloads come from the captures read as JS produces them, with the
backslashes a template literal silently drops restored: 121 separator
values across 14 services - OMS URIs (OMSOBJ:<Class>\<repo>.<id>),
BUILTIN\Administrators, SAS\sasdemo and D:\... / C:\... paths.
Staging is the preview of the next release, and what it should preview first is
the deployment itself. DC routes to /deploy while makedata is still on the drive
(at <appLoc>/services/admin) and stops doing so once the wizard has run it and it
has removed itself - so a drive that carries the services and the licence but no
control library opens on the deploy screen, and the visitor deploys the control
library from the wizard into their own IndexedDB drive.
The seeding step is now gated on a tag: only the demo (which a visitor should be
able to use straight away) runs makedata before the drive is captured. The
staging path gets a guard of its own, because a seed that leaked into it would
only be visible as a site that opens on the home screen: makedata must be on the
drive and no mpe_tables.json may be there.
CONTRIBUTING updated: the two sites now ship different drives from the same
build.
The captures wrote values like OMSOBJ:SASLibrary\A59LNVZG.B500000K inside a
template literal, where \A is an escape, so the separator never reached the
client: it served OMSOBJ:SASLibraryA59LNVZG.B500000K. The same happened to
D:\MyDemo\Sankey and BUILTIN\Administrators. Converting the service to webout()
faithfully preserved the corrupted value - and the metadetails ASSOCURI, the one
field fixed by hand at conversion time, was left as the only member that kept its
separator, which is the inconsistency the review named.
Seventeen values corrected across five services:
lineage/fetchtablelineage info[0].LIBURI 1
lineage/getmetacols metacols[].COLURI 11
lineage/getmetatables PATH_SCHEMA, TABLEURI 2
public/refreshlibinfo libinfo[0].OWNERS 1
usernav/usermembersbyrole sasgroups[].URIMEM,
sasmembers[].URIMEM 2
Verified by deploying both trees on one server and comparing the parsed
responses: those seventeen fields are the only differences, and every other
member of all thirteen converted services is unchanged (metadetails' two
ASSOCURI, already carrying its separator, still differ from the capture for the
same reason).
Anything that splits a URI or a Windows path on the separator, or displays
OWNERS or PATH_SCHEMA, now sees the value the SAS metadata server returns.
3.14.0 ships @sasjs/lint 5.0.0, so the studio reports OS commands, and it is the
release the estate runs. It still declares @sasjs/utils ^4.7.0, so the envelope
formats from utils 4.8 (SYSCC as text, DATE7, TIME5, E8601DT26.6) are not yet
guaranteed by the floor - a fresh install resolves 4.8.0, a cached tree need not.
The comment says so, and the pin moves again when the floor lands.
Verified on a local 3.14.0 install (clean, resolving utils 4.8.0): the mock
estate's startupservice returns SYSDATE 04NOV26, SYSTIME 12:44, SYSCC "0" (text),
END_DTTM 2026-11-04T12:44:14.954000 (no Z). With that estate seeded, and against
a fresh app location, licensing.cy.ts passes 10/10 and metanav.cy.ts 1/1 on
Chromium.
The other nine services that answer with a captured response also assigned it to
_webout, envelope and all. They now emit their members through the runtime's
webout():
lineage/fetchtablelineage info, finalfinal, flatdata, idlookup
lineage/getmetacols metacols 11 rows
lineage/getmetatables metatables 1 row
usernav/usermembers users 2 rows
usernav/usermembersbygroup sasmembers 2 rows
usernav/usergroupsbymember groups 1 row
usernav/usermembersbyrole sasgroups 1 row, sasmembers 1 row
usernav/userroles roles 46 rows
public/refreshlibinfo libinfo 1 row
Each payload was taken from the response the old service serves, then both
trees were deployed on one server (/Public/app/dc and /Public/app/dcnew) and
compared as parsed JSON with the envelope keys excluded: all nine are equal row
for row, including the escaped values (the OMS URIs keep their backslash, and
refreshlibinfo keeps the quoted paths).
The envelope is the runtime's now, so SYSCC is numeric 0 rather than the string
"0", SYSDATE/SYSTIME are today's, SYSSITE is reported (these captures carried a
lower-cased syssite, which the client never read), and the SAS-only
_METAUSER/_METAPERSON are gone.
lineage/fetchlineage is deliberately left alone: its captured body is not valid
JSON today (bare quotes in the graphviz labels, raw newlines inside the STRING
values, and a lost backslash), so there is no payload to preserve and the
intended text needs a decision rather than a rewrite. Tracked separately.
The four metanav services assigned a captured 2022 response to _webout, envelope
and all, instead of using the runtime's webout() like every other service. The
members are unchanged:
metarepos outrepos 2 rows
metaobjects objects 1 row
metatypes types 71 rows
metadetails attributes 12 rows
associations 2 rows
Both trees were deployed on one server, at /Public/app/dc and /Public/app/dcnew,
and the responses compared as parsed JSON with the envelope keys excluded:
outrepos, objects and types are equal row for row.
metadetails changes in one place. The capture wrote the OMS URI as
OMSOBJ:Property\A59..., but \A is an escape in a JavaScript template literal,
so the backslash was consumed and the client received OMSOBJ:PropertyA59... -
a URI that cannot resolve. The member now carries the backslash the SAS
metadata server returns.
The envelope comes from the runtime now, which also drops three artefacts of
the recording: SYSCC "4" on metarepos (an error code from the captured day,
which the client reads as a SAS error), SYSDATE/SYSTIME frozen at 26SEP22, and
the SAS-only _METAUSER/_METAPERSON.
The mock services call webout(), as the server's runtime does. The browser mock
runtime that @sasjs/cli ships is what provides it there, and the 6.5.0 this repo
was pinned to does not: its runtime-core.js has no reference to the webout
prelude, and the build writes no webout-prelude.json for it to load. Every
service that calls webout() therefore throws "ReferenceError: webout is not
defined" in a published mock site while the build itself reports success.
6.9.0 is the first release that ships the prelude; this pins 6.10.0, the current
release. The transitive @sasjs/adapter and @sasjs/utils entries move with it.
The guard that keeps a runtime without webout()'s options argument from serving a
thinner response tested webout.length >= 4. Function.length stops counting at the
first default or rest parameter, so a runtime that does carry the option but
declares it `options = {}` or gathers the tail with `...args` reports three, and
every service that calls the guard refuses to start on a working runtime.
It now asserts the outcome instead: the member the option produces is looked for
in the response body the program is writing. `$sasdata`, `$viewdata`, `$vals` are
the members the client reads for column formats and labels, and their presence is
the thing that actually has to hold.
Where it runs matters too, and the first attempt got that wrong: called after
webout('CLOSE') the throw leaves a complete body behind, and the server serves it
- the caller sees a normal response with SYSCC 0 and no metadata, which is exactly
the silent failure this exists to prevent. It is called before the envelope is
closed, so an unhonoured option fails the run.
@sasjs/utils 4.7 makes webout() emit SYSSITE from a configured site id,
defaulting to 123, where %webout takes &syssite. Until 3.13.0 the prelude
wrote an empty string, and dc's client routes an expired licence key and a
wrong-organisation key only when startup_site_id is truthy - so on the mock
estate both pages were unreachable and licensing.cy.ts failed three cases.
Three values reached `run:` blocks through `${{ }}`, so the runner pasted them
into the script before the shell ever saw it - which makes the value part of the
code rather than an argument to it.
- `BASE="${{ github.base_ref }}"` (build.yaml, lighthouse.yaml) - a ref name in
the script.
- `echo ${{ secrets.SHEET_PWD }} | gpg --passphrase-fd 0` (all four workflows) -
a passphrase in the script. A quote in it ends the quoting, and a command
substitution in it runs.
- two `replace-in-files --replacement=...${{ secrets.CYPRESS_* }}...` lines
(build.yaml, release.yaml) - the same, inside a replacement string.
Each now arrives as a step `env:` value and is expanded as `"$VAR"`, so the shell
only ever sees data. Checked the replacement lines side by side with the old form:
identical for an ordinary value, and the new form keeps a value containing a space
intact where the old one word-splits it, and survives a value containing a single
quote where the old one is a shell syntax error.
The passphrase pipe also moves from `echo` to `printf '%s\n'`, so the byte the
decryption sees does not depend on how the shell's `echo` treats backslashes.
Review finding: when every body is refused by carriesClause, or the response
carries no FILTER_HASH, capture.hash stays empty - and storedClauseFor('')
then spends its 60 retries before reporting "no clause for <var> under",
which names neither the refused body nor the missing hash.
The hash is now asserted non-empty before the read, with the case's own
message, so a broken delivery fails where it happened.
The call count stays where it was - after the data load - because moving it
ahead of the read would check it before the app has issued the second POST
that the count exists to catch.
The duplicate check waited a fixed 250ms after reading the stored row, which is
wrong in both directions: on a slow runner the window can elapse before the first
response callback has incremented the counter, and a second POST landing after it
passes with two deliveries.
The app's sequence for an apply is validatefilter, then the data load for the
filtered route, so the load is the completion signal: the count is asserted after
it, and nothing depends on how fast the machine is. A delivery issued later than
that is a second apply, not a duplicate of this one.
Also corrects the ordering paragraph in cypress-specs.sh: it still said specs are
appended, in a file that now places va-ddc-host ahead of metanav. va-ddc-host does
not depend on the estate being unlicensed either - it passed 3/3 immediately after
metanav planted a key in the full CI run - so the bullet says what is true and
metanav stays the only spec with a fixed place.
Review finding: the matcher only accepted the variable name at the start of the
body or after a comma or newline, so a delivery wrapped any other way was not
counted - and the case then failed at the call-count assertion, reading as
"the app never called validatefilter". Reproduced against the previous
implementation: a urlencoded form value, a form value that starts with the
name, a JSON object keyed on the column and a JSON string value holding the
row all matched nothing.
The body is now expanded into the candidate strings it can carry - the raw
text, the percent-decoded text, each form field's value, and every string leaf
of a JSON body - and the name has to be a whole CSV field of one of them.
Field equality rather than delimiters means a quoted literal, a longer name,
an SQL fragment or another variable's name is still refused, and there is no
regular expression over user text to get wrong.
Also restores the ordering note in cypress-specs.sh: va-ddc-host asserts
nothing about licensing, so it runs before metanav, the one spec that plants a
licence key.
The pin moved to 3.12.3 with the webout() change but the comment above it still
named 3.12.2 as the current release - true when it was written, wrong under the
pin it now sits on. The comment also only carried the /ddc-host reason, which
does not explain this bump.
3.12.3 is the first server release declaring @sasjs/utils ^4.6.0, and that is
what gives webout() the options argument getdata, viewdata and getcolvals pass as
{ showmeta }. Say so where the constant is, so the next person to move it knows
the metadata members depend on it.
The services in this branch pass { showmeta } to webout(), which is
@sasjs/utils 4.6 and up. sasjs/server 3.12.3 is the first release whose
package.json asks for ^4.6.0 (a clean install resolves 4.6.1), so the pin
moves from 3.12.2 to 3.12.3 in every workflow that starts the mock estate.
Verified against the published package rather than the release notes:
`npm view @sasjs/server@3.12.3 dependencies.@sasjs/utils` answers ^4.6.0, and
`npm install @sasjs/server@3.12.3` in an empty directory installs
@sasjs/utils 4.6.1, whose webout prelude carries showmeta.
Review of the release diff: `npm install -g @sasjs/cli` and
`replace-in-files-cli` were unpinned, so CI ran whatever release was current
and a bad one would reach the pipeline unreviewed - next to a SASjs server
that is pinned to an exact version.
The CLI now comes from the version sas/package.json pins, which is the shape
docs.yaml and release-assets.yaml already use, so there is one source for it.
replace-in-files-cli, pm2 and wait-on are pinned to the versions in use
(4.0.0, 7.1.2, 9.5.1).
Both node expressions are checked from the directories they run in:
`require('./sas/package.json')` at the root and `require('../../package.json')`
in sas/mocks/sasjs both answer 6.5.0.
The three services that pass { showmeta } - getdata, viewdata, getcolvals - rely
on webout()'s fourth argument, which arrived in @sasjs/utils 4.6.0. @sasjs/server
3.12 declares ^4.4.0, so an install can resolve 4.5.x, and there the argument is
dropped silently: the response still looks complete but $sasdata, $viewdata and
$vals are absent, so the client's format and label paths stop being exercised
while the mock estate reports success.
requireWeboutOptions() checks that the generated webout() takes an options
argument (arity, because the option names live outside the callable function) and
throws naming the member, the cause and both fixes. A 4.6 runtime is unchanged.
The matcher tested the variable as a substring of the whole request body, so a
RAW_VALUE, an SQL fragment or another variable whose name contains it was counted
as another delivery of this clause and failed the count. It now requires the name
between its CSV delimiters, which is the filterquery VARIABLE_NM field; a quoted
literal cannot match, because its delimiters are quotes.
cy.wait('@validatefilter') used the 5s default, where the polling loop it replaced
tolerated 15s, and the message only arrives once the framed page has rendered.
It now waits for longerCommandTimeout, as the rest of this spec does.
The capture assertion also carries the tail of the host page's log in its message:
the failure reads "expected -1 to be above -1" otherwise, and the log is what says
whether the app never exposed its buffer or the frame never loaded.
Two review findings:
- The load handler re-posted pendingPayload on every load event of the frame,
so an in-app navigation or a self-reload posted a filter that belonged to
the mount which had already delivered it, and logged it as that mount's.
The handler now takes the payload (clearing pendingPayload and
pendingTiming) before posting, so a later load has nothing to send.
- The README illustrated report.json with one parameter while the committed
file carries three, and the prose below reads as though one parameter were
the shape. The example block is gone: the file is the example, and the keys
are documented as a table with the parameter rules kept underneath.
The last eleven services still built the response envelope through
dcMockUtils' webOutOpen/webOutObj/webOutClose. They now use the webout() the
SASjs Server emits into every JS program, so dcMockUtils no longer emulates
%webout at all - the helper existed only for these callers and is deleted
with them.
That needed the options object added to webout() in @sasjs/utils 4.6
(showmeta, maxobs), which is what a clean install of @sasjs/server@3.12.2
resolves to (^4.4.0 -> 4.6.1, checked). A server whose install still holds
4.4/4.5 refuses the option by name rather than ignoring it, so the failure is
loud rather than a response that is quietly missing its column metadata.
getdata, viewdata and getcolvals send their metadata as
`webout('OBJ', name, rows, { showmeta })`, and every other call keeps its
member name and rows unchanged.
One call was positional all along: startupservice's xlmaps is an array of
arrays, which the old helper serialised as-is and webout() refuses for OBJ.
It is now webout('ARR', 'xlmaps', xlmaps) - caught by the comparison run
below, not by the rewrite.
Two gaps in the guard that decides which validatefilter response belongs to the
case.
A body Cypress parses into an object stringified to "[object Object]", so no
call matched and the case failed on the count with nothing naming the body shape.
The body is now serialised the way the response already was - a string is used as
it is, anything else is JSON.stringify'd.
The count was asserted immediately after the first response, which is before a
second POST from the same apply could reach its own callback, so a duplicate read
as one call. The count is now asserted after the stored row has been read back,
with a settle that covers a second POST still in flight when the read returned.
Two review findings on host.js:
- postToFrame() used '*' as targetOrigin, so the filter clause went to
whatever document occupied the frame. The frame is the app on report.server,
so that origin is the only one that should receive it; the origin is now
resolved once from report.server (new URL(...).origin) and a report.server
that is not a URL logs a visible error instead of widening delivery. The
page also says so when it is not on the frame's origin, because the capture
probe cannot read across origins and that would otherwise look like a
harness failure.
- startEarlyReplay() accepted any __vaLastMessage once it had posted, and on
a remount to the same URL the old document is still on screen carrying the
buffer the previous mount captured. A capture now only counts once the
document has been replaced (mountDocument), both in the interval and in the
load-event check. Keying to the document rather than waiting for a load
event keeps the first post where the early window is - the interval starts
exactly as before.
allan
dc