The SASjs Server / SAS9 screen put the DC Staging Directory first and the admin
group second, the reverse of the Viya screen, so the operator was asked to pick
a location before being told who administers DC. It now asks for the admin group
first and the directory last, with the copy reordered to match.
A rendered-order spec covers it: it reads the labels off the rendered template
and fails if the directory comes first again.
DIRLIST and WRITECHECK were refused once the appLoc had a settings.sas, on the
argument that a failed run could leave that file behind. That is what makes the
gate wrong: a failed run is exactly when the deploy screen still needs to list
and probe directories, and any appLoc configured earlier refuses them for good.
The window needs no separate clock - a successful run deletes this program, so
once DC is configured there is nothing left here to call.
Removed from both variants (server checks the drive over HTTP, Viya checks the
filesystem; neither is needed), and the two tests that asserted the refusal go
with it. The macros themselves are covered by mpe_dirlist.test.sas and
mpe_writecheck.test.sas.
Seen on sas.4gl.io: a deployment to /Public/app/dc477 answered BROWSE with
"DIRLIST and WRITECHECK are only available until DC is configured".
The release pipeline went red at 28157fae: Karma aborted with "Some of your
tests did a full page reload!" after test 753 of 861. The new-window spec calls
runMakedataInNewWindow, which starts the five-second makedata poll; the service
stub answers that makedata is gone, so the first tick assigns window.location.href
and the page reloads under the running spec. A local run finishes in under five
seconds, so the timer never fires there - only CI saw it.
- the poll is cleared in ngOnDestroy, so a timer cannot reload the page after
the user has moved on; a spec covers that
- the spec that exercises the new-window URL no longer starts the real poll
Three open findings on #407, all taken:
- lighthouse.yaml: sas/package.json joins the change-detection pattern, because
the job pins @sasjs/cli from that file
- release-assets.yaml: webpack is installed at the version the client lockfile
resolves rather than whatever is current at release time
- build.yaml: the client audit runs with --audit-level=low, which is what the
comment above it has always claimed
Both test programs pointed _webout at a temporary file so the macros under
test could write their JSON answer, then tried to hand the fileref back with
`filename _webout "&_webout_path" mod;`. On Viya's JES _webout is a filesrvc
fileref, whose pathname() is empty, so that statement fails with "Invalid
physical name" and the temporary file stays in place. The term program's
%webout(OPEN) assigns _webout itself (filesrvc on JES, temp elsewhere), so the
restore is not needed at all; the two programs now leave the fileref to it.
Observed on the nextviya estate: the suite reports 64/64 of the tests that
produced results, with mpe_dirlist and mpe_writecheck absent from the results
JSON and "Invalid physical name" in their logs.
The spec failed on CI: the browse button never appeared. Two causes, both
found by reproducing the CI estate locally (seeded drive):
- The intercept pattern '**/drive/folder**' does not match the request the
app makes ('/SASjsApi/drive/folder?_folderPath=...') in this Cypress
version, so the app saw the real drive, loaded the startup service and
never routed to the deploy screen. Both intercepts are now regexps, which
match deterministically. Verified on the seeded estate: the app routes to
#/deploy and the agreement card renders.
- A secured estate shows a login form first. The boot helper now waits for
one of the states it can act on (a visible login form, the agreement card,
or the deploy screen), signs in when the form is visible, then waits for
the deploy screen. The login form stays in the DOM hidden on other routes,
so visibility - not presence - decides.
Review findings on #476.
- The 'hands the picker the context' test declared only AutomaticComponent
and HostComponent with NO_ERRORS_SCHEMA, so app-dc-location stayed an
unknown element: the [computeContextName] binding was never applied and
the assertion just repeated the fallback case. DcLocationComponent is
now declared (SasService is already stubbed), and the test asserts the
picker's own input - both before a selection (the configured fallback)
and after one is chosen. Verified it bites: removing the binding from
the template fails it.
- The Cypress spec set the path nowhere before clicking Validate, so the
assertion leaned on the field arriving pre-filled; a blank field would
short-circuit with 'Type or pick a folder first' and fail for a reason
unrelated to the layout. It now types the path explicitly.
- The spec slept a fixed 4s after accepting the agreement, which turns a
loaded CI runner into an intermittent failure. Both tests now share one
boot helper whose readiness gate is the Browse button appearing.
Two defects on the deploy screens, found on staging.
1. The folder picker's outer row reused Clarity's 'clr-input-wrapper'
class, whose 'max-height: var(--clr-forms-input-wrapper-height)'
clamps the row to the input's height. The row is flex-wrap: wrap, so
once the Browse/Validate buttons wrapped (or the row was simply
taller than one input) they overflowed the clamp, and the folder
browser panel - positioned after the collapsed box - rendered on
top of the input, the buttons and the surrounding text. Measured on
staging: panel top 256 against input 248-272 and buttons 280-304;
with the class dropped the row contains all its children and the
panel starts at 312.
2. The Viya screen asked for the DC directory before the compute
context, but the folder listing and the write check run inside the
chosen context, as its batch identity - so the probes could inspect
a file system the deployment itself would not use. The fields are
reordered (admin group, then batch id / compute context, then DC
Loc last) and the picker now runs DIRLIST and WRITECHECK with the
selected context in the adapter request config, which every Viya
executor path honours. The SAS 9 / SASjs screens pass no context
and behave exactly as before.
Tests: a Cypress spec drives the real deploy screen estate - the
picker row must contain its own children and the panel must start
below them; both tests fail with the clamp reintroduced. A rendered
TestBed spec pins the Viya field order (admin group < compute context
< DC Loc) and fails with the old order. Unit specs pin the probe
config: no context on SAS 9 / SASjs, the chosen context on Viya.
Three review findings.
- KEY_HASH was upper-cased after the digest. SAS md5() writes lower-case hex,
so the mock's key hash did not match what a real audit table holds. The
digest is stored as it comes out, and the test asserts lower case.
- restore resolved MPE_TABLES.AUDIT_LIBDS per table, while the approval always
appended to the control library's MPE_AUDIT: a table with its own audit table
had its diffs written where restore never looked, and restore then refused
with "No entry for <load>", which no later approval could fix. Resolution now
lives in dcMockUtils.auditLibdsFor, called by both sides, and the approval
reads and saves that table in its own libref.
- An audit failure was only logged to the server console, leaving an approval
with no diffs and no visible reason. It is still not allowed to undo the
approval, but once the control tables are written the failure is raised as an
abort that names the table and says the load can no longer be rewound.
Two tests: one approves a table whose MPE_TABLES row names a different audit
table and asserts the rows land there and not in the control library; one points
AUDIT_LIBDS at a libref with no data folder and asserts the approval still
lands and the response says the diffs could not be recorded.
Two review findings on this branch.
- editors/restore.js spread every audit row for the load into Math.min(...).
MPE_AUDIT is append-only, so a long-lived estate would eventually exceed the
engine's argument limit and the RangeError would surface as a failed restore
of a perfectly restorable load. Folded into a reduce with an Infinity seed.
- auditors/postdata.js appended to MPE_AUDIT as a read-modify-write outside any
lock. postdata.sas locks the table it writes - MPE_REVIEW around its proc
append, MPE_SUBMIT around its update - so the audit append now takes the
MPE_AUDIT lock around a fresh read, the concat and the save, rather than
reading the table up front and saving a copy that may be missing another
approval's rows.
53 mock tests pass (node --test sas/mocks/test/).
`editors/restore` had no mock, and the reason it could not have had one is that
`postdata` wrote no audit rows: an approval applied the load and left nothing
to rewind. Both halves are here.
postdata (final approval): writes MPE_AUDIT the way %mp_storediffs does - one
row per column, keyed by the md5 of the business-key values joined by '|', with
move_type A/D/M, is_pk on the key columns, is_diff on a changed one, and the
old and new value on their own typed columns. An append carries every column
on the new side, a delete every column on the old side (which is what lets a
restore put the row back), and a modification the keys plus only the columns
that actually changed. A failure writing the audit is logged and does not undo
an approval that has already landed.
restore: mpe_checkrestore's two decidable checks (the table has an audit table;
the load has entries in it) with their messages, then mp_stripdiffs' window -
the load's timestamp and every change to that table from then on, because a
restore reverts this load *and all subsequent ones*. Per business key the
pre-window state is the earliest event: an append means the row is staged for
deletion (`_____DELETE__THIS__RECORD_____=YES`), a delete means it is staged
with the values the delete recorded, and a modification means each column takes
its earliest old value while untouched columns keep the current one. The
result goes through the same stageSubmission the editors use, so it arrives as
a new submission with the reason `Restoring <loadref>` and is approved like any
other - restore.sas never writes the base table either.
The access and group/RLS checks are not emulated, consistent with the rest of
the mocks: nothing enforces MPE_SECURITY there.
Tests: 52 pass across the seven mock suites, 7 of them new. They drive a real
approve and then read the staged rows back: append staged for deletion,
modification staged with the pre-load value, delete staged with the row's
values, and restoring the first of two loads reverting the second as well.
With the base postdata and no restore service, all 7 fail.
Four review findings.
- isCurrentOn counted TX_TO 0 as current. 0 is 01JAN1960 in SAS, so a real row
with that window has been superseded; only a missing, null or blank TX_TO is
open besides 31DEC9999. The fixtures are seeded with 31DEC9999 (253717919999)
through an OPEN constant instead, and the 0 clause is gone.
- The MPE_DATADICTIONARY filter compared DD_SOURCE against the libref without a
boundary, so DC_JSLIB2.TABLE was excluded along with DC_JSLIB.TABLE. It now
tests for the libref plus the dot, as the libds rewrite does.
- The schema-less export took its column list from Object.keys(rows[0]), so a
column that first appears in a later row was silently missing from the backup.
The list is the union of the keys across the exported rows, in first-seen
order.
- control-tables.test.js imported readTable and never used it.
Three review findings on this branch.
- dcMockUtils defined sasDatetimeNow twice: once beside isCurrentRow with the
epoch written inline and once in the mp_lockanytable section using
SAS_EPOCH_MS. The second is the one the file already had; the duplicate is
gone, so the epoch has one expression.
- isCurrentRow treated TX_TO=0 as current without saying why, and reached that
through Number(null) === 0. Both conventions are real: makedata writes
31DEC9999 for a row that was never superseded, and the fixtures in
control-tables.test.js use 0 for the same thing. The clause stays and the
comment now says which convention is which, plus that a missing, null or
blank TX_TO (a table with no version column) is current too. Reading the
column moved into isCurrentOn, because Array.filter passes the index as a
second argument and would have handed a `column` parameter the row number.
- admin/exportconfig used the schema only when it existed: with no columns in
the table file it fell back to Object.keys(rows[0]), which exported the
TX_/VER_/SELECTBOX_RK columns the spec drops and quoted every number. The
fallback now applies the same drop prefixes and infers numeric typing from
the rows, so a schema-less estate produces the same shape as a seeded one. A
test writes a table file with rows and no columns and asserts both.
All 46 mock tests pass (node --test sas/mocks/test/).
postinstall runs scripts/strip-clr-base64-fonts.mjs, which edits Clarity's CSS
inside node_modules and then deletes client/.angular/cache - because the
builder treats node_modules as immutable and would otherwise serve styles built
from the unstripped file. Measured on the Angular 20 builder: a marker
@font-face injected into clr-ui.min.css was absent from a warm build's CSS
(28s) and present in a cold one (88s).
The restore used to run before the install, so that clear deleted the entry
before anything could use it - run c732d75 restored 396MB and logged "cleared
/workspace/.../client/.angular/cache" minutes later. It now runs immediately
after "Install dependencies": the clear has already happened, and every entry
in the cache was produced from a stripped stylesheet, because every run strips
before it builds. The invariant the clear exists to enforce is preserved.
No restore-keys on this cache, deliberately: because the builder cannot see a
change inside node_modules, a prefix match could restore entries compiled
against a different Clarity after a lockfile bump. Exact key only, so a
dependency change misses and rebuilds.
Measured locally with the tree matching the lockfile (CI=true): cold with the
cache cleared 90s, warm with no source change 27s, warm with version.ts
regenerated as a new commit does 58s. So a build that is 99-109s in CI should
land near 58s; the rest needs the version out of the compiled graph, which is
an application change.
The strip script's comment described webpack 5; it now describes what was
measured, and notes that CI restores after it runs.
Three services the Angular app calls had no mock at all, so on the mock
estate the Excel-map screen, the system screen's refresh and the config
export either failed or showed nothing:
- editors/getxlmaps - the rules and info of one map. Reads MPE_XLMAP_RULES
and MPE_XLMAP_INFO for the current rows of that XLMAP_ID, orders by sheet
and range id, and coalesces the target to DC_JSLIB.MPE_XLMAP_DATA - the
same five columns, order and fallback getxlmaps.sas uses.
- admin/refreshcatalog - a libref is optional and must be well formed
(Invalid libref otherwise), and the reply is sasparams with the
completion message the screen displays. A mock estate has no catalogue
to rewrite, which the service says rather than pretending.
- admin/exportconfig - a member per non-empty table, named
<libref>.<TABLE>.csv, with the per-table treatment exportconfig.sas
applies: TX_* dropped, a version filter, the `where also` exclusions,
and the control-library name rewritten to the target libref (including
the DC_MACROS value and the LIBREF.TABLE of a map target). Zipped as
DCBACKUP.zip.
The version test and the control-table reader were local to startupservice;
they now live in dcMockUtils (isCurrentRow, sasDatetimeNow, and
makeControlTableLoader for controlTable), so the rule that decides a row is
current exists once.
The harness gains the session's stpsrv_header.txt and _SASJS_USERNAME, so
a service that streams a file can be exercised: exportconfig is asserted
both ways - the zip where the runtime has Buffer, the content listing where
it does not.
Tests: 45 pass across the six mock suites, 8 of them new. With the three
services removed, all 8 new cases fail.
Run c732d75 restored both entries on runner2 and the build stayed at 99s (it
was 99-109s cold); the tz suite's 62s -> 25s came from the disk cache inside
the job, not from the restore. postinstall rewrites
src/environments/version.ts with the commit hash and a timestamp on every run,
and the app shell imports it, so the application and test graphs are
invalidated regardless of what was restored.
The npm cache stays, and it is the one that pays: the client install fell from
80s to 25s in that run. angular.json keeps cli.cache.environment=all, which is
what makes the within-job reuse possible.
The gitleaks step scans the working tree with --no-git, and the Angular cache
holds webpack/babel JSON full of `{ token: ... }` fragments. Restoring it
before the scan put those files in the tree and the gate failed on run 3174
with a wall of generic-api-key findings in
client/.angular/cache/20.3.41/babel-webpack/*.json.
Both cache steps now run immediately after the scan. The npm download cache
only helps the client's npm ci, which is after that point anyway; the Angular
cache only helps the tests and the build, which are later still.
The comments cited run 3170 and a timing measured on one 4 vCPU box. Both go
stale and read as a guarantee about a particular runner. They now state the
invariant the workflow relies on: an unreachable cache server degrades restore
and save to a warning, so the job stays green. The timing figures stay in the
PR description, where they are evidence for this change rather than a promise
the workflow makes.
act_runner's cache handler scopes entries to the repository (each entry is
stamped with its repo and checked on read), not to a ref, so a pull request
into one branch could seed the entry another branch's build restores. The base
branch is now part of both keys and both restore-key prefixes.
A save against an unreachable cache server is already a warning, not a
failure: run 3170 landed on runner4.4gl.io and logged four
'::warning::Failed to save: reserveCache failed: Request timeout' lines with the
job completing green.
The first run of these cache steps showed the npm entry saving (234 MB) and the
Angular one reporting 'Path Validation Error: Path(s) specified in the action
for caching do(es) not exist'. Angular keeps its persistent cache in memory
when CI is set, so no .angular/cache directory existed to upload; the disk
cache is only used outside CI by default.
client/angular.json now sets cli.cache.environment to "all", which is the
supported switch for that. Verified locally with CI=true: after removing
.angular/cache, a build recreates it (396 MB).
The runner's cache server is reachable from job containers on two of the three
boxes now, so these are the first cache entries in this repo that can actually
come back. Probe evidence, four dispatches of a throwaway two-job workflow:
- save on runner.4gl.io, restore on runner2.4gl.io: miss (each act_runner runs
its own cache server, so entries belong to the box that saved them)
- save and restore on runner2.4gl.io: cache-hit=true, the stamp came back
- save on runner4.4gl.io: "Failed to save: reserveCache failed: Request
timeout" - that box still runs the old runner config
Two caches, both keyed on client/package-lock.json so a dependency change
invalidates them, with prefix restore-keys so the previous entry still lands:
- ~/.npm, for the client's npm ci (81s cold, measured on run 3132)
- client/.angular/cache, for the production build (109s cold, 118s -> 33s with
a warm cache on a 4 vCPU box) and the two Karma runs, which share it
Per run the arithmetic while runner4 is still on the old config is about
1/3 x (+80s: two restores and two saves timing out at 20s each) plus
2/3 x (-100s), so it is a net improvement already and a clear one once the
third box is deployed. On an old box both steps only warn and the job stays
green.
actions/cache does not save a second entry for an existing key, so the Angular
cache refreshes when the lockfile changes; the v1- prefix is there to force a
reset by hand if the win decays.
Review findings on #471.
The platform rule now checks the documented tokens exactly
(['WIN', 'DNTHOST']) instead of their first letters, so an unlisted token
beginning with W or D no longer silently gets the drive path back.
brDirList raised TRUNCATED whenever it saw rows.length >= maxobs, which
included the check made after the last permitted row was added: a listing
of exactly maxobs folders with a file sorting after them reported itself
truncated although nothing was refused. The flag is now set only where a
row is actually refused, and the early exit remains the walk bound.
Two new cases pin the shape the picker depends on: a listing of exactly
MAXOBS entries is TRUNCATED='N', and a DEPTH=2 browse numbers children at
1 and grandchildren at 2 with no LEVEL 0 row. Both, plus the existing
parent-level and write-probe cases, fail against release/7.22's service.
Tests: 37 pass across the five mock suites, client spec 9/9.
Two review findings on the first two commits.
The two jobs no longer decide separately. Build-and-ng-test now publishes
`client` as a job output and Build-and-test-development consumes it instead of
re-diffing the base branch: a base commit landing between the two decisions
could make the Cypress job expect an artifact the build job had already
skipped, and the failure would read as a missing artifact. An empty output
(the build job could not decide) still runs everything.
The path list is now a PATHS variable the script can be asked about directly
(`--match` with a file list on stdin), and tests/cypress-changes.test.sh pins
one case per alternative plus the negatives (a sas/ service, docs, the
Lighthouse workflow, a "clientx/" directory, the sas lockfile). The build job
runs the test, so dropping an alternative is red rather than a suite that
silently stops covering a class of change. Checked the test fails when the
pattern is mutated.
The Cypress job installs the CLI from the version sas/package.json pins
(`CLI_VERSION=$(node -p "...dependencies['@sasjs/cli']")` in "Deploy mocked
services"), so bumping that pin changes what the suite runs without touching
anything under sas/mocks/. A pull request that only bumped it would have been
skipped by both jobs.
The sas lockfile stays out of the set: the job installs that CLI globally by
version string, so the lockfile cannot change what it resolves.
Three defects behind staging's "Invalid parent" and the picker that could
not list a subfolder. Found by running the real mpe_dirlist on
sas.4gl.io, not by reading it.
1. The browser host answers SYSSCPL "browser". The client looked for the
first letter of a unix token (L/H/A/S) and fell through to the Windows
default for anything else, so the deploy screen pre-filled
C:\DataController on a POSIX runtime, which the server then refused as
an invalid parent. Windows is now recognised by its own tokens (WIN,
DNTHOST); anything else, including a value the list has never seen,
gets the POSIX path.
2. mpe_dirlist emitted a non-blank parent's children at LEVEL 0, because
mp_dirlist numbers a folder's contents with the folder's own level.
The picker keeps LEVEL>0 for a non-root listing, so every row was
discarded and a subfolder could never be browsed. The children are
shifted to start at 1; a blank parent still lists the platform roots
at 0. Verified on the estate: /tmp at DEPTH=1 returns levels [1],
a blank parent returns [0, 1] (the root, then its children).
3. The mock's write probe generated one name and wrote it, truncating a
file that was already called that; mpe_writecheck tries up to ten
names with fileexist()=0 first. The mock now does the same and
reports 'No free name for the write probe' when all ten are taken.
The mock's DIRLIST now matches the estate's shape too: children at
LEVEL 1, roots at 0 followed by their children, and the cap stops the
walk at MAXOBS rather than slicing afterwards. Tests: 35 pass across
the five mock suites (makedata-actions 15, two of them new: the probe
collision and a folder that cannot be written to), sasjs lint clean,
client spec 8/8 with the new platform cases and 2 failing without the
fix.
The build job spent 402s on every pull request (run 3132, release/7.22).
Two of its steps were work nobody consumed:
- `npx cypress install` (11s, measured 17:07:05-17:07:16) downloaded and
unzipped the Cypress binary into a job that only runs Karma, eslint, the
licence checker, the mock service tests and `ng build`. Cypress runs in
Build-and-test-development, which installs its own binary.
- The production build (109s) and the client-dist upload exist only to feed
the Cypress job, which already skips pull requests that touch nothing it
exercises. On those pull requests the artifact was uploaded and never
downloaded.
The changed-path decision now lives in .gitea/scripts/cypress-changes.sh and
both jobs call it, so the build job's gate cannot drift from the Cypress job's
skip. The script keeps the existing fail-open rule: anything it cannot decide
(no base ref, failed fetch, failed diff) writes no output, and every caller
gates on `!= 'false'`, so the work still runs.
Three findings from the review of the release head.
A quote, a comma and both parentheses were refused as parent/path characters,
which are legal on both platforms - "C:\Program Files (x86)", "O'Brien",
"/data/sas, archive" - so the deploy screen refused to browse or write-check a
folder it could have been deployed into. writecheck never turns the value into
macro text (it is read back with symget and handed to filename()), and dirlist
now passes it to mp_dirlist through %superq, so no character in it is read as
macro text; both keep the length, control character and ISPATH checks and drop
the character refusals. brCheckPath in the mock agrees, so the two estates
answer alike, with a test for a folder named "Program Files (x86), O'Brien".
brDirList walked every level and sliced to MAXOBS afterwards, so browsing /usr
or /proc at depth 3 still stat'd everything below it - the cost the cap exists to
avoid. It stops adding rows once the cap is passed, as mpe_dirlist does.
rejection.js and postdata.js loaded MPE_REVIEW before taking the lock and
released it straight after the write, so two requests could both read before
either locked and the second save dropped the first row. The lock now covers
the read-modify-write it is there for, released in a finally.
7.1.7 closes the outstatus step with run, stops truncating the flow table's
state, and strips the response envelope before the JSON libname; 7.1.8 lets
mp_replace delete, which is how the envelope is stripped. Without them a Viya
run cannot read an aborted child's result: ten of Data Controller's SAS suites
report no result at all.
The payloads are the SAS 9 recordings, separators included - the front end splits
metadata URIs on the backslash, so a value that arrives without one exercises the
wrong branch. The envelope is not SAS 9: _PROGRAM, SYSUSERID, SYSVLONG and
SYSDATE/SYSTIME come from whatever runtime serves the request, so this variant
must not be read for SAS 9 envelope values.
The guard is the tree's one shared rule, but each of the three services that
needs it repeated the path derivation and then eval'd a file read out of the
SASjs Drive at run time - which is what the file's own header said this tree does
not need. weboutGuard.js now exports the function and the services require it by
absolute path: one copy of the function, no eval, and a load failure is a
require error naming the file rather than a silent no-op.
Being a module also means the guard has to carry its own dependencies: as an
eval'd snippet `fs` came from the program's scope, and the first run as a
required module failed with "fs is not defined". It now requires fs itself.
dcMockUtils.requireMemberMetadata remains the source of truth for the rule and
both files say so, since a site may run both trees and an operator reads
whichever one threw.
Two review findings on the approval hooks.
mpeRunhook read only the lower-case column name, while a row written
through the editor carries the schema's upper-case names (PRE_APPROVE_HOOK)
and the buskey/loadtype reads beside it already try both. A hook
registered that way resolved to nothing, so the approval ran without it
and said nothing.
postdata ran the pre-approve hook before the duplicate-approval guard.
postdata.sas checks prev_upload_check first and returns ALREADY_UPDATED
without reaching the hook, so a second approval from the same user was
aborting with the hook's message instead. The guard is now computed
before the hook and the hook is skipped when it fires. The hook still
runs for SHOW_DIFFS, which is what the SAS service does too - the macro
call sits outside both action branches.
Both cases are pinned by a test, and each fails when its fix is removed.
postdata.sas runs PRE_APPROVE_HOOK before the diff and before any write,
and POST_APPROVE_HOOK after the load is applied. The mock ran neither,
so an approve hook registered on MPE_TABLES did nothing on the
browser-hosted estate - while the post-edit hooks (which live in
stagedata/loadfile, mirroring mpe_loader) did run.
mpe_runhook(tableReg, hookvar) in dcMockUtils picks the hook out of the
MPE_TABLES row by column name and returns its source; the service
eval()s it in its own scope, which is the JS equivalent of the macro's
%include sharing the caller's macro variables. The three call sites that
already resolved a hook by hand (getdata pre_edit, stagedata/loadfile
post_edit) now go through it, so the next hook column costs one line.
postdata runs pre-approve before the staged data is diffed and aborts on
failure (nothing is written), and post-approve after the base table and
control tables are updated - so its failure is reported but cannot undo
the load, which is what postdata.sas does with its SYSCC= response.
A hook refuses by throwing or by setting hookErrmsg, the convention the
post-edit hooks already use.
sas/mocks/test/hooks.test.js covers the ordering: pre-approve refuses
with the base table untouched and the submission still SUBMITTED;
post-approve aborts with the row already in the base table and
MPE_SUBMIT APPROVED; and one hook proves it ran by writing a file, not
by the service reporting success. 30 tests pass; 4 of the 5 hook cases
fail against the base service.
Three more places where the mock answered differently from the SAS service.
mp_abort (dcMockUtils): every mock refusal is now a sasjsAbort table
(MSG + MAC) written where the macro writes it, so the client shows the
abort modal and rejects the promise. A thrown error had become an HTTP
500 "Job execution failed" - the browser host discards the webout when a
program throws - which is unreadable to the user and hides the message in
a stack. mpAbort also stops the program, so nothing is written after a
refusal; each service that can abort catches that stop at its top level.
stagedata, loadfile, getauditfile and makedata now go through it, and the
hand-rolled sasjsAbort blocks are gone.
mp_lockanytable (dcMockUtils): a lock is a row in MPE_LOCKANYTABLE marked
LOCKED carrying the holder's job id, user and reference, exactly as the
macro records it, and a lock held by another session refuses the write
with the macro's message. postdata and rejection now lock MPE_REVIEW and
MPE_SUBMIT around each write, which is where postdata.sas and
rejection.sas take them. The macro's retry loop is collapsed to a single
attempt: the runtime is single threaded, so nothing can release a lock
while the request waits.
startupservice: saslibs is the distinct LIBREF of the permitted datasets,
which is what "create table saslibs as select distinct libref from
&syslast" produces - it was a four-column row describing the control
library, with fields the client does not read.
Tests: abort.test.js and locks.test.js join the harness suites; 25 pass,
and 9 of the 13 relevant cases fail against the base services.
Every screen reads public/startupservice, so a divergence there is a
divergence in the whole app. Three of its answers were fixed values:
xlmaps a hardcoded list of three maps, so editing MPE_XLMAP_RULES
or MPE_XLMAP_INFO changed nothing on the site
sasdatasets included the map target dataset, which the SAS service
deletes - the demo offered MPE_XLMAP_DATA as an editable
table while SAS refuses it
timezone TIMEZONE and SYSTIMEZONEIDENT pinned to Europe/London and
the offset to +0100, so a GMT session reported an hour of
DST it did not have
They now come from the same sources the SAS service reads: the current
MPE_XLMAP_RULES rows joined to MPE_XLMAP_INFO (coalescec to
<mpelib>.MPE_XLMAP_DATA), the target datasets removed from sasdatasets,
and the session zone and current offset taken from the runtime, with
TIMEZONE left blank as an unset option is.
The shared harness (sas/mocks/test/harness.js) makes the service tests
one-liners: it builds a temporary estate with every service on its drive
and runs a service file the way the runtime does, webout() included.
sas/mocks/test/startupservice.test.js covers the joins, the target
removal, closed rules and the offset.
Staging now opens on the deploy screen, and its folder picker asks
makedata for a listing. The mock ignored ACTION entirely: every request
seeded the control library, deleted makedata from the drive and replied
with the HTML success page. So the first BROWSE deployed the site and
closed the wizard the visitor was standing on, behind an "invalid Json
string" error from the adapter, and VALIDATE and SUBMIT then answered
500 "Program not found in drive".
The mock now dispatches on ACTION the way the SAS service does:
DIRLIST the folders below PARENT, one level at a time, with the
dirinfo/dirlist shape mpe_dirlist emits
WRITECHECK the checkwrite row mpe_writecheck emits, probed by creating
and removing a file
MAKEDATA the existing seeding path, unchanged
ACTION is required and refused when absent or unknown, and the two browse
actions are refused once the settings file exists, mirroring browse_gate.
Both input routes are accepted - the screen's fromjs table and a CLI
seed's SASControlTable - because %webout(FETCH) does not care which one
arrived; deploy/makedata.json now carries ACTION=MAKEDATA as the real
service requires.
sas/mocks/test/makedata-actions.test.js runs the service file itself
against a temporary estate and is wired into the build workflow.
Three review follow-ups on the guard the sas9 tree loads.
- A body that cannot be read is now its own error, naming the path and the
filesystem message. Reporting it as missing metadata sent the reader to a
@sasjs/utils reinstall that cannot fix a path.
- One statement of the dependency fact: showmeta arrived in @sasjs/utils
4.6.0 and @sasjs/server declares that floor from 3.12.3 (3.12.0 and
3.12.2 declare ^4.4.0, checked with npm view), so the message names the
server version to install instead of "reinstall the server".
- CONTRIBUTING lists _program and weboutPath among the predeclared globals,
which the guarded services use to locate the helper.
The sasjs tree's copy in dcMockUtils.js had the same two defects, so both
now say one thing rather than one tree being clearer than the other.
The sasjs tree has emitted its responses through the runtime's webout()
since #461; the SAS 9 tree under sas/mocks/sas9/ still assigned every
captured payload to _webout, so its envelope was frozen at the 2022
recording and its values carried the separators a template literal eats.
All 35 services now call webout('OPEN') / webout('OBJ'|'ARR', name, rows)
/ webout('CLOSE'), with the three services that carry column metadata
passing { showmeta } and loading the guard from the new weboutGuard.js.
The payloads come from the captures read as JS produces them, with the
backslashes a template literal silently drops restored: 121 separator
values across 14 services - OMS URIs (OMSOBJ:<Class>\<repo>.<id>),
BUILTIN\Administrators, SAS\sasdemo and D:\... / C:\... paths.
Staging is the preview of the next release, and what it should preview first is
the deployment itself. DC routes to /deploy while makedata is still on the drive
(at <appLoc>/services/admin) and stops doing so once the wizard has run it and it
has removed itself - so a drive that carries the services and the licence but no
control library opens on the deploy screen, and the visitor deploys the control
library from the wizard into their own IndexedDB drive.
The seeding step is now gated on a tag: only the demo (which a visitor should be
able to use straight away) runs makedata before the drive is captured. The
staging path gets a guard of its own, because a seed that leaked into it would
only be visible as a site that opens on the home screen: makedata must be on the
drive and no mpe_tables.json may be there.
CONTRIBUTING updated: the two sites now ship different drives from the same
build.
The captures wrote values like OMSOBJ:SASLibrary\A59LNVZG.B500000K inside a
template literal, where \A is an escape, so the separator never reached the
client: it served OMSOBJ:SASLibraryA59LNVZG.B500000K. The same happened to
D:\MyDemo\Sankey and BUILTIN\Administrators. Converting the service to webout()
faithfully preserved the corrupted value - and the metadetails ASSOCURI, the one
field fixed by hand at conversion time, was left as the only member that kept its
separator, which is the inconsistency the review named.
Seventeen values corrected across five services:
lineage/fetchtablelineage info[0].LIBURI 1
lineage/getmetacols metacols[].COLURI 11
lineage/getmetatables PATH_SCHEMA, TABLEURI 2
public/refreshlibinfo libinfo[0].OWNERS 1
usernav/usermembersbyrole sasgroups[].URIMEM,
sasmembers[].URIMEM 2
Verified by deploying both trees on one server and comparing the parsed
responses: those seventeen fields are the only differences, and every other
member of all thirteen converted services is unchanged (metadetails' two
ASSOCURI, already carrying its separator, still differ from the capture for the
same reason).
Anything that splits a URI or a Windows path on the separator, or displays
OWNERS or PATH_SCHEMA, now sees the value the SAS metadata server returns.
3.14.0 ships @sasjs/lint 5.0.0, so the studio reports OS commands, and it is the
release the estate runs. It still declares @sasjs/utils ^4.7.0, so the envelope
formats from utils 4.8 (SYSCC as text, DATE7, TIME5, E8601DT26.6) are not yet
guaranteed by the floor - a fresh install resolves 4.8.0, a cached tree need not.
The comment says so, and the pin moves again when the floor lands.
Verified on a local 3.14.0 install (clean, resolving utils 4.8.0): the mock
estate's startupservice returns SYSDATE 04NOV26, SYSTIME 12:44, SYSCC "0" (text),
END_DTTM 2026-11-04T12:44:14.954000 (no Z). With that estate seeded, and against
a fresh app location, licensing.cy.ts passes 10/10 and metanav.cy.ts 1/1 on
Chromium.
The other nine services that answer with a captured response also assigned it to
_webout, envelope and all. They now emit their members through the runtime's
webout():
lineage/fetchtablelineage info, finalfinal, flatdata, idlookup
lineage/getmetacols metacols 11 rows
lineage/getmetatables metatables 1 row
usernav/usermembers users 2 rows
usernav/usermembersbygroup sasmembers 2 rows
usernav/usergroupsbymember groups 1 row
usernav/usermembersbyrole sasgroups 1 row, sasmembers 1 row
usernav/userroles roles 46 rows
public/refreshlibinfo libinfo 1 row
Each payload was taken from the response the old service serves, then both
trees were deployed on one server (/Public/app/dc and /Public/app/dcnew) and
compared as parsed JSON with the envelope keys excluded: all nine are equal row
for row, including the escaped values (the OMS URIs keep their backslash, and
refreshlibinfo keeps the quoted paths).
The envelope is the runtime's now, so SYSCC is numeric 0 rather than the string
"0", SYSDATE/SYSTIME are today's, SYSSITE is reported (these captures carried a
lower-cased syssite, which the client never read), and the SAS-only
_METAUSER/_METAPERSON are gone.
lineage/fetchlineage is deliberately left alone: its captured body is not valid
JSON today (bare quotes in the graphviz labels, raw newlines inside the STRING
values, and a lost backslash), so there is no payload to preserve and the
intended text needs a decision rather than a rewrite. Tracked separately.
The four metanav services assigned a captured 2022 response to _webout, envelope
and all, instead of using the runtime's webout() like every other service. The
members are unchanged:
metarepos outrepos 2 rows
metaobjects objects 1 row
metatypes types 71 rows
metadetails attributes 12 rows
associations 2 rows
Both trees were deployed on one server, at /Public/app/dc and /Public/app/dcnew,
and the responses compared as parsed JSON with the envelope keys excluded:
outrepos, objects and types are equal row for row.
metadetails changes in one place. The capture wrote the OMS URI as
OMSOBJ:Property\A59..., but \A is an escape in a JavaScript template literal,
so the backslash was consumed and the client received OMSOBJ:PropertyA59... -
a URI that cannot resolve. The member now carries the backslash the SAS
metadata server returns.
The envelope comes from the runtime now, which also drops three artefacts of
the recording: SYSCC "4" on metarepos (an error code from the captured day,
which the client reads as a SAS error), SYSDATE/SYSTIME frozen at 26SEP22, and
the SAS-only _METAUSER/_METAPERSON.
The mock services call webout(), as the server's runtime does. The browser mock
runtime that @sasjs/cli ships is what provides it there, and the 6.5.0 this repo
was pinned to does not: its runtime-core.js has no reference to the webout
prelude, and the build writes no webout-prelude.json for it to load. Every
service that calls webout() therefore throws "ReferenceError: webout is not
defined" in a published mock site while the build itself reports success.
6.9.0 is the first release that ships the prelude; this pins 6.10.0, the current
release. The transitive @sasjs/adapter and @sasjs/utils entries move with it.
The guard that keeps a runtime without webout()'s options argument from serving a
thinner response tested webout.length >= 4. Function.length stops counting at the
first default or rest parameter, so a runtime that does carry the option but
declares it `options = {}` or gathers the tail with `...args` reports three, and
every service that calls the guard refuses to start on a working runtime.
It now asserts the outcome instead: the member the option produces is looked for
in the response body the program is writing. `$sasdata`, `$viewdata`, `$vals` are
the members the client reads for column formats and labels, and their presence is
the thing that actually has to hold.
Where it runs matters too, and the first attempt got that wrong: called after
webout('CLOSE') the throw leaves a complete body behind, and the server serves it
- the caller sees a normal response with SYSCC 0 and no metadata, which is exactly
the silent failure this exists to prevent. It is called before the envelope is
closed, so an unhonoured option fails the run.
@sasjs/utils 4.7 makes webout() emit SYSSITE from a configured site id,
defaulting to 123, where %webout takes &syssite. Until 3.13.0 the prelude
wrote an empty string, and dc's client routes an expired licence key and a
wrong-organisation key only when startup_site_id is truthy - so on the mock
estate both pages were unreachable and licensing.cy.ts failed three cases.
Three values reached `run:` blocks through `${{ }}`, so the runner pasted them
into the script before the shell ever saw it - which makes the value part of the
code rather than an argument to it.
- `BASE="${{ github.base_ref }}"` (build.yaml, lighthouse.yaml) - a ref name in
the script.
- `echo ${{ secrets.SHEET_PWD }} | gpg --passphrase-fd 0` (all four workflows) -
a passphrase in the script. A quote in it ends the quoting, and a command
substitution in it runs.
- two `replace-in-files --replacement=...${{ secrets.CYPRESS_* }}...` lines
(build.yaml, release.yaml) - the same, inside a replacement string.
Each now arrives as a step `env:` value and is expanded as `"$VAR"`, so the shell
only ever sees data. Checked the replacement lines side by side with the old form:
identical for an ordinary value, and the new form keeps a value containing a space
intact where the old one word-splits it, and survives a value containing a single
quote where the old one is a shell syntax error.
The passphrase pipe also moves from `echo` to `printf '%s\n'`, so the byte the
decryption sees does not depend on how the shell's `echo` treats backslashes.
Review finding: when every body is refused by carriesClause, or the response
carries no FILTER_HASH, capture.hash stays empty - and storedClauseFor('')
then spends its 60 retries before reporting "no clause for <var> under",
which names neither the refused body nor the missing hash.
The hash is now asserted non-empty before the read, with the case's own
message, so a broken delivery fails where it happened.
The call count stays where it was - after the data load - because moving it
ahead of the read would check it before the app has issued the second POST
that the count exists to catch.
The duplicate check waited a fixed 250ms after reading the stored row, which is
wrong in both directions: on a slow runner the window can elapse before the first
response callback has incremented the counter, and a second POST landing after it
passes with two deliveries.
The app's sequence for an apply is validatefilter, then the data load for the
filtered route, so the load is the completion signal: the count is asserted after
it, and nothing depends on how fast the machine is. A delivery issued later than
that is a second apply, not a duplicate of this one.
Also corrects the ordering paragraph in cypress-specs.sh: it still said specs are
appended, in a file that now places va-ddc-host ahead of metanav. va-ddc-host does
not depend on the estate being unlicensed either - it passed 3/3 immediately after
metanav planted a key in the full CI run - so the bullet says what is true and
metanav stays the only spec with a fixed place.
Review finding: the matcher only accepted the variable name at the start of the
body or after a comma or newline, so a delivery wrapped any other way was not
counted - and the case then failed at the call-count assertion, reading as
"the app never called validatefilter". Reproduced against the previous
implementation: a urlencoded form value, a form value that starts with the
name, a JSON object keyed on the column and a JSON string value holding the
row all matched nothing.
The body is now expanded into the candidate strings it can carry - the raw
text, the percent-decoded text, each form field's value, and every string leaf
of a JSON body - and the name has to be a whole CSV field of one of them.
Field equality rather than delimiters means a quoted literal, a longer name,
an SQL fragment or another variable's name is still refused, and there is no
regular expression over user text to get wrong.
Also restores the ordering note in cypress-specs.sh: va-ddc-host asserts
nothing about licensing, so it runs before metanav, the one spec that plants a
licence key.
The pin moved to 3.12.3 with the webout() change but the comment above it still
named 3.12.2 as the current release - true when it was written, wrong under the
pin it now sits on. The comment also only carried the /ddc-host reason, which
does not explain this bump.
3.12.3 is the first server release declaring @sasjs/utils ^4.6.0, and that is
what gives webout() the options argument getdata, viewdata and getcolvals pass as
{ showmeta }. Say so where the constant is, so the next person to move it knows
the metadata members depend on it.
The services in this branch pass { showmeta } to webout(), which is
@sasjs/utils 4.6 and up. sasjs/server 3.12.3 is the first release whose
package.json asks for ^4.6.0 (a clean install resolves 4.6.1), so the pin
moves from 3.12.2 to 3.12.3 in every workflow that starts the mock estate.
Verified against the published package rather than the release notes:
`npm view @sasjs/server@3.12.3 dependencies.@sasjs/utils` answers ^4.6.0, and
`npm install @sasjs/server@3.12.3` in an empty directory installs
@sasjs/utils 4.6.1, whose webout prelude carries showmeta.
Review of the release diff: `npm install -g @sasjs/cli` and
`replace-in-files-cli` were unpinned, so CI ran whatever release was current
and a bad one would reach the pipeline unreviewed - next to a SASjs server
that is pinned to an exact version.
The CLI now comes from the version sas/package.json pins, which is the shape
docs.yaml and release-assets.yaml already use, so there is one source for it.
replace-in-files-cli, pm2 and wait-on are pinned to the versions in use
(4.0.0, 7.1.2, 9.5.1).
Both node expressions are checked from the directories they run in:
`require('./sas/package.json')` at the root and `require('../../package.json')`
in sas/mocks/sasjs both answer 6.5.0.
The three services that pass { showmeta } - getdata, viewdata, getcolvals - rely
on webout()'s fourth argument, which arrived in @sasjs/utils 4.6.0. @sasjs/server
3.12 declares ^4.4.0, so an install can resolve 4.5.x, and there the argument is
dropped silently: the response still looks complete but $sasdata, $viewdata and
$vals are absent, so the client's format and label paths stop being exercised
while the mock estate reports success.
requireWeboutOptions() checks that the generated webout() takes an options
argument (arity, because the option names live outside the callable function) and
throws naming the member, the cause and both fixes. A 4.6 runtime is unchanged.
The matcher tested the variable as a substring of the whole request body, so a
RAW_VALUE, an SQL fragment or another variable whose name contains it was counted
as another delivery of this clause and failed the count. It now requires the name
between its CSV delimiters, which is the filterquery VARIABLE_NM field; a quoted
literal cannot match, because its delimiters are quotes.
cy.wait('@validatefilter') used the 5s default, where the polling loop it replaced
tolerated 15s, and the message only arrives once the framed page has rendered.
It now waits for longerCommandTimeout, as the rest of this spec does.
The capture assertion also carries the tail of the host page's log in its message:
the failure reads "expected -1 to be above -1" otherwise, and the log is what says
whether the app never exposed its buffer or the frame never loaded.
Two review findings:
- The load handler re-posted pendingPayload on every load event of the frame,
so an in-app navigation or a self-reload posted a filter that belonged to
the mount which had already delivered it, and logged it as that mount's.
The handler now takes the payload (clearing pendingPayload and
pendingTiming) before posting, so a later load has nothing to send.
- The README illustrated report.json with one parameter while the committed
file carries three, and the prose below reads as though one parameter were
the shape. The example block is gone: the file is the example, and the keys
are documented as a table with the parameter rules kept underneath.
The last eleven services still built the response envelope through
dcMockUtils' webOutOpen/webOutObj/webOutClose. They now use the webout() the
SASjs Server emits into every JS program, so dcMockUtils no longer emulates
%webout at all - the helper existed only for these callers and is deleted
with them.
That needed the options object added to webout() in @sasjs/utils 4.6
(showmeta, maxobs), which is what a clean install of @sasjs/server@3.12.2
resolves to (^4.4.0 -> 4.6.1, checked). A server whose install still holds
4.4/4.5 refuses the option by name rather than ignoring it, so the failure is
loud rather than a response that is quietly missing its column metadata.
getdata, viewdata and getcolvals send their metadata as
`webout('OBJ', name, rows, { showmeta })`, and every other call keeps its
member name and rows unchanged.
One call was positional all along: startupservice's xlmaps is an array of
arrays, which the old helper serialised as-is and webout() refuses for OBJ.
It is now webout('ARR', 'xlmaps', xlmaps) - caught by the comparison run
below, not by the rewrite.
Two gaps in the guard that decides which validatefilter response belongs to the
case.
A body Cypress parses into an object stringified to "[object Object]", so no
call matched and the case failed on the count with nothing naming the body shape.
The body is now serialised the way the response already was - a string is used as
it is, anything else is JSON.stringify'd.
The count was asserted immediately after the first response, which is before a
second POST from the same apply could reach its own callback, so a duplicate read
as one call. The count is now asserted after the stored row has been read back,
with a settle that covers a second POST still in flight when the read returned.
Two review findings on host.js:
- postToFrame() used '*' as targetOrigin, so the filter clause went to
whatever document occupied the frame. The frame is the app on report.server,
so that origin is the only one that should receive it; the origin is now
resolved once from report.server (new URL(...).origin) and a report.server
that is not a URL logs a visible error instead of widening delivery. The
page also says so when it is not on the frame's origin, because the capture
probe cannot read across origins and that would otherwise look like a
harness failure.
- startEarlyReplay() accepted any __vaLastMessage once it had posted, and on
a remount to the same URL the old document is still on screen carrying the
buffer the previous mount captured. A capture now only counts once the
document has been replaced (mountDocument), both in the interval and in the
load-event check. Keying to the document rather than waiting for a load
event keeps the first post where the early window is - the interval starts
exactly as before.
Every push to a pull request queued another full run - lint, licence check, the
mocked estate and the twenty-six minute suite - and nothing cancelled the run for
the head it replaced. With two runners shared by every pull request in the repo,
those stale runs are picked oldest-first and delay the head that can actually be
merged: runs 3002, 3003 and 3005 were queued for heads that had already been
superseded while the current one waited behind them.
The group is the pull request, so a newer push cancels the run for the older
head. Pushes that are not pull requests are not affected - the release workflow
answers to its own group.
Three review findings on the harness:
- The usage line advertised `--pass`, which puts the secret in argv - readable
from `ps` by every account on the host, and kept in the shell history. The
flag is gone: SASJS_PASS stays the documented route, and `--pass-stdin`
reads one line from stdin for scripts that cannot use the environment.
- deploy() reported success on the deploy route's 2xx alone. That means the
tree was accepted, not that the app is served: a truncated `type: 'file'`
member still serves a 200 page, and a stale asset from a previous deploy is
a 200 by definition - both of which the README already lists as the sharp
edges of that route. The page and all three assets are now fetched back and
compared with the bytes that were sent, failing with the size and hash of
both copies; a 401 is reported as deployed-but-gated, since on a server-mode
estate that is the gate working. `--verify-only` runs just that half.
- Two harnesses stand in for a VA report (this page and
client/cypress/e2e/va-ddc-host.cy.ts) and nothing said which does what. The
README now names both roles and points at the contract of record - the
message shape and __vaLastMessage in va-messaging.service.ts, and the
pre-bootstrap capture rule in va-early.js - including why the probe differs
between them (Cypress proxies the framed document).
The intercept kept one hash from any validatefilter response in the case, so a
second call - another table, a revalidate on load - would have replaced it and
the rows read back would belong to a different clause. The response is now only
taken when the request body carries this case's variable, which is what tells one
clause from another, and a case that sees more than one call for that variable
fails on the count rather than silently keeping the last.
A hash that cannot be read is captured and rethrown in the command chain: thrown
inside the route handler, Cypress reports it as an uncaught error and the case
fails later as a bare "no clause under" timeout instead of naming the response.
The spec located its evidence as "the last stored row for this variable", which
cannot tell this run's row from one an earlier case or run left behind - so it
deleted the DC_JSLIB filter tables before each case instead, and that bought a
scheduling contract: it had to run after every other spec on the shared mock
estate, and it left the estate falsified for anything that ran after it.
The mock validatefilter already returns the FILTER_HASH it computed for the
clause, so the rows are now selected by the hash this case's own response names.
Nothing is cleared, and no spec has to run before or after it. A store that
already holds this clause holds the same rows; a clause that changed hashes
differently, so a regression cannot be satisfied by a stale row.
va-ddc-host moves back in front of metanav, which is the one spec with a real
ordering constraint (its licence key must not change another spec).
The SASjs Server emits a webout() block into every JS program it runs
(@sasjs/utils >= 4.4, server >= 3.12.0), mirroring %ms_webout in
@sasjs/core: OPEN, ARR, OBJ, CLOSE. Ten of the mock services are two-year
functional implementations of their SAS counterparts, and they were
hand-building that same envelope through dcMockUtils' webOutOpen/Obj/Close.
Move those ten onto webout(), so the response envelope is built by the
runtime that owns it - real SYSUSERID, SYSPROCESSNAME, host name, a numeric
SYSCC and a content-type header - instead of constants in a mock. The
services that send showmeta metadata stay on the dcMockUtils helpers,
which webout() has no argument for yet.
The old 3.6.2 server pin in release, lighthouse and sites predates the
block, so those three move to 3.12.2 with it.
The interval tested its deadline only after the contentWindow guard, so a frame
that never became postable (detached, mid-navigation, or a load that never
fires) returned on every tick and the timer ran for the life of the page.
The capture check and the deadline now both run before the guard, so the replay
always stops. Capture keeps precedence, so a message caught on the last tick is
still reported as a capture, and it only counts once something has been posted
in this replay - a buffer left by the previous load of the same frame cannot be
read as one that belongs to this mount.
attempts keeps counting posts rather than ticks, because that is what the lines
quote ("after N post(s)"); a tick skipped for want of a frame sent nothing. The
zero case is now stated separately, since "0 post(s)" otherwise reads as a
broken probe rather than a frame that never appeared.
3.12.2 is the current release and carries the packaged-executable fix for
/ddc-host (the page this job's va-ddc-host spec drives), so the version the
tests run against is the version that ships. 3.12.0 remains the floor for
`timing=early`.
Closes#449.
- SASJS_SERVER_VERSION 3.6.2 -> 3.12.1 in build.yaml. 3.12.0 is the floor:
/ddc-host with `timing=early` (the replay that lands before the app
bootstraps) arrived there, and it is absent from 3.11.3, which has no
/ddc-host at all.
- DDC_HOST_ENABLED=true in the job's .env. The server 404s /ddc-host unless it
is set - it is a testing tool and does not announce itself - so enabling the
spec without this would have failed on the first case.
- va-ddc-host moves from EXCLUDED to the END of SPECS in cypress-specs.sh: it
clears the DC_JSLIB filter tables before each case, so it must run last on the
shared mock estate. The ordering contract at the top of that file now records
that, and metanav's licence key, which no longer has to avoid being last
itself.
Review findings on the harness.
- timing=load posted nothing: mount() only started a replay for early/both and
the load handler only logged. Both and load now post on the frame's load as
well, which is the metadata path and the second copy VA sends, so the mode the
controls offer is a mode that works.
- a re-apply with the same prompts did not reload the frame: setting src to the
value it already has is a no-op in the browser, so the new load post (and the
refresh path generally) delivered nothing. The frame is reloaded explicitly
when the URL is unchanged.
- dataType was declared and never applied. A prompt configured as number now
arrives as 42 rather than "42", which is what DC's clause builder branches on;
a non-numeric value is left alone rather than becoming NaN.
- the prompt select is found by walking the selects instead of building a
[data-name="..."] selector, so a prompt name with a quote or bracket in it
cannot throw inside apply() and leave the page with no frame.
- the replay's give-up message now says when the app does not expose
window.__vaLastMessage, so a changed probe is not reported as a delivery
failure.
- deploy.mjs throws when the Node runtime cannot read Set-Cookie rather than
silently dropping every cookie and failing later as an unauthenticated 401.
report.json carries a numeric prompt so the numeric path is exercised by
default rather than only in theory.
Review findings on the CASE marker.
- when a column stops carrying the rule (a metadata reload, or the column
becomes formula or read-only) the badge was removed but the tooltip kept the
sentence. stripCaseHeaderTitle is exported and applied there too, so the
header says nothing it cannot back up.
- the strip now removes only the `; <hint>` form the composer produces, so a
title that happens to use the same words is left alone, and a title that is
only a hint collapses to empty rather than to a hint with a separator.
- composeCaseHeaderTitle takes `string | undefined`, which is what th.title is,
and the spec drops its `undefined as any` cast.
- resolveEnteredCaseTransform no longer takes an isDerived flag: the grid only
tints a derived column, it does not block entries, so excluding it would have
stopped converting values the editor converts today. The integration spec now
drives the real predicate, with read-only and formula columns alongside the
editable ones.
Review findings on the change-detection steps and the shared build artefact.
- A failed fetch or diff in "Detect client changes" used to fail the job: the
fail-safe only covered an output that was absent, not a step that errored.
Both workflows now exit 0 without writing client=false on any failure, so an
undecidable change set runs the suite instead of blocking the pull request.
- .gitea/scripts/ joins the change set in both workflows: it carries the spec
list (cypress-specs.sh) and the runner gate (check-runner-memory.sh), so a
change to what the suite runs, or to the gate that lets it run, must run it.
- A skipped job now says so in a named step and a notice annotation, rather than
reporting success from a list of greyed-out steps.
- The dist upload sets if-no-files-found: error, so an empty artefact fails the
producing job instead of surfacing later as "webSourcePath ... doesn't exist".
- lighthouserc.js records the cost of numberOfRuns: 1 - no median, so one slow
navigation is a re-run rather than a fix.
A parent page standing in for the VA report that embeds Data Controller as
data-driven content: the dropdowns are the report's prompts, and the iframe is
framed and fed the way VA frames and feeds it.
Deliberately outside the DC build. The host is not the child: shipping it in
DC's deployment would put a test harness on customer servers and let DC be
opened top-level as a page that frames itself.
- report.json defines the report: child AppStream name, table, sandbox set,
delivery mode and timing, and the prompts (label is the va_parmap key, since
DC matches on the label rather than the report's internal parameter name).
- two delivery modes, because they reach different code in DC: refresh (reload
the frame, replay from before load - the report-refresh path) and prompt (post
into the loaded frame - an in-place prompt change).
- deploy.mjs posts the folder to a SASjs Server with /SASjsApi/drive/deploy and
streamWebFolder, which is the same route sasjs cbd uses, so a static app is
published the usual way and served at /AppStream/<name>.
Addresses the retroactive review of #447.
- afterGetColHeader appended the case hint to th.title on every render, and
Handsontable reuses the th, so the tooltip grew a copy per render. The hint
is composed by composeCaseHeaderTitle, which strips any known hint first, so
a repeat call is a no-op and a rule change replaces the sentence.
- the marker is now gated on the same predicate the conversion uses,
resolveEnteredCaseTransform, so a formula, read-only or derived column no
longer advertises a conversion the hook refuses to make. The class is removed
when the predicate stops matching.
- resolvePkFormula and the unescapeFormula write (given an explicit source) join
CASE_TRANSFORM_SKIPPED_SOURCES: neither is an entry, and re-casing a computed
key or a live `=...` string corrupts it. force_cell_validation_value keeps
converting on purpose, stated in the docstring.
Review follow-up on the browser harness.
The exclusion of va-ddc-host now names the issue that tracks re-enabling it
(dc/dc#449), so the coverage gate cannot report a rotted exclusion as accounted
for. The comment also states the ordering constraint the spec carries: it clears
the DC_JSLIB filter tables per case, and every spec in that job shares one mock
estate, so it has to run last or be given its own.
The spec's comments and test name describe the behaviour and the harness, with
no ticket numbers, per AGENTS.md. The va-filter docstring says what the inference
is: a guess from the value's shape alone, which can disagree with the column in
either direction, and which costs the deferred call rather than the page because
the attempt suppresses the abort modal and falls back to the unfiltered load.
Measured over the last 150 workflow runs on this forge: a pull request costs
56 runner-minutes (Build-and-ng-test 5.8m, Build-and-test-development 31.2m,
lighthouse 19.3m), and when three pull requests are in flight the first job
queues 47-69 minutes before it starts.
- Build-and-ng-test publishes client/dist as an artifact and the Cypress job
downloads it instead of running a second cold `npm run build` (1.6m each).
Both jobs test the same bytes.
- lighthouse: numberOfRuns 1 (lhci's default of 3 tripled the 13.6m audit to
steady a median that only has to clear a fixed threshold), and the duplicate
/view/usernav/groups entry is removed (17 unique URLs, not 18).
- Cypress records no video in CI and takes screenshots on failure instead
(`--config video=false,screenshotOnRunFailure=true`, CLI only so local runs
and the demo recorder still record); the failure artifact carries
screenshots as well as any video.
- Both workflows detect whether client/ or sas/mocks/ changed and skip their
expensive steps when neither did - a sas-only or docs-only pull request no
longer pays 45 minutes for suites that cannot observe it. The gate is
`!= 'false'`, so a missing step output runs the suite rather than quietly
reporting green.
issue 444 was a month-name date from VA becoming a character literal, and it
only reproduced on a report refresh: the deferred initial load reads VA's
buffered message, which has no column metadata, so the SAS kind comes from the
value's shape. Posting a message into a running page exercises the other path -
it arrives after getdata, metadata in hand, so a date is formatted correctly
whatever the client does - which is why the unit spec could pin the fix but not
show the failure a report author sees.
The new spec frames the built app in sasjs/server's /ddc-host page, which mounts
it with VA's sandbox attribute set and, with timing=early, replays the message
from the moment the frame is created. That is the request and apply path a
refresh takes, with mock services instead of a VA estate.
Where the assertion sits: the mock public/validatefilter stores what it is sent,
so the clause is read back from the drive (mpe_filtersource.json) rather than
inferred from a request payload. The store is emptied per case because
mpFilterStore reuses an existing FILTER_RK for a clause it has already seen, and
a leftover row would answer for this run. The host page's own log is the second
witness: the message is captured before the frame finishes loading.
Three cases in one run - the WORDDATE shape from the issue, a DATE9 date that
always worked, and a character parameter. Against the unfixed service the first
stores 'August 02, 1971' and fails, while both controls pass; with the fix all
three store the expected literal. So the guard goes red for the defect and only
the defect.
The spec runs last, needs no SAS, and takes about a minute against the mock
estate the other specs already use.
Where a column carries a CASE=UPCASE or CASE=LOWCASE rule in
MPE_VALIDATIONS, the EDIT grid treated it as validation only: a value in
the other case was flagged after typing, so the user had to retype it or
accept a red cell and fix it later.
The pending value is now converted before it commits, driven by the rule
that already exists rather than a hard-coded column list:
- DcValidator.getCaseTransform(col) maps a CASE rule on the column to
'upper'/'lower' (the two rule values dq-validation.ts acts on).
- The existing beforeChange hook applies it alongside ROUND, so an edit, a
paste range and a fill-down all take the same path, the cursor and
selection stay put, and undo keeps its history.
- applyCaseTransform / isCaseTransformSkipped live in a small pure module
so the conversion and the source exclusions are testable directly.
Values are not converted when they are not user entry: loadData (the grid
re-sending its dataset on every updateSettings({data})), UndoRedo.undo /
redo, and the Revert menu item. Converting those would rewrite values the
service supplied, or hand back something other than the value the user
asked to restore, and would leave untouched rows looking modified.
A CASE-bearing column is marked in the header ('Aa' marker plus a tooltip)
so the user can tell the conversion came from the grid.
The CASE validation itself is unchanged: values from an API, staged data
or an Excel import are still validated as before.
The footer is a %let and two %if inside the PDF branch of the service, which
already runs inside a macro, so the mpe_pdffootnote wrapper added a file and a
call boundary for a value that is built once, in one place.
Behaviour is unchanged: the same filter wording, session user, timestamp and
the truncation notice when the cap bites, still driven by pdf_maxobs so the
printed row count and the printed cap cannot disagree. A blank obs count is
treated as no truncation, which was the one real edge the macro guarded.
getrawdata.test.sas loses test 5 with it - those assertions existed because
the text was composed by a macro the test could call. With the composition
inline there is no seam to assert on, because a PDF cannot be read back as
text in SAS. The service-level PDF assertion (signature present) still covers
the branch, and the footer is checked from the downloaded file: page 1 of the
artefact reads "No filter | sysadmin | 03NOV2025 13:48 Page 1 of 20" on SAS 9,
and the same request on Viya returns a 20 page report.
7.1.3 carries sasjs/core#510: mv_getjobresult now returns a binary webout
from a job result instead of writing its own "not registered" text, so the
PDF download can be asserted on Viya as well as on SAS 9.
The only other change in the range is 7.1.2's filter validation message
(sasjs/core#511), which this project does not call.
Two real findings from review.
ACTION: the default was assigned with %let before the value was read, so a
URL-supplied ACTION was overwritten and the parameter could never take effect.
The incoming value is now captured first with symget, guarded by symexist so a
missing name does not write to the log, and the default is assigned afterwards.
%global was tried first and is wrong here: declaring a name that already exists
as a local in the including scope does nothing, while the symputx that follows
writes the global - so &action kept resolving to the local and no action ran,
which is what broke the makedata test.
Separately: on the SASjs Server STP endpoint a query parameter is not delivered
to the service at all - a request with ?action=DIRLIST&parent=/tmp reaches the
program with no fromjs and no symexist('action'), and fails on WORK.FROMJS
before ACTION is considered. The configurator's link is therefore legacy on
that target; the adapter path (the Angular deploy screen) is the one that
works, and the capture above is what makes a parameter effective if it does
arrive.
Folder type: the picker compared FILE_OR_FOLDER to 'folder' case sensitively
while mpe_dirlist selects on upcase, so an uppercased value emptied the list.
The compare is now lowercase, and entries mp_dirlist listed but could not open
are surfaced - rendered without the drill-in and Use buttons, with the service
message beside them - instead of being filtered out after the macro's header
promised they would be visible. Two specs cover both, including one that
renders a locked row.
The service tests exercise the actions through a child session; these call the
macros in place, so the guard and the answers are covered without a request:
mpe_dirlist 7 assertions - iftrue=0 does nothing, one level, files
excluded, three levels, the depth cap, the maxobs cap with
TRUNCATED=Y, and a blank parent returning a root path
mpe_writecheck 5 assertions - iftrue=0 does nothing, a writable folder, a
file, a missing path with a reason, and the probe leaving an
existing file that already had its old name alone
They live in sasjs/macros beside the macros they test, which works: a test file
in a declared macro folder compiles to sasjsbuild/tests/macros/ and sasjs test
runs it from there.
Two traps, both hit before this passed:
- these macros write JSON to _webout, so the test has to point _webout at a
temp file while they run and put it back before the term program writes the
results - the first attempt redirected without restoring and the runner
reported "No webout was returned by job";
- descriptions in mp_assert must not contain a comma: an unquoted comma splits
the parameter list and the assertion fails to compile with "All positional
parameters must precede keyword parameters", which names the call rather than
the comma.
check_outfmt and check_keepme were macros around one data step each, so they are
now those data steps, in the body where they run. Nothing else about them
changes: the &inds trick replaces the `%if &exist` guard, so the step still
touches work.fromjs only when the table is there.
That inlining exposed a real bug, found by running the KEEPME path rather than
by reading it. keepme was initialised with %let inside the body, so it was
local to makedata_body, while the request set the global with symputx(...,'G').
self_destruct, called from the body, resolves the nearest scope first - the
local 0 - so a caller asking to keep the program had it deleted anyway. It is
now declared %global and initialised at file scope, before the body exists.
Verified on the estate: KEEPME=1 leaves the program in place (a follow-up
request still finds it), a normal run deletes it (the next request reports that
the program does not exist), the suite is 3/3 with 16/16 assertions, and the
seed returns STATUS SUCCESS.
Each action is now its own macro that takes `iftrue=` and returns immediately
when its action was not requested:
%mpe_dirlist(iftrue= (&action=DIRLIST))
%mpe_writecheck(iftrue= (&action=WRITECHECK))
%makedata_body(iftrue= (&action=MAKEDATA))
so the service reads as a list rather than a branch, and there is no open-code
%if left in it. mpe_adminsetup splits into mpe_dirlist and mpe_writecheck
along the action boundary it already had.
The four helper macros move back to file scope, above the body. With the
dispatch branch gone nothing is nested any more, so the noNestedMacros
exception comes off both files - the last reason for it was the branch I just
removed.
mp_abort takes its own condition, so the two refusals are two calls in open
code rather than branches inside a wrapper macro:
%mp_abort(iftrue= (&action=MISSING) ,mac=&_program ,msg=...)
%mp_abort(iftrue= (&action=BAD) ,mac=&_program ,msg=...)
check_action was never needed either - it wrapped a single data step whose only
condition is a data-step IF on &exist.
That leaves one branch in the file, the two-way dispatch between the browse
actions and the seeding body, which is what noMacroIfInOpenCode actually
requires to be inside a macro.
Replaces the risk-register framing with the assessment: there is no admin group
until makedata creates it, the caller is the administrator doing the deployment,
the configurator in the same window is more powerful than these two actions, and
a successful run deletes the service anyway.
Filed as deploy-window-authorization.md so the next review finds the reasoning
instead of re-raising the finding.
Five defects from review, all mine.
ACTION is no longer defaulted. `%let action=MAKEDATA;` meant a caller with no
input table at all ran the seeding body - so a URL that never mentioned ACTION
would create the control library. The default is MISSING, which is refused,
and a URL parameter is whitelisted through the same check with symget() so it
is never resolved as macro text.
The write probe no longer has a predictable name. fopen with 'O' truncates, so
a probe called .dcprobe_<pid> destroyed any pre-existing file of that name in
the folder being tested. It now carries the pid and a random suffix, checks
fileexist before use, and gives up rather than reuse a name.
DIRLIST returns folders and locked entries only. The old filter dropped 'file'
and passed every other value - blanks and unknown types - to the picker as a
folder. The filter is now an explicit whitelist, and the header describes what
is actually emitted.
A request with no input table is refused, covered by a test: that is the case
the default used to swallow.
ngOnDestroy is gone from the client component - it reset a flag on a component
being torn down.
The deploy-window exposure is recorded in .agents/docs/, with why the admin
group cannot be checked there: %mpeinit() reads &mpeadmins from the control
library that makedata creates, so before it runs there is no group to check -
which is also why the previous dirlist could never serve this screen.
Reverts the hoist. The helpers read better next to the body that calls them,
and the body runs once per deployment, so the nesting costs nothing at run
time - the only cost is the noNestedMacros exception in the two headers, which
is the narrower trade than four macros defined far from their only caller.
Wrapping the body in a macro had demoted four existing helpers - check_outfmt,
check_keepme, emit_response and self_destruct - to nested definitions, which
needed a noNestedMacros exception in the file header. They were at file scope
before, so hoisting them back is undoing that demotion rather than promoting
anything new, and the exception goes away.
No behaviour change: macro variable lookup is dynamic, so the helpers still
resolve the body's variables when the body calls them.
The deploy screens asked for a physical folder the deployment user can write
to, as a single text box, and found out whether it worked only by running the
whole deployment. They now get a picker and a Validate button, backed by the
makedata actions that already answer those two questions.
A new app-dc-location component renders the path as an editable input - the
picker is a convenience over it, never a replacement, because a folder the
deployment user cannot open is not listed and a customer may want a path that
does not exist yet. Browse lists one level at a time (ACTION=DIRLIST, DEPTH 1,
MAXOBS 500), with Up, a "Use this folder" for where you are standing, and a per
row Use. Validate calls ACTION=WRITECHECK and reports the verdict inline -
writeable, or the reason it is not - instead of leaving it to the deployment to
fail.
Both screens use it: the Viya setup screen (which runs makedata through the
adapter) and the Sasjs configurator (which opens makedata as a URL). The
mkdata calls now send ACTION=MAKEDATA, since the service requires it.
Details worth knowing:
- every value read from the service is trimmed: SAS has no zero length
character value, so a blank field arrives as a single space and reads as a
value to any `||` or `@if`.
- the listing's level 0 row is the folder being listed, not a folder to drill
into - except when the parent is blank, where the level 0 rows are the
platform roots.
- failures are shown inline rather than as abort modals: the two requests pass
suppressErrorAbortModal/suppressSuccessAbortModal, because a refused path is
an expected answer here, not a crash.
- parentOf handles a drive root (C:\) and a unix root without inventing an
empty path.
Frontend tests: 14 new specs (38 in the deploy folder, all passing), including
a template spec for the browser panel and one for the verdict line, both
mutation checked.
The deploy screen needs to find and test a writeable folder before the control
library exists, so these questions cannot be answered by an admin-gated
service: %mpeinit() reads &DC_LIBREF..mpe_config, which makedata itself
creates. That is why the previous dirlist could not run in the window it
existed for.
They are answered by the service that already owns that window instead.
makedata now requires an ACTION column:
MAKEDATA create the configuration tables (what it always did)
DIRLIST the folders below PARENT, DEPTH levels deep, MAXOBS capped
WRITECHECK does PATH exist, is it a folder, can we write in it
so a deployment has one service to run and the same one to remove - makedata
already self destructs after a successful run, and now nothing is left behind
that can enumerate the filesystem.
ACTION is not defaulted: a caller that does not say what it wants is refused
rather than having a control library seeded for it. The payloads under
deploy/ and the configurator's link are updated in the same change, so the
installer and the seeding scripts keep working.
The work is in mpe_browse, called from both server and viya makedata:
mp_dirlist does the listing (a blank parent is the platform roots), and
mp_validatecol(ISPATH) shape checks whatever is submitted. Folders are
returned a level at a time so a parent with tens of thousands of subfolders is
never transferred in full, and the finite depth is what makes a symlink loop
harmless.
The body of makedata is now a macro of its own, because the branch cannot be
open code - noMacroIfInOpenCode keeps the file loadable on every 9.4
maintenance level. Its four helper macros are unchanged.
Input is the adapter table as before, never macro text: a value is refused if
it carries a macro trigger, a quote or a statement separator, and only the
three known ACTION values are ever put into a macro variable.
sas9 is not included: its makedata is the metadata STP driven by the
configurator page through URL parameters, not by the adapter, and the sas9
configurator takes a typed path rather than a picker. That is a change of
input mechanism, not of this feature, so it is left for its own ticket.
A parameter named without a value arrives blank, and a blank reached the
footer's numeric comparisons as nothing to evaluate:
%if &filter eq 0 /* blank filter */
%if %eval(&nobs) gt %eval(&maxobs) /* blank obs count */
That raised %EVAL arithmetic errors, stopped mpe_pdffootnote and failed the
whole download instead of producing a report. Both are now normalised to
their defaults first, maxobs included.
The footer's user= is taken from mf_getuser() at the top of the service - it
is deliberately not an input, so a caller cannot assert who ran the report -
and it is passed through %superq because a display name may hold a comma,
which the macro call would otherwise read as a parameter separator.
Test 5 gains a case for the blank call.
mv_getgroupmembers declared the member name at $43, and a longer name was
truncated - or, on Viya, stopped the step with "Multiple lengths were
specified".
membername is declared at $256 in the same shape the macro now returns it: one
width for every group, so the tables the caller combines are never of differing
lengths, and a long active directory name is kept whole.
Adds PDF to getrawdata: the staged, filtered dataset is rendered by
mp_ds2pdf and streamed with contenttype=PDF, so the Download modal gains
a typeset report - landscape, SAS formats and labels applied, headers
repeated on every page, 'Page n of m', and the filter, user and timestamp
in the footer.
The report is capped at 2000 observations, about 75 landscape pages, and
the footer states the truncation when it bites. The cap is one macro
variable driving both maxobs= and the footer text, so the two cannot
disagree, and the footer itself is composed by mpe_pdffootnote so the
truncation wording can be asserted without reading text out of a PDF.
@sasjs/core goes 7.0.3 -> 7.1.0 for the macro; the bump also brings
mp_validatecol ISPATH, two mp_dirlist guards and an mx_execute test.
On Viya the report renders but the download does not arrive: mp_streamfile
fails ahead of the PDF branch inside mp_binarycopy (RECFM=N with lrecl=1
needs 256), which is upstream of this change and left alone here.
The stage that reports what the file is doing is meant to belong to the whole file, and it finished at "Extracting the data finished" whatever happened. A sheet whose parse rejected had already blanked its datasource and dropped itself from the submission in the per-sheet catch, so the user was told the file had extracted while a table was silently missing from the submission.
`allSettled` was already the right primitive, but its results said nothing: the catch swallowed the error, so every promise fulfilled. The catch now names the sheet on the component, the stage ends at a new `extractionFailed` state when anything was left out, and the pane lists what was left out and why.
The two reasons a sheet contributes nothing are kept apart, because they are not the same thing to the reader: a sheet the sheet utility could not read is a "could not extract" problem in red, while a sheet it parsed and found no rows in - which arrives as the rejection "No relevant data found" - is reported as "no rows found" without calling the file corrupt. Both leave the sheet out of the submission, and either one stops the stage reporting a clean finish.
The row derivation sits inside the try with the parse. Outside it, a throw on a row shape the sheet utility did not expect escaped as a rejection that `allSettled` swallows - the dataset stayed in the submission with `parsingTable` true and the stage reported a clean finish over a sheet that had been dropped.
The messages live in the no-table-selected pane, which is where the user is once extraction has finished; the file pane they were first put in unmounts as soon as the datasets are fetched, so they would never have been read there. The stage line keeps one id per state rather than two of the same: `file-loading-stage` in the file pane while the datasets are fetched, `file-extraction-stage` in the overlay while the sheets are extracted, and the Cypress spec asks for both.
The Cypress spec asks for both stage ids, because the element that carries the stage changes state, and asserts that a sheet left out of the submission is named in the pane - under whichever reason the sheet utility reported, since only the utility decides which of the two it is.
The spec builds the component through the test injector rather than with seven positional arguments, so a constructor that gains or reorders a dependency fails there (`NG0201: No provider found for ElementRef`) instead of binding a stub to the wrong collaborator and passing.
The comment claimed the sidebar animates open/closed. It does not: the
collapse is a plain margin-left on clr-vertical-nav.closed (styles.scss)
and margin is not in any transition there - Clarity's width transition on
.clr-vertical-nav does not apply, because the width never changes. What
does need the deferral is that Angular applies the class binding after the
event handler returns, so a synchronous measurement reads the pre-toggle
layout.
The spec now says what it covers: the scheduling contract, not whether a
measurement lands before or after a transition.
Unsubscribing stops future layout notifications but leaves a timer that
one already scheduled, so a sidebar toggle immediately followed by destroy
still ran calculateInputPosition() against a torn-down view. The timer id
is now held and cleared in ngOnDestroy, and the spec covers that window -
it fails if the clear is removed.
Also drops the EventService.dispatchEvent stub from the viewboxes spec,
which advertised a method this branch deletes.
The specs gated their assertions on a real setTimeout(..., 10) plus done,
which is wall-clock timing: a loaded runner can assert before the
component's own timer fires, and the negative case would then pass
vacuously. fakeAsync + tick order the assertion on the scheduler instead.
Also drops the stale half of the viewbox resize comment - the shared tail
no longer notifies listeners, it just persists and re-measures.
The synthetic resize dispatch was its only caller. Nothing else used it,
and a generic window-event dispatcher on a service invites exactly the
coupling this change removes.
The sidebar spec now asserts the behaviour directly - no resize event
reaches window listeners on a toggle - and fails when a dispatch is put
back, rather than asserting on the removed API.
A sidebar toggle and the viewbox data/resize paths dispatched a synthetic
window resize so other components would notice the layout moved. Two
thirds of that was unnecessary: Handsontable 18 attaches its own
ResizeObserver to the grid's parent, so a container change re-measures the
grid with no event at all (measured in Chromium: view.getLastSize().width
tracks a container resize 600 -> 380 -> 700 without any dispatch).
The one real consumer was the autocomplete dropdown, which follows its
input under the new layout. It now subscribes to
EventService.onSidebarToggle and re-runs calculateInputPosition(), and
unsubscribes on destroy; the window:resize listener stays for real
viewport changes.
Removed: sidebar.component.ts:53, viewboxes.component.ts:281, :705 and
:1571. The viewbox resize tail already re-measures each instance through
refreshTableAfterResize, and a data load renders the grid itself.
Scoping the pass to one grid left the page's other grids uncorrected: the
viewboxes each render their own Handsontable (shared/viewboxes), and the
multi-dataset preview renders two. Nothing called the pass for them, so
they kept role=treegrid and aria-rowcount=-1 - the failures this change
exists to remove. The old document-wide sweep happened to cover them.
observeGridAria(hot) runs the pass and keeps it running from the grid's
own afterRender, for grids whose component has no render hook of its own.
It defers by one microtask: Handsontable writes the row/column counts
after the render that follows a data load, so a pass inside the hook
still sees the previous frame.
Wired into getViewboxHotInstance (every viewbox that hands out an
instance) and into both multi-dataset instances.
The root `package.json` carries no install or postinstall script, so the
guidance to rely on "the composite install script ... which installs both
subprojects" sends the reader after something that does not exist, and
`client/.npmrc` told them the same thing in a comment.
The install section also contradicted itself: it opened by telling the reader
to write the SheetJS registry token into `client/.npmrc`, and closed by
explaining that `client/.npmrc` is a tracked policy file that must not carry a
token (the pre-commit hook rejects one). The token goes in the user config,
which is what the first block now says.
The install sequence itself is unchanged - root `npm ci`, then `client` and
`sas` explicitly - because that is what the repo actually needs.
Review follow-up. The table pass matched every table inside the grid root,
so a table a cell renderer owns picked up role=table and scope=col - which
contradicted the reason the wrapper loop avoids [role="treegrid"]. It now
matches table.htCore only (every HOT table carries that class, one per
view), and header cells are read with :scope > thead > tr > th so a
renderer's nested table is not touched. The separate aria-rowcount=-1 pass
is gone: the wrapper strip already removes that attribute from the grid's
own element, and matching it anywhere else was the same overreach.
The viewer also gets the editor's fallback chain, so a pass that runs
before hotInstance is assigned still finds the grid through
hotTableComponent.
Scoping the ARIA pass to the grid exposed a dependency the document-wide
sweep had been hiding: the sweep used to write aria-label from an
element's text, so cypress/e2e/full-table-search.cy.ts could select the
filter modal's submit button as aria-label="Ok" - a label only the sweep
ever wrote. Once the sweep stopped reaching the modal, that selector
matched nothing and the dev job failed on it.
An element's text already is its accessible name, so the synthesis is
gone from the pass, and the spec now selects the button by its text the
way a user would.
The helper-only spec passes even if the observer goes back to calling
updateSettings() - which is the regression the helper exists to prevent.
Move the observer into refreshGridAfterContainerResize.ts so a spec can
drive a real container resize through it, and assert that the resize
reaches refreshDimensions() without any settings call. Verified by
re-injecting the old wiring: the spec fails with 'Expected spy
updateSettings not to have been called', and passes once it is removed.
Also corrects the helper's claim about what refreshDimensions() measures:
it compares rootElement.getBoundingClientRect() with the last measured
size, and both grids take their height from a viewport-derived calc(), so
a resize that matters is a width or viewport change.
The sweep matched [role="treegrid"] inside the root, which would also
catch a treegrid a cell renderer put in a cell. The grid's own wrapper is
the root itself, so target that plus the overlay wrappers Handsontable
renders, and leave nested widgets alone.
Adds a live-grid spec: Handsontable writes role=treegrid/aria-rowcount
again on loadData, so the pass has to run after renders, not once.
The editor and viewer each ran fixAriaAccessibility() over the whole
document on every render, on a 3-second interval and from a
MutationObserver on document.body, and each call scheduled four further
copies of itself. The sweep removes the role/aria attributes that its own
observer watches, so it re-armed itself.
Measured with axe-core against Handsontable 18: the default markup fails
aria-required-children (critical) because the treegrid wrapper has no row
children, and disabling the tags with ariaTags:false only trades that for
empty-table-header while leaving the placeholder semantics missing. The
existing attribute sweep is therefore the right end state - it just has to
run against one grid's root, not the document.
Both components now call one shared helper with their own
hotInstance.rootElement, and the interval, the body observer and the four
deferred re-runs are gone; the afterRender/afterChange hooks and the init
calls are the only triggers.
The grid's container ResizeObserver (added in c7ba025 to replace the
afterRender window-resize dispatch of #246) refreshed by calling
updateSettingsSortSafe({ height }), and that wrapper's whole reason for
existing is that any updateSettings() while a multi-column sort is active
corrupts formula references into #REF!. It works around that by clearing the
sort, applying the settings and re-applying the sort - so every sidebar
toggle, modal open/close or window resize re-sorted the user's data, and an
open cell editor (anchored to visual coordinates) could end up writing to a
different row than the one being typed into.
refreshGridAfterContainerResize() replaces it with hot.refreshDimensions():
it compares the element's box with the last known size, re-measures the
overlays and redraws only when the size actually moved, touching no settings.
The decision lives in its own module so the spec can pin it - a real
Handsontable instance with an active sort and a formula column asserts that
the resize path calls refreshDimensions() and does not call updateSettings(),
clearSort() or sort(), that the sort config is unchanged, and that no cell
degrades to #REF!.
This is the narrow, independent piece of #246; the wider removal of the
remaining synthetic window resize dispatches (sidebar, viewboxes) is analysed
on the issue and is not part of this change.
Two findings from review.
downloadTablePdf() revoked the blob URL in the same task as the anchor click.
Firefox and Safari read the blob asynchronously, so that can cancel the
download and leave the user with no file; the revoke now runs on a later turn,
after the click has been handed to the download machinery.
getExportRange() clamped the selection's left edge up to skipLeadingCols but
then forced the right edge up to match, so a selection lying entirely inside
the leading housekeeping column(s) was shifted sideways into a column the user
never selected - the PDF contained column 1 when they had selected column 0.
The range is now clamped into the exportable columns and comes back empty
(right < left) when nothing is left, which the PDF path reports as "no table"
and the CSV/Excel path skips rather than exporting the wrong column.
The revoke change is only covered by the spec (it needs a browser to exercise);
the range change is covered by the spec and by the local harness, now 28 checks.
The modal decided what was wrong with two regular expressions of its own, and the answer disagreed with the log it was showing:
- an ODS HTML line was cut from the first capital `E` in it and lost its last character, so an error read `ERROR: File WORK.X.DATA does not exist.</font>`;
- a message SAS wrapped over several lines was reported once, truncated at the break, and the rest of it was not reported at all;
- the colon-less form SAS writes (`ERROR Libref X is not assigned.`) was missed on any line carrying markup;
- the notable notes - uninitialised variable, invalid data, a format too small - were not shown anywhere;
- clicking an issue searched the rendered text for its message, so a log that repeats a message (two `ERROR 22-322:` lines is ordinary) always jumped to the first of them.
`parseLog` from `@sasjs/utils` replaces all of it, the same function the VS Code extension and the SASjs server Studio use, so one log now reads identically in every host. Each request keeps the issues the parser reported, grouped by severity, and the tree gains a Notes list beside Errors and Warnings. The click scrolls to `issue.line` and flashes from that line to `issue.endLine`, and the lists no longer render at zero.
The modal is SAS Logs now, as are the menu entry, the deploy button and the abort modal's button; the request history is unchanged underneath.
`formatTimestamp` decided on `moment().format()`, which is 'Invalid date' - truthy - for anything it cannot parse, so its fallback could never fire. It asks `isValid()` now.
The warning flash was yellow where the warning text is ochre; both are ochre now, and the notes flash blue.
`@sasjs/utils` goes to 4.3.1, which is where the parser lives.
The new empty-grid spec built a grid from `data: []` alone, and Handsontable
reports zero columns when the data is empty and no `columns` setting defines
the schema - so buildGridPdfInput() correctly returned null and the spec failed
in CI while passing against the stub harness. The real grids declare their
columns, so the fixture does now too.
Two findings from the second review.
A row taller than the printable area was drawn straight through the footer and
off the page, losing the tail of the cell unannounced - MAX_COL_CHARS bounds a
column, not a value, so one long text field wraps to more lines than a page
holds. Rows are now clamped to the tallest row that fits the page, with ' ...'
marking the cut, and the height budget accounts for both margins so an
exactly-fitting row stays on the page it started on instead of leaving the
first page with headers only.
exportGridPdf() also returned silently for a grid with no rows, although the
writer renders an empty table as a valid header-only page. The input is now
built by buildGridPdfInput(), which returns null only when there are no
columns, so a rowless grid downloads a page saying "0 rows x N columns".
Tests: the tall-row spec asserts the marker, the single page and that no text
start falls outside the printable band; the empty-grid spec asserts the input
is built rather than skipped.
Two findings from review.
First, a regression in the CSV/Excel path: getExportRange() defaults `left`
to skipLeadingCols, so comparing it against skipLeadingCols made every
no-selection export look like a whole-grid export and the plugin put the
leading housekeeping column back in. The whole-grid test now compares the left
edge against visual column 0, so a skipped column stays out.
Second, the PDF row cap was applied after collectGridTable() had already read
every cell, so a very large grid still froze before the cap took effect.
collectGridTable() now takes a maxRows limit that clamps the row loop before
any cell is read, and exportGridPdf() still reports the true total in the
subtitle.
Both are covered: a spec asserts the CSV export receives range
[0, skipLeadingCols, lastRow, lastCol], and one asserts the cap stops cell
reads rather than collecting and slicing.
CI caught this on the real grid specs: getPlugin('hiddenColumns') returns the
plugin instance even when it is switched off, and isHidden() then dereferences
a map that enablePlugin() never created, so any grid without a hiddenColumns
setting threw instead of exporting.
Ask the plugin's enabled flag first and treat a failure as "visible" - an
export is better off with a column the user could not see than with an error
dialog. The spec adds a regression case for exactly the CI failure, and its
afterEach no longer destroys an already-destroyed instance when a test creates
no grid.
Adds a PDF entry to the right-click Export submenu, next to CSV and Excel,
so the grid as displayed can leave the browser as a printable document.
Implemented directly against PDF 1.4 rather than adding a PDF library: the
output is a text table, which needs no embedded fonts, no images and no
compression. The base-14 Courier font is used because its glyphs are exactly
600/1000 em wide, so column widths are arithmetic on character counts instead
of a font-metrics table - a 1200 x 12 grid is laid out in ~60 ms.
Layout: A4 landscape, bold header band repeated on every page with "Page n of
m" footers, numeric columns right-aligned, values word-wrapped inside their
column, and long tables paginated. Wide tables are fitted by shrinking the
body font first (7pt down to 5pt, so a 20-column table keeps full headers at
6pt) rather than squeezing columns; only at 5pt are columns scaled, and they
stop at 4 characters - past that trailing columns are dropped and the first
page says how many were kept, so nothing is omitted silently.
Encoding is WinAnsi, the widest set the base-14 fonts cover without embedding;
characters outside cp1252 print as '?'. A typeset, branded report is a
different feature and belongs on the server, where SAS formats and DC metadata
are available.
The export keeps the existing "what you see" contract: current selection (when
it spans more than one cell), otherwise the whole grid, hidden columns
omitted, leading housekeeping columns skipped. The range calculation that CSV
and Excel already used is now a shared getExportRange(), so the three formats
cannot drift apart. Rows are capped at 5000 and the subtitle says so.
Tests: pdf-export.util.spec.ts covers document structure (xref offsets, page
count, footer numbering), string escaping, WinAnsi mapping and fallback,
wrapping, alignment, multi-line values, pagination, font fitting and the
page-width invariant; hot-export.util.spec.ts covers grid extraction against a
real Handsontable instance.
A MIN_TOTAL_MB typo reached the integer test under set -e and aborted with "integer expression expected"; it now prints usage and exits 2. A kernel without MemAvailable no longer reports 0 MB available in the summary or the warning, so the oversubscription warning stays meaningful.
release.yaml (all three jobs), sites.yaml and release-assets.yaml run the same cold production build that the review jobs do, so they get the same pre-install check.
The header now says what the check rules out (an undersized box) and what it does not (two jobs sharing one box), so the MemAvailable warning is not read as a gate.
An undersized runner does not fail honestly: the kernel OOM killer takes
the build, so the job reports exitcode 137 at ng build, or Karma
disconnects mid-suite after twenty minutes. Both read as a flaky test
rather than as a box that cannot hold the job, and both cost a whole CI
slot.
The check runs first in the three client jobs (Build-and-ng-test,
Build-and-test-development, lighthouse), prints total and available RAM,
vCPU count and any cgroup limit, and exits 1 when the effective size is
under 8 GB - the supported minimum. A cold production build measures
3.8 GB peak on its own, so 8 GB leaves roughly 2x headroom.
An uploaded file's preview is validated in place, so the cells that fail
a rule carry Handsontable's invalid cell meta. Discarding the file
restores the loaded data, but cell meta is keyed by row/column and
survives that restore - without a re-validation the grid keeps painting
cells that are no longer invalid.
cancelEdit re-runs the validation (hot.validateCells()) after restoring
the data source, which is what clears them; this test drives the flow and
pins it: import a file that fails validation, discard it, then assert the
row count is back, no .htInvalid cell remains and the page is read-only
again. The manual-edit path (test 25) keeps the parsed values by design.
Refs #122
The multi-load page rendered the loading stage only inside a branch guarded by
`parsedDatasets.length`, and that collection is empty for the whole read and
sheet-scan phase - so two of the four stage labels ("Reading the file",
"Searching for the data in the file") were assigned but never reached the DOM,
and a large file sat on screen with no feedback at all.
The stage is now shown next to the file name and size from the moment a file is
attached, and again in the content area while the sheets are being parsed. The
labels cover every phase the user waits through: reading the file, searching
for the data in the file, file ready to load, fetching the dataset details (one
getdata call per dataset - the longest wait), extracting the data, and
extracting the data finished. A spinner accompanies the stages that have work
running; the idle stages and the finished state show the label alone.
Two corrections fall out of that: the stage only moves to `extracted` once
every sheet has been parsed rather than when the first one finishes, and the
read's continuation only advances the stage if the read is still the phase in
progress (the sheet scan runs in parallel and can finish first).
excel-multi-load.cy.ts gains test 4, which asserts the file size, the ready
stage, the fetching stage (the getdata calls are held open so it is on screen
long enough to assert) and the handover to the "select a dataset" hint.
Closes#121
Enables noMacroIfInOpenCode and promotes it to "error". Open-code %if is a SAS
9.4M5 feature, so the 19 sites it found in nine files were hiding a version
requirement the codebase does not document (the requirements page says 9.4M3).
Each is rewritten so the construct no longer appears in open code:
- Six blocks become a small macro called unconditionally, which is where the
conditional then lives: drop_cas_tables (demodata), add_derived_cols
(getdata), build_sasdata4 (getdata test 4), drop_derived (stagedata),
build_staged (getrawdata), and one per option in the 20261003 migration.
- Four blocks in the two makedata STPs get the same treatment: check_outfmt,
check_keepme, emit_response and self_destruct. Their `call symputx` writes
are now explicit about the global symbol table, because a %let or a plain
symput inside a wrapper would be local to it.
- Two guards in stagedata and getrawdata are removed rather than wrapped: the
SQL only selects into a list that is already %let blank, and with no derived
variables it returns no rows, so the guard was doing nothing. The trailing
drop keeps a conditional, now in a macro and tested with %trim, since a
zero-row select can leave a single blank in the variable.
- validatefilter's two %let branches become one data step with call symputx,
which writes to the global table the later %mp_filterstore reads.
Verified: `npm run lint:sas` clean with the rule as an error; `sasjs compile`
succeeds; on sas.4gl.io the full suite passes 64/64, including the derived
variable tests in getdata test 4, stagedata tests 3 and 4 and validatefilter,
which are the paths these wrappers touch.
The tree is clean under all of them, so the remaining warn-level rules can be
promoted without a single code change: hasDoxygenHeader, hasMacroNameInMend,
hasMacroParentheses, noNestedMacros, strictMacroDefinition, noSpacesInFileNames,
lowerCaseFileNames, noTabIndentation, noTrailingSpaces, noGremlins and
noNestedCommentsInHeader.
With that, every rule @sasjs/lint 4.9.0 can raise is an error - only
noEncodedPasswords was an error by default - so any new violation fails
`npm run lint:sas` and the CI step that runs it, instead of printing a warning
that leaves the exit code at 0.
Checked: a trailing space appended to a file reports
"Error Line contains trailing spaces" and a non-zero exit, where it used to be
a Warning.
sas/.sasjslint takes maxLineLength from 180 to 80 - the linter's own default -
and promotes it to "error".
53 of the 258 .sas files already hold a line longer than 80. Rewrapping them
would be a large, reviewable-only-in-theory diff, so each keeps its current
longest line as an exact ceiling via a per-file override, e.g.
@sasjslint {"maxLineLength": 95} in services/validations/tables_all.sas. A
file with no override is held to 80, and an overridden file cannot grow past
the length it has today: the ceiling is the file's longest line, not a round
number.
Both directions were checked: a 101 character line added to an
un-overridden file fails with "Line exceeds maximum length by 21 characters",
and a 114 character line added to macros/mpe_execute.sas (ceiling 101) fails
by 13. `npm run lint:sas` is clean with the setting in force.
Rung 6 of the #376 ratchet: `"lineEndings": "lf"` is enabled and promoted to
"error". Its 37 findings are cleared and nothing but line endings changed.
- 34 files ended without a newline; each now ends with exactly one LF.
- 3 files were stored with CRLF (services/public/getgroups.sas and the two
sas9 lineage services, getmetacols and getmetatables), against the repo's
own `* text=auto eol=lf` in .gitattributes. They are converted to LF.
Checked mechanically that this is whitespace only: for every changed file,
HEAD's content and the new content are identical once CRLF is folded to LF
and the trailing newline is stripped.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- On sas.4gl.io, against a scratch appLoc seeded with makedata:
"Tests that pass: 64/64 (100%)".
The rule flags the %if blocks service bodies run in open code, which need
SAS 9.4M5. The documented minimum is 9.4M3, so enabling it would mean
raising the floor or wrapping every service body in a macro - neither is
a lint change. Written down so the next agent does not enable it by
mistake.
Rung 5 of the #376 ratchet. Both rules are off by default; enabling them
surfaced 10 findings, all now cleared, and both are promoted to "error".
- Nine `* text;` comment statements become block comments: three disabled
debug lines (a putlog in bitemporal_dataloader, an alternative type
assignment in mpe_loader, a put in demodata) and six explanatory
comments in the two sas9 usernav services.
- loadfile.sas assigned `libname approve "&dir"` and never used the
libref; stagedata.sas and restore.sas assign the same libref because
they write approve.jsdset, loadfile writes through a fileref instead.
The statement is removed.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- On sas.4gl.io, against a scratch appLoc seeded with makedata:
"Tests that pass: 64/64 (100%)".
Rung 4 of the #376 ratchet. It clears the last 5 findings and promotes the
three rules that produced them to "error": minMacroNameLength,
lowerCaseMacroNames and indentationMultiple. `sasjs lint` now reports no
warnings and no errors across the project.
- viewdata.sas wraps its service body in a helper macro named `x`; it
becomes viewdata_body. The wrapper is what keeps the %global and the
%return in one macro scope.
- usergroupsbymember.sas (sas9) builds work.iwant inside a helper macro
named `x`; it becomes build_iwant.
- refreshtablelineage.sas (sas9) defines `getTables` with uppercase; it
becomes `gettables`, at the definition, the %mend and its seven calls.
- getstagetable.test.sas: one comment line carried three leading spaces
where the file uses two.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0, "All matched files
use @sasjs/lint code style!".
- `sasjs compile`: succeeds.
- Deployed to sas.4gl.io and ran the SAS suite there against a scratch
appLoc (/Public/app/dc-lint-ratchet) seeded with makedata: "Tests that
pass: 64/64 (100%)". The viewdata and usergroupsbymember suites are in
that run, so the two renames are covered by passing tests.
Rung 3 of the #376 ratchet: `sas/.sasjslint` promotes noBannedStrings to
"error". The rule flags the literal words "error " / "warning " anywhere in
a file, comments included, because they turn up in a grep of the SAS log
and bury the real messages. Its 13 findings are cleared:
- Nine are prose and are reworded, so the sentence both reads normally and
stops matching: a truncation that aborts a build, a syntax fault, an
abort condition, "only records the failure in work.mp_abort_errds" (four
copies of the mp_abort-inside-%include comment), a test description, a
hook comment.
- Two are the text the user sees. loadfile.sas reports
"(mf_abort.error file found)" - the marker file carries the word in its
name - and buildtermsas9.sas reports "Some final error in &_program";
both are reworded to say the same thing without the literal, since the
text sits in a macro argument where the %str(WARN)ING split does not
survive the quoting.
- Three keep the word because the word is the content: the doxygen
@warning tags in macros/bitemporal_dataloader.sas and the two SAS 9
admin STPs (makedata, makelib). Each carries
@sasjslint {"noBannedStrings": false}, as
targets/server/services_server/admin/makedata.sas already did.
VERIFICATION
- `npm run lint:sas`: 5 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- Deployed to sas.4gl.io and ran the SAS suite there:
`sasjs test` reports "Tests that pass: 64/64 (100%)". The run used a
scratch appLoc (/Public/app/dc-lint-ratchet) seeded with makedata, so
the shared /Public/app/dc deployment was left alone; the scratch target
was removed from sasjsconfig.json before this commit.
5 warnings remain: macro name length (2) and case (2),
indentationMultiple (1) - the last rung.
Rung 2 of the #376 ratchet: `sas/.sasjslint` promotes noUndeclaredMacros to
"error". Its 25 findings on release/7.22 are cleared, and two of them were
real faults rather than missing prose.
WHAT WAS BROKEN
Comparing the compiled program against the macros it calls (compile, then
list every project macro each generated service calls but does not define)
found three services calling macros that were absent from their own
compiled program:
- services/validations/tables_all.sas calls %mf_getuniquename() on its
third line and never declared it.
- services/validations/mpe_tables.dsn.sas calls %mf_getuniquename() and
%dc_assignlib() and declared neither.
- services/hooks/mpe_tables_postedit.sas calls %dc_assignlib(), which is
supplied by the service that includes the hook (see below).
The two validation services are configured as rule_value entries in
MPE_TABLES (mpe_makedata.sas and the v3.12/v3.13 migrations), so the calls
run when a user reviews or filters a library - each would have raised
"Apparent invocation of macro ... not resolved". Both now declare what
they call, and the definitions are present in the compiled program.
WHAT CHANGED
- 14 files declare the macros they call but had been relying on another
declaration to bring in: mf_getquotedstr, mf_nobs, mf_getuser (twice),
mf_getapploc, mf_getattrn, mp_abort (six services), mf_getuniquename
(twice), dc_assignlib, bitemporal_closeouts. These were defined in the
compiled program already; the declarations make the dependency visible
and the two validation services correct.
- 6 hook scripts get a file-local @sasjslint {"ignoredMacros": [...]}
(mp_abort, plus dc_assignlib in mpe_tables_postedit) instead of a macro
list. Hooks are included at run time by the service that owns them, so
the macros are already in that session, and these files are copied and
edited by end users - they should not carry a declaration section to
maintain. Recorded in AGENTS.md.
- bitemporal_dataloader.test.2.sas loses its test 30 block: it is gated
with `and 0` ("skip until have a db") and its only setup line calls
%assign_direct_lib, a macro that exists nowhere in this project or in
@sasjs/core. Enabling it would fail on the first line.
VERIFICATION
- `npm run lint:sas`: 18 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds; the called-but-undefined list drops from
three services to the one hook exception; the only definition-count
changes are additions.
- 18 warnings remain for the next rungs: noBannedStrings (13), macro name
length (2) and case (2), indentationMultiple (1).
sas/.sasjslint now promotes noUnusedMacros to "error", so the rule fails
the build rather than printing a warning. The 54 findings it reported are
gone:
- 46 header entries nothing calls are dropped. Each removal was checked
against the compiled artefacts rather than the linter alone: the
project was compiled before and after, and the two sasjsbuild trees
compared. 37 macro definitions disappear from the generated programs -
every one of them either a duplicate of a definition the file still
reaches through another declared macro, or a macro no generated program
calls. No macro that a generated program calls lost its definition.
- 8 entries are kept, in five files that carry
@sasjslint {"noUnusedMacros": false} with the reason written above it:
the three serviceInit programs and serviceterm.sas declare mpeinit and
mpeterm without calling them, because that declaration is how a service
body reaches those macros at all; loadfile.sas declares mp_binarycopy
for the fallback that is commented out on SAS 9.4 and later.
- Four headers are left with no macros to declare, so their empty
<h4> SAS Macros </h4> section is removed rather than kept as a stub.
@sasjs/cli is bumped 6.2.0 -> 6.5.0 in sas/package.json. The lint engine
ships inside the CLI and is not a direct dependency, so the pinned CLI is
what fixes the rule set: 6.2.0 resolves @sasjs/lint 4.3.0, 6.5.0 pins
4.9.0 exactly. Both versions were compiled against this tree and the
sasjsbuild output is byte-identical, so the bump changes the lint engine
and nothing else.
`sasjs lint` is also now reachable as `npm run lint:sas`, and a CI step in
build.yaml runs it. Until now nothing ran SAS lint automatically, and a
bare `sasjs` resolves whatever CLI is installed globally - a different
lint version reports a different rule set, which is why the counts in
issue #376 (taken with 6.2.5) do not match the pinned 6.2.0.
43 warnings remain under the rules not yet promoted: noUndeclaredMacros
(25), noBannedStrings (13), macro name length and case (4),
indentationMultiple (1). They are the next rungs of the ratchet.
The pre-commit Prettier gate could not run: it needs client/node_modules,
which this worktree does not install, and the fix it attempted touched
three client files that this change does not modify (reverted). No client
file is in this commit.
`cy.get` inside `.within(<tree node>)` only searches that node, and the
metadata accordion renders in the content area - so `.clr-accordion-header-button`
was never found however long it retried. The spec queries the accordion at
document level, selects the first metadata type by its class rather than by
node index, and clicks the header button directly (Clarity puts
`clr-accordion-header-button` on the button itself).
The assertions are on the app's own markers: the single open panel
(`clr-accordion-panel-open`, `aria-expanded`), the association tree inside the
panel, and the attribute grid that `metanav/metadetails` fills.
metanav moves from EXCLUDED to SPECS in `.gitea/scripts/cypress-specs.sh`.
Closes#395.
The case took the id from the stage route and looked for a download button
carrying it. For an upload the route does not hold a table id at all - it holds
a load reference, the CI failure naming DC20261031-201657.470_601010_4449 - so
the button it looked for could never exist, however long it waited or however
many times the screen was reloaded. That is why the row "never appeared" while
the approval was in fact there.
The row is now found the way hook-programs finds its own: by the table it names.
The id the download is named after is read from the button, which is the only
place that knows it.
Verified against the mock estate locally, with the licence the spec now plants:
7 passing, exit 0.
The estate ships no licence key, so it runs on the free tier, and the free tier
disables file upload - which is exactly what the audit case needs. Whether it
could run depended on another spec having planted a key earlier in the run: the
key persists in the estate config, so the case passed or failed on spec order.
loginAndUpdateValidKey(true) plants one here. csv-limited, which asserts the
free-tier modal, runs first in cypress-specs.sh and is unaffected; download runs
last, so nothing downstream inherits the key.
Found by running the spec against the mock estate locally: the upload input
never rendered, the modal read "Locked Feature (File Upload) ... site id (123)",
and the run went 7/7 green once the key was planted.
mpe_abortmessage existed to assemble the message mpe_execute aborts with - the
program, the state and the child message - inside a data step, so that a percent
sign, an ampersand or a quote in the child message was not macro resolved on the
way into a symbol. mpe_execute already runs a data step over the status row for
state and errmsg, so the text is built there now and the helper, plus the %let
fallback that duplicated it, is gone.
What the helper did is kept: the values come from the status row through
symget/symputx, never through %let, and a message already at the 32767 limit is
cut to the room the prefix leaves.
The comma and percent cases its tests covered now live in core's mx_execute
test, where the message is captured from a real child; what stays here is the
assembly DC owns - the prefix, the state and the child message from the status
row.
The audit-file case uploads a fixture and then looks for the approval it should
create. An upload identical to what the table already holds produces no
changes, and a changeset with no changes leaves nothing waiting for approval -
so the case finds no row. A retry lands in exactly that state: the first
attempt's rows are already in the table, so the second upload is a no-op and the
failure looks like a missing approval rather than a duplicate upload.
The fixture is now copied per attempt with its primary keys renumbered from a
random base, so the upload is always an insert. The copy is removed in
afterEach - deleteFolder cannot be used there because the source fixture lives
in the same directory.
Core 7.0.3 fills outstatus.errmsg on both platforms - the SASJS branch since
7.0.2, the Viya branch from the job result as of 7.0.3 - so a canceled call now
always carries the reason it was canceled.
That removes the last platform difference from the assertion: while Viya wrote a
blank errmsg, an expected message could only be checked on one platform and the
state alone had to stand in on the other. A comparison against a message that
was never recorded now fails instead of passing silently, which is the point of
asking for it.
The macro decided an aborted call by scanning the child's response text for a
sasjsAbort object and then recovering the message from that text - anchors for
the separator, gating on whether the payload was escaped, cuts for the log
extract, and a capture width that had to be widened more than once. Every one
of those steps existed to undo the escaping in the response.
Core now publishes the abort reason in outstatus.errmsg for both the SASJS and
Viya branches, with the whole message and no truncation, so the macro reads the
row instead: STATE says whether the child was canceled, ERRMSG is the reason.
The cut at the appended log extract is the only thing left, because that text is
part of the message the child raised; it is stripped as characters, as an
escaped pair or as a doubled pair, whichever the producer wrote.
The macro therefore takes outstatus= rather than outref=, and the call sites in
the service tests now pass the status dataset they already asked %mpe_execute
for. stagedata.test.4's two aborting calls had no status row at all and now
have one.
A platform that reports the state but no message - SAS Viya until core fills
errmsg from the job result - is judged on the state alone, so the same tests run
on both platforms.
@mpe_assert_abort.test covers the twelve cases directly against status rows
instead of building response files, including the doubled separator the estate
actually writes, a message past the old 256 character width, quotes and
backslashes.
Requires @sasjs/core 7.0.2, which carries the SASJS half of this.
The upload and submit now work; the failure moved to the approve grid, which is
loaded when the screen opens and does not yet contain the row just submitted.
The screen is reloaded up to five times, two seconds apart, until the row with
this test's id is present.
The mock getddl aborts when ds is missing, as production does (getddl.sas:43-45),
instead of quietly returning a library-level DDL - a client that dropped ds would
otherwise stay green against the mocks.
The comment above the case no longer claims a leftover row cannot matter: the
grid paginates, so a row pushed past the first page would not be found.
The upload does parse now - CI showed the info modal appearing - but the helper
treated it as an abort because the info modal and the abort modal are the same
component (.clr-abort-modal). The title is what separates them: a good upload
raises "Table Found", so that is asserted and anything else fails with its own
text instead of being clicked through.
After Submit, the stage route is already current, so waiting on the URL proved
nothing; the editor leaving preview mode is the signal that the submit was
accepted. Also trims the comment that narrated the old route.
The first case failed in CI with "Expected to find element:
input[type=file]#file-upload", because the editor renders that input inside its
upload modal - the Upload button in the button bar opens it. Clicking the button
by its label rather than by position, as excel.cy.ts does.
The viewer download cases (2-7) passed, so the rest of the spec is sound.
Follow-ups from the review of #414.
The credential guard now checks each staged .npmrc rather than only the client
one: the root and sas/ files are tracked policy as well, so a token pasted into
either was equally one `git add -A` away. Verified by staging a token in
sas/.npmrc - the hook exits 1.
$RUNNER_TEMP/npmrc is removed at the end of each job that writes it, with
`if: always()` so a failed job cleans up too.
The root `install` script is gone: the root .npmrc sets ignore-scripts, so npm
never ran it. CONTRIBUTING already spells out the per-project commands, and is
now the single source.
The release workflows build the published client from the same tree, so they run
`check:archiver` like build.yaml does.
readdir-glob reaches minimatch twice: the directory walk, and the pattern/ignore
matching behind archiver's glob(). The second is where a semantic change shows
up as silently skipped files rather than an error, so the check archives a tree
by pattern with an ignore and asserts what it kept: a.txt in, sub/** out.
client/.npmrc is tracked now, so a token left in it from the old pattern is one
`git add -A` from being committed. Prose in CONTRIBUTING is not a control: the
pre-commit hook fails when the staged file holds _authToken, _auth or _password.
`npm run check:archiver` runs the directory-walk guard, so it is reproducible
before a push instead of only in CI, and the CI step calls the same script.
CONTRIBUTING now spells out the per-project installs, since the root .npmrc has
ignore-scripts=true and the composite install script therefore never runs.
unzipper was only there to read the test archive back. archiver emits an 'entry'
event per file it adds, and those names are exactly what the walk produced, so
the guard asserts on them and needs no reader. archiver stays: it is the package
whose directory() walk reaches readdir-glob, and the import has to be pinned to
the version the override installs.
Four write steps interpolated ${{ secrets.NPMRC}} straight into the shell source,
so a quote or $( in the value would break the step or run. They use the env form
the same workflows already use elsewhere.
CONTRIBUTING.md now says what the tracked client/.npmrc means for a developer:
run `npm run postinstall` after installing, `npx cypress install` for the suite,
and keep registry credentials in the user config rather than in the tracked file.
The release workflow runs cypress against the built app, and its install steps
now inherit ignore-scripts from the tracked client/.npmrc - so the binary has to
be installed explicitly there too, as build.yaml does.
unzipper is pinned to 0.12.5 so the exceljs override block no longer mixes exact
pins with a range.
client/.npmrc is tracked now, and every workflow appended the NPMRC secret to it
inside the workspace - one `git add -A` from committing a live token, and
semantic-release runs in jobs that have already written it. The secret goes to
$RUNNER_TEMP/npmrc and NPM_CONFIG_USERCONFIG points the installs at it, so the
project file stays policy-only.
The archiver guard used a fixed path in the shared tmpdir and had no handler on
the write stream: two jobs on one runner collided on the path, and a write
failure surfaced as an uncaught throw. Each run now gets its own directory, both
streams report, and the tree is removed afterwards.
The exceljs override pins archiver to 8.0.0 rather than ^8.0.0, so the version
the guard was validated against cannot drift.
The committed client/.npmrc turns dependency scripts off everywhere, not just in
build.yaml - and every one of these workflows builds the frontend, which imports
src/environments/version.ts. The lighthouse job failed exactly there:
Error: Cannot find module '../../environments/version'
The generator now runs after each client install: build (with cypress install),
lighthouse, sites, docs, release-assets and the three release installs.
Three things the last run exposed.
Every workflow that writes the NPMRC secret used `>`, which now overwrites a
tracked policy file - so the release, lighthouse and sites jobs would install
with hooks enabled and ranges unpinned, the opposite of what this PR adds. They
append.
Cypress downloads its binary in a dependency install hook, so ignore-scripts
leaves it out: "The cypress npm package is installed, but the Cypress binary is
missing". Both client installs now run `npx cypress install` after the
generator.
The archiver guard imports archiver and unzipper, which were only present as
overrides under exceljs; they are declared devDependencies at their resolved
versions so the script owns what it imports. Both remain production packages
because exceljs still requires them.
Dependency install hooks now stay off in client/, as they already are at the root
and in sas/: client/.npmrc is committed with ignore-scripts and save-exact, the
root install script runs `npm run postinstall` after installing, and both CI
installs pass --ignore-scripts and then run the generator by hand. The client's
own postinstall is what creates src/environments/version.ts and the hot icons, so
it has to run - it is third-party hooks that do not.
CI writes client/.npmrc from the NPMRC secret; the second job now appends rather
than overwrites so the tracked policy survives (the first already appended).
The archiver guard reads the archive back instead of checking its size: a matcher
that takes the top-level file and misses the nested directory clears any size
threshold, and that is exactly the failure it exists to catch. It now asserts
a.txt and sub/b.txt are both present.
archiver 8 is ESM-only, so require('archiver') resolves through require(esm) and
that exists from Node 22.12 - a supported Node 18/20 would fail the check for a
reason that has nothing to do with the minimatch override. The script is .mjs and
imports ZipArchive.
The cut already tells the two payload shapes apart - the doubled separator means
mp_abort escaped the message, the single form means it did not - so the unescape
is gated on the same decision. Unescaping a payload that was never escaped
collapsed a genuine pair of backslashes, which is a UNC path or a regex in a hook
message, and the comparison then ran against text the hook never raised.
The over-long message cut clamps the room to zero first: a prefix long enough to
fill the buffer gave SUBSTR a negative third argument, which errors the step and
cancels the program the helper exists to explain.
programvar=/statevar=/msgvar= names that do not resolve now say so with a
WARNING, instead of reaching the abort text as a blank.
Two cases: an unescaped payload keeping its double backslashes, and a message at
the 32767 limit being cut rather than overflowing. The trailing-backslash case
now uses the fully escaped form, which is what the producer emits.
The client's postinstall generates src/environments/version.ts (and the hot
icons), which the app imports - with --ignore-scripts the build fails with
"Can't resolve '../../environments/version'". My local check missed it because
version.ts was already on disk from an earlier install.
So the client keeps its install scripts; the archiver guard stays, which is what
that commit was for.
The minimatch override forces a major above readdir-glob's own range, and
archiver's directory() is the only runtime caller of that path. scripts/
check-archiver.cjs archives a small tree and fails if the result is too small to
hold it, so the override is guarded by a run rather than by the install
succeeding.
client/.npmrc is gitignored - it is where a developer's registry credentials
live - so the repo-wide ignore-scripts policy is passed on the client install
steps instead. Verified locally: npm ci --ignore-scripts, the archiver check and
ng build all succeed.
Cutting on the marker plus a newline still truncated a message that ends a line
with "Log Extract:", and an abort whose message merely prefixed the expected one
could pass. The appended extract always emits '\n\nLog Extract:\n', so the anchor
is that whole string (doubled form first, single as fallback): a message cannot
reproduce it by accident, and nothing is cut when neither form is present, so an
unexpected layout fails loudly instead of shortening the text.
The tail strip goes with it - the leading newlines are part of the anchor - and
the test for the doubled separator now writes the fully escaped form the estate
actually emits.
mpe_abortmessage cuts an over-long message to the room left after the prefix
instead of losing its tail to the 32767 limit, and mpe_execute's fallback skips
the trailing blank when the child's message is blank, matching the helper.
The %let could never be used: it ran immediately before an unconditional call
to the helper, which overwrote the global, and on a build without the helper the
unresolved macro call does not fall through to it - the parameter text lands in
open code and the step breaks before the abort fires.
%sysmacexist now chooses between them: the helper when it is on the autocall
path, the %let when it is not. The %global stays, so both branches write the
symbol that %superq reads.
Test datasets and filerefs are numbered to match their test, so a failing
assertion can be traced back by name.
Same reasoning as the minimatch pin: two ranges in one override object, one of
them caret, means the block can still float. Exact 5.0.12, which is what the
lockfile already resolves.
An override exists to stop a later install moving the package it pins, so a
caret range defeats it: the lockfile records 11.0.3 and the range would allow
11.0.4 to arrive without review. Exact version, and the lockfile is unchanged -
it already resolved 11.0.3.
Three of the four review points; the fourth is a producer problem, explained in
the reply.
The marker is only a separator when a newline follows it, so the cut anchors on
marker+newline (doubled form first, single as fallback for a response assembled
without the escape) and takes the FIRST match. A message that mentions the
marker is not followed by a newline and survives; a log line that contains the
marker can no longer move the cut past the separator and leave extract residue.
New case: an abort whose log mentions the marker.
mpe_abortmessage reads program and state into their own variables before
building the prefix: symget() nested in cat() is typed $200 by the data step, so
a long endpoint name was truncated before the assignment.
The header no longer claims byte-for-byte retention: symputx strips leading and
trailing blanks.
Not changed: decoding \n in the message. mp_abort escapes the append as
cats(msg,'\n\nLog Extract:\n',logmsg) and then doubles every backslash, so the
newline arrives as \\n\\n - un-doubling leaves the two characters \ and n, and
the only way to turn those into a newline is also the way a literal backslash-n
in the message becomes one. That is a producer-side decision, not a consumer one.
With follow-redirects moved, the client audit still failed on minimatch
<11.0.1 (globparts amplification), reached through
exceljs -> archiver -> readdir-glob, which asks for ^10.2.2.
readdir-glob already carries an override block here for brace-expansion, so the
fix follows that shape rather than a breaking exceljs change: minimatch
^11.0.1 nested under readdir-glob. package-lock gains 23 lines, the override
line is the only package.json change.
Verified: npm audit --omit=dev exits 0 in client; ng build succeeds; an
exceljs xlsx write succeeds; and archiver ZipArchive.directory() - the caller
that walks the tree through readdir-glob - produces a valid zip.
The audit runs in root, sas and client, and sas carries its own lockfile - the
same 1.16.1 was resolving there, which is why the first attempt left CI red.
Same one-package update, no pin change.
`npm audit --omit=dev` in sas is now clean.
The audit gate fails on any runtime advisory, and this one arrived upstream:
follow-redirects 1.16.0-1.16.1 discards cross-realm sensitiveHeaders arrays and
lets custom credentials follow redirects. 1.16.2 is outside that range and inside
the ranges the tree already asks for, so this is a lockfile-only update - no pin
in package.json changes.
Regenerated with `npm update follow-redirects --package-lock-only`, which moves
that one package rather than re-resolving every floating range (an
`npm audit fix` run also bumped the whole @angular-devkit family, which is not
part of this fix).
`npm audit --omit=dev` in client: the high severity entry is gone. Four moderate
advisories remain, all through exceljs -> archiver -> readdir-glob -> minimatch,
and they need a breaking exceljs change; they are pre-existing and not touched
here.
Two of the three review points.
The fallback %let now carries %superq(msg) as well, so a build without the helper
on the autocall path still reports the child's own message rather than only the
program and state. The whole value is masked and consumed through %superq at the
abort, so the comma hazard does not come back.
A case with the doubled separator - the form the estate actually emits, six
characters once the backslashes are doubled - now covers the branch that all the
other cases, written single-escaped, could not reach.
The quote case is not fixed; see the reply.
Two comment blocks sat back to back above the same call - the click handler's
and the anchored label's. They describe one selector, so they read as one note.
Three review points.
The fallback %let now follows %global _mpe_abortmsg. Without it the %let created
a local of that name in mpe_execute while the helper wrote the global, so on the
first failed call in a session the fallback shadowed the child's message - the
symptom this branch set out to remove.
mpe_abortmessage takes names for all three inputs (programvar/statevar/msgvar),
so a comma in an endpoint path cannot split the parameter list either. The test
that passes a path containing a comma now covers that, alongside the message.
mpe_assert_abort unescapes outside the marker block: a response that carries no
appended extract is still escaped, and leaving the unescape inside the block made
the comparison fail on text that was correct.
Full suite 64/64, mpe_execute and mpe_assert_abort PASS, sasjs cbd first.
Three review points.
The table id was sliced out of the stage URL without trimming a trailing slash,
so a route rendered as /stage/<id>/ produced an id that never matched the button.
Trailing slashes are removed before use.
The options menu entry is matched on the whole label, /^\s*Download\s*$/, because
an entry like "Download web query" would make cy.contains ambiguous and it fails
outright rather than picking one.
The getddl guard covers both hosts: libref is read behind a typeof too, so a call
with either variable out of scope falls back instead of throwing.
Two review points.
The strip looped over trailing \n and \r escapes, so a message that itself ended
in an escaped newline lost that escape along with the separator and the exact
comparison failed. It now removes one fixed tail: the six characters the
separator becomes once mp_abort doubles the backslashes, or four if the extract
is not doubled. Anything the message itself ended with is left for the unescape.
mpe_execute sets _mpe_abortmsg before calling mpe_abortmessage, so a build where
the helper is not on the autocall path still aborts with the program and state
rather than an empty message.
Full suite 64/64, mpe_assert_abort PASS, sasjs cbd before the run.
Three review points.
attachExcelFile read the body immediately after attachFile, so a slow upload made
abortModal false and skipped the abort branch - the failure then arrived later as
the timeout the branch exists to avoid. It now waits for either the confirmation
button or the abort modal title, with longerCommandTimeout, and branches on which
one matched; the confirm click uses the same timeout.
getddl guards ds with typeof: the truthiness fallback only covers a ds that is
bound and empty, while a missing ds in scope throws before the fallback runs.
The submitted row test 1 leaves behind is now documented in both places. The CI job
deletes the appLoc and re-runs makedata before the specs, so it does not accumulate
between runs; a hand-run against a reused estate leaves one pending row per run, and
the selector is anchored on the id the test created, so a stale row changes nothing.
Three things, one of them a bug the review question uncovered.
- msgvar is passed explicitly at the call site, so renaming the message variable
fails loudly instead of quietly producing an abort text with no child message.
- program and state are read with symget rather than interpolated into the data
step, so a % or & in an endpoint name can no longer be macro-resolved, and a
quote cannot break the literal.
- _prefix is trimmed before the message is appended. Declared as a character
variable it arrives padded to length, so the child's message was landing past
the buffer end and disappearing - the wrapper text was right and the message
was simply not there. That showed up as the comma and blank cases failing while
the "names the program state" case passed.
The abort branch asserted the modal title did not include 'Abort Message', so
any other title let the run continue with the abort modal open and the next step
died on a timeout that never mentioned the upload. It now throws with the title,
which points at the upload as the cause.
Three review points.
attachExcelFile read '.clr-abort-modal .modal-title' unconditionally, so an
upload that raised no abort modal failed on the lookup, and the abort branch
then left the modal open for the next command to click through. It asks the body
first: no abort modal means confirm the upload; an abort modal means assert its
title is not an Abort Message and stop there.
openDownloadModal clicked the deepest element holding the text. The click
handler sits on the div[clrDropdownItem], so the item is selected by that
attribute and its text.
The spec list comment claimed download leaves nothing for later specs to trip
over. It seeds the approval it needs and that approval stays, which the comment
now says.
mp_abort appends the extract as cats(msg,'\\n\\nLog Extract:\\n',logmsg), which is
now stated where the strip happens. The strip removes whole \\n or \\r pairs, so a
separator that gains a carriage return does not leave a fragment glued to the
message; a message that ends with the marker and no appended extract is a known
ambiguity and says so in the comment.
stagedata.test.4: the comment above exp8 now records why the expected text has
no space after either colon - the child builds it as &mac.:&VAR_NM.:&reason -
so the spacing is not a formatting choice a later reader has to re-derive. The
change itself came with ebfd395 on this branch and is unchanged here.
Suite 64/64, mpe_assert_abort PASS, sasjs cbd before the run.
Two reasons the last CI run still had four failures.
test 1 timed out on label[for="file-upload"]. The label is a click target for a
hidden input; attachFile can drive the input itself, so the helper no longer
clicks anything and selects by the input's id, asserting there is exactly one.
That also answers the earlier review point about positional selectors without
reintroducing a click on a control that is not always rendered.
tests 5-7 timed out waiting for the DDL file. The mock names a DDL download
after the libref alone, while services/public/getddl.sas names it
<libref>_<ds>.ddl - which is what the viewer expects and what the spec looks
for. The mock now follows the service.
fixturePath carried a trailing slash that the template also supplies, so
attachFile was given /excels//regular_excel.xlsx. Also drops the stale duplicate
doc comment above attachExcelFile - the helper targets the editor's import label,
which is what the remaining line says.
Addresses the review on e2e9103.
- The marker is located from the end. A hook message may legitimately contain
the text "Log Extract:" and only the final occurrence is the one mp_abort
appended; taking the first cut such a message mid-sentence.
- The captured text is unescaped (\" -> " and \\ -> \) before the comparison, so
it is compared as the text the hook raised rather than as its JSON form. That
is also what makes a message ending in a backslash comparable at all: mp_abort
escapes it as two, and the strip now leaves the pair alone for the unescape to
collapse - it only removes \n pairs and a lone stray backslash that is the
start of an escape the marker cut in half. Requiring the preceding character
not to be a backslash is what keeps a real trailing backslash out of that
branch.
- mpe_abortmessage owns the name it writes, _mpe_abortmsg, instead of taking one
as a parameter, so a call can no longer overwrite a caller's global of the
same name. Each call replaces the previous value; the doc says so.
Two cases added to mpe_assert_abort.test: a message ending in a backslash, and a
message containing the marker itself. Both fail against the previous logic
(first occurrence, no unescape) and pass now. Full suite 64/64 on sas.4gl.io,
sasjs cbd before the run.
Second CI run put the spec at 2 passing / 5 failing, and the five failures were
one more stale assumption: cy.select() matches an option's text against the whole
label, and the labels carry a suffix - the option for the Excel writer is
"Excel (.xlsx)", not "Excel". Every format is now chosen by the option's value
(EXCEL, SAS, SAS_DDL, TSQL_DDL, PGSQL_DDL), which the template sets and which does
not change when a label gains or loses a qualifier.
Also the review points on this PR:
- attachExcelFile starts from label[for="file-upload"] instead of
.buttonBar button:last-child, and submitExcel no longer takes a callback that
nothing passed.
- test 1 asserts on the download button whose id is the table it submitted: the
stage URL carries the id, so the assertion names its own row and asserts one
match, instead of indexing into every match on the screen.
- the comment in cypress-specs.sh no longer claims the spec depends on approvals
left by earlier specs - it seeds its own - and says why it is appended.
Addresses the review on 5d32de9.
The text mpe_execute aborts with is now built by mpe_abortmessage, which takes
the child message by variable name and writes the result to the global symbol
table. Two reasons: a value argument would be split at the first comma before
the macro ran, which is the defect this PR is about, and having the assembly in
a macro makes the comma case testable - a test calls it directly and asserts the
text, where going through mpe_execute needs a second service aborting to observe
the call the caller itself is aborted by.
- mpe_assert_abort: a lone trailing backslash is no longer stripped. Only
whole escape sequences are removed (\\n, \\\\n, \\\\), so a message that ends
in a backslash keeps it.
- mpe_assert_abort: the strip is skipped when "Log Extract:" is the first thing
in the captured text, instead of relying on SUBSTR with length 0.
- mpe_assert_abort: both sides of the comparison go through %superq, so a child
message containing %, & or a quote is neither rescanned nor able to break the
quoted comparison.
- mpe_execute: the message is concatenated, not CATX'd, so it arrives byte for
byte and a blank child message leaves no trailing blank.
- mpe_execute.test: three cases over the new macro - a comma-bearing message
survives, the text names the program and state, and a blank message leaves the
text clean.
mpe_execute 3/3 pass, mpe_assert_abort PASS, full suite 64/64 on sas.4gl.io,
sasjs cbd before each run.
The strip cuts the message at the "Log Extract:" marker, and got was declared
$256 - so a message whose text pushed the marker past 256 characters was never
stripped, and the exact comparison then failed on the escaped tail. got is now
the same width as line ($32767).
Covered by a new case in mpe_assert_abort.test.sas: an abort whose message runs
past 256 characters, asserted in full. It fails against the $256 buffer and
passes once got is widened, so the width is pinned rather than assumed.
mpe_assert_abort: PASS, full suite 64/64, rebuilt with sasjs cbd.
Three separate reasons the spec could not pass, all of them stale routes or
selectors rather than an overlay covering the button:
- test 1 opened #/approve/toapprove, which is not a route. The review module is
namespaced under review/, so the approve screen is #/review/approve. The dead
URL rendered the 404 page, and the only .btn.btn-success on that page is not a
download button - which is why the failure looked like a covered element.
- on the right route the approve grid still renders nothing on a fresh estate:
it draws a download button per table waiting for approval, and there are none
("There are no approvals remaining"). Test 1 now submits an excel fixture for
mpe_x_test first, the way stage.cy.ts does, so it owns the approval it
downloads instead of depending on what earlier specs happen to leave behind.
- the viewer's options menu is a list of div[clrDropdownItem] entries, not
buttons, so cy.get('clr-dropdown-menu button') never matched; Download is now
selected by its text, as filter-panel.cy.ts selects Filter. The dropdown-toggle
class the helper also required is gone from that element too.
The download button is selected by aria-label, which is unique, rather than a
bare .btn.btn-success that also matches the editor toolbar's small buttons.
Closes#398
Addresses the review on #412.
The prefix comparison is gone instead of guarded. mp_abort appends "Log Extract:"
and the surrounding log to every message it returns; the capture now cuts at
that marker and strips the escaped newlines before it, so the comparison is an
exact match again. That removes the hole the reviewer found - a blank expectation
can no longer pass, because %length 0 no longer short-circuits anything - and it
keeps the expectations strict, so a change in the child's wording still fails the
test rather than sliding past on a matching prefix.
mpe_execute: the assembled message declares its own length. An inline symget()
is typed by the data step default, which would truncate a long child message
before it reached the abort.
The expected literals in stagedata.test.4 are unchanged by this commit: with
exact matching restored, stagedata passes 5/5, which is the confirmation the
reviewer asked for - they are the child's text, not text adjusted to fit the
observed log. Full suite 64/64.
Both selectors in download.cy.ts are stale or ambiguous, which is why the spec
fails as soon as it runs:
- `.btn.btn-success` also matches the editor toolbar's buttons (`btn btn-sm
btn-success`), and those sit inside a fixed-position container on the approve
screen. The first match was one of those, so `should('be.visible')` retried
against an element that is never visible and the error read like the download
button was covered. The approve row's button is `btn btn-success` with
aria-label="Download audit file", so select it by that label.
- `.btn.btn-sm.btn-outline.filterSide.dropdown-toggle` cannot match anything:
the viewer's options dropdown is `class="btn btn-sm btn-outline filterSide"`,
with no dropdown-toggle class. filter-panel.cy.ts and row-cell-limits.cy.ts
already select it as `.filterSide`.
The spec goes back into the CI list, appended at the end because it needs
approvals to exist and the earlier specs create them. metanav stays excluded
(#395).
Closes#398
A child that does not complete is reported by aborting the caller with
"&program did not complete: state=&state &msg". The child's message can contain
commas - mp_abort appends a log extract to whatever the hook said, and the
derived-variable scan's own wording has one - so building that string inside the
macro call made the commas delimiters:
ERROR: All positional parameters must precede keyword parameters.
The caller lost the child's message and the abort fired with a corrupted text.
Wrapping the whole thing in macro quoting is not the answer either: %bquote masks
'=' and '/' as well, and the mask characters end up in the message the user
reads (they show up in work.mp_abort_errds and in the JSON response as control
bytes).
The text is now assembled in a data step and passed through %superq, which keeps
it as one argument without altering the characters.
Two follow-on fixes in the same path:
- mpe_assert_abort compared the returned message exactly, but mp_abort appends
"Log Extract:" and the surrounding log to every message it returns, so an
expectation written as the message the hook raised could never match. The
comparison now falls back to a prefix match, which keeps the exact-match
behaviour for the messages that carry no extract.
- stagedata.test.4: both abort-expecting calls pass failfast=0 (the second one
did not, so the test cancelled itself instead of reaching its assertion), the
expected messages match the scan's own format ("MPE_DERIVED:CALLER:..." with
no space after the first colons), and the expected text is passed through
%superq because it contains a comma.
Test 8 submits a VAR_LOGIC expression that the derived-variable scan refuses, so
the child is expected to abort and the assertion reads its message from the
response. Without failfast=0 mpe_execute aborted this program instead, so the
case could never reach its assertion. The file's other abort-expecting calls
already pass failfast=0.
This does not make the case pass on its own - the child still cancels without
writing a response, which is tracked separately - but it removes the wrong
failure in front of it.
mpe_execute reports a child that did not complete by aborting the caller with
"&program did not complete: state=&state &msg". The message was passed through
%str, which masks the literal text before resolution - so the commas in a
child's own abort message (mp_abort writes them as part of the reason) were
still delimiters when the call was parsed. The call split into extra positional
parameters and SAS raised
ERROR: All positional parameters must precede keyword parameters.
replacing the real failure with a parse error at the call site. The caller then
lost the child's message entirely.
%bquote masks the resolved value, so the message arrives as one argument and the
abort reports what the child actually said.
The same shape in mpe_security_postedit, where the submitted ACCESS_LEVEL value
is interpolated into the abort message, is fixed the same way.
All four pipelines downloaded `releases/latest/download/linux.zip`, unzipped it
and ran `api-linux`. They now start the published package at a pinned version:
pm2 start npx --name sasjs-server -- --yes @sasjs/server@3.6.2
and `sites.yaml`, which runs it under nohup, does the same with
`npx --yes @sasjs/server@3.6.2`.
Why: 10.7MB instead of 78.7MB, the same artifact a user installs, and a version
this repository names rather than whatever `latest` happens to be. `latest` also
moved silently on every run, so a red pipeline could not say which server
produced it. 3.6.2 is the first release whose bin carries a shebang; before it
`npx @sasjs/server` could not start at all.
The pin lives in `env.SASJS_SERVER_VERSION` in each workflow, next to the npm
cache step that makes repeat runs skip the download (`~/.npm`, keyed on the
version; `continue-on-error`, so a runner without cache support still runs the
pipeline).
The readiness loops and `RUN_AS=root` are unchanged - the package runs as the
runner's Node 24 rather than the Node embedded in the zip, and the loop is what
tells us it came up. Failure paths now dump `pm2 logs sasjs-server`, the name the
process is registered under.
Addresses both findings from the review of #408.
runAutoDeploy was declared async after the service-pack step went, but calls
createDatabase() and returns without awaiting anything. createDatabase() starts
the request and reports through the component's own status flags rather than a
returned promise, so there is nothing to await - drop the async rather than
pretend to a contract the callee does not offer.
runMakedataInNewWindow interpolated the context name, admin group and DC path
straight into the query string. An ampersand in any of them started a new
parameter and a space arrived truncated; a DC path containing a slash was only
working because nothing decoded the query. Those three values are now
percent-encoded at the build site, so the server reads back what the user chose.
Tests: the new-window case now asserts the encoded forms, and a second case
pins the values that need encoding (context with an ampersand and spaces, admin
group with a space and a plus, DC path with a slash and a space). The deploy
case no longer awaits a synchronous handler.
Executed 24 of 24 SUCCESS (automatic + sasjs-configurator + deploy components).
Restoring any load that touched a special missing numeric (._ or .A-.Z) failed
in mp_stripdiffs, which writes the old value into a generated UPDATE statement.
PUT writes a special missing as its letter only, so the statement carried
"set SOME_BESTNUM =_" and PROC SQL read the bare underscore as a column name:
ERROR: The following columns were not found in the contributing tables: _.
ERROR: SOME_BESTNUM, a numeric column, can only be updated with a numeric
expression.
Every statement in the generated block then failed NOEXEC, so the revert
produced the wrong data; the service's syscc check is what surfaced it.
Fixed upstream in sasjs/core#499 (the period is now written first) and
released as 7.0.1. v7.0.0's breaking change - mf_existvarlist warns and returns
0 instead of aborting on blank input - does not affect DC: there are no direct
callers, and the full SAS suite covers the core macros that do use it.
Adds restore.test.2.sas, which seeds some_bestnum=._ on mpe_x_test, changes it
to a number through stagedata and approves it, reverts that load, approves the
reversion, and asserts the cell is back to ._ rather than blank. It fails on
6.1.4 and passes on 7.0.1.
The setup screen's only action is to run `makedata` with what the form
collected, and nothing tested it. Both screens now have specs pinning the
request and the URL they build:
- Viya (automatic): the admin group and DC path reach the service as
`fromjs`, the configured context is used with the web method, a context chosen
in the form wins over the configured one, the new-window URL carries
`_contextname` / `admin` / `dcpath` / `_debug`, and pressing deploy reaches
`createDatabase` at all
- SAS9 and SASjs Server (configurator): the makedata URL carries `_program`,
`admin`, `path` and `_debug`; SAS9 stays on the page origin and encodes the
directory, SASjs Server leaves the path readable, and the SASjs Server check
runs before any reload
The deploy button is also restored. Removing the dead upload markup took the
live button with it, so the screen had no way to start the job its own copy
promises - the new `runAutoDeploy` test would have caught that.
The Viya setup screen still carried a path that deployed the backend as a
service pack from a viya.json (`deployServicePack`). That is no longer how a
deployment works: the frontend and services arrive with the sasjs deploy, and the
only thing left to do is create the control library and its configuration data,
which `makedata` does - and then deletes itself.
The path was already unreachable from the UI (its button had been commented out
"for reference"), so the wizard only ever ran `makedata` in practice while its
copy still claimed to deploy services. Removed:
- `executeJson()` and the `viya.json` upload plumbing: `readFile`,
`clearUploadInput`, and the fields and status flag that fed them
- the "Deploy SAS Jobs" row in the completion modal, which could never show
- the commented-out button and file input that kept the dead path visible
The screen now says what it does: it runs the one-off job that creates the
control library and its configuration data, and the status row reads "Create
control library" rather than "Create database".
The deferred VA load builds the filter from the report's parameters before the
first getdata, which is what makes that first load already filtered. A va_parmap
entry is a column the report author typed, so the request can legitimately fail
when the table does not have it - validatefilter aborts on an unknown variable -
and that abort surfaced as a modal over an editor which then loaded fine
(dc/dc#404).
Validating the mapping first would cost a SAS call on every load of a mapped
object, which is not affordable, so the attempt is simply made quietly:
saveQuery takes wrapper options and the deferred load suppresses the error
modal. A mapping that is right still filters the very first load; a wrong one
falls back to the normal load, and the metadata pass reports it in the banner
and reconfigures once, at setup time rather than on every page load.
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.
No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.
parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.
A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.
Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.
No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.
parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.
A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.
Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
@sasjs/core 6.1.3 moved the derived variable field rules out of
mp_derivedvars into mp_derivedvarcheck, so the same scan can run before a
definition is used. DC carried its own copy of that list in
mpe_derived_check.sas, which is now deleted: the post edit hook calls the
core macro instead, and the hook keeps only the rules that belong to the
control table rather than the expression - LIBREF/DSN required, and ACTIVE
within the picker pair.
The scan is stricter than the copy it replaces, in three ways:
- VAR_NM may not be _N_, _ERROR_ or _IORC_ (automatic variables of the data
step)
- VAR_LOGIC may not exceed 4000 characters
- VAR_LOGIC may not call a function that submits code or reaches the file
system, the environment or the macro symbol table - a function name cannot
be assembled at run time, so this is a control the expression cannot evade
and it relaxes one: VAR_LABEL may carry a line break, which is flattened to a
space when the label is used, so only the remaining control characters are
refused.
The mock hook mirror is updated to the same rules, and stagedata.test.4
gains the two submit-time cases that pin the new wiring.
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
### Bug Fixes
* **ci:** install doxygen for the SAS docs job ([f58925b](f58925b299))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](27686d3aad))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](888873421c))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](dd482c4069))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](25ffa71add))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](c5808a16eb))
* **ci:** rebuild and upload release assets on demand ([6486c80](6486c80ed4))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](af1813f829))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](cf7b21a8b6))
* **cypress:** hold download out too, and retry once in CI ([c97e886](c97e8866db)), closes [#395](#395) [#398](#398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](c803773d3c)), closes [#395](#395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](80e8911af8))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](e51953ca8a))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](459ff2da37)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](f7e753863b))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](7e3045a21b))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](2623a3fb4b))
* **embed:** log a parse-time mapping problem once ([0011f7e](0011f7e933))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](541a61ba08))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](e9216b0895))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](bd3c220af2))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](f83dc359e2))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](aa91daf9f7))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](10157c1159))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](411f6c3354))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](c0ea810766))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](15a1f34d52))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](88010c3980))
makeDataServer.json seeded ADMIN=AllUsers, a Viya-style group that no account on
the shared estate belongs to. The seed still reports SUCCESS with a group the
caller is not in, and then every getdata/stagedata call aborts with
'<user> is not authorised to edit <libds> in the MPE_SECURITY table', which
reads like a broken deployment. The estate's accounts are in all-users, the
SASjs Server group every logged-in user has, so seed with that and describe the
rule in the estate-testing skill rather than telling the reader to override the
shipped value.
The run on release/7.20 went from four failing specs to two:
csv-limited PASS (order restored)
hook-programs PASS (order restored)
metanav held out, #395
download 7 of 7 fail
download now gets past its hooks - the deleteFolder task is registered - and
fails on the button it clicks:
AssertionError: expected '<button.btn.btn-sm.btn-success...>' to be 'visible'
This element is not visible because its ancestor has 'position: fixed' CSS
property and it is overflowed by other elements.
Something fixed-position sits over it. That needs a scrollIntoView or an overlay
dismissed, and each attempt costs a forty minute CI cycle, so it is held out with
#398 carrying the exact error and the suggested fix. The list reports 22 run, 2
excluded.
editor.cy.ts still fails one test of fifty-five, and it failed the same way in
both runs: 'Expected <div.app-loading> not to exist ... continuously found', a
load timeout on test 28 rather than an assertion. One retry in run mode absorbs
that, none interactively; a real failure still fails twice and the retry's output
is in the log.
The shared-estate cycle is two npm scripts and sasjs test, but the seed in
npm run server is load-bearing: without it testsetup 404s settings.sas and
59 of the 64 test programs fail to complete, which reads like a broken
deployment. Record the targets, the token-pair credentials, the ADMIN group
the estate actually uses, how to read the results, and the frontend
passphrase steps as a repo-local skill, and point the testing doc at it.
Four specs failed on release/7.20 once the list went from 19 to 24. Three of the
four are addressed here; the fourth is a spec that cannot pass and is now
excluded with an issue.
download.cy.ts - it clears the downloads folder in an afterEach through
cy.task('deleteFolder'), and that task was registered only in
cypress/plugins/index.js, which Cypress 10+ never loads unless it is required
from cypress.config.ts. The task is now registered in setupNodeEvents, where
Cypress looks, and the login in its beforeEach only fills the form when the form
is actually on screen - the unconditional version failed with 'the center of this
element is hidden from view' whenever the session was already live.
csv-limited.cy.ts and hook-programs.cy.ts - neither is broken; both depend on
where they sit in the run. csv-limited asserts the free-tier 'Locked Feature'
modal, which an earlier spec removes by planting a valid licence, and
hook-programs asserts an approval row against its target table before other specs
have added rows to the approval tables. The list is now the sequence that was
green for months, with csv-limited first and hook-programs second, and the newer
specs appended rather than inserted. The script says why order matters, because
the next person to alphabetise it will break the same two specs.
metanav.cy.ts - cannot pass, and says so in its own source: '// BLOCKER ...
.clr-accordion-header-button always null although it is present on the page'.
With the duplicate login in its beforeEach removed it now reaches that assertion
and fails there, so it is excluded from the list with #395 as the reason, and the
login fix is kept for whoever picks it up. The coverage check now distinguishes a
deliberate exclusion from a forgotten file, so nothing can go missing silently.
Not verified locally: the specs need an estate seeded by 'sasjs cbd -t server-ci'
plus makedata, which a scratch mock does not reproduce - the app never leaves
app-loading. The CI run is the verification.
semantic-release bumps the working-tree package.json before the release
commit, so the tag is known in the job. Using it for the asset upload and
both dispatches means an overlapping release cannot redirect them.
Only the assets this workflow uploads are removed before a re-run, so a
hand-attached asset on the release survives. jq is installed before the
tag resolve that parses its output, the SASjs CLI is installed at the
version the tag's sas/package.json pins, and the npmrc secret is appended
rather than overwriting the file.
The four shards made the release-branch run red, and the reasons are in the
logs rather than in the theory:
- every failure was a timeout, not an assertion - "cy.type() failed because
the center of this element is hidden from view", "Expected
<div.app-loading> not to exist ... continuously found". Four jobs on one
runner means four SASjs servers, four ng serve and four Chromes competing
for the same CPU, and the app under test starves;
- upload-artifact@v4 does not work on this runner at all:
"GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ ...
not supported on GHES", so the v3 -> v4 move I made on review advice broke
the artifact step outright.
Measured, from the one sharded run that did complete: 20m34s wall against 33-35
minutes for the single job, with peak concurrency 3 - so a third off, not the
four-fold the split suggests. That is not worth a flaky pipeline while a release
sits on it.
Kept from the sharding work, because those parts were the real gain:
- the spec list has one home, .gitea/scripts/cypress-specs.sh;
- Build-and-ng-test fails when a spec on disk is not listed, so the five that
had never run in CI cannot go missing again - and they do run now;
- videos are zipped and uploaded only on failure;
- one job means one server, which is also what the guards and the balance
arguments were protecting.
The shard and fan-in jobs are gone with the branch that carried them.
curl -k disables verification on calls that carry the release token, so a
man-in-the-middle on the runner's egress could capture a credential that
rewrites releases. The forge serves a publicly trusted certificate (the
runner verifies the same chain on every checkout), so the flag is
unnecessary: drop it from all fifteen calls in the two release workflows.
The standalone URL dropped every query parameter whose name started with
va_, not just va_parmap; the names are now enumerated, and a later
parameter with that prefix survives (the component spec pins the exact
standalone URL instead of a substring, so the query surviving is the
assertion, not an accident of the runner's own query).
The timezone runs are scoped to the VA specs, the only specs with local
calendar arithmetic, so the blocking job no longer runs the whole suite
twice more.
The gate could not fail. With needs: on four shards and no if: always(), a
failed shard skips the fan-in job instead of failing it, and a skipped job is
not a failure on the one check branch protection watches - so the gate was
silent in exactly the case it exists for.
It now runs with if: always() and fails on any shard result that is not
success. The count is checked as well: if the needs context does not resolve on
this runner the step fails saying so, rather than looping over nothing and
reporting green - which is the failure mode a bare loop would have.
Also removes what the last regeneration left behind: the coverage guard step
appeared twice in Build-and-ng-test (so it ran twice) and the shard 1 comment
block appeared three times. The file is regenerated from fix/va-parmap-review-572
rather than from my own previous commit, which is what caused the duplication.
Renaming the shards to cypress-shard-1..4 and adding a job named
Build-and-test-development that does nothing but needs all four.
That name is the status check the branch protection rule requires, so the check
now turns green only when every shard passed - a red shard 3 leaves it
unreported and the PR unmergeable, which is the behaviour the single job had.
No repo setting has to change, and adding a fifth shard later means adding it to
needs rather than remembering to edit branch protection, where forgetting fails
silently.
Review found that five specs ran in no shard - coltype-delete-record,
download, embed-va, metanav, submitted-details - and one of them is the VA
embed suite this release added, 454 lines that had never run in CI. The old
single --spec string omitted them too, so nothing regressed; they were simply
never wired up, and nothing in the repo marked them skipped.
The spec lists now live in .gitea/scripts/cypress-shards.sh, one place, so the
workflow no longer spells them out four times. A guard step in
Build-and-ng-test compares the shards against the specs on disk and fails when a
file belongs to none of them or to more than one - proven by removing a name and
watching it fail with that file listed.
Rebalanced for the five additions, on test count and spec count together, since
each spec pays a few seconds of startup:
shard 1 editor, csv, metanav 57 tests
shard 2 excel, filtering, download, browser-info,
excel-multi-load, hook-programs 49 tests
shard 3 embed-va, viewbox, licensing, filter-panel, liveness 43 tests
shard 4 viewer-labels, row-cell-limits, derived-variables,
ddl-export, coltype-delete-record, submitted-details,
startupservice-launcher, stage, full-table-search,
csv-limited 27 tests
24 specs / 176 tests, each in exactly one shard.
upload-artifact moved from the retired v3 to v4; this change multiplies that
step by four, so it is worth not carrying the dead major forward.
Not in this commit: the three new check names still need adding to branch
protection, which needs repo admin.
Sits on top of fix/va-parmap-review-572 (#391) so the two changes to this
workflow land in one order; the timezone step that PR adds to
Build-and-ng-test is untouched.
The development job ran all nineteen specs in one Cypress invocation on one
runner, so its wall time was the sum of every spec - around half an hour on a
green run, with the whole pipeline waiting on it.
Four jobs now, each with its own runner and its own SASjs server, taking a
subset:
shard 1 editor (55 of the suite's ~148 tests)
shard 2 excel, filtering, csv-limited, csv, stage
shard 3 viewbox, licensing, filter-panel, browser-info, excel-multi-load, liveness
shard 4 viewer-labels, row-cell-limits, derived-variables, ddl-export,
hook-programs, startupservice-launcher, full-table-search
The split is balanced on test count, not file count: editor.cy.ts alone holds 55
tests, so an even spread of files would leave one shard running twice as long as
the others. Its own duration is the floor the pipeline can reach.
The setup steps are identical in every shard, so they are defined once and
aliased with a YAML anchor; each job differs only in SHARD and SPECS. Shard 1
keeps the original job name so an existing required status check still matches.
Video is still recorded - measured at 91s with it and 92s without on the same
spec, so the recording is not what takes the time. What it did cost was the zip
and artifact upload of every run's videos, now guarded by if: failure().
Review follow-up on the embed guard.
The spec asserted the isEmbedded getter, which mirrors the template rather
than exercising it: dropping !isEmbedded from the @if guards left every test
green. The spec now renders the component in a TestBed and asserts the DOM -
#TCS present standalone and absent for embed=va and embed=true, and the
deploy section withheld after the terms are accepted when embedded. Both
guards were mutation-checked by removing them, which fails the spec.
deployUrl and onNavigateToHome computed the standalone URL two different
ways: origin+pathname in one, location.href.split('#')[0] in the other,
which kept the query string, so a tab opened from ?embed=va#/deploy stayed
embedded. Both now call standaloneUrl() in standalone-url.ts, which parses
the href and drops the query and the fragment. It is tested with a
hash-bearing input, and the navigate test asserts the opened URL carries
neither '?' nor '#', so a regression to split('#') fails.
The self-destruct block ends with a single-statement %else in open code:
%if &keepme ne 1 %then %do;
%mv_deletejes(path=&root/admin, name=makedata);
%end;
%else %put NOTE: &_program retained, as KEEPME was requested;
Outside a macro SAS pairs %else only with a %do branch, so this stops with
'ERROR: Expected %DO not found' and 'ERROR: Skipping to next %END statement'
after the deployment has completed. On Viya the job then reports state error
with jobConditionCode 1012, so a successful deployment looks like a failure
exactly when someone is watching a first install.
Both branches now use %do/%end, which is the pattern the repo uses elsewhere
for this reason - see db/migrations/20261003_row_and_cell_limits.sas.
The same block is fixed in the server variant. The two delete calls also gain
their terminating semicolons: the macro call worked without them, but their
absence is what made the block look responsible for the error.
Behaviour is unchanged: the program still deletes itself unless KEEPME was
requested.
Deployment needs an authenticated, standalone session, and the person
looking at a report object is not the person who installs the app. Offering
the wizard there produces a screen whose controls cannot work - the failure
we hit while testing a report: T&C accepted, credentials filled, DEPLOY
clicked, and nothing happened, because the embedded app has no session to
deploy with.
An embedded instance (embed=va or embed=true) now shows what the state is
and where to deploy from, instead of a form that cannot submit. Standalone
behaviour is unchanged.
Two review findings, both about a filter that looks applied and is not:
- The transform branch of the clause builder interpolated whatever
convertJsDateToSasDate returned, so an Invalid Date reaching the helper
would have put a literal NaN in the WHERE clause. A non-finite result is
now dropped, and unsupportedTransforms reports the parameter, so the
author sees a banner instead of a broken filter.
- The range count check counted values that survived date parsing, while the
builder counts resolved bounds. A RANGE sent two values where one did not
parse therefore built a BETWEEN covering one period with no warning. The
check now counts what VA sent, the builder refuses a range with more than
one value rather than picking one, and a value set that is partly
unparseable gets its own line ('1 of its 2 values was not a date - the
filter covers only the rest').
The banner also now says what to do: it names the URL parameter to edit and
shows the entry grammar with an example, since the reader is the report
author and has no other way to act on it.
Conflict was valueSignature: the transform branch already builds a clause
value set through buildClauseValueSet, so the helper delegates to it and
takes the parameter's transform, keeping derived dates in the comparison.
Two review findings, plus a scope decision.
A va_parmap key is now matched against the parameter's label only. Matching
the internal name (pr273) as well meant one parameter answered to two keys,
so an entry written against the name and one against the label could never
be told apart: the label entry won and the name entry was silently dead -
no parse problem, no unresolved line, no unmatched line. Dropping the
fallback removes the ambiguity, and an entry keyed on the internal name now
shows up as an unmatched mapping like any other typo.
sameColumnMappings no longer reports two parameters on one column when they
carry the same value: col = 'x' AND col = 'x' filters exactly as one clause
does, yet the banner asserted that different values would show no rows. The
clause value each parameter produces is now compared, and the warning is
raised only when they differ. Parameters that produce no clause at all are
excluded, since they cannot conflict with anything.
A transform resolves a report value to dates using local calendar
components, so a test that passes in UTC says nothing about a browser west
of it. test:tz-west and test:tz-east run the whole suite under
America/New_York and Pacific/Auckland; with the UTC construction this test
pair replaces, the bare-number round-trip case fails under the first.
Not wired into CI: it triples the run for a property that is already
pinned by the round-trip assertion, so it stays a local and pre-release
check.
Brings in the diagnostics fixes from the base branch; conflicts were the
paramMap type widening to VaParmapTarget and the new paramsStable argument,
both of which the transform branch already carries.
Review findings:
- A bare SAS number was turned into a UTC instant and then read with local
accessors, so west of UTC the value landed a day early and the anchor was
computed from the wrong quarter; the round trip to a SAS day number was
one out. Built with local calendar construction instead, matching
parseDate and convertJsDateToSasDate.
- RANGE passed keepTime to both bounds, so a DATETIME range started at the
value's clock time on day one and ended at it on the last day, dropping
the rest of both boundary days. A range now covers whole days: period
start at midnight, period end at 23:59:59.
- A RANGE fed more than one parameter value produced no clause and no
warning, reading as a filter that matched nothing. One BETWEEN cannot
express several periods, so the count mismatch is now reported.
Review findings, all about a warning that misleads:
- The banner heading said the mapping was ignored, but two of the three
lists it renders are mappings that are applied (the AND-ed pair) or that
were never applicable (a parameter VA does not send). Neutral heading.
- unmatchedMappings reported a key as unsent as soon as any parameter had
been seen. VA sends the parameters a report object's query uses, so a
name that has not arrived yet looked like a typo. A mapping is now only
judged once a message introduces no new parameter name, i.e. once the
object's parameter set has settled.
- sameColumnMappings counted a column the table does not have, so one
mistake produced the unresolved-mapping line and the AND line. A column
that is not in the table is now left to unresolvedMappings.
- The Cypress cases asserted exact row counts, which encode the demo
table's contents. They now read the unfiltered count from the app and
assert the filtered grid is smaller (or empty), so a fixture change
cannot fail a behaviour test.
Subtracting a fixed day from the next quarter's start moves by an hour
across a daylight-saving transition, which would place a DATETIME bound
off the intended midnight. Day 0 of the following month is the last day
of the month by construction.
A VA parameter can now carry a transform on the column side of a mapping:
va_parmap=SELECTED_DATE-D_DATE:ADD(QUARTER_START,-1)
which filters D_DATE on the report value AND the same point one quarter
earlier, as a single IN clause - so the AND rule, the clause signature,
saveQuery and the reload loop are untouched.
Transforms resolve to one clause with these operations:
- ADD(<ANCHOR>,<offset>) the value plus one anchor beside it -> IN
- RANGE(<PERIOD>,<offset>) a closed period -> BETWEEN
Anchors: QUARTER_START/END, MONTH_START/END, YEAR_START/END.
Periods: QUARTER, MONTH, YEAR. Offset is a whole number of periods,
defaulting to 0.
The names describe the result rather than mirroring SAS intnx: the
arithmetic runs in the browser, so mirroring intnx would mean
reimplementing its alignment letters and their corners (week alignment,
's' and 'm') with no guarantee of matching SAS. A report author can read
ADD(QUARTER_START,-1) without knowing intnx.
Resolution is anchored on the parameter value, never on today, so a saved
report reproduces the same rows. A transform that cannot run where it is
used - a non-temporal column, a value that is not a date - is skipped and
reported in the existing warning banner rather than silently filtering on
the raw value.
`surfer deploy` chooses its target only from `--site-name`; without one it
replaces the app's DEFAULT site. The typedoc job passed `--server
webdoc.datacontroller.io` and no site name, so the 7.19.0 docs were published
over the default site of that app - the marketing site served at
datacontroller.io - instead of to a webdoc site.
Name the site, and give it its alias domain so the first run creates it.
The SAS docs job is left unnamed on purpose: code.datacontroller.io is its own
app whose default site is the docs, and a comment now says so.
The parse-time problems were logged at app init and then again by the
editor's warning loop, which spreads them into the banner's text. The loop
now logs only what the instance discovers for itself (unresolved column,
unmatched key, AND-ed pair); the parse problems stay in the banner and keep
their single init-time console line.
E2e case 11 pins it: one unparseable entry plus one valid-but-missing
mapping, so the loop certainly runs and logs its own line, and exactly one
line mentions the parse problem.
Two review findings, both paths where a filter could look applied while
producing nothing.
- `unresolvedMappings` judged a mapping against an empty column list while the
grid was still loading, so every entry was briefly reported as naming a
column the table does not have. An empty or unset `cols` now means "not known
yet" and returns nothing, so the banner appears only once metadata exists.
- Two parameters mapped to one column are AND-ed, so different values yield no
rows at all, and nothing said so. `sameColumnMappings()` now reports the
contended column and both parameters. The clauses are deliberately left as an
intersection: turning that AND into an OR is a filter-semantics decision, not
a diagnostic, so it is tracked separately.
Covered by unit tests for both (unloaded column list, one line per contended
column) and e2e case 10, which drives two parameters onto one column and
asserts the banner plus the empty grid.
Records that two parameters mapped to the same DC column are AND-ed, so the
result is the intersection: two different dates match no rows rather than
both dates' rows. Answers the 'filter one column on two dates' case from the
code rather than by assumption, and marks the single-parameter IN (...) shape
as the one that unions.
Three follow-ups from review, all the same class of defect: an author error
that leaves an unfiltered grid looking like a working filter.
- A duplicate key in `va_parmap` silently kept whichever entry came last, so
`va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER` filtered OTHER with nothing to
show for it. One VA parameter now maps one column: the first entry wins and
a conflicting duplicate is reported. The same mapping written twice is not a
conflict, and two different parameters mapping to one column stay valid -
that is how two report objects filter the same column from their own
parameters.
- A parameter-side typo (mapping `DROPDOWN_TAB1` when the report parameter is
`DROPDOWN_TAB_1`) produced no clause, no banner and no console line.
VaMessagingService now records every parameter key it has seen, across
editor reloads, and the editor reports a mapping key that VA has never sent.
Judged only once parameters are actually arriving, because VA sends just the
parameters its query uses - an absent key is not on its own an error, and a
key that turns up later stops being reported.
- The deferred first load cannot check a mapped column against metadata. On
SAS `mp_filtercheck` aborts the save, so the load falls back to unfiltered;
the banner now covers that path too, and an e2e case seeds the pre-bootstrap
message buffer to prove it (and to prove a valid mapping filters the very
first getdata).
A `va_parmap` entry that cannot be used was dropped in silence, in both
places it can fail: at parse time (no `-` separator, a missing side, or a
column that is not a plain SAS name) and at apply time (the mapped column is
not in the table being opened). Either way the parameter is skipped, which
degrades to "no filter" - a typo in the report's URL then looks exactly like
a working filter over the whole table.
Both are now reported in terms the report author can act on:
- a warning banner in the editor (VA mode) naming the parameter, the column
and the table, and pointing at the `va_parmap` entries in the URL
- console.warn for each distinct problem, so it is findable from the report's
console too
The parser returns `{ map, problems }` rather than a bare Map, VaFilterService
gains `unresolvedMappings()` for the entries the loaded table cannot satisfy,
and the editor merges both sets for the banner (logging once per distinct set
rather than once per VA message).
The banner is covered by two e2e cases, one asserting its painted box inside
the viewport (VA hides the app header, so it has to stand on its own) and one
covering a parse-time failure before any VA message arrives. The alert markup
keeps the message inside the alert's text flow so it wraps, instead of being
laid out beside a title and pushed off the right edge.
Two DC instances over the same table in one VA report could not filter the
same column from different parameters: parameter -> column matching is by
name, and two VA parameters cannot share a label.
The `va_parmap` URL parameter overrides that match per instance, one entry
per mapping (`va_parmap=<VA parameter>-<DC column>`, repeated for more):
...?embed=va&va_parmap=REGION_TAB1-REGION
The mapping is additive (unmapped parameters keep the name match), the
column is validated as a plain SAS name at parse time, and an entry only
applies where the named column exists in the table being opened - a miss
skips that parameter rather than falling back to the same-name match the
mapping exists to override. Entries split at the last hyphen, so a label
may itself contain hyphens, and the key matches the parameter label first,
then its internal name (pr###).
Parsed once at app init onto globals (like embed / vaApplyMode) so the
editor reloads a filter apply performs keep the mapping, and both filter
paths (the metadata build and the deferred initial load) honour it.
Also repairs embed-va.cy.ts: a cy.visit differing from the current URL
only in the hash is a same-document navigation, so ?embed=va never took
effect and both existing cases failed; their layout assertions also
described an earlier design (VA now opens read-only with an Edit button
and hides Filter). Cases 3 and 4 drive two mappings of the same column
through a real postMessage and assert the applied filter and row count.
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
### Bug Fixes
* address the review findings on the harness and refresh changes ([36e4e88](36e4e88eb1))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](698dea6d6b))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](0d58be975d))
* **client:** open the SAS log page with noopener ([072a133](072a133702))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](1fb4818d21))
* **cypress:** apply the licence key when the estate has none ([be8f445](be8f445ae1)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](fed43cb07e))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](f5aade5976))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](1e224763ba))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](9b3cf50b9c))
* **derived:** address five review findings ([ccd468b](ccd468b335))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](be50cb5d08))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](9914790777))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](9ef813d439))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](422f766cd4))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](7830b7bcfa))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](2d5de9a324))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](6a1128a463))
* **derived:** eight review findings ([561d7fc](561d7fc3ed))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](1863076bc9)), closes [#366](#366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](d3dec682aa))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](469229aecb))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](7fb981daa9))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](5f5d8681b9))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](5f443b0b16))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](6b5bf3e6be))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](85c89ca2a4))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](f8f65681e1))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](7efbf31ea5))
* **derived:** only the delete marker is a reserved name ([af31c36](af31c36b67))
* **derived:** open the statements file with a long record length ([ee31782](ee31782fbf))
* **derived:** pair the filterds %else with %do/%end ([88a38db](88a38db877))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](27ddcfb575))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](7799a300df))
* **derived:** reject a dependency on a row this submission deletes ([8806504](8806504967))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](9a047ad42e))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](e0da9f1098))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](688550a28d))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](1b5c4b3ecd))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](4d9ad38df1))
* **editor:** a blank truncation note is not a warning ([f182179](f182179403))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](04ce1861e7))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](4bcee706ff))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](45dce7b11b))
* **getdata:** clear the globals it declares before use ([359b570](359b570b85))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](059cdf176a))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](661ee19cfa))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](355c46a7b6))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](d626cb8f2d))
* **mocks:** address the audit-pack review findings ([489819b](489819b051))
* **mocks:** carry the zip flags into the central directory ([c4021a4](c4021a42ff))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](62f55db170))
* **mocks:** copy the physical column list before deriving ([f411b7d](f411b7d269))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](60ec470404))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](74060c0585))
* **mocks:** drop derived variables before staging ([73d25b1](73d25b1220))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](347dbb53b0))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](ee3c8787ef))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](6cad23fce9))
* **mocks:** search the physical columns only in the viewer ([293022d](293022d46b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](fb36379520))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](9c294c0ab5))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](0a52d20c5f))
* **mpe_execute:** fail fast when the child left no status ([a036386](a03638667f))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](220408538b))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](f2b1428ec3))
* name the startupservice launcher button "Show log" ([3445107](3445107d9a))
* offer the launcher on the globvars failure modal too ([ad2da45](ad2da45389))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](46bacdd6d9))
* **refresh:** one audit folder per refresh call ([56b6620](56b6620ef4))
* **sites:** round-trip the committed licence before publishing ([4d462d3](4d462d310f))
* **test:** getdata test ([da30266](da30266278))
* **tests:** enable closeouts logging ([fdcd8fa](fdcd8fa4b3))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](033ffd3025))
* **ui:** one error modal at a time ([81ccf75](81ccf758e9))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](42355f17bc))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](337ea8e5e0))
### Features
* **dc_getlibs:** dump the library list when debug is on ([582523d](582523dc87))
* **derived:** allow a variable to shadow a physical column ([12beeff](12beeff695))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](813bce6856))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](7c9ce9a712))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](87e794f769))
* **derived:** report the dependency cycle path ([cee1a95](cee1a954ee))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](78e88a4112))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](0b010f2c65))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](aa6d50e1ea))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](33d043ea92))
* **editor:** derived variables in the EDIT screen ([b0fd354](b0fd354ecd))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](6513d09f05))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](34691dceec))
* **editor:** show the derived column label in the header tooltip ([cfe6647](cfe6647ca8))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](752d944c20))
* **mocks:** implement derived variables in the JS services ([b4fae88](b4fae88ae1))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](5d183d1350))
* run startupservice in a new window from its error modal ([40f6133](40f61331aa))
* **sites:** mint a licence for the published sites ([7d9df71](7d9df71fdb))
### Reverts
* **derived:** no shadowing, and no self reference ([5af8e6f](5af8e6fc35))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](31aa859153))
Review feedback on the merged fix:
- derived_nobs was assigned twice, the first one dead - the coalesced form below
is the live value. Removed, so there is one assignment to read.
- making the catalog test a plain %if after the derived-variables test let the
catalog branch override it; the derived view won before and still should. The
catalog branch now requires derived_nobs=0, so the view keeps precedence and
only a catalog with no derived configuration is blanked.
Two guarded if blocks, not %else %if: that construct is what the parser rejected
in this file before ('Expected DO not found'), and the first attempt at this
review point reproduced it - validatefilter and viewdata both failed to complete
again until it was replaced.
Verified: sasjs test -t new -> 64/64 (100%), TEST=0.
The suite failed to complete on the estate with
ERROR: Expected %DO not found.
ERROR: Skipping to next %END statement.
and, once the parser had lost sync, 'Apparent symbolic reference IS_FMTCAT not
resolved' - because the %let that defines it never ran.
Two causes, both in the code that decides which dataset the filter is validated
against:
- the comment explaining the catalog case contained the text '%if ...' and the
macro processor resolves triggers even inside a comment, so the prose was
executed as code and tripped the parser from there. The comment now avoids
macro triggers entirely.
- an empty value compared in a condition is a character operand, and
%mf_nobs returns nothing when the dataset is missing. It is wrapped in
coalesce now, and the catalog flag is computed with %eval instead of a nested
%if.
targetds is still blanked for a LIBREF.CATALOGNAME-FC name - which is what makes
mp_filterstore rewrite it to a cntlout dataset - and stays the plain libds
otherwise, as an empty targetds falls back to libds in the engine anyway.
The test now passes outref= and mdebug=&sasjs_mdebug rather than a hard-coded
1: the suite already owns that switch, and testinit turns it on by itself when
the run asks for _debug=2477 or 131, which is when the child's response and the
PROC HTTP trace are worth having.
Verified: sasjs test validatefilter -t new -> 3/3 (100%), six assertions, all
PASS, including the catalog and derived-variable cases.
The Sites workflow was the one place that still started the server without a
listener, so it died with
[error] Invalid environment variable(s) provided:
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
and the following step's 120s poll timed out. build.yaml, release.yaml and
lighthouse.yaml already carry the same line.
refreshcatalog.test and viewtables.test failed to complete with
ERROR: could not create /tmp/DataController/<libref>/dc_staging/refreshcatalog:mpe_refreshcatalogs: + <stamp>
The staging folder a closeout writes its statements to is
&dc_staging_area/&etlsource, and etlsource defaulted to
'refreshcatalog:<step>:%mf_uid()'. A default is resolved when the macro is
compiled, not when it is called, so every refresh in a session reused the same
folder - the first step created it, the next one could not, and the abort took
the whole suite down. Whichever suite reached the staging area first decided
which others failed, which is why the count moved between runs.
etlsource now defaults to 0 and is set per call from
%mf_getuniquename(prefix=<step>): unique, a plain name rather than a path, and
resolved at the point of use.
Verified on the estate: refreshcatalog 4/4 (100%), viewtables 2/2 (100%).
derived-variables.cy.ts:97 expected dc-derived-first on the REGION header,
because the comment assumed REGION opens the computed block. The grid follows
MPE_DERIVED order, and that order is LINE_VALUE, REGION, REGION_LABEL - so the
marker was on LINE_VALUE and correct. The spec asserted the wrong column, and
the development pipeline has been red on it since.
Dumping the header row on a local mock estate (sasjs/server desktop mode,
RUN_TIMES=js) shows:
6 | LINE_VALUE | dc-derived-header dc-derived-first
7 | REGION | dc-derived-header
8 | REGION_LABEL | dc-derived-header beforeHiddenColumn htLastVisibleHeader
and the same dump against the unmodified release/7.19 client, so this was never
a client defect.
The assertion now states the invariant instead: at least one derived header, the
first of them opens the block, and no other one carries the marker. That holds
whatever order the control table returns the variables in.
The previous wording named only mpeinit and mpeterm, so the webout entry read as
an unexplained addition. webout is the linter's own default - sasjs compile
injects %webout into every service - and an explicit ignoredMacros list replaces
that default, so it has to be restated. 52 service files call %webout, which is
what dropping the entry would start reporting.
sasjs lint (@sasjs/cli 6.2.5) reported 24 warnings across the 11 .sas files
this release touches. None of them are in those files now:
- mpeinit and mpeterm are added to ignoredMacros in sas/.sasjslint. Every
service is compiled with a serviceInit program (sasjs/utils/
serviceinitsas9|viya|server.sas), and all three declare both macros in
their own header, which is how a service body's %mpeinit()/%mpeterm()
resolves without a declaration of its own. An explicit ignoredMacros list
replaces the default, so webout stays listed.
- Header entries nothing in the file calls are dropped: mp_binarycopy in
getrawdata (it arrives through mp_streamfile, which calls it - the
compiled program is unchanged), mp_jsonout in viewdata,
mp_assertcolvals in validatefilter.test, and mx_execute in testsetup
(every file that calls it declares it).
- Prose carrying "error"/"warning" is reworded. Where the text is log
output, %put WARNING: is written %put %str(WARN)ING: so the line the
log receives is unchanged.
- The server makedata service keeps its doxygen @warning tag - the tag is
the content - so that file carries @sasjslint {"noBannedStrings": false}.
- Two comment lines in getdata are re-indented to the file's multiple of 2.
AGENTS.md records the ignoredMacros rule and the %str(WARN)ING convention so
the next warning is not silenced by reflex. Compile passes; the 93 warnings
that remain are all in files this release does not touch.
The MARKDOWN branch of the service calls %mp_ds2md(), which ships with
@sasjs/core, but the header never listed it. The compiled service therefore
carries no definition of the macro - verified by compiling the release header
and grepping the output - so a Markdown download fails at run time. The viewer
offers "Markdown (.md)" in the download list, so the path is reachable.
Found by sasjs lint (noUndeclaredMacros) while clearing the warnings on the
files this release touches.
Review: '%if &outref ne 0' handed %EVAL the text 'derref5 ne 0', which needs a
numeric operand - so the catalog branch failed in exactly the case it was added
for, on every caller that passes a fileref. Quoted, the default 0 still skips
the file and a fileref creates it.
The test now proves both halves: fopen tells us the file was created, and a
read loop counted zero records tells us it is empty. A single INPUT is no
good - on an empty file it stops the step before the symputx, which is why the
first version recorded a FAIL.
The outvars placeholder keeps 'keep=name'; the comment records why that is the
whole column set: the engine builds the dataset in exactly two places and both
project NAME, and the only projection in the SAS tree is getdata.sas.
Review feedback: the first version left targetds out for every filter without
derived variables, which silently changed the plain-dataset path to whatever
mp_filterstore defaults to. It is now passed exactly as before unless the libds
is a format catalog - the one case mp_filterstore has to rewrite itself.
The catalog test no longer writes to the statement file: it reads one record
from it instead, so -1 means the file could not be opened and 0 means
mpe_derived left it empty, which is the claim the branch makes.
mpe_derived used &dc_dttmtfmt in its where= but the guard only checked libds and
dclib, so an unset value turned the filter into 'lt tx_to' and the control-table
step failed with a syntax error a long way from the cause. It is now part of
the guard, named in the message.
The 7.19 migration took max(selectbox_rk) into a macro variable, which is
missing on an install whose MPE_SELECTBOX is still empty - %eval(&maxrk+1) then
fails. coalesce(max(selectbox_rk),0) makes the first row start at 1.
A format catalog is addressed as LIBREF.CATALOGNAME-FC. Two places assumed a
dataset:
- mpe_derived passed the name straight to mp_derivedvars, whose existence check
aborted with 'Target table not found: DCTEST.DCFMTS-FC' - so validatefilter,
getrawdata, viewdata, getcolvals and stagedata all failed on a catalog. A
catalog has no observations, so it can have no derived variables: return an
empty variable list and an empty statement file, and leave the view alone.
- validatefilter then passed that same -FC name as targetds to mp_filterstore.
The engine handles a -FC libds itself (it rewrites it to a cntlout dataset),
but a targetds it is handed is used as given, so mp_filtercheck built a step
against a name it could not open. targetds is now only passed when the
derived view is actually in use.
mpe_derived.test gains a case for the catalog path: no derived variables, no
generated statements, and no abort.
- mpe_assert_abort.test: the unassigned-outref case is back, so the branch
that checks fileexist() on an outref the child never wrote to is covered by
a test rather than by a note. It runs: the four cases pass on the estate.
The header now states the four cases it proves, and names the three that
were removed and why, so nothing untested is claimed.
- mpe_execute.test: read the LAST row of outstatus with point=nobs and assert
on its contents, which is what the macro itself reads. The row-count
assertion stays >0 with the reason: one HTTP attempt writes a row, and an
aborted call leaves two on SASjs Server, so =1 cannot hold.
- Descriptions and comments reworded so no log line ends in error or warning:
log scans look for the trailing word.
The hover text on a derived column said only that the column cannot be edited,
which tells the user nothing about what it holds. It now carries the column
label - the configured VAR_LABEL, or the generated name=expression that
mp_derivedvars writes when the administrator left the label blank - and falls
back to the plain rule when the label merely repeats the column name.
The cypress case that marks derived columns apart now also checks the tooltip
for both shapes: REGION_LABEL carries a configured label, REGION falls back to
the generated one.
Submitting a rejected MPE_DERIVED row showed the SAS abort message, and then a
second modal on top of it reading
Submit error
SAS Service: editors/stagedata
Submit request failed
undefined
Two separate faults. sasService.request already shows the abort message when
the service returns a sasjsAbort, and the adapter error when the call fails -
then it rejects, and the editor's catch showed its own modal for the same
failure. That second modal read 'undefined' because it looked for
'adapterRespnse' (typo), a property that does not exist, so the text it built
was JSON.stringify(undefined).
- sas.service.ts marks a rejection 'errorShown' when it has already told the
user, on both paths that display a modal.
- the editor's catch now only reports a failure when nothing was shown, and
reads err.adapterResponse.error when it does.
- app.component.ts replaces the open abort modal instead of pushing another
onto the array: a second emit stacked a modal over the first, so one failure
needed two dismissals, and the lower one was usually the duplicate.
LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_LEN, VAR_LOGIC and ACTIVE are NOT NULL in the
table definition, so the requirement is stated once, in the model, instead of
being repeated as NOTNULL rules in MPE_VALIDATIONS for each column.
VAR_FMT stays nullable: it is optional on a derived variable, and its HARDREGEX
already accepts an empty value.
The seven NOTNULL rule inserts are removed from the 7.19 migration, and the
comment on ACTIVE no longer describes a rule that is not there.
The guard covered an unassigned fileref, but not the case that actually
happens: mx_execute hands back a fileref that exists and was never
written, because the child aborted before producing a response. Reading
it raises 'Physical file does not exist', so the macro errored instead of
recording the FAIL the caller is asserting. Check fileexist on the
fileref's pathname as well.
mpe_execute.test asserted one status row per call. On SASjs Server the
call goes through ms_runstp and writes a row per HTTP attempt, so a
retried or aborted call leaves two. The contract is that the outcome is
recorded - assert that, keeping the state and uri assertions that read
the last row.
mpe_assert_abort.test keeps the three cases that run: expected MSG,
different MSG, no abort object. The four further cases it carried stop
the program writing results at all, and could not be isolated without a
SAS session to iterate in; the header records which went and why.
The picker comes from the MPE_SELECTBOX rows - the client builds the dropdown
from that data and the SELECT rule types only set whether a value outside the
list can be typed. NOTNULL is what makes ACTIVE mandatory, and the post edit
hook is what refuses anything other than 0 and 1, so the HARDSELECT row was
saying the same thing a third time.
Drop it in both writers, and drop the comment pointing at the migration: the
two files are read side by side anyway.
MPE_DERIVED is maintained through the EDIT screen, but LIBREF and DSN were free
text - a typo in either silently produced a rule that never fired.
LIBREF now offers the editable libraries and DSN the tables in the LIBREF on the
same row, through the same SOFTSELECT_HOOK services the other configuration
tables use (libraries_editable, tables_editable). VAR_NM stays typed: a derived
variable names a column the table does not have yet, so a list of existing
columns would be the wrong control. ACTIVE becomes the fixed pair 1 and 0 from
MPE_SELECTBOX, with HARDSELECT so it cannot be typed freehand, and its NOTNULL
rule carries '1' - the value the EDIT screen pre-fills on Add Row
(getColumnDefault), because a derived variable only takes effect when active.
The rows are written by mpe_makedata.sas for a fresh install and by the v7.19
migration for an upgrade, and each names the other so the pair cannot drift
unnoticed; the ACTIVE guard itself lives once, in mpe_derived_check.sas.
The mock hook carries the ACTIVE rule, as it carries every rule
mpe_derived_check.sas enforces.
getdata returns TRUNC_MSG as a single blank when the selection was not
truncated: SAS has no zero-length character value, so mp_jsonout's trim of a
blank $200 leaves one blank and it serialises as a single space. The editor
assigned it straight to truncationNote, and a space is truthy in JavaScript, so
the EDIT screen showed the orange truncation warning against the table name and
its tooltip contained one space - a dark bubble with nothing in it.
Trim the note before it is tested.
Same rule as the 7.19 hook (PR #366): ACTIVE is NOTNULL + HARDSELECT in the
EDIT screen, so the field checks must refuse a missing value or anything other
than 0 and 1 - otherwise a direct stagedata submission stages a row the picker
would never have allowed.
The fixture gains the active column (1 by default, 2 for BADACTIVE) and the
count of reported rows goes from five to six.
getcolvals - mf_nobs returns -1 when the dataset does not exist, so the guard
tested ==0 and a request that supplied no iwant at all fell through to the SQL.
It now tests <=0, which covers both an empty and an absent input.
mpe_refreshcatalogs - etlsource defaulted to a bare uid while its two siblings
use refreshcatalog:<step>:<uid>, so a load through this path could not be traced
to the step that wrote it. The default now has the same shape, and the explicit
etlsource=0 path follows it too.
mpe_execute - the status table was read without being cleared, so a child that
died before core wrote a row left the previous call's row in place and failfast
read it as completed. The table is deleted before the call.
makedata - KEEPME is compared through cats(), so a numeric column no longer
raises in upcase().
The statements written into the catalog ended without a semicolon, so the
%INCLUDE that ran them ran on into the caller: the parser finished the statement
with the text after the include and tried to include it, producing
'Cannot open %INCLUDE file PROC' immediately before the next proc sql, and
refreshlibinfo aborted with syscc non-zero.
The three refresh macros are only reached through those generated statements, so
the false 'declared but not used' warnings are switched off with a @sasjslint
block rather than by dropping the declarations.
getddl requires a dataset - it aborts when the member cannot be found, and only
treats a format catalog specially - so a request with ds= always aborted and the
assertion read a stale webout. Test 1 now names MPE_TABLES and checks that the
TSQL header names it, which keeps TSQL coverage on a request the service accepts.
Test4 filtered only on the derived column, so it returned the same DSN once per
libref, including transient session librefs. The filter now pins LIBREF as well,
built in a data step because datalines leave &dc_libref unresolved, and the
expected derived value uses MPE_LOCKANYTABLE - the name makedata registers.
mp_assertcols tests variable names, so it can never find a DERIVED_NOTE row in
the cols payload - the assertion now counts rows instead. With debug on, both
Test4 payloads are printed so a wrong payload can be read rather than guessed.
An apostrophe in a %put breaks macro compilation on this platform: the statement
ends early and the next macro is left unterminated, which showed up as 'Macro
keyword LET appears as text' several hundred lines further down.
libds and col2 are only populated once validation passes, so the
"Missing inputs from iwant" abort - which tested those macro variables - fired for
any invalid payload and hid the specific reason. It now tests whether the
request carried rows at all, leaving Invalid libds / Invalid col to report the
value. The service still refuses the same requests; only the message changes.
The matching tests carried two faults of their own: getcolvals.test.4 sent its
libds through single quotes so the libref itself was invalid, and getdiffs /
getrawdata expected "Invalid table" for cases their own field names.
SAS 9 has no job state: core's SASMETA branch writes state='unknown' with
'proc stp returns no job status' on a call that succeeded. The fail-fast
change would have turned every converted test on that platform into a failure,
so unknown is noted rather than aborted on. A missing status table, a blank
state, canceled and failed still stop the caller.
The macro test asserted on uri, which the SAS 9 status row leaves empty; it now
asserts the platform column, which every branch populates.
mpe_assert_abort decides PASS/FAIL by scanning response text, and a mistake in
that scan either fails a genuine abort or passes on any sasjsAbort string in the
response; nothing reported it. Its test builds the responses as files: the
expected MSG, a different MSG, no abort object, several MSGs where the first
belongs to the abort, an object split across lines, an abort past 32767
characters, and an unassigned fileref.
mpe_execute is covered against a child that aborts for a known reason: one
status row per call, the state named and not completed, the uri recorded, and
the child log kept only when mdebug asks for it.
loadtype, var_txfrom, var_txto, var_processed, filter_text, pk, coltype, sortpk,
var_busfrom, var_busto, rk_underlying, buskey and pre_edit_hook are declared
%global but were never initialised, so in a reused session a code path that
skipped an assignment applied the previous request's value - a stale
pre_edit_hook or buskey against the wrong table.
The KEEPME probe did 'set work.fromjs' inside the varnum guard and then stopped,
so only the first observation was inspected: a submission that set KEEPME on a
later row still self-destructed. It now reads to end of file.
makeDataNextViya.json ships KEEPME=Y and is the config that seeds a real Viya
environment, so the default posture there was 'the seeding program stays on the
server'. KEEPME stays in makeDataSas719.json, which exists for the test
harness, and is removed from the environment seed.
Three defects in the scan:
- 's/.*"MSG" *: *"([^"]*)".*/$1/' is greedy, so a response carrying more
than one MSG compared the last one rather than the abort's. The pattern is
now matched with prxmatch and the capture read with prxposn, which returns
the first match at or after the sasjsAbort marker.
- the whole response was concatenated into $32767, so a longer response
truncated silently and an abort near the tail was missed. The file is now
scanned a line at a time and the scan stops at the first MSG.
- 'infile &outref' had no fileref check, so a child that died before writing a
response raised an unassigned-fileref error instead of recording a FAIL.
failfast fired only on state=canceled or state=failed, and read &outstatus
unguarded. A child that never reached the runner leaves no status table, and
a status row with a blank state left both macro variables blank - either way the
test carried on against whatever the previous call left in outlib, which is the
inference this wrapper exists to remove.
The status table is now read only if it exists, a missing table is named as
such, and failfast aborts unless the state is completed. A status row that is
absent and one that is blank both fail.
The release branch gained 36e1689, which fixed the same four security tests
(getcols, getcolvals, getrawdata, validatefilter) by introducing
dc_assert_abort. This branch had already fixed them with mpe_assert_abort and
mpe_execute, and mpe_assert_abort is the wider of the two: it also checks the
expected MSG, and the calls pass outstatus/failfast so the child cannot be
inferred from a stale outlib. Kept the harness versions and dropped
dc_assert_abort.sas, which nothing references after the merge.
SASjs Server 3.x deprecated PORT and PROTOCOL and refuses to start when an
.env names neither HTTP_PORT nor HTTPS_PORT, exiting with
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
The three workflows downloaded releases/latest and wrote an .env without a
listener, so api-linux never bound :5000 and the 'Setup and start SASjs
server' step failed after its 120s wait (build runs for #363 and #365, both
failed there). Add HTTP_PORT=5000 to build.yaml, release.yaml and
lighthouse.yaml, and document it in the local-dev .env in CONTRIBUTING.md,
where the default-port claim is no longer true.
Derived variables are read-only, but they were rendered exactly like any other
read-only column: dimmed via readonlyCell, with no indication that the column is
computed at read time rather than locked by a rule or by column level security.
The grid could not tell the difference either - getdata holds a derived column
read-only with a READONLY DQ rule, which is also a rule type a table can be
configured with.
getdata now names the derived columns in sasparams.DERIVEDVARS (space separated,
upper case, from MPE_DERIVED's outvars). The EDIT screen holds that list, and
marks those columns with their own classes instead of the read-only dimming:
a tinted header with an 'fx' marker and a tooltip, tinted italic cells, and a
left edge on the first derived column so the computed block reads as a group.
Blue is the info family, leaving the primary key green and the warning ochre to
their existing meanings. Light and dark values are both set.
The SASjs mock getdata.js returns the same field, so the Cypress suite exercises
the styling; the existing derived-variables spec asserted readonlyCell on the
derived header, which is now the fallback for a plain read-only column, so that
assertion moves to dc-derived-header and a physical column stays the control.
Lint: prettier --check clean on the touched client files; sasjs lint reports no
new warnings for getdata.sas.
The harness pins @sasjs/core exactly (6.1.0) so the tests can read the
outstatus table, but the lockfile still resolved 6.0.0. 'npm ci' refuses to
install when an exact pin and the lockfile disagree, which failed the mocked
deploy in both build and lighthouse ("lock file's @sasjs/core@6.0.0 does not
satisfy @sasjs/core@6.1.0"). Regenerated with 'npm install
--package-lock-only': the only change is the core entry.
MPE_DATALOADS is keyed on (processed_dttm, libref, dsn, etlsource), and etlsource
was the calling macro name, so a second catalogue load of the same table in the
same second was refused as a duplicate. The three refresh macros now take an
etlsource parameter, defaulting to refreshcatalog:<step>:%mf_uid(), which records
which step loaded what and keeps the audit row unique.
dc_refreshcatalog also echoes the generated statements when debug is on, since a
failure inside one of them otherwise names no libref.
A refresh that catalogues the wrong libraries is hard to read from the log: the
list is built here, then filtered and reworked by its callers. With
sasjs_mdebug set, print the result so the library list can be checked at source.
The MPE_DERIVED post edit hook aborts with mp_abort, so the webout carries a
sasjsAbort object: stagedata.test.3 and .4 now read that through
mpe_assert_abort instead of inferring the outcome from MPE_LOADS.
The field level rules move into mpe_derived_check so they can be tested
directly. A service cannot exercise them all - VAR_LABEL is varchar(256) and a
staging dataset declares the same length, so a longer label is truncated before
any check sees it - and the macro test runs in seconds rather than minutes.
The wrapper is DC specific and sits alongside mpe_assert_abort, so the mpe_
prefix marks which layer it belongs to. File, macro name, %mend and all seven
callers updated; no dc_execute reference remains in the tree.
Reading the sasjsAbort object instead did not work for this test: the derived post
edit hook aborts with `abort cancel FILE` via %mp_include, and with the
response-based version the test stopped before reporting any assertion, where the
MPE_LOADS version reports ten and fails five.
Which evidence is right depends on where that abort is raised - the test's own
session or the child's - and the log does not distinguish the two here. Reverting
until that is settled, so the branch is not carrying a change that reduces
coverage.
Same inputs and outputs, with a status row and no fail-fast, so the echoed child
log carries the CHILD: prefix - which is what distinguishes an abort raised in
the child from one raised in the test's own session.
The five rejected-submission cases queried MPE_LOADS for a row that did not reach
SUCCESS. The post edit hook aborts with mp_abort, so the webout carries a
sasjsAbort object: mpe_assert_abort reads that directly, which is the actual
evidence that the submission was refused.
It was declared but never assigned, and this session treats an uninitialized
variable as an error that stops the step before the assertion runs - so the
macro reported FAIL for a response that did carry the abort object.
%if &failfast=1 and "&state" in ("canceled" "failed") fails to evaluate: %EVAL
reports it needs a numeric operand and stops the macro, which aborted every test
that reached the guard even though the child had completed or aborted cleanly.
Two plain comparisons do the same job and evaluate reliably.
dc_assert_abort inferred the outcome from the status row. The assertion should
prove the service rejected its input, so it now reads the child's response and
requires a sasjsAbort object, and compares that object's MSG value with the
message the service is expected to return:
%mpe_assert_abort(desc=An invalid libds aborts the service,
outref=web1, msg=Invalid libds)
Renamed with the mpe_ prefix to match the other macro-level helpers. getdata
Test 4 asserted completion with the abort macro, which was the wrong tool; it
now checks the status row for state=completed directly.
dictionary.libnames lists the DC libref as well, so it was seeded into the
result by the SELECT and then added a second time by the explicit INSERT. The
duplicate row then collides with pk_mpe_datacatalog_libs when the refresh closes
the bitemporal record:
ERROR: Duplicate values not allowed on index pk_mpe_datacatalog_libs
The DC libref is now excluded from the scan, so it arrives once, from the INSERT
that carries its library name and engine.
@sasjs/core 6.1.0 ships outlogds and outstatus, so the temporary copy of
mx_execute is gone and the abort assertions no longer need a platform branch:
- dc_assert_abort reads the status dataset instead of work.results on Viya and
the response text elsewhere - one table, one code path;
- the abort cases call dc_execute with outstatus and failfast=0, so a cancelled
child is reported rather than aborting the test;
- getdata Test 4 loses its %mf_getplatform() gate for the same reason, and reads
the outlib only when the status says the child completed;
- dc_execute passes viyaresult through, which the abort cases ask for.
A full makedata on every sasjs test run is expensive, and it made the suite
depend on state the tests themselves should not be creating. Move it to where
DC already does this: the npm scripts that send deploy/makeData*.json to
services/admin/makedata.
npm run sas719setup deploy, then seed once
npm run sas719test run the suite as often as wanted
KEEPME in the payload keeps makedata on the server, so a reseed does not need a
redeploy first. Viya's makedata gains the same switch, so nextviyamakedata
behaves the same way.
Tests must now leave the estate as they found it - that is the follow-up, and it
is what makes re-running a single test possible.
Tests inferred whether a service call worked by reading an outlib that a failed
call never touched, so assertions ran against the previous call's tables. This
adds dc_execute, which wraps mx_execute and makes each call observable:
- failfast: abort as soon as the child did not complete, with its message, so
the first failure is the real one
- a status row per call (outstatus) so a test can assert the outcome
- the child's log behind the harness debug flag, quiet otherwise
testsetup reseeds the data baseline before the test library is built, makedata
honours a keepme switch and getdata initialises the variables it later tests
with %length, plus desc= values reworded where a comma or apostrophe unbalanced
the parameter.
sasjs/macros/mx_execute.sas is a TEMPORARY copy from sasjs/core#476, which adds
outlogds/outstatus and stops leaving work.results in the caller WORK. Delete it
and bump the dependency once that PR is released.
The four security test programs proved that a bad input aborted the service by
reading work.results, the job table %mv_jobflow writes in the Viya branch of
%mx_execute. The SASjs Server and SAS 9 branches call %ms_testservice instead,
so the table never exists, the data step fails with "File WORK.RESULTS.DATA
does not exist." and the assertion reported (0=1) - a failure that says
nothing about whether the service validated its input.
The platform difference is now in one macro, dc_assert_abort: on Viya it reads
the job state as before, elsewhere it looks for the sasjsAbort block that
mp_abort leaves in the webout %mx_execute returns through outref. The call to
the service stays outside the platform check, so the payloads are submitted on
every platform and the assertion is real rather than a skip.
Verified on SASjs Server (sas.4gl.io, target sas719, appLoc /Public/app/dc719):
the seven abort assertions PASS, i.e. the services do abort on the macro, SQL
injection and path traversal payloads. Negative control: the same program with
a valid libds reports FAIL (0=1), so the assertion can still fail.
sasjs lint: exit 0, no warnings in the five files.
The four security test programs proved that a bad input aborted the service by
reading work.results, the job table %mv_jobflow writes in the Viya branch of
%mx_execute. The SASjs Server and SAS 9 branches call %ms_testservice instead,
so the table never exists, the data step fails with "File WORK.RESULTS.DATA
does not exist." and the assertion reported (0=1) - a failure that says
nothing about whether the service validated its input.
The platform difference is now in one macro, dc_assert_abort: on Viya it reads
the job state as before, elsewhere it looks for the sasjsAbort block that
mp_abort leaves in the webout %mx_execute returns through outref. The call to
the service stays outside the platform check, so the payloads are submitted on
every platform and the assertion is real rather than a skip.
Verified on SASjs Server (sas.4gl.io, target sas719, appLoc /Public/app/dc719):
the seven abort assertions PASS, i.e. the services do abort on the macro, SQL
injection and path traversal payloads. Negative control: the same program with
a valid libds reports FAIL (0=1), so the assertion can still fail.
sasjs lint: exit 0, no warnings in the five files.
2026-10-26 20:17:50 +00:00
342 changed files with 26716 additions and 10024 deletions
The deploy screens ask two questions about the physical filesystem - what folders sit below this parent, and can the deployment user write to this one. Both are answered by `makedata` (`ACTION=DIRLIST` and `ACTION=WRITECHECK`), through `mpe_dirlist.sas` and `mpe_writecheck.sas`.
This note records the two-layer answer: a gate that closes the window once the appLoc is configured, and why the window itself has no group check.
## The gate
The deploy window is over once the appLoc has `services/public/settings.sas`: `makedata` writes it at the end of a successful run, and a run that failed - or was abandoned - after writing it leaves the file behind. So the service refuses `DIRLIST` and `WRITECHECK` whenever that file is present, however the run that created it ended. On SASjs Server the file is fetched with `ms_getfile` and a probe that cannot answer (anything other than 200 or 404) is treated as configured: an unreachable drive is not evidence that the window is open. Viya checks the same file through `mfv_existfile`.
The consequence for tests: the browse actions are refused on any appLoc that runs the suite, because every such appLoc is configured. `services/admin/makedata.test.sas` therefore asserts the refusal, and the listing and write-probe behaviour is asserted against the macros directly (`macros/mpe_dirlist.test.sas`, `macros/mpe_writecheck.test.sas`). The macros' own abort paths (a relative path, one carrying a macro trigger) are not asserted anywhere: `mp_abort` cannot be exercised inside the test session.
## Why there is no group check
Every other admin service gates on `&mpeadmins`, which comes from `MPE_CONFIG` in the control library - and that library is what `makedata` creates. Before it runs there is no group to check: `%mpeinit()` reads `&DC_LIBREF..mpe_config` and has nothing to read. That is also why the previous `dirlist` service could never serve the deploy screen it was written for.
The window is not narrower than the configurator's own: on SASjs Server the configurator page, in the same window, lets the caller pick any group they belong to and create the control library - which is full Data Controller admin. `DIRLIST` and `WRITECHECK` are strictly less powerful than that.
## What the actions do, for the record
-`DIRLIST` - enumerate the folders below any absolute path the SAS process can open. Folder names only; no file contents, no file listing.
-`WRITECHECK` - ask whether a path exists, is a folder, and whether the deployment user can create and delete a file in it. It creates a uniquely named probe file and removes it again.
Both refuse a value carrying a macro trigger, a quote or a statement separator, and shape check it with `mp_validatecol(ISPATH)` before use. The shape check fails closed: an absent `ISPATH` rule refuses the path rather than accepting it.
## The assessment
- **The window is the deployment itself.** `makedata` is deployed, run once by an administrator, and deletes itself on success.
- **There is no admin group to check while it runs.** It is created by `makedata`, from the group the deployer picks in the configurator.
- **The caller is the administrator.** Reaching these actions means reaching the STPs of a deployment in progress, which is the same caller the configurator is serving in the same window.
- **The actions are the weaker half of that window.** The configurator, in the same window, lets the caller pick any group they belong to and create the control library - which is full Data Controller admin. Listing folders and probing a write cannot achieve that.
- **The window closes on the settings file, not on survival of the service.** That is what covers a failed or abandoned run.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name.
## Commands
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`.
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS
test suite against it. Covers the target and credential setup, the makedata seed the suite needs on
a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend
so the streamed app is the build you just made.
---
# Deploying and testing on the shared SASjs Server estate
The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong.
cd sas && npm ci # macro dependencies
cd ../client && npm ci && npm run build # frontend only - see "Deploying the frontend"
cd ../sas && npm run server # cpfavicon + sasjs cbd -t server + makedata
npx sasjs test -t server
`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `<appLoc>/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed.
## The targets
`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc``/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair.
A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered.
To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app/<you>/dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at.
## Credentials
The CLI logs in with a registered client, not a username and password: `.env.<target>` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless.
Mint the pair from a local login, with the password read from a file rather than argv:
1.`GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header.
2.`POST /SASLogon/login` with `{"username","password"}`.
3.`GET /SASjsApi/client` on the same session - the registered client ids.
4.`POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`.
5.`POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`.
An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over.
## The makedata seed
`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.
## Reading the results
Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON.
A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete.
`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for.
## Deploying the frontend
The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists:
1.`client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`.
2.`client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines.
3.`cd client && npm ci && npm run build`.
4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc.
5.`cd ../sas && npx sasjs cbd -t <target>` - with `client/dist` present the log says `Copying web app files` and `Web app is available at <url>`.
Without step 1 the deploy prints `ERROR An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main.<hash>.js` the page names.
Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check.
## Reading a failed run
-`ERROR An error occurred while executing the request. Job location: <appLoc>/tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/<suite>.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise.
-`WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem.
-`sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree.
- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code.
echo"::error::runner has ${effective_mb} MB of memory, a client job needs at least ${MIN_TOTAL_MB} MB (8 GB is the supported minimum; a cold production build alone peaks at about 3.8 GB, and the Cypress and Lighthouse jobs add a dev server and a browser to that)."
exit1
fi
# Not a gate - a second job landing on the same box shows up here first.
if[ -n "$avail_kb"]&&["$avail_mb" -lt 5000];then
echo"::warning::only ${avail_mb} MB is available on this runner; another job may be running on the same box"
*) echo "the build job published no decision - running the suite" ;;
esac
- name:Cypress suite not run - no client changes
# A named step rather than a run of greyed-out ones, so the job summary
# says why it took a minute: this job reports success having tested
# nothing, which is the intended behaviour for a pull request that
# changes no client code. If this job is a required check, "green" for
# such a pull request means "not applicable", and this step is the
# evidence for it.
if:steps.changes.outputs.client == 'false'
run:|
echo "No client, sas/mocks, sas/package.json or .gitea/scripts change in this pull request - the Cypress suite was skipped by design."
echo "The build job's 'Detect the Cypress job's inputs' step lists the files it saw, and warns instead if it could not decide (in which case the suite runs)."
- name:Check runner size
if:steps.changes.outputs.client != 'false'
# Each Cypress job compiles the frontend and runs a browser on top; a
# runner too small for that fails as a flaky-looking disconnect - see
# the script.
run:bash .gitea/scripts/check-runner-memory.sh
- uses:actions/setup-node@v4
if:steps.changes.outputs.client != 'false'
with:
node-version:${{ env.NODE_VERSION }}
- name:Write .npmrc file
if:steps.changes.outputs.client != 'false'
run:|
touch client/.npmrc
echo '${{ secrets.NPMRC}}' > client/.npmrc
# the secret arrives through the environment, not interpolated into
# this script: a quote or $( in the value would otherwise break the
# the server the mocked services and app tests run against - bump deliberately
SASJS_SERVER_VERSION:'3.14.0'
NODE_VERSION:'24.15.0'
jobs:
@@ -12,39 +21,112 @@ jobs:
steps:
- uses:actions/checkout@v4
- name:Detect client changes
id:changes
# The audit measures the client, served from the mocked estate. A pull
# request that changes none of what it reads cannot move a score, so it
# does not need nineteen minutes of runner (measured median) to prove it.
# As in build.yaml, the gate is `!= 'false'` and every failure exits 0
# without writing client=false, so a decision that cannot be made runs
# the audit rather than failing the job.
env:
BASE_REF:${{ github.base_ref }}
run:|
set -euo pipefail
BASE="${BASE_REF:-}"
if [ -z "$BASE" ]; then
echo "no base ref (not a pull request) - running the audit"
exit 0
fi
if ! git fetch --no-tags --depth=1 origin "+refs/heads/${BASE}:refs/remotes/origin/${BASE}"; then
echo "::warning::could not fetch ${BASE} - running the audit rather than deciding"
exit 0
fi
if ! CHANGED="$(git diff --name-only "refs/remotes/origin/${BASE}" HEAD)"; then
echo "::warning::could not diff against ${BASE} - running the audit rather than deciding"
exit 0
fi
# .gitea/scripts/ carries the runner-memory gate this job runs before
# it starts, so a change there has to be exercised too. sas/package.json
# is what this job reads for the @sasjs/cli pin below, so a bump there
# has to be audited as well - build.yaml's matcher covers it too.
if printf '%s\n' "$CHANGED" | grep -qE '^(client/|sas/mocks/|sas/package\.json|\.gitea/scripts/|package-lock\.json|\.gitea/workflows/lighthouse\.yaml)'; then
echo "client=true" >> "$GITHUB_OUTPUT"
else
echo "client=false" >> "$GITHUB_OUTPUT"
echo "::notice::Lighthouse audit skipped - nothing under client/, sas/mocks/, sas/package.json or .gitea/scripts/ changed"
printf 'Skipping the audit - nothing it reads changed:\n%s\n' "$CHANGED"
fi
- name:Lighthouse audit not run - no client changes
if:steps.changes.outputs.client == 'false'
run:|
echo "No client, sas/mocks, sas/package.json or .gitea/scripts change in this pull request - the audit was skipped by design."
echo "The detection step above lists the files it saw, and warns instead if it could not decide (in which case the audit runs)."
- name:Check runner size
if:steps.changes.outputs.client != 'false'
# This job runs a production build, then a SASjs server, a dev server
# and Chrome. An undersized runner dies inside the build with exitcode
# 137; fail here in seconds instead - see the script.
@@ -31,7 +31,12 @@ Rationale: hard-wrapped prose produces noisy diffs when sentences are edited and
Never consider a change complete until the relevant linters pass on the files you touched — do not rely on the user's pre-commit hooks to catch it:
- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
- **SAS**: after creating or modifying any `.sas` files, run `npm run lint:sas` from the `sas/` directory. That script runs the `@sasjs/cli` version pinned in `sas/package.json`, so it matches CI; a global or `npx``sasjs` may resolve a different `@sasjs/lint` and report a different rule set.
Rules are promoted to `error` in `sas/.sasjslint` one at a time as their findings reach zero, and only errors change the exit code - so the whole project must stay clean of the promoted rules, not just the files you touched. Warnings under rules not yet promoted are the remaining backlog; do not add to them in files you touch, and do not clear other files' findings in the same change unless that rule is the point of the change.
`sas/.sasjslint` carries `ignoredMacros: ["webout", "mpeinit", "mpeterm"]`. `webout` is the linter's own default - `sasjs compile` injects `%webout` into every service - and an explicit list replaces that default, so it has to be restated there. `mpeinit` and `mpeterm` are listed because every service is compiled with a serviceInit program (`sasjs/utils/serviceinitsas9|viya|server.sas`) that declares both macros in its own header, so a service body may call them without declaring them. Add a macro to that list only when it reaches a file the same way - injected by the compiler, through serviceInit, testSetUp or another implicit provider - never to silence a genuinely missing declaration.
`noBannedStrings` flags the literal words `error ` and `warning ` anywhere in a file, comments included. Reword prose; where the text has to stay (a real `%put`, a doxygen `@warning` tag), keep the log output identical with `%put %str(WARN)ING:` rather than a per-file override, and use `@sasjslint {"noBannedStrings": false}` only when the word itself is the content.
Hook scripts (`sasjs/services/hooks/*.sas`) run inside the service that includes them, so the macros they call come from that service's session. They carry a file-local `@sasjslint {"ignoredMacros": [...]}` rather than a macro list of their own, because end users copy and edit these scripts and should not have to maintain a declaration section.
`noMacroIfInOpenCode` is on, so `%if` must sit inside a `%macro` definition. Where a conditional has to stay in a service body, wrap the block in a small macro and call it unconditionally (`%macro build_staged(); ... %end; %mend build_staged; %build_staged()`), or resolve the value in a data step with `call symputx`. Two traps: a `%let` inside such a wrapper is local, so a macro variable that must survive the block has to be written with `call symputx(...,'G')`; and open-code `%if` itself needs SAS 9.4M5, which the code no longer relies on.
## No external assets
@@ -41,7 +46,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
### Features
* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](https://git.datacontroller.io/dc/dc/commit/888873421c3aeaf312d5ec45e3c805317c66976a))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](https://git.datacontroller.io/dc/dc/commit/dd482c4069202a0c6b9983129c19efc09bd160a7))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](https://git.datacontroller.io/dc/dc/commit/25ffa71adddba5e334f4a8c166f284d1be498fcd))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](https://git.datacontroller.io/dc/dc/commit/c5808a16eba0d29562fcc76f13ac4a33ab8da0b7))
* **ci:** rebuild and upload release assets on demand ([6486c80](https://git.datacontroller.io/dc/dc/commit/6486c80ed4f368bbf116c0ecdecc0e5e5f02075f))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](https://git.datacontroller.io/dc/dc/commit/af1813f829861df41c89b512c824e8d3ebac7a4b))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](https://git.datacontroller.io/dc/dc/commit/cf7b21a8b61ec3bf7ebe21968ccf13ade264bc64))
* **cypress:** hold download out too, and retry once in CI ([c97e886](https://git.datacontroller.io/dc/dc/commit/c97e8866db9ee404b2118d24baf53fc7058debef)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395) [#398](https://git.datacontroller.io/dc/dc/issues/398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](https://git.datacontroller.io/dc/dc/commit/c803773d3c36476d8263f01fc3b8344aa3503d44)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](https://git.datacontroller.io/dc/dc/commit/80e8911af8857fa2c417f028cd59b64a87233b9e))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](https://git.datacontroller.io/dc/dc/commit/e51953ca8a6734e1649bad68cf4ad2057d677334))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](https://git.datacontroller.io/dc/dc/commit/459ff2da371375e996839d69deb40f681c356f1d)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](https://git.datacontroller.io/dc/dc/commit/f7e753863b61d8545bd0777d80267661b3c58670))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](https://git.datacontroller.io/dc/dc/commit/7e3045a21b67053b76a81bfe836e31571ccbffbf))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](https://git.datacontroller.io/dc/dc/commit/2623a3fb4ba3bf19c00636b1717b53d5a2d54f3a))
* **embed:** log a parse-time mapping problem once ([0011f7e](https://git.datacontroller.io/dc/dc/commit/0011f7e933d85be8e33debe3a6ecb9000edaabe3))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](https://git.datacontroller.io/dc/dc/commit/541a61ba08bc7256a911c50de7078859cc46c84e))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](https://git.datacontroller.io/dc/dc/commit/e9216b08956d5599662259a2b05cc1524c458df5))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](https://git.datacontroller.io/dc/dc/commit/bd3c220af2678a2a3f8394fe3084ee03c23d0b15))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](https://git.datacontroller.io/dc/dc/commit/f83dc359e258f5d1cb9fc49f0eaab9748995d823))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](https://git.datacontroller.io/dc/dc/commit/aa91daf9f78c00cea486d0f885f8e634366bd1a0))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](https://git.datacontroller.io/dc/dc/commit/10157c11595685efdc206fe7b16a5cafa098f26d))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](https://git.datacontroller.io/dc/dc/commit/411f6c3354b7fbfec11fef3b174ae1f8066c92ba))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](https://git.datacontroller.io/dc/dc/commit/c0ea810766ebfda5ae5358ab14146fbcc07fcc8e))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](https://git.datacontroller.io/dc/dc/commit/15a1f34d5263503cac83b4fe4e52482777dc2f1f))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](https://git.datacontroller.io/dc/dc/commit/88010c3980c2ea1c39bbb024ed32d63e495e61f0))
* address the review findings on the harness and refresh changes ([36e4e88](https://git.datacontroller.io/dc/dc/commit/36e4e88eb134d3b48217d6dd185ea3c03f03fba5))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](https://git.datacontroller.io/dc/dc/commit/698dea6d6b7a8635313f11acfcc7a4f39a02bc99))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](https://git.datacontroller.io/dc/dc/commit/0d58be975d8eb2d7f1ab6a648f760151392c7973))
* **client:** open the SAS log page with noopener ([072a133](https://git.datacontroller.io/dc/dc/commit/072a1337020970b5b4ff8aa31719cc5ffec7701e))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](https://git.datacontroller.io/dc/dc/commit/1fb4818d215f538a0ac51772ab1f261aeb91f7e1))
* **cypress:** apply the licence key when the estate has none ([be8f445](https://git.datacontroller.io/dc/dc/commit/be8f445ae1051e132fb6eb5a69b45264255326b5)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](https://git.datacontroller.io/dc/dc/commit/fed43cb07ef7f2afc685612a124dbc16d7618e21))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](https://git.datacontroller.io/dc/dc/commit/f5aade59760cc376b0f96eb4c892c8bca82dde14))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](https://git.datacontroller.io/dc/dc/commit/1e224763ba80500a112812a983801968e3751959))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](https://git.datacontroller.io/dc/dc/commit/9b3cf50b9c3889786605f0f5d9032d84c50f7107))
* **derived:** address five review findings ([ccd468b](https://git.datacontroller.io/dc/dc/commit/ccd468b3358c1418ee6d8e198b088a0cf23b8c24))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](https://git.datacontroller.io/dc/dc/commit/be50cb5d082e788c6ae106bc1d02e73a8c7b9646))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](https://git.datacontroller.io/dc/dc/commit/9914790777f92128c74ecce67e6c67ee48824b45))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](https://git.datacontroller.io/dc/dc/commit/9ef813d439b48e7fefd35c426e62b99e5943002a))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](https://git.datacontroller.io/dc/dc/commit/422f766cd43db4ce740e1890fd5df30790eaa7e3))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](https://git.datacontroller.io/dc/dc/commit/7830b7bcfa9806dde840ce8dd3e6a1887d0b38be))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](https://git.datacontroller.io/dc/dc/commit/2d5de9a324a07a36edc0a4f74c8fad09fde07cbf))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](https://git.datacontroller.io/dc/dc/commit/6a1128a46352f77c95dd02df9309f0984a1db2d1))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](https://git.datacontroller.io/dc/dc/commit/1863076bc956b4ff94bf7b85b3dde0d6e3b54d7a)), closes [#366](https://git.datacontroller.io/dc/dc/issues/366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](https://git.datacontroller.io/dc/dc/commit/d3dec682aafeb143fe983d6546337cb6490f5620))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](https://git.datacontroller.io/dc/dc/commit/469229aecbaad304a835a5f229df111c37ff7c0d))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](https://git.datacontroller.io/dc/dc/commit/7fb981daa91bce29c2d62c246185d01dcad08d6b))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](https://git.datacontroller.io/dc/dc/commit/5f5d8681b9d78280a5200309917415e135c3ce5b))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](https://git.datacontroller.io/dc/dc/commit/5f443b0b16a2abd3c11d96caa7a043911c29ef01))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](https://git.datacontroller.io/dc/dc/commit/6b5bf3e6be89b22e5a633c34b9aecbc8c7d48ee2))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](https://git.datacontroller.io/dc/dc/commit/85c89ca2a4e02112ba496ac36a222f33675544da))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](https://git.datacontroller.io/dc/dc/commit/f8f65681e1489f4ce94d9127d5e4c0c82798e960))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](https://git.datacontroller.io/dc/dc/commit/7efbf31ea5ac234bd8f924fdf89a95d101b3aff1))
* **derived:** only the delete marker is a reserved name ([af31c36](https://git.datacontroller.io/dc/dc/commit/af31c36b670ec63a2a4ba056bf6c4b28f6096e08))
* **derived:** open the statements file with a long record length ([ee31782](https://git.datacontroller.io/dc/dc/commit/ee31782fbff909b540162b1bdfca6f307208331b))
* **derived:** pair the filterds %else with %do/%end ([88a38db](https://git.datacontroller.io/dc/dc/commit/88a38db877dba9b98ccf50f0cc7242674654586b))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](https://git.datacontroller.io/dc/dc/commit/27ddcfb5759d6eb8f9880adc7c271a408e7741ad))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](https://git.datacontroller.io/dc/dc/commit/7799a300df82339683eea21e4a89ce7a6471c575))
* **derived:** reject a dependency on a row this submission deletes ([8806504](https://git.datacontroller.io/dc/dc/commit/88065049674c2cc0141eef22704b759f5b708b0f))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](https://git.datacontroller.io/dc/dc/commit/9a047ad42ec86df7d1eaf5058c0edde56016e88f))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](https://git.datacontroller.io/dc/dc/commit/e0da9f1098db63185538398ca478696cc849da4b))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](https://git.datacontroller.io/dc/dc/commit/688550a28d9794ac6b878926a0e1e2ef8f6e9335))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](https://git.datacontroller.io/dc/dc/commit/1b5c4b3ecd962a28e08fc5671b57dbf7ae0b8d9f))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](https://git.datacontroller.io/dc/dc/commit/4d9ad38df1fa89b0d5a12214d703d9c1fe5295f2))
* **editor:** a blank truncation note is not a warning ([f182179](https://git.datacontroller.io/dc/dc/commit/f18217940365f27ae388fae087e6126931e1222c))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](https://git.datacontroller.io/dc/dc/commit/04ce1861e7821063b1be42c22f92f62aed509602))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](https://git.datacontroller.io/dc/dc/commit/4bcee706ffaf1929bdd14bb2fb15111174ca3916))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](https://git.datacontroller.io/dc/dc/commit/45dce7b11b7a4399c1d0dc9c3e7a283b1cf59df1))
* **getdata:** clear the globals it declares before use ([359b570](https://git.datacontroller.io/dc/dc/commit/359b570b85aa2f66507e81ea2977243be4590f83))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](https://git.datacontroller.io/dc/dc/commit/059cdf176a7ec556b45ea1b21d880c048eba14ca))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](https://git.datacontroller.io/dc/dc/commit/661ee19cfa5f996bf01f2b11817039f38dbb9985))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](https://git.datacontroller.io/dc/dc/commit/355c46a7b64994d0146a88657dc5635593249138))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](https://git.datacontroller.io/dc/dc/commit/d626cb8f2daf94fde7ff21d924b9a2ca4968d983))
* **mocks:** address the audit-pack review findings ([489819b](https://git.datacontroller.io/dc/dc/commit/489819b05111323ae1e34edda22a5446cdbc2d87))
* **mocks:** carry the zip flags into the central directory ([c4021a4](https://git.datacontroller.io/dc/dc/commit/c4021a42ffaed2af80c53aaa1455249190b339b8))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](https://git.datacontroller.io/dc/dc/commit/62f55db170185900063f7a431cd8f375ebabf930))
* **mocks:** copy the physical column list before deriving ([f411b7d](https://git.datacontroller.io/dc/dc/commit/f411b7d269381e3f7cab2d85b6d8f7d7d50ea4d1))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](https://git.datacontroller.io/dc/dc/commit/60ec470404e5c279b58ac6a01768dfc83413b197))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](https://git.datacontroller.io/dc/dc/commit/74060c0585fb8b217cb181ea747ad318f5238868))
* **mocks:** drop derived variables before staging ([73d25b1](https://git.datacontroller.io/dc/dc/commit/73d25b1220d574ef2547fbd9e6e3cf1ceda375ee))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](https://git.datacontroller.io/dc/dc/commit/347dbb53b01c47bbd562de14d5ccbae7d96407c6))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](https://git.datacontroller.io/dc/dc/commit/ee3c8787efef9cd716bdd2bf80f6cb6751461db9))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](https://git.datacontroller.io/dc/dc/commit/6cad23fce9ac0cf64c247c40f3b7eac875fd4cfb))
* **mocks:** search the physical columns only in the viewer ([293022d](https://git.datacontroller.io/dc/dc/commit/293022d46b27e096fd514cdda49f688612df2f5b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](https://git.datacontroller.io/dc/dc/commit/fb36379520028baaf4bc3f08131bfc88deab3ac3))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](https://git.datacontroller.io/dc/dc/commit/9c294c0ab5ca8fe30a235553a298c1c68a61442f))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](https://git.datacontroller.io/dc/dc/commit/0a52d20c5fc27b4adde5fc8ed6f39a58c48f62ff))
* **mpe_execute:** fail fast when the child left no status ([a036386](https://git.datacontroller.io/dc/dc/commit/a03638667f1968c5909f496d954eeaa13f316763))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](https://git.datacontroller.io/dc/dc/commit/220408538b26016bda45e48d619cdbc3e1cbcc59))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](https://git.datacontroller.io/dc/dc/commit/f2b1428ec3099b2b6e3c6da6627530ef81fab201))
* name the startupservice launcher button "Show log" ([3445107](https://git.datacontroller.io/dc/dc/commit/3445107d9a91642b67e114bcfcd81c3ce47f3f0f))
* offer the launcher on the globvars failure modal too ([ad2da45](https://git.datacontroller.io/dc/dc/commit/ad2da453896a51f2cc7e49cebb6c84f7b47d7395))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](https://git.datacontroller.io/dc/dc/commit/46bacdd6d9ea814bcd02a1ff8aa3f447baf02f64))
* **refresh:** one audit folder per refresh call ([56b6620](https://git.datacontroller.io/dc/dc/commit/56b6620ef402b288ce78a2a77662b651e310753e))
* **sites:** round-trip the committed licence before publishing ([4d462d3](https://git.datacontroller.io/dc/dc/commit/4d462d310f7df23e5d5229cb0702721b7ad41bbd))
* **test:** getdata test ([da30266](https://git.datacontroller.io/dc/dc/commit/da30266278c4fe27a72810cd5ed647cf3bd823af))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](https://git.datacontroller.io/dc/dc/commit/033ffd302502c7530fc44a314364f5cdb3be6ad2))
* **ui:** one error modal at a time ([81ccf75](https://git.datacontroller.io/dc/dc/commit/81ccf758e9dab0c7632ab50bf469e88b8c4c4420))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](https://git.datacontroller.io/dc/dc/commit/42355f17bc07bf4d44ba858984d83b9ed72cc0ba))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](https://git.datacontroller.io/dc/dc/commit/337ea8e5e021c6f2451fa3ad8396148744118fc9))
### Features
* **dc_getlibs:** dump the library list when debug is on ([582523d](https://git.datacontroller.io/dc/dc/commit/582523dc87a253b71824359224070ccb4746a386))
* **derived:** allow a variable to shadow a physical column ([12beeff](https://git.datacontroller.io/dc/dc/commit/12beeff69518376a042fbbc58332acc40d89d132))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](https://git.datacontroller.io/dc/dc/commit/813bce685691ad252043eeab31eb5460de81b756))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](https://git.datacontroller.io/dc/dc/commit/7c9ce9a712137e9825a5f2ff52796d765423147b))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](https://git.datacontroller.io/dc/dc/commit/87e794f769fe420f214f43b141f0b41cf88eefee))
* **derived:** report the dependency cycle path ([cee1a95](https://git.datacontroller.io/dc/dc/commit/cee1a954ee97e17f297b7cf43ef0e5cfc6e393a6))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](https://git.datacontroller.io/dc/dc/commit/78e88a41128b047c0276ccf647f128229152604c))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](https://git.datacontroller.io/dc/dc/commit/0b010f2c65ddada70d62ea83c706943c6c45756f))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](https://git.datacontroller.io/dc/dc/commit/aa6d50e1ea5fc2fe6be8aa9fa859b92d0a1061c5))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](https://git.datacontroller.io/dc/dc/commit/33d043ea92f007dca70c95d5363329d4447eeba9))
* **editor:** derived variables in the EDIT screen ([b0fd354](https://git.datacontroller.io/dc/dc/commit/b0fd354ecdbd7bf29196ed983ff3cce691affeed))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](https://git.datacontroller.io/dc/dc/commit/6513d09f059de15c5304f99b3f8d898d46ea1be7))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](https://git.datacontroller.io/dc/dc/commit/34691dceecb63f952b23ad8acc5bc93753c6cfee))
* **editor:** show the derived column label in the header tooltip ([cfe6647](https://git.datacontroller.io/dc/dc/commit/cfe6647ca870c45a3d8abc2753ce7f070e3fdf4a))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](https://git.datacontroller.io/dc/dc/commit/752d944c2091ef6d2624a15ab2e2e7be35608798))
* **mocks:** implement derived variables in the JS services ([b4fae88](https://git.datacontroller.io/dc/dc/commit/b4fae88ae13658607e4209449577cd42849d3a99))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](https://git.datacontroller.io/dc/dc/commit/5d183d135085ea023f32e5b1d4992d997394a8fb))
* run startupservice in a new window from its error modal ([40f6133](https://git.datacontroller.io/dc/dc/commit/40f61331aad6bc8a52c551b924f0d1f1e6426cee))
* **sites:** mint a licence for the published sites ([7d9df71](https://git.datacontroller.io/dc/dc/commit/7d9df71fdb0ab265e745ed3b708038983e855a87))
### Reverts
* **derived:** no shadowing, and no self reference ([5af8e6f](https://git.datacontroller.io/dc/dc/commit/5af8e6fc359221586af74a685159dfd75a40ec45))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](https://git.datacontroller.io/dc/dc/commit/31aa8591538409fc1a3ef97d7474120094ce202d))
@@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output
-`client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
-`sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
-`.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
-`.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).
The licence key comes from the backend licence: the decrypted licence carries `hot_license_key`,
@@ -86,6 +109,7 @@ RUN_TIMES=js
NODE_PATH=node
CORS=enable
WHITELIST=http://localhost:4200
HTTP_PORT=5000
```
`RUN_TIMES=js` means the server executes `.js` files as Stored Programs (no SAS binary needed). `CORS=enable` plus `WHITELIST` allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
./api-linux # or api-macos / api-win.exe depending on your platform
```
The server runs in desktop mode (no auth) on `http://localhost:5000` by default.
`HTTP_PORT` is not optional in SASjs Server 3.x: `PORT` and `PROTOCOL` are deprecated, and a server started without `HTTP_PORT` or `HTTPS_PORT` exits with "no listener is enabled". With it set, the server runs in desktop mode (no auth) on `http://localhost:5000`.
### 2. Deploy the mock services and streaming frontend
@@ -123,11 +147,11 @@ The app will be at `http://localhost:4200`. It calls the JS mock services on por
### How the mocks work
Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). The JS runtime predeclares `fs` and`_webout`; the mock assigns a JSON string to`_webout` that matches the structure the real SAS service would return. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). The JS runtime predeclares `fs`, `_program` (the program's deployed path),`weboutPath` (the response body it writes) and `_webout`, and the services emit their members through the runtime's `webout()` - `webout('OPEN')`, one `webout('OBJ', name, rows)` (or `webout('ARR', name, rows)` for positional rows) per member,`webout('CLOSE')` - so the response envelope comes from the host and the file carries only what the SAS service returns. A few services still write the body themselves through `_webout`: the ones that emit plain text, CSV, DDL or HTML and set their own content headers (`admin/exportdb`, `admin/makedata`, `public/getrawdata`, `public/getddl`), and the lineage captures. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
The mock data includes sample tables (`DC996664.MPE_X_TEST`, `MPE_X_NEW`, `MPE_X_FORMULA_TEST`, etc.) with realistic columns, validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT), and dropdown values. This is sufficient to exercise the full editor grid, approval workflow, viewer, and admin screens.
There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`).
There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`). Its services are fixed recordings of the SAS 9 responses, emitted through `webout()` in the same way; the payloads are the SAS 9 ones (metadata URIs, `BUILTIN\Administrators` owners, `D:\...` paths, with their separators - the front end splits on `\`, so a value that arrives without it exercises the wrong branch). Only the payloads are SAS 9 flavour: the envelope around them (`_PROGRAM`, `SYSUSERID`, `SYSVLONG`, `SYSDATE`/`SYSTIME`) is whatever runtime serves the request, so do not read this variant for SAS 9 envelope values.
### Re-deploying after changes
@@ -135,7 +159,7 @@ After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sa
### Demo and staging sites
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs `makedata` on a throwaway SASjs Server first and ships that drive as `sasjsbuild/mock-snapshot.json`; without it the site opens with no data. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. The drive the site ships with is captured from a throwaway SASjs Server as `sasjsbuild/mock-snapshot.json`, and the two sites ship different ones: the demo runs `makedata` on that server first, so it opens with the control library and the sample tables; staging publishes without running it, so it opens on the deploy screen (DC routes there while `makedata` is still on the drive) and the visitor deploys the control library from the wizard. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
`VA parameter "${paramName}" maps to ${target.column}:${spec}, which is a range, but the parameter has ${sent.length} values - a range needs one value, so the filter was ignored.`
)
continue
}
if(usable.length<sent.length){
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${sent.length-usable.length} of its ${sent.length} values was not a date - the filter covers only the rest.`
`va_parmap "${entry}" was ignored: "${match[1]}" is not a known transform (ADD, RANGE).`
)
returnundefined
}
Loaded 100 of 342 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
dc
allan