Reviews 893 and 894 on #518.
894-1 and 894-2 - a -FC request named a format catalog, and the mock answered
it with the sibling dataset. Stripping the suffix for validation made the name
canonical but also changed which object was resolved, so LIBREF.NAME-FC returned
NAME's rows for a name that is not a catalog at all. Now the MPE_TABLES
registration decides: if it carries a row for NAME-FC the member is a catalog
and is streamed as LIBREF.NAME.ext; if it does not, the request is not a catalog
and the estate's own answer is reproduced - syscc=1012 from %mp_cntlout,
recorded as public/getrawdata/catalog_suffix_on_dataset. The catalog the fixture
does hold, MPE_X_CATALOG, has no format entries, so its listing is empty; that
is recorded in the parity README beside the other fixture divergences rather
than left as an unexplained empty body. The test that pinned the object swap now
asserts both halves.
894-3 - getcolvals answered a well-formed libref it does not have with "Invalid
libds", and a malformed one with "Target table not found": the two rules were
the wrong way round. getcolvals.sas validates the SHAPE first with
%mpe_validatecol(...,LIBDS,...) and that is the only "Invalid libds"; the table
is then left to %mp_derivedvars, which aborts "Target table not found: <LIBDS>"
with its own MAC. Both are now recorded: public/getcolvals/wrong_libref uses a
valid-length NOLIB and pins the resolution abort, and the new
public/getcolvals/malformed_libds pins the shape abort with an 11-character
libref. fault-paths.test.js went with them.
893-2 - loadTableFrom duplicated loadTableData, so the two could drift: one
loadTableFile(libref, tableName) now serves both, with the control library as
the default.
893-3 - an unreadable or malformed table file was swallowed into null and
surfaced as "Target table not found". It now throws naming the file, because a
broken fixture should not send the next reader looking for a missing table; an
absent member still returns null and keeps the recorded abort.
Tests: 149 pass. Each of the three behaviour changes was run against the
previous code first and fails there - swallowing parse errors, answering the
catalog with the dataset, and restoring the inverted getcolvals rule.
parity/check.js ok on all 25 scenarios.
Reviews 890 and 892 on #517, both about getrawdata.js in this branch.
890 - the -FC strip fed only the shape guard. `withoutCatalog` was computed for
the LIBDS test and then dropped, so the filename, the table name and the library
were all taken from the unstripped value: a request the guard accepted resolved
nothing and came back as an empty download named LIBREF.DS-FC.ext.
getrawdata.sas strips the suffix and then works with the stripped pair
(`table=cats("&lib..",ds)`), so the mock does that too - `requestTable` is the
stripped, upcased name and everything after the guard uses it: the guard, the
filename, the library it reads and the name in an abort.
892 - `Number()` is not ISINT's rule. I checked the premise against the estate
rather than the macro reading, because the two disagree here: it answers
"Invalid filter" for 1,000 and 1abc (which a leading-prefix read would accept),
and for 0x10, which Number() reads as 16. What the estate does accept is
7.0, 1e3, +5, 007, " 7 " and -3, each of which then fails downstream on the
filter RK rather than on validation. The mock now matches the value with a
number pattern first and then asks whether it is whole, which reproduces all of
that; the test's claim is corrected to the recorded answers, and the comment no
longer says the reading was taken from the macro.
Tests: download-headers.test.js is 14 tests. Both new assertions go red against
the previous code - leaving the unstripped name downstream fails the catalog
test, and dropping the shape match fails the filter test on 0x10. Whole mock
suite: 147 tests, 147 pass; parity/check.js unchanged.
Stacks on #517, which fixed the download headers of this service.
The mock loaded every table from the control library's folder, whatever LIBDS the
request named, and answered an unresolvable table with a 200 whose body was the
headers and nothing else. getrawdata.sas assigns the library it was given
(%dc_assignlib(READ,&lib)) and reads the data from there; the control library
supplies only the registration row that names the validity columns to drop. So:
- the table now loads from data/<LIBREF>/, which is where makedata and
createdemodata write each library's files, and the LIBDS is upcased first as
`call symputx('table',upcase(_table))` does
- a table that cannot be resolved aborts before anything is written, with the
message and MAC the estate answers - recorded, not inferred:
NOLIB.MPE_TABLES -> Target table not found: NOLIB.MPE_TABLES MP_ABORT
DC634301.NO_SUCH_TABLE -> Target table not found: DC634301.NO_SUCH_TABLE
WORK.NOPE -> Target table not found: WORK.NOPE
The abort is mp_derivedvars', so the MAC is that macro's own default, and the
message carries the upcased LIBDS rather than the service name.
Recorded: new parity scenario public/getrawdata/missing_table, golden taken from
the synthetic estate (one accepted difference, SYSCC, as for the getddl abort).
Tests: download-headers.test.js gains two - a download of TESTDATA.DEMO_ORDERS,
which lives outside the control library and used to answer an empty file, and
the three unresolvable-table cases above. Both go red against the old code:
loading from the control folder fails the first, and removing the abort fails
the second. Whole mock suite: 146 tests, 146 pass; parity/check.js ok on all 23
scenarios.
Review 886 on #517, both findings.
Finding 1 - getrawdata validates three inputs, and the mock only had the third.
getrawdata.sas aborts, in this order and all with mac=&_program..sas, before it
streams anything:
table shape -> "Invalid table"
filter integer -> "Invalid filter"
type list -> "Invalid type"
The mock now reproduces all three, from the rules of the macros the service
calls: %mpe_validatecol(...,LIBDS,...) strips a trailing -FC and applies
%mp_validatecol's /^[_a-z]\w{0,7}\.[_a-z]\w{0,31}$/i, and
%mp_validatecol(...,ISINT,...) is input(??best32.) with mod 1 = 0, so 7.0 and
1e3 are integers and abc and 1.5 are not. Each answer was recorded from the
synthetic estate first:
{table:'NOT_A_LIBDS'} -> Invalid table
{filter:'abc'} -> Invalid filter
no table at all -> Invalid table (was a
ReferenceError before)
{type:'FOOBAR',table:'NOT_A_LIBDS',filter:'abc'} -> Invalid table
Finding 2 - the tests waited 100ms for a fire-and-forget write. The cause was
the writers, not the tests: getddl, getrawdata and exportdb used the async
fs.writeFile for the headers file, while %mfs_httpheader (fopen/fput/fclose)
and dcMockUtils.mpStreamfile are synchronous. Those three now use
fs.writeFileSync, so the file is complete when the program returns; the waits
are gone, the positive assertions read the file directly, and the absence
assertions mean "never written" rather than "not written yet". No polling
remains in the file.
Also: mock-settings.test.js renamed the control library to DC7QATEST to prove
getrawdata follows the settings file - nine characters, which the estate
rejects as "Invalid table" for a LIBREF. The fixture is now DC7QATST. That was
a test defect the new validation exposed, not a product one.
Tests: download-headers.test.js is 11 tests, and the two new ones were run
against the old code first - dropping either check fails its own test. Whole
mock suite: 144 tests, 144 pass. parity/check.js unchanged.
Finding 1 on #510 was that getddl sent an octet-stream, quoted-filename header
the estate never sends. The same defect was in every other download path, so
this fixes the class rather than the one service. Every value below was read
from the synthetic estate's own response, except where noted.
public/getrawdata, from getrawdata.sas and its recorded answers:
SAS application/text LIBREF.DS.sas
CSV application/csv LIBREF.DS.csv
EXCEL openxml sheet LIBREF.DS.xlsx (from the service source:
PROC EXPORT xlsx fails on the synthetic
estate with syscc=1012, so it cannot be
recorded there)
MARKDOWN text/markdown LIBREF.DS.md
PDF application/pdf LIBREF.DS.pdf
- the filename is the upcased LIBREF.DS plus the type's extension, unquoted
- WEBCSV and WEBTAB send no headers at all: the estate writes those rows
straight to _webout, and a content-disposition would advertise a download it
does not offer
- a type outside that list aborts "Invalid type" with mac=&_program..sas before
anything is streamed, which is what the estate answers for FOOBAR
admin/exportdb: %mp_streamfile(contenttype=TEXT, outname=&dc_libref..ddl), so
application/text and an unquoted <libref>.ddl - the mock sent text/plain.
The shared mpStreamfile helper in dcMockUtils (exportconfig, getauditfile) now
writes one header per line with the filename unquoted, as %mfs_httpheader does,
so the zip downloads stop quoting a name the estate does not quote.
Tests: sas/mocks/test/download-headers.test.js (renamed from
getddl-headers.test.js) now covers all four paths - 9 tests. Each of the new
assertions was run against the old behaviour first and fails there: text/csv
instead of application/csv, a quoted filename, and WEBCSV aborting instead of
streaming. Whole mock suite: 142 tests, 142 pass.
The parity gate masks the whole generated-by comment and carries no response
headers, so neither the local-time fix nor the header shape was visible to any
test. Both are now unit tests in their own file, which fixes TZ to a zone three
hours from UTC so a return to the UTC getters cannot pass by coincidence:
- the printed clock is the UTC clock plus the zone's offset, checked to within
the two minutes a run can take
- the header file holds exactly the two lines %mfs_httpheader writes,
application/text and an unquoted filename
- an aborted request still describes no download
Each was run against the pre-fix behaviour first: the UTC getters fail the
first test and the old single-line quoted header fails the second. Whole mock
suite: 136 tests, 136 pass.
Six findings from the review of #510, each checked against the synthetic
estate's own public/getddl response before being changed:
- the download is an application/text attachment, not octet-stream, and the
filename is unquoted: the estate sends `content-type: application/text` and
`content-disposition: attachment; filename=<libref>_<ds>.ddl`
- the DDL header timestamp is the session's local time, as %sysfunc(datetime(),
datetime19.) is, not UTC - the estate answers 07NOV2025:22:27 where UTC
getters give 21:27, and the projection masks the comment, so nothing else
could see the drift
- the headers are written after the DDL is generated, as %mp_streamfile runs
after %mp_ds2ddl, so nothing is sent before a request that cannot produce a
file has aborted
- the refusal of an unknown flavour was invented: getddl.sas passes it to
mp_ds2ddl, which has branches for SAS, TSQL and PGSQL only and leaves the
file holding its opening comment. The estate answers that as a 200 download,
and the mock now does too; a new parity scenario, unknown_flavour, pins it
- the "all forty control tables" claim in the DDL comment overstated what the
committed goldens show: MPE_TABLES is the table recorded, in three flavours
- getcols: drop the dead `libref = knownLib` assignment
Tests: node --test sas/mocks/test/*.test.js - 133 pass, 0 fail, and
parity/check.js reports 0 differences for every getddl scenario.
The README still listed a LIBDS naming a library the estate does not have as
outstanding work - the mock falls back to searching every libref and answers.
That is done: the library is validated first and the fallback is gone, and the
entry would have contradicted the code whichever of the two pull requests landed
second.
Folds in #515 (the two getcols aborts, the NOLIB scenario and its unit test) so
the mock parity work is one branch rather than one per finding.
getcols.sas answers two different things and the previous commit collapsed them.
"Invalid libds" comes from the shape check (%mp_validatecol - a libref of eight
characters or fewer, a member of 32 or fewer), and only from there. A LIBDS that
passes the shape and cannot be resolved reaches the second abort, msg=syscc=&syscc.
Probed on the synthetic estate, that second abort says syscc=3000 with a SAS
syntax error behind it for a well-formed unknown library, for a missing member in
a known library, and for WORK.NO_SUCH alike - a defect in getcols.sas rather than
a designed message. The mock reproduces it as it is, because that is what the
client renders in production today, and the comment says so rather than
pretending it is designed.
The scenario is re-recorded with NOLIB.MPE_TABLES, which passes the shape check;
the old value, NO_SUCH_LIB, is eleven characters and only ever exercised the
shape abort, so its golden proved less than its name claimed. The unit test now
covers the malformed, the unresolvable and the answerable cases separately.
The index map and the schema are two lists of the same column names, and nothing
else joins them: a later rename in one would leave getddl emitting CREATE UNIQUE
INDEX on a column the table does not have, while the test that exists to check
every table has a key kept passing.
Each uniqueIndex column must now resolve to a column of its own table,
case-insensitively, so a rename on either side fails here instead of in a
download.
Merges release/7.23, which carries the 4.14.0 pinned estate and the lazy-route
revert.
Review findings on this pull request, and one CI failure they predicted.
The CI failure was the case change, not the DDL: getdata's DTVARS, DTTMVARS and
TMVARS still listed col.name, so they read " some_date" while every header and
row key was "SOME_DATE", and the client matches those lists against the headers
case-sensitively - an Excel upload stopped treating those columns as dates
(the Cypress failure was '21958' where '1960-02-12' was expected, across several
specs). They go through payloadKey now, like every other name in the payload.
Also from the review: the payloadKey doc block had displaced mpGetcols' JSDoc,
so mpGetcols was undocumented and payloadKey was described as taking a table's
columns; it sits above the block again. ddlDialectColumn's unreachable null
return is gone, and an unrecognised flavour aborts rather than quietly
answering SAS DDL under a label it was not asked for.
The intermittent suite failure the reviewer saw is the harness pruning a
session directory under a pending async header write; #512 carries the same
fix, and this branch needs it too or the gate fails at random.
getrawdata's CSV header still upcases inline: that service carries its own
helpers rather than loading dcMockUtils, so payloadKey is not in its scope.
getcols.sas validates LIBDS against the libraries it can assign before it reads
anything (%mpe_validatecol, then dc_assignlib), so a LIBDS naming a library the
estate does not have aborts "Invalid libds". The mock searched every libref
instead, so NO_SUCH_LIB.MPE_TABLES answered with the columns of the copy that
does exist in the control library - a request the estate refuses, answered with
data from somewhere else.
The named library is resolved the way the estate assigns it, and a missing or
malformed LIBDS reaches the same abort. A member missing from a library that
does exist is left alone: the estate fails inside its query there, and there is
no message worth copying.
Recorded as public/getcols/wrong_libref.
Also fixes an intermittent suite failure this work exposed: the harness pruned
earlier session directories, so a service whose headers are written with the
async fs.writeFile could land after its directory was removed, and the ENOENT
took the test process down. Each run still gets its own directory - that is what
stops a stale body being served - and the estate is removed when the process
exits.
public/getddl answered the SAS text for both remaining flavours, which the
download menu labels by name. Each is generated from the same table metadata
now, and each was checked against the estate's own output before it went in:
all forty control tables, byte for byte, in all three dialects.
The dialects differ in more than type names. Postgres quotes with double quotes
and declares its key as a second statement after the table; TSQL brackets the
identifiers, carries the key inside the create table as a UNIQUE NONCLUSTERED
index, and turns every column LABEL into a sys.sp_addextendedproperty block
after the GO. Only a format whose name starts with DATETIME becomes a timestamp
in either - a DATE9. or E8601DT26.6 column stays DOUBLE PRECISION or
decimal(18,2), which is what the estate does and is not what a format list would
suggest.
Recorded as public/getddl/mpe_tables_pgsql and public/getddl/mpe_tables_tsql,
so each dialect is a gate rather than something re-checked by hand.
Revert commit 7d5f1e85.
The lazy split makes the Cypress suite fail, reproducibly, on two separate
runs of two different trees that contain it:
run 3441 (cc43512) viewer-labels 1, full-table-search 1, viewbox 1,
row-cell-limits 4
run 3449 (5aa8cd7) viewer-labels 4, full-table-search 1, viewbox 1,
row-cell-limits 4, deploy-folder-picker 7
Both runs show the same failure family, and it is always the click that
selects a node by name:
CypressError: cy.click() failed because the page updated as a result of
this command ... the subject is no longer attached to the DOM
AssertionError: Expected to find element:
#hotTable .ht_clone_top .htCore thead button.changeType, but never found it
AssertionError: ... with the selector: .clr-treenode-link but never did
The tree is re-rendered while the spec is mid-chain: the click starts a
navigation, the route's chunk is fetched, and the node that was clicked is
detached before the chain continues. With the modules in the main bundle there
was no fetch, so the race almost never fired; a chunk fetch widens it enough to
fail four to five specs per run.
Revert restores a green suite on the release branch. The split re-lands with
the specs' click chains broken up so a re-render cannot detach a subject the
chain still holds - that is the change that makes the test robust to the app
doing what it does, rather than keeping the app eager to keep the test passing.
Not reverted: #507 (text-encoding and handsontable polyfill imports), whose own
run 3442 was green with the suite in this state.
Measured cost of the revert: the audit at 3449 read 0.93/0.80/0.79/0.70 on
performance with both the compressed server and the split in place; the split
alone moved those routes by roughly +0.05 when it landed (run 3442 vs 3441),
so the revert gives back that much and keeps the compression gain.
public/getddl answered one fixed Postgres-flavoured body for every request -
DC_JSLIB.MPE_X_TEST whatever dataset was asked for - which is the last read path
the client calls where the mock invented its answer.
It now builds the SAS-flavour DDL from the table file, the way mp_ds2ddl builds
it from dictionary.columns and dictionary.indexes: the header, the create table
with each column's type, format, NOT NULL and label, and the unique index. The
list-output spacing was derived from the estate's own answers and reproduces
them byte for byte for all forty control tables.
Recorded as public/getddl/mpe_tables, so a mock change that alters the text
fails the gate. The header credits a user and a timestamp, so the projection
masks both - a user id is not something to publish, and the comparison is about
the DDL.
TSQL and PGSQL are not generated yet: a caller asking for one gets the SAS text,
and the README says so rather than inventing a second dialect without a
recording to check it against.
A control table on a real estate is created with a unique index - MPE_TABLES on
tx_to/libref/dsn, MPE_X_TEST on primary_key_field, and so on - and mp_ds2ddl
turns that into the CREATE UNIQUE INDEX block of a DDL download. The mock's
tables had columns and rows and no key at all, which is why getddl could only
ever answer a fixed sample.
Each table file now carries `uniqueIndex` (name plus the ordered columns), taken
from the synthetic estate's own DDL: 39 of the 40 control tables have one,
MPE_REQUESTS is the exception the estate also has, and the format catalog marker
has none because it is not a dataset.
The columns also move to the case SAS creates them with. A table declared with
`alert_event` keeps that spelling in dictionary.columns and in the DDL, while
every payload the client reads reports the name upper case. The files now store
the creation case and the services build their keys through payloadKey(), which
fixes the DDL and leaves the payloads alone.
Two metadata gaps went with them: MPE_LOADS.csv_dir and MPE_MAXKEYVALUES.keytable
are NOT NULL on the estate and were not here.
EOF2
echo "--- second commit ---"; git log --oneline -1 | cat
Three claims in the comment block did not survive checking: the 3.14.0
paragraph described the utils floor as still open when 4.14.0 is the release
that raises it, the 3.12.3 paragraph named a guard that does not exist, and
the 4.0.0 paragraph was incomplete.
The logout one is the interesting correction: the estate does call
GET /SASLogon/logout - @sasjs/adapter's logOut() is a GET on that route (client
sas.service.ts logout() -> sasjsAdapter.logOut() -> AuthManager.logOut()), so
the paragraph now says what the estate does touch and why the status change
does not matter to it.
4.14.0 is the first @sasjs/server release that compresses its responses and
sends cache headers for the app it streams (sasjs/server#575), which is what
the audit has been measuring: the app is a 4.83MB build output, and the
previous pin served it raw and uncacheable.
Verified on the published package before pinning it here: 4.14.0 boots, the
mock estate deploys and seeds against it, and a deployed asset now answers
Content-Encoding: gzip
Cache-Control: public, max-age=31536000, immutable
with index.html answering max-age=0, must-revalidate. Measured on the previous
pin, same build and machine: FCP 8.48s and 0.34 performance raw, 2.81s and
0.49 with the same bytes compressed.
4.0.0's breaking change is the auth statuses (401 where 403 was, and the
undocumented DELETE on logout is gone). The client treats any non-2xx as a
refusal; the suite running against this pin is the check.
sites.yaml and release.yaml move with it - same mocked estate, one version.
The catch that keeps the app booting on a build layout without the root
package.json left nothing behind: a misversioned build looked like a clean one.
It warns now, so the fallback is diagnosable from the console the app already
logs to.
Only E8601DT, E8601DA and E8601TM are printed as ISO. The zoned E8601DZ and
E8601TZ, and the interval forms E8601DTM/DN/MN, fall back to the SAS shape -
they would need their offset or interval suffix printed too, and no table on the
estate carries one. The comment now says that instead of implying the whole
family is covered.
The case added for the settings read deleted services/settings.js, which the
reader tolerates on its own, so it said nothing about reading a moved library.
It is replaced with the two cases that do:
- a drive whose control library is TESTFIX, with settings.js naming it, seeds
the fixtures into TESTFIX and creates no DC_JSLIB at all;
- a drive with no compiled dcSettings.js, the parser the readers load, still
seeds into the default rather than dying at load.
The settings guard also stops accepting a bare dcReadSettings( call: that
function only exists once dcSettings.js has been loaded, so the call now counts
as a read only when the same file loads the parser. dcInit/loadDcSettings are
accepted as before.
The runtime package went with the import, but its types package stayed - in
dependencies, so a type-only package was installed and shipped. Nothing
references the types now that the call site uses the platform TextDecoder.
Review found two more loadChildren in the eager form, in
home-routing.module.ts: excel-maps (XLMapModule) and multi-load
(MultiDatasetModule). They were inside the home chunk, so opening the app's
default screen /home/tables pulled both.
Both now return dynamic imports, so each gets its own chunk (47 kB apiece) and
the tables screen no longer pays for them. The initial total is unchanged -
home was already lazy - which is why the earlier measurement did not show
them.
Two findings on #506.
createdemodata eval'd the compiled settings program unconditionally, so a drive
that has the fixtures but not that file went down at load with an ENOENT before
it could report anything. The read is guarded: an absent program evals a stub
that answers the same default dcReadSettings gives for a missing file, and the
service settles on the control library makedata creates instead of failing. A
settings program that exists and does not parse is still a broken deployment,
and is left to fail loudly.
viewIsoDateTime and viewIsoTime clamped a rounded fraction at scale-1 instead of
carrying it, so a value 1e-7 under the next second printed .999999 of the second
before it where SAS prints the next second, .000000. Both now round to the
printed precision first and split the result, which carries into the second.
Both are pinned: createdemodata seeds with settings.js deleted, and an
E8601DT26.6 value one rounding error short of the next minute prints 03:05:00.
app.routing.ts already declared loadChildren, but as the eager form - a
function returning the module class - so Angular bundled every screen into
the initial chunk and loaded it before the first paint. Only exceljs, which
uses a dynamic import, was ever split.
Each route now returns a dynamic import, so webpack emits a chunk per screen:
before after
main 8.96 MB 3.50 MB
initial total 10.29 MB 4.83 MB
initial transfer 2.29 MB 849.30 kB
The modules declare no providers, so a lazy module cannot introduce a second
instance of a service the shell already holds - the failure this change would
otherwise risk.
polyfills.ts imported the grid so that every route parsed it before the app
bootstrapped - the login screen included. Every module that uses Handsontable
imports it itself (editor, viewer, viewboxes, the shared utils), so the
polyfill import was redundant as well as misplaced.
It does not move the grid out of the initial bundle on its own: the viewer is
an eager route, so Handsontable still ships in main until the routes are split
(504). What it fixes is the pretence that the grid is a polyfill.
text-encoding was imported for one call in the licence service:
new encoding.TextDecoder().decode(value). TextDecoder has been native in
every browser this app targets for years, so the polyfill bought nothing and
cost 701 kB in the initial bundle (encoding-indexes.js 607 kB + encoding.js
94 kB).
Measured on the production build: main 8.96 MB -> 8.41 MB raw, initial total
10.29 MB -> 9.74 MB. The dependency goes with it.
Three findings from the release review.
src/version.ts read the root package.json unguarded on the no-tag path: a build
layout without that file threw at module load, taking the app down to report a
version. It falls back to v0.0.0 now.
lighthouse-report.mjs looked for its marker in the first fifty comments, so on
a pull request past that count the "updated in place" contract silently became
a new comment on every push. It pages until the marker is found, or until a
page comes back short.
lighthouserc.js justified the 0.85 accessibility floor as sitting "just below
the observed minimum" while the assertion takes the median - a route 0.09 under
the floor passes when the median clears it. The comment now says what is
measured, that this is the trade the threshold makes, and that
aggregationMethod 'pessimistic' is the lever if the floor must hold for every
route (accessibility would pass that way today, performance would not).
createdemodata was the one service the settings migration left naming
DC_JSLIB, with a comment asking the reader to keep two constants in step. On a
deployment that moves the library it fails with "run services/admin/makedata
first" whatever makedata did.
It reads dcReadSettings now, the way getrawdata does - loading dcSettings.js
alone, since this service's own table helpers would collide with dcMockUtils.
The test that exists to catch exactly this exempted any file whose text
mentions settings.js, and createdemodata's comment mentioned it, so the
constant passed. The exemption now requires a call to a parser
(dcReadSettings/dcInit/loadDcSettings), which is what "reads the settings"
means.
EOF2
fix(build): guard the version fallback, and page the Lighthouse comment
Three findings from the release review.
src/version.ts read the root package.json unguarded on the no-tag path: a build
layout without that file threw at module load, taking the app down to report a
version. It falls back to v0.0.0 now.
lighthouse-report.mjs looked for its marker in the first fifty comments, so on
a pull request past that count the "updated in place" contract silently became
a new comment on every push. It pages until the marker is found, or until a
page comes back short.
lighthouserc.js justified the 0.85 accessibility floor as sitting "just below
the observed minimum" while the assertion takes the median - a route 0.09 under
the floor passes when the median clears it. The comment now says what is
measured, that this is the trade the threshold makes, and that
aggregationMethod 'pessimistic' is the lever if the floor must hold for every
route (accessibility would pass that way today, performance would not).
The viewer printed one shape for every datetime - a two-digit year and a
two-decimal fraction - whatever format the column carried, while the same
payload published that format in `$viewdata.vars`. A column formatted
E8601DT26.6 therefore answered 02JAN26:03:04:05.00 where the estate prints
2026-01-02T03:04:05.000000.
The E8601 family is now printed as ISO 8601, at the precision its format names
(a width with no fraction prints none), and every other format keeps the shape
the recorded response pins, which is what the DATETIME19.3 columns already
matched.
Two comments outran their evidence.
The getcolvals docstring said no fixture reached a decorating format, while the
same commit adds a dollar8.2 one that does - the padding and decimals are
pinned, the symbol is not. COMMA and SIZEKMG are the formats with no fixture,
and it says that now.
The _PROGRAM exemption was justified by the path being uniform across
deployments. It is not: it carries the target's appLoc. The reason that holds is
that the path is the route of the request that was just made and is already in
that request's URL, so publishing it adds nothing. Both the projection comment
and the README say that instead.
Three findings, all of them about the same field.
`sasNumericFormatted` only parsed `BEST<w>.<d>`, so a DOLLAR8.2 column fell back
to a fixed width of 12 while `$vals` declared $8./8 - the value contradicted the
metadata a caller sizes the column from. Any `NAME<w>.<d>` now yields the width
and decimals, and the declared width is the fallback; a decorating format is
padded and rounded but still arrives without its symbol, which the comment says
rather than an implementation it does not have.
The `formattedWidth` ternary had a temporal arm that nothing could reach - every
temporal column is numeric, so the numeric arm answered it - and the docstring
justified a 200 fallback no call site asked for. The width is now computed once,
before the values are read, from the numeric arm or the character length, and the
fallback is the caller's.
The formatting itself had no test with a value in it: the MPE_AUDIT fixture is
empty, so only metadata was asserted. A mock table with rows and a formatted
numeric now pins the padding and the decimals, for best12.2 and for dollar8.2.
The comment and the README explained the exemption by what it buys - a MAC
compared rather than masked - which was not the reason. A service route is not
sensitive and is the same on every deployment, so it is dropped with the rest of
the envelope but not denied; that the abort's MAC therefore arrives as it is is
a consequence, and the diff comparing it is incidental.
The manifest lists a report under its route while that report's own finalUrl
can have lost the fragment, so matching the two on the URL printed the same
run twice - once under #/home/tables and once under an empty label. Dedupe
by file instead, and label a navigation that ended on the app root as '/'.
Four findings from the review of this branch, all of them comparisons against a
live estate response.
`public/getcolvals` hardcoded best12 for every numeric column. The estate
formats `formatted` with put() at the column's own format - mf_getVarFormat
with force=1 - so a BEST32. column reports $32./32 and pads the value to 32,
and the metadata's unformatted copy keeps the column's label rather than the
variable name. A widthless format now gets its own default width: the estate
answers $16./16 for a DATETIME. column, where the fallback was 200.
The same service also searched every libref, so a library the estate does not
have still answered for a table that exists elsewhere. The named libref is
resolved first and refused as "Invalid libds" - the message %mpe_validatecol
gives, before the member is looked for; a missing member in a known library
stays "Target table not found".
`public/getcols` sorted the column list with localeCompare, the ordering this
branch already fixed in viewtables. SAS compares the character value, where
'_' sorts after every letter, so A_B comes after AB; the list now sorts that
way, with a fixture whose names differ only in underscore placement.
An empty getcolvals result answered empty strings where the estate sends one
row of SAS missings - a blank for the character and null for the number.
Recorded from the estate: a formatted number (MPE_AUDIT.OLDVAL_NUM), a
widthless temporal (MPE_DATASTATUS_CATS.CREATED) and a wrong library that names
a real table. The header-file test now polls for the line instead of waiting a
fixed 20ms.
The `<SERVICE>` placeholder this branch added to publish the getddl abort also
projected the mock's MAC, so a mock reporting a different program would have
compared equal to the estate. The service path is the app's own route, identical
on every deployment, so it is not an identifier to protect: `_PROGRAM` is now
dropped from the response but exempt from the deny list, and the MAC is compared
as it arrives. The golden is re-recorded with the literal path.
The FQDN class grows a lookbehind so a path ending in `.sas` is not read as a
hostname - the abort's MAC ends that way.
The README says which envelope member is dropped but not denied, and its runbook
step 1 now names the raw capture that holds the control library: the committed
startup golden masks `DCLIB` to `<LIBREF>`, so the step could not be followed
from it.
The audit's numbers were only visible in the job log and the artifact. One
comment per pull request, rewritten in place on every push (the body carries
a marker the script looks for, so a re-run replaces the table instead of
adding a second one), with a row per route and the lowest accessibility
score against the gate.
.gitea/scripts/lighthouse-report.mjs reads the reports, and posts through the
API with the workflow's token; a comment that cannot be posted logs and
leaves the job's verdict alone, because the audit's own step is the gate.
The job needs pull-requests: write (and issues: write, which is where Gitea
keeps pull request comments) for that and nothing else.
The collect comment still argued numberOfRuns:1 from "accessibility 1 - a
fixed score that does not vary with load", which is no longer the threshold
and was only ever true of an empty estate. It now states the trade against
the scores the audit actually measures (0.87-0.96, issue 497).
The assert comment records why the floor is global rather than per route:
lhci strips the fragment, so the seventeen routes are one URL to it and the
assertion aggregates their median - assertMatrix with matchingUrlPattern
cannot express per-page allowances here.
Also drops include-hidden-files from the artifact upload: that is a v4.4
input and v3 warns about it as unexpected on every run.
`public/getddl` for a dataset that is not there and `public/getcolvals` for a
table that is not there - the two aborts the mock used to answer with a body.
Both carry one accepted difference, `SYSCC`, for the reason the existing abort
scenarios already record: the estate's wrapper adds it to an aborted response,
and a JS program's envelope is finalised at CLOSE.
The README gains the refresh runbook the status section kept promising: deploy
first, how to find the estate's control library, what each gate line means,
and how to read the two refusals.
It also names what cannot be recorded yet and why. The "not authorised" abort
needs a non-admin login, because %mpe_accesscheck lets the admin group through
and the synthetic estate's user is in it. The getddl body needs index metadata
the mock's table files do not carry. And getcols' libref tolerance is a
behaviour change that needs a survey of its callers first.
Two requests that the estate refuses were answered as if they had worked.
`public/getddl` streamed its fixed sample body for any request at all - a
dataset it has never heard of, an unassigned library - and the message it did
throw for a missing `ds` carried two dots where SAS writes one. It now resolves
the member through the mock's own table files and aborts with SAS's message,
"Dataset <libref>.<ds> was not found", reported against the program because
that is the `mac` the service passes to `%mp_abort`.
`public/getcolvals` answered an empty dropdown for a table that is not there.
The estate aborts from `%mp_derivedvars` before any values are selected, so the
mock does too, with that macro's own MAC.
The projection needed one rule to publish either golden: `%mp_abort(mac=
&_program)` puts the service's own path in the payload, and `_PROGRAM` is an
envelope literal the deny list refuses. A service path is now `<SERVICE>`,
which is the app's own route rather than anything about the estate.
Both rules are asserted in test/fault-paths.test.js.
Recorded from the synthetic estate: the library browse (public/viewtables),
a dropdown's values for a character column and for a numeric one
(public/getcolvals), and an Excel map (editors/getxlmaps). The first four
scenarios recorded in the earlier pass now gate the download that already
matched - public/getrawdata had no differences left but still carried
"gated": false, so a regression there would have been reported and not failed.
public/getgroups is deliberately not recorded. Its answer is the SASjs
Server's own group list - all-users, public, and every group created on that
install - so a golden would be a copy of one estate's setup, and the
difference is in the rows rather than in a path that `accepted` can name. The
mock derives its groups from MPE_GROUPS, and the startup scenario already
accepts the admin group's name as estate state. The README now says so, and
names the two paths still to align: the security abort a table the caller may
not see produces, and getddl's hardcoded body.
The parity recorder found two payload differences that no test covered.
`public/viewtables` listed the mock's own file name for the format catalog -
MPE_X_CATALOG, a dataset the estate does not have - where SAS lists a catalog
as <name>-FC. A member with no variables is the catalog, so it is named that
way, and the list is now sorted in SAS byte order: '_' is 0x5F and sorts after
'L', so MPE_XLMAP_RULES precedes MPE_X_CATALOG-FC, which localeCompare got
backwards.
`public/getcolvals` reported the metadata of a column it had not looked at:
$32767. and a length of 32767 for the formatted value whatever the column, an
empty SASFORMAT for a column whose format is blank (SAS answers a space), and
a bare number where put(x,best.) right-justifies into best12. The metadata now
describes the two variables the service creates - $<len>. at the column's own
length, best12 for a number, the format's width for a temporal - and the value
is formatted to match.
Each rule is asserted in test/read-paths-payload.test.js so a regression names
the rule rather than a JSON path.
Seeding the estate made the audit measure the app with data in it for the
first time, and accessibility is 0.87-0.96 across the seventeen URLs rather
than 1.00. Those are real defects, tracked in 497; the gate cannot stay at 1
while they exist, and leaving it there would block every pull request for a
reason unrelated to what they change.
0.85 sits just below the observed minimum so a busy runner does not turn the
threshold into a coin flip, and goes back to 1 as 497 clears.
The server refuses to run with CORS=enable and no WHITELIST, which is what
made the job fail at startup. CORS is not needed here and was only ever
needed because the audit used to run against ng serve on :4200: the app and
the services are both served by this server on :5000, so every call is
same-origin. The Cypress job keeps its CORS setting, because there the
browser really is on another origin.
The audit's URLs are mostly data views - tables, view/data, the review
lists, excel-maps, multi-load - and the job deployed the services without
the database behind them. makedata builds what a real deployment builds,
createdemodata adds the Cypress fixtures (TESTDATA, the search and wide
viewers, the multi-load library), and both run here before lhci, exactly as
the Cypress job runs them. makedata's payload asks for OUTFMT=JSON, so a
failed seed fails the job instead of auditing an empty estate.
Two things carried over from lighthouse.yaml verbatim did not hold:
- The job ran cd sas/mocks, which has no sasjsconfig.json. The CLI then
searched for the project and npm ci installed whatever package.json the
prefix walk found (sas/, 159 packages), neither of which is what the step
is for. It now cds into sas/mocks/sasjs, as the Cypress job does, and
installs nothing: the pinned global CLI carries @sasjs/core.
- The built copy was patched to appLoc /proj/sasjs/genesis-mocks, but the
only target this job deploys - server-ci - publishes services at
/Public/app/dc, which is also what the production build already carries.
The audited app was therefore calling a path with no services behind it.
The patch step goes: the build's own appLoc is the deployed one, so the
two cannot drift, and the unused replace-in-files install goes with it.
Its docstring still described the trimmed comparison the previous commit
removed, which is exactly the kind of comment a later reader follows back into
the bug. It now states that cells are kept as they arrive, padding included,
that a leading or trailing blank is part of a SAS character value, and that only
the framing - the quotes and the line endings the state machine consumes - is
removed.
Rebased onto release/7.23 after #492 and #494 merged: the scenario list carried
two entries with the id mpe_tables (public/viewdata from #492, public/getcols
from this PR), which the merge made a conflict; both are kept, keyed by service
and id together, which is how the golden path and the checker already treat
them. 114 pass, 0 fail, eight scenarios ok.
Three findings from review.
- the abort envelope: no service depended on the CLOSE-time assignment. Members
are on the session's webout file as they are added, and the server serves that
file whether or not the program finished (api/src/controllers/session.ts
sessionOutput returns webout.txt, catching to ''), so a caller receives what
was flushed; every abort writes sasjsAbort first, so an abort is never an
empty body. The comment now says that, says that dropping CLOSE is estate
parity (a SAS soft abort stops before the envelope is finalised), and keeps
the note that the estate's wrapper is what adds SYSCC to its own aborts.
- parseCsvTable trimmed every cell, quoted content included, so the download
comparison could not see a leading or trailing blank - the very class of
difference the viewer work just established. Cells are kept as they arrive,
a quote opens a field only at its start, and a test pins all three cases.
Removing the trim changed no scenario: all seven still ok.
- provenance: parity/README.md now has a section saying where each _meta field
comes from. sas is the envelope's SYSVLONG, which only a SAS runtime emits, so
a golden carrying it cannot have come from the mock; a scenario that answers
with something other than a SAS JSON envelope (the getrawdata CSV) records
unknown, and that is why that one golden says so while its accepted
differences still come from a live response. estate is the target's declared
flag, which record.js refuses to do without; dc is the repository version at
record time and is named as the one field not read from the response.
Four findings from review.
- mpAbort's comment still said the runtime puts SYSCC on an aborted envelope,
which is what I believed before checking: a JS program stops before the
envelope is finalised, so an abort carries neither SYSCC nor SYSENCODING. The
comment now says that, and says that the SAS estate's wrapper is what adds
SYSCC to an abort - the difference the abort scenarios declare.
- runService removes the previous run's session directory when it creates the
next one: a directory is only read back through estate.weboutPath, which has
moved by then, and a test file that runs many services was leaving one behind
per call.
- brCheckPath: both callers are macro-bound (a DIRLIST parent and a WRITECHECK
target that is written into a filename statement), so rejecting % and & at
every call is right; the comment says so now instead of leaving a reader to
check.
- getdata's abort message is the recorded text - 'dataset X.Y does not
exist!!' with MAC mpestp_getdata.sas, from the estate's own response - so the
mock keeps it rather than inventing a diagnosis. The control-library mismatch
that could put a misleading name in that sentence is the hardcoded DC_JSLIB,
and #494 removes it.
Four findings from review.
- stagedata: the abort needs a registry that loaded and does not hold the row; a
drive where MPE_TABLES itself is absent is a different failure and the message
would be a lie.
- record.js checks for {{LIBREF}} before the first substitution rather than only
before the POST, so a scenario that names its table in the query string fails
with the same instruction instead of recording against an empty libref.
- makedata_actions: the macro triggers are asserted one at a time ('%', '&', and
a control character), so removing one from brCheckPath fails the test rather
than being masked by the other; the path-shape case is its own test.
- the getcols drop is parity, not a regression: the estate's getcols returns
NAME, LENGTH, VARNUM, LABEL, FMTNAME, FORMAT, TYPE, DDTYPE and nothing else,
and the picker's column-level security comes from getdata's dqrules/READONLY.
A fallback that needs CLS_RULE is a change to getcols.sas, not to its mock.
The abort shape change (no SYSCC/SYSENCODING on a soft abort) is estate-wide and
deliberate - it is what an aborted SAS response looks like.
Four findings from review.
- stagedata aborted on the target's data file existing, which is the state of a
first load into a newly registered table - staging there has to work. The gate
is the registration now (finalReg, already computed above), and the comment
says why: a table registered nowhere is the case a user hits when it is dropped
between loading the grid and saving.
- getcols rebuilt a character column's FORMAT as '$<length>.' whatever the column
declared, so a $CHAR10. or $UPCASE. column was described with a format it does
not have. mpGetcols' own FORMAT wins when there is one, the length-derived
default applies only when there is none, and an undeclared length no longer
produces '$0.'.
- getrawdata pushed the CSV header row for every requested type, so the EXCEL,
MARKDOWN and PDF bodies - which stream the same lines under their own content
type - began with a CSV header. Only type=CSV gets one now.
- the harness put SYSCC on the envelope at OPEN, claiming the runtime does that
for a program that never closes. It does not: sasjs/server's JS runtime
finalises the envelope at CLOSE (createJSProgram.ts -> weboutFinaliseStatement),
which is why a JS abort carries neither SYSCC nor SYSENCODING. The stub is back
to adding both at CLOSE, and the two abort scenarios declare the difference
they now show: the SAS estate's wrapper adds SYSCC to an abort, a JS program
stops before the envelope is finalised. The message and macro name the client
renders are unchanged.
check.js - seven scenarios, five at zero accepted differences.
Suite: 99 pass, 0 fail.
The runtime creates a session directory per execution - <root>/sessions/<id>
with webout.txt and stpsrv_header.txt beside it - so a program that only assigns
_webout cannot be served the previous request's body. The harness used one
directory for the whole estate, which is how a stagedata abort once answered for
a download.
runService now creates <root>/sessions/s<n> per call and points weboutPath and
_SASJS_WEBOUT_HEADERS at it, as the runtime does, and estate.weboutPath follows
the most recent run so a caller can still inspect a response afterwards. The
hooks test writes beside weboutPath and reads it back from the session directory
of that run rather than from a hardcoded s1.
Two more scenarios, and both needed recorder support that did not exist.
editors/stagedata on a table that is registered nowhere (0 accepted
differences). An edit submission sends *two* table inputs - the control table
and the edited rows as jsdata - so the recorder now posts several parts and the
harness declares several inputs, as the runtime does. The mock answered a
completed body where the service aborts with "Target table not found"; it aborts
that way now, and mpAbort no longer closes the body, because a SAS soft abort
does not reach the state that carries SYSENCODING.
public/getrawdata (0 accepted differences). A download answers CSV rather than
JSON, so scenarios can now declare format: csv and the body is parsed to
{header, rows} on both sides before the diff - SAS quotes and pads each cell to
the variable's width, so a text diff would fail on spacing that says nothing.
The mock was missing the header row, was exporting the registration's validity
columns (getrawdata.sas drops them: data staged(drop=&txfrom &txto)), and broke
any value containing a newline across two records because it did not quote it.
The harness now clears the response file before each run. Without that, a
service that only assigns _webout was handed the *previous* service's body -
which was hiding a stale-result bug in a DIRLIST test: the mock accepted
/tmp/%sysfunc(system(id)) as a parent, and the test passed because it was
reading the abort from the run before it. brCheckPath refuses '&' and '%' now,
which is the guard the test was written for.
check.js - seven scenarios, five of them at zero accepted differences:
ok public/startupservice/default (8 accepted difference(s))
ok editors/getdata/mpe_tables (1 accepted difference(s))
ok public/viewlibs/default (1 accepted difference(s))
ok public/getcols/mpe_tables (0 accepted difference(s))
ok editors/getdata/missing_table (0 accepted difference(s))
ok editors/stagedata/target_missing (0 accepted difference(s))
ok public/getrawdata/selectbox (0 accepted difference(s))
read-paths.test.js covers the two-input abort and the download's header/validity
columns/quoted newline. Suite: 99 pass, 0 fail.
Two more scenarios, both read-only, and both silent in the mock until now -
neither of them broke a Cypress spec, they just tested less than they claimed.
public/getcols (0 accepted differences). The filter panel builds its operator
set and value quoting from this list, and the mock answered the *getdata* column
shape: CLS_RULE, MEMLABEL, DESC, LONGDESC and COLTYPE, with FORMAT blank for
every column. The estate answers NAME, LENGTH, VARNUM, LABEL, FMTNAME, FORMAT,
TYPE, DDTYPE - VARNUM being the column's position in the table rather than in
the sorted list - so the mock now emits exactly that.
editors/getdata on a table that is registered nowhere (0 accepted differences).
getdata.sas aborts, and the client renders the message; the mock answered an
empty grid with an empty dsmeta instead, which is a different answer rather than
a quieter one. It now aborts with the same message and macro name, and - like
mp_abort - stops before CLOSE.
That last point needed the harness to be more faithful: the runtime puts SYSCC on
the envelope whether or not the program closed it, while SYSENCODING only appears
on a completed body. The stub adds SYSCC at OPEN now and SYSENCODING at CLOSE, so
an aborted response matches the estate's shape instead of being accepted as a
difference.
record.js also cuts the response at SASJS_LOGS_SEPARATOR before parsing: an
aborted call comes back with the log appended, the same way a _debug response
does, and the recorder used to fail on the trailing text.
check.js:
ok public/startupservice/default (8 accepted difference(s))
ok editors/getdata/mpe_tables (1 accepted difference(s))
ok public/viewlibs/default (1 accepted difference(s))
ok public/getcols/mpe_tables (0 accepted difference(s))
ok editors/getdata/missing_table (0 accepted difference(s))
read-paths.test.js pins both fixes; the suite is 97 pass, 0 fail.
The audit needs the production build, and an artifact belongs to the run that
produced it (needs: cannot cross workflows), so as a separate workflow it
compiled the frontend a second time - 92-104s measured - and then started an
ng serve that nothing audited: every URL in client/lighthouserc.js is on
:5000, which the SASjs server feeds from client/dist.
As a job in build.yaml it takes the change decision from the build job
(the same inputs gate both), downloads client-dist, and patches appLoc in the
built index.html instead of compiling a patched copy of the source. The
second compile, the dev server and the source generator in this job are gone;
what remains is node_modules for lhci, the pinned CLI, the SASjs server and
the mock estate deploy.
The check name moves from 'Lighthouse Checks / lighthouse' to
'Build / lighthouse'; there are no required contexts to update. The
changed-path matcher already covers everything this job reads, so its only
change is a comment and the fixture that used to assert lighthouse.yaml was
not an input.
Three findings from review.
- the trailing-blank strip now matches only blanks. SAS pads with blanks, so a
value that ends in a tab is not padding and is not silently altered while the
same character mid-string survives.
- the `typeof val === 'number'` arm was unreachable: both branches above return
text (viewCell and formatSasValue), so the ternary is gone and a comment says
why a string is all that arrives.
- provenance: the docstrings cited "the recorded response" as the authority, and
the golden's `_meta.estate` says synthetic, which invites exactly the doubt you
raised. They now name the file and the field that answers it - `_meta.sas` is
9.4M9, read from the live envelope's SYSVLONG, which only a SAS runtime emits
and which the mock cannot produce - and point at the Provenance section added
to parity/README.md in #493.
Three findings from review.
- getrawdata no longer parses settings.js itself. The reader moved into
services/dcSettings.js, which dcMockUtils.loadDcSettings calls and which a
service that cannot eval the whole utility file (its own formatSasValue and
SAS_EPOCH_MS would collide) can load on its own. One file, one parser: the
greedy object match now parses JSON where it can and falls back to a
quote-agnostic pattern, and the ENOENT-only catch is in that one place.
- a behavioural test moves the control library and its data and asserts a
getrawdata export still comes from it. This was the least-covered path in the
change - static tests cannot follow a reader that does not load the utilities.
- viewlibs' comment still said a `|| 'DC_JSLIB'` fallback mattered; it lives in
loadDcSettings (which defaults dcLibref to DC_JSLIB), so the comment says that
and the dead `||` stays gone.
Three findings from review.
- viewdata trimmed both ends of a character cell. A leading blank is part of a
SAS character value and the viewer renders it; only the trailing blanks SAS
pads to are noise, so the cell is trimmed at the end and an all-blank cell is
still SAS missing. viewdata-cells.test.js fails on the previous version.
- viewDate floored a stored day count while the ISO branch rounded, so the same
instant could land on different days depending on how the row carried it. Both
round now.
- the two label members stay different. The finding was that cols[].LABEL uses
the data dictionary business label while $viewdata.vars[.].label uses the
physical one; the recorded estate response does exactly that - cols DSN is
"Dataset Name to be edited", $viewdata.vars.DSN.label is "DSN" - so unifying
them would break parity. The comment says so, and a test asserts the split
against the recorded answer so the next reader does not tidy it away.
Five findings from review.
- getrawdata's settings reader was a second parser that only matched a
double-quoted key and value, so a hand-edited settings.js would leave this one
service on DC_JSLIB while the estate used the configured library. The object is
parsed as JSON first (what makedata writes) and a quote-agnostic pattern is the
fallback, so both a written and an edited file are read the same way.
- the bare catch treated every read failure as "makedata has not run": an
EACCES or an encoding failure now surfaces instead of silently defaulting.
- `dcInit(...).dcLibref || 'DC_JSLIB'` was unreachable - loadDcSettings already
defaults dcLibref to DC_JSLIB - so the `||` is gone from all fourteen services;
it would have masked a changed settings shape as a silent default.
- the "Load shared DC mock utilities" comment had ended up below the eval it
describes in getdynamiccolvals, getcols and getcolvals; it describes the eval,
so it sits above it, and the note about which tables live in the control
library no longer names DC_JSLIB.
- merge order with #493 (getcols, getrawdata, stagedata): whichever lands second
is rebased; the getrawdata hunks are separate from that PR's CSV changes.
The settings guard now also accepts a `let` declaration and a service that keeps
the literal as a default while reading settings.js (getrawdata does).
Three findings from review, and tests that hold them.
- a DATE below the year 1000 printed three digits ('999'), so DATE9.'s
four-digit year is now zero-padded
- a stored ISO text reached a TIME column through the datetime branch, giving
seconds since 1960 rather than seconds since midnight; a time's day is
discarded, and a text without a zone is read as UTC rather than in the host's
zone (the control box sits east of UTC, which shifted every value by an hour)
- the datetime fraction is clamped at 99 as the time branch already was, so a
fraction that rounds up cannot print three digits
viewdata-temporal.test.js builds a table with DATE, TIME and DATETIME columns and
covers each renderer plus a stored ISO string; the year and the ISO cases fail on
the previous version of the service.
Three findings from review.
- validatefilter used fetchTable/colVal above the eval that defines them, so it
threw on every call - the conversion moved the eval down past the parse. The
eval and the dcInit read are back above the parse, with a comment saying why
they have to be. Reproduced and checked: the service answers now.
- a source-level test walks sas/mocks/sasjs/services/**/*.js and asserts the
invariant the copy-paste is supposed to keep: a service that loads the
utilities reads the settings after them, calls no utility before them, and
does not name the control library; makedata is exempt (it creates it) and a
hook is exempt (its body is eval'd in the calling service's scope). This is the
guard that would have caught the validatefilter slip, and it costs no estate.
- getrawdata reads dc_libref out of services/settings.js rather than naming
DC_JSLIB, so the documented CSV row-diff path works on a deployment with a
different DCPATH. It cannot eval dcMockUtils - its own formatSasValue and
SAS_EPOCH_MS would collide - so it reads the one value it needs; createdemodata
stays as it is, because it creates the library.
Two findings from review.
- viewTime was reached with the unrounded value (viewCell passed the number
straight through for TIME), so a stored 3661.9999 produced a fraction of 100
and printed "01:01:01.100". The value is rounded to two decimals on entry, as
the DATETIME branch already did, and the fraction is clamped as well so a
rounding edge cannot reach three digits.
- a temporal column with no format fell back on DATETIME19.3 whatever its
ddtype, so DATE and TIME columns were labelled as datetimes. The default
follows the ddtype now: DATE9., TIME8., DATETIME19.3.
viewdata's scenario is still at zero accepted differences, and the suite is 96
pass, 0 fail.
makedata writes services/settings.js - the JS twin of the settings program the
backend generates in %mpe_init - with dc_libref, dc_admin_group, dc_staging_area
and dcpath. dcInit() reads it. getdata and viewlibs used it; the other thirteen
services that load dcMockUtils still carried const dcLibref = 'DC_JSLIB' and
built their paths from it, so a deployment with a different DCPATH kept working
in two services and broke in the rest.
Each of those services now loads dcMockUtils first and takes its libref from
dcInit(driveRoot, appLoc).dcLibref, with 'DC_JSLIB' only as the default for a
drive makedata has not seeded. Services that computed paths from the constant
(stagedata's staging directory, getapprovals' data directory, gethistory's) have
the eval above that line, so the derived paths come from the settings too.
Two services do not load dcMockUtils at all - createdemodata and getrawdata
carry their own table and format helpers - so they keep naming the constant, and
say why in a comment: adding the eval just for the settings would collide with
their own helper names.
mock-settings.test.js gains a case that would fail without this: it renames the
control library, moves the data, rewrites the settings file, and asserts a row
written into MPE_SUBMIT comes back from approvers/getapprovals. With the
hardcoded service it is empty, which is the failure mode this removes.
The viewer was the third read path the harness could not see. Recorded from the
seeded estate it showed 364 differences; all of them were the mock's, and none
remain.
- cols: sorted by column name, and a column LABEL comes from the
MPE_DATADICTIONARY row for the lib.ds.column when there is one - viewdata.sas
augments mp_getcols the same way getdata does. The format name is upper case,
as SAS reports it.
- rows: a temporal value is rendered the way the estate renders it - two-digit
year and a two-decimal fraction ("01JAN60:00:00:00.00") - rather than by the
format label on the column, which reads DATETIME19.3. A character value is
trimmed and an all-blank one is a space, matching the missing-value rule the
data rows and the synthetic no-data row already share.
- $viewdata: the column's own metadata (num, 8 bytes, DATETIME19.3). The mock
reused the entry built for the edit grid, where the temporal columns have been
converted to character, so the viewer was told $200 char.
- sasparams.PK_FIELDS: the validity column ahead of the business key, which is
what viewdata answers ("TX_TO LIBREF DSN").
sas/mocks/test/viewdata.test.js pins the datetime text and the empty-table row.
getdata padded every blank character cell to a space, including one the row
holds as an explicit ''. That broke nine editor specs: the grid classifies a row
by comparing what it holds against the payload it was given (classifyRow), so a
payload whose empty strings had become spaces made every row read as modified on
load - the row header showed +/- and DC.ROW_STATUS answered 'M' where 'U' is
correct.
The rule is now the one SAS expresses: a cell the row does not carry is missing
and reaches the client as one space, and a value the row does carry is passed
through. MPE_TABLES rows omit their blank columns, so the recorded golden still
matches; the TESTDATA formula fixture stores '', and keeps it.
getdata-blanks.test.js pins both sides.
Every existing case set one state and left the rest empty, so the order the
component relies on was untested: a licence problem and an over-capacity state
were never rendered for a user who is also unregistered. Two cases now do that
and assert the problem/limit message with the registration text absent.
Making the problem state lose to the unregistered state turns the first one
red.
The banner is the only route into the licensing pages now, so it needs a state
for every way the licence can block the user, and each state needs to point at
the page that can fix it:
- a licence problem with no free tier (the startupservice-without-globvars
case, which previously fell through to 'not registered') reports the problem
and opens its encoded page;
- free tier keeps its update-key and problem links;
- over capacity, and an unregistered user who cannot register, point at
licensing/limit, which had no entry point once the forced navigation went;
- an unregistered user points at licensing/register when they can register;
- the expiry banner keeps its update link.
Moving the banner out of app.component means its states can be tested: ten
TestBed cases assert the message and the rendered link for each, including that
nothing renders before the startup data lands or when the licence is in order.
currentUserCanRegister finally has a reader - it decides between the register
and limit links - and the Cypress limit case follows the banner instead of
expecting a redirect.
A startupservice failure sets isAppActivated false but never emits
startupDataLoaded, so the banner would have claimed the user is not registered
when the app had not started at all. The banner now waits for the startup data,
which is the same flag that removes the loading page.
The incorrect-site-id subscription deactivated the app and then navigated to
the licensing key page. With the banner as the entry point the app stays where
it is: licenceProblem already carries the URL the banner's More details link
opens, and deactivateApp's redirect argument had no other caller.
Startup no longer moves the user: an unactivated app stays on the page it
loaded instead of being sent to licensing/key, and an unregistered user is no
longer sent to licensing/register. The banner in app.component reports the
licence state and links to the licensing page - Register while the current user
is not registered, Update key for free tier, and More details for a licence
problem - so the licence page is somewhere the user goes rather than somewhere
they are put.
With nothing forcing the user onto licensing/register there is nothing to lock
them onto it either, so LicensingGuard (canDeactivate) and its route hook go.
A licence problem found while the user is already on a licensing page still
opens the page that explains it (navigateWithinLicensing), which keeps the
apply-key error flow - and its specs - unchanged.
The licensing spec follows the new route through the banner for the expired,
invalid and wrong-organisation cases, and updateLicenseKeyQuick waits for the
reboot that applying keys triggers before the caller continues.
Startup navigation is pinned by app.service.spec: an unactivated app on any
page, licensing included, is left where it is.
Nine findings from review, all taken.
Row reads (getdata):
- approversForTable reads every column through field(): the security and group
rows are seeded with UPPERCASE keys and an edited row comes back lower-cased,
so a direct property read compared undefined with 'UNDEFINED' and found no
grant - silently falling back to the admin group for every table.
- tx_to is handled by rowInForce(): epoch seconds, which is what Date.now()/1000
assumes, with an ISO string accepted; a blank, absent or unparseable end date
is open-ended rather than expired, so a rule or a grant that omits it is no
longer dropped as NaN. The dqrules filter reads rule_active/base_lib/base_ds
through field() for the same reason.
Rows and the text that describes them (getdata):
- the current-rows clause in FILTER_TEXT is now applied to the selection: the
grid no longer shows expired rows while telling the user they were excluded.
- the primary-key sort resolves each key to the column's own casing, so a
lowercase DDL sorts on real values instead of comparing undefined.
Settings (getdata, viewlibs):
- dcLibref falls back to 'DC_JSLIB' like dcAdminGroup already did: a deploy that
has not run makedata has no settings.js, and viewlibs must still list the
control library.
Wording that had gone stale:
- the parity README's Status paragraph now states the invariant (every scenario
is gated; gated:false is advisory) instead of counts and a list of differences
that were fixed, and the scenarios.json notes say what each scenario pins now.
- viewdata's comment states the contract rather than the behaviour it replaced,
and the synthetic no-data row writes the same space for a blank character cell
as the data rows do.
Tests: getdata-rules.test.js covers an uppercase-keyed APPROVE grant and an
open-ended rule (both fail on the previous code); viewdata.test.js covers the
numeric missing and the empty-table row.
String(undefined) put the literal text "undefined" into every cell of a column
the row does not carry - the ordinary case in MPE_TABLES, where nine columns are
blank on every row - so the viewer rendered the word instead of an empty cell.
A value the row does not carry is SAS's missing, so a character column gets a
space and a numeric one '.', which is what the empty-table branch beside it
already did.
sas/mocks/test/viewdata.test.js fails without the change and passes with it.
The two scenarios recorded in the previous commit are now gated, with one
declared difference each instead of 388 and 4. The diffs were the mock's, and
they are fixed rather than accepted:
- cols: dropped the DESC key SAS never emits, and took LABEL/LONGDESC from the
MPE_DATADICTIONARY rows for this lib.ds.column, as getdata.sas augments
mp_getcols. COLTYPE is blank for DATE/DATETIME/TIME, which is what the estate
answers.
- sasdata: sorted by the business key; character values trimmed the way the
JSON writer writes them, with an all-blank value as a single space.
- dqrules: the active MPE_VALIDATIONS rows unioned with the schema's NOTNULL
(limited to the columns left in the selection, which is why a NOT NULL on
tx_from never reached the grid) and the derived columns' READONLY, then
de-duplicated and sorted by column and rule type with X left at the missing
value - getdata.sas's union/sort/by-group logic.
- sasparams: the current-rows clause mpe_filtermaster appends, and blank
character values as the single space SAS sends.
- maxvarlengths: the value as stored (trimmed), and the key columns in upper
case, which is how the selection spells them.
- approvers: computed from MPE_SECURITY's APPROVE rows with the admin group as
the fallback, instead of a hardcoded sasdemo.
- viewlibs: the shape %mp_getlibs answers - engine in LIBRARYID, blank ENGINE,
libref as LIBRARYNAME.
The control library and the admin group now come from services/settings.js
through dcInit(), which is the mock's %mpe_init(): makedata already writes
dc_libref, dc_admin_group and dcpath there, so a deployment with a different
DCPATH no longer needs a service change. test/mock-settings.test.js proves it by
renaming the library and moving the data. The remaining services still carry
const dcLibref = 'DC_JSLIB' and compute paths from it at file scope, so each is
a separate change.
The sas9 mock keeps its own getdata/viewlibs variant, so it is untouched: the
goldens are recorded from the SASjs Server estate.
appLocked is constructed as false and nothing sets it: the two intended
writers in licence.service.ts were commented out, and the only reader was
canDeactivate. The branch could not fire in the app, so the guard's tested
contract was not the one that ships. Remove the branch, its spec case and the
now-unused subject.
The route-params handler and the isAppActivated subscription both called the
same redirect check, so a load ran it twice. They now feed one combineLatest
subscription that assigns the action and decides the redirects, which still
covers a licence state arriving after the page loads and a switch between
licensing actions without a reload.
The same subscription sends an unknown :action home. Only the values in
LicenseActions have a page, so a URL like licensing/whatever previously
rendered the component with no action and left an empty page.
Licensing spec: 12/12 locally in 49s, including the new unknown-action case.
gotoLicensingUpdate() deferred visitPage('licensing/update') into a
cy.location().then() callback, so callers that queued commands after the bare
call had those commands enqueued ahead of the visit and ran them against
whichever page the app was on. It now returns the chainable and the callers
chain on it, so nothing is queued before the navigation.
waitForAppSettled() also gains .card-header: the key, update and limit pages
render the licensing card even when there is no error to show, and an install
that is not activated yet has no nav tree, no demo banner and no username
field, so the previous set could time out on a page the spec was already on.
Licensing spec: 11/11 locally in 43s.
Dropping canActivate left licensing/register reachable by URL on an install
that is already registered. Nothing there can succeed: accepting the EULA posts
public/registeruser, which aborts for an already-registered user, so a
bookmark, a reload or a back navigation into the page ended in an error.
The component now sends an activated install home whenever the register action
is reached, watching isAppActivated so the decision also covers a licence state
that lands after the page loads. That is the part of the old canActivate
behaviour the register page still needs, without closing the other licensing
pages.
The licensing spec's terms helper now waits for the EULA card to disappear:
accepting it reloads the app, so the card is the marker that the reload
happened, and a refused registration fails there rather than in the selector
wait that follows.
The guard now holds the registration page only while isAppActivated is
false. A null state means startup data has not landed, so no registration has
begun and there is nothing to protect; a failed startupservice does not leave
it null either - app.service.ts sets false on the malformed, missing-property
and request-failure paths - but a route loaded before that point should not
depend on it.
The spec case for the unknown state flips to expect leaving; reverting the
condition makes it red.
An activated install that reaches licensing/register (stale bookmark,
hand-typed URL) had nothing to register: accepting the EULA posted
public/registeruser, which aborts for an already-registered user, and every
navigation out was then refused, trapping the tab until a hard reload. The
lockout now applies only while isAppActivated is not true, which is the state
it was written for.
The force query parameter was read only by the deleted canActivate, so it is
dropped from the five licensing URLs and from the AppService spec rather than
left as a dead parameter pinned by a test.
Adds unit coverage for the guard's canDeactivate branches.
52 fixed waits (350s declared) replaced by explicit conditions: the app
settling out of .app-loading, and the licensing page's own markers. With the
licensing routes rendered normally these waits observe real state, so the
spec runs in the same wall-clock (1:32 against 1:33) without arbitrary
timeouts to tune - three consecutive local runs at 10/10.
An activated app that loaded a licensing URL (bookmark, browser reload, or
cy.visit in a spec) was redirected to /home as soon as the startupservice
response landed, and LicensingGuard redirected there too while
isAppActivated was still null. Both redirects meant the page only existed
for the window between the router rendering it and startup completing - so
the same link worked or failed depending on whether a key was installed.
Drop both: the guard keeps only canDeactivate, which still holds a user on
the registration page and blocks leaving a locked app. Unactivated apps are
still sent to licensing/key from any other page.
Two more scenarios, and the recorder could not take either of them as it stood.
- editors/getdata reads a table input, not URL parameters, so record.js can now
post one: it writes the informational header the server expects (the first
line is the SAS input statement, not a list of column names), posts it as
multipart/form-data, and takes --libref so the estate's control library name
stays out of scenarios.json. replay.js hands the mock its own rows through
mockInput, since the two sides do not share a libref.
- the test harness's webout() stub now honours the showmeta option and writes
the body as members are added. Without them the mocks' own
requireMemberMetadata() guard throws - it reads the body before CLOSE - and a
service that stopped sending formats and labels would have passed here.
- --scenario accepts service/id, because two screens legitimately call their
first service 'default'.
The two goldens are recorded but marked gated: false: the gate prints their
difference counts and does not fail, and check.js labels them ADVISORY. They are
a work list, not an acceptance.
editors/getdata/mpe_tables - 388 differences, all of them the mock's:
cols carries a DESC key SAS never emits and misses the LABEL/LONGDESC/COLTYPE
SAS reads from MPE_DATADICTIONARY; sasdata leaves blank character values out
instead of padding them and is not ordered by the business key; dqrules differ
in content and order; approvers is hardcoded to one user where SAS computes it
and returns none.
public/viewlibs - 4 differences: SAS answers LIBRARYID='V9' with a blank ENGINE
and LIBRARYNAME equal to the libref, and lists SASHELP; the mock answers
LIBRARYID=<libref>, ENGINE='V9', LIBRARYNAME='Data Controller(<libref>)' and has
no SASHELP.
dsmeta is excluded on purpose: it is the engine's report on the dataset (owner,
host, encoding, paths, created dates), estate state rather than behaviour, and
the projection refused the recorded response until it was dropped - the site id
and the service account name were in it.
Two documentation findings from review:
- The comment in createdemodata.js and the parity README said the excel
fixtures hold "3, 18, 34, 79", which points at values the dropped rule never
saw. The uploads that the rule applies to go to dc_jslib, and
regular_excel.xlsx has five rows where SOME_SHORTNUM is 5 - that is the value
the 1-5 HARDREGEX rejects. The 3s belong to a multi-load fixture whose sheet
targets DC996664.MPE_X_TEST, a libref that carries no seeded validations
because getdata filters MPE_VALIDATIONS on base_lib === requestedLibref and
createdemodata copies only data files into it. Both places now name
regular_excel.xlsx and the value 5.
- README:9 promised the scan refuses an "eight-digit id" while the added class
is \bDC\d{8}\b. It now says "eight-digit DC id", which is what the scan does
and what the date case in parity-golden.test.js relies on.
Checked the fixture rather than the diff: column H (SOME_SHORTNUM) of
regular_excel.xlsx holds 18, 34, 79, 74, 98 ... and exactly five rows of 5.
The previous commit added "synthetic": true to the server target to stop the
recorder claiming an estate it had not checked. That put a harness-only key
into the shared build config, where nothing in sasjs reads it and a maintainer
could reasonably read it as a sasjs feature.
The recorder now carries the list of target names that point at the estate
makedata builds (SYNTHETIC_TARGETS = ['server'], which is sas.4gl.io locally)
and refuses anything else, so sasjsconfig.json is untouched. The declaration is
the same one - it cannot be derived from a response, because makedata is the
installer and a customer's control library holds the same rows - it just lives
with the tool that acts on it.
--target server passes the guard (and needs a token, as before); --target
server-mihajlo is refused with a message naming the allowed target.
The previous commit removed every validation rule SAS seeds for MPE_X_TEST,
which fixed the excel specs and broke editor 3: the cell dropdown they click for
comes from the HARDSELECT_HOOK on SOME_NUM, and that hook had gone with the rest.
Only one of SAS's rules actually conflicts with the fixtures - the HARDREGEX on
SOME_SHORTNUM, which rejects 1-5 while duplicate_row_excel.xlsx and friends hold
3, 18, 34, 79. Everything else stays, so the mock's rule set for the table is
again exactly what it was before the seed work: SOFTSELECT on SOME_BESTNUM,
HARDREGEX and SOFTREGEX on SOME_CHAR, SOFTREGEX on PRIMARY_KEY_FIELD and the
SOME_NUM hook.
The parity README now says this, rather than claiming the table is rule-free,
and createdemodata.test.js asserts both halves - the hook is present and the
SOME_SHORTNUM rule is not.
The parity gate's Cypress job went red after the seed work landed, and the two
causes are the same change seen from the fixture side:
- makedata now carries SAS's demo validation rules for MPE_X_TEST, including the
HARDREGEX on SOME_CHAR that blocks any text but 'the'/'data'. The excel
fixtures upload arbitrary values into that table and expect the submission to
reach the approval screen, so ten excel specs timed out with no Approve
button. createdemodata drops MPE_X_TEST's rules, which is what editor.cy.ts
already documents ("MPE_X_TEST stays clean to match the excel fixtures").
- the LABELs the viewer-labels spec asserts went with the hand-written schema.
createdemodata restores them, because a SAS estate has none and the control
library should not claim otherwise.
Both are declared as fixture divergences in the parity README and asserted in
createdemodata.test.js, so the next change to either side fails a test instead
of a Cypress run.
Review findings:
- the SYSTIMEZONEOFFSET acceptance depended on the host's zone; both entry
points pin TZ=UTC before the harness loads, so the accepted list is the same
everywhere.
- the sas9 mock still published '+0100'; it now answers seconds east of UTC
like the standard estate.
- the README promised an eight-digit id scan that IDENTIFIER_CLASSES no longer
had. A DC-prefixed eight-digit class is back (a plain digit class would fire
on a yyyymmdd date), with a case proving a date is not refused.
Six findings from review:
- estate: 'synthetic' was a constant. The target now declares "synthetic": true
in sasjsconfig.json and record.js refuses any other target, so the claim in
the file is one the tool had to check before it wrote anything.
- project() dropped the key when scrubbing a value, so a top-level member named
like a secret was published unredacted. The key is passed through, and
checkPublishable now runs secretViolations on the projection, so the recorder
refuses before writing rather than leaving it to the next pull request.
- denyList restated a subset of ENVELOPE_NEVER. It is built from that list now,
so an error text or a hostname in a discarded field is caught too (values
shorter than four characters are skipped - they would match half the file).
- replay() projected with golden._meta.members while the test validated
scenario.members, so editing the scenario had no effect until a re-record.
It projects with the scenario's list and reports a member-list mismatch.
- replay() deleted SYSCC/SYSENCODING from the expected side when the harness
omitted them, which would hide a service that stopped emitting them. The
harness now adds the two members the way the server does, and they are
compared.
- the eight-digit siteId scan class matched any eight-digit payload value (a
date, an amount) and refused clean goldens. Removed; the site id is caught by
the deny list, which now covers it.
Four findings:
- the replay was never run by CI, so the README's "a mock change fails the
pull request" was not true. parity-check.test.js now replays every scenario
through a shared parity/replay.js (which check.js also uses), fails on a
difference that is not declared and on a declared one that no longer differs,
and fails a scenario that has no golden. The differences that remain are
declared per scenario with a reason: the estate's admin group, its
registration state, its timezone and SAS's space padding of blank characters.
- recording stamped tls: verified for a non-https target while sending the
bearer token in the clear. A non-https target is refused outright now, and
--insecure still records tls: unverified.
- the committed-golden gate never applied the secret-key rule. secretViolations
now walks a golden (and a fresh projection) for secret-named fields whose
value is not <REDACTED>, with a case proving a hand-typed licence key fails.
- arg() returned argv[index + 1] unconditionally, so a trailing --ca reached
readFileSync(undefined). Flags without a value are a usage error now.
The replay also found a real mock defect, fixed here: SYSTIMEZONEOFFSET was
emitted as +HHMM while SAS answers seconds east of UTC ('3600'), and the mock's
own test asserted the wrong form - exactly the drift this harness exists for.
The sasdatasets sort used localeCompare, which puts MPE_X_CATALOG-FC before
MPE_XLMAP_RULES; SAS sorts on the byte value, so the mock now does too.
Two findings from review:
- record.js turned certificate verification off unconditionally, on the request
whose response becomes the golden. Verification is now the default; --ca
trusts a private estate's own certificate and --insecure is opt-in, warns on
stderr and is recorded as _meta.tls so a golden taken over unverified
transport cannot be mistaken for one that was not. The estate's certificate
verifies, so the committed golden is tls: verified.
- the libref placeholder ran before the staging-id one, so a staging id came out
as <LIBREF>-1_2_3. The staging rule now runs first and matches both minted
shapes (DC<date>-<time>_<n>_<n>, and the mock's millisecond form); a project()
case pins the ordering.
The pipeline cannot reach a SAS estate, so the goldens are refreshed where the
credentials are - an agent or a maintainer running record.js - and the CI half
stays offline: scan the committed goldens, replay them against a mock estate.
Say that in the harness README so the next reader does not go looking for a
scheduled refresh job that cannot exist.
The mock services are a second implementation of the SAS backend, and the only
thing comparing them was a human reading two files. This adds the harness that
compares their output instead.
A golden is a projection of a SAS response, never a copy: project.js keeps a
fixed member list per scenario, drops the whole envelope (hostname, site id,
user, home paths, job ids, _PROGRAM), substitutes the identifiers the diff
needs (<LIBREF>, <PATH>, <EMAIL>, <STAGING_ID>, <DATETIME>), and redacts secrets
by key name. The recorder then refuses to publish if any identifier class
survives, or if any literal the envelope just discarded still appears in the
projection - which is what catches a hostname echoed inside an error message.
record.js records from a live estate into .captures/ (gitignored) and writes
only the projection to golden/. check.js replays a golden against a mock estate
built by the test harness and prints field-level differences; it needs no
server and no credentials. parity-golden.test.js re-runs the identifier scan
over every committed golden, so a hand-edited file cannot slip through either.
Phase 0 ships one scenario, public/startupservice. The replay is advisory and
already reports twelve differences worth a look (admin group casing, the
registration counters, timezone representation, the sasdatasets order, blank
versus empty Excel map descriptions); the corpus, the merge gate and the
nightly drift job follow.
Stacked on the mocks context in the parent branch: it seeds the mock estate with
seededEstate(), which that branch adds.
The fixture's schema entry lived in makedata's schema object (as a second,
later key in the same literal) and did not survive the move to createdemodata.
saveTable falls back to the file already on the drive, so a reseed looked
fine - but CI deletes the appLoc before seeding, and a fresh estate wrote
mpe_x_search.json with columns: [], losing the column order the
full-table-search spec derives its counts from.
Carry the nine columns (name/type/length/format/label/notnull) in
createdemodata's schema, and add a case that every fixture table is written
with column metadata covering its rows, plus the MPE_X_SEARCH order. It fails
against the previous service.
The panel heading reads 'Configuration (#2 DC_JSLIB.MPE_ALERTS)', so the check
parses lib.dataset out of it and compares it whole. A substring test accepted
another library's table of the same name - this estate seeds MPE_X_TEST under
two libraries - and any name that merely contains the target, which is the
mix-up the check exists to catch.
Verified on the local stack: 12 passing in 2:24 and 2:22.
createdemodata assigned tables.MPE_DERIVED outright, so any row makedata wrote
there would be wiped by a fixture run - and the idempotency test did not count
the table, so nothing would have said so. Filter the TESTDATA rows out and
append, in the same shape as the tables above, and count MPE_DERIVED in the
idempotency test.
New case: a non-TESTDATA derived row survives a createdemodata run exactly
once. It fails against the previous service and passes now.
The table link is looked up inside the matched library node rather than across
the whole picker, so a table of the same name under another expanded library
cannot be clicked instead, and the library fragment is escaped before it
becomes a RegExp - a dot or a bracket in a library name no longer matches more
than the intended node.
The configuration panel is checked after the ~600-800ms window in which
loadData() re-selects the viewbox it just loaded, rather than 400ms into it,
so a moved panel is caught at the click instead of in the column list that
follows.
Verified on the local stack: 12 passing in 2:24 and 2:22.
- createdemodata drops the unused servicesDir binding.
- the DC996664 copy no longer swallows a failure: a broken multi-load fixture
now fails the seed where it happened instead of surfacing in a Cypress spec.
- makedata answers OUTFMT=JSON with a makedata_response member (as the SAS
service does) and the CLI payload asks for it, so the `|| true` comes off
makedata in build.yaml, release.yaml and sites.yaml - a failed seed names the
service that failed instead of reappearing as "createdemodata needs the
control library".
- the makedata comment points at sas/mocks/test/makedata-parity.test.js, not a
macro that does not exist.
makedata seeded two things at once: the control library, and the tables the
Cypress suite needs. The estate it produced was therefore not the estate a
SAS deployment produces - and it had drifted: MPE_MAXKEYVALUES and
MPE_SIGNOFFS were never created, and MPE_CONFIG, MPE_VALIDATIONS, MPE_TABLES,
MPE_SELECTBOX and MPE_XLMAP_RULES were missing seed rows that mpe_makedata.sas
inserts.
makedata now transcribes mpe_makedatamodel.sas and mpe_makedata.sas: every
model table with its columns, and every seed row the macro writes. The
Cypress fixtures move to services/admin/createdemodata.js, which the test
estate runs after makedata; it strips and re-adds its rows so a reseed does
not accumulate.
Checked against real SAS output on the shared estate (fresh deploy of the
server target, makedata, then getrawdata for each control table and a
dictionary.columns probe): all 40 tables and 339 columns match on name, type,
length, format and label, and the seeded rows match row for row apart from
the environment (libref, user id, group names) and the sample rows. That run
found six defects a source-only comparison could not: INT columns are 8 bytes
not 4, one label had lost a double space, the DC_MACROS setting held the SAS
expression instead of a path, the email templates held an escaped newline,
75 selectbox rows held %mf_increment(rk) instead of their key, and one
validation row was truncated at a ';' inside its regex.
makedata-parity.test.js parses both macros and compares them with the seeded
estate, and fails on a value that does not fit its column type, so the next
drift fails a test instead of shipping. The CI workflows, the Sites demo seed
and CONTRIBUTING call createdemodata after makedata.
loadData() re-selects a viewbox about 600ms after its data lands, and that can
land after the row click that opens its configuration - leaving the panel on
the other viewbox and the test looking for a column that table does not have.
The panel's heading names the viewbox it is showing, so the click is repeated
until the heading matches rather than trusting the click once.
The row is only clicked when the panel is not already on that viewbox, since
selectViewbox() toggles and a second click would deselect it.
Verified on the local stack: 12 passing in 2:30 and 2:20, against 5:22 for the
same spec before the change.
openTableFromViewboxTree paused 300ms before looking at each rendered node,
which on a library of this size cost ~13.4s of the ~20s a viewbox test takes.
The helper now expands the library and clicks the table's link, retrying the
expansion when the picker re-renders underneath it.
It also waits for the picker's own state before returning: every requested
table listed, no row spinner, and the ~600ms window in which loadData()
re-selects the box that just loaded. Without those, the next step read the
other viewbox's column list and failed on a column it does not have.
Measured on the local stack: the spec runs 2:18-3:15 against 5:22 before, with
all 12 tests passing in both.
The SAS Logs modal rendered the log, the source code and the generated code as one text block each, so no line could be referred to and the error, warning and note links had to locate a line by arithmetic. Render one row per line instead: every line carries its number, a wrapped line hangs under the text rather than under the number, and the numbers stay in view when a long log line is scrolled sideways.
The links now scroll to the row itself. The old offset was the measured line height multiplied by the line index, and two things were wrong with it: clientHeight rounds the height to a whole pixel, and the wrapper's padding was not counted. In a 168-line log at the stylesheet's own metrics the early links landed within a few pixels of their line while the later ones did not; at an 18.5px line height, which is what 16px monospace at 1.15 produces, the scroll for line 43 was 10.5px out and for line 127 was 52.5px out, so the highlight sat above the line it named and the drift grew with the line number. A row's own position cannot drift: every link lands its line at the top of the log, less the wrapper's 1px border, and flashes the right row.
The flash is a class on the rows rather than an absolutely positioned overlay, so a message SAS wrapped over several lines flashes every line it runs to.
One row per line costs about 0.25ms in Angular - a binding plus a sanitizer pass each - so rendering a whole log would freeze the modal: 20,000 lines took 5.3s to open. Each pane renders a window of 400 lines with a "Show more" line underneath, a jump renders through to the line it was given before scrolling, and opening is now flat at 165-280ms for 1,000, 5,000 and 20,000 lines.
git-describe only sees tags reachable from HEAD. CI checks out a single commit with no tags, so it reports the bare hash and every version in the app reads "v.<hash>" - including the production release builds. The root package.json carries the released version (semantic-release bumps it before the shipped frontend is built), so use it as the tag when git-describe found none, and fill semverString from it for the browser_info diagnostics.
The job installed the same lockfile the build job had just installed and paid
the downloads again (92s), and npx cypress install fetched its binary on every
run. Both now come from the cache the build job already fills, with the
Cypress binary keyed on the lockfile.
sas/mocks/test/ holds the node:test harnesses that read the payload files, and
the job deploys sas/mocks/sasjs/ and nothing else - so a pull request that
changes only those tests cannot change what the browser is served, yet it
started the Cypress suite. The file list is filtered before the match, so any
other file under sas/mocks/ still counts, a change that touches the tests next
to a service included.
The matcher's cases cover both directions.
The capture assigned its body to _webout in one template literal, and that
literal was not JSON: raw line breaks inside the graphviz label and an
unescaped quote in labelloc = "t". fromsas, clickableids, info and flatdata
are members now, written as string literals that keep the label's quotes and
line breaks, with the envelope supplied by the runtime.
sas/mocks/test/sasjs-webout.test.js holds the six services that set their own
content headers, and pins this capture's members. CONTRIBUTING.md lists the
same six.
Each fault shape the tree repairs now has a case that builds it - drive,
OMSOBJ, BUILTIN and DOMAIN - so a heuristic that stops detecting one fails the
suite instead of leaving only the exact-value assertion behind.
A drive letter is path-initial when nothing word-like precedes it, so a path
after <, >, =, [ or a comma is checked too; prose that puts a word before a
colon still is not a drive.
The two cases added for a path after a tag and after a comma were written as
single backslashes in the source, so each one evaluated to a value with no
separator at all and pinned nothing. The value is now assembled from
String.fromCharCode, with an assertion that it starts with the two separators
the case needs.
A service whose payload does not arrive as (kind, name, rows) contributed no
members and passed the tree-wide test with nothing checked; the loop now
records the file with no members captured.
The header describes what the values and the test do. The ticket numbers it
carried are gone: AGENTS.md allows a tracker reference only as a link, and the
behaviour reads without one.
A file that does not emit through webout() and is not a listed helper fails a
scope test, so a renamed emitter shrinks the run set loudly instead of leaving
the replay over an empty set. Every webout kind is captured, a service that
needs a global the sandbox lacks names itself in the error, and the header
states the scope the scan covers.
The header described the captures as live template literals, which this tree
no longer carries; the values now travel through webout() and the same
backslash is one escape away from the same damage.
The replay covers the files that call webout(), so a shared module added to
the tree is skipped instead of failing with a require error, while the
_webout assertion still reads every file - a service that writes its own body
cannot slip past the filter.
A UNC prefix is legal after any character that is not a word character, a
colon or another separator, so a path inside a tag or after a comma keeps its
prefix instead of reading as a doubled separator.
The whole-tree check treated any two consecutive backslashes as a fault, so a
UNC library path (\\server\share\...) failed on a correct value. A run of two
is now legal only where a UNC path puts it - the start of a value or of a quoted
path - and two separators anywhere else, plus any run of three, are still a
fault. faultsIn gains a case for each direction.
The sandbox require delegated relative requests to this test file's own
require, so a service requiring a sibling module resolved it under
sas/mocks/test. A relative request now resolves from the service file; builtins
and absolute paths delegate unchanged.
getdata's ENGINEHOST metadata rows carried a Windows path and an account
name with no separators, which SAS writes and the rest of this tree
already has: the rows read C:DataControllerDC996664mpe_x_test.sas7bdat
and BUILTINAdministrators, against refreshlibinfo's C:\DataController\
DC996664 library and BUILTIN\Administrators owner in the same recording.
Both values now carry single separators.
sas/mocks/test/sas9-separators.test.js replays all 36 files of the tree
under a webout() stub and fails on a doubled separator, or a missing one
in an OMSOBJ URI, a drive path or a BUILTIN/DOMAIN account. It also pins
the two corrected rows and asserts that no service assigns _webout.
The CSV / Excel / PDF export submenu never reached the screen. HOT's
ExportFile plugin contributes a context-menu entry under the reserved key
`export_file`, and ContextMenu's ItemsFactory.setPredefinedItems folds the
plugin's entry over the grid's entry at the same key. That folded entry
carries `hidden() { return this.getSettings().exportFile === undefined }`,
and no grid in this app sets `exportFile`, so the whole item - submenu and
all - was filtered out of the menu. Seen on the 7.22.0 build: the viewer's
context menu offers Copy with header, Copy header only and Show labels, and
nothing else, so the client-side PDF export could not be reached at all.
Give the item a key of the app's own, EXPORT_MENU_KEY, and use it at the
three grids that build the menu (viewer, editor, viewboxes). The submenu
keys move with it.
export-menu.spec.ts pins both halves against a real grid, because the
merge happens inside the plugin at menu-build time and no unit test of the
item object can see it: configured under `export_file` the item is absent,
configured under EXPORT_MENU_KEY it is present.
The 7.22 release's sas-docs job failed with surfer's "Run surfer config first,
or provide --server ... --password" message, and its log shows SURFER_PASSWORD
empty - the CODE_DATACONTROLLER_IO secret resolves to nothing. Authenticate
with WEBDOC_DATACONTROLLER_IO, the password the typedoc job in this same
workflow already uses: a Cloudron app password belongs to the user, and the
same user owns both docs apps, so one credential covers both.
build.yaml, lighthouse.yaml and sites.yaml were moved to Node 24.15.0 on
2026-10-28 (d84a1d50); release.yaml, release-assets.yaml and docs.yaml were left
on 24.5.0. The release workflow builds the same client and runs the same Cypress
suite as build.yaml, so it should not be the one place pinning a different
runtime - and docs.yaml builds the frontend for typedoc with it too.
The release workflow recorded video for every spec and uploaded
cypress-videos.zip on every run, green or red. Recording eleven specs costs
encode time each run and the recording is only ever opened after a failure, so
run with video=false,screenshotOnRunFailure=true - the same pair build.yaml
already passes - and zip screenshots plus videos only under if: failure().
# [7.22.0](https://git.datacontroller.io/dc/dc/compare/v7.21.0...v7.22.0) (2026-11-05)
### Bug Fixes
* **browse:** accept legal folder names, cap the walk, and lock the read ([3572d56](3572d56cfb))
* **build:** build the mock sites with a CLI whose runtime provides webout() ([b1eab98](b1eab98200))
* **build:** publish the changed-path decision and pin the matcher with tests ([ecc8702](ecc87021c3))
* **build:** treat a sas/package.json bump as a Cypress input ([48df795](48df795202))
* **client:** cancel a queued dropdown reposition on destroy ([590b841](590b841f24))
* **client:** drop the synthetic window resize dispatches ([2535baa](2535baa5f7))
* **client:** keep the ARIA pass on the viewbox and multi-dataset grids too ([cb30469](cb30469e79))
* **client:** restrict the table pass to Handsontable's own tables, add the viewer fallback ([46c6aec](46c6aec37a))
* **client:** scope the ARIA pass to each grid and drive it from the render hooks ([bf1e7d6](bf1e7d6b15))
* **client:** stop synthesising aria-labels; name the filter modal's submit in the test ([32d3d49](32d3d49b59))
* **client:** strip only this grid's own wrapper, and pin the re-strip after a data load ([f282d1a](f282d1a44b))
* **deploy:** ACTION cannot seed by default, and the probe stops overwriting ([e84cfaa](e84cfaaabd))
* **deploy:** ask for the admin group before the staging directory on SASjs ([5c64691](5c646913e4))
* **deploy:** await nothing, and encode the values in the makedata URL ([d67a0e5](d67a0e5d5f))
* **deploy:** drop the deprecated service-pack deploy from the setup screen ([59e09e2](59e09e26a5))
* **deploy:** gate the browse actions on the settings file ([5eb8492](5eb8492742))
* **deploy:** make the folder picker work on both estates ([fb38309](fb38309e98))
* **deploy:** match the documented SYSSCPL set and stop guessing at the cap ([5f338ff](5f338ff437))
* **deploy:** read a URL-supplied ACTION before defaulting, and show locked folders ([addb4a5](addb4a56ea))
* **deploy:** stop the makedata poll outliving its screen, and align the release gates ([c808c41](c808c41333)), closes [#407](#407)
* **deploy:** unclamp the folder picker row and probe in the chosen context ([1330925](133092502c))
* **deps:** bump @sasjs/core to 7.0.1 so special missings can be restored ([3d2eefd](3d2eefdd7b)), closes [sasjs/core#499](sasjs/core#499)
* **deps:** bump @sasjs/core to 7.1.3 so a filter abort names the clause ([c7c1815](c7c181518c))
* **editor:** a container resize must not clear an active sort ([5bd7b5d](5bd7b5d724)), closes [#246](#246) [#246](#246)
* **editor:** drop the stale case hint, and narrow the strip to our own form ([bd2739c](bd2739cde4))
* **editor:** one predicate for the CASE marker, and no stacking in the tooltip ([18b0546](18b0546dcb))
* **embed:** a rejected mapping must not block the page ([d845b9e](d845b9e9b7)), closes [dc/dc#404](#404)
* **embed:** read a month-name date from VA as a date, not a character value ([30cefcf](30cefcf2dc))
* **getgroupmembers:** widen membername to $256 for long names ([196d6c9](196d6c931d))
* **lint:** enable noSingleAsteriskComments and noUnusedLibnames ([9071801](9071801ac5)), closes [#376](#376)
* **lint:** make noBannedStrings an error and clear its 13 findings ([0b85f7b](0b85f7bf1e)), closes [#376](#376)
* **lint:** make noUndeclaredMacros an error, and fix what it found ([af28a79](af28a79bd5)), closes [#376](#376)
* **lint:** make noUnusedMacros an error and clear its 54 findings ([c8c0858](c8c0858295)), closes [#376](#376)
* **lint:** rename the short and mixed-case helper macros, and clear the last warnings ([7da67eb](7da67eb17b)), closes [#376](#376)
* **makedata:** let the browse actions live as long as the service does ([912fae4](912fae448b))
* **mocks:** 0 is a closed TX_TO, a libref boundary, and the union of the row keys ([67d9cfd](67d9cfd1bd))
* **mocks:** abort like mp_abort, lock like mp_lockanytable, derive saslibs ([ca60208](ca602082ff))
* **mocks:** answer the deploy screen's DIRLIST and WRITECHECK actions ([e1c3c6e](e1c3c6e872))
* **mocks:** derive startupservice's maps and timezone instead of hardcoding ([87fd46c](87fd46c825))
* **mocks:** fold the audit window, and append MPE_AUDIT under its lock ([5ecc7be](5ecc7bebec))
* **mocks:** keep the OMS and path separators the captures lost ([7d9d466](7d9d466c3e))
* **mocks:** one sasDatetimeNow, a documented TX_TO=0, and a shaped fallback export ([e53c50d](e53c50d4e2))
* **mocks:** read hook columns in either case, and guard duplicates first ([349a2f3](349a2f314c))
* **mocks:** read the metadata member from the response, not the function's arity ([09e9f19](09e9f192c5))
* **mocks:** refuse a runtime whose webout() cannot carry column metadata ([482435f](482435f318))
* **mocks:** resolve the audit table once, record lower-case md5, and report a failed audit write ([1a9f594](1a9f59478b))
* **mocks:** run approval hooks, as mpe_runhook does ([f0208bb](f0208bb0b0))
* **mocks:** separate a body read failure from a missing showmeta floor ([744c89b](744c89b919))
* **mpe_abortmessage:** decide once whether the payload was escaped ([af89126](af89126acf))
* **mpe_abortmessage:** read program and state by name, and trim the prefix ([68b89ca](68b89caf63))
* **mpe_assert_abort:** anchor the cut on the whole escaped separator ([e74a7ba](e74a7ba115))
* **mpe_assert_abort:** compare the hook's message, not the log extract ([8b426c2](8b426c27fd))
* **mpe_assert_abort:** compare the unescaped message, cut at the last marker ([e68d221](e68d221354))
* **mpe_assert_abort:** cut at the separator, not the last marker ([4c7e869](4c7e869369))
* **mpe_assert_abort:** name the separator and tolerate a carriage return ([7b09ade](7b09ade97c))
* **mpe_assert_abort:** size the capture buffer for the whole message ([5d32de9](5d32de9df9))
* **mpe_assert_abort:** strip the separator tail, not every trailing escape ([b29bc1e](b29bc1e7ae))
* **mpe_execute:** assemble the abort text in a macro that can be tested ([e2e9103](e2e91039cb))
* **mpe_execute:** keep a child abort message intact when reporting it ([ebfd395](ebfd39582f))
* **mpe_execute:** keep the child message in the fallback, and test the doubled separator ([11587e2](11587e2f24))
* **mpe_execute:** make the abort-text fallback reachable ([24b8cbe](24b8cbe9dc))
* **mpe_execute:** mask a child abort message before it aborts the caller ([20dbf00](20dbf00fb0))
* **mpe_execute:** one symbol for the abort text, and no values in the call ([700b906](700b906d69))
* **multi-dataset:** a sheet that failed to extract is not a clean finish ([3129648](312964864e))
* **multi-dataset:** show the file size and loading stage while the file loads ([45d6c02](45d6c025c9)), closes [#121](#121)
* **report:** treat a blank PDF footer parameter as its default ([0f729d7](0f729d7101))
* **test:** let the dirlist and writecheck results reach the caller on Viya ([ed678f0](ed678f0b46))
* **test:** run metanav.cy.ts in CI and open a metadata object ([02a8590](02a8590310))
* **va-host:** bound the early replay even when the frame is not postable ([f6d5a04](f6d5a0495e))
* **va-host:** deliver in load mode, honour dataType, and survive odd prompt names ([7d4155d](7d4155dc6c))
* **va-host:** deliver the load-mount message once, and document report.json from the file ([90784b3](90784b342f))
* **va-host:** keep the deploy password off argv, and verify the app is served ([7c63a5c](7c63a5c93f))
* **va-host:** post to the frame's own origin, and key the replay to the document ([9cec292](9cec292681))
* **viewer:** clamp over-tall PDF rows, always emit an otherwise-empty page ([10229d5](10229d5efe))
* **viewer:** defer the blob URL revoke, reject a selection inside skipped columns ([b5bb704](b5bb704a4f))
* **viewer:** keep the PDF export working where hiddenColumns is not enabled ([f84d4ca](f84d4ca7af))
* **viewer:** review findings - CSV skipped column, PDF row cap before reading ([2c4be5e](2c4be5e94b))
### Features
* **admin:** find and test a writeable location during deploy ([442c595](442c595c2b))
* **deploy:** pick and validate the DC location from the deploy screens ([6eeab2e](6eeab2ed16))
* **editor:** convert CASE rule values on entry ([c4eef49](c4eef498b2))
* **logs:** read the logs modal with the shared log parser ([aabc578](aabc578d5e))
* **mocks:** add the control-table services the screens call ([a1615ee](a1615ee8a7))
* **mocks:** build every JS stored program's response with webout() ([ce6bb5d](ce6bb5d4cd))
* **mocks:** emit the lineage, usernav and libinfo responses through webout() ([1f54fad](1f54fad3fe))
* **mocks:** emit the metanav responses through webout() ([e1d877f](e1d877f21d))
* **mocks:** emit the sas9 services through webout() ([7be3f84](7be3f84073)), closes [#461](#461)
* **mocks:** restore a load, and record the diffs it needs ([1a60c68](1a60c689f4))
* **mocks:** use the server's webout() in the JS stored programs ([ccb67ed](ccb67ed5a0))
* **va-host:** emulate a VA report that embeds DC, as a deployable AppStream app ([2185db9](2185db911d))
* **viewer:** download a server-side PDF report ([67ca129](67ca129c4d))
* **viewer:** export the grid as PDF from the context menu ([e58e735](e58e735610))
### Reverts
* keep the helpers nested inside makedata_body ([4424171](4424171b2f))
The SASjs Server / SAS9 screen put the DC Staging Directory first and the admin
group second, the reverse of the Viya screen, so the operator was asked to pick
a location before being told who administers DC. It now asks for the admin group
first and the directory last, with the copy reordered to match.
A rendered-order spec covers it: it reads the labels off the rendered template
and fails if the directory comes first again.
DIRLIST and WRITECHECK were refused once the appLoc had a settings.sas, on the
argument that a failed run could leave that file behind. That is what makes the
gate wrong: a failed run is exactly when the deploy screen still needs to list
and probe directories, and any appLoc configured earlier refuses them for good.
The window needs no separate clock - a successful run deletes this program, so
once DC is configured there is nothing left here to call.
Removed from both variants (server checks the drive over HTTP, Viya checks the
filesystem; neither is needed), and the two tests that asserted the refusal go
with it. The macros themselves are covered by mpe_dirlist.test.sas and
mpe_writecheck.test.sas.
Seen on sas.4gl.io: a deployment to /Public/app/dc477 answered BROWSE with
"DIRLIST and WRITECHECK are only available until DC is configured".
The release pipeline went red at 28157fae: Karma aborted with "Some of your
tests did a full page reload!" after test 753 of 861. The new-window spec calls
runMakedataInNewWindow, which starts the five-second makedata poll; the service
stub answers that makedata is gone, so the first tick assigns window.location.href
and the page reloads under the running spec. A local run finishes in under five
seconds, so the timer never fires there - only CI saw it.
- the poll is cleared in ngOnDestroy, so a timer cannot reload the page after
the user has moved on; a spec covers that
- the spec that exercises the new-window URL no longer starts the real poll
Three open findings on #407, all taken:
- lighthouse.yaml: sas/package.json joins the change-detection pattern, because
the job pins @sasjs/cli from that file
- release-assets.yaml: webpack is installed at the version the client lockfile
resolves rather than whatever is current at release time
- build.yaml: the client audit runs with --audit-level=low, which is what the
comment above it has always claimed
Both test programs pointed _webout at a temporary file so the macros under
test could write their JSON answer, then tried to hand the fileref back with
`filename _webout "&_webout_path" mod;`. On Viya's JES _webout is a filesrvc
fileref, whose pathname() is empty, so that statement fails with "Invalid
physical name" and the temporary file stays in place. The term program's
%webout(OPEN) assigns _webout itself (filesrvc on JES, temp elsewhere), so the
restore is not needed at all; the two programs now leave the fileref to it.
Observed on the nextviya estate: the suite reports 64/64 of the tests that
produced results, with mpe_dirlist and mpe_writecheck absent from the results
JSON and "Invalid physical name" in their logs.
The spec failed on CI: the browse button never appeared. Two causes, both
found by reproducing the CI estate locally (seeded drive):
- The intercept pattern '**/drive/folder**' does not match the request the
app makes ('/SASjsApi/drive/folder?_folderPath=...') in this Cypress
version, so the app saw the real drive, loaded the startup service and
never routed to the deploy screen. Both intercepts are now regexps, which
match deterministically. Verified on the seeded estate: the app routes to
#/deploy and the agreement card renders.
- A secured estate shows a login form first. The boot helper now waits for
one of the states it can act on (a visible login form, the agreement card,
or the deploy screen), signs in when the form is visible, then waits for
the deploy screen. The login form stays in the DOM hidden on other routes,
so visibility - not presence - decides.
Review findings on #476.
- The 'hands the picker the context' test declared only AutomaticComponent
and HostComponent with NO_ERRORS_SCHEMA, so app-dc-location stayed an
unknown element: the [computeContextName] binding was never applied and
the assertion just repeated the fallback case. DcLocationComponent is
now declared (SasService is already stubbed), and the test asserts the
picker's own input - both before a selection (the configured fallback)
and after one is chosen. Verified it bites: removing the binding from
the template fails it.
- The Cypress spec set the path nowhere before clicking Validate, so the
assertion leaned on the field arriving pre-filled; a blank field would
short-circuit with 'Type or pick a folder first' and fail for a reason
unrelated to the layout. It now types the path explicitly.
- The spec slept a fixed 4s after accepting the agreement, which turns a
loaded CI runner into an intermittent failure. Both tests now share one
boot helper whose readiness gate is the Browse button appearing.
Two defects on the deploy screens, found on staging.
1. The folder picker's outer row reused Clarity's 'clr-input-wrapper'
class, whose 'max-height: var(--clr-forms-input-wrapper-height)'
clamps the row to the input's height. The row is flex-wrap: wrap, so
once the Browse/Validate buttons wrapped (or the row was simply
taller than one input) they overflowed the clamp, and the folder
browser panel - positioned after the collapsed box - rendered on
top of the input, the buttons and the surrounding text. Measured on
staging: panel top 256 against input 248-272 and buttons 280-304;
with the class dropped the row contains all its children and the
panel starts at 312.
2. The Viya screen asked for the DC directory before the compute
context, but the folder listing and the write check run inside the
chosen context, as its batch identity - so the probes could inspect
a file system the deployment itself would not use. The fields are
reordered (admin group, then batch id / compute context, then DC
Loc last) and the picker now runs DIRLIST and WRITECHECK with the
selected context in the adapter request config, which every Viya
executor path honours. The SAS 9 / SASjs screens pass no context
and behave exactly as before.
Tests: a Cypress spec drives the real deploy screen estate - the
picker row must contain its own children and the panel must start
below them; both tests fail with the clamp reintroduced. A rendered
TestBed spec pins the Viya field order (admin group < compute context
< DC Loc) and fails with the old order. Unit specs pin the probe
config: no context on SAS 9 / SASjs, the chosen context on Viya.
Three review findings.
- KEY_HASH was upper-cased after the digest. SAS md5() writes lower-case hex,
so the mock's key hash did not match what a real audit table holds. The
digest is stored as it comes out, and the test asserts lower case.
- restore resolved MPE_TABLES.AUDIT_LIBDS per table, while the approval always
appended to the control library's MPE_AUDIT: a table with its own audit table
had its diffs written where restore never looked, and restore then refused
with "No entry for <load>", which no later approval could fix. Resolution now
lives in dcMockUtils.auditLibdsFor, called by both sides, and the approval
reads and saves that table in its own libref.
- An audit failure was only logged to the server console, leaving an approval
with no diffs and no visible reason. It is still not allowed to undo the
approval, but once the control tables are written the failure is raised as an
abort that names the table and says the load can no longer be rewound.
Two tests: one approves a table whose MPE_TABLES row names a different audit
table and asserts the rows land there and not in the control library; one points
AUDIT_LIBDS at a libref with no data folder and asserts the approval still
lands and the response says the diffs could not be recorded.
Two review findings on this branch.
- editors/restore.js spread every audit row for the load into Math.min(...).
MPE_AUDIT is append-only, so a long-lived estate would eventually exceed the
engine's argument limit and the RangeError would surface as a failed restore
of a perfectly restorable load. Folded into a reduce with an Infinity seed.
- auditors/postdata.js appended to MPE_AUDIT as a read-modify-write outside any
lock. postdata.sas locks the table it writes - MPE_REVIEW around its proc
append, MPE_SUBMIT around its update - so the audit append now takes the
MPE_AUDIT lock around a fresh read, the concat and the save, rather than
reading the table up front and saving a copy that may be missing another
approval's rows.
53 mock tests pass (node --test sas/mocks/test/).
`editors/restore` had no mock, and the reason it could not have had one is that
`postdata` wrote no audit rows: an approval applied the load and left nothing
to rewind. Both halves are here.
postdata (final approval): writes MPE_AUDIT the way %mp_storediffs does - one
row per column, keyed by the md5 of the business-key values joined by '|', with
move_type A/D/M, is_pk on the key columns, is_diff on a changed one, and the
old and new value on their own typed columns. An append carries every column
on the new side, a delete every column on the old side (which is what lets a
restore put the row back), and a modification the keys plus only the columns
that actually changed. A failure writing the audit is logged and does not undo
an approval that has already landed.
restore: mpe_checkrestore's two decidable checks (the table has an audit table;
the load has entries in it) with their messages, then mp_stripdiffs' window -
the load's timestamp and every change to that table from then on, because a
restore reverts this load *and all subsequent ones*. Per business key the
pre-window state is the earliest event: an append means the row is staged for
deletion (`_____DELETE__THIS__RECORD_____=YES`), a delete means it is staged
with the values the delete recorded, and a modification means each column takes
its earliest old value while untouched columns keep the current one. The
result goes through the same stageSubmission the editors use, so it arrives as
a new submission with the reason `Restoring <loadref>` and is approved like any
other - restore.sas never writes the base table either.
The access and group/RLS checks are not emulated, consistent with the rest of
the mocks: nothing enforces MPE_SECURITY there.
Tests: 52 pass across the seven mock suites, 7 of them new. They drive a real
approve and then read the staged rows back: append staged for deletion,
modification staged with the pre-load value, delete staged with the row's
values, and restoring the first of two loads reverting the second as well.
With the base postdata and no restore service, all 7 fail.
Four review findings.
- isCurrentOn counted TX_TO 0 as current. 0 is 01JAN1960 in SAS, so a real row
with that window has been superseded; only a missing, null or blank TX_TO is
open besides 31DEC9999. The fixtures are seeded with 31DEC9999 (253717919999)
through an OPEN constant instead, and the 0 clause is gone.
- The MPE_DATADICTIONARY filter compared DD_SOURCE against the libref without a
boundary, so DC_JSLIB2.TABLE was excluded along with DC_JSLIB.TABLE. It now
tests for the libref plus the dot, as the libds rewrite does.
- The schema-less export took its column list from Object.keys(rows[0]), so a
column that first appears in a later row was silently missing from the backup.
The list is the union of the keys across the exported rows, in first-seen
order.
- control-tables.test.js imported readTable and never used it.
Three review findings on this branch.
- dcMockUtils defined sasDatetimeNow twice: once beside isCurrentRow with the
epoch written inline and once in the mp_lockanytable section using
SAS_EPOCH_MS. The second is the one the file already had; the duplicate is
gone, so the epoch has one expression.
- isCurrentRow treated TX_TO=0 as current without saying why, and reached that
through Number(null) === 0. Both conventions are real: makedata writes
31DEC9999 for a row that was never superseded, and the fixtures in
control-tables.test.js use 0 for the same thing. The clause stays and the
comment now says which convention is which, plus that a missing, null or
blank TX_TO (a table with no version column) is current too. Reading the
column moved into isCurrentOn, because Array.filter passes the index as a
second argument and would have handed a `column` parameter the row number.
- admin/exportconfig used the schema only when it existed: with no columns in
the table file it fell back to Object.keys(rows[0]), which exported the
TX_/VER_/SELECTBOX_RK columns the spec drops and quoted every number. The
fallback now applies the same drop prefixes and infers numeric typing from
the rows, so a schema-less estate produces the same shape as a seeded one. A
test writes a table file with rows and no columns and asserts both.
All 46 mock tests pass (node --test sas/mocks/test/).
postinstall runs scripts/strip-clr-base64-fonts.mjs, which edits Clarity's CSS
inside node_modules and then deletes client/.angular/cache - because the
builder treats node_modules as immutable and would otherwise serve styles built
from the unstripped file. Measured on the Angular 20 builder: a marker
@font-face injected into clr-ui.min.css was absent from a warm build's CSS
(28s) and present in a cold one (88s).
The restore used to run before the install, so that clear deleted the entry
before anything could use it - run c732d75 restored 396MB and logged "cleared
/workspace/.../client/.angular/cache" minutes later. It now runs immediately
after "Install dependencies": the clear has already happened, and every entry
in the cache was produced from a stripped stylesheet, because every run strips
before it builds. The invariant the clear exists to enforce is preserved.
No restore-keys on this cache, deliberately: because the builder cannot see a
change inside node_modules, a prefix match could restore entries compiled
against a different Clarity after a lockfile bump. Exact key only, so a
dependency change misses and rebuilds.
Measured locally with the tree matching the lockfile (CI=true): cold with the
cache cleared 90s, warm with no source change 27s, warm with version.ts
regenerated as a new commit does 58s. So a build that is 99-109s in CI should
land near 58s; the rest needs the version out of the compiled graph, which is
an application change.
The strip script's comment described webpack 5; it now describes what was
measured, and notes that CI restores after it runs.
Three services the Angular app calls had no mock at all, so on the mock
estate the Excel-map screen, the system screen's refresh and the config
export either failed or showed nothing:
- editors/getxlmaps - the rules and info of one map. Reads MPE_XLMAP_RULES
and MPE_XLMAP_INFO for the current rows of that XLMAP_ID, orders by sheet
and range id, and coalesces the target to DC_JSLIB.MPE_XLMAP_DATA - the
same five columns, order and fallback getxlmaps.sas uses.
- admin/refreshcatalog - a libref is optional and must be well formed
(Invalid libref otherwise), and the reply is sasparams with the
completion message the screen displays. A mock estate has no catalogue
to rewrite, which the service says rather than pretending.
- admin/exportconfig - a member per non-empty table, named
<libref>.<TABLE>.csv, with the per-table treatment exportconfig.sas
applies: TX_* dropped, a version filter, the `where also` exclusions,
and the control-library name rewritten to the target libref (including
the DC_MACROS value and the LIBREF.TABLE of a map target). Zipped as
DCBACKUP.zip.
The version test and the control-table reader were local to startupservice;
they now live in dcMockUtils (isCurrentRow, sasDatetimeNow, and
makeControlTableLoader for controlTable), so the rule that decides a row is
current exists once.
The harness gains the session's stpsrv_header.txt and _SASJS_USERNAME, so
a service that streams a file can be exercised: exportconfig is asserted
both ways - the zip where the runtime has Buffer, the content listing where
it does not.
Tests: 45 pass across the six mock suites, 8 of them new. With the three
services removed, all 8 new cases fail.
Run c732d75 restored both entries on runner2 and the build stayed at 99s (it
was 99-109s cold); the tz suite's 62s -> 25s came from the disk cache inside
the job, not from the restore. postinstall rewrites
src/environments/version.ts with the commit hash and a timestamp on every run,
and the app shell imports it, so the application and test graphs are
invalidated regardless of what was restored.
The npm cache stays, and it is the one that pays: the client install fell from
80s to 25s in that run. angular.json keeps cli.cache.environment=all, which is
what makes the within-job reuse possible.
The gitleaks step scans the working tree with --no-git, and the Angular cache
holds webpack/babel JSON full of `{ token: ... }` fragments. Restoring it
before the scan put those files in the tree and the gate failed on run 3174
with a wall of generic-api-key findings in
client/.angular/cache/20.3.41/babel-webpack/*.json.
Both cache steps now run immediately after the scan. The npm download cache
only helps the client's npm ci, which is after that point anyway; the Angular
cache only helps the tests and the build, which are later still.
The comments cited run 3170 and a timing measured on one 4 vCPU box. Both go
stale and read as a guarantee about a particular runner. They now state the
invariant the workflow relies on: an unreachable cache server degrades restore
and save to a warning, so the job stays green. The timing figures stay in the
PR description, where they are evidence for this change rather than a promise
the workflow makes.
act_runner's cache handler scopes entries to the repository (each entry is
stamped with its repo and checked on read), not to a ref, so a pull request
into one branch could seed the entry another branch's build restores. The base
branch is now part of both keys and both restore-key prefixes.
A save against an unreachable cache server is already a warning, not a
failure: run 3170 landed on runner4.4gl.io and logged four
'::warning::Failed to save: reserveCache failed: Request timeout' lines with the
job completing green.
The first run of these cache steps showed the npm entry saving (234 MB) and the
Angular one reporting 'Path Validation Error: Path(s) specified in the action
for caching do(es) not exist'. Angular keeps its persistent cache in memory
when CI is set, so no .angular/cache directory existed to upload; the disk
cache is only used outside CI by default.
client/angular.json now sets cli.cache.environment to "all", which is the
supported switch for that. Verified locally with CI=true: after removing
.angular/cache, a build recreates it (396 MB).
The runner's cache server is reachable from job containers on two of the three
boxes now, so these are the first cache entries in this repo that can actually
come back. Probe evidence, four dispatches of a throwaway two-job workflow:
- save on runner.4gl.io, restore on runner2.4gl.io: miss (each act_runner runs
its own cache server, so entries belong to the box that saved them)
- save and restore on runner2.4gl.io: cache-hit=true, the stamp came back
- save on runner4.4gl.io: "Failed to save: reserveCache failed: Request
timeout" - that box still runs the old runner config
Two caches, both keyed on client/package-lock.json so a dependency change
invalidates them, with prefix restore-keys so the previous entry still lands:
- ~/.npm, for the client's npm ci (81s cold, measured on run 3132)
- client/.angular/cache, for the production build (109s cold, 118s -> 33s with
a warm cache on a 4 vCPU box) and the two Karma runs, which share it
Per run the arithmetic while runner4 is still on the old config is about
1/3 x (+80s: two restores and two saves timing out at 20s each) plus
2/3 x (-100s), so it is a net improvement already and a clear one once the
third box is deployed. On an old box both steps only warn and the job stays
green.
actions/cache does not save a second entry for an existing key, so the Angular
cache refreshes when the lockfile changes; the v1- prefix is there to force a
reset by hand if the win decays.
Review findings on #471.
The platform rule now checks the documented tokens exactly
(['WIN', 'DNTHOST']) instead of their first letters, so an unlisted token
beginning with W or D no longer silently gets the drive path back.
brDirList raised TRUNCATED whenever it saw rows.length >= maxobs, which
included the check made after the last permitted row was added: a listing
of exactly maxobs folders with a file sorting after them reported itself
truncated although nothing was refused. The flag is now set only where a
row is actually refused, and the early exit remains the walk bound.
Two new cases pin the shape the picker depends on: a listing of exactly
MAXOBS entries is TRUNCATED='N', and a DEPTH=2 browse numbers children at
1 and grandchildren at 2 with no LEVEL 0 row. Both, plus the existing
parent-level and write-probe cases, fail against release/7.22's service.
Tests: 37 pass across the five mock suites, client spec 9/9.
Two review findings on the first two commits.
The two jobs no longer decide separately. Build-and-ng-test now publishes
`client` as a job output and Build-and-test-development consumes it instead of
re-diffing the base branch: a base commit landing between the two decisions
could make the Cypress job expect an artifact the build job had already
skipped, and the failure would read as a missing artifact. An empty output
(the build job could not decide) still runs everything.
The path list is now a PATHS variable the script can be asked about directly
(`--match` with a file list on stdin), and tests/cypress-changes.test.sh pins
one case per alternative plus the negatives (a sas/ service, docs, the
Lighthouse workflow, a "clientx/" directory, the sas lockfile). The build job
runs the test, so dropping an alternative is red rather than a suite that
silently stops covering a class of change. Checked the test fails when the
pattern is mutated.
The Cypress job installs the CLI from the version sas/package.json pins
(`CLI_VERSION=$(node -p "...dependencies['@sasjs/cli']")` in "Deploy mocked
services"), so bumping that pin changes what the suite runs without touching
anything under sas/mocks/. A pull request that only bumped it would have been
skipped by both jobs.
The sas lockfile stays out of the set: the job installs that CLI globally by
version string, so the lockfile cannot change what it resolves.
Three defects behind staging's "Invalid parent" and the picker that could
not list a subfolder. Found by running the real mpe_dirlist on
sas.4gl.io, not by reading it.
1. The browser host answers SYSSCPL "browser". The client looked for the
first letter of a unix token (L/H/A/S) and fell through to the Windows
default for anything else, so the deploy screen pre-filled
C:\DataController on a POSIX runtime, which the server then refused as
an invalid parent. Windows is now recognised by its own tokens (WIN,
DNTHOST); anything else, including a value the list has never seen,
gets the POSIX path.
2. mpe_dirlist emitted a non-blank parent's children at LEVEL 0, because
mp_dirlist numbers a folder's contents with the folder's own level.
The picker keeps LEVEL>0 for a non-root listing, so every row was
discarded and a subfolder could never be browsed. The children are
shifted to start at 1; a blank parent still lists the platform roots
at 0. Verified on the estate: /tmp at DEPTH=1 returns levels [1],
a blank parent returns [0, 1] (the root, then its children).
3. The mock's write probe generated one name and wrote it, truncating a
file that was already called that; mpe_writecheck tries up to ten
names with fileexist()=0 first. The mock now does the same and
reports 'No free name for the write probe' when all ten are taken.
The mock's DIRLIST now matches the estate's shape too: children at
LEVEL 1, roots at 0 followed by their children, and the cap stops the
walk at MAXOBS rather than slicing afterwards. Tests: 35 pass across
the five mock suites (makedata-actions 15, two of them new: the probe
collision and a folder that cannot be written to), sasjs lint clean,
client spec 8/8 with the new platform cases and 2 failing without the
fix.
The build job spent 402s on every pull request (run 3132, release/7.22).
Two of its steps were work nobody consumed:
- `npx cypress install` (11s, measured 17:07:05-17:07:16) downloaded and
unzipped the Cypress binary into a job that only runs Karma, eslint, the
licence checker, the mock service tests and `ng build`. Cypress runs in
Build-and-test-development, which installs its own binary.
- The production build (109s) and the client-dist upload exist only to feed
the Cypress job, which already skips pull requests that touch nothing it
exercises. On those pull requests the artifact was uploaded and never
downloaded.
The changed-path decision now lives in .gitea/scripts/cypress-changes.sh and
both jobs call it, so the build job's gate cannot drift from the Cypress job's
skip. The script keeps the existing fail-open rule: anything it cannot decide
(no base ref, failed fetch, failed diff) writes no output, and every caller
gates on `!= 'false'`, so the work still runs.
Three findings from the review of the release head.
A quote, a comma and both parentheses were refused as parent/path characters,
which are legal on both platforms - "C:\Program Files (x86)", "O'Brien",
"/data/sas, archive" - so the deploy screen refused to browse or write-check a
folder it could have been deployed into. writecheck never turns the value into
macro text (it is read back with symget and handed to filename()), and dirlist
now passes it to mp_dirlist through %superq, so no character in it is read as
macro text; both keep the length, control character and ISPATH checks and drop
the character refusals. brCheckPath in the mock agrees, so the two estates
answer alike, with a test for a folder named "Program Files (x86), O'Brien".
brDirList walked every level and sliced to MAXOBS afterwards, so browsing /usr
or /proc at depth 3 still stat'd everything below it - the cost the cap exists to
avoid. It stops adding rows once the cap is passed, as mpe_dirlist does.
rejection.js and postdata.js loaded MPE_REVIEW before taking the lock and
released it straight after the write, so two requests could both read before
either locked and the second save dropped the first row. The lock now covers
the read-modify-write it is there for, released in a finally.
7.1.7 closes the outstatus step with run, stops truncating the flow table's
state, and strips the response envelope before the JSON libname; 7.1.8 lets
mp_replace delete, which is how the envelope is stripped. Without them a Viya
run cannot read an aborted child's result: ten of Data Controller's SAS suites
report no result at all.
The payloads are the SAS 9 recordings, separators included - the front end splits
metadata URIs on the backslash, so a value that arrives without one exercises the
wrong branch. The envelope is not SAS 9: _PROGRAM, SYSUSERID, SYSVLONG and
SYSDATE/SYSTIME come from whatever runtime serves the request, so this variant
must not be read for SAS 9 envelope values.
The guard is the tree's one shared rule, but each of the three services that
needs it repeated the path derivation and then eval'd a file read out of the
SASjs Drive at run time - which is what the file's own header said this tree does
not need. weboutGuard.js now exports the function and the services require it by
absolute path: one copy of the function, no eval, and a load failure is a
require error naming the file rather than a silent no-op.
Being a module also means the guard has to carry its own dependencies: as an
eval'd snippet `fs` came from the program's scope, and the first run as a
required module failed with "fs is not defined". It now requires fs itself.
dcMockUtils.requireMemberMetadata remains the source of truth for the rule and
both files say so, since a site may run both trees and an operator reads
whichever one threw.
Two review findings on the approval hooks.
mpeRunhook read only the lower-case column name, while a row written
through the editor carries the schema's upper-case names (PRE_APPROVE_HOOK)
and the buskey/loadtype reads beside it already try both. A hook
registered that way resolved to nothing, so the approval ran without it
and said nothing.
postdata ran the pre-approve hook before the duplicate-approval guard.
postdata.sas checks prev_upload_check first and returns ALREADY_UPDATED
without reaching the hook, so a second approval from the same user was
aborting with the hook's message instead. The guard is now computed
before the hook and the hook is skipped when it fires. The hook still
runs for SHOW_DIFFS, which is what the SAS service does too - the macro
call sits outside both action branches.
Both cases are pinned by a test, and each fails when its fix is removed.
postdata.sas runs PRE_APPROVE_HOOK before the diff and before any write,
and POST_APPROVE_HOOK after the load is applied. The mock ran neither,
so an approve hook registered on MPE_TABLES did nothing on the
browser-hosted estate - while the post-edit hooks (which live in
stagedata/loadfile, mirroring mpe_loader) did run.
mpe_runhook(tableReg, hookvar) in dcMockUtils picks the hook out of the
MPE_TABLES row by column name and returns its source; the service
eval()s it in its own scope, which is the JS equivalent of the macro's
%include sharing the caller's macro variables. The three call sites that
already resolved a hook by hand (getdata pre_edit, stagedata/loadfile
post_edit) now go through it, so the next hook column costs one line.
postdata runs pre-approve before the staged data is diffed and aborts on
failure (nothing is written), and post-approve after the base table and
control tables are updated - so its failure is reported but cannot undo
the load, which is what postdata.sas does with its SYSCC= response.
A hook refuses by throwing or by setting hookErrmsg, the convention the
post-edit hooks already use.
sas/mocks/test/hooks.test.js covers the ordering: pre-approve refuses
with the base table untouched and the submission still SUBMITTED;
post-approve aborts with the row already in the base table and
MPE_SUBMIT APPROVED; and one hook proves it ran by writing a file, not
by the service reporting success. 30 tests pass; 4 of the 5 hook cases
fail against the base service.
Three more places where the mock answered differently from the SAS service.
mp_abort (dcMockUtils): every mock refusal is now a sasjsAbort table
(MSG + MAC) written where the macro writes it, so the client shows the
abort modal and rejects the promise. A thrown error had become an HTTP
500 "Job execution failed" - the browser host discards the webout when a
program throws - which is unreadable to the user and hides the message in
a stack. mpAbort also stops the program, so nothing is written after a
refusal; each service that can abort catches that stop at its top level.
stagedata, loadfile, getauditfile and makedata now go through it, and the
hand-rolled sasjsAbort blocks are gone.
mp_lockanytable (dcMockUtils): a lock is a row in MPE_LOCKANYTABLE marked
LOCKED carrying the holder's job id, user and reference, exactly as the
macro records it, and a lock held by another session refuses the write
with the macro's message. postdata and rejection now lock MPE_REVIEW and
MPE_SUBMIT around each write, which is where postdata.sas and
rejection.sas take them. The macro's retry loop is collapsed to a single
attempt: the runtime is single threaded, so nothing can release a lock
while the request waits.
startupservice: saslibs is the distinct LIBREF of the permitted datasets,
which is what "create table saslibs as select distinct libref from
&syslast" produces - it was a four-column row describing the control
library, with fields the client does not read.
Tests: abort.test.js and locks.test.js join the harness suites; 25 pass,
and 9 of the 13 relevant cases fail against the base services.
Every screen reads public/startupservice, so a divergence there is a
divergence in the whole app. Three of its answers were fixed values:
xlmaps a hardcoded list of three maps, so editing MPE_XLMAP_RULES
or MPE_XLMAP_INFO changed nothing on the site
sasdatasets included the map target dataset, which the SAS service
deletes - the demo offered MPE_XLMAP_DATA as an editable
table while SAS refuses it
timezone TIMEZONE and SYSTIMEZONEIDENT pinned to Europe/London and
the offset to +0100, so a GMT session reported an hour of
DST it did not have
They now come from the same sources the SAS service reads: the current
MPE_XLMAP_RULES rows joined to MPE_XLMAP_INFO (coalescec to
<mpelib>.MPE_XLMAP_DATA), the target datasets removed from sasdatasets,
and the session zone and current offset taken from the runtime, with
TIMEZONE left blank as an unset option is.
The shared harness (sas/mocks/test/harness.js) makes the service tests
one-liners: it builds a temporary estate with every service on its drive
and runs a service file the way the runtime does, webout() included.
sas/mocks/test/startupservice.test.js covers the joins, the target
removal, closed rules and the offset.
Staging now opens on the deploy screen, and its folder picker asks
makedata for a listing. The mock ignored ACTION entirely: every request
seeded the control library, deleted makedata from the drive and replied
with the HTML success page. So the first BROWSE deployed the site and
closed the wizard the visitor was standing on, behind an "invalid Json
string" error from the adapter, and VALIDATE and SUBMIT then answered
500 "Program not found in drive".
The mock now dispatches on ACTION the way the SAS service does:
DIRLIST the folders below PARENT, one level at a time, with the
dirinfo/dirlist shape mpe_dirlist emits
WRITECHECK the checkwrite row mpe_writecheck emits, probed by creating
and removing a file
MAKEDATA the existing seeding path, unchanged
ACTION is required and refused when absent or unknown, and the two browse
actions are refused once the settings file exists, mirroring browse_gate.
Both input routes are accepted - the screen's fromjs table and a CLI
seed's SASControlTable - because %webout(FETCH) does not care which one
arrived; deploy/makedata.json now carries ACTION=MAKEDATA as the real
service requires.
sas/mocks/test/makedata-actions.test.js runs the service file itself
against a temporary estate and is wired into the build workflow.
Three review follow-ups on the guard the sas9 tree loads.
- A body that cannot be read is now its own error, naming the path and the
filesystem message. Reporting it as missing metadata sent the reader to a
@sasjs/utils reinstall that cannot fix a path.
- One statement of the dependency fact: showmeta arrived in @sasjs/utils
4.6.0 and @sasjs/server declares that floor from 3.12.3 (3.12.0 and
3.12.2 declare ^4.4.0, checked with npm view), so the message names the
server version to install instead of "reinstall the server".
- CONTRIBUTING lists _program and weboutPath among the predeclared globals,
which the guarded services use to locate the helper.
The sasjs tree's copy in dcMockUtils.js had the same two defects, so both
now say one thing rather than one tree being clearer than the other.
The sasjs tree has emitted its responses through the runtime's webout()
since #461; the SAS 9 tree under sas/mocks/sas9/ still assigned every
captured payload to _webout, so its envelope was frozen at the 2022
recording and its values carried the separators a template literal eats.
All 35 services now call webout('OPEN') / webout('OBJ'|'ARR', name, rows)
/ webout('CLOSE'), with the three services that carry column metadata
passing { showmeta } and loading the guard from the new weboutGuard.js.
The payloads come from the captures read as JS produces them, with the
backslashes a template literal silently drops restored: 121 separator
values across 14 services - OMS URIs (OMSOBJ:<Class>\<repo>.<id>),
BUILTIN\Administrators, SAS\sasdemo and D:\... / C:\... paths.
Staging is the preview of the next release, and what it should preview first is
the deployment itself. DC routes to /deploy while makedata is still on the drive
(at <appLoc>/services/admin) and stops doing so once the wizard has run it and it
has removed itself - so a drive that carries the services and the licence but no
control library opens on the deploy screen, and the visitor deploys the control
library from the wizard into their own IndexedDB drive.
The seeding step is now gated on a tag: only the demo (which a visitor should be
able to use straight away) runs makedata before the drive is captured. The
staging path gets a guard of its own, because a seed that leaked into it would
only be visible as a site that opens on the home screen: makedata must be on the
drive and no mpe_tables.json may be there.
CONTRIBUTING updated: the two sites now ship different drives from the same
build.
The captures wrote values like OMSOBJ:SASLibrary\A59LNVZG.B500000K inside a
template literal, where \A is an escape, so the separator never reached the
client: it served OMSOBJ:SASLibraryA59LNVZG.B500000K. The same happened to
D:\MyDemo\Sankey and BUILTIN\Administrators. Converting the service to webout()
faithfully preserved the corrupted value - and the metadetails ASSOCURI, the one
field fixed by hand at conversion time, was left as the only member that kept its
separator, which is the inconsistency the review named.
Seventeen values corrected across five services:
lineage/fetchtablelineage info[0].LIBURI 1
lineage/getmetacols metacols[].COLURI 11
lineage/getmetatables PATH_SCHEMA, TABLEURI 2
public/refreshlibinfo libinfo[0].OWNERS 1
usernav/usermembersbyrole sasgroups[].URIMEM,
sasmembers[].URIMEM 2
Verified by deploying both trees on one server and comparing the parsed
responses: those seventeen fields are the only differences, and every other
member of all thirteen converted services is unchanged (metadetails' two
ASSOCURI, already carrying its separator, still differ from the capture for the
same reason).
Anything that splits a URI or a Windows path on the separator, or displays
OWNERS or PATH_SCHEMA, now sees the value the SAS metadata server returns.
3.14.0 ships @sasjs/lint 5.0.0, so the studio reports OS commands, and it is the
release the estate runs. It still declares @sasjs/utils ^4.7.0, so the envelope
formats from utils 4.8 (SYSCC as text, DATE7, TIME5, E8601DT26.6) are not yet
guaranteed by the floor - a fresh install resolves 4.8.0, a cached tree need not.
The comment says so, and the pin moves again when the floor lands.
Verified on a local 3.14.0 install (clean, resolving utils 4.8.0): the mock
estate's startupservice returns SYSDATE 04NOV26, SYSTIME 12:44, SYSCC "0" (text),
END_DTTM 2026-11-04T12:44:14.954000 (no Z). With that estate seeded, and against
a fresh app location, licensing.cy.ts passes 10/10 and metanav.cy.ts 1/1 on
Chromium.
The other nine services that answer with a captured response also assigned it to
_webout, envelope and all. They now emit their members through the runtime's
webout():
lineage/fetchtablelineage info, finalfinal, flatdata, idlookup
lineage/getmetacols metacols 11 rows
lineage/getmetatables metatables 1 row
usernav/usermembers users 2 rows
usernav/usermembersbygroup sasmembers 2 rows
usernav/usergroupsbymember groups 1 row
usernav/usermembersbyrole sasgroups 1 row, sasmembers 1 row
usernav/userroles roles 46 rows
public/refreshlibinfo libinfo 1 row
Each payload was taken from the response the old service serves, then both
trees were deployed on one server (/Public/app/dc and /Public/app/dcnew) and
compared as parsed JSON with the envelope keys excluded: all nine are equal row
for row, including the escaped values (the OMS URIs keep their backslash, and
refreshlibinfo keeps the quoted paths).
The envelope is the runtime's now, so SYSCC is numeric 0 rather than the string
"0", SYSDATE/SYSTIME are today's, SYSSITE is reported (these captures carried a
lower-cased syssite, which the client never read), and the SAS-only
_METAUSER/_METAPERSON are gone.
lineage/fetchlineage is deliberately left alone: its captured body is not valid
JSON today (bare quotes in the graphviz labels, raw newlines inside the STRING
values, and a lost backslash), so there is no payload to preserve and the
intended text needs a decision rather than a rewrite. Tracked separately.
The four metanav services assigned a captured 2022 response to _webout, envelope
and all, instead of using the runtime's webout() like every other service. The
members are unchanged:
metarepos outrepos 2 rows
metaobjects objects 1 row
metatypes types 71 rows
metadetails attributes 12 rows
associations 2 rows
Both trees were deployed on one server, at /Public/app/dc and /Public/app/dcnew,
and the responses compared as parsed JSON with the envelope keys excluded:
outrepos, objects and types are equal row for row.
metadetails changes in one place. The capture wrote the OMS URI as
OMSOBJ:Property\A59..., but \A is an escape in a JavaScript template literal,
so the backslash was consumed and the client received OMSOBJ:PropertyA59... -
a URI that cannot resolve. The member now carries the backslash the SAS
metadata server returns.
The envelope comes from the runtime now, which also drops three artefacts of
the recording: SYSCC "4" on metarepos (an error code from the captured day,
which the client reads as a SAS error), SYSDATE/SYSTIME frozen at 26SEP22, and
the SAS-only _METAUSER/_METAPERSON.
The mock services call webout(), as the server's runtime does. The browser mock
runtime that @sasjs/cli ships is what provides it there, and the 6.5.0 this repo
was pinned to does not: its runtime-core.js has no reference to the webout
prelude, and the build writes no webout-prelude.json for it to load. Every
service that calls webout() therefore throws "ReferenceError: webout is not
defined" in a published mock site while the build itself reports success.
6.9.0 is the first release that ships the prelude; this pins 6.10.0, the current
release. The transitive @sasjs/adapter and @sasjs/utils entries move with it.
The guard that keeps a runtime without webout()'s options argument from serving a
thinner response tested webout.length >= 4. Function.length stops counting at the
first default or rest parameter, so a runtime that does carry the option but
declares it `options = {}` or gathers the tail with `...args` reports three, and
every service that calls the guard refuses to start on a working runtime.
It now asserts the outcome instead: the member the option produces is looked for
in the response body the program is writing. `$sasdata`, `$viewdata`, `$vals` are
the members the client reads for column formats and labels, and their presence is
the thing that actually has to hold.
Where it runs matters too, and the first attempt got that wrong: called after
webout('CLOSE') the throw leaves a complete body behind, and the server serves it
- the caller sees a normal response with SYSCC 0 and no metadata, which is exactly
the silent failure this exists to prevent. It is called before the envelope is
closed, so an unhonoured option fails the run.
@sasjs/utils 4.7 makes webout() emit SYSSITE from a configured site id,
defaulting to 123, where %webout takes &syssite. Until 3.13.0 the prelude
wrote an empty string, and dc's client routes an expired licence key and a
wrong-organisation key only when startup_site_id is truthy - so on the mock
estate both pages were unreachable and licensing.cy.ts failed three cases.
Three values reached `run:` blocks through `${{ }}`, so the runner pasted them
into the script before the shell ever saw it - which makes the value part of the
code rather than an argument to it.
- `BASE="${{ github.base_ref }}"` (build.yaml, lighthouse.yaml) - a ref name in
the script.
- `echo ${{ secrets.SHEET_PWD }} | gpg --passphrase-fd 0` (all four workflows) -
a passphrase in the script. A quote in it ends the quoting, and a command
substitution in it runs.
- two `replace-in-files --replacement=...${{ secrets.CYPRESS_* }}...` lines
(build.yaml, release.yaml) - the same, inside a replacement string.
Each now arrives as a step `env:` value and is expanded as `"$VAR"`, so the shell
only ever sees data. Checked the replacement lines side by side with the old form:
identical for an ordinary value, and the new form keeps a value containing a space
intact where the old one word-splits it, and survives a value containing a single
quote where the old one is a shell syntax error.
The passphrase pipe also moves from `echo` to `printf '%s\n'`, so the byte the
decryption sees does not depend on how the shell's `echo` treats backslashes.
Review finding: when every body is refused by carriesClause, or the response
carries no FILTER_HASH, capture.hash stays empty - and storedClauseFor('')
then spends its 60 retries before reporting "no clause for <var> under",
which names neither the refused body nor the missing hash.
The hash is now asserted non-empty before the read, with the case's own
message, so a broken delivery fails where it happened.
The call count stays where it was - after the data load - because moving it
ahead of the read would check it before the app has issued the second POST
that the count exists to catch.
The duplicate check waited a fixed 250ms after reading the stored row, which is
wrong in both directions: on a slow runner the window can elapse before the first
response callback has incremented the counter, and a second POST landing after it
passes with two deliveries.
The app's sequence for an apply is validatefilter, then the data load for the
filtered route, so the load is the completion signal: the count is asserted after
it, and nothing depends on how fast the machine is. A delivery issued later than
that is a second apply, not a duplicate of this one.
Also corrects the ordering paragraph in cypress-specs.sh: it still said specs are
appended, in a file that now places va-ddc-host ahead of metanav. va-ddc-host does
not depend on the estate being unlicensed either - it passed 3/3 immediately after
metanav planted a key in the full CI run - so the bullet says what is true and
metanav stays the only spec with a fixed place.
Review finding: the matcher only accepted the variable name at the start of the
body or after a comma or newline, so a delivery wrapped any other way was not
counted - and the case then failed at the call-count assertion, reading as
"the app never called validatefilter". Reproduced against the previous
implementation: a urlencoded form value, a form value that starts with the
name, a JSON object keyed on the column and a JSON string value holding the
row all matched nothing.
The body is now expanded into the candidate strings it can carry - the raw
text, the percent-decoded text, each form field's value, and every string leaf
of a JSON body - and the name has to be a whole CSV field of one of them.
Field equality rather than delimiters means a quoted literal, a longer name,
an SQL fragment or another variable's name is still refused, and there is no
regular expression over user text to get wrong.
Also restores the ordering note in cypress-specs.sh: va-ddc-host asserts
nothing about licensing, so it runs before metanav, the one spec that plants a
licence key.
The pin moved to 3.12.3 with the webout() change but the comment above it still
named 3.12.2 as the current release - true when it was written, wrong under the
pin it now sits on. The comment also only carried the /ddc-host reason, which
does not explain this bump.
3.12.3 is the first server release declaring @sasjs/utils ^4.6.0, and that is
what gives webout() the options argument getdata, viewdata and getcolvals pass as
{ showmeta }. Say so where the constant is, so the next person to move it knows
the metadata members depend on it.
The services in this branch pass { showmeta } to webout(), which is
@sasjs/utils 4.6 and up. sasjs/server 3.12.3 is the first release whose
package.json asks for ^4.6.0 (a clean install resolves 4.6.1), so the pin
moves from 3.12.2 to 3.12.3 in every workflow that starts the mock estate.
Verified against the published package rather than the release notes:
`npm view @sasjs/server@3.12.3 dependencies.@sasjs/utils` answers ^4.6.0, and
`npm install @sasjs/server@3.12.3` in an empty directory installs
@sasjs/utils 4.6.1, whose webout prelude carries showmeta.
Review of the release diff: `npm install -g @sasjs/cli` and
`replace-in-files-cli` were unpinned, so CI ran whatever release was current
and a bad one would reach the pipeline unreviewed - next to a SASjs server
that is pinned to an exact version.
The CLI now comes from the version sas/package.json pins, which is the shape
docs.yaml and release-assets.yaml already use, so there is one source for it.
replace-in-files-cli, pm2 and wait-on are pinned to the versions in use
(4.0.0, 7.1.2, 9.5.1).
Both node expressions are checked from the directories they run in:
`require('./sas/package.json')` at the root and `require('../../package.json')`
in sas/mocks/sasjs both answer 6.5.0.
The three services that pass { showmeta } - getdata, viewdata, getcolvals - rely
on webout()'s fourth argument, which arrived in @sasjs/utils 4.6.0. @sasjs/server
3.12 declares ^4.4.0, so an install can resolve 4.5.x, and there the argument is
dropped silently: the response still looks complete but $sasdata, $viewdata and
$vals are absent, so the client's format and label paths stop being exercised
while the mock estate reports success.
requireWeboutOptions() checks that the generated webout() takes an options
argument (arity, because the option names live outside the callable function) and
throws naming the member, the cause and both fixes. A 4.6 runtime is unchanged.
The matcher tested the variable as a substring of the whole request body, so a
RAW_VALUE, an SQL fragment or another variable whose name contains it was counted
as another delivery of this clause and failed the count. It now requires the name
between its CSV delimiters, which is the filterquery VARIABLE_NM field; a quoted
literal cannot match, because its delimiters are quotes.
cy.wait('@validatefilter') used the 5s default, where the polling loop it replaced
tolerated 15s, and the message only arrives once the framed page has rendered.
It now waits for longerCommandTimeout, as the rest of this spec does.
The capture assertion also carries the tail of the host page's log in its message:
the failure reads "expected -1 to be above -1" otherwise, and the log is what says
whether the app never exposed its buffer or the frame never loaded.
Two review findings:
- The load handler re-posted pendingPayload on every load event of the frame,
so an in-app navigation or a self-reload posted a filter that belonged to
the mount which had already delivered it, and logged it as that mount's.
The handler now takes the payload (clearing pendingPayload and
pendingTiming) before posting, so a later load has nothing to send.
- The README illustrated report.json with one parameter while the committed
file carries three, and the prose below reads as though one parameter were
the shape. The example block is gone: the file is the example, and the keys
are documented as a table with the parameter rules kept underneath.
The last eleven services still built the response envelope through
dcMockUtils' webOutOpen/webOutObj/webOutClose. They now use the webout() the
SASjs Server emits into every JS program, so dcMockUtils no longer emulates
%webout at all - the helper existed only for these callers and is deleted
with them.
That needed the options object added to webout() in @sasjs/utils 4.6
(showmeta, maxobs), which is what a clean install of @sasjs/server@3.12.2
resolves to (^4.4.0 -> 4.6.1, checked). A server whose install still holds
4.4/4.5 refuses the option by name rather than ignoring it, so the failure is
loud rather than a response that is quietly missing its column metadata.
getdata, viewdata and getcolvals send their metadata as
`webout('OBJ', name, rows, { showmeta })`, and every other call keeps its
member name and rows unchanged.
One call was positional all along: startupservice's xlmaps is an array of
arrays, which the old helper serialised as-is and webout() refuses for OBJ.
It is now webout('ARR', 'xlmaps', xlmaps) - caught by the comparison run
below, not by the rewrite.
Two gaps in the guard that decides which validatefilter response belongs to the
case.
A body Cypress parses into an object stringified to "[object Object]", so no
call matched and the case failed on the count with nothing naming the body shape.
The body is now serialised the way the response already was - a string is used as
it is, anything else is JSON.stringify'd.
The count was asserted immediately after the first response, which is before a
second POST from the same apply could reach its own callback, so a duplicate read
as one call. The count is now asserted after the stored row has been read back,
with a settle that covers a second POST still in flight when the read returned.
Two review findings on host.js:
- postToFrame() used '*' as targetOrigin, so the filter clause went to
whatever document occupied the frame. The frame is the app on report.server,
so that origin is the only one that should receive it; the origin is now
resolved once from report.server (new URL(...).origin) and a report.server
that is not a URL logs a visible error instead of widening delivery. The
page also says so when it is not on the frame's origin, because the capture
probe cannot read across origins and that would otherwise look like a
harness failure.
- startEarlyReplay() accepted any __vaLastMessage once it had posted, and on
a remount to the same URL the old document is still on screen carrying the
buffer the previous mount captured. A capture now only counts once the
document has been replaced (mountDocument), both in the interval and in the
load-event check. Keying to the document rather than waiting for a load
event keeps the first post where the early window is - the interval starts
exactly as before.
Every push to a pull request queued another full run - lint, licence check, the
mocked estate and the twenty-six minute suite - and nothing cancelled the run for
the head it replaced. With two runners shared by every pull request in the repo,
those stale runs are picked oldest-first and delay the head that can actually be
merged: runs 3002, 3003 and 3005 were queued for heads that had already been
superseded while the current one waited behind them.
The group is the pull request, so a newer push cancels the run for the older
head. Pushes that are not pull requests are not affected - the release workflow
answers to its own group.
Three review findings on the harness:
- The usage line advertised `--pass`, which puts the secret in argv - readable
from `ps` by every account on the host, and kept in the shell history. The
flag is gone: SASJS_PASS stays the documented route, and `--pass-stdin`
reads one line from stdin for scripts that cannot use the environment.
- deploy() reported success on the deploy route's 2xx alone. That means the
tree was accepted, not that the app is served: a truncated `type: 'file'`
member still serves a 200 page, and a stale asset from a previous deploy is
a 200 by definition - both of which the README already lists as the sharp
edges of that route. The page and all three assets are now fetched back and
compared with the bytes that were sent, failing with the size and hash of
both copies; a 401 is reported as deployed-but-gated, since on a server-mode
estate that is the gate working. `--verify-only` runs just that half.
- Two harnesses stand in for a VA report (this page and
client/cypress/e2e/va-ddc-host.cy.ts) and nothing said which does what. The
README now names both roles and points at the contract of record - the
message shape and __vaLastMessage in va-messaging.service.ts, and the
pre-bootstrap capture rule in va-early.js - including why the probe differs
between them (Cypress proxies the framed document).
The intercept kept one hash from any validatefilter response in the case, so a
second call - another table, a revalidate on load - would have replaced it and
the rows read back would belong to a different clause. The response is now only
taken when the request body carries this case's variable, which is what tells one
clause from another, and a case that sees more than one call for that variable
fails on the count rather than silently keeping the last.
A hash that cannot be read is captured and rethrown in the command chain: thrown
inside the route handler, Cypress reports it as an uncaught error and the case
fails later as a bare "no clause under" timeout instead of naming the response.
The spec located its evidence as "the last stored row for this variable", which
cannot tell this run's row from one an earlier case or run left behind - so it
deleted the DC_JSLIB filter tables before each case instead, and that bought a
scheduling contract: it had to run after every other spec on the shared mock
estate, and it left the estate falsified for anything that ran after it.
The mock validatefilter already returns the FILTER_HASH it computed for the
clause, so the rows are now selected by the hash this case's own response names.
Nothing is cleared, and no spec has to run before or after it. A store that
already holds this clause holds the same rows; a clause that changed hashes
differently, so a regression cannot be satisfied by a stale row.
va-ddc-host moves back in front of metanav, which is the one spec with a real
ordering constraint (its licence key must not change another spec).
The SASjs Server emits a webout() block into every JS program it runs
(@sasjs/utils >= 4.4, server >= 3.12.0), mirroring %ms_webout in
@sasjs/core: OPEN, ARR, OBJ, CLOSE. Ten of the mock services are two-year
functional implementations of their SAS counterparts, and they were
hand-building that same envelope through dcMockUtils' webOutOpen/Obj/Close.
Move those ten onto webout(), so the response envelope is built by the
runtime that owns it - real SYSUSERID, SYSPROCESSNAME, host name, a numeric
SYSCC and a content-type header - instead of constants in a mock. The
services that send showmeta metadata stay on the dcMockUtils helpers,
which webout() has no argument for yet.
The old 3.6.2 server pin in release, lighthouse and sites predates the
block, so those three move to 3.12.2 with it.
The interval tested its deadline only after the contentWindow guard, so a frame
that never became postable (detached, mid-navigation, or a load that never
fires) returned on every tick and the timer ran for the life of the page.
The capture check and the deadline now both run before the guard, so the replay
always stops. Capture keeps precedence, so a message caught on the last tick is
still reported as a capture, and it only counts once something has been posted
in this replay - a buffer left by the previous load of the same frame cannot be
read as one that belongs to this mount.
attempts keeps counting posts rather than ticks, because that is what the lines
quote ("after N post(s)"); a tick skipped for want of a frame sent nothing. The
zero case is now stated separately, since "0 post(s)" otherwise reads as a
broken probe rather than a frame that never appeared.
3.12.2 is the current release and carries the packaged-executable fix for
/ddc-host (the page this job's va-ddc-host spec drives), so the version the
tests run against is the version that ships. 3.12.0 remains the floor for
`timing=early`.
Closes#449.
- SASJS_SERVER_VERSION 3.6.2 -> 3.12.1 in build.yaml. 3.12.0 is the floor:
/ddc-host with `timing=early` (the replay that lands before the app
bootstraps) arrived there, and it is absent from 3.11.3, which has no
/ddc-host at all.
- DDC_HOST_ENABLED=true in the job's .env. The server 404s /ddc-host unless it
is set - it is a testing tool and does not announce itself - so enabling the
spec without this would have failed on the first case.
- va-ddc-host moves from EXCLUDED to the END of SPECS in cypress-specs.sh: it
clears the DC_JSLIB filter tables before each case, so it must run last on the
shared mock estate. The ordering contract at the top of that file now records
that, and metanav's licence key, which no longer has to avoid being last
itself.
Review findings on the harness.
- timing=load posted nothing: mount() only started a replay for early/both and
the load handler only logged. Both and load now post on the frame's load as
well, which is the metadata path and the second copy VA sends, so the mode the
controls offer is a mode that works.
- a re-apply with the same prompts did not reload the frame: setting src to the
value it already has is a no-op in the browser, so the new load post (and the
refresh path generally) delivered nothing. The frame is reloaded explicitly
when the URL is unchanged.
- dataType was declared and never applied. A prompt configured as number now
arrives as 42 rather than "42", which is what DC's clause builder branches on;
a non-numeric value is left alone rather than becoming NaN.
- the prompt select is found by walking the selects instead of building a
[data-name="..."] selector, so a prompt name with a quote or bracket in it
cannot throw inside apply() and leave the page with no frame.
- the replay's give-up message now says when the app does not expose
window.__vaLastMessage, so a changed probe is not reported as a delivery
failure.
- deploy.mjs throws when the Node runtime cannot read Set-Cookie rather than
silently dropping every cookie and failing later as an unauthenticated 401.
report.json carries a numeric prompt so the numeric path is exercised by
default rather than only in theory.
Review findings on the CASE marker.
- when a column stops carrying the rule (a metadata reload, or the column
becomes formula or read-only) the badge was removed but the tooltip kept the
sentence. stripCaseHeaderTitle is exported and applied there too, so the
header says nothing it cannot back up.
- the strip now removes only the `; <hint>` form the composer produces, so a
title that happens to use the same words is left alone, and a title that is
only a hint collapses to empty rather than to a hint with a separator.
- composeCaseHeaderTitle takes `string | undefined`, which is what th.title is,
and the spec drops its `undefined as any` cast.
- resolveEnteredCaseTransform no longer takes an isDerived flag: the grid only
tints a derived column, it does not block entries, so excluding it would have
stopped converting values the editor converts today. The integration spec now
drives the real predicate, with read-only and formula columns alongside the
editable ones.
Review findings on the change-detection steps and the shared build artefact.
- A failed fetch or diff in "Detect client changes" used to fail the job: the
fail-safe only covered an output that was absent, not a step that errored.
Both workflows now exit 0 without writing client=false on any failure, so an
undecidable change set runs the suite instead of blocking the pull request.
- .gitea/scripts/ joins the change set in both workflows: it carries the spec
list (cypress-specs.sh) and the runner gate (check-runner-memory.sh), so a
change to what the suite runs, or to the gate that lets it run, must run it.
- A skipped job now says so in a named step and a notice annotation, rather than
reporting success from a list of greyed-out steps.
- The dist upload sets if-no-files-found: error, so an empty artefact fails the
producing job instead of surfacing later as "webSourcePath ... doesn't exist".
- lighthouserc.js records the cost of numberOfRuns: 1 - no median, so one slow
navigation is a re-run rather than a fix.
A parent page standing in for the VA report that embeds Data Controller as
data-driven content: the dropdowns are the report's prompts, and the iframe is
framed and fed the way VA frames and feeds it.
Deliberately outside the DC build. The host is not the child: shipping it in
DC's deployment would put a test harness on customer servers and let DC be
opened top-level as a page that frames itself.
- report.json defines the report: child AppStream name, table, sandbox set,
delivery mode and timing, and the prompts (label is the va_parmap key, since
DC matches on the label rather than the report's internal parameter name).
- two delivery modes, because they reach different code in DC: refresh (reload
the frame, replay from before load - the report-refresh path) and prompt (post
into the loaded frame - an in-place prompt change).
- deploy.mjs posts the folder to a SASjs Server with /SASjsApi/drive/deploy and
streamWebFolder, which is the same route sasjs cbd uses, so a static app is
published the usual way and served at /AppStream/<name>.
Addresses the retroactive review of #447.
- afterGetColHeader appended the case hint to th.title on every render, and
Handsontable reuses the th, so the tooltip grew a copy per render. The hint
is composed by composeCaseHeaderTitle, which strips any known hint first, so
a repeat call is a no-op and a rule change replaces the sentence.
- the marker is now gated on the same predicate the conversion uses,
resolveEnteredCaseTransform, so a formula, read-only or derived column no
longer advertises a conversion the hook refuses to make. The class is removed
when the predicate stops matching.
- resolvePkFormula and the unescapeFormula write (given an explicit source) join
CASE_TRANSFORM_SKIPPED_SOURCES: neither is an entry, and re-casing a computed
key or a live `=...` string corrupts it. force_cell_validation_value keeps
converting on purpose, stated in the docstring.
Review follow-up on the browser harness.
The exclusion of va-ddc-host now names the issue that tracks re-enabling it
(dc/dc#449), so the coverage gate cannot report a rotted exclusion as accounted
for. The comment also states the ordering constraint the spec carries: it clears
the DC_JSLIB filter tables per case, and every spec in that job shares one mock
estate, so it has to run last or be given its own.
The spec's comments and test name describe the behaviour and the harness, with
no ticket numbers, per AGENTS.md. The va-filter docstring says what the inference
is: a guess from the value's shape alone, which can disagree with the column in
either direction, and which costs the deferred call rather than the page because
the attempt suppresses the abort modal and falls back to the unfiltered load.
Measured over the last 150 workflow runs on this forge: a pull request costs
56 runner-minutes (Build-and-ng-test 5.8m, Build-and-test-development 31.2m,
lighthouse 19.3m), and when three pull requests are in flight the first job
queues 47-69 minutes before it starts.
- Build-and-ng-test publishes client/dist as an artifact and the Cypress job
downloads it instead of running a second cold `npm run build` (1.6m each).
Both jobs test the same bytes.
- lighthouse: numberOfRuns 1 (lhci's default of 3 tripled the 13.6m audit to
steady a median that only has to clear a fixed threshold), and the duplicate
/view/usernav/groups entry is removed (17 unique URLs, not 18).
- Cypress records no video in CI and takes screenshots on failure instead
(`--config video=false,screenshotOnRunFailure=true`, CLI only so local runs
and the demo recorder still record); the failure artifact carries
screenshots as well as any video.
- Both workflows detect whether client/ or sas/mocks/ changed and skip their
expensive steps when neither did - a sas-only or docs-only pull request no
longer pays 45 minutes for suites that cannot observe it. The gate is
`!= 'false'`, so a missing step output runs the suite rather than quietly
reporting green.
issue 444 was a month-name date from VA becoming a character literal, and it
only reproduced on a report refresh: the deferred initial load reads VA's
buffered message, which has no column metadata, so the SAS kind comes from the
value's shape. Posting a message into a running page exercises the other path -
it arrives after getdata, metadata in hand, so a date is formatted correctly
whatever the client does - which is why the unit spec could pin the fix but not
show the failure a report author sees.
The new spec frames the built app in sasjs/server's /ddc-host page, which mounts
it with VA's sandbox attribute set and, with timing=early, replays the message
from the moment the frame is created. That is the request and apply path a
refresh takes, with mock services instead of a VA estate.
Where the assertion sits: the mock public/validatefilter stores what it is sent,
so the clause is read back from the drive (mpe_filtersource.json) rather than
inferred from a request payload. The store is emptied per case because
mpFilterStore reuses an existing FILTER_RK for a clause it has already seen, and
a leftover row would answer for this run. The host page's own log is the second
witness: the message is captured before the frame finishes loading.
Three cases in one run - the WORDDATE shape from the issue, a DATE9 date that
always worked, and a character parameter. Against the unfixed service the first
stores 'August 02, 1971' and fails, while both controls pass; with the fix all
three store the expected literal. So the guard goes red for the defect and only
the defect.
The spec runs last, needs no SAS, and takes about a minute against the mock
estate the other specs already use.
Where a column carries a CASE=UPCASE or CASE=LOWCASE rule in
MPE_VALIDATIONS, the EDIT grid treated it as validation only: a value in
the other case was flagged after typing, so the user had to retype it or
accept a red cell and fix it later.
The pending value is now converted before it commits, driven by the rule
that already exists rather than a hard-coded column list:
- DcValidator.getCaseTransform(col) maps a CASE rule on the column to
'upper'/'lower' (the two rule values dq-validation.ts acts on).
- The existing beforeChange hook applies it alongside ROUND, so an edit, a
paste range and a fill-down all take the same path, the cursor and
selection stay put, and undo keeps its history.
- applyCaseTransform / isCaseTransformSkipped live in a small pure module
so the conversion and the source exclusions are testable directly.
Values are not converted when they are not user entry: loadData (the grid
re-sending its dataset on every updateSettings({data})), UndoRedo.undo /
redo, and the Revert menu item. Converting those would rewrite values the
service supplied, or hand back something other than the value the user
asked to restore, and would leave untouched rows looking modified.
A CASE-bearing column is marked in the header ('Aa' marker plus a tooltip)
so the user can tell the conversion came from the grid.
The CASE validation itself is unchanged: values from an API, staged data
or an Excel import are still validated as before.
The footer is a %let and two %if inside the PDF branch of the service, which
already runs inside a macro, so the mpe_pdffootnote wrapper added a file and a
call boundary for a value that is built once, in one place.
Behaviour is unchanged: the same filter wording, session user, timestamp and
the truncation notice when the cap bites, still driven by pdf_maxobs so the
printed row count and the printed cap cannot disagree. A blank obs count is
treated as no truncation, which was the one real edge the macro guarded.
getrawdata.test.sas loses test 5 with it - those assertions existed because
the text was composed by a macro the test could call. With the composition
inline there is no seam to assert on, because a PDF cannot be read back as
text in SAS. The service-level PDF assertion (signature present) still covers
the branch, and the footer is checked from the downloaded file: page 1 of the
artefact reads "No filter | sysadmin | 03NOV2025 13:48 Page 1 of 20" on SAS 9,
and the same request on Viya returns a 20 page report.
7.1.3 carries sasjs/core#510: mv_getjobresult now returns a binary webout
from a job result instead of writing its own "not registered" text, so the
PDF download can be asserted on Viya as well as on SAS 9.
The only other change in the range is 7.1.2's filter validation message
(sasjs/core#511), which this project does not call.
Two real findings from review.
ACTION: the default was assigned with %let before the value was read, so a
URL-supplied ACTION was overwritten and the parameter could never take effect.
The incoming value is now captured first with symget, guarded by symexist so a
missing name does not write to the log, and the default is assigned afterwards.
%global was tried first and is wrong here: declaring a name that already exists
as a local in the including scope does nothing, while the symputx that follows
writes the global - so &action kept resolving to the local and no action ran,
which is what broke the makedata test.
Separately: on the SASjs Server STP endpoint a query parameter is not delivered
to the service at all - a request with ?action=DIRLIST&parent=/tmp reaches the
program with no fromjs and no symexist('action'), and fails on WORK.FROMJS
before ACTION is considered. The configurator's link is therefore legacy on
that target; the adapter path (the Angular deploy screen) is the one that
works, and the capture above is what makes a parameter effective if it does
arrive.
Folder type: the picker compared FILE_OR_FOLDER to 'folder' case sensitively
while mpe_dirlist selects on upcase, so an uppercased value emptied the list.
The compare is now lowercase, and entries mp_dirlist listed but could not open
are surfaced - rendered without the drill-in and Use buttons, with the service
message beside them - instead of being filtered out after the macro's header
promised they would be visible. Two specs cover both, including one that
renders a locked row.
The service tests exercise the actions through a child session; these call the
macros in place, so the guard and the answers are covered without a request:
mpe_dirlist 7 assertions - iftrue=0 does nothing, one level, files
excluded, three levels, the depth cap, the maxobs cap with
TRUNCATED=Y, and a blank parent returning a root path
mpe_writecheck 5 assertions - iftrue=0 does nothing, a writable folder, a
file, a missing path with a reason, and the probe leaving an
existing file that already had its old name alone
They live in sasjs/macros beside the macros they test, which works: a test file
in a declared macro folder compiles to sasjsbuild/tests/macros/ and sasjs test
runs it from there.
Two traps, both hit before this passed:
- these macros write JSON to _webout, so the test has to point _webout at a
temp file while they run and put it back before the term program writes the
results - the first attempt redirected without restoring and the runner
reported "No webout was returned by job";
- descriptions in mp_assert must not contain a comma: an unquoted comma splits
the parameter list and the assertion fails to compile with "All positional
parameters must precede keyword parameters", which names the call rather than
the comma.
check_outfmt and check_keepme were macros around one data step each, so they are
now those data steps, in the body where they run. Nothing else about them
changes: the &inds trick replaces the `%if &exist` guard, so the step still
touches work.fromjs only when the table is there.
That inlining exposed a real bug, found by running the KEEPME path rather than
by reading it. keepme was initialised with %let inside the body, so it was
local to makedata_body, while the request set the global with symputx(...,'G').
self_destruct, called from the body, resolves the nearest scope first - the
local 0 - so a caller asking to keep the program had it deleted anyway. It is
now declared %global and initialised at file scope, before the body exists.
Verified on the estate: KEEPME=1 leaves the program in place (a follow-up
request still finds it), a normal run deletes it (the next request reports that
the program does not exist), the suite is 3/3 with 16/16 assertions, and the
seed returns STATUS SUCCESS.
Each action is now its own macro that takes `iftrue=` and returns immediately
when its action was not requested:
%mpe_dirlist(iftrue= (&action=DIRLIST))
%mpe_writecheck(iftrue= (&action=WRITECHECK))
%makedata_body(iftrue= (&action=MAKEDATA))
so the service reads as a list rather than a branch, and there is no open-code
%if left in it. mpe_adminsetup splits into mpe_dirlist and mpe_writecheck
along the action boundary it already had.
The four helper macros move back to file scope, above the body. With the
dispatch branch gone nothing is nested any more, so the noNestedMacros
exception comes off both files - the last reason for it was the branch I just
removed.
mp_abort takes its own condition, so the two refusals are two calls in open
code rather than branches inside a wrapper macro:
%mp_abort(iftrue= (&action=MISSING) ,mac=&_program ,msg=...)
%mp_abort(iftrue= (&action=BAD) ,mac=&_program ,msg=...)
check_action was never needed either - it wrapped a single data step whose only
condition is a data-step IF on &exist.
That leaves one branch in the file, the two-way dispatch between the browse
actions and the seeding body, which is what noMacroIfInOpenCode actually
requires to be inside a macro.
Replaces the risk-register framing with the assessment: there is no admin group
until makedata creates it, the caller is the administrator doing the deployment,
the configurator in the same window is more powerful than these two actions, and
a successful run deletes the service anyway.
Filed as deploy-window-authorization.md so the next review finds the reasoning
instead of re-raising the finding.
Five defects from review, all mine.
ACTION is no longer defaulted. `%let action=MAKEDATA;` meant a caller with no
input table at all ran the seeding body - so a URL that never mentioned ACTION
would create the control library. The default is MISSING, which is refused,
and a URL parameter is whitelisted through the same check with symget() so it
is never resolved as macro text.
The write probe no longer has a predictable name. fopen with 'O' truncates, so
a probe called .dcprobe_<pid> destroyed any pre-existing file of that name in
the folder being tested. It now carries the pid and a random suffix, checks
fileexist before use, and gives up rather than reuse a name.
DIRLIST returns folders and locked entries only. The old filter dropped 'file'
and passed every other value - blanks and unknown types - to the picker as a
folder. The filter is now an explicit whitelist, and the header describes what
is actually emitted.
A request with no input table is refused, covered by a test: that is the case
the default used to swallow.
ngOnDestroy is gone from the client component - it reset a flag on a component
being torn down.
The deploy-window exposure is recorded in .agents/docs/, with why the admin
group cannot be checked there: %mpeinit() reads &mpeadmins from the control
library that makedata creates, so before it runs there is no group to check -
which is also why the previous dirlist could never serve this screen.
Reverts the hoist. The helpers read better next to the body that calls them,
and the body runs once per deployment, so the nesting costs nothing at run
time - the only cost is the noNestedMacros exception in the two headers, which
is the narrower trade than four macros defined far from their only caller.
Wrapping the body in a macro had demoted four existing helpers - check_outfmt,
check_keepme, emit_response and self_destruct - to nested definitions, which
needed a noNestedMacros exception in the file header. They were at file scope
before, so hoisting them back is undoing that demotion rather than promoting
anything new, and the exception goes away.
No behaviour change: macro variable lookup is dynamic, so the helpers still
resolve the body's variables when the body calls them.
The deploy screens asked for a physical folder the deployment user can write
to, as a single text box, and found out whether it worked only by running the
whole deployment. They now get a picker and a Validate button, backed by the
makedata actions that already answer those two questions.
A new app-dc-location component renders the path as an editable input - the
picker is a convenience over it, never a replacement, because a folder the
deployment user cannot open is not listed and a customer may want a path that
does not exist yet. Browse lists one level at a time (ACTION=DIRLIST, DEPTH 1,
MAXOBS 500), with Up, a "Use this folder" for where you are standing, and a per
row Use. Validate calls ACTION=WRITECHECK and reports the verdict inline -
writeable, or the reason it is not - instead of leaving it to the deployment to
fail.
Both screens use it: the Viya setup screen (which runs makedata through the
adapter) and the Sasjs configurator (which opens makedata as a URL). The
mkdata calls now send ACTION=MAKEDATA, since the service requires it.
Details worth knowing:
- every value read from the service is trimmed: SAS has no zero length
character value, so a blank field arrives as a single space and reads as a
value to any `||` or `@if`.
- the listing's level 0 row is the folder being listed, not a folder to drill
into - except when the parent is blank, where the level 0 rows are the
platform roots.
- failures are shown inline rather than as abort modals: the two requests pass
suppressErrorAbortModal/suppressSuccessAbortModal, because a refused path is
an expected answer here, not a crash.
- parentOf handles a drive root (C:\) and a unix root without inventing an
empty path.
Frontend tests: 14 new specs (38 in the deploy folder, all passing), including
a template spec for the browser panel and one for the verdict line, both
mutation checked.
The deploy screen needs to find and test a writeable folder before the control
library exists, so these questions cannot be answered by an admin-gated
service: %mpeinit() reads &DC_LIBREF..mpe_config, which makedata itself
creates. That is why the previous dirlist could not run in the window it
existed for.
They are answered by the service that already owns that window instead.
makedata now requires an ACTION column:
MAKEDATA create the configuration tables (what it always did)
DIRLIST the folders below PARENT, DEPTH levels deep, MAXOBS capped
WRITECHECK does PATH exist, is it a folder, can we write in it
so a deployment has one service to run and the same one to remove - makedata
already self destructs after a successful run, and now nothing is left behind
that can enumerate the filesystem.
ACTION is not defaulted: a caller that does not say what it wants is refused
rather than having a control library seeded for it. The payloads under
deploy/ and the configurator's link are updated in the same change, so the
installer and the seeding scripts keep working.
The work is in mpe_browse, called from both server and viya makedata:
mp_dirlist does the listing (a blank parent is the platform roots), and
mp_validatecol(ISPATH) shape checks whatever is submitted. Folders are
returned a level at a time so a parent with tens of thousands of subfolders is
never transferred in full, and the finite depth is what makes a symlink loop
harmless.
The body of makedata is now a macro of its own, because the branch cannot be
open code - noMacroIfInOpenCode keeps the file loadable on every 9.4
maintenance level. Its four helper macros are unchanged.
Input is the adapter table as before, never macro text: a value is refused if
it carries a macro trigger, a quote or a statement separator, and only the
three known ACTION values are ever put into a macro variable.
sas9 is not included: its makedata is the metadata STP driven by the
configurator page through URL parameters, not by the adapter, and the sas9
configurator takes a typed path rather than a picker. That is a change of
input mechanism, not of this feature, so it is left for its own ticket.
A parameter named without a value arrives blank, and a blank reached the
footer's numeric comparisons as nothing to evaluate:
%if &filter eq 0 /* blank filter */
%if %eval(&nobs) gt %eval(&maxobs) /* blank obs count */
That raised %EVAL arithmetic errors, stopped mpe_pdffootnote and failed the
whole download instead of producing a report. Both are now normalised to
their defaults first, maxobs included.
The footer's user= is taken from mf_getuser() at the top of the service - it
is deliberately not an input, so a caller cannot assert who ran the report -
and it is passed through %superq because a display name may hold a comma,
which the macro call would otherwise read as a parameter separator.
Test 5 gains a case for the blank call.
mv_getgroupmembers declared the member name at $43, and a longer name was
truncated - or, on Viya, stopped the step with "Multiple lengths were
specified".
membername is declared at $256 in the same shape the macro now returns it: one
width for every group, so the tables the caller combines are never of differing
lengths, and a long active directory name is kept whole.
Adds PDF to getrawdata: the staged, filtered dataset is rendered by
mp_ds2pdf and streamed with contenttype=PDF, so the Download modal gains
a typeset report - landscape, SAS formats and labels applied, headers
repeated on every page, 'Page n of m', and the filter, user and timestamp
in the footer.
The report is capped at 2000 observations, about 75 landscape pages, and
the footer states the truncation when it bites. The cap is one macro
variable driving both maxobs= and the footer text, so the two cannot
disagree, and the footer itself is composed by mpe_pdffootnote so the
truncation wording can be asserted without reading text out of a PDF.
@sasjs/core goes 7.0.3 -> 7.1.0 for the macro; the bump also brings
mp_validatecol ISPATH, two mp_dirlist guards and an mx_execute test.
On Viya the report renders but the download does not arrive: mp_streamfile
fails ahead of the PDF branch inside mp_binarycopy (RECFM=N with lrecl=1
needs 256), which is upstream of this change and left alone here.
The stage that reports what the file is doing is meant to belong to the whole file, and it finished at "Extracting the data finished" whatever happened. A sheet whose parse rejected had already blanked its datasource and dropped itself from the submission in the per-sheet catch, so the user was told the file had extracted while a table was silently missing from the submission.
`allSettled` was already the right primitive, but its results said nothing: the catch swallowed the error, so every promise fulfilled. The catch now names the sheet on the component, the stage ends at a new `extractionFailed` state when anything was left out, and the pane lists what was left out and why.
The two reasons a sheet contributes nothing are kept apart, because they are not the same thing to the reader: a sheet the sheet utility could not read is a "could not extract" problem in red, while a sheet it parsed and found no rows in - which arrives as the rejection "No relevant data found" - is reported as "no rows found" without calling the file corrupt. Both leave the sheet out of the submission, and either one stops the stage reporting a clean finish.
The row derivation sits inside the try with the parse. Outside it, a throw on a row shape the sheet utility did not expect escaped as a rejection that `allSettled` swallows - the dataset stayed in the submission with `parsingTable` true and the stage reported a clean finish over a sheet that had been dropped.
The messages live in the no-table-selected pane, which is where the user is once extraction has finished; the file pane they were first put in unmounts as soon as the datasets are fetched, so they would never have been read there. The stage line keeps one id per state rather than two of the same: `file-loading-stage` in the file pane while the datasets are fetched, `file-extraction-stage` in the overlay while the sheets are extracted, and the Cypress spec asks for both.
The Cypress spec asks for both stage ids, because the element that carries the stage changes state, and asserts that a sheet left out of the submission is named in the pane - under whichever reason the sheet utility reported, since only the utility decides which of the two it is.
The spec builds the component through the test injector rather than with seven positional arguments, so a constructor that gains or reorders a dependency fails there (`NG0201: No provider found for ElementRef`) instead of binding a stub to the wrong collaborator and passing.
The comment claimed the sidebar animates open/closed. It does not: the
collapse is a plain margin-left on clr-vertical-nav.closed (styles.scss)
and margin is not in any transition there - Clarity's width transition on
.clr-vertical-nav does not apply, because the width never changes. What
does need the deferral is that Angular applies the class binding after the
event handler returns, so a synchronous measurement reads the pre-toggle
layout.
The spec now says what it covers: the scheduling contract, not whether a
measurement lands before or after a transition.
Unsubscribing stops future layout notifications but leaves a timer that
one already scheduled, so a sidebar toggle immediately followed by destroy
still ran calculateInputPosition() against a torn-down view. The timer id
is now held and cleared in ngOnDestroy, and the spec covers that window -
it fails if the clear is removed.
Also drops the EventService.dispatchEvent stub from the viewboxes spec,
which advertised a method this branch deletes.
The specs gated their assertions on a real setTimeout(..., 10) plus done,
which is wall-clock timing: a loaded runner can assert before the
component's own timer fires, and the negative case would then pass
vacuously. fakeAsync + tick order the assertion on the scheduler instead.
Also drops the stale half of the viewbox resize comment - the shared tail
no longer notifies listeners, it just persists and re-measures.
The synthetic resize dispatch was its only caller. Nothing else used it,
and a generic window-event dispatcher on a service invites exactly the
coupling this change removes.
The sidebar spec now asserts the behaviour directly - no resize event
reaches window listeners on a toggle - and fails when a dispatch is put
back, rather than asserting on the removed API.
A sidebar toggle and the viewbox data/resize paths dispatched a synthetic
window resize so other components would notice the layout moved. Two
thirds of that was unnecessary: Handsontable 18 attaches its own
ResizeObserver to the grid's parent, so a container change re-measures the
grid with no event at all (measured in Chromium: view.getLastSize().width
tracks a container resize 600 -> 380 -> 700 without any dispatch).
The one real consumer was the autocomplete dropdown, which follows its
input under the new layout. It now subscribes to
EventService.onSidebarToggle and re-runs calculateInputPosition(), and
unsubscribes on destroy; the window:resize listener stays for real
viewport changes.
Removed: sidebar.component.ts:53, viewboxes.component.ts:281, :705 and
:1571. The viewbox resize tail already re-measures each instance through
refreshTableAfterResize, and a data load renders the grid itself.
Scoping the pass to one grid left the page's other grids uncorrected: the
viewboxes each render their own Handsontable (shared/viewboxes), and the
multi-dataset preview renders two. Nothing called the pass for them, so
they kept role=treegrid and aria-rowcount=-1 - the failures this change
exists to remove. The old document-wide sweep happened to cover them.
observeGridAria(hot) runs the pass and keeps it running from the grid's
own afterRender, for grids whose component has no render hook of its own.
It defers by one microtask: Handsontable writes the row/column counts
after the render that follows a data load, so a pass inside the hook
still sees the previous frame.
Wired into getViewboxHotInstance (every viewbox that hands out an
instance) and into both multi-dataset instances.
The root `package.json` carries no install or postinstall script, so the
guidance to rely on "the composite install script ... which installs both
subprojects" sends the reader after something that does not exist, and
`client/.npmrc` told them the same thing in a comment.
The install section also contradicted itself: it opened by telling the reader
to write the SheetJS registry token into `client/.npmrc`, and closed by
explaining that `client/.npmrc` is a tracked policy file that must not carry a
token (the pre-commit hook rejects one). The token goes in the user config,
which is what the first block now says.
The install sequence itself is unchanged - root `npm ci`, then `client` and
`sas` explicitly - because that is what the repo actually needs.
Review follow-up. The table pass matched every table inside the grid root,
so a table a cell renderer owns picked up role=table and scope=col - which
contradicted the reason the wrapper loop avoids [role="treegrid"]. It now
matches table.htCore only (every HOT table carries that class, one per
view), and header cells are read with :scope > thead > tr > th so a
renderer's nested table is not touched. The separate aria-rowcount=-1 pass
is gone: the wrapper strip already removes that attribute from the grid's
own element, and matching it anywhere else was the same overreach.
The viewer also gets the editor's fallback chain, so a pass that runs
before hotInstance is assigned still finds the grid through
hotTableComponent.
Scoping the ARIA pass to the grid exposed a dependency the document-wide
sweep had been hiding: the sweep used to write aria-label from an
element's text, so cypress/e2e/full-table-search.cy.ts could select the
filter modal's submit button as aria-label="Ok" - a label only the sweep
ever wrote. Once the sweep stopped reaching the modal, that selector
matched nothing and the dev job failed on it.
An element's text already is its accessible name, so the synthesis is
gone from the pass, and the spec now selects the button by its text the
way a user would.
The helper-only spec passes even if the observer goes back to calling
updateSettings() - which is the regression the helper exists to prevent.
Move the observer into refreshGridAfterContainerResize.ts so a spec can
drive a real container resize through it, and assert that the resize
reaches refreshDimensions() without any settings call. Verified by
re-injecting the old wiring: the spec fails with 'Expected spy
updateSettings not to have been called', and passes once it is removed.
Also corrects the helper's claim about what refreshDimensions() measures:
it compares rootElement.getBoundingClientRect() with the last measured
size, and both grids take their height from a viewport-derived calc(), so
a resize that matters is a width or viewport change.
The sweep matched [role="treegrid"] inside the root, which would also
catch a treegrid a cell renderer put in a cell. The grid's own wrapper is
the root itself, so target that plus the overlay wrappers Handsontable
renders, and leave nested widgets alone.
Adds a live-grid spec: Handsontable writes role=treegrid/aria-rowcount
again on loadData, so the pass has to run after renders, not once.
The editor and viewer each ran fixAriaAccessibility() over the whole
document on every render, on a 3-second interval and from a
MutationObserver on document.body, and each call scheduled four further
copies of itself. The sweep removes the role/aria attributes that its own
observer watches, so it re-armed itself.
Measured with axe-core against Handsontable 18: the default markup fails
aria-required-children (critical) because the treegrid wrapper has no row
children, and disabling the tags with ariaTags:false only trades that for
empty-table-header while leaving the placeholder semantics missing. The
existing attribute sweep is therefore the right end state - it just has to
run against one grid's root, not the document.
Both components now call one shared helper with their own
hotInstance.rootElement, and the interval, the body observer and the four
deferred re-runs are gone; the afterRender/afterChange hooks and the init
calls are the only triggers.
The grid's container ResizeObserver (added in c7ba025 to replace the
afterRender window-resize dispatch of #246) refreshed by calling
updateSettingsSortSafe({ height }), and that wrapper's whole reason for
existing is that any updateSettings() while a multi-column sort is active
corrupts formula references into #REF!. It works around that by clearing the
sort, applying the settings and re-applying the sort - so every sidebar
toggle, modal open/close or window resize re-sorted the user's data, and an
open cell editor (anchored to visual coordinates) could end up writing to a
different row than the one being typed into.
refreshGridAfterContainerResize() replaces it with hot.refreshDimensions():
it compares the element's box with the last known size, re-measures the
overlays and redraws only when the size actually moved, touching no settings.
The decision lives in its own module so the spec can pin it - a real
Handsontable instance with an active sort and a formula column asserts that
the resize path calls refreshDimensions() and does not call updateSettings(),
clearSort() or sort(), that the sort config is unchanged, and that no cell
degrades to #REF!.
This is the narrow, independent piece of #246; the wider removal of the
remaining synthetic window resize dispatches (sidebar, viewboxes) is analysed
on the issue and is not part of this change.
Two findings from review.
downloadTablePdf() revoked the blob URL in the same task as the anchor click.
Firefox and Safari read the blob asynchronously, so that can cancel the
download and leave the user with no file; the revoke now runs on a later turn,
after the click has been handed to the download machinery.
getExportRange() clamped the selection's left edge up to skipLeadingCols but
then forced the right edge up to match, so a selection lying entirely inside
the leading housekeeping column(s) was shifted sideways into a column the user
never selected - the PDF contained column 1 when they had selected column 0.
The range is now clamped into the exportable columns and comes back empty
(right < left) when nothing is left, which the PDF path reports as "no table"
and the CSV/Excel path skips rather than exporting the wrong column.
The revoke change is only covered by the spec (it needs a browser to exercise);
the range change is covered by the spec and by the local harness, now 28 checks.
The modal decided what was wrong with two regular expressions of its own, and the answer disagreed with the log it was showing:
- an ODS HTML line was cut from the first capital `E` in it and lost its last character, so an error read `ERROR: File WORK.X.DATA does not exist.</font>`;
- a message SAS wrapped over several lines was reported once, truncated at the break, and the rest of it was not reported at all;
- the colon-less form SAS writes (`ERROR Libref X is not assigned.`) was missed on any line carrying markup;
- the notable notes - uninitialised variable, invalid data, a format too small - were not shown anywhere;
- clicking an issue searched the rendered text for its message, so a log that repeats a message (two `ERROR 22-322:` lines is ordinary) always jumped to the first of them.
`parseLog` from `@sasjs/utils` replaces all of it, the same function the VS Code extension and the SASjs server Studio use, so one log now reads identically in every host. Each request keeps the issues the parser reported, grouped by severity, and the tree gains a Notes list beside Errors and Warnings. The click scrolls to `issue.line` and flashes from that line to `issue.endLine`, and the lists no longer render at zero.
The modal is SAS Logs now, as are the menu entry, the deploy button and the abort modal's button; the request history is unchanged underneath.
`formatTimestamp` decided on `moment().format()`, which is 'Invalid date' - truthy - for anything it cannot parse, so its fallback could never fire. It asks `isValid()` now.
The warning flash was yellow where the warning text is ochre; both are ochre now, and the notes flash blue.
`@sasjs/utils` goes to 4.3.1, which is where the parser lives.
The new empty-grid spec built a grid from `data: []` alone, and Handsontable
reports zero columns when the data is empty and no `columns` setting defines
the schema - so buildGridPdfInput() correctly returned null and the spec failed
in CI while passing against the stub harness. The real grids declare their
columns, so the fixture does now too.
Two findings from the second review.
A row taller than the printable area was drawn straight through the footer and
off the page, losing the tail of the cell unannounced - MAX_COL_CHARS bounds a
column, not a value, so one long text field wraps to more lines than a page
holds. Rows are now clamped to the tallest row that fits the page, with ' ...'
marking the cut, and the height budget accounts for both margins so an
exactly-fitting row stays on the page it started on instead of leaving the
first page with headers only.
exportGridPdf() also returned silently for a grid with no rows, although the
writer renders an empty table as a valid header-only page. The input is now
built by buildGridPdfInput(), which returns null only when there are no
columns, so a rowless grid downloads a page saying "0 rows x N columns".
Tests: the tall-row spec asserts the marker, the single page and that no text
start falls outside the printable band; the empty-grid spec asserts the input
is built rather than skipped.
Two findings from review.
First, a regression in the CSV/Excel path: getExportRange() defaults `left`
to skipLeadingCols, so comparing it against skipLeadingCols made every
no-selection export look like a whole-grid export and the plugin put the
leading housekeeping column back in. The whole-grid test now compares the left
edge against visual column 0, so a skipped column stays out.
Second, the PDF row cap was applied after collectGridTable() had already read
every cell, so a very large grid still froze before the cap took effect.
collectGridTable() now takes a maxRows limit that clamps the row loop before
any cell is read, and exportGridPdf() still reports the true total in the
subtitle.
Both are covered: a spec asserts the CSV export receives range
[0, skipLeadingCols, lastRow, lastCol], and one asserts the cap stops cell
reads rather than collecting and slicing.
CI caught this on the real grid specs: getPlugin('hiddenColumns') returns the
plugin instance even when it is switched off, and isHidden() then dereferences
a map that enablePlugin() never created, so any grid without a hiddenColumns
setting threw instead of exporting.
Ask the plugin's enabled flag first and treat a failure as "visible" - an
export is better off with a column the user could not see than with an error
dialog. The spec adds a regression case for exactly the CI failure, and its
afterEach no longer destroys an already-destroyed instance when a test creates
no grid.
Adds a PDF entry to the right-click Export submenu, next to CSV and Excel,
so the grid as displayed can leave the browser as a printable document.
Implemented directly against PDF 1.4 rather than adding a PDF library: the
output is a text table, which needs no embedded fonts, no images and no
compression. The base-14 Courier font is used because its glyphs are exactly
600/1000 em wide, so column widths are arithmetic on character counts instead
of a font-metrics table - a 1200 x 12 grid is laid out in ~60 ms.
Layout: A4 landscape, bold header band repeated on every page with "Page n of
m" footers, numeric columns right-aligned, values word-wrapped inside their
column, and long tables paginated. Wide tables are fitted by shrinking the
body font first (7pt down to 5pt, so a 20-column table keeps full headers at
6pt) rather than squeezing columns; only at 5pt are columns scaled, and they
stop at 4 characters - past that trailing columns are dropped and the first
page says how many were kept, so nothing is omitted silently.
Encoding is WinAnsi, the widest set the base-14 fonts cover without embedding;
characters outside cp1252 print as '?'. A typeset, branded report is a
different feature and belongs on the server, where SAS formats and DC metadata
are available.
The export keeps the existing "what you see" contract: current selection (when
it spans more than one cell), otherwise the whole grid, hidden columns
omitted, leading housekeeping columns skipped. The range calculation that CSV
and Excel already used is now a shared getExportRange(), so the three formats
cannot drift apart. Rows are capped at 5000 and the subtitle says so.
Tests: pdf-export.util.spec.ts covers document structure (xref offsets, page
count, footer numbering), string escaping, WinAnsi mapping and fallback,
wrapping, alignment, multi-line values, pagination, font fitting and the
page-width invariant; hot-export.util.spec.ts covers grid extraction against a
real Handsontable instance.
A MIN_TOTAL_MB typo reached the integer test under set -e and aborted with "integer expression expected"; it now prints usage and exits 2. A kernel without MemAvailable no longer reports 0 MB available in the summary or the warning, so the oversubscription warning stays meaningful.
release.yaml (all three jobs), sites.yaml and release-assets.yaml run the same cold production build that the review jobs do, so they get the same pre-install check.
The header now says what the check rules out (an undersized box) and what it does not (two jobs sharing one box), so the MemAvailable warning is not read as a gate.
An undersized runner does not fail honestly: the kernel OOM killer takes
the build, so the job reports exitcode 137 at ng build, or Karma
disconnects mid-suite after twenty minutes. Both read as a flaky test
rather than as a box that cannot hold the job, and both cost a whole CI
slot.
The check runs first in the three client jobs (Build-and-ng-test,
Build-and-test-development, lighthouse), prints total and available RAM,
vCPU count and any cgroup limit, and exits 1 when the effective size is
under 8 GB - the supported minimum. A cold production build measures
3.8 GB peak on its own, so 8 GB leaves roughly 2x headroom.
An uploaded file's preview is validated in place, so the cells that fail
a rule carry Handsontable's invalid cell meta. Discarding the file
restores the loaded data, but cell meta is keyed by row/column and
survives that restore - without a re-validation the grid keeps painting
cells that are no longer invalid.
cancelEdit re-runs the validation (hot.validateCells()) after restoring
the data source, which is what clears them; this test drives the flow and
pins it: import a file that fails validation, discard it, then assert the
row count is back, no .htInvalid cell remains and the page is read-only
again. The manual-edit path (test 25) keeps the parsed values by design.
Refs #122
The multi-load page rendered the loading stage only inside a branch guarded by
`parsedDatasets.length`, and that collection is empty for the whole read and
sheet-scan phase - so two of the four stage labels ("Reading the file",
"Searching for the data in the file") were assigned but never reached the DOM,
and a large file sat on screen with no feedback at all.
The stage is now shown next to the file name and size from the moment a file is
attached, and again in the content area while the sheets are being parsed. The
labels cover every phase the user waits through: reading the file, searching
for the data in the file, file ready to load, fetching the dataset details (one
getdata call per dataset - the longest wait), extracting the data, and
extracting the data finished. A spinner accompanies the stages that have work
running; the idle stages and the finished state show the label alone.
Two corrections fall out of that: the stage only moves to `extracted` once
every sheet has been parsed rather than when the first one finishes, and the
read's continuation only advances the stage if the read is still the phase in
progress (the sheet scan runs in parallel and can finish first).
excel-multi-load.cy.ts gains test 4, which asserts the file size, the ready
stage, the fetching stage (the getdata calls are held open so it is on screen
long enough to assert) and the handover to the "select a dataset" hint.
Closes#121
Enables noMacroIfInOpenCode and promotes it to "error". Open-code %if is a SAS
9.4M5 feature, so the 19 sites it found in nine files were hiding a version
requirement the codebase does not document (the requirements page says 9.4M3).
Each is rewritten so the construct no longer appears in open code:
- Six blocks become a small macro called unconditionally, which is where the
conditional then lives: drop_cas_tables (demodata), add_derived_cols
(getdata), build_sasdata4 (getdata test 4), drop_derived (stagedata),
build_staged (getrawdata), and one per option in the 20261003 migration.
- Four blocks in the two makedata STPs get the same treatment: check_outfmt,
check_keepme, emit_response and self_destruct. Their `call symputx` writes
are now explicit about the global symbol table, because a %let or a plain
symput inside a wrapper would be local to it.
- Two guards in stagedata and getrawdata are removed rather than wrapped: the
SQL only selects into a list that is already %let blank, and with no derived
variables it returns no rows, so the guard was doing nothing. The trailing
drop keeps a conditional, now in a macro and tested with %trim, since a
zero-row select can leave a single blank in the variable.
- validatefilter's two %let branches become one data step with call symputx,
which writes to the global table the later %mp_filterstore reads.
Verified: `npm run lint:sas` clean with the rule as an error; `sasjs compile`
succeeds; on sas.4gl.io the full suite passes 64/64, including the derived
variable tests in getdata test 4, stagedata tests 3 and 4 and validatefilter,
which are the paths these wrappers touch.
The tree is clean under all of them, so the remaining warn-level rules can be
promoted without a single code change: hasDoxygenHeader, hasMacroNameInMend,
hasMacroParentheses, noNestedMacros, strictMacroDefinition, noSpacesInFileNames,
lowerCaseFileNames, noTabIndentation, noTrailingSpaces, noGremlins and
noNestedCommentsInHeader.
With that, every rule @sasjs/lint 4.9.0 can raise is an error - only
noEncodedPasswords was an error by default - so any new violation fails
`npm run lint:sas` and the CI step that runs it, instead of printing a warning
that leaves the exit code at 0.
Checked: a trailing space appended to a file reports
"Error Line contains trailing spaces" and a non-zero exit, where it used to be
a Warning.
sas/.sasjslint takes maxLineLength from 180 to 80 - the linter's own default -
and promotes it to "error".
53 of the 258 .sas files already hold a line longer than 80. Rewrapping them
would be a large, reviewable-only-in-theory diff, so each keeps its current
longest line as an exact ceiling via a per-file override, e.g.
@sasjslint {"maxLineLength": 95} in services/validations/tables_all.sas. A
file with no override is held to 80, and an overridden file cannot grow past
the length it has today: the ceiling is the file's longest line, not a round
number.
Both directions were checked: a 101 character line added to an
un-overridden file fails with "Line exceeds maximum length by 21 characters",
and a 114 character line added to macros/mpe_execute.sas (ceiling 101) fails
by 13. `npm run lint:sas` is clean with the setting in force.
Rung 6 of the #376 ratchet: `"lineEndings": "lf"` is enabled and promoted to
"error". Its 37 findings are cleared and nothing but line endings changed.
- 34 files ended without a newline; each now ends with exactly one LF.
- 3 files were stored with CRLF (services/public/getgroups.sas and the two
sas9 lineage services, getmetacols and getmetatables), against the repo's
own `* text=auto eol=lf` in .gitattributes. They are converted to LF.
Checked mechanically that this is whitespace only: for every changed file,
HEAD's content and the new content are identical once CRLF is folded to LF
and the trailing newline is stripped.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- On sas.4gl.io, against a scratch appLoc seeded with makedata:
"Tests that pass: 64/64 (100%)".
The rule flags the %if blocks service bodies run in open code, which need
SAS 9.4M5. The documented minimum is 9.4M3, so enabling it would mean
raising the floor or wrapping every service body in a macro - neither is
a lint change. Written down so the next agent does not enable it by
mistake.
Rung 5 of the #376 ratchet. Both rules are off by default; enabling them
surfaced 10 findings, all now cleared, and both are promoted to "error".
- Nine `* text;` comment statements become block comments: three disabled
debug lines (a putlog in bitemporal_dataloader, an alternative type
assignment in mpe_loader, a put in demodata) and six explanatory
comments in the two sas9 usernav services.
- loadfile.sas assigned `libname approve "&dir"` and never used the
libref; stagedata.sas and restore.sas assign the same libref because
they write approve.jsdset, loadfile writes through a fileref instead.
The statement is removed.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- On sas.4gl.io, against a scratch appLoc seeded with makedata:
"Tests that pass: 64/64 (100%)".
Rung 4 of the #376 ratchet. It clears the last 5 findings and promotes the
three rules that produced them to "error": minMacroNameLength,
lowerCaseMacroNames and indentationMultiple. `sasjs lint` now reports no
warnings and no errors across the project.
- viewdata.sas wraps its service body in a helper macro named `x`; it
becomes viewdata_body. The wrapper is what keeps the %global and the
%return in one macro scope.
- usergroupsbymember.sas (sas9) builds work.iwant inside a helper macro
named `x`; it becomes build_iwant.
- refreshtablelineage.sas (sas9) defines `getTables` with uppercase; it
becomes `gettables`, at the definition, the %mend and its seven calls.
- getstagetable.test.sas: one comment line carried three leading spaces
where the file uses two.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0, "All matched files
use @sasjs/lint code style!".
- `sasjs compile`: succeeds.
- Deployed to sas.4gl.io and ran the SAS suite there against a scratch
appLoc (/Public/app/dc-lint-ratchet) seeded with makedata: "Tests that
pass: 64/64 (100%)". The viewdata and usergroupsbymember suites are in
that run, so the two renames are covered by passing tests.
Rung 3 of the #376 ratchet: `sas/.sasjslint` promotes noBannedStrings to
"error". The rule flags the literal words "error " / "warning " anywhere in
a file, comments included, because they turn up in a grep of the SAS log
and bury the real messages. Its 13 findings are cleared:
- Nine are prose and are reworded, so the sentence both reads normally and
stops matching: a truncation that aborts a build, a syntax fault, an
abort condition, "only records the failure in work.mp_abort_errds" (four
copies of the mp_abort-inside-%include comment), a test description, a
hook comment.
- Two are the text the user sees. loadfile.sas reports
"(mf_abort.error file found)" - the marker file carries the word in its
name - and buildtermsas9.sas reports "Some final error in &_program";
both are reworded to say the same thing without the literal, since the
text sits in a macro argument where the %str(WARN)ING split does not
survive the quoting.
- Three keep the word because the word is the content: the doxygen
@warning tags in macros/bitemporal_dataloader.sas and the two SAS 9
admin STPs (makedata, makelib). Each carries
@sasjslint {"noBannedStrings": false}, as
targets/server/services_server/admin/makedata.sas already did.
VERIFICATION
- `npm run lint:sas`: 5 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- Deployed to sas.4gl.io and ran the SAS suite there:
`sasjs test` reports "Tests that pass: 64/64 (100%)". The run used a
scratch appLoc (/Public/app/dc-lint-ratchet) seeded with makedata, so
the shared /Public/app/dc deployment was left alone; the scratch target
was removed from sasjsconfig.json before this commit.
5 warnings remain: macro name length (2) and case (2),
indentationMultiple (1) - the last rung.
Rung 2 of the #376 ratchet: `sas/.sasjslint` promotes noUndeclaredMacros to
"error". Its 25 findings on release/7.22 are cleared, and two of them were
real faults rather than missing prose.
WHAT WAS BROKEN
Comparing the compiled program against the macros it calls (compile, then
list every project macro each generated service calls but does not define)
found three services calling macros that were absent from their own
compiled program:
- services/validations/tables_all.sas calls %mf_getuniquename() on its
third line and never declared it.
- services/validations/mpe_tables.dsn.sas calls %mf_getuniquename() and
%dc_assignlib() and declared neither.
- services/hooks/mpe_tables_postedit.sas calls %dc_assignlib(), which is
supplied by the service that includes the hook (see below).
The two validation services are configured as rule_value entries in
MPE_TABLES (mpe_makedata.sas and the v3.12/v3.13 migrations), so the calls
run when a user reviews or filters a library - each would have raised
"Apparent invocation of macro ... not resolved". Both now declare what
they call, and the definitions are present in the compiled program.
WHAT CHANGED
- 14 files declare the macros they call but had been relying on another
declaration to bring in: mf_getquotedstr, mf_nobs, mf_getuser (twice),
mf_getapploc, mf_getattrn, mp_abort (six services), mf_getuniquename
(twice), dc_assignlib, bitemporal_closeouts. These were defined in the
compiled program already; the declarations make the dependency visible
and the two validation services correct.
- 6 hook scripts get a file-local @sasjslint {"ignoredMacros": [...]}
(mp_abort, plus dc_assignlib in mpe_tables_postedit) instead of a macro
list. Hooks are included at run time by the service that owns them, so
the macros are already in that session, and these files are copied and
edited by end users - they should not carry a declaration section to
maintain. Recorded in AGENTS.md.
- bitemporal_dataloader.test.2.sas loses its test 30 block: it is gated
with `and 0` ("skip until have a db") and its only setup line calls
%assign_direct_lib, a macro that exists nowhere in this project or in
@sasjs/core. Enabling it would fail on the first line.
VERIFICATION
- `npm run lint:sas`: 18 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds; the called-but-undefined list drops from
three services to the one hook exception; the only definition-count
changes are additions.
- 18 warnings remain for the next rungs: noBannedStrings (13), macro name
length (2) and case (2), indentationMultiple (1).
sas/.sasjslint now promotes noUnusedMacros to "error", so the rule fails
the build rather than printing a warning. The 54 findings it reported are
gone:
- 46 header entries nothing calls are dropped. Each removal was checked
against the compiled artefacts rather than the linter alone: the
project was compiled before and after, and the two sasjsbuild trees
compared. 37 macro definitions disappear from the generated programs -
every one of them either a duplicate of a definition the file still
reaches through another declared macro, or a macro no generated program
calls. No macro that a generated program calls lost its definition.
- 8 entries are kept, in five files that carry
@sasjslint {"noUnusedMacros": false} with the reason written above it:
the three serviceInit programs and serviceterm.sas declare mpeinit and
mpeterm without calling them, because that declaration is how a service
body reaches those macros at all; loadfile.sas declares mp_binarycopy
for the fallback that is commented out on SAS 9.4 and later.
- Four headers are left with no macros to declare, so their empty
<h4> SAS Macros </h4> section is removed rather than kept as a stub.
@sasjs/cli is bumped 6.2.0 -> 6.5.0 in sas/package.json. The lint engine
ships inside the CLI and is not a direct dependency, so the pinned CLI is
what fixes the rule set: 6.2.0 resolves @sasjs/lint 4.3.0, 6.5.0 pins
4.9.0 exactly. Both versions were compiled against this tree and the
sasjsbuild output is byte-identical, so the bump changes the lint engine
and nothing else.
`sasjs lint` is also now reachable as `npm run lint:sas`, and a CI step in
build.yaml runs it. Until now nothing ran SAS lint automatically, and a
bare `sasjs` resolves whatever CLI is installed globally - a different
lint version reports a different rule set, which is why the counts in
issue #376 (taken with 6.2.5) do not match the pinned 6.2.0.
43 warnings remain under the rules not yet promoted: noUndeclaredMacros
(25), noBannedStrings (13), macro name length and case (4),
indentationMultiple (1). They are the next rungs of the ratchet.
The pre-commit Prettier gate could not run: it needs client/node_modules,
which this worktree does not install, and the fix it attempted touched
three client files that this change does not modify (reverted). No client
file is in this commit.
`cy.get` inside `.within(<tree node>)` only searches that node, and the
metadata accordion renders in the content area - so `.clr-accordion-header-button`
was never found however long it retried. The spec queries the accordion at
document level, selects the first metadata type by its class rather than by
node index, and clicks the header button directly (Clarity puts
`clr-accordion-header-button` on the button itself).
The assertions are on the app's own markers: the single open panel
(`clr-accordion-panel-open`, `aria-expanded`), the association tree inside the
panel, and the attribute grid that `metanav/metadetails` fills.
metanav moves from EXCLUDED to SPECS in `.gitea/scripts/cypress-specs.sh`.
Closes#395.
The case took the id from the stage route and looked for a download button
carrying it. For an upload the route does not hold a table id at all - it holds
a load reference, the CI failure naming DC20261031-201657.470_601010_4449 - so
the button it looked for could never exist, however long it waited or however
many times the screen was reloaded. That is why the row "never appeared" while
the approval was in fact there.
The row is now found the way hook-programs finds its own: by the table it names.
The id the download is named after is read from the button, which is the only
place that knows it.
Verified against the mock estate locally, with the licence the spec now plants:
7 passing, exit 0.
The estate ships no licence key, so it runs on the free tier, and the free tier
disables file upload - which is exactly what the audit case needs. Whether it
could run depended on another spec having planted a key earlier in the run: the
key persists in the estate config, so the case passed or failed on spec order.
loginAndUpdateValidKey(true) plants one here. csv-limited, which asserts the
free-tier modal, runs first in cypress-specs.sh and is unaffected; download runs
last, so nothing downstream inherits the key.
Found by running the spec against the mock estate locally: the upload input
never rendered, the modal read "Locked Feature (File Upload) ... site id (123)",
and the run went 7/7 green once the key was planted.
mpe_abortmessage existed to assemble the message mpe_execute aborts with - the
program, the state and the child message - inside a data step, so that a percent
sign, an ampersand or a quote in the child message was not macro resolved on the
way into a symbol. mpe_execute already runs a data step over the status row for
state and errmsg, so the text is built there now and the helper, plus the %let
fallback that duplicated it, is gone.
What the helper did is kept: the values come from the status row through
symget/symputx, never through %let, and a message already at the 32767 limit is
cut to the room the prefix leaves.
The comma and percent cases its tests covered now live in core's mx_execute
test, where the message is captured from a real child; what stays here is the
assembly DC owns - the prefix, the state and the child message from the status
row.
The audit-file case uploads a fixture and then looks for the approval it should
create. An upload identical to what the table already holds produces no
changes, and a changeset with no changes leaves nothing waiting for approval -
so the case finds no row. A retry lands in exactly that state: the first
attempt's rows are already in the table, so the second upload is a no-op and the
failure looks like a missing approval rather than a duplicate upload.
The fixture is now copied per attempt with its primary keys renumbered from a
random base, so the upload is always an insert. The copy is removed in
afterEach - deleteFolder cannot be used there because the source fixture lives
in the same directory.
Core 7.0.3 fills outstatus.errmsg on both platforms - the SASJS branch since
7.0.2, the Viya branch from the job result as of 7.0.3 - so a canceled call now
always carries the reason it was canceled.
That removes the last platform difference from the assertion: while Viya wrote a
blank errmsg, an expected message could only be checked on one platform and the
state alone had to stand in on the other. A comparison against a message that
was never recorded now fails instead of passing silently, which is the point of
asking for it.
The macro decided an aborted call by scanning the child's response text for a
sasjsAbort object and then recovering the message from that text - anchors for
the separator, gating on whether the payload was escaped, cuts for the log
extract, and a capture width that had to be widened more than once. Every one
of those steps existed to undo the escaping in the response.
Core now publishes the abort reason in outstatus.errmsg for both the SASJS and
Viya branches, with the whole message and no truncation, so the macro reads the
row instead: STATE says whether the child was canceled, ERRMSG is the reason.
The cut at the appended log extract is the only thing left, because that text is
part of the message the child raised; it is stripped as characters, as an
escaped pair or as a doubled pair, whichever the producer wrote.
The macro therefore takes outstatus= rather than outref=, and the call sites in
the service tests now pass the status dataset they already asked %mpe_execute
for. stagedata.test.4's two aborting calls had no status row at all and now
have one.
A platform that reports the state but no message - SAS Viya until core fills
errmsg from the job result - is judged on the state alone, so the same tests run
on both platforms.
@mpe_assert_abort.test covers the twelve cases directly against status rows
instead of building response files, including the doubled separator the estate
actually writes, a message past the old 256 character width, quotes and
backslashes.
Requires @sasjs/core 7.0.2, which carries the SASJS half of this.
The upload and submit now work; the failure moved to the approve grid, which is
loaded when the screen opens and does not yet contain the row just submitted.
The screen is reloaded up to five times, two seconds apart, until the row with
this test's id is present.
The mock getddl aborts when ds is missing, as production does (getddl.sas:43-45),
instead of quietly returning a library-level DDL - a client that dropped ds would
otherwise stay green against the mocks.
The comment above the case no longer claims a leftover row cannot matter: the
grid paginates, so a row pushed past the first page would not be found.
The upload does parse now - CI showed the info modal appearing - but the helper
treated it as an abort because the info modal and the abort modal are the same
component (.clr-abort-modal). The title is what separates them: a good upload
raises "Table Found", so that is asserted and anything else fails with its own
text instead of being clicked through.
After Submit, the stage route is already current, so waiting on the URL proved
nothing; the editor leaving preview mode is the signal that the submit was
accepted. Also trims the comment that narrated the old route.
The first case failed in CI with "Expected to find element:
input[type=file]#file-upload", because the editor renders that input inside its
upload modal - the Upload button in the button bar opens it. Clicking the button
by its label rather than by position, as excel.cy.ts does.
The viewer download cases (2-7) passed, so the rest of the spec is sound.
Follow-ups from the review of #414.
The credential guard now checks each staged .npmrc rather than only the client
one: the root and sas/ files are tracked policy as well, so a token pasted into
either was equally one `git add -A` away. Verified by staging a token in
sas/.npmrc - the hook exits 1.
$RUNNER_TEMP/npmrc is removed at the end of each job that writes it, with
`if: always()` so a failed job cleans up too.
The root `install` script is gone: the root .npmrc sets ignore-scripts, so npm
never ran it. CONTRIBUTING already spells out the per-project commands, and is
now the single source.
The release workflows build the published client from the same tree, so they run
`check:archiver` like build.yaml does.
readdir-glob reaches minimatch twice: the directory walk, and the pattern/ignore
matching behind archiver's glob(). The second is where a semantic change shows
up as silently skipped files rather than an error, so the check archives a tree
by pattern with an ignore and asserts what it kept: a.txt in, sub/** out.
client/.npmrc is tracked now, so a token left in it from the old pattern is one
`git add -A` from being committed. Prose in CONTRIBUTING is not a control: the
pre-commit hook fails when the staged file holds _authToken, _auth or _password.
`npm run check:archiver` runs the directory-walk guard, so it is reproducible
before a push instead of only in CI, and the CI step calls the same script.
CONTRIBUTING now spells out the per-project installs, since the root .npmrc has
ignore-scripts=true and the composite install script therefore never runs.
unzipper was only there to read the test archive back. archiver emits an 'entry'
event per file it adds, and those names are exactly what the walk produced, so
the guard asserts on them and needs no reader. archiver stays: it is the package
whose directory() walk reaches readdir-glob, and the import has to be pinned to
the version the override installs.
Four write steps interpolated ${{ secrets.NPMRC}} straight into the shell source,
so a quote or $( in the value would break the step or run. They use the env form
the same workflows already use elsewhere.
CONTRIBUTING.md now says what the tracked client/.npmrc means for a developer:
run `npm run postinstall` after installing, `npx cypress install` for the suite,
and keep registry credentials in the user config rather than in the tracked file.
The release workflow runs cypress against the built app, and its install steps
now inherit ignore-scripts from the tracked client/.npmrc - so the binary has to
be installed explicitly there too, as build.yaml does.
unzipper is pinned to 0.12.5 so the exceljs override block no longer mixes exact
pins with a range.
client/.npmrc is tracked now, and every workflow appended the NPMRC secret to it
inside the workspace - one `git add -A` from committing a live token, and
semantic-release runs in jobs that have already written it. The secret goes to
$RUNNER_TEMP/npmrc and NPM_CONFIG_USERCONFIG points the installs at it, so the
project file stays policy-only.
The archiver guard used a fixed path in the shared tmpdir and had no handler on
the write stream: two jobs on one runner collided on the path, and a write
failure surfaced as an uncaught throw. Each run now gets its own directory, both
streams report, and the tree is removed afterwards.
The exceljs override pins archiver to 8.0.0 rather than ^8.0.0, so the version
the guard was validated against cannot drift.
The committed client/.npmrc turns dependency scripts off everywhere, not just in
build.yaml - and every one of these workflows builds the frontend, which imports
src/environments/version.ts. The lighthouse job failed exactly there:
Error: Cannot find module '../../environments/version'
The generator now runs after each client install: build (with cypress install),
lighthouse, sites, docs, release-assets and the three release installs.
Three things the last run exposed.
Every workflow that writes the NPMRC secret used `>`, which now overwrites a
tracked policy file - so the release, lighthouse and sites jobs would install
with hooks enabled and ranges unpinned, the opposite of what this PR adds. They
append.
Cypress downloads its binary in a dependency install hook, so ignore-scripts
leaves it out: "The cypress npm package is installed, but the Cypress binary is
missing". Both client installs now run `npx cypress install` after the
generator.
The archiver guard imports archiver and unzipper, which were only present as
overrides under exceljs; they are declared devDependencies at their resolved
versions so the script owns what it imports. Both remain production packages
because exceljs still requires them.
Dependency install hooks now stay off in client/, as they already are at the root
and in sas/: client/.npmrc is committed with ignore-scripts and save-exact, the
root install script runs `npm run postinstall` after installing, and both CI
installs pass --ignore-scripts and then run the generator by hand. The client's
own postinstall is what creates src/environments/version.ts and the hot icons, so
it has to run - it is third-party hooks that do not.
CI writes client/.npmrc from the NPMRC secret; the second job now appends rather
than overwrites so the tracked policy survives (the first already appended).
The archiver guard reads the archive back instead of checking its size: a matcher
that takes the top-level file and misses the nested directory clears any size
threshold, and that is exactly the failure it exists to catch. It now asserts
a.txt and sub/b.txt are both present.
archiver 8 is ESM-only, so require('archiver') resolves through require(esm) and
that exists from Node 22.12 - a supported Node 18/20 would fail the check for a
reason that has nothing to do with the minimatch override. The script is .mjs and
imports ZipArchive.
The cut already tells the two payload shapes apart - the doubled separator means
mp_abort escaped the message, the single form means it did not - so the unescape
is gated on the same decision. Unescaping a payload that was never escaped
collapsed a genuine pair of backslashes, which is a UNC path or a regex in a hook
message, and the comparison then ran against text the hook never raised.
The over-long message cut clamps the room to zero first: a prefix long enough to
fill the buffer gave SUBSTR a negative third argument, which errors the step and
cancels the program the helper exists to explain.
programvar=/statevar=/msgvar= names that do not resolve now say so with a
WARNING, instead of reaching the abort text as a blank.
Two cases: an unescaped payload keeping its double backslashes, and a message at
the 32767 limit being cut rather than overflowing. The trailing-backslash case
now uses the fully escaped form, which is what the producer emits.
The client's postinstall generates src/environments/version.ts (and the hot
icons), which the app imports - with --ignore-scripts the build fails with
"Can't resolve '../../environments/version'". My local check missed it because
version.ts was already on disk from an earlier install.
So the client keeps its install scripts; the archiver guard stays, which is what
that commit was for.
The minimatch override forces a major above readdir-glob's own range, and
archiver's directory() is the only runtime caller of that path. scripts/
check-archiver.cjs archives a small tree and fails if the result is too small to
hold it, so the override is guarded by a run rather than by the install
succeeding.
client/.npmrc is gitignored - it is where a developer's registry credentials
live - so the repo-wide ignore-scripts policy is passed on the client install
steps instead. Verified locally: npm ci --ignore-scripts, the archiver check and
ng build all succeed.
Cutting on the marker plus a newline still truncated a message that ends a line
with "Log Extract:", and an abort whose message merely prefixed the expected one
could pass. The appended extract always emits '\n\nLog Extract:\n', so the anchor
is that whole string (doubled form first, single as fallback): a message cannot
reproduce it by accident, and nothing is cut when neither form is present, so an
unexpected layout fails loudly instead of shortening the text.
The tail strip goes with it - the leading newlines are part of the anchor - and
the test for the doubled separator now writes the fully escaped form the estate
actually emits.
mpe_abortmessage cuts an over-long message to the room left after the prefix
instead of losing its tail to the 32767 limit, and mpe_execute's fallback skips
the trailing blank when the child's message is blank, matching the helper.
The %let could never be used: it ran immediately before an unconditional call
to the helper, which overwrote the global, and on a build without the helper the
unresolved macro call does not fall through to it - the parameter text lands in
open code and the step breaks before the abort fires.
%sysmacexist now chooses between them: the helper when it is on the autocall
path, the %let when it is not. The %global stays, so both branches write the
symbol that %superq reads.
Test datasets and filerefs are numbered to match their test, so a failing
assertion can be traced back by name.
Same reasoning as the minimatch pin: two ranges in one override object, one of
them caret, means the block can still float. Exact 5.0.12, which is what the
lockfile already resolves.
An override exists to stop a later install moving the package it pins, so a
caret range defeats it: the lockfile records 11.0.3 and the range would allow
11.0.4 to arrive without review. Exact version, and the lockfile is unchanged -
it already resolved 11.0.3.
Three of the four review points; the fourth is a producer problem, explained in
the reply.
The marker is only a separator when a newline follows it, so the cut anchors on
marker+newline (doubled form first, single as fallback for a response assembled
without the escape) and takes the FIRST match. A message that mentions the
marker is not followed by a newline and survives; a log line that contains the
marker can no longer move the cut past the separator and leave extract residue.
New case: an abort whose log mentions the marker.
mpe_abortmessage reads program and state into their own variables before
building the prefix: symget() nested in cat() is typed $200 by the data step, so
a long endpoint name was truncated before the assignment.
The header no longer claims byte-for-byte retention: symputx strips leading and
trailing blanks.
Not changed: decoding \n in the message. mp_abort escapes the append as
cats(msg,'\n\nLog Extract:\n',logmsg) and then doubles every backslash, so the
newline arrives as \\n\\n - un-doubling leaves the two characters \ and n, and
the only way to turn those into a newline is also the way a literal backslash-n
in the message becomes one. That is a producer-side decision, not a consumer one.
With follow-redirects moved, the client audit still failed on minimatch
<11.0.1 (globparts amplification), reached through
exceljs -> archiver -> readdir-glob, which asks for ^10.2.2.
readdir-glob already carries an override block here for brace-expansion, so the
fix follows that shape rather than a breaking exceljs change: minimatch
^11.0.1 nested under readdir-glob. package-lock gains 23 lines, the override
line is the only package.json change.
Verified: npm audit --omit=dev exits 0 in client; ng build succeeds; an
exceljs xlsx write succeeds; and archiver ZipArchive.directory() - the caller
that walks the tree through readdir-glob - produces a valid zip.
The audit runs in root, sas and client, and sas carries its own lockfile - the
same 1.16.1 was resolving there, which is why the first attempt left CI red.
Same one-package update, no pin change.
`npm audit --omit=dev` in sas is now clean.
The audit gate fails on any runtime advisory, and this one arrived upstream:
follow-redirects 1.16.0-1.16.1 discards cross-realm sensitiveHeaders arrays and
lets custom credentials follow redirects. 1.16.2 is outside that range and inside
the ranges the tree already asks for, so this is a lockfile-only update - no pin
in package.json changes.
Regenerated with `npm update follow-redirects --package-lock-only`, which moves
that one package rather than re-resolving every floating range (an
`npm audit fix` run also bumped the whole @angular-devkit family, which is not
part of this fix).
`npm audit --omit=dev` in client: the high severity entry is gone. Four moderate
advisories remain, all through exceljs -> archiver -> readdir-glob -> minimatch,
and they need a breaking exceljs change; they are pre-existing and not touched
here.
Two of the three review points.
The fallback %let now carries %superq(msg) as well, so a build without the helper
on the autocall path still reports the child's own message rather than only the
program and state. The whole value is masked and consumed through %superq at the
abort, so the comma hazard does not come back.
A case with the doubled separator - the form the estate actually emits, six
characters once the backslashes are doubled - now covers the branch that all the
other cases, written single-escaped, could not reach.
The quote case is not fixed; see the reply.
Two comment blocks sat back to back above the same call - the click handler's
and the anchored label's. They describe one selector, so they read as one note.
Three review points.
The fallback %let now follows %global _mpe_abortmsg. Without it the %let created
a local of that name in mpe_execute while the helper wrote the global, so on the
first failed call in a session the fallback shadowed the child's message - the
symptom this branch set out to remove.
mpe_abortmessage takes names for all three inputs (programvar/statevar/msgvar),
so a comma in an endpoint path cannot split the parameter list either. The test
that passes a path containing a comma now covers that, alongside the message.
mpe_assert_abort unescapes outside the marker block: a response that carries no
appended extract is still escaped, and leaving the unescape inside the block made
the comparison fail on text that was correct.
Full suite 64/64, mpe_execute and mpe_assert_abort PASS, sasjs cbd first.
Three review points.
The table id was sliced out of the stage URL without trimming a trailing slash,
so a route rendered as /stage/<id>/ produced an id that never matched the button.
Trailing slashes are removed before use.
The options menu entry is matched on the whole label, /^\s*Download\s*$/, because
an entry like "Download web query" would make cy.contains ambiguous and it fails
outright rather than picking one.
The getddl guard covers both hosts: libref is read behind a typeof too, so a call
with either variable out of scope falls back instead of throwing.
Two review points.
The strip looped over trailing \n and \r escapes, so a message that itself ended
in an escaped newline lost that escape along with the separator and the exact
comparison failed. It now removes one fixed tail: the six characters the
separator becomes once mp_abort doubles the backslashes, or four if the extract
is not doubled. Anything the message itself ended with is left for the unescape.
mpe_execute sets _mpe_abortmsg before calling mpe_abortmessage, so a build where
the helper is not on the autocall path still aborts with the program and state
rather than an empty message.
Full suite 64/64, mpe_assert_abort PASS, sasjs cbd before the run.
Three review points.
attachExcelFile read the body immediately after attachFile, so a slow upload made
abortModal false and skipped the abort branch - the failure then arrived later as
the timeout the branch exists to avoid. It now waits for either the confirmation
button or the abort modal title, with longerCommandTimeout, and branches on which
one matched; the confirm click uses the same timeout.
getddl guards ds with typeof: the truthiness fallback only covers a ds that is
bound and empty, while a missing ds in scope throws before the fallback runs.
The submitted row test 1 leaves behind is now documented in both places. The CI job
deletes the appLoc and re-runs makedata before the specs, so it does not accumulate
between runs; a hand-run against a reused estate leaves one pending row per run, and
the selector is anchored on the id the test created, so a stale row changes nothing.
Three things, one of them a bug the review question uncovered.
- msgvar is passed explicitly at the call site, so renaming the message variable
fails loudly instead of quietly producing an abort text with no child message.
- program and state are read with symget rather than interpolated into the data
step, so a % or & in an endpoint name can no longer be macro-resolved, and a
quote cannot break the literal.
- _prefix is trimmed before the message is appended. Declared as a character
variable it arrives padded to length, so the child's message was landing past
the buffer end and disappearing - the wrapper text was right and the message
was simply not there. That showed up as the comma and blank cases failing while
the "names the program state" case passed.
The abort branch asserted the modal title did not include 'Abort Message', so
any other title let the run continue with the abort modal open and the next step
died on a timeout that never mentioned the upload. It now throws with the title,
which points at the upload as the cause.
Three review points.
attachExcelFile read '.clr-abort-modal .modal-title' unconditionally, so an
upload that raised no abort modal failed on the lookup, and the abort branch
then left the modal open for the next command to click through. It asks the body
first: no abort modal means confirm the upload; an abort modal means assert its
title is not an Abort Message and stop there.
openDownloadModal clicked the deepest element holding the text. The click
handler sits on the div[clrDropdownItem], so the item is selected by that
attribute and its text.
The spec list comment claimed download leaves nothing for later specs to trip
over. It seeds the approval it needs and that approval stays, which the comment
now says.
mp_abort appends the extract as cats(msg,'\\n\\nLog Extract:\\n',logmsg), which is
now stated where the strip happens. The strip removes whole \\n or \\r pairs, so a
separator that gains a carriage return does not leave a fragment glued to the
message; a message that ends with the marker and no appended extract is a known
ambiguity and says so in the comment.
stagedata.test.4: the comment above exp8 now records why the expected text has
no space after either colon - the child builds it as &mac.:&VAR_NM.:&reason -
so the spacing is not a formatting choice a later reader has to re-derive. The
change itself came with ebfd395 on this branch and is unchanged here.
Suite 64/64, mpe_assert_abort PASS, sasjs cbd before the run.
Two reasons the last CI run still had four failures.
test 1 timed out on label[for="file-upload"]. The label is a click target for a
hidden input; attachFile can drive the input itself, so the helper no longer
clicks anything and selects by the input's id, asserting there is exactly one.
That also answers the earlier review point about positional selectors without
reintroducing a click on a control that is not always rendered.
tests 5-7 timed out waiting for the DDL file. The mock names a DDL download
after the libref alone, while services/public/getddl.sas names it
<libref>_<ds>.ddl - which is what the viewer expects and what the spec looks
for. The mock now follows the service.
fixturePath carried a trailing slash that the template also supplies, so
attachFile was given /excels//regular_excel.xlsx. Also drops the stale duplicate
doc comment above attachExcelFile - the helper targets the editor's import label,
which is what the remaining line says.
Addresses the review on e2e9103.
- The marker is located from the end. A hook message may legitimately contain
the text "Log Extract:" and only the final occurrence is the one mp_abort
appended; taking the first cut such a message mid-sentence.
- The captured text is unescaped (\" -> " and \\ -> \) before the comparison, so
it is compared as the text the hook raised rather than as its JSON form. That
is also what makes a message ending in a backslash comparable at all: mp_abort
escapes it as two, and the strip now leaves the pair alone for the unescape to
collapse - it only removes \n pairs and a lone stray backslash that is the
start of an escape the marker cut in half. Requiring the preceding character
not to be a backslash is what keeps a real trailing backslash out of that
branch.
- mpe_abortmessage owns the name it writes, _mpe_abortmsg, instead of taking one
as a parameter, so a call can no longer overwrite a caller's global of the
same name. Each call replaces the previous value; the doc says so.
Two cases added to mpe_assert_abort.test: a message ending in a backslash, and a
message containing the marker itself. Both fail against the previous logic
(first occurrence, no unescape) and pass now. Full suite 64/64 on sas.4gl.io,
sasjs cbd before the run.
Second CI run put the spec at 2 passing / 5 failing, and the five failures were
one more stale assumption: cy.select() matches an option's text against the whole
label, and the labels carry a suffix - the option for the Excel writer is
"Excel (.xlsx)", not "Excel". Every format is now chosen by the option's value
(EXCEL, SAS, SAS_DDL, TSQL_DDL, PGSQL_DDL), which the template sets and which does
not change when a label gains or loses a qualifier.
Also the review points on this PR:
- attachExcelFile starts from label[for="file-upload"] instead of
.buttonBar button:last-child, and submitExcel no longer takes a callback that
nothing passed.
- test 1 asserts on the download button whose id is the table it submitted: the
stage URL carries the id, so the assertion names its own row and asserts one
match, instead of indexing into every match on the screen.
- the comment in cypress-specs.sh no longer claims the spec depends on approvals
left by earlier specs - it seeds its own - and says why it is appended.
Addresses the review on 5d32de9.
The text mpe_execute aborts with is now built by mpe_abortmessage, which takes
the child message by variable name and writes the result to the global symbol
table. Two reasons: a value argument would be split at the first comma before
the macro ran, which is the defect this PR is about, and having the assembly in
a macro makes the comma case testable - a test calls it directly and asserts the
text, where going through mpe_execute needs a second service aborting to observe
the call the caller itself is aborted by.
- mpe_assert_abort: a lone trailing backslash is no longer stripped. Only
whole escape sequences are removed (\\n, \\\\n, \\\\), so a message that ends
in a backslash keeps it.
- mpe_assert_abort: the strip is skipped when "Log Extract:" is the first thing
in the captured text, instead of relying on SUBSTR with length 0.
- mpe_assert_abort: both sides of the comparison go through %superq, so a child
message containing %, & or a quote is neither rescanned nor able to break the
quoted comparison.
- mpe_execute: the message is concatenated, not CATX'd, so it arrives byte for
byte and a blank child message leaves no trailing blank.
- mpe_execute.test: three cases over the new macro - a comma-bearing message
survives, the text names the program and state, and a blank message leaves the
text clean.
mpe_execute 3/3 pass, mpe_assert_abort PASS, full suite 64/64 on sas.4gl.io,
sasjs cbd before each run.
The strip cuts the message at the "Log Extract:" marker, and got was declared
$256 - so a message whose text pushed the marker past 256 characters was never
stripped, and the exact comparison then failed on the escaped tail. got is now
the same width as line ($32767).
Covered by a new case in mpe_assert_abort.test.sas: an abort whose message runs
past 256 characters, asserted in full. It fails against the $256 buffer and
passes once got is widened, so the width is pinned rather than assumed.
mpe_assert_abort: PASS, full suite 64/64, rebuilt with sasjs cbd.
Three separate reasons the spec could not pass, all of them stale routes or
selectors rather than an overlay covering the button:
- test 1 opened #/approve/toapprove, which is not a route. The review module is
namespaced under review/, so the approve screen is #/review/approve. The dead
URL rendered the 404 page, and the only .btn.btn-success on that page is not a
download button - which is why the failure looked like a covered element.
- on the right route the approve grid still renders nothing on a fresh estate:
it draws a download button per table waiting for approval, and there are none
("There are no approvals remaining"). Test 1 now submits an excel fixture for
mpe_x_test first, the way stage.cy.ts does, so it owns the approval it
downloads instead of depending on what earlier specs happen to leave behind.
- the viewer's options menu is a list of div[clrDropdownItem] entries, not
buttons, so cy.get('clr-dropdown-menu button') never matched; Download is now
selected by its text, as filter-panel.cy.ts selects Filter. The dropdown-toggle
class the helper also required is gone from that element too.
The download button is selected by aria-label, which is unique, rather than a
bare .btn.btn-success that also matches the editor toolbar's small buttons.
Closes#398
Addresses the review on #412.
The prefix comparison is gone instead of guarded. mp_abort appends "Log Extract:"
and the surrounding log to every message it returns; the capture now cuts at
that marker and strips the escaped newlines before it, so the comparison is an
exact match again. That removes the hole the reviewer found - a blank expectation
can no longer pass, because %length 0 no longer short-circuits anything - and it
keeps the expectations strict, so a change in the child's wording still fails the
test rather than sliding past on a matching prefix.
mpe_execute: the assembled message declares its own length. An inline symget()
is typed by the data step default, which would truncate a long child message
before it reached the abort.
The expected literals in stagedata.test.4 are unchanged by this commit: with
exact matching restored, stagedata passes 5/5, which is the confirmation the
reviewer asked for - they are the child's text, not text adjusted to fit the
observed log. Full suite 64/64.
Both selectors in download.cy.ts are stale or ambiguous, which is why the spec
fails as soon as it runs:
- `.btn.btn-success` also matches the editor toolbar's buttons (`btn btn-sm
btn-success`), and those sit inside a fixed-position container on the approve
screen. The first match was one of those, so `should('be.visible')` retried
against an element that is never visible and the error read like the download
button was covered. The approve row's button is `btn btn-success` with
aria-label="Download audit file", so select it by that label.
- `.btn.btn-sm.btn-outline.filterSide.dropdown-toggle` cannot match anything:
the viewer's options dropdown is `class="btn btn-sm btn-outline filterSide"`,
with no dropdown-toggle class. filter-panel.cy.ts and row-cell-limits.cy.ts
already select it as `.filterSide`.
The spec goes back into the CI list, appended at the end because it needs
approvals to exist and the earlier specs create them. metanav stays excluded
(#395).
Closes#398
A child that does not complete is reported by aborting the caller with
"&program did not complete: state=&state &msg". The child's message can contain
commas - mp_abort appends a log extract to whatever the hook said, and the
derived-variable scan's own wording has one - so building that string inside the
macro call made the commas delimiters:
ERROR: All positional parameters must precede keyword parameters.
The caller lost the child's message and the abort fired with a corrupted text.
Wrapping the whole thing in macro quoting is not the answer either: %bquote masks
'=' and '/' as well, and the mask characters end up in the message the user
reads (they show up in work.mp_abort_errds and in the JSON response as control
bytes).
The text is now assembled in a data step and passed through %superq, which keeps
it as one argument without altering the characters.
Two follow-on fixes in the same path:
- mpe_assert_abort compared the returned message exactly, but mp_abort appends
"Log Extract:" and the surrounding log to every message it returns, so an
expectation written as the message the hook raised could never match. The
comparison now falls back to a prefix match, which keeps the exact-match
behaviour for the messages that carry no extract.
- stagedata.test.4: both abort-expecting calls pass failfast=0 (the second one
did not, so the test cancelled itself instead of reaching its assertion), the
expected messages match the scan's own format ("MPE_DERIVED:CALLER:..." with
no space after the first colons), and the expected text is passed through
%superq because it contains a comma.
Test 8 submits a VAR_LOGIC expression that the derived-variable scan refuses, so
the child is expected to abort and the assertion reads its message from the
response. Without failfast=0 mpe_execute aborted this program instead, so the
case could never reach its assertion. The file's other abort-expecting calls
already pass failfast=0.
This does not make the case pass on its own - the child still cancels without
writing a response, which is tracked separately - but it removes the wrong
failure in front of it.
mpe_execute reports a child that did not complete by aborting the caller with
"&program did not complete: state=&state &msg". The message was passed through
%str, which masks the literal text before resolution - so the commas in a
child's own abort message (mp_abort writes them as part of the reason) were
still delimiters when the call was parsed. The call split into extra positional
parameters and SAS raised
ERROR: All positional parameters must precede keyword parameters.
replacing the real failure with a parse error at the call site. The caller then
lost the child's message entirely.
%bquote masks the resolved value, so the message arrives as one argument and the
abort reports what the child actually said.
The same shape in mpe_security_postedit, where the submitted ACCESS_LEVEL value
is interpolated into the abort message, is fixed the same way.
All four pipelines downloaded `releases/latest/download/linux.zip`, unzipped it
and ran `api-linux`. They now start the published package at a pinned version:
pm2 start npx --name sasjs-server -- --yes @sasjs/server@3.6.2
and `sites.yaml`, which runs it under nohup, does the same with
`npx --yes @sasjs/server@3.6.2`.
Why: 10.7MB instead of 78.7MB, the same artifact a user installs, and a version
this repository names rather than whatever `latest` happens to be. `latest` also
moved silently on every run, so a red pipeline could not say which server
produced it. 3.6.2 is the first release whose bin carries a shebang; before it
`npx @sasjs/server` could not start at all.
The pin lives in `env.SASJS_SERVER_VERSION` in each workflow, next to the npm
cache step that makes repeat runs skip the download (`~/.npm`, keyed on the
version; `continue-on-error`, so a runner without cache support still runs the
pipeline).
The readiness loops and `RUN_AS=root` are unchanged - the package runs as the
runner's Node 24 rather than the Node embedded in the zip, and the loop is what
tells us it came up. Failure paths now dump `pm2 logs sasjs-server`, the name the
process is registered under.
Addresses both findings from the review of #408.
runAutoDeploy was declared async after the service-pack step went, but calls
createDatabase() and returns without awaiting anything. createDatabase() starts
the request and reports through the component's own status flags rather than a
returned promise, so there is nothing to await - drop the async rather than
pretend to a contract the callee does not offer.
runMakedataInNewWindow interpolated the context name, admin group and DC path
straight into the query string. An ampersand in any of them started a new
parameter and a space arrived truncated; a DC path containing a slash was only
working because nothing decoded the query. Those three values are now
percent-encoded at the build site, so the server reads back what the user chose.
Tests: the new-window case now asserts the encoded forms, and a second case
pins the values that need encoding (context with an ampersand and spaces, admin
group with a space and a plus, DC path with a slash and a space). The deploy
case no longer awaits a synchronous handler.
Executed 24 of 24 SUCCESS (automatic + sasjs-configurator + deploy components).
Restoring any load that touched a special missing numeric (._ or .A-.Z) failed
in mp_stripdiffs, which writes the old value into a generated UPDATE statement.
PUT writes a special missing as its letter only, so the statement carried
"set SOME_BESTNUM =_" and PROC SQL read the bare underscore as a column name:
ERROR: The following columns were not found in the contributing tables: _.
ERROR: SOME_BESTNUM, a numeric column, can only be updated with a numeric
expression.
Every statement in the generated block then failed NOEXEC, so the revert
produced the wrong data; the service's syscc check is what surfaced it.
Fixed upstream in sasjs/core#499 (the period is now written first) and
released as 7.0.1. v7.0.0's breaking change - mf_existvarlist warns and returns
0 instead of aborting on blank input - does not affect DC: there are no direct
callers, and the full SAS suite covers the core macros that do use it.
Adds restore.test.2.sas, which seeds some_bestnum=._ on mpe_x_test, changes it
to a number through stagedata and approves it, reverts that load, approves the
reversion, and asserts the cell is back to ._ rather than blank. It fails on
6.1.4 and passes on 7.0.1.
The setup screen's only action is to run `makedata` with what the form
collected, and nothing tested it. Both screens now have specs pinning the
request and the URL they build:
- Viya (automatic): the admin group and DC path reach the service as
`fromjs`, the configured context is used with the web method, a context chosen
in the form wins over the configured one, the new-window URL carries
`_contextname` / `admin` / `dcpath` / `_debug`, and pressing deploy reaches
`createDatabase` at all
- SAS9 and SASjs Server (configurator): the makedata URL carries `_program`,
`admin`, `path` and `_debug`; SAS9 stays on the page origin and encodes the
directory, SASjs Server leaves the path readable, and the SASjs Server check
runs before any reload
The deploy button is also restored. Removing the dead upload markup took the
live button with it, so the screen had no way to start the job its own copy
promises - the new `runAutoDeploy` test would have caught that.
The Viya setup screen still carried a path that deployed the backend as a
service pack from a viya.json (`deployServicePack`). That is no longer how a
deployment works: the frontend and services arrive with the sasjs deploy, and the
only thing left to do is create the control library and its configuration data,
which `makedata` does - and then deletes itself.
The path was already unreachable from the UI (its button had been commented out
"for reference"), so the wizard only ever ran `makedata` in practice while its
copy still claimed to deploy services. Removed:
- `executeJson()` and the `viya.json` upload plumbing: `readFile`,
`clearUploadInput`, and the fields and status flag that fed them
- the "Deploy SAS Jobs" row in the completion modal, which could never show
- the commented-out button and file input that kept the dead path visible
The screen now says what it does: it runs the one-off job that creates the
control library and its configuration data, and the status row reads "Create
control library" rather than "Create database".
The deferred VA load builds the filter from the report's parameters before the
first getdata, which is what makes that first load already filtered. A va_parmap
entry is a column the report author typed, so the request can legitimately fail
when the table does not have it - validatefilter aborts on an unknown variable -
and that abort surfaced as a modal over an editor which then loaded fine
(dc/dc#404).
Validating the mapping first would cost a SAS call on every load of a mapped
object, which is not affordable, so the attempt is simply made quietly:
saveQuery takes wrapper options and the deferred load suppresses the error
modal. A mapping that is right still filters the very first load; a wrong one
falls back to the normal load, and the metadata pass reports it in the banner
and reconfigures once, at setup time rather than on every page load.
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.
No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.
parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.
A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.
Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.
No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.
parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.
A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.
Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
@sasjs/core 6.1.3 moved the derived variable field rules out of
mp_derivedvars into mp_derivedvarcheck, so the same scan can run before a
definition is used. DC carried its own copy of that list in
mpe_derived_check.sas, which is now deleted: the post edit hook calls the
core macro instead, and the hook keeps only the rules that belong to the
control table rather than the expression - LIBREF/DSN required, and ACTIVE
within the picker pair.
The scan is stricter than the copy it replaces, in three ways:
- VAR_NM may not be _N_, _ERROR_ or _IORC_ (automatic variables of the data
step)
- VAR_LOGIC may not exceed 4000 characters
- VAR_LOGIC may not call a function that submits code or reaches the file
system, the environment or the macro symbol table - a function name cannot
be assembled at run time, so this is a control the expression cannot evade
and it relaxes one: VAR_LABEL may carry a line break, which is flattened to a
space when the label is used, so only the remaining control characters are
refused.
The mock hook mirror is updated to the same rules, and stagedata.test.4
gains the two submit-time cases that pin the new wiring.
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
### Bug Fixes
* **ci:** install doxygen for the SAS docs job ([f58925b](f58925b299))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](27686d3aad))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](888873421c))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](dd482c4069))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](25ffa71add))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](c5808a16eb))
* **ci:** rebuild and upload release assets on demand ([6486c80](6486c80ed4))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](af1813f829))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](cf7b21a8b6))
* **cypress:** hold download out too, and retry once in CI ([c97e886](c97e8866db)), closes [#395](#395) [#398](#398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](c803773d3c)), closes [#395](#395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](80e8911af8))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](e51953ca8a))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](459ff2da37)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](f7e753863b))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](7e3045a21b))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](2623a3fb4b))
* **embed:** log a parse-time mapping problem once ([0011f7e](0011f7e933))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](541a61ba08))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](e9216b0895))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](bd3c220af2))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](f83dc359e2))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](aa91daf9f7))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](10157c1159))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](411f6c3354))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](c0ea810766))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](15a1f34d52))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](88010c3980))
makeDataServer.json seeded ADMIN=AllUsers, a Viya-style group that no account on
the shared estate belongs to. The seed still reports SUCCESS with a group the
caller is not in, and then every getdata/stagedata call aborts with
'<user> is not authorised to edit <libds> in the MPE_SECURITY table', which
reads like a broken deployment. The estate's accounts are in all-users, the
SASjs Server group every logged-in user has, so seed with that and describe the
rule in the estate-testing skill rather than telling the reader to override the
shipped value.
The run on release/7.20 went from four failing specs to two:
csv-limited PASS (order restored)
hook-programs PASS (order restored)
metanav held out, #395
download 7 of 7 fail
download now gets past its hooks - the deleteFolder task is registered - and
fails on the button it clicks:
AssertionError: expected '<button.btn.btn-sm.btn-success...>' to be 'visible'
This element is not visible because its ancestor has 'position: fixed' CSS
property and it is overflowed by other elements.
Something fixed-position sits over it. That needs a scrollIntoView or an overlay
dismissed, and each attempt costs a forty minute CI cycle, so it is held out with
#398 carrying the exact error and the suggested fix. The list reports 22 run, 2
excluded.
editor.cy.ts still fails one test of fifty-five, and it failed the same way in
both runs: 'Expected <div.app-loading> not to exist ... continuously found', a
load timeout on test 28 rather than an assertion. One retry in run mode absorbs
that, none interactively; a real failure still fails twice and the retry's output
is in the log.
The shared-estate cycle is two npm scripts and sasjs test, but the seed in
npm run server is load-bearing: without it testsetup 404s settings.sas and
59 of the 64 test programs fail to complete, which reads like a broken
deployment. Record the targets, the token-pair credentials, the ADMIN group
the estate actually uses, how to read the results, and the frontend
passphrase steps as a repo-local skill, and point the testing doc at it.
Four specs failed on release/7.20 once the list went from 19 to 24. Three of the
four are addressed here; the fourth is a spec that cannot pass and is now
excluded with an issue.
download.cy.ts - it clears the downloads folder in an afterEach through
cy.task('deleteFolder'), and that task was registered only in
cypress/plugins/index.js, which Cypress 10+ never loads unless it is required
from cypress.config.ts. The task is now registered in setupNodeEvents, where
Cypress looks, and the login in its beforeEach only fills the form when the form
is actually on screen - the unconditional version failed with 'the center of this
element is hidden from view' whenever the session was already live.
csv-limited.cy.ts and hook-programs.cy.ts - neither is broken; both depend on
where they sit in the run. csv-limited asserts the free-tier 'Locked Feature'
modal, which an earlier spec removes by planting a valid licence, and
hook-programs asserts an approval row against its target table before other specs
have added rows to the approval tables. The list is now the sequence that was
green for months, with csv-limited first and hook-programs second, and the newer
specs appended rather than inserted. The script says why order matters, because
the next person to alphabetise it will break the same two specs.
metanav.cy.ts - cannot pass, and says so in its own source: '// BLOCKER ...
.clr-accordion-header-button always null although it is present on the page'.
With the duplicate login in its beforeEach removed it now reaches that assertion
and fails there, so it is excluded from the list with #395 as the reason, and the
login fix is kept for whoever picks it up. The coverage check now distinguishes a
deliberate exclusion from a forgotten file, so nothing can go missing silently.
Not verified locally: the specs need an estate seeded by 'sasjs cbd -t server-ci'
plus makedata, which a scratch mock does not reproduce - the app never leaves
app-loading. The CI run is the verification.
semantic-release bumps the working-tree package.json before the release
commit, so the tag is known in the job. Using it for the asset upload and
both dispatches means an overlapping release cannot redirect them.
Only the assets this workflow uploads are removed before a re-run, so a
hand-attached asset on the release survives. jq is installed before the
tag resolve that parses its output, the SASjs CLI is installed at the
version the tag's sas/package.json pins, and the npmrc secret is appended
rather than overwriting the file.
The four shards made the release-branch run red, and the reasons are in the
logs rather than in the theory:
- every failure was a timeout, not an assertion - "cy.type() failed because
the center of this element is hidden from view", "Expected
<div.app-loading> not to exist ... continuously found". Four jobs on one
runner means four SASjs servers, four ng serve and four Chromes competing
for the same CPU, and the app under test starves;
- upload-artifact@v4 does not work on this runner at all:
"GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ ...
not supported on GHES", so the v3 -> v4 move I made on review advice broke
the artifact step outright.
Measured, from the one sharded run that did complete: 20m34s wall against 33-35
minutes for the single job, with peak concurrency 3 - so a third off, not the
four-fold the split suggests. That is not worth a flaky pipeline while a release
sits on it.
Kept from the sharding work, because those parts were the real gain:
- the spec list has one home, .gitea/scripts/cypress-specs.sh;
- Build-and-ng-test fails when a spec on disk is not listed, so the five that
had never run in CI cannot go missing again - and they do run now;
- videos are zipped and uploaded only on failure;
- one job means one server, which is also what the guards and the balance
arguments were protecting.
The shard and fan-in jobs are gone with the branch that carried them.
curl -k disables verification on calls that carry the release token, so a
man-in-the-middle on the runner's egress could capture a credential that
rewrites releases. The forge serves a publicly trusted certificate (the
runner verifies the same chain on every checkout), so the flag is
unnecessary: drop it from all fifteen calls in the two release workflows.
The standalone URL dropped every query parameter whose name started with
va_, not just va_parmap; the names are now enumerated, and a later
parameter with that prefix survives (the component spec pins the exact
standalone URL instead of a substring, so the query surviving is the
assertion, not an accident of the runner's own query).
The timezone runs are scoped to the VA specs, the only specs with local
calendar arithmetic, so the blocking job no longer runs the whole suite
twice more.
The gate could not fail. With needs: on four shards and no if: always(), a
failed shard skips the fan-in job instead of failing it, and a skipped job is
not a failure on the one check branch protection watches - so the gate was
silent in exactly the case it exists for.
It now runs with if: always() and fails on any shard result that is not
success. The count is checked as well: if the needs context does not resolve on
this runner the step fails saying so, rather than looping over nothing and
reporting green - which is the failure mode a bare loop would have.
Also removes what the last regeneration left behind: the coverage guard step
appeared twice in Build-and-ng-test (so it ran twice) and the shard 1 comment
block appeared three times. The file is regenerated from fix/va-parmap-review-572
rather than from my own previous commit, which is what caused the duplication.
Renaming the shards to cypress-shard-1..4 and adding a job named
Build-and-test-development that does nothing but needs all four.
That name is the status check the branch protection rule requires, so the check
now turns green only when every shard passed - a red shard 3 leaves it
unreported and the PR unmergeable, which is the behaviour the single job had.
No repo setting has to change, and adding a fifth shard later means adding it to
needs rather than remembering to edit branch protection, where forgetting fails
silently.
Review found that five specs ran in no shard - coltype-delete-record,
download, embed-va, metanav, submitted-details - and one of them is the VA
embed suite this release added, 454 lines that had never run in CI. The old
single --spec string omitted them too, so nothing regressed; they were simply
never wired up, and nothing in the repo marked them skipped.
The spec lists now live in .gitea/scripts/cypress-shards.sh, one place, so the
workflow no longer spells them out four times. A guard step in
Build-and-ng-test compares the shards against the specs on disk and fails when a
file belongs to none of them or to more than one - proven by removing a name and
watching it fail with that file listed.
Rebalanced for the five additions, on test count and spec count together, since
each spec pays a few seconds of startup:
shard 1 editor, csv, metanav 57 tests
shard 2 excel, filtering, download, browser-info,
excel-multi-load, hook-programs 49 tests
shard 3 embed-va, viewbox, licensing, filter-panel, liveness 43 tests
shard 4 viewer-labels, row-cell-limits, derived-variables,
ddl-export, coltype-delete-record, submitted-details,
startupservice-launcher, stage, full-table-search,
csv-limited 27 tests
24 specs / 176 tests, each in exactly one shard.
upload-artifact moved from the retired v3 to v4; this change multiplies that
step by four, so it is worth not carrying the dead major forward.
Not in this commit: the three new check names still need adding to branch
protection, which needs repo admin.
Sits on top of fix/va-parmap-review-572 (#391) so the two changes to this
workflow land in one order; the timezone step that PR adds to
Build-and-ng-test is untouched.
The development job ran all nineteen specs in one Cypress invocation on one
runner, so its wall time was the sum of every spec - around half an hour on a
green run, with the whole pipeline waiting on it.
Four jobs now, each with its own runner and its own SASjs server, taking a
subset:
shard 1 editor (55 of the suite's ~148 tests)
shard 2 excel, filtering, csv-limited, csv, stage
shard 3 viewbox, licensing, filter-panel, browser-info, excel-multi-load, liveness
shard 4 viewer-labels, row-cell-limits, derived-variables, ddl-export,
hook-programs, startupservice-launcher, full-table-search
The split is balanced on test count, not file count: editor.cy.ts alone holds 55
tests, so an even spread of files would leave one shard running twice as long as
the others. Its own duration is the floor the pipeline can reach.
The setup steps are identical in every shard, so they are defined once and
aliased with a YAML anchor; each job differs only in SHARD and SPECS. Shard 1
keeps the original job name so an existing required status check still matches.
Video is still recorded - measured at 91s with it and 92s without on the same
spec, so the recording is not what takes the time. What it did cost was the zip
and artifact upload of every run's videos, now guarded by if: failure().
Review follow-up on the embed guard.
The spec asserted the isEmbedded getter, which mirrors the template rather
than exercising it: dropping !isEmbedded from the @if guards left every test
green. The spec now renders the component in a TestBed and asserts the DOM -
#TCS present standalone and absent for embed=va and embed=true, and the
deploy section withheld after the terms are accepted when embedded. Both
guards were mutation-checked by removing them, which fails the spec.
deployUrl and onNavigateToHome computed the standalone URL two different
ways: origin+pathname in one, location.href.split('#')[0] in the other,
which kept the query string, so a tab opened from ?embed=va#/deploy stayed
embedded. Both now call standaloneUrl() in standalone-url.ts, which parses
the href and drops the query and the fragment. It is tested with a
hash-bearing input, and the navigate test asserts the opened URL carries
neither '?' nor '#', so a regression to split('#') fails.
The self-destruct block ends with a single-statement %else in open code:
%if &keepme ne 1 %then %do;
%mv_deletejes(path=&root/admin, name=makedata);
%end;
%else %put NOTE: &_program retained, as KEEPME was requested;
Outside a macro SAS pairs %else only with a %do branch, so this stops with
'ERROR: Expected %DO not found' and 'ERROR: Skipping to next %END statement'
after the deployment has completed. On Viya the job then reports state error
with jobConditionCode 1012, so a successful deployment looks like a failure
exactly when someone is watching a first install.
Both branches now use %do/%end, which is the pattern the repo uses elsewhere
for this reason - see db/migrations/20261003_row_and_cell_limits.sas.
The same block is fixed in the server variant. The two delete calls also gain
their terminating semicolons: the macro call worked without them, but their
absence is what made the block look responsible for the error.
Behaviour is unchanged: the program still deletes itself unless KEEPME was
requested.
Deployment needs an authenticated, standalone session, and the person
looking at a report object is not the person who installs the app. Offering
the wizard there produces a screen whose controls cannot work - the failure
we hit while testing a report: T&C accepted, credentials filled, DEPLOY
clicked, and nothing happened, because the embedded app has no session to
deploy with.
An embedded instance (embed=va or embed=true) now shows what the state is
and where to deploy from, instead of a form that cannot submit. Standalone
behaviour is unchanged.
Two review findings, both about a filter that looks applied and is not:
- The transform branch of the clause builder interpolated whatever
convertJsDateToSasDate returned, so an Invalid Date reaching the helper
would have put a literal NaN in the WHERE clause. A non-finite result is
now dropped, and unsupportedTransforms reports the parameter, so the
author sees a banner instead of a broken filter.
- The range count check counted values that survived date parsing, while the
builder counts resolved bounds. A RANGE sent two values where one did not
parse therefore built a BETWEEN covering one period with no warning. The
check now counts what VA sent, the builder refuses a range with more than
one value rather than picking one, and a value set that is partly
unparseable gets its own line ('1 of its 2 values was not a date - the
filter covers only the rest').
The banner also now says what to do: it names the URL parameter to edit and
shows the entry grammar with an example, since the reader is the report
author and has no other way to act on it.
Conflict was valueSignature: the transform branch already builds a clause
value set through buildClauseValueSet, so the helper delegates to it and
takes the parameter's transform, keeping derived dates in the comparison.
Two review findings, plus a scope decision.
A va_parmap key is now matched against the parameter's label only. Matching
the internal name (pr273) as well meant one parameter answered to two keys,
so an entry written against the name and one against the label could never
be told apart: the label entry won and the name entry was silently dead -
no parse problem, no unresolved line, no unmatched line. Dropping the
fallback removes the ambiguity, and an entry keyed on the internal name now
shows up as an unmatched mapping like any other typo.
sameColumnMappings no longer reports two parameters on one column when they
carry the same value: col = 'x' AND col = 'x' filters exactly as one clause
does, yet the banner asserted that different values would show no rows. The
clause value each parameter produces is now compared, and the warning is
raised only when they differ. Parameters that produce no clause at all are
excluded, since they cannot conflict with anything.
A transform resolves a report value to dates using local calendar
components, so a test that passes in UTC says nothing about a browser west
of it. test:tz-west and test:tz-east run the whole suite under
America/New_York and Pacific/Auckland; with the UTC construction this test
pair replaces, the bare-number round-trip case fails under the first.
Not wired into CI: it triples the run for a property that is already
pinned by the round-trip assertion, so it stays a local and pre-release
check.
Brings in the diagnostics fixes from the base branch; conflicts were the
paramMap type widening to VaParmapTarget and the new paramsStable argument,
both of which the transform branch already carries.
Review findings:
- A bare SAS number was turned into a UTC instant and then read with local
accessors, so west of UTC the value landed a day early and the anchor was
computed from the wrong quarter; the round trip to a SAS day number was
one out. Built with local calendar construction instead, matching
parseDate and convertJsDateToSasDate.
- RANGE passed keepTime to both bounds, so a DATETIME range started at the
value's clock time on day one and ended at it on the last day, dropping
the rest of both boundary days. A range now covers whole days: period
start at midnight, period end at 23:59:59.
- A RANGE fed more than one parameter value produced no clause and no
warning, reading as a filter that matched nothing. One BETWEEN cannot
express several periods, so the count mismatch is now reported.
Review findings, all about a warning that misleads:
- The banner heading said the mapping was ignored, but two of the three
lists it renders are mappings that are applied (the AND-ed pair) or that
were never applicable (a parameter VA does not send). Neutral heading.
- unmatchedMappings reported a key as unsent as soon as any parameter had
been seen. VA sends the parameters a report object's query uses, so a
name that has not arrived yet looked like a typo. A mapping is now only
judged once a message introduces no new parameter name, i.e. once the
object's parameter set has settled.
- sameColumnMappings counted a column the table does not have, so one
mistake produced the unresolved-mapping line and the AND line. A column
that is not in the table is now left to unresolvedMappings.
- The Cypress cases asserted exact row counts, which encode the demo
table's contents. They now read the unfiltered count from the app and
assert the filtered grid is smaller (or empty), so a fixture change
cannot fail a behaviour test.
Subtracting a fixed day from the next quarter's start moves by an hour
across a daylight-saving transition, which would place a DATETIME bound
off the intended midnight. Day 0 of the following month is the last day
of the month by construction.
A VA parameter can now carry a transform on the column side of a mapping:
va_parmap=SELECTED_DATE-D_DATE:ADD(QUARTER_START,-1)
which filters D_DATE on the report value AND the same point one quarter
earlier, as a single IN clause - so the AND rule, the clause signature,
saveQuery and the reload loop are untouched.
Transforms resolve to one clause with these operations:
- ADD(<ANCHOR>,<offset>) the value plus one anchor beside it -> IN
- RANGE(<PERIOD>,<offset>) a closed period -> BETWEEN
Anchors: QUARTER_START/END, MONTH_START/END, YEAR_START/END.
Periods: QUARTER, MONTH, YEAR. Offset is a whole number of periods,
defaulting to 0.
The names describe the result rather than mirroring SAS intnx: the
arithmetic runs in the browser, so mirroring intnx would mean
reimplementing its alignment letters and their corners (week alignment,
's' and 'm') with no guarantee of matching SAS. A report author can read
ADD(QUARTER_START,-1) without knowing intnx.
Resolution is anchored on the parameter value, never on today, so a saved
report reproduces the same rows. A transform that cannot run where it is
used - a non-temporal column, a value that is not a date - is skipped and
reported in the existing warning banner rather than silently filtering on
the raw value.
`surfer deploy` chooses its target only from `--site-name`; without one it
replaces the app's DEFAULT site. The typedoc job passed `--server
webdoc.datacontroller.io` and no site name, so the 7.19.0 docs were published
over the default site of that app - the marketing site served at
datacontroller.io - instead of to a webdoc site.
Name the site, and give it its alias domain so the first run creates it.
The SAS docs job is left unnamed on purpose: code.datacontroller.io is its own
app whose default site is the docs, and a comment now says so.
The parse-time problems were logged at app init and then again by the
editor's warning loop, which spreads them into the banner's text. The loop
now logs only what the instance discovers for itself (unresolved column,
unmatched key, AND-ed pair); the parse problems stay in the banner and keep
their single init-time console line.
E2e case 11 pins it: one unparseable entry plus one valid-but-missing
mapping, so the loop certainly runs and logs its own line, and exactly one
line mentions the parse problem.
Two review findings, both paths where a filter could look applied while
producing nothing.
- `unresolvedMappings` judged a mapping against an empty column list while the
grid was still loading, so every entry was briefly reported as naming a
column the table does not have. An empty or unset `cols` now means "not known
yet" and returns nothing, so the banner appears only once metadata exists.
- Two parameters mapped to one column are AND-ed, so different values yield no
rows at all, and nothing said so. `sameColumnMappings()` now reports the
contended column and both parameters. The clauses are deliberately left as an
intersection: turning that AND into an OR is a filter-semantics decision, not
a diagnostic, so it is tracked separately.
Covered by unit tests for both (unloaded column list, one line per contended
column) and e2e case 10, which drives two parameters onto one column and
asserts the banner plus the empty grid.
Records that two parameters mapped to the same DC column are AND-ed, so the
result is the intersection: two different dates match no rows rather than
both dates' rows. Answers the 'filter one column on two dates' case from the
code rather than by assumption, and marks the single-parameter IN (...) shape
as the one that unions.
Three follow-ups from review, all the same class of defect: an author error
that leaves an unfiltered grid looking like a working filter.
- A duplicate key in `va_parmap` silently kept whichever entry came last, so
`va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER` filtered OTHER with nothing to
show for it. One VA parameter now maps one column: the first entry wins and
a conflicting duplicate is reported. The same mapping written twice is not a
conflict, and two different parameters mapping to one column stay valid -
that is how two report objects filter the same column from their own
parameters.
- A parameter-side typo (mapping `DROPDOWN_TAB1` when the report parameter is
`DROPDOWN_TAB_1`) produced no clause, no banner and no console line.
VaMessagingService now records every parameter key it has seen, across
editor reloads, and the editor reports a mapping key that VA has never sent.
Judged only once parameters are actually arriving, because VA sends just the
parameters its query uses - an absent key is not on its own an error, and a
key that turns up later stops being reported.
- The deferred first load cannot check a mapped column against metadata. On
SAS `mp_filtercheck` aborts the save, so the load falls back to unfiltered;
the banner now covers that path too, and an e2e case seeds the pre-bootstrap
message buffer to prove it (and to prove a valid mapping filters the very
first getdata).
A `va_parmap` entry that cannot be used was dropped in silence, in both
places it can fail: at parse time (no `-` separator, a missing side, or a
column that is not a plain SAS name) and at apply time (the mapped column is
not in the table being opened). Either way the parameter is skipped, which
degrades to "no filter" - a typo in the report's URL then looks exactly like
a working filter over the whole table.
Both are now reported in terms the report author can act on:
- a warning banner in the editor (VA mode) naming the parameter, the column
and the table, and pointing at the `va_parmap` entries in the URL
- console.warn for each distinct problem, so it is findable from the report's
console too
The parser returns `{ map, problems }` rather than a bare Map, VaFilterService
gains `unresolvedMappings()` for the entries the loaded table cannot satisfy,
and the editor merges both sets for the banner (logging once per distinct set
rather than once per VA message).
The banner is covered by two e2e cases, one asserting its painted box inside
the viewport (VA hides the app header, so it has to stand on its own) and one
covering a parse-time failure before any VA message arrives. The alert markup
keeps the message inside the alert's text flow so it wraps, instead of being
laid out beside a title and pushed off the right edge.
Two DC instances over the same table in one VA report could not filter the
same column from different parameters: parameter -> column matching is by
name, and two VA parameters cannot share a label.
The `va_parmap` URL parameter overrides that match per instance, one entry
per mapping (`va_parmap=<VA parameter>-<DC column>`, repeated for more):
...?embed=va&va_parmap=REGION_TAB1-REGION
The mapping is additive (unmapped parameters keep the name match), the
column is validated as a plain SAS name at parse time, and an entry only
applies where the named column exists in the table being opened - a miss
skips that parameter rather than falling back to the same-name match the
mapping exists to override. Entries split at the last hyphen, so a label
may itself contain hyphens, and the key matches the parameter label first,
then its internal name (pr###).
Parsed once at app init onto globals (like embed / vaApplyMode) so the
editor reloads a filter apply performs keep the mapping, and both filter
paths (the metadata build and the deferred initial load) honour it.
Also repairs embed-va.cy.ts: a cy.visit differing from the current URL
only in the hash is a same-document navigation, so ?embed=va never took
effect and both existing cases failed; their layout assertions also
described an earlier design (VA now opens read-only with an Edit button
and hides Filter). Cases 3 and 4 drive two mappings of the same column
through a real postMessage and assert the applied filter and row count.
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
### Bug Fixes
* address the review findings on the harness and refresh changes ([36e4e88](36e4e88eb1))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](698dea6d6b))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](0d58be975d))
* **client:** open the SAS log page with noopener ([072a133](072a133702))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](1fb4818d21))
* **cypress:** apply the licence key when the estate has none ([be8f445](be8f445ae1)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](fed43cb07e))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](f5aade5976))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](1e224763ba))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](9b3cf50b9c))
* **derived:** address five review findings ([ccd468b](ccd468b335))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](be50cb5d08))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](9914790777))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](9ef813d439))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](422f766cd4))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](7830b7bcfa))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](2d5de9a324))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](6a1128a463))
* **derived:** eight review findings ([561d7fc](561d7fc3ed))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](1863076bc9)), closes [#366](#366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](d3dec682aa))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](469229aecb))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](7fb981daa9))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](5f5d8681b9))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](5f443b0b16))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](6b5bf3e6be))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](85c89ca2a4))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](f8f65681e1))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](7efbf31ea5))
* **derived:** only the delete marker is a reserved name ([af31c36](af31c36b67))
* **derived:** open the statements file with a long record length ([ee31782](ee31782fbf))
* **derived:** pair the filterds %else with %do/%end ([88a38db](88a38db877))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](27ddcfb575))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](7799a300df))
* **derived:** reject a dependency on a row this submission deletes ([8806504](8806504967))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](9a047ad42e))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](e0da9f1098))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](688550a28d))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](1b5c4b3ecd))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](4d9ad38df1))
* **editor:** a blank truncation note is not a warning ([f182179](f182179403))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](04ce1861e7))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](4bcee706ff))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](45dce7b11b))
* **getdata:** clear the globals it declares before use ([359b570](359b570b85))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](059cdf176a))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](661ee19cfa))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](355c46a7b6))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](d626cb8f2d))
* **mocks:** address the audit-pack review findings ([489819b](489819b051))
* **mocks:** carry the zip flags into the central directory ([c4021a4](c4021a42ff))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](62f55db170))
* **mocks:** copy the physical column list before deriving ([f411b7d](f411b7d269))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](60ec470404))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](74060c0585))
* **mocks:** drop derived variables before staging ([73d25b1](73d25b1220))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](347dbb53b0))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](ee3c8787ef))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](6cad23fce9))
* **mocks:** search the physical columns only in the viewer ([293022d](293022d46b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](fb36379520))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](9c294c0ab5))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](0a52d20c5f))
* **mpe_execute:** fail fast when the child left no status ([a036386](a03638667f))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](220408538b))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](f2b1428ec3))
* name the startupservice launcher button "Show log" ([3445107](3445107d9a))
* offer the launcher on the globvars failure modal too ([ad2da45](ad2da45389))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](46bacdd6d9))
* **refresh:** one audit folder per refresh call ([56b6620](56b6620ef4))
* **sites:** round-trip the committed licence before publishing ([4d462d3](4d462d310f))
* **test:** getdata test ([da30266](da30266278))
* **tests:** enable closeouts logging ([fdcd8fa](fdcd8fa4b3))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](033ffd3025))
* **ui:** one error modal at a time ([81ccf75](81ccf758e9))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](42355f17bc))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](337ea8e5e0))
### Features
* **dc_getlibs:** dump the library list when debug is on ([582523d](582523dc87))
* **derived:** allow a variable to shadow a physical column ([12beeff](12beeff695))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](813bce6856))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](7c9ce9a712))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](87e794f769))
* **derived:** report the dependency cycle path ([cee1a95](cee1a954ee))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](78e88a4112))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](0b010f2c65))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](aa6d50e1ea))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](33d043ea92))
* **editor:** derived variables in the EDIT screen ([b0fd354](b0fd354ecd))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](6513d09f05))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](34691dceec))
* **editor:** show the derived column label in the header tooltip ([cfe6647](cfe6647ca8))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](752d944c20))
* **mocks:** implement derived variables in the JS services ([b4fae88](b4fae88ae1))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](5d183d1350))
* run startupservice in a new window from its error modal ([40f6133](40f61331aa))
* **sites:** mint a licence for the published sites ([7d9df71](7d9df71fdb))
### Reverts
* **derived:** no shadowing, and no self reference ([5af8e6f](5af8e6fc35))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](31aa859153))
Review feedback on the merged fix:
- derived_nobs was assigned twice, the first one dead - the coalesced form below
is the live value. Removed, so there is one assignment to read.
- making the catalog test a plain %if after the derived-variables test let the
catalog branch override it; the derived view won before and still should. The
catalog branch now requires derived_nobs=0, so the view keeps precedence and
only a catalog with no derived configuration is blanked.
Two guarded if blocks, not %else %if: that construct is what the parser rejected
in this file before ('Expected DO not found'), and the first attempt at this
review point reproduced it - validatefilter and viewdata both failed to complete
again until it was replaced.
Verified: sasjs test -t new -> 64/64 (100%), TEST=0.
The suite failed to complete on the estate with
ERROR: Expected %DO not found.
ERROR: Skipping to next %END statement.
and, once the parser had lost sync, 'Apparent symbolic reference IS_FMTCAT not
resolved' - because the %let that defines it never ran.
Two causes, both in the code that decides which dataset the filter is validated
against:
- the comment explaining the catalog case contained the text '%if ...' and the
macro processor resolves triggers even inside a comment, so the prose was
executed as code and tripped the parser from there. The comment now avoids
macro triggers entirely.
- an empty value compared in a condition is a character operand, and
%mf_nobs returns nothing when the dataset is missing. It is wrapped in
coalesce now, and the catalog flag is computed with %eval instead of a nested
%if.
targetds is still blanked for a LIBREF.CATALOGNAME-FC name - which is what makes
mp_filterstore rewrite it to a cntlout dataset - and stays the plain libds
otherwise, as an empty targetds falls back to libds in the engine anyway.
The test now passes outref= and mdebug=&sasjs_mdebug rather than a hard-coded
1: the suite already owns that switch, and testinit turns it on by itself when
the run asks for _debug=2477 or 131, which is when the child's response and the
PROC HTTP trace are worth having.
Verified: sasjs test validatefilter -t new -> 3/3 (100%), six assertions, all
PASS, including the catalog and derived-variable cases.
The Sites workflow was the one place that still started the server without a
listener, so it died with
[error] Invalid environment variable(s) provided:
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
and the following step's 120s poll timed out. build.yaml, release.yaml and
lighthouse.yaml already carry the same line.
refreshcatalog.test and viewtables.test failed to complete with
ERROR: could not create /tmp/DataController/<libref>/dc_staging/refreshcatalog:mpe_refreshcatalogs: + <stamp>
The staging folder a closeout writes its statements to is
&dc_staging_area/&etlsource, and etlsource defaulted to
'refreshcatalog:<step>:%mf_uid()'. A default is resolved when the macro is
compiled, not when it is called, so every refresh in a session reused the same
folder - the first step created it, the next one could not, and the abort took
the whole suite down. Whichever suite reached the staging area first decided
which others failed, which is why the count moved between runs.
etlsource now defaults to 0 and is set per call from
%mf_getuniquename(prefix=<step>): unique, a plain name rather than a path, and
resolved at the point of use.
Verified on the estate: refreshcatalog 4/4 (100%), viewtables 2/2 (100%).
derived-variables.cy.ts:97 expected dc-derived-first on the REGION header,
because the comment assumed REGION opens the computed block. The grid follows
MPE_DERIVED order, and that order is LINE_VALUE, REGION, REGION_LABEL - so the
marker was on LINE_VALUE and correct. The spec asserted the wrong column, and
the development pipeline has been red on it since.
Dumping the header row on a local mock estate (sasjs/server desktop mode,
RUN_TIMES=js) shows:
6 | LINE_VALUE | dc-derived-header dc-derived-first
7 | REGION | dc-derived-header
8 | REGION_LABEL | dc-derived-header beforeHiddenColumn htLastVisibleHeader
and the same dump against the unmodified release/7.19 client, so this was never
a client defect.
The assertion now states the invariant instead: at least one derived header, the
first of them opens the block, and no other one carries the marker. That holds
whatever order the control table returns the variables in.
The previous wording named only mpeinit and mpeterm, so the webout entry read as
an unexplained addition. webout is the linter's own default - sasjs compile
injects %webout into every service - and an explicit ignoredMacros list replaces
that default, so it has to be restated. 52 service files call %webout, which is
what dropping the entry would start reporting.
sasjs lint (@sasjs/cli 6.2.5) reported 24 warnings across the 11 .sas files
this release touches. None of them are in those files now:
- mpeinit and mpeterm are added to ignoredMacros in sas/.sasjslint. Every
service is compiled with a serviceInit program (sasjs/utils/
serviceinitsas9|viya|server.sas), and all three declare both macros in
their own header, which is how a service body's %mpeinit()/%mpeterm()
resolves without a declaration of its own. An explicit ignoredMacros list
replaces the default, so webout stays listed.
- Header entries nothing in the file calls are dropped: mp_binarycopy in
getrawdata (it arrives through mp_streamfile, which calls it - the
compiled program is unchanged), mp_jsonout in viewdata,
mp_assertcolvals in validatefilter.test, and mx_execute in testsetup
(every file that calls it declares it).
- Prose carrying "error"/"warning" is reworded. Where the text is log
output, %put WARNING: is written %put %str(WARN)ING: so the line the
log receives is unchanged.
- The server makedata service keeps its doxygen @warning tag - the tag is
the content - so that file carries @sasjslint {"noBannedStrings": false}.
- Two comment lines in getdata are re-indented to the file's multiple of 2.
AGENTS.md records the ignoredMacros rule and the %str(WARN)ING convention so
the next warning is not silenced by reflex. Compile passes; the 93 warnings
that remain are all in files this release does not touch.
The MARKDOWN branch of the service calls %mp_ds2md(), which ships with
@sasjs/core, but the header never listed it. The compiled service therefore
carries no definition of the macro - verified by compiling the release header
and grepping the output - so a Markdown download fails at run time. The viewer
offers "Markdown (.md)" in the download list, so the path is reachable.
Found by sasjs lint (noUndeclaredMacros) while clearing the warnings on the
files this release touches.
2026-10-27 21:27:09 +00:00
431 changed files with 41446 additions and 14307 deletions
The deploy screens ask two questions about the physical filesystem - what folders sit below this parent, and can the deployment user write to this one. Both are answered by `makedata` (`ACTION=DIRLIST` and `ACTION=WRITECHECK`), through `mpe_dirlist.sas` and `mpe_writecheck.sas`.
This note records the two-layer answer: a gate that closes the window once the appLoc is configured, and why the window itself has no group check.
## The gate
The deploy window is over once the appLoc has `services/public/settings.sas`: `makedata` writes it at the end of a successful run, and a run that failed - or was abandoned - after writing it leaves the file behind. So the service refuses `DIRLIST` and `WRITECHECK` whenever that file is present, however the run that created it ended. On SASjs Server the file is fetched with `ms_getfile` and a probe that cannot answer (anything other than 200 or 404) is treated as configured: an unreachable drive is not evidence that the window is open. Viya checks the same file through `mfv_existfile`.
The consequence for tests: the browse actions are refused on any appLoc that runs the suite, because every such appLoc is configured. `services/admin/makedata.test.sas` therefore asserts the refusal, and the listing and write-probe behaviour is asserted against the macros directly (`macros/mpe_dirlist.test.sas`, `macros/mpe_writecheck.test.sas`). The macros' own abort paths (a relative path, one carrying a macro trigger) are not asserted anywhere: `mp_abort` cannot be exercised inside the test session.
## Why there is no group check
Every other admin service gates on `&mpeadmins`, which comes from `MPE_CONFIG` in the control library - and that library is what `makedata` creates. Before it runs there is no group to check: `%mpeinit()` reads `&DC_LIBREF..mpe_config` and has nothing to read. That is also why the previous `dirlist` service could never serve the deploy screen it was written for.
The window is not narrower than the configurator's own: on SASjs Server the configurator page, in the same window, lets the caller pick any group they belong to and create the control library - which is full Data Controller admin. `DIRLIST` and `WRITECHECK` are strictly less powerful than that.
## What the actions do, for the record
-`DIRLIST` - enumerate the folders below any absolute path the SAS process can open. Folder names only; no file contents, no file listing.
-`WRITECHECK` - ask whether a path exists, is a folder, and whether the deployment user can create and delete a file in it. It creates a uniquely named probe file and removes it again.
Both refuse a value carrying a macro trigger, a quote or a statement separator, and shape check it with `mp_validatecol(ISPATH)` before use. The shape check fails closed: an absent `ISPATH` rule refuses the path rather than accepting it.
## The assessment
- **The window is the deployment itself.** `makedata` is deployed, run once by an administrator, and deletes itself on success.
- **There is no admin group to check while it runs.** It is created by `makedata`, from the group the deployer picks in the configurator.
- **The caller is the administrator.** Reaching these actions means reaching the STPs of a deployment in progress, which is the same caller the configurator is serving in the same window.
- **The actions are the weaker half of that window.** The configurator, in the same window, lets the caller pick any group they belong to and create the control library - which is full Data Controller admin. Listing folders and probing a write cannot achieve that.
- **The window closes on the settings file, not on survival of the service.** That is what covers a failed or abandoned run.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name.
## Commands
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`.
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
@@ -46,8 +46,12 @@ Don't use for: SAS backend tests (`sasjs test -t 4gl`), Angular unit tests (`npm
cd sas/mocks && sasjs request services/admin/makedata -t server-ci \
-d deploy/makedata.json -o ./makedata_out.json
cd sas/mocks && sasjs request services/admin/createdemodata -t server-ci
`makedata`replies with HTML (it is normally called as a URL redirect), so the CLI reports `invalid Json string`. That is expected - check the drive instead.
`makedata`builds only what a SAS deployment builds (the control library);
`createdemodata` adds the Cypress fixtures and refuses to run without it.
The payload asks for `OUTFMT=JSON`, so both calls exit non-zero on a failed
seed rather than reporting `invalid Json string`.
4.**Frontend.** Point the `sasjs` tag in `client/src/index.html` at the mock backend (`serverUrl="http://localhost:5000"`, `appLoc="/Public/app/dc"`, `serverType="SASJS"` - a local-only edit, the repo default is `serverUrl=""`), then `npx ng serve --host 0.0.0.0 --port 4200`. `client/cypress.env.json` needs `username`/`password` entries; any values work, `loginAndUpdateValidKey()` no-ops when there is no login form.
Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS
test suite against it. Covers the target and credential setup, the makedata seed the suite needs on
a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend
so the streamed app is the build you just made.
---
# Deploying and testing on the shared SASjs Server estate
The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong.
cd sas && npm ci # macro dependencies
cd ../client && npm ci && npm run build # frontend only - see "Deploying the frontend"
cd ../sas && npm run server # cpfavicon + sasjs cbd -t server + makedata
npx sasjs test -t server
`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `<appLoc>/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed.
## The targets
`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc``/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair.
A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered.
To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app/<you>/dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at.
## Credentials
The CLI logs in with a registered client, not a username and password: `.env.<target>` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless.
Mint the pair from a local login, with the password read from a file rather than argv:
1.`GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header.
2.`POST /SASLogon/login` with `{"username","password"}`.
3.`GET /SASjsApi/client` on the same session - the registered client ids.
4.`POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`.
5.`POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`.
An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over.
## The makedata seed
`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.
## Reading the results
Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON.
A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete.
`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for.
## Deploying the frontend
The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists:
1.`client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`.
2.`client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines.
3.`cd client && npm ci && npm run build`.
4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc.
5.`cd ../sas && npx sasjs cbd -t <target>` - with `client/dist` present the log says `Copying web app files` and `Web app is available at <url>`.
Without step 1 the deploy prints `ERROR An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main.<hash>.js` the page names.
Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check.
## Reading a failed run
-`ERROR An error occurred while executing the request. Job location: <appLoc>/tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/<suite>.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise.
-`WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem.
-`sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree.
- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code.
echo"::error::runner has ${effective_mb} MB of memory, a client job needs at least ${MIN_TOTAL_MB} MB (8 GB is the supported minimum; a cold production build alone peaks at about 3.8 GB, and the Cypress and Lighthouse jobs add a dev server and a browser to that)."
exit1
fi
# Not a gate - a second job landing on the same box shows up here first.
if[ -n "$avail_kb"]&&["$avail_mb" -lt 5000];then
echo"::warning::only ${avail_mb} MB is available on this runner; another job may be running on the same box"
*) echo "the build job published no decision - running the suite" ;;
esac
- name:Cypress suite not run - no client changes
# A named step rather than a run of greyed-out ones, so the job summary
# says why it took a minute: this job reports success having tested
# nothing, which is the intended behaviour for a pull request that
# changes no client code. If this job is a required check, "green" for
# such a pull request means "not applicable", and this step is the
# evidence for it.
if:steps.changes.outputs.client == 'false'
run:|
echo "No client, sas/mocks, sas/package.json or .gitea/scripts change in this pull request - the Cypress suite was skipped by design."
echo "The build job's 'Detect the Cypress job's inputs' step lists the files it saw, and warns instead if it could not decide (in which case the suite runs)."
- name:Check runner size
if:steps.changes.outputs.client != 'false'
# Each Cypress job compiles the frontend and runs a browser on top; a
# runner too small for that fails as a flaky-looking disconnect - see
# the script.
run:bash .gitea/scripts/check-runner-memory.sh
- uses:actions/setup-node@v4
if:steps.changes.outputs.client != 'false'
with:
node-version:${{ env.NODE_VERSION }}
- name:Write .npmrc file
if:steps.changes.outputs.client != 'false'
run:|
touch client/.npmrc
echo '${{ secrets.NPMRC}}' > client/.npmrc
# the secret arrives through the environment, not interpolated into
# this script: a quote or $( in the value would otherwise break the
*) echo "the build job published no decision - running the audit" ;;
esac
- name:Lighthouse audit not run - no client changes
# A named step rather than a run of greyed-out ones, so the job summary
# says why it took a second: this job reports success having audited
# nothing, which is the intended behaviour for a pull request that
# changes none of its inputs.
if:steps.changes.outputs.client == 'false'
run:|
echo "No client, sas/mocks, sas/package.json or .gitea/scripts change in this pull request - the audit was skipped by design."
echo "The build job's 'Detect the Cypress job's inputs' step lists the files it saw, and warns instead if it could not decide (in which case the audit runs)."
- name:Check runner size
if:steps.changes.outputs.client != 'false'
# This job runs a SASjs server, Chrome and the full mock estate; an
# undersized runner dies inside one of them with exitcode 137. Fail here
# in seconds instead - see the script.
run:bash .gitea/scripts/check-runner-memory.sh
- uses:actions/setup-node@v4
if:steps.changes.outputs.client != 'false'
with:
node-version:${{ env.NODE_VERSION }}
- name:Write .npmrc file
if:steps.changes.outputs.client != 'false'
run:|
# the registry secret arrives through the environment, not interpolated
@@ -31,7 +31,12 @@ Rationale: hard-wrapped prose produces noisy diffs when sentences are edited and
Never consider a change complete until the relevant linters pass on the files you touched — do not rely on the user's pre-commit hooks to catch it:
- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
- **SAS**: after creating or modifying any `.sas` files, run `npm run lint:sas` from the `sas/` directory. That script runs the `@sasjs/cli` version pinned in `sas/package.json`, so it matches CI; a global or `npx``sasjs` may resolve a different `@sasjs/lint` and report a different rule set.
Rules are promoted to `error` in `sas/.sasjslint` one at a time as their findings reach zero, and only errors change the exit code - so the whole project must stay clean of the promoted rules, not just the files you touched. Warnings under rules not yet promoted are the remaining backlog; do not add to them in files you touch, and do not clear other files' findings in the same change unless that rule is the point of the change.
`sas/.sasjslint` carries `ignoredMacros: ["webout", "mpeinit", "mpeterm"]`. `webout` is the linter's own default - `sasjs compile` injects `%webout` into every service - and an explicit list replaces that default, so it has to be restated there. `mpeinit` and `mpeterm` are listed because every service is compiled with a serviceInit program (`sasjs/utils/serviceinitsas9|viya|server.sas`) that declares both macros in its own header, so a service body may call them without declaring them. Add a macro to that list only when it reaches a file the same way - injected by the compiler, through serviceInit, testSetUp or another implicit provider - never to silence a genuinely missing declaration.
`noBannedStrings` flags the literal words `error ` and `warning ` anywhere in a file, comments included. Reword prose; where the text has to stay (a real `%put`, a doxygen `@warning` tag), keep the log output identical with `%put %str(WARN)ING:` rather than a per-file override, and use `@sasjslint {"noBannedStrings": false}` only when the word itself is the content.
Hook scripts (`sasjs/services/hooks/*.sas`) run inside the service that includes them, so the macros they call come from that service's session. They carry a file-local `@sasjslint {"ignoredMacros": [...]}` rather than a macro list of their own, because end users copy and edit these scripts and should not have to maintain a declaration section.
`noMacroIfInOpenCode` is on, so `%if` must sit inside a `%macro` definition. Where a conditional has to stay in a service body, wrap the block in a small macro and call it unconditionally (`%macro build_staged(); ... %end; %mend build_staged; %build_staged()`), or resolve the value in a data step with `call symputx`. Two traps: a `%let` inside such a wrapper is local, so a macro variable that must survive the block has to be written with `call symputx(...,'G')`; and open-code `%if` itself needs SAS 9.4M5, which the code no longer relies on.
## No external assets
@@ -41,7 +46,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
* **browse:** accept legal folder names, cap the walk, and lock the read ([3572d56](https://git.datacontroller.io/dc/dc/commit/3572d56cfb0e438512fac202829c8409388b1dac))
* **build:** build the mock sites with a CLI whose runtime provides webout() ([b1eab98](https://git.datacontroller.io/dc/dc/commit/b1eab982006393bb23f8d19076a715055087d4fc))
* **build:** publish the changed-path decision and pin the matcher with tests ([ecc8702](https://git.datacontroller.io/dc/dc/commit/ecc87021c3e398f7d2b451971e9d618d3dd33d04))
* **build:** treat a sas/package.json bump as a Cypress input ([48df795](https://git.datacontroller.io/dc/dc/commit/48df79520217afbc55a5b7f6a67a500efc70db07))
* **client:** cancel a queued dropdown reposition on destroy ([590b841](https://git.datacontroller.io/dc/dc/commit/590b841f24329e6d1aefab881895f794fc3738ed))
* **client:** drop the synthetic window resize dispatches ([2535baa](https://git.datacontroller.io/dc/dc/commit/2535baa5f7ad169f48678df3f0f55ca18b5eaa9a))
* **client:** keep the ARIA pass on the viewbox and multi-dataset grids too ([cb30469](https://git.datacontroller.io/dc/dc/commit/cb30469e794b4fb4c87f5c382f36236b837e1855))
* **client:** restrict the table pass to Handsontable's own tables, add the viewer fallback ([46c6aec](https://git.datacontroller.io/dc/dc/commit/46c6aec37a836722fff61337fde4cab0d97723d8))
* **client:** scope the ARIA pass to each grid and drive it from the render hooks ([bf1e7d6](https://git.datacontroller.io/dc/dc/commit/bf1e7d6b1583ef32d492840c508d2a4b0fb1a1df))
* **client:** stop synthesising aria-labels; name the filter modal's submit in the test ([32d3d49](https://git.datacontroller.io/dc/dc/commit/32d3d49b59fd158c5790c9e1a4dd3cfc4f532877))
* **client:** strip only this grid's own wrapper, and pin the re-strip after a data load ([f282d1a](https://git.datacontroller.io/dc/dc/commit/f282d1a44bfe458c1e17415e1e3ba8b484f61518))
* **deploy:** ACTION cannot seed by default, and the probe stops overwriting ([e84cfaa](https://git.datacontroller.io/dc/dc/commit/e84cfaaabdf6fb91b0d58524f8a1ad7a4d2673f3))
* **deploy:** ask for the admin group before the staging directory on SASjs ([5c64691](https://git.datacontroller.io/dc/dc/commit/5c646913e453699ee8d5e7aedcae8b92b125ebde))
* **deploy:** await nothing, and encode the values in the makedata URL ([d67a0e5](https://git.datacontroller.io/dc/dc/commit/d67a0e5d5f06eac2d3aba220f43eb0c7f9a2e4af))
* **deploy:** drop the deprecated service-pack deploy from the setup screen ([59e09e2](https://git.datacontroller.io/dc/dc/commit/59e09e26a5f4432fbfb397287f1fdd4f656d761e))
* **deploy:** gate the browse actions on the settings file ([5eb8492](https://git.datacontroller.io/dc/dc/commit/5eb8492742a6ede252bc4f40dd33932700792718))
* **deploy:** make the folder picker work on both estates ([fb38309](https://git.datacontroller.io/dc/dc/commit/fb38309e98a6455ffe1d2cc501d5c1f9e0a694ba))
* **deploy:** match the documented SYSSCPL set and stop guessing at the cap ([5f338ff](https://git.datacontroller.io/dc/dc/commit/5f338ff43754ef58ba45c81d9929115c3f4715a3))
* **deploy:** read a URL-supplied ACTION before defaulting, and show locked folders ([addb4a5](https://git.datacontroller.io/dc/dc/commit/addb4a56ea4e7cac2bf92956fcc30b593bba7eee))
* **deploy:** stop the makedata poll outliving its screen, and align the release gates ([c808c41](https://git.datacontroller.io/dc/dc/commit/c808c41333a1dd46008c38fe60169585e03bfe68)), closes [#407](https://git.datacontroller.io/dc/dc/issues/407)
* **deploy:** unclamp the folder picker row and probe in the chosen context ([1330925](https://git.datacontroller.io/dc/dc/commit/133092502cf4832ab9df7472b690e646ed0765ff))
* **deps:** bump @sasjs/core to 7.0.1 so special missings can be restored ([3d2eefd](https://git.datacontroller.io/dc/dc/commit/3d2eefdd7b815e8dcff7989393a6055468859bbd)), closes [sasjs/core#499](https://git.datacontroller.io/sasjs/core/issues/499)
* **deps:** bump @sasjs/core to 7.1.3 so a filter abort names the clause ([c7c1815](https://git.datacontroller.io/dc/dc/commit/c7c181518cbdb209fc73830bb791e87186bcfc20))
* **editor:** a container resize must not clear an active sort ([5bd7b5d](https://git.datacontroller.io/dc/dc/commit/5bd7b5d724db1590d211c25fa6b5cd3568ead36c)), closes [#246](https://git.datacontroller.io/dc/dc/issues/246) [#246](https://git.datacontroller.io/dc/dc/issues/246)
* **editor:** drop the stale case hint, and narrow the strip to our own form ([bd2739c](https://git.datacontroller.io/dc/dc/commit/bd2739cde41f4e2132059d04b75f0e9ed4cf33d9))
* **editor:** one predicate for the CASE marker, and no stacking in the tooltip ([18b0546](https://git.datacontroller.io/dc/dc/commit/18b0546dcbe647a5a64209535c04132c4c68bd17))
* **embed:** a rejected mapping must not block the page ([d845b9e](https://git.datacontroller.io/dc/dc/commit/d845b9e9b74a87be907f2455e55dbf21c7dfdd36)), closes [dc/dc#404](https://git.datacontroller.io/dc/dc/issues/404)
* **embed:** read a month-name date from VA as a date, not a character value ([30cefcf](https://git.datacontroller.io/dc/dc/commit/30cefcf2dcfd78ccb963d86b36668248fec9fa1e))
* **getgroupmembers:** widen membername to $256 for long names ([196d6c9](https://git.datacontroller.io/dc/dc/commit/196d6c931d44527250f59dd1f89db259a24774ad))
* **lint:** enable noSingleAsteriskComments and noUnusedLibnames ([9071801](https://git.datacontroller.io/dc/dc/commit/9071801ac51ac1bc726c9430389d34bddc7cd1c0)), closes [#376](https://git.datacontroller.io/dc/dc/issues/376)
* **lint:** make noBannedStrings an error and clear its 13 findings ([0b85f7b](https://git.datacontroller.io/dc/dc/commit/0b85f7bf1e59543ac195e37dd0a9e4c58b6befb6)), closes [#376](https://git.datacontroller.io/dc/dc/issues/376)
* **lint:** make noUndeclaredMacros an error, and fix what it found ([af28a79](https://git.datacontroller.io/dc/dc/commit/af28a79bd56234f17545c5bc3df4969464779264)), closes [#376](https://git.datacontroller.io/dc/dc/issues/376)
* **lint:** make noUnusedMacros an error and clear its 54 findings ([c8c0858](https://git.datacontroller.io/dc/dc/commit/c8c0858295f40ba734403ba24a4cae02b5464554)), closes [#376](https://git.datacontroller.io/dc/dc/issues/376)
* **lint:** rename the short and mixed-case helper macros, and clear the last warnings ([7da67eb](https://git.datacontroller.io/dc/dc/commit/7da67eb17bccf2e6635d01a007872294b5d1708a)), closes [#376](https://git.datacontroller.io/dc/dc/issues/376)
* **makedata:** let the browse actions live as long as the service does ([912fae4](https://git.datacontroller.io/dc/dc/commit/912fae448b55130db13b3e684ea66b846d3d51c7))
* **mocks:** 0 is a closed TX_TO, a libref boundary, and the union of the row keys ([67d9cfd](https://git.datacontroller.io/dc/dc/commit/67d9cfd1bd9eb9512ddd9d222221dcf674552d9f))
* **mocks:** abort like mp_abort, lock like mp_lockanytable, derive saslibs ([ca60208](https://git.datacontroller.io/dc/dc/commit/ca602082ffc69f4cfca2de878fbf842a57efc194))
* **mocks:** answer the deploy screen's DIRLIST and WRITECHECK actions ([e1c3c6e](https://git.datacontroller.io/dc/dc/commit/e1c3c6e8727d928a10ddd2838563d13efcf33ddc))
* **mocks:** derive startupservice's maps and timezone instead of hardcoding ([87fd46c](https://git.datacontroller.io/dc/dc/commit/87fd46c825f592927df0e172bdfc91f0d13f81ef))
* **mocks:** fold the audit window, and append MPE_AUDIT under its lock ([5ecc7be](https://git.datacontroller.io/dc/dc/commit/5ecc7bebec104ef451e3d405c4c1728e39d8c2b0))
* **mocks:** keep the OMS and path separators the captures lost ([7d9d466](https://git.datacontroller.io/dc/dc/commit/7d9d466c3e9e9863e897bd1f663664274e660153))
* **mocks:** one sasDatetimeNow, a documented TX_TO=0, and a shaped fallback export ([e53c50d](https://git.datacontroller.io/dc/dc/commit/e53c50d4e2b4b8e3b3097bd415f2e2bbf940793d))
* **mocks:** read hook columns in either case, and guard duplicates first ([349a2f3](https://git.datacontroller.io/dc/dc/commit/349a2f314cbcbf6c706e201ccb5faa3412f934f4))
* **mocks:** read the metadata member from the response, not the function's arity ([09e9f19](https://git.datacontroller.io/dc/dc/commit/09e9f192c5dbc64f2812c2bac90b61d3e1e7346a))
* **mocks:** resolve the audit table once, record lower-case md5, and report a failed audit write ([1a9f594](https://git.datacontroller.io/dc/dc/commit/1a9f59478b059ac59c284f45cdd08fa6428e430c))
* **mocks:** run approval hooks, as mpe_runhook does ([f0208bb](https://git.datacontroller.io/dc/dc/commit/f0208bb0b00cce7e6fa1e5015aa95b4286509bdb))
* **mocks:** separate a body read failure from a missing showmeta floor ([744c89b](https://git.datacontroller.io/dc/dc/commit/744c89b91972f08130797a2650daf49cce0e6eb7))
* **mpe_abortmessage:** decide once whether the payload was escaped ([af89126](https://git.datacontroller.io/dc/dc/commit/af89126acf38d9aeee320b5784df87cdd48148e8))
* **mpe_abortmessage:** read program and state by name, and trim the prefix ([68b89ca](https://git.datacontroller.io/dc/dc/commit/68b89caf63678ace2bde61414b5b4bac2d419a9d))
* **mpe_assert_abort:** anchor the cut on the whole escaped separator ([e74a7ba](https://git.datacontroller.io/dc/dc/commit/e74a7ba1153f83eb436235fa17a90826e09af81f))
* **mpe_assert_abort:** compare the hook's message, not the log extract ([8b426c2](https://git.datacontroller.io/dc/dc/commit/8b426c27fd37a071a3df0417bbda3e1ec2edc291))
* **mpe_assert_abort:** compare the unescaped message, cut at the last marker ([e68d221](https://git.datacontroller.io/dc/dc/commit/e68d221354a3ab94d2975ca0949dee7b4f0c4137))
* **mpe_assert_abort:** cut at the separator, not the last marker ([4c7e869](https://git.datacontroller.io/dc/dc/commit/4c7e8693698ef7da6dd94e053cb6183a536004a3))
* **mpe_assert_abort:** name the separator and tolerate a carriage return ([7b09ade](https://git.datacontroller.io/dc/dc/commit/7b09ade97c4283fcff49a3e3d3b42a88085e575d))
* **mpe_assert_abort:** size the capture buffer for the whole message ([5d32de9](https://git.datacontroller.io/dc/dc/commit/5d32de9df97208e87e9baf7b60113ea155264510))
* **mpe_assert_abort:** strip the separator tail, not every trailing escape ([b29bc1e](https://git.datacontroller.io/dc/dc/commit/b29bc1e7ae6fa0c166e0649d96df1bc9aed9d54c))
* **mpe_execute:** assemble the abort text in a macro that can be tested ([e2e9103](https://git.datacontroller.io/dc/dc/commit/e2e91039cbf67cdc4306d4abb9d1db000c8ca1a3))
* **mpe_execute:** keep a child abort message intact when reporting it ([ebfd395](https://git.datacontroller.io/dc/dc/commit/ebfd39582fe6abe7249ce696cd79ff2a8962d53b))
* **mpe_execute:** keep the child message in the fallback, and test the doubled separator ([11587e2](https://git.datacontroller.io/dc/dc/commit/11587e2f2433fc68fc9e382b7fd3c3136a4edd34))
* **mpe_execute:** make the abort-text fallback reachable ([24b8cbe](https://git.datacontroller.io/dc/dc/commit/24b8cbe9dc3dc41a7fdd75d36de09dc7d7e4ca17))
* **mpe_execute:** mask a child abort message before it aborts the caller ([20dbf00](https://git.datacontroller.io/dc/dc/commit/20dbf00fb0b468549330cee049ff17bcdd99d5ea))
* **mpe_execute:** one symbol for the abort text, and no values in the call ([700b906](https://git.datacontroller.io/dc/dc/commit/700b906d69a20a73165335c8acbeb95f70dc1fea))
* **multi-dataset:** a sheet that failed to extract is not a clean finish ([3129648](https://git.datacontroller.io/dc/dc/commit/312964864ef5d908e181610245c46cd2e78b86cb))
* **multi-dataset:** show the file size and loading stage while the file loads ([45d6c02](https://git.datacontroller.io/dc/dc/commit/45d6c025c94946b7543f88ebaf92baae341d8a10)), closes [#121](https://git.datacontroller.io/dc/dc/issues/121)
* **report:** treat a blank PDF footer parameter as its default ([0f729d7](https://git.datacontroller.io/dc/dc/commit/0f729d7101ad14d556f7d356ffeb5366072e6be5))
* **test:** let the dirlist and writecheck results reach the caller on Viya ([ed678f0](https://git.datacontroller.io/dc/dc/commit/ed678f0b4616da0cef7f61b5e8b890cbdc26e912))
* **test:** run metanav.cy.ts in CI and open a metadata object ([02a8590](https://git.datacontroller.io/dc/dc/commit/02a8590310afce6c43a9218b67824cc6ac988710))
* **va-host:** bound the early replay even when the frame is not postable ([f6d5a04](https://git.datacontroller.io/dc/dc/commit/f6d5a0495e922ef3f3c14b0c291e4f8301a865ba))
* **va-host:** deliver in load mode, honour dataType, and survive odd prompt names ([7d4155d](https://git.datacontroller.io/dc/dc/commit/7d4155dc6ca0f3ec788a9e33719e9e29fe82d470))
* **va-host:** deliver the load-mount message once, and document report.json from the file ([90784b3](https://git.datacontroller.io/dc/dc/commit/90784b342f82d480fd67fdbcad5f65e8cd9c59e0))
* **va-host:** keep the deploy password off argv, and verify the app is served ([7c63a5c](https://git.datacontroller.io/dc/dc/commit/7c63a5c93f6efb64afa13e183fd2e8e91ff42679))
* **va-host:** post to the frame's own origin, and key the replay to the document ([9cec292](https://git.datacontroller.io/dc/dc/commit/9cec292681fbc8d331dc7e9ad063c350529de1a3))
* **viewer:** clamp over-tall PDF rows, always emit an otherwise-empty page ([10229d5](https://git.datacontroller.io/dc/dc/commit/10229d5efe3f1fa652cb4c168d40045bd3788771))
* **viewer:** defer the blob URL revoke, reject a selection inside skipped columns ([b5bb704](https://git.datacontroller.io/dc/dc/commit/b5bb704a4fb346c71c7159814b7e9bdc246e80a0))
* **viewer:** keep the PDF export working where hiddenColumns is not enabled ([f84d4ca](https://git.datacontroller.io/dc/dc/commit/f84d4ca7af55b577c644db28b0c375cc0ea4e9b5))
* **viewer:** review findings - CSV skipped column, PDF row cap before reading ([2c4be5e](https://git.datacontroller.io/dc/dc/commit/2c4be5e94b0f4590e96deac72bbaac36d5297784))
### Features
* **admin:** find and test a writeable location during deploy ([442c595](https://git.datacontroller.io/dc/dc/commit/442c595c2bb5983ba177d35b49d37c31a8a433d9))
* **deploy:** pick and validate the DC location from the deploy screens ([6eeab2e](https://git.datacontroller.io/dc/dc/commit/6eeab2ed16655c7e57ad4b8d3d678832fd225d14))
* **editor:** convert CASE rule values on entry ([c4eef49](https://git.datacontroller.io/dc/dc/commit/c4eef498b22af47258cd5cfac317e2020c8b7e31))
* **logs:** read the logs modal with the shared log parser ([aabc578](https://git.datacontroller.io/dc/dc/commit/aabc578d5ec787837450e4a12ade307b7cda08fc))
* **mocks:** add the control-table services the screens call ([a1615ee](https://git.datacontroller.io/dc/dc/commit/a1615ee8a711ccc4a4ad8a7b7638b3fd6a26b18e))
* **mocks:** build every JS stored program's response with webout() ([ce6bb5d](https://git.datacontroller.io/dc/dc/commit/ce6bb5d4cde50d47d2a5f7d2d3066ff086172603))
* **mocks:** emit the lineage, usernav and libinfo responses through webout() ([1f54fad](https://git.datacontroller.io/dc/dc/commit/1f54fad3fe7d0a42387f7e5e9fa661409f892a86))
* **mocks:** emit the metanav responses through webout() ([e1d877f](https://git.datacontroller.io/dc/dc/commit/e1d877f21d0882ee2220ce143583d2d6fb080cf8))
* **mocks:** emit the sas9 services through webout() ([7be3f84](https://git.datacontroller.io/dc/dc/commit/7be3f8407384babdf59979e91f706b5e8025579a)), closes [#461](https://git.datacontroller.io/dc/dc/issues/461)
* **mocks:** restore a load, and record the diffs it needs ([1a60c68](https://git.datacontroller.io/dc/dc/commit/1a60c689f4a1b7b3b374829a669638fc493a899f))
* **mocks:** use the server's webout() in the JS stored programs ([ccb67ed](https://git.datacontroller.io/dc/dc/commit/ccb67ed5a0981a9fb36f44214e364c08b3384e5c))
* **va-host:** emulate a VA report that embeds DC, as a deployable AppStream app ([2185db9](https://git.datacontroller.io/dc/dc/commit/2185db911d0333ef006cca7dc7a0f3d6aa7884a6))
* **viewer:** download a server-side PDF report ([67ca129](https://git.datacontroller.io/dc/dc/commit/67ca129c4db3eaead56a287811c578634ff59b19))
* **viewer:** export the grid as PDF from the context menu ([e58e735](https://git.datacontroller.io/dc/dc/commit/e58e7356103df350cf8ee1dd6e9eb59c8db57dd4))
### Reverts
* keep the helpers nested inside makedata_body ([4424171](https://git.datacontroller.io/dc/dc/commit/4424171b2f1cc89a7235c415557c72e0b9e7148a))
* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
### Features
* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](https://git.datacontroller.io/dc/dc/commit/888873421c3aeaf312d5ec45e3c805317c66976a))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](https://git.datacontroller.io/dc/dc/commit/dd482c4069202a0c6b9983129c19efc09bd160a7))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](https://git.datacontroller.io/dc/dc/commit/25ffa71adddba5e334f4a8c166f284d1be498fcd))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](https://git.datacontroller.io/dc/dc/commit/c5808a16eba0d29562fcc76f13ac4a33ab8da0b7))
* **ci:** rebuild and upload release assets on demand ([6486c80](https://git.datacontroller.io/dc/dc/commit/6486c80ed4f368bbf116c0ecdecc0e5e5f02075f))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](https://git.datacontroller.io/dc/dc/commit/af1813f829861df41c89b512c824e8d3ebac7a4b))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](https://git.datacontroller.io/dc/dc/commit/cf7b21a8b61ec3bf7ebe21968ccf13ade264bc64))
* **cypress:** hold download out too, and retry once in CI ([c97e886](https://git.datacontroller.io/dc/dc/commit/c97e8866db9ee404b2118d24baf53fc7058debef)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395) [#398](https://git.datacontroller.io/dc/dc/issues/398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](https://git.datacontroller.io/dc/dc/commit/c803773d3c36476d8263f01fc3b8344aa3503d44)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](https://git.datacontroller.io/dc/dc/commit/80e8911af8857fa2c417f028cd59b64a87233b9e))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](https://git.datacontroller.io/dc/dc/commit/e51953ca8a6734e1649bad68cf4ad2057d677334))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](https://git.datacontroller.io/dc/dc/commit/459ff2da371375e996839d69deb40f681c356f1d)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](https://git.datacontroller.io/dc/dc/commit/f7e753863b61d8545bd0777d80267661b3c58670))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](https://git.datacontroller.io/dc/dc/commit/7e3045a21b67053b76a81bfe836e31571ccbffbf))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](https://git.datacontroller.io/dc/dc/commit/2623a3fb4ba3bf19c00636b1717b53d5a2d54f3a))
* **embed:** log a parse-time mapping problem once ([0011f7e](https://git.datacontroller.io/dc/dc/commit/0011f7e933d85be8e33debe3a6ecb9000edaabe3))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](https://git.datacontroller.io/dc/dc/commit/541a61ba08bc7256a911c50de7078859cc46c84e))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](https://git.datacontroller.io/dc/dc/commit/e9216b08956d5599662259a2b05cc1524c458df5))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](https://git.datacontroller.io/dc/dc/commit/bd3c220af2678a2a3f8394fe3084ee03c23d0b15))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](https://git.datacontroller.io/dc/dc/commit/f83dc359e258f5d1cb9fc49f0eaab9748995d823))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](https://git.datacontroller.io/dc/dc/commit/aa91daf9f78c00cea486d0f885f8e634366bd1a0))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](https://git.datacontroller.io/dc/dc/commit/10157c11595685efdc206fe7b16a5cafa098f26d))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](https://git.datacontroller.io/dc/dc/commit/411f6c3354b7fbfec11fef3b174ae1f8066c92ba))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](https://git.datacontroller.io/dc/dc/commit/c0ea810766ebfda5ae5358ab14146fbcc07fcc8e))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](https://git.datacontroller.io/dc/dc/commit/15a1f34d5263503cac83b4fe4e52482777dc2f1f))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](https://git.datacontroller.io/dc/dc/commit/88010c3980c2ea1c39bbb024ed32d63e495e61f0))
* address the review findings on the harness and refresh changes ([36e4e88](https://git.datacontroller.io/dc/dc/commit/36e4e88eb134d3b48217d6dd185ea3c03f03fba5))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](https://git.datacontroller.io/dc/dc/commit/698dea6d6b7a8635313f11acfcc7a4f39a02bc99))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](https://git.datacontroller.io/dc/dc/commit/0d58be975d8eb2d7f1ab6a648f760151392c7973))
* **client:** open the SAS log page with noopener ([072a133](https://git.datacontroller.io/dc/dc/commit/072a1337020970b5b4ff8aa31719cc5ffec7701e))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](https://git.datacontroller.io/dc/dc/commit/1fb4818d215f538a0ac51772ab1f261aeb91f7e1))
* **cypress:** apply the licence key when the estate has none ([be8f445](https://git.datacontroller.io/dc/dc/commit/be8f445ae1051e132fb6eb5a69b45264255326b5)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](https://git.datacontroller.io/dc/dc/commit/fed43cb07ef7f2afc685612a124dbc16d7618e21))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](https://git.datacontroller.io/dc/dc/commit/f5aade59760cc376b0f96eb4c892c8bca82dde14))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](https://git.datacontroller.io/dc/dc/commit/1e224763ba80500a112812a983801968e3751959))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](https://git.datacontroller.io/dc/dc/commit/9b3cf50b9c3889786605f0f5d9032d84c50f7107))
* **derived:** address five review findings ([ccd468b](https://git.datacontroller.io/dc/dc/commit/ccd468b3358c1418ee6d8e198b088a0cf23b8c24))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](https://git.datacontroller.io/dc/dc/commit/be50cb5d082e788c6ae106bc1d02e73a8c7b9646))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](https://git.datacontroller.io/dc/dc/commit/9914790777f92128c74ecce67e6c67ee48824b45))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](https://git.datacontroller.io/dc/dc/commit/9ef813d439b48e7fefd35c426e62b99e5943002a))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](https://git.datacontroller.io/dc/dc/commit/422f766cd43db4ce740e1890fd5df30790eaa7e3))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](https://git.datacontroller.io/dc/dc/commit/7830b7bcfa9806dde840ce8dd3e6a1887d0b38be))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](https://git.datacontroller.io/dc/dc/commit/2d5de9a324a07a36edc0a4f74c8fad09fde07cbf))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](https://git.datacontroller.io/dc/dc/commit/6a1128a46352f77c95dd02df9309f0984a1db2d1))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](https://git.datacontroller.io/dc/dc/commit/1863076bc956b4ff94bf7b85b3dde0d6e3b54d7a)), closes [#366](https://git.datacontroller.io/dc/dc/issues/366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](https://git.datacontroller.io/dc/dc/commit/d3dec682aafeb143fe983d6546337cb6490f5620))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](https://git.datacontroller.io/dc/dc/commit/469229aecbaad304a835a5f229df111c37ff7c0d))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](https://git.datacontroller.io/dc/dc/commit/7fb981daa91bce29c2d62c246185d01dcad08d6b))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](https://git.datacontroller.io/dc/dc/commit/5f5d8681b9d78280a5200309917415e135c3ce5b))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](https://git.datacontroller.io/dc/dc/commit/5f443b0b16a2abd3c11d96caa7a043911c29ef01))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](https://git.datacontroller.io/dc/dc/commit/6b5bf3e6be89b22e5a633c34b9aecbc8c7d48ee2))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](https://git.datacontroller.io/dc/dc/commit/85c89ca2a4e02112ba496ac36a222f33675544da))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](https://git.datacontroller.io/dc/dc/commit/f8f65681e1489f4ce94d9127d5e4c0c82798e960))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](https://git.datacontroller.io/dc/dc/commit/7efbf31ea5ac234bd8f924fdf89a95d101b3aff1))
* **derived:** only the delete marker is a reserved name ([af31c36](https://git.datacontroller.io/dc/dc/commit/af31c36b670ec63a2a4ba056bf6c4b28f6096e08))
* **derived:** open the statements file with a long record length ([ee31782](https://git.datacontroller.io/dc/dc/commit/ee31782fbff909b540162b1bdfca6f307208331b))
* **derived:** pair the filterds %else with %do/%end ([88a38db](https://git.datacontroller.io/dc/dc/commit/88a38db877dba9b98ccf50f0cc7242674654586b))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](https://git.datacontroller.io/dc/dc/commit/27ddcfb5759d6eb8f9880adc7c271a408e7741ad))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](https://git.datacontroller.io/dc/dc/commit/7799a300df82339683eea21e4a89ce7a6471c575))
* **derived:** reject a dependency on a row this submission deletes ([8806504](https://git.datacontroller.io/dc/dc/commit/88065049674c2cc0141eef22704b759f5b708b0f))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](https://git.datacontroller.io/dc/dc/commit/9a047ad42ec86df7d1eaf5058c0edde56016e88f))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](https://git.datacontroller.io/dc/dc/commit/e0da9f1098db63185538398ca478696cc849da4b))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](https://git.datacontroller.io/dc/dc/commit/688550a28d9794ac6b878926a0e1e2ef8f6e9335))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](https://git.datacontroller.io/dc/dc/commit/1b5c4b3ecd962a28e08fc5671b57dbf7ae0b8d9f))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](https://git.datacontroller.io/dc/dc/commit/4d9ad38df1fa89b0d5a12214d703d9c1fe5295f2))
* **editor:** a blank truncation note is not a warning ([f182179](https://git.datacontroller.io/dc/dc/commit/f18217940365f27ae388fae087e6126931e1222c))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](https://git.datacontroller.io/dc/dc/commit/04ce1861e7821063b1be42c22f92f62aed509602))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](https://git.datacontroller.io/dc/dc/commit/4bcee706ffaf1929bdd14bb2fb15111174ca3916))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](https://git.datacontroller.io/dc/dc/commit/45dce7b11b7a4399c1d0dc9c3e7a283b1cf59df1))
* **getdata:** clear the globals it declares before use ([359b570](https://git.datacontroller.io/dc/dc/commit/359b570b85aa2f66507e81ea2977243be4590f83))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](https://git.datacontroller.io/dc/dc/commit/059cdf176a7ec556b45ea1b21d880c048eba14ca))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](https://git.datacontroller.io/dc/dc/commit/661ee19cfa5f996bf01f2b11817039f38dbb9985))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](https://git.datacontroller.io/dc/dc/commit/355c46a7b64994d0146a88657dc5635593249138))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](https://git.datacontroller.io/dc/dc/commit/d626cb8f2daf94fde7ff21d924b9a2ca4968d983))
* **mocks:** address the audit-pack review findings ([489819b](https://git.datacontroller.io/dc/dc/commit/489819b05111323ae1e34edda22a5446cdbc2d87))
* **mocks:** carry the zip flags into the central directory ([c4021a4](https://git.datacontroller.io/dc/dc/commit/c4021a42ffaed2af80c53aaa1455249190b339b8))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](https://git.datacontroller.io/dc/dc/commit/62f55db170185900063f7a431cd8f375ebabf930))
* **mocks:** copy the physical column list before deriving ([f411b7d](https://git.datacontroller.io/dc/dc/commit/f411b7d269381e3f7cab2d85b6d8f7d7d50ea4d1))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](https://git.datacontroller.io/dc/dc/commit/60ec470404e5c279b58ac6a01768dfc83413b197))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](https://git.datacontroller.io/dc/dc/commit/74060c0585fb8b217cb181ea747ad318f5238868))
* **mocks:** drop derived variables before staging ([73d25b1](https://git.datacontroller.io/dc/dc/commit/73d25b1220d574ef2547fbd9e6e3cf1ceda375ee))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](https://git.datacontroller.io/dc/dc/commit/347dbb53b01c47bbd562de14d5ccbae7d96407c6))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](https://git.datacontroller.io/dc/dc/commit/ee3c8787efef9cd716bdd2bf80f6cb6751461db9))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](https://git.datacontroller.io/dc/dc/commit/6cad23fce9ac0cf64c247c40f3b7eac875fd4cfb))
* **mocks:** search the physical columns only in the viewer ([293022d](https://git.datacontroller.io/dc/dc/commit/293022d46b27e096fd514cdda49f688612df2f5b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](https://git.datacontroller.io/dc/dc/commit/fb36379520028baaf4bc3f08131bfc88deab3ac3))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](https://git.datacontroller.io/dc/dc/commit/9c294c0ab5ca8fe30a235553a298c1c68a61442f))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](https://git.datacontroller.io/dc/dc/commit/0a52d20c5fc27b4adde5fc8ed6f39a58c48f62ff))
* **mpe_execute:** fail fast when the child left no status ([a036386](https://git.datacontroller.io/dc/dc/commit/a03638667f1968c5909f496d954eeaa13f316763))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](https://git.datacontroller.io/dc/dc/commit/220408538b26016bda45e48d619cdbc3e1cbcc59))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](https://git.datacontroller.io/dc/dc/commit/f2b1428ec3099b2b6e3c6da6627530ef81fab201))
* name the startupservice launcher button "Show log" ([3445107](https://git.datacontroller.io/dc/dc/commit/3445107d9a91642b67e114bcfcd81c3ce47f3f0f))
* offer the launcher on the globvars failure modal too ([ad2da45](https://git.datacontroller.io/dc/dc/commit/ad2da453896a51f2cc7e49cebb6c84f7b47d7395))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](https://git.datacontroller.io/dc/dc/commit/46bacdd6d9ea814bcd02a1ff8aa3f447baf02f64))
* **refresh:** one audit folder per refresh call ([56b6620](https://git.datacontroller.io/dc/dc/commit/56b6620ef402b288ce78a2a77662b651e310753e))
* **sites:** round-trip the committed licence before publishing ([4d462d3](https://git.datacontroller.io/dc/dc/commit/4d462d310f7df23e5d5229cb0702721b7ad41bbd))
* **test:** getdata test ([da30266](https://git.datacontroller.io/dc/dc/commit/da30266278c4fe27a72810cd5ed647cf3bd823af))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](https://git.datacontroller.io/dc/dc/commit/033ffd302502c7530fc44a314364f5cdb3be6ad2))
* **ui:** one error modal at a time ([81ccf75](https://git.datacontroller.io/dc/dc/commit/81ccf758e9dab0c7632ab50bf469e88b8c4c4420))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](https://git.datacontroller.io/dc/dc/commit/42355f17bc07bf4d44ba858984d83b9ed72cc0ba))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](https://git.datacontroller.io/dc/dc/commit/337ea8e5e021c6f2451fa3ad8396148744118fc9))
### Features
* **dc_getlibs:** dump the library list when debug is on ([582523d](https://git.datacontroller.io/dc/dc/commit/582523dc87a253b71824359224070ccb4746a386))
* **derived:** allow a variable to shadow a physical column ([12beeff](https://git.datacontroller.io/dc/dc/commit/12beeff69518376a042fbbc58332acc40d89d132))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](https://git.datacontroller.io/dc/dc/commit/813bce685691ad252043eeab31eb5460de81b756))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](https://git.datacontroller.io/dc/dc/commit/7c9ce9a712137e9825a5f2ff52796d765423147b))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](https://git.datacontroller.io/dc/dc/commit/87e794f769fe420f214f43b141f0b41cf88eefee))
* **derived:** report the dependency cycle path ([cee1a95](https://git.datacontroller.io/dc/dc/commit/cee1a954ee97e17f297b7cf43ef0e5cfc6e393a6))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](https://git.datacontroller.io/dc/dc/commit/78e88a41128b047c0276ccf647f128229152604c))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](https://git.datacontroller.io/dc/dc/commit/0b010f2c65ddada70d62ea83c706943c6c45756f))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](https://git.datacontroller.io/dc/dc/commit/aa6d50e1ea5fc2fe6be8aa9fa859b92d0a1061c5))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](https://git.datacontroller.io/dc/dc/commit/33d043ea92f007dca70c95d5363329d4447eeba9))
* **editor:** derived variables in the EDIT screen ([b0fd354](https://git.datacontroller.io/dc/dc/commit/b0fd354ecdbd7bf29196ed983ff3cce691affeed))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](https://git.datacontroller.io/dc/dc/commit/6513d09f059de15c5304f99b3f8d898d46ea1be7))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](https://git.datacontroller.io/dc/dc/commit/34691dceecb63f952b23ad8acc5bc93753c6cfee))
* **editor:** show the derived column label in the header tooltip ([cfe6647](https://git.datacontroller.io/dc/dc/commit/cfe6647ca870c45a3d8abc2753ce7f070e3fdf4a))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](https://git.datacontroller.io/dc/dc/commit/752d944c2091ef6d2624a15ab2e2e7be35608798))
* **mocks:** implement derived variables in the JS services ([b4fae88](https://git.datacontroller.io/dc/dc/commit/b4fae88ae13658607e4209449577cd42849d3a99))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](https://git.datacontroller.io/dc/dc/commit/5d183d135085ea023f32e5b1d4992d997394a8fb))
* run startupservice in a new window from its error modal ([40f6133](https://git.datacontroller.io/dc/dc/commit/40f61331aad6bc8a52c551b924f0d1f1e6426cee))
* **sites:** mint a licence for the published sites ([7d9df71](https://git.datacontroller.io/dc/dc/commit/7d9df71fdb0ab265e745ed3b708038983e855a87))
### Reverts
* **derived:** no shadowing, and no self reference ([5af8e6f](https://git.datacontroller.io/dc/dc/commit/5af8e6fc359221586af74a685159dfd75a40ec45))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](https://git.datacontroller.io/dc/dc/commit/31aa8591538409fc1a3ef97d7474120094ce202d))
@@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output
-`client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
-`sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
-`.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
-`.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).
The licence key comes from the backend licence: the decrypted licence carries `hot_license_key`,
@@ -108,6 +131,13 @@ sasjs cbd -t server-ci
This compiles, builds, and deploys the JS mock services plus the Angular frontend (as a streaming app) to the SASjs Server. The `server-ci` target in `sas/mocks/sasjs/sasjsconfig.json` points to `http://localhost:5000` with appLoc `/Public/app/dc` and stream service name `clickme`.
The drive still needs seeding - the control library first, then the Cypress fixtures:
After deploy, the streamed app is available at: `http://localhost:5000/AppStream/clickme/`
### 3. Run the Angular dev server (optional, for hot reload)
@@ -124,11 +154,13 @@ The app will be at `http://localhost:4200`. It calls the JS mock services on por
### How the mocks work
Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). The JS runtime predeclares `fs` and`_webout`; the mock assigns a JSON string to`_webout` that matches the structure the real SAS service would return. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). `admin/createdemodata.js` is the exception: it has no SAS counterpart, because it exists only to give the test estate tables a deployed site would not have. The JS runtime predeclares `fs`, `_program` (the program's deployed path),`weboutPath` (the response body it writes) and `_webout`, and the services emit their members through the runtime's `webout()` - `webout('OPEN')`, one `webout('OBJ', name, rows)` (or `webout('ARR', name, rows)` for positional rows) per member,`webout('CLOSE')` - so the response envelope comes from the host and the file carries only what the SAS service returns. A few services still write the body themselves through `_webout`: the ones that emit plain text, CSV, DDL, HTML or a file download and set their own content headers (`admin/exportconfig`, `admin/exportdb`, `admin/makedata`, `auditors/getauditfile`, `public/getddl`, `public/getrawdata`). `sas/mocks/test/sasjs-webout.test.js` holds that list, so a service that goes back to writing its own body fails the suite. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
The mock data includes sample tables (`DC996664.MPE_X_TEST`, `MPE_X_NEW`, `MPE_X_FORMULA_TEST`, etc.) with realistic columns, validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT), and dropdown values. This is sufficient to exercise the full editor grid, approval workflow, viewer, and admin screens.
`admin/makedata` builds only what the SAS service builds: the control library defined by `mpe_makedatamodel.sas`, seeded with the rows `mpe_makedata.sas` inserts. `sas/mocks/test/makedata-parity.test.js` reads both macros and fails when the mock estate drifts from them, so the mock and a deployed site open with the same tables and the same configuration.
There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`).
The fixtures the Cypress suite needs but a DC install does not have live in `admin/createdemodata`, which runs after makedata: the `TESTDATA` demo library (`MPE_X_NEW`, `MPE_X_FORMULA_TEST`, `DERIVED_DEMO`, etc.) with its validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT) and dropdown values, the `MPE_X_SEARCH` and `MPE_X_WIDE` viewer fixtures, and the `DC996664` multi-load library. It is safe to run twice, and refuses to run before makedata.
There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`). Its services are fixed recordings of the SAS 9 responses, emitted through `webout()` in the same way; the payloads are the SAS 9 ones (metadata URIs, `BUILTIN\Administrators` owners, `D:\...` paths, with their separators - the front end splits on `\`, so a value that arrives without it exercises the wrong branch). Only the payloads are SAS 9 flavour: the envelope around them (`_PROGRAM`, `SYSUSERID`, `SYSVLONG`, `SYSDATE`/`SYSTIME`) is whatever runtime serves the request, so do not read this variant for SAS 9 envelope values.
### Re-deploying after changes
@@ -136,7 +168,7 @@ After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sa
### Demo and staging sites
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs `makedata` on a throwaway SASjs Server first and ships that drive as `sasjsbuild/mock-snapshot.json`; without it the site opens with no data. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. The drive the site ships with is captured from a throwaway SASjs Server as `sasjsbuild/mock-snapshot.json`, and the two sites ship different ones: the demo runs `makedata` on that server first, so it opens with the control library and the sample tables; staging publishes without running it, so it opens on the deploy screen (DC routes there while `makedata` is still on the drive) and the visitor deploys the control library from the wizard. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
`VA parameter "${paramName}" maps to ${target.column}:${spec}, which is a range, but the parameter has ${sent.length} values - a range needs one value, so the filter was ignored.`
)
continue
}
if(usable.length<sent.length){
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${sent.length-usable.length} of its ${sent.length} values was not a date - the filter covers only the rest.`
Loaded 100 of 431 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
dc
allan