An override exists to stop a later install moving the package it pins, so a
caret range defeats it: the lockfile records 11.0.3 and the range would allow
11.0.4 to arrive without review. Exact version, and the lockfile is unchanged -
it already resolved 11.0.3.
With follow-redirects moved, the client audit still failed on minimatch
<11.0.1 (globparts amplification), reached through
exceljs -> archiver -> readdir-glob, which asks for ^10.2.2.
readdir-glob already carries an override block here for brace-expansion, so the
fix follows that shape rather than a breaking exceljs change: minimatch
^11.0.1 nested under readdir-glob. package-lock gains 23 lines, the override
line is the only package.json change.
Verified: npm audit --omit=dev exits 0 in client; ng build succeeds; an
exceljs xlsx write succeeds; and archiver ZipArchive.directory() - the caller
that walks the tree through readdir-glob - produces a valid zip.
The audit runs in root, sas and client, and sas carries its own lockfile - the
same 1.16.1 was resolving there, which is why the first attempt left CI red.
Same one-package update, no pin change.
`npm audit --omit=dev` in sas is now clean.
The audit gate fails on any runtime advisory, and this one arrived upstream:
follow-redirects 1.16.0-1.16.1 discards cross-realm sensitiveHeaders arrays and
lets custom credentials follow redirects. 1.16.2 is outside that range and inside
the ranges the tree already asks for, so this is a lockfile-only update - no pin
in package.json changes.
Regenerated with `npm update follow-redirects --package-lock-only`, which moves
that one package rather than re-resolving every floating range (an
`npm audit fix` run also bumped the whole @angular-devkit family, which is not
part of this fix).
`npm audit --omit=dev` in client: the high severity entry is gone. Four moderate
advisories remain, all through exceljs -> archiver -> readdir-glob -> minimatch,
and they need a breaking exceljs change; they are pre-existing and not touched
here.
Test 8 submits a VAR_LOGIC expression that the derived-variable scan refuses, so
the child is expected to abort and the assertion reads its message from the
response. Without failfast=0 mpe_execute aborted this program instead, so the
case could never reach its assertion. The file's other abort-expecting calls
already pass failfast=0.
This does not make the case pass on its own - the child still cancels without
writing a response, which is tracked separately - but it removes the wrong
failure in front of it.
mpe_execute reports a child that did not complete by aborting the caller with
"&program did not complete: state=&state &msg". The message was passed through
%str, which masks the literal text before resolution - so the commas in a
child's own abort message (mp_abort writes them as part of the reason) were
still delimiters when the call was parsed. The call split into extra positional
parameters and SAS raised
ERROR: All positional parameters must precede keyword parameters.
replacing the real failure with a parse error at the call site. The caller then
lost the child's message entirely.
%bquote masks the resolved value, so the message arrives as one argument and the
abort reports what the child actually said.
The same shape in mpe_security_postedit, where the submitted ACCESS_LEVEL value
is interpolated into the abort message, is fixed the same way.
Addresses both findings from the review of #408.
runAutoDeploy was declared async after the service-pack step went, but calls
createDatabase() and returns without awaiting anything. createDatabase() starts
the request and reports through the component's own status flags rather than a
returned promise, so there is nothing to await - drop the async rather than
pretend to a contract the callee does not offer.
runMakedataInNewWindow interpolated the context name, admin group and DC path
straight into the query string. An ampersand in any of them started a new
parameter and a space arrived truncated; a DC path containing a slash was only
working because nothing decoded the query. Those three values are now
percent-encoded at the build site, so the server reads back what the user chose.
Tests: the new-window case now asserts the encoded forms, and a second case
pins the values that need encoding (context with an ampersand and spaces, admin
group with a space and a plus, DC path with a slash and a space). The deploy
case no longer awaits a synchronous handler.
Executed 24 of 24 SUCCESS (automatic + sasjs-configurator + deploy components).
Restoring any load that touched a special missing numeric (._ or .A-.Z) failed
in mp_stripdiffs, which writes the old value into a generated UPDATE statement.
PUT writes a special missing as its letter only, so the statement carried
"set SOME_BESTNUM =_" and PROC SQL read the bare underscore as a column name:
ERROR: The following columns were not found in the contributing tables: _.
ERROR: SOME_BESTNUM, a numeric column, can only be updated with a numeric
expression.
Every statement in the generated block then failed NOEXEC, so the revert
produced the wrong data; the service's syscc check is what surfaced it.
Fixed upstream in sasjs/core#499 (the period is now written first) and
released as 7.0.1. v7.0.0's breaking change - mf_existvarlist warns and returns
0 instead of aborting on blank input - does not affect DC: there are no direct
callers, and the full SAS suite covers the core macros that do use it.
Adds restore.test.2.sas, which seeds some_bestnum=._ on mpe_x_test, changes it
to a number through stagedata and approves it, reverts that load, approves the
reversion, and asserts the cell is back to ._ rather than blank. It fails on
6.1.4 and passes on 7.0.1.
The setup screen's only action is to run `makedata` with what the form
collected, and nothing tested it. Both screens now have specs pinning the
request and the URL they build:
- Viya (automatic): the admin group and DC path reach the service as
`fromjs`, the configured context is used with the web method, a context chosen
in the form wins over the configured one, the new-window URL carries
`_contextname` / `admin` / `dcpath` / `_debug`, and pressing deploy reaches
`createDatabase` at all
- SAS9 and SASjs Server (configurator): the makedata URL carries `_program`,
`admin`, `path` and `_debug`; SAS9 stays on the page origin and encodes the
directory, SASjs Server leaves the path readable, and the SASjs Server check
runs before any reload
The deploy button is also restored. Removing the dead upload markup took the
live button with it, so the screen had no way to start the job its own copy
promises - the new `runAutoDeploy` test would have caught that.
The Viya setup screen still carried a path that deployed the backend as a
service pack from a viya.json (`deployServicePack`). That is no longer how a
deployment works: the frontend and services arrive with the sasjs deploy, and the
only thing left to do is create the control library and its configuration data,
which `makedata` does - and then deletes itself.
The path was already unreachable from the UI (its button had been commented out
"for reference"), so the wizard only ever ran `makedata` in practice while its
copy still claimed to deploy services. Removed:
- `executeJson()` and the `viya.json` upload plumbing: `readFile`,
`clearUploadInput`, and the fields and status flag that fed them
- the "Deploy SAS Jobs" row in the completion modal, which could never show
- the commented-out button and file input that kept the dead path visible
The screen now says what it does: it runs the one-off job that creates the
control library and its configuration data, and the status row reads "Create
control library" rather than "Create database".
The deferred VA load builds the filter from the report's parameters before the
first getdata, which is what makes that first load already filtered. A va_parmap
entry is a column the report author typed, so the request can legitimately fail
when the table does not have it - validatefilter aborts on an unknown variable -
and that abort surfaced as a modal over an editor which then loaded fine
(dc/dc#404).
Validating the mapping first would cost a SAS call on every load of a mapped
object, which is not affordable, so the attempt is simply made quietly:
saveQuery takes wrapper options and the deferred load suppresses the error
modal. A mapping that is right still filters the very first load; a wrong one
falls back to the normal load, and the metadata pass reports it in the banner
and reconfigures once, at setup time rather than on every page load.
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.
No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.
parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.
A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.
Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.
No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.
parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.
A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.
Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
@sasjs/core 6.1.3 moved the derived variable field rules out of
mp_derivedvars into mp_derivedvarcheck, so the same scan can run before a
definition is used. DC carried its own copy of that list in
mpe_derived_check.sas, which is now deleted: the post edit hook calls the
core macro instead, and the hook keeps only the rules that belong to the
control table rather than the expression - LIBREF/DSN required, and ACTIVE
within the picker pair.
The scan is stricter than the copy it replaces, in three ways:
- VAR_NM may not be _N_, _ERROR_ or _IORC_ (automatic variables of the data
step)
- VAR_LOGIC may not exceed 4000 characters
- VAR_LOGIC may not call a function that submits code or reaches the file
system, the environment or the macro symbol table - a function name cannot
be assembled at run time, so this is a control the expression cannot evade
and it relaxes one: VAR_LABEL may carry a line break, which is flattened to a
space when the label is used, so only the remaining control characters are
refused.
The mock hook mirror is updated to the same rules, and stagedata.test.4
gains the two submit-time cases that pin the new wiring.
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
### Bug Fixes
* **ci:** install doxygen for the SAS docs job ([f58925b](f58925b299))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](27686d3aad))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](888873421c))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](dd482c4069))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](25ffa71add))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](c5808a16eb))
* **ci:** rebuild and upload release assets on demand ([6486c80](6486c80ed4))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](af1813f829))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](cf7b21a8b6))
* **cypress:** hold download out too, and retry once in CI ([c97e886](c97e8866db)), closes [#395](#395) [#398](#398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](c803773d3c)), closes [#395](#395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](80e8911af8))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](e51953ca8a))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](459ff2da37)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](f7e753863b))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](7e3045a21b))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](2623a3fb4b))
* **embed:** log a parse-time mapping problem once ([0011f7e](0011f7e933))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](541a61ba08))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](e9216b0895))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](bd3c220af2))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](f83dc359e2))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](aa91daf9f7))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](10157c1159))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](411f6c3354))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](c0ea810766))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](15a1f34d52))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](88010c3980))
makeDataServer.json seeded ADMIN=AllUsers, a Viya-style group that no account on
the shared estate belongs to. The seed still reports SUCCESS with a group the
caller is not in, and then every getdata/stagedata call aborts with
'<user> is not authorised to edit <libds> in the MPE_SECURITY table', which
reads like a broken deployment. The estate's accounts are in all-users, the
SASjs Server group every logged-in user has, so seed with that and describe the
rule in the estate-testing skill rather than telling the reader to override the
shipped value.
The run on release/7.20 went from four failing specs to two:
csv-limited PASS (order restored)
hook-programs PASS (order restored)
metanav held out, #395
download 7 of 7 fail
download now gets past its hooks - the deleteFolder task is registered - and
fails on the button it clicks:
AssertionError: expected '<button.btn.btn-sm.btn-success...>' to be 'visible'
This element is not visible because its ancestor has 'position: fixed' CSS
property and it is overflowed by other elements.
Something fixed-position sits over it. That needs a scrollIntoView or an overlay
dismissed, and each attempt costs a forty minute CI cycle, so it is held out with
#398 carrying the exact error and the suggested fix. The list reports 22 run, 2
excluded.
editor.cy.ts still fails one test of fifty-five, and it failed the same way in
both runs: 'Expected <div.app-loading> not to exist ... continuously found', a
load timeout on test 28 rather than an assertion. One retry in run mode absorbs
that, none interactively; a real failure still fails twice and the retry's output
is in the log.
The shared-estate cycle is two npm scripts and sasjs test, but the seed in
npm run server is load-bearing: without it testsetup 404s settings.sas and
59 of the 64 test programs fail to complete, which reads like a broken
deployment. Record the targets, the token-pair credentials, the ADMIN group
the estate actually uses, how to read the results, and the frontend
passphrase steps as a repo-local skill, and point the testing doc at it.
Four specs failed on release/7.20 once the list went from 19 to 24. Three of the
four are addressed here; the fourth is a spec that cannot pass and is now
excluded with an issue.
download.cy.ts - it clears the downloads folder in an afterEach through
cy.task('deleteFolder'), and that task was registered only in
cypress/plugins/index.js, which Cypress 10+ never loads unless it is required
from cypress.config.ts. The task is now registered in setupNodeEvents, where
Cypress looks, and the login in its beforeEach only fills the form when the form
is actually on screen - the unconditional version failed with 'the center of this
element is hidden from view' whenever the session was already live.
csv-limited.cy.ts and hook-programs.cy.ts - neither is broken; both depend on
where they sit in the run. csv-limited asserts the free-tier 'Locked Feature'
modal, which an earlier spec removes by planting a valid licence, and
hook-programs asserts an approval row against its target table before other specs
have added rows to the approval tables. The list is now the sequence that was
green for months, with csv-limited first and hook-programs second, and the newer
specs appended rather than inserted. The script says why order matters, because
the next person to alphabetise it will break the same two specs.
metanav.cy.ts - cannot pass, and says so in its own source: '// BLOCKER ...
.clr-accordion-header-button always null although it is present on the page'.
With the duplicate login in its beforeEach removed it now reaches that assertion
and fails there, so it is excluded from the list with #395 as the reason, and the
login fix is kept for whoever picks it up. The coverage check now distinguishes a
deliberate exclusion from a forgotten file, so nothing can go missing silently.
Not verified locally: the specs need an estate seeded by 'sasjs cbd -t server-ci'
plus makedata, which a scratch mock does not reproduce - the app never leaves
app-loading. The CI run is the verification.
semantic-release bumps the working-tree package.json before the release
commit, so the tag is known in the job. Using it for the asset upload and
both dispatches means an overlapping release cannot redirect them.
Only the assets this workflow uploads are removed before a re-run, so a
hand-attached asset on the release survives. jq is installed before the
tag resolve that parses its output, the SASjs CLI is installed at the
version the tag's sas/package.json pins, and the npmrc secret is appended
rather than overwriting the file.
The four shards made the release-branch run red, and the reasons are in the
logs rather than in the theory:
- every failure was a timeout, not an assertion - "cy.type() failed because
the center of this element is hidden from view", "Expected
<div.app-loading> not to exist ... continuously found". Four jobs on one
runner means four SASjs servers, four ng serve and four Chromes competing
for the same CPU, and the app under test starves;
- upload-artifact@v4 does not work on this runner at all:
"GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ ...
not supported on GHES", so the v3 -> v4 move I made on review advice broke
the artifact step outright.
Measured, from the one sharded run that did complete: 20m34s wall against 33-35
minutes for the single job, with peak concurrency 3 - so a third off, not the
four-fold the split suggests. That is not worth a flaky pipeline while a release
sits on it.
Kept from the sharding work, because those parts were the real gain:
- the spec list has one home, .gitea/scripts/cypress-specs.sh;
- Build-and-ng-test fails when a spec on disk is not listed, so the five that
had never run in CI cannot go missing again - and they do run now;
- videos are zipped and uploaded only on failure;
- one job means one server, which is also what the guards and the balance
arguments were protecting.
The shard and fan-in jobs are gone with the branch that carried them.
curl -k disables verification on calls that carry the release token, so a
man-in-the-middle on the runner's egress could capture a credential that
rewrites releases. The forge serves a publicly trusted certificate (the
runner verifies the same chain on every checkout), so the flag is
unnecessary: drop it from all fifteen calls in the two release workflows.
The standalone URL dropped every query parameter whose name started with
va_, not just va_parmap; the names are now enumerated, and a later
parameter with that prefix survives (the component spec pins the exact
standalone URL instead of a substring, so the query surviving is the
assertion, not an accident of the runner's own query).
The timezone runs are scoped to the VA specs, the only specs with local
calendar arithmetic, so the blocking job no longer runs the whole suite
twice more.
The gate could not fail. With needs: on four shards and no if: always(), a
failed shard skips the fan-in job instead of failing it, and a skipped job is
not a failure on the one check branch protection watches - so the gate was
silent in exactly the case it exists for.
It now runs with if: always() and fails on any shard result that is not
success. The count is checked as well: if the needs context does not resolve on
this runner the step fails saying so, rather than looping over nothing and
reporting green - which is the failure mode a bare loop would have.
Also removes what the last regeneration left behind: the coverage guard step
appeared twice in Build-and-ng-test (so it ran twice) and the shard 1 comment
block appeared three times. The file is regenerated from fix/va-parmap-review-572
rather than from my own previous commit, which is what caused the duplication.
Renaming the shards to cypress-shard-1..4 and adding a job named
Build-and-test-development that does nothing but needs all four.
That name is the status check the branch protection rule requires, so the check
now turns green only when every shard passed - a red shard 3 leaves it
unreported and the PR unmergeable, which is the behaviour the single job had.
No repo setting has to change, and adding a fifth shard later means adding it to
needs rather than remembering to edit branch protection, where forgetting fails
silently.
Review found that five specs ran in no shard - coltype-delete-record,
download, embed-va, metanav, submitted-details - and one of them is the VA
embed suite this release added, 454 lines that had never run in CI. The old
single --spec string omitted them too, so nothing regressed; they were simply
never wired up, and nothing in the repo marked them skipped.
The spec lists now live in .gitea/scripts/cypress-shards.sh, one place, so the
workflow no longer spells them out four times. A guard step in
Build-and-ng-test compares the shards against the specs on disk and fails when a
file belongs to none of them or to more than one - proven by removing a name and
watching it fail with that file listed.
Rebalanced for the five additions, on test count and spec count together, since
each spec pays a few seconds of startup:
shard 1 editor, csv, metanav 57 tests
shard 2 excel, filtering, download, browser-info,
excel-multi-load, hook-programs 49 tests
shard 3 embed-va, viewbox, licensing, filter-panel, liveness 43 tests
shard 4 viewer-labels, row-cell-limits, derived-variables,
ddl-export, coltype-delete-record, submitted-details,
startupservice-launcher, stage, full-table-search,
csv-limited 27 tests
24 specs / 176 tests, each in exactly one shard.
upload-artifact moved from the retired v3 to v4; this change multiplies that
step by four, so it is worth not carrying the dead major forward.
Not in this commit: the three new check names still need adding to branch
protection, which needs repo admin.
Sits on top of fix/va-parmap-review-572 (#391) so the two changes to this
workflow land in one order; the timezone step that PR adds to
Build-and-ng-test is untouched.
The development job ran all nineteen specs in one Cypress invocation on one
runner, so its wall time was the sum of every spec - around half an hour on a
green run, with the whole pipeline waiting on it.
Four jobs now, each with its own runner and its own SASjs server, taking a
subset:
shard 1 editor (55 of the suite's ~148 tests)
shard 2 excel, filtering, csv-limited, csv, stage
shard 3 viewbox, licensing, filter-panel, browser-info, excel-multi-load, liveness
shard 4 viewer-labels, row-cell-limits, derived-variables, ddl-export,
hook-programs, startupservice-launcher, full-table-search
The split is balanced on test count, not file count: editor.cy.ts alone holds 55
tests, so an even spread of files would leave one shard running twice as long as
the others. Its own duration is the floor the pipeline can reach.
The setup steps are identical in every shard, so they are defined once and
aliased with a YAML anchor; each job differs only in SHARD and SPECS. Shard 1
keeps the original job name so an existing required status check still matches.
Video is still recorded - measured at 91s with it and 92s without on the same
spec, so the recording is not what takes the time. What it did cost was the zip
and artifact upload of every run's videos, now guarded by if: failure().
dc
allan