3.12.2 is the current release and carries the packaged-executable fix for
/ddc-host (the page this job's va-ddc-host spec drives), so the version the
tests run against is the version that ships. 3.12.0 remains the floor for
`timing=early`.
Closes#449.
- SASJS_SERVER_VERSION 3.6.2 -> 3.12.1 in build.yaml. 3.12.0 is the floor:
/ddc-host with `timing=early` (the replay that lands before the app
bootstraps) arrived there, and it is absent from 3.11.3, which has no
/ddc-host at all.
- DDC_HOST_ENABLED=true in the job's .env. The server 404s /ddc-host unless it
is set - it is a testing tool and does not announce itself - so enabling the
spec without this would have failed on the first case.
- va-ddc-host moves from EXCLUDED to the END of SPECS in cypress-specs.sh: it
clears the DC_JSLIB filter tables before each case, so it must run last on the
shared mock estate. The ordering contract at the top of that file now records
that, and metanav's licence key, which no longer has to avoid being last
itself.
Review findings on the change-detection steps and the shared build artefact.
- A failed fetch or diff in "Detect client changes" used to fail the job: the
fail-safe only covered an output that was absent, not a step that errored.
Both workflows now exit 0 without writing client=false on any failure, so an
undecidable change set runs the suite instead of blocking the pull request.
- .gitea/scripts/ joins the change set in both workflows: it carries the spec
list (cypress-specs.sh) and the runner gate (check-runner-memory.sh), so a
change to what the suite runs, or to the gate that lets it run, must run it.
- A skipped job now says so in a named step and a notice annotation, rather than
reporting success from a list of greyed-out steps.
- The dist upload sets if-no-files-found: error, so an empty artefact fails the
producing job instead of surfacing later as "webSourcePath ... doesn't exist".
- lighthouserc.js records the cost of numberOfRuns: 1 - no median, so one slow
navigation is a re-run rather than a fix.
Measured over the last 150 workflow runs on this forge: a pull request costs
56 runner-minutes (Build-and-ng-test 5.8m, Build-and-test-development 31.2m,
lighthouse 19.3m), and when three pull requests are in flight the first job
queues 47-69 minutes before it starts.
- Build-and-ng-test publishes client/dist as an artifact and the Cypress job
downloads it instead of running a second cold `npm run build` (1.6m each).
Both jobs test the same bytes.
- lighthouse: numberOfRuns 1 (lhci's default of 3 tripled the 13.6m audit to
steady a median that only has to clear a fixed threshold), and the duplicate
/view/usernav/groups entry is removed (17 unique URLs, not 18).
- Cypress records no video in CI and takes screenshots on failure instead
(`--config video=false,screenshotOnRunFailure=true`, CLI only so local runs
and the demo recorder still record); the failure artifact carries
screenshots as well as any video.
- Both workflows detect whether client/ or sas/mocks/ changed and skip their
expensive steps when neither did - a sas-only or docs-only pull request no
longer pays 45 minutes for suites that cannot observe it. The gate is
`!= 'false'`, so a missing step output runs the suite rather than quietly
reporting green.
An undersized runner does not fail honestly: the kernel OOM killer takes
the build, so the job reports exitcode 137 at ng build, or Karma
disconnects mid-suite after twenty minutes. Both read as a flaky test
rather than as a box that cannot hold the job, and both cost a whole CI
slot.
The check runs first in the three client jobs (Build-and-ng-test,
Build-and-test-development, lighthouse), prints total and available RAM,
vCPU count and any cgroup limit, and exits 1 when the effective size is
under 8 GB - the supported minimum. A cold production build measures
3.8 GB peak on its own, so 8 GB leaves roughly 2x headroom.
sas/.sasjslint now promotes noUnusedMacros to "error", so the rule fails
the build rather than printing a warning. The 54 findings it reported are
gone:
- 46 header entries nothing calls are dropped. Each removal was checked
against the compiled artefacts rather than the linter alone: the
project was compiled before and after, and the two sasjsbuild trees
compared. 37 macro definitions disappear from the generated programs -
every one of them either a duplicate of a definition the file still
reaches through another declared macro, or a macro no generated program
calls. No macro that a generated program calls lost its definition.
- 8 entries are kept, in five files that carry
@sasjslint {"noUnusedMacros": false} with the reason written above it:
the three serviceInit programs and serviceterm.sas declare mpeinit and
mpeterm without calling them, because that declaration is how a service
body reaches those macros at all; loadfile.sas declares mp_binarycopy
for the fallback that is commented out on SAS 9.4 and later.
- Four headers are left with no macros to declare, so their empty
<h4> SAS Macros </h4> section is removed rather than kept as a stub.
@sasjs/cli is bumped 6.2.0 -> 6.5.0 in sas/package.json. The lint engine
ships inside the CLI and is not a direct dependency, so the pinned CLI is
what fixes the rule set: 6.2.0 resolves @sasjs/lint 4.3.0, 6.5.0 pins
4.9.0 exactly. Both versions were compiled against this tree and the
sasjsbuild output is byte-identical, so the bump changes the lint engine
and nothing else.
`sasjs lint` is also now reachable as `npm run lint:sas`, and a CI step in
build.yaml runs it. Until now nothing ran SAS lint automatically, and a
bare `sasjs` resolves whatever CLI is installed globally - a different
lint version reports a different rule set, which is why the counts in
issue #376 (taken with 6.2.5) do not match the pinned 6.2.0.
43 warnings remain under the rules not yet promoted: noUndeclaredMacros
(25), noBannedStrings (13), macro name length and case (4),
indentationMultiple (1). They are the next rungs of the ratchet.
The pre-commit Prettier gate could not run: it needs client/node_modules,
which this worktree does not install, and the fix it attempted touched
three client files that this change does not modify (reverted). No client
file is in this commit.
Follow-ups from the review of #414.
The credential guard now checks each staged .npmrc rather than only the client
one: the root and sas/ files are tracked policy as well, so a token pasted into
either was equally one `git add -A` away. Verified by staging a token in
sas/.npmrc - the hook exits 1.
$RUNNER_TEMP/npmrc is removed at the end of each job that writes it, with
`if: always()` so a failed job cleans up too.
The root `install` script is gone: the root .npmrc sets ignore-scripts, so npm
never ran it. CONTRIBUTING already spells out the per-project commands, and is
now the single source.
The release workflows build the published client from the same tree, so they run
`check:archiver` like build.yaml does.
client/.npmrc is tracked now, so a token left in it from the old pattern is one
`git add -A` from being committed. Prose in CONTRIBUTING is not a control: the
pre-commit hook fails when the staged file holds _authToken, _auth or _password.
`npm run check:archiver` runs the directory-walk guard, so it is reproducible
before a push instead of only in CI, and the CI step calls the same script.
CONTRIBUTING now spells out the per-project installs, since the root .npmrc has
ignore-scripts=true and the composite install script therefore never runs.
Four write steps interpolated ${{ secrets.NPMRC}} straight into the shell source,
so a quote or $( in the value would break the step or run. They use the env form
the same workflows already use elsewhere.
CONTRIBUTING.md now says what the tracked client/.npmrc means for a developer:
run `npm run postinstall` after installing, `npx cypress install` for the suite,
and keep registry credentials in the user config rather than in the tracked file.
client/.npmrc is tracked now, and every workflow appended the NPMRC secret to it
inside the workspace - one `git add -A` from committing a live token, and
semantic-release runs in jobs that have already written it. The secret goes to
$RUNNER_TEMP/npmrc and NPM_CONFIG_USERCONFIG points the installs at it, so the
project file stays policy-only.
The archiver guard used a fixed path in the shared tmpdir and had no handler on
the write stream: two jobs on one runner collided on the path, and a write
failure surfaced as an uncaught throw. Each run now gets its own directory, both
streams report, and the tree is removed afterwards.
The exceljs override pins archiver to 8.0.0 rather than ^8.0.0, so the version
the guard was validated against cannot drift.
Three things the last run exposed.
Every workflow that writes the NPMRC secret used `>`, which now overwrites a
tracked policy file - so the release, lighthouse and sites jobs would install
with hooks enabled and ranges unpinned, the opposite of what this PR adds. They
append.
Cypress downloads its binary in a dependency install hook, so ignore-scripts
leaves it out: "The cypress npm package is installed, but the Cypress binary is
missing". Both client installs now run `npx cypress install` after the
generator.
The archiver guard imports archiver and unzipper, which were only present as
overrides under exceljs; they are declared devDependencies at their resolved
versions so the script owns what it imports. Both remain production packages
because exceljs still requires them.
Dependency install hooks now stay off in client/, as they already are at the root
and in sas/: client/.npmrc is committed with ignore-scripts and save-exact, the
root install script runs `npm run postinstall` after installing, and both CI
installs pass --ignore-scripts and then run the generator by hand. The client's
own postinstall is what creates src/environments/version.ts and the hot icons, so
it has to run - it is third-party hooks that do not.
CI writes client/.npmrc from the NPMRC secret; the second job now appends rather
than overwrites so the tracked policy survives (the first already appended).
The archiver guard reads the archive back instead of checking its size: a matcher
that takes the top-level file and misses the nested directory clears any size
threshold, and that is exactly the failure it exists to catch. It now asserts
a.txt and sub/b.txt are both present.
archiver 8 is ESM-only, so require('archiver') resolves through require(esm) and
that exists from Node 22.12 - a supported Node 18/20 would fail the check for a
reason that has nothing to do with the minimatch override. The script is .mjs and
imports ZipArchive.
The client's postinstall generates src/environments/version.ts (and the hot
icons), which the app imports - with --ignore-scripts the build fails with
"Can't resolve '../../environments/version'". My local check missed it because
version.ts was already on disk from an earlier install.
So the client keeps its install scripts; the archiver guard stays, which is what
that commit was for.
The minimatch override forces a major above readdir-glob's own range, and
archiver's directory() is the only runtime caller of that path. scripts/
check-archiver.cjs archives a small tree and fails if the result is too small to
hold it, so the override is guarded by a run rather than by the install
succeeding.
client/.npmrc is gitignored - it is where a developer's registry credentials
live - so the repo-wide ignore-scripts policy is passed on the client install
steps instead. Verified locally: npm ci --ignore-scripts, the archiver check and
ng build all succeed.
All four pipelines downloaded `releases/latest/download/linux.zip`, unzipped it
and ran `api-linux`. They now start the published package at a pinned version:
pm2 start npx --name sasjs-server -- --yes @sasjs/server@3.6.2
and `sites.yaml`, which runs it under nohup, does the same with
`npx --yes @sasjs/server@3.6.2`.
Why: 10.7MB instead of 78.7MB, the same artifact a user installs, and a version
this repository names rather than whatever `latest` happens to be. `latest` also
moved silently on every run, so a red pipeline could not say which server
produced it. 3.6.2 is the first release whose bin carries a shebang; before it
`npx @sasjs/server` could not start at all.
The pin lives in `env.SASJS_SERVER_VERSION` in each workflow, next to the npm
cache step that makes repeat runs skip the download (`~/.npm`, keyed on the
version; `continue-on-error`, so a runner without cache support still runs the
pipeline).
The readiness loops and `RUN_AS=root` are unchanged - the package runs as the
runner's Node 24 rather than the Node embedded in the zip, and the loop is what
tells us it came up. Failure paths now dump `pm2 logs sasjs-server`, the name the
process is registered under.
The four shards made the release-branch run red, and the reasons are in the
logs rather than in the theory:
- every failure was a timeout, not an assertion - "cy.type() failed because
the center of this element is hidden from view", "Expected
<div.app-loading> not to exist ... continuously found". Four jobs on one
runner means four SASjs servers, four ng serve and four Chromes competing
for the same CPU, and the app under test starves;
- upload-artifact@v4 does not work on this runner at all:
"GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ ...
not supported on GHES", so the v3 -> v4 move I made on review advice broke
the artifact step outright.
Measured, from the one sharded run that did complete: 20m34s wall against 33-35
minutes for the single job, with peak concurrency 3 - so a third off, not the
four-fold the split suggests. That is not worth a flaky pipeline while a release
sits on it.
Kept from the sharding work, because those parts were the real gain:
- the spec list has one home, .gitea/scripts/cypress-specs.sh;
- Build-and-ng-test fails when a spec on disk is not listed, so the five that
had never run in CI cannot go missing again - and they do run now;
- videos are zipped and uploaded only on failure;
- one job means one server, which is also what the guards and the balance
arguments were protecting.
The shard and fan-in jobs are gone with the branch that carried them.
The standalone URL dropped every query parameter whose name started with
va_, not just va_parmap; the names are now enumerated, and a later
parameter with that prefix survives (the component spec pins the exact
standalone URL instead of a substring, so the query surviving is the
assertion, not an accident of the runner's own query).
The timezone runs are scoped to the VA specs, the only specs with local
calendar arithmetic, so the blocking job no longer runs the whole suite
twice more.
The gate could not fail. With needs: on four shards and no if: always(), a
failed shard skips the fan-in job instead of failing it, and a skipped job is
not a failure on the one check branch protection watches - so the gate was
silent in exactly the case it exists for.
It now runs with if: always() and fails on any shard result that is not
success. The count is checked as well: if the needs context does not resolve on
this runner the step fails saying so, rather than looping over nothing and
reporting green - which is the failure mode a bare loop would have.
Also removes what the last regeneration left behind: the coverage guard step
appeared twice in Build-and-ng-test (so it ran twice) and the shard 1 comment
block appeared three times. The file is regenerated from fix/va-parmap-review-572
rather than from my own previous commit, which is what caused the duplication.
Renaming the shards to cypress-shard-1..4 and adding a job named
Build-and-test-development that does nothing but needs all four.
That name is the status check the branch protection rule requires, so the check
now turns green only when every shard passed - a red shard 3 leaves it
unreported and the PR unmergeable, which is the behaviour the single job had.
No repo setting has to change, and adding a fifth shard later means adding it to
needs rather than remembering to edit branch protection, where forgetting fails
silently.
Review found that five specs ran in no shard - coltype-delete-record,
download, embed-va, metanav, submitted-details - and one of them is the VA
embed suite this release added, 454 lines that had never run in CI. The old
single --spec string omitted them too, so nothing regressed; they were simply
never wired up, and nothing in the repo marked them skipped.
The spec lists now live in .gitea/scripts/cypress-shards.sh, one place, so the
workflow no longer spells them out four times. A guard step in
Build-and-ng-test compares the shards against the specs on disk and fails when a
file belongs to none of them or to more than one - proven by removing a name and
watching it fail with that file listed.
Rebalanced for the five additions, on test count and spec count together, since
each spec pays a few seconds of startup:
shard 1 editor, csv, metanav 57 tests
shard 2 excel, filtering, download, browser-info,
excel-multi-load, hook-programs 49 tests
shard 3 embed-va, viewbox, licensing, filter-panel, liveness 43 tests
shard 4 viewer-labels, row-cell-limits, derived-variables,
ddl-export, coltype-delete-record, submitted-details,
startupservice-launcher, stage, full-table-search,
csv-limited 27 tests
24 specs / 176 tests, each in exactly one shard.
upload-artifact moved from the retired v3 to v4; this change multiplies that
step by four, so it is worth not carrying the dead major forward.
Not in this commit: the three new check names still need adding to branch
protection, which needs repo admin.
Sits on top of fix/va-parmap-review-572 (#391) so the two changes to this
workflow land in one order; the timezone step that PR adds to
Build-and-ng-test is untouched.
The development job ran all nineteen specs in one Cypress invocation on one
runner, so its wall time was the sum of every spec - around half an hour on a
green run, with the whole pipeline waiting on it.
Four jobs now, each with its own runner and its own SASjs server, taking a
subset:
shard 1 editor (55 of the suite's ~148 tests)
shard 2 excel, filtering, csv-limited, csv, stage
shard 3 viewbox, licensing, filter-panel, browser-info, excel-multi-load, liveness
shard 4 viewer-labels, row-cell-limits, derived-variables, ddl-export,
hook-programs, startupservice-launcher, full-table-search
The split is balanced on test count, not file count: editor.cy.ts alone holds 55
tests, so an even spread of files would leave one shard running twice as long as
the others. Its own duration is the floor the pipeline can reach.
The setup steps are identical in every shard, so they are defined once and
aliased with a YAML anchor; each job differs only in SHARD and SPECS. Shard 1
keeps the original job name so an existing required status check still matches.
Video is still recorded - measured at 91s with it and 92s without on the same
spec, so the recording is not what takes the time. What it did cost was the zip
and artifact upload of every run's videos, now guarded by if: failure().
SASjs Server 3.x deprecated PORT and PROTOCOL and refuses to start when an
.env names neither HTTP_PORT nor HTTPS_PORT, exiting with
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
The three workflows downloaded releases/latest and wrote an .env without a
listener, so api-linux never bound :5000 and the 'Setup and start SASjs
server' step failed after its 120s wait (build runs for #363 and #365, both
failed there). Add HTTP_PORT=5000 to build.yaml, release.yaml and
lighthouse.yaml, and document it in the local-dev .env in CONTRIBUTING.md,
where the default-port claim is no longer true.
The step's inner guard killed Cypress at 30 minutes (exitcode 124) with the
19-spec list still on spec 18, so the job failed without a single failed spec -
seen on a07728b and on the previous head. The suite has grown to roughly 33-36
minutes; the inner guard is now 45 minutes and the step 50.
When the startupservice fails, the modal is a dead end: the adapter returned
no log, so there is nothing to act on. The modal now offers a button that runs
the same service through the JES / STP web app in a new browser window, where
the SAS log is returned as the page body.
`_debug=131` is always added (the log is the point), and `_executionTasks` is
never included, so on Viya the run is not submitted as a Compute task and
cannot hit the execution-task timeout. The URL is built by `buildServiceUrl`,
which works for SAS 9, Viya and SASjs Server, and is the same shape the deploy
configurator already uses to open makedata.
The startup failure modals now carry the service name, which is what shows the
button: all three failure branches (malformed response, missing properties,
request rejected) pass `public/startupservice`.
It applies a licence key via loginAndUpdateValidKey, and csv-limited
asserts the free-tier restriction - so it must not run before that spec.
Running last keeps the licence state every other spec sees unchanged.
Review feedback on this PR: with .npmrc setting ignore-scripts=true the prepare
script never runs, so the native hooks in .git-hooks/ do not activate on a fresh
clone and a commit can go unchecked.
Both standard routes are added, the way sasjs/cli and sasjs/lint carry them:
- .pre-commit-config.yaml pins the gitleaks hook to v8.30.1, so `pre-commit
install` wires the secret scan without the one-time core.hooksPath command.
- Build-and-ng-test runs the same pinned @nogoo9/gitleaks binary as a step,
because `pre-commit install` is itself a manual command: CI is the enforcement
point. --no-git scans the working tree, so the gate is what the pull request
adds rather than the repo's history.
The scan runs before the job writes client/.npmrc from the NPMRC secret - that
file carries the private registry token, so scanning after it would flag the
token CI injects itself.
.npmrc is unchanged - ignore-scripts=true stays - and the native hooks stay for
anyone who prefers them. CONTRIBUTING.md names the framework route next to the
existing one-time command.
Verified: npm ci followed by the scan reports no leaks on a fresh checkout, the
client lint check passes, and pre-commit validate-config plus pre-commit run
--all-files both pass.
VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.
The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.
Recent sasjs/server releases refuse to run as root unless RUN_AS names
an account (security hardening, "refuse to run as root unless RUN_AS
names an account"). The CI runners are root, so the server exited at
startup and pm2 restart-looped (103 restarts), leaving :5000
unreachable - which is what failed the mocked-estate deploy and the
Lighthouse job.
The server's own message offers RUN_AS=root as the deliberate override;
opt in explicitly for the throwaway CI container.
Both workflows start api-linux with `pm2 start api-linux --wait-ready`
and then hit http://localhost:5000 in the next step. pm2 can return
while the app is still 'launching' (it does not always block on the
ready signal), so the deploy step races the server and fails with
"Couldn't connect to server".
Wait up to 120s for the server to answer, and print `pm2 list` plus the
api-linux log if it never does, so a genuine startup failure is visible
in the job output instead of a bare ECONNREFUSED.
Adds a fourth admin action to the System information screen - Export DC
Library DDL - with a secondary flavour chooser (SAS, PGSQL, TSQL) and an
optional schema box for the DB flavours. The selection is passed to the
existing services/admin/exportdb service in a new window as URL params.
Also bumps @sasjs/adapter to ^4.19.1, adds a Cypress spec for the three
flavour states, and adds a TSQL case to the exportdb SAS test.
Closes#179
hook-programs.cy.ts was in no CI spec list, so it never ran - the only
end-to-end coverage of the new mock hook execution (mockHookSource plus
the PRE/POST_EDIT_HOOK blocks) and of the loader's target re-registration
check was dead weight. Add it to the development job's --spec list.
It goes second, not last: csv-limited.cy.ts asserts the free tier and so
must keep the first slot (before any spec applies a licence key), and test
2 reads the approval queue, which is paged oldest first - a changeset
submitted by an earlier spec pushes its own row off the first page. The
spec header records the constraint. Full list verified locally: 15 specs,
132 tests, all passing.
browser-info.cy.ts asserts the diagnostics tables on the adapter payload but was not in the job's explicit --spec list, so it never ran. Append it so the feature is covered by CI.
The server-ci target streams the Angular frontend (CONTRIBUTING.md documents
http://localhost:5000/AppStream/clickme/), but webSourcePath was
"`../../../../client/dist". The leading backtick makes the first path segment
literal, so it happened to resolve to <repo>/client/dist by accident; the
correct value relative to the sasjs project root (sas/mocks) is
../../client/dist.
The real problem was that CI job 2 never built the client, so client/dist did
not exist when the mock services were deployed: the deploy logged
"webSourcePath: <repo>/client/dist present in 'streamConfig' doesn't exist"
and streamed no frontend, while the job still passed. Build the production
frontend before deploying the mocks so the streamed app is actually produced.
The viewer's search box searches every column of a table - character columns
by case-sensitive CONTAINS and numeric columns by exact equality - but against
the JS mock backend the box did nothing, because the viewdata mock ignored
SEARCHTYPE/SEARCHVAL (and FILTER_RK) entirely.
Mock changes (sas/mocks/sasjs/services/public/viewdata.js):
- apply the stored filter first via mpeFilterMaster, mirroring
%mpe_filtermaster(VIEW,...) in viewdata.sas, and return its WHERE text as
sasparams.FILTER_TEXT (previously hard-coded blank, so the viewer's info bar
never showed a clause and the filter was never applied to the rows)
- mirror %mp_searchdata: CHAR = case-sensitive CONTAINS across every character
column, NUM = exact equality across every numeric column, only CHAR/NUM
trigger a search, search values stripped of % & ; " like the SAS service
- cap the rows the way the real service does (MAXROWS 500; NOBS is the uncapped
filtered count for a normal view, the capped match count for a search)
- a search with no matches now returns no rows, so the client shows its
"No data found with given conditions" panel; the single-empty-row fallback
stays for the normal-view-empty-table case
Mock data (sas/mocks/sasjs/services/admin/makedata.js):
- new MPE_X_SEARCH demo table: 1000 deterministic, obviously fictional
deep-sea survey rows, registered in MPE_TABLES. Seeded so a partial search
hits a value buried in the middle of a long text column, a value spanning two
columns, a repeated exact numeric, and rows that deliberately do not match.
New spec (client/cypress/e2e/full-table-search.cy.ts):
- seven tests covering open/full view, partial character search, case
sensitivity, numeric exact match, no match, search within a filter, and
clearing the search. No artificial waits - state assertions only.
- added to the Cypress spec list in .gitea/workflows/build.yaml
Also adds the dc-cypress skill (.agents/skills/dc-cypress) covering the mock
backend setup, the spec conventions, the selectors that actually work in the
viewer filter modal, and the fixed 1280x720 video capture.
- Mirrors the review page's toggle: getstagetable.sas now also webouts
fmt_stagetable (SAS-formatted), selectFormattedRows() picks between it
and the raw stagetable, falling back to raw if fmt data is absent
- Adds stage.cy.ts e2e coverage and wires it into both build.yaml and
release.yaml's cypress --spec lists (previously out of sync with
each other on viewbox.cy.ts too)
dc
allan
Yury