`surfer deploy` chooses its target only from `--site-name`; without one it
replaces the app's DEFAULT site. The typedoc job passed `--server
webdoc.datacontroller.io` and no site name, so the 7.19.0 docs were published
over the default site of that app - the marketing site served at
datacontroller.io - instead of to a webdoc site.
Name the site, and give it its alias domain so the first run creates it.
The SAS docs job is left unnamed on purpose: code.datacontroller.io is its own
app whose default site is the docs, and a comment now says so.
The Sites workflow was the one place that still started the server without a
listener, so it died with
[error] Invalid environment variable(s) provided:
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
and the following step's 120s poll timed out. build.yaml, release.yaml and
lighthouse.yaml already carry the same line.
SASjs Server 3.x deprecated PORT and PROTOCOL and refuses to start when an
.env names neither HTTP_PORT nor HTTPS_PORT, exiting with
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
The three workflows downloaded releases/latest and wrote an .env without a
listener, so api-linux never bound :5000 and the 'Setup and start SASjs
server' step failed after its 120s wait (build runs for #363 and #365, both
failed there). Add HTTP_PORT=5000 to build.yaml, release.yaml and
lighthouse.yaml, and document it in the local-dev .env in CONTRIBUTING.md,
where the default-port claim is no longer true.
The pre-publish check only asserted the pair had two different values, which a
committed pair can satisfy while being broken (one half regenerated, a value
truncated on copy). It now imports the activation key as pkcs8 RSA-OAEP,
decrypts the licence key, parses the payload and exits non-zero otherwise, so a
drifted pair fails the deploy instead of publishing a site that cannot decrypt
its own licence.
CONTRIBUTING records that the two values are one key and are regenerated
together, and that the .gitleaksignore entries are line-scoped.
The step's inner guard killed Cypress at 30 minutes (exitcode 124) with the
19-spec list still on spec 18, so the job failed without a single failed spec -
seen on a07728b and on the previous head. The suite has grown to roughly 33-36
minutes; the inner guard is now 45 minutes and the step 50.
The published sites get the same key pair on every deploy, committed as
sas/mocks/mock-licence.json (site_id 123, the mock estate's SYSSITE) rather than
generated in the pipeline. The pair sits outside the deployed sync folder, so the
CI estate stays unlicensed and the Cypress specs keep exercising the free tier.
The Sites job copies the pair into the drive before the snapshot and refuses a
pair whose two values are identical, which is how the app spots an HTTP-format
key it cannot decrypt in a secure context. The pair trips the generic-api-key
scanner, so .gitleaksignore carries the two fingerprints.
The demo and staging sites ran on the free tier, so uploads and the other
licensed features were unavailable to anyone looking at them. The Sites job now
mints a self-contained key pair (site_id 123, the estate's SYSSITE) into the
drive before the snapshot, so both sites are fully usable.
Minted per deploy rather than committed, and the CI estate stays unlicensed so
the specs keep exercising the free tier.
The estate now ships no licence key at all - the placeholder word 'invalid' made
startupservice return an unusable key pair and produced the decrypt-failure page
on every HTTPS site. An unlicensed estate simply runs on the free tier.
loginAndUpdateValidKey therefore no longer infers 'already licensed' from the
nav tree: it detects the free-tier banner (#demo-banner) and uploads a key
through the licensing screen, which posts to admin/registerkey - the API the
licence page's upload button uses.
The dispatch endpoint resolves a bare name as a branch, so the release tag needs
the refs/tags/ prefix, and the route is scoped to the actions unit - contents
does not grant it. Without both, the step failed and the demo stayed on the
previous release.
The release tag is created by the release job's semantic-release step, i.e. as
the Actions user, and the forge suppresses ref events from that user, so the
Sites workflow's `on: push: tags` trigger never fires and the demo would stay on
the previous release. Dispatch the workflow with the new tag after the assets
are uploaded instead.
Emptying mock-storage/licence.json dropped every CI run to the free tier, where
the upload button is absent, so excel.cy.ts failed all 24 tests: the licensing
spec registers a key only when the activation page is shown, and an unlicensed
estate activates straight into the app.
The placeholder pair stays in the fixture, and the Sites job strips it from the
drive before the snapshot is taken, so the published sites still run unlicensed
without the decrypt-failure page.
Two review findings: the makedata call ends in || true (needed for its expected
HTML reply), so a genuine failure published a data-less site as a green build -
the drive is now checked for mpe_tables.json and at least ten tables before the
snapshot is taken. And the concurrency group is keyed on the target site rather
than the branch, so two release branches publishing staging cannot interleave.
The demo now deploys when semantic-release creates the tag, so it can never go
live on a commit whose release then fails, and the checkout carries the tag the
client's build stamp reports.
demo.datacontroller.io is now a multi-site Surfer instance: a release branch
deploys the staging.datacontroller.io site, and a push to main deploys the demo
(default site). Both run the same browser-hosted mock build.
Pushes to release/** now build the mocks as a browser-hosted site (SASjs CLI
6.1.0, streamConfig.mockWeb) and publish it to the Surfer app behind
demo.datacontroller.io, so the demo always matches the release branch.
The browser build seeds its drive from the compiled services, which carries no
DC data, so the job runs makedata on a throwaway SASjs Server and ships that
drive as mock-snapshot.json.
When the startupservice fails, the modal is a dead end: the adapter returned
no log, so there is nothing to act on. The modal now offers a button that runs
the same service through the JES / STP web app in a new browser window, where
the SAS log is returned as the page body.
`_debug=131` is always added (the log is the point), and `_executionTasks` is
never included, so on Viya the run is not submitted as a Compute task and
cannot hit the execution-task timeout. The URL is built by `buildServiceUrl`,
which works for SAS 9, Viya and SASjs Server, and is the same shape the deploy
configurator already uses to open makedata.
The startup failure modals now carry the service name, which is what shows the
button: all three failure branches (malformed response, missing properties,
request rejected) pass `public/startupservice`.
It applies a licence key via loginAndUpdateValidKey, and csv-limited
asserts the free-tier restriction - so it must not run before that spec.
Running last keeps the licence state every other spec sees unchanged.
The "Upload assets to release" step builds the new body with
`jq -n --arg body ... '$body + "\n" + ...'` and no `-r`. jq then prints the
result as a JSON string literal - quoted, with `\n` escapes - which the
PATCH that follows encodes a second time, so the stored release body is a
quoted one-liner instead of markdown. On v7.17.0 this rendered the whole
release as a single escaped line in the markdown box.
Add `-r` so the concatenated notes are emitted as raw text. Verified by
running the step's jq invocation against sample notes: without `-r` the
stored body contains literal `\n`, with `-r` it contains real newlines.
release.yaml only runs on a push to main, so this cannot be exercised by
CI on the PR itself.
Review feedback on this PR: with .npmrc setting ignore-scripts=true the prepare
script never runs, so the native hooks in .git-hooks/ do not activate on a fresh
clone and a commit can go unchecked.
Both standard routes are added, the way sasjs/cli and sasjs/lint carry them:
- .pre-commit-config.yaml pins the gitleaks hook to v8.30.1, so `pre-commit
install` wires the secret scan without the one-time core.hooksPath command.
- Build-and-ng-test runs the same pinned @nogoo9/gitleaks binary as a step,
because `pre-commit install` is itself a manual command: CI is the enforcement
point. --no-git scans the working tree, so the gate is what the pull request
adds rather than the repo's history.
The scan runs before the job writes client/.npmrc from the NPMRC secret - that
file carries the private registry token, so scanning after it would flag the
token CI injects itself.
.npmrc is unchanged - ignore-scripts=true stays - and the native hooks stay for
anyone who prefers them. CONTRIBUTING.md names the framework route next to the
existing one-time command.
Verified: npm ci followed by the scan reports no leaks on a fresh checkout, the
client lint check passes, and pre-commit validate-config plus pre-commit run
--all-files both pass.
VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.
The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.
release.yaml starts the SASjs Server the same way as build.yaml and
lighthouse.yaml, so the same root guard (sasjs/server v1.9.0) breaks it.
It runs on pushes to main and has not run since the guard shipped, so
the next release would have failed at the mock deploy.
Recent sasjs/server releases refuse to run as root unless RUN_AS names
an account (security hardening, "refuse to run as root unless RUN_AS
names an account"). The CI runners are root, so the server exited at
startup and pm2 restart-looped (103 restarts), leaving :5000
unreachable - which is what failed the mocked-estate deploy and the
Lighthouse job.
The server's own message offers RUN_AS=root as the deliberate override;
opt in explicitly for the throwaway CI container.
Both workflows start api-linux with `pm2 start api-linux --wait-ready`
and then hit http://localhost:5000 in the next step. pm2 can return
while the app is still 'launching' (it does not always block on the
ready signal), so the deploy step races the server and fails with
"Couldn't connect to server".
Wait up to 120s for the server to answer, and print `pm2 list` plus the
api-linux log if it never does, so a genuine startup failure is visible
in the job output instead of a bare ECONNREFUSED.
Adds a fourth admin action to the System information screen - Export DC
Library DDL - with a secondary flavour chooser (SAS, PGSQL, TSQL) and an
optional schema box for the DB flavours. The selection is passed to the
existing services/admin/exportdb service in a new window as URL params.
Also bumps @sasjs/adapter to ^4.19.1, adds a Cypress spec for the three
flavour states, and adds a TSQL case to the exportdb SAS test.
Closes#179
hook-programs.cy.ts was in no CI spec list, so it never ran - the only
end-to-end coverage of the new mock hook execution (mockHookSource plus
the PRE/POST_EDIT_HOOK blocks) and of the loader's target re-registration
check was dead weight. Add it to the development job's --spec list.
It goes second, not last: csv-limited.cy.ts asserts the free tier and so
must keep the first slot (before any spec applies a licence key), and test
2 reads the approval queue, which is paged oldest first - a changeset
submitted by an earlier spec pushes its own row off the first page. The
spec header records the constraint. Full list verified locally: 15 specs,
132 tests, all passing.
browser-info.cy.ts asserts the diagnostics tables on the adapter payload but was not in the job's explicit --spec list, so it never ran. Append it so the feature is covered by CI.
The server-ci target streams the Angular frontend (CONTRIBUTING.md documents
http://localhost:5000/AppStream/clickme/), but webSourcePath was
"`../../../../client/dist". The leading backtick makes the first path segment
literal, so it happened to resolve to <repo>/client/dist by accident; the
correct value relative to the sasjs project root (sas/mocks) is
../../client/dist.
The real problem was that CI job 2 never built the client, so client/dist did
not exist when the mock services were deployed: the deploy logged
"webSourcePath: <repo>/client/dist present in 'streamConfig' doesn't exist"
and streamed no frontend, while the job still passed. Build the production
frontend before deploying the mocks so the streamed app is actually produced.
The release job now hashes every asset it uploads (frontend.zip, the SAS 9
and Viya deployment programs, the SASjs Server bundle) into a SHA256SUMS
file with basename-only paths, and attaches it to the release alongside
the assets. The release body gains a "Verifying the download" section -
emitted by .gitea/scripts/verify-section.sh so the YAML scalar stays
clean - with the sha256sum -c command and a note on what a checksum does
and does not prove.
The upload loop is rewritten from an inline list to a bash array shared
by the hashing and upload steps, so the asset set cannot drift between
what is hashed and what is uploaded.
Simulated end to end locally (stubbed curl/jq): 8 uploads fire with
correct paths, SHA256SUMS verifies with sha256sum -c, a tampered file
fails the check, and the assembled release body keeps the existing
notes with the new section and installation footer on their own lines.
The release job zipped ./client/dist, so every archive entry carried a
client/dist/ prefix and an unzip dropped the app into a client/dist
subfolder. Zip from inside dist instead, so index.html, assets and the
bundled viya.json sit at the root of the archive.
Closes#147
The viewer's search box searches every column of a table - character columns
by case-sensitive CONTAINS and numeric columns by exact equality - but against
the JS mock backend the box did nothing, because the viewdata mock ignored
SEARCHTYPE/SEARCHVAL (and FILTER_RK) entirely.
Mock changes (sas/mocks/sasjs/services/public/viewdata.js):
- apply the stored filter first via mpeFilterMaster, mirroring
%mpe_filtermaster(VIEW,...) in viewdata.sas, and return its WHERE text as
sasparams.FILTER_TEXT (previously hard-coded blank, so the viewer's info bar
never showed a clause and the filter was never applied to the rows)
- mirror %mp_searchdata: CHAR = case-sensitive CONTAINS across every character
column, NUM = exact equality across every numeric column, only CHAR/NUM
trigger a search, search values stripped of % & ; " like the SAS service
- cap the rows the way the real service does (MAXROWS 500; NOBS is the uncapped
filtered count for a normal view, the capped match count for a search)
- a search with no matches now returns no rows, so the client shows its
"No data found with given conditions" panel; the single-empty-row fallback
stays for the normal-view-empty-table case
Mock data (sas/mocks/sasjs/services/admin/makedata.js):
- new MPE_X_SEARCH demo table: 1000 deterministic, obviously fictional
deep-sea survey rows, registered in MPE_TABLES. Seeded so a partial search
hits a value buried in the middle of a long text column, a value spanning two
columns, a repeated exact numeric, and rows that deliberately do not match.
New spec (client/cypress/e2e/full-table-search.cy.ts):
- seven tests covering open/full view, partial character search, case
sensitivity, numeric exact match, no match, search within a filter, and
clearing the search. No artificial waits - state assertions only.
- added to the Cypress spec list in .gitea/workflows/build.yaml
Also adds the dc-cypress skill (.agents/skills/dc-cypress) covering the mock
backend setup, the spec conventions, the selectors that actually work in the
viewer filter modal, and the fixed 1280x720 video capture.
- Mirrors the review page's toggle: getstagetable.sas now also webouts
fmt_stagetable (SAS-formatted), selectFormattedRows() picks between it
and the raw stagetable, falling back to raw if fmt data is absent
- Adds stage.cy.ts e2e coverage and wires it into both build.yaml and
release.yaml's cypress --spec lists (previously out of sync with
each other on viewbox.cy.ts too)
Adds a URL-driven toggle (?labels=true, mirroring embed=va) that
swaps grid column headers between NAME and LABEL in both the Viewer
and Editor, with a "Show labels"/"Show names" item in each grid's
right-click context menu. The header-info popup now shows NAME first
regardless of display mode. Sorting/filtering/data binding stay
NAME-based throughout — only the displayed header text changes.
New shared, unit-tested utils (parse-labels-param, display-col-headers,
col-info-html) back both pages. Mock data (viewdata.js, getdata.js)
gains a couple of columns with LABEL != NAME plus a blank-LABEL case,
covered by a new viewer-labels.cy.ts e2e spec now wired into CI.
Also fixes a pre-existing gap in cypress/support/commands.ts: the
isLoggedIn/loginAndUpdateValidKey custom commands had no Cypress
Chainable type augmentation, and moment was imported as a namespace
import despite its `export =` typing making it uncallable that way.
dc
allan
Yury