makedata seeded two things at once: the control library, and the tables the
Cypress suite needs. The estate it produced was therefore not the estate a
SAS deployment produces - and it had drifted: MPE_MAXKEYVALUES and
MPE_SIGNOFFS were never created, and MPE_CONFIG, MPE_VALIDATIONS, MPE_TABLES,
MPE_SELECTBOX and MPE_XLMAP_RULES were missing seed rows that mpe_makedata.sas
inserts.
makedata now transcribes mpe_makedatamodel.sas and mpe_makedata.sas: every
model table with its columns, and every seed row the macro writes. The
Cypress fixtures move to services/admin/createdemodata.js, which the test
estate runs after makedata; it strips and re-adds its rows so a reseed does
not accumulate.
Checked against real SAS output on the shared estate (fresh deploy of the
server target, makedata, then getrawdata for each control table and a
dictionary.columns probe): all 40 tables and 339 columns match on name, type,
length, format and label, and the seeded rows match row for row apart from
the environment (libref, user id, group names) and the sample rows. That run
found six defects a source-only comparison could not: INT columns are 8 bytes
not 4, one label had lost a double space, the DC_MACROS setting held the SAS
expression instead of a path, the email templates held an escaped newline,
75 selectbox rows held %mf_increment(rk) instead of their key, and one
validation row was truncated at a ';' inside its regex.
makedata-parity.test.js parses both macros and compares them with the seeded
estate, and fails on a value that does not fit its column type, so the next
drift fails a test instead of shipping. The CI workflows, the Sites demo seed
and CONTRIBUTING call createdemodata after makedata.
The capture assigned its body to _webout in one template literal, and that
literal was not JSON: raw line breaks inside the graphviz label and an
unescaped quote in labelloc = "t". fromsas, clickableids, info and flatdata
are members now, written as string literals that keep the label's quotes and
line breaks, with the envelope supplied by the runtime.
sas/mocks/test/sasjs-webout.test.js holds the six services that set their own
content headers, and pins this capture's members. CONTRIBUTING.md lists the
same six.
The payloads are the SAS 9 recordings, separators included - the front end splits
metadata URIs on the backslash, so a value that arrives without one exercises the
wrong branch. The envelope is not SAS 9: _PROGRAM, SYSUSERID, SYSVLONG and
SYSDATE/SYSTIME come from whatever runtime serves the request, so this variant
must not be read for SAS 9 envelope values.
Three review follow-ups on the guard the sas9 tree loads.
- A body that cannot be read is now its own error, naming the path and the
filesystem message. Reporting it as missing metadata sent the reader to a
@sasjs/utils reinstall that cannot fix a path.
- One statement of the dependency fact: showmeta arrived in @sasjs/utils
4.6.0 and @sasjs/server declares that floor from 3.12.3 (3.12.0 and
3.12.2 declare ^4.4.0, checked with npm view), so the message names the
server version to install instead of "reinstall the server".
- CONTRIBUTING lists _program and weboutPath among the predeclared globals,
which the guarded services use to locate the helper.
The sasjs tree's copy in dcMockUtils.js had the same two defects, so both
now say one thing rather than one tree being clearer than the other.
The sasjs tree has emitted its responses through the runtime's webout()
since #461; the SAS 9 tree under sas/mocks/sas9/ still assigned every
captured payload to _webout, so its envelope was frozen at the 2022
recording and its values carried the separators a template literal eats.
All 35 services now call webout('OPEN') / webout('OBJ'|'ARR', name, rows)
/ webout('CLOSE'), with the three services that carry column metadata
passing { showmeta } and loading the guard from the new weboutGuard.js.
The payloads come from the captures read as JS produces them, with the
backslashes a template literal silently drops restored: 121 separator
values across 14 services - OMS URIs (OMSOBJ:<Class>\<repo>.<id>),
BUILTIN\Administrators, SAS\sasdemo and D:\... / C:\... paths.
Staging is the preview of the next release, and what it should preview first is
the deployment itself. DC routes to /deploy while makedata is still on the drive
(at <appLoc>/services/admin) and stops doing so once the wizard has run it and it
has removed itself - so a drive that carries the services and the licence but no
control library opens on the deploy screen, and the visitor deploys the control
library from the wizard into their own IndexedDB drive.
The seeding step is now gated on a tag: only the demo (which a visitor should be
able to use straight away) runs makedata before the drive is captured. The
staging path gets a guard of its own, because a seed that leaked into it would
only be visible as a site that opens on the home screen: makedata must be on the
drive and no mpe_tables.json may be there.
CONTRIBUTING updated: the two sites now ship different drives from the same
build.
The root `package.json` carries no install or postinstall script, so the
guidance to rely on "the composite install script ... which installs both
subprojects" sends the reader after something that does not exist, and
`client/.npmrc` told them the same thing in a comment.
The install section also contradicted itself: it opened by telling the reader
to write the SheetJS registry token into `client/.npmrc`, and closed by
explaining that `client/.npmrc` is a tracked policy file that must not carry a
token (the pre-commit hook rejects one). The token goes in the user config,
which is what the first block now says.
The install sequence itself is unchanged - root `npm ci`, then `client` and
`sas` explicitly - because that is what the repo actually needs.
client/.npmrc is tracked now, so a token left in it from the old pattern is one
`git add -A` from being committed. Prose in CONTRIBUTING is not a control: the
pre-commit hook fails when the staged file holds _authToken, _auth or _password.
`npm run check:archiver` runs the directory-walk guard, so it is reproducible
before a push instead of only in CI, and the CI step calls the same script.
CONTRIBUTING now spells out the per-project installs, since the root .npmrc has
ignore-scripts=true and the composite install script therefore never runs.
Four write steps interpolated ${{ secrets.NPMRC}} straight into the shell source,
so a quote or $( in the value would break the step or run. They use the env form
the same workflows already use elsewhere.
CONTRIBUTING.md now says what the tracked client/.npmrc means for a developer:
run `npm run postinstall` after installing, `npx cypress install` for the suite,
and keep registry credentials in the user config rather than in the tracked file.
SASjs Server 3.x deprecated PORT and PROTOCOL and refuses to start when an
.env names neither HTTP_PORT nor HTTPS_PORT, exiting with
- no listener is enabled
- set HTTP_PORT and/or HTTPS_PORT (either may be used alone)
The three workflows downloaded releases/latest and wrote an .env without a
listener, so api-linux never bound :5000 and the 'Setup and start SASjs
server' step failed after its 120s wait (build runs for #363 and #365, both
failed there). Add HTTP_PORT=5000 to build.yaml, release.yaml and
lighthouse.yaml, and document it in the local-dev .env in CONTRIBUTING.md,
where the default-port claim is no longer true.
The pre-publish check only asserted the pair had two different values, which a
committed pair can satisfy while being broken (one half regenerated, a value
truncated on copy). It now imports the activation key as pkcs8 RSA-OAEP,
decrypts the licence key, parses the payload and exits non-zero otherwise, so a
drifted pair fails the deploy instead of publishing a site that cannot decrypt
its own licence.
CONTRIBUTING records that the two values are one key and are regenerated
together, and that the .gitleaksignore entries are line-scoped.
The published sites get the same key pair on every deploy, committed as
sas/mocks/mock-licence.json (site_id 123, the mock estate's SYSSITE) rather than
generated in the pipeline. The pair sits outside the deployed sync folder, so the
CI estate stays unlicensed and the Cypress specs keep exercising the free tier.
The Sites job copies the pair into the drive before the snapshot and refuses a
pair whose two values are identical, which is how the app spots an HTTP-format
key it cannot decrypt in a secure context. The pair trips the generic-api-key
scanner, so .gitleaksignore carries the two fingerprints.
The demo and staging sites ran on the free tier, so uploads and the other
licensed features were unavailable to anyone looking at them. The Sites job now
mints a self-contained key pair (site_id 123, the estate's SYSSITE) into the
drive before the snapshot, so both sites are fully usable.
Minted per deploy rather than committed, and the CI estate stays unlicensed so
the specs keep exercising the free tier.
The estate now ships no licence key at all - the placeholder word 'invalid' made
startupservice return an unusable key pair and produced the decrypt-failure page
on every HTTPS site. An unlicensed estate simply runs on the free tier.
loginAndUpdateValidKey therefore no longer infers 'already licensed' from the
nav tree: it detects the free-tier banner (#demo-banner) and uploads a key
through the licensing screen, which posts to admin/registerkey - the API the
licence page's upload button uses.
Emptying mock-storage/licence.json dropped every CI run to the free tier, where
the upload button is absent, so excel.cy.ts failed all 24 tests: the licensing
spec registers a key only when the activation page is shown, and an unlicensed
estate activates straight into the app.
The placeholder pair stays in the fixture, and the Sites job strips it from the
drive before the snapshot is taken, so the published sites still run unlicensed
without the decrypt-failure page.
Two review findings: the makedata call ends in || true (needed for its expected
HTML reply), so a genuine failure published a data-less site as a green build -
the drive is now checked for mpe_tables.json and at least ten tables before the
snapshot is taken. And the concurrency group is keyed on the target site rather
than the branch, so two release branches publishing staging cannot interleave.
The demo now deploys when semantic-release creates the tag, so it can never go
live on a commit whose release then fails, and the checkout carries the tag the
client's build stamp reports.
demo.datacontroller.io is now a multi-site Surfer instance: a release branch
deploys the staging.datacontroller.io site, and a push to main deploys the demo
(default site). Both run the same browser-hosted mock build.
Pushes to release/** now build the mocks as a browser-hosted site (SASjs CLI
6.1.0, streamConfig.mockWeb) and publish it to the Surfer app behind
demo.datacontroller.io, so the demo always matches the release branch.
The browser build seeds its drive from the compiled services, which carries no
DC data, so the job runs makedata on a throwaway SASjs Server and ships that
drive as mock-snapshot.json.
The key now comes from the backend licence (hot_license_key on the decrypted
licence), falling back to DEFAULT_HOT_LICENCE_KEY, so the hotLicenceKey tag
attribute the docs told you to set has no reader. Also reword the missing-tag
comment in parseDcAdapterSettings, which claimed there was no case to handle
while the boot screen reports one.
Review feedback on this PR: with .npmrc setting ignore-scripts=true the prepare
script never runs, so the native hooks in .git-hooks/ do not activate on a fresh
clone and a commit can go unchecked.
Both standard routes are added, the way sasjs/cli and sasjs/lint carry them:
- .pre-commit-config.yaml pins the gitleaks hook to v8.30.1, so `pre-commit
install` wires the secret scan without the one-time core.hooksPath command.
- Build-and-ng-test runs the same pinned @nogoo9/gitleaks binary as a step,
because `pre-commit install` is itself a manual command: CI is the enforcement
point. --no-git scans the working tree, so the gate is what the pull request
adds rather than the repo's history.
The scan runs before the job writes client/.npmrc from the NPMRC secret - that
file carries the private registry token, so scanning after it would flag the
token CI injects itself.
.npmrc is unchanged - ignore-scripts=true stays - and the native hooks stay for
anyone who prefers them. CONTRIBUTING.md names the framework route next to the
existing one-time command.
Verified: npm ci followed by the scan reports no leaks on a fresh checkout, the
client lint check passes, and pre-commit validate-config plus pre-commit run
--all-files both pass.
dc