demo.datacontroller.io is now a multi-site Surfer instance: a release branch
deploys the staging.datacontroller.io site, and a push to main deploys the demo
(default site). Both run the same browser-hosted mock build.
Pushes to release/** now build the mocks as a browser-hosted site (SASjs CLI
6.1.0, streamConfig.mockWeb) and publish it to the Surfer app behind
demo.datacontroller.io, so the demo always matches the release branch.
The browser build seeds its drive from the compiled services, which carries no
DC data, so the job runs makedata on a throwaway SASjs Server and ships that
drive as mock-snapshot.json.
streamConfig.mockWeb turns the mock site on for the target whose services are JavaScript, so sasjs build produces sasjsbuild/mock alongside the deploy artefact. mockWeb is independent of streamWeb, so the same target still deploys for real.
VAR_NM carries CASE=UPCASE and the post edit hook upcases every non-delete
staged row, so a name typed in the grid reaches submit uppercase: a delete plus
a re-add of it is the same raw primary key and mpe_loader's dup check rejects
that at approval. Test 6's comment claimed the opposite, so it now states the
truth - only a stored name that is not uppercase reaches submit as two PKs, ie
a name written outside the editor.
Test 7 covers the other arm of the exclusion in the same check: setting an
inactive row active again while another active row depends on it. That starting
state is also one the hook refuses to create, so the case is defensive; both
tests prove the guard does not block the submission that repairs the data.
The inactive set behind the bad-dependency check admitted every delete-marked
staged name, so a submission that delete-marks one case variant of a variable
and stages the same name again as a live active row (the two are distinct keys
to the loader's case sensitive duplicate check) matched the dep against the
inactive copy and aborted a configuration the engine resolves fine.
Both arms now exclude a name that this submission also stages as a live active
row - the staged one, and the control row it replaces. stagedata.test.4 gets a
Test 6 for that shape: delete 'Base_X', stage 'BASE_X', with a surviving active
row declaring VAR_DEPS=BASE_X, which must stage successfully.
Verified: sasjs lint (no new warnings), sasjs cb -t nextviya with @sasjs/core
6.0.0, and the built output carries the new predicate and the test. No SAS
estate was reachable from this box (nextviya and sas.4gl.io both time out), so
the test has not been executed on a real engine yet.
@sasjs/core 6.0.0 renamed the session globals to a _SASJS_ prefix and sets
_SASJS_WORK only; SASJSWORK no longer exists (and sasjs/server's pre-program
compat aliases never covered it). testsetup.sas still read &sasjswork in both
the SASJS and SASMETA arms of %update_settings, so under the new pin those
filename statements built unrooted paths and ms_createfile / mm_createstp wrote
outside the session work directory, breaking sasjs test setup on SASJS and SAS9.
mp_init() is called at the top of the script, so _SASJS_WORK is defined here.
Two conflicts, both resolved by keeping the intent of each side:
- sas/mocks/sasjs/services/editors/getdata.js: the PK derivation and the
pkFirst() ordering from this branch, then the derived-variable block from the
release branch. The derived columns are added to the schema before the
filter, so the PK-first ordering still applies to the full column set.
- client/package-lock.json: taken from this branch. The merged package.json
differs from the release branch only in the fast-uri override (4.2.3 here,
3.1.8 there); the release branch already carries the same @angular 20.3.34
and @sasjs/adapter 5.1.2 pins, so the lock is unchanged from 74060c0.
Gates on the merged tree: npm audit --omit=dev 0 vulnerabilities, prettier
clean, 603/603 unit tests, ng build, license-checker. Mock services re-probed
against a local SASjs Server: MPE_ROW_LEVEL_SECURITY publishes PKCNT 10 with
the business keys first and no RLS_RK, an edit diffs as UPDATED 1 with the
base RK, and the derived table TESTDATA.DERIVED_DEMO still serves its computed
columns.
Review follow-ups on the rk_underlying handling:
- mpeAssignRetainedKey (dcMockUtils) mirrors %mp_retainedkey: any submitted RK
column is dropped and every row's RK is re-derived from the business key, so a
stale RK in a hand-built payload cannot survive into the diff. Business keys
compare the way SAS does - case-sensitive, trailing blanks ignored.
- editors/loadfile excludes the RK from its expected-columns check for these
tables, as loadfile.sas does, and runs the derivation before staging, so a
real CSV upload (which legitimately omits the RK) is accepted and its diff
matches instead of reporting every row as NEW.
- both staging paths add the derived column to the staged headers, so
MPE_SUBMIT.INPUT_VARS counts the width mpe_loader would see.
The npm audit gate started failing on advisories that landed upstream, so
nothing can merge until the client tree is patched:
- @angular/* 20.3.33 -> 20.3.34 (XSRF token leak in HttpClient, SSR DoS in
the router)
- @sasjs/adapter 5.1.0 -> 5.1.2 (brings a patched axios)
- fast-uri override 3.1.8 -> 4.2.3 (the override was pinning the vulnerable
version; ajv only needs it for URI resolution)
- lockfile refresh takes moment, undici, follow-redirects and brace-expansion
to their patched versions, and re-resolves the @angular-devkit build
toolchain within its existing ^ range
Verified in client/: npm audit --omit=dev reports 0 vulnerabilities, npm run
test:headless is 589/589, npm run build succeeds, lint:check and
license-checker pass.
getdata.sas exposes coalesce(rk_underlying, buskey) as the PK and drops the
RK column, so the editor shows the business key as read-only and never sends
the RK. The JS mock still published the RK as the PK and left it in the grid,
and stagedata had no loader to derive it back, so an edit to such a table
(MPE_ROW_LEVEL_SECURITY, MPE_SELECTBOX) could not be diffed.
The mock now mirrors that contract: PK/PKCNT come from rk_underlying, the RK
column is withheld and the PK variables lead COLHEADERS, and stagedata assigns
the RK from the business key - keeping the one the base row already holds and
minting the next value for a new key.
CLI 6 ships the core 6.0.0 macros, so builds now carry the _SASJS_ session
variables; Data Controller reads none of them and calls %mp_init() with no
arguments. Adapter 5.1.2 is a patch on the 5.1.0 already in use.
Verified locally: sasjs cb -t nextviya and sasjs lint, client unit suite
603/603 and the production build.
The inactive set behind the VAR_DEPS check took staged rows that are not
delete-marked with active ne 1, plus control rows with active ne 1. A staged
DELETE of a relied-on row was in neither arm, so it passed every guard and the
engine aborted every read after approval. Delete-marked staged rows are now in
the inactive set, and stagedata.test.4 covers it.
v6 renames the session macro variables to the _SASJS_ prefix and removes
mp_init(prefix=). Data Controller calls %mp_init() with no arguments and reads
none of the renamed variables, so no code change is needed; every macro file
present in 5.7.2 is still present in 6.0.0.
%if %then %let ...; %else %let ...; is not valid in open code (SAS stops with
"Expected %DO not found"), which is why these sites used %do/%end blocks. One
%let with %sysfunc(ifc()) needs no blocks and reads as what it is: a choice
between two dataset names. The droplist block in getrawdata/viewdata keeps its
%if %then %do; there is no %else to pair.
A hook is %included from %mp_include, so its open-code %let runs in that
macro's local scope: a %global for the same name inside the hook raises
"exists in a local environment", sets SYSCC, and puts the session into syntax
check mode - every check after it silently skipped while the step still
aborts. The derived hook now sets its (renamed) control variables in the
including scope and does not %global them.
The RLS hook also referenced rls_msg before assigning it whenever there was no
derived conflict, which produced an unresolved-symbol WARNING and SYSCC=4 on
every clean rule submission. It is initialised now.
Four cases: a dependency on an inactive row aborts, deactivating a relied-on
row aborts, an inactive row does not block an unrelated variable, and a
delete-marked RLS rule naming a derived variable loads.
The graph now holds only effective active rows, and a declared dependency that
resolves to an inactive row (existing, or one this submission deactivates) is
rejected: the engine reads only active rows and aborts every read of the table
on the unresolved reference.
The derived-variable / RLS guard goes back to rejecting every scope. Exempting
VIEW rules left them to fail a moment later in mp_filtercheck, which validates
against the physical table where the column does not exist, so the pairing is
refused up front with a message that says why.
RLS batches now exclude delete-marked rows, matching the badloadtypes check, so
a rule being removed is not validated against the derived configuration.
The dependency graph built at submit time pulled every open current row of
MPE_DERIVED into the union, while the engine only reads active=1 rows. An
inactive row that is still open could therefore abort a submission over a
cycle or a stale reference the engine never generates. The union now filters
active=1, matching mpe_derived.sas.
Both derived-variable guards also matched any active row level security rule,
whatever its scope. A VIEW rule is applied on the view the EDIT screen reads,
where a derived column exists, so naming one there is legitimate; only EDIT and
ALL rules run against the submitted data, where the column is absent. The
derived hook now filters rls_scope in ('EDIT','ALL'), and the RLS hook carries
rls_scope through work.batches to filter its side of the same guard, matching
the staging-side check in the same service.
The globvars check in licence.service reports a failed startupservice run as
well, so its modal needs the service name for the new-window button - that was
the fourth startup failure path and the only one still a dead end.
Also guard the decodeURIComponent call in the Cypress drive/folder intercept,
so a lone '%' in a URL cannot throw inside the handler and abort the request.
When the startupservice fails, the modal is a dead end: the adapter returned
no log, so there is nothing to act on. The modal now offers a button that runs
the same service through the JES / STP web app in a new browser window, where
the SAS log is returned as the page body.
`_debug=131` is always added (the log is the point), and `_executionTasks` is
never included, so on Viya the run is not submitted as a Compute task and
cannot hit the execution-task timeout. The URL is built by `buildServiceUrl`,
which works for SAS 9, Viya and SASjs Server, and is the same shape the deploy
configurator already uses to open makedata.
The startup failure modals now carry the service name, which is what shows the
button: all three failure branches (malformed response, missing properties,
request rejected) pass `public/startupservice`.
Confirming the "source file will be discarded" warning coerced every
numeric column of the preview with parseInt, which truncated the
fraction: 0.901045951946194 became 0, and 1234.56 became 1234.
parseFloat converts the same values without dropping precision, and a
cell the parser could not read as a number is left as text so the
editor's numeric validation flags it instead of showing NaN.
Adds an Excel spec case that reads the first data row before and after
the switch to edit mode and asserts the decimal value survives.
dctest.wideboy is 1004 columns (row_id, 500 char, 500 numeric, plus the
field and position loop variables), so the Test 2 banner says 200 x 1004 =
200,800 cells, matching the @details block. The row cap is unchanged.
The audit gate fails on advisories published against the pinned versions:
@angular/common 20.3.33 carries the XSRF token leakage reported in
GHSA-jfgx-762w-99vr, which reaches forms, platform-browser,
platform-browser-dynamic and router through their peer dependency on
common, and follow-redirects 1.16.0 carries GHSA-q3jh-wrc6-jgq8 and
GHSA-8r9p-f939-6h3c.
Both are patch bumps. The lockfile is edited in place - the nine
@angular pins and the follow-redirects entry - rather than re-resolved,
because a full install prunes the stale entries the lockfile carries and
turns a pin change into hundreds of unrelated lines. axios allows
^1.16.0, so the transitive bump needs no parent change.
npm audit --omit=dev reports 0 vulnerabilities at the repo root, in sas/
and in client/. ng build and the client unit suite (589) pass.
The doxygen list omitted mpeinit and mpeterm, which the service calls - both
are now declared the way the other services that use mpeinit declare it. The
compiler inlines them, as it does for those services. webout stays out of the
list: it is supplied by the framework rather than a macro in the repo, so that
one warning needs an exception in the linter.
dctest.wideboy has 1004 columns, not 1001 - row_id, char1-char500,
num1-num500, plus the loop variables field and position and the scratch
variable alphabet - so the test header's cell arithmetic is 200 x 1004 =
200,800. The 199 row cap is unchanged, floor(200000/1004) is still 199.
The MPE_X_WIDE mock comment said 200,400 cells for 200 x 1001 columns, which
was the old counting that added the client-owned delete column. The mock now
counts the table's own columns, so it is 200,200.
The message is built in three short assignments rather than one 161
character %let, which .sasjslint rejects (maxLineLength 107). The
resulting text is unchanged.
The getdata comment explained the truncation by contrast with the abort it
replaces. State what the code does and why instead. Same for the two
not-refused phrasings in the mock and the test header.
A selection over DC_MAXOBS_WEBEDIT or DC_MAXCELLS_WEBEDIT was refused with
"Table is too big", leaving the user with no grid and no way to see what the
filter had matched. getdata now serves the first rows that fit and flags it:
- the row cap is min(DC_MAXOBS_WEBEDIT, DC_MAXCELLS_WEBEDIT / columns), so a
wide table is still bounded by the cell budget
- sasparams carries TRUNCATED (0/1) and TRUNC_MSG, naming the limit reached
- the EDIT screen shows a warning triangle next to the table name, with the
message in a tooltip and the advice to filter
The mock mirrors the new behaviour, the Cypress spec asserts the truncated
grid and the tooltip (real hover, opacity checked so a hidden tooltip cannot
pass), and the SAS limit test now asserts the served row counts and message
rather than a cancelled child job.
- the join against MPE_DERIVED compared an upcased var_nm with the rule's
column as stored, so a case variant slipped past the explanatory rejection
and fell through to the generic filter check. Both sides are upcased.
- the invariant was enforced only from the RLS side: configuring a derived
variable for a table whose active rule already names it passed the derived
hook. The same check now runs there, against the incoming libref/dsn/var_nm.
A table's metadata columns are named per table - VAR_TXFROM, VAR_TXTO and
VAR_PROCESSED in MPE_TABLES hold those names, and they can be anything (the
bitemporal demo table uses VER_FROM_DTTM and VER_TO_DTTM). When they exist
they are physical columns of the target, so the engine's collision check
already refuses a VAR_NM that matches one, whatever it is called.
The hard-coded list treated three example names as special: it refused them on
tables that use other names, and told the reader something untrue about what
is reserved.
- getdata.test now checks the child job state before reading its outlib: a
cancelled child returns no webout, so the previous call's members were being
asserted against and the failures described the wrong problem.
- validatefilter.test's inputdatasets call needs a _PROGRAM column, which
mx_execute reads to decide what to run - without it the job stopped at
'The _PROGRAM column must exist'.
- stagedata.test.3 covers the new RLS check: an MPE_ROW_LEVEL_SECURITY rule
naming a derived variable must be rejected at save time.
- getrawdata selected its filter target with a bare '%else %let' in open code,
the shape that stops the step with 'Expected %DO not found'. Both branches
are %do/%end now, so the download service applies a filter at all.
- the conditional drop in the approve.jsdset step and the one against
MPSEARCH.vwsearch are dead: work.jsdata and work.vwsearch are already
rewritten without the derived columns, so the drops only warn.
- the post-edit hook checked that a reference is declared, but not that a
declared dependency is referenced. The engine rejects the stale shape at read
time, so the same check now rejects it at submit time.
- a row level security rule naming a derived variable is rejected when the rule
is saved, with a message that says why, instead of failing later at submit.
- getdata.test's cleanup deleted only DERIVED_NOTE, leaving DERIVED_NOTE2 with
a dependency on a variable that no longer exists, which aborted every call in
the following run - the cause of the repeated no-result runs.
Both services selected the filter target with
'%if cond %then %let x=y; %else %let x=z;'. In open code the %else is not
paired with the %then and the whole step stops with 'Expected %DO not found',
so neither service ever applied a filter: every assertion downstream failed and
the job produced no results at all.
Found on a real estate - the pattern works inside a macro definition, which is
why the same construct elsewhere in the codebase is fine.
Carries the abort-shape fix from sasjs/core#468 - the cycle abort no longer
re-enters its placement loop, and the validation aborts are driven by a count
at the point of call.
- mpe_derived_postedit ran %local at the top level of a file that is included
at execution time, so the hook stopped before validating anything and every
MPE_DERIVED submission failed. The macro language now lives in a macro
definition, errflag and errmsg are declared global, and the file calls it.
- the cycle check aborts after leaving its loop, and the self, undeclared and
field aborts are driven by a count at the point of call rather than by a
wrapping %if, so an abort cannot be outlived. The same shapes are fixed
upstream in sasjs/core#468.
- a case-variant duplicate survived mpe_loader, which compares raw values:
the hook now rejects it after normalisation and names the variable.
- stagedata dropped the derived columns from the output of the security step
rather than from its input, so a crafted submission could still satisfy an
RLS clause through a derived column. work.jsdata itself is rewritten now.
- the VAR_FMT validation rule rejected the bare w.d forms that the hook and
the engine accept.
- the migration carried the same four dangling insert headers as mpe_makedata,
so the v7.19 migration failed for existing installs.
- the filter fixtures in getdata.test and validatefilter.test had been caught
by the variable rename: the filter schema column is VARIABLE_NM.
- the TX_FROM/TX_TO definitions agree across the ddl, the migration and
mpe_makedatamodel.
@sasjs/core bumped to 5.7.1, which carries the where= fix.
Four stray insert lines and a missing one left the MPE_VALIDATIONS block for
MPE_DERIVED malformed, so makedata aborted and the control table was never
created - found by building the library on a real estate.
The format assertions in the test now read the emitted statement, because proc
contents on a view does not report a format in full ('DATE' for DATE9.).
mpe_derived carried its own copy of the derived variable engine. That engine
now ships in @sasjs/core 5.7.0 as mp_derivedvars, so the local copy is gone and
mpe_derived is a wrapper that supplies what is Data Controller's own: the
control table, the bitemporal filter that picks the current active row, and the
list of metadata names that may not be used as a variable.
The macro goes from 511 lines to 102. The contract and the generated statement
format are unchanged, so the callers in the six services need no edit.
A test covers the wrapper's own responsibility: a closed version of a row and
an inactive row are both ignored.
A label at the 256 character limit produces an attrib statement of about 300
characters, which the default record length wraps, splitting the statement and
breaking the view.
Same harness fix as the core test: the setup inserts create the SQL automatic
macro variables, so the snapshot has to come after them, and the framework's
_SASJS_WEBOUT_HEADERS is ignored.
A configured label replaces the default NAME=expression; when it is missing or
blank the default is used, so existing rows are unaffected.
- MPE_DERIVED gains VAR_LABEL varchar(256), in the ddl, mpe_makedatamodel and
the v7.19 migration
- mpe_derived uses it, through the same escaping and 256-char truncation as
the default label
- the hook trims it; the mock mirrors the fallback
- test 1 now sets a label on BAND and asserts it
The label had 256 chars of storage, so the doubled quotes were truncated on
assignment; a cut between the halves of an escape leaves an unterminated
literal and the view fails to build. The label now holds 512 chars, quotes
are doubled, then the 256 boundary is applied, retreating one character when
the cut lands on an odd quote count.
GHSA-jfgx-762w-99vr (XSRF token leakage in the HTTP client, an incomplete fix
for CVE-2025-66035) covers @angular/common 20.0.0 - 20.3.33, and the 'Check
audit' step fails on any vulnerability, so every branch is red until this
lands.
The nine packages are pinned exactly and must move together - Angular pins its
siblings at the same patch level - so package.json and the lock are updated as
a set. The local tarball for @sheet/crypto is only present in encrypted form,
so npm cannot run here; the client build and unit tests in CI are the proof.
Allowing a derived variable to reuse a physical column's name was wrong: an
intra-row calculation (SUMMER = SUMMER + 1) implies a retained value, which
depends on the order rows are read. The same argument rules out a variable
listing itself as a dependency.
Reverted:
- the name collision is an error again ('VAR_NM collides with a column in the
target table')
- VAR_DEPS may not list the variable itself
- the physical column list is back to mf_getvarlist - the type/length check
only existed to support shadowing
- the droplist guards in stagedata / viewdata / getrawdata, and the duplicate
suppression in getdata, are no longer needed
- the docs and both test suites drop the shadowing case
dc
allan