The graph now holds only effective active rows, and a declared dependency that
resolves to an inactive row (existing, or one this submission deactivates) is
rejected: the engine reads only active rows and aborts every read of the table
on the unresolved reference.
The derived-variable / RLS guard goes back to rejecting every scope. Exempting
VIEW rules left them to fail a moment later in mp_filtercheck, which validates
against the physical table where the column does not exist, so the pairing is
refused up front with a message that says why.
RLS batches now exclude delete-marked rows, matching the badloadtypes check, so
a rule being removed is not validated against the derived configuration.
The dependency graph built at submit time pulled every open current row of
MPE_DERIVED into the union, while the engine only reads active=1 rows. An
inactive row that is still open could therefore abort a submission over a
cycle or a stale reference the engine never generates. The union now filters
active=1, matching mpe_derived.sas.
Both derived-variable guards also matched any active row level security rule,
whatever its scope. A VIEW rule is applied on the view the EDIT screen reads,
where a derived column exists, so naming one there is legitimate; only EDIT and
ALL rules run against the submitted data, where the column is absent. The
derived hook now filters rls_scope in ('EDIT','ALL'), and the RLS hook carries
rls_scope through work.batches to filter its side of the same guard, matching
the staging-side check in the same service.
The globvars check in licence.service reports a failed startupservice run as
well, so its modal needs the service name for the new-window button - that was
the fourth startup failure path and the only one still a dead end.
Also guard the decodeURIComponent call in the Cypress drive/folder intercept,
so a lone '%' in a URL cannot throw inside the handler and abort the request.
When the startupservice fails, the modal is a dead end: the adapter returned
no log, so there is nothing to act on. The modal now offers a button that runs
the same service through the JES / STP web app in a new browser window, where
the SAS log is returned as the page body.
`_debug=131` is always added (the log is the point), and `_executionTasks` is
never included, so on Viya the run is not submitted as a Compute task and
cannot hit the execution-task timeout. The URL is built by `buildServiceUrl`,
which works for SAS 9, Viya and SASjs Server, and is the same shape the deploy
configurator already uses to open makedata.
The startup failure modals now carry the service name, which is what shows the
button: all three failure branches (malformed response, missing properties,
request rejected) pass `public/startupservice`.
Confirming the "source file will be discarded" warning coerced every
numeric column of the preview with parseInt, which truncated the
fraction: 0.901045951946194 became 0, and 1234.56 became 1234.
parseFloat converts the same values without dropping precision, and a
cell the parser could not read as a number is left as text so the
editor's numeric validation flags it instead of showing NaN.
Adds an Excel spec case that reads the first data row before and after
the switch to edit mode and asserts the decimal value survives.
dctest.wideboy is 1004 columns (row_id, 500 char, 500 numeric, plus the
field and position loop variables), so the Test 2 banner says 200 x 1004 =
200,800 cells, matching the @details block. The row cap is unchanged.
The audit gate fails on advisories published against the pinned versions:
@angular/common 20.3.33 carries the XSRF token leakage reported in
GHSA-jfgx-762w-99vr, which reaches forms, platform-browser,
platform-browser-dynamic and router through their peer dependency on
common, and follow-redirects 1.16.0 carries GHSA-q3jh-wrc6-jgq8 and
GHSA-8r9p-f939-6h3c.
Both are patch bumps. The lockfile is edited in place - the nine
@angular pins and the follow-redirects entry - rather than re-resolved,
because a full install prunes the stale entries the lockfile carries and
turns a pin change into hundreds of unrelated lines. axios allows
^1.16.0, so the transitive bump needs no parent change.
npm audit --omit=dev reports 0 vulnerabilities at the repo root, in sas/
and in client/. ng build and the client unit suite (589) pass.
The doxygen list omitted mpeinit and mpeterm, which the service calls - both
are now declared the way the other services that use mpeinit declare it. The
compiler inlines them, as it does for those services. webout stays out of the
list: it is supplied by the framework rather than a macro in the repo, so that
one warning needs an exception in the linter.
dctest.wideboy has 1004 columns, not 1001 - row_id, char1-char500,
num1-num500, plus the loop variables field and position and the scratch
variable alphabet - so the test header's cell arithmetic is 200 x 1004 =
200,800. The 199 row cap is unchanged, floor(200000/1004) is still 199.
The MPE_X_WIDE mock comment said 200,400 cells for 200 x 1001 columns, which
was the old counting that added the client-owned delete column. The mock now
counts the table's own columns, so it is 200,200.
The message is built in three short assignments rather than one 161
character %let, which .sasjslint rejects (maxLineLength 107). The
resulting text is unchanged.
The getdata comment explained the truncation by contrast with the abort it
replaces. State what the code does and why instead. Same for the two
not-refused phrasings in the mock and the test header.
A selection over DC_MAXOBS_WEBEDIT or DC_MAXCELLS_WEBEDIT was refused with
"Table is too big", leaving the user with no grid and no way to see what the
filter had matched. getdata now serves the first rows that fit and flags it:
- the row cap is min(DC_MAXOBS_WEBEDIT, DC_MAXCELLS_WEBEDIT / columns), so a
wide table is still bounded by the cell budget
- sasparams carries TRUNCATED (0/1) and TRUNC_MSG, naming the limit reached
- the EDIT screen shows a warning triangle next to the table name, with the
message in a tooltip and the advice to filter
The mock mirrors the new behaviour, the Cypress spec asserts the truncated
grid and the tooltip (real hover, opacity checked so a hidden tooltip cannot
pass), and the SAS limit test now asserts the served row counts and message
rather than a cancelled child job.
- the join against MPE_DERIVED compared an upcased var_nm with the rule's
column as stored, so a case variant slipped past the explanatory rejection
and fell through to the generic filter check. Both sides are upcased.
- the invariant was enforced only from the RLS side: configuring a derived
variable for a table whose active rule already names it passed the derived
hook. The same check now runs there, against the incoming libref/dsn/var_nm.
A table's metadata columns are named per table - VAR_TXFROM, VAR_TXTO and
VAR_PROCESSED in MPE_TABLES hold those names, and they can be anything (the
bitemporal demo table uses VER_FROM_DTTM and VER_TO_DTTM). When they exist
they are physical columns of the target, so the engine's collision check
already refuses a VAR_NM that matches one, whatever it is called.
The hard-coded list treated three example names as special: it refused them on
tables that use other names, and told the reader something untrue about what
is reserved.
- getdata.test now checks the child job state before reading its outlib: a
cancelled child returns no webout, so the previous call's members were being
asserted against and the failures described the wrong problem.
- validatefilter.test's inputdatasets call needs a _PROGRAM column, which
mx_execute reads to decide what to run - without it the job stopped at
'The _PROGRAM column must exist'.
- stagedata.test.3 covers the new RLS check: an MPE_ROW_LEVEL_SECURITY rule
naming a derived variable must be rejected at save time.
- getrawdata selected its filter target with a bare '%else %let' in open code,
the shape that stops the step with 'Expected %DO not found'. Both branches
are %do/%end now, so the download service applies a filter at all.
- the conditional drop in the approve.jsdset step and the one against
MPSEARCH.vwsearch are dead: work.jsdata and work.vwsearch are already
rewritten without the derived columns, so the drops only warn.
- the post-edit hook checked that a reference is declared, but not that a
declared dependency is referenced. The engine rejects the stale shape at read
time, so the same check now rejects it at submit time.
- a row level security rule naming a derived variable is rejected when the rule
is saved, with a message that says why, instead of failing later at submit.
- getdata.test's cleanup deleted only DERIVED_NOTE, leaving DERIVED_NOTE2 with
a dependency on a variable that no longer exists, which aborted every call in
the following run - the cause of the repeated no-result runs.
Both services selected the filter target with
'%if cond %then %let x=y; %else %let x=z;'. In open code the %else is not
paired with the %then and the whole step stops with 'Expected %DO not found',
so neither service ever applied a filter: every assertion downstream failed and
the job produced no results at all.
Found on a real estate - the pattern works inside a macro definition, which is
why the same construct elsewhere in the codebase is fine.
Carries the abort-shape fix from sasjs/core#468 - the cycle abort no longer
re-enters its placement loop, and the validation aborts are driven by a count
at the point of call.
- mpe_derived_postedit ran %local at the top level of a file that is included
at execution time, so the hook stopped before validating anything and every
MPE_DERIVED submission failed. The macro language now lives in a macro
definition, errflag and errmsg are declared global, and the file calls it.
- the cycle check aborts after leaving its loop, and the self, undeclared and
field aborts are driven by a count at the point of call rather than by a
wrapping %if, so an abort cannot be outlived. The same shapes are fixed
upstream in sasjs/core#468.
- a case-variant duplicate survived mpe_loader, which compares raw values:
the hook now rejects it after normalisation and names the variable.
- stagedata dropped the derived columns from the output of the security step
rather than from its input, so a crafted submission could still satisfy an
RLS clause through a derived column. work.jsdata itself is rewritten now.
- the VAR_FMT validation rule rejected the bare w.d forms that the hook and
the engine accept.
- the migration carried the same four dangling insert headers as mpe_makedata,
so the v7.19 migration failed for existing installs.
- the filter fixtures in getdata.test and validatefilter.test had been caught
by the variable rename: the filter schema column is VARIABLE_NM.
- the TX_FROM/TX_TO definitions agree across the ddl, the migration and
mpe_makedatamodel.
@sasjs/core bumped to 5.7.1, which carries the where= fix.
Four stray insert lines and a missing one left the MPE_VALIDATIONS block for
MPE_DERIVED malformed, so makedata aborted and the control table was never
created - found by building the library on a real estate.
The format assertions in the test now read the emitted statement, because proc
contents on a view does not report a format in full ('DATE' for DATE9.).
mpe_derived carried its own copy of the derived variable engine. That engine
now ships in @sasjs/core 5.7.0 as mp_derivedvars, so the local copy is gone and
mpe_derived is a wrapper that supplies what is Data Controller's own: the
control table, the bitemporal filter that picks the current active row, and the
list of metadata names that may not be used as a variable.
The macro goes from 511 lines to 102. The contract and the generated statement
format are unchanged, so the callers in the six services need no edit.
A test covers the wrapper's own responsibility: a closed version of a row and
an inactive row are both ignored.
A label at the 256 character limit produces an attrib statement of about 300
characters, which the default record length wraps, splitting the statement and
breaking the view.
Same harness fix as the core test: the setup inserts create the SQL automatic
macro variables, so the snapshot has to come after them, and the framework's
_SASJS_WEBOUT_HEADERS is ignored.
A configured label replaces the default NAME=expression; when it is missing or
blank the default is used, so existing rows are unaffected.
- MPE_DERIVED gains VAR_LABEL varchar(256), in the ddl, mpe_makedatamodel and
the v7.19 migration
- mpe_derived uses it, through the same escaping and 256-char truncation as
the default label
- the hook trims it; the mock mirrors the fallback
- test 1 now sets a label on BAND and asserts it
The label had 256 chars of storage, so the doubled quotes were truncated on
assignment; a cut between the halves of an escape leaves an unterminated
literal and the view fails to build. The label now holds 512 chars, quotes
are doubled, then the 256 boundary is applied, retreating one character when
the cut lands on an odd quote count.
GHSA-jfgx-762w-99vr (XSRF token leakage in the HTTP client, an incomplete fix
for CVE-2025-66035) covers @angular/common 20.0.0 - 20.3.33, and the 'Check
audit' step fails on any vulnerability, so every branch is red until this
lands.
The nine packages are pinned exactly and must move together - Angular pins its
siblings at the same patch level - so package.json and the lock are updated as
a set. The local tarball for @sheet/crypto is only present in encrypted form,
so npm cannot run here; the client build and unit tests in CI are the proof.
Allowing a derived variable to reuse a physical column's name was wrong: an
intra-row calculation (SUMMER = SUMMER + 1) implies a retained value, which
depends on the order rows are read. The same argument rules out a variable
listing itself as a dependency.
Reverted:
- the name collision is an error again ('VAR_NM collides with a column in the
target table')
- VAR_DEPS may not list the variable itself
- the physical column list is back to mf_getvarlist - the type/length check
only existed to support shadowing
- the droplist guards in stagedata / viewdata / getrawdata, and the duplicate
suppression in getdata, are no longer needed
- the docs and both test suites drop the shadowing case
cols1 already holds the physical column, so appending the derived row for the
same name would send it twice. The derived row is now appended only when the
name is new; a shadowed column keeps the physical label and format, while its
value comes from the view.
With shadowing allowed, a derived name can also be a physical column. The
derived droplist was built from every derived name, so such a column would
have been dropped on submission (losing the user's edit) and from the view
and raw downloads (losing a real column).
stagedata, viewdata and getrawdata now subtract the target's own columns
from the droplist, so only genuinely virtual columns are dropped.
SUMMER = SUMMER + 1 is a legitimate definition: the name matches an existing
column, the expression reads the physical value, and the view carries the
derived one under the same name.
- mpe_derived now takes the physical column list from mp_getcols, so it has
type and length in one pass; a shadowed name must match the column's type,
and its length may not exceed it (numeric lengths must be equal), as attrib
cannot change either on an existing variable
- a self reference is only an error when the name is not a physical column
- self references are not graph edges, so the dependency sort is unaffected
- the post edit hook no longer rejects a self reference, since it cannot see
the target library; it still rejects cycles between distinct names
- mpe_derived now uses mf_getuniquename for its working tables, matching the
core macro, and carries the same placement sequence fix
- test 4 covers shadowing, docs cover the rule
Two high advisories were published against 1.16.0 in the last few hours
(GHSA-q3jh-wrc6-jgq8, GHSA-8r9p-f939-6h3c), and the 'Check audit' step fails
on any vulnerability, so every run after 18:33 UTC today is red - including
the ones on this branch that had passed earlier.
Lockfile only, no source change: 1.16.1 satisfies the ^1.15.6 range axios
asks for. Worth cherry-picking to main, since it blocks every PR.
Same defect as the core review found in mp_derivedvars: the final assemble
merged the placement table by name, but that table is built in dependency
rounds and is not name ordered, so the BY-merge failed whenever the rounds
were not alphabetical, and seq was a BY-group counter rather than the
dependency order.
Each round now stamps seq=_n_+nplaced, the merge runs on the name-sorted
tables, and the result is sorted by seq. Also guards the physical-column
in-list for a zero-column table, and checks libds exists before running.
The cycle check was already in the post edit hook and in mpe_derived - the
topological sort places a variable only once its derived inputs are placed,
and if a round places nothing the remainder is a cycle. What it reported was
just the set of names that could not be placed, which does not tell the
administrator what to change.
The message now walks the edges from the first unresolved variable and names
the loop, eg:
VAR_DEPENDENCIES contains a cycle: A -> B -> C -> A
Names are collected in a dotted seen list, so a lead-in to the loop is shown
as well and the walk stops at the first repeat.
The test fixtures now state the inputs their expressions actually use:
LIBREF DSN, DERIVED_NOTE, PRIMARY_KEY_FIELD, and none for a literal-only
expression. The getdata label assertion follows the new <VAR_NM>=<expr>
label.
A sequence number has to be maintained by the administrator: inserting a
variable means renumbering everything after it, and a wrong number fails
silently - the derived value comes back missing, not wrong.
VAR_DEPENDENCIES names every input the expression uses, physical columns and
derived variables alike, eg 'REGION STATE'. The statements are emitted in
dependency order, so a variable can build on another one with no sequence to
maintain, and a cycle is rejected.
The declaration is checked against the expression in both directions: a
derived variable referenced by VAR_LOGIC must be declared, and a declared
name that the expression does not reference is an error; a declared name must
resolve to a column of the target table or to a derived variable of the same
table.
Changes:
- MPE_DERIVED: VAR_ORDER (num) becomes VAR_DEPENDENCIES (varchar(4000)), in
the ddl, mpe_makedatamodel and the v7.19 migration
- mpe_derived: dependency validation and a topological sort replace the
VAR_ORDER sort and the forward-reference check
- mpe_derived_postedit: checks self reference, undeclared derived references
and cycles at submit time, across the incoming rows plus the current rows
they do not replace
- MPE_VALIDATIONS: the VAR_ORDER MINVAL/MAXVAL rules are replaced by UPCASE
and a character pattern on VAR_DEPENDENCIES
- mocks: the dependency order is applied in JS, so a chained variable works
even when the rows are loaded child first
- docs and tests updated
Found running the macro on a real Viya compute server:
- verify() was applied to fixed width variables, so the blank padding was
always outside the allowed character list and every name was rejected. The
check now runs on strip(name) and strip(fmt).
- the empty outvars dataset used 'length name $32; stop;', which the Viya
compute server rejects with 'Variable name is uninitialized'. It now
assigns the variable and subsets with 'if 0;'.
- the label is now <VAR_NM>=<expression> rather than 'DERIVED: <expression>',
so the grid shows the variable name as well as the logic.
mpe_loader already sorts work.staging_ds by the business key
(buskey='LIBREF DSN VAR_NM') and aborts on duplicates before the post edit
hook runs, so the same check in the hook could never fire.
The check in %mpe_derived stays: it reads the control table, which can also
be edited outside the app.
Client side rules on the table itself, so mistakes show up in the grid:
- NOTNULL on LIBREF, DSN, VAR_NM, VAR_ORDER, VAR_TYPE, VAR_LEN, VAR_LOGIC
- UPCASE on LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_FMT
- MINVAL 1 / MAXVAL 999 on VAR_ORDER, MINVAL 1 / MAXVAL 32767 on VAR_LEN
- HARDREGEX on VAR_NM, VAR_FMT and VAR_LOGIC
- HARDSELECT on VAR_TYPE, with C and N in MPE_SELECTBOX
Seeded in mpe_makedata and in the v7.19 migration. The post edit hook stays
the server side enforcement, since these rules are evaluated in the browser.
Also fixes the MPE_DERIVED create in mpe_makedatamodel, which still used the
old VARIABLE_* column names.
A post edit hook on MPE_DERIVED applies the configuration rules when a row
is submitted, rather than only when the EDIT screen next builds the view.
It normalises LIBREF, DSN, VAR_NM, VAR_TYPE and VAR_FMT, and rejects:
- a missing LIBREF, DSN or VAR_NM
- an invalid or reserved name, or one starting with a digit
- a VAR_ORDER that is not a positive integer
- a VAR_TYPE other than C or N
- a VAR_LEN outside 1-32767 (character) or 1-8 (numeric)
- a VAR_FMT that is not a plain format name
- a VAR_LOGIC that is not a single expression
- the same VAR_NM twice for one table
- a reference to a derived variable that is not defined earlier
The name collision with a physical column is left to %mpe_derived, which runs
with the target library assigned; the hook does not have it.
MPE_DERIVED columns:
VARIABLE_NM -> VAR_NM
VARIABLE_ORDER -> VAR_ORDER
VARIABLE_TYPE -> VAR_TYPE
VARIABLE_LENGTH -> VAR_LEN
VARIABLE_FORMAT -> VAR_FMT
VARIABLE_LOGIC -> VAR_LOGIC
Updated in the ddl, mpe_makedatamodel, the migration, the MPE_TABLES buskey,
the macro (reads and validation messages), the tests, the mocks and the docs.
The macro carries the ordering and statement-generation logic, so it is
worth a test of its own:
- character, numeric and chained variables: statements, view, columns
- label and default/configured formats
- a filter on a derived column
- the no-config path: no view, no statements, empty outvars
The configuration checks call mp_abort, so they stay out of the suite.
Review follow-up.
- stagedata: build the derived droplist before the ULOAD filter, and drop the
derived columns in the badrecords step, so the row-level-security check runs
against physical columns only. A submitted derived value can no longer
satisfy an RLS clause that names a derived column.
- viewdata: the full-table search view now drops the derived columns too, so
a free-text search cannot match a value in an EDIT-only column.
- mpe_derived: upcase the libds when matching MPE_DERIVED, so a lower-case
table reference still finds its configuration instead of silently showing
no derived columns.
dc
allan