A table's metadata columns are named per table - VAR_TXFROM, VAR_TXTO and
VAR_PROCESSED in MPE_TABLES hold those names, and they can be anything (the
bitemporal demo table uses VER_FROM_DTTM and VER_TO_DTTM). When they exist
they are physical columns of the target, so the engine's collision check
already refuses a VAR_NM that matches one, whatever it is called.
The hard-coded list treated three example names as special: it refused them on
tables that use other names, and told the reader something untrue about what
is reserved.
- getdata.test now checks the child job state before reading its outlib: a
cancelled child returns no webout, so the previous call's members were being
asserted against and the failures described the wrong problem.
- validatefilter.test's inputdatasets call needs a _PROGRAM column, which
mx_execute reads to decide what to run - without it the job stopped at
'The _PROGRAM column must exist'.
- stagedata.test.3 covers the new RLS check: an MPE_ROW_LEVEL_SECURITY rule
naming a derived variable must be rejected at save time.
- getrawdata selected its filter target with a bare '%else %let' in open code,
the shape that stops the step with 'Expected %DO not found'. Both branches
are %do/%end now, so the download service applies a filter at all.
- the conditional drop in the approve.jsdset step and the one against
MPSEARCH.vwsearch are dead: work.jsdata and work.vwsearch are already
rewritten without the derived columns, so the drops only warn.
- the post-edit hook checked that a reference is declared, but not that a
declared dependency is referenced. The engine rejects the stale shape at read
time, so the same check now rejects it at submit time.
- a row level security rule naming a derived variable is rejected when the rule
is saved, with a message that says why, instead of failing later at submit.
- getdata.test's cleanup deleted only DERIVED_NOTE, leaving DERIVED_NOTE2 with
a dependency on a variable that no longer exists, which aborted every call in
the following run - the cause of the repeated no-result runs.
Both services selected the filter target with
'%if cond %then %let x=y; %else %let x=z;'. In open code the %else is not
paired with the %then and the whole step stops with 'Expected %DO not found',
so neither service ever applied a filter: every assertion downstream failed and
the job produced no results at all.
Found on a real estate - the pattern works inside a macro definition, which is
why the same construct elsewhere in the codebase is fine.
Carries the abort-shape fix from sasjs/core#468 - the cycle abort no longer
re-enters its placement loop, and the validation aborts are driven by a count
at the point of call.
- mpe_derived_postedit ran %local at the top level of a file that is included
at execution time, so the hook stopped before validating anything and every
MPE_DERIVED submission failed. The macro language now lives in a macro
definition, errflag and errmsg are declared global, and the file calls it.
- the cycle check aborts after leaving its loop, and the self, undeclared and
field aborts are driven by a count at the point of call rather than by a
wrapping %if, so an abort cannot be outlived. The same shapes are fixed
upstream in sasjs/core#468.
- a case-variant duplicate survived mpe_loader, which compares raw values:
the hook now rejects it after normalisation and names the variable.
- stagedata dropped the derived columns from the output of the security step
rather than from its input, so a crafted submission could still satisfy an
RLS clause through a derived column. work.jsdata itself is rewritten now.
- the VAR_FMT validation rule rejected the bare w.d forms that the hook and
the engine accept.
- the migration carried the same four dangling insert headers as mpe_makedata,
so the v7.19 migration failed for existing installs.
- the filter fixtures in getdata.test and validatefilter.test had been caught
by the variable rename: the filter schema column is VARIABLE_NM.
- the TX_FROM/TX_TO definitions agree across the ddl, the migration and
mpe_makedatamodel.
@sasjs/core bumped to 5.7.1, which carries the where= fix.
Four stray insert lines and a missing one left the MPE_VALIDATIONS block for
MPE_DERIVED malformed, so makedata aborted and the control table was never
created - found by building the library on a real estate.
The format assertions in the test now read the emitted statement, because proc
contents on a view does not report a format in full ('DATE' for DATE9.).
mpe_derived carried its own copy of the derived variable engine. That engine
now ships in @sasjs/core 5.7.0 as mp_derivedvars, so the local copy is gone and
mpe_derived is a wrapper that supplies what is Data Controller's own: the
control table, the bitemporal filter that picks the current active row, and the
list of metadata names that may not be used as a variable.
The macro goes from 511 lines to 102. The contract and the generated statement
format are unchanged, so the callers in the six services need no edit.
A test covers the wrapper's own responsibility: a closed version of a row and
an inactive row are both ignored.
A label at the 256 character limit produces an attrib statement of about 300
characters, which the default record length wraps, splitting the statement and
breaking the view.
Same harness fix as the core test: the setup inserts create the SQL automatic
macro variables, so the snapshot has to come after them, and the framework's
_SASJS_WEBOUT_HEADERS is ignored.
A configured label replaces the default NAME=expression; when it is missing or
blank the default is used, so existing rows are unaffected.
- MPE_DERIVED gains VAR_LABEL varchar(256), in the ddl, mpe_makedatamodel and
the v7.19 migration
- mpe_derived uses it, through the same escaping and 256-char truncation as
the default label
- the hook trims it; the mock mirrors the fallback
- test 1 now sets a label on BAND and asserts it
The label had 256 chars of storage, so the doubled quotes were truncated on
assignment; a cut between the halves of an escape leaves an unterminated
literal and the view fails to build. The label now holds 512 chars, quotes
are doubled, then the 256 boundary is applied, retreating one character when
the cut lands on an odd quote count.
GHSA-jfgx-762w-99vr (XSRF token leakage in the HTTP client, an incomplete fix
for CVE-2025-66035) covers @angular/common 20.0.0 - 20.3.33, and the 'Check
audit' step fails on any vulnerability, so every branch is red until this
lands.
The nine packages are pinned exactly and must move together - Angular pins its
siblings at the same patch level - so package.json and the lock are updated as
a set. The local tarball for @sheet/crypto is only present in encrypted form,
so npm cannot run here; the client build and unit tests in CI are the proof.
Allowing a derived variable to reuse a physical column's name was wrong: an
intra-row calculation (SUMMER = SUMMER + 1) implies a retained value, which
depends on the order rows are read. The same argument rules out a variable
listing itself as a dependency.
Reverted:
- the name collision is an error again ('VAR_NM collides with a column in the
target table')
- VAR_DEPS may not list the variable itself
- the physical column list is back to mf_getvarlist - the type/length check
only existed to support shadowing
- the droplist guards in stagedata / viewdata / getrawdata, and the duplicate
suppression in getdata, are no longer needed
- the docs and both test suites drop the shadowing case
cols1 already holds the physical column, so appending the derived row for the
same name would send it twice. The derived row is now appended only when the
name is new; a shadowed column keeps the physical label and format, while its
value comes from the view.
With shadowing allowed, a derived name can also be a physical column. The
derived droplist was built from every derived name, so such a column would
have been dropped on submission (losing the user's edit) and from the view
and raw downloads (losing a real column).
stagedata, viewdata and getrawdata now subtract the target's own columns
from the droplist, so only genuinely virtual columns are dropped.
SUMMER = SUMMER + 1 is a legitimate definition: the name matches an existing
column, the expression reads the physical value, and the view carries the
derived one under the same name.
- mpe_derived now takes the physical column list from mp_getcols, so it has
type and length in one pass; a shadowed name must match the column's type,
and its length may not exceed it (numeric lengths must be equal), as attrib
cannot change either on an existing variable
- a self reference is only an error when the name is not a physical column
- self references are not graph edges, so the dependency sort is unaffected
- the post edit hook no longer rejects a self reference, since it cannot see
the target library; it still rejects cycles between distinct names
- mpe_derived now uses mf_getuniquename for its working tables, matching the
core macro, and carries the same placement sequence fix
- test 4 covers shadowing, docs cover the rule
Two high advisories were published against 1.16.0 in the last few hours
(GHSA-q3jh-wrc6-jgq8, GHSA-8r9p-f939-6h3c), and the 'Check audit' step fails
on any vulnerability, so every run after 18:33 UTC today is red - including
the ones on this branch that had passed earlier.
Lockfile only, no source change: 1.16.1 satisfies the ^1.15.6 range axios
asks for. Worth cherry-picking to main, since it blocks every PR.
Same defect as the core review found in mp_derivedvars: the final assemble
merged the placement table by name, but that table is built in dependency
rounds and is not name ordered, so the BY-merge failed whenever the rounds
were not alphabetical, and seq was a BY-group counter rather than the
dependency order.
Each round now stamps seq=_n_+nplaced, the merge runs on the name-sorted
tables, and the result is sorted by seq. Also guards the physical-column
in-list for a zero-column table, and checks libds exists before running.
The cycle check was already in the post edit hook and in mpe_derived - the
topological sort places a variable only once its derived inputs are placed,
and if a round places nothing the remainder is a cycle. What it reported was
just the set of names that could not be placed, which does not tell the
administrator what to change.
The message now walks the edges from the first unresolved variable and names
the loop, eg:
VAR_DEPENDENCIES contains a cycle: A -> B -> C -> A
Names are collected in a dotted seen list, so a lead-in to the loop is shown
as well and the walk stops at the first repeat.
The test fixtures now state the inputs their expressions actually use:
LIBREF DSN, DERIVED_NOTE, PRIMARY_KEY_FIELD, and none for a literal-only
expression. The getdata label assertion follows the new <VAR_NM>=<expr>
label.
A sequence number has to be maintained by the administrator: inserting a
variable means renumbering everything after it, and a wrong number fails
silently - the derived value comes back missing, not wrong.
VAR_DEPENDENCIES names every input the expression uses, physical columns and
derived variables alike, eg 'REGION STATE'. The statements are emitted in
dependency order, so a variable can build on another one with no sequence to
maintain, and a cycle is rejected.
The declaration is checked against the expression in both directions: a
derived variable referenced by VAR_LOGIC must be declared, and a declared
name that the expression does not reference is an error; a declared name must
resolve to a column of the target table or to a derived variable of the same
table.
Changes:
- MPE_DERIVED: VAR_ORDER (num) becomes VAR_DEPENDENCIES (varchar(4000)), in
the ddl, mpe_makedatamodel and the v7.19 migration
- mpe_derived: dependency validation and a topological sort replace the
VAR_ORDER sort and the forward-reference check
- mpe_derived_postedit: checks self reference, undeclared derived references
and cycles at submit time, across the incoming rows plus the current rows
they do not replace
- MPE_VALIDATIONS: the VAR_ORDER MINVAL/MAXVAL rules are replaced by UPCASE
and a character pattern on VAR_DEPENDENCIES
- mocks: the dependency order is applied in JS, so a chained variable works
even when the rows are loaded child first
- docs and tests updated
Found running the macro on a real Viya compute server:
- verify() was applied to fixed width variables, so the blank padding was
always outside the allowed character list and every name was rejected. The
check now runs on strip(name) and strip(fmt).
- the empty outvars dataset used 'length name $32; stop;', which the Viya
compute server rejects with 'Variable name is uninitialized'. It now
assigns the variable and subsets with 'if 0;'.
- the label is now <VAR_NM>=<expression> rather than 'DERIVED: <expression>',
so the grid shows the variable name as well as the logic.
mpe_loader already sorts work.staging_ds by the business key
(buskey='LIBREF DSN VAR_NM') and aborts on duplicates before the post edit
hook runs, so the same check in the hook could never fire.
The check in %mpe_derived stays: it reads the control table, which can also
be edited outside the app.
Client side rules on the table itself, so mistakes show up in the grid:
- NOTNULL on LIBREF, DSN, VAR_NM, VAR_ORDER, VAR_TYPE, VAR_LEN, VAR_LOGIC
- UPCASE on LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_FMT
- MINVAL 1 / MAXVAL 999 on VAR_ORDER, MINVAL 1 / MAXVAL 32767 on VAR_LEN
- HARDREGEX on VAR_NM, VAR_FMT and VAR_LOGIC
- HARDSELECT on VAR_TYPE, with C and N in MPE_SELECTBOX
Seeded in mpe_makedata and in the v7.19 migration. The post edit hook stays
the server side enforcement, since these rules are evaluated in the browser.
Also fixes the MPE_DERIVED create in mpe_makedatamodel, which still used the
old VARIABLE_* column names.
A post edit hook on MPE_DERIVED applies the configuration rules when a row
is submitted, rather than only when the EDIT screen next builds the view.
It normalises LIBREF, DSN, VAR_NM, VAR_TYPE and VAR_FMT, and rejects:
- a missing LIBREF, DSN or VAR_NM
- an invalid or reserved name, or one starting with a digit
- a VAR_ORDER that is not a positive integer
- a VAR_TYPE other than C or N
- a VAR_LEN outside 1-32767 (character) or 1-8 (numeric)
- a VAR_FMT that is not a plain format name
- a VAR_LOGIC that is not a single expression
- the same VAR_NM twice for one table
- a reference to a derived variable that is not defined earlier
The name collision with a physical column is left to %mpe_derived, which runs
with the target library assigned; the hook does not have it.
MPE_DERIVED columns:
VARIABLE_NM -> VAR_NM
VARIABLE_ORDER -> VAR_ORDER
VARIABLE_TYPE -> VAR_TYPE
VARIABLE_LENGTH -> VAR_LEN
VARIABLE_FORMAT -> VAR_FMT
VARIABLE_LOGIC -> VAR_LOGIC
Updated in the ddl, mpe_makedatamodel, the migration, the MPE_TABLES buskey,
the macro (reads and validation messages), the tests, the mocks and the docs.
The macro carries the ordering and statement-generation logic, so it is
worth a test of its own:
- character, numeric and chained variables: statements, view, columns
- label and default/configured formats
- a filter on a derived column
- the no-config path: no view, no statements, empty outvars
The configuration checks call mp_abort, so they stay out of the suite.
Review follow-up.
- stagedata: build the derived droplist before the ULOAD filter, and drop the
derived columns in the badrecords step, so the row-level-security check runs
against physical columns only. A submitted derived value can no longer
satisfy an RLS clause that names a derived column.
- viewdata: the full-table search view now drops the derived columns too, so
a free-text search cannot match a value in an EDIT-only column.
- mpe_derived: upcase the libds when matching MPE_DERIVED, so a lower-case
table reference still finds its configuration instead of silently showing
no derived columns.
A derived variable may need to reference one defined before it (a label that
concatenates a derived region, for example), so the order has to be
configured rather than assumed.
- MPE_DERIVED: new VARIABLE_ORDER column (ddl, makedatamodel, migration).
- mpe_derived: statements are generated in VARIABLE_ORDER then name sequence.
VARIABLE_ORDER is required and must be a positive integer. A reference to a
derived variable with an equal or higher order is rejected, because SAS
would otherwise treat it as uninitialised and return a missing value with no
error; references to lower-order variables are allowed.
- mock: same ordering, each variable compiled against the physical columns
plus those defined earlier, so the frontend can exercise a chained variable.
- makedata: VARIABLE_ORDER on both demo rows plus a chained REGION_LABEL.
- cypress: asserts the chained value in the grid.
- tests: the getdata test seeds a second variable built on the first.
- docs: VARIABLE_ORDER documented, with the ordered rule replacing the
self-contained one.
Review follow-up.
- viewdata / getrawdata: apply the filter against the derived view when one
exists, so a saved filter that references a derived column no longer aborts
with "Variable ... is not on file" in the VIEW screen or a download. The
derived columns themselves are dropped from that output - they remain an
EDIT screen feature.
- mpe_derived: reject a VARIABLE_FORMAT containing a semicolon, macro trigger
or comment, and require it to look like a format name, so it cannot inject a
statement into the generated data step.
- mpe_derived: reject a VARIABLE_LOGIC that references another derived
variable (word-boundary match), which was documented but not enforced.
- mpe_derived: stmt length 4100 so a 4000-char expression plus the name and
assignment cannot lose its trailing semicolon.
- mock tokenizer: allowlist the punctuation the expression subset uses, so
anything else is rejected instead of being evaluated.
It applies a licence key via loginAndUpdateValidKey, and csv-limited
asserts the free-tier restriction - so it must not run before that spec.
Running last keeps the licence state every other spec sees unchanged.
So the EDIT screen can be exercised before a SAS server is available.
- dcMockUtils: mpeDerived() reads MPE_DERIVED for a table and mpeDeriveRows()
adds the computed columns to the rows and to the schema, before the filter
runs - the same order as the real service.
- compileSasExpression() covers the subset of SAS expression syntax the mock
configs use (ifc, cats, catx, upcase, lowcase, strip, trim, length, substr,
index, comparisons, AND/OR and IN). Anything else leaves the column missing,
as a failing SAS expression would. The real evaluation is done by SAS.
- getdata / getcolvals apply the derivation before filtering, so a filter on a
derived column works and dropdown values come from the derived data.
- makedata: MPE_DERIVED schema and rows, a new TESTDATA.DERIVED_DEMO table
(REGION, LINE_VALUE) and its MPE_TABLES registration.
- cypress: derived-variables.cy.ts covers display and filtering on a character
and a numeric derived column.
Also bumps @sasjs/core to 5.6.0, which provides the mp_filterstore targetds=
parameter the validatefilter change depends on.
Adds user-defined computed variables, configured per table in the new
MPE_DERIVED control table. A derived variable is declared with a name,
type, length, optional format and a SAS expression.
The expression is applied in a DATA step view, and the filter is applied
by the step that reads the view, so the variables exist before the
filter is evaluated and can therefore be filtered on. A WHERE clause
cannot see a variable created in the same data step, hence the view.
- getdata: derived columns are added to the COLS payload (read only,
labelled DERIVED: <expression>) and to the data returned to the grid
- getcolvals: values are taken from the derived view, so filter dropdowns
work for derived columns
- validatefilter: the filter is validated against the view, and stored
against the physical table (needs mp_filterstore targetds=)
- stagedata: derived columns are dropped before the approval package
- mpe_makedata / mpe_makedatamodel / ddl / migration: new MPE_DERIVED
table, registered in MPE_TABLES
Values are display-time only - nothing derived is ever written back.
MPE_CONFIG.VAR_DESC is varchar(300), but the description written for
DC_MAXCELLS_WEBEDIT is 336 characters, so the insert truncates it. The
truncation raises a WARNING, which sets SYSCC, and makedata aborts on
`&syscc ne 0` immediately after %mpe_makedata():
WARNING: Character expression will be truncated when assigned to
character column var_desc.
ERROR: Execution terminated by an ABORT CANCEL statement
The abort happens before the settings job is written, so a fresh deploy
ends with no control library and no settings.sas at all. It goes
unnoticed because the SAS test suite reuses whatever library is already
on the estate - the first time it bites is a deploy from scratch.
The description is shortened to 201 characters in both places it is
written - mpe_makedata (new installs) and the 20261003 row/cell limits
migration (existing installs). The rationale it carried is user-visible
help text, so it moves to a SAS comment above each insert rather than
being dropped.
Verified on a real Viya estate, deploying from scratch:
- before: makedata aborted, "Err during VIYA1337 build", no settings.sas
- after: makedata completed with zero warnings, and the SAS test suite
then ran 59/59 green
@sasjs/utils is imported directly across the client but was left as a caret
range in the 7.18 bump, while every other dependency in the file is exact.
Pin it to 4.0.0, which is the version the lockfile already resolves.
jsrsasign is declared but referenced nowhere in the tracked source - the only
other mentions are the lockfile entry and a historical CHANGELOG line - so
drop it.
Lockfile regenerated with npm install --package-lock-only; the only deltas are
the pinned range and the removed jsrsasign tree. 570/570 unit tests pass.
Finding from the review at ece67d3.
dynamicWhereClauseCreator's IN branch mapped the entry through element.values
only when building the literal; the two fallbacks emitted the unmapped value, so
a formatted label could still reach RAW_VALUE. A temporal entry whose mapped
value is not a whole day / second (sasTemporalLiteral returns null for a
genuinely fractional value) or whose label finds no match in element.values
would fall through to the quoted label on a numeric column, which mp_filtercheck
rejects and which aborts the getcolvals load inside the swallowed catch.
The mapped value is now assigned once and used in all three branches, as
createFilterQueryTable's IN branch does.
One unit case added for the non-literal fallback. 582/582 unit tests pass.
Two findings from the review at 8f98ebe.
The IN branch read this.clauses.queryObj[groupInd][clauseInd].value, but the
entries sit at queryObj[groupInd].elements[clauseInd].value, so the forEach
dereferenced undefined and threw. dynamicWhereClauseCreator runs inside
getValues / getValuesLoadMore, where the surrounding try/catch swallowed the
TypeError and the value list silently failed to load. It now reads
element.value, like the BETWEEN and equality branches.
The BETWEEN branch composed element.value directly, so a temporal BETWEEN
picked from the value list sent the formatted strings into RAW_VALUE, which
mp_filtercheck rejects on a numeric column and which aborts the values load.
Both ends now go through the same formatted-to-unformatted lookup the IN branch
and createFilterQueryTable use, in one unformattedValue helper shared by the
two branches.
query.component gains a spec covering both branches. 581/581 unit tests pass.
Finding from the review at be79f3a.
createFilterQueryTable's IN / NOT IN branch read element.val directly, so it
produced a temporal literal only because whereClauseCreator had already mutated
query.value[i].val from the formatted label to the unformatted count in place
before submit. Any path reaching the branch without the preview having run
first would have put the formatted string into RAW_VALUE.
The entry now goes through the same unformattedValue() lookup as the BETWEEN
ends. An already-unformatted value passes through unchanged, since it no longer
matches a formatted label.
Two unit cases added: an IN entry carrying the formatted label, and one already
carrying the internal count. 578/578 unit tests pass.
Two findings from the review at 72a7963.
whereClauseCreator's guard could not match a BETWEEN: checkOperator composes
it as `<from> AND <to>` and the letters of AND are outside the character class,
so the temporal rewrite was skipped and the filter-panel preview kept the
internal day counts (`BETWEEN 42 AND 43`). The separator is removed before the
test, so a BETWEEN pair is shown as literals like a single value or an IN list.
Binding the range dropdown to column.unformatted made the field render the
internal value: setOptionValue assigns the clicked option's value to the input's
ngModel, so after choosing 12FEB1960 the field read 42. The option value is the
formatted label again, as the single-value list and the IN modal already are,
and createFilterQueryTable's BETWEEN branch maps each end back to its
unformatted value before building the literal - the same lookup
whereClauseCreator applies for the preview.
Three new unit cases pin the BETWEEN preview and the BETWEEN RAW_VALUE in both
directions. 576/576 unit tests pass.
With the pickers off, a temporal BETWEEN takes both of its values from a
soft-select over the column's values, so the clause is composed from whatever
that list binds. The test chooses 12FEB1960 for both ends of SOME_DATE and
asserts the applied clause carries the literals and still selects the 5 rows
the internal value selected.
Typing into the range fields - which is what the existing helper does - does
not exercise the binding, so both values are picked from the datalist instead.
Also accepts the evaluation agreement in beforeAll when it is presented. It
is per-origin localStorage, so a fresh browser profile lands on it and the nav
tree never renders; the accept is guarded, so it is a no-op once an earlier
spec in the run has accepted it.
Two findings from the review at f710d87.
whereClauseCreator's temporal rewrite is a regex pass over the composed value
string, so it rewrote every digit run in it. A date typed by hand as
12/02/1960 composed to 12/02/1960 and the clause preview rendered
'13JAN1960'd/'03JAN1960'd/'14MAY1965'd. The pass now runs only when the
composed value is nothing but the numbers the clause is built from - a single
value, a BETWEEN pair or an IN list - so a typed value is left alone while an
internal day count is still shown as a literal.
The range dropdown bound [value]="column.formatted", where the single-value
dropdown binds column.unformatted. whereClauseCreator's BETWEEN mapping writes
the unformatted value to a local copy rather than back onto the emitted clause,
so the formatted strings reached createFilterQueryTable, sasTemporalLiteral
returned null for them, and RAW_VALUE came out as `12FEB1960 AND 12FEB1960` -
unquoted, which mp_filtercheck rejects for a numeric column and so aborts the
save. The option value is now the unformatted value, with the formatted text
kept as the label, matching the single-value dropdown.
573/573 unit tests pass, including two new cases pinning the preview: a typed
value is left alone, and an internal day count is still shown as a literal.
sasTemporalLiteral rendered the fraction with String(fraction).slice(1), which
emits raw IEEE-754 noise - 42.3 came out as '00:00:42.29999999999999716't and
1.1 as .10000000000000009. Only the 0.5 case in the spec was clean.
SAS temporal values are whole days and whole seconds, so the value is rounded to
seconds first, which absorbs the noise a double picks up on the way here
(1.9999999999999 is simply the second 2). A value that is genuinely fractional
does not land on a second, and rounding it would silently widen the filter, so
it now returns null and the caller sends the raw number instead.
sasTimeFromSeconds has no fraction left to render, so that branch is gone.
Also corrects the doc examples, which paired 34322 with '20APR2042'd - 34322
days after 1960-01-01 is '20DEC2053'd, which is what the spec already asserted.
Same fix in the mock's parser comment.
571/571 unit tests pass.
sas/package.json
@sasjs/cli 4.20.5 -> 4.20.9
@sasjs/core 5.2.8 -> 5.5.2
client/package.json
@sasjs/adapter 4.19.1 -> 5.1.0
@sasjs/utils ^3.6.0 -> ^4.0.0
Both client bumps are majors, and both are safe here:
- @sasjs/adapter 5.0.0 raised engines.node to >= 22. The workflows run Node
24.15.0 (24.5.0 in release.yaml), so the floor is satisfied. 5.1.0 also
repackaged the adapter as compiled modules - DC imports only the package
root (SASjsConfig, UploadFile, SASjsRequest), which index.d.ts still
re-exports through ./types, and the package ships no exports map, so the
imports DC does make are unaffected.
- @sasjs/utils 4.0.0 stopped re-exporting ./fs from the package root. DC
imports @sasjs/utils/input/validators and @sasjs/utils/types/serverType,
neither of which is fs, and both still ship in 4.0.0.
@sasjs/adapter 5.1.0 depends on @sasjs/utils 4.0.0, so the two move together.
The shelljs override in sas/package.json is dropped: @sasjs/cli 4.20.9 no
longer depends on shelljs, so nothing in the tree resolves to it any more and
the pin was dead config. The nanoid pin is untouched.
Lockfiles regenerated with npm install --package-lock-only --ignore-scripts,
then validated by enumerating every added, removed and version-changed entry
and tracing each one back to @sasjs/cli, @sasjs/core, @sasjs/adapter or
@sasjs/utils. No entry was pruned as extraneous.
Verified by reproducing the Build-and-ng-test gates locally:
npm audit --omit=dev 0 vulnerabilities in root, sas/ and client/
npm run lint:check pass
npm run license-checker pass
npm run test:headless 563/563 SUCCESS
npm run build pass
5.4.0 carries the mp_filtercheck change that accepts a SAS temporal literal
in RAW_VALUE, which is what the filter literal work needs. Without it the
literal is rejected when the filter is stored, so the clause stays numeric on
a real estate even though the mocks (which do not run the SAS macro) pass.
Verified the build inlines it from the project copy, not the CLI's nested one:
sasjsbuild/server_compileTree.json resolves mp_filtercheck.sas from
node_modules/@sasjs/core, and the compiled server.sas carries the fix.
The clause shown above the grid (sasparams.FILTER_TEXT) is the resolved WHERE
clause, so a date filter read SOME_DATE = 34322 rather than a date. The
client now sends a SAS literal for a DATE, DATETIME or TIME column -
'20APR2042'd, '01JAN1960:00:00:42'dt, '00:00:42't - which SAS accepts in the
same place, so the applied filter is unchanged and the clause is readable.
HelperService.sasTemporalLiteral builds the literal arithmetically in UTC, so
the day cannot shift with the browser timezone, and fractional seconds are
preserved. It returns null for anything that is not a temporal value, and the
callers then send the raw value exactly as before.
Used by createFilterQueryTable (the applied filter), whereClauseCreator (the
clause preview in the filter panel), dynamicWhereClauseCreator (the filter
getcolvals applies to the value list) and the VA filter service.
Two fixes fall out of the same code paths:
- the dynamic where clause read the second BETWEEN value from
queryObj[groupInd][clauseInd], which is undefined, so a BETWEEN clause threw;
- an IN over a temporal column sent the *formatted* value, which the backend
refuses for a numeric column - the entries are now mapped back to the
unformatted value before the literal is built.
The mock filter parser reads the literals too. A time literal carries no date
part, so it needs its own pattern - the date pattern alone silently matched
nothing and a time filter returned no rows.
filtering.cy.ts asserts the new clause text and, for every temporal case, the
row count, so a clause that reads well but filters differently cannot pass.
Review on #343:
- initSetup's retry branch called itself from a bare setTimeout, outside the
callService promise chain, so the reserved-column throw on a retry never
reached the .catch: getdataError was never set and the user was left on the
loading spinner. Wrap the retry call and route it through the same failure
handler the .catch uses (now a private handleGetdataFailure, shared by
both).
- The multi-dataset catch logged and returned, leaving an empty table
selection with the file still attached and only a console.warn. Surface it
with eventService.showInfoModal, the pattern this file already uses for
other upload problems.
Review on #343:
- buildDatasetsObjects() runs inside the Promise.allSettled(...).then(...)
chain in onStartParsingFile, which had no .catch. prepareDeleteRecordColumn
throws on a response carrying the reserved delete-record column, so in this
flow that became an unhandled rejection: the loop aborted, no dataset was
parsed and no state was shown. Wrap the call and drop the parse, the way
fetchDataset's own catch handles a failed getdata.
- The rename-to-Delete? block still used the literal name while the same
block in editor.component.ts uses DELETE_RECORD_COLUMN_NAME. Use the
constant here too.
getdata.sas synthesised _____DELETE__THIS__RECORD_____ itself - it added
the column to work.out, hoisted it in front of the PKs, set it to "No" on
every row, gave it a No/Yes dropdown in the metadata and forced MAXLEN=3
in maxvarlengths. In the editor that column is a constant: it is only ever
"No" on load, and its Yes/No rule is already injected client-side
(deleteRecordColumnRule.ts, since #253's vertical-array fix). So the
backend was sending a column the client already owns.
The editor now creates the column, the same way EDIT_STATUS / dc.row_status
is client-only:
- getdata.sas no longer emits it (data steps, the outdata hoist, the
ctrlOptions dropdown, the maxvarlengths override).
- deleteRecordColumn.ts asserts the response does NOT carry the name, then
prepends it to COLHEADERS. Everything downstream (DcValidator's rules,
columnHeader, headerArray) is keyed off COLHEADERS order, so the delete
column is still index 0 and the grid is unchanged.
- Every row is seeded to 'No' before the validator or HyperFormula sees it.
- A response that does carry the name now throws instead of rendering a
table whose delete column would shadow a real column of that name. The
throw lands in the existing getdata .catch, which sets getdataError and
shows the error state. The name is reserved; the docs note is in the
sibling docs PR.
The multi-dataset loader (multi-dataset.component.ts) builds its own
DcValidator from the same responses, so it goes through the same helper.
Mocks updated to the new contract (server + sas9 getdata); postdata,
getstagetable and makedata are untouched - they consume or seed the flag
legitimately, and the upload contract in excel.md / files.md is unchanged.
Verified: client unit tests 570/570, npm run build clean, sasjs lint clean
on the changed SAS, sasjs compile -t server succeeds, and against the
mocked estate coltype-delete-record.cy.ts 2/2, editor.cy.ts 55/55,
excel.cy.ts 23/23 and row-cell-limits.cy.ts 5/5.
The "Upload assets to release" step builds the new body with
`jq -n --arg body ... '$body + "\n" + ...'` and no `-r`. jq then prints the
result as a JSON string literal - quoted, with `\n` escapes - which the
PATCH that follows encodes a second time, so the stored release body is a
quoted one-liner instead of markdown. On v7.17.0 this rendered the whole
release as a single escaped line in the markdown box.
Add `-r` so the concatenated notes are emitted as raw text. Verified by
running the step's jq invocation against sample notes: without `-r` the
stored body contains literal `\n`, with `-r` it contains real newlines.
release.yaml only runs on a push to main, so this cannot be exercised by
CI on the PR itself.
8fdb468 removed the only component matching app-manual-deploy and the
.clear-memory-button rule inside it, leaving the block in styles.scss with no
consumer anywhere in the tree.
It is the old Viya deployment form - the one built around uploading a JSON build
file - and it is not coming back. It was reachable only through the manualdeploy
route, which goes with it, so the Viya deploy screen is now the automatic flow
only.
That takes the manualdeploy-only plumbing in deploy.component with it (autodeploy
and its dead commented-out toggle, adminGroups/getAdminGroups, jsonFile) along
with the now-unused Router and LoggerService injections, and the section's two
spec files.
The key now comes from the backend licence (hot_license_key on the decrypted
licence), falling back to DEFAULT_HOT_LICENCE_KEY, so the hotLicenceKey tag
attribute the docs told you to set has no reader. Also reword the missing-tag
comment in parseDcAdapterSettings, which claimed there was no case to handle
while the boot screen reports one.
The tag is a mandatory part of index.html - it carries serverUrl, serverType and
the rest of the adapter config - so the missing-tag branch (deactivateApp plus an
abort modal) goes. A missing tag is reported on the tag step itself instead,
which is where the rest of the boot diagnostics already live, and the settings
parse is null-safe so nothing throws on the way there.
Both were hangovers from the earlier deployment flow.
adminGroup is a DC setting (DC_ADMIN_GROUP, shown on the System screen), not a
tag attribute. Its only readers were the deploy form's default in
deploy.component.ts and manual.component.ts, and both already fall back to
SASAdministrators when it is absent, so the fallback is now the whole logic.
hotLicenceKey was the Handsontable licence key. The customer's own key arrives
in the decrypted licence (hot_license_key), so the client carries the free-tier
default itself and no longer reads it from index.html.
With neither attribute on the tag, the boot-screen dump has nothing to hide:
HIDDEN_TAG_ATTRIBUTES is gone and every attribute is shown, so a wrong or
misspelt one is visible. That closes the review finding on hiding adminGroup.
The missing-tag branch also records why it carries no error state, closing the
other carried finding.
It is a hangover from the earlier deployment flow: its only consumers are the
deploy form's default (deploy.component.ts setDeployDefaults and
manual.component.ts), both of which already fall back to SASAdministrators when
it is absent. The attribute stays readable, so an install that still sets it is
unaffected.
hotLicenceKey is left in place - it is not a deployment setting. It is the
Handsontable licence key, seeded into freeTierLicenceData.hot_license_key and
pushed to hotTable.licenseKey in six components, and nothing on the SAS side
supplies it.
Clarity's `xl` modal size is a fixed 57.6rem (1152px), so the SAS log pane -
which is `white-space: pre` and deliberately never wraps - needed left/right
scrolling to read a line. Give the requests modal a 90vw dialog instead; the
log pane itself is untouched, so lines still do not wrap.
The <sasjs> tag in index.html is re-read on every load, so the debug switch fell
back to the tag's value whenever the page was refreshed. Store the state the user
picks from the UI in localStorage, and reinstate it when the adapter is built, so
a refresh keeps the state from before.
Render the tag attributes as a two-column grid under the label instead of a
single comma-joined line, and leave the admin group and the licence key off the
boot screen.
The missing-tag error state goes too: the tag ships in index.html, so an absent
tag is not a case the boot screen needs to report.
The check row renders the detail in a narrow right-hand column, which squeezed
the tag dump into a clipped block next to the label. Long details now wrap onto
their own full-width line, left aligned, via a `stacked` flag on the step.
The boot screen lists the startup checks (app location, Viya deploy, startup
service). Add a fourth step that shows the contents of the <sasjs> tag, so the
effective configuration (serverType, debug, useComputeApi, ...) is visible
without having to read the page source. The step is marked as errored when the
tag is missing, which is the state that triggers the abort modal.
The step is appended after the existing three so the indices the startup flow
drives (sas.service / app.service) are unchanged. HTML lower-cases attribute
names in the DOM, so known attributes are mapped back to their canonical
spelling for display; unknown ones are kept as-is so a misspelt attribute is
still visible.
A file chosen before the identity calls settle runs onSasFileChange ->
addPrecodeLines while selectedAdminGroup is still '', baking '%let admin=;'
into the generated script; the action buttons then enable against that stale
header once the groups load. Disable both file inputs while
adminGroupsLoading, and cover it with a template spec that asserts the
binding (verified red without the guard).
The admin-skip conditional used open-code %if/%then/%do, which needs SAS
9.4M5+. The suite also runs on the sas9 target, where an older maintenance
release would fail to compile the test. Wrap the body in
%macro getdata_rls_test() ... %mend and invoke it, so the conditional sits
inside a macro definition (as testsetup.sas's update_settings does).
- The startupservice test leaves trailing %let dc_admin_group lines in the
deployed settings.sas, so the skip guard compares against whatever the
file resolves to; keep using &dc_admin_group (not &mpeadmins, which is
never set in a test session).
- Selectbox RLS test now derives the restricted SOME_BESTNUM value from the
data (testsetup reseeds MPE_X_TEST with random values) and asserts the
user filter matches at least one value so the grid cannot be empty.
- Replace the open-code %return (invalid outside a macro definition) with
an %if/%else %do guard around the demote + assert section.
RLS is skipped for admins, and test sessions never resolve &mpeadmins
(testinit.sas runs without mpeinit), so the test checks the runner's
identity groups against &dc_admin_group instead. Admin runners skip
the assertions with an explicit PASS row; everyone else gets the real
RLS selectbox checks. Also documents the filter (row limit) and distinct
value preconditions.
The cell dropdown (selectbox) values sourced from a library.member.column
reference are read directly from the source table in editors/getdata.sas
(%dq_selects), without applying Row Level Security. The grid rows are
filtered by %mpe_filtermaster, but the dropdown was not, so values from
restricted rows leaked into the dropdown.
This runs %mpe_filtermaster over the source table and reads the distinct
values through the filtered view. EDIT mode mirrors the grid on this
(editor) page. A data step view is used because a fileref cannot be %inc'd
inside a proc sql where clause. The per-source view, filter table and
fileref are dropped at the end of each loop iteration.
getdata.test.2.sas restricts SOME_BESTNUM in MPE_X_TEST with a Row Level
Security rule (only the value 44 is visible) and asserts the selectbox
values returned by getdata are restricted to 44. Without the fix the
dropdown returns the other distinct values and the test fails. It is named
.test.2 because the row/cell limits test added on the target branch already
occupies .test.1.
Closes#340
The fallback added in b0e47fc names a group when the identity calls fail, but
the deploy should not proceed on a group the user never saw or chose. The
three actions that consume it - the precode download, SUBMIT (executeSAS) and
Create Database - are now disabled while the group list is empty, so a failed
or empty identity call cannot produce a deploy script or a makedata run. A
note next to the group select says why they are disabled.
The template spec asserts both states by rendering the real template.
When the identity calls fail the group lists stay empty and selectedAdminGroup
stayed '' - the manual section, unlike the automatic one, set no default - so
addPrecodeLines wrote `%let admin=;` and createDatabase carried an empty
ADMIN to makedata.
Settle the selection on both paths: applyAdminGroupFilter now also runs when
the calls fail, and falls back the way the automatic section does
(dcAdapterSettings.adminGroup, else SASAdministrators). The filter previously
sat inside the try, so the failure path skipped it entirely.
The automatic deploy's getAdminGroups has the identical raw fetch() - no
credentials, no res.ok check, and no catch, so a 401 surfaced as an unhandled
rejection while the list stayed empty. Same fix as the manual section; this
path predates the release branch.
The "Show all groups" toggle was the only ngModel control in the form without
a name or ngModelOptions.standalone, so Angular threw NG01352 on render -
dev-mode only, so a production build hid it, but ng serve and the Cypress job
did not, and the form renders on every visit (needsLogin defaults to false).
Added both, matching the neighbouring controls.
Both Viya identity calls used raw fetch() with no credentials and no res.ok
check. The deploy app can be served from a different origin to the Viya
server, where the default same-origin credential mode sends no session cookie
and Viya answers 401 - fetch still resolves, the body parses to no items, and
the screen silently rendered an empty group list that the generated deploy
script then carried as an empty admin value. Passed credentials: 'include'
(matching withCredentials on the app's other Viya calls) and threw on a
non-ok response so the existing error path reports it.
Adds a spec that renders the real template through its module - the other
specs construct the class directly, so nothing ran the template directives,
which is why CI could not catch the ngModel guard.
public/getcols publishes the SAS vocabulary (TYPE C/N, DDTYPE separating
DATE/DATETIME/TIME) but the picker compares TYPE exactly against num/char, so
the fallback list has to be mapped before it is published. getCols does that
with the rule mergeColsRules uses on the normal path: the date and time types
are numeric.
Adds the missing spec for the store's fallback, covering the mapping, the
pass-through, and the empty-payload guard.
The fallback rewrite changed the published TYPE from C/N to the client's
num/char, which is a breaking change to a public service. TYPE goes back to
the SAS vocabulary it has always used, and the client maps it to its own
num/char at the store boundary instead (sas-store.getCols). ddtype, fmtname,
format and label are unchanged, so the payload is a superset of the old one.
getcols.test now asserts the SAS contract (TYPE is C/N and agrees with ddtype)
rather than the client vocabulary, and the mock mirrors the service.
The rewrite dropped format and label from the payload of this public service.
They are restored alongside the new fmtname/ddtype so an existing caller still
finds them, and the mock carries format too so it mirrors the service. TYPE
keeps the picker's num/char vocabulary - the picker compares it exactly, so the
old C/N cannot be kept - and that vocabulary change is now documented in the
service.
setQueryVariablesFromTable fell back to public/getcols, which can return an
empty cols list when the dataset has no metadata. Publishing that pushes []
into the filter picker, whose subscriber reads cols[0].NAME - the throw is
caught, but it leaves the picker half-updated (this.cols assigned, the
filter.cols assignment below the throwing line never run). Skip the store
update when the fallback is empty.
The entries this PR bumps (@angular/animations, common, compiler, core,
forms, platform-browser, platform-browser-dynamic, router at 20.3.33;
@sasjs/adapter 4.19.1; @angular/compiler-cli 20.3.33) carried ^ ranges even
though .npmrc sets save-exact=true, so the manifest drifted on every manual
edit. Pinned them to the versions the lockfile resolves. @angular/cli is
pinned too (to 20.3.37) as the reviewer listed it. npm ci verified against
the lockfile afterwards.
Each of the three blocks ended with a single-statement %else %put, which the
Viya compute engine rejects in open code (outside a %macro) with
'Expected %DO not found' - so the migration aborted whenever it was run as a
job. Both branches now use %do/%end. Verified by executing the script as a
real Viya job: all three options updated and inserted, no errors.
SAS tests:
- getcols.test: the new TYPE-vocabulary assertion had an unquoted comma in
desc=, so the argument split and the whole test job aborted with ERROR 180.
Quoted it with %str() - the test now runs (shown RED/GREEN against the
length type $4 line it guards).
- getdata.test.1 (new): covers the EDIT row and cell limits. dctest.wideboy
filtered to 300 rows trips DC_MAXOBS_WEBEDIT; filtered to 200 rows (200 x
1005 = 201,000 cells) trips DC_MAXCELLS_WEBEDIT.
- getdata.test: test 1 fetched MPE_TABLES unfiltered, which now exceeds the
250 row limit on a populated estate. Filtered it to one libref and moved
the query assertion from test=NONE to test=ALL.
- exportdb.test: TSQL test now also asserts the export contains no INSERT.
Client:
- manual (Viya) deploy: the SAS Admin group dropdown was never populated
(ngOnInit was empty). It now fetches the groups and defaults the list to
the ones the user is a member of, with a 'Show all groups' toggle -
mirroring the automatic deploy section. Adds three unit tests.
Review feedback on this PR. The cypress cases exercise the mock, which derives
TYPE in JS and never passes through the $1 variable the length statement
widens, so dropping `length type $4;` again would pass everything CI runs.
getcols.test.sas already runs the service end to end, so its second case now
also counts the rows whose TYPE does not match their DDTYPE - 'char' for C,
'num' for N/DATE/DATETIME/TIME. That covers the vocabulary and the mapping it
is derived from, so a truncated 'n'/'c' and a swapped mapping both fail it.
The suite runs via `sasjs test` against a Viya target, not in CI.
sasjs lint reports no warning on the file.
Review feedback on this PR. type comes from %mp_getcols as a $1 holding 'C' or
'N' - the macro assigns those with no length statement - and this step sets
cols1, so type inherited that $1. Assigning 'num' and 'char' to it truncated
them to 'n' and 'c', and the picker compares the vocabulary exactly, so against
the real backend every filter built from the fallback list gave numeric columns
the character operator set and submitted character values unquoted, which
%mp_filtercheck rejects.
length type $4; now precedes the SET, which is the documented way to change an
input variable's length. ddtype needs nothing: the macro creates it with
'CHARACTER', so it is already $9.
The sibling data steps are not affected - getdata.sas (vars3) and mpe_loader.sas
rename the incoming type to type2 first, so type is a new variable there and its
length comes from the first assignment ('char', 4), which holds both values.
sasjs lint reports no warning on this file. The cypress case cannot cover this:
the mock sets TYPE in full in JS, and CI has no SAS engine to run the service.
Review feedback on this PR: with .npmrc setting ignore-scripts=true the prepare
script never runs, so the native hooks in .git-hooks/ do not activate on a fresh
clone and a commit can go unchecked.
Both standard routes are added, the way sasjs/cli and sasjs/lint carry them:
- .pre-commit-config.yaml pins the gitleaks hook to v8.30.1, so `pre-commit
install` wires the secret scan without the one-time core.hooksPath command.
- Build-and-ng-test runs the same pinned @nogoo9/gitleaks binary as a step,
because `pre-commit install` is itself a manual command: CI is the enforcement
point. --no-git scans the working tree, so the gate is what the pull request
adds rather than the repo's history.
The scan runs before the job writes client/.npmrc from the NPMRC secret - that
file carries the private registry token, so scanning after it would flag the
token CI injects itself.
.npmrc is unchanged - ignore-scripts=true stays - and the native hooks stay for
anyone who prefers them. CONTRIBUTING.md names the framework route next to the
existing one-time command.
Verified: npm ci followed by the scan reports no leaks on a fresh checkout, the
client lint check passes, and pre-commit validate-config plus pre-commit run
--all-files both pass.
Review feedback on this PR.
- The fallback list bypassed mergeColsRules, so col.TYPE was never set and every
column reached the picker as char: char operators for numerics, and unquoted
character values, which %mp_filtercheck rejects on the value list and on the
submit alike. getcols.sas and the mock now emit TYPE in the client's
vocabulary - char for C, num for N/DATE/DATETIME/TIME - and the cypress case
no longer stops at selecting the variable: it sets a value, submits, and
asserts the applied clause reads SITE_NAME = 'Bristol', so the quoting is the
assertion.
- The temporal exclusion is dropped, matching getdata.sas and viewdata.sas,
whose cols payloads include those columns - the drop is on the data, not on
the cols. The mock's getdata.js cols had the same divergence and is aligned;
it keeps the exclusion for the rows. That also removes the SAS/mock
disagreement on the excluded set, since there is no set.
- client/.npmrc added with ignore-scripts and save-exact (plus legacy-peer-deps
and fund, matching the root), so a local `cd client && npm i` is guarded the
way the CI's is.
Verified against the running mock: for MPE_CONFIG (TXTEMPORAL) getcols returns
TX_FROM/TX_TO with TYPE=num and DDTYPE=DATETIME, the char columns as TYPE=char,
and editors/getdata returns cols including both temporal columns while the rows
still exclude them. row-cell-limits.cy.ts is 5/5.
The mock typed columns from its own format list while the services typed them
from the macro's, so the two could disagree - the review of the getcols.sas
change found exactly that, with the mock covering YYMMDD/MMDDYY/DDMMYY/E8601DT
and the service only the DATETIME/DATE/TIME prefixes.
mpGetcols() now mirrors the macro: NAME (upcased), TYPE (C/N), LENGTH, LABEL,
VARNUM, FORMAT, FMTNAME and DDTYPE (CHARACTER/NUMERIC/DATETIME/DATE/TIME), with
the macro's date list. editors/getdata.js, public/viewdata.js and
public/getcols.js use it, the last mapping CHARACTER/NUMERIC to C/N the way the
service does. getDdType() delegates to it, so there is one inference, not two.
Verified by diffing the column list every fixture table returns from all three
services before and after: 49 tables x 3 services, 9 differences, all of them
FMTNAME on E8601DT columns - the old code rewrote it to DATETIME, the new code
leaves E8601DT, which is what %mp_getcols puts in fmtname. FMTNAME is not read
at runtime. Every DDTYPE, TYPE, LENGTH, FORMAT and LABEL is identical.
Review feedback on this PR, plus the conflict with the base.
Conflict: row-cell-limits.cy.ts, where both branches added a case. Both are
kept - the config-driven limits case is 4, the filter-picker-after-abort case
is 5 - with the helpers from each side.
- %local in open code is a macro error ("not valid in open code"), logged on
every call, and the variables are never created - so a table with no
mpe_tables row left the INTO variables unresolved. It is %global now, the
way getdata.sas declares the same three variables.
- The hand-rolled format-prefix test typed only DATETIME/DATE/TIME, so YYMMDD,
MMDDYY, DDMMYY, MONYY, E8601DA, B8601DA, E8601DT and NLDATM columns reached
the picker as numeric. The proc contents step is replaced with
%mp_getcols(&ds, outds=cols1) - the call getdata.sas already makes - so the
fallback list is typed exactly as the normal path types it, with the
temporal-column exclusion applied on top.
Review feedback on this PR.
- getdata.js read MPE_CONFIG from the requested table's own library, so the
read failed for every non-control libref and both limits silently fell back
to the shipped defaults. It now reads mpeDataDir, as viewdata.js already
did. Reproduced first: with DC_MAXOBS_WEBEDIT=5, DC_JSLIB.MPE_X_TEST
aborted while TESTDATA.DEMO_01 loaded untouched.
- exportdb.js mapped DATETIME to TIMESTAMP for TSQL as well, and T-SQL's
TIMESTAMP is a deprecated synonym for rowversion, so datetime columns were
declared as rowversions. TSQL now gets DATETIME2; PGSQL still gets
TIMESTAMP. DOUBLE PRECISION is left alone - it is a T-SQL synonym for
float, not an invalid type.
- loadDcConfig read the config row as direct properties, but a row that has
been through the app's own edit-and-approve flow comes back from the adapter
with upper-case keys, so the approved option was skipped and fell back to
its default - the same silent-default failure by another route. It now
reads every field through colVal, like the rest of the mocks.
- row-cell-limits.cy.ts gains a case that moves DC_MAXCELLS_WEBEDIT to a
non-default value through the editor, asserts that a non-control-libref
table is then refused, and puts the option back. It fails if the read is
dropped: verified by reinstating the getdata.js bug and watching it fail.
A limit aborting the table request left the filter picker with no variables,
so the "please filter and try again" advice could not be acted on: the
picker's list came from the columns that arrived with the table, and there
was no table.
The list now falls back to public/getcols, which reads the column metadata
without the rows. The normal path is unchanged - the service is only called
when the caller has no columns to hand - so nothing extra is requested for a
table that loaded.
- SasStoreService.getCols() calls public/getcols and returns cols.
- SasStoreService.setQueryVariablesFromTable() takes the columns the caller
already has and reaches for the service only when they are empty.
- editor openQb() and viewboxes openFilter() use it (the viewer's filter is
reachable after a VIEW abort, the editor's after an EDIT abort).
- public/getcols.sas keeps ddtype (already computed, dropped by the keep) and
excludes the temporal columns exactly as the editor does, so the fallback
list matches the table's own.
- New mock public/getcols.js mirrors the service.
Verified: the mock's list matches editors/getdata for every fixture except the
aborting one (MPE_X_SEARCH: getdata 0 cols, getcols 9), and a new Cypress case
in row-cell-limits.cy.ts aborts a too-big EDIT, opens the filter and asserts
all nine columns are offered. 89 Cypress tests green across the six affected
specs.
The mocked estate could not exercise what 7.17 changed.
- getdata.js and viewdata.js hard-coded their limits, so editing
DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT / DC_MAXOBS_WEBVIEW in DC made
no difference. They now read them from MPE_CONFIG via a new
loadDcConfig/dcConfigNumber pair in dcMockUtils, which mirrors the loop
in the generated settings.sas: DC-scope rows only, var_active=1 only,
current rows only (now < tx_to), latest tx_from wins. An option that is
absent or switched off falls back to the service's own default, which is
what %symexist does in the real service.
- There was no mock for services/admin/exportdb, so the System screen's
Export DC Library DDL action hit a service that did not exist. The new
mock renders every table in the DC libref as basic DDL - SAS, PGSQL and
TSQL flavours, schema defaulting to the libref, a unique index from the
MPE_TABLES buskey, and the download headers mp_streamfile produces. DDL
only: the real service appends inserts for SAS and PGSQL, but
serialising the library's rows in a mock buys no test value.
ddl-export.cy.ts gains two cases that fetch the URL the action opens and
assert the DDL, so the spec covers the service and not just the URL.
Review follow-ups on the row/cell limits:
- getdata.sas checked the limits twice - once before the sort that caps
work.out, and again after PRE_EDIT_HOOK. It is now a single check,
after both, so the counts cover everything that could reach the
browser: the rows the selection matched, plus anything a hook has
added. The payload cap moves to where work.outdata is built, which is
the only place that needs it. Each mp_abort also names its option once
- the limit was repeated in the message text as well as the condition.
- The migration closed only var_active=1 rows but inserted active rows
unconditionally, so an option a site had switched off was silently
switched back on. It now leaves those alone and logs a note.
- row-cell-limits.cy.ts test 1 matched a regex against the grid's text,
which the fixture's own columns satisfy (DEPTH_M reaches 4293,
SAMPLE_COUNT 4210), so it did not pin the raised VIEW cap. It now
reads the PK column - the first td of a .ht_master row - and requires
values in the PK range 1001-2000 with a maximum above 1500, which the
old 500 row cap could never render.
VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.
The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.
braces (GHSA-vfj7-8cjw-p6xm) arrives via @sasjs/cli > shelljs > fast-glob
> micromatch > braces and has no fixed version - braces 3.0.3 is the
latest release, and npm's only remedy is downgrading @sasjs/cli to
4.13.1. shelljs 0.9.0 is what introduced fast-glob, so pinning shelljs
to 0.8.5 removes braces, micromatch and fast-glob from the tree
entirely. The CLI only uses shelljs for ls/cp/rm/exec, unchanged
between the two versions - sasjs lint and sasjs cb -t server both still
pass.
handsontable and @handsontable/angular-wrapper 18.0.0 -> 18.1.1, plus the
two repo-owned places that pin the version.
18.1 fixes two bugs this repo's specs had pinned as failure modes:
- "Sheet size limit exceeded" (upstream #10672): the formulas plugin used
to push the grid's maxRows into the HyperFormula engine as its sheet
size limit, so a licensing cap below an existing table's row count made
the engine reject the whole table. initSetup works around it with
Math.max(dataSource.length, editor_rows_allowed).
- Formula corruption on updateSettings while a sort is active: a bare {}
settings object was enough to turn formula cells into #REF!.
updateSettingsSortSafe clears and restores the sort around every call.
Both specs now assert the fixed behaviour instead, so a future regression
shows up. DC's own workarounds are untouched.
18.1.1 also fixes a filters plugin crash that lands on DC's usage exactly
(upstream #13480): editing a cell in a filtered column, or replacing the
data with a filter active, after updateSettings() with the filters option.
The Angular wrapper calls updateSettings on every update, and DC's viewer
and viewboxes run filters: true.
18.1.0 is mostly a performance release (viewport cell-meta release, index
translation, single-pass layout, bulk operations).
Also:
- licenseChecker.js: add the 18.1.1 handsontable packages to
excludePackages, or the build stops at the license-checker step (the
library's licence string is non-standard).
- _hot-icons.scss: regenerated by scripts/gen-hot-icons.mjs; drops the
Pikaday rules because 18.1 removed the leftover Pikaday theme styles.
- dc-validator.spec.ts: 18.1 tightened ColumnSettings['validator'] to its
real union (RemoveIndexSignature), so narrow it the way the production
code already does.
release.yaml starts the SASjs Server the same way as build.yaml and
lighthouse.yaml, so the same root guard (sasjs/server v1.9.0) breaks it.
It runs on pushes to main and has not run since the guard shipped, so
the next release would have failed at the mock deploy.
Recent sasjs/server releases refuse to run as root unless RUN_AS names
an account (security hardening, "refuse to run as root unless RUN_AS
names an account"). The CI runners are root, so the server exited at
startup and pm2 restart-looped (103 restarts), leaving :5000
unreachable - which is what failed the mocked-estate deploy and the
Lighthouse job.
The server's own message offers RUN_AS=root as the deliberate override;
opt in explicitly for the throwaway CI container.
Both workflows start api-linux with `pm2 start api-linux --wait-ready`
and then hit http://localhost:5000 in the next step. pm2 can return
while the app is still 'launching' (it does not always block on the
ready signal), so the deploy step races the server and fails with
"Couldn't connect to server".
Wait up to 120s for the server to answer, and print `pm2 list` plus the
api-linux log if it never does, so a genuine startup failure is visible
in the job output instead of a bare ECONNREFUSED.
dc