A format catalog is addressed as LIBREF.CATALOGNAME-FC. Two places assumed a
dataset:
- mpe_derived passed the name straight to mp_derivedvars, whose existence check
aborted with 'Target table not found: DCTEST.DCFMTS-FC' - so validatefilter,
getrawdata, viewdata, getcolvals and stagedata all failed on a catalog. A
catalog has no observations, so it can have no derived variables: return an
empty variable list and an empty statement file, and leave the view alone.
- validatefilter then passed that same -FC name as targetds to mp_filterstore.
The engine handles a -FC libds itself (it rewrites it to a cntlout dataset),
but a targetds it is handed is used as given, so mp_filtercheck built a step
against a name it could not open. targetds is now only passed when the
derived view is actually in use.
mpe_derived.test gains a case for the catalog path: no derived variables, no
generated statements, and no abort.
- mpe_assert_abort.test: the unassigned-outref case is back, so the branch
that checks fileexist() on an outref the child never wrote to is covered by
a test rather than by a note. It runs: the four cases pass on the estate.
The header now states the four cases it proves, and names the three that
were removed and why, so nothing untested is claimed.
- mpe_execute.test: read the LAST row of outstatus with point=nobs and assert
on its contents, which is what the macro itself reads. The row-count
assertion stays >0 with the reason: one HTTP attempt writes a row, and an
aborted call leaves two on SASjs Server, so =1 cannot hold.
- Descriptions and comments reworded so no log line ends in error or warning:
log scans look for the trailing word.
LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_LEN, VAR_LOGIC and ACTIVE are NOT NULL in the
table definition, so the requirement is stated once, in the model, instead of
being repeated as NOTNULL rules in MPE_VALIDATIONS for each column.
VAR_FMT stays nullable: it is optional on a derived variable, and its HARDREGEX
already accepts an empty value.
The seven NOTNULL rule inserts are removed from the 7.19 migration, and the
comment on ACTIVE no longer describes a rule that is not there.
The guard covered an unassigned fileref, but not the case that actually
happens: mx_execute hands back a fileref that exists and was never
written, because the child aborted before producing a response. Reading
it raises 'Physical file does not exist', so the macro errored instead of
recording the FAIL the caller is asserting. Check fileexist on the
fileref's pathname as well.
mpe_execute.test asserted one status row per call. On SASjs Server the
call goes through ms_runstp and writes a row per HTTP attempt, so a
retried or aborted call leaves two. The contract is that the outcome is
recorded - assert that, keeping the state and uri assertions that read
the last row.
mpe_assert_abort.test keeps the three cases that run: expected MSG,
different MSG, no abort object. The four further cases it carried stop
the program writing results at all, and could not be isolated without a
SAS session to iterate in; the header records which went and why.
The picker comes from the MPE_SELECTBOX rows - the client builds the dropdown
from that data and the SELECT rule types only set whether a value outside the
list can be typed. NOTNULL is what makes ACTIVE mandatory, and the post edit
hook is what refuses anything other than 0 and 1, so the HARDSELECT row was
saying the same thing a third time.
Drop it in both writers, and drop the comment pointing at the migration: the
two files are read side by side anyway.
MPE_DERIVED is maintained through the EDIT screen, but LIBREF and DSN were free
text - a typo in either silently produced a rule that never fired.
LIBREF now offers the editable libraries and DSN the tables in the LIBREF on the
same row, through the same SOFTSELECT_HOOK services the other configuration
tables use (libraries_editable, tables_editable). VAR_NM stays typed: a derived
variable names a column the table does not have yet, so a list of existing
columns would be the wrong control. ACTIVE becomes the fixed pair 1 and 0 from
MPE_SELECTBOX, with HARDSELECT so it cannot be typed freehand, and its NOTNULL
rule carries '1' - the value the EDIT screen pre-fills on Add Row
(getColumnDefault), because a derived variable only takes effect when active.
The rows are written by mpe_makedata.sas for a fresh install and by the v7.19
migration for an upgrade, and each names the other so the pair cannot drift
unnoticed; the ACTIVE guard itself lives once, in mpe_derived_check.sas.
The mock hook carries the ACTIVE rule, as it carries every rule
mpe_derived_check.sas enforces.
Same rule as the 7.19 hook (PR #366): ACTIVE is NOTNULL + HARDSELECT in the
EDIT screen, so the field checks must refuse a missing value or anything other
than 0 and 1 - otherwise a direct stagedata submission stages a row the picker
would never have allowed.
The fixture gains the active column (1 by default, 2 for BADACTIVE) and the
count of reported rows goes from five to six.
getcolvals - mf_nobs returns -1 when the dataset does not exist, so the guard
tested ==0 and a request that supplied no iwant at all fell through to the SQL.
It now tests <=0, which covers both an empty and an absent input.
mpe_refreshcatalogs - etlsource defaulted to a bare uid while its two siblings
use refreshcatalog:<step>:<uid>, so a load through this path could not be traced
to the step that wrote it. The default now has the same shape, and the explicit
etlsource=0 path follows it too.
mpe_execute - the status table was read without being cleared, so a child that
died before core wrote a row left the previous call's row in place and failfast
read it as completed. The table is deleted before the call.
makedata - KEEPME is compared through cats(), so a numeric column no longer
raises in upcase().
The statements written into the catalog ended without a semicolon, so the
%INCLUDE that ran them ran on into the caller: the parser finished the statement
with the text after the include and tried to include it, producing
'Cannot open %INCLUDE file PROC' immediately before the next proc sql, and
refreshlibinfo aborted with syscc non-zero.
The three refresh macros are only reached through those generated statements, so
the false 'declared but not used' warnings are switched off with a @sasjslint
block rather than by dropping the declarations.
getddl requires a dataset - it aborts when the member cannot be found, and only
treats a format catalog specially - so a request with ds= always aborted and the
assertion read a stale webout. Test 1 now names MPE_TABLES and checks that the
TSQL header names it, which keeps TSQL coverage on a request the service accepts.
Test4 filtered only on the derived column, so it returned the same DSN once per
libref, including transient session librefs. The filter now pins LIBREF as well,
built in a data step because datalines leave &dc_libref unresolved, and the
expected derived value uses MPE_LOCKANYTABLE - the name makedata registers.
mp_assertcols tests variable names, so it can never find a DERIVED_NOTE row in
the cols payload - the assertion now counts rows instead. With debug on, both
Test4 payloads are printed so a wrong payload can be read rather than guessed.
An apostrophe in a %put breaks macro compilation on this platform: the statement
ends early and the next macro is left unterminated, which showed up as 'Macro
keyword LET appears as text' several hundred lines further down.
libds and col2 are only populated once validation passes, so the
"Missing inputs from iwant" abort - which tested those macro variables - fired for
any invalid payload and hid the specific reason. It now tests whether the
request carried rows at all, leaving Invalid libds / Invalid col to report the
value. The service still refuses the same requests; only the message changes.
The matching tests carried two faults of their own: getcolvals.test.4 sent its
libds through single quotes so the libref itself was invalid, and getdiffs /
getrawdata expected "Invalid table" for cases their own field names.
SAS 9 has no job state: core's SASMETA branch writes state='unknown' with
'proc stp returns no job status' on a call that succeeded. The fail-fast
change would have turned every converted test on that platform into a failure,
so unknown is noted rather than aborted on. A missing status table, a blank
state, canceled and failed still stop the caller.
The macro test asserted on uri, which the SAS 9 status row leaves empty; it now
asserts the platform column, which every branch populates.
mpe_assert_abort decides PASS/FAIL by scanning response text, and a mistake in
that scan either fails a genuine abort or passes on any sasjsAbort string in the
response; nothing reported it. Its test builds the responses as files: the
expected MSG, a different MSG, no abort object, several MSGs where the first
belongs to the abort, an object split across lines, an abort past 32767
characters, and an unassigned fileref.
mpe_execute is covered against a child that aborts for a known reason: one
status row per call, the state named and not completed, the uri recorded, and
the child log kept only when mdebug asks for it.
loadtype, var_txfrom, var_txto, var_processed, filter_text, pk, coltype, sortpk,
var_busfrom, var_busto, rk_underlying, buskey and pre_edit_hook are declared
%global but were never initialised, so in a reused session a code path that
skipped an assignment applied the previous request's value - a stale
pre_edit_hook or buskey against the wrong table.
The KEEPME probe did 'set work.fromjs' inside the varnum guard and then stopped,
so only the first observation was inspected: a submission that set KEEPME on a
later row still self-destructed. It now reads to end of file.
makeDataNextViya.json ships KEEPME=Y and is the config that seeds a real Viya
environment, so the default posture there was 'the seeding program stays on the
server'. KEEPME stays in makeDataSas719.json, which exists for the test
harness, and is removed from the environment seed.
Three defects in the scan:
- 's/.*"MSG" *: *"([^"]*)".*/$1/' is greedy, so a response carrying more
than one MSG compared the last one rather than the abort's. The pattern is
now matched with prxmatch and the capture read with prxposn, which returns
the first match at or after the sasjsAbort marker.
- the whole response was concatenated into $32767, so a longer response
truncated silently and an abort near the tail was missed. The file is now
scanned a line at a time and the scan stops at the first MSG.
- 'infile &outref' had no fileref check, so a child that died before writing a
response raised an unassigned-fileref error instead of recording a FAIL.
failfast fired only on state=canceled or state=failed, and read &outstatus
unguarded. A child that never reached the runner leaves no status table, and
a status row with a blank state left both macro variables blank - either way the
test carried on against whatever the previous call left in outlib, which is the
inference this wrapper exists to remove.
The status table is now read only if it exists, a missing table is named as
such, and failfast aborts unless the state is completed. A status row that is
absent and one that is blank both fail.
Derived variables are read-only, but they were rendered exactly like any other
read-only column: dimmed via readonlyCell, with no indication that the column is
computed at read time rather than locked by a rule or by column level security.
The grid could not tell the difference either - getdata holds a derived column
read-only with a READONLY DQ rule, which is also a rule type a table can be
configured with.
getdata now names the derived columns in sasparams.DERIVEDVARS (space separated,
upper case, from MPE_DERIVED's outvars). The EDIT screen holds that list, and
marks those columns with their own classes instead of the read-only dimming:
a tinted header with an 'fx' marker and a tooltip, tinted italic cells, and a
left edge on the first derived column so the computed block reads as a group.
Blue is the info family, leaving the primary key green and the warning ochre to
their existing meanings. Light and dark values are both set.
The SASjs mock getdata.js returns the same field, so the Cypress suite exercises
the styling; the existing derived-variables spec asserted readonlyCell on the
derived header, which is now the fallback for a plain read-only column, so that
assertion moves to dc-derived-header and a physical column stays the control.
Lint: prettier --check clean on the touched client files; sasjs lint reports no
new warnings for getdata.sas.
The harness pins @sasjs/core exactly (6.1.0) so the tests can read the
outstatus table, but the lockfile still resolved 6.0.0. 'npm ci' refuses to
install when an exact pin and the lockfile disagree, which failed the mocked
deploy in both build and lighthouse ("lock file's @sasjs/core@6.0.0 does not
satisfy @sasjs/core@6.1.0"). Regenerated with 'npm install
--package-lock-only': the only change is the core entry.
MPE_DATALOADS is keyed on (processed_dttm, libref, dsn, etlsource), and etlsource
was the calling macro name, so a second catalogue load of the same table in the
same second was refused as a duplicate. The three refresh macros now take an
etlsource parameter, defaulting to refreshcatalog:<step>:%mf_uid(), which records
which step loaded what and keeps the audit row unique.
dc_refreshcatalog also echoes the generated statements when debug is on, since a
failure inside one of them otherwise names no libref.
A refresh that catalogues the wrong libraries is hard to read from the log: the
list is built here, then filtered and reworked by its callers. With
sasjs_mdebug set, print the result so the library list can be checked at source.
The MPE_DERIVED post edit hook aborts with mp_abort, so the webout carries a
sasjsAbort object: stagedata.test.3 and .4 now read that through
mpe_assert_abort instead of inferring the outcome from MPE_LOADS.
The field level rules move into mpe_derived_check so they can be tested
directly. A service cannot exercise them all - VAR_LABEL is varchar(256) and a
staging dataset declares the same length, so a longer label is truncated before
any check sees it - and the macro test runs in seconds rather than minutes.
The wrapper is DC specific and sits alongside mpe_assert_abort, so the mpe_
prefix marks which layer it belongs to. File, macro name, %mend and all seven
callers updated; no dc_execute reference remains in the tree.
Reading the sasjsAbort object instead did not work for this test: the derived post
edit hook aborts with `abort cancel FILE` via %mp_include, and with the
response-based version the test stopped before reporting any assertion, where the
MPE_LOADS version reports ten and fails five.
Which evidence is right depends on where that abort is raised - the test's own
session or the child's - and the log does not distinguish the two here. Reverting
until that is settled, so the branch is not carrying a change that reduces
coverage.
Same inputs and outputs, with a status row and no fail-fast, so the echoed child
log carries the CHILD: prefix - which is what distinguishes an abort raised in
the child from one raised in the test's own session.
The five rejected-submission cases queried MPE_LOADS for a row that did not reach
SUCCESS. The post edit hook aborts with mp_abort, so the webout carries a
sasjsAbort object: mpe_assert_abort reads that directly, which is the actual
evidence that the submission was refused.
It was declared but never assigned, and this session treats an uninitialized
variable as an error that stops the step before the assertion runs - so the
macro reported FAIL for a response that did carry the abort object.
%if &failfast=1 and "&state" in ("canceled" "failed") fails to evaluate: %EVAL
reports it needs a numeric operand and stops the macro, which aborted every test
that reached the guard even though the child had completed or aborted cleanly.
Two plain comparisons do the same job and evaluate reliably.
dc_assert_abort inferred the outcome from the status row. The assertion should
prove the service rejected its input, so it now reads the child's response and
requires a sasjsAbort object, and compares that object's MSG value with the
message the service is expected to return:
%mpe_assert_abort(desc=An invalid libds aborts the service,
outref=web1, msg=Invalid libds)
Renamed with the mpe_ prefix to match the other macro-level helpers. getdata
Test 4 asserted completion with the abort macro, which was the wrong tool; it
now checks the status row for state=completed directly.
dictionary.libnames lists the DC libref as well, so it was seeded into the
result by the SELECT and then added a second time by the explicit INSERT. The
duplicate row then collides with pk_mpe_datacatalog_libs when the refresh closes
the bitemporal record:
ERROR: Duplicate values not allowed on index pk_mpe_datacatalog_libs
The DC libref is now excluded from the scan, so it arrives once, from the INSERT
that carries its library name and engine.
@sasjs/core 6.1.0 ships outlogds and outstatus, so the temporary copy of
mx_execute is gone and the abort assertions no longer need a platform branch:
- dc_assert_abort reads the status dataset instead of work.results on Viya and
the response text elsewhere - one table, one code path;
- the abort cases call dc_execute with outstatus and failfast=0, so a cancelled
child is reported rather than aborting the test;
- getdata Test 4 loses its %mf_getplatform() gate for the same reason, and reads
the outlib only when the status says the child completed;
- dc_execute passes viyaresult through, which the abort cases ask for.
A full makedata on every sasjs test run is expensive, and it made the suite
depend on state the tests themselves should not be creating. Move it to where
DC already does this: the npm scripts that send deploy/makeData*.json to
services/admin/makedata.
npm run sas719setup deploy, then seed once
npm run sas719test run the suite as often as wanted
KEEPME in the payload keeps makedata on the server, so a reseed does not need a
redeploy first. Viya's makedata gains the same switch, so nextviyamakedata
behaves the same way.
Tests must now leave the estate as they found it - that is the follow-up, and it
is what makes re-running a single test possible.
Tests inferred whether a service call worked by reading an outlib that a failed
call never touched, so assertions ran against the previous call's tables. This
adds dc_execute, which wraps mx_execute and makes each call observable:
- failfast: abort as soon as the child did not complete, with its message, so
the first failure is the real one
- a status row per call (outstatus) so a test can assert the outcome
- the child's log behind the harness debug flag, quiet otherwise
testsetup reseeds the data baseline before the test library is built, makedata
honours a keepme switch and getdata initialises the variables it later tests
with %length, plus desc= values reworded where a comma or apostrophe unbalanced
the parameter.
sasjs/macros/mx_execute.sas is a TEMPORARY copy from sasjs/core#476, which adds
outlogds/outstatus and stops leaving work.results in the caller WORK. Delete it
and bump the dependency once that PR is released.
The four security test programs proved that a bad input aborted the service by
reading work.results, the job table %mv_jobflow writes in the Viya branch of
%mx_execute. The SASjs Server and SAS 9 branches call %ms_testservice instead,
so the table never exists, the data step fails with "File WORK.RESULTS.DATA
does not exist." and the assertion reported (0=1) - a failure that says
nothing about whether the service validated its input.
The platform difference is now in one macro, dc_assert_abort: on Viya it reads
the job state as before, elsewhere it looks for the sasjsAbort block that
mp_abort leaves in the webout %mx_execute returns through outref. The call to
the service stays outside the platform check, so the payloads are submitted on
every platform and the assertion is real rather than a skip.
Verified on SASjs Server (sas.4gl.io, target sas719, appLoc /Public/app/dc719):
the seven abort assertions PASS, i.e. the services do abort on the macro, SQL
injection and path traversal payloads. Negative control: the same program with
a valid libds reports FAIL (0=1), so the assertion can still fail.
sasjs lint: exit 0, no warnings in the five files.
The review read the cap as a session option that is never restored, so every later
data step - including the derived-column merge into work.cols1 - would be cut to
&maxrows observations, and a table wide enough to hit the cell limit would lose
columns rather than rows.
On this head the reset is there, two lines after the data step (`options obs=max;`
at what was line 360), so the leak described is not live. It is worth removing the
dependency on that line though: the cap is now a data set option on the SET that
builds the payload, so there is no session option to leak and no later step can be
cut by it, whether or not a reset is remembered.
The existing cell-limit test asserted the row count only - 199 rows for a
1004-column table - which would have passed with 805 columns missing. It now also
compares the COLS payload against dictionary.columns, so the metadata being whole
is tested rather than assumed.
The other obs pair in this program (`options obs=10000` around the filtered
selection, restored at line 290) is deliberately left alone: it bounds the input
rows that the WHERE clause reads, which a data set option on the SET cannot do.
VAR_LABEL was the one free-text column with no rule at all, in the hook or in the
MPE_VALIDATIONS seed. The suggestion in review was an allowlist, but a label has
no reason to exclude any character - quotes are doubled by %mp_derivedvars before
the attrib statement is built, and the value never reaches a macro context - so
the rule is a denylist of what cannot be a single-line column label:
* a control character, which covers the line breaks and tabs that would carry
through into the generated DATA step and render as nothing in a header
* a label over 256 characters, which the engine truncates silently on a quote
boundary, so the configured value and the applied one differ with nothing
saying so
Everything else passes: any script, and the semicolon, ampersand, percent and
comment markers that VAR_FMT and VAR_LOGIC refuse. Implemented in the hook
(anycntrl, klength), as a HARDREGEX in all three seed writers (migration,
mpe_makedata.sas, the mock's makedata.js) so the grid refuses it before submit,
and in the mock hook for the deployed mock.
SAS: new stagedata.test.4 cases 8-10 - a label with an embedded LF is rejected, a
label of free text is accepted, and 257 characters is rejected.
Verified against the deployed mock: the tab and 257-character cases abort with the
right message, free text and UTF-8 pass, and a 256-character label passes. The
newline case cannot be driven through `sasjs request` - the mock server splits a
jsdata row at an embedded newline, so the row arrives short - hence the SAS test.
#359 has landed, so the CLI bump this branch was waiting on is now in the base
and the merge brings it in. One conflict, in
client/cypress/support/commands.ts: both sides reworked loginAndUpdateValidKey.
The base version decides from the body captured before login, which is where
the original bug was - on an estate that shows a login form that capture has no
nav tree, so the command early-returned and never applied a key, and the upload
specs then found no file input. This branch re-reads the loaded page and treats
the licence banner as the signal. Kept this side; prettier re-indented the
result. Every other file merged automatically, and the hookErrmsg veto in
stagedata.js and loadfile.js survives the merge.
Two divergences from the server hook, both of which reject a valid submission:
- VAR_DEP_UNION joins on libref, dsn and name, so a derived variable called
STATE on one table does not make STATE a derived name on another. The mock
kept a flat set of names, so a physical column named STATE was reported as
an undeclared reference whenever any other table derived a variable of that
name.
- The control-table arm filters staging_ds for the tables being submitted, so
another table's configuration cannot block this one. The mock read the whole
MPE_DERIVED table, so a cycle declared on an unrelated table blocked every
submission.
Reproduced both against the previous head - a physical STATE reference and an
unrelated CYC_A/CYC_B cycle were each rejected - and both pass after, while the
cycle, undeclared-reference and stale-dependency cases on the submitted table
still fire.
Both come up in review every time these lines are read: dqrules is the
request-local array the filter/map builds, and work.derived_vars is written by
%mpe_derived before it can return, empty when the table has no derived
variables. Said where the code is, rather than in a review reply.
has() returned true for a character outside the allowed set, which every call
site read as the opposite; it is hasInvalidChar now.
The cycle rejection was the one message without the MPE_DERIVED: prefix. The
SAS hook produced the same text, so both are prefixed rather than letting the
mock drift away from the message a real deployment shows.
loadfile names the submitted rows 'rows' and never defined jsDataRows, so
eval'd there the hook threw a ReferenceError - which the caller logged and
ignored. The rows are bound under the name the hook reads, and a hook that
throws now rejects the submission instead of being swallowed: a hook whose job
is to reject cannot fail open.
The hook's current-row test also required a TX_TO in the future, while the
read-time engine (mpeDerived) treats a missing or zero TX_TO as current, so a
dependency on an open-ended row validated here and broke at read time. Both the
graph and the inactive set, and the RLS guard, now use the engine's predicate.
The VAR_FMT message named '//*' where the check is for /* and */; it now names
the markers it rejects, in the hook and in its SAS counterpart.
The MPE_TABLES row registered services/hooks/mpe_derived_postedit, but no
mock counterpart existed, so a submission to the now-visible control table skipped
the checks the server hook runs and only failed later, when the EDIT screen
tried to build the view.
The hook mirrors mpe_derived_postedit.sas in its order: normalise the incoming
rows, reject a duplicate name and a bad field, then check the dependency graph
the resulting configuration would have - an inactive dependency, a self
reference, a reference the row does not declare, a declared input the
expression never uses, a name an active RLS rule uses, and a cycle, whose path
is reported. The collision with a physical column stays with the read-time
engine, as on the server.
A hook rejects a submission by setting hookErrmsg, which stagedata/loadfile
turn into a sasjsAbort before anything is staged; that is the mock's equivalent
of the hook's %mp_abort.
physicalColumns held the same array the derive helper reads, so every later
use in the file - cols, the search, the row loop, the empty-row fallback and
vars - depended on that helper returning a new list rather than extending the
one it was given. Taking a copy makes the invariant local to this file.
viewdata.sas drops the derived columns in the vwsearch data step that
mp_searchdata then reads, so a search matches on the columns the grid shows.
The mock handed the full column list, so a search could match a row on a
derived column that is never displayed.
dc
allan