sasjs lint (@sasjs/cli 6.2.5) reported 24 warnings across the 11 .sas files
this release touches. None of them are in those files now:
- mpeinit and mpeterm are added to ignoredMacros in sas/.sasjslint. Every
service is compiled with a serviceInit program (sasjs/utils/
serviceinitsas9|viya|server.sas), and all three declare both macros in
their own header, which is how a service body's %mpeinit()/%mpeterm()
resolves without a declaration of its own. An explicit ignoredMacros list
replaces the default, so webout stays listed.
- Header entries nothing in the file calls are dropped: mp_binarycopy in
getrawdata (it arrives through mp_streamfile, which calls it - the
compiled program is unchanged), mp_jsonout in viewdata,
mp_assertcolvals in validatefilter.test, and mx_execute in testsetup
(every file that calls it declares it).
- Prose carrying "error"/"warning" is reworded. Where the text is log
output, %put WARNING: is written %put %str(WARN)ING: so the line the
log receives is unchanged.
- The server makedata service keeps its doxygen @warning tag - the tag is
the content - so that file carries @sasjslint {"noBannedStrings": false}.
- Two comment lines in getdata are re-indented to the file's multiple of 2.
AGENTS.md records the ignoredMacros rule and the %str(WARN)ING convention so
the next warning is not silenced by reflex. Compile passes; the 93 warnings
that remain are all in files this release does not touch.
The MARKDOWN branch of the service calls %mp_ds2md(), which ships with
@sasjs/core, but the header never listed it. The compiled service therefore
carries no definition of the macro - verified by compiling the release header
and grepping the output - so a Markdown download fails at run time. The viewer
offers "Markdown (.md)" in the download list, so the path is reachable.
Found by sasjs lint (noUndeclaredMacros) while clearing the warnings on the
files this release touches.
%if %then %let ...; %else %let ...; is not valid in open code (SAS stops with
"Expected %DO not found"), which is why these sites used %do/%end blocks. One
%let with %sysfunc(ifc()) needs no blocks and reads as what it is: a choice
between two dataset names. The droplist block in getrawdata/viewdata keeps its
%if %then %do; there is no %else to pair.
- getrawdata selected its filter target with a bare '%else %let' in open code,
the shape that stops the step with 'Expected %DO not found'. Both branches
are %do/%end now, so the download service applies a filter at all.
- the conditional drop in the approve.jsdset step and the one against
MPSEARCH.vwsearch are dead: work.jsdata and work.vwsearch are already
rewritten without the derived columns, so the drops only warn.
- the post-edit hook checked that a reference is declared, but not that a
declared dependency is referenced. The engine rejects the stale shape at read
time, so the same check now rejects it at submit time.
- a row level security rule naming a derived variable is rejected when the rule
is saved, with a message that says why, instead of failing later at submit.
- getdata.test's cleanup deleted only DERIVED_NOTE, leaving DERIVED_NOTE2 with
a dependency on a variable that no longer exists, which aborted every call in
the following run - the cause of the repeated no-result runs.
Allowing a derived variable to reuse a physical column's name was wrong: an
intra-row calculation (SUMMER = SUMMER + 1) implies a retained value, which
depends on the order rows are read. The same argument rules out a variable
listing itself as a dependency.
Reverted:
- the name collision is an error again ('VAR_NM collides with a column in the
target table')
- VAR_DEPS may not list the variable itself
- the physical column list is back to mf_getvarlist - the type/length check
only existed to support shadowing
- the droplist guards in stagedata / viewdata / getrawdata, and the duplicate
suppression in getdata, are no longer needed
- the docs and both test suites drop the shadowing case
With shadowing allowed, a derived name can also be a physical column. The
derived droplist was built from every derived name, so such a column would
have been dropped on submission (losing the user's edit) and from the view
and raw downloads (losing a real column).
stagedata, viewdata and getrawdata now subtract the target's own columns
from the droplist, so only genuinely virtual columns are dropped.
Review follow-up.
- viewdata / getrawdata: apply the filter against the derived view when one
exists, so a saved filter that references a derived column no longer aborts
with "Variable ... is not on file" in the VIEW screen or a download. The
derived columns themselves are dropped from that output - they remain an
EDIT screen feature.
- mpe_derived: reject a VARIABLE_FORMAT containing a semicolon, macro trigger
or comment, and require it to look like a format name, so it cannot inject a
statement into the generated data step.
- mpe_derived: reject a VARIABLE_LOGIC that references another derived
variable (word-boundary match), which was documented but not enforced.
- mpe_derived: stmt length 4100 so a 4000-char expression plus the name and
assignment cannot lose its trailing semicolon.
- mock tokenizer: allowlist the punctuation the expression subset uses, so
anything else is rejected instead of being evaluated.
mpe_accesscheck and validatefilter validated their libds input with
mp_validatecol(LIBDS), which rejects the LIBREF.CATALOGNAME-FC form that
Data Controller uses to address a format catalog. A format-catalog load
or filter therefore aborted with "Invalid base_table" / "Invalid
filter_table" - stagedata, getdata and postdata all reach mpe_accesscheck
through the edit/approve path.
Add mpe_validatecol, a wrapper that permits the catalog form: the -FC
suffix is matched exactly and the remainder is validated as a strict
LIBREF.DATASET, so the whole value is covered and a caller that needs the
catalog reference downstream (MPE_SECURITY stores it with the suffix)
still receives it. getrawdata and getcolvals had grown an inline version
of this check that scanned on the dash and validated only the prefix,
leaving whatever followed it unvalidated; the wrapper replaces both.
mpe_validatecol.test.sas asserts the matrix - plain libds, catalog form,
and payloads that smuggle content past a valid libds prefix.
Security fixes for the input-validation gaps in the public download and
metadata services, plus missing in-code admin gates:
- mpe_accesscheck: validate base_table (LIBDS) and access_level before
they reach the authorisation query, and escape embedded quotes in the
SQL literals (defence in depth for direct macro callers)
- getrawdata: validate table (LIBDS / format-catalog form), filter
(integer) and type before they are used; read all inputs with symget
in a data step so macro content cannot execute at a resolution boundary
- getdiffs: validate libds, table and stp_diffs_csv before the access
check and the staging-file stream path
- getcols, getcolvals, validatefilter: read the IWANT inputs with symget
in a data step and validate (LIBDS / SAS name) before use
- admin dirlist, refreshlibs, refreshcatalog, exportconfig: require
membership of the DC administrators group (the admin folder prefix is
not an access control)
- admin dirlist: read parent with symget and reject macro characters
Tests (all proven RED on the vulnerable services first, then GREEN on
the fix): getrawdata.test.1, getdiffs.test, getcols.test,
getcolvals.test.4, validatefilter.test.1, dirlist.test,
refreshcatalog.test.1
dc
allan