Enables noMacroIfInOpenCode and promotes it to "error". Open-code %if is a SAS
9.4M5 feature, so the 19 sites it found in nine files were hiding a version
requirement the codebase does not document (the requirements page says 9.4M3).
Each is rewritten so the construct no longer appears in open code:
- Six blocks become a small macro called unconditionally, which is where the
conditional then lives: drop_cas_tables (demodata), add_derived_cols
(getdata), build_sasdata4 (getdata test 4), drop_derived (stagedata),
build_staged (getrawdata), and one per option in the 20261003 migration.
- Four blocks in the two makedata STPs get the same treatment: check_outfmt,
check_keepme, emit_response and self_destruct. Their `call symputx` writes
are now explicit about the global symbol table, because a %let or a plain
symput inside a wrapper would be local to it.
- Two guards in stagedata and getrawdata are removed rather than wrapped: the
SQL only selects into a list that is already %let blank, and with no derived
variables it returns no rows, so the guard was doing nothing. The trailing
drop keeps a conditional, now in a macro and tested with %trim, since a
zero-row select can leave a single blank in the variable.
- validatefilter's two %let branches become one data step with call symputx,
which writes to the global table the later %mp_filterstore reads.
Verified: `npm run lint:sas` clean with the rule as an error; `sasjs compile`
succeeds; on sas.4gl.io the full suite passes 64/64, including the derived
variable tests in getdata test 4, stagedata tests 3 and 4 and validatefilter,
which are the paths these wrappers touch.
sas/.sasjslint takes maxLineLength from 180 to 80 - the linter's own default -
and promotes it to "error".
53 of the 258 .sas files already hold a line longer than 80. Rewrapping them
would be a large, reviewable-only-in-theory diff, so each keeps its current
longest line as an exact ceiling via a per-file override, e.g.
@sasjslint {"maxLineLength": 95} in services/validations/tables_all.sas. A
file with no override is held to 80, and an overridden file cannot grow past
the length it has today: the ceiling is the file's longest line, not a round
number.
Both directions were checked: a 101 character line added to an
un-overridden file fails with "Line exceeds maximum length by 21 characters",
and a 114 character line added to macros/mpe_execute.sas (ceiling 101) fails
by 13. `npm run lint:sas` is clean with the setting in force.
Rung 6 of the #376 ratchet: `"lineEndings": "lf"` is enabled and promoted to
"error". Its 37 findings are cleared and nothing but line endings changed.
- 34 files ended without a newline; each now ends with exactly one LF.
- 3 files were stored with CRLF (services/public/getgroups.sas and the two
sas9 lineage services, getmetacols and getmetatables), against the repo's
own `* text=auto eol=lf` in .gitattributes. They are converted to LF.
Checked mechanically that this is whitespace only: for every changed file,
HEAD's content and the new content are identical once CRLF is folded to LF
and the trailing newline is stripped.
VERIFICATION
- `npm run lint:sas`: 0 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- On sas.4gl.io, against a scratch appLoc seeded with makedata:
"Tests that pass: 64/64 (100%)".
Rung 3 of the #376 ratchet: `sas/.sasjslint` promotes noBannedStrings to
"error". The rule flags the literal words "error " / "warning " anywhere in
a file, comments included, because they turn up in a grep of the SAS log
and bury the real messages. Its 13 findings are cleared:
- Nine are prose and are reworded, so the sentence both reads normally and
stops matching: a truncation that aborts a build, a syntax fault, an
abort condition, "only records the failure in work.mp_abort_errds" (four
copies of the mp_abort-inside-%include comment), a test description, a
hook comment.
- Two are the text the user sees. loadfile.sas reports
"(mf_abort.error file found)" - the marker file carries the word in its
name - and buildtermsas9.sas reports "Some final error in &_program";
both are reworded to say the same thing without the literal, since the
text sits in a macro argument where the %str(WARN)ING split does not
survive the quoting.
- Three keep the word because the word is the content: the doxygen
@warning tags in macros/bitemporal_dataloader.sas and the two SAS 9
admin STPs (makedata, makelib). Each carries
@sasjslint {"noBannedStrings": false}, as
targets/server/services_server/admin/makedata.sas already did.
VERIFICATION
- `npm run lint:sas`: 5 warnings, 0 errors, exit 0.
- `sasjs compile`: succeeds.
- Deployed to sas.4gl.io and ran the SAS suite there:
`sasjs test` reports "Tests that pass: 64/64 (100%)". The run used a
scratch appLoc (/Public/app/dc-lint-ratchet) seeded with makedata, so
the shared /Public/app/dc deployment was left alone; the scratch target
was removed from sasjsconfig.json before this commit.
5 warnings remain: macro name length (2) and case (2),
indentationMultiple (1) - the last rung.
sas/.sasjslint now promotes noUnusedMacros to "error", so the rule fails
the build rather than printing a warning. The 54 findings it reported are
gone:
- 46 header entries nothing calls are dropped. Each removal was checked
against the compiled artefacts rather than the linter alone: the
project was compiled before and after, and the two sasjsbuild trees
compared. 37 macro definitions disappear from the generated programs -
every one of them either a duplicate of a definition the file still
reaches through another declared macro, or a macro no generated program
calls. No macro that a generated program calls lost its definition.
- 8 entries are kept, in five files that carry
@sasjslint {"noUnusedMacros": false} with the reason written above it:
the three serviceInit programs and serviceterm.sas declare mpeinit and
mpeterm without calling them, because that declaration is how a service
body reaches those macros at all; loadfile.sas declares mp_binarycopy
for the fallback that is commented out on SAS 9.4 and later.
- Four headers are left with no macros to declare, so their empty
<h4> SAS Macros </h4> section is removed rather than kept as a stub.
@sasjs/cli is bumped 6.2.0 -> 6.5.0 in sas/package.json. The lint engine
ships inside the CLI and is not a direct dependency, so the pinned CLI is
what fixes the rule set: 6.2.0 resolves @sasjs/lint 4.3.0, 6.5.0 pins
4.9.0 exactly. Both versions were compiled against this tree and the
sasjsbuild output is byte-identical, so the bump changes the lint engine
and nothing else.
`sasjs lint` is also now reachable as `npm run lint:sas`, and a CI step in
build.yaml runs it. Until now nothing ran SAS lint automatically, and a
bare `sasjs` resolves whatever CLI is installed globally - a different
lint version reports a different rule set, which is why the counts in
issue #376 (taken with 6.2.5) do not match the pinned 6.2.0.
43 warnings remain under the rules not yet promoted: noUndeclaredMacros
(25), noBannedStrings (13), macro name length and case (4),
indentationMultiple (1). They are the next rungs of the ratchet.
The pre-commit Prettier gate could not run: it needs client/node_modules,
which this worktree does not install, and the fix it attempted touched
three client files that this change does not modify (reverted). No client
file is in this commit.
mpe_derived used &dc_dttmtfmt in its where= but the guard only checked libds and
dclib, so an unset value turned the filter into 'lt tx_to' and the control-table
step failed with a syntax error a long way from the cause. It is now part of
the guard, named in the message.
The 7.19 migration took max(selectbox_rk) into a macro variable, which is
missing on an install whose MPE_SELECTBOX is still empty - %eval(&maxrk+1) then
fails. coalesce(max(selectbox_rk),0) makes the first row start at 1.
LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_LEN, VAR_LOGIC and ACTIVE are NOT NULL in the
table definition, so the requirement is stated once, in the model, instead of
being repeated as NOTNULL rules in MPE_VALIDATIONS for each column.
VAR_FMT stays nullable: it is optional on a derived variable, and its HARDREGEX
already accepts an empty value.
The seven NOTNULL rule inserts are removed from the 7.19 migration, and the
comment on ACTIVE no longer describes a rule that is not there.
The picker comes from the MPE_SELECTBOX rows - the client builds the dropdown
from that data and the SELECT rule types only set whether a value outside the
list can be typed. NOTNULL is what makes ACTIVE mandatory, and the post edit
hook is what refuses anything other than 0 and 1, so the HARDSELECT row was
saying the same thing a third time.
Drop it in both writers, and drop the comment pointing at the migration: the
two files are read side by side anyway.
MPE_DERIVED is maintained through the EDIT screen, but LIBREF and DSN were free
text - a typo in either silently produced a rule that never fired.
LIBREF now offers the editable libraries and DSN the tables in the LIBREF on the
same row, through the same SOFTSELECT_HOOK services the other configuration
tables use (libraries_editable, tables_editable). VAR_NM stays typed: a derived
variable names a column the table does not have yet, so a list of existing
columns would be the wrong control. ACTIVE becomes the fixed pair 1 and 0 from
MPE_SELECTBOX, with HARDSELECT so it cannot be typed freehand, and its NOTNULL
rule carries '1' - the value the EDIT screen pre-fills on Add Row
(getColumnDefault), because a derived variable only takes effect when active.
The rows are written by mpe_makedata.sas for a fresh install and by the v7.19
migration for an upgrade, and each names the other so the pair cannot drift
unnoticed; the ACTIVE guard itself lives once, in mpe_derived_check.sas.
The mock hook carries the ACTIVE rule, as it carries every rule
mpe_derived_check.sas enforces.
VAR_LABEL was the one free-text column with no rule at all, in the hook or in the
MPE_VALIDATIONS seed. The suggestion in review was an allowlist, but a label has
no reason to exclude any character - quotes are doubled by %mp_derivedvars before
the attrib statement is built, and the value never reaches a macro context - so
the rule is a denylist of what cannot be a single-line column label:
* a control character, which covers the line breaks and tabs that would carry
through into the generated DATA step and render as nothing in a header
* a label over 256 characters, which the engine truncates silently on a quote
boundary, so the configured value and the applied one differ with nothing
saying so
Everything else passes: any script, and the semicolon, ampersand, percent and
comment markers that VAR_FMT and VAR_LOGIC refuse. Implemented in the hook
(anycntrl, klength), as a HARDREGEX in all three seed writers (migration,
mpe_makedata.sas, the mock's makedata.js) so the grid refuses it before submit,
and in the mock hook for the deployed mock.
SAS: new stagedata.test.4 cases 8-10 - a label with an embedded LF is rejected, a
label of free text is accepted, and 257 characters is rejected.
Verified against the deployed mock: the tab and 257-character cases abort with the
right message, free text and UTF-8 pass, and a 256-character label passes. The
newline case cannot be driven through `sasjs request` - the mock server splits a
jsdata row at an embedded newline, so the row arrives short - hence the SAS test.
- mpe_derived_postedit ran %local at the top level of a file that is included
at execution time, so the hook stopped before validating anything and every
MPE_DERIVED submission failed. The macro language now lives in a macro
definition, errflag and errmsg are declared global, and the file calls it.
- the cycle check aborts after leaving its loop, and the self, undeclared and
field aborts are driven by a count at the point of call rather than by a
wrapping %if, so an abort cannot be outlived. The same shapes are fixed
upstream in sasjs/core#468.
- a case-variant duplicate survived mpe_loader, which compares raw values:
the hook now rejects it after normalisation and names the variable.
- stagedata dropped the derived columns from the output of the security step
rather than from its input, so a crafted submission could still satisfy an
RLS clause through a derived column. work.jsdata itself is rewritten now.
- the VAR_FMT validation rule rejected the bare w.d forms that the hook and
the engine accept.
- the migration carried the same four dangling insert headers as mpe_makedata,
so the v7.19 migration failed for existing installs.
- the filter fixtures in getdata.test and validatefilter.test had been caught
by the variable rename: the filter schema column is VARIABLE_NM.
- the TX_FROM/TX_TO definitions agree across the ddl, the migration and
mpe_makedatamodel.
@sasjs/core bumped to 5.7.1, which carries the where= fix.
A configured label replaces the default NAME=expression; when it is missing or
blank the default is used, so existing rows are unaffected.
- MPE_DERIVED gains VAR_LABEL varchar(256), in the ddl, mpe_makedatamodel and
the v7.19 migration
- mpe_derived uses it, through the same escaping and 256-char truncation as
the default label
- the hook trims it; the mock mirrors the fallback
- test 1 now sets a label on BAND and asserts it
A sequence number has to be maintained by the administrator: inserting a
variable means renumbering everything after it, and a wrong number fails
silently - the derived value comes back missing, not wrong.
VAR_DEPENDENCIES names every input the expression uses, physical columns and
derived variables alike, eg 'REGION STATE'. The statements are emitted in
dependency order, so a variable can build on another one with no sequence to
maintain, and a cycle is rejected.
The declaration is checked against the expression in both directions: a
derived variable referenced by VAR_LOGIC must be declared, and a declared
name that the expression does not reference is an error; a declared name must
resolve to a column of the target table or to a derived variable of the same
table.
Changes:
- MPE_DERIVED: VAR_ORDER (num) becomes VAR_DEPENDENCIES (varchar(4000)), in
the ddl, mpe_makedatamodel and the v7.19 migration
- mpe_derived: dependency validation and a topological sort replace the
VAR_ORDER sort and the forward-reference check
- mpe_derived_postedit: checks self reference, undeclared derived references
and cycles at submit time, across the incoming rows plus the current rows
they do not replace
- MPE_VALIDATIONS: the VAR_ORDER MINVAL/MAXVAL rules are replaced by UPCASE
and a character pattern on VAR_DEPENDENCIES
- mocks: the dependency order is applied in JS, so a chained variable works
even when the rows are loaded child first
- docs and tests updated
Client side rules on the table itself, so mistakes show up in the grid:
- NOTNULL on LIBREF, DSN, VAR_NM, VAR_ORDER, VAR_TYPE, VAR_LEN, VAR_LOGIC
- UPCASE on LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_FMT
- MINVAL 1 / MAXVAL 999 on VAR_ORDER, MINVAL 1 / MAXVAL 32767 on VAR_LEN
- HARDREGEX on VAR_NM, VAR_FMT and VAR_LOGIC
- HARDSELECT on VAR_TYPE, with C and N in MPE_SELECTBOX
Seeded in mpe_makedata and in the v7.19 migration. The post edit hook stays
the server side enforcement, since these rules are evaluated in the browser.
Also fixes the MPE_DERIVED create in mpe_makedatamodel, which still used the
old VARIABLE_* column names.
A post edit hook on MPE_DERIVED applies the configuration rules when a row
is submitted, rather than only when the EDIT screen next builds the view.
It normalises LIBREF, DSN, VAR_NM, VAR_TYPE and VAR_FMT, and rejects:
- a missing LIBREF, DSN or VAR_NM
- an invalid or reserved name, or one starting with a digit
- a VAR_ORDER that is not a positive integer
- a VAR_TYPE other than C or N
- a VAR_LEN outside 1-32767 (character) or 1-8 (numeric)
- a VAR_FMT that is not a plain format name
- a VAR_LOGIC that is not a single expression
- the same VAR_NM twice for one table
- a reference to a derived variable that is not defined earlier
The name collision with a physical column is left to %mpe_derived, which runs
with the target library assigned; the hook does not have it.
MPE_DERIVED columns:
VARIABLE_NM -> VAR_NM
VARIABLE_ORDER -> VAR_ORDER
VARIABLE_TYPE -> VAR_TYPE
VARIABLE_LENGTH -> VAR_LEN
VARIABLE_FORMAT -> VAR_FMT
VARIABLE_LOGIC -> VAR_LOGIC
Updated in the ddl, mpe_makedatamodel, the migration, the MPE_TABLES buskey,
the macro (reads and validation messages), the tests, the mocks and the docs.
A derived variable may need to reference one defined before it (a label that
concatenates a derived region, for example), so the order has to be
configured rather than assumed.
- MPE_DERIVED: new VARIABLE_ORDER column (ddl, makedatamodel, migration).
- mpe_derived: statements are generated in VARIABLE_ORDER then name sequence.
VARIABLE_ORDER is required and must be a positive integer. A reference to a
derived variable with an equal or higher order is rejected, because SAS
would otherwise treat it as uninitialised and return a missing value with no
error; references to lower-order variables are allowed.
- mock: same ordering, each variable compiled against the physical columns
plus those defined earlier, so the frontend can exercise a chained variable.
- makedata: VARIABLE_ORDER on both demo rows plus a chained REGION_LABEL.
- cypress: asserts the chained value in the grid.
- tests: the getdata test seeds a second variable built on the first.
- docs: VARIABLE_ORDER documented, with the ordered rule replacing the
self-contained one.
Adds user-defined computed variables, configured per table in the new
MPE_DERIVED control table. A derived variable is declared with a name,
type, length, optional format and a SAS expression.
The expression is applied in a DATA step view, and the filter is applied
by the step that reads the view, so the variables exist before the
filter is evaluated and can therefore be filtered on. A WHERE clause
cannot see a variable created in the same data step, hence the view.
- getdata: derived columns are added to the COLS payload (read only,
labelled DERIVED: <expression>) and to the data returned to the grid
- getcolvals: values are taken from the derived view, so filter dropdowns
work for derived columns
- validatefilter: the filter is validated against the view, and stored
against the physical table (needs mp_filterstore targetds=)
- stagedata: derived columns are dropped before the approval package
- mpe_makedata / mpe_makedatamodel / ddl / migration: new MPE_DERIVED
table, registered in MPE_TABLES
Values are display-time only - nothing derived is ever written back.
MPE_CONFIG.VAR_DESC is varchar(300), but the description written for
DC_MAXCELLS_WEBEDIT is 336 characters, so the insert truncates it. The
truncation raises a WARNING, which sets SYSCC, and makedata aborts on
`&syscc ne 0` immediately after %mpe_makedata():
WARNING: Character expression will be truncated when assigned to
character column var_desc.
ERROR: Execution terminated by an ABORT CANCEL statement
The abort happens before the settings job is written, so a fresh deploy
ends with no control library and no settings.sas at all. It goes
unnoticed because the SAS test suite reuses whatever library is already
on the estate - the first time it bites is a deploy from scratch.
The description is shortened to 201 characters in both places it is
written - mpe_makedata (new installs) and the 20261003 row/cell limits
migration (existing installs). The rationale it carried is user-visible
help text, so it moves to a SAS comment above each insert rather than
being dropped.
Verified on a real Viya estate, deploying from scratch:
- before: makedata aborted, "Err during VIYA1337 build", no settings.sas
- after: makedata completed with zero warnings, and the SAS test suite
then ran 59/59 green
Each of the three blocks ended with a single-statement %else %put, which the
Viya compute engine rejects in open code (outside a %macro) with
'Expected %DO not found' - so the migration aborted whenever it was run as a
job. Both branches now use %do/%end. Verified by executing the script as a
real Viya job: all three options updated and inserted, no errors.
Review follow-ups on the row/cell limits:
- getdata.sas checked the limits twice - once before the sort that caps
work.out, and again after PRE_EDIT_HOOK. It is now a single check,
after both, so the counts cover everything that could reach the
browser: the rows the selection matched, plus anything a hook has
added. The payload cap moves to where work.outdata is built, which is
the only place that needs it. Each mp_abort also names its option once
- the limit was repeated in the message text as well as the condition.
- The migration closed only var_active=1 rows but inserted active rows
unconditionally, so an option a site had switched off was silently
switched back on. It now leaves those alone and logs a note.
- row-cell-limits.cy.ts test 1 matched a regex against the grid's text,
which the fixture's own columns satisfy (DEPTH_M reaches 4293,
SAMPLE_COUNT 4210), so it did not pin the raised VIEW cap. It now
reads the PK column - the first td of a .ht_master row - and requires
values in the PK range 1001-2000 with a maximum above 1500, which the
old 500 row cap could never render.
VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.
The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.
BREAKING CHANGE: Introduction of 4 new tables for capturing information related to catalogs and their objects. Migration script prepared and available in the DB folder (usual place)
dc
allan