Compare commits

..
Author SHA1 Message Date
dc 11587e2f24 fix(mpe_execute): keep the child message in the fallback, and test the doubled separator
Lighthouse Checks / lighthouse (pull_request) In progress
Build / Build-and-ng-test (pull_request) Failing after 1m55s
Build / Build-and-test-development (pull_request) Skipped
Two of the three review points.

The fallback %let now carries %superq(msg) as well, so a build without the helper
on the autocall path still reports the child's own message rather than only the
program and state. The whole value is masked and consumed through %superq at the
abort, so the comma hazard does not come back.

A case with the doubled separator - the form the estate actually emits, six
characters once the backslashes are doubled - now covers the branch that all the
other cases, written single-escaped, could not reach.

The quote case is not fixed; see the reply.
2026-10-31 00:56:36 +00:00
dc 700b906d69 fix(mpe_execute): one symbol for the abort text, and no values in the call
Build / Build-and-ng-test (pull_request) Failing after 2m1s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 16m45s
Three review points.

The fallback %let now follows %global _mpe_abortmsg. Without it the %let created
a local of that name in mpe_execute while the helper wrote the global, so on the
first failed call in a session the fallback shadowed the child's message - the
symptom this branch set out to remove.

mpe_abortmessage takes names for all three inputs (programvar/statevar/msgvar),
so a comma in an endpoint path cannot split the parameter list either. The test
that passes a path containing a comma now covers that, alongside the message.

mpe_assert_abort unescapes outside the marker block: a response that carries no
appended extract is still escaped, and leaving the unescape inside the block made
the comparison fail on text that was correct.

Full suite 64/64, mpe_execute and mpe_assert_abort PASS, sasjs cbd first.
2026-10-31 00:22:40 +00:00
dc b29bc1e7ae fix(mpe_assert_abort): strip the separator tail, not every trailing escape
Build / Build-and-ng-test (pull_request) Successful in 4m46s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m50s
Build / Build-and-test-development (pull_request) Successful in 27m30s
Two review points.

The strip looped over trailing \n and \r escapes, so a message that itself ended
in an escaped newline lost that escape along with the separator and the exact
comparison failed. It now removes one fixed tail: the six characters the
separator becomes once mp_abort doubles the backslashes, or four if the extract
is not doubled. Anything the message itself ended with is left for the unescape.

mpe_execute sets _mpe_abortmsg before calling mpe_abortmessage, so a build where
the helper is not on the autocall path still aborts with the program and state
rather than an empty message.

Full suite 64/64, mpe_assert_abort PASS, sasjs cbd before the run.
2026-10-30 23:53:42 +00:00
dc 68b89caf63 fix(mpe_abortmessage): read program and state by name, and trim the prefix
Build / Build-and-ng-test (pull_request) Successful in 4m52s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m58s
Build / Build-and-test-development (pull_request) Successful in 29m26s
Three things, one of them a bug the review question uncovered.

- msgvar is passed explicitly at the call site, so renaming the message variable
  fails loudly instead of quietly producing an abort text with no child message.
- program and state are read with symget rather than interpolated into the data
  step, so a % or & in an endpoint name can no longer be macro-resolved, and a
  quote cannot break the literal.
- _prefix is trimmed before the message is appended. Declared as a character
  variable it arrives padded to length, so the child's message was landing past
  the buffer end and disappearing - the wrapper text was right and the message
  was simply not there. That showed up as the comma and blank cases failing while
  the "names the program state" case passed.
2026-10-30 23:20:32 +00:00
dc 7b09ade97c fix(mpe_assert_abort): name the separator and tolerate a carriage return
Build / Build-and-ng-test (pull_request) Successful in 4m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m43s
Build / Build-and-test-development (pull_request) Successful in 27m53s
mp_abort appends the extract as cats(msg,'\\n\\nLog Extract:\\n',logmsg), which is
now stated where the strip happens.  The strip removes whole \\n or \\r pairs, so a
separator that gains a carriage return does not leave a fragment glued to the
message; a message that ends with the marker and no appended extract is a known
ambiguity and says so in the comment.

stagedata.test.4: the comment above exp8 now records why the expected text has
no space after either colon - the child builds it as &mac.:&VAR_NM.:&reason -
so the spacing is not a formatting choice a later reader has to re-derive.  The
change itself came with ebfd395 on this branch and is unchanged here.

Suite 64/64, mpe_assert_abort PASS, sasjs cbd before the run.
2026-10-30 22:27:53 +00:00
dc e68d221354 fix(mpe_assert_abort): compare the unescaped message, cut at the last marker
Build / Build-and-ng-test (pull_request) Successful in 4m45s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m1s
Build / Build-and-test-development (pull_request) Successful in 27m55s
Addresses the review on e2e9103.

- The marker is located from the end. A hook message may legitimately contain
  the text "Log Extract:" and only the final occurrence is the one mp_abort
  appended; taking the first cut such a message mid-sentence.
- The captured text is unescaped (\" -> " and \\ -> \) before the comparison, so
  it is compared as the text the hook raised rather than as its JSON form.  That
  is also what makes a message ending in a backslash comparable at all: mp_abort
  escapes it as two, and the strip now leaves the pair alone for the unescape to
  collapse - it only removes \n pairs and a lone stray backslash that is the
  start of an escape the marker cut in half.  Requiring the preceding character
  not to be a backslash is what keeps a real trailing backslash out of that
  branch.
- mpe_abortmessage owns the name it writes, _mpe_abortmsg, instead of taking one
  as a parameter, so a call can no longer overwrite a caller's global of the
  same name.  Each call replaces the previous value; the doc says so.

Two cases added to mpe_assert_abort.test: a message ending in a backslash, and a
message containing the marker itself.  Both fail against the previous logic
(first occurrence, no unescape) and pass now.  Full suite 64/64 on sas.4gl.io,
sasjs cbd before the run.
2026-10-30 21:23:27 +00:00
dc e2e91039cb fix(mpe_execute): assemble the abort text in a macro that can be tested
Build / Build-and-ng-test (pull_request) Successful in 5m1s
Lighthouse Checks / lighthouse (pull_request) Successful in 16m43s
Build / Build-and-test-development (pull_request) Successful in 27m20s
Addresses the review on 5d32de9.

The text mpe_execute aborts with is now built by mpe_abortmessage, which takes
the child message by variable name and writes the result to the global symbol
table.  Two reasons: a value argument would be split at the first comma before
the macro ran, which is the defect this PR is about, and having the assembly in
a macro makes the comma case testable - a test calls it directly and asserts the
text, where going through mpe_execute needs a second service aborting to observe
the call the caller itself is aborted by.

- mpe_assert_abort: a lone trailing backslash is no longer stripped.  Only
  whole escape sequences are removed (\\n, \\\\n, \\\\), so a message that ends
  in a backslash keeps it.
- mpe_assert_abort: the strip is skipped when "Log Extract:" is the first thing
  in the captured text, instead of relying on SUBSTR with length 0.
- mpe_assert_abort: both sides of the comparison go through %superq, so a child
  message containing %, & or a quote is neither rescanned nor able to break the
  quoted comparison.
- mpe_execute: the message is concatenated, not CATX'd, so it arrives byte for
  byte and a blank child message leaves no trailing blank.
- mpe_execute.test: three cases over the new macro - a comma-bearing message
  survives, the text names the program and state, and a blank message leaves the
  text clean.

mpe_execute 3/3 pass, mpe_assert_abort PASS, full suite 64/64 on sas.4gl.io,
sasjs cbd before each run.
2026-10-30 20:49:58 +00:00
dc 5d32de9df9 fix(mpe_assert_abort): size the capture buffer for the whole message
Build / Build-and-ng-test (pull_request) Successful in 4m56s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m45s
Build / Build-and-test-development (pull_request) Successful in 27m49s
The strip cuts the message at the "Log Extract:" marker, and got was declared
$256 - so a message whose text pushed the marker past 256 characters was never
stripped, and the exact comparison then failed on the escaped tail. got is now
the same width as line ($32767).

Covered by a new case in mpe_assert_abort.test.sas: an abort whose message runs
past 256 characters, asserted in full. It fails against the $256 buffer and
passes once got is widened, so the width is pinned rather than assumed.

mpe_assert_abort: PASS, full suite 64/64, rebuilt with sasjs cbd.
2026-10-30 20:01:19 +00:00
dc 8b426c27fd fix(mpe_assert_abort): compare the hook's message, not the log extract
Build / Build-and-ng-test (pull_request) Successful in 4m45s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m16s
Build / Build-and-test-development (pull_request) Successful in 30m39s
Addresses the review on #412.

The prefix comparison is gone instead of guarded. mp_abort appends "Log Extract:"
and the surrounding log to every message it returns; the capture now cuts at
that marker and strips the escaped newlines before it, so the comparison is an
exact match again. That removes the hole the reviewer found - a blank expectation
can no longer pass, because %length 0 no longer short-circuits anything - and it
keeps the expectations strict, so a change in the child's wording still fails the
test rather than sliding past on a matching prefix.

mpe_execute: the assembled message declares its own length. An inline symget()
is typed by the data step default, which would truncate a long child message
before it reached the abort.

The expected literals in stagedata.test.4 are unchanged by this commit: with
exact matching restored, stagedata passes 5/5, which is the confirmation the
reviewer asked for - they are the child's text, not text adjusted to fit the
observed log. Full suite 64/64.
2026-10-30 18:28:55 +00:00
dc ebfd39582f fix(mpe_execute): keep a child abort message intact when reporting it
Build / Build-and-ng-test (pull_request) Successful in 5m1s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m17s
Build / Build-and-test-development (pull_request) Successful in 30m39s
A child that does not complete is reported by aborting the caller with
"&program did not complete: state=&state &msg". The child's message can contain
commas - mp_abort appends a log extract to whatever the hook said, and the
derived-variable scan's own wording has one - so building that string inside the
macro call made the commas delimiters:

  ERROR: All positional parameters must precede keyword parameters.

The caller lost the child's message and the abort fired with a corrupted text.
Wrapping the whole thing in macro quoting is not the answer either: %bquote masks
'=' and '/' as well, and the mask characters end up in the message the user
reads (they show up in work.mp_abort_errds and in the JSON response as control
bytes).

The text is now assembled in a data step and passed through %superq, which keeps
it as one argument without altering the characters.

Two follow-on fixes in the same path:

- mpe_assert_abort compared the returned message exactly, but mp_abort appends
  "Log Extract:" and the surrounding log to every message it returns, so an
  expectation written as the message the hook raised could never match. The
  comparison now falls back to a prefix match, which keeps the exact-match
  behaviour for the messages that carry no extract.
- stagedata.test.4: both abort-expecting calls pass failfast=0 (the second one
  did not, so the test cancelled itself instead of reaching its assertion), the
  expected messages match the scan's own format ("MPE_DERIVED:CALLER:..." with
  no space after the first colons), and the expected text is passed through
  %superq because it contains a comma.
2026-10-30 17:52:20 +00:00
allan 8862798d65 Merge pull request 'fix(deps): bump @sasjs/core to 7.0.1 so special missings can be restored' (#409) from fix/restore-special-missings into release/7.22
Sites / deploy (push) Successful in 2m44s
Build / Build-and-ng-test (pull_request) Successful in 4m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m28s
Build / Build-and-test-development (pull_request) Successful in 28m17s
Reviewed-on: #409
2026-10-30 17:23:32 +00:00
dc dfc5c6ebb8 test(stagedata): do not let an expected child abort cancel the test
Build / Build-and-ng-test (pull_request) Successful in 4m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m27s
Build / Build-and-test-development (pull_request) Successful in 28m3s
Test 8 submits a VAR_LOGIC expression that the derived-variable scan refuses, so
the child is expected to abort and the assertion reads its message from the
response. Without failfast=0 mpe_execute aborted this program instead, so the
case could never reach its assertion. The file's other abort-expecting calls
already pass failfast=0.

This does not make the case pass on its own - the child still cancels without
writing a response, which is tracked separately - but it removes the wrong
failure in front of it.
2026-10-30 17:19:50 +00:00
dc 20dbf00fb0 fix(mpe_execute): mask a child abort message before it aborts the caller
mpe_execute reports a child that did not complete by aborting the caller with
"&program did not complete: state=&state &msg". The message was passed through
%str, which masks the literal text before resolution - so the commas in a
child's own abort message (mp_abort writes them as part of the reason) were
still delimiters when the call was parsed. The call split into extra positional
parameters and SAS raised

  ERROR: All positional parameters must precede keyword parameters.

replacing the real failure with a parse error at the call site. The caller then
lost the child's message entirely.

%bquote masks the resolved value, so the message arrives as one argument and the
abort reports what the child actually said.

The same shape in mpe_security_postedit, where the submitted ACCESS_LEVEL value
is interpolated into the abort message, is fixed the same way.
2026-10-30 17:19:50 +00:00
allan 9d82d7545e Merge pull request 'fix(deploy): drop the deprecated service-pack deploy from the setup screen' (#408) from chore/deprecate-service-pack-deploy into release/7.22
Sites / deploy (push) Successful in 2m45s
Build / Build-and-ng-test (pull_request) Successful in 4m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m7s
Build / Build-and-test-development (pull_request) Successful in 27m39s
Reviewed-on: #408
2026-10-30 17:04:26 +00:00
dc d67a0e5d5f fix(deploy): await nothing, and encode the values in the makedata URL
Build / Build-and-ng-test (pull_request) Successful in 4m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m19s
Build / Build-and-test-development (pull_request) Successful in 27m47s
Addresses both findings from the review of #408.

runAutoDeploy was declared async after the service-pack step went, but calls
createDatabase() and returns without awaiting anything. createDatabase() starts
the request and reports through the component's own status flags rather than a
returned promise, so there is nothing to await - drop the async rather than
pretend to a contract the callee does not offer.

runMakedataInNewWindow interpolated the context name, admin group and DC path
straight into the query string. An ampersand in any of them started a new
parameter and a space arrived truncated; a DC path containing a slash was only
working because nothing decoded the query. Those three values are now
percent-encoded at the build site, so the server reads back what the user chose.

Tests: the new-window case now asserts the encoded forms, and a second case
pins the values that need encoding (context with an ampersand and spaces, admin
group with a space and a plus, DC path with a slash and a space). The deploy
case no longer awaits a synchronous handler.

Executed 24 of 24 SUCCESS (automatic + sasjs-configurator + deploy components).
2026-10-30 16:14:28 +00:00
dc 3d2eefdd7b fix(deps): bump @sasjs/core to 7.0.1 so special missings can be restored
Build / Build-and-ng-test (pull_request) Successful in 4m29s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m9s
Build / Build-and-test-development (pull_request) Successful in 27m34s
Restoring any load that touched a special missing numeric (._ or .A-.Z) failed
in mp_stripdiffs, which writes the old value into a generated UPDATE statement.
PUT writes a special missing as its letter only, so the statement carried
"set SOME_BESTNUM =_" and PROC SQL read the bare underscore as a column name:

  ERROR: The following columns were not found in the contributing tables: _.
  ERROR: SOME_BESTNUM, a numeric column, can only be updated with a numeric
    expression.

Every statement in the generated block then failed NOEXEC, so the revert
produced the wrong data; the service's syscc check is what surfaced it.

Fixed upstream in sasjs/core#499 (the period is now written first) and
released as 7.0.1. v7.0.0's breaking change - mf_existvarlist warns and returns
0 instead of aborting on blank input - does not affect DC: there are no direct
callers, and the full SAS suite covers the core macros that do use it.

Adds restore.test.2.sas, which seeds some_bestnum=._ on mpe_x_test, changes it
to a number through stagedata and approves it, reverts that load, approves the
reversion, and asserts the cell is back to ._ rather than blank. It fails on
6.1.4 and passes on 7.0.1.
2026-10-30 16:07:31 +00:00
dc e3819e4350 test(deploy): cover the makedata hand-off on both setup screens
Build / Build-and-ng-test (pull_request) Successful in 4m47s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m2s
Build / Build-and-test-development (pull_request) Successful in 28m5s
The setup screen's only action is to run `makedata` with what the form
collected, and nothing tested it. Both screens now have specs pinning the
request and the URL they build:

- Viya (automatic): the admin group and DC path reach the service as
  `fromjs`, the configured context is used with the web method, a context chosen
  in the form wins over the configured one, the new-window URL carries
  `_contextname` / `admin` / `dcpath` / `_debug`, and pressing deploy reaches
  `createDatabase` at all
- SAS9 and SASjs Server (configurator): the makedata URL carries `_program`,
  `admin`, `path` and `_debug`; SAS9 stays on the page origin and encodes the
  directory, SASjs Server leaves the path readable, and the SASjs Server check
  runs before any reload

The deploy button is also restored. Removing the dead upload markup took the
live button with it, so the screen had no way to start the job its own copy
promises - the new `runAutoDeploy` test would have caught that.
2026-10-30 08:28:48 +00:00
dc 59e09e26a5 fix(deploy): drop the deprecated service-pack deploy from the setup screen
Build / Build-and-ng-test (pull_request) Successful in 4m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m54s
Build / Build-and-test-development (pull_request) Successful in 29m50s
The Viya setup screen still carried a path that deployed the backend as a
service pack from a viya.json (`deployServicePack`). That is no longer how a
deployment works: the frontend and services arrive with the sasjs deploy, and the
only thing left to do is create the control library and its configuration data,
which `makedata` does - and then deletes itself.

The path was already unreachable from the UI (its button had been commented out
"for reference"), so the wizard only ever ran `makedata` in practice while its
copy still claimed to deploy services. Removed:

- `executeJson()` and the `viya.json` upload plumbing: `readFile`,
  `clearUploadInput`, and the fields and status flag that fed them
- the "Deploy SAS Jobs" row in the completion modal, which could never show
- the commented-out button and file input that kept the dead path visible

The screen now says what it does: it runs the one-off job that creates the
control library and its configuration data, and the status row reads "Create
control library" rather than "Create database".
2026-10-29 18:34:35 +00:00
allan eaa3b3e619 Merge pull request 'fix(embed): a rejected mapping must not block the page' (#406) from fix/va-parmap-column into release/7.22
Sites / deploy (push) Successful in 2m51s
Build / Build-and-ng-test (pull_request) Successful in 4m55s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m45s
Build / Build-and-test-development (pull_request) Successful in 28m56s
Reviewed-on: #406
2026-10-29 18:29:30 +00:00
dc d845b9e9b7 fix(embed): a rejected mapping must not block the page
Build / Build-and-ng-test (pull_request) Successful in 4m55s
Lighthouse Checks / lighthouse (pull_request) Successful in 16m59s
Build / Build-and-test-development (pull_request) Successful in 27m58s
The deferred VA load builds the filter from the report's parameters before the
first getdata, which is what makes that first load already filtered. A va_parmap
entry is a column the report author typed, so the request can legitimately fail
when the table does not have it - validatefilter aborts on an unknown variable -
and that abort surfaced as a modal over an editor which then loaded fine
(dc/dc#404).

Validating the mapping first would cost a SAS call on every load of a mapped
object, which is not affordable, so the attempt is simply made quietly:
saveQuery takes wrapper options and the deferred load suppresses the error
modal. A mapping that is right still filters the very first load; a wrong one
falls back to the normal load, and the metadata pass reports it in the banner
and reconfigures once, at setup time rather than on every page load.
2026-10-29 18:22:41 +00:00
semantic-release-bot 428e70a4c1 chore(release): 7.21.0 [skip ci]
# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)

### Bug Fixes

* **client:** parse word-date values from VA parameters ([405b28d](405b28dccc))
* **client:** parse word-date values from VA parameters ([ce5b1aa](ce5b1aa961))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](eeffa92f9e))

### Features

* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](a4a0f9cb2f))
2026-10-29 16:32:45 +00:00
allan 94e040c006 Merge pull request 'feat: Release/7.21' (#401) from release/7.21 into main
Release / Build-production-and-ng-test (push) Successful in 3m30s
Release / Build-and-test-development (push) Successful in 20m56s
Release / release (push) Successful in 5m15s
Reviewed-on: #401
2026-10-29 16:05:27 +00:00
allan cdc0ab2a91 Merge pull request 'fix(client): parse word-date values, and seed demo dates that repeat (release/7.21)' (#405) from fix/va-word-date-release into release/7.21
Sites / deploy (push) Successful in 2m56s
Build / Build-and-ng-test (pull_request) Successful in 4m58s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m9s
Build / Build-and-test-development (pull_request) Successful in 27m56s
Reviewed-on: #405
2026-10-29 15:08:00 +00:00
dc 8449ab3987 test(data): seed the demo table with dates that repeat
Build / Build-and-ng-test (pull_request) Successful in 4m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m26s
Build / Build-and-test-development (pull_request) Canceled after 18m20s
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.

No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
2026-10-29 14:42:33 +00:00
dc 405b28dccc fix(client): parse word-date values from VA parameters
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.

parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.

A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.

Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
2026-10-29 14:42:33 +00:00
allan 559221fad4 Merge pull request 'fix(client): parse word-date values, and seed demo dates that repeat' (#403) from fix/va-date-worddate into main
Release / Build-production-and-ng-test (push) Canceled after 10s
Release / Build-and-test-development (push) Canceled after 0s
Release / release (push) Canceled after 0s
Reviewed-on: #403
2026-10-29 14:39:39 +00:00
dc 999e16b8fc test(data): seed the demo table with dates that repeat
Build / Build-and-ng-test (pull_request) Successful in 4m6s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m45s
Build / Build-and-test-development (pull_request) Successful in 26m22s
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.

No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
2026-10-29 14:05:29 +00:00
dc ce5b1aa961 fix(client): parse word-date values from VA parameters
Build / Build-and-ng-test (pull_request) Successful in 3m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m17s
Build / Build-and-test-development (pull_request) Failing after 27m2s
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.

parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.

A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.

Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
2026-10-29 13:43:47 +00:00
allan f74100a818 Merge pull request 'feat(derived): apply the core field scan in the MPE_DERIVED hook' (#402) from refactor/derivedvarcheck into release/7.21
Sites / deploy (push) Successful in 3m23s
Build / Build-and-ng-test (pull_request) Successful in 6m10s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m15s
Build / Build-and-test-development (pull_request) Successful in 29m46s
Reviewed-on: #402
2026-10-29 11:25:26 +00:00
dc a4a0f9cb2f feat(derived): apply the core field scan in the MPE_DERIVED hook
Build / Build-and-ng-test (pull_request) Successful in 6m6s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m15s
Build / Build-and-test-development (pull_request) Successful in 32m40s
@sasjs/core 6.1.3 moved the derived variable field rules out of
mp_derivedvars into mp_derivedvarcheck, so the same scan can run before a
definition is used.  DC carried its own copy of that list in
mpe_derived_check.sas, which is now deleted: the post edit hook calls the
core macro instead, and the hook keeps only the rules that belong to the
control table rather than the expression - LIBREF/DSN required, and ACTIVE
within the picker pair.

The scan is stricter than the copy it replaces, in three ways:

- VAR_NM may not be _N_, _ERROR_ or _IORC_ (automatic variables of the data
  step)
- VAR_LOGIC may not exceed 4000 characters
- VAR_LOGIC may not call a function that submits code or reaches the file
  system, the environment or the macro symbol table - a function name cannot
  be assembled at run time, so this is a control the expression cannot evade

and it relaxes one: VAR_LABEL may carry a line break, which is flattened to a
space when the label is used, so only the remaining control characters are
refused.

The mock hook mirror is updated to the same rules, and stagedata.test.4
gains the two submit-time cases that pin the new wiring.
2026-10-29 10:51:00 +00:00
allan e6a5cce475 Merge pull request 'fix(makedata): name the sasjs server group in the estate seed' (#400) from fix/estate-seed-admin into release/7.21
Sites / deploy (push) Successful in 4m20s
Build / Build-and-ng-test (pull_request) Successful in 6m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m47s
Build / Build-and-test-development (pull_request) Successful in 33m53s
Reviewed-on: #400
2026-10-29 08:33:52 +00:00
semantic-release-bot 97b4cd4d61 chore(release): 7.20.0 [skip ci]
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)

### Bug Fixes

* **ci:** install doxygen for the SAS docs job ([f58925b](f58925b299))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](27686d3aad))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](888873421c))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](dd482c4069))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](25ffa71add))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](c5808a16eb))
* **ci:** rebuild and upload release assets on demand ([6486c80](6486c80ed4))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](af1813f829))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](cf7b21a8b6))
* **cypress:** hold download out too, and retry once in CI ([c97e886](c97e8866db)), closes [#395](#395) [#398](#398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](c803773d3c)), closes [#395](#395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](80e8911af8))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](e51953ca8a))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](459ff2da37)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](f7e753863b))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](7e3045a21b))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](2623a3fb4b))
* **embed:** log a parse-time mapping problem once ([0011f7e](0011f7e933))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](541a61ba08))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](e9216b0895))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](bd3c220af2))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](f83dc359e2))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](aa91daf9f7))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](10157c1159))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](411f6c3354))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](c0ea810766))

### Features

* **embed:** map VA parameters to DC columns from the URL ([15a1f34](15a1f34d52))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](88010c3980))
2026-10-28 23:11:50 +00:00
dc eeffa92f9e fix(makedata): name the sasjs server group in the estate seed
Build / Build-and-ng-test (pull_request) Successful in 6m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m54s
Build / Build-and-test-development (pull_request) Successful in 34m49s
makeDataServer.json seeded ADMIN=AllUsers, a Viya-style group that no account on
the shared estate belongs to. The seed still reports SUCCESS with a group the
caller is not in, and then every getdata/stagedata call aborts with
'<user> is not authorised to edit <libds> in the MPE_SECURITY table', which
reads like a broken deployment. The estate's accounts are in all-users, the
SASjs Server group every logged-in user has, so seed with that and describe the
rule in the estate-testing skill rather than telling the reader to override the
shipped value.
2026-10-28 22:48:02 +00:00
allan dd25be8e75 Merge pull request 'Release/7.20' (#387) from release/7.20 into main
Release / Build-production-and-ng-test (push) Successful in 4m19s
Release / Build-and-test-development (push) Successful in 28m0s
Release / release (push) Successful in 7m31s
Reviewed-on: #387
2026-10-28 22:35:17 +00:00
allan 2e40749052 Merge pull request 'docs(skills): add the estate deploy-and-test procedure' (#397) from docs/estate-testing-skill into release/7.20
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #397
2026-10-28 22:34:37 +00:00
allan fc687f8643 Merge pull request 'fix(cypress): hold download out too, and retry once in CI' (#399) from fix/cypress-release-specs into release/7.20
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #399
2026-10-28 22:34:22 +00:00
dc c97e8866db fix(cypress): hold download out too, and retry once in CI
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 2m15s
Build / Build-and-test-development (pull_request) Canceled after 0s
The run on release/7.20 went from four failing specs to two:

    csv-limited   PASS   (order restored)
    hook-programs PASS   (order restored)
    metanav       held out, #395
    download      7 of 7 fail

download now gets past its hooks - the deleteFolder task is registered - and
fails on the button it clicks:

    AssertionError: expected '<button.btn.btn-sm.btn-success...>' to be 'visible'
    This element is not visible because its ancestor has 'position: fixed' CSS
    property and it is overflowed by other elements.

Something fixed-position sits over it. That needs a scrollIntoView or an overlay
dismissed, and each attempt costs a forty minute CI cycle, so it is held out with
#398 carrying the exact error and the suggested fix. The list reports 22 run, 2
excluded.

editor.cy.ts still fails one test of fifty-five, and it failed the same way in
both runs: 'Expected <div.app-loading> not to exist ... continuously found', a
load timeout on test 28 rather than an assertion. One retry in run mode absorbs
that, none interactively; a real failure still fails twice and the retry's output
is in the log.
2026-10-28 22:20:32 +00:00
dc ad061946c3 docs(skills): add the estate deploy-and-test procedure
Build / Build-and-ng-test (pull_request) Successful in 6m56s
Lighthouse Checks / lighthouse (pull_request) Canceled after 19m45s
Build / Build-and-test-development (pull_request) Canceled after 16m16s
The shared-estate cycle is two npm scripts and sasjs test, but the seed in
npm run server is load-bearing: without it testsetup 404s settings.sas and
59 of the 64 test programs fail to complete, which reads like a broken
deployment. Record the targets, the token-pair credentials, the ADMIN group
the estate actually uses, how to read the results, and the frontend
passphrase steps as a repo-local skill, and point the testing doc at it.
2026-10-28 22:03:34 +00:00
allan 914a1606bb Merge pull request 'fix(cypress): register the deleteFolder task, restore the spec order, hold metanav out' (#396) from fix/cypress-release-specs into release/7.20
Sites / deploy (push) Successful in 3m40s
Build / Build-and-ng-test (pull_request) Successful in 6m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m38s
Build / Build-and-test-development (pull_request) Failing after 41m54s
Reviewed-on: #396
2026-10-28 21:37:20 +00:00
dc c803773d3c fix(cypress): register the deleteFolder task, restore the spec order, hold metanav out
Build / Build-and-ng-test (pull_request) Successful in 6m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m9s
Build / Build-and-test-development (pull_request) Failing after 41m32s
Four specs failed on release/7.20 once the list went from 19 to 24. Three of the
four are addressed here; the fourth is a spec that cannot pass and is now
excluded with an issue.

download.cy.ts - it clears the downloads folder in an afterEach through
cy.task('deleteFolder'), and that task was registered only in
cypress/plugins/index.js, which Cypress 10+ never loads unless it is required
from cypress.config.ts. The task is now registered in setupNodeEvents, where
Cypress looks, and the login in its beforeEach only fills the form when the form
is actually on screen - the unconditional version failed with 'the center of this
element is hidden from view' whenever the session was already live.

csv-limited.cy.ts and hook-programs.cy.ts - neither is broken; both depend on
where they sit in the run. csv-limited asserts the free-tier 'Locked Feature'
modal, which an earlier spec removes by planting a valid licence, and
hook-programs asserts an approval row against its target table before other specs
have added rows to the approval tables. The list is now the sequence that was
green for months, with csv-limited first and hook-programs second, and the newer
specs appended rather than inserted. The script says why order matters, because
the next person to alphabetise it will break the same two specs.

metanav.cy.ts - cannot pass, and says so in its own source: '// BLOCKER ...
.clr-accordion-header-button always null although it is present on the page'.
With the duplicate login in its beforeEach removed it now reaches that assertion
and fails there, so it is excluded from the list with #395 as the reason, and the
login fix is kept for whoever picks it up. The coverage check now distinguishes a
deliberate exclusion from a forgotten file, so nothing can go missing silently.

Not verified locally: the specs need an estate seeded by 'sasjs cbd -t server-ci'
plus makedata, which a scratch mock does not reproduce - the app never leaves
app-loading. The CI run is the verification.
2026-10-28 21:29:50 +00:00
allan b8eea5063c Merge pull request 'ci(build): run the Cypress suite in one job again, keeping the spec-list guard' (#394) from ci/unshard-cypress into release/7.20
Sites / deploy (push) Successful in 3m46s
Build / Build-and-ng-test (pull_request) Successful in 6m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m57s
Build / Build-and-test-development (pull_request) Failing after 37m40s
Reviewed-on: #394
2026-10-28 19:38:08 +00:00
allan 0a172b0c37 Merge pull request 'fix(ci): address the release workflow review findings' (#393) from fix/release-review-566-573 into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 6m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m10s
Build / cypress-shard-1 (pull_request) Failing after 13m19s
Build / cypress-shard-2 (pull_request) Failing after 13m58s
Build / cypress-shard-4 (pull_request) Failing after 14m23s
Build / cypress-shard-3 (pull_request) Successful in 17m37s
Build / Build-and-test-development (pull_request) Failing after 2s
Reviewed-on: #393
2026-10-28 19:37:37 +00:00
dc@4gl.io 888873421c fix(ci): publish against the tag this job cuts, not releases/latest
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
semantic-release bumps the working-tree package.json before the release
commit, so the tag is known in the job. Using it for the asset upload and
both dispatches means an overlapping release cannot redirect them.
2026-10-28 19:36:13 +00:00
dc@4gl.io af1813f829 fix(ci): use the lockfile and the pinned CLI in the docs job 2026-10-28 19:36:05 +00:00
dc@4gl.io 27686d3aad fix(ci): keep foreign release assets, install jq first, pin the CLI
Only the assets this workflow uploads are removed before a re-run, so a
hand-attached asset on the release survives. jq is installed before the
tag resolve that parses its output, the SASjs CLI is installed at the
version the tag's sas/package.json pins, and the npmrc secret is appended
rather than overwriting the file.
2026-10-28 19:35:57 +00:00
dc d84a1d50f4 ci(build): run the Cypress suite in one job again, keeping the spec-list guard
Build / Build-and-ng-test (pull_request) Successful in 6m49s
Lighthouse Checks / lighthouse (pull_request) Canceled after 1m17s
Build / Build-and-test-development (pull_request) Canceled after 0s
The four shards made the release-branch run red, and the reasons are in the
logs rather than in the theory:

  - every failure was a timeout, not an assertion - "cy.type() failed because
    the center of this element is hidden from view", "Expected
    <div.app-loading> not to exist ... continuously found".  Four jobs on one
    runner means four SASjs servers, four ng serve and four Chromes competing
    for the same CPU, and the app under test starves;
  - upload-artifact@v4 does not work on this runner at all:
    "GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ ...
    not supported on GHES", so the v3 -> v4 move I made on review advice broke
    the artifact step outright.

Measured, from the one sharded run that did complete: 20m34s wall against 33-35
minutes for the single job, with peak concurrency 3 - so a third off, not the
four-fold the split suggests.  That is not worth a flaky pipeline while a release
sits on it.

Kept from the sharding work, because those parts were the real gain:

  - the spec list has one home, .gitea/scripts/cypress-specs.sh;
  - Build-and-ng-test fails when a spec on disk is not listed, so the five that
    had never run in CI cannot go missing again - and they do run now;
  - videos are zipped and uploaded only on failure;
  - one job means one server, which is also what the guards and the balance
    arguments were protecting.

The shard and fan-in jobs are gone with the branch that carried them.
2026-10-28 19:27:51 +00:00
dc@4gl.io cf7b21a8b6 fix(ci): verify the forge certificate on the release API calls
Build / Build-and-ng-test (pull_request) Successful in 6m48s
Lighthouse Checks / lighthouse (pull_request) Canceled after 10m53s
Build / cypress-shard-1 (pull_request) Canceled after 1m20s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Failing after 3s
curl -k disables verification on calls that carry the release token, so a
man-in-the-middle on the runner's egress could capture a credential that
rewrites releases. The forge serves a publicly trusted certificate (the
runner verifies the same chain on every checkout), so the flag is
unnecessary: drop it from all fifteen calls in the two release workflows.
2026-10-28 19:21:57 +00:00
allan 60589c78b6 Merge pull request 'fix(embed): report a name-keyed va_parmap entry, and keep the app query on the standalone URL' (#391) from fix/va-parmap-review-572 into release/7.20
Sites / deploy (push) Successful in 3m46s
Build / Build-and-ng-test (pull_request) Successful in 6m19s
Build / cypress-shard-1 (pull_request) Failing after 13m58s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m17s
Build / cypress-shard-2 (pull_request) Failing after 14m33s
Build / cypress-shard-3 (pull_request) Failing after 4m36s
Build / cypress-shard-4 (pull_request) Failing after 14m10s
Build / Build-and-test-development (pull_request) Failing after 6s
Reviewed-on: #391
2026-10-28 19:02:31 +00:00
allan 8e2cce4150 Merge pull request 'ci(build): split the Cypress suite into four shards, and keep videos for failures' (#392) from ci/parallel-cypress into fix/va-parmap-review-572
Build / Build-and-ng-test (pull_request) Canceled after 10s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 11s
Build / Build-and-test-development (pull_request) Failing after 2s
Reviewed-on: #392
2026-10-28 19:01:59 +00:00
dc@4gl.io f7e753863b fix(deploy): strip the embed parameters by name, and scope the timezone runs to the VA specs
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 1s
The standalone URL dropped every query parameter whose name started with
va_, not just va_parmap; the names are now enumerated, and a later
parameter with that prefix survives (the component spec pins the exact
standalone URL instead of a substring, so the query surviving is the
assertion, not an accident of the runner's own query).

The timezone runs are scoped to the VA specs, the only specs with local
calendar arithmetic, so the blocking job no longer runs the whole suite
twice more.
2026-10-28 18:58:26 +00:00
dc febfea3e5e ci(build): make the fan-in gate fail, and drop the duplicated blocks
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The gate could not fail.  With needs: on four shards and no if: always(), a
failed shard skips the fan-in job instead of failing it, and a skipped job is
not a failure on the one check branch protection watches - so the gate was
silent in exactly the case it exists for.

It now runs with if: always() and fails on any shard result that is not
success.  The count is checked as well: if the needs context does not resolve on
this runner the step fails saying so, rather than looping over nothing and
reporting green - which is the failure mode a bare loop would have.

Also removes what the last regeneration left behind: the coverage guard step
appeared twice in Build-and-ng-test (so it ran twice) and the shard 1 comment
block appeared three times.  The file is regenerated from fix/va-parmap-review-572
rather than from my own previous commit, which is what caused the duplication.
2026-10-28 18:55:18 +00:00
dc 3580d6e9eb ci(build): gate the shards behind the job name branch protection already requires
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Renaming the shards to cypress-shard-1..4 and adding a job named
Build-and-test-development that does nothing but needs all four.

That name is the status check the branch protection rule requires, so the check
now turns green only when every shard passed - a red shard 3 leaves it
unreported and the PR unmergeable, which is the behaviour the single job had.
No repo setting has to change, and adding a fifth shard later means adding it to
needs rather than remembering to edit branch protection, where forgetting fails
silently.
2026-10-28 18:31:27 +00:00
dc bc9e3ae50f ci(build): give the shard lists one home, cover the five missing specs, and guard it
Build / Build-and-ng-test (pull_request) Successful in 7m36s
Lighthouse Checks / lighthouse (pull_request) Canceled after 1m41s
Build / Build-and-test-development (pull_request) Canceled after 0s
Build / Build-and-test-development-2 (pull_request) Canceled after 0s
Build / Build-and-test-development-3 (pull_request) Canceled after 0s
Build / Build-and-test-development-4 (pull_request) Canceled after 0s
Review found that five specs ran in no shard - coltype-delete-record,
download, embed-va, metanav, submitted-details - and one of them is the VA
embed suite this release added, 454 lines that had never run in CI.  The old
single --spec string omitted them too, so nothing regressed; they were simply
never wired up, and nothing in the repo marked them skipped.

The spec lists now live in .gitea/scripts/cypress-shards.sh, one place, so the
workflow no longer spells them out four times.  A guard step in
Build-and-ng-test compares the shards against the specs on disk and fails when a
file belongs to none of them or to more than one - proven by removing a name and
watching it fail with that file listed.

Rebalanced for the five additions, on test count and spec count together, since
each spec pays a few seconds of startup:

    shard 1  editor, csv, metanav                                    57 tests
    shard 2  excel, filtering, download, browser-info,
             excel-multi-load, hook-programs                         49 tests
    shard 3  embed-va, viewbox, licensing, filter-panel, liveness     43 tests
    shard 4  viewer-labels, row-cell-limits, derived-variables,
             ddl-export, coltype-delete-record, submitted-details,
             startupservice-launcher, stage, full-table-search,
             csv-limited                                             27 tests

24 specs / 176 tests, each in exactly one shard.

upload-artifact moved from the retired v3 to v4; this change multiplies that
step by four, so it is worth not carrying the dead major forward.

Not in this commit: the three new check names still need adding to branch
protection, which needs repo admin.
2026-10-28 18:26:45 +00:00
dc 330448c496 ci(build): split the Cypress suite into four shards, and keep videos for failures
Build / Build-and-ng-test (pull_request) Successful in 7m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 0s
Build / Build-and-test-development-2 (pull_request) Canceled after 0s
Build / Build-and-test-development-3 (pull_request) Canceled after 0s
Build / Build-and-test-development-4 (pull_request) Canceled after 0s
Sits on top of fix/va-parmap-review-572 (#391) so the two changes to this
workflow land in one order; the timezone step that PR adds to
Build-and-ng-test is untouched.

The development job ran all nineteen specs in one Cypress invocation on one
runner, so its wall time was the sum of every spec - around half an hour on a
green run, with the whole pipeline waiting on it.

Four jobs now, each with its own runner and its own SASjs server, taking a
subset:

    shard 1  editor (55 of the suite's ~148 tests)
    shard 2  excel, filtering, csv-limited, csv, stage
    shard 3  viewbox, licensing, filter-panel, browser-info, excel-multi-load, liveness
    shard 4  viewer-labels, row-cell-limits, derived-variables, ddl-export,
             hook-programs, startupservice-launcher, full-table-search

The split is balanced on test count, not file count: editor.cy.ts alone holds 55
tests, so an even spread of files would leave one shard running twice as long as
the others.  Its own duration is the floor the pipeline can reach.

The setup steps are identical in every shard, so they are defined once and
aliased with a YAML anchor; each job differs only in SHARD and SPECS.  Shard 1
keeps the original job name so an existing required status check still matches.

Video is still recorded - measured at 91s with it and 92s without on the same
spec, so the recording is not what takes the time.  What it did cost was the zip
and artifact upload of every run's videos, now guarded by if: failure().
2026-10-28 18:21:38 +00:00
dc 00257f9160 ci(client): run the client suite in two more timezones
Lighthouse Checks / lighthouse (pull_request) Successful in 20m42s
Build / Build-and-ng-test (pull_request) Successful in 7m8s
Build / Build-and-test-development (pull_request) Canceled after 0s
2026-10-28 18:04:01 +00:00
dc e51953ca8a fix(deploy): keep the app query when opening the standalone URL 2026-10-28 18:03:53 +00:00
dc aa91daf9f7 fix(embed): record the parameter label only, so a name-keyed va_parmap is reported 2026-10-28 18:03:45 +00:00
allan 00df87f189 Merge pull request 'fix(makedata): pair the %else of the keepme block so it parses in open code' (#389) from fix/makedata-selfdestruct into release/7.20
Sites / deploy (push) Successful in 3m43s
Build / Build-and-ng-test (pull_request) Successful in 5m5s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m48s
Build / Build-and-test-development (pull_request) Canceled after 32m30s
Reviewed-on: #389
2026-10-28 17:09:17 +00:00
allan 65699bdd39 Merge pull request 'fix(deploy): do not offer the deploy wizard inside a report' (#388) from feat/embed-not-configured into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m6s
Build / Build-and-test-development (pull_request) Successful in 34m34s
Reviewed-on: #388
2026-10-28 17:08:40 +00:00
dc 459ff2da37 fix(deploy): one definition of the standalone URL, and test the guards
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Successful in 33m54s
Review follow-up on the embed guard.

The spec asserted the isEmbedded getter, which mirrors the template rather
than exercising it: dropping !isEmbedded from the @if guards left every test
green. The spec now renders the component in a TestBed and asserts the DOM -
#TCS present standalone and absent for embed=va and embed=true, and the
deploy section withheld after the terms are accepted when embedded. Both
guards were mutation-checked by removing them, which fails the spec.

deployUrl and onNavigateToHome computed the standalone URL two different
ways: origin+pathname in one, location.href.split('#')[0] in the other,
which kept the query string, so a tab opened from ?embed=va#/deploy stayed
embedded. Both now call standaloneUrl() in standalone-url.ts, which parses
the href and drops the query and the fragment. It is tested with a
hash-bearing input, and the navigate test asserts the opened URL carries
neither '?' nor '#', so a regression to split('#') fails.
2026-10-28 17:05:35 +00:00
dc c0ea810766 fix(makedata): pair the %else of the keepme block so it parses in open code
Build / Build-and-ng-test (pull_request) Successful in 4m54s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m1s
Build / Build-and-test-development (pull_request) Successful in 32m10s
The self-destruct block ends with a single-statement %else in open code:

    %if &keepme ne 1 %then %do;
      %mv_deletejes(path=&root/admin, name=makedata);
    %end;
    %else %put NOTE: &_program retained, as KEEPME was requested;

Outside a macro SAS pairs %else only with a %do branch, so this stops with
'ERROR: Expected %DO not found' and 'ERROR: Skipping to next %END statement'
after the deployment has completed. On Viya the job then reports state error
with jobConditionCode 1012, so a successful deployment looks like a failure
exactly when someone is watching a first install.

Both branches now use %do/%end, which is the pattern the repo uses elsewhere
for this reason - see db/migrations/20261003_row_and_cell_limits.sas.

The same block is fixed in the server variant. The two delete calls also gain
their terminating semicolons: the macro call worked without them, but their
absence is what made the block look responsible for the error.

Behaviour is unchanged: the program still deletes itself unless KEEPME was
requested.
2026-10-28 16:38:21 +00:00
allan 62e43034e4 Merge pull request 'feat(embed): resolve va_parmap values through named date anchors' (#386) from feat/va-parmap-transforms into release/7.20
Sites / deploy (push) Successful in 3m46s
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m7s
Build / Build-and-test-development (pull_request) Successful in 34m1s
Reviewed-on: #386
2026-10-28 15:08:45 +00:00
dc 80e8911af8 fix(deploy): do not offer the deploy wizard inside a report
Build / Build-and-ng-test (pull_request) Successful in 5m10s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m36s
Build / Build-and-test-development (pull_request) Successful in 33m18s
Deployment needs an authenticated, standalone session, and the person
looking at a report object is not the person who installs the app. Offering
the wizard there produces a screen whose controls cannot work - the failure
we hit while testing a report: T&C accepted, credentials filled, DEPLOY
clicked, and nothing happened, because the embedded app has no session to
deploy with.

An embedded instance (embed=va or embed=true) now shows what the state is
and where to deploy from, instead of a form that cannot submit. Standalone
behaviour is unchanged.
2026-10-28 15:08:02 +00:00
dc bd3c220af2 fix(embed): never build a clause from a value that did not convert, and say so in the banner
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m25s
Build / Build-and-test-development (pull_request) Successful in 33m8s
Two review findings, both about a filter that looks applied and is not:

- The transform branch of the clause builder interpolated whatever
  convertJsDateToSasDate returned, so an Invalid Date reaching the helper
  would have put a literal NaN in the WHERE clause. A non-finite result is
  now dropped, and unsupportedTransforms reports the parameter, so the
  author sees a banner instead of a broken filter.
- The range count check counted values that survived date parsing, while the
  builder counts resolved bounds. A RANGE sent two values where one did not
  parse therefore built a BETWEEN covering one period with no warning. The
  check now counts what VA sent, the builder refuses a range with more than
  one value rather than picking one, and a value set that is partly
  unparseable gets its own line ('1 of its 2 values was not a date - the
  filter covers only the rest').

The banner also now says what to do: it names the URL parameter to edit and
shows the entry grammar with an example, since the reader is the report
author and has no other way to act on it.
2026-10-28 12:44:05 +00:00
allan 5e8910f238 Merge pull request 'feat(embed): map VA parameters to DC columns from the URL' (#383) from feat/va-param-map into release/7.20
Sites / deploy (push) Successful in 3m44s
Build / Build-and-ng-test (pull_request) Successful in 4m58s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m54s
Build / Build-and-test-development (pull_request) Successful in 33m9s
Reviewed-on: #383
2026-10-28 12:41:09 +00:00
dc 221dd8a15e Merge feat/va-param-map: label-only mapping keys and value-aware same-column warning
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m49s
Conflict was valueSignature: the transform branch already builds a clause
value set through buildClauseValueSet, so the helper delegates to it and
takes the parameter's transform, keeping derived dates in the comparison.
2026-10-28 12:34:32 +00:00
dc e9216b0895 fix(embed): match a mapping on the parameter label only, and warn only when the values differ
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Canceled after 0s
Two review findings, plus a scope decision.

A va_parmap key is now matched against the parameter's label only. Matching
the internal name (pr273) as well meant one parameter answered to two keys,
so an entry written against the name and one against the label could never
be told apart: the label entry won and the name entry was silently dead -
no parse problem, no unresolved line, no unmatched line. Dropping the
fallback removes the ambiguity, and an entry keyed on the internal name now
shows up as an unmatched mapping like any other typo.

sameColumnMappings no longer reports two parameters on one column when they
carry the same value: col = 'x' AND col = 'x' filters exactly as one clause
does, yet the banner asserted that different values would show no rows. The
clause value each parameter produces is now compared, and the warning is
raised only when they differ. Parameters that produce no clause at all are
excluded, since they cannot conflict with anything.
2026-10-28 12:32:47 +00:00
dc 37d81c902a test(embed): run the suite in two more timezones
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m37s
Build / Build-and-test-development (pull_request) Canceled after 0s
A transform resolves a report value to dates using local calendar
components, so a test that passes in UTC says nothing about a browser west
of it. test:tz-west and test:tz-east run the whole suite under
America/New_York and Pacific/Auckland; with the UTC construction this test
pair replaces, the bare-number round-trip case fails under the first.

Not wired into CI: it triples the run for a property that is already
pinned by the round-trip assertion, so it stays a local and pre-release
check.
2026-10-28 12:27:42 +00:00
dc 689b5b6709 Merge feat/va-param-map into feat/va-parmap-transforms
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m29s
Build / Build-and-test-development (pull_request) Successful in 33m58s
Brings in the diagnostics fixes from the base branch; conflicts were the
paramMap type widening to VaParmapTarget and the new paramsStable argument,
both of which the transform branch already carries.
2026-10-28 12:24:36 +00:00
dc 10157c1159 fix(embed): resolve transform values as local dates and give ranges whole days
Review findings:

- A bare SAS number was turned into a UTC instant and then read with local
  accessors, so west of UTC the value landed a day early and the anchor was
  computed from the wrong quarter; the round trip to a SAS day number was
  one out. Built with local calendar construction instead, matching
  parseDate and convertJsDateToSasDate.
- RANGE passed keepTime to both bounds, so a DATETIME range started at the
  value's clock time on day one and ended at it on the last day, dropping
  the rest of both boundary days. A range now covers whole days: period
  start at midnight, period end at 23:59:59.
- A RANGE fed more than one parameter value produced no clause and no
  warning, reading as a filter that matched nothing. One BETWEEN cannot
  express several periods, so the count mismatch is now reported.
2026-10-28 12:20:49 +00:00
dc 541a61ba08 fix(embed): make the va_parmap diagnostics say what they mean
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s
Build / Build-and-test-development (pull_request) Successful in 33m26s
Review findings, all about a warning that misleads:

- The banner heading said the mapping was ignored, but two of the three
  lists it renders are mappings that are applied (the AND-ed pair) or that
  were never applicable (a parameter VA does not send). Neutral heading.
- unmatchedMappings reported a key as unsent as soon as any parameter had
  been seen. VA sends the parameters a report object's query uses, so a
  name that has not arrived yet looked like a typo. A mapping is now only
  judged once a message introduces no new parameter name, i.e. once the
  object's parameter set has settled.
- sameColumnMappings counted a column the table does not have, so one
  mistake produced the unresolved-mapping line and the AND line. A column
  that is not in the table is now left to unresolvedMappings.
- The Cypress cases asserted exact row counts, which encode the demo
  table's contents. They now read the unfiltered count from the app and
  assert the filtered grid is smaller (or empty), so a fixture change
  cannot fail a behaviour test.
2026-10-28 12:20:37 +00:00
allan ad50acf647 Merge pull request 'fix(ci): publish the typedoc to the webdoc site, not the app default' (#385) from fix/webdoc-named-site into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #385
Reviewed-by: Allan <allan@4gl.io>
2026-10-28 12:10:16 +00:00
dc cd1ce4a7e2 test(embed): format the transform spec to satisfy prettier --check
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
2026-10-28 12:01:11 +00:00
dc 7e3045a21b fix(embed): build quarter ends from the calendar, not minus 24 hours
Build / Build-and-ng-test (pull_request) Failing after 2m2s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Canceled after 6m38s
Subtracting a fixed day from the next quarter's start moves by an hour
across a daylight-saving transition, which would place a DATETIME bound
off the intended midnight. Day 0 of the following month is the last day
of the month by construction.
2026-10-28 11:57:23 +00:00
dc 88010c3980 feat(embed): resolve va_parmap values through named date anchors
Build / Build-and-ng-test (pull_request) Failing after 1m55s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m22s
A VA parameter can now carry a transform on the column side of a mapping:

  va_parmap=SELECTED_DATE-D_DATE:ADD(QUARTER_START,-1)

which filters D_DATE on the report value AND the same point one quarter
earlier, as a single IN clause - so the AND rule, the clause signature,
saveQuery and the reload loop are untouched.

Transforms resolve to one clause with these operations:

- ADD(<ANCHOR>,<offset>)   the value plus one anchor beside it -> IN
- RANGE(<PERIOD>,<offset>) a closed period -> BETWEEN

Anchors: QUARTER_START/END, MONTH_START/END, YEAR_START/END.
Periods: QUARTER, MONTH, YEAR. Offset is a whole number of periods,
defaulting to 0.

The names describe the result rather than mirroring SAS intnx: the
arithmetic runs in the browser, so mirroring intnx would mean
reimplementing its alignment letters and their corners (week alignment,
's' and 'm') with no guarantee of matching SAS. A report author can read
ADD(QUARTER_START,-1) without knowing intnx.

Resolution is anchored on the parameter value, never on today, so a saved
report reproduces the same rows. A transform that cannot run where it is
used - a non-temporal column, a value that is not a date - is skipped and
reported in the existing warning banner rather than silently filtering on
the raw value.
2026-10-28 11:52:53 +00:00
dc c5808a16eb fix(ci): publish the typedoc to the webdoc site, not the app default
Build / Build-and-ng-test (pull_request) Successful in 5m42s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m39s
Build / Build-and-test-development (pull_request) Canceled after 19m50s
`surfer deploy` chooses its target only from `--site-name`; without one it
replaces the app's DEFAULT site. The typedoc job passed `--server
webdoc.datacontroller.io` and no site name, so the 7.19.0 docs were published
over the default site of that app - the marketing site served at
datacontroller.io - instead of to a webdoc site.

Name the site, and give it its alias domain so the first run creates it.
The SAS docs job is left unnamed on purpose: code.datacontroller.io is its own
app whose default site is the docs, and a comment now says so.
2026-10-28 11:49:25 +00:00
dc 0011f7e933 fix(embed): log a parse-time mapping problem once
Build / Build-and-ng-test (pull_request) Successful in 5m0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m41s
Build / Build-and-test-development (pull_request) Successful in 31m54s
The parse-time problems were logged at app init and then again by the
editor's warning loop, which spreads them into the banner's text. The loop
now logs only what the instance discovers for itself (unresolved column,
unmatched key, AND-ed pair); the parse problems stay in the banner and keep
their single init-time console line.

E2e case 11 pins it: one unparseable entry plus one valid-but-missing
mapping, so the loop certainly runs and logs its own line, and exactly one
line mentions the parse problem.
2026-10-28 11:13:10 +00:00
dc 4ed1e0684d test(embed): prove va_parmap case-insensitivity end to end
Build / Build-and-ng-test (pull_request) Successful in 5m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m42s
Build / Build-and-test-development (pull_request) Successful in 33m51s
2026-10-28 09:57:58 +00:00
dc 2623a3fb4b fix(embed): defer the column check until metadata loads, report AND-ed pairs
Build / Build-and-ng-test (pull_request) Successful in 5m50s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m50s
Build / Build-and-test-development (pull_request) Successful in 33m51s
Two review findings, both paths where a filter could look applied while
producing nothing.

- `unresolvedMappings` judged a mapping against an empty column list while the
  grid was still loading, so every entry was briefly reported as naming a
  column the table does not have. An empty or unset `cols` now means "not known
  yet" and returns nothing, so the banner appears only once metadata exists.
- Two parameters mapped to one column are AND-ed, so different values yield no
  rows at all, and nothing said so. `sameColumnMappings()` now reports the
  contended column and both parameters. The clauses are deliberately left as an
  intersection: turning that AND into an OR is a filter-semantics decision, not
  a diagnostic, so it is tracked separately.

Covered by unit tests for both (unloaded column list, one line per contended
column) and e2e case 10, which drives two parameters onto one column and
asserts the banner plus the empty grid.
2026-10-28 09:54:48 +00:00
dc 388f418bf4 test(embed): pin the semantics of two params on one column
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m39s
Build / Build-and-test-development (pull_request) Failing after 35m57s
Records that two parameters mapped to the same DC column are AND-ed, so the
result is the intersection: two different dates match no rows rather than
both dates' rows. Answers the 'filter one column on two dates' case from the
code rather than by assumption, and marks the single-parameter IN (...) shape
as the one that unions.
2026-10-28 09:35:36 +00:00
dc f83dc359e2 fix(embed): one parameter maps one column, and report typos on both sides
Build / Build-and-ng-test (pull_request) Successful in 5m42s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m20s
Build / Build-and-test-development (pull_request) Successful in 33m57s
Three follow-ups from review, all the same class of defect: an author error
that leaves an unfiltered grid looking like a working filter.

- A duplicate key in `va_parmap` silently kept whichever entry came last, so
  `va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER` filtered OTHER with nothing to
  show for it. One VA parameter now maps one column: the first entry wins and
  a conflicting duplicate is reported. The same mapping written twice is not a
  conflict, and two different parameters mapping to one column stay valid -
  that is how two report objects filter the same column from their own
  parameters.
- A parameter-side typo (mapping `DROPDOWN_TAB1` when the report parameter is
  `DROPDOWN_TAB_1`) produced no clause, no banner and no console line.
  VaMessagingService now records every parameter key it has seen, across
  editor reloads, and the editor reports a mapping key that VA has never sent.
  Judged only once parameters are actually arriving, because VA sends just the
  parameters its query uses - an absent key is not on its own an error, and a
  key that turns up later stops being reported.
- The deferred first load cannot check a mapped column against metadata. On
  SAS `mp_filtercheck` aborts the save, so the load falls back to unfiltered;
  the banner now covers that path too, and an e2e case seeds the pre-bootstrap
  message buffer to prove it (and to prove a valid mapping filters the very
  first getdata).
2026-10-28 09:34:02 +00:00
dc 411f6c3354 fix(embed): warn when a va_parmap entry is ignored
Build / Build-and-ng-test (pull_request) Successful in 5m48s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m9s
Build / Build-and-test-development (pull_request) Successful in 33m7s
A `va_parmap` entry that cannot be used was dropped in silence, in both
places it can fail: at parse time (no `-` separator, a missing side, or a
column that is not a plain SAS name) and at apply time (the mapped column is
not in the table being opened). Either way the parameter is skipped, which
degrades to "no filter" - a typo in the report's URL then looks exactly like
a working filter over the whole table.

Both are now reported in terms the report author can act on:

- a warning banner in the editor (VA mode) naming the parameter, the column
  and the table, and pointing at the `va_parmap` entries in the URL
- console.warn for each distinct problem, so it is findable from the report's
  console too

The parser returns `{ map, problems }` rather than a bare Map, VaFilterService
gains `unresolvedMappings()` for the entries the loaded table cannot satisfy,
and the editor merges both sets for the banner (logging once per distinct set
rather than once per VA message).

The banner is covered by two e2e cases, one asserting its painted box inside
the viewport (VA hides the app header, so it has to stand on its own) and one
covering a parse-time failure before any VA message arrives. The alert markup
keeps the message inside the alert's text flow so it wraps, instead of being
laid out beside a title and pushed off the right edge.
2026-10-28 09:08:30 +00:00
allan f042ca82ea Merge pull request 'fix(ci): publish the docs with surfer 7, and rebuild release assets on demand' (#382) from fix/ci-surfer-app-password into release/7.20
Sites / deploy (push) Successful in 3m49s
Reviewed-on: #382
2026-10-28 08:49:33 +00:00
dc 15a1f34d52 feat(embed): map VA parameters to DC columns from the URL
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m20s
Build / Build-and-test-development (pull_request) Successful in 32m46s
Two DC instances over the same table in one VA report could not filter the
same column from different parameters: parameter -> column matching is by
name, and two VA parameters cannot share a label.

The `va_parmap` URL parameter overrides that match per instance, one entry
per mapping (`va_parmap=<VA parameter>-<DC column>`, repeated for more):

  ...?embed=va&va_parmap=REGION_TAB1-REGION

The mapping is additive (unmapped parameters keep the name match), the
column is validated as a plain SAS name at parse time, and an entry only
applies where the named column exists in the table being opened - a miss
skips that parameter rather than falling back to the same-name match the
mapping exists to override. Entries split at the last hyphen, so a label
may itself contain hyphens, and the key matches the parameter label first,
then its internal name (pr###).

Parsed once at app init onto globals (like embed / vaApplyMode) so the
editor reloads a filter apply performs keep the mapping, and both filter
paths (the metadata build and the deferred initial load) honour it.

Also repairs embed-va.cy.ts: a cy.visit differing from the current URL
only in the hash is a same-document navigation, so ?embed=va never took
effect and both existing cases failed; their layout assertions also
described an earlier design (VA now opens read-only with an Edit button
and hides Filter). Cases 3 and 4 drive two mappings of the same column
through a real postMessage and assert the applied filter and row count.
2026-10-28 08:41:15 +00:00
dc 6486c80ed4 fix(ci): rebuild and upload release assets on demand
Build / Build-and-ng-test (pull_request) Successful in 5m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m33s
Build / Build-and-test-development (pull_request) Successful in 31m45s
2026-10-28 02:07:31 +00:00
dc dd482c4069 fix(ci): publish docs after the assets, with the webdoc app password 2026-10-28 02:05:10 +00:00
dc f58925b299 fix(ci): install doxygen for the SAS docs job 2026-10-28 02:02:57 +00:00
dc 25ffa71add fix(ci): publish the code documentation with surfer 7 and app passwords 2026-10-28 01:57:23 +00:00
semantic-release-bot 424b111737 chore(release): 7.19.0 [skip ci]
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)

### Bug Fixes

* address the review findings on the harness and refresh changes ([36e4e88](36e4e88eb1))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](698dea6d6b))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](0d58be975d))
* **client:** open the SAS log page with noopener ([072a133](072a133702))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](1fb4818d21))
* **cypress:** apply the licence key when the estate has none ([be8f445](be8f445ae1)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](fed43cb07e))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](f5aade5976))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](1e224763ba))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](9b3cf50b9c))
* **derived:** address five review findings ([ccd468b](ccd468b335))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](be50cb5d08))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](9914790777))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](9ef813d439))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](422f766cd4))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](7830b7bcfa))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](2d5de9a324))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](6a1128a463))
* **derived:** eight review findings ([561d7fc](561d7fc3ed))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](1863076bc9)), closes [#366](#366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](d3dec682aa))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](469229aecb))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](7fb981daa9))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](5f5d8681b9))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](5f443b0b16))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](6b5bf3e6be))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](85c89ca2a4))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](f8f65681e1))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](7efbf31ea5))
* **derived:** only the delete marker is a reserved name ([af31c36](af31c36b67))
* **derived:** open the statements file with a long record length ([ee31782](ee31782fbf))
* **derived:** pair the filterds %else with %do/%end ([88a38db](88a38db877))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](27ddcfb575))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](7799a300df))
* **derived:** reject a dependency on a row this submission deletes ([8806504](8806504967))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](9a047ad42e))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](e0da9f1098))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](688550a28d))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](1b5c4b3ecd))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](4d9ad38df1))
* **editor:** a blank truncation note is not a warning ([f182179](f182179403))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](04ce1861e7))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](4bcee706ff))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](45dce7b11b))
* **getdata:** clear the globals it declares before use ([359b570](359b570b85))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](059cdf176a))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](661ee19cfa))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](355c46a7b6))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](d626cb8f2d))
* **mocks:** address the audit-pack review findings ([489819b](489819b051))
* **mocks:** carry the zip flags into the central directory ([c4021a4](c4021a42ff))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](62f55db170))
* **mocks:** copy the physical column list before deriving ([f411b7d](f411b7d269))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](60ec470404))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](74060c0585))
* **mocks:** drop derived variables before staging ([73d25b1](73d25b1220))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](347dbb53b0))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](ee3c8787ef))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](6cad23fce9))
* **mocks:** search the physical columns only in the viewer ([293022d](293022d46b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](fb36379520))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](9c294c0ab5))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](0a52d20c5f))
* **mpe_execute:** fail fast when the child left no status ([a036386](a03638667f))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](220408538b))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](f2b1428ec3))
* name the startupservice launcher button "Show log" ([3445107](3445107d9a))
* offer the launcher on the globvars failure modal too ([ad2da45](ad2da45389))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](46bacdd6d9))
* **refresh:** one audit folder per refresh call ([56b6620](56b6620ef4))
* **sites:** round-trip the committed licence before publishing ([4d462d3](4d462d310f))
* **test:** getdata test ([da30266](da30266278))
* **tests:** enable closeouts logging ([fdcd8fa](fdcd8fa4b3))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](033ffd3025))
* **ui:** one error modal at a time ([81ccf75](81ccf758e9))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](42355f17bc))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](337ea8e5e0))

### Features

* **dc_getlibs:** dump the library list when debug is on ([582523d](582523dc87))
* **derived:** allow a variable to shadow a physical column ([12beeff](12beeff695))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](813bce6856))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](7c9ce9a712))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](87e794f769))
* **derived:** report the dependency cycle path ([cee1a95](cee1a954ee))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](78e88a4112))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](0b010f2c65))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](aa6d50e1ea))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](33d043ea92))
* **editor:** derived variables in the EDIT screen ([b0fd354](b0fd354ecd))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](6513d09f05))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](34691dceec))
* **editor:** show the derived column label in the header tooltip ([cfe6647](cfe6647ca8))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](752d944c20))
* **mocks:** implement derived variables in the JS services ([b4fae88](b4fae88ae1))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](5d183d1350))
* run startupservice in a new window from its error modal ([40f6133](40f61331aa))
* **sites:** mint a licence for the published sites ([7d9df71](7d9df71fdb))

### Reverts

* **derived:** no shadowing, and no self reference ([5af8e6f](5af8e6fc35))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](31aa859153))
2026-10-28 01:42:51 +00:00
allan 849153d4a2 Merge pull request 'ci(release): build the frontend before deploying mocked services' (#381) from fix/ci-release-build-frontend into main
Release / Build-production-and-ng-test (push) Successful in 4m42s
Release / Build-and-test-development (push) Successful in 25m42s
Release / release (push) Failing after 6m56s
Reviewed-on: #381
2026-10-28 01:08:49 +00:00
dc bc9e0996ec ci(release): build the frontend before deploying mocked services
Build / Build-and-ng-test (pull_request) Canceled after 1m32s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m52s
2026-10-28 01:07:00 +00:00
allan 66039c91d7 Merge pull request 'feat: release 7.19 - derived variables, startupservice launcher, core v6' (#348) from release/7.19 into main
Release / Build-production-and-ng-test (push) Successful in 4m13s
Release / Build-and-test-development (push) Failing after 1m52s
Release / release (push) Skipped
Reviewed-on: #348
2026-10-28 00:52:14 +00:00
allan abb1ecbcab Merge pull request 'fix(validatefilter): drop the dead nobs assignment, keep targetds branches exclusive' (#380) from fix/validatefilter-branches into release/7.19
Sites / deploy (push) Successful in 4m2s
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m9s
Build / Build-and-test-development (pull_request) Successful in 33m26s
Reviewed-on: #380
2026-10-28 00:07:31 +00:00
dc 337ea8e5e0 fix(validatefilter): drop the dead nobs assignment, keep targetds precedence
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Review feedback on the merged fix:

- derived_nobs was assigned twice, the first one dead - the coalesced form below
  is the live value.  Removed, so there is one assignment to read.
- making the catalog test a plain %if after the derived-variables test let the
  catalog branch override it; the derived view won before and still should.  The
  catalog branch now requires derived_nobs=0, so the view keeps precedence and
  only a catalog with no derived configuration is blanked.

Two guarded if blocks, not %else %if: that construct is what the parser rejected
in this file before ('Expected DO not found'), and the first attempt at this
review point reproduced it - validatefilter and viewdata both failed to complete
again until it was replaced.

Verified: sasjs test -t new -> 64/64 (100%), TEST=0.
2026-10-28 00:05:19 +00:00
allan 709acfad73 Merge pull request 'fix(validatefilter): build targetds without macro triggers in comments' (#379) from fix/validatefilter-targetds into release/7.19
Sites / deploy (push) Successful in 3m50s
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 14m43s
Reviewed-on: #379
2026-10-27 23:46:46 +00:00
dc 42355f17bc fix(validatefilter): build targetds without macro triggers in comments
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m37s
Build / Build-and-test-development (pull_request) Successful in 33m19s
The suite failed to complete on the estate with

  ERROR: Expected %DO not found.
  ERROR: Skipping to next %END statement.

and, once the parser had lost sync, 'Apparent symbolic reference IS_FMTCAT not
resolved' - because the %let that defines it never ran.

Two causes, both in the code that decides which dataset the filter is validated
against:

- the comment explaining the catalog case contained the text '%if ...' and the
  macro processor resolves triggers even inside a comment, so the prose was
  executed as code and tripped the parser from there.  The comment now avoids
  macro triggers entirely.
- an empty value compared in a condition is a character operand, and
  %mf_nobs returns nothing when the dataset is missing.  It is wrapped in
  coalesce now, and the catalog flag is computed with %eval instead of a nested
  %if.

targetds is still blanked for a LIBREF.CATALOGNAME-FC name - which is what makes
mp_filterstore rewrite it to a cntlout dataset - and stays the plain libds
otherwise, as an empty targetds falls back to libds in the engine anyway.

The test now passes outref= and mdebug=&sasjs_mdebug rather than a hard-coded
1: the suite already owns that switch, and testinit turns it on by itself when
the run asks for _debug=2477 or 131, which is when the child's response and the
PROC HTTP trace are worth having.

Verified: sasjs test validatefilter -t new -> 3/3 (100%), six assertions, all
PASS, including the catalog and derived-variable cases.
2026-10-27 23:46:19 +00:00
dc 6ef2eec011 ci(sites): start the SASjs server with HTTP_PORT=5000
Sites / deploy (push) Successful in 3m45s
Build / Build-and-ng-test (pull_request) Successful in 5m5s
Build / Build-and-test-development (pull_request) Canceled after 5m11s
Lighthouse Checks / lighthouse (pull_request) Canceled after 7m59s
The Sites workflow was the one place that still started the server without a
listener, so it died with

  [error] Invalid environment variable(s) provided:
   - no listener is enabled
   - set HTTP_PORT and/or HTTPS_PORT (either may be used alone)

and the following step's 120s poll timed out. build.yaml, release.yaml and
lighthouse.yaml already carry the same line.
2026-10-27 23:26:26 +00:00
allan 1ef3e0a97b Merge pull request 'fix(refresh): one audit folder per refresh call' (#378) from fix/refresh-etlsource into release/7.19
Sites / deploy (push) Canceled after 2m14s
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 13m28s
Reviewed-on: #378
2026-10-27 23:12:33 +00:00
dc 56b6620ef4 fix(refresh): one audit folder per refresh call
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 18m57s
refreshcatalog.test and viewtables.test failed to complete with

  ERROR: could not create /tmp/DataController/<libref>/dc_staging/refreshcatalog:mpe_refreshcatalogs: + <stamp>

The staging folder a closeout writes its statements to is
&dc_staging_area/&etlsource, and etlsource defaulted to
'refreshcatalog:<step>:%mf_uid()'. A default is resolved when the macro is
compiled, not when it is called, so every refresh in a session reused the same
folder - the first step created it, the next one could not, and the abort took
the whole suite down. Whichever suite reached the staging area first decided
which others failed, which is why the count moved between runs.

etlsource now defaults to 0 and is set per call from
%mf_getuniquename(prefix=<step>): unique, a plain name rather than a path, and
resolved at the point of use.

Verified on the estate: refreshcatalog 4/4 (100%), viewtables 2/2 (100%).
2026-10-27 23:08:47 +00:00
allan fd272eec89 Merge pull request 'test(editor): assert the derived block marker, not a column name' (#377) from fix/derived-block-spec into release/7.19
Sites / deploy (push) Failing after 5m38s
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 1m32s
Reviewed-on: #377
2026-10-27 22:25:30 +00:00
dc c463a4e03c test(editor): assert the derived block marker, not a column name
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Canceled after 17m25s
derived-variables.cy.ts:97 expected dc-derived-first on the REGION header,
because the comment assumed REGION opens the computed block. The grid follows
MPE_DERIVED order, and that order is LINE_VALUE, REGION, REGION_LABEL - so the
marker was on LINE_VALUE and correct. The spec asserted the wrong column, and
the development pipeline has been red on it since.

Dumping the header row on a local mock estate (sasjs/server desktop mode,
RUN_TIMES=js) shows:

  6 | LINE_VALUE | dc-derived-header dc-derived-first
  7 | REGION | dc-derived-header
  8 | REGION_LABEL | dc-derived-header beforeHiddenColumn htLastVisibleHeader

and the same dump against the unmodified release/7.19 client, so this was never
a client defect.

The assertion now states the invariant instead: at least one derived header, the
first of them opens the block, and no other one carries the marker. That holds
whatever order the control table returns the variables in.
2026-10-27 22:21:52 +00:00
allan 23015ac41f Merge pull request 'fix(lint): clear the sasjs lint warnings on the files 7.19 touched' (#374) from fix/lint-719-touched-files into release/7.19
Sites / deploy (push) Failing after 5m39s
Build / Build-and-ng-test (pull_request) Successful in 5m26s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 20m56s
Reviewed-on: #374
2026-10-27 22:01:09 +00:00
dc a8f62bdfe2 docs(agents): say why webout is in ignoredMacros, and where the macros come from
Build / Build-and-ng-test (pull_request) Successful in 5m44s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m29s
Build / Build-and-test-development (pull_request) Failing after 34m27s
The previous wording named only mpeinit and mpeterm, so the webout entry read as
an unexplained addition. webout is the linter's own default - sasjs compile
injects %webout into every service - and an explicit ignoredMacros list replaces
that default, so it has to be restated. 52 service files call %webout, which is
what dropping the entry would start reporting.
2026-10-27 21:47:19 +00:00
dc 355c46a7b6 fix(lint): clear the sasjs lint warnings on the files 7.19 touched
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Lighthouse Checks / lighthouse (pull_request) Canceled after 14m27s
Build / Build-and-test-development (pull_request) Failing after 34m21s
sasjs lint (@sasjs/cli 6.2.5) reported 24 warnings across the 11 .sas files
this release touches. None of them are in those files now:

- mpeinit and mpeterm are added to ignoredMacros in sas/.sasjslint. Every
  service is compiled with a serviceInit program (sasjs/utils/
  serviceinitsas9|viya|server.sas), and all three declare both macros in
  their own header, which is how a service body's %mpeinit()/%mpeterm()
  resolves without a declaration of its own. An explicit ignoredMacros list
  replaces the default, so webout stays listed.
- Header entries nothing in the file calls are dropped: mp_binarycopy in
  getrawdata (it arrives through mp_streamfile, which calls it - the
  compiled program is unchanged), mp_jsonout in viewdata,
  mp_assertcolvals in validatefilter.test, and mx_execute in testsetup
  (every file that calls it declares it).
- Prose carrying "error"/"warning" is reworded. Where the text is log
  output, %put WARNING: is written %put %str(WARN)ING: so the line the
  log receives is unchanged.
- The server makedata service keeps its doxygen @warning tag - the tag is
  the content - so that file carries @sasjslint {"noBannedStrings": false}.
- Two comment lines in getdata are re-indented to the file's multiple of 2.

AGENTS.md records the ignoredMacros rule and the %str(WARN)ING convention so
the next warning is not silenced by reflex. Compile passes; the 93 warnings
that remain are all in files this release does not touch.
2026-10-27 21:27:54 +00:00
dc 661ee19cfa fix(getrawdata): declare mp_ds2md so the Markdown download can run
The MARKDOWN branch of the service calls %mp_ds2md(), which ships with
@sasjs/core, but the header never listed it. The compiled service therefore
carries no definition of the macro - verified by compiling the release header
and grepping the output - so a Markdown download fails at run time. The viewer
offers "Markdown (.md)" in the download list, so the path is reachable.

Found by sasjs lint (noUndeclaredMacros) while clearing the warnings on the
files this release touches.
2026-10-27 21:27:09 +00:00
73 changed files with 4835 additions and 681 deletions

No files matched your search

+2 -1
View File
@@ -1,10 +1,11 @@
# Testing
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name.
## Commands
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`.
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
+79
View File
@@ -0,0 +1,79 @@
---
name: dc-estate-testing
description: >
Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS
test suite against it. Covers the target and credential setup, the makedata seed the suite needs on
a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend
so the streamed app is the build you just made.
---
# Deploying and testing on the shared SASjs Server estate
The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong.
cd sas && npm ci # macro dependencies
cd ../client && npm ci && npm run build # frontend only - see "Deploying the frontend"
cd ../sas && npm run server # cpfavicon + sasjs cbd -t server + makedata
npx sasjs test -t server
`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `<appLoc>/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed.
## The targets
`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc` `/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair.
A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered.
To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app/<you>/dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at.
## Credentials
The CLI logs in with a registered client, not a username and password: `.env.<target>` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless.
Mint the pair from a local login, with the password read from a file rather than argv:
1. `GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header.
2. `POST /SASLogon/login` with `{"username","password"}`.
3. `GET /SASjsApi/client` on the same session - the registered client ids.
4. `POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`.
5. `POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`.
An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over.
## The makedata seed
`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.
## Reading the results
Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON.
A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete.
`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for.
## Deploying the frontend
The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists:
1. `client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`.
2. `client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines.
3. `cd client && npm ci && npm run build`.
4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc.
5. `cd ../sas && npx sasjs cbd -t <target>` - with `client/dist` present the log says `Copying web app files` and `Web app is available at <url>`.
Without step 1 the deploy prints `ERROR An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main.<hash>.js` the page names.
Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check.
## Reading a failed run
- `ERROR An error occurred while executing the request. Job location: <appLoc>/tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/<suite>.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise.
- `WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem.
- `sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree.
- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code.
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
# The Cypress specs CI runs, and the order they run in.
#
# ORDER MATTERS. These specs share one mock estate that is seeded once per job,
# so several of them depend on the state they find:
#
# - csv-limited asserts the free-tier "Locked Feature" modal, which an earlier
# spec can remove by planting a valid licence. It has always run first.
# - hook-programs asserts that an edit raises an approval row against its
# target table; it runs early, before other specs have added rows to the
# approval tables.
#
# Specs added later are appended rather than inserted, so the sequence that was
# known green keeps its shape. metanav is deliberately absent - see
# https://git.datacontroller.io/dc/dc/issues/395.
#
# download is held out for a different reason - its download button is covered by
# a fixed-position element, see
# https://git.datacontroller.io/dc/dc/issues/398.
set -euo pipefail
SPECS="csv-limited hook-programs liveness editor excel-multi-load excel csv filtering filter-panel licensing viewer-labels full-table-search viewbox stage browser-info ddl-export row-cell-limits derived-variables startupservice-launcher coltype-delete-record submitted-details embed-va"
# Specs that exist but are held out of CI, each with the reason. Listed here so
# the coverage check below can tell a deliberate exclusion from a forgotten file.
EXCLUDED="metanav download"
case "${1:-}" in
all)
# comma separated, the form `cypress run --spec` takes
for name in $SPECS; do printf 'cypress/e2e/%s.cy.ts,' "$name"; done | sed 's/,$//'
;;
coverage)
on_disk=$(cd "$2/client" && ls cypress/e2e/*.cy.ts | xargs -n1 basename | sed 's/\.cy\.ts$//' | sort)
listed=$(printf '%s\n' $SPECS $EXCLUDED | sort)
missing=$(comm -23 <(echo "$on_disk") <(echo "$listed"))
if [ -n "$missing" ]; then
echo "Specs that are neither run nor explicitly excluded:"
echo "$missing"
echo "Add them to SPECS (or EXCLUDED, with a reason) in .gitea/scripts/cypress-specs.sh"
exit 1
fi
echo "All $(echo "$on_disk" | wc -l) specs are accounted for: $(echo $SPECS | wc -w) run, $(echo $EXCLUDED | wc -w) excluded."
;;
*)
echo "usage: $0 {all|coverage <repo-root>}" >&2
exit 2
;;
esac
+42 -3
View File
@@ -75,11 +75,33 @@ jobs:
cd client
npm run test:headless
- name: Angular Tests in other timezones
# The VA filter clauses are built from local calendar dates, so a
# UTC-only run cannot see a date that slips a day west of UTC or east of
# it. Only the VA specs carry that property, so both scripts run that
# subset rather than the whole suite a second and third time.
run: |
cd client
npm run test:tz-west
npm run test:tz-east
- name: Every Cypress spec is in the CI list
# The spec list used to be spelled out inline in --spec, and five files
# quietly fell out of it - among them the VA embed suite. Fails when a
# spec on disk is not listed.
run: bash .gitea/scripts/cypress-specs.sh coverage .
- name: Production Build
run: |
cd client
npm run build
# Cypress shard 1 of 4. The setup repeats in every shard because each
# one gets its own runner and its own SASjs server; that fixed cost is what
# limits how far sharding helps. The specs it runs come from
# .gitea/scripts/cypress-specs.sh.
Build-and-test-development:
runs-on: ubuntu-latest
needs: Build-and-ng-test
@@ -182,9 +204,20 @@ jobs:
sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
- name: Prepare and run frontend and cypress
# One job, one SASjs server, one ng serve, one Chrome: four of each on one
# runner starved the app under test and every failure in that run was a
# timeout, not an assertion.
timeout-minutes: 50
run: |
cd ./client
# The spec list lives in one place, and an empty list is a broken
# workflow rather than a green run with no tests.
SPECS="$(bash ../.gitea/scripts/cypress-specs.sh all)"
if [ -z "$SPECS" ]; then
echo "the spec list resolved to nothing"
exit 1
fi
echo "specs: $SPECS"
mv ./cypress.env.example.json ./cypress.env.json
replace-in-files --regex='"username".*' --replacement='"username":"'${{ secrets.CYPRESS_USERNAME_SASJS }}'",' ./cypress.env.json
replace-in-files --regex='"password".*' --replacement='"password":"'${{ secrets.CYPRESS_PWD_SASJS }}'" ' ./cypress.env.json
@@ -199,16 +232,22 @@ jobs:
# Start frontend and run cypress
# timeout 2700: SIGTERM after 45 min so Cypress can flush video/screenshots
# before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/hook-programs.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/filter-panel.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts,cypress/e2e/browser-info.cy.ts,cypress/e2e/ddl-export.cy.ts,cypress/e2e/row-cell-limits.cy.ts,cypress/e2e/derived-variables.cy.ts,cypress/e2e/startupservice-launcher.cy.ts"
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --spec "$SPECS"
- name: Zip Cypress videos
if: always()
# Only on a failure: a green run has nothing to watch, and zipping and
# uploading every run's videos is the part of the video cost that lands
# on the clock.
if: failure()
run: |
mkdir -p ./client/cypress/videos
zip -r cypress-videos ./client/cypress/videos
- name: Add cypress videos artifacts
if: always()
if: failure()
# v3, not v4: this runner reports "GHESNotSupportedError:
# @actions/artifact v2.0.0+, upload-artifact@v4+ ... not supported on
# GHES", so the current major cannot upload here at all.
uses: actions/upload-artifact@v3
with:
name: cypress-videos.zip
+121
View File
@@ -0,0 +1,121 @@
name: Publish code documentation
run-name: Publishing typedoc and SAS docs
# The code documentation sites are published here rather than inline in the
# release job, so that they can be republished on their own - after a secret
# rotation, or for a release whose artefact upload failed (the release job
# cannot simply be re-run: semantic-release refuses once the tag exists).
on:
workflow_dispatch:
env:
NODE_VERSION: '24.5.0'
jobs:
typedoc:
runs-on: ubuntu-latest
# One deploy per site at a time: `surfer deploy` swaps the site atomically,
# so two runs racing is not corruption, but it is a wasted build and the
# last swap wins.
concurrency:
group: surfer-webdoc
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Write .npmrc file
run: |
printf '%s\n' "$NPMRC" >> client/.npmrc
echo "legacy-peer-deps=true" >> client/.npmrc
shell: bash
env:
NPMRC: ${{ secrets.NPMRC }}
- name: Install dependencies
run: |
cd client
# Decrypt and Install sheet
echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
shell: bash
env:
SHEET_PWD: ${{ secrets.SHEET_PWD }}
- name: Build typedoc
run: |
cd client
npm run compodoc:build
# Deploy tooling is pinned to an exact version: an unpinned `latest` would
# let a breaking or compromised upstream release publish straight to the
# live site. `deploy` (v7) replaces the site atomically, and it takes the
# Cloudron username plus an app password - the v6 `put --token` flow this
# replaces is gone.
#
# webdoc is a NAMED site on the app whose default site is the marketing
# site, so the name and its alias domain are both passed. Surfer picks
# the target only from `--site-name` (it travels as the Surfer-Deployment
# header); without it `deploy` replaces the app's DEFAULT site, which is
# what serves datacontroller.io - so an unnamed deploy published the
# Typedoc over the top of the marketing site instead of to webdoc.
- name: Deploy typedoc
env:
SURFER_PASSWORD: ${{ secrets.WEBDOC_DATACONTROLLER_IO }}
REF_NAME: ${{ gitea.ref_name }}
COMMIT_SHA: ${{ gitea.sha }}
run: |
set -euo pipefail
npx --yes @cloudron/surfer@7.2.0 deploy client/documentation \
--server webdoc.datacontroller.io \
--site-name webdoc \
--domain webdoc.datacontroller.io \
--username allan \
--password "$SURFER_PASSWORD" \
--message "Typedoc from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
sas-docs:
runs-on: ubuntu-latest
concurrency:
group: surfer-code-datacontroller
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Install dependencies
run: |
# `sasjs doc` shells out to doxygen
apt-get update
apt-get install -y doxygen
cd sas
npm ci
CLI_VERSION=$(node -p "require('./package.json').dependencies['@sasjs/cli']")
npm i -g "@sasjs/cli@${CLI_VERSION}"
- name: Build SAS docs
run: |
cd sas
sasjs doc
# No `--site-name` here on purpose: code.datacontroller.io is its own app
# and the SAS docs are its DEFAULT site, so naming one would create a
# separate site and leave the served copy stale. Do not copy the named
# form from the typedoc job above - that app's default site is the
# marketing site, which is why it must not be the target.
- name: Deploy SAS docs
env:
SURFER_PASSWORD: ${{ secrets.CODE_DATACONTROLLER_IO }}
REF_NAME: ${{ gitea.ref_name }}
COMMIT_SHA: ${{ gitea.sha }}
run: |
set -euo pipefail
npx --yes @cloudron/surfer@7.2.0 deploy sas/sasjsbuild/sasdocs \
--server code.datacontroller.io \
--username allan \
--password "$SURFER_PASSWORD" \
--message "SAS docs from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
+209
View File
@@ -0,0 +1,209 @@
name: Release assets
run-name: Building and uploading the release assets
# Builds the downloadable artefacts for an existing release and uploads them.
#
# The release job cuts the tag before it builds, so a failure anywhere between
# the semantic-release step and the asset upload leaves a published release with
# no files - and that job cannot simply be re-run, because semantic-release
# refuses once the tag exists. This workflow rebuilds the artefacts from the tag
# and attaches them, so the release can be completed without cutting a new one.
on:
workflow_dispatch:
inputs:
tag:
description: 'Release tag to build and upload (blank = the latest release)'
required: false
default: ''
env:
NODE_VERSION: '24.5.0'
permissions:
contents: write
jobs:
assets:
runs-on: ubuntu-latest
# Two runs uploading the same asset names would race the delete-then-upload
# below.
concurrency:
group: release-assets
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Write .npmrc file
run: |
printf '%s\n' "$NPMRC" >> client/.npmrc
echo "legacy-peer-deps=true" >> client/.npmrc
shell: bash
env:
NPMRC: ${{ secrets.NPMRC }}
- name: Install system packages
# jq parses the release JSON below, so it is installed before the tag
# resolve.
run: |
apt-get update
apt-get install -y zip jq
- name: Resolve the release tag
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
REQUESTED_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
if [ -n "${REQUESTED_TAG}" ]; then
TAG="${REQUESTED_TAG}"
# Fail here rather than after a ten-minute build if the tag is not
# cut yet or has no release.
curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${TAG}" > /dev/null
else
TAG=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest" | jq -r '.tag_name')
fi
echo "Building assets for ${TAG}"
echo "TAG=${TAG}" >> "$GITHUB_ENV"
- name: Check out the tag
run: |
set -euo pipefail
git fetch --force --tags origin "+refs/tags/${TAG}:refs/tags/${TAG}"
git checkout --force "refs/tags/${TAG}"
- name: Install the SASjs CLI the tag pins
run: |
cd sas
CLI_VERSION=$(node -p "require('./package.json').dependencies['@sasjs/cli']")
npm i -g "@sasjs/cli@${CLI_VERSION}"
- name: Frontend build
run: |
cd client
# Decrypt and Install sheet
echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
npm i webpack
npm run build
shell: bash
env:
SHEET_PWD: ${{ secrets.SHEET_PWD }}
- name: Build SAS9 EBI Release
run: |
cd sas
npm i
sasjs c -t sas9
rm -rf sasjsbuild/tests
sasjs b -t sas9
cp sasjsbuild/mysas9deploy.sas ./demostream_sas9.sas
# remove streamed component and rebuild SAS 9 services
rm -rf sasjsbuild/services/web9
rm sasjsbuild/services/clickme.sas
sasjs b -t sas9
cp sasjsbuild/mysas9deploy.sas ./sas9.sas
- name: Build SASjs Server Release
run: |
cd sas
cp sasjs/utils/favicon.ico ../client/dist/favicon.ico
sasjs c -t server
rm -rf sasjsbuild/tests
server_apploc="/Public/app/dc"
sed -i "s|apploc=\"[^\"]*\"|apploc=\"${server_apploc}\"|g" sasjsbuild/services/web/index.html
sasjs b -t server
cp sasjsbuild/server.json.zip ./sasjs_server.json.zip
- name: Build Viya Release
run: |
cd sas
sasjs c -t viya
rm -rf sasjsbuild/tests
sed -i -e 's/servertype="SASJS"/servertype="SASVIYA"/g' sasjsbuild/services/DC.html
sasjs b -t viya
cp sasjsbuild/viya.sas ./viya.sas
# compile Viya Full deploy (without web)
rm -rf sasjsbuild/services/web
rm sasjsbuild/services/DC.html
sasjs b -t viya
cp sasjsbuild/viya.sas ./viya_noweb.sas
cp sasjsbuild/viya.json ./viya_noweb.json
- name: Zip Frontend (including viya.json for full viya deploy)
run: |
cd sas
cp sasjsbuild/viya.json ../client/dist/viya.json
# Zip from *inside* dist so the archive holds the frontend files at its
# root rather than under a client/dist/ prefix (see #147).
cd ../client/dist
zip -r ../../frontend.zip .
- name: Upload assets to the release
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
RELEASE_JSON=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${TAG}")
RELEASE_ID=$(echo "$RELEASE_JSON" | jq -r '.id')
RELEASE_BODY=$(echo "$RELEASE_JSON" | jq -r '.body')
ASSETS=(frontend.zip
sas/demostream_sas9.sas
sas/viya.sas
sas/sasjs_server.json.zip
sas/sas9.sas
sas/viya_noweb.sas
sas/viya_noweb.json)
: > SHA256SUMS
for f in "${ASSETS[@]}"; do
sum=$(sha256sum "$f" | cut -d' ' -f1)
printf '%s %s\n' "$sum" "${f##*/}" >> SHA256SUMS
done
cat SHA256SUMS
# Replacing the assets means removing the ones this workflow uploads,
# and a second run must not append the verification section twice. An
# asset the release carries from another source is left alone.
OWNED=$(printf '%s\n' "${ASSETS[@]##*/}" SHA256SUMS)
echo "$RELEASE_JSON" | jq -r '.assets[]? | "\(.id) \(.name)"' |
while read -r asset_id asset_name; do
if printf '%s\n' "$OWNED" | grep -qxF -- "$asset_name"; then
echo "Removing previous ${asset_name}"
curl --fail-with-body -sS -X DELETE -H "$AUTH_HEADER" "$BASE/releases/${RELEASE_ID}/assets/${asset_id}"
else
echo "Leaving ${asset_name} in place: this workflow did not upload it"
fi
done
if echo "$RELEASE_BODY" | grep -q "## Verifying the download"; then
echo "Release body already carries the verification section"
else
VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
'$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
curl --fail-with-body -sS -X PATCH \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg body "$NEW_BODY" '{draft:false, body:$body}')" \
"$BASE/releases/$RELEASE_ID"
fi
URL="$BASE/releases/${RELEASE_ID}/assets"
for f in "${ASSETS[@]}"; do
echo "Uploading $f ..."
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
done
echo "Uploading SHA256SUMS ..."
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
+48 -19
View File
@@ -107,6 +107,16 @@ jobs:
echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
- name: Build frontend for web streaming
# The server-ci target streams client/dist as the 'clickme' web app, so
# the production build has to exist before the mocked services are
# deployed. Without it `sasjs cbd` reports
# "webSourcePath: .../client/dist ... doesn't exist", streams no
# frontend and the deploy step fails.
run: |
cd client
npm run build
- name: Setup and start SASjs server
run: |
npm i -g pm2
@@ -271,6 +281,11 @@ jobs:
semantic-release
# The tag this job cuts, for the steps that publish against it: a
# `releases/latest` lookup would pick whichever release landed last
# if two overlap.
echo "RELEASE_TAG=v$(node -p "require('./package.json').version")" >> "$GITHUB_ENV"
- name: Frontend Build
description: Must be created AFTER the release as the version (git tag) is used in the interface
run: |
@@ -331,19 +346,6 @@ jobs:
cd ../client/dist
zip -r ../../frontend.zip .
- name: Release Typedoc
run: |
cd client
npm -g install cloudron-surfer
npm run compodoc:build
surfer put --token ${{ secrets.TSDOC_TOKEN }} --server webdoc.datacontroller.io documentation/* /
- name: Release code.datacontroller.io
run: |
cd sas
sasjs doc
surfer put --token ${{ secrets.CODE_DATACONTROLLER_IO }} --server code.datacontroller.io sasjsbuild/sasdocs/* /
- name: Upload assets to release
env:
# Use the same ephemeral per-job Gitea Actions token that the
@@ -358,7 +360,7 @@ jobs:
AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
RELEASE_JSON=$(curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest")
RELEASE_JSON=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${RELEASE_TAG}")
RELEASE_ID=$(echo "$RELEASE_JSON" | jq -r '.id')
RELEASE_BODY=$(echo "$RELEASE_JSON" | jq -r '.body')
@@ -392,7 +394,7 @@ jobs:
VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
'$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
curl -k --fail-with-body -sS -X PATCH \
curl --fail-with-body -sS -X PATCH \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg body "$NEW_BODY" '{draft:false, body:$body}')" \
@@ -402,10 +404,37 @@ jobs:
URL="$BASE/releases/$RELEASE_ID/assets"
for f in "${ASSETS[@]}"; do
echo "Uploading $f ..."
curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
done
echo "Uploading SHA256SUMS ..."
curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
- name: Publish the code documentation
# Typedoc (webdoc) and the SAS docs (code) are published by docs.yaml,
# not inline here. Two reasons: the v6 `surfer put --token` flow is gone
# in Surfer 7, which takes a Cloudron username plus an app password; and
# a dispatch means a stale or failed docs deploy can be republished
# without re-running this job, which semantic-release refuses once the
# tag exists.
#
# This runs AFTER the release assets are uploaded: the documentation is
# a side channel, and a docs problem must not cost a cut release its
# artefacts.
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
TAG="${RELEASE_TAG}"
echo "Dispatching code documentation for ${TAG}"
curl --fail-with-body -sS -X POST \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
"$BASE/actions/workflows/docs.yaml/dispatches"
- name: Publish the demo site for this release
# The tag is created by this job's semantic-release step, so it carries
@@ -420,12 +449,12 @@ jobs:
AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
TAG=$(curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest" | jq -r '.tag_name')
TAG="${RELEASE_TAG}"
echo "Dispatching Sites for ${TAG}"
# the dispatch endpoint resolves a bare name as a branch; a tag needs
# the refs/tags/ prefix
curl -k --fail-with-body -sS -X POST \
curl --fail-with-body -sS -X POST \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
+1
View File
@@ -75,6 +75,7 @@ jobs:
touch .env
echo RUN_TIMES=js >> .env
echo NODE_PATH=node >> .env
echo HTTP_PORT=5000 >> .env
# The server refuses to run as root unless RUN_AS names an account
echo RUN_AS=root >> .env
nohup ./api-linux > server.log 2>&1 &
+3 -1
View File
@@ -32,6 +32,8 @@ Never consider a change complete until the relevant linters pass on the files yo
- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
`sas/.sasjslint` carries `ignoredMacros: ["webout", "mpeinit", "mpeterm"]`. `webout` is the linter's own default - `sasjs compile` injects `%webout` into every service - and an explicit list replaces that default, so it has to be restated there. `mpeinit` and `mpeterm` are listed because every service is compiled with a serviceInit program (`sasjs/utils/serviceinitsas9|viya|server.sas`) that declares both macros in its own header, so a service body may call them without declaring them. Add a macro to that list only when it reaches a file the same way - injected by the compiler, through serviceInit, testSetUp or another implicit provider - never to silence a genuinely missing declaration.
`noBannedStrings` flags the literal words `error ` and `warning ` anywhere in a file, comments included. Reword prose; where the text has to stay (a real `%put`, a doxygen `@warning` tag), keep the log output identical with `%put %str(WARN)ING:` rather than a per-file override, and use `@sasjslint {"noBannedStrings": false}` only when the word itself is the content.
## No external assets
@@ -41,7 +43,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
## Code comments and test names
+164
View File
@@ -1,3 +1,167 @@
# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)
### Bug Fixes
* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
### Features
* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
### Bug Fixes
* **ci:** install doxygen for the SAS docs job ([f58925b](https://git.datacontroller.io/dc/dc/commit/f58925b29956f1f753f0a367fdcd0afe7f3f51a0))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](https://git.datacontroller.io/dc/dc/commit/27686d3aad5e34ace59ca1b735a17cca1a4025e1))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](https://git.datacontroller.io/dc/dc/commit/888873421c3aeaf312d5ec45e3c805317c66976a))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](https://git.datacontroller.io/dc/dc/commit/dd482c4069202a0c6b9983129c19efc09bd160a7))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](https://git.datacontroller.io/dc/dc/commit/25ffa71adddba5e334f4a8c166f284d1be498fcd))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](https://git.datacontroller.io/dc/dc/commit/c5808a16eba0d29562fcc76f13ac4a33ab8da0b7))
* **ci:** rebuild and upload release assets on demand ([6486c80](https://git.datacontroller.io/dc/dc/commit/6486c80ed4f368bbf116c0ecdecc0e5e5f02075f))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](https://git.datacontroller.io/dc/dc/commit/af1813f829861df41c89b512c824e8d3ebac7a4b))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](https://git.datacontroller.io/dc/dc/commit/cf7b21a8b61ec3bf7ebe21968ccf13ade264bc64))
* **cypress:** hold download out too, and retry once in CI ([c97e886](https://git.datacontroller.io/dc/dc/commit/c97e8866db9ee404b2118d24baf53fc7058debef)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395) [#398](https://git.datacontroller.io/dc/dc/issues/398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](https://git.datacontroller.io/dc/dc/commit/c803773d3c36476d8263f01fc3b8344aa3503d44)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](https://git.datacontroller.io/dc/dc/commit/80e8911af8857fa2c417f028cd59b64a87233b9e))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](https://git.datacontroller.io/dc/dc/commit/e51953ca8a6734e1649bad68cf4ad2057d677334))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](https://git.datacontroller.io/dc/dc/commit/459ff2da371375e996839d69deb40f681c356f1d)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](https://git.datacontroller.io/dc/dc/commit/f7e753863b61d8545bd0777d80267661b3c58670))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](https://git.datacontroller.io/dc/dc/commit/7e3045a21b67053b76a81bfe836e31571ccbffbf))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](https://git.datacontroller.io/dc/dc/commit/2623a3fb4ba3bf19c00636b1717b53d5a2d54f3a))
* **embed:** log a parse-time mapping problem once ([0011f7e](https://git.datacontroller.io/dc/dc/commit/0011f7e933d85be8e33debe3a6ecb9000edaabe3))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](https://git.datacontroller.io/dc/dc/commit/541a61ba08bc7256a911c50de7078859cc46c84e))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](https://git.datacontroller.io/dc/dc/commit/e9216b08956d5599662259a2b05cc1524c458df5))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](https://git.datacontroller.io/dc/dc/commit/bd3c220af2678a2a3f8394fe3084ee03c23d0b15))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](https://git.datacontroller.io/dc/dc/commit/f83dc359e258f5d1cb9fc49f0eaab9748995d823))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](https://git.datacontroller.io/dc/dc/commit/aa91daf9f78c00cea486d0f885f8e634366bd1a0))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](https://git.datacontroller.io/dc/dc/commit/10157c11595685efdc206fe7b16a5cafa098f26d))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](https://git.datacontroller.io/dc/dc/commit/411f6c3354b7fbfec11fef3b174ae1f8066c92ba))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](https://git.datacontroller.io/dc/dc/commit/c0ea810766ebfda5ae5358ab14146fbcc07fcc8e))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](https://git.datacontroller.io/dc/dc/commit/15a1f34d5263503cac83b4fe4e52482777dc2f1f))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](https://git.datacontroller.io/dc/dc/commit/88010c3980c2ea1c39bbb024ed32d63e495e61f0))
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
### Bug Fixes
* address the review findings on the harness and refresh changes ([36e4e88](https://git.datacontroller.io/dc/dc/commit/36e4e88eb134d3b48217d6dd185ea3c03f03fba5))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](https://git.datacontroller.io/dc/dc/commit/698dea6d6b7a8635313f11acfcc7a4f39a02bc99))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](https://git.datacontroller.io/dc/dc/commit/0d58be975d8eb2d7f1ab6a648f760151392c7973))
* **client:** open the SAS log page with noopener ([072a133](https://git.datacontroller.io/dc/dc/commit/072a1337020970b5b4ff8aa31719cc5ffec7701e))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](https://git.datacontroller.io/dc/dc/commit/1fb4818d215f538a0ac51772ab1f261aeb91f7e1))
* **cypress:** apply the licence key when the estate has none ([be8f445](https://git.datacontroller.io/dc/dc/commit/be8f445ae1051e132fb6eb5a69b45264255326b5)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](https://git.datacontroller.io/dc/dc/commit/fed43cb07ef7f2afc685612a124dbc16d7618e21))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](https://git.datacontroller.io/dc/dc/commit/f5aade59760cc376b0f96eb4c892c8bca82dde14))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](https://git.datacontroller.io/dc/dc/commit/1e224763ba80500a112812a983801968e3751959))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](https://git.datacontroller.io/dc/dc/commit/9b3cf50b9c3889786605f0f5d9032d84c50f7107))
* **derived:** address five review findings ([ccd468b](https://git.datacontroller.io/dc/dc/commit/ccd468b3358c1418ee6d8e198b088a0cf23b8c24))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](https://git.datacontroller.io/dc/dc/commit/be50cb5d082e788c6ae106bc1d02e73a8c7b9646))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](https://git.datacontroller.io/dc/dc/commit/9914790777f92128c74ecce67e6c67ee48824b45))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](https://git.datacontroller.io/dc/dc/commit/9ef813d439b48e7fefd35c426e62b99e5943002a))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](https://git.datacontroller.io/dc/dc/commit/422f766cd43db4ce740e1890fd5df30790eaa7e3))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](https://git.datacontroller.io/dc/dc/commit/7830b7bcfa9806dde840ce8dd3e6a1887d0b38be))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](https://git.datacontroller.io/dc/dc/commit/2d5de9a324a07a36edc0a4f74c8fad09fde07cbf))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](https://git.datacontroller.io/dc/dc/commit/6a1128a46352f77c95dd02df9309f0984a1db2d1))
* **derived:** eight review findings ([561d7fc](https://git.datacontroller.io/dc/dc/commit/561d7fc3ed2a460835619962cf4dcb29d3dc87c7))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](https://git.datacontroller.io/dc/dc/commit/1863076bc956b4ff94bf7b85b3dde0d6e3b54d7a)), closes [#366](https://git.datacontroller.io/dc/dc/issues/366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](https://git.datacontroller.io/dc/dc/commit/d3dec682aafeb143fe983d6546337cb6490f5620))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](https://git.datacontroller.io/dc/dc/commit/469229aecbaad304a835a5f229df111c37ff7c0d))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](https://git.datacontroller.io/dc/dc/commit/7fb981daa91bce29c2d62c246185d01dcad08d6b))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](https://git.datacontroller.io/dc/dc/commit/5f5d8681b9d78280a5200309917415e135c3ce5b))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](https://git.datacontroller.io/dc/dc/commit/5f443b0b16a2abd3c11d96caa7a043911c29ef01))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](https://git.datacontroller.io/dc/dc/commit/6b5bf3e6be89b22e5a633c34b9aecbc8c7d48ee2))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](https://git.datacontroller.io/dc/dc/commit/85c89ca2a4e02112ba496ac36a222f33675544da))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](https://git.datacontroller.io/dc/dc/commit/f8f65681e1489f4ce94d9127d5e4c0c82798e960))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](https://git.datacontroller.io/dc/dc/commit/7efbf31ea5ac234bd8f924fdf89a95d101b3aff1))
* **derived:** only the delete marker is a reserved name ([af31c36](https://git.datacontroller.io/dc/dc/commit/af31c36b670ec63a2a4ba056bf6c4b28f6096e08))
* **derived:** open the statements file with a long record length ([ee31782](https://git.datacontroller.io/dc/dc/commit/ee31782fbff909b540162b1bdfca6f307208331b))
* **derived:** pair the filterds %else with %do/%end ([88a38db](https://git.datacontroller.io/dc/dc/commit/88a38db877dba9b98ccf50f0cc7242674654586b))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](https://git.datacontroller.io/dc/dc/commit/27ddcfb5759d6eb8f9880adc7c271a408e7741ad))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](https://git.datacontroller.io/dc/dc/commit/7799a300df82339683eea21e4a89ce7a6471c575))
* **derived:** reject a dependency on a row this submission deletes ([8806504](https://git.datacontroller.io/dc/dc/commit/88065049674c2cc0141eef22704b759f5b708b0f))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](https://git.datacontroller.io/dc/dc/commit/9a047ad42ec86df7d1eaf5058c0edde56016e88f))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](https://git.datacontroller.io/dc/dc/commit/e0da9f1098db63185538398ca478696cc849da4b))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](https://git.datacontroller.io/dc/dc/commit/688550a28d9794ac6b878926a0e1e2ef8f6e9335))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](https://git.datacontroller.io/dc/dc/commit/1b5c4b3ecd962a28e08fc5671b57dbf7ae0b8d9f))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](https://git.datacontroller.io/dc/dc/commit/4d9ad38df1fa89b0d5a12214d703d9c1fe5295f2))
* **editor:** a blank truncation note is not a warning ([f182179](https://git.datacontroller.io/dc/dc/commit/f18217940365f27ae388fae087e6126931e1222c))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](https://git.datacontroller.io/dc/dc/commit/04ce1861e7821063b1be42c22f92f62aed509602))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](https://git.datacontroller.io/dc/dc/commit/4bcee706ffaf1929bdd14bb2fb15111174ca3916))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](https://git.datacontroller.io/dc/dc/commit/45dce7b11b7a4399c1d0dc9c3e7a283b1cf59df1))
* **getdata:** clear the globals it declares before use ([359b570](https://git.datacontroller.io/dc/dc/commit/359b570b85aa2f66507e81ea2977243be4590f83))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](https://git.datacontroller.io/dc/dc/commit/059cdf176a7ec556b45ea1b21d880c048eba14ca))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](https://git.datacontroller.io/dc/dc/commit/661ee19cfa5f996bf01f2b11817039f38dbb9985))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](https://git.datacontroller.io/dc/dc/commit/355c46a7b64994d0146a88657dc5635593249138))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](https://git.datacontroller.io/dc/dc/commit/d626cb8f2daf94fde7ff21d924b9a2ca4968d983))
* **mocks:** address the audit-pack review findings ([489819b](https://git.datacontroller.io/dc/dc/commit/489819b05111323ae1e34edda22a5446cdbc2d87))
* **mocks:** carry the zip flags into the central directory ([c4021a4](https://git.datacontroller.io/dc/dc/commit/c4021a42ffaed2af80c53aaa1455249190b339b8))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](https://git.datacontroller.io/dc/dc/commit/62f55db170185900063f7a431cd8f375ebabf930))
* **mocks:** copy the physical column list before deriving ([f411b7d](https://git.datacontroller.io/dc/dc/commit/f411b7d269381e3f7cab2d85b6d8f7d7d50ea4d1))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](https://git.datacontroller.io/dc/dc/commit/60ec470404e5c279b58ac6a01768dfc83413b197))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](https://git.datacontroller.io/dc/dc/commit/74060c0585fb8b217cb181ea747ad318f5238868))
* **mocks:** drop derived variables before staging ([73d25b1](https://git.datacontroller.io/dc/dc/commit/73d25b1220d574ef2547fbd9e6e3cf1ceda375ee))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](https://git.datacontroller.io/dc/dc/commit/347dbb53b01c47bbd562de14d5ccbae7d96407c6))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](https://git.datacontroller.io/dc/dc/commit/ee3c8787efef9cd716bdd2bf80f6cb6751461db9))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](https://git.datacontroller.io/dc/dc/commit/6cad23fce9ac0cf64c247c40f3b7eac875fd4cfb))
* **mocks:** search the physical columns only in the viewer ([293022d](https://git.datacontroller.io/dc/dc/commit/293022d46b27e096fd514cdda49f688612df2f5b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](https://git.datacontroller.io/dc/dc/commit/fb36379520028baaf4bc3f08131bfc88deab3ac3))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](https://git.datacontroller.io/dc/dc/commit/9c294c0ab5ca8fe30a235553a298c1c68a61442f))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](https://git.datacontroller.io/dc/dc/commit/0a52d20c5fc27b4adde5fc8ed6f39a58c48f62ff))
* **mpe_execute:** fail fast when the child left no status ([a036386](https://git.datacontroller.io/dc/dc/commit/a03638667f1968c5909f496d954eeaa13f316763))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](https://git.datacontroller.io/dc/dc/commit/220408538b26016bda45e48d619cdbc3e1cbcc59))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](https://git.datacontroller.io/dc/dc/commit/f2b1428ec3099b2b6e3c6da6627530ef81fab201))
* name the startupservice launcher button "Show log" ([3445107](https://git.datacontroller.io/dc/dc/commit/3445107d9a91642b67e114bcfcd81c3ce47f3f0f))
* offer the launcher on the globvars failure modal too ([ad2da45](https://git.datacontroller.io/dc/dc/commit/ad2da453896a51f2cc7e49cebb6c84f7b47d7395))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](https://git.datacontroller.io/dc/dc/commit/46bacdd6d9ea814bcd02a1ff8aa3f447baf02f64))
* **refresh:** one audit folder per refresh call ([56b6620](https://git.datacontroller.io/dc/dc/commit/56b6620ef402b288ce78a2a77662b651e310753e))
* **sites:** round-trip the committed licence before publishing ([4d462d3](https://git.datacontroller.io/dc/dc/commit/4d462d310f7df23e5d5229cb0702721b7ad41bbd))
* **test:** getdata test ([da30266](https://git.datacontroller.io/dc/dc/commit/da30266278c4fe27a72810cd5ed647cf3bd823af))
* **tests:** enable closeouts logging ([fdcd8fa](https://git.datacontroller.io/dc/dc/commit/fdcd8fa4b3e7a7a119c662974d70dc9eea80bdae))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](https://git.datacontroller.io/dc/dc/commit/033ffd302502c7530fc44a314364f5cdb3be6ad2))
* **ui:** one error modal at a time ([81ccf75](https://git.datacontroller.io/dc/dc/commit/81ccf758e9dab0c7632ab50bf469e88b8c4c4420))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](https://git.datacontroller.io/dc/dc/commit/42355f17bc07bf4d44ba858984d83b9ed72cc0ba))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](https://git.datacontroller.io/dc/dc/commit/337ea8e5e021c6f2451fa3ad8396148744118fc9))
### Features
* **dc_getlibs:** dump the library list when debug is on ([582523d](https://git.datacontroller.io/dc/dc/commit/582523dc87a253b71824359224070ccb4746a386))
* **derived:** allow a variable to shadow a physical column ([12beeff](https://git.datacontroller.io/dc/dc/commit/12beeff69518376a042fbbc58332acc40d89d132))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](https://git.datacontroller.io/dc/dc/commit/813bce685691ad252043eeab31eb5460de81b756))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](https://git.datacontroller.io/dc/dc/commit/7c9ce9a712137e9825a5f2ff52796d765423147b))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](https://git.datacontroller.io/dc/dc/commit/87e794f769fe420f214f43b141f0b41cf88eefee))
* **derived:** report the dependency cycle path ([cee1a95](https://git.datacontroller.io/dc/dc/commit/cee1a954ee97e17f297b7cf43ef0e5cfc6e393a6))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](https://git.datacontroller.io/dc/dc/commit/78e88a41128b047c0276ccf647f128229152604c))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](https://git.datacontroller.io/dc/dc/commit/0b010f2c65ddada70d62ea83c706943c6c45756f))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](https://git.datacontroller.io/dc/dc/commit/aa6d50e1ea5fc2fe6be8aa9fa859b92d0a1061c5))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](https://git.datacontroller.io/dc/dc/commit/33d043ea92f007dca70c95d5363329d4447eeba9))
* **editor:** derived variables in the EDIT screen ([b0fd354](https://git.datacontroller.io/dc/dc/commit/b0fd354ecdbd7bf29196ed983ff3cce691affeed))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](https://git.datacontroller.io/dc/dc/commit/6513d09f059de15c5304f99b3f8d898d46ea1be7))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](https://git.datacontroller.io/dc/dc/commit/34691dceecb63f952b23ad8acc5bc93753c6cfee))
* **editor:** show the derived column label in the header tooltip ([cfe6647](https://git.datacontroller.io/dc/dc/commit/cfe6647ca870c45a3d8abc2753ce7f070e3fdf4a))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](https://git.datacontroller.io/dc/dc/commit/752d944c2091ef6d2624a15ab2e2e7be35608798))
* **mocks:** implement derived variables in the JS services ([b4fae88](https://git.datacontroller.io/dc/dc/commit/b4fae88ae13658607e4209449577cd42849d3a99))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](https://git.datacontroller.io/dc/dc/commit/5d183d135085ea023f32e5b1d4992d997394a8fb))
* run startupservice in a new window from its error modal ([40f6133](https://git.datacontroller.io/dc/dc/commit/40f61331aad6bc8a52c551b924f0d1f1e6426cee))
* **sites:** mint a licence for the published sites ([7d9df71](https://git.datacontroller.io/dc/dc/commit/7d9df71fdb0ab265e745ed3b708038983e855a87))
### Reverts
* **derived:** no shadowing, and no self reference ([5af8e6f](https://git.datacontroller.io/dc/dc/commit/5af8e6fc359221586af74a685159dfd75a40ec45))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](https://git.datacontroller.io/dc/dc/commit/31aa8591538409fc1a3ef97d7474120094ce202d))
# [7.18.0](https://git.datacontroller.io/dc/dc/compare/v7.17.0...v7.18.0) (2026-10-19)
+1 -1
View File
@@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output
- `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
- `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).
## Roles
+22
View File
@@ -1,3 +1,4 @@
import fs from 'fs'
import { defineConfig } from 'cypress'
export default defineConfig({
@@ -34,7 +35,28 @@ export default defineConfig({
e2e: {
video: true,
// One retry in a CI run, none interactively. `editor.cy.ts` test 28
// intermittently fails on "app-loading ... continuously found" - a slow load,
// not an assertion - and a single retry costs a few seconds where a red
// pipeline costs forty minutes. Failures that are real still fail twice, and
// the retried attempt's output is in the run log.
retries: { runMode: 1, openMode: 0 },
setupNodeEvents(on, config) {
// `download.cy.ts` clears the downloads folder before each test through
// `cy.task('deleteFolder')`. That task was registered in
// cypress/plugins/index.js, which Cypress 10+ never loads unless it is
// required from here - so the task had no handler and the spec failed on
// its first hook.
on('task', {
deleteFolder(folderName: string) {
return new Promise((resolve, reject) => {
fs.rm(folderName, { recursive: true, force: true }, (err) =>
err ? reject(err) : resolve(null)
)
})
}
})
// Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
// render deterministically regardless of the runner's system locale.
on('before:browser:launch', (browser, launchOptions) => {
+23 -2
View File
@@ -93,8 +93,29 @@ context('derived variables: ', function () {
'not.have.class',
'dc-derived-cell'
)
// REGION is the first derived column, so it opens the computed block
headerCell('REGION').should('have.class', 'dc-derived-first')
// The computed block opens at whichever derived column comes first in the
// grid - MPE_DERIVED order, so LINE_VALUE here, not REGION. Assert the
// invariant rather than a name: exactly one derived header carries the
// marker, and it is the first one.
cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
const derived = [...$ths].filter((th) =>
th.classList.contains('dc-derived-header')
)
expect(derived.length, 'derived headers').to.be.greaterThan(0)
expect(
derived[0].classList.contains('dc-derived-first'),
'the first derived header opens the block'
).to.be.true
derived
.slice(1)
.forEach(
(th) =>
expect(
th.classList.contains('dc-derived-first'),
'only the first derived header carries the marker'
).to.be.false
)
})
// the hover text carries the label, which is what says what the column is:
// the configured VAR_LABEL where there is one, otherwise the generated
+14 -3
View File
@@ -17,9 +17,20 @@ context('download files test: ', function () {
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
cy.get('input.username').type(username)
cy.get('input.password').type(password)
cy.get('.login-group button').click()
// beforeAll already logged in. Typing into the login form unconditionally
// failed with "the center of this element is hidden from view" whenever the
// session was live and the form was not on screen, so only fill it when it
// is actually shown.
cy.get('body').then(($body) => {
const usernameInput = $body.find('input.username')[0]
if (usernameInput && !Cypress.dom.isHidden(usernameInput)) {
cy.get('input.username').type(username)
cy.get('input.password').type(password)
cy.get('.login-group button').click()
}
})
visitPage('home')
})
+437 -17
View File
@@ -6,44 +6,90 @@ const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
// SAS Visual Analytics data-driven content embed mode (`?embed=va`):
// - chrome is hidden (same as embed=true),
// - the editor opens in edit mode immediately (not read-only),
// - the only action button is a single Submit below the grid (stubbed in v1),
// - the top button bar and Add Record are hidden.
// - the editor opens READ-ONLY (the user clicks Edit, as in normal use),
// - the Filter button is hidden - VA filters replace the filter builder,
// - the Add Row / Add Record buttons are hidden.
//
// The detailed PK+label data-merge logic is covered by the VaMessagingService
// unit spec; here we only assert the deterministic UI layout and that pushing a
// VA postMessage does not break the page.
// unit spec; here we assert the deterministic UI layout, that pushing a VA
// postMessage does not break the page, and (case 3) that a `va_parmap` URL
// mapping filters the column it names end to end.
/**
* Assert on the row count the editor shows. The assertion runs inside a Cy
* `should`, so it retries until the grid has loaded: reading the text once would
* capture the count from before a filter is applied.
*/
const parseRows = (text: string): number => {
const match = /\(([\d,]+) rows?/.exec(text)
return match ? Number(match[1].replace(/,/g, '')) : NaN
}
const expectRows = (check: (rows: number) => void) =>
cy.get('h4', { timeout: longerCommandTimeout }).should(($h4) => {
check(parseRows($h4.text()))
})
/** Assert the grid shrank, without pinning a fixture-specific number. */
const expectFewerRowsThan = (baseline: number) =>
expectRows((rows) => {
expect(rows, 'filtered row count').to.be.lessThan(baseline)
})
/** Read the unfiltered row count from the app itself, retrying until it loads. */
const readRowCount = (): Cypress.Chainable<number> =>
expectRows((rows) => {
expect(rows, 'row count').to.be.greaterThan(0)
})
.get('h4')
.invoke('text')
.then(parseRows)
context('embed=va (VA data-driven content) tests: ', function () {
// How many rows the table has unfiltered. Captured once, from the app itself.
let unfilteredRows = 0
this.beforeAll(() => {
cy.visit(`${hostUrl}/SASLogon/logout`)
cy.loginAndUpdateValidKey()
// Baseline for the relative row-count assertions: open the table with no VA
// message and record what the editor itself reports.
cy.visit(`${hostUrl}${appLocation}/#/editor/DC_JSLIB.MPE_X_TEST`, {
timeout: 60000
})
readRowCount().then((count) => {
unfilteredRows = count
})
})
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
// A cy.visit that differs from the current URL only in the hash is a
// same-document navigation, so the app is not re-bootstrapped. Reload so
// each test starts from a fresh document (the previous test may have left
// ?embed=va in the URL, which the app reads once at init).
cy.reload()
visitPage('home')
})
it('1 | opens the editor in VA mode: chrome hidden, editable, single bottom Submit', () => {
it('1 | opens the editor in VA mode: chrome hidden, read-only, no Filter button', () => {
openEditorInVaMode('mpe_x_test', () => {
// Chrome hidden (same as embed=true)
cy.get('header.app-header').should('not.exist')
// Grid present and in edit mode (read-only Filter/Edit/Upload bar absent)
// Grid present, read-only: the user enters edit mode with Edit, as usual
cy.get('#hotTable', { timeout: longerCommandTimeout }).should('exist')
cy.get('.btnCtrl button').contains('Edit').should('not.exist')
cy.get('.btnCtrl button').contains('Edit').should('exist')
// The filter builder is replaced by VA filters
cy.get('.btnCtrl button').contains('Filter').should('not.exist')
// Top edit-mode buttons hidden in VA mode
// Not in edit mode, so no edit-mode controls
cy.get('.btnCtrl button.btn-outline-danger').should('not.exist') // Cancel
cy.get('.btnCtrl').contains('Add Row').should('not.exist')
cy.get('.btnCtrl').contains('Submit').should('not.exist')
// Bottom Add Record hidden
// Bottom Add Record hidden in VA mode
cy.contains('button', 'Add Record').should('not.exist')
// Exactly one Submit button, below the grid, disabled in v1
cy.get('button').filter(':contains("Submit")').should('have.length', 1)
cy.get('button').contains('Submit').should('be.disabled')
})
})
@@ -69,14 +115,382 @@ context('embed=va (VA data-driven content) tests: ', function () {
// Editor still loads in VA mode after the message (chrome hidden, grid up)
cy.get('#hotTable', { timeout: longerCommandTimeout }).should('exist')
cy.get('header.app-header').should('not.exist')
cy.get('button').contains('Submit').should('exist')
cy.get('.btnCtrl button').contains('Edit').should('exist')
})
})
it('3 | va_parmap filters a column from a parameter that is not its name (tab 1)', () => {
// Two DC instances over one table (one per report tab), each filtering the
// same DC column from its own report parameter. Without the URL mapping the
// parameter label is not a column name, so nothing would filter at all. The
// mapping here is written in lower case against an upper-case label and
// column, which pins the case-insensitivity of both sides end to end.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
// The applied filter names the MAPPED column, and the grid shrinks to
// the rows matching that value (Option 2 occurs twice in this table).
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
'SOME_DROPDOWN'
)
cy.get('.infoBar-text').should('contain.text', "'Option 2'")
expectFewerRowsThan(unfilteredRows)
},
'va_parmap=dropdown_tab1-some_dropdown'
)
})
it('4 | the other tab maps its own parameter to the same column', () => {
// Same table, same column, a different parameter label and value (Option 1
// occurs four times) - the per-instance URL mapping is what keeps the two
// tabs independent.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB2',
dataType: 'string',
value: 'Option 1'
})
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
'SOME_DROPDOWN'
)
cy.get('.infoBar-text').should('contain.text', "'Option 1'")
expectFewerRowsThan(unfilteredRows)
},
'va_parmap=DROPDOWN_TAB2-SOME_DROPDOWN'
)
})
it('5 | warns in the grid when a mapping names a column the table lacks', () => {
// Written by hand in the report's URL, so a typo must be visible to the
// report author: the parameter is skipped, and the reason is shown rather
// than leaving an unfiltered grid that looks like a working filter.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'VA parameter mapping problem'
)
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
cy.get('.va-parmap-warning').should('contain.text', 'NOT_A_COLUMN')
// VA mode hides the app header, so the banner must paint inside the
// viewport on its own - assert the real box, not just its presence.
cy.get('.va-parmap-warning').should(($alert) => {
const rect = $alert[0].getBoundingClientRect()
expect(rect.width, 'banner width').to.be.greaterThan(100)
expect(rect.height, 'banner height').to.be.greaterThan(20)
expect(rect.top, 'banner top').to.be.at.least(0)
expect(rect.bottom, 'banner bottom').to.be.at.most(
Cypress.config('viewportHeight')
)
})
cy.screenshot('va-parmap-banner', { capture: 'viewport' })
// Nothing was filtered: the row count matches the unfiltered baseline.
expectRows((rows) => expect(rows).to.equal(unfilteredRows))
},
'va_parmap=DROPDOWN_TAB1-NOT_A_COLUMN'
)
})
it('6 | warns straight away when a mapping entry cannot be parsed', () => {
// No separator, so the entry never becomes a mapping - reported before any
// VA message arrives.
openEditorInVaMode(
'mpe_x_test',
() => {
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'it needs a "-" between the VA parameter and the column name'
)
},
'va_parmap=DROPDOWN_TAB1'
)
})
it('7 | warns when a mapping names a parameter VA is not sending', () => {
// Parameter-side typo: the mapping never applies, so without the banner this
// is indistinguishable from a working filter over every row.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
// One message only tells us this object's parameter set is still being
// discovered; the warning waits for a message that introduces no new name.
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'not.exist'
)
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'names VA parameter "dropdown_tab_1", which VA is not sending'
)
expectRows((rows) => expect(rows).to.equal(unfilteredRows))
},
'va_parmap=DROPDOWN_TAB_1-SOME_DROPDOWN'
)
})
it('8 | applies a mapping from the buffered message on the very first load', () => {
// VA posts once on iframe load, before the app bootstraps, so the first
// getdata is already filtered from that buffered message - with no column
// metadata available, the mapping is what names the column.
visitWithBufferedMessage(
'va_parmap=DROPDOWN_TAB1-SOME_DROPDOWN',
{
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
},
() => {
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
"SOME_DROPDOWN = 'Option 2'"
)
expectFewerRowsThan(unfilteredRows)
cy.url().should('match', /\/editor\/[^/]+\/\d+/)
}
)
})
it('9 | a bad mapping on the first load still ends at the banner', () => {
// The deferred first load cannot check the column against metadata, so its
// saveQuery may fail and fall back to an unfiltered load - the banner is
// what stops that reading as a working filter.
visitWithBufferedMessage(
'va_parmap=DROPDOWN_TAB1-NOT_A_COLUMN',
{
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
},
() => {
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'NOT_A_COLUMN'
)
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
// Deliberately no row-count assertion: this path cannot validate the
// column (no metadata yet), so the outcome depends on the platform - on
// SAS, mp_filtercheck aborts the save and the load falls back to
// unfiltered; the JS mock stores the filter and returns 0 rows. The
// banner is the part that must hold either way, because both outcomes
// are otherwise indistinguishable from a working filter.
}
)
})
it('10 | warns when two parameters filter the same column (an empty grid)', () => {
// Clauses are AND-ed, so two different values on one column produce no rows
// at all - the last silent-empty-grid path. Reported rather than rewritten,
// so the author decides whether to use one parameter carrying both values.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage(
{
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 1'
},
{
name: 'pr2',
label: 'DROPDOWN_TAB2',
dataType: 'string',
value: 'Option 2'
}
)
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'combined with AND'
)
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB2')
// The consequence: rows must match both clauses, and the two values
// differ, so none can match.
expectRows((rows) => expect(rows, 'matching rows').to.equal(0))
},
'va_parmap=DROPDOWN_TAB1-SOME_DROPDOWN&va_parmap=DROPDOWN_TAB2-SOME_DROPDOWN'
)
})
it('12 | a transform adds the previous quarter to the filter', () => {
// ADD(QUARTER_START,-1) is the reported requirement: one clause carrying the
// selected date and the same point one quarter earlier. 2026-10-01 = 24380,
// 2026-07-01 = 24288 in SAS days; nothing in the fixture matches either, so
// the assertion is on the clause rather than the row count.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'SELECTED_DATE',
dataType: 'string',
value: '01OCT2026'
})
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
'SOME_DATE IN ('
)
cy.get('.infoBar-text').should('contain.text', '24380')
cy.get('.infoBar-text').should('contain.text', '24288')
cy.get('h4').should('contain.text', '(0 rows')
},
'va_parmap=SELECTED_DATE-SOME_DATE:ADD(QUARTER_START,-1)'
)
})
it('13 | warns when a transform lands on a column it cannot apply to', () => {
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'SELECTED_DATE',
dataType: 'string',
value: 'x'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'is not a date or datetime column'
)
cy.get('h4').should('contain.text', '(7 rows')
},
'va_parmap=SELECTED_DATE-SOME_CHAR:ADD(QUARTER_START,-1)'
)
})
it('11 | logs a parse-time problem once, however many messages arrive', () => {
// The parse problem is logged at app init; the editor's own loop must not log
// it again. The second entry is valid-but-missing, so the loop definitely runs
// and logs its own line - which is what makes this a real assertion.
cy.visit(
`${hostUrl}${appLocation}/index.html#/editor/DC_JSLIB.MPE_X_TEST?embed=va&va_parmap=DROPDOWN_TAB1&va_parmap=DROPDOWN_TAB2-NOT_A_COLUMN`,
{
onBeforeLoad(win: any) {
win.__warns = []
const original = win.console.warn.bind(win.console)
win.console.warn = (...args: any[]) => {
win.__warns.push(args.join(' '))
original(...args)
}
}
}
)
cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
'not.exist'
)
pushVaMessage({
name: 'pr2',
label: 'DROPDOWN_TAB2',
dataType: 'string',
value: 'Option 2'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'NOT_A_COLUMN'
)
cy.window().then((win: any) => {
const warns: string[] = win.__warns || []
const parseLines = warns.filter((w) =>
w.includes('va_parmap "DROPDOWN_TAB1"')
)
const runtimeLines = warns.filter((w) => w.includes('NOT_A_COLUMN'))
expect(parseLines.length, warns.join(' | ')).to.eq(1)
expect(runtimeLines.length, warns.join(' | ')).to.eq(1)
})
})
})
/**
* Loads the editor with a VA message already in the pre-bootstrap buffer, the
* way VA's own postMessage arrives before Angular starts. A distinct pathname is
* used so this is a real document load: a cy.visit that differs only in the hash
* is a same-document navigation and would not re-bootstrap the app.
*/
const visitWithBufferedMessage = (
extraParams: string,
parameter: any,
callback?: any
) => {
cy.visit(
`${hostUrl}${appLocation}/index.html#/editor/DC_JSLIB.MPE_X_TEST?embed=va&${extraParams}`,
{
onBeforeLoad(win: any) {
win.__vaLastMessage = {
t: Date.now(),
origin: win.location.origin,
data: {
version: '1',
resultName: 'dd1',
rowCount: 0,
availableRowCount: 0,
data: [],
columns: [],
parameters: [parameter]
}
}
}
}
)
cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
if (callback) callback()
}
/** Pushes one VA data-driven content message carrying the given parameters. */
const pushVaMessage = (...parameters: any[]) => {
cy.window().then((win) => {
win.postMessage(
{
version: '1',
resultName: 'dd1',
rowCount: 0,
availableRowCount: 0,
data: [],
columns: [],
parameters
},
'*'
)
})
}
// Opens a table from the tree (which routes to #/editor/LIB.TABLE), then
// re-visits the same route with ?embed=va so the app parses VA embed mode.
const openEditorInVaMode = (tablename: string, callback?: any) => {
// `extraParams` is appended to that query string (e.g. va_parmap entries).
const openEditorInVaMode = (
tablename: string,
callback?: any,
extraParams?: string
) => {
openTableFromTree(libraryToOpenIncludes, tablename)
cy.get('#hotTable', { timeout: longerCommandTimeout })
@@ -84,7 +498,13 @@ const openEditorInVaMode = (tablename: string, callback?: any) => {
.then(() => {
cy.url().then((url) => {
const separator = url.includes('?') ? '&' : '?'
cy.visit(`${url}${separator}embed=va`)
const extra = extraParams ? `&${extraParams}` : ''
cy.visit(`${url}${separator}embed=va${extra}`)
// A cy.visit that differs from the current URL only in the hash is a
// same-document navigation, so the app is not re-bootstrapped and the
// embed parameter (read once at app init) never takes effect. Reload to
// bootstrap it, as the VA iframe load does.
cy.reload()
cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
'not.exist'
)
+4 -5
View File
@@ -14,11 +14,10 @@ context('metanav tests: ', function () {
})
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
cy.get('input.username').type(username)
cy.get('input.password').type(password)
cy.get('.login-group button').click()
// beforeAll has already logged in. Typing into the login form here failed
// once the session existed - the form is not on screen, so Cypress reported
// "the center of this element is hidden from view" and the whole suite was
// skipped behind that hook.
visitPage('view/metadata')
})
+3 -1
View File
@@ -37,7 +37,9 @@
"compodoc:build-and-serve": "compodoc -p tsconfig.doc.json -s --name 'Data Controller Client'",
"compodoc:serve": "compodoc -s --name 'Data Controller Client'",
"lighthouse": "lhci autorun",
"ng": "ng"
"ng": "ng",
"test:tz-west": "TZ=America/New_York npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'",
"test:tz-east": "TZ=Pacific/Auckland npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'"
},
"private": true,
"dependencies": {
+18
View File
@@ -1,4 +1,5 @@
import { QueryClause } from './models/TableData'
import { VaParmapTarget } from './shared/utils/parse-va-parmap'
/**
* Filtering cache info, to be reused when filtering modal is re-open
@@ -61,6 +62,21 @@ export const globals: {
* Defaults to 'live'; toggled by the Auto-apply checkbox in the editor.
*/
vaApplyMode: 'live' | 'confirm'
/**
* Explicit VA parameter -> DC column mappings, parsed from the `va_parmap`
* URL parameter (see parseVaParmap). Empty unless configured; entries override
* VA's default same-name (parameter label -> column) resolution, so two DC
* instances over one table can filter the same column from differently-named
* VA parameters.
*/
vaParamMap: Map<string, VaParmapTarget>
/**
* `va_parmap` entries that could not be used at all (not a plain SAS column
* name, or missing a side of the `-`). Surfaced to the report author, because
* these are hand-written in the report's URL and a typo otherwise looks like
* a working, unfiltered view.
*/
vaParmapProblems: string[]
rootParam: string
dcLib: string
xlmaps: XLMapListItem[]
@@ -77,6 +93,8 @@ export const globals: {
} = {
embed: false,
vaApplyMode: 'live',
vaParamMap: new Map(),
vaParmapProblems: [],
rootParam: <string>'',
dcLib: '',
xlmaps: [],
+16
View File
@@ -12,6 +12,7 @@ import '@clr/icons'
import '@clr/icons/shapes/all-shapes'
import { globals } from './_globals'
import { parseEmbedParam } from './shared/utils/parse-embed-param'
import { parseVaParmap } from './shared/utils/parse-va-parmap'
import moment from 'moment'
import { EventService } from './services/event.service'
import { AppService } from './services/app.service'
@@ -163,6 +164,21 @@ export class AppComponent {
globals.embed = embed
this.embed = embed
}
// VA parameter -> DC column mappings for this instance, held on globals
// (like `embed` and `vaApplyMode`) so in-app navigation that rebuilds the
// editor keeps them - e.g. returning from the stage / approve pages.
if (globals.embed === 'va') {
const parmap = parseVaParmap(hashQuery ?? '')
globals.vaParamMap = parmap.map
globals.vaParmapProblems = parmap.problems
// Visible in the browser console as well as the editor banner: the entry
// is hand-written in the report's URL, so a typo has to be findable when
// someone is looking at the report rather than at DC's UI.
for (const problem of parmap.problems) {
// eslint-disable-next-line no-console
console.warn('Data Controller: ' + problem)
}
}
this.subscribeToShowAbortModal()
this.subscribeToRequestsModal()
+31 -2
View File
@@ -1,7 +1,36 @@
<main class="content-area position-relative">
<!-- An embedded instance (a report object) has no report author who can deploy:
showing the wizard there only produces a screen whose buttons cannot work,
because the deployment needs an authenticated, non-embedded session. Name
the state instead, and keep the wizard for standalone use. -->
@if (isEmbedded) {
<div class="clr-row">
<div id="not-configured" class="card">
<div class="card-header">Data Controller is not configured</div>
<div class="card-block">
<div class="card-text">
<p class="mt-0">
This Data Controller instance has not been deployed yet, so it
cannot be shown inside a report.
</p>
<p>
Deployment has to be done once, in its own browser tab, by someone
who can write to the application location:
<a [href]="deployUrl" target="_blank">{{ deployUrl }}</a>
</p>
<p>
When it is configured, reload the report and this object will show
the editor.
</p>
</div>
</div>
</div>
</div>
}
<div class="clr-row">
<!-- T&C section -->
@if (step === 0) {
@if (!isEmbedded && step === 0) {
<div id="TCS" class="card">
<div class="card-header">Terms and Conditions</div>
<div class="card-block">
@@ -35,7 +64,7 @@
</div>
<!-- T&C section end -->
@if (step > 0) {
@if (!isEmbedded && step > 0) {
@switch (true) {
@case (sasJsConfig.serverType === ServerType.SasViya) {
<div class="autodeploy-section card">
@@ -0,0 +1,229 @@
import { NO_ERRORS_SCHEMA } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { ServerType } from '@sasjs/utils/types/serverType'
import { DeployComponent } from './deploy.component'
import { standaloneUrl } from './standalone-url'
import { globals } from '../_globals'
import { AppStoreService } from '../services/app-store.service'
import { SasService } from '../services/sas.service'
/**
* An embedded instance cannot deploy itself: the deployment needs an
* authenticated, standalone session, and the person looking at a report object
* is not the person who installs the app. The component therefore withholds the
* wizard and names the state instead.
*
* These cases render the template, so they fail if the `!isEmbedded` guards in
* deploy.component.html are dropped - asserting the getter alone would not.
*/
describe('DeployComponent embed guard', () => {
let fixture: ComponentFixture<DeployComponent>
let component: DeployComponent
const originalEmbed = globals.embed
const configure = () =>
TestBed.configureTestingModule({
declarations: [DeployComponent],
providers: [
{
provide: AppStoreService,
useValue: {
getDcAdapterSettings: () => ({
appLoc: '/Public/app/dc',
dcPath: '/tmp/dc'
})
}
},
{
provide: SasService,
useValue: {
getSasjsInstance: () => ({}),
getSasjsConfig: () => ({ serverType: ServerType.Sasjs })
}
}
],
// The wizard renders child components this spec does not exercise.
schemas: [NO_ERRORS_SCHEMA]
}).compileComponents()
const render = () => {
fixture = TestBed.createComponent(DeployComponent)
component = fixture.componentInstance
fixture.detectChanges()
}
const tcsSection = () =>
fixture.nativeElement.querySelector('#TCS') as HTMLElement | null
const notConfigured = () =>
fixture.nativeElement.querySelector('#not-configured') as HTMLElement | null
const deploySection = () =>
fixture.nativeElement.querySelector(
'.autodeploy-section'
) as HTMLElement | null
afterEach(() => {
globals.embed = originalEmbed
TestBed.resetTestingModule()
})
it('offers the wizard when running in its own tab', () => {
globals.embed = false
configure()
render()
expect(tcsSection()).toBeTruthy()
expect(notConfigured()).toBeNull()
expect(component.isEmbedded).toBeFalse()
})
it('replaces the wizard with the notice for a VA embed', () => {
globals.embed = 'va'
configure()
render()
expect(tcsSection()).toBeNull()
expect(notConfigured()).toBeTruthy()
})
it('replaces the wizard with the notice for a plain embed too', () => {
globals.embed = true
configure()
render()
expect(tcsSection()).toBeNull()
expect(notConfigured()).toBeTruthy()
})
it('names the standalone URL in the embedded notice', () => {
globals.embed = 'va'
configure()
render()
const link = notConfigured()!.querySelector('a') as HTMLAnchorElement
expect(link.getAttribute('href')).toBe(component.deployUrl)
expect(link.textContent).toBe(component.deployUrl)
})
it('withholds the deploy section after the terms are accepted, when embedded', () => {
globals.embed = 'va'
configure()
render()
component.step = 1
fixture.detectChanges()
expect(deploySection()).toBeNull()
})
it('shows the deploy section after the terms are accepted, standalone', () => {
globals.embed = false
configure()
render()
component.step = 1
fixture.detectChanges()
expect(deploySection()).toBeTruthy()
})
})
describe('DeployComponent onNavigateToHome', () => {
const originalEmbed = globals.embed
const originalHref = window.location.href
afterEach(() => {
globals.embed = originalEmbed
history.replaceState(null, '', originalHref)
TestBed.resetTestingModule()
})
it('opens the standalone URL, merging an embed tab into the app URL', () => {
// Karma serves the runner with its own query, so the app URL is set here
// rather than read from window.location: the assertion has to see the
// app's query survive the embed parameters being dropped. The shape is
// the real SAS 9 one - the app's `_program` query in the search part,
// the embed switches in the hash query.
history.replaceState(
null,
'',
'/deploy-tc?_program=/Public/app/dc&_debug=131#/editor?embed=va&va_parmap=A-B'
)
const open = spyOn(window, 'open')
TestBed.configureTestingModule({
declarations: [DeployComponent],
providers: [
{
provide: AppStoreService,
useValue: { getDcAdapterSettings: () => ({ appLoc: '', dcPath: '' }) }
},
{
provide: SasService,
useValue: {
getSasjsInstance: () => ({}),
getSasjsConfig: () => ({ serverType: ServerType.Sasjs })
}
}
],
schemas: [NO_ERRORS_SCHEMA]
})
const fixture = TestBed.createComponent(DeployComponent)
fixture.componentInstance.onNavigateToHome()
const opened = open.calls.mostRecent().args[0] as string
expect(opened).toBe(
window.location.origin + '/deploy-tc?_program=/Public/app/dc&_debug=131'
)
expect(opened).not.toContain('#')
expect(opened).not.toContain('embed=')
})
})
describe('standaloneUrl', () => {
const origin = window.location.origin
it('keeps the path of a plain URL', () => {
expect(standaloneUrl(`${origin}/SASJobExecution/dc`)).toBe(
`${origin}/SASJobExecution/dc`
)
})
it('keeps a query the app needs, and drops the hash route', () => {
expect(standaloneUrl(`${origin}/path?x=1#/deploy`)).toBe(
`${origin}/path?x=1`
)
})
it('keeps `_program=`, which is how SAS 9 and Viya address the app', () => {
// The whole app URL is the query here: /SASJobExecution/?_program=... is
// the standalone app, so dropping the query would open a blank executor.
const sas9 =
`${origin}/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice` +
'&_debug=131'
expect(standaloneUrl(sas9)).toBe(sas9)
})
it('drops a bare embed query', () => {
expect(standaloneUrl(`${origin}/app?embed=va`)).toBe(`${origin}/app`)
})
it('drops the va_parmap entries along with the embed switch', () => {
expect(
standaloneUrl(
`${origin}/app?embed=va&va_parmap=A-B&va_parmap=C-D#/editor/x`
)
).toBe(`${origin}/app`)
})
it('keeps a query parameter that merely starts with va_', () => {
expect(standaloneUrl(`${origin}/app?embed=va&va_rowlimit=100`)).toBe(
`${origin}/app?va_rowlimit=100`
)
})
it('resolves a relative URL against the current origin', () => {
expect(standaloneUrl('/app/#/deploy')).toBe(`${origin}/app/`)
})
})
+19 -1
View File
@@ -4,6 +4,8 @@ import { SASjsConfig } from '@sasjs/adapter'
import { ServerType } from '@sasjs/utils/types/serverType'
import { AppStoreService } from '../services/app-store.service'
import { DcAdapterSettings } from '../models/DcAdapterSettings'
import { globals } from '../_globals'
import { standaloneUrl } from './standalone-url'
@Component({
selector: 'app-deploy',
@@ -44,6 +46,20 @@ export class DeployComponent implements OnInit {
this.setDeployDefaults()
}
/**
* True when DC is running inside a report rather than in its own tab. The
* deployment needs an authenticated, standalone session, so the wizard is not
* offered here - a report object cannot deploy the app that it is displaying.
*/
get isEmbedded(): boolean {
return !!globals.embed
}
/** The standalone URL an administrator deploys from, for the embedded notice. */
get deployUrl(): string {
return standaloneUrl(window.location.href)
}
/**
* Setting default values used for deploy request
*/
@@ -60,6 +76,8 @@ export class DeployComponent implements OnInit {
}
public onNavigateToHome() {
window.open(location.href.split('#')[0], '_blank')
// Same helper as deployUrl: splitting on '#' left the query string in
// place, so a tab opened from `?embed=va#/deploy` stayed embedded.
window.open(standaloneUrl(window.location.href), '_blank')
}
}
@@ -8,20 +8,6 @@
@if (autodeployDone) {
<p class="m-0 align-self-start">Done</p>
<hr class="w-100" />
@if (autoDeployStatus.deployServicePack !== null) {
<div class="deploy-status-row">
@if (autoDeployStatus.deployServicePack === true) {
<clr-icon
class="deploy-success"
shape="success-standard"
></clr-icon>
}
@if (!autoDeployStatus.deployServicePack === false) {
<clr-icon class="deploy-error" shape="times-circle"></clr-icon>
}
<p>Deploy SAS Jobs</p>
</div>
}
@if (autoDeployStatus.runMakeData !== null) {
<div class="deploy-status-row">
@if (autoDeployStatus.runMakeData) {
@@ -33,16 +19,13 @@
@if (autoDeployStatus.runMakeData === false) {
<clr-icon class="deploy-error" shape="times-circle"></clr-icon>
}
<p>Create database</p>
<p>Create control library</p>
</div>
}
<hr class="w-100" />
<div class="buttons">
<button (click)="navigateToHome()" class="btn btn-primary mt-15 mr-0">
@if (
autoDeployStatus.deployServicePack === false ||
autoDeployStatus.runMakeData === false
) {
@if (autoDeployStatus.runMakeData === false) {
<clr-icon class="deploy-error" shape="times-circle"></clr-icon>
}
LAUNCH
@@ -78,9 +61,9 @@
<h4 class="text-center my-15">Viya Deploy</h4>
<p class="deploy-intro">
Configure and deploy Data Controller to your Viya environment. Deployment
creates the backend SAS services and the control library used by the
application.
Configure Data Controller for your Viya environment. The frontend and services
are already in place; this runs the one-off job that creates the control
library and its configuration data, then removes itself.
</p>
<hr />
@@ -248,19 +231,8 @@
</div>
}
<!-- Keeping this for a reference in case future VIYA changes and starts allowing separate backend and frontend) -->
<!-- <clr-checkbox-wrapper>
<input
clrCheckbox
[(ngModel)]="recreateDatabase"
(click)="recreateDatabaseClicked($event)"
type="checkbox"
checked
/>
<label>Recreate database</label>
</clr-checkbox-wrapper> -->
<!-- Runs the one-off makedata job. This is the screen's only action: the
frontend and services are already deployed. -->
<hr />
<button
@@ -269,53 +241,3 @@
>
Deploy
</button>
<!-- Keeping this for a reference in case future VIYA changes and starts allowing separate backend and frontend) -->
<!-- <button
(click)="executeJson()"
class="btn-autodeploy btn btn-primary d-inline-block mr-10"
[disabled]="!jsonFile"
>
Deploy {{ !jsonFile ? '(json file is not available)' : '' }}
</button> -->
<!-- <button
(click)="uploadJsonAuto.click()"
class="btn-autodeploy btn btn-primary d-inline-block mr-10"
>
Upload different file to deploy
</button>
<input
#uploadJsonAuto
type="file"
hidden
(click)="clearUploadInput($event)"
(change)="onJsonFileChange($event)"
/> -->
<clr-modal [(clrModalOpen)]="recreateDatabaseModal" [clrModalClosable]="false">
<h3 class="modal-title">Warning</h3>
<div class="modal-body">
This action will recreate the database (if it exists). For an initial
deployment, this is expected. If this is a re-deployment, you will lose any
existing tables in <strong>{{ dcPath }}</strong
>.
</div>
<div class="modal-footer">
<button
type="button"
class="btn btn-sm btn-primary"
(click)="recreateDatabaseModal = false; recreateDatabase = false"
>
Cancel
</button>
<button
type="button"
class="btn btn-sm btn-primary"
(click)="recreateDatabaseModal = false; recreateDatabase = true"
>
Ok
</button>
</div>
</clr-modal>
@@ -0,0 +1,153 @@
import { SASjsConfig } from '@sasjs/adapter'
import { AutomaticComponent } from './automatic.component'
/**
* The Viya setup screen has one job: run `makedata` with the selections the
* form collected - the admin group, the DC path, and the compute context the
* job runs in. These tests pin that payload and the new-window URL, because a
* screen whose only action silently stopped calling the service would still
* compile, still render, and still look like it worked.
*/
describe('AutomaticComponent (Viya setup)', () => {
const build = () => {
const requests: any[][] = []
const sasJs: any = {
request: (...args: any[]) => {
requests.push(args)
return Promise.resolve({ result: [{}] })
}
}
const sasService: any = {
getSasjsConfig: () => new SASjsConfig(),
getExecutionPath: () => '/SASJobExecution/',
shouldLogin: { next: () => undefined },
viyaMakedataSuccessfull: () => Promise.resolve(true),
updateFileContent: () => Promise.resolve()
}
const component = new AutomaticComponent(
{
showAbortModal: () => undefined,
showInfoModal: () => undefined
} as any,
{ downloadFile: () => undefined } as any,
sasService,
{} as any,
{ error: () => undefined } as any,
{} as any
)
component.sasJs = sasJs
const config = {
contextName: 'Compute Reusable',
serverUrl: 'https://viya.example.com',
appLoc: '/Users/demo/dc'
} as any
component.sasJsConfig = config
// The new-window URL is built from the adapter config the component holds,
// while the request path reads the @Input one.
component.sasjsConfig = config
component.appLoc = '/Users/demo/dc'
component.dcPath = '/export/dc'
component.selectedAdminGroup = 'viyagroup18'
return { component, requests, sasService }
}
it('posts makedata with the admin group and the DC path', async () => {
const { component, requests } = build()
component.deployInNewWindow = false
await component.createDatabase()
expect(requests[0][0]).toBe('services/admin/makedata')
expect(requests[0][1]).toEqual({
fromjs: [{ ADMIN: 'viyagroup18', DCPATH: '/export/dc' }]
})
})
it('runs the job in the configured context, on the web method', async () => {
const { component, requests } = build()
component.deployInNewWindow = false
await component.createDatabase()
expect(requests[0][2]).toEqual({
useComputeApi: null,
contextName: 'Compute Reusable',
debug: true
})
})
it('prefers the context chosen in the form over the configured one', async () => {
const { component, requests } = build()
component.deployInNewWindow = false
component.computeContexts = [
{ id: 'ctx-1', name: 'Selected Context' } as any
]
component.selectedComputeContext = 'ctx-1'
await component.createDatabase()
expect(requests[0][2].contextName).toBe('Selected Context')
})
it('opens makedata in a new window with the selections as URL parameters', () => {
const { component } = build()
const opened: string[] = []
spyOn(window, 'open').and.callFake((url?: any) => {
opened.push(String(url))
return null
})
spyOn(component as any, 'pollMakedataCompletion')
component.deployInNewWindow = true
component.createDatabase()
expect(opened.length).toBe(1)
expect(opened[0]).toContain('/Users/demo/dc/services/admin/makedata')
expect(opened[0]).toContain('_contextname=Compute%20Reusable')
expect(opened[0]).toContain('admin=viyagroup18')
expect(opened[0]).toContain('dcpath=%2Fexport%2Fdc')
expect(opened[0]).toContain('_debug=131')
})
it('percent-encodes the values that need it in the new-window URL', () => {
// A context name or admin group can hold characters that would otherwise
// split the query string - an ampersand would start a new parameter and a
// space would arrive truncated.
const { component } = build()
const opened: string[] = []
spyOn(window, 'open').and.callFake((url?: any) => {
opened.push(String(url))
return null
})
component.runMakedataInNewWindow({
contextName: 'SAS Job Execution & co',
admin: 'group 1+2',
dcPath: '/export/dc data'
})
expect(opened[0]).toContain('_contextname=SAS%20Job%20Execution%20%26%20co')
expect(opened[0]).toContain('admin=group%201%2B2')
expect(opened[0]).toContain('dcpath=%2Fexport%2Fdc%20data')
// the parameter names themselves are not encoded
expect(opened[0]).toContain('&admin=group%201%2B2')
})
it('starts the job when deploy is pressed', () => {
// The screen's only button calls this; a version where it stopped doing so
// would leave the setup page unable to do anything at all.
const { component } = build()
const spy = spyOn(component, 'createDatabase').and.returnValue(
Promise.resolve() as any
)
component.runAutoDeploy()
expect(spy).toHaveBeenCalled()
})
})
@@ -51,17 +51,8 @@ export class AutomaticComponent implements OnInit {
public selectedComputeContext: string = ''
public makeDataResponse: string = ''
public jsonFile: any = null
public autodeploying: boolean = false
public autodeployDone: boolean = false
public recreateDatabaseModal: boolean = false
public isSubmittingJson: boolean = false
public isJsonSubmitted: boolean = false
/**
* Default was `false` when deploy was done with frontend and backend separately.
* Now we are using only streaming app, so we always want to recreate database (makedata)
*/
public recreateDatabase: boolean = true
public createDatabaseLoading: boolean = false
public adminGroupsLoading: boolean = false
public currentUserInfoLoading: boolean = false
@@ -86,18 +77,14 @@ export class AutomaticComponent implements OnInit {
public currentUserInfo: ViyaApiCurrentUser | null = null
public computeContexts: ComputeContextItem[] = []
/** autoDeployStatus
* This object presents the status for two steps that we have for deploy.
* `deployServicePack` - Creating services based on `viya.json`
* `runMakeData` - Running `makedata` service
* If any of them is `null` or `false` that means step failed
* and will be shown to user on deploy done modal.
/**
* Status of the one step this screen runs: `makedata`, which creates the
* control library and its configuration data and then deletes itself.
* `null` means it has not run yet, `false` that it failed.
*/
public autoDeployStatus: {
deployServicePack: any
runMakeData: any
} = {
deployServicePack: null,
runMakeData: null
}
@@ -420,61 +407,15 @@ export class AutomaticComponent implements OnInit {
}
/**
* Executes sas.json file to deploy the backend
* Method will first try to run the `auto deploy`
* If that fails the rest of the code is ignored.
* If request is successfull, method will continue to try
* to create database if checkbox is toggled on
* Runs the deployment. The control library and its configuration data are
* created by `makedata`, which self-deletes once it has run; the frontend and
* services arrived with the sasjs deploy, so there is nothing else to deploy
* from here.
*/
public async executeJson() {
this.isSubmittingJson = true
try {
let uploadJsonFile = await this.sasJs.deployServicePack(
this.jsonFile,
this.dcAdapterSettings?.appLoc,
undefined,
undefined,
true
)
this.autoDeployStatus.deployServicePack = true
this.isJsonSubmitted = true
} catch (ex: any) {
let textEx = ''
if (typeof ex.message !== 'string') {
textEx = JSON.stringify(ex).replace(/\\/gm, '')
} else {
textEx = ex.message
}
this.autoDeployStatus.deployServicePack = false
this.eventService.showInfoModal(
'Deploy error',
`Exception: \n ${textEx !== '' ? textEx : ex}`
)
this.autodeploying = false
this.autodeployDone = false
return
}
this.isSubmittingJson = false
}
public async runAutoDeploy(executeJson: boolean = false) {
public runAutoDeploy() {
if (!this.deployInNewWindow) this.autodeploying = true
if (executeJson) {
this.executeJson()
}
if (this.recreateDatabase) {
this.createDatabase()
} else {
if (!this.deployInNewWindow) this.autodeployDone = true
}
this.createDatabase()
}
/**
@@ -578,9 +519,9 @@ export class AutomaticComponent implements OnInit {
let serverUrl = this.sasjsConfig.serverUrl
let appLoc = this.sasjsConfig.appLoc
const execPath = this.sasService.getExecutionPath()
let contextname = `&_contextname=${params.contextName}`
let admin = `&admin=${params.admin}`
let dcPath = `&dcpath=${params.dcPath}`
let contextname = `&_contextname=${encodeURIComponent(params.contextName)}`
let admin = `&admin=${encodeURIComponent(params.admin)}`
let dcPath = `&dcpath=${encodeURIComponent(params.dcPath)}`
// Debug is ALWAYS enabled for makedata — it runs only once during
// deployment and the full log is needed to diagnose any issues.
let debug = '&_debug=131'
@@ -708,22 +649,6 @@ export class AutomaticComponent implements OnInit {
this.deployService.downloadFile(content, filename, extension)
}
public async onJsonFileChange(event: any) {
let file = event.target.files[0]
this.jsonFile = await this.deployService.readFile(file)
}
public recreateDatabaseClicked(event: Event) {
;(<HTMLInputElement>event.target).checked === true
? (this.recreateDatabaseModal = true)
: ''
}
public clearUploadInput(event: Event) {
this.deployService.clearUploadInput(event)
}
public openSasRequestsModal() {
this.eventService.openRequestsModal()
}
@@ -0,0 +1,88 @@
import { ServerType } from '@sasjs/utils/types/serverType'
import { SasjsConfiguratorComponent } from './sasjs-configurator.component'
/**
* The SAS9 / SASjs Server setup screen cannot deploy anything: it builds the
* one-off `makedata` URL and opens it, with the admin group and the directory
* the control library will live in. These tests pin the URL shape, because a
* wrong parameter name or a lost encode still produces a page that opens and
* merely fails inside SAS, where nobody is looking.
*/
describe('SasjsConfiguratorComponent (SAS9 / SASjs setup)', () => {
const build = (serverType: ServerType) => {
const sasService: any = {
getExecutionPath: () => '/SASStoredProcess/do',
sasjsMakedataChecking: () => Promise.resolve(false),
logout: () => undefined
}
const component = new SasjsConfiguratorComponent(
sasService,
{ getServerInfo: () => ({ subscribe: () => undefined }) } as any,
{ replaceState: () => undefined } as any
)
component.sasJsConfig = {
serverType,
serverUrl: 'https://sas9.example.com',
appLoc: '/User Folders/dc'
} as any
component.dcDirectory = '/tmp/DataController'
component.dcAdminGroup = 'DCADMINS'
const opened: string[] = []
spyOn(window, 'open').and.callFake((url?: any) => {
opened.push(String(url))
return null
})
return { component, opened, sasService }
}
it('opens makedata on SAS9 with the admin group and the directory', () => {
const { component, opened } = build(ServerType.Sas9)
component.makeData()
expect(opened.length).toBe(1)
expect(opened[0]).toContain(
'_program=/User Folders/dc/services/admin/makedata'
)
expect(opened[0]).toContain('admin=DCADMINS')
expect(opened[0]).toContain('path=%2Ftmp%2FDataController')
expect(opened[0]).toContain('_debug=131')
})
it('stays on the page origin for SAS9 rather than the configured server', () => {
const { component, opened } = build(ServerType.Sas9)
component.makeData()
// A SAS9 deployment is reached through the URL the user is already on; the
// configured serverUrl would point the browser at a host it may not reach.
expect(opened[0].startsWith(window.location.origin)).toBe(true)
expect(opened[0]).not.toContain('sas9.example.com')
})
it('keeps the directory readable for SASjs Server', () => {
const { component, opened } = build(ServerType.Sasjs)
component.makeData()
// The SASjs branch does not encode the path, so a plain directory stays as
// written; encoding it there would make makedata look for a folder with %2F
// in its name.
expect(opened[0]).toContain('path=/tmp/DataController')
})
it('checks the SASjs Server deployment before reloading the app', () => {
const { component, sasService } = build(ServerType.Sasjs)
const spy = spyOn(sasService, 'sasjsMakedataChecking').and.returnValue(
Promise.resolve(false)
)
component.makeData()
expect(spy).toHaveBeenCalled()
})
})
+29
View File
@@ -0,0 +1,29 @@
/**
* The URL a Data Controller instance runs at when it is not embedded: the same
* URL, without the parts that make it an embed - the `embed` switch, the
* `va_parmap` entries, and the hash route.
*
* The rest of the query is kept. On SAS 9 and Viya the app is addressed
* entirely through it (`/SASJobExecution/?_program=/.../startupservice`), so
* dropping the query would open a blank executor rather than the app, and any
* other parameter the deployment relies on would go the same way.
*
* Only these two parameters are embed parameters. A later parameter that
* happens to start with `va_` belongs to the app, not the embed, so it stays.
*
* Parsing with URL also handles the relative hrefs the adapter can hand back,
* which plain string splicing cannot.
*/
const EMBED_PARAMS = ['embed', 'va_parmap']
export const standaloneUrl = (href: string): string => {
const url = new URL(href, window.location.origin)
url.hash = ''
for (const key of Array.from(url.searchParams.keys())) {
if (EMBED_PARAMS.includes(key.toLowerCase())) {
url.searchParams.delete(key)
}
}
return url.toString()
}
@@ -459,6 +459,43 @@
<!--<hr>-->
<!--{{ validationDone }}-->
<!-- VA embed: something about a `va_parmap` entry in this report
object's URL needs the report author's attention - it could not be
used, it was applied but AND-ed against another parameter, or it
names a parameter VA never sends. Shown as a banner, not just a
console warning, because
the entries are written by the report author and a typo otherwise
looks like a working filter over the whole table. -->
@if (isVaEmbed && vaMappingWarnings.length) {
<clr-alert
class="va-parmap-warning"
[clrAlertType]="'warning'"
[clrAlertClosable]="false"
>
<clr-alert-item>
<div class="alert-text">
<strong>VA parameter mapping problem</strong>
@for (warning of vaMappingWarnings; track warning) {
<div>{{ warning }}</div>
}
<!-- The reader is the report author, who can only fix this in
the report: name the parameter they edit, and say what a
correct entry looks like, so the message is actionable
without the docs. -->
<div>
Fix: edit the <code>va_parmap</code> entries in the URL of
this Data Controller object in the report. An entry is
<code>parameter-column</code>, optionally
<code>:ADD(anchor,offset)</code> or
<code>:RANGE(period,offset)</code> - for example
<code>va_parmap=SEL_DATE-D_DATE:ADD(QUARTER_START,-1)</code
>.
</div>
</div>
</clr-alert-item>
</clr-alert>
}
<div class="hot-wrapper clr-flex-1">
<hot-table
#hotInstance
@@ -27,3 +27,15 @@
font-style: italic;
opacity: 0.7;
}
/* VA embed: `va_parmap` diagnostic banner. Clarity's alert lays `.alert-title`
and `.alert-text` out on one row, which pushes a long message off the right
edge; this keeps the whole message in the flow and wraps it. */
.va-parmap-warning {
width: 100%;
.alert-text {
white-space: normal;
overflow-wrap: anywhere;
}
}
+104 -19
View File
@@ -73,6 +73,7 @@ import {
} from '../shared/dc-validator/utils/autoEscapedCellTracker'
import { HyperFormula } from 'hyperformula'
import { parseLabelsParam } from '../shared/utils/parse-labels-param'
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
import { getDisplayColHeaders } from '../shared/utils/display-col-headers'
import { buildColInfoHtml } from '../shared/utils/col-info-html'
import { globals } from '../_globals'
@@ -485,6 +486,46 @@ export class EditorComponent
get isVaEmbed() {
return globals.embed === 'va'
}
/**
* Explicit VA parameter -> DC column mappings (`va_parmap`) for this instance.
* Parsed once at app init (see parseVaParmap) and held on globals so the
* editor reloads a filter apply performs keep using the same mapping.
*/
get vaParamMap(): Map<string, VaParmapTarget> {
return globals.vaParamMap
}
/** `va_parmap` entries that name a column this table does not have. */
public vaUnresolvedMappings: string[] = []
/** `va_parmap` keys that no VA message has named (a parameter-side typo). */
public vaUnmatchedMappings: string[] = []
/** Parameters in this message that filter the same column (an AND-ed pair). */
public vaSameColumnMappings: string[] = []
/** Transform mappings this message cannot apply (non-temporal column or value). */
public vaTransformProblems: string[] = []
/** The warning text last written to the console, so repeated VA messages log once. */
private vaWarningsReported = ''
/**
* Everything wrong with this instance's `va_parmap` entries, in the report
* author's terms: entries that could not be parsed at all, plus entries that
* name a column the open table does not have. Shown as a banner in the editor
* - a report developer is usually not going to read the browser console.
*/
get vaMappingWarnings(): string[] {
return [...globals.vaParmapProblems, ...this.vaRuntimeWarnings]
}
/**
* The warnings this instance discovers for itself, i.e. everything except the
* parse-time problems - those are logged once at app init (`app.component`),
* so logging them again here would double every line in the console.
*/
get vaRuntimeWarnings(): string[] {
return [
...this.vaUnresolvedMappings,
...this.vaUnmatchedMappings,
...this.vaSameColumnMappings,
...this.vaTransformProblems
]
}
/**
* VA filter mode. Live = auto-apply each VA change; confirm = stage the change
* and wait for the user to click Apply. Backed by globals (toggled by the
@@ -3602,11 +3643,20 @@ export class EditorComponent
const msg = this.vaMessaging.latestMessage()
if (!msg) return false
const clauses = this.vaFilter.buildInitialClauses(msg)
const clauses = this.vaFilter.buildInitialClauses(msg, this.vaParamMap)
if (clauses.length === 0) return false
try {
const res: any = await this.sasStoreService.saveQuery(libds, clauses)
// A `va_parmap` entry is a column the report author typed, so this
// request can legitimately fail: the table may not have that column, and
// validatefilter aborts on an unknown variable (dc/dc#404). Attempting it
// is still worth it - when the mapping is right the very first getdata is
// already filtered - but the failure must not stop the page. An abort
// modal here covers an editor that is about to load fine; the mapping
// banner names the bad entry once the real metadata arrives.
const res: any = await this.sasStoreService.saveQuery(libds, clauses, {
suppressErrorAbortModal: true
})
const id = res?.result?.[0]?.FILTER_RK
const table = res?.result?.[0]?.FILTER_TABLE
if (id === undefined || table === undefined) return false
@@ -4022,19 +4072,6 @@ export class EditorComponent
// rules must stay index-aligned with headerColumns).
this.headerColumns.push(EDIT_STATUS_COLUMN_NAME)
/**
* columnHeader is the same list, taken once it is final (Delete? renamed,
* EDIT_STATUS appended) - it is the grid's own column order, which is what
* every hook that maps a visual column index to a name needs.
*
* This has to happen HERE, before the updateSettings below installs
* afterGetColHeader / afterGetCellMeta: those fire during the render that
* updateSettings triggers, and reading a columnHeader that was still being
* assigned further down meant they saw the previous table's columns (or
* nothing). That is what put the derived-block marker on the wrong column.
*/
this.columnHeader = this.headerColumns.slice()
if (response.data.sasparams[0].DTVARS !== '') {
this.dateHeaders = response.data.sasparams[0].DTVARS.split(' ')
}
@@ -4156,9 +4193,8 @@ export class EditorComponent
this.dataSourceUnchanged = this.helperService.deepClone(this.dataSource)
this.overlayFormulaRawValuesOnUnchanged(this.dataSourceUnchanged)
// columnHeader is a copy of headerColumns taken above, once the Delete?
// rename and the EDIT_STATUS append had been applied - the hooks read it
// for grid-index-to-name mapping, so the two must not drift apart again.
// Note: this.headerColumns and this.columnHeader contains same data
// need to resolve redundancy
// default schema - includes NOTNULL defaults from DQ rules
for (let i = 0; i < this.headerColumns.length; i++) {
@@ -4517,6 +4553,9 @@ export class EditorComponent
// hot.getPlugin('manualColumnFreeze').freezeColumn(0);
this.queryText = response.data.sasparams[0].FILTER_TEXT
this.columnHeader = response.data.sasparams[0].COLHEADERS.split(',')
// First column is always used to mark records for deletion
this.columnHeader[0] = 'Delete?'
this.readOnlyFields = response.data.sasparams[0].PKCNT
hot.addHook(
@@ -5059,7 +5098,53 @@ export class EditorComponent
this.updateVaFilterText(msg)
const clauses = this.vaFilter.buildClauses(msg, labelToVar, this.cols)
// A `va_parmap` entry naming a column this table does not have is skipped by
// buildClauses. Report it (banner + console, once per distinct set) so the
// report author sees a misconfiguration instead of an unfiltered grid.
this.vaUnresolvedMappings = this.vaFilter.unresolvedMappings(
msg,
this.vaParamMap,
this.cols,
this.libds
)
// A key naming no parameter VA has ever sent is a typo on the parameter
// side: the mapping never applies. Only judged once parameters are actually
// arriving, since VA sends just the parameters its query uses.
this.vaUnmatchedMappings = this.vaFilter.unmatchedMappings(
this.vaParamMap,
this.vaMessaging.seenParamKeys,
this.vaMessaging.lastMessageParamsStable
)
// Two parameters filtering one column are AND-ed, so differing values leave
// an empty grid. Reported, not rewritten: turning that AND into an OR is a
// filter-semantics decision, taken in buildClauses and not in a diagnostic.
this.vaSameColumnMappings = this.vaFilter.sameColumnMappings(
msg,
this.vaParamMap,
this.cols
)
// A transform that cannot run here (char column, value that is not a date) is
// skipped by the clause builder, so the banner has to say why.
this.vaTransformProblems = this.vaFilter.unsupportedTransforms(
msg,
this.vaParamMap,
this.cols
)
const warnings = this.vaRuntimeWarnings.join(' | ')
if (warnings !== this.vaWarningsReported) {
this.vaWarningsReported = warnings
for (const warning of this.vaRuntimeWarnings) {
// eslint-disable-next-line no-console
console.warn('Data Controller: ' + warning)
}
}
const clauses = this.vaFilter.buildClauses(
msg,
labelToVar,
this.cols,
this.vaParamMap
)
const signature = this.vaFilter.signature(clauses)
if (signature === this.vaMessaging.filterSignature) {
-18
View File
@@ -26,22 +26,4 @@ export class DeployService {
document.body.removeChild(downloadLink)
}
}
public readFile(file: File): Promise<any> {
return new Promise((resolve, reject) => {
let JsonFileReader = new FileReader()
JsonFileReader.onload = () => {
if (JsonFileReader.result) {
resolve(JSON.parse(JsonFileReader.result.toString()))
}
}
JsonFileReader.readAsText(file)
})
}
public clearUploadInput(event: Event) {
;(<HTMLInputElement>event.target).value = ''
}
}
@@ -116,6 +116,53 @@ describe('SasStoreService', () => {
expect(seen.length).toBe(0)
})
})
describe('saveQuery', () => {
const capture = (adapterResponse: any = { result: [] }) => {
const calls: any[][] = []
const store = new SasStoreService(
{
request: (...args: any[]) => {
calls.push(args)
return Promise.resolve({ adapterResponse })
}
} as any,
{} as any,
{} as any
)
return { store, calls }
}
it('posts the filter to public/validatefilter', async () => {
const { store, calls } = capture()
await store.saveQuery('DC.MY_TABLE', [
{ VARIABLE_NM: 'A', OPERATOR_NM: '=', RAW_VALUE: "'x'" } as any
])
expect(calls[0][0]).toBe('public/validatefilter')
expect(calls[0][1].iwant[0].filter_table).toBe('DC.MY_TABLE')
expect(calls[0][1].filterquery.length).toBe(1)
})
it('passes wrapper options through, so a rejected filter can stay quiet', async () => {
// The VA deferred load attempts a filter built from a va_parmap entry the
// table may not have, and must not cover a page that loads fine with an
// abort modal (dc/dc#404).
const { store, calls } = capture()
await store.saveQuery('DC.MY_TABLE', [], {
suppressErrorAbortModal: true
})
expect(calls[0][3]).toEqual({ suppressErrorAbortModal: true })
})
it('sends no wrapper options when the caller gives none', async () => {
const { store, calls } = capture()
await store.saveQuery('DC.MY_TABLE', [])
expect(calls[0][3]).toBeUndefined()
})
})
})
/**
+20 -2
View File
@@ -8,6 +8,7 @@ import { HelperService } from './helper.service'
import { QueryClause } from '../models/TableData'
import { globals } from '../_globals'
import { FilterClause, FilterGroup, FilterQuery } from '../models/FilterQuery'
import { RequestWrapperOptions } from '../models/request-wrapper/RequestWrapperOptions'
import {
$DataFormats,
EditorsGetDataSASResponse,
@@ -395,13 +396,30 @@ export class SasStoreService {
).adapterResponse
}
public async saveQuery(libds: string, filterQuery: QueryClause[]) {
/**
* Persists a filter and returns its FILTER_RK.
*
* `wrapperOptions` reaches the request wrapper, so a caller that can live with
* a rejected filter (the VA deferred load, whose `va_parmap` entry may name a
* column the table does not have) can suppress the abort modal that would
* otherwise cover a page which is about to load perfectly well.
*/
public async saveQuery(
libds: string,
filterQuery: QueryClause[],
wrapperOptions?: RequestWrapperOptions
) {
let tables = {
iwant: [{ filter_table: libds }],
filterquery: filterQuery
}
const res = await this.sasService.request('public/validatefilter', tables)
const res = await this.sasService.request(
'public/validatefilter',
tables,
undefined,
wrapperOptions
)
this.filter.next(res)
@@ -1,4 +1,5 @@
import { VaFilterService } from './va-filter.service'
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
import { VaMessage } from './va-messaging.service'
import { HelperService } from './helper.service'
@@ -14,6 +15,7 @@ function msg(parameters: any[]): VaMessage {
describe('VaFilterService', () => {
let service: VaFilterService
let helperService: HelperService
// DC column metadata used by the metadata path. cellValidation[0] is the
// delete column and must be skipped by buildLabelToVarMap.
@@ -42,7 +44,8 @@ describe('VaFilterService', () => {
]
beforeEach(() => {
service = new VaFilterService(new HelperService({} as any))
helperService = new HelperService({} as any)
service = new VaFilterService(helperService)
})
const labelToVar = () =>
@@ -128,6 +131,41 @@ describe('VaFilterService', () => {
expect(build([{ label: 'SOME_DATE', value: '2025/03/31' }])).toHaveSize(1)
})
it('parses word dates, in either order and with an optional comma', () => {
const expected = "'01JAN2026'd"
// The reported shape: a date parameter displayed as WORDDATE.
expect(
build([{ label: 'SOME_DATE', value: 'January 1, 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: 'Jan 1, 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: 'Jan. 1, 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: '1 January 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: '1-Jan-2026' }])[0].RAW_VALUE
).toBe(expected)
})
it('matches a spelled-out month case-insensitively, and keeps DATE9 working', () => {
expect(
build([{ label: 'SOME_DATE', value: 'DECEMBER 31, 2025' }])[0].RAW_VALUE
).toBe("'31DEC2025'd")
expect(
build([{ label: 'SOME_DATE', value: '01AUG1962' }])[0].RAW_VALUE
).toBe("'01AUG1962'd")
})
it('still drops a month-name value that has no day', () => {
// A month and year alone is not a date - accepted shapes always carry a
// day, so this must not silently become the first of the month.
expect(build([{ label: 'SOME_DATE', value: 'January 2026' }])).toEqual([])
})
it('resolves ambiguous slash dates day-first (never US), disambiguating by value', () => {
const iso = build([{ label: 'SOME_DATE', value: '2025-03-31' }])[0]
.RAW_VALUE
@@ -199,7 +237,694 @@ describe('VaFilterService', () => {
})
})
describe('buildClauses with an explicit va_parmap', () => {
const build = (parameters: any[], paramMap: Map<string, VaParmapTarget>) =>
service.buildClauses(msg(parameters), labelToVar(), cols, paramMap)
it('overrides the same-name match, including the value formatting', () => {
// The VA label IS a DC column here (SOME_CHAR), but the mapping points at
// a different one - the numeric column - so the value is formatted numeric.
const map = new Map([['some_char', { column: 'SOME_NUM' }]])
expect(build([{ label: 'SOME_CHAR', value: '10' }], map)).toEqual([
{
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: 'SOME_NUM',
OPERATOR_NM: '=',
RAW_VALUE: '10'
}
])
})
it('lets two differently-named params filter the same column', () => {
// The multi-tab case: one DC instance per report tab, each with its own
// VA parameter, both mapping to the same DC column.
const map = new Map([
['region_tab1', { column: 'SOME_CHAR' }],
['region_tab2', { column: 'SOME_CHAR' }]
])
const clauses = build(
[
{ label: 'REGION_TAB1', value: 'north' },
{ label: 'REGION_TAB2', value: 'south' }
],
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['SOME_CHAR', "'north'"],
['SOME_CHAR', "'south'"]
])
})
it('matches the parameter label only, not its internal name', () => {
// The internal name is the report's own bookkeeping. Matching it as well
// meant one parameter answered to two keys, so an entry written against the
// name and one against the label silently fought over it.
const map = new Map([['pr273', { column: 'SOME_NUM' }]])
expect(
build([{ name: 'pr273', label: 'Tab_1_Value', value: '7' }], map)
).toEqual([])
// The label still matches, case-insensitively.
expect(
build(
[{ name: 'pr273', label: 'Tab_1_Value', value: 'x' }],
new Map([['tab_1_value', { column: 'SOME_CHAR' }]])
)[0].VARIABLE_NM
).toBe('SOME_CHAR')
})
it('uses the column case from the loaded metadata', () => {
const map = new Map([['region_tab1', { column: 'some_char' }]])
expect(
build([{ label: 'REGION_TAB1', value: 'x' }], map)[0].VARIABLE_NM
).toBe('SOME_CHAR')
})
it('leaves unmapped params on the same-name path', () => {
const map = new Map([['region_tab1', { column: 'SOME_NUM' }]])
const clauses = build(
[
{ label: 'REGION_TAB1', value: '10' },
{ label: 'SOME_CHAR', value: 'x' }
],
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['SOME_NUM', '10'],
['SOME_CHAR', "'x'"]
])
})
it('skips a mapping whose column is not in this table (no fallback)', () => {
// Falling back to the same-name match would silently apply the filter the
// mapping exists to override, so the param is dropped instead.
const map = new Map([['some_char', { column: 'NOT_IN_THIS_TABLE' }]])
expect(build([{ label: 'SOME_CHAR', value: 'x' }], map)).toEqual([])
})
it('ANDs two parameters that map to the same column - it does not union them', () => {
// What a report gets when one column's two values arrive in two
// parameters: every clause is AND-ed, so the result is the INTERSECTION.
// Two different dates therefore match no rows at all rather than both
// dates' rows - a single parameter carrying both values is the shape that
// ORs, via `IN (...)`.
const map = new Map([
['param_a', { column: 'SOME_DATE' }],
['param_b', { column: 'SOME_DATE' }]
])
const one = build([{ label: 'PARAM_A', value: '01JUL1960' }], map)[0]
const other = build([{ label: 'PARAM_B', value: '01JAN1961' }], map)[0]
const clauses = build(
[
{ label: 'PARAM_A', value: '01JUL1960' },
{ label: 'PARAM_B', value: '01JAN1961' }
],
map
)
expect(one.RAW_VALUE).not.toBe(other.RAW_VALUE)
expect(clauses).toHaveSize(2)
expect(clauses.map((c) => c.RAW_VALUE)).toEqual([
one.RAW_VALUE,
other.RAW_VALUE
])
for (const clause of clauses) {
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
expect(clause.OPERATOR_NM).toBe('=')
expect(clause.GROUP_LOGIC).toBe('AND')
expect(clause.SUBGROUP_LOGIC).toBe('AND')
expect(clause.SUBGROUP_ID).toBe(0)
}
})
it('applies a transform to the clause it builds', () => {
// ADD(QUARTER_START,-1) is the reported requirement: the selected row plus
// the row for the previous quarter, as one IN clause so the AND rule is
// untouched. 2026-10-01 = 24380, 2026-07-01 = 24288 in SAS days.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: '01OCT2026' }], map)
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
expect(clause.OPERATOR_NM).toBe('IN')
expect(clause.RAW_VALUE).toBe('(24380,24288)')
})
it('applies a RANGE as BETWEEN', () => {
// RANGE(QUARTER,-1) for a mid-quarter value: 2026-04-01..2026-06-30
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: '1962-08-15' }], map)
expect(clause.OPERATOR_NM).toBe('BETWEEN')
expect(clause.RAW_VALUE).toContain(' AND ')
expect(clause.RAW_VALUE.split(' AND ')).toHaveSize(2)
})
it('keeps equality when the value already sits on the anchor', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: 0
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: '01OCT2026' }], map)
expect(clause.OPERATOR_NM).toBe('=')
expect(clause.RAW_VALUE).toBe('24380')
})
it('drops a transform on a column it cannot apply to', () => {
// A character column has no dates to shift; the parameter is skipped rather
// than filtered on the raw value the author did not see.
const map = new Map([
[
'selected',
{
column: 'SOME_CHAR',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
expect(build([{ label: 'SELECTED', value: 'x' }], map)).toEqual([])
})
it('adds nothing when the value is unparseable as a date', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
expect(build([{ label: 'SELECTED', value: 'not a date' }], map)).toEqual(
[]
)
})
it('round-trips a bare SAS day number unchanged', () => {
// VA may send the SAS internal value rather than a formatted string. The
// Date a transform works from is local, so the number must come back out of
// the clause identical - a UTC-built Date shifts a day west of UTC.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: 24380 }], map)
expect(clause.RAW_VALUE).toBe('(24380,24288)')
})
it('reports a range mapping fed more than one value', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
const problems = service.unsupportedTransforms(
msg([{ label: 'SELECTED', value: ['01APR2026', '01JUL2026'] }]),
map,
[{ NAME: 'SOME_DATE', DDTYPE: 'DATE', FMTNAME: 'DATE9' }]
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('is a range')
expect(problems[0]).toContain('2 values')
})
it('refuses a range fed more than one value, rather than picking one', () => {
// Two values, one of which does not parse: building a BETWEEN from the
// survivor would silently filter one period when VA selected two.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
expect(
build([{ label: 'SELECTED', value: ['01APR2026', 'not a date'] }], map)
).toEqual([])
})
it('never lets a non-finite date reach the clause', () => {
// convertJsDateToSasDate returning NaN would otherwise put a literal NaN in
// the WHERE clause.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
spyOn(helperService, 'convertJsDateToSasDate').and.returnValue(NaN as any)
expect(build([{ label: 'SELECTED', value: '01OCT2026' }], map)).toEqual(
[]
)
})
it('reports a partly unparseable value set instead of covering the rest', () => {
// One good value and one that is not a date: the clause still filters on
// the good one, so the author has to be told the other was dropped.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const problems = service.unsupportedTransforms(
msg([{ label: 'SELECTED', value: ['01OCT2026', 'not a date'] }]),
map,
cols
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('1 of its 2 values was not a date')
})
it('reports a range holding a value that does not parse', () => {
// Counted on what VA sent, not on what parsed: otherwise the second value
// vanishes and the range looks like a clean single-value case.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
const problems = service.unsupportedTransforms(
msg([{ label: 'SELECTED', value: ['01APR2026', 'not a date'] }]),
map,
cols
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('is a range')
expect(problems[0]).toContain('2 values')
})
it('applies a transform on the deferred initial load too', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const [clause] = service.buildInitialClauses(
msg([{ name: 'pr1', label: 'SELECTED', value: '01OCT2026' }]),
map
)
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
expect(clause.RAW_VALUE).toBe('(24380,24288)')
})
it('behaves exactly as before with an empty map', () => {
expect(
service.buildClauses(
msg([{ label: 'SOME_CHAR', value: 'x' }]),
labelToVar(),
cols
)
).toEqual(build([{ label: 'SOME_CHAR', value: 'x' }], new Map()))
})
})
describe('unresolvedMappings (what the report author is warned about)', () => {
const report = (
parameters: any[],
map: Map<string, VaParmapTarget>,
libds?: string
) => service.unresolvedMappings(msg(parameters), map, cols, libds)
it('reports a mapping whose column is not in the loaded table', () => {
expect(
report(
[{ label: 'REGION_TAB1', value: 'x' }],
new Map([['region_tab1', { column: 'NOT_A_COLUMN' }]]),
'DC_JSLIB.MPE_X_TEST'
)
).toEqual([
'VA parameter "REGION_TAB1" is mapped to NOT_A_COLUMN, which is not a column of DC_JSLIB.MPE_X_TEST - the filter was ignored.'
])
})
it('says "this table" when the table name is not supplied', () => {
expect(
report(
[{ label: 'REGION_TAB1', value: 'x' }],
new Map([['region_tab1', { column: 'NOPE' }]])
)[0]
).toContain('is not a column of this table')
})
it('reports nothing when the mapping resolves, or the param is unmapped', () => {
expect(
report(
[{ label: 'REGION_TAB1', value: 'x' }],
new Map([['region_tab1', { column: 'some_char' }]])
)
).toEqual([])
expect(report([{ label: 'SOME_CHAR', value: 'x' }], new Map())).toEqual(
[]
)
// A mapping whose parameter this message does not carry is not an error.
expect(
report([], new Map([['region_tab1', { column: 'NOPE' }]]))
).toEqual([])
})
it('reports each mapping once, not once per message parameter', () => {
const problems = report(
[
{ label: 'REGION_TAB1', value: '1' },
{ name: 'pr99', label: 'REGION_TAB1', value: '2' }
],
new Map([['region_tab1', { column: 'NOPE' }]])
)
expect(problems).toHaveSize(1)
})
it('treats an unloaded column list as "not known yet", not as a bad mapping', () => {
// While the grid is still loading there is no metadata to judge against, so
// reporting would flash the banner on every load.
expect(
service.unresolvedMappings(
msg([{ label: 'REGION_TAB1', value: 'x' }]),
new Map([['region_tab1', { column: 'NOPE' }]]),
[],
'DC_JSLIB.MPE_X_TEST'
)
).toEqual([])
expect(
service.unresolvedMappings(
msg([{ label: 'REGION_TAB1', value: 'x' }]),
new Map([['region_tab1', { column: 'NOPE' }]]),
undefined as any,
'DC_JSLIB.MPE_X_TEST'
)
).toEqual([])
})
})
describe('sameColumnMappings (two parameters, one column)', () => {
it('reports an AND-ed pair, naming both parameters and the column', () => {
const map = new Map([
['dropdown_tab1', { column: 'SOME_DROPDOWN' }],
['dropdown_tab2', { column: 'SOME_DROPDOWN' }]
])
const problems = service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'Option 1' },
{ label: 'DROPDOWN_TAB2', value: 'Option 2' }
]),
map
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('"DROPDOWN_TAB1" and "DROPDOWN_TAB2"')
expect(problems[0]).toContain('all filter SOME_DROPDOWN')
expect(problems[0]).toContain('combined with AND')
})
it('says nothing for one parameter per column', () => {
const map = new Map([
['dropdown_tab1', { column: 'SOME_DROPDOWN' }],
['num_tab1', { column: 'SOME_NUM' }]
])
expect(
service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'Option 1' },
{ label: 'NUM_TAB1', value: '1' }
]),
map
)
).toEqual([])
expect(
service.sameColumnMappings(
msg([{ label: 'DROPDOWN_TAB1', value: 'Option 1' }]),
map
)
).toEqual([])
expect(service.sameColumnMappings(msg([]), map)).toEqual([])
})
it('says nothing when the AND-ed parameters carry the same value', () => {
// col = 'x' AND col = 'x' filters exactly as one clause does, so a banner
// claiming "different values will show no rows" would be wrong.
const map = new Map([
['dropdown_tab1', { column: 'SOME_CHAR' }],
['dropdown_tab2', { column: 'SOME_CHAR' }]
])
expect(
service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'x' },
{ label: 'DROPDOWN_TAB2', value: 'x' }
]),
map,
cols
)
).toEqual([])
// One differing value and the intersection can be empty: reported.
const problems = service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'x' },
{ label: 'DROPDOWN_TAB2', value: 'y' }
]),
map,
cols
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('"DROPDOWN_TAB1" and "DROPDOWN_TAB2"')
})
it('leaves a column the table does not have to unresolvedMappings', () => {
// One cause, one line: the column is missing, so the AND story is moot.
const map = new Map([
['a', { column: 'NOPE' }],
['b', { column: 'NOPE' }]
])
const cols = [
{ LIBREF: 'DC_JSLIB', TABLE: 'MPE_X_TEST', VARIABLE: 'SOME_CHAR' }
]
expect(
service.sameColumnMappings(
msg([
{ label: 'A', value: '1' },
{ label: 'B', value: '2' }
]),
map,
cols
)
).toEqual([])
// Without metadata the pair is still reported: the AND semantics hold
// whatever the table turns out to be.
expect(
service.sameColumnMappings(
msg([
{ label: 'A', value: '1' },
{ label: 'B', value: '2' }
]),
map
)
).toHaveSize(1)
})
it('reports one line per contended column', () => {
const map = new Map([
['a', { column: 'SOME_DROPDOWN' }],
['b', { column: 'SOME_DROPDOWN' }],
['c', { column: 'SOME_NUM' }],
['d', { column: 'SOME_NUM' }]
])
const problems = service.sameColumnMappings(
msg([
{ label: 'A', value: '1' },
{ label: 'B', value: '2' },
{ label: 'C', value: '3' },
{ label: 'D', value: '4' }
]),
map
)
expect(problems).toHaveSize(2)
})
})
describe('unmatchedMappings (parameter-side typo)', () => {
const seen = (...keys: string[]) => new Set(keys)
it('reports a key that no message has named', () => {
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen('dropdown_tab_1', 'pr1'),
true
)
).toEqual([
'This object\'s va_parmap names VA parameter "dropdown_tab1", which VA is not sending - check the spelling.'
])
})
it('says nothing until a parameter has actually been seen', () => {
// Without evidence that this object receives parameters at all, an absent
// key proves nothing - VA sends only the parameters its query uses.
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen()
)
).toEqual([])
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
undefined
)
).toEqual([])
})
it('says nothing when every key has been seen', () => {
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen('dropdown_tab1'),
true
)
).toEqual([])
expect(
service.unmatchedMappings(new Map(), seen('anything'), true)
).toEqual([])
})
it('says nothing while the parameter set is still growing', () => {
// A message that introduces a new parameter name means this object's set
// has not settled, so an absent key may arrive in the next message.
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen('pr1', 'pr2'),
false
)
).toEqual([])
})
})
describe('buildInitialClauses (no metadata, infer kind from value)', () => {
it('uses an explicit mapping instead of the label', () => {
// The deferred first load has no column metadata, so a mapped column is
// used as given and the SAS kind still comes from the value shape.
const map = new Map([
['region_tab1', { column: 'REGION' }],
['amt_tab1', { column: 'AMOUNT' }]
])
const clauses = service.buildInitialClauses(
msg([
{ name: 'pr1', label: 'REGION_TAB1', dataType: 'string', value: 'n' },
{ name: 'pr2', label: 'AMT_TAB1', dataType: 'number', value: '10' }
]),
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['REGION', "'n'"],
['AMOUNT', '10']
])
})
it('falls back to the label for params the map does not name', () => {
const map = new Map([['region_tab2', { column: 'REGION' }]])
const clauses = service.buildInitialClauses(
msg([
{ name: 'pr1', label: 'REGION_TAB1', value: 'n' },
{ name: 'pr2', label: 'REGION_TAB2', value: 's' }
]),
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['REGION_TAB1', "'n'"],
['REGION', "'s'"]
])
})
it('uses the VA label as the column and infers kinds by value shape', () => {
const clauses = service.buildInitialClauses(
msg([
+461 -47
View File
@@ -2,6 +2,12 @@ import { Injectable } from '@angular/core'
import { HelperService } from './helper.service'
import { VaMessage } from './va-messaging.service'
import { QueryClause } from '../models/TableData'
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
import {
applyVaTransform,
isRangeTransform,
VaTransform
} from '../shared/utils/va-parmap-transform'
/** SAS data-type kind used to format a VA filter value into a DC RAW_VALUE. */
type VaColumnKind = 'char' | 'numeric' | 'time' | 'date' | 'datetime'
@@ -54,11 +60,19 @@ export class VaFilterService {
* Translates VA `parameters` into DC filter clauses. Each parameter is matched
* to a DC column by label; a single value becomes `col = value`, multiple
* values become `col IN (...)`. Values are formatted for the column's SAS kind.
*
* `paramMap` (from the URL `va_parmap` parameter, see parseVaParmap) overrides
* that same-name resolution for the parameters it names, so two DC instances
* over one table can filter the same column from differently-named VA
* parameters. A mapped column must exist in `cols` - if it does not the
* parameter is skipped rather than falling back to the same-name match, which
* is exactly the behaviour the mapping exists to override.
*/
buildClauses(
msg: VaMessage,
labelToVar: Map<string, string>,
cols: any[]
cols: any[],
paramMap?: Map<string, VaParmapTarget>
): QueryClause[] {
const clauses: QueryClause[] = []
for (const param of msg.parameters || []) {
@@ -67,60 +81,194 @@ export class VaFilterService {
.toString()
.trim()
.toLowerCase()
// A VA param whose label isn't a DC column (beyond case, which is handled)
// simply doesn't map — skip it.
const varName = labelToVar.get(key)
// An explicit URL mapping wins over the label->column lookup. The mapped
// column has to exist in THIS table, else the entry does not apply here
// (and the same-name fallback would be the behaviour it exists to
// override).
const mapped = this.mappedColumn(param, paramMap)
const mappedCol = mapped ? this.findCol(mapped, cols) : undefined
if (mapped && !mappedCol) continue
// The table's own case, so the clause matches what the metadata build
// produces for a same-name match.
const varName: string = mappedCol
? mappedCol.NAME
: (labelToVar.get(key) ?? '')
if (!varName) continue
const col = this.findCol(varName, cols)
const col = mappedCol ?? this.findCol(varName, cols)
const kind = this.columnKind(col)
const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
const rawValues = Array.isArray(param.value) ? param.value : [param.value]
// Format/validate every value for the column's SAS type. Numeric columns
// must receive a finite number (untrusted postMessage input is
// interpolated into a SAS WHERE clause, so a non-numeric value here would
// be an injection vector); temporal -> SAS internal number; char ->
// escaped. Invalid values are dropped.
const formatted = rawValues
.filter((v) => v !== undefined && v !== null && v !== '')
.map((v) => this.formatValueByKind(v, kind, order))
.filter((v): v is string => v !== null)
if (formatted.length === 0) continue
clauses.push(this.clause(varName, formatted))
const built = this.buildClauseValueSet(
param,
kind,
order,
this.mappedTarget(param, paramMap)?.transform
)
if (!built) continue
clauses.push({
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: varName,
OPERATOR_NM: built.operator,
RAW_VALUE: built.rawValue
})
}
return clauses
}
/**
* The operator and raw value for one parameter, after any transform.
*
* Values are formatted for the column's SAS kind: numeric columns must receive
* a finite number (untrusted postMessage input is interpolated into a WHERE
* clause, so anything else would be an injection vector); temporal values
* become SAS internal numbers; char values are escaped. Invalid values are
* dropped, and a transform that cannot be applied to the column's kind yields
* nothing at all - the caller reports that separately rather than filtering on
* a value the author did not choose.
*/
private buildClauseValueSet(
param: any,
kind: VaColumnKind,
order: 'YMD' | 'DMY' | 'MDY' | undefined,
transform?: VaTransform
): { operator: '=' | 'IN' | 'BETWEEN'; rawValue: string } | null {
const rawValues = (
Array.isArray(param.value) ? param.value : [param.value]
).filter((v: any) => v !== undefined && v !== null && v !== '')
if (rawValues.length === 0) return null
if (!transform) {
const formatted = rawValues
.map((v: any) => this.formatValueByKind(v, kind, order))
.filter((v: string | null): v is string => v !== null)
if (formatted.length === 0) return null
return formatted.length === 1
? { operator: '=', rawValue: formatted[0] }
: { operator: 'IN', rawValue: `(${formatted.join(',')})` }
}
// Transforms resolve to dates, so only temporal columns can take one.
if (kind !== 'date' && kind !== 'datetime') return null
const unit = kind === 'datetime' ? 'seconds' : 'days'
const values = new Set<string>()
for (const raw of rawValues) {
const resolved = this.valueAsDate(raw, kind, order)
if (!resolved) continue
for (const date of applyVaTransform(
resolved,
transform,
kind === 'datetime'
)) {
// A non-finite result (an Invalid Date reaching the helper) must never be
// interpolated into the clause: `NaN` is not a SAS value. Dropped here,
// and reported by unsupportedTransforms so the parameter is not silently
// filtered on nothing.
const sasDate = this.helperService.convertJsDateToSasDate(date, unit)
if (!Number.isFinite(Number(sasDate))) continue
values.add(`${sasDate}`)
}
}
const formatted = Array.from(values)
if (formatted.length === 0) return null
if (isRangeTransform(transform)) {
// One BETWEEN carries one period, so more than one value has no clause -
// dropping it here (rather than building a BETWEEN from whichever value
// parsed) is what unsupportedTransforms' count warning then describes.
if (rawValues.length > 1) return null
if (formatted.length !== 2) return null
return { operator: 'BETWEEN', rawValue: formatted.join(' AND ') }
}
return formatted.length === 1
? { operator: '=', rawValue: formatted[0] }
: { operator: 'IN', rawValue: `(${formatted.join(',')})` }
}
/**
* A VA value as a local Date, for a transform to work from. Formatted strings
* are parsed by shape; a bare number is already the SAS internal value, so it
* is converted back from days (date) or seconds (datetime) since 1960-01-01.
*
* Built with local calendar construction, matching `parseDate` and
* `HelperService.convertJsDateToSasDate`: a UTC instant read back through local
* accessors lands on the previous day west of UTC, and the round trip to a SAS
* day number would be one out.
*/
private valueAsDate(
value: any,
kind: VaColumnKind,
order?: 'YMD' | 'DMY' | 'MDY'
): Date | null {
const s = String(value).trim()
if (s === '') return null
if (this.isBareNumber(s)) {
const internal = Number(s)
if (kind !== 'datetime') return new Date(1960, 0, 1 + internal)
// Seconds since midnight 1960-01-01, split into whole days and a remainder
// so the constructor normalises the clock time locally.
const days = Math.floor(internal / 86400)
const seconds = internal - days * 86400
return new Date(1960, 0, 1 + days, 0, 0, seconds)
}
return kind === 'datetime'
? this.parseDatetime(s, order)
: this.parseDate(s, order)
}
/**
* Pre-getdata variant used on the deferred initial load, when DC column
* metadata isn't available. Uses the VA `label` as the column name and infers
* the SAS kind from the value shape, so the clause matches what the metadata
* build later produces (no reconcile reload).
*
* `paramMap` overrides the label-as-column rule for the parameters it names,
* so the very first (already filtered) load also honours an explicit mapping.
*/
buildInitialClauses(msg: VaMessage): QueryClause[] {
buildInitialClauses(
msg: VaMessage,
paramMap?: Map<string, VaParmapTarget>
): QueryClause[] {
const clauses: QueryClause[] = []
for (const param of msg.parameters || []) {
if (!param || param.value === undefined || param.value === null) continue
// The DC column name is the VA `label`; `name` is VA's internal id (pr###).
const varName = (param.label ?? param.name ?? '').toString().trim()
const target = this.mappedTarget(param, paramMap)
// Without a mapping the DC column name is the VA `label`; `name` is VA's
// internal id (pr###). A `va_parmap` entry wins over both.
const varName = (target?.column ?? param.label ?? param.name ?? '')
.toString()
.trim()
if (!varName) continue
const rawValues = Array.isArray(param.value) ? param.value : [param.value]
const rawValues = (
Array.isArray(param.value) ? param.value : [param.value]
) as any[]
const nonEmpty = rawValues.filter(
(v) => v !== undefined && v !== null && v !== ''
)
if (nonEmpty.length === 0) continue
const formatted = nonEmpty
.map((v) =>
this.formatValueByKind(
v,
this.inferKindFromValue(String(v).trim(), param.dataType)
)
)
.filter((v): v is string => v !== null)
if (formatted.length === 0) continue
clauses.push(this.clause(varName, formatted))
// A transform only ever produces dates, so it fixes the kind: inferring it
// from the value would see a formatted date either way, but with the
// transform the column is known to be temporal.
const kind = target?.transform
? ('date' as VaColumnKind)
: this.inferKindFromValue(String(nonEmpty[0]).trim(), param.dataType)
const built = this.buildClauseValueSet(
param,
kind,
undefined,
target?.transform
)
if (!built) continue
clauses.push({
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: varName,
OPERATOR_NM: built.operator,
RAW_VALUE: built.rawValue
})
}
return clauses
}
@@ -141,16 +289,254 @@ export class VaFilterService {
)
}
private clause(varName: string, formatted: string[]): QueryClause {
return {
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: varName,
OPERATOR_NM: formatted.length === 1 ? '=' : 'IN',
RAW_VALUE:
formatted.length === 1 ? formatted[0] : `(${formatted.join(',')})`
/**
* Transform mappings in this message that cannot be applied, one plain-English
* line each: the column is not temporal, or the value is not a date the browser
* can parse. `buildClauseValueSet` skips those parameters, so this is what stops
* the misconfiguration reading as a filter that simply matched nothing.
*
* A missing column is left to `unresolvedMappings`, which already reports it.
*/
unsupportedTransforms(
msg: VaMessage,
paramMap: Map<string, VaParmapTarget> | undefined,
cols: any[]
): string[] {
if (!paramMap || paramMap.size === 0) return []
if (!cols || cols.length === 0) return []
const problems = new Set<string>()
for (const param of msg.parameters || []) {
const target = this.mappedTarget(param, paramMap)
if (!target?.transform) continue
const transform = target.transform
const col = this.findCol(target.column, cols)
if (!col) continue
const kind = this.columnKind(col)
const paramName = (param.label ?? param.name ?? '').toString().trim()
const spec =
target.transform.op === 'ADD'
? `ADD(${target.transform.anchor},${target.transform.offset})`
: `RANGE(${target.transform.period},${target.transform.offset})`
if (kind !== 'date' && kind !== 'datetime') {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${target.column} is not a date or datetime column - the filter was ignored.`
)
continue
}
const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
const sent = (
Array.isArray(param.value) ? param.value : [param.value]
).filter((v: any) => v !== undefined && v !== null && v !== '')
const usable = sent.filter((v: any) => {
const date = this.valueAsDate(v, kind, order)
if (!date) return false
// Same test the clause builder applies, so this count matches what it uses.
return applyVaTransform(date, transform, kind === 'datetime').every(
(resolved) => {
const sasDate = this.helperService.convertJsDateToSasDate(
resolved,
kind === 'datetime' ? 'seconds' : 'days'
)
return Number.isFinite(Number(sasDate))
}
)
})
if (usable.length === 0) {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but its value was not a date - the filter was ignored.`
)
continue
}
// One BETWEEN can only express one period. A multi-select would have to
// become an OR of ranges, which the clause model does not carry, so say so
// rather than building a BETWEEN from whichever value happened to parse.
// Counted on what VA sent: a second value that does not parse must not make
// the single-value case look clean.
if (isRangeTransform(target.transform) && sent.length > 1) {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, which is a range, but the parameter has ${sent.length} values - a range needs one value, so the filter was ignored.`
)
continue
}
if (usable.length < sent.length) {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${sent.length - usable.length} of its ${sent.length} values was not a date - the filter covers only the rest.`
)
}
}
return Array.from(problems)
}
/**
* `va_parmap` entries that this message actually uses but that name a column
* the loaded table does not have, one plain-English line each. `buildClauses`
* skips those parameters (deliberately: falling back to the same-name match
* would silently apply the filter the mapping exists to override), so this is
* what makes the misconfiguration visible to the report author - otherwise the
* grid simply shows every row, which looks like a working filter.
*/
unresolvedMappings(
msg: VaMessage,
paramMap: Map<string, VaParmapTarget> | undefined,
cols: any[],
libds?: string
): string[] {
const problems = new Set<string>()
if (!paramMap || paramMap.size === 0) return []
// No metadata yet is not the same as "the table lacks this column": an empty
// `cols` means the grid is still loading, so there is nothing to judge yet.
// Reporting here would flash a misconfiguration banner during every load.
if (!cols || cols.length === 0) return []
for (const param of msg.parameters || []) {
const mapped = this.mappedColumn(param, paramMap)
if (!mapped || this.findCol(mapped, cols)) continue
const paramName = (param.label ?? param.name ?? '').toString().trim()
problems.add(
`VA parameter "${paramName}" is mapped to ${mapped}, which is not a column of ${
libds || 'this table'
} - the filter was ignored.`
)
}
return Array.from(problems)
}
/**
* `va_parmap` keys that no VA message has named, one plain-English line each.
* A key that matches nothing is a parameter-side typo: the mapping never
* applies and the grid stays unfiltered, which looks the same as a working
* filter.
*
* `seenKeys` must be every key observed across messages so far. `paramsStable`
* says the latest message introduced no new parameter name, i.e. this object's
* parameter set has settled - until then a missing name may simply not have
* arrived, because VA sends only the parameters its query uses.
*/
unmatchedMappings(
paramMap: Map<string, VaParmapTarget> | undefined,
seenKeys: Set<string> | undefined,
paramsStable = false
): string[] {
if (!paramMap || paramMap.size === 0) return []
if (!seenKeys || seenKeys.size === 0) return []
if (!paramsStable) return []
const problems: string[] = []
for (const key of Array.from(paramMap.keys())) {
if (!seenKeys.has(key)) {
problems.push(
`This object's va_parmap names VA parameter "${key}", which VA is not sending - check the spelling.`
)
}
}
return problems
}
/**
* Parameters in this message that filter the same column with *different*
* values, one plain-English line per column. Clauses are AND-ed, so two
* parameters on one column are an intersection: with different values the grid
* comes back empty, which looks like a working filter. Two parameters carrying
* the same value intersect cleanly and filter correctly, so they are not
* reported - a banner there would call a working configuration broken.
*
* Reported rather than rewritten - folding them into one `IN (...)` would
* silently turn an AND into an OR, which is a semantics decision, not a
* diagnostic.
*/
sameColumnMappings(
msg: VaMessage,
paramMap: Map<string, VaParmapTarget> | undefined,
cols?: any[]
): string[] {
if (!paramMap || paramMap.size === 0) return []
// column -> the parameters on it, each with the clause value it produces
const byColumn = new Map<string, { name: string; value: string }[]>()
for (const param of msg.parameters || []) {
const mapped = this.mappedColumn(param, paramMap)
if (!mapped) continue
// A column the table lacks is unresolvedMappings' finding; counting it here
// would report one cause as two symptoms.
const col =
cols && cols.length > 0 ? this.findCol(mapped, cols) : undefined
if (cols && cols.length > 0 && !col) continue
// A parameter that produces no clause cannot conflict with anything.
const value = this.valueSignature(
param,
col,
this.mappedTarget(param, paramMap)?.transform
)
if (value === null) continue
const column = mapped.toUpperCase()
const params = byColumn.get(column) || []
const paramName = (param.label ?? '').toString().trim()
if (paramName && !params.some((p) => p.name === paramName)) {
params.push({ name: paramName, value })
}
byColumn.set(column, params)
}
const problems: string[] = []
for (const [column, params] of Array.from(byColumn.entries())) {
if (params.length < 2) continue
// Same value on every parameter: the AND is a no-op, not an empty grid.
const distinct = new Set(params.map((p) => p.value))
if (distinct.size === 1) continue
problems.push(
`VA parameters ${params
.map((p) => `"${p.name}"`)
.join(
' and '
)} all filter ${column}. Parameters are combined with AND, so rows must match all of them - different values will show no rows. Use one parameter carrying both values to match either.`
)
}
return problems
}
/**
* The DC column an explicit URL mapping (`va_parmap`) assigns to this
* parameter, or undefined when the parameter is not mapped. The key is matched
* case-insensitively against the parameter's `label`.
*/
private mappedTarget(
param: any,
paramMap?: Map<string, VaParmapTarget>
): VaParmapTarget | undefined {
if (!paramMap || paramMap.size === 0) return undefined
// The label only. The internal name (pr273) is an implementation detail of
// the report, and matching both made one parameter answer to two keys: a
// mapping written against the name could never be distinguished from a typo
// once the label was also mapped, and the second entry silently lost.
const key = (param.label ?? '').toString().trim().toLowerCase()
if (!key) return undefined
return paramMap.get(key)
}
/**
* The clause value a parameter would filter its column with, as a comparable
* string, or null when it produces no clause at all. Used to tell an AND of two
* identical values (harmless) from an AND of two different ones (empty grid).
*
* Delegates to the clause builder so a transform's derived dates take part in
* the comparison: two parameters adding the same quarter are the same clause.
*/
private valueSignature(
param: any,
col: any,
transform?: VaTransform
): string | null {
const built = this.buildClauseValueSet(
param,
this.columnKind(col),
this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT),
transform
)
return built ? built.rawValue : null
}
/** The mapped column name, for callers that do not need the transform. */
private mappedColumn(
param: any,
paramMap?: Map<string, VaParmapTarget>
): string | undefined {
return this.mappedTarget(param, paramMap)?.column
}
/** The loaded column spec whose NAME matches varName (case-insensitive). */
@@ -321,14 +707,42 @@ export class VaFilterService {
return this.parseMonthNameDate(value) ?? this.parseNumericDate(value, order)
}
/** DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62) -> local Date, or null. */
/**
* Month-name date shapes -> local Date, or null. Three families:
* - DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62);
* - word date with the month first, comma optional: "January 1, 2026",
* "Jan 1 2026", "Jan. 1, 2026";
* - word date with the day first: "1 January 2026", "1-Jan-2026".
*
* A date parameter's display format decides which of these the report author
* sees, and VA sends the formatted text, so a report using WORDDATE (or a
* long month name in any order) has to parse here rather than being dropped.
* The month name may be abbreviated or spelled out; only its first three
* letters are matched.
*/
private parseMonthNameDate(value: string): Date | null {
const m = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
if (!m) return null
const [, day, monthAbbr, year] = m
const monthIndex = VaFilterService.SAS_MONTHS.indexOf(
monthAbbr.toUpperCase()
)
const monthByName = (name: string): number =>
VaFilterService.SAS_MONTHS.indexOf(
name.replace(/\./g, '').slice(0, 3).toUpperCase()
)
const date9 = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
if (date9) {
const [, day, monthAbbr, year] = date9
const monthIndex = monthByName(monthAbbr)
if (monthIndex < 0) return null
return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
}
const monthFirst =
/^([A-Za-z]{3,9})\.?[\s-]+(\d{1,2}),?[\s-]+(\d{2}|\d{4})$/.exec(value)
const dayFirst =
/^(\d{1,2})[\s-]+([A-Za-z]{3,9})\.?,?[\s-]+(\d{2}|\d{4})$/.exec(value)
const name = monthFirst ? monthFirst[1] : dayFirst ? dayFirst[2] : null
if (!name) return null
const day = monthFirst ? monthFirst[2] : dayFirst![1]
const year = monthFirst ? monthFirst[3] : dayFirst![3]
const monthIndex = monthByName(name)
if (monthIndex < 0) return null
return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
}
@@ -1,4 +1,6 @@
import { VaMessagingService, VaMessage } from './va-messaging.service'
import { VaFilterService } from './va-filter.service'
import { HelperService } from './helper.service'
// Sample message taken from the official SAS DDC docs (brush column included).
const SAMPLE = {
@@ -51,6 +53,57 @@ describe('VaMessagingService', () => {
})
})
describe('parameter keys (what the va_parmap typo diagnostic is judged on)', () => {
const PARAMS = [
{ name: 'pr273', label: 'Tab_1_Date', value: '2025-03-31' },
{ name: 'pr274', label: 'Tab 2 Region', value: 'north' }
]
const withParams = () => ({ ...SAMPLE, parameters: PARAMS })
it('records each parameter label once, across messages', () => {
service.parseMessage({ data: withParams() } as MessageEvent)
service.parseMessage({ data: withParams() } as MessageEvent)
expect(Array.from(service.seenParamKeys)).toEqual([
'tab_1_date',
'tab 2 region'
])
// The second message introduced no new label, so the set has settled.
expect(service.lastMessageParamsStable).toBeTrue()
})
it('records the label only, so a mapping written against the internal name is reported', () => {
// `pr273` is VA's own bookkeeping. A va_parmap entry written against it
// never matches a parameter, and recording the name here would make the
// typo diagnostic believe VA is sending it - leaving the author with an
// unfiltered grid and no warning at all.
service.parseMessage({ data: withParams() } as MessageEvent)
service.parseMessage({ data: withParams() } as MessageEvent)
const filter = new VaFilterService(new HelperService({} as any))
const seen = service.seenParamKeys
const stable = service.lastMessageParamsStable
expect(
filter.unmatchedMappings(
new Map([['pr273', { column: 'D_DATE' }]]),
seen,
stable
)
).toEqual([
'This object\'s va_parmap names VA parameter "pr273", which VA is not sending - check the spelling.'
])
// The label is a real key, so the same map written that way is silent.
expect(
filter.unmatchedMappings(
new Map([['tab_1_date', { column: 'D_DATE' }]]),
seen,
stable
)
).toEqual([])
})
})
describe('dataColumns', () => {
it('excludes brush columns and keeps original indices', () => {
const msg = service.parseMessage({ data: SAMPLE } as MessageEvent)!
@@ -82,6 +82,26 @@ export class VaMessagingService {
/** DC variable names that should be visible (the rest are hidden). */
visibleColumns: string[] | null = null
/**
* Lower-cased label of every parameter any VA message has carried so far, kept
* across editor reloads. Lets the editor tell a `va_parmap` key that names no
* parameter in the report (a typo, so the filter would silently never apply)
* from one that is merely absent from the current message - VA only sends the
* parameters its query uses.
*
* Labels only: that is the key a mapping is matched on, and recording the
* internal name as well would make a name-keyed entry - which never applies -
* look like a parameter VA is sending.
*/
seenParamKeys = new Set<string>()
/**
* True when every parameter name in the latest message had already been seen.
* VA sends only the parameters a report object's query uses, and sends them per
* object, so a name that is absent from the first messages may simply not have
* arrived yet. A stable parameter set is what makes absence meaningful.
*/
lastMessageParamsStable = false
/** True once the pre-bootstrap early buffer has been drained (drain-once). */
private earlyDrained = false
@@ -187,6 +207,22 @@ export class VaMessagingService {
if (typeof data.resultName !== 'string') return null
if (!Array.isArray(data.columns) || !Array.isArray(data.data)) return null
const parameters = Array.isArray(data.parameters) ? data.parameters : []
const keys: string[] = []
for (const param of parameters) {
// The label only, which is the key a `va_parmap` entry is matched on. The
// internal name (`pr273`) is the report's own bookkeeping: recording it
// here would let the typo diagnostic below believe a name-keyed entry is
// being sent when no code path can ever apply it.
const key = (param?.label ?? '').toString().trim().toLowerCase()
if (key && !keys.includes(key)) keys.push(key)
}
// Judged before the keys are recorded: a message that introduces a new name
// means the set is still growing, so a missing name proves nothing yet.
this.lastMessageParamsStable =
keys.length > 0 && keys.every((key) => this.seenParamKeys.has(key))
for (const key of keys) this.seenParamKeys.add(key)
return {
version: data.version,
resultName: data.resultName,
@@ -194,7 +230,7 @@ export class VaMessagingService {
availableRowCount: data.availableRowCount,
data: data.data,
columns: data.columns,
parameters: Array.isArray(data.parameters) ? data.parameters : []
parameters
}
}
@@ -0,0 +1,215 @@
import { parseVaParmap } from './parse-va-parmap'
describe('parseVaParmap', () => {
it('returns an empty map and no problems when the parameter is absent', () => {
for (const query of ['', 'embed=va', 'embed=va&labels=true']) {
const parsed = parseVaParmap(query)
expect(parsed.map.size).toBe(0)
expect(parsed.problems).toEqual([])
}
})
it('maps a VA parameter label to a DC column', () => {
const { map, problems } = parseVaParmap(
'embed=va&va_parmap=REGION_TAB1-REGION'
)
expect(map.get('region_tab1')?.column).toBe('REGION')
expect(map.size).toBe(1)
expect(problems).toEqual([])
})
it('collects repeated entries, so two parameters can map to one column', () => {
// The multi-tab case: each DC instance filters the same DC column from a
// differently-named VA parameter.
const { map } = parseVaParmap(
'embed=va&va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION'
)
expect(map.get('region_tab1')?.column).toBe('REGION')
expect(map.get('region_tab2')?.column).toBe('REGION')
expect(map.size).toBe(2)
})
it('is case-insensitive on the key and trims both sides', () => {
const { map } = parseVaParmap('va_parmap=%20Region_Tab1%20-%20region%20')
expect(map.get('region_tab1')?.column).toBe('region')
})
it('decodes a key containing a space (VA labels are free text)', () => {
expect(
parseVaParmap('va_parmap=My+Region-DEPT').map.get('my region')?.column
).toBe('DEPT')
})
it('splits at the LAST hyphen, so a label may contain hyphens', () => {
expect(
parseVaParmap('va_parmap=Sales-2024-DEPT').map.get('sales-2024')?.column
).toBe('DEPT')
})
describe('entries that cannot be used', () => {
it('reports an entry with no separator', () => {
const { map, problems } = parseVaParmap('va_parmap=DEPT')
expect(map.size).toBe(0)
expect(problems).toEqual([
'va_parmap "DEPT" was ignored: it needs a "-" between the VA parameter and the column name.'
])
})
it('reports an entry with no parameter name', () => {
const { map, problems } = parseVaParmap('va_parmap=-DEPT')
expect(map.size).toBe(0)
expect(problems).toEqual([
'va_parmap "-DEPT" was ignored: the VA parameter name is missing.'
])
})
it('reports an entry with no column name', () => {
const { map, problems } = parseVaParmap('va_parmap=REGION_TAB1-')
expect(map.size).toBe(0)
expect(problems).toEqual([
'va_parmap "REGION_TAB1-" was ignored: "" is not a valid SAS column name.'
])
})
it('reports a column that is not a plain SAS name (injection guard)', () => {
// The column is interpolated into a SAS WHERE clause, so anything beyond a
// SAS name is refused at parse time rather than carried into a clause.
for (const entry of [
'A-x;drop',
'A-REGION%20OR%201=1',
"A-RE'GION",
'A-1ABC'
]) {
const { map, problems } = parseVaParmap(`va_parmap=${entry}`)
expect(map.size).toBe(0)
expect(problems.length).toBe(1)
expect(problems[0]).toContain('is not a valid SAS column name')
}
expect(parseVaParmap('va_parmap=A-A_B9').map.get('a')?.column).toBe(
'A_B9'
)
})
it('keeps the usable entries and reports only the bad one', () => {
const { map, problems } = parseVaParmap(
'va_parmap=GOOD-REGION&va_parmap=BROKEN'
)
expect(map.get('good')?.column).toBe('REGION')
expect(problems.length).toBe(1)
expect(problems[0]).toContain('va_parmap "BROKEN"')
})
})
describe('transforms', () => {
it('parses an ADD anchor with an offset', () => {
const { map, problems } = parseVaParmap(
'va_parmap=SELECTED-D_DATE:ADD(QUARTER_START,-1)'
)
expect(problems).toEqual([])
expect(map.get('selected')?.column).toBe('D_DATE')
expect(map.get('selected')?.transform).toEqual({
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
})
it('defaults the offset to zero and accepts lower case', () => {
const { map, problems } = parseVaParmap(
'va_parmap=sel-d_date:add(quarter_start)'
)
expect(problems).toEqual([])
expect(map.get('sel')?.transform).toEqual({
op: 'ADD',
anchor: 'QUARTER_START',
offset: 0
})
})
it('parses a RANGE period', () => {
const { map, problems } = parseVaParmap(
'va_parmap=sel-D_DATE:RANGE(quarter,-1)'
)
expect(problems).toEqual([])
expect(map.get('sel')?.transform).toEqual({
op: 'RANGE',
period: 'QUARTER',
offset: -1
})
})
it('leaves a mapping without a transform as a plain column', () => {
const { map } = parseVaParmap('va_parmap=sel-D_DATE')
expect(map.get('sel')?.column).toBe('D_DATE')
expect(map.get('sel')?.transform).toBeUndefined()
})
it('reports an unknown operation, anchor, period or offset', () => {
const cases: [string, string][] = [
['sel-D_DATE:SHIFT(QUARTER_START,-1)', 'is not a known transform'],
['sel-D_DATE:ADD(FORTNIGHT_START,-1)', 'is not a known anchor'],
['sel-D_DATE:RANGE(DECADE,-1)', 'is not a known period'],
['sel-D_DATE:ADD(QUARTER_START,3.5)', 'is not a whole number'],
['sel-D_DATE:ADD', 'is not a transform']
]
for (const [entry, expected] of cases) {
const { map, problems } = parseVaParmap(`va_parmap=${entry}`)
expect(`${entry}: ${map.size}`).toBe(`${entry}: 0`)
expect(`${entry}: ${problems.length}`).toBe(`${entry}: 1`)
expect(problems[0]).toContain(expected)
}
})
it('keeps the parameter label free to contain a colon', () => {
// The key side is untouched by the transform grammar: only the column side
// is split, so a label with a colon still maps.
const { map, problems } = parseVaParmap(
'va_parmap=Sales:2026-D_DATE:ADD(QUARTER_START,-1)'
)
expect(problems).toEqual([])
expect(map.get('sales:2026')?.column).toBe('D_DATE')
expect(map.get('sales:2026')?.transform).toEqual({
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
})
})
describe('one parameter, one column', () => {
it('keeps the first mapping and reports a conflicting duplicate', () => {
const { map, problems } = parseVaParmap(
'va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER'
)
expect(map.get('dept')?.column).toBe('REGION')
expect(problems).toEqual([
'va_parmap "DEPT-OTHER" was ignored: VA parameter "dept" already maps to REGION, and a parameter can filter only one column.'
])
})
it('treats a duplicate key as a duplicate across case', () => {
const { map, problems } = parseVaParmap(
'va_parmap=Dept-REGION&va_parmap=DEPT-SOMETHING_ELSE'
)
expect(map.get('dept')?.column).toBe('REGION')
expect(problems.length).toBe(1)
})
it('accepts the same mapping written twice (no ambiguity)', () => {
const { map, problems } = parseVaParmap(
'va_parmap=DEPT-REGION&va_parmap=dept-region'
)
expect(map.get('dept')?.column).toBe('REGION')
expect(problems).toEqual([])
})
it('still allows two parameters to map to one column', () => {
// The multi-object case: different parameters, same DC column.
const { map, problems } = parseVaParmap(
'va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION'
)
expect(map.size).toBe(2)
expect(problems).toEqual([])
})
})
})
@@ -0,0 +1,192 @@
import {
VA_ANCHORS,
VA_PERIODS,
VaAnchor,
VaPeriod,
VaTransform
} from './va-parmap-transform'
/**
* What a VA parameter's mapping resolves to: the DC column, and optionally a
* transform that adds derived dates to the filter (e.g. the previous quarter's
* row alongside the selected one).
*/
export interface VaParmapTarget {
column: string
transform?: VaTransform
}
/** Result of `parseVaParmap` - the usable mappings, and what was ignored. */
export interface VaParmap {
/** Lower-cased VA parameter key -> DC column (and transform, if any). */
map: Map<string, VaParmapTarget>
/** One plain-English line per `va_parmap` entry that was ignored. */
problems: string[]
}
/**
* Parses the `va_parmap` entries out of a hash query string (the part after `?`
* in `window.location.hash`).
*
* Format - one mapping per occurrence, each `<VA parameter key>-<DC column>`:
*
* ?embed=va&va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION
*
* Repeating the parameter (rather than packing a comma-separated list into one
* value) keeps the grammar unambiguous: VA parameter labels are free text, so
* any list separator we picked could also appear inside a label.
*
* Each entry is split at its LAST `-`. A SAS column name can never contain a
* hyphen, so the final one always delimits the column, and a label that does
* contain hyphens (e.g. `Sales-2024`) still maps correctly.
*
* The key is matched case-insensitively, and trailing/leading blanks are
* trimmed, against a VA parameter's `label`, which is the same key the
* label->column match uses. The internal `name` (e.g. `pr273`) is not a key: it
* is the report's own bookkeeping, so an entry written against it never applies
* and is reported by `unmatchedMappings`.
*
* An entry that cannot be used is reported in `problems` rather than dropped in
* silence: these entries are hand-written in the report's DDC URL, usually by
* the report author, and a typo would otherwise degrade to "no filter" - which
* looks exactly like a working filter over the whole table.
*
* Returns an empty map when the parameter is absent, which leaves the default
* same-name resolution untouched (existing embeds are unaffected).
*
* Modeled on `parseEmbedParam` / `parseLabelsParam`, which read the same hash
* query.
*/
export function parseVaParmap(hashQuery: string): VaParmap {
const map = new Map<string, VaParmapTarget>()
const problems: string[] = []
if (!hashQuery) return { map, problems }
const entries = new URLSearchParams(hashQuery).getAll('va_parmap')
for (const entry of entries) {
// A transform sits at the end, in parentheses: `<param>-<column>:OP(args)`.
// It is lifted off FIRST, because its arguments may contain a hyphen (a
// negative offset) and the key/column split below uses the hyphen.
let body = entry ?? ''
let transformSpec = ''
if (body.trim().endsWith(')')) {
const open = body.lastIndexOf('(')
const colon = open > 0 ? body.lastIndexOf(':', open) : -1
if (colon > 0) {
transformSpec = body.slice(colon + 1).trim()
body = body.slice(0, colon)
}
}
// Last hyphen delimits the DC column (see above).
const separator = body.lastIndexOf('-')
if (separator < 0) {
problems.push(
`va_parmap "${entry}" was ignored: it needs a "-" between the VA parameter and the column name.`
)
continue
}
const key = body.slice(0, separator).trim().toLowerCase()
const column = body.slice(separator + 1).trim()
if (!key) {
problems.push(
`va_parmap "${entry}" was ignored: the VA parameter name is missing.`
)
continue
}
if (column.includes(':')) {
problems.push(
`va_parmap "${entry}" was ignored: "${column}" is not a transform - expected e.g. ADD(QUARTER_START,-1).`
)
continue
}
// The column name is interpolated into a SAS WHERE clause, so only a plain
// SAS name is accepted - anything else is refused rather than mapped.
if (!/^[A-Za-z_][A-Za-z0-9_]{0,31}$/.test(column)) {
problems.push(
`va_parmap "${entry}" was ignored: "${column}" is not a valid SAS column name.`
)
continue
}
const transform = transformSpec
? parseTransform(transformSpec, entry, problems)
: undefined
if (transformSpec && !transform) continue
// One VA parameter filters one column. A second entry for the same parameter
// is ambiguous, so the FIRST one wins and the conflict is reported - the
// alternative (last one wins, silently) would apply a filter the author
// never sees the origin of. Two parameters mapping to the same column is
// fine and is how two report objects filter one column from their own
// parameters.
const existing = map.get(key)
if (existing) {
if (existing.column.toUpperCase() !== column.toUpperCase()) {
problems.push(
`va_parmap "${entry}" was ignored: VA parameter "${key}" already maps to ${existing.column}, and a parameter can filter only one column.`
)
}
continue
}
map.set(key, { column, transform })
}
return { map, problems }
}
/**
* Parses `OP(arg[, arg])`, returning undefined (and pushing a problem line) when
* the operation or one of its arguments is not in the allow-list.
*
* ADD takes an anchor and an optional offset; RANGE takes a period and an
* optional offset. The offset is whole periods, negative for earlier.
*/
function parseTransform(
spec: string,
entry: string,
problems: string[]
): VaTransform | undefined {
const match = /^([A-Za-z]+)\s*\(([^)]*)\)$/.exec(spec)
if (!match) {
problems.push(
`va_parmap "${entry}" was ignored: "${spec}" is not a transform - expected e.g. ADD(QUARTER_START,-1).`
)
return undefined
}
const op = match[1].toUpperCase()
const args = match[2]
.split(',')
.map((arg) => arg.trim())
.filter((arg) => arg.length > 0)
const offsetArg = args[1] === undefined ? '0' : args[1]
if (!/^-?\d+$/.test(offsetArg)) {
problems.push(
`va_parmap "${entry}" was ignored: the offset "${offsetArg}" is not a whole number of periods.`
)
return undefined
}
const offset = Number(offsetArg)
if (op === 'ADD') {
const anchor = (args[0] || '').toUpperCase()
if (!VA_ANCHORS.includes(anchor as VaAnchor)) {
problems.push(
`va_parmap "${entry}" was ignored: "${args[0] || ''}" is not a known anchor (${VA_ANCHORS.join(', ')}).`
)
return undefined
}
return { op: 'ADD', anchor: anchor as VaAnchor, offset }
}
if (op === 'RANGE') {
const period = (args[0] || '').toUpperCase()
if (!VA_PERIODS.includes(period as VaPeriod)) {
problems.push(
`va_parmap "${entry}" was ignored: "${args[0] || ''}" is not a known period (${VA_PERIODS.join(', ')}).`
)
return undefined
}
return { op: 'RANGE', period: period as VaPeriod, offset }
}
problems.push(
`va_parmap "${entry}" was ignored: "${match[1]}" is not a known transform (ADD, RANGE).`
)
return undefined
}
@@ -0,0 +1,194 @@
import {
applyVaTransform,
isRangeTransform,
VA_ANCHORS,
VA_PERIODS,
VaTransform
} from './va-parmap-transform'
/** Local y-m-d, so assertions do not depend on the runner's timezone offset. */
const ymd = (d: Date) =>
`${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(
d.getDate()
).padStart(2, '0')}`
const date = (y: number, m: number, d: number) => new Date(y, m - 1, d)
describe('va_parmap transforms', () => {
it('exposes only allow-listed anchors and periods', () => {
expect(VA_ANCHORS).toContain('QUARTER_START')
expect(VA_PERIODS).toEqual(['QUARTER', 'MONTH', 'YEAR'])
})
describe('ADD - the value plus one anchor beside it', () => {
it('adds the previous quarter start, which is the reported requirement', () => {
// intnx('qtr', dt, -1, 'b') for a value on a quarter boundary
const dates = applyVaTransform(date(2026, 10, 1), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2026-10-01', '2026-07-01'])
})
it('resolves a mid-quarter value to its own value plus the previous quarter start', () => {
const dates = applyVaTransform(date(2026, 8, 15), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2026-08-15', '2026-04-01'])
})
it('de-duplicates when the value already sits on the anchor', () => {
// 2026-01-01 is a quarter start, so "0 quarters back" adds nothing
const dates = applyVaTransform(date(2026, 1, 1), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: 0
})
expect(dates.map(ymd)).toEqual(['2026-01-01'])
})
it('resolves the other anchors at their boundaries', () => {
const at = date(2026, 8, 15)
expect(
ymd(
applyVaTransform(at, {
op: 'ADD',
anchor: 'QUARTER_END',
offset: 0
})[1]
)
).toBe('2026-09-30')
expect(
ymd(
applyVaTransform(at, { op: 'ADD', anchor: 'MONTH_END', offset: 0 })[1]
)
).toBe('2026-08-31')
expect(
ymd(
applyVaTransform(at, {
op: 'ADD',
anchor: 'YEAR_START',
offset: -1
})[1]
)
).toBe('2025-01-01')
expect(
ymd(
applyVaTransform(at, { op: 'ADD', anchor: 'YEAR_END', offset: 0 })[1]
)
).toBe('2026-12-31')
})
it('handles February and leap years at month end', () => {
expect(
ymd(
applyVaTransform(date(2026, 3, 1), {
op: 'ADD',
anchor: 'MONTH_END',
offset: -1
})[1]
)
).toBe('2026-02-28')
expect(
ymd(
applyVaTransform(date(2024, 3, 15), {
op: 'ADD',
anchor: 'MONTH_END',
offset: -1
})[1]
)
).toBe('2024-02-29')
})
it('crosses a year boundary when the offset takes it out of the year', () => {
const dates = applyVaTransform(date(2027, 1, 1), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2027-01-01', '2026-10-01'])
})
})
describe('RANGE - a closed period', () => {
it('covers the previous quarter', () => {
const dates = applyVaTransform(date(2026, 8, 15), {
op: 'RANGE',
period: 'QUARTER',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2026-04-01', '2026-06-30'])
})
it('covers the current month and the previous year', () => {
expect(
applyVaTransform(date(2026, 8, 15), {
op: 'RANGE',
period: 'MONTH',
offset: 0
}).map(ymd)
).toEqual(['2026-08-01', '2026-08-31'])
expect(
applyVaTransform(date(2026, 8, 15), {
op: 'RANGE',
period: 'YEAR',
offset: -1
}).map(ymd)
).toEqual(['2025-01-01', '2025-12-31'])
})
it('reports itself as a range so the caller picks BETWEEN', () => {
expect(
isRangeTransform({ op: 'RANGE', period: 'QUARTER', offset: 0 })
).toBe(true)
expect(
isRangeTransform({
op: 'ADD',
anchor: 'QUARTER_START',
offset: 0
} as VaTransform)
).toBe(false)
})
})
describe('time of day', () => {
it('gives a datetime range whole days, not the value clock time', () => {
// A BETWEEN that starts at 13:45 on day one loses the earlier part of that
// day, and one that ends at 13:45 on the last day loses the rest of it.
const value = new Date(2026, 7, 15, 13, 45, 30)
const [start, end] = applyVaTransform(
value,
{ op: 'RANGE', period: 'QUARTER', offset: 0 },
true
)
expect([
start.getHours(),
start.getMinutes(),
start.getSeconds()
]).toEqual([0, 0, 0])
expect([end.getHours(), end.getMinutes(), end.getSeconds()]).toEqual([
23, 59, 59
])
expect(ymd(start)).toBe('2026-07-01')
expect(ymd(end)).toBe('2026-09-30')
})
it('keeps the time for datetime columns and drops it otherwise', () => {
const value = new Date(2026, 9, 1, 13, 45, 30)
const add: VaTransform = {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
}
const kept = applyVaTransform(value, add, true)[1]
expect([kept.getHours(), kept.getMinutes(), kept.getSeconds()]).toEqual([
13, 45, 30
])
const dropped = applyVaTransform(value, add, false)[1]
expect([dropped.getHours(), dropped.getMinutes()]).toEqual([0, 0])
})
})
})
@@ -0,0 +1,151 @@
/**
* Anchors and periods a `va_parmap` transform can resolve a report value to.
*
* The names describe the result rather than mirroring SAS `intnx`, because the
* arithmetic runs in the browser: emulating `intnx`'s alignment letters (`b`,
* `e`, `s`, `m`, never mind week alignment and fiscal quarters) would invite
* assumptions we could not honour. Each anchor below is one small function, so
* adding one is a function plus a test row.
*/
export const VA_ANCHORS = [
'QUARTER_START',
'QUARTER_END',
'MONTH_START',
'MONTH_END',
'YEAR_START',
'YEAR_END'
] as const
export type VaAnchor = (typeof VA_ANCHORS)[number]
export const VA_PERIODS = ['QUARTER', 'MONTH', 'YEAR'] as const
export type VaPeriod = (typeof VA_PERIODS)[number]
/**
* A parsed transform: the operation plus its validated arguments.
*
* `ADD` yields the value and one anchor beside it (a list of dates), which is
* how "this period's row plus the previous quarter's" is expressed. `RANGE`
* yields a closed period (two dates) for a BETWEEN.
*/
export type VaTransform =
| { op: 'ADD'; anchor: VaAnchor; offset: number }
| { op: 'RANGE'; period: VaPeriod; offset: number }
/** True when the transform produces a closed range rather than a list. */
export function isRangeTransform(transform: VaTransform): boolean {
return transform.op === 'RANGE'
}
/** Local midnight on the first day of the month `offset` months from `date`. */
function monthStart(date: Date, offset: number): Date {
return new Date(date.getFullYear(), date.getMonth() + offset, 1)
}
/** Local midnight on the last day of the month `offset` months from `date`. */
function monthEnd(date: Date, offset: number): Date {
return new Date(date.getFullYear(), date.getMonth() + offset + 1, 0)
}
/** First day of the quarter `offset` quarters from the one containing `date`. */
function quarterStart(date: Date, offset: number): Date {
const firstMonthOfQuarter = 3 * Math.floor(date.getMonth() / 3)
return new Date(date.getFullYear(), firstMonthOfQuarter + offset * 3, 1)
}
/** Last day of the quarter `offset` quarters from the one containing `date`. */
function quarterEnd(date: Date, offset: number): Date {
// Day 0 of a month is the last day of the month before it: calendar
// construction, so a daylight-saving transition inside the quarter cannot
// shift the result by an hour the way subtracting 24 hours would.
const nextStart = quarterStart(date, offset + 1)
return new Date(nextStart.getFullYear(), nextStart.getMonth(), 0)
}
/** First day of the year `offset` years from the one containing `date`. */
function yearStart(date: Date, offset: number): Date {
return new Date(date.getFullYear() + offset, 0, 1)
}
/** Last day of the year `offset` years from the one containing `date`. */
function yearEnd(date: Date, offset: number): Date {
return new Date(date.getFullYear() + offset, 11, 31)
}
/** Resolve one anchor, keeping the time of day of `date` for datetime columns. */
function resolveAnchor(
anchor: VaAnchor,
date: Date,
offset: number,
keepTime: boolean
): Date {
let resolved: Date
switch (anchor) {
case 'QUARTER_START':
resolved = quarterStart(date, offset)
break
case 'QUARTER_END':
resolved = quarterEnd(date, offset)
break
case 'MONTH_START':
resolved = monthStart(date, offset)
break
case 'MONTH_END':
resolved = monthEnd(date, offset)
break
case 'YEAR_START':
resolved = yearStart(date, offset)
break
default:
resolved = yearEnd(date, offset)
break
}
if (keepTime) {
resolved.setHours(date.getHours(), date.getMinutes(), date.getSeconds(), 0)
}
return resolved
}
/**
* The dates a transform adds to a mapping, in order.
*
* `ADD` returns the report's own value plus the anchor beside it, de-duplicated -
* so a value that already sits on the anchor yields one date, and the clause
* stays an equality rather than an `IN` with a repeat. `RANGE` returns the
* period's first and last day.
*
* `keepTime` carries the time of day across for datetime columns, since a
* DATETIME filter compares to the second; date columns use midnight.
*/
export function applyVaTransform(
value: Date,
transform: VaTransform,
keepTime = false
): Date[] {
if (transform.op === 'RANGE') {
// A range covers whole days: the bounds are the period's own, never the
// value's clock time. Reusing keepTime here would start the range mid-morning
// on day one and end it mid-morning on the last day, dropping the rest of
// both boundary days from a BETWEEN.
const start = resolveAnchor(
(transform.period + '_START') as VaAnchor,
value,
transform.offset,
false
)
const end = resolveAnchor(
(transform.period + '_END') as VaAnchor,
value,
transform.offset,
false
)
if (keepTime) end.setHours(23, 59, 59, 0)
return [start, end]
}
const anchor = resolveAnchor(
transform.anchor,
value,
transform.offset,
keepTime
)
return anchor.getTime() === value.getTime() ? [value] : [value, anchor]
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "dcfrontend",
"version": "7.18.0",
"version": "7.21.0",
"description": "Data Controller",
"devDependencies": {
"@nogoo9/gitleaks": "8.30.1-post.2",
+1
View File
@@ -11,5 +11,6 @@
"noNestedMacros": true,
"hasMacroParentheses": true,
"strictMacroDefinition": true,
"ignoredMacros": ["webout", "mpeinit", "mpeterm"],
"ignoreList":["sasjs/scratch"]
}
+1 -1
View File
@@ -1,7 +1,7 @@
{
"fromjs": [
{
"ADMIN": "AllUsers",
"ADMIN": "all-users",
"DCPATH": "/tmp/dcdata",
"OUTFMT": "JSON"
}
@@ -13,10 +13,12 @@
// The rules mirror services/hooks/mpe_derived_postedit.sas, in its order:
// normalise the incoming rows, reject a duplicate name and a bad field, then
// check the dependency graph that the resulting configuration would have (the
// incoming rows plus the current rows they do not replace). As on the server,
// the collision between VAR_NM and a physical column is left to the read-time
// engine, which runs with the target library assigned - the hook does not have
// it here either.
// incoming rows plus the current rows they do not replace). The field rules
// are core's mp_derivedvarcheck, which the server hook calls and the read-time
// engine runs too - this file spells them out a second time, so a change there
// has to be mirrored here. As on the server, the collision between VAR_NM and
// a physical column is left to the read-time engine, which runs with the target
// library assigned - the hook does not have it here either.
// ─────────────────────────────────────────────────────────────────────────────
;(function mpeDerivedPostedit() {
@@ -83,9 +85,45 @@
seen.add(key)
}
// ─── Field level checks, the rules the engine applies at read time ─────────
// C0 controls and DEL: what a single-line label cannot carry
// ─── Field level checks ────────────────────────────────────────────────────
// Two passes, in the order the SAS hook runs them. The first is the fields
// this hook owns - the control table key, which the loader normalises but
// does not require, and ACTIVE, the picker pair the EDIT screen offers.
for (const row of rows) {
if (isDeleted(row)) continue
const name = field(row, 'VAR_NM')
let reason = ''
if (!field(row, 'LIBREF') || !field(row, 'DSN'))
reason = 'LIBREF and DSN are required'
// ACTIVE is a picker in the EDIT screen (HARDSELECT over the
// MPE_SELECTBOX pair) - the same pair the hook enforces
else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
reason = 'ACTIVE must be 1 or 0'
if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
}
// The second pass is the rules for the derived variable itself, which are
// core's (mp_derivedvarcheck): the engine runs the same macro at read time,
// so a definition is measured against one rule list wherever it is checked.
// C0 controls and DEL: what a label cannot carry once a line break has been
// flattened to a space, which is what the engine does with it
const CONTROL_CHARS = /[\u0000-\u001F\u007F]/
const flatten = (v) => v.replace(/[\r\n]+/g, ' ')
const AUTOMATIC_VARS = ['_N_', '_ERROR_', '_IORC_']
// the capabilities refused by name - a function name cannot be assembled at
// run time, so naming them is a control the expression cannot evade
const DENY_FUNCS = new RegExp(
'\\b(DOSUBL|JAVAOBJ|JAVAMETA|SYMGET|SYMPUTX?|SYSGET|GETOPTION' +
'|PATHNAME|FILEEXIST|FOPEN|FREAD|FGET|FCLOSE' +
'|DOPEN|DREAD|DNUM|MOPEN|SLEEP' +
'|FILENAME|FDELETE|FCOPY|DCREATE|DLCREATE)\\s*\\('
)
// quoted literals are data, not calls
const unquoted = (v) =>
v.replace(/'([^']|'')*'/g, ' ').replace(/"([^"]|"")*"/g, ' ')
// true when the value carries a character outside the allowed set
const hasInvalidChar = (value, chars) =>
@@ -93,8 +131,6 @@
for (const row of rows) {
if (isDeleted(row)) continue
const libref = field(row, 'LIBREF')
const dsn = field(row, 'DSN')
const name = field(row, 'VAR_NM')
const type = field(row, 'VAR_TYPE')
const len = Number(colVal(row, 'VAR_LEN'))
@@ -103,19 +139,16 @@
const logic = strip(colVal(row, 'VAR_LOGIC'))
let reason = ''
if (!libref || !dsn) reason = 'LIBREF and DSN are required'
else if (!name) reason = 'VAR_NM is required'
if (!name) reason = 'VAR_NM is required'
else if (hasInvalidChar(name, NAME_CHARS))
reason = 'VAR_NM must be a valid SAS name'
else if ('0123456789'.includes(name[0]))
reason = 'VAR_NM must not start with a digit'
else if (AUTOMATIC_VARS.includes(name))
reason = 'VAR_NM is an automatic variable of the data step'
else if (name === DELETE_MARK) reason = 'VAR_NM is reserved'
else if (type !== 'C' && type !== 'N')
reason = 'VAR_TYPE must be C or N'
// ACTIVE is a picker in the EDIT screen (HARDSELECT over the
// MPE_SELECTBOX pair) - the same pair mpe_derived_check.sas enforces
else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
reason = 'ACTIVE must be 1 or 0'
else if (!Number.isInteger(len) || len < 1)
reason = 'VAR_LEN must be a positive integer'
else if (type === 'C' && len > 32767)
@@ -132,20 +165,24 @@
reason = 'VAR_FMT must be a format name, without ; & % or /* */'
else if (hasInvalidChar(fmt, FMT_CHARS))
reason = 'VAR_FMT must be a format name'
// free text, in any script - what is refused is anything that cannot be
// a single-line column label, and a value the engine would cut to 256
else if (CONTROL_CHARS.test(label))
reason =
'VAR_LABEL must be plain text - no line breaks or control characters'
// free text, in any script - what is refused is a control character,
// and a value the engine would cut to 256
else if (CONTROL_CHARS.test(flatten(label)))
reason = 'VAR_LABEL must be plain text - no control characters'
else if ([...label].length > 256)
reason = 'VAR_LABEL exceeds 256 characters'
else if (!logic) reason = 'VAR_LOGIC is required'
else if (logic.length > 4000)
reason = 'VAR_LOGIC exceeds 4000 characters'
else if (logic.includes(';'))
reason = 'VAR_LOGIC must be a single expression - remove the ;'
else if (logic.includes('&') || logic.includes('%'))
reason = 'VAR_LOGIC must not contain macro triggers'
else if (logic.includes('/*') || logic.includes('*/'))
reason = 'VAR_LOGIC must not contain comments'
else if (DENY_FUNCS.test(unquoted(logic).toUpperCase()))
reason =
'VAR_LOGIC may not call a function that submits code or reaches the file system, the environment or the macro symbol table'
if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
}
+10 -4
View File
@@ -7,7 +7,7 @@
"name": "dc-sas",
"dependencies": {
"@sasjs/cli": "6.2.0",
"@sasjs/core": "6.1.0"
"@sasjs/core": "7.0.1"
}
},
"node_modules/@asamuzakjp/css-color": {
@@ -199,6 +199,12 @@
"@sasjs/lint": "^4.3.0"
}
},
"node_modules/@sasjs/cli/node_modules/@sasjs/core": {
"version": "6.1.4",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.4.tgz",
"integrity": "sha512-vIF2LIftxPWg4CvO3i5raEfrHFHY/6S+MqVGpjRR71dgjLRJEYNyPJgGqD6BCZwHv5V6/2nDnBi/BIIMiLXfzA==",
"license": "MIT"
},
"node_modules/@sasjs/cli/node_modules/@sasjs/utils": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.1.0.tgz",
@@ -235,9 +241,9 @@
}
},
"node_modules/@sasjs/core": {
"version": "6.1.0",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.0.tgz",
"integrity": "sha512-U0Ekq/TXVMquYV+AG6urBQSUgqUvV37yduufy68QQVgDj3PJiLUJDcOXKOGbhWGD+Nv877VuPCviGYqrqbjXvg==",
"version": "7.0.1",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-7.0.1.tgz",
"integrity": "sha512-M75475GuOMTmbmrB8V/Hgcwm2j3PdIHlgmkbjtT88ALSQuzRPBAjEK2IDDYIUn6rsVJG9aZWvUpRV7fzrOjK/w==",
"license": "MIT"
},
"node_modules/@sasjs/lint": {
+1 -1
View File
@@ -34,7 +34,7 @@
"private": true,
"dependencies": {
"@sasjs/cli": "6.2.0",
"@sasjs/core": "6.1.0"
"@sasjs/core": "7.0.1"
},
"overrides": {
"nanoid": "3.3.18"
+49
View File
@@ -0,0 +1,49 @@
/**
@file
@brief builds the message mpe_execute aborts with when a child fails
@details The child's own message is carried into the abort text as it stands,
so a comma, a blank or a quote in it survives the trip. The text is built in
a data step rather than with %str at the abort call site, where a comma in the
child's message split the call into extra positional parameters.
The child message is named rather than passed by value: a value argument
would itself be split at the first comma before this macro ever ran. The
result is written to the global symbol table under a name this macro owns -
_mpe_abortmsg - so a call cannot overwrite a caller's own global. Each call
replaces the previous value; read it before the next call.
@param [in] programvar Name of the macro variable holding the _PROGRAM
endpoint - a name, not the value, so a comma in a path cannot split the call
@param [in] statevar Name of the macro variable holding the recorded state
@param [in] msgvar Name of the macro variable holding the child's message
@author 4GL Apps Ltd
@copyright 4GL Apps Ltd. This code may only be used within Data Controller
and may not be re-distributed or re-sold without the express permission of
4GL Apps Ltd.
**/
%macro mpe_abortmessage(programvar=program,statevar=state,msgvar=msg);
data _null_;
/* explicit lengths: an inline symget() is typed by the data step default,
which truncates a long child message */
length _prefix $32767 _msg $32767 _out $32767;
/* every value is read by name: interpolating program or state into this
data step would let a %, & or quote in an endpoint name be macro-resolved
or break the literal */
/* cat, not cats: cats strips the blanks around each argument, which would
close up the space before 'did' */
_prefix=cat(symget("&programvar"),' did not complete: state=',
symget("&statevar"));
_msg=symget("&msgvar");
/* plain concatenation, not CATX: the child's message is kept byte for
byte, blank and all, so an exact comparison downstream can rely on it */
/* trim the prefix: it is declared as a character variable, so it arrives
padded to length and the child's message would land past the buffer end */
if missing(_msg) then _out=trim(_prefix);
else _out=trim(_prefix)||' '||_msg;
call symputx('_mpe_abortmsg',_out,'g');
run;
%mend mpe_abortmessage;
+41 -5
View File
@@ -13,7 +13,7 @@
than the abort's own.
An unassigned outref (a child that died before writing a response) records a
FAIL rather than raising an unassigned-fileref error in the data step.
FAIL rather than letting the data step fail on an unassigned fileref.
Usage:
@@ -47,17 +47,21 @@
%let abortseen=0;
%if %sysfunc(fileref(&outref))>0 %then %do;
%put WARNING: &sysmacroname: fileref &outref is not assigned - the child left no response to read;
%put %str(WARN)ING: &sysmacroname: fileref &outref is not assigned - the child left no response to read;
%end;
%else %if %sysfunc(fileexist(%sysfunc(pathname(&outref)))) ne 1 %then %do;
/* the fileref exists but nothing was written to it: a child that aborted
before it produced a response. Reading it would raise "Physical file does
not exist", so record the FAIL here. */
%put WARNING: &sysmacroname: &outref has no response written to it - the child produced nothing;
%put %str(WARN)ING: &sysmacroname: &outref has no response written to it - the child produced nothing;
%end;
%else %do;
data _null_;
length line $32767 got $256;
/* got is the same width as line: the strip of the appended log extract can
only fire if the whole message reached the buffer, and a hook message can
run past 256 characters */
length line $32767 got $32767;
length _marker 8 _from 8;
retain prxid 0 inabort 0 found 0 abortseen 0;
if _n_=1 then prxid=prxparse('/"MSG" *: *"([^"]*)"/');
infile &outref end=eof lrecl=32767;
@@ -77,6 +81,38 @@ data _null_;
end;
end;
call symputx('abortseen',abortseen,'l');
/* mp_abort appends "Log Extract:" and the surrounding log to the message it
returns; what is compared is the message the hook raised, not that tail, so
the expectation stays an exact match. Cut at the LAST marker: a message
may legitimately contain the text "Log Extract:" itself, and only the final
occurrence is the appended one.
A message that *ends* with the marker and has no extract appended is cut
short - accepted, because mp_abort always appends one and the alternative
is to guess which occurrence is the producer's */
_marker=0;
_from=1;
do while(index(substr(got,_from),'Log Extract:')>0);
_marker=_from+index(substr(got,_from),'Log Extract:')-1;
_from=_marker+1;
end;
if _marker>0 then do;
if _marker=1 then got='';
else got=substr(got,1,_marker-1);
/* mp_abort appends the extract as cats(msg,'\n\nLog Extract:\n',logmsg)
and then escapes the message, so the separator arrives as two escaped
newlines - six characters once the backslashes are doubled.
Remove exactly that tail: a message that ends in its own escaped newline
keeps it, which a loop over trailing escapes got wrong. */
if length(got)>=6 and substr(got,length(got)-5,6)='\\n\\n' then
got=substr(got,1,length(got)-6);
else if length(got)>=4 and substr(got,length(got)-3,4)='\n\n' then
got=substr(got,1,length(got)-4);
end;
/* unescape what mp_abort escaped, so the comparison is against the text the
hook raised rather than against its JSON form. This is independent of the
tail strip: a response that carries no appended extract is still escaped. */
got=tranwrd(got,'\"','"');
got=tranwrd(got,'\\','\');
if found=1 then call symputx('seen',got,'l');
stop;
run;
@@ -84,7 +120,7 @@ run;
%if &abortseen=1 %then %do;
%if "%superq(msg)"="0" %then %let ok=1;
%else %if "&seen"="%superq(msg)" %then %let ok=1;
%else %if "%superq(seen)"="%superq(msg)" %then %let ok=1;
%end;
%if "&seen" ne "" %then %put NOTE: &sysmacroname: sasjsAbort MSG=&seen;
+125 -3
View File
@@ -4,8 +4,9 @@
@details The macro decides PASS/FAIL by scanning the child's response text, so
the cases that matter can be built as files instead of services: an abort with
the expected message, an abort with a different one, no abort at all, and a
fileref the child never wrote to.
the expected message, an abort with a different one, no abort at all, a
fileref the child never wrote to, and a message long enough to test the width
of the capture buffer.
Each case asserts on the PASS/FAIL the macro recorded, so a regression in the
scan fails here rather than silently passing an abort assertion in a service
@@ -38,6 +39,9 @@
%let rc_split=work.assert_rc5;
%let rc_long=work.assert_rc6;
%let rc_noref=work.assert_rc7;
%let rc_bslash=work.assert_rc8;
%let rc_marker=work.assert_rc9;
%let rc_doubled=work.assert_rc11;
/**
* Test 1 - an abort whose MSG matches is a PASS
@@ -101,7 +105,85 @@ run;
)
/**
* Test 5 - the recorded results
* Test 5 - a message longer than the capture buffer is compared in full
*
* The strip of the appended log extract only works if the whole message made
* it into the capture buffer, so this case is longer than 256 characters -
* the width the buffer had while the strip was added.
*/
%let f5=%mf_getuniquefileref();
%let longmsg=;
data _null_;
file &f5 termstr=crlf;
length msgtxt $1000;
msgtxt=repeat('a clause that runs on ',30);
put '{"sasjsAbort": [{"MSG": "' msgtxt +(-1) '\n\nLog Extract:\nNOTE: a log line"}]}';
call symputx('longmsg',msgtxt,'l');
run;
%mpe_assert_abort(
desc=an abort with a message past the capture buffer,
outref=&f5,
msg=%superq(longmsg),
outds=&rc_long
)
/**
* Test 6 - a message that ends in a backslash keeps it
*
* mp_abort escapes a real backslash as two, so the captured text ends in the
* escape of that backslash followed by the escaped newlines of the log
* extract. Only the newline escapes may be stripped.
*/
%let f6=%mf_getuniquefileref();
data _null_;
file &f6 termstr=crlf;
put '{"sasjsAbort": [{"MSG": "path C:\\temp\\\n\nLog Extract:\nNOTE: a log line"}]}';
run;
%let msg_trail=path C:\temp\;
%mpe_assert_abort(
desc=an abort whose message ends in a backslash,
outref=&f6,
msg=%superq(msg_trail),
outds=&rc_bslash
)
/**
* Test 7 - a message that itself contains "Log Extract:" is not cut at it
*/
%let f8=%mf_getuniquefileref();
data _null_;
file &f8 termstr=crlf;
put '{"sasjsAbort": [{"MSG": "the check said Log Extract: once\n\nLog Extract:\nNOTE: a log line"}]}';
run;
%mpe_assert_abort(
desc=an abort whose message contains the extract marker,
outref=&f8,
msg=the check said Log Extract: once,
outds=&rc_marker
)
/**
* Test 8 - the six-character separator, as the estate emits it
*/
%let f10=%mf_getuniquefileref();
data _null_;
file &f10 termstr=crlf;
put '{"sasjsAbort": [{"MSG": "doubled\\n\\nLog Extract:\nNOTE: a log line"}]}';
run;
%mpe_assert_abort(
desc=an abort whose separator is doubled,
outref=&f10,
msg=doubled,
outds=&rc_doubled
)
/**
* Test 9 - the recorded results
*/
@@ -121,6 +203,22 @@ data _null_;
set &rc_noref;
call symputx('res_noref',test_result,'g');
run;
data _null_;
set &rc_long;
call symputx('res_long',test_result,'g');
run;
data _null_;
set &rc_bslash;
call symputx('res_bslash',test_result,'g');
run;
data _null_;
set &rc_marker;
call symputx('res_marker',test_result,'g');
run;
data _null_;
set &rc_doubled;
call symputx('res_doubled',test_result,'g');
run;
%mp_assert(
iftrue=("&res_any"="PASS"),
@@ -143,6 +241,30 @@ run;
%mp_assert(
iftrue=("&res_long"="PASS"),
desc=%str(Checking an abort message longer than the capture buffer matches in full),
outds=work.test_results
)
%mp_assert(
iftrue=("&res_bslash"="PASS"),
desc=%str(Checking a message ending in a backslash compares in full),
outds=work.test_results
)
%mp_assert(
iftrue=("&res_marker"="PASS"),
desc=%str(Checking a message containing the extract marker is not cut at it),
outds=work.test_results
)
%mp_assert(
iftrue=("&res_doubled"="PASS"),
desc=%str(Checking the doubled separator is stripped),
outds=work.test_results
)
%mp_assert(
iftrue=("&res_noref"="FAIL"),
desc=%str(Checking an unassigned outref fails the assertion, not the step),
+13 -3
View File
@@ -49,9 +49,19 @@
being maintained. A cycle is rejected.
Semicolons, macro triggers and comments are rejected, so a row can only ever
be an expression. A derived variable may not reuse the name of a physical
column, and may not reference itself: an intra-row calculation would imply a
retained value, which depends on the order rows happen to arrive in.
be an expression, and an expression that would let the row submit code of its
own - DOSUBL, or a function reaching the file system, the environment or the
macro symbol table - is refused by name.
The field level rules themselves are core's: mp_derivedvarcheck scans the
definitions and mp_derivedvars calls it, which is the same macro the MPE_DERIVED
post edit hook runs at submit time (services/hooks/mpe_derived_postedit). A
row is therefore measured against one rule list, whether it is being staged
or read.
A derived variable may not reuse the name of a physical column, and may not
reference itself: an intra-row calculation would imply a retained value,
which depends on the order rows happen to arrive in.
<h4> SAS Macros </h4>
@li mp_derivedvars.sas
-75
View File
@@ -1,75 +0,0 @@
/**
@file
@brief Field level checks for a staged MPE_DERIVED submission
@details Writes one row per staged row that breaks a rule, with the reason in
REASON, so a caller can report the first failure and abort. Rows marked for
deletion are skipped - they are being removed, so their fields are not loaded.
The rules are the ones %mpe_derived applies at read time, so a submission
cannot stage a row the engine would then refuse.
@param inds the staged rows, already normalised by the caller
@param outds the dataset to create: one row per failing row, plus REASON
@author 4GL Apps Ltd
@copyright 4GL Apps Ltd. This code may only be used within Data Controller
and may not be re-distributed or re-sold without the express permission of
4GL Apps Ltd.
**/
%macro mpe_derived_check(inds=work.staging_ds, outds=work.derived_check);
data &outds;
set &inds;
length reason $200;
if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
call missing(reason);
if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
else if var_nm='' then reason='VAR_NM is required';
else if verify(strip(var_nm),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_')>0
then reason='VAR_NM must be a valid SAS name';
else if verify(substr(var_nm,1,1),'0123456789')=0
then reason='VAR_NM must not start with a digit';
else if var_nm='_____DELETE__THIS__RECORD_____'
then reason='VAR_NM is reserved';
else if var_type not in ('C','N')
then reason='VAR_TYPE must be C or N';
/* ACTIVE is a picker in the EDIT screen (HARDSELECT over the MPE_SELECTBOX
pair) - the same pair is enforced here, so a submission that bypasses the
grid cannot land a value the screen would never offer. */
else if missing(active) or active not in (0,1)
then reason='ACTIVE must be 1 or 0';
else if missing(var_len) or var_len ne int(var_len) or var_len<1
then reason='VAR_LEN must be a positive integer';
else if var_type='C' and var_len>32767
then reason='VAR_LEN exceeds 32767 for a character variable';
else if var_type='N' and var_len>8
then reason='VAR_LEN exceeds 8 for a numeric variable';
else if index(var_fmt,';')>0 or indexc(var_fmt,'&%')>0
or index(var_fmt,'/*')>0 or index(var_fmt,'*/')>0
then reason='VAR_FMT must be a format name, without ; & % or /* */';
else if verify(strip(var_fmt),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.')>0
then reason='VAR_FMT must be a format name';
/* VAR_LABEL is free text: any character is allowed, in any script, and the
quotes in it are escaped by %mp_derivedvars when the attrib statement is
built. What is refused is anything that cannot be a single-line column
label - a control character or a line break - and a value long enough to
be cut down silently, since the engine truncates the label to 256. */
else if anycntrl(var_label)>0
then reason='VAR_LABEL must be plain text - no line breaks or control characters';
else if klength(var_label)>256
then reason='VAR_LABEL exceeds 256 characters';
else if missing(var_logic)
then reason='VAR_LOGIC is required';
else if index(var_logic,';')>0
then reason='VAR_LOGIC must be a single expression - remove the ;';
else if indexc(var_logic,'&%')>0
then reason='VAR_LOGIC must not contain macro triggers';
else if index(var_logic,'/*')>0 or index(var_logic,'*/')>0
then reason='VAR_LOGIC must not contain comments';
if not missing(reason) then output;
run;
%mend mpe_derived_check;
-194
View File
@@ -1,194 +0,0 @@
/**
@file
@brief testing the mpe_derived_check macro
@details Checks each field level rule the MPE_DERIVED post edit hook applies
to a staged submission. The macro is exercised directly so the rules can be
probed with values a service cannot carry: VAR_LABEL is varchar(256) in the
table and a staging dataset declares the same length, so a longer label is
truncated before any check would see it.
<h4> SAS Macros </h4>
@li mf_nobs.sas
@li mp_assert.sas
@li mpe_derived_check.sas
@author 4GL Apps Ltd
@copyright 4GL Apps Ltd. This code may only be used within Data Controller
and may not be re-distributed or re-sold without the express permission of
4GL Apps Ltd.
**/
/**
* Test 1 - one row per failing rule, and only failing rows
*/
data work.staging_ds;
length libref $8 dsn $32 var_nm $32 var_type $1 var_len 8
var_fmt $32 var_label $4000 var_deps $4000 var_logic $4000
active 8 _____DELETE__THIS__RECORD_____ $3;
libref='DCTEST';
dsn='DC_DERIV';
/* valid unless a row says otherwise - ACTIVE is mandatory in the screen and
the check enforces the same pair here */
active=1;
_____DELETE__THIS__RECORD_____='No';
/* meets every rule */
var_nm='GOOD';
var_type='N';
var_len=8;
var_fmt='';
var_label='a plain label';
var_deps='';
var_logic='1';
output;
/* delete marked rows are being removed, so their fields are not loaded */
var_nm='DELME';
var_label='first line'||'0A'x||'second line';
_____DELETE__THIS__RECORD_____='Yes';
output;
_____DELETE__THIS__RECORD_____='No';
/* free text is allowed, including quotes and punctuation */
var_nm='FREETXT';
var_label='It''s "quoted", plain text: 100 percent';
output;
/* a line break cannot be a single line column label */
var_nm='LBLBREAK';
var_label='first line'||'0A'x||'second line';
output;
/* longer than the table holds, so it would be cut down silently */
var_nm='LBLLONG';
var_label=repeat('x',300);
output;
/* a format name, not an expression */
var_nm='BADFMT';
var_fmt='DATE9;';
var_label='fmt';
output;
/* one expression, no statement separator, no macro triggers */
var_nm='BADLOGIC';
var_fmt='';
var_logic='x; y';
output;
/* C or N */
var_nm='BADTYPE';
var_logic='1';
var_type='X';
output;
/* ACTIVE is a picker in the screen: only 1 or 0 */
var_nm='BADACTIVE';
var_type='C';
var_logic='1';
active=2;
output;
stop;
run;
%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
/**
* Test 2 - the reported rows, rule by rule
*/
data work.expected;
length var_nm $32 reason $200;
var_nm='LBLBREAK';
reason='VAR_LABEL must be plain text - no line breaks or control characters';
output;
var_nm='LBLLONG';
reason='VAR_LABEL exceeds 256 characters';
output;
var_nm='BADFMT';
reason='VAR_FMT must be a format name, without ; & % or /* */';
output;
var_nm='BADLOGIC';
reason='VAR_LOGIC must be a single expression - remove the ;';
output;
var_nm='BADTYPE';
reason='VAR_TYPE must be C or N';
output;
var_nm='BADACTIVE';
reason='ACTIVE must be 1 or 0';
output;
run;
proc sql;
create table work.cmp as
select e.var_nm
,case when a.reason=e.reason then 1 else 0 end as same
from work.expected e
left join work.derived_check a
on upcase(strip(a.var_nm))=upcase(e.var_nm);
quit;
data _null_;
set work.cmp;
call symputx(cats('rule_',var_nm),same,'l');
run;
%mp_assert(
iftrue=(%mf_nobs(work.derived_check)=6),
desc=%str(Checking only the six failing rows are reported),
outds=work.test_results
)
%mp_assert(
iftrue=(&rule_LBLBREAK=1),
desc=%str(Checking a label with a line break is rejected),
outds=work.test_results
)
%mp_assert(
iftrue=(&rule_LBLLONG=1),
desc=%str(Checking a label over 256 characters is rejected),
outds=work.test_results
)
%mp_assert(
iftrue=(&rule_BADFMT=1),
desc=%str(Checking a format carrying a statement separator is rejected),
outds=work.test_results
)
%mp_assert(
iftrue=(&rule_BADLOGIC=1),
desc=%str(Checking a logic value with a statement separator is rejected),
outds=work.test_results
)
%mp_assert(
iftrue=(&rule_BADTYPE=1),
desc=%str(Checking a VAR_TYPE other than C or N is rejected),
outds=work.test_results
)
%mp_assert(
iftrue=(&rule_BADACTIVE=1),
desc=%str(Checking an ACTIVE outside 1 or 0 is rejected),
outds=work.test_results
)
/**
* Test 3 - a row that meets every rule, a free text label, and a delete
* marked row are all left alone
*/
proc sql noprint;
select count(*) into: notchecked
from work.derived_check
where upcase(strip(var_nm)) in ('GOOD','FREETXT','DELME');
quit;
%mp_assert(
iftrue=(&notchecked=0),
desc=%str(Checking a valid row, a free text label and a delete marked row are not reported),
outds=work.test_results
)
+14 -1
View File
@@ -45,6 +45,7 @@
<h4> SAS Macros </h4>
@li mp_abort.sas
@li mpe_abortmessage.sas
@li mx_execute.sas
**/
@@ -149,10 +150,22 @@
%if "%upcase(&state)"="UNKNOWN" %then
%put NOTE: &sysmacroname: &program returned no job status (SAS 9) - the outcome was not measured;
%else %if "%upcase(&state)" ne "COMPLETED" %then %do;
/* the child's own message is carried into the abort verbatim - see
mpe_abortmessage, which builds the text so a comma in either the message
or an endpoint path cannot split the call, and owns the global it writes
to (_mpe_abortmsg).
The %let is the fallback for a build where the helper is not on the
autocall path: the abort still names the program and state rather than
arriving blank. It has to be global, because a plain %let inside this
macro would create a local of that name and win on the first failed call,
dropping the child's message the helper then wrote. */
%global _mpe_abortmsg;
%let _mpe_abortmsg=%superq(program) did not complete: state=%superq(state) %superq(msg);
%mpe_abortmessage(programvar=program, statevar=state, msgvar=msg)
%mp_abort(
iftrue=(1=1)
,mac=&sysmacroname
,msg=%str(&program did not complete: state=&state &msg)
,msg=%superq(_mpe_abortmsg)
)
%end;
%end;
+46
View File
@@ -16,6 +16,7 @@
@li mf_getuniquefileref.sas
@li mf_nobs.sas
@li mp_assert.sas
@li mpe_abortmessage.sas
@li mpe_execute.sas
@author 4GL Apps Ltd
@@ -123,3 +124,48 @@ run;
desc=%str(Checking mdebug=1 keeps the child log),
outds=work.test_results
)
/**
* Test 3 - the abort text carries a child message containing a comma
*
* Assembling that text with %str at the abort call site split the call at the
* first comma, so the message the caller saw was not the child's. The text is
* built by mpe_abortmessage, which names the message variable instead of
* taking its value - a value argument would be split before the macro ran.
*/
/* a plain %let: %str would store its mask characters in the value, and the
macro reads the value with symget(). The same text is kept in csvmsg so both
sides of the comparison below go through %superq - the macro's value carries
its own quoting around the comma, and comparing that against a %str-masked
literal does not match */
%let msg=alpha, beta and gamma are not valid;
%let csvmsg=alpha, beta and gamma are not valid;
%let prog=/services/example, with a comma;
%let st=ABORTED;
%mpe_abortmessage(programvar=prog, statevar=st)
%mp_assert(
iftrue=(%index(%superq(_mpe_abortmsg),%superq(csvmsg))>0),
desc=%str(Checking a comma in the child message survives into the abort text),
outds=work.test_results
)
%mp_assert(
iftrue=(%index(%superq(_mpe_abortmsg),%str(did not complete: state=ABORTED))>0),
desc=%str(Checking the abort text names the program state),
outds=work.test_results
)
/**
* Test 4 - a blank child message leaves no trailing blank behind
*/
%let msg=;
%mpe_abortmessage(programvar=prog, statevar=st)
%mp_assert(
iftrue=("%superq(_mpe_abortmsg)"="/services/example, with a comma did not complete: state=ABORTED"),
desc=%str(Checking a blank child message leaves the abort text clean),
outds=work.test_results
)
+8 -6
View File
@@ -80,9 +80,9 @@ insert into &lib..mpe_config set
!!' client side. The DC_MAXCELLS_WEBEDIT limit is applied at the same'
!!' time, whichever is reached first.';
/* Why the cell limit exists, kept out of var_desc - MPE_CONFIG.VAR_DESC is
varchar(300), and a truncation WARNING aborts this whole build. A wide
table is expensive long before it is tall - 250 rows of a 1000 column
table is a quarter of a million cells. */
varchar(300), and a truncated value raises a diagnostic that aborts this
whole build. A wide table is expensive long before it is tall - 250 rows of
a 1000 column table is a quarter of a million cells. */
insert into &lib..mpe_config set
tx_from=0
,tx_to='31DEC9999:23:59:59'dt
@@ -2214,8 +2214,8 @@ insert into &lib..MPE_VALIDATIONS set
,rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
/* test soft regex - PRIMARY_KEY_FIELD should be an integer (yellow if it
contains a decimal point). All generated keys are integers, so no
warning shows until a demo user enters a decimal. */
contains a decimal point). All generated keys are integers, so nothing is
flagged until a demo user enters a decimal. */
insert into &lib..MPE_VALIDATIONS set
tx_from=0
,base_lib="&lib"
@@ -2355,7 +2355,9 @@ insert into &lib..MPE_VALIDATIONS set
,some_char="&x bottles of beer on the wall"
,some_dropdown='Option 1'
,some_num=ranuni(0)
,some_date=round(ranuni(0)*1000,1)
/* First of the month, cycling through two years: a randomly spread date
gave every row its own day, so a date filter never returned many rows. */
,some_date=intnx('month','01JAN1960'd,mod(&x,24),'S')
,some_datetime=round(ranuni(0)*50000,1)
,some_time=round(ranuni(0)*100,1)
,some_shortnum=6+round(ranuni(0)*94,1)
+5 -4
View File
@@ -3,14 +3,15 @@
@brief Refreshes the data catalog
@details Assumes library is already assigned.
@param [in] etlsource identifies this load in the MPE_DATALOADS audit row.
@param [in] etlsource= (0) Defaults to %mf_getuniquename(prefix=refreshcatalogs); it
identifies this load in the MPE_DATALOADS audit row.
Usage:
%mpe_refreshcatalogs(sashelp)
<h4> SAS Macros </h4>
@li mf_uid.sas
@li mf_getuniquename.sas
@li mp_abort.sas
@li bitemporal_dataloader.sas
@@ -20,11 +21,11 @@
**/
%macro mpe_refreshcatalogs(lib,cat=#all
,etlsource=refreshcatalog:mpe_refreshcatalogs:%mf_uid()
,etlsource=0
);
%let lib=%upcase(&lib);
%let cat=%upcase(&cat);
%if "&etlsource"="0" %then %let etlsource=refreshcatalog:%sysmacroname:%mf_uid();
%if "&etlsource"="0" %then %let etlsource=%mf_getuniquename(prefix=refreshcatalogs);
%put running &sysmacroname &lib for &cat;
proc sql;
+8 -3
View File
@@ -3,12 +3,12 @@
@brief Refreshes the library data catalog
@details
@param [in] etlsource= (refreshcatalog:<step>:%mf_uid()) identifies this load
@param [in] etlsource= (0) Defaults to %mf_getuniquename(prefix=refreshlibs), identifies this load
in the MPE_DATALOADS audit row. That table is keyed on etlsource together
with processed_dttm, libref and dsn, so the uid keeps each load distinct.
<h4> SAS Macros </h4>
@li mf_uid.sas
@li mf_getuniquename.sas
@li dc_getlibs.sas
@li mf_getplatform.sas
@li bitemporal_dataloader.sas
@@ -22,8 +22,13 @@
**/
%macro mpe_refreshlibs(lib=0
,etlsource=refreshcatalog:mpe_refreshlibs:%mf_uid()
,etlsource=0
);
/* a default of 0 rather than a %mf_uid() expression: a default is
resolved when the macro is compiled, so it would be one value for the
whole session and the audit folder below it would collide on the
second call - as it did between the refresh steps of one run */
%if "&etlsource"="0" %then %let etlsource=%mf_getuniquename(prefix=refreshlibs);
%dc_getlibs(outds=work.mm_getLibs)
proc sort data=mm_getlibs;
+8 -3
View File
@@ -3,7 +3,7 @@
@brief Refreshes the data catalog
@details
@param [in] etlsource= (refreshcatalog:<step>:%mf_uid()) identifies this load
@param [in] etlsource= (0) Defaults to %mf_getuniquename(prefix=refreshtables), identifies this load
in the MPE_DATALOADS audit row. That table is keyed on etlsource together
with processed_dttm, libref and dsn, so the uid keeps each load distinct.
Usage:
@@ -11,7 +11,7 @@
%mpe_refreshtables(sashelp)
<h4> SAS Macros </h4>
@li mf_uid.sas
@li mf_getuniquename.sas
@li mp_abort.sas
@li bitemporal_dataloader.sas
@li mf_existfeature.sas
@@ -24,8 +24,13 @@
**/
%macro mpe_refreshtables(lib,ds=#all
,etlsource=refreshcatalog:mpe_refreshtables:%mf_uid()
,etlsource=0
);
/* a default of 0 rather than a %mf_uid() expression: a default is
resolved when the macro is compiled, so it would be one value for the
whole session and the audit folder below it would collide on the
second call - as it did between the refresh steps of one run */
%if "&etlsource"="0" %then %let etlsource=%mf_getuniquename(prefix=refreshtables);
%let lib=%upcase(&lib);
%let ds=%upcase(&ds);
%local engine; %let engine=%mf_getengine(&lib);
+4 -4
View File
@@ -186,9 +186,9 @@ run;
%global loadtype var_txfrom var_txto var_processed filter_text pk coltype
sortpk var_busfrom var_busto rk_underlying buskey pre_edit_hook;
/* clear first: a global survives the call, so in a reused session it would still
* hold the previous request's values, and any path that skips an assignment
* would apply them to this one */
/* clear first: a global survives the call, so in a reused session it would
still hold the previous request's values, and any path that skips an
assignment would apply them to this one */
%let loadtype=;
%let var_txfrom=;
%let var_txto=;
@@ -552,7 +552,7 @@ select upcase(loadtype)
%mpestp_getdata()
/* mp_abort cannot exit cleanly from inside a %include, so an abort there
(eg in a hook script run via mp_include) only records the error in
(eg in a hook script run via mp_include) only records the failure in
work.mp_abort_errds. mode=INCLUDE picks that up and aborts properly here. */
%mp_abort(mode=INCLUDE)
@@ -65,8 +65,8 @@ select count(*) into: admin_check
where groupname="&dc_admin_group";
quit;
%if &admin_check>0 %then %do;
%put WARNING: %mf_getuser() is in the &dc_admin_group group;
%put WARNING: Row Level Security is skipped for admins - test skipped;
%put %str(WARN)ING: %mf_getuser() is in the &dc_admin_group group;
%put %str(WARN)ING: Row Level Security is skipped for admins - test skipped;
data work.test_results;
length test_result $4 test_description $256 test_comments $256;
test_result='PASS';
@@ -0,0 +1,172 @@
/**
@file
@brief testing restore of a change to a special missing value
@details A cell holding a special missing (._ .A-.Z) and changed to an
ordinary number must be reverted to that special missing. mp_stripdiffs
writes the old value into a generated UPDATE statement, and PUT drops the
leading period of a special missing, so the statement used to be
"set SOME_BESTNUM =_" - which PROC SQL reads as a column name.
<h4> SAS Macros </h4>
@li mp_assert.sas
@li mx_execute.sas
**/
%let _program=&appLoc/services/editors/restore;
/* seed the base table with a special missing - the audit stores whatever the
base holds, so this is the value the revert has to write back */
proc sql noprint;
update &dclib..mpe_x_test set some_bestnum=._
where primary_key_field=3;
quit;
/**
* load 1 - change that cell to an ordinary number
*/
data work.sascontroltable;
action='LOAD';
message="special missing prep";
libds="&dclib..MPE_X_TEST";
output;
stop;
run;
data work.jsdata;
set &dclib..mpe_x_test(where=(primary_key_field=3)
rename=(some_date=dt2 SOME_DATETIME=dttm2 SOME_TIME=tm2)
);
/* the adapter sends these as strings */
some_date=put(dt2,date9.);
SOME_DATETIME=put(dttm2,datetime19.);
some_time=put(tm2,time.);
drop dt2 dttm2 tm2;
_____DELETE__THIS__RECORD_____='No';
some_bestnum=42;
run;
%mx_execute(&appLoc/services/editors/stagedata,
viyacontext=&defaultcontext,
inputdatasets=work.jsdata work.sascontroltable,
outlib=web1,
mdebug=&sasjs_mdebug
)
%let status=0;
data _null_;
set web1.sasparams;
if status='SUCCESS' then call symputx('status',1);
call symputx('dsid',dsid);
run;
%mp_assert(
iftrue=(&status=1 and &syscc=0),
desc=Checking the special missing change was staged,
outds=work.test_results
)
data work.sascontroltable;
ACTION='APPROVE_TABLE';
TABLE="&dsid";
DIFFTIME="%sysfunc(datetime())";
output;
stop;
run;
%mx_execute(&appLoc/services/auditors/postdata,
viyacontext=&defaultcontext,
inputdatasets=work.sascontroltable,
outlib=web3,
outref=wb3,
mdebug=&sasjs_mdebug
)
%let status=0;
data _null_;
set web3.apparams;
if response='SUCCESS!' then call symputx('status',1);
run;
%mp_assert(
iftrue=(&status=1 and &syscc=0),
desc=Checking the change was approved,
outds=work.test_results
)
/* the load reference to revert */
%let loadref=0;
data _null_;
set &dclib..mpe_submit end=last;
if last then call symputx('loadref',table_id);
run;
/**
* revert it - the old value in the diff is the special missing
*/
data work.restore_in;
load_ref="&loadref";
output;
stop;
run;
%mx_execute(&_program,
viyacontext=&defaultcontext,
inputdatasets=work.restore_in,
outlib=web1,
mdebug=&sasjs_mdebug
)
%let newref=0;
data _null_;
set web1.restore_out;
call symputx('newref',loadref);
run;
%mp_assert(
iftrue=(&newref ne 0 and &syscc=0),
desc=Checking the revert was submitted,
outds=work.test_results
)
/* approve the reversion */
data work.sascontroltable;
ACTION='APPROVE_TABLE';
TABLE="&newref";
DIFFTIME="%sysfunc(datetime())";
output;
stop;
run;
%mx_execute(&appLoc/services/auditors/postdata,
viyacontext=&defaultcontext,
inputdatasets=work.sascontroltable,
outlib=web3,
outref=wb3,
mdebug=&sasjs_mdebug
)
%let status=0;
data _null_;
set web3.apparams;
if response='SUCCESS!' then call symputx('status',1);
run;
%mp_assert(
iftrue=(&status=1 and &syscc=0),
desc=Checking the reversion was approved,
outds=work.test_results
)
/* and the cell is back to the special missing, not a number and not blank */
%let spmcheck=0;
data _null_;
set &dclib..mpe_x_test(where=(primary_key_field=3));
if some_bestnum=._ then call symputx('spmcheck',1);
putlog some_bestnum=;
run;
%mp_assert(
iftrue=(&spmcheck=1),
desc=Checking the special missing was restored,
outds=work.test_results
)
/* put the fixture back as makedata left it */
proc sql noprint;
update &dclib..mpe_x_test set some_bestnum=44
where primary_key_field=3;
quit;
+1 -1
View File
@@ -294,7 +294,7 @@ run;
,dc_dttmtfmt=&dc_dttmtfmt
)
/* mp_abort cannot exit cleanly from inside a %include, so an abort there
(eg in a hook script run via mp_include) only records the error in
(eg in a hook script run via mp_include) only records the failure in
work.mp_abort_errds. mode=INCLUDE picks that up and aborts properly here. */
%mp_abort(mode=INCLUDE)
+114 -4
View File
@@ -6,16 +6,19 @@
table - the engine reads only active rows, so the dependency can never
resolve at run time
2. a row is staged with active=0 while a surviving active row depends on it
3. a staged row breaks a field rule of the shared scan (core's
mp_derivedvarcheck), or carries an ACTIVE value the EDIT screen's picker
cannot produce
They should NOT abort when:
3. a new active row is added to a table that also holds an inactive row -
4. a new active row is added to a table that also holds an inactive row -
the inactive row is not part of the dependency graph, so it cannot
create a cycle or a stale reference
4. a delete-marked RLS rule naming a derived variable is submitted - the
5. a delete-marked RLS rule naming a derived variable is submitted - the
rule is being removed, so its variables are not validated
5. a delete and re-add of the same variable is submitted together - the
6. a delete and re-add of the same variable is submitted together - the
re-added active row resolves the dependency
6. an inactive row is set active again while another active row depends on
7. an inactive row is set active again while another active row depends on
it - the re-activation is what makes the dependency resolvable
The staged loads are aborted by the post edit hook. The assertion fetches the
@@ -553,6 +556,113 @@ run;
)
/**
* Test 8 - a field rule of the shared scan (mp_derivedvarcheck, core) is
* applied at submit time: an expression that could submit code of its own is
* refused by name, with the reason the scan gives. This is the same scan the
* engine runs at read time, so the case pins that the hook calls it rather
* than carrying a rule list of its own.
*/
data work.sascontroltable;
action='LOAD';
message='A refused function in VAR_LOGIC should abort';
libds="&dc_libref..MPE_DERIVED";
output;
stop;
run;
data work.jsdata;
length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
VAR_LOGIC $4000 VAR_FMT $32
_____DELETE__THIS__RECORD_____ $3;
LIBREF='DCTEST';
DSN='DC_DERIV';
VAR_NM='CALLER';
VAR_DEPS='';
VAR_TYPE='N';
VAR_LEN=8;
VAR_FMT='';
/* no semicolon in the payload: the single-expression rule is tested before
the function-name rule, so a quoted semicolon would abort on that first */
VAR_LOGIC='dosubl("data _null_")';
ACTIVE=1;
_____DELETE__THIS__RECORD_____='No';
run;
/* the child is expected to abort here, so the failure must be asserted rather
than aborting this program - hence failfast=0 */
%mpe_execute(&_program,
viyacontext=&defaultcontext,
inputdatasets=work.sascontroltable work.jsdata,
outlib=web8,
outref=wb8,
failfast=0,
mdebug=&sasjs_mdebug
)
/* the assertion compares the abort's MSG exactly, so the expected value is
assembled on one line: a literal wrapped across two source lines carries the
newline and the continuation indent with it.
The text has no space after either colon because that is how the child builds
it - mpe_derived emits &mac.:&VAR_NM.:&reason, and the reason itself is
'VAR_LOGIC may not call a function that' || ' reaches the file system, ...'.
Spacing here is the child's, not a formatting choice. */
%let exp8=MPE_DERIVED:CALLER:VAR_LOGIC may not call a function that;
%let exp8=&exp8 submits code or reaches the file system,;
%let exp8=&exp8 the environment or the macro symbol table;
%mpe_assert_abort(
desc=Checking a refused function in VAR_LOGIC is rejected,
outref=wb8,
msg=%superq(exp8)
)
/**
* Test 9 - the fields the hook owns rather than the scan: ACTIVE is a picker
* in the EDIT screen (HARDSELECT over the MPE_SELECTBOX pair), so a submission
* that bypasses the grid cannot land a value the screen would never offer.
*/
data work.sascontroltable;
action='LOAD';
message='An ACTIVE value the screen cannot offer should abort';
libds="&dc_libref..MPE_DERIVED";
output;
stop;
run;
data work.jsdata;
length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
VAR_LOGIC $4000 VAR_FMT $32
_____DELETE__THIS__RECORD_____ $3;
LIBREF='DCTEST';
DSN='DC_DERIV';
VAR_NM='BADACT';
VAR_DEPS='';
VAR_TYPE='N';
VAR_LEN=8;
VAR_FMT='';
VAR_LOGIC='1';
ACTIVE=2;
_____DELETE__THIS__RECORD_____='No';
run;
/* this child is expected to abort too - assert the failure rather than
cancelling this program */
%mpe_execute(&_program,
viyacontext=&defaultcontext,
inputdatasets=work.sascontroltable work.jsdata,
outlib=web9,
outref=wb9,
failfast=0,
mdebug=&sasjs_mdebug
)
%mpe_assert_abort(
desc=Checking an ACTIVE value outside the picker pair is rejected,
outref=wb9,
msg=MPE_DERIVED:BADACT:ACTIVE must be 1 or 0
)
/* tidy up so repeated runs start clean */
proc sql noprint;
delete from &dc_libref..mpe_derived
@@ -34,7 +34,7 @@
<h4> SAS Macros </h4>
@li mf_nobs.sas
@li mp_abort.sas
@li mpe_derived_check.sas
@li mp_derivedvarcheck.sas
<h4> Related Macros </h4>
@li mpe_derived.sas
@@ -100,14 +100,43 @@ run;
)
/**
* Field level checks - the same rules %mpe_derived applies at read time
* Field level checks. The rules for the derived variable itself - the name,
* the type, length, format, label and the expression - are core's, in
* mp_derivedvarcheck, so this hook and the read time engine (%mpe_derived ->
* mp_derivedvars) scan a row with the same rules and report the same reasons.
* What is added here are the two fields that belong to Data Controller rather
* than to the expression: the control table key, which the loader normalises
* but does not require, and ACTIVE, which the EDIT screen offers as a picker
* over the MPE_SELECTBOX pair - so a submission that bypasses the grid cannot
* land a value the screen would never offer.
*/
%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
data work.derived_dcfield(keep=name reason);
set work.staging_ds;
if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
length name $32 reason $200;
call missing(reason);
name=upcase(strip(var_nm));
if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
else if missing(active) or active not in (0,1)
then reason='ACTIVE must be 1 or 0';
if not missing(reason) then output;
run;
%mp_derivedvarcheck(inds=work.staging_ds
,outds=work.derived_field
,reserved=%str(_____DELETE__THIS__RECORD_____)
)
/* the control table fields are reported before the expression's own rules, so
which of the two passes found a row does not change the message it gets */
data work.derived_check;
set work.derived_dcfield work.derived_field;
run;
%if %mf_nobs(work.derived_check)>0 %then %do;
data _null_;
set work.derived_check(obs=1);
call symputx('der_errmsg',cats('MPE_DERIVED: ',var_nm,': ',reason));
call symputx('der_errmsg',cats('MPE_DERIVED: ',name,': ',reason));
call symputx('der_errflag',1);
run;
%end;
@@ -35,5 +35,7 @@ run;
%mp_abort(iftrue=(&errval=1)
,mac=mpe_security_postedit.sas
,msg=%str(&errmsg)
/* %bquote: the submitted value is interpolated into this message, and a
comma in it would split this call into extra positional parameters */
,msg=%bquote(&errmsg)
)
+1 -1
View File
@@ -15,10 +15,10 @@
@li mf_existfeature.sas
@li dc_assignlib.sas
@li mp_abort.sas
@li mp_binarycopy.sas
@li mp_cntlout.sas
@li mp_ds2cards.sas
@li mp_ds2csv.sas
@li mp_ds2md.sas
@li mp_streamfile.sas
@li mp_validatecol.sas
@li mf_nobs.sas
+21 -13
View File
@@ -78,25 +78,33 @@ run;
outview=work.vw_derived,
outvars=work.derived_vars
)
%let derived_nobs=%mf_nobs(work.derived_vars);
/* A format catalog (LIBREF.CATALOGNAME-FC) must be left for mp_filterstore to
rewrite into a cntlout dataset, and a -FC name handed over as targetds is not
something it can read - so pass no targetds at all in that case. An empty
targetds makes the engine fall back to libds, which is what a plain dataset
wants anyway.
/* targetds is passed as before for a plain dataset, and as the view when the
filter can reference derived columns. A format catalog is the one case that
must be left alone: mp_filterstore rewrites a -FC libds to a cntlout dataset
itself, and a -FC name handed over as targetds is not something it can read.
The trailing comma lets the call below read targetds=..., queryds=... */
%let targetds_opt=;
No macro triggers in this comment: the macro processor resolves them even
inside a comment, so prose about conditions is itself executed and the parser
reports "Expected DO not found" from here. Same reason the values below are
plain LET statements with no trailing comma. */
%let derived_nobs=%sysfunc(coalesce(%mf_nobs(work.derived_vars),0));
%let is_fmtcat=%eval(%length(&ds)>3 and "%upcase(%substr(&ds,%length(&ds)-2,3))"="-FC");
%let targetds_val=&ds;
%if &derived_nobs>0 %then %do;
%let targetds_opt=targetds=work.vw_derived,;
%let targetds_val=work.vw_derived;
%end;
%else %if %length(&ds)>3 and "%upcase(%substr(&ds,%length(&ds)-2,3))" ne "-FC" %then %do;
%let targetds_opt=targetds=&ds,;
/* the derived view keeps precedence: a catalog is only blanked when there is no
derived configuration to validate against. Two plain %if blocks rather than
%else %if - that construct is what the parser rejected here before. */
%if &derived_nobs=0 and &is_fmtcat=1 %then %do;
%let targetds_val=;
%end;
%mp_filterstore(
libds=&ds,
&targetds_opt
queryds=work.filterquery,
libds=&ds
,targetds=&targetds_val
,queryds=work.filterquery,
filter_summary=&dc_libref..mpe_filteranytable,
filter_detail=&dc_libref..mpe_filtersource,
lock_table=&dc_libref..mpe_lockanytable,
@@ -5,7 +5,6 @@
<h4> SAS Macros </h4>
@li mx_execute.sas
@li mp_assertdsobs.sas
@li mp_assertcolvals.sas
@li mf_getuniquefileref.sas
**/
@@ -34,8 +33,15 @@ run;
%mx_execute(&_program,
viyacontext=&defaultcontext,
inputfiles=&f1:iwant &f2:filterquery,
outlib=web1
outlib=web1,
outref=web1ref,
mdebug=&sasjs_mdebug
)
data _null_;
infile web1ref;
input;
putlog _infile_;
run;
data result;
set web1.result;
run;
@@ -64,7 +70,8 @@ run;
viyacontext=&defaultcontext,
inputdatasets=work.iwant work.filterquery,
outlib=web2,
outref=web2ref
outref=web2ref,
mdebug=&sasjs_mdebug
)
data _null_;
infile web2ref;
@@ -119,7 +126,8 @@ run;
viyacontext=&defaultcontext,
inputdatasets=work.iwant work.filterquery,
outlib=web3,
outref=web3ref
outref=web3ref,
mdebug=&sasjs_mdebug
)
data _null_;
infile web3ref;
-1
View File
@@ -55,7 +55,6 @@
@li mp_cntlout.sas
@li mp_getcols.sas
@li mp_getpk.sas
@li mp_jsonout.sas
@li mp_searchdata.sas
@li mp_validatecol.sas
@li mpe_columnlevelsecurity.sas
@@ -11,6 +11,8 @@
overwriting actual data. Pass a KEEPME column in the input data to keep
them - test harnesses reseed before every run and cannot redeploy first.
@sasjslint {"noBannedStrings": false}
<h4> SAS Macros </h4>
@li mf_getapploc.sas
@li mf_mkdir.sas
@@ -215,9 +217,11 @@ run;
* caller asked to keep them (KEEPME in the input data). Keeping them is for
* test harnesses, which reseed before every run and cannot redeploy first. */
%if &keepme ne 1 %then %do;
%ms_deletefile(&root/services/admin/configurator.sas)
%ms_deletefile(&_program..sas)
%ms_deletefile(&root/services/admin/configurator.sas);
%ms_deletefile(&_program..sas);
%end;
%else %do;
%put NOTE: &_program retained, as KEEPME was requested;
%end;
%else %put NOTE: &_program retained, as KEEPME was requested;
@@ -298,6 +298,8 @@ run;
/* keep the program when KEEPME was requested, so a reseed does not need a
* redeploy first */
%if &keepme ne 1 %then %do;
%mv_deletejes(path=&root/admin, name=makedata)
%mv_deletejes(path=&root/admin, name=makedata);
%end;
%else %do;
%put NOTE: &_program retained, as KEEPME was requested;
%end;
%else %put NOTE: &_program retained, as KEEPME was requested;
-1
View File
@@ -17,7 +17,6 @@
@li mddl_dc_difftable.sas
@li mddl_dc_locktable.sas
@li mp_init.sas
@li mx_execute.sas
@li ms_createfile.sas
@li ms_deletefile.sas
@li mx_getcode.sas