Compare commits

..
Author SHA1 Message Date
dc 468d2e9a80 test(download): check for the modal before using it, and click the menu item
Build / Build-and-ng-test (pull_request) In progress
Lighthouse Checks / lighthouse (pull_request) In progress
Three review points.

attachExcelFile read '.clr-abort-modal .modal-title' unconditionally, so an
upload that raised no abort modal failed on the lookup, and the abort branch
then left the modal open for the next command to click through.  It asks the body
first: no abort modal means confirm the upload; an abort modal means assert its
title is not an Abort Message and stop there.

openDownloadModal clicked the deepest element holding the text.  The click
handler sits on the div[clrDropdownItem], so the item is selected by that
attribute and its text.

The spec list comment claimed download leaves nothing for later specs to trip
over.  It seeds the approval it needs and that approval stays, which the comment
now says.
2026-10-30 22:28:29 +00:00
dc f50a0de9ae test(download): drive the file input directly and name DDL files as getddl does
Build / Build-and-ng-test (pull_request) Successful in 4m35s
Build / Build-and-test-development (pull_request) In progress
Lighthouse Checks / lighthouse (pull_request) Failing after 11m5s
Two reasons the last CI run still had four failures.

test 1 timed out on label[for="file-upload"].  The label is a click target for a
hidden input; attachFile can drive the input itself, so the helper no longer
clicks anything and selects by the input's id, asserting there is exactly one.
That also answers the earlier review point about positional selectors without
reintroducing a click on a control that is not always rendered.

tests 5-7 timed out waiting for the DDL file.  The mock names a DDL download
after the libref alone, while services/public/getddl.sas names it
<libref>_<ds>.ddl - which is what the viewer expects and what the spec looks
for.  The mock now follows the service.
2026-10-30 22:18:03 +00:00
dc 628092d136 test(download): single separator in the fixture path
Build / Build-and-ng-test (pull_request) Successful in 4m49s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m1s
Build / Build-and-test-development (pull_request) Failing after 35m54s
fixturePath carried a trailing slash that the template also supplies, so
attachFile was given /excels//regular_excel.xlsx.  Also drops the stale duplicate
doc comment above attachExcelFile - the helper targets the editor's import label,
which is what the remaining line says.
2026-10-30 21:23:56 +00:00
dc 3e4070ae49 test(download): select the download format by value
Build / Build-and-ng-test (pull_request) Successful in 5m23s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m58s
Build / Build-and-test-development (pull_request) Failing after 35m4s
Second CI run put the spec at 2 passing / 5 failing, and the five failures were
one more stale assumption: cy.select() matches an option's text against the whole
label, and the labels carry a suffix - the option for the Excel writer is
"Excel (.xlsx)", not "Excel". Every format is now chosen by the option's value
(EXCEL, SAS, SAS_DDL, TSQL_DDL, PGSQL_DDL), which the template sets and which does
not change when a label gains or loses a qualifier.

Also the review points on this PR:

- attachExcelFile starts from label[for="file-upload"] instead of
  .buttonBar button:last-child, and submitExcel no longer takes a callback that
  nothing passed.
- test 1 asserts on the download button whose id is the table it submitted: the
  stage URL carries the id, so the assertion names its own row and asserts one
  match, instead of indexing into every match on the screen.
- the comment in cypress-specs.sh no longer claims the spec depends on approvals
  left by earlier specs - it seeds its own - and says why it is appended.
2026-10-30 21:04:41 +00:00
dc 847d329d5c test(download): point the spec at the live route and the dropdown markup
Build / Build-and-ng-test (pull_request) Successful in 4m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m52s
Build / Build-and-test-development (pull_request) Failing after 34m0s
Three separate reasons the spec could not pass, all of them stale routes or
selectors rather than an overlay covering the button:

- test 1 opened #/approve/toapprove, which is not a route. The review module is
  namespaced under review/, so the approve screen is #/review/approve. The dead
  URL rendered the 404 page, and the only .btn.btn-success on that page is not a
  download button - which is why the failure looked like a covered element.

- on the right route the approve grid still renders nothing on a fresh estate:
  it draws a download button per table waiting for approval, and there are none
  ("There are no approvals remaining"). Test 1 now submits an excel fixture for
  mpe_x_test first, the way stage.cy.ts does, so it owns the approval it
  downloads instead of depending on what earlier specs happen to leave behind.

- the viewer's options menu is a list of div[clrDropdownItem] entries, not
  buttons, so cy.get('clr-dropdown-menu button') never matched; Download is now
  selected by its text, as filter-panel.cy.ts selects Filter. The dropdown-toggle
  class the helper also required is gone from that element too.

The download button is selected by aria-label, which is unique, rather than a
bare .btn.btn-success that also matches the editor toolbar's small buttons.

Closes #398
2026-10-30 19:54:14 +00:00
dc c1e7314417 test(download): target the approve button and the viewer options dropdown
Build / Build-and-ng-test (pull_request) Successful in 4m39s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m28s
Build / Build-and-test-development (pull_request) Failing after 36m15s
Both selectors in download.cy.ts are stale or ambiguous, which is why the spec
fails as soon as it runs:

- `.btn.btn-success` also matches the editor toolbar's buttons (`btn btn-sm
  btn-success`), and those sit inside a fixed-position container on the approve
  screen. The first match was one of those, so `should('be.visible')` retried
  against an element that is never visible and the error read like the download
  button was covered. The approve row's button is `btn btn-success` with
  aria-label="Download audit file", so select it by that label.

- `.btn.btn-sm.btn-outline.filterSide.dropdown-toggle` cannot match anything:
  the viewer's options dropdown is `class="btn btn-sm btn-outline filterSide"`,
  with no dropdown-toggle class. filter-panel.cy.ts and row-cell-limits.cy.ts
  already select it as `.filterSide`.

The spec goes back into the CI list, appended at the end because it needs
approvals to exist and the earlier specs create them. metanav stays excluded
(#395).

Closes #398
2026-10-30 18:21:14 +00:00
allan 8862798d65 Merge pull request 'fix(deps): bump @sasjs/core to 7.0.1 so special missings can be restored' (#409) from fix/restore-special-missings into release/7.22
Sites / deploy (push) Successful in 2m44s
Build / Build-and-ng-test (pull_request) Successful in 4m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m28s
Build / Build-and-test-development (pull_request) Successful in 28m17s
Reviewed-on: #409
2026-10-30 17:23:32 +00:00
dc dfc5c6ebb8 test(stagedata): do not let an expected child abort cancel the test
Build / Build-and-ng-test (pull_request) Successful in 4m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m27s
Build / Build-and-test-development (pull_request) Successful in 28m3s
Test 8 submits a VAR_LOGIC expression that the derived-variable scan refuses, so
the child is expected to abort and the assertion reads its message from the
response. Without failfast=0 mpe_execute aborted this program instead, so the
case could never reach its assertion. The file's other abort-expecting calls
already pass failfast=0.

This does not make the case pass on its own - the child still cancels without
writing a response, which is tracked separately - but it removes the wrong
failure in front of it.
2026-10-30 17:19:50 +00:00
dc 20dbf00fb0 fix(mpe_execute): mask a child abort message before it aborts the caller
mpe_execute reports a child that did not complete by aborting the caller with
"&program did not complete: state=&state &msg". The message was passed through
%str, which masks the literal text before resolution - so the commas in a
child's own abort message (mp_abort writes them as part of the reason) were
still delimiters when the call was parsed. The call split into extra positional
parameters and SAS raised

  ERROR: All positional parameters must precede keyword parameters.

replacing the real failure with a parse error at the call site. The caller then
lost the child's message entirely.

%bquote masks the resolved value, so the message arrives as one argument and the
abort reports what the child actually said.

The same shape in mpe_security_postedit, where the submitted ACCESS_LEVEL value
is interpolated into the abort message, is fixed the same way.
2026-10-30 17:19:50 +00:00
allan 9d82d7545e Merge pull request 'fix(deploy): drop the deprecated service-pack deploy from the setup screen' (#408) from chore/deprecate-service-pack-deploy into release/7.22
Sites / deploy (push) Successful in 2m45s
Build / Build-and-ng-test (pull_request) Successful in 4m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m7s
Build / Build-and-test-development (pull_request) Successful in 27m39s
Reviewed-on: #408
2026-10-30 17:04:26 +00:00
dc d67a0e5d5f fix(deploy): await nothing, and encode the values in the makedata URL
Build / Build-and-ng-test (pull_request) Successful in 4m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m19s
Build / Build-and-test-development (pull_request) Successful in 27m47s
Addresses both findings from the review of #408.

runAutoDeploy was declared async after the service-pack step went, but calls
createDatabase() and returns without awaiting anything. createDatabase() starts
the request and reports through the component's own status flags rather than a
returned promise, so there is nothing to await - drop the async rather than
pretend to a contract the callee does not offer.

runMakedataInNewWindow interpolated the context name, admin group and DC path
straight into the query string. An ampersand in any of them started a new
parameter and a space arrived truncated; a DC path containing a slash was only
working because nothing decoded the query. Those three values are now
percent-encoded at the build site, so the server reads back what the user chose.

Tests: the new-window case now asserts the encoded forms, and a second case
pins the values that need encoding (context with an ampersand and spaces, admin
group with a space and a plus, DC path with a slash and a space). The deploy
case no longer awaits a synchronous handler.

Executed 24 of 24 SUCCESS (automatic + sasjs-configurator + deploy components).
2026-10-30 16:14:28 +00:00
dc 3d2eefdd7b fix(deps): bump @sasjs/core to 7.0.1 so special missings can be restored
Build / Build-and-ng-test (pull_request) Successful in 4m29s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m9s
Build / Build-and-test-development (pull_request) Successful in 27m34s
Restoring any load that touched a special missing numeric (._ or .A-.Z) failed
in mp_stripdiffs, which writes the old value into a generated UPDATE statement.
PUT writes a special missing as its letter only, so the statement carried
"set SOME_BESTNUM =_" and PROC SQL read the bare underscore as a column name:

  ERROR: The following columns were not found in the contributing tables: _.
  ERROR: SOME_BESTNUM, a numeric column, can only be updated with a numeric
    expression.

Every statement in the generated block then failed NOEXEC, so the revert
produced the wrong data; the service's syscc check is what surfaced it.

Fixed upstream in sasjs/core#499 (the period is now written first) and
released as 7.0.1. v7.0.0's breaking change - mf_existvarlist warns and returns
0 instead of aborting on blank input - does not affect DC: there are no direct
callers, and the full SAS suite covers the core macros that do use it.

Adds restore.test.2.sas, which seeds some_bestnum=._ on mpe_x_test, changes it
to a number through stagedata and approves it, reverts that load, approves the
reversion, and asserts the cell is back to ._ rather than blank. It fails on
6.1.4 and passes on 7.0.1.
2026-10-30 16:07:31 +00:00
dc e3819e4350 test(deploy): cover the makedata hand-off on both setup screens
Build / Build-and-ng-test (pull_request) Successful in 4m47s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m2s
Build / Build-and-test-development (pull_request) Successful in 28m5s
The setup screen's only action is to run `makedata` with what the form
collected, and nothing tested it. Both screens now have specs pinning the
request and the URL they build:

- Viya (automatic): the admin group and DC path reach the service as
  `fromjs`, the configured context is used with the web method, a context chosen
  in the form wins over the configured one, the new-window URL carries
  `_contextname` / `admin` / `dcpath` / `_debug`, and pressing deploy reaches
  `createDatabase` at all
- SAS9 and SASjs Server (configurator): the makedata URL carries `_program`,
  `admin`, `path` and `_debug`; SAS9 stays on the page origin and encodes the
  directory, SASjs Server leaves the path readable, and the SASjs Server check
  runs before any reload

The deploy button is also restored. Removing the dead upload markup took the
live button with it, so the screen had no way to start the job its own copy
promises - the new `runAutoDeploy` test would have caught that.
2026-10-30 08:28:48 +00:00
dc 59e09e26a5 fix(deploy): drop the deprecated service-pack deploy from the setup screen
Build / Build-and-ng-test (pull_request) Successful in 4m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m54s
Build / Build-and-test-development (pull_request) Successful in 29m50s
The Viya setup screen still carried a path that deployed the backend as a
service pack from a viya.json (`deployServicePack`). That is no longer how a
deployment works: the frontend and services arrive with the sasjs deploy, and the
only thing left to do is create the control library and its configuration data,
which `makedata` does - and then deletes itself.

The path was already unreachable from the UI (its button had been commented out
"for reference"), so the wizard only ever ran `makedata` in practice while its
copy still claimed to deploy services. Removed:

- `executeJson()` and the `viya.json` upload plumbing: `readFile`,
  `clearUploadInput`, and the fields and status flag that fed them
- the "Deploy SAS Jobs" row in the completion modal, which could never show
- the commented-out button and file input that kept the dead path visible

The screen now says what it does: it runs the one-off job that creates the
control library and its configuration data, and the status row reads "Create
control library" rather than "Create database".
2026-10-29 18:34:35 +00:00
allan eaa3b3e619 Merge pull request 'fix(embed): a rejected mapping must not block the page' (#406) from fix/va-parmap-column into release/7.22
Sites / deploy (push) Successful in 2m51s
Build / Build-and-ng-test (pull_request) Successful in 4m55s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m45s
Build / Build-and-test-development (pull_request) Successful in 28m56s
Reviewed-on: #406
2026-10-29 18:29:30 +00:00
dc d845b9e9b7 fix(embed): a rejected mapping must not block the page
Build / Build-and-ng-test (pull_request) Successful in 4m55s
Lighthouse Checks / lighthouse (pull_request) Successful in 16m59s
Build / Build-and-test-development (pull_request) Successful in 27m58s
The deferred VA load builds the filter from the report's parameters before the
first getdata, which is what makes that first load already filtered. A va_parmap
entry is a column the report author typed, so the request can legitimately fail
when the table does not have it - validatefilter aborts on an unknown variable -
and that abort surfaced as a modal over an editor which then loaded fine
(dc/dc#404).

Validating the mapping first would cost a SAS call on every load of a mapped
object, which is not affordable, so the attempt is simply made quietly:
saveQuery takes wrapper options and the deferred load suppresses the error
modal. A mapping that is right still filters the very first load; a wrong one
falls back to the normal load, and the metadata pass reports it in the banner
and reconfigures once, at setup time rather than on every page load.
2026-10-29 18:22:41 +00:00
semantic-release-bot 428e70a4c1 chore(release): 7.21.0 [skip ci]
# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)

### Bug Fixes

* **client:** parse word-date values from VA parameters ([405b28d](405b28dccc))
* **client:** parse word-date values from VA parameters ([ce5b1aa](ce5b1aa961))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](eeffa92f9e))

### Features

* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](a4a0f9cb2f))
2026-10-29 16:32:45 +00:00
allan 94e040c006 Merge pull request 'feat: Release/7.21' (#401) from release/7.21 into main
Release / Build-production-and-ng-test (push) Successful in 3m30s
Release / Build-and-test-development (push) Successful in 20m56s
Release / release (push) Successful in 5m15s
Reviewed-on: #401
2026-10-29 16:05:27 +00:00
allan cdc0ab2a91 Merge pull request 'fix(client): parse word-date values, and seed demo dates that repeat (release/7.21)' (#405) from fix/va-word-date-release into release/7.21
Sites / deploy (push) Successful in 2m56s
Build / Build-and-ng-test (pull_request) Successful in 4m58s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m9s
Build / Build-and-test-development (pull_request) Successful in 27m56s
Reviewed-on: #405
2026-10-29 15:08:00 +00:00
dc 8449ab3987 test(data): seed the demo table with dates that repeat
Build / Build-and-ng-test (pull_request) Successful in 4m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m26s
Build / Build-and-test-development (pull_request) Canceled after 18m20s
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.

No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
2026-10-29 14:42:33 +00:00
dc 405b28dccc fix(client): parse word-date values from VA parameters
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.

parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.

A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.

Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
2026-10-29 14:42:33 +00:00
allan 559221fad4 Merge pull request 'fix(client): parse word-date values, and seed demo dates that repeat' (#403) from fix/va-date-worddate into main
Release / Build-production-and-ng-test (push) Canceled after 10s
Release / Build-and-test-development (push) Canceled after 0s
Release / release (push) Canceled after 0s
Reviewed-on: #403
2026-10-29 14:39:39 +00:00
dc 999e16b8fc test(data): seed the demo table with dates that repeat
Build / Build-and-ng-test (pull_request) Successful in 4m6s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m45s
Build / Build-and-test-development (pull_request) Successful in 26m22s
The demo rows had a random date each, so a date filter over the demo table
matched at most a couple of rows - which is not enough to show a date filter
working, in a report or in a test. The generated rows now cycle through 24
first-of-month dates over two years, about 20 rows each.

No test asserts a specific generated date; the fixed rows at the top of the
table are unchanged.
2026-10-29 14:05:29 +00:00
dc ce5b1aa961 fix(client): parse word-date values from VA parameters
Build / Build-and-ng-test (pull_request) Successful in 3m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 17m17s
Build / Build-and-test-development (pull_request) Failing after 27m2s
A date parameter whose display format is not DATE9 or a numeric date reached
DC as text like "January 1, 2026" and the value was dropped: the clause
builder produced nothing for that parameter, so with no mapping the editor
filtered on the other parameters only, with no banner to say the date had been
ignored. Reproduced inside a report - a SOME_DATE parameter set to
"January 1, 2026" left the FILTER line reading
(( SOME_DROPDOWN = 'Option 1' )) and the grid at the unfiltered row count.

parseMonthNameDate now accepts a month name spelled out or abbreviated, with
the day before or after it, comma optional and "." or "-" as separators, so
WORDDATE-style values ("January 1, 2026", "Jan 1, 2026", "1 January 2026",
"1-Jan-2026") convert like DATE9 does. Only the first three letters of the
month name are matched, case-insensitively.

A month and year with no day ("January 2026") is still rejected rather than
being read as the first of the month - inventing a day would silently filter
on rows the author did not choose.

Unit tests cover each accepted shape, the case-insensitive match, DATE9 still
parsing as before, and the month-without-day rejection.
2026-10-29 13:43:47 +00:00
allan f74100a818 Merge pull request 'feat(derived): apply the core field scan in the MPE_DERIVED hook' (#402) from refactor/derivedvarcheck into release/7.21
Sites / deploy (push) Successful in 3m23s
Build / Build-and-ng-test (pull_request) Successful in 6m10s
Lighthouse Checks / lighthouse (pull_request) Successful in 18m15s
Build / Build-and-test-development (pull_request) Successful in 29m46s
Reviewed-on: #402
2026-10-29 11:25:26 +00:00
dc a4a0f9cb2f feat(derived): apply the core field scan in the MPE_DERIVED hook
Build / Build-and-ng-test (pull_request) Successful in 6m6s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m15s
Build / Build-and-test-development (pull_request) Successful in 32m40s
@sasjs/core 6.1.3 moved the derived variable field rules out of
mp_derivedvars into mp_derivedvarcheck, so the same scan can run before a
definition is used.  DC carried its own copy of that list in
mpe_derived_check.sas, which is now deleted: the post edit hook calls the
core macro instead, and the hook keeps only the rules that belong to the
control table rather than the expression - LIBREF/DSN required, and ACTIVE
within the picker pair.

The scan is stricter than the copy it replaces, in three ways:

- VAR_NM may not be _N_, _ERROR_ or _IORC_ (automatic variables of the data
  step)
- VAR_LOGIC may not exceed 4000 characters
- VAR_LOGIC may not call a function that submits code or reaches the file
  system, the environment or the macro symbol table - a function name cannot
  be assembled at run time, so this is a control the expression cannot evade

and it relaxes one: VAR_LABEL may carry a line break, which is flattened to a
space when the label is used, so only the remaining control characters are
refused.

The mock hook mirror is updated to the same rules, and stagedata.test.4
gains the two submit-time cases that pin the new wiring.
2026-10-29 10:51:00 +00:00
allan e6a5cce475 Merge pull request 'fix(makedata): name the sasjs server group in the estate seed' (#400) from fix/estate-seed-admin into release/7.21
Sites / deploy (push) Successful in 4m20s
Build / Build-and-ng-test (pull_request) Successful in 6m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m47s
Build / Build-and-test-development (pull_request) Successful in 33m53s
Reviewed-on: #400
2026-10-29 08:33:52 +00:00
semantic-release-bot 97b4cd4d61 chore(release): 7.20.0 [skip ci]
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)

### Bug Fixes

* **ci:** install doxygen for the SAS docs job ([f58925b](f58925b299))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](27686d3aad))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](888873421c))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](dd482c4069))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](25ffa71add))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](c5808a16eb))
* **ci:** rebuild and upload release assets on demand ([6486c80](6486c80ed4))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](af1813f829))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](cf7b21a8b6))
* **cypress:** hold download out too, and retry once in CI ([c97e886](c97e8866db)), closes [#395](#395) [#398](#398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](c803773d3c)), closes [#395](#395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](80e8911af8))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](e51953ca8a))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](459ff2da37)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](f7e753863b))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](7e3045a21b))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](2623a3fb4b))
* **embed:** log a parse-time mapping problem once ([0011f7e](0011f7e933))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](541a61ba08))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](e9216b0895))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](bd3c220af2))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](f83dc359e2))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](aa91daf9f7))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](10157c1159))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](411f6c3354))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](c0ea810766))

### Features

* **embed:** map VA parameters to DC columns from the URL ([15a1f34](15a1f34d52))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](88010c3980))
2026-10-28 23:11:50 +00:00
dc eeffa92f9e fix(makedata): name the sasjs server group in the estate seed
Build / Build-and-ng-test (pull_request) Successful in 6m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m54s
Build / Build-and-test-development (pull_request) Successful in 34m49s
makeDataServer.json seeded ADMIN=AllUsers, a Viya-style group that no account on
the shared estate belongs to. The seed still reports SUCCESS with a group the
caller is not in, and then every getdata/stagedata call aborts with
'<user> is not authorised to edit <libds> in the MPE_SECURITY table', which
reads like a broken deployment. The estate's accounts are in all-users, the
SASjs Server group every logged-in user has, so seed with that and describe the
rule in the estate-testing skill rather than telling the reader to override the
shipped value.
2026-10-28 22:48:02 +00:00
allan dd25be8e75 Merge pull request 'Release/7.20' (#387) from release/7.20 into main
Release / Build-production-and-ng-test (push) Successful in 4m19s
Release / Build-and-test-development (push) Successful in 28m0s
Release / release (push) Successful in 7m31s
Reviewed-on: #387
2026-10-28 22:35:17 +00:00
allan 2e40749052 Merge pull request 'docs(skills): add the estate deploy-and-test procedure' (#397) from docs/estate-testing-skill into release/7.20
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #397
2026-10-28 22:34:37 +00:00
allan fc687f8643 Merge pull request 'fix(cypress): hold download out too, and retry once in CI' (#399) from fix/cypress-release-specs into release/7.20
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #399
2026-10-28 22:34:22 +00:00
dc c97e8866db fix(cypress): hold download out too, and retry once in CI
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 2m15s
Build / Build-and-test-development (pull_request) Canceled after 0s
The run on release/7.20 went from four failing specs to two:

    csv-limited   PASS   (order restored)
    hook-programs PASS   (order restored)
    metanav       held out, #395
    download      7 of 7 fail

download now gets past its hooks - the deleteFolder task is registered - and
fails on the button it clicks:

    AssertionError: expected '<button.btn.btn-sm.btn-success...>' to be 'visible'
    This element is not visible because its ancestor has 'position: fixed' CSS
    property and it is overflowed by other elements.

Something fixed-position sits over it. That needs a scrollIntoView or an overlay
dismissed, and each attempt costs a forty minute CI cycle, so it is held out with
#398 carrying the exact error and the suggested fix. The list reports 22 run, 2
excluded.

editor.cy.ts still fails one test of fifty-five, and it failed the same way in
both runs: 'Expected <div.app-loading> not to exist ... continuously found', a
load timeout on test 28 rather than an assertion. One retry in run mode absorbs
that, none interactively; a real failure still fails twice and the retry's output
is in the log.
2026-10-28 22:20:32 +00:00
dc ad061946c3 docs(skills): add the estate deploy-and-test procedure
Build / Build-and-ng-test (pull_request) Successful in 6m56s
Lighthouse Checks / lighthouse (pull_request) Canceled after 19m45s
Build / Build-and-test-development (pull_request) Canceled after 16m16s
The shared-estate cycle is two npm scripts and sasjs test, but the seed in
npm run server is load-bearing: without it testsetup 404s settings.sas and
59 of the 64 test programs fail to complete, which reads like a broken
deployment. Record the targets, the token-pair credentials, the ADMIN group
the estate actually uses, how to read the results, and the frontend
passphrase steps as a repo-local skill, and point the testing doc at it.
2026-10-28 22:03:34 +00:00
allan 914a1606bb Merge pull request 'fix(cypress): register the deleteFolder task, restore the spec order, hold metanav out' (#396) from fix/cypress-release-specs into release/7.20
Sites / deploy (push) Successful in 3m40s
Build / Build-and-ng-test (pull_request) Successful in 6m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m38s
Build / Build-and-test-development (pull_request) Failing after 41m54s
Reviewed-on: #396
2026-10-28 21:37:20 +00:00
dc c803773d3c fix(cypress): register the deleteFolder task, restore the spec order, hold metanav out
Build / Build-and-ng-test (pull_request) Successful in 6m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m9s
Build / Build-and-test-development (pull_request) Failing after 41m32s
Four specs failed on release/7.20 once the list went from 19 to 24. Three of the
four are addressed here; the fourth is a spec that cannot pass and is now
excluded with an issue.

download.cy.ts - it clears the downloads folder in an afterEach through
cy.task('deleteFolder'), and that task was registered only in
cypress/plugins/index.js, which Cypress 10+ never loads unless it is required
from cypress.config.ts. The task is now registered in setupNodeEvents, where
Cypress looks, and the login in its beforeEach only fills the form when the form
is actually on screen - the unconditional version failed with 'the center of this
element is hidden from view' whenever the session was already live.

csv-limited.cy.ts and hook-programs.cy.ts - neither is broken; both depend on
where they sit in the run. csv-limited asserts the free-tier 'Locked Feature'
modal, which an earlier spec removes by planting a valid licence, and
hook-programs asserts an approval row against its target table before other specs
have added rows to the approval tables. The list is now the sequence that was
green for months, with csv-limited first and hook-programs second, and the newer
specs appended rather than inserted. The script says why order matters, because
the next person to alphabetise it will break the same two specs.

metanav.cy.ts - cannot pass, and says so in its own source: '// BLOCKER ...
.clr-accordion-header-button always null although it is present on the page'.
With the duplicate login in its beforeEach removed it now reaches that assertion
and fails there, so it is excluded from the list with #395 as the reason, and the
login fix is kept for whoever picks it up. The coverage check now distinguishes a
deliberate exclusion from a forgotten file, so nothing can go missing silently.

Not verified locally: the specs need an estate seeded by 'sasjs cbd -t server-ci'
plus makedata, which a scratch mock does not reproduce - the app never leaves
app-loading. The CI run is the verification.
2026-10-28 21:29:50 +00:00
allan b8eea5063c Merge pull request 'ci(build): run the Cypress suite in one job again, keeping the spec-list guard' (#394) from ci/unshard-cypress into release/7.20
Sites / deploy (push) Successful in 3m46s
Build / Build-and-ng-test (pull_request) Successful in 6m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m57s
Build / Build-and-test-development (pull_request) Failing after 37m40s
Reviewed-on: #394
2026-10-28 19:38:08 +00:00
allan 0a172b0c37 Merge pull request 'fix(ci): address the release workflow review findings' (#393) from fix/release-review-566-573 into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 6m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m10s
Build / cypress-shard-1 (pull_request) Failing after 13m19s
Build / cypress-shard-2 (pull_request) Failing after 13m58s
Build / cypress-shard-4 (pull_request) Failing after 14m23s
Build / cypress-shard-3 (pull_request) Successful in 17m37s
Build / Build-and-test-development (pull_request) Failing after 2s
Reviewed-on: #393
2026-10-28 19:37:37 +00:00
dc@4gl.io 888873421c fix(ci): publish against the tag this job cuts, not releases/latest
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
semantic-release bumps the working-tree package.json before the release
commit, so the tag is known in the job. Using it for the asset upload and
both dispatches means an overlapping release cannot redirect them.
2026-10-28 19:36:13 +00:00
dc@4gl.io af1813f829 fix(ci): use the lockfile and the pinned CLI in the docs job 2026-10-28 19:36:05 +00:00
dc@4gl.io 27686d3aad fix(ci): keep foreign release assets, install jq first, pin the CLI
Only the assets this workflow uploads are removed before a re-run, so a
hand-attached asset on the release survives. jq is installed before the
tag resolve that parses its output, the SASjs CLI is installed at the
version the tag's sas/package.json pins, and the npmrc secret is appended
rather than overwriting the file.
2026-10-28 19:35:57 +00:00
dc d84a1d50f4 ci(build): run the Cypress suite in one job again, keeping the spec-list guard
Build / Build-and-ng-test (pull_request) Successful in 6m49s
Lighthouse Checks / lighthouse (pull_request) Canceled after 1m17s
Build / Build-and-test-development (pull_request) Canceled after 0s
The four shards made the release-branch run red, and the reasons are in the
logs rather than in the theory:

  - every failure was a timeout, not an assertion - "cy.type() failed because
    the center of this element is hidden from view", "Expected
    <div.app-loading> not to exist ... continuously found".  Four jobs on one
    runner means four SASjs servers, four ng serve and four Chromes competing
    for the same CPU, and the app under test starves;
  - upload-artifact@v4 does not work on this runner at all:
    "GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ ...
    not supported on GHES", so the v3 -> v4 move I made on review advice broke
    the artifact step outright.

Measured, from the one sharded run that did complete: 20m34s wall against 33-35
minutes for the single job, with peak concurrency 3 - so a third off, not the
four-fold the split suggests.  That is not worth a flaky pipeline while a release
sits on it.

Kept from the sharding work, because those parts were the real gain:

  - the spec list has one home, .gitea/scripts/cypress-specs.sh;
  - Build-and-ng-test fails when a spec on disk is not listed, so the five that
    had never run in CI cannot go missing again - and they do run now;
  - videos are zipped and uploaded only on failure;
  - one job means one server, which is also what the guards and the balance
    arguments were protecting.

The shard and fan-in jobs are gone with the branch that carried them.
2026-10-28 19:27:51 +00:00
dc@4gl.io cf7b21a8b6 fix(ci): verify the forge certificate on the release API calls
Build / Build-and-ng-test (pull_request) Successful in 6m48s
Lighthouse Checks / lighthouse (pull_request) Canceled after 10m53s
Build / cypress-shard-1 (pull_request) Canceled after 1m20s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Failing after 3s
curl -k disables verification on calls that carry the release token, so a
man-in-the-middle on the runner's egress could capture a credential that
rewrites releases. The forge serves a publicly trusted certificate (the
runner verifies the same chain on every checkout), so the flag is
unnecessary: drop it from all fifteen calls in the two release workflows.
2026-10-28 19:21:57 +00:00
allan 60589c78b6 Merge pull request 'fix(embed): report a name-keyed va_parmap entry, and keep the app query on the standalone URL' (#391) from fix/va-parmap-review-572 into release/7.20
Sites / deploy (push) Successful in 3m46s
Build / Build-and-ng-test (pull_request) Successful in 6m19s
Build / cypress-shard-1 (pull_request) Failing after 13m58s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m17s
Build / cypress-shard-2 (pull_request) Failing after 14m33s
Build / cypress-shard-3 (pull_request) Failing after 4m36s
Build / cypress-shard-4 (pull_request) Failing after 14m10s
Build / Build-and-test-development (pull_request) Failing after 6s
Reviewed-on: #391
2026-10-28 19:02:31 +00:00
allan 8e2cce4150 Merge pull request 'ci(build): split the Cypress suite into four shards, and keep videos for failures' (#392) from ci/parallel-cypress into fix/va-parmap-review-572
Build / Build-and-ng-test (pull_request) Canceled after 10s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 11s
Build / Build-and-test-development (pull_request) Failing after 2s
Reviewed-on: #392
2026-10-28 19:01:59 +00:00
dc@4gl.io f7e753863b fix(deploy): strip the embed parameters by name, and scope the timezone runs to the VA specs
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 1s
The standalone URL dropped every query parameter whose name started with
va_, not just va_parmap; the names are now enumerated, and a later
parameter with that prefix survives (the component spec pins the exact
standalone URL instead of a substring, so the query surviving is the
assertion, not an accident of the runner's own query).

The timezone runs are scoped to the VA specs, the only specs with local
calendar arithmetic, so the blocking job no longer runs the whole suite
twice more.
2026-10-28 18:58:26 +00:00
dc febfea3e5e ci(build): make the fan-in gate fail, and drop the duplicated blocks
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The gate could not fail.  With needs: on four shards and no if: always(), a
failed shard skips the fan-in job instead of failing it, and a skipped job is
not a failure on the one check branch protection watches - so the gate was
silent in exactly the case it exists for.

It now runs with if: always() and fails on any shard result that is not
success.  The count is checked as well: if the needs context does not resolve on
this runner the step fails saying so, rather than looping over nothing and
reporting green - which is the failure mode a bare loop would have.

Also removes what the last regeneration left behind: the coverage guard step
appeared twice in Build-and-ng-test (so it ran twice) and the shard 1 comment
block appeared three times.  The file is regenerated from fix/va-parmap-review-572
rather than from my own previous commit, which is what caused the duplication.
2026-10-28 18:55:18 +00:00
dc 3580d6e9eb ci(build): gate the shards behind the job name branch protection already requires
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / cypress-shard-1 (pull_request) Canceled after 0s
Build / cypress-shard-2 (pull_request) Canceled after 0s
Build / cypress-shard-3 (pull_request) Canceled after 0s
Build / cypress-shard-4 (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Renaming the shards to cypress-shard-1..4 and adding a job named
Build-and-test-development that does nothing but needs all four.

That name is the status check the branch protection rule requires, so the check
now turns green only when every shard passed - a red shard 3 leaves it
unreported and the PR unmergeable, which is the behaviour the single job had.
No repo setting has to change, and adding a fifth shard later means adding it to
needs rather than remembering to edit branch protection, where forgetting fails
silently.
2026-10-28 18:31:27 +00:00
dc bc9e3ae50f ci(build): give the shard lists one home, cover the five missing specs, and guard it
Build / Build-and-ng-test (pull_request) Successful in 7m36s
Lighthouse Checks / lighthouse (pull_request) Canceled after 1m41s
Build / Build-and-test-development (pull_request) Canceled after 0s
Build / Build-and-test-development-2 (pull_request) Canceled after 0s
Build / Build-and-test-development-3 (pull_request) Canceled after 0s
Build / Build-and-test-development-4 (pull_request) Canceled after 0s
Review found that five specs ran in no shard - coltype-delete-record,
download, embed-va, metanav, submitted-details - and one of them is the VA
embed suite this release added, 454 lines that had never run in CI.  The old
single --spec string omitted them too, so nothing regressed; they were simply
never wired up, and nothing in the repo marked them skipped.

The spec lists now live in .gitea/scripts/cypress-shards.sh, one place, so the
workflow no longer spells them out four times.  A guard step in
Build-and-ng-test compares the shards against the specs on disk and fails when a
file belongs to none of them or to more than one - proven by removing a name and
watching it fail with that file listed.

Rebalanced for the five additions, on test count and spec count together, since
each spec pays a few seconds of startup:

    shard 1  editor, csv, metanav                                    57 tests
    shard 2  excel, filtering, download, browser-info,
             excel-multi-load, hook-programs                         49 tests
    shard 3  embed-va, viewbox, licensing, filter-panel, liveness     43 tests
    shard 4  viewer-labels, row-cell-limits, derived-variables,
             ddl-export, coltype-delete-record, submitted-details,
             startupservice-launcher, stage, full-table-search,
             csv-limited                                             27 tests

24 specs / 176 tests, each in exactly one shard.

upload-artifact moved from the retired v3 to v4; this change multiplies that
step by four, so it is worth not carrying the dead major forward.

Not in this commit: the three new check names still need adding to branch
protection, which needs repo admin.
2026-10-28 18:26:45 +00:00
dc 330448c496 ci(build): split the Cypress suite into four shards, and keep videos for failures
Build / Build-and-ng-test (pull_request) Successful in 7m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 0s
Build / Build-and-test-development-2 (pull_request) Canceled after 0s
Build / Build-and-test-development-3 (pull_request) Canceled after 0s
Build / Build-and-test-development-4 (pull_request) Canceled after 0s
Sits on top of fix/va-parmap-review-572 (#391) so the two changes to this
workflow land in one order; the timezone step that PR adds to
Build-and-ng-test is untouched.

The development job ran all nineteen specs in one Cypress invocation on one
runner, so its wall time was the sum of every spec - around half an hour on a
green run, with the whole pipeline waiting on it.

Four jobs now, each with its own runner and its own SASjs server, taking a
subset:

    shard 1  editor (55 of the suite's ~148 tests)
    shard 2  excel, filtering, csv-limited, csv, stage
    shard 3  viewbox, licensing, filter-panel, browser-info, excel-multi-load, liveness
    shard 4  viewer-labels, row-cell-limits, derived-variables, ddl-export,
             hook-programs, startupservice-launcher, full-table-search

The split is balanced on test count, not file count: editor.cy.ts alone holds 55
tests, so an even spread of files would leave one shard running twice as long as
the others.  Its own duration is the floor the pipeline can reach.

The setup steps are identical in every shard, so they are defined once and
aliased with a YAML anchor; each job differs only in SHARD and SPECS.  Shard 1
keeps the original job name so an existing required status check still matches.

Video is still recorded - measured at 91s with it and 92s without on the same
spec, so the recording is not what takes the time.  What it did cost was the zip
and artifact upload of every run's videos, now guarded by if: failure().
2026-10-28 18:21:38 +00:00
dc 00257f9160 ci(client): run the client suite in two more timezones
Lighthouse Checks / lighthouse (pull_request) Successful in 20m42s
Build / Build-and-ng-test (pull_request) Successful in 7m8s
Build / Build-and-test-development (pull_request) Canceled after 0s
2026-10-28 18:04:01 +00:00
dc e51953ca8a fix(deploy): keep the app query when opening the standalone URL 2026-10-28 18:03:53 +00:00
dc aa91daf9f7 fix(embed): record the parameter label only, so a name-keyed va_parmap is reported 2026-10-28 18:03:45 +00:00
allan 00df87f189 Merge pull request 'fix(makedata): pair the %else of the keepme block so it parses in open code' (#389) from fix/makedata-selfdestruct into release/7.20
Sites / deploy (push) Successful in 3m43s
Build / Build-and-ng-test (pull_request) Successful in 5m5s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m48s
Build / Build-and-test-development (pull_request) Canceled after 32m30s
Reviewed-on: #389
2026-10-28 17:09:17 +00:00
allan 65699bdd39 Merge pull request 'fix(deploy): do not offer the deploy wizard inside a report' (#388) from feat/embed-not-configured into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m6s
Build / Build-and-test-development (pull_request) Successful in 34m34s
Reviewed-on: #388
2026-10-28 17:08:40 +00:00
dc 459ff2da37 fix(deploy): one definition of the standalone URL, and test the guards
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Successful in 33m54s
Review follow-up on the embed guard.

The spec asserted the isEmbedded getter, which mirrors the template rather
than exercising it: dropping !isEmbedded from the @if guards left every test
green. The spec now renders the component in a TestBed and asserts the DOM -
#TCS present standalone and absent for embed=va and embed=true, and the
deploy section withheld after the terms are accepted when embedded. Both
guards were mutation-checked by removing them, which fails the spec.

deployUrl and onNavigateToHome computed the standalone URL two different
ways: origin+pathname in one, location.href.split('#')[0] in the other,
which kept the query string, so a tab opened from ?embed=va#/deploy stayed
embedded. Both now call standaloneUrl() in standalone-url.ts, which parses
the href and drops the query and the fragment. It is tested with a
hash-bearing input, and the navigate test asserts the opened URL carries
neither '?' nor '#', so a regression to split('#') fails.
2026-10-28 17:05:35 +00:00
dc c0ea810766 fix(makedata): pair the %else of the keepme block so it parses in open code
Build / Build-and-ng-test (pull_request) Successful in 4m54s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m1s
Build / Build-and-test-development (pull_request) Successful in 32m10s
The self-destruct block ends with a single-statement %else in open code:

    %if &keepme ne 1 %then %do;
      %mv_deletejes(path=&root/admin, name=makedata);
    %end;
    %else %put NOTE: &_program retained, as KEEPME was requested;

Outside a macro SAS pairs %else only with a %do branch, so this stops with
'ERROR: Expected %DO not found' and 'ERROR: Skipping to next %END statement'
after the deployment has completed. On Viya the job then reports state error
with jobConditionCode 1012, so a successful deployment looks like a failure
exactly when someone is watching a first install.

Both branches now use %do/%end, which is the pattern the repo uses elsewhere
for this reason - see db/migrations/20261003_row_and_cell_limits.sas.

The same block is fixed in the server variant. The two delete calls also gain
their terminating semicolons: the macro call worked without them, but their
absence is what made the block look responsible for the error.

Behaviour is unchanged: the program still deletes itself unless KEEPME was
requested.
2026-10-28 16:38:21 +00:00
allan 62e43034e4 Merge pull request 'feat(embed): resolve va_parmap values through named date anchors' (#386) from feat/va-parmap-transforms into release/7.20
Sites / deploy (push) Successful in 3m46s
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m7s
Build / Build-and-test-development (pull_request) Successful in 34m1s
Reviewed-on: #386
2026-10-28 15:08:45 +00:00
dc 80e8911af8 fix(deploy): do not offer the deploy wizard inside a report
Build / Build-and-ng-test (pull_request) Successful in 5m10s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m36s
Build / Build-and-test-development (pull_request) Successful in 33m18s
Deployment needs an authenticated, standalone session, and the person
looking at a report object is not the person who installs the app. Offering
the wizard there produces a screen whose controls cannot work - the failure
we hit while testing a report: T&C accepted, credentials filled, DEPLOY
clicked, and nothing happened, because the embedded app has no session to
deploy with.

An embedded instance (embed=va or embed=true) now shows what the state is
and where to deploy from, instead of a form that cannot submit. Standalone
behaviour is unchanged.
2026-10-28 15:08:02 +00:00
dc bd3c220af2 fix(embed): never build a clause from a value that did not convert, and say so in the banner
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m25s
Build / Build-and-test-development (pull_request) Successful in 33m8s
Two review findings, both about a filter that looks applied and is not:

- The transform branch of the clause builder interpolated whatever
  convertJsDateToSasDate returned, so an Invalid Date reaching the helper
  would have put a literal NaN in the WHERE clause. A non-finite result is
  now dropped, and unsupportedTransforms reports the parameter, so the
  author sees a banner instead of a broken filter.
- The range count check counted values that survived date parsing, while the
  builder counts resolved bounds. A RANGE sent two values where one did not
  parse therefore built a BETWEEN covering one period with no warning. The
  check now counts what VA sent, the builder refuses a range with more than
  one value rather than picking one, and a value set that is partly
  unparseable gets its own line ('1 of its 2 values was not a date - the
  filter covers only the rest').

The banner also now says what to do: it names the URL parameter to edit and
shows the entry grammar with an example, since the reader is the report
author and has no other way to act on it.
2026-10-28 12:44:05 +00:00
allan 5e8910f238 Merge pull request 'feat(embed): map VA parameters to DC columns from the URL' (#383) from feat/va-param-map into release/7.20
Sites / deploy (push) Successful in 3m44s
Build / Build-and-ng-test (pull_request) Successful in 4m58s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m54s
Build / Build-and-test-development (pull_request) Successful in 33m9s
Reviewed-on: #383
2026-10-28 12:41:09 +00:00
dc 221dd8a15e Merge feat/va-param-map: label-only mapping keys and value-aware same-column warning
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m49s
Conflict was valueSignature: the transform branch already builds a clause
value set through buildClauseValueSet, so the helper delegates to it and
takes the parameter's transform, keeping derived dates in the comparison.
2026-10-28 12:34:32 +00:00
dc e9216b0895 fix(embed): match a mapping on the parameter label only, and warn only when the values differ
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Canceled after 0s
Two review findings, plus a scope decision.

A va_parmap key is now matched against the parameter's label only. Matching
the internal name (pr273) as well meant one parameter answered to two keys,
so an entry written against the name and one against the label could never
be told apart: the label entry won and the name entry was silently dead -
no parse problem, no unresolved line, no unmatched line. Dropping the
fallback removes the ambiguity, and an entry keyed on the internal name now
shows up as an unmatched mapping like any other typo.

sameColumnMappings no longer reports two parameters on one column when they
carry the same value: col = 'x' AND col = 'x' filters exactly as one clause
does, yet the banner asserted that different values would show no rows. The
clause value each parameter produces is now compared, and the warning is
raised only when they differ. Parameters that produce no clause at all are
excluded, since they cannot conflict with anything.
2026-10-28 12:32:47 +00:00
dc 37d81c902a test(embed): run the suite in two more timezones
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m37s
Build / Build-and-test-development (pull_request) Canceled after 0s
A transform resolves a report value to dates using local calendar
components, so a test that passes in UTC says nothing about a browser west
of it. test:tz-west and test:tz-east run the whole suite under
America/New_York and Pacific/Auckland; with the UTC construction this test
pair replaces, the bare-number round-trip case fails under the first.

Not wired into CI: it triples the run for a property that is already
pinned by the round-trip assertion, so it stays a local and pre-release
check.
2026-10-28 12:27:42 +00:00
dc 689b5b6709 Merge feat/va-param-map into feat/va-parmap-transforms
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m29s
Build / Build-and-test-development (pull_request) Successful in 33m58s
Brings in the diagnostics fixes from the base branch; conflicts were the
paramMap type widening to VaParmapTarget and the new paramsStable argument,
both of which the transform branch already carries.
2026-10-28 12:24:36 +00:00
dc 10157c1159 fix(embed): resolve transform values as local dates and give ranges whole days
Review findings:

- A bare SAS number was turned into a UTC instant and then read with local
  accessors, so west of UTC the value landed a day early and the anchor was
  computed from the wrong quarter; the round trip to a SAS day number was
  one out. Built with local calendar construction instead, matching
  parseDate and convertJsDateToSasDate.
- RANGE passed keepTime to both bounds, so a DATETIME range started at the
  value's clock time on day one and ended at it on the last day, dropping
  the rest of both boundary days. A range now covers whole days: period
  start at midnight, period end at 23:59:59.
- A RANGE fed more than one parameter value produced no clause and no
  warning, reading as a filter that matched nothing. One BETWEEN cannot
  express several periods, so the count mismatch is now reported.
2026-10-28 12:20:49 +00:00
dc 541a61ba08 fix(embed): make the va_parmap diagnostics say what they mean
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s
Build / Build-and-test-development (pull_request) Successful in 33m26s
Review findings, all about a warning that misleads:

- The banner heading said the mapping was ignored, but two of the three
  lists it renders are mappings that are applied (the AND-ed pair) or that
  were never applicable (a parameter VA does not send). Neutral heading.
- unmatchedMappings reported a key as unsent as soon as any parameter had
  been seen. VA sends the parameters a report object's query uses, so a
  name that has not arrived yet looked like a typo. A mapping is now only
  judged once a message introduces no new parameter name, i.e. once the
  object's parameter set has settled.
- sameColumnMappings counted a column the table does not have, so one
  mistake produced the unresolved-mapping line and the AND line. A column
  that is not in the table is now left to unresolvedMappings.
- The Cypress cases asserted exact row counts, which encode the demo
  table's contents. They now read the unfiltered count from the app and
  assert the filtered grid is smaller (or empty), so a fixture change
  cannot fail a behaviour test.
2026-10-28 12:20:37 +00:00
allan ad50acf647 Merge pull request 'fix(ci): publish the typedoc to the webdoc site, not the app default' (#385) from fix/webdoc-named-site into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #385
Reviewed-by: Allan <allan@4gl.io>
2026-10-28 12:10:16 +00:00
dc cd1ce4a7e2 test(embed): format the transform spec to satisfy prettier --check
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
2026-10-28 12:01:11 +00:00
dc 7e3045a21b fix(embed): build quarter ends from the calendar, not minus 24 hours
Build / Build-and-ng-test (pull_request) Failing after 2m2s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Canceled after 6m38s
Subtracting a fixed day from the next quarter's start moves by an hour
across a daylight-saving transition, which would place a DATETIME bound
off the intended midnight. Day 0 of the following month is the last day
of the month by construction.
2026-10-28 11:57:23 +00:00
dc 88010c3980 feat(embed): resolve va_parmap values through named date anchors
Build / Build-and-ng-test (pull_request) Failing after 1m55s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m22s
A VA parameter can now carry a transform on the column side of a mapping:

  va_parmap=SELECTED_DATE-D_DATE:ADD(QUARTER_START,-1)

which filters D_DATE on the report value AND the same point one quarter
earlier, as a single IN clause - so the AND rule, the clause signature,
saveQuery and the reload loop are untouched.

Transforms resolve to one clause with these operations:

- ADD(<ANCHOR>,<offset>)   the value plus one anchor beside it -> IN
- RANGE(<PERIOD>,<offset>) a closed period -> BETWEEN

Anchors: QUARTER_START/END, MONTH_START/END, YEAR_START/END.
Periods: QUARTER, MONTH, YEAR. Offset is a whole number of periods,
defaulting to 0.

The names describe the result rather than mirroring SAS intnx: the
arithmetic runs in the browser, so mirroring intnx would mean
reimplementing its alignment letters and their corners (week alignment,
's' and 'm') with no guarantee of matching SAS. A report author can read
ADD(QUARTER_START,-1) without knowing intnx.

Resolution is anchored on the parameter value, never on today, so a saved
report reproduces the same rows. A transform that cannot run where it is
used - a non-temporal column, a value that is not a date - is skipped and
reported in the existing warning banner rather than silently filtering on
the raw value.
2026-10-28 11:52:53 +00:00
dc c5808a16eb fix(ci): publish the typedoc to the webdoc site, not the app default
Build / Build-and-ng-test (pull_request) Successful in 5m42s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m39s
Build / Build-and-test-development (pull_request) Canceled after 19m50s
`surfer deploy` chooses its target only from `--site-name`; without one it
replaces the app's DEFAULT site. The typedoc job passed `--server
webdoc.datacontroller.io` and no site name, so the 7.19.0 docs were published
over the default site of that app - the marketing site served at
datacontroller.io - instead of to a webdoc site.

Name the site, and give it its alias domain so the first run creates it.
The SAS docs job is left unnamed on purpose: code.datacontroller.io is its own
app whose default site is the docs, and a comment now says so.
2026-10-28 11:49:25 +00:00
dc 0011f7e933 fix(embed): log a parse-time mapping problem once
Build / Build-and-ng-test (pull_request) Successful in 5m0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m41s
Build / Build-and-test-development (pull_request) Successful in 31m54s
The parse-time problems were logged at app init and then again by the
editor's warning loop, which spreads them into the banner's text. The loop
now logs only what the instance discovers for itself (unresolved column,
unmatched key, AND-ed pair); the parse problems stay in the banner and keep
their single init-time console line.

E2e case 11 pins it: one unparseable entry plus one valid-but-missing
mapping, so the loop certainly runs and logs its own line, and exactly one
line mentions the parse problem.
2026-10-28 11:13:10 +00:00
dc 4ed1e0684d test(embed): prove va_parmap case-insensitivity end to end
Build / Build-and-ng-test (pull_request) Successful in 5m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m42s
Build / Build-and-test-development (pull_request) Successful in 33m51s
2026-10-28 09:57:58 +00:00
dc 2623a3fb4b fix(embed): defer the column check until metadata loads, report AND-ed pairs
Build / Build-and-ng-test (pull_request) Successful in 5m50s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m50s
Build / Build-and-test-development (pull_request) Successful in 33m51s
Two review findings, both paths where a filter could look applied while
producing nothing.

- `unresolvedMappings` judged a mapping against an empty column list while the
  grid was still loading, so every entry was briefly reported as naming a
  column the table does not have. An empty or unset `cols` now means "not known
  yet" and returns nothing, so the banner appears only once metadata exists.
- Two parameters mapped to one column are AND-ed, so different values yield no
  rows at all, and nothing said so. `sameColumnMappings()` now reports the
  contended column and both parameters. The clauses are deliberately left as an
  intersection: turning that AND into an OR is a filter-semantics decision, not
  a diagnostic, so it is tracked separately.

Covered by unit tests for both (unloaded column list, one line per contended
column) and e2e case 10, which drives two parameters onto one column and
asserts the banner plus the empty grid.
2026-10-28 09:54:48 +00:00
dc 388f418bf4 test(embed): pin the semantics of two params on one column
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m39s
Build / Build-and-test-development (pull_request) Failing after 35m57s
Records that two parameters mapped to the same DC column are AND-ed, so the
result is the intersection: two different dates match no rows rather than
both dates' rows. Answers the 'filter one column on two dates' case from the
code rather than by assumption, and marks the single-parameter IN (...) shape
as the one that unions.
2026-10-28 09:35:36 +00:00
dc f83dc359e2 fix(embed): one parameter maps one column, and report typos on both sides
Build / Build-and-ng-test (pull_request) Successful in 5m42s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m20s
Build / Build-and-test-development (pull_request) Successful in 33m57s
Three follow-ups from review, all the same class of defect: an author error
that leaves an unfiltered grid looking like a working filter.

- A duplicate key in `va_parmap` silently kept whichever entry came last, so
  `va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER` filtered OTHER with nothing to
  show for it. One VA parameter now maps one column: the first entry wins and
  a conflicting duplicate is reported. The same mapping written twice is not a
  conflict, and two different parameters mapping to one column stay valid -
  that is how two report objects filter the same column from their own
  parameters.
- A parameter-side typo (mapping `DROPDOWN_TAB1` when the report parameter is
  `DROPDOWN_TAB_1`) produced no clause, no banner and no console line.
  VaMessagingService now records every parameter key it has seen, across
  editor reloads, and the editor reports a mapping key that VA has never sent.
  Judged only once parameters are actually arriving, because VA sends just the
  parameters its query uses - an absent key is not on its own an error, and a
  key that turns up later stops being reported.
- The deferred first load cannot check a mapped column against metadata. On
  SAS `mp_filtercheck` aborts the save, so the load falls back to unfiltered;
  the banner now covers that path too, and an e2e case seeds the pre-bootstrap
  message buffer to prove it (and to prove a valid mapping filters the very
  first getdata).
2026-10-28 09:34:02 +00:00
dc 411f6c3354 fix(embed): warn when a va_parmap entry is ignored
Build / Build-and-ng-test (pull_request) Successful in 5m48s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m9s
Build / Build-and-test-development (pull_request) Successful in 33m7s
A `va_parmap` entry that cannot be used was dropped in silence, in both
places it can fail: at parse time (no `-` separator, a missing side, or a
column that is not a plain SAS name) and at apply time (the mapped column is
not in the table being opened). Either way the parameter is skipped, which
degrades to "no filter" - a typo in the report's URL then looks exactly like
a working filter over the whole table.

Both are now reported in terms the report author can act on:

- a warning banner in the editor (VA mode) naming the parameter, the column
  and the table, and pointing at the `va_parmap` entries in the URL
- console.warn for each distinct problem, so it is findable from the report's
  console too

The parser returns `{ map, problems }` rather than a bare Map, VaFilterService
gains `unresolvedMappings()` for the entries the loaded table cannot satisfy,
and the editor merges both sets for the banner (logging once per distinct set
rather than once per VA message).

The banner is covered by two e2e cases, one asserting its painted box inside
the viewport (VA hides the app header, so it has to stand on its own) and one
covering a parse-time failure before any VA message arrives. The alert markup
keeps the message inside the alert's text flow so it wraps, instead of being
laid out beside a title and pushed off the right edge.
2026-10-28 09:08:30 +00:00
allan f042ca82ea Merge pull request 'fix(ci): publish the docs with surfer 7, and rebuild release assets on demand' (#382) from fix/ci-surfer-app-password into release/7.20
Sites / deploy (push) Successful in 3m49s
Reviewed-on: #382
2026-10-28 08:49:33 +00:00
dc 15a1f34d52 feat(embed): map VA parameters to DC columns from the URL
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m20s
Build / Build-and-test-development (pull_request) Successful in 32m46s
Two DC instances over the same table in one VA report could not filter the
same column from different parameters: parameter -> column matching is by
name, and two VA parameters cannot share a label.

The `va_parmap` URL parameter overrides that match per instance, one entry
per mapping (`va_parmap=<VA parameter>-<DC column>`, repeated for more):

  ...?embed=va&va_parmap=REGION_TAB1-REGION

The mapping is additive (unmapped parameters keep the name match), the
column is validated as a plain SAS name at parse time, and an entry only
applies where the named column exists in the table being opened - a miss
skips that parameter rather than falling back to the same-name match the
mapping exists to override. Entries split at the last hyphen, so a label
may itself contain hyphens, and the key matches the parameter label first,
then its internal name (pr###).

Parsed once at app init onto globals (like embed / vaApplyMode) so the
editor reloads a filter apply performs keep the mapping, and both filter
paths (the metadata build and the deferred initial load) honour it.

Also repairs embed-va.cy.ts: a cy.visit differing from the current URL
only in the hash is a same-document navigation, so ?embed=va never took
effect and both existing cases failed; their layout assertions also
described an earlier design (VA now opens read-only with an Edit button
and hides Filter). Cases 3 and 4 drive two mappings of the same column
through a real postMessage and assert the applied filter and row count.
2026-10-28 08:41:15 +00:00
dc 6486c80ed4 fix(ci): rebuild and upload release assets on demand
Build / Build-and-ng-test (pull_request) Successful in 5m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m33s
Build / Build-and-test-development (pull_request) Successful in 31m45s
2026-10-28 02:07:31 +00:00
dc dd482c4069 fix(ci): publish docs after the assets, with the webdoc app password 2026-10-28 02:05:10 +00:00
dc f58925b299 fix(ci): install doxygen for the SAS docs job 2026-10-28 02:02:57 +00:00
dc 25ffa71add fix(ci): publish the code documentation with surfer 7 and app passwords 2026-10-28 01:57:23 +00:00
semantic-release-bot 424b111737 chore(release): 7.19.0 [skip ci]
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)

### Bug Fixes

* address the review findings on the harness and refresh changes ([36e4e88](36e4e88eb1))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](698dea6d6b))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](0d58be975d))
* **client:** open the SAS log page with noopener ([072a133](072a133702))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](1fb4818d21))
* **cypress:** apply the licence key when the estate has none ([be8f445](be8f445ae1)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](fed43cb07e))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](f5aade5976))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](1e224763ba))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](9b3cf50b9c))
* **derived:** address five review findings ([ccd468b](ccd468b335))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](be50cb5d08))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](9914790777))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](9ef813d439))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](422f766cd4))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](7830b7bcfa))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](2d5de9a324))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](6a1128a463))
* **derived:** eight review findings ([561d7fc](561d7fc3ed))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](1863076bc9)), closes [#366](#366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](d3dec682aa))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](469229aecb))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](7fb981daa9))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](5f5d8681b9))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](5f443b0b16))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](6b5bf3e6be))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](85c89ca2a4))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](f8f65681e1))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](7efbf31ea5))
* **derived:** only the delete marker is a reserved name ([af31c36](af31c36b67))
* **derived:** open the statements file with a long record length ([ee31782](ee31782fbf))
* **derived:** pair the filterds %else with %do/%end ([88a38db](88a38db877))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](27ddcfb575))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](7799a300df))
* **derived:** reject a dependency on a row this submission deletes ([8806504](8806504967))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](9a047ad42e))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](e0da9f1098))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](688550a28d))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](1b5c4b3ecd))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](4d9ad38df1))
* **editor:** a blank truncation note is not a warning ([f182179](f182179403))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](04ce1861e7))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](4bcee706ff))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](45dce7b11b))
* **getdata:** clear the globals it declares before use ([359b570](359b570b85))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](059cdf176a))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](661ee19cfa))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](355c46a7b6))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](d626cb8f2d))
* **mocks:** address the audit-pack review findings ([489819b](489819b051))
* **mocks:** carry the zip flags into the central directory ([c4021a4](c4021a42ff))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](62f55db170))
* **mocks:** copy the physical column list before deriving ([f411b7d](f411b7d269))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](60ec470404))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](74060c0585))
* **mocks:** drop derived variables before staging ([73d25b1](73d25b1220))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](347dbb53b0))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](ee3c8787ef))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](6cad23fce9))
* **mocks:** search the physical columns only in the viewer ([293022d](293022d46b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](fb36379520))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](9c294c0ab5))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](0a52d20c5f))
* **mpe_execute:** fail fast when the child left no status ([a036386](a03638667f))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](220408538b))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](f2b1428ec3))
* name the startupservice launcher button "Show log" ([3445107](3445107d9a))
* offer the launcher on the globvars failure modal too ([ad2da45](ad2da45389))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](46bacdd6d9))
* **refresh:** one audit folder per refresh call ([56b6620](56b6620ef4))
* **sites:** round-trip the committed licence before publishing ([4d462d3](4d462d310f))
* **test:** getdata test ([da30266](da30266278))
* **tests:** enable closeouts logging ([fdcd8fa](fdcd8fa4b3))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](033ffd3025))
* **ui:** one error modal at a time ([81ccf75](81ccf758e9))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](42355f17bc))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](337ea8e5e0))

### Features

* **dc_getlibs:** dump the library list when debug is on ([582523d](582523dc87))
* **derived:** allow a variable to shadow a physical column ([12beeff](12beeff695))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](813bce6856))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](7c9ce9a712))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](87e794f769))
* **derived:** report the dependency cycle path ([cee1a95](cee1a954ee))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](78e88a4112))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](0b010f2c65))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](aa6d50e1ea))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](33d043ea92))
* **editor:** derived variables in the EDIT screen ([b0fd354](b0fd354ecd))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](6513d09f05))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](34691dceec))
* **editor:** show the derived column label in the header tooltip ([cfe6647](cfe6647ca8))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](752d944c20))
* **mocks:** implement derived variables in the JS services ([b4fae88](b4fae88ae1))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](5d183d1350))
* run startupservice in a new window from its error modal ([40f6133](40f61331aa))
* **sites:** mint a licence for the published sites ([7d9df71](7d9df71fdb))

### Reverts

* **derived:** no shadowing, and no self reference ([5af8e6f](5af8e6fc35))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](31aa859153))
2026-10-28 01:42:51 +00:00
allan 849153d4a2 Merge pull request 'ci(release): build the frontend before deploying mocked services' (#381) from fix/ci-release-build-frontend into main
Release / Build-production-and-ng-test (push) Successful in 4m42s
Release / Build-and-test-development (push) Successful in 25m42s
Release / release (push) Failing after 6m56s
Reviewed-on: #381
2026-10-28 01:08:49 +00:00
dc bc9e0996ec ci(release): build the frontend before deploying mocked services
Build / Build-and-ng-test (pull_request) Canceled after 1m32s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m52s
2026-10-28 01:07:00 +00:00
allan 66039c91d7 Merge pull request 'feat: release 7.19 - derived variables, startupservice launcher, core v6' (#348) from release/7.19 into main
Release / Build-production-and-ng-test (push) Successful in 4m13s
Release / Build-and-test-development (push) Failing after 1m52s
Release / release (push) Skipped
Reviewed-on: #348
2026-10-28 00:52:14 +00:00
allan abb1ecbcab Merge pull request 'fix(validatefilter): drop the dead nobs assignment, keep targetds branches exclusive' (#380) from fix/validatefilter-branches into release/7.19
Sites / deploy (push) Successful in 4m2s
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m9s
Build / Build-and-test-development (pull_request) Successful in 33m26s
Reviewed-on: #380
2026-10-28 00:07:31 +00:00
dc 337ea8e5e0 fix(validatefilter): drop the dead nobs assignment, keep targetds precedence
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Review feedback on the merged fix:

- derived_nobs was assigned twice, the first one dead - the coalesced form below
  is the live value.  Removed, so there is one assignment to read.
- making the catalog test a plain %if after the derived-variables test let the
  catalog branch override it; the derived view won before and still should.  The
  catalog branch now requires derived_nobs=0, so the view keeps precedence and
  only a catalog with no derived configuration is blanked.

Two guarded if blocks, not %else %if: that construct is what the parser rejected
in this file before ('Expected DO not found'), and the first attempt at this
review point reproduced it - validatefilter and viewdata both failed to complete
again until it was replaced.

Verified: sasjs test -t new -> 64/64 (100%), TEST=0.
2026-10-28 00:05:19 +00:00
allan 709acfad73 Merge pull request 'fix(validatefilter): build targetds without macro triggers in comments' (#379) from fix/validatefilter-targetds into release/7.19
Sites / deploy (push) Successful in 3m50s
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 14m43s
Reviewed-on: #379
2026-10-27 23:46:46 +00:00
dc 42355f17bc fix(validatefilter): build targetds without macro triggers in comments
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m37s
Build / Build-and-test-development (pull_request) Successful in 33m19s
The suite failed to complete on the estate with

  ERROR: Expected %DO not found.
  ERROR: Skipping to next %END statement.

and, once the parser had lost sync, 'Apparent symbolic reference IS_FMTCAT not
resolved' - because the %let that defines it never ran.

Two causes, both in the code that decides which dataset the filter is validated
against:

- the comment explaining the catalog case contained the text '%if ...' and the
  macro processor resolves triggers even inside a comment, so the prose was
  executed as code and tripped the parser from there.  The comment now avoids
  macro triggers entirely.
- an empty value compared in a condition is a character operand, and
  %mf_nobs returns nothing when the dataset is missing.  It is wrapped in
  coalesce now, and the catalog flag is computed with %eval instead of a nested
  %if.

targetds is still blanked for a LIBREF.CATALOGNAME-FC name - which is what makes
mp_filterstore rewrite it to a cntlout dataset - and stays the plain libds
otherwise, as an empty targetds falls back to libds in the engine anyway.

The test now passes outref= and mdebug=&sasjs_mdebug rather than a hard-coded
1: the suite already owns that switch, and testinit turns it on by itself when
the run asks for _debug=2477 or 131, which is when the child's response and the
PROC HTTP trace are worth having.

Verified: sasjs test validatefilter -t new -> 3/3 (100%), six assertions, all
PASS, including the catalog and derived-variable cases.
2026-10-27 23:46:19 +00:00
dc 6ef2eec011 ci(sites): start the SASjs server with HTTP_PORT=5000
Sites / deploy (push) Successful in 3m45s
Build / Build-and-ng-test (pull_request) Successful in 5m5s
Build / Build-and-test-development (pull_request) Canceled after 5m11s
Lighthouse Checks / lighthouse (pull_request) Canceled after 7m59s
The Sites workflow was the one place that still started the server without a
listener, so it died with

  [error] Invalid environment variable(s) provided:
   - no listener is enabled
   - set HTTP_PORT and/or HTTPS_PORT (either may be used alone)

and the following step's 120s poll timed out. build.yaml, release.yaml and
lighthouse.yaml already carry the same line.
2026-10-27 23:26:26 +00:00
allan 1ef3e0a97b Merge pull request 'fix(refresh): one audit folder per refresh call' (#378) from fix/refresh-etlsource into release/7.19
Sites / deploy (push) Canceled after 2m14s
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 13m28s
Reviewed-on: #378
2026-10-27 23:12:33 +00:00
dc 56b6620ef4 fix(refresh): one audit folder per refresh call
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 18m57s
refreshcatalog.test and viewtables.test failed to complete with

  ERROR: could not create /tmp/DataController/<libref>/dc_staging/refreshcatalog:mpe_refreshcatalogs: + <stamp>

The staging folder a closeout writes its statements to is
&dc_staging_area/&etlsource, and etlsource defaulted to
'refreshcatalog:<step>:%mf_uid()'. A default is resolved when the macro is
compiled, not when it is called, so every refresh in a session reused the same
folder - the first step created it, the next one could not, and the abort took
the whole suite down. Whichever suite reached the staging area first decided
which others failed, which is why the count moved between runs.

etlsource now defaults to 0 and is set per call from
%mf_getuniquename(prefix=<step>): unique, a plain name rather than a path, and
resolved at the point of use.

Verified on the estate: refreshcatalog 4/4 (100%), viewtables 2/2 (100%).
2026-10-27 23:08:47 +00:00
allan fd272eec89 Merge pull request 'test(editor): assert the derived block marker, not a column name' (#377) from fix/derived-block-spec into release/7.19
Sites / deploy (push) Failing after 5m38s
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 1m32s
Reviewed-on: #377
2026-10-27 22:25:30 +00:00
dc c463a4e03c test(editor): assert the derived block marker, not a column name
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Canceled after 17m25s
derived-variables.cy.ts:97 expected dc-derived-first on the REGION header,
because the comment assumed REGION opens the computed block. The grid follows
MPE_DERIVED order, and that order is LINE_VALUE, REGION, REGION_LABEL - so the
marker was on LINE_VALUE and correct. The spec asserted the wrong column, and
the development pipeline has been red on it since.

Dumping the header row on a local mock estate (sasjs/server desktop mode,
RUN_TIMES=js) shows:

  6 | LINE_VALUE | dc-derived-header dc-derived-first
  7 | REGION | dc-derived-header
  8 | REGION_LABEL | dc-derived-header beforeHiddenColumn htLastVisibleHeader

and the same dump against the unmodified release/7.19 client, so this was never
a client defect.

The assertion now states the invariant instead: at least one derived header, the
first of them opens the block, and no other one carries the marker. That holds
whatever order the control table returns the variables in.
2026-10-27 22:21:52 +00:00
allan 23015ac41f Merge pull request 'fix(lint): clear the sasjs lint warnings on the files 7.19 touched' (#374) from fix/lint-719-touched-files into release/7.19
Sites / deploy (push) Failing after 5m39s
Build / Build-and-ng-test (pull_request) Successful in 5m26s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 20m56s
Reviewed-on: #374
2026-10-27 22:01:09 +00:00
dc a8f62bdfe2 docs(agents): say why webout is in ignoredMacros, and where the macros come from
Build / Build-and-ng-test (pull_request) Successful in 5m44s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m29s
Build / Build-and-test-development (pull_request) Failing after 34m27s
The previous wording named only mpeinit and mpeterm, so the webout entry read as
an unexplained addition. webout is the linter's own default - sasjs compile
injects %webout into every service - and an explicit ignoredMacros list replaces
that default, so it has to be restated. 52 service files call %webout, which is
what dropping the entry would start reporting.
2026-10-27 21:47:19 +00:00
dc 355c46a7b6 fix(lint): clear the sasjs lint warnings on the files 7.19 touched
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Lighthouse Checks / lighthouse (pull_request) Canceled after 14m27s
Build / Build-and-test-development (pull_request) Failing after 34m21s
sasjs lint (@sasjs/cli 6.2.5) reported 24 warnings across the 11 .sas files
this release touches. None of them are in those files now:

- mpeinit and mpeterm are added to ignoredMacros in sas/.sasjslint. Every
  service is compiled with a serviceInit program (sasjs/utils/
  serviceinitsas9|viya|server.sas), and all three declare both macros in
  their own header, which is how a service body's %mpeinit()/%mpeterm()
  resolves without a declaration of its own. An explicit ignoredMacros list
  replaces the default, so webout stays listed.
- Header entries nothing in the file calls are dropped: mp_binarycopy in
  getrawdata (it arrives through mp_streamfile, which calls it - the
  compiled program is unchanged), mp_jsonout in viewdata,
  mp_assertcolvals in validatefilter.test, and mx_execute in testsetup
  (every file that calls it declares it).
- Prose carrying "error"/"warning" is reworded. Where the text is log
  output, %put WARNING: is written %put %str(WARN)ING: so the line the
  log receives is unchanged.
- The server makedata service keeps its doxygen @warning tag - the tag is
  the content - so that file carries @sasjslint {"noBannedStrings": false}.
- Two comment lines in getdata are re-indented to the file's multiple of 2.

AGENTS.md records the ignoredMacros rule and the %str(WARN)ING convention so
the next warning is not silenced by reflex. Compile passes; the 93 warnings
that remain are all in files this release does not touch.
2026-10-27 21:27:54 +00:00
dc 661ee19cfa fix(getrawdata): declare mp_ds2md so the Markdown download can run
The MARKDOWN branch of the service calls %mp_ds2md(), which ships with
@sasjs/core, but the header never listed it. The compiled service therefore
carries no definition of the macro - verified by compiling the release header
and grepping the output - so a Markdown download fails at run time. The viewer
offers "Markdown (.md)" in the download list, so the path is reachable.

Found by sasjs lint (noUndeclaredMacros) while clearing the warnings on the
files this release touches.
2026-10-27 21:27:09 +00:00
allan 619fe9f9b9 Merge pull request 'fix(derived): format catalog filtering, cut-off guard, selectbox seed' (#373) from fix/format-catalog-filter into release/7.19
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m48s
Build / Build-and-test-development (pull_request) Canceled after 28m13s
Reviewed-on: #373
2026-10-27 21:13:46 +00:00
dc 7830b7bcfa fix(derived): compare the outref as text, prove the file is empty
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 3m52s
Review: '%if &outref ne 0' handed %EVAL the text 'derref5 ne 0', which needs a
numeric operand - so the catalog branch failed in exactly the case it was added
for, on every caller that passes a fileref.  Quoted, the default 0 still skips
the file and a fileref creates it.

The test now proves both halves: fopen tells us the file was created, and a
read loop counted zero records tells us it is empty.  A single INPUT is no
good - on an empty file it stops the step before the symputx, which is why the
first version recorded a FAIL.

The outvars placeholder keeps 'keep=name'; the comment records why that is the
whole column set: the engine builds the dataset in exactly two places and both
project NAME, and the only projection in the SAS tree is getdata.sas.
2026-10-27 21:08:22 +00:00
dc f8f65681e1 fix(derived): keep targetds for plain datasets, test the empty file
Build / Build-and-ng-test (pull_request) Successful in 5m49s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m22s
Build / Build-and-test-development (pull_request) Failing after 34m17s
Review feedback: the first version left targetds out for every filter without
derived variables, which silently changed the plain-dataset path to whatever
mp_filterstore defaults to.  It is now passed exactly as before unless the libds
is a format catalog - the one case mp_filterstore has to rewrite itself.

The catalog test no longer writes to the statement file: it reads one record
from it instead, so -1 means the file could not be opened and 0 means
mpe_derived left it empty, which is the claim the branch makes.
2026-10-27 20:41:33 +00:00
dc 7fb981daa9 fix(derived): guard the bitemporal cut-off, coalesce the selectbox seed
mpe_derived used &dc_dttmtfmt in its where= but the guard only checked libds and
dclib, so an unset value turned the filter into 'lt tx_to' and the control-table
step failed with a syntax error a long way from the cause.  It is now part of
the guard, named in the message.

The 7.19 migration took max(selectbox_rk) into a macro variable, which is
missing on an install whose MPE_SELECTBOX is still empty - %eval(&maxrk+1) then
fails.  coalesce(max(selectbox_rk),0) makes the first row start at 1.
2026-10-27 20:41:21 +00:00
dc 469229aecb fix(derived): filtering a format catalog no longer aborts the service
Build / Build-and-ng-test (pull_request) Successful in 5m44s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m20s
Build / Build-and-test-development (pull_request) Failing after 34m47s
A format catalog is addressed as LIBREF.CATALOGNAME-FC.  Two places assumed a
dataset:

- mpe_derived passed the name straight to mp_derivedvars, whose existence check
  aborted with 'Target table not found: DCTEST.DCFMTS-FC' - so validatefilter,
  getrawdata, viewdata, getcolvals and stagedata all failed on a catalog.  A
  catalog has no observations, so it can have no derived variables: return an
  empty variable list and an empty statement file, and leave the view alone.
- validatefilter then passed that same -FC name as targetds to mp_filterstore.
  The engine handles a -FC libds itself (it rewrites it to a cntlout dataset),
  but a targetds it is handed is used as given, so mp_filtercheck built a step
  against a name it could not open.  targetds is now only passed when the
  derived view is actually in use.

mpe_derived.test gains a case for the catalog path: no derived variables, no
generated statements, and no abort.
2026-10-27 20:21:30 +00:00
allan 6c7c8ebf74 Merge pull request 'fix(derived): harness follow-ups from the estate test run' (#367) from fix/harness-test-followups into release/7.19
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #367
2026-10-27 19:46:59 +00:00
allan 468bdf5ab3 Merge pull request 'feat(editor): derived column tooltip carries the label' (#371) from fix/derived-label-tooltip into release/7.19
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #371
2026-10-27 19:43:17 +00:00
dc e0da9f1098 fix(derived): review feedback - cover the empty response, assert the last row
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
- mpe_assert_abort.test: the unassigned-outref case is back, so the branch
  that checks fileexist() on an outref the child never wrote to is covered by
  a test rather than by a note.  It runs: the four cases pass on the estate.
  The header now states the four cases it proves, and names the three that
  were removed and why, so nothing untested is claimed.
- mpe_execute.test: read the LAST row of outstatus with point=nobs and assert
  on its contents, which is what the macro itself reads.  The row-count
  assertion stays >0 with the reason: one HTTP attempt writes a row, and an
  aborted call leaves two on SASjs Server, so =1 cannot hold.
- Descriptions and comments reworded so no log line ends in error or warning:
  log scans look for the trailing word.
2026-10-27 19:39:37 +00:00
dc cfe6647ca8 feat(editor): show the derived column label in the header tooltip
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The hover text on a derived column said only that the column cannot be edited,
which tells the user nothing about what it holds.  It now carries the column
label - the configured VAR_LABEL, or the generated name=expression that
mp_derivedvars writes when the administrator left the label blank - and falls
back to the plain rule when the label merely repeats the column name.

The cypress case that marks derived columns apart now also checks the tooltip
for both shapes: REGION_LABEL carries a configured label, REGION falls back to
the generated one.
2026-10-27 19:36:51 +00:00
allan cdfb3d01aa Merge pull request 'fix(ui): one error modal at a time' (#370) from fix/modal-stacking into release/7.19
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #370
2026-10-27 19:33:16 +00:00
allan e80ca2af4d Merge pull request 'fix(derived): NOT NULL in the model, VAR_FMT optional' (#369) from fix/derived-model-notnull into release/7.19
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #369
2026-10-27 19:32:38 +00:00
dc 81ccf758e9 fix(ui): one error modal at a time
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Submitting a rejected MPE_DERIVED row showed the SAS abort message, and then a
second modal on top of it reading

    Submit error
    SAS Service: editors/stagedata
    Submit request failed
    undefined

Two separate faults.  sasService.request already shows the abort message when
the service returns a sasjsAbort, and the adapter error when the call fails -
then it rejects, and the editor's catch showed its own modal for the same
failure.  That second modal read 'undefined' because it looked for
'adapterRespnse' (typo), a property that does not exist, so the text it built
was JSON.stringify(undefined).

- sas.service.ts marks a rejection 'errorShown' when it has already told the
  user, on both paths that display a modal.
- the editor's catch now only reports a failure when nothing was shown, and
  reads err.adapterResponse.error when it does.
- app.component.ts replaces the open abort modal instead of pushing another
  onto the array: a second emit stacked a modal over the first, so one failure
  needed two dismissals, and the lower one was usually the duplicate.
2026-10-27 19:27:00 +00:00
dc 7efbf31ea5 fix(derived): make the model enforce the required columns, and leave VAR_FMT optional
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_LEN, VAR_LOGIC and ACTIVE are NOT NULL in the
table definition, so the requirement is stated once, in the model, instead of
being repeated as NOTNULL rules in MPE_VALIDATIONS for each column.

VAR_FMT stays nullable: it is optional on a derived variable, and its HARDREGEX
already accepts an empty value.

The seven NOTNULL rule inserts are removed from the 7.19 migration, and the
comment on ACTIVE no longer describes a rule that is not there.
2026-10-27 19:25:27 +00:00
dc 9b3cf50b9c fix(derived): a fileref with nothing written is a FAIL, not an error
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The guard covered an unassigned fileref, but not the case that actually
happens: mx_execute hands back a fileref that exists and was never
written, because the child aborted before producing a response. Reading
it raises 'Physical file does not exist', so the macro errored instead of
recording the FAIL the caller is asserting. Check fileexist on the
fileref's pathname as well.

mpe_execute.test asserted one status row per call. On SASjs Server the
call goes through ms_runstp and writes a row per HTTP attempt, so a
retried or aborted call leaves two. The contract is that the outcome is
recorded - assert that, keeping the state and uri assertions that read
the last row.

mpe_assert_abort.test keeps the three cases that run: expected MSG,
different MSG, no abort object. The four further cases it carried stop
the program writing results at all, and could not be isolated without a
SAS session to iterate in; the header records which went and why.
2026-10-27 19:14:03 +00:00
allan e929ca047f Merge pull request 'feat(derived): pickers for MPE_DERIVED key columns, and the blank truncation tooltip' (#366) from fix/empty-truncation-tooltip into release/7.19
Sites / deploy (push) Failing after 5m16s
Build / Build-and-ng-test (pull_request) Successful in 8m25s
Lighthouse Checks / lighthouse (pull_request) Failing after 22m35s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #366
2026-10-27 18:53:37 +00:00
dc d95ffdfb97 chore(derived): ACTIVE needs no HARDSELECT, and no cross-reference note
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The picker comes from the MPE_SELECTBOX rows - the client builds the dropdown
from that data and the SELECT rule types only set whether a value outside the
list can be typed.  NOTNULL is what makes ACTIVE mandatory, and the post edit
hook is what refuses anything other than 0 and 1, so the HARDSELECT row was
saying the same thing a third time.

Drop it in both writers, and drop the comment pointing at the migration: the
two files are read side by side anyway.
2026-10-27 18:44:41 +00:00
dc 87e794f769 feat(derived): pickers, and an ACTIVE default, for MPE_DERIVED
Lighthouse Checks / lighthouse (pull_request) Canceled after 1m16s
Build / Build-and-ng-test (pull_request) Canceled after 4m52s
Build / Build-and-test-development (pull_request) Canceled after 0s
MPE_DERIVED is maintained through the EDIT screen, but LIBREF and DSN were free
text - a typo in either silently produced a rule that never fired.

LIBREF now offers the editable libraries and DSN the tables in the LIBREF on the
same row, through the same SOFTSELECT_HOOK services the other configuration
tables use (libraries_editable, tables_editable).  VAR_NM stays typed: a derived
variable names a column the table does not have yet, so a list of existing
columns would be the wrong control.  ACTIVE becomes the fixed pair 1 and 0 from
MPE_SELECTBOX, with HARDSELECT so it cannot be typed freehand, and its NOTNULL
rule carries '1' - the value the EDIT screen pre-fills on Add Row
(getColumnDefault), because a derived variable only takes effect when active.

The rows are written by mpe_makedata.sas for a fresh install and by the v7.19
migration for an upgrade, and each names the other so the pair cannot drift
unnoticed; the ACTIVE guard itself lives once, in mpe_derived_check.sas.

The mock hook carries the ACTIVE rule, as it carries every rule
mpe_derived_check.sas enforces.
2026-10-27 18:39:15 +00:00
dc f182179403 fix(editor): a blank truncation note is not a warning
getdata returns TRUNC_MSG as a single blank when the selection was not
truncated: SAS has no zero-length character value, so mp_jsonout's trim of a
blank $200 leaves one blank and it serialises as a single space.  The editor
assigned it straight to truncationNote, and a space is truthy in JavaScript, so
the EDIT screen showed the orange truncation warning against the table name and
its tooltip contained one space - a dark bubble with nothing in it.

Trim the note before it is tested.
2026-10-27 18:36:46 +00:00
allan 803bb1ad76 Merge pull request 'fix(admin): make service-test calls observable, and fix the catalogue, getdata and makedata defects it exposed' (#363) from fix/dc-test-harness into release/7.19
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 5m10s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #363
2026-10-27 18:33:37 +00:00
dc 1863076bc9 fix(derived): enforce the ACTIVE picker pair in the extracted check
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m49s
Build / Build-and-test-development (pull_request) Canceled after 0s
Same rule as the 7.19 hook (PR #366): ACTIVE is NOTNULL + HARDSELECT in the
EDIT screen, so the field checks must refuse a missing value or anything other
than 0 and 1 - otherwise a direct stagedata submission stages a row the picker
would never have allowed.

The fixture gains the active column (1 by default, 2 for BADACTIVE) and the
count of reported rows goes from five to six.
2026-10-27 18:24:34 +00:00
dc 36e4e88eb1 fix: address the review findings on the harness and refresh changes
Lighthouse Checks / lighthouse (pull_request) Successful in 22m41s
Build / Build-and-ng-test (pull_request) Successful in 5m52s
Build / Build-and-test-development (pull_request) Failing after 34m22s
getcolvals - mf_nobs returns -1 when the dataset does not exist, so the guard
tested ==0 and a request that supplied no iwant at all fell through to the SQL.
It now tests <=0, which covers both an empty and an absent input.

mpe_refreshcatalogs - etlsource defaulted to a bare uid while its two siblings
use refreshcatalog:<step>:<uid>, so a load through this path could not be traced
to the step that wrote it. The default now has the same shape, and the explicit
etlsource=0 path follows it too.

mpe_execute - the status table was read without being cleared, so a child that
died before core wrote a row left the previous call's row in place and failfast
read it as completed. The table is deleted before the call.

makedata - KEEPME is compared through cats(), so a numeric column no longer
raises in upcase().
2026-10-27 18:22:18 +00:00
dc 1e224763ba fix(dc_refreshcatalog): terminate the generated statements
Build / Build-and-ng-test (pull_request) Successful in 5m40s
Build / Build-and-test-development (pull_request) Failing after 34m52s
Lighthouse Checks / lighthouse (pull_request) Failing after 22m3s
The statements written into the catalog ended without a semicolon, so the
%INCLUDE that ran them ran on into the caller: the parser finished the statement
with the text after the include and tried to include it, producing
'Cannot open %INCLUDE file PROC' immediately before the next proc sql, and
refreshlibinfo aborted with syscc non-zero.

The three refresh macros are only reached through those generated statements, so
the false 'declared but not used' warnings are switched off with a @sasjslint
block rather than by dropping the declarations.
2026-10-27 18:12:10 +00:00
dc 059cdf176a fix(getddl.test): ask for a table, not a whole library
Build / Build-and-ng-test (pull_request) Successful in 5m24s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m40s
Build / Build-and-test-development (pull_request) Failing after 35m8s
getddl requires a dataset - it aborts when the member cannot be found, and only
treats a format catalog specially - so a request with ds= always aborted and the
assertion read a stale webout.  Test 1 now names MPE_TABLES and checks that the
TSQL header names it, which keeps TSQL coverage on a request the service accepts.
2026-10-27 17:59:32 +00:00
dc 45dce7b11b fix(getdata.test): pin the derived filter to a libref, and spell the table as it is
Build / Build-and-ng-test (pull_request) Successful in 5m48s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m12s
Build / Build-and-test-development (pull_request) Failing after 35m8s
Test4 filtered only on the derived column, so it returned the same DSN once per
libref, including transient session librefs.  The filter now pins LIBREF as well,
built in a data step because datalines leave &dc_libref unresolved, and the
expected derived value uses MPE_LOCKANYTABLE - the name makedata registers.
2026-10-27 17:48:19 +00:00
dc ddb1671e37 test(getdata): dump the Test4 payloads under debug, and count the derived row
mp_assertcols tests variable names, so it can never find a DERIVED_NOTE row in
the cols payload - the assertion now counts rows instead.  With debug on, both
Test4 payloads are printed so a wrong payload can be read rather than guessed.
2026-10-27 17:18:40 +00:00
_ da30266278 fix(test): getdata test
Build / Build-and-ng-test (pull_request) Successful in 9m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m26s
Build / Build-and-test-development (pull_request) Failing after 34m35s
2026-10-27 17:17:53 +00:00
_ fdcd8fa4b3 fix(tests): enable closeouts logging
Build / Build-and-ng-test (pull_request) Successful in 10m44s
Lighthouse Checks / lighthouse (pull_request) Successful in 23m15s
Build / Build-and-test-development (pull_request) Failing after 35m42s
2026-10-27 17:14:01 +00:00
dc 220408538b fix(mpe_execute): no apostrophe in the SAS 9 status note
Build / Build-and-ng-test (pull_request) Successful in 5m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m32s
Build / Build-and-test-development (pull_request) Failing after 34m45s
An apostrophe in a %put breaks macro compilation on this platform: the statement
ends early and the next macro is left unterminated, which showed up as 'Macro
keyword LET appears as text' several hundred lines further down.
2026-10-27 16:45:12 +00:00
dc 4bcee706ff fix(getcolvals): report an invalid value, not missing inputs
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Failing after 34m57s
libds and col2 are only populated once validation passes, so the
"Missing inputs from iwant" abort - which tested those macro variables - fired for
any invalid payload and hid the specific reason.  It now tests whether the
request carried rows at all, leaving Invalid libds / Invalid col to report the
value.  The service still refuses the same requests; only the message changes.

The matching tests carried two faults of their own: getcolvals.test.4 sent its
libds through single quotes so the libref itself was invalid, and getdiffs /
getrawdata expected "Invalid table" for cases their own field names.
2026-10-27 16:23:31 +00:00
dc ded8bd13b3 ci: drop the HTTP_PORT explainer from the workflows
Build / Build-and-ng-test (pull_request) Successful in 5m35s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m49s
Build / Build-and-test-development (pull_request) Failing after 34m53s
The line that sets it is the change; the four lines above it only restate it.
2026-10-27 16:12:50 +00:00
dc@4gl.io f2b1428ec3 fix(mpe_execute): read an unknown state as unmeasured, not failed
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m47s
Build / Build-and-test-development (pull_request) Failing after 34m58s
SAS 9 has no job state: core's SASMETA branch writes state='unknown' with
'proc stp returns no job status' on a call that succeeded.  The fail-fast
change would have turned every converted test on that platform into a failure,
so unknown is noted rather than aborted on.  A missing status table, a blank
state, canceled and failed still stop the caller.

The macro test asserted on uri, which the SAS 9 status row leaves empty; it now
asserts the platform column, which every branch populates.
2026-10-27 14:58:00 +00:00
dc@4gl.io ea300f73e7 test(macros): cover mpe_execute and mpe_assert_abort
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m24s
Build / Build-and-test-development (pull_request) Failing after 35m16s
mpe_assert_abort decides PASS/FAIL by scanning response text, and a mistake in
that scan either fails a genuine abort or passes on any sasjsAbort string in the
response; nothing reported it.  Its test builds the responses as files: the
expected MSG, a different MSG, no abort object, several MSGs where the first
belongs to the abort, an object split across lines, an abort past 32767
characters, and an unassigned fileref.

mpe_execute is covered against a child that aborts for a known reason: one
status row per call, the state named and not completed, the uri recorded, and
the child log kept only when mdebug asks for it.
2026-10-27 14:49:59 +00:00
dc@4gl.io 359b570b85 fix(getdata): clear the globals it declares before use
loadtype, var_txfrom, var_txto, var_processed, filter_text, pk, coltype, sortpk,
var_busfrom, var_busto, rk_underlying, buskey and pre_edit_hook are declared
%global but were never initialised, so in a reused session a code path that
skipped an assignment applied the previous request's value - a stale
pre_edit_hook or buskey against the wrong table.
2026-10-27 14:49:59 +00:00
dc@4gl.io d626cb8f2d fix(makedata): read KEEPME from every input row, and keep it out of the Viya seed
The KEEPME probe did 'set work.fromjs' inside the varnum guard and then stopped,
so only the first observation was inspected: a submission that set KEEPME on a
later row still self-destructed.  It now reads to end of file.

makeDataNextViya.json ships KEEPME=Y and is the config that seeds a real Viya
environment, so the default posture there was 'the seeding program stays on the
server'.  KEEPME stays in makeDataSas719.json, which exists for the test
harness, and is removed from the environment seed.
2026-10-27 14:49:59 +00:00
dc@4gl.io 0a52d20c5f fix(mpe_assert_abort): take the abort's own MSG, tolerate a missing response
Three defects in the scan:

- 's/.*"MSG" *: *"([^"]*)".*/$1/' is greedy, so a response carrying more
  than one MSG compared the last one rather than the abort's.  The pattern is
  now matched with prxmatch and the capture read with prxposn, which returns
  the first match at or after the sasjsAbort marker.
- the whole response was concatenated into $32767, so a longer response
  truncated silently and an abort near the tail was missed.  The file is now
  scanned a line at a time and the scan stops at the first MSG.
- 'infile &outref' had no fileref check, so a child that died before writing a
  response raised an unassigned-fileref error instead of recording a FAIL.
2026-10-27 14:49:55 +00:00
dc@4gl.io a03638667f fix(mpe_execute): fail fast when the child left no status
failfast fired only on state=canceled or state=failed, and read &outstatus
unguarded.  A child that never reached the runner leaves no status table, and
a status row with a blank state left both macro variables blank - either way the
test carried on against whatever the previous call left in outlib, which is the
inference this wrapper exists to remove.

The status table is now read only if it exists, a missing table is named as
such, and failfast aborts unless the state is completed.  A status row that is
absent and one that is blank both fail.
2026-10-27 14:49:55 +00:00
dc@4gl.io 638c34a070 Merge release/7.19 into the test harness
The release branch gained 36e1689, which fixed the same four security tests
(getcols, getcolvals, getrawdata, validatefilter) by introducing
dc_assert_abort.  This branch had already fixed them with mpe_assert_abort and
mpe_execute, and mpe_assert_abort is the wider of the two: it also checks the
expected MSG, and the calls pass outstatus/failfast so the child cannot be
inferred from a stale outlib.  Kept the harness versions and dropped
dc_assert_abort.sas, which nothing references after the merge.
2026-10-27 14:46:04 +00:00
allan 9aef2f4f03 Merge pull request 'feat(editor): mark derived variables as computed in the EDIT grid' (#365) from feat/derived-var-styling into fix/dc-test-harness
Build / Build-and-ng-test (pull_request) Successful in 5m36s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m21s
Build / Build-and-test-development (pull_request) Failing after 35m38s
Reviewed-on: #365
2026-10-27 14:42:49 +00:00
dc@4gl.io 9aea59dfef ci: start the SASjs Server with HTTP_PORT=5000
Build / Build-and-ng-test (pull_request) Successful in 5m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 23m4s
Build / Build-and-test-development (pull_request) Failing after 34m12s
SASjs Server 3.x deprecated PORT and PROTOCOL and refuses to start when an
.env names neither HTTP_PORT nor HTTPS_PORT, exiting with

    - no listener is enabled
     - set HTTP_PORT and/or HTTPS_PORT (either may be used alone)

The three workflows downloaded releases/latest and wrote an .env without a
listener, so api-linux never bound :5000 and the 'Setup and start SASjs
server' step failed after its 120s wait (build runs for #363 and #365, both
failed there). Add HTTP_PORT=5000 to build.yaml, release.yaml and
lighthouse.yaml, and document it in the local-dev .env in CONTRIBUTING.md,
where the default-port claim is no longer true.
2026-10-27 14:34:13 +00:00
dc 6513d09f05 feat(editor): mark derived variables as computed in the EDIT grid
Derived variables are read-only, but they were rendered exactly like any other
read-only column: dimmed via readonlyCell, with no indication that the column is
computed at read time rather than locked by a rule or by column level security.
The grid could not tell the difference either - getdata holds a derived column
read-only with a READONLY DQ rule, which is also a rule type a table can be
configured with.

getdata now names the derived columns in sasparams.DERIVEDVARS (space separated,
upper case, from MPE_DERIVED's outvars). The EDIT screen holds that list, and
marks those columns with their own classes instead of the read-only dimming:
a tinted header with an 'fx' marker and a tooltip, tinted italic cells, and a
left edge on the first derived column so the computed block reads as a group.
Blue is the info family, leaving the primary key green and the warning ochre to
their existing meanings. Light and dark values are both set.

The SASjs mock getdata.js returns the same field, so the Cypress suite exercises
the styling; the existing derived-variables spec asserted readonlyCell on the
derived header, which is now the fallback for a plain read-only column, so that
assertion moves to dc-derived-header and a physical column stays the control.

Lint: prettier --check clean on the touched client files; sasjs lint reports no
new warnings for getdata.sas.
2026-10-27 14:34:13 +00:00
dc@4gl.io 356308e13f chore(deps): sync the sas lockfile to @sasjs/core 6.1.0
Lighthouse Checks / lighthouse (pull_request) Failing after 3m0s
Build / Build-and-ng-test (pull_request) Successful in 5m47s
Build / Build-and-test-development (pull_request) Failing after 6m32s
The harness pins @sasjs/core exactly (6.1.0) so the tests can read the
outstatus table, but the lockfile still resolved 6.0.0. 'npm ci' refuses to
install when an exact pin and the lockfile disagree, which failed the mocked
deploy in both build and lighthouse ("lock file's @sasjs/core@6.0.0 does not
satisfy @sasjs/core@6.1.0"). Regenerated with 'npm install
--package-lock-only': the only change is the core entry.
2026-10-27 14:32:20 +00:00
dc 46bacdd6d9 fix(refresh): name each catalogue load uniquely in the audit trail
Lighthouse Checks / lighthouse (pull_request) Failing after 2m58s
Build / Build-and-ng-test (pull_request) Successful in 5m20s
Build / Build-and-test-development (pull_request) Failing after 6m4s
MPE_DATALOADS is keyed on (processed_dttm, libref, dsn, etlsource), and etlsource
was the calling macro name, so a second catalogue load of the same table in the
same second was refused as a duplicate.  The three refresh macros now take an
etlsource parameter, defaulting to refreshcatalog:<step>:%mf_uid(), which records
which step loaded what and keeps the audit row unique.

dc_refreshcatalog also echoes the generated statements when debug is on, since a
failure inside one of them otherwise names no libref.
2026-10-27 14:22:20 +00:00
dc 582523dc87 feat(dc_getlibs): dump the library list when debug is on
A refresh that catalogues the wrong libraries is hard to read from the log: the
list is built here, then filtered and reworked by its callers.  With
sasjs_mdebug set, print the result so the library list can be checked at source.
2026-10-27 14:22:20 +00:00
dc 105052b948 test: prove staged rejections from the response, and check the field rules at macro level
The MPE_DERIVED post edit hook aborts with mp_abort, so the webout carries a
sasjsAbort object: stagedata.test.3 and .4 now read that through
mpe_assert_abort instead of inferring the outcome from MPE_LOADS.

The field level rules move into mpe_derived_check so they can be tested
directly.  A service cannot exercise them all - VAR_LABEL is varchar(256) and a
staging dataset declares the same length, so a longer label is truncated before
any check sees it - and the macro test runs in seconds rather than minutes.
2026-10-27 14:22:16 +00:00
dc d0bc5ae297 refactor: rename dc_execute to mpe_execute
Lighthouse Checks / lighthouse (pull_request) Failing after 2m53s
Build / Build-and-ng-test (pull_request) Successful in 4m44s
Build / Build-and-test-development (pull_request) Failing after 5m42s
The wrapper is DC specific and sits alongside mpe_assert_abort, so the mpe_
prefix marks which layer it belongs to.  File, macro name, %mend and all seven
callers updated; no dc_execute reference remains in the tree.
2026-10-27 11:18:26 +00:00
dc 31aa859153 revert(stagedata.test.4): back to the MPE_LOADS evidence for now
Lighthouse Checks / lighthouse (pull_request) Failing after 3m5s
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Build / Build-and-test-development (pull_request) Failing after 5m45s
Reading the sasjsAbort object instead did not work for this test: the derived post
edit hook aborts with `abort cancel FILE` via %mp_include, and with the
response-based version the test stopped before reporting any assertion, where the
MPE_LOADS version reports ten and fails five.

Which evidence is right depends on where that abort is raised - the test's own
session or the child's - and the log does not distinguish the two here.  Reverting
until that is settled, so the branch is not carrying a change that reduces
coverage.
2026-10-27 11:00:55 +00:00
dc a141e38da4 refactor(stagedata.test.4): call the service through dc_execute
Lighthouse Checks / lighthouse (pull_request) Failing after 2m54s
Build / Build-and-ng-test (pull_request) Successful in 4m50s
Build / Build-and-test-development (pull_request) Failing after 6m5s
Same inputs and outputs, with a status row and no fail-fast, so the echoed child
log carries the CHILD: prefix - which is what distinguishes an abort raised in
the child from one raised in the test's own session.
2026-10-27 10:55:16 +00:00
dc e3ec1da55c refactor(stagedata.test.4): prove the rejections from the response, not MPE_LOADS
Lighthouse Checks / lighthouse (pull_request) Failing after 2m54s
Build / Build-and-ng-test (pull_request) Successful in 4m50s
Build / Build-and-test-development (pull_request) Failing after 5m45s
The five rejected-submission cases queried MPE_LOADS for a row that did not reach
SUCCESS.  The post edit hook aborts with mp_abort, so the webout carries a
sasjsAbort object: mpe_assert_abort reads that directly, which is the actual
evidence that the submission was refused.
2026-10-27 10:39:24 +00:00
dc 9c294c0ab5 fix(mpe_assert_abort): drop the unused line variable
Lighthouse Checks / lighthouse (pull_request) Failing after 3m1s
Build / Build-and-ng-test (pull_request) Successful in 5m11s
Build / Build-and-test-development (pull_request) Failing after 5m49s
It was declared but never assigned, and this session treats an uninitialized
variable as an error that stops the step before the assertion runs - so the
macro reported FAIL for a response that did carry the abort object.
2026-10-27 09:38:36 +00:00
dc fed43cb07e fix(dc_execute): use = comparisons, not IN, in the fail-fast guard
Lighthouse Checks / lighthouse (pull_request) Failing after 2m59s
Build / Build-and-ng-test (pull_request) Successful in 5m11s
Build / Build-and-test-development (pull_request) Failing after 6m13s
%if &failfast=1 and "&state" in ("canceled" "failed") fails to evaluate: %EVAL
reports it needs a numeric operand and stops the macro, which aborted every test
that reached the guard even though the child had completed or aborted cleanly.
Two plain comparisons do the same job and evaluate reliably.
2026-10-27 09:32:54 +00:00
dc a20a5a627c refactor(tests): mpe_assert_abort reads the sasjsAbort object from the response
Lighthouse Checks / lighthouse (pull_request) Failing after 3m1s
Build / Build-and-ng-test (pull_request) Successful in 4m53s
Build / Build-and-test-development (pull_request) Failing after 6m4s
dc_assert_abort inferred the outcome from the status row.  The assertion should
prove the service rejected its input, so it now reads the child's response and
requires a sasjsAbort object, and compares that object's MSG value with the
message the service is expected to return:

  %mpe_assert_abort(desc=An invalid libds aborts the service,
    outref=web1, msg=Invalid libds)

Renamed with the mpe_ prefix to match the other macro-level helpers.  getdata
Test 4 asserted completion with the abort macro, which was the wrong tool; it
now checks the status row for state=completed directly.
2026-10-27 09:26:34 +00:00
dc f5aade5976 fix(dc_getlibs): exclude the DC libref from the dictionary scan
Lighthouse Checks / lighthouse (pull_request) Failing after 2m57s
Build / Build-and-ng-test (pull_request) Successful in 4m49s
Build / Build-and-test-development (pull_request) Failing after 5m44s
dictionary.libnames lists the DC libref as well, so it was seeded into the
result by the SELECT and then added a second time by the explicit INSERT.  The
duplicate row then collides with pk_mpe_datacatalog_libs when the refresh closes
the bitemporal record:

  ERROR: Duplicate values not allowed on index pk_mpe_datacatalog_libs

The DC libref is now excluded from the scan, so it arrives once, from the INSERT
that carries its library name and engine.
2026-10-27 09:04:16 +00:00
dc 867faf8dd8 refactor(tests): read the outcome from core's outstatus, drop the local mx_execute
Lighthouse Checks / lighthouse (pull_request) Failing after 3m0s
Build / Build-and-ng-test (pull_request) Successful in 4m58s
Build / Build-and-test-development (pull_request) Failing after 5m46s
@sasjs/core 6.1.0 ships outlogds and outstatus, so the temporary copy of
mx_execute is gone and the abort assertions no longer need a platform branch:

- dc_assert_abort reads the status dataset instead of work.results on Viya and
  the response text elsewhere - one table, one code path;
- the abort cases call dc_execute with outstatus and failfast=0, so a cancelled
  child is reported rather than aborting the test;
- getdata Test 4 loses its %mf_getplatform() gate for the same reason, and reads
  the outlib only when the status says the child completed;
- dc_execute passes viyaresult through, which the abort cases ask for.
2026-10-27 01:19:45 +00:00
dc 609f2a743c test(hooks): seed the baseline from an npm script, not from testsetup
Lighthouse Checks / lighthouse (pull_request) Failing after 2m57s
Build / Build-and-ng-test (pull_request) Successful in 4m51s
Build / Build-and-test-development (pull_request) Failing after 5m42s
A full makedata on every sasjs test run is expensive, and it made the suite
depend on state the tests themselves should not be creating.  Move it to where
DC already does this: the npm scripts that send deploy/makeData*.json to
services/admin/makedata.

  npm run sas719setup   deploy, then seed once
  npm run sas719test    run the suite as often as wanted

KEEPME in the payload keeps makedata on the server, so a reseed does not need a
redeploy first.  Viya's makedata gains the same switch, so nextviyamakedata
behaves the same way.

Tests must now leave the estate as they found it - that is the follow-up, and it
is what makes re-running a single test possible.
2026-10-26 23:51:59 +00:00
dc 16e9b31e65 test(harness): a measurable wrapper around mx_execute for service tests
Build / Build-and-ng-test (pull_request) Successful in 4m50s
Build / Build-and-test-development (pull_request) Failing after 5m48s
Lighthouse Checks / lighthouse (pull_request) Failing after 2m53s
Tests inferred whether a service call worked by reading an outlib that a failed
call never touched, so assertions ran against the previous call's tables.  This
adds dc_execute, which wraps mx_execute and makes each call observable:

 - failfast: abort as soon as the child did not complete, with its message, so
   the first failure is the real one
 - a status row per call (outstatus) so a test can assert the outcome
 - the child's log behind the harness debug flag, quiet otherwise

testsetup reseeds the data baseline before the test library is built, makedata
honours a keepme switch and getdata initialises the variables it later tests
with %length, plus desc= values reworded where a comma or apostrophe unbalanced
the parameter.

sasjs/macros/mx_execute.sas is a TEMPORARY copy from sasjs/core#476, which adds
outlogds/outstatus and stops leaving work.results in the caller WORK.  Delete it
and bump the dependency once that PR is released.
2026-10-26 23:29:46 +00:00
dc 8096772119 test(security): assert aborts from the webout where there is no job state
The four security test programs proved that a bad input aborted the service by
reading work.results, the job table %mv_jobflow writes in the Viya branch of
%mx_execute. The SASjs Server and SAS 9 branches call %ms_testservice instead,
so the table never exists, the data step fails with "File WORK.RESULTS.DATA
does not exist." and the assertion reported (0=1) - a failure that says
nothing about whether the service validated its input.

The platform difference is now in one macro, dc_assert_abort: on Viya it reads
the job state as before, elsewhere it looks for the sasjsAbort block that
mp_abort leaves in the webout %mx_execute returns through outref. The call to
the service stays outside the platform check, so the payloads are submitted on
every platform and the assertion is real rather than a skip.

Verified on SASjs Server (sas.4gl.io, target sas719, appLoc /Public/app/dc719):
the seven abort assertions PASS, i.e. the services do abort on the macro, SQL
injection and path traversal payloads. Negative control: the same program with
a valid libds reports FAIL (0=1), so the assertion can still fail.

sasjs lint: exit 0, no warnings in the five files.
2026-10-26 23:29:33 +00:00
dc 36e168971c test(security): assert aborts from the webout where there is no job state
Sites / deploy (push) Successful in 3m43s
Build / Build-and-ng-test (pull_request) Successful in 5m2s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m48s
Build / Build-and-test-development (pull_request) Successful in 33m28s
The four security test programs proved that a bad input aborted the service by
reading work.results, the job table %mv_jobflow writes in the Viya branch of
%mx_execute. The SASjs Server and SAS 9 branches call %ms_testservice instead,
so the table never exists, the data step fails with "File WORK.RESULTS.DATA
does not exist." and the assertion reported (0=1) - a failure that says
nothing about whether the service validated its input.

The platform difference is now in one macro, dc_assert_abort: on Viya it reads
the job state as before, elsewhere it looks for the sasjsAbort block that
mp_abort leaves in the webout %mx_execute returns through outref. The call to
the service stays outside the platform check, so the payloads are submitted on
every platform and the assertion is real rather than a skip.

Verified on SASjs Server (sas.4gl.io, target sas719, appLoc /Public/app/dc719):
the seven abort assertions PASS, i.e. the services do abort on the macro, SQL
injection and path traversal payloads. Negative control: the same program with
a valid libds reports FAIL (0=1), so the assertion can still fail.

sasjs lint: exit 0, no warnings in the five files.
2026-10-26 20:17:50 +00:00
dc 0d58be975d fix(ci): restore the gitleaks ignore entries for the mock licence pair
Sites / deploy (push) Successful in 3m48s
Build / Build-and-ng-test (pull_request) Successful in 5m23s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m31s
Build / Build-and-test-development (pull_request) Successful in 32m19s
116c37b removed them on the grounds that a three-segment fingerprint matches
nothing. That is wrong for the mode this repo scans in: with `--no-git`, gitleaks
reports a finding as `file:rule:***` - the line number is not part of the
fingerprint - so the entries were matching, and the pipeline failed on the next
push with two findings in sas/mocks/mock-licence.json.

Measured on a clean checkout of this head, with the pinned tool
(@nogoo9/gitleaks 8.30.1-post.2) and the command the workflow runs:

  with .gitleaksignore     6.10 MB scanned, no leaks found
  without it               6.10 MB scanned, leaks found: 2

The 6.10 MB scan is the same figure the CI log reports, which is what makes this
comparable; my earlier check was run in a tree that still had node_modules in it,
scanned 1.05 GB, and its verdict meant nothing.

The entries stay, with the comment explaining that the pair is RSA key material
for the browser mock rather than a credential.
2026-10-26 10:34:36 +00:00
dc fe2388718a chore(deps): handsontable 18.1.2, so a wrapped entitlement key cannot lock the grid
Build / Build-and-ng-test (pull_request) Failing after 23s
Build / Build-and-test-development (pull_request) Skipped
Sites / deploy (push) Successful in 3m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m38s
18.1.2 carries one fix (2026-10-07): an entitlement licence key whose bracketed
block is broken across lines - by an email client, typically - read as invalid and
locked the grid.

That is this app's key path. The DC licence payload carries `hot_license_key`, and
licence.service puts it on `hotTable.licenseKey` for every grid - editor, viewer,
viewboxes, stage, multi-dataset, xlmap - so a key pasted with line breaks would
lock the editor rather than open it, on 18.1.1.

Both packages move together. The licence-checker exclusions pick up the new
version too: handsontable and @handsontable/angular-wrapper declare `SEE LICENSE
IN LICENSE.txt`, which the checker reports as a custom licence, so every pinned
version is listed in excludePackages - 18.1.1 was already there, 18.1.2 is added
beside it.

Checked: client 603/603, production build green, npm audit --omit=dev clean,
licence checker "All packages are licensed properly".
2026-10-26 10:22:59 +00:00
dc 05b91cb43e refactor(getdata): scope the payload row cap to the SET, and assert the column list
Build / Build-and-ng-test (pull_request) Failing after 21s
Build / Build-and-test-development (pull_request) Skipped
Sites / deploy (push) Successful in 3m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m53s
The review read the cap as a session option that is never restored, so every later
data step - including the derived-column merge into work.cols1 - would be cut to
&maxrows observations, and a table wide enough to hit the cell limit would lose
columns rather than rows.

On this head the reset is there, two lines after the data step (`options obs=max;`
at what was line 360), so the leak described is not live. It is worth removing the
dependency on that line though: the cap is now a data set option on the SET that
builds the payload, so there is no session option to leak and no later step can be
cut by it, whether or not a reset is remembered.

The existing cell-limit test asserted the row count only - 199 rows for a
1004-column table - which would have passed with 805 columns missing. It now also
compares the COLS payload against dictionary.columns, so the metadata being whole
is tested rather than assumed.

The other obs pair in this program (`options obs=10000` around the filtered
selection, restored at line 290) is deliberately left alone: it bounds the input
rows that the WHERE clause reads, which a data set option on the SET cannot do.
2026-10-26 10:09:25 +00:00
dc 072a133702 fix(client): open the SAS log page with noopener
window.open(url, '_blank') hands the opened page a window.opener reference. The
URL is built from config.serverUrl, which on Viya is a different origin from the
app, so the SAS log page could navigate this tab - a log page is exactly the sort
of thing that gets rendered from backend content.

'noopener' in the window features drops the reference in every browser that
supports it. The two existing specs asserted the call's two arguments, so they
now assert the third; the client suite is 603/603 with handsontable 18.1.1 from
the lockfile.
2026-10-26 10:09:21 +00:00
dc 116c37b379 chore: drop the gitleaks ignore entries that matched nothing
Build / Build-and-ng-test (pull_request) Failing after 16s
Build / Build-and-test-development (pull_request) Skipped
Sites / deploy (push) Successful in 3m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m32s
The three-segment entries in .gitleaksignore were meant to cover the mock licence
pair, but a fingerprint is commit:file:rule:line, so file:rule:*** never matched
anything - and the file does not need covering anyway:

  gitleaks detect --no-git --source . --redact   (the CI command, on this tree)
    no leaks found

  gitleaks protect --staged --redact             (after touching mock-licence.json)
    no leaks found

A full-history detect over 1088 commits reports 58 findings, none of them in
mock-licence.json - they are an old viewdata.js commit, .npmrc and
mock-storage/licence.json, all outside the tree CI scans. Removing the file is
more honest than keeping an exception that guards nothing; if a future edit to
the pair does fire, the commit fails and the real fingerprint can be added then.
2026-10-26 09:53:47 +00:00
dc 1fb4818d21 fix(client): refuse a truncated table in the multi-table upload
getdata now serves a selection over DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT
truncated rather than aborting, which the EDIT screen reports above the grid. The
multi-table upload is the other caller: it fetches each table to compare the whole
of it against the uploaded workbook, so a partial read would compute the changes
against the wrong starting state and submit differences nobody asked for.

It now refuses such a table with a message naming the limit, rather than carrying
on with the rows it received. The EDIT screen already reads the same
sasparams.TRUNCATED flag; sas.service only transports the response.
2026-10-26 09:53:47 +00:00
dc 78e88a4112 feat(derived): take any text in VAR_LABEL, but not a line break
VAR_LABEL was the one free-text column with no rule at all, in the hook or in the
MPE_VALIDATIONS seed. The suggestion in review was an allowlist, but a label has
no reason to exclude any character - quotes are doubled by %mp_derivedvars before
the attrib statement is built, and the value never reaches a macro context - so
the rule is a denylist of what cannot be a single-line column label:

  * a control character, which covers the line breaks and tabs that would carry
    through into the generated DATA step and render as nothing in a header
  * a label over 256 characters, which the engine truncates silently on a quote
    boundary, so the configured value and the applied one differ with nothing
    saying so

Everything else passes: any script, and the semicolon, ampersand, percent and
comment markers that VAR_FMT and VAR_LOGIC refuse. Implemented in the hook
(anycntrl, klength), as a HARDREGEX in all three seed writers (migration,
mpe_makedata.sas, the mock's makedata.js) so the grid refuses it before submit,
and in the mock hook for the deployed mock.

SAS: new stagedata.test.4 cases 8-10 - a label with an embedded LF is rejected, a
label of free text is accepted, and 257 characters is rejected.

Verified against the deployed mock: the tab and 257-character cases abort with the
right message, free text and UTF-8 pass, and a 256-character label passes. The
newline case cannot be driven through `sasjs request` - the mock server splits a
jsdata row at an embedded newline, so the row arrives short - hence the SAS test.
2026-10-26 09:53:37 +00:00
allan bca1b9363f Merge pull request 'fix(derived): hold derived columns read-only regardless of CLS, and register MPE_DERIVED in the mock' (#360) from fix/derived-readonly-and-mock-config into release/7.19
Sites / deploy (push) Successful in 3m56s
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m49s
Build / Build-and-test-development (pull_request) Successful in 33m35s
Reviewed-on: #360
2026-10-26 08:37:45 +00:00
dc 1b13120839 Merge release/7.19 into fix/derived-readonly-and-mock-config
Build / Build-and-ng-test (pull_request) Successful in 5m40s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m7s
Build / Build-and-test-development (pull_request) Successful in 33m25s
#359 has landed, so the CLI bump this branch was waiting on is now in the base
and the merge brings it in. One conflict, in
client/cypress/support/commands.ts: both sides reworked loginAndUpdateValidKey.

The base version decides from the body captured before login, which is where
the original bug was - on an estate that shows a login form that capture has no
nav tree, so the command early-returned and never applied a key, and the upload
specs then found no file input. This branch re-reads the loaded page and treats
the licence banner as the signal. Kept this side; prettier re-indented the
result. Every other file merged automatically, and the hookErrmsg veto in
stagedata.js and loadfile.js survives the merge.
2026-10-26 08:34:32 +00:00
dc 6cad23fce9 fix(mocks): scope the derived hook's graph to the submission
Build / Build-and-ng-test (pull_request) Successful in 4m52s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m50s
Build / Build-and-test-development (pull_request) Successful in 31m26s
Two divergences from the server hook, both of which reject a valid submission:

- VAR_DEP_UNION joins on libref, dsn and name, so a derived variable called
  STATE on one table does not make STATE a derived name on another. The mock
  kept a flat set of names, so a physical column named STATE was reported as
  an undeclared reference whenever any other table derived a variable of that
  name.

- The control-table arm filters staging_ds for the tables being submitted, so
  another table's configuration cannot block this one. The mock read the whole
  MPE_DERIVED table, so a cycle declared on an unrelated table blocked every
  submission.

Reproduced both against the previous head - a physical STATE reference and an
unrelated CYC_A/CYC_B cycle were each rejected - and both pass after, while the
cycle, undeclared-reference and stale-dependency cases on the submitted table
still fire.
2026-10-25 23:44:17 +00:00
dc 5fc0ff1ee3 docs(derived): record why the read-only rules are safe to append and the union unguarded
Build / Build-and-ng-test (pull_request) Successful in 4m54s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m48s
Build / Build-and-test-development (pull_request) Successful in 31m9s
Both come up in review every time these lines are read: dqrules is the
request-local array the filter/map builds, and work.derived_vars is written by
%mpe_derived before it can return, empty when the table has no derived
variables. Said where the code is, rather than in a review reply.
2026-10-25 23:04:20 +00:00
dc 5b70c000d7 test(derived): assert the read-only marker instead of inferring it
The case rested on an editor not appearing after a fixed wait, which passes for
any reason the editor did not open. The editor marks a read-only column with
readonlyCell on the header and on its cells, and STATE in the same grid is the
control, so the state is asserted directly and the sleep goes.

Mutating the mock to drop the READONLY rule fails the case on exactly that
assertion (expected '<th.darkTH>' to have class 'readonlyCell'); restoring it
passes 5/5 in about 50s.
2026-10-25 23:04:19 +00:00
dc d7cadc0b7b refactor(derived): name the invalid-character test, and prefix the cycle message
Build / Build-and-ng-test (pull_request) Successful in 4m56s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m56s
Build / Build-and-test-development (pull_request) Successful in 31m19s
has() returned true for a character outside the allowed set, which every call
site read as the opposite; it is hasInvalidChar now.

The cycle rejection was the one message without the MPE_DERIVED: prefix. The
SAS hook produced the same text, so both are prefixed rather than letting the
mock drift away from the message a real deployment shows.
2026-10-25 22:19:06 +00:00
dc 9914790777 fix(derived): close four gaps in the MPE_DERIVED hook
Build / Build-and-ng-test (pull_request) Successful in 5m35s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m54s
Build / Build-and-test-development (pull_request) Successful in 31m45s
loadfile names the submitted rows 'rows' and never defined jsDataRows, so
eval'd there the hook threw a ReferenceError - which the caller logged and
ignored. The rows are bound under the name the hook reads, and a hook that
throws now rejects the submission instead of being swallowed: a hook whose job
is to reject cannot fail open.

The hook's current-row test also required a TX_TO in the future, while the
read-time engine (mpeDerived) treats a missing or zero TX_TO as current, so a
dependency on an open-ended row validated here and broke at read time. Both the
graph and the inactive set, and the RLS guard, now use the engine's predicate.

The VAR_FMT message named '//*' where the check is for /* and */; it now names
the markers it rejects, in the hook and in its SAS counterpart.
2026-10-25 21:39:19 +00:00
dc e0ee99c178 test(derived): wait for the editor before asserting its absence
should('not.exist') passes the moment the element is missing, so a timeout on
it does not widen the window: an editor that opens after the assertion has
already passed goes unnoticed. The wait gives it that window, which is what the
case needs to rule the cell's editability out.
2026-10-25 21:39:19 +00:00
dc 5d183d1350 feat(mocks): validate a submitted MPE_DERIVED row like the hook does
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m53s
Build / Build-and-test-development (pull_request) Successful in 33m15s
The MPE_TABLES row registered services/hooks/mpe_derived_postedit, but no
mock counterpart existed, so a submission to the now-visible control table skipped
the checks the server hook runs and only failed later, when the EDIT screen
tried to build the view.

The hook mirrors mpe_derived_postedit.sas in its order: normalise the incoming
rows, reject a duplicate name and a bad field, then check the dependency graph
the resulting configuration would have - an inactive dependency, a self
reference, a reference the row does not declare, a declared input the
expression never uses, a name an active RLS rule uses, and a cycle, whose path
is reported.  The collision with a physical column stays with the read-time
engine, as on the server.

A hook rejects a submission by setting hookErrmsg, which stagedata/loadfile
turn into a sasjsAbort before anything is staged; that is the mock's equivalent
of the hook's %mp_abort.
2026-10-25 20:59:15 +00:00
dc fa7a468fd3 test(derived): state the negative assertion's wait explicitly
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m14s
Build / Build-and-test-development (pull_request) Successful in 33m49s
The window a 'not to exist' assertion covers is its timeout, so it is set
here rather than inherited from the config default.
2026-10-25 20:50:38 +00:00
dc d14395a0f8 test(derived): enter edit mode, and add the positive control
Build / Build-and-ng-test (pull_request) Successful in 4m52s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m48s
Build / Build-and-test-development (pull_request) Failing after 33m17s
The screen opens read-only until the edit button is pressed, so the case
asserted nothing about the column: no cell opened an editor. It now enters edit
mode first, asserts the editor's absence directly rather than typing at
whatever happens to be focused, and a sibling case proves a physical column in
the same grid does open one - so the silence is about the column, not the
screen.

Mutating the mock to drop the READONLY rule fails the derived case and passes
the physical one; restoring it passes both (5/5, ~48s).
2026-10-25 20:30:09 +00:00
dc f411b7d269 fix(mocks): copy the physical column list before deriving
physicalColumns held the same array the derive helper reads, so every later
use in the file - cols, the search, the row loop, the empty-row fallback and
vars - depended on that helper returning a new list rather than extending the
one it was given. Taking a copy makes the invariant local to this file.
2026-10-25 20:30:09 +00:00
dc be8f445ae1 fix(cypress): apply the licence key when the estate has none
Build / Build-and-ng-test (pull_request) Successful in 5m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m36s
Build / Build-and-test-development (pull_request) Successful in 32m51s
loginAndUpdateValidKey decided whether a key was already working by looking
for the nav tree, and looked for it in the body captured before login.  A free
tier estate renders the tree as well, so on an unlicensed estate the command
took the 'already licensed' branch every time and never applied a key.  Every
upload spec then lost its feature set: onShowUploadModal shows the locked
feature dialog instead of the panel, so input#file-upload never renders.

The loaded page is read again after login, the licence banner decides whether
a key is needed, and an estate with no key is left untouched unless the caller
asked for one - the free tier specs are written against that state.

Measured on an unlicensed mock estate: excel.cy.ts 24 failing in 14m 1s before,
24 passing in 5m 44s after; csv.cy.ts 1 failing before, 1 passing after.
2026-10-25 19:31:14 +00:00
dc 293022d46b fix(mocks): search the physical columns only in the viewer
Build / Build-and-ng-test (pull_request) Successful in 5m55s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m56s
Build / Build-and-test-development (pull_request) Failing after 36m0s
viewdata.sas drops the derived columns in the vwsearch data step that
mp_searchdata then reads, so a search matches on the columns the grid shows.
The mock handed the full column list, so a search could match a row on a
derived column that is never displayed.
2026-10-25 15:32:57 +00:00
dc 60ec470404 fix(mocks): derive before filtering in the viewer, and drop the columns
Build / Build-and-ng-test (pull_request) Successful in 5m11s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m0s
Build / Build-and-test-development (pull_request) Failing after 36m4s
viewdata.sas builds the derived view and filters against it, so a filter stored
on a derived column resolves in the VIEW screen; the mock served the physical
table only, so the same filter matched nothing and the viewer showed its
no-data row.  The mock now derives ahead of the filter and reports the physical
columns only, as viewdata.sas drops them from the output.
2026-10-25 15:20:15 +00:00
dc ee3c8787ef fix(mocks): register MPE_DERIVED and seed its field rules
Build / Build-and-ng-test (pull_request) Successful in 5m4s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m25s
Build / Build-and-test-development (pull_request) Failing after 36m2s
The mock seeded MPE_DERIVED's rows and schema but no MPE_TABLES registration,
and the table tree lists registered tables only, so the hosted mock showed no
MPE_DERIVED at all - the derived variables could be seen on a table but not
configured.  The field rules and the VAR_TYPE dropdown were missing for the
same reason.

Mirrors mpe_makedata.sas and the v7.19 migration: the MPE_TABLES row
(TXTEMPORAL, buskey LIBREF DSN VAR_NM, mpe_derived_postedit hook), the 19
MPE_VALIDATIONS rows and the two VAR_TYPE selectbox values.
2026-10-25 15:17:43 +00:00
dc 5f443b0b16 fix(derived): hold a derived column read-only whatever the table's security
COLS.CLS_RULE is the only thing that marks a column read-only in the grid,
and the client applies it only when CLS_FLAG is set - which column level
security drives.  A table with no CLS rules therefore rendered its derived
columns as ordinary editable cells, although the values are computed at read
time and dropped from the submission.

getdata now unions a READONLY dq rule per derived variable, which the client
honours unconditionally, and the mock emits the same rule.  The SAS test
asserts the rule; the cypress spec double-clicks a derived cell and asserts the
value is unchanged.
2026-10-25 15:17:40 +00:00
allan 3e8f200a1a Merge pull request 'chore(deps): @sasjs/cli 6.2.0, so the browser mock serves file downloads' (#359) from chore/cli-6.2-mock-downloads into release/7.19
Sites / deploy (push) Successful in 4m4s
Build / Build-and-ng-test (pull_request) Successful in 5m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m3s
Build / Build-and-test-development (pull_request) Successful in 37m27s
Reviewed-on: #359
2026-10-25 14:26:18 +00:00
dc 03b67c44c7 chore(deps): @sasjs/cli 6.2.0, so the browser mock serves file downloads
Build / Build-and-ng-test (pull_request) Successful in 5m24s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m18s
Build / Build-and-test-development (pull_request) Failing after 35m40s
The browser mock site answered every service with text/plain, because the
service worker ignored the headers a service writes to
_SASJS_WEBOUT_HEADERS. CLI 6.2.0 applies the declared content type and
content disposition, and carries a byte body, so the downloads a mock
declares (public/getrawdata in DC) arrive as real files under the name the
service chose.
2026-10-25 13:57:07 +00:00
dc 4d462d310f fix(sites): round-trip the committed licence before publishing
Sites / deploy (push) Successful in 3m51s
Build / Build-and-ng-test (pull_request) Successful in 5m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m8s
Build / Build-and-test-development (pull_request) Successful in 36m44s
The pre-publish check only asserted the pair had two different values, which a
committed pair can satisfy while being broken (one half regenerated, a value
truncated on copy). It now imports the activation key as pkcs8 RSA-OAEP,
decrypts the licence key, parses the payload and exits non-zero otherwise, so a
drifted pair fails the deploy instead of publishing a site that cannot decrypt
its own licence.

CONTRIBUTING records that the two values are one key and are regenerated
together, and that the .gitleaksignore entries are line-scoped.
2026-10-24 23:54:04 +00:00
dc 1f1028a2a5 ci(build): give the Cypress suite the time it now needs
Sites / deploy (push) Successful in 3m44s
Build / Build-and-ng-test (pull_request) Successful in 5m8s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s
Build / Build-and-test-development (pull_request) Successful in 36m42s
The step's inner guard killed Cypress at 30 minutes (exitcode 124) with the
19-spec list still on spec 18, so the job failed without a single failed spec -
seen on a07728b and on the previous head. The suite has grown to roughly 33-36
minutes; the inner guard is now 45 minutes and the step 50.
2026-10-24 23:00:14 +00:00
dc a07728be2b refactor(sites): bundle the site licence instead of minting it
Sites / deploy (push) Successful in 3m59s
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m49s
Build / Build-and-test-development (pull_request) Failing after 35m42s
The published sites get the same key pair on every deploy, committed as
sas/mocks/mock-licence.json (site_id 123, the mock estate's SYSSITE) rather than
generated in the pipeline. The pair sits outside the deployed sync folder, so the
CI estate stays unlicensed and the Cypress specs keep exercising the free tier.

The Sites job copies the pair into the drive before the snapshot and refuses a
pair whose two values are identical, which is how the app spots an HTTP-format
key it cannot decrypt in a secure context. The pair trips the generic-api-key
scanner, so .gitleaksignore carries the two fingerprints.
2026-10-24 22:18:07 +00:00
dc 7d9df71fdb feat(sites): mint a licence for the published sites
Sites / deploy (push) Successful in 4m4s
Build / Build-and-ng-test (pull_request) Successful in 5m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Failing after 36m15s
The demo and staging sites ran on the free tier, so uploads and the other
licensed features were unavailable to anyone looking at them. The Sites job now
mints a self-contained key pair (site_id 123, the estate's SYSSITE) into the
drive before the snapshot, so both sites are fully usable.

Minted per deploy rather than committed, and the CI estate stays unlicensed so
the specs keep exercising the free tier.
2026-10-24 20:32:08 +00:00
dc 75b4acf1b1 test(cypress): upload the licence on a free-tier estate
Sites / deploy (push) Canceled after 1m40s
Build / Build-and-ng-test (pull_request) Successful in 5m43s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m10s
Build / Build-and-test-development (pull_request) Failing after 36m22s
The estate now ships no licence key at all - the placeholder word 'invalid' made
startupservice return an unusable key pair and produced the decrypt-failure page
on every HTTPS site. An unlicensed estate simply runs on the free tier.

loginAndUpdateValidKey therefore no longer infers 'already licensed' from the
nav tree: it detects the free-tier banner (#demo-banner) and uploads a key
through the licensing screen, which posts to admin/registerkey - the API the
licence page's upload button uses.
2026-10-24 20:30:22 +00:00
dc 73d25b1220 fix(mocks): drop derived variables before staging
stagedata.sas removes the MPE_DERIVED droplist from the submitted data before
anything is staged, and writes its approval CSV from that stripped dataset - the
mock staged the derived columns into both.
2026-10-24 20:30:17 +00:00
dc c4021a42ff fix(mocks): carry the zip flags into the central directory
`flags` was scoped to the entry loop and read by the central-directory loop, so
mpZipPackage threw a ReferenceError on every call - the audit download was
broken outright. The value now travels with each central entry.
2026-10-24 20:30:16 +00:00
dc 698dea6d6b fix(ci): dispatch the demo deploy with a tag ref and the actions scope
The dispatch endpoint resolves a bare name as a branch, so the release tag needs
the refs/tags/ prefix, and the route is scoped to the actions unit - contents
does not grant it. Without both, the step failed and the demo stayed on the
previous release.
2026-10-24 20:30:16 +00:00
dc 489819b051 fix(mocks): address the audit-pack review findings
Sites / deploy (push) Successful in 3m52s
Build / Build-and-ng-test (pull_request) Successful in 5m27s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m5s
Build / Build-and-test-development (pull_request) Successful in 33m8s
- zip entries carry flag bit 11 when the name is non-ASCII, so an extractor
  reads `BKP_` plus an accented filename back by name instead of CP437
- loadfile and stagedata name the staged CSV after the libds the request
  arrived with, before a hook can re-point the submission
- stagedata writes that CSV at all (%mp_ds2csv does), so a grid submission's
  audit pack carries the same artefact as an upload's
- loadfile writes no BKP_ file: loadfile.sas copies one only for a second,
  non-CSV fileref, which the mock does not accept
- getauditfile says access is unenforced rather than that there is no
  permission store (MPE_SECURITY is seeded)
- the special-missings clip no longer claims the estate carries a valid key
2026-10-24 20:06:09 +00:00
dc 0b7cf6a724 ci(release): dispatch the demo deploy once the release is published
The release tag is created by the release job's semantic-release step, i.e. as
the Actions user, and the forge suppresses ref events from that user, so the
Sites workflow's `on: push: tags` trigger never fires and the demo would stay on
the previous release. Dispatch the workflow with the new tag after the assets
are uploaded instead.
2026-10-24 20:06:08 +00:00
dc 62f55db170 fix(mocks): clear the licence for published sites, not for the estate
Emptying mock-storage/licence.json dropped every CI run to the free tier, where
the upload button is absent, so excel.cy.ts failed all 24 tests: the licensing
spec registers a key only when the activation page is shown, and an unlicensed
estate activates straight into the app.

The placeholder pair stays in the fixture, and the Sites job strips it from the
drive before the snapshot is taken, so the published sites still run unlicensed
without the decrypt-failure page.
2026-10-24 20:06:03 +00:00
dc 752d944c20 feat(mocks): add getauditfile and the audit-folder artefacts
Sites / deploy (push) Successful in 3m51s
Build / Build-and-ng-test (pull_request) Successful in 5m29s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Failing after 36m12s
The audit download the review screens offer (services/auditors/getauditfile)
had no JS counterpart, so the browser build answered "program not found".

getauditfile.js is the port of getauditfile.sas: table id -> libds from
MPE_SUBMIT, the approval folder zipped under contents/, streamed as
<table>.zip. The zip is built by mpZipPackage and streamed by mpStreamfile in
dcMockUtils, the JS counterparts of the ODS package and %mp_streamfile; the
browser host has a text-only webout, so there the service describes the pack
instead of streaming it.

loadfile.js now leaves the two files loadfile.sas does in the folder as well:
the cleaned CSV and BKP_<upload>.
2026-10-24 19:34:13 +00:00
dc 50f09c8db8 ci(sites): guard the seed and serialise deploys per site
Sites / deploy (push) Successful in 3m47s
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m41s
Build / Build-and-test-development (pull_request) Failing after 36m16s
Two review findings: the makedata call ends in || true (needed for its expected
HTML reply), so a genuine failure published a data-less site as a green build -
the drive is now checked for mpe_tables.json and at least ten tables before the
snapshot is taken. And the concurrency group is keyed on the target site rather
than the branch, so two release branches publishing staging cannot interleave.
2026-10-24 19:14:56 +00:00
dc fb36379520 fix(mocks): stop the mock advertising invalid licence keys
mock-storage/licence.json held the placeholder word 'invalid' for both keys, so
startupservice returned them as globvars and every HTTPS site showed the
'Licence key is invalid / Error decrypting license key' page. An empty pair is
what an unlicensed install has, and it lands on the free tier instead.
2026-10-24 19:14:52 +00:00
dc 6e70d203d1 ci(sites): publish the demo from the release tag, not from main
Sites / deploy (push) Successful in 3m45s
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Successful in 32m25s
The demo now deploys when semantic-release creates the tag, so it can never go
live on a commit whose release then fails, and the checkout carries the tag the
client's build stamp reports.
2026-10-24 19:05:38 +00:00
dc 3ec53076e7 ci(sites): use the staging site name, with the domain as its alias
Sites / deploy (push) Successful in 3m38s
Build / Build-and-ng-test (pull_request) Successful in 5m40s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m10s
Build / Build-and-test-development (pull_request) Successful in 32m44s
2026-10-24 17:39:17 +00:00
dc bb8d007e2d ci(sites): publish release branches to staging and main to demo
Sites / deploy (push) Failing after 3m38s
Build / Build-and-ng-test (pull_request) Successful in 5m17s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m11s
Build / Build-and-test-development (pull_request) Successful in 33m0s
demo.datacontroller.io is now a multi-site Surfer instance: a release branch
deploys the staging.datacontroller.io site, and a push to main deploys the demo
(default site). Both run the same browser-hosted mock build.
2026-10-24 17:31:07 +00:00
dc a128b609f8 ci(demo): publish release branches to demo.datacontroller.io
Demo preview / deploy (push) Successful in 3m47s
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m18s
Build / Build-and-test-development (pull_request) Successful in 30m56s
Pushes to release/** now build the mocks as a browser-hosted site (SASjs CLI
6.1.0, streamConfig.mockWeb) and publish it to the Surfer app behind
demo.datacontroller.io, so the demo always matches the release branch.

The browser build seeds its drive from the compiled services, which carries no
DC data, so the job runs makedata on a throwaway SASjs Server and ships that
drive as mock-snapshot.json.
2026-10-24 16:49:02 +00:00
allan 37abc3adb7 Merge pull request 'build(mocks): emit the browser build for the server-ci target' (#358) from feat/browser-mock-build into feat/derived-vars
Build / Build-and-ng-test (pull_request) Successful in 5m21s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m25s
Build / Build-and-test-development (pull_request) Successful in 31m31s
Reviewed-on: #358
2026-10-24 13:27:37 +00:00
sasjs ab36ff8671 build(mocks): emit the browser build for the server-ci target
Build / Build-and-ng-test (pull_request) Failing after 1m49s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m48s
streamConfig.mockWeb turns the mock site on for the target whose services are JavaScript, so sasjs build produces sasjsbuild/mock alongside the deploy artefact. mockWeb is independent of streamWeb, so the same target still deploys for real.
2026-10-24 13:24:53 +00:00
dc 56ed429183 test(derived): correct the re-add rationale, cover re-activation
Build / Build-and-ng-test (pull_request) Successful in 5m3s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m24s
Build / Build-and-test-development (pull_request) Successful in 31m16s
VAR_NM carries CASE=UPCASE and the post edit hook upcases every non-delete
staged row, so a name typed in the grid reaches submit uppercase: a delete plus
a re-add of it is the same raw primary key and mpe_loader's dup check rejects
that at approval. Test 6's comment claimed the opposite, so it now states the
truth - only a stored name that is not uppercase reaches submit as two PKs, ie
a name written outside the editor.

Test 7 covers the other arm of the exclusion in the same check: setting an
inactive row active again while another active row depends on it. That starting
state is also one the hook refuses to create, so the case is defensive; both
tests prove the guard does not block the submission that repairs the data.
2026-10-24 11:18:12 +00:00
dc 6b5bf3e6be fix(derived): ignore a re-added name when collecting inactive rows
Build / Build-and-ng-test (pull_request) Successful in 4m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m45s
Build / Build-and-test-development (pull_request) Successful in 31m8s
The inactive set behind the bad-dependency check admitted every delete-marked
staged name, so a submission that delete-marks one case variant of a variable
and stages the same name again as a live active row (the two are distinct keys
to the loader's case sensitive duplicate check) matched the dep against the
inactive copy and aborted a configuration the engine resolves fine.

Both arms now exclude a name that this submission also stages as a live active
row - the staged one, and the control row it replaces. stagedata.test.4 gets a
Test 6 for that shape: delete 'Base_X', stage 'BASE_X', with a surviving active
row declaring VAR_DEPS=BASE_X, which must stage successfully.

Verified: sasjs lint (no new warnings), sasjs cb -t nextviya with @sasjs/core
6.0.0, and the built output carries the new predicate and the test. No SAS
estate was reachable from this box (nextviya and sas.4gl.io both time out), so
the test has not been executed on a real engine yet.
2026-10-24 10:04:58 +00:00
dc 033ffd3025 fix(tests): read the work location from _SASJS_WORK
@sasjs/core 6.0.0 renamed the session globals to a _SASJS_ prefix and sets
_SASJS_WORK only; SASJSWORK no longer exists (and sasjs/server's pre-program
compat aliases never covered it). testsetup.sas still read &sasjswork in both
the SASJS and SASMETA arms of %update_settings, so under the new pin those
filename statements built unrooted paths and ms_createfile / mm_createstp wrote
outside the session work directory, breaking sasjs test setup on SASJS and SAS9.

mp_init() is called at the top of the script, so _SASJS_WORK is defined here.
2026-10-24 10:04:53 +00:00
allan ac97252b02 Merge pull request 'fix(mocks): key rk_underlying tables on their business key' (#357) from fix/mock-rk-underlying-pk into feat/derived-vars
Build / Build-and-ng-test (pull_request) Successful in 4m55s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m30s
Build / Build-and-test-development (pull_request) Successful in 30m57s
Reviewed-on: #357
2026-10-24 09:24:39 +00:00
dc 821bf9d1ca Merge branch 'feat/derived-vars' into fix/mock-rk-underlying-pk
Build / Build-and-ng-test (pull_request) Successful in 5m8s
Lighthouse Checks / lighthouse (pull_request) Successful in 23m2s
Build / Build-and-test-development (pull_request) Successful in 31m10s
Two conflicts, both resolved by keeping the intent of each side:

- sas/mocks/sasjs/services/editors/getdata.js: the PK derivation and the
  pkFirst() ordering from this branch, then the derived-variable block from the
  release branch. The derived columns are added to the schema before the
  filter, so the PK-first ordering still applies to the full column set.
- client/package-lock.json: taken from this branch. The merged package.json
  differs from the release branch only in the fast-uri override (4.2.3 here,
  3.1.8 there); the release branch already carries the same @angular 20.3.34
  and @sasjs/adapter 5.1.2 pins, so the lock is unchanged from 74060c0.

Gates on the merged tree: npm audit --omit=dev 0 vulnerabilities, prettier
clean, 603/603 unit tests, ng build, license-checker. Mock services re-probed
against a local SASjs Server: MPE_ROW_LEVEL_SECURITY publishes PKCNT 10 with
the business keys first and no RLS_RK, an edit diffs as UPDATED 1 with the
base RK, and the derived table TESTDATA.DERIVED_DEMO still serves its computed
columns.
2026-10-24 08:40:44 +00:00
dc 74060c0585 fix(mocks): derive the retained key wherever rows are staged
Build / Build-and-ng-test (pull_request) Successful in 4m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m49s
Build / Build-and-test-development (pull_request) Successful in 29m37s
Review follow-ups on the rk_underlying handling:

- mpeAssignRetainedKey (dcMockUtils) mirrors %mp_retainedkey: any submitted RK
  column is dropped and every row's RK is re-derived from the business key, so a
  stale RK in a hand-built payload cannot survive into the diff. Business keys
  compare the way SAS does - case-sensitive, trailing blanks ignored.
- editors/loadfile excludes the RK from its expected-columns check for these
  tables, as loadfile.sas does, and runs the derivation before staging, so a
  real CSV upload (which legitimately omits the RK) is accepted and its diff
  matches instead of reporting every row as NEW.
- both staging paths add the derived column to the staged headers, so
  MPE_SUBMIT.INPUT_VARS counts the width mpe_loader would see.
2026-10-24 00:32:25 +00:00
dc 0ba7450b65 chore(deps): clear the client audit advisories
Build / Build-and-ng-test (pull_request) Successful in 4m52s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m38s
Build / Build-and-test-development (pull_request) Successful in 29m19s
The npm audit gate started failing on advisories that landed upstream, so
nothing can merge until the client tree is patched:

- @angular/* 20.3.33 -> 20.3.34 (XSRF token leak in HttpClient, SSR DoS in
  the router)
- @sasjs/adapter 5.1.0 -> 5.1.2 (brings a patched axios)
- fast-uri override 3.1.8 -> 4.2.3 (the override was pinning the vulnerable
  version; ajv only needs it for URI resolution)
- lockfile refresh takes moment, undici, follow-redirects and brace-expansion
  to their patched versions, and re-resolves the @angular-devkit build
  toolchain within its existing ^ range

Verified in client/: npm audit --omit=dev reports 0 vulnerabilities, npm run
test:headless is 589/589, npm run build succeeds, lint:check and
license-checker pass.
2026-10-23 23:53:15 +00:00
dc 347dbb53b0 fix(mocks): key rk_underlying tables on their business key
Build / Build-and-ng-test (pull_request) Failing after 1m55s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 19m24s
getdata.sas exposes coalesce(rk_underlying, buskey) as the PK and drops the
RK column, so the editor shows the business key as read-only and never sends
the RK. The JS mock still published the RK as the PK and left it in the grid,
and stagedata had no loader to derive it back, so an edit to such a table
(MPE_ROW_LEVEL_SECURITY, MPE_SELECTBOX) could not be diffed.

The mock now mirrors that contract: PK/PKCNT come from rk_underlying, the RK
column is withheld and the PK variables lead COLHEADERS, and stagedata assigns
the RK from the business key - keeping the one the base row already holds and
minting the next value for a new key.
2026-10-23 23:30:57 +00:00
dc 2dc2885c3f chore(deps): @sasjs/cli 6.0.0 and adapter 5.1.2
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m6s
Build / Build-and-test-development (pull_request) Successful in 31m32s
CLI 6 ships the core 6.0.0 macros, so builds now carry the _SASJS_ session
variables; Data Controller reads none of them and calls %mp_init() with no
arguments. Adapter 5.1.2 is a patch on the 5.1.0 already in use.

Verified locally: sasjs cb -t nextviya and sasjs lint, client unit suite
603/603 and the production build.
2026-10-23 19:58:09 +00:00
dc 8806504967 fix(derived): reject a dependency on a row this submission deletes
Build / Build-and-ng-test (pull_request) Successful in 5m13s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m52s
Build / Build-and-test-development (pull_request) Successful in 31m25s
The inactive set behind the VAR_DEPS check took staged rows that are not
delete-marked with active ne 1, plus control rows with active ne 1. A staged
DELETE of a relied-on row was in neither arm, so it passed every guard and the
engine aborted every read after approval. Delete-marked staged rows are now in
the inactive set, and stagedata.test.4 covers it.
2026-10-23 17:07:05 +00:00
dc aa07401766 chore(deps): bump @sasjs/core to 6.0.0
v6 renames the session macro variables to the _SASJS_ prefix and removes
mp_init(prefix=). Data Controller calls %mp_init() with no arguments and reads
none of the renamed variables, so no code change is needed; every macro file
present in 5.7.2 is still present in 6.0.0.
2026-10-23 17:07:05 +00:00
dc 4166679dbf refactor(derived): pick the filter dataset with a single %let
Build / Build-and-ng-test (pull_request) Successful in 6m4s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m50s
Build / Build-and-test-development (pull_request) Successful in 32m25s
%if %then %let ...; %else %let ...; is not valid in open code (SAS stops with
"Expected %DO not found"), which is why these sites used %do/%end blocks. One
%let with %sysfunc(ifc()) needs no blocks and reads as what it is: a choice
between two dataset names. The droplist block in getrawdata/viewdata keeps its
%if %then %do; there is no %else to pair.
2026-10-23 11:29:43 +00:00
dc 688550a28d fix(derived): stop the hooks aborting on their own macro variables
Lighthouse Checks / lighthouse (pull_request) Successful in 21m0s
Build / Build-and-ng-test (pull_request) Successful in 5m13s
Build / Build-and-test-development (pull_request) Successful in 32m7s
A hook is %included from %mp_include, so its open-code %let runs in that
macro's local scope: a %global for the same name inside the hook raises
"exists in a local environment", sets SYSCC, and puts the session into syntax
check mode - every check after it silently skipped while the step still
aborts. The derived hook now sets its (renamed) control variables in the
including scope and does not %global them.

The RLS hook also referenced rls_msg before assigning it whenever there was no
derived conflict, which produced an unresolved-symbol WARNING and SYSCC=4 on
every clean rule submission. It is initialised now.
2026-10-23 10:32:49 +00:00
dc bd4ac50a69 test(derived): cover the MPE_DERIVED submit guards through stagedata
Four cases: a dependency on an inactive row aborts, deactivating a relied-on
row aborts, an inactive row does not block an unrelated variable, and a
delete-marked RLS rule naming a derived variable loads.
2026-10-23 09:44:32 +00:00
dc 7799a300df fix(derived): refuse dependencies on inactive rows, and keep the RLS scope guard coherent
The graph now holds only effective active rows, and a declared dependency that
resolves to an inactive row (existing, or one this submission deactivates) is
rejected: the engine reads only active rows and aborts every read of the table
on the unresolved reference.

The derived-variable / RLS guard goes back to rejecting every scope. Exempting
VIEW rules left them to fail a moment later in mp_filtercheck, which validates
against the physical table where the column does not exist, so the pairing is
refused up front with a message that says why.

RLS batches now exclude delete-marked rows, matching the badloadtypes check, so
a rule being removed is not validated against the derived configuration.
2026-10-23 09:44:32 +00:00
dc 85c89ca2a4 fix(derived): ignore inactive rows and VIEW rules in the submit guards
Build / Build-and-ng-test (pull_request) Successful in 5m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m43s
Build / Build-and-test-development (pull_request) Successful in 33m13s
The dependency graph built at submit time pulled every open current row of
MPE_DERIVED into the union, while the engine only reads active=1 rows. An
inactive row that is still open could therefore abort a submission over a
cycle or a stale reference the engine never generates. The union now filters
active=1, matching mpe_derived.sas.

Both derived-variable guards also matched any active row level security rule,
whatever its scope. A VIEW rule is applied on the view the EDIT screen reads,
where a derived column exists, so naming one there is legitimate; only EDIT and
ALL rules run against the submitted data, where the column is absent. The
derived hook now filters rls_scope in ('EDIT','ALL'), and the RLS hook carries
rls_scope through work.batches to filter its side of the same guard, matching
the staging-side check in the same service.
2026-10-23 09:08:13 +00:00
allan 96914f1cbd Merge pull request 'feat: run startupservice in a new window from its error modal' (#356) from feat/startupservice-launcher into feat/derived-vars
Build / Build-and-ng-test (pull_request) Successful in 5m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m52s
Build / Build-and-test-development (pull_request) Successful in 33m34s
Reviewed-on: #356
2026-10-23 09:01:05 +00:00
dc 3445107d9a fix: name the startupservice launcher button "Show log"
Build / Build-and-ng-test (pull_request) Successful in 5m13s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m26s
Build / Build-and-test-development (pull_request) Successful in 31m20s
2026-10-23 03:01:23 +00:00
dc ad2da45389 fix: offer the launcher on the globvars failure modal too
Build / Build-and-ng-test (pull_request) Successful in 5m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m10s
Build / Build-and-test-development (pull_request) Successful in 31m25s
The globvars check in licence.service reports a failed startupservice run as
well, so its modal needs the service name for the new-window button - that was
the fourth startup failure path and the only one still a dead end.

Also guard the decodeURIComponent call in the Cypress drive/folder intercept,
so a lone '%' in a URL cannot throw inside the handler and abort the request.
2026-10-23 02:17:49 +00:00
dc de71b4456e chore: merge feat/derived-vars into the startupservice launcher branch
Build / Build-and-ng-test (pull_request) Successful in 5m43s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m45s
Build / Build-and-test-development (pull_request) Successful in 32m14s
2026-10-23 01:46:08 +00:00
dc 40f61331aa feat: run startupservice in a new window from its error modal
Build / Build-and-ng-test (pull_request) Failing after 1m56s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 19m25s
When the startupservice fails, the modal is a dead end: the adapter returned
no log, so there is nothing to act on. The modal now offers a button that runs
the same service through the JES / STP web app in a new browser window, where
the SAS log is returned as the page body.

`_debug=131` is always added (the log is the point), and `_executionTasks` is
never included, so on Viya the run is not submitted as a Compute task and
cannot hit the execution-task timeout. The URL is built by `buildServiceUrl`,
which works for SAS 9, Viya and SASjs Server, and is the same shape the deploy
configurator already uses to open makedata.

The startup failure modals now carry the service name, which is what shows the
button: all three failure branches (malformed response, missing properties,
request rejected) pass `public/startupservice`.
2026-10-23 01:15:31 +00:00
allan 0dd55c9796 Merge pull request 'fix(editor): keep decimal values when an uploaded file is edited manually' (#354) from fix/349-upload-edit-decimals into feat/derived-vars
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m8s
Build / Build-and-test-development (pull_request) Successful in 31m6s
Reviewed-on: #354
2026-10-22 22:27:59 +00:00
dc 04ce1861e7 fix(editor): keep decimal values when an uploaded file is edited manually
Build / Build-and-ng-test (pull_request) Successful in 5m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m2s
Build / Build-and-test-development (pull_request) Successful in 31m4s
Confirming the "source file will be discarded" warning coerced every
numeric column of the preview with parseInt, which truncated the
fraction: 0.901045951946194 became 0, and 1234.56 became 1234.
parseFloat converts the same values without dropping precision, and a
cell the parser could not read as a number is left as text so the
editor's numeric validation flags it instead of showing NaN.

Adds an Excel spec case that reads the first data row before and after
the switch to edit mode and asserts the decimal value survives.
2026-10-22 21:03:05 +00:00
allan 7ac620cc22 Merge pull request 'feat(editor): serve a truncated selection with a warning instead of aborting' (#352) from feat/edit-limit-warning into feat/derived-vars
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m0s
Build / Build-and-test-development (pull_request) Successful in 31m9s
Reviewed-on: #352
2026-10-22 19:32:00 +00:00
dc f38d0f80ca chore(getdata): correct the Test 2 banner column count
Build / Build-and-ng-test (pull_request) Successful in 5m45s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m14s
Build / Build-and-test-development (pull_request) Successful in 31m21s
dctest.wideboy is 1004 columns (row_id, 500 char, 500 numeric, plus the
field and position loop variables), so the Test 2 banner says 200 x 1004 =
200,800 cells, matching the @details block.  The row cap is unchanged.
2026-10-22 18:13:42 +00:00
dc 9de56f8a0b chore(deps): bump Angular to 20.3.34 and follow-redirects to 1.16.1
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m27s
Build / Build-and-test-development (pull_request) Successful in 32m8s
The audit gate fails on advisories published against the pinned versions:
@angular/common 20.3.33 carries the XSRF token leakage reported in
GHSA-jfgx-762w-99vr, which reaches forms, platform-browser,
platform-browser-dynamic and router through their peer dependency on
common, and follow-redirects 1.16.0 carries GHSA-q3jh-wrc6-jgq8 and
GHSA-8r9p-f939-6h3c.

Both are patch bumps.  The lockfile is edited in place - the nine
@angular pins and the follow-redirects entry - rather than re-resolved,
because a full install prunes the stale entries the lockfile carries and
turns a pin change into hundreds of unrelated lines.  axios allows
^1.16.0, so the transitive bump needs no parent change.

npm audit --omit=dev reports 0 vulnerabilities at the repo root, in sas/
and in client/.  ng build and the client unit suite (589) pass.
2026-10-22 17:58:03 +00:00
dc 7df4cb29cc chore(getdata): declare mpeinit and mpeterm, correct the fixture counts
Build / Build-and-ng-test (pull_request) Failing after 1m49s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m55s
The doxygen list omitted mpeinit and mpeterm, which the service calls - both
are now declared the way the other services that use mpeinit declare it.  The
compiler inlines them, as it does for those services.  webout stays out of the
list: it is supplied by the framework rather than a macro in the repo, so that
one warning needs an exception in the linter.

dctest.wideboy has 1004 columns, not 1001 - row_id, char1-char500,
num1-num500, plus the loop variables field and position and the scratch
variable alphabet - so the test header's cell arithmetic is 200 x 1004 =
200,800.  The 199 row cap is unchanged, floor(200000/1004) is still 199.

The MPE_X_WIDE mock comment said 200,400 cells for 200 x 1001 columns, which
was the old counting that added the client-owned delete column.  The mock now
counts the table's own columns, so it is 200,200.
2026-10-22 17:44:56 +00:00
dc 66c233602f style(sas): keep the truncation message within the lint line limit
Build / Build-and-ng-test (pull_request) Failing after 1m51s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 19m45s
The message is built in three short assignments rather than one 161
character %let, which .sasjslint rejects (maxLineLength 107).  The
resulting text is unchanged.
2026-10-22 16:56:27 +00:00
dc 5585ae481e docs(comments): describe the limit handling, not the behaviour it replaced
Build / Build-and-ng-test (pull_request) Failing after 1m57s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 19m53s
The getdata comment explained the truncation by contrast with the abort it
replaces.  State what the code does and why instead.  Same for the two
not-refused phrasings in the mock and the test header.
2026-10-22 16:49:21 +00:00
dc 34691dceec feat(editor): serve a truncated selection with a warning instead of aborting
Build / Build-and-ng-test (pull_request) Failing after 3m13s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 19m53s
A selection over DC_MAXOBS_WEBEDIT or DC_MAXCELLS_WEBEDIT was refused with
"Table is too big", leaving the user with no grid and no way to see what the
filter had matched.  getdata now serves the first rows that fit and flags it:

- the row cap is min(DC_MAXOBS_WEBEDIT, DC_MAXCELLS_WEBEDIT / columns), so a
  wide table is still bounded by the cell budget
- sasparams carries TRUNCATED (0/1) and TRUNC_MSG, naming the limit reached
- the EDIT screen shows a warning triangle next to the table name, with the
  message in a tooltip and the advice to filter

The mock mirrors the new behaviour, the Cypress spec asserts the truncated
grid and the tooltip (real hover, opacity checked so a hidden tooltip cannot
pass), and the SAS limit test now asserts the served row counts and message
rather than a cancelled child job.
2026-10-22 16:38:37 +00:00
dc 422f766cd4 fix(derived): close the two gaps in the derived/RLS invariant
Lighthouse Checks / lighthouse (pull_request) Successful in 21m46s
Build / Build-and-ng-test (pull_request) Successful in 5m56s
Build / Build-and-test-development (pull_request) Successful in 30m41s
- the join against MPE_DERIVED compared an upcased var_nm with the rule's
  column as stored, so a case variant slipped past the explanatory rejection
  and fell through to the generic filter check.  Both sides are upcased.
- the invariant was enforced only from the RLS side: configuring a derived
  variable for a table whose active rule already names it passed the derived
  hook.  The same check now runs there, against the incoming libref/dsn/var_nm.
2026-10-21 22:31:56 +00:00
dc af31c36b67 fix(derived): only the delete marker is a reserved name
Build / Build-and-ng-test (pull_request) Successful in 5m54s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m52s
Build / Build-and-test-development (pull_request) Successful in 33m6s
A table's metadata columns are named per table - VAR_TXFROM, VAR_TXTO and
VAR_PROCESSED in MPE_TABLES hold those names, and they can be anything (the
bitemporal demo table uses VER_FROM_DTTM and VER_TO_DTTM).  When they exist
they are physical columns of the target, so the engine's collision check
already refuses a VAR_NM that matches one, whatever it is called.

The hard-coded list treated three example names as special: it refused them on
tables that use other names, and told the reader something untrue about what
is reserved.
2026-10-21 22:23:10 +00:00
dc c5a8de38b3 test(derived): guard the getdata fixture, fix two harness faults, cover the RLS rejection
Build / Build-and-ng-test (pull_request) Successful in 5m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m56s
Build / Build-and-test-development (pull_request) Successful in 32m39s
- getdata.test now checks the child job state before reading its outlib: a
  cancelled child returns no webout, so the previous call's members were being
  asserted against and the failures described the wrong problem.
- validatefilter.test's inputdatasets call needs a _PROGRAM column, which
  mx_execute reads to decide what to run - without it the job stopped at
  'The _PROGRAM column must exist'.
- stagedata.test.3 covers the new RLS check: an MPE_ROW_LEVEL_SECURITY rule
  naming a derived variable must be rejected at save time.
2026-10-21 17:59:49 +00:00
dc ccd468b335 fix(derived): address five review findings
Build / Build-and-ng-test (pull_request) Successful in 6m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m40s
Build / Build-and-test-development (pull_request) Successful in 32m51s
- getrawdata selected its filter target with a bare '%else %let' in open code,
  the shape that stops the step with 'Expected %DO not found'. Both branches
  are %do/%end now, so the download service applies a filter at all.
- the conditional drop in the approve.jsdset step and the one against
  MPSEARCH.vwsearch are dead: work.jsdata and work.vwsearch are already
  rewritten without the derived columns, so the drops only warn.
- the post-edit hook checked that a reference is declared, but not that a
  declared dependency is referenced. The engine rejects the stale shape at read
  time, so the same check now rejects it at submit time.
- a row level security rule naming a derived variable is rejected when the rule
  is saved, with a message that says why, instead of failing later at submit.
- getdata.test's cleanup deleted only DERIVED_NOTE, leaving DERIVED_NOTE2 with
  a dependency on a variable that no longer exists, which aborted every call in
  the following run - the cause of the repeated no-result runs.
2026-10-21 14:02:18 +00:00
dc 88a38db877 fix(derived): pair the filterds %else with %do/%end
Build / Build-and-ng-test (pull_request) Successful in 6m7s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m54s
Build / Build-and-test-development (pull_request) Successful in 33m33s
Both services selected the filter target with
'%if cond %then %let x=y; %else %let x=z;'. In open code the %else is not
paired with the %then and the whole step stops with 'Expected %DO not found',
so neither service ever applied a filter: every assertion downstream failed and
the job produced no results at all.

Found on a real estate - the pattern works inside a macro definition, which is
why the same construct elsewhere in the codebase is fine.
2026-10-21 13:34:02 +00:00
dc 4c198c6e31 chore(deps): @sasjs/core 5.7.2
Build / Build-and-ng-test (pull_request) Successful in 5m59s
Lighthouse Checks / lighthouse (pull_request) Successful in 23m0s
Build / Build-and-test-development (pull_request) Successful in 34m6s
Carries the abort-shape fix from sasjs/core#468 - the cycle abort no longer
re-enters its placement loop, and the validation aborts are driven by a count
at the point of call.
2026-10-21 13:04:31 +00:00
dc 561d7fc3ed fix(derived): eight review findings
Build / Build-and-ng-test (pull_request) Successful in 5m26s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m0s
Build / Build-and-test-development (pull_request) Successful in 33m29s
- mpe_derived_postedit ran %local at the top level of a file that is included
  at execution time, so the hook stopped before validating anything and every
  MPE_DERIVED submission failed. The macro language now lives in a macro
  definition, errflag and errmsg are declared global, and the file calls it.
- the cycle check aborts after leaving its loop, and the self, undeclared and
  field aborts are driven by a count at the point of call rather than by a
  wrapping %if, so an abort cannot be outlived. The same shapes are fixed
  upstream in sasjs/core#468.
- a case-variant duplicate survived mpe_loader, which compares raw values:
  the hook now rejects it after normalisation and names the variable.
- stagedata dropped the derived columns from the output of the security step
  rather than from its input, so a crafted submission could still satisfy an
  RLS clause through a derived column. work.jsdata itself is rewritten now.
- the VAR_FMT validation rule rejected the bare w.d forms that the hook and
  the engine accept.
- the migration carried the same four dangling insert headers as mpe_makedata,
  so the v7.19 migration failed for existing installs.
- the filter fixtures in getdata.test and validatefilter.test had been caught
  by the variable rename: the filter schema column is VARIABLE_NM.
- the TX_FROM/TX_TO definitions agree across the ddl, the migration and
  mpe_makedatamodel.

@sasjs/core bumped to 5.7.1, which carries the where= fix.
2026-10-21 12:54:34 +00:00
dc 9a047ad42e fix(derived): repair the validation rows for MPE_DERIVED
Build / Build-and-ng-test (pull_request) Successful in 6m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m5s
Build / Build-and-test-development (pull_request) Successful in 32m21s
Four stray insert lines and a missing one left the MPE_VALIDATIONS block for
MPE_DERIVED malformed, so makedata aborted and the control table was never
created - found by building the library on a real estate.

The format assertions in the test now read the emitted statement, because proc
contents on a view does not report a format in full ('DATE' for DATE9.).
2026-10-21 12:06:35 +00:00
dc 616a2928e7 refactor(derived): use mp_derivedvars from @sasjs/core
Build / Build-and-ng-test (pull_request) Successful in 5m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m28s
Build / Build-and-test-development (pull_request) Successful in 34m10s
mpe_derived carried its own copy of the derived variable engine. That engine
now ships in @sasjs/core 5.7.0 as mp_derivedvars, so the local copy is gone and
mpe_derived is a wrapper that supplies what is Data Controller's own: the
control table, the bitemporal filter that picks the current active row, and the
list of metadata names that may not be used as a variable.

The macro goes from 511 lines to 102. The contract and the generated statement
format are unchanged, so the callers in the six services need no edit.

A test covers the wrapper's own responsibility: a closed version of a row and
an inactive row are both ignored.
2026-10-21 11:27:37 +00:00
dc ee31782fbf fix(derived): open the statements file with a long record length
Lighthouse Checks / lighthouse (pull_request) Successful in 20m13s
Build / Build-and-ng-test (pull_request) Successful in 5m3s
Build / Build-and-test-development (pull_request) Successful in 31m9s
A label at the 256 character limit produces an attrib statement of about 300
characters, which the default record length wraps, splitting the statement and
breaking the view.
2026-10-21 10:15:33 +00:00
dc ace4a66701 test(derived): snapshot the scope after the setup SQL
Build / Build-and-ng-test (pull_request) Successful in 5m29s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m24s
Build / Build-and-test-development (pull_request) Successful in 31m30s
Same harness fix as the core test: the setup inserts create the SQL automatic
macro variables, so the snapshot has to come after them, and the framework's
_SASJS_WEBOUT_HEADERS is ignored.
2026-10-21 09:10:59 +00:00
dc 7c9ce9a712 feat(derived): optional VAR_LABEL on MPE_DERIVED
Build / Build-and-ng-test (pull_request) Successful in 5m27s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m23s
Build / Build-and-test-development (pull_request) Successful in 31m25s
A configured label replaces the default NAME=expression; when it is missing or
blank the default is used, so existing rows are unaffected.

- MPE_DERIVED gains VAR_LABEL varchar(256), in the ddl, mpe_makedatamodel and
  the v7.19 migration
- mpe_derived uses it, through the same escaping and 256-char truncation as
  the default label
- the hook trims it; the mock mirrors the fallback
- test 1 now sets a label on BAND and asserts it
2026-10-21 08:14:53 +00:00
dc 1b5c4b3ecd fix(derived): truncate the derived label on an even quote boundary
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m0s
Build / Build-and-test-development (pull_request) Successful in 31m34s
The label had 256 chars of storage, so the doubled quotes were truncated on
assignment; a cut between the halves of an escape leaves an unterminated
literal and the view fails to build.  The label now holds 512 chars, quotes
are doubled, then the 256 boundary is applied, retreating one character when
the cut lands on an odd quote count.
2026-10-21 01:49:04 +00:00
dc 5f5d8681b9 fix(derived): guard the empty physical column list in the dep check
Build / Build-and-ng-test (pull_request) Successful in 5m21s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m56s
Build / Build-and-test-development (pull_request) Successful in 31m28s
Same fix as the core review: mf_getvarlist returns nothing for a zero-column
table, so the dependency query compiled as 'not in ()'.
2026-10-21 01:05:10 +00:00
dc 9f70c56b58 chore(deps): @angular/* 20.3.34
Lighthouse Checks / lighthouse (pull_request) Successful in 20m25s
Build / Build-and-ng-test (pull_request) Successful in 4m57s
Build / Build-and-test-development (pull_request) Successful in 30m35s
GHSA-jfgx-762w-99vr (XSRF token leakage in the HTTP client, an incomplete fix
for CVE-2025-66035) covers @angular/common 20.0.0 - 20.3.33, and the 'Check
audit' step fails on any vulnerability, so every branch is red until this
lands.

The nine packages are pinned exactly and must move together - Angular pins its
siblings at the same patch level - so package.json and the lock are updated as
a set.  The local tarball for @sheet/crypto is only present in encrypted form,
so npm cannot run here; the client build and unit tests in CI are the proof.
2026-10-20 23:00:07 +00:00
dc 5af8e6fc35 revert(derived): no shadowing, and no self reference
Allowing a derived variable to reuse a physical column's name was wrong: an
intra-row calculation (SUMMER = SUMMER + 1) implies a retained value, which
depends on the order rows are read.  The same argument rules out a variable
listing itself as a dependency.

Reverted:
- the name collision is an error again ('VAR_NM collides with a column in the
  target table')
- VAR_DEPS may not list the variable itself
- the physical column list is back to mf_getvarlist - the type/length check
  only existed to support shadowing
- the droplist guards in stagedata / viewdata / getrawdata, and the duplicate
  suppression in getdata, are no longer needed
- the docs and both test suites drop the shadowing case
2026-10-20 23:00:01 +00:00
dc 6a1128a463 fix(derived): do not duplicate a shadowed column in the grid payload
Build / Build-and-ng-test (pull_request) Failing after 1m57s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m45s
cols1 already holds the physical column, so appending the derived row for the
same name would send it twice.  The derived row is now appended only when the
name is new; a shadowed column keeps the physical label and format, while its
value comes from the view.
2026-10-20 22:28:44 +00:00
dc 2d5de9a324 fix(derived): do not drop a shadowed column from staging or downloads
Build / Build-and-ng-test (pull_request) Failing after 2m0s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Failing after 4m48s
With shadowing allowed, a derived name can also be a physical column.  The
derived droplist was built from every derived name, so such a column would
have been dropped on submission (losing the user's edit) and from the view
and raw downloads (losing a real column).

stagedata, viewdata and getrawdata now subtract the target's own columns
from the droplist, so only genuinely virtual columns are dropped.
2026-10-20 22:27:19 +00:00
dc 12beeff695 feat(derived): allow a variable to shadow a physical column
Build / Build-and-ng-test (pull_request) Failing after 1m57s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 21m6s
SUMMER = SUMMER + 1 is a legitimate definition: the name matches an existing
column, the expression reads the physical value, and the view carries the
derived one under the same name.

- mpe_derived now takes the physical column list from mp_getcols, so it has
  type and length in one pass; a shadowed name must match the column's type,
  and its length may not exceed it (numeric lengths must be equal), as attrib
  cannot change either on an existing variable
- a self reference is only an error when the name is not a physical column
- self references are not graph edges, so the dependency sort is unaffected
- the post edit hook no longer rejects a self reference, since it cannot see
  the target library; it still rejects cycles between distinct names
- mpe_derived now uses mf_getuniquename for its working tables, matching the
  core macro, and carries the same placement sequence fix
- test 4 covers shadowing, docs cover the rule
2026-10-20 22:25:16 +00:00
dc 1055a0bc4a chore(deps): follow-redirects 1.16.1
Build / Build-and-ng-test (pull_request) Successful in 5m50s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m30s
Build / Build-and-test-development (pull_request) Successful in 31m45s
Two high advisories were published against 1.16.0 in the last few hours
(GHSA-q3jh-wrc6-jgq8, GHSA-8r9p-f939-6h3c), and the 'Check audit' step fails
on any vulnerability, so every run after 18:33 UTC today is red - including
the ones on this branch that had passed earlier.

Lockfile only, no source change: 1.16.1 satisfies the ^1.15.6 range axios
asks for.  Worth cherry-picking to main, since it blocks every PR.
2026-10-20 21:34:21 +00:00
dc d2de3ce88a test(derived): pin the macro's scope in mpe_derived.test.sas
Build / Build-and-ng-test (pull_request) Failing after 1m58s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 21m2s
2026-10-20 21:26:30 +00:00
dc be50cb5d08 fix(derived): carry the placement sequence in mpe_derived
Build / Build-and-ng-test (pull_request) Failing after 1m53s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 21m11s
Same defect as the core review found in mp_derivedvars: the final assemble
merged the placement table by name, but that table is built in dependency
rounds and is not name ordered, so the BY-merge failed whenever the rounds
were not alphabetical, and seq was a BY-group counter rather than the
dependency order.

Each round now stamps seq=_n_+nplaced, the merge runs on the name-sorted
tables, and the result is sorted by seq.  Also guards the physical-column
in-list for a zero-column table, and checks libds exists before running.
2026-10-20 21:24:03 +00:00
dc ee5d7c678e refactor(derived): VAR_DEPS instead of VAR_DEPENDENCIES
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Successful in 31m50s
Shorter column name, same meaning: every input the expression uses,
physical columns and derived variables alike.
2026-10-20 18:33:24 +00:00
dc cee1a954ee feat(derived): report the dependency cycle path
Build / Build-and-ng-test (pull_request) Successful in 5m21s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m59s
Build / Build-and-test-development (pull_request) Successful in 31m35s
The cycle check was already in the post edit hook and in mpe_derived - the
topological sort places a variable only once its derived inputs are placed,
and if a round places nothing the remainder is a cycle.  What it reported was
just the set of names that could not be placed, which does not tell the
administrator what to change.

The message now walks the edges from the first unresolved variable and names
the loop, eg:

  VAR_DEPENDENCIES contains a cycle: A -> B -> C -> A

Names are collected in a dotted seen list, so a lead-in to the loop is shown
as well and the walk stops at the first repeat.
2026-10-20 17:52:04 +00:00
dc 5f6569b9e5 test(derived): declare the real dependencies in the SAS tests
Build / Build-and-ng-test (pull_request) Successful in 6m5s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m21s
Build / Build-and-test-development (pull_request) Successful in 32m59s
The test fixtures now state the inputs their expressions actually use:
LIBREF DSN, DERIVED_NOTE, PRIMARY_KEY_FIELD, and none for a literal-only
expression.  The getdata label assertion follows the new <VAR_NM>=<expr>
label.
2026-10-20 17:06:59 +00:00
dc 813bce6856 feat(derived): declare VAR_DEPENDENCIES instead of VAR_ORDER
Build / Build-and-ng-test (pull_request) Successful in 5m49s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m29s
Build / Build-and-test-development (pull_request) Successful in 32m28s
A sequence number has to be maintained by the administrator: inserting a
variable means renumbering everything after it, and a wrong number fails
silently - the derived value comes back missing, not wrong.

VAR_DEPENDENCIES names every input the expression uses, physical columns and
derived variables alike, eg 'REGION STATE'.  The statements are emitted in
dependency order, so a variable can build on another one with no sequence to
maintain, and a cycle is rejected.

The declaration is checked against the expression in both directions: a
derived variable referenced by VAR_LOGIC must be declared, and a declared
name that the expression does not reference is an error; a declared name must
resolve to a column of the target table or to a derived variable of the same
table.

Changes:
- MPE_DERIVED: VAR_ORDER (num) becomes VAR_DEPENDENCIES (varchar(4000)), in
  the ddl, mpe_makedatamodel and the v7.19 migration
- mpe_derived: dependency validation and a topological sort replace the
  VAR_ORDER sort and the forward-reference check
- mpe_derived_postedit: checks self reference, undeclared derived references
  and cycles at submit time, across the incoming rows plus the current rows
  they do not replace
- MPE_VALIDATIONS: the VAR_ORDER MINVAL/MAXVAL rules are replaced by UPCASE
  and a character pattern on VAR_DEPENDENCIES
- mocks: the dependency order is applied in JS, so a chained variable works
  even when the rows are loaded child first
- docs and tests updated
2026-10-20 16:51:08 +00:00
dc 4d9ad38df1 fix(derived): verify on stripped values, and label with the variable name
Build / Build-and-ng-test (pull_request) Successful in 5m47s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m49s
Build / Build-and-test-development (pull_request) Successful in 32m2s
Found running the macro on a real Viya compute server:

- verify() was applied to fixed width variables, so the blank padding was
  always outside the allowed character list and every name was rejected.  The
  check now runs on strip(name) and strip(fmt).
- the empty outvars dataset used 'length name $32; stop;', which the Viya
  compute server rejects with 'Variable name is uninitialized'.  It now
  assigns the variable and subsets with 'if 0;'.
- the label is now <VAR_NM>=<expression> rather than 'DERIVED: <expression>',
  so the grid shows the variable name as well as the logic.
2026-10-20 11:51:58 +00:00
dc 8bae21230e refactor(derived): drop the duplicate name check from the hook
Lighthouse Checks / lighthouse (pull_request) Successful in 21m9s
Build / Build-and-ng-test (pull_request) Successful in 4m59s
Build / Build-and-test-development (pull_request) Successful in 31m21s
mpe_loader already sorts work.staging_ds by the business key
(buskey='LIBREF DSN VAR_NM') and aborts on duplicates before the post edit
hook runs, so the same check in the hook could never fire.

The check in %mpe_derived stays: it reads the control table, which can also
be edited outside the app.
2026-10-20 09:56:52 +00:00
dc 27ddcfb575 fix(derived): raise the VAR_ORDER ceiling to 10000
Build / Build-and-ng-test (pull_request) Successful in 6m7s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m47s
Build / Build-and-test-development (pull_request) Successful in 32m0s
The client side MAXVAL cannot express 'integer', so integrality stays in the
macro and the hook. The ceiling is only there to catch a typo.
2026-10-20 09:07:18 +00:00
dc aa6d50e1ea feat(derived): validation rules and a VAR_TYPE dropdown for MPE_DERIVED
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Build / Build-and-test-development (pull_request) Canceled after 7m45s
Lighthouse Checks / lighthouse (pull_request) Canceled after 19m16s
Client side rules on the table itself, so mistakes show up in the grid:
  - NOTNULL on LIBREF, DSN, VAR_NM, VAR_ORDER, VAR_TYPE, VAR_LEN, VAR_LOGIC
  - UPCASE on LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_FMT
  - MINVAL 1 / MAXVAL 999 on VAR_ORDER, MINVAL 1 / MAXVAL 32767 on VAR_LEN
  - HARDREGEX on VAR_NM, VAR_FMT and VAR_LOGIC
  - HARDSELECT on VAR_TYPE, with C and N in MPE_SELECTBOX

Seeded in mpe_makedata and in the v7.19 migration. The post edit hook stays
the server side enforcement, since these rules are evaluated in the browser.

Also fixes the MPE_DERIVED create in mpe_makedatamodel, which still used the
old VARIABLE_* column names.
2026-10-20 09:01:55 +00:00
dc 0b010f2c65 feat(derived): validate MPE_DERIVED at submit time
Build / Build-and-ng-test (pull_request) Successful in 6m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m24s
Build / Build-and-test-development (pull_request) Successful in 32m50s
A post edit hook on MPE_DERIVED applies the configuration rules when a row
is submitted, rather than only when the EDIT screen next builds the view.

It normalises LIBREF, DSN, VAR_NM, VAR_TYPE and VAR_FMT, and rejects:
  - a missing LIBREF, DSN or VAR_NM
  - an invalid or reserved name, or one starting with a digit
  - a VAR_ORDER that is not a positive integer
  - a VAR_TYPE other than C or N
  - a VAR_LEN outside 1-32767 (character) or 1-8 (numeric)
  - a VAR_FMT that is not a plain format name
  - a VAR_LOGIC that is not a single expression
  - the same VAR_NM twice for one table
  - a reference to a derived variable that is not defined earlier

The name collision with a physical column is left to %mpe_derived, which runs
with the target library assigned; the hook does not have it.
2026-10-20 08:20:52 +00:00
dc 52803225df refactor(derived): shorten the MPE_DERIVED column names to VAR_*
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m5s
Build / Build-and-test-development (pull_request) Successful in 32m53s
MPE_DERIVED columns:
  VARIABLE_NM     -> VAR_NM
  VARIABLE_ORDER  -> VAR_ORDER
  VARIABLE_TYPE   -> VAR_TYPE
  VARIABLE_LENGTH -> VAR_LEN
  VARIABLE_FORMAT -> VAR_FMT
  VARIABLE_LOGIC  -> VAR_LOGIC

Updated in the ddl, mpe_makedatamodel, the migration, the MPE_TABLES buskey,
the macro (reads and validation messages), the tests, the mocks and the docs.
2026-10-20 07:57:49 +00:00
dc 3ede98fb18 test(derived): cover mpe_derived directly
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m0s
Build / Build-and-test-development (pull_request) Successful in 31m0s
The macro carries the ordering and statement-generation logic, so it is
worth a test of its own:

- character, numeric and chained variables: statements, view, columns
- label and default/configured formats
- a filter on a derived column
- the no-config path: no view, no statements, empty outvars

The configuration checks call mp_abort, so they stay out of the suite.
2026-10-20 03:31:08 +00:00
dc 9ef813d439 fix(derived): close the RLS and search gaps on derived columns
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m12s
Build / Build-and-test-development (pull_request) Successful in 30m59s
Review follow-up.

- stagedata: build the derived droplist before the ULOAD filter, and drop the
  derived columns in the badrecords step, so the row-level-security check runs
  against physical columns only.  A submitted derived value can no longer
  satisfy an RLS clause that names a derived column.
- viewdata: the full-table search view now drops the derived columns too, so
  a free-text search cannot match a value in an EDIT-only column.
- mpe_derived: upcase the libds when matching MPE_DERIVED, so a lower-case
  table reference still finds its configuration instead of silently showing
  no derived columns.
2026-10-20 02:49:24 +00:00
dc 33d043ea92 feat(derived): VARIABLE_ORDER, so a derived variable can build on another
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m15s
Build / Build-and-test-development (pull_request) Successful in 31m0s
A derived variable may need to reference one defined before it (a label that
concatenates a derived region, for example), so the order has to be
configured rather than assumed.

- MPE_DERIVED: new VARIABLE_ORDER column (ddl, makedatamodel, migration).
- mpe_derived: statements are generated in VARIABLE_ORDER then name sequence.
  VARIABLE_ORDER is required and must be a positive integer. A reference to a
  derived variable with an equal or higher order is rejected, because SAS
  would otherwise treat it as uninitialised and return a missing value with no
  error; references to lower-order variables are allowed.
- mock: same ordering, each variable compiled against the physical columns
  plus those defined earlier, so the frontend can exercise a chained variable.
- makedata: VARIABLE_ORDER on both demo rows plus a chained REGION_LABEL.
- cypress: asserts the chained value in the grid.
- tests: the getdata test seeds a second variable built on the first.
- docs: VARIABLE_ORDER documented, with the ordered rule replacing the
  self-contained one.
2026-10-20 02:09:37 +00:00
dc d3dec682aa fix(derived): enforce the documented limits, and handle derived filters in VIEW
Build / Build-and-ng-test (pull_request) Successful in 5m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m59s
Build / Build-and-test-development (pull_request) Successful in 30m51s
Review follow-up.

- viewdata / getrawdata: apply the filter against the derived view when one
  exists, so a saved filter that references a derived column no longer aborts
  with "Variable ... is not on file" in the VIEW screen or a download.  The
  derived columns themselves are dropped from that output - they remain an
  EDIT screen feature.
- mpe_derived: reject a VARIABLE_FORMAT containing a semicolon, macro trigger
  or comment, and require it to look like a format name, so it cannot inject a
  statement into the generated data step.
- mpe_derived: reject a VARIABLE_LOGIC that references another derived
  variable (word-boundary match), which was documented but not enforced.
- mpe_derived: stmt length 4100 so a 4000-char expression plus the name and
  assignment cannot lose its trailing semicolon.
- mock tokenizer: allowlist the punctuation the expression subset uses, so
  anything else is rejected instead of being evaluated.
2026-10-20 01:24:50 +00:00
dc 05ad5544e2 ci: run the derived-variables spec last
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m58s
Build / Build-and-test-development (pull_request) Successful in 30m47s
It applies a licence key via loginAndUpdateValidKey, and csv-limited
asserts the free-tier restriction - so it must not run before that spec.
Running last keeps the licence state every other spec sees unchanged.
2026-10-20 00:43:22 +00:00
dc 8e82a7c857 ci: run the derived-variables cypress spec
Build / Build-and-ng-test (pull_request) Successful in 6m6s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m7s
Build / Build-and-test-development (pull_request) Failing after 31m36s
The cypress step takes an explicit spec list, so a new spec has to be
added to it to run.
2026-10-20 00:03:04 +00:00
dc 11ae41d626 test(cypress): open the EDIT screen filter from the editor, not the viewer
Build / Build-and-ng-test (pull_request) Successful in 5m52s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m35s
Build / Build-and-test-development (pull_request) Canceled after 24m37s
The editor has no .filterSide button - the filter is opened from the
btnView button above the grid.
2026-10-19 23:38:57 +00:00
dc b4fae88ae1 feat(mocks): implement derived variables in the JS services
Build / Build-and-ng-test (pull_request) Successful in 6m5s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m27s
Build / Build-and-test-development (pull_request) Successful in 32m52s
So the EDIT screen can be exercised before a SAS server is available.

- dcMockUtils: mpeDerived() reads MPE_DERIVED for a table and mpeDeriveRows()
  adds the computed columns to the rows and to the schema, before the filter
  runs - the same order as the real service.
- compileSasExpression() covers the subset of SAS expression syntax the mock
  configs use (ifc, cats, catx, upcase, lowcase, strip, trim, length, substr,
  index, comparisons, AND/OR and IN). Anything else leaves the column missing,
  as a failing SAS expression would. The real evaluation is done by SAS.
- getdata / getcolvals apply the derivation before filtering, so a filter on a
  derived column works and dropdown values come from the derived data.
- makedata: MPE_DERIVED schema and rows, a new TESTDATA.DERIVED_DEMO table
  (REGION, LINE_VALUE) and its MPE_TABLES registration.
- cypress: derived-variables.cy.ts covers display and filtering on a character
  and a numeric derived column.

Also bumps @sasjs/core to 5.6.0, which provides the mp_filterstore targetds=
parameter the validatefilter change depends on.
2026-10-19 23:18:46 +00:00
dc b0fd354ecd feat(editor): derived variables in the EDIT screen
Build / Build-and-ng-test (pull_request) Successful in 5m21s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m24s
Build / Build-and-test-development (pull_request) Successful in 31m28s
Adds user-defined computed variables, configured per table in the new
MPE_DERIVED control table. A derived variable is declared with a name,
type, length, optional format and a SAS expression.

The expression is applied in a DATA step view, and the filter is applied
by the step that reads the view, so the variables exist before the
filter is evaluated and can therefore be filtered on. A WHERE clause
cannot see a variable created in the same data step, hence the view.

- getdata: derived columns are added to the COLS payload (read only,
  labelled DERIVED: <expression>) and to the data returned to the grid
- getcolvals: values are taken from the derived view, so filter dropdowns
  work for derived columns
- validatefilter: the filter is validated against the view, and stored
  against the physical table (needs mp_filterstore targetds=)
- stagedata: derived columns are dropped before the approval package
- mpe_makedata / mpe_makedatamodel / ddl / migration: new MPE_DERIVED
  table, registered in MPE_TABLES

Values are display-time only - nothing derived is ever written back.
2026-10-19 22:58:19 +00:00
semantic-release-bot 779006b0bf chore(release): 7.18.0 [skip ci]
# [7.18.0](https://git.datacontroller.io/dc/dc/compare/v7.17.0...v7.18.0) (2026-10-19)

### Bug Fixes

* **deps:** bump @sasjs/core to 5.4.0 ([19fb032](19fb032f0d))
* **deps:** pin @sasjs/utils and drop the unused jsrsasign ([05d2d1e](05d2d1e08e))
* **editor:** own the delete-record column client-side ([56025cc](56025cc052))
* **editor:** surface the reserved-column refusal on every getdata path ([b26f557](b26f5571b6))
* **filters:** do not turn a fractional temporal value into a literal ([f710d87](f710d871ff))
* **filters:** fix the dynamic where clause deref and map its temporal values ([ece67d3](ece67d37ba))
* **filters:** keep the range field readable and cover a BETWEEN in the rewrite ([be79f3a](be79f3a1ea))
* **filters:** map an IN entry to its unformatted value before the literal ([8f98ebe](8f98ebe326))
* **filters:** stop the temporal rewrite corrupting a typed value, and bind the range dropdown to the unformatted value ([a5f61d4](a5f61d4a73))
* **filters:** use the unformatted value in every IN branch ([cc01f54](cc01f540c2))
* **makedata:** shorten the DC_MAXCELLS_WEBEDIT description to fit var_desc ([a51d9e5](a51d9e5574))
* **multi-dataset:** surface the reserved-column guard, use the name constant ([2b8a2fa](2b8a2fa103))
* **release:** emit the release body as raw text, not a JSON string ([a05554e](a05554e67a))

### Features

* **filters:** show dates as date literals in the filter query ([662d000](662d00032c))
2026-10-19 14:14:13 +00:00
allan 6f65bb691f Merge pull request 'feat: release 7.18 - sasjs dependency bumps, filter date literals and editor fixes' (#345) from feat/release-7.18 into main
Release / Build-production-and-ng-test (push) Successful in 5m29s
Release / Build-and-test-development (push) Successful in 25m7s
Release / release (push) Successful in 8m12s
Reviewed-on: #345
2026-10-19 13:39:26 +00:00
allan 3d21a0134c Merge pull request 'fix(makedata): shorten the DC_MAXCELLS_WEBEDIT description to fit var_desc' (#346) from fix/makedata-vardesc-length into feat/release-7.18
Build / Build-and-ng-test (pull_request) Successful in 5m54s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m22s
Build / Build-and-test-development (pull_request) Successful in 32m5s
Reviewed-on: #346
2026-10-19 13:39:02 +00:00
dc a51d9e5574 fix(makedata): shorten the DC_MAXCELLS_WEBEDIT description to fit var_desc
Lighthouse Checks / lighthouse (pull_request) Successful in 20m46s
Build / Build-and-ng-test (pull_request) Successful in 5m23s
Build / Build-and-test-development (pull_request) Successful in 30m13s
MPE_CONFIG.VAR_DESC is varchar(300), but the description written for
DC_MAXCELLS_WEBEDIT is 336 characters, so the insert truncates it.  The
truncation raises a WARNING, which sets SYSCC, and makedata aborts on
`&syscc ne 0` immediately after %mpe_makedata():

    WARNING: Character expression will be truncated when assigned to
      character column var_desc.
    ERROR: Execution terminated by an ABORT CANCEL statement

The abort happens before the settings job is written, so a fresh deploy
ends with no control library and no settings.sas at all.  It goes
unnoticed because the SAS test suite reuses whatever library is already
on the estate - the first time it bites is a deploy from scratch.

The description is shortened to 201 characters in both places it is
written - mpe_makedata (new installs) and the 20261003 row/cell limits
migration (existing installs).  The rationale it carried is user-visible
help text, so it moves to a SAS comment above each insert rather than
being dropped.

Verified on a real Viya estate, deploying from scratch:
  - before: makedata aborted, "Err during VIYA1337 build", no settings.sas
  - after:  makedata completed with zero warnings, and the SAS test suite
            then ran 59/59 green
2026-10-19 12:22:13 +00:00
hermes 026930d3e4 Merge pull request 'feat(filters): show dates as date literals in the filter query' (#344) from feat/filter-date-literals into feat/release-7.18
Lighthouse Checks / lighthouse (pull_request) Successful in 20m29s
Build / Build-and-ng-test (pull_request) Successful in 5m6s
Build / Build-and-test-development (pull_request) Successful in 29m19s
2026-10-17 20:10:37 +00:00
dc 05d2d1e08e fix(deps): pin @sasjs/utils and drop the unused jsrsasign
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m50s
Build / Build-and-test-development (pull_request) Successful in 29m57s
@sasjs/utils is imported directly across the client but was left as a caret
range in the 7.18 bump, while every other dependency in the file is exact.
Pin it to 4.0.0, which is the version the lockfile already resolves.

jsrsasign is declared but referenced nowhere in the tracked source - the only
other mentions are the lockfile entry and a historical CHANGELOG line - so
drop it.

Lockfile regenerated with npm install --package-lock-only; the only deltas are
the pinned range and the removed jsrsasign tree. 570/570 unit tests pass.
2026-10-17 19:32:14 +00:00
dc cc01f540c2 fix(filters): use the unformatted value in every IN branch
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m55s
Build / Build-and-test-development (pull_request) Successful in 29m58s
Finding from the review at ece67d3.

dynamicWhereClauseCreator's IN branch mapped the entry through element.values
only when building the literal; the two fallbacks emitted the unmapped value, so
a formatted label could still reach RAW_VALUE.  A temporal entry whose mapped
value is not a whole day / second (sasTemporalLiteral returns null for a
genuinely fractional value) or whose label finds no match in element.values
would fall through to the quoted label on a numeric column, which mp_filtercheck
rejects and which aborts the getcolvals load inside the swallowed catch.

The mapped value is now assigned once and used in all three branches, as
createFilterQueryTable's IN branch does.

One unit case added for the non-literal fallback.  582/582 unit tests pass.
2026-10-16 21:44:08 +00:00
dc ece67d37ba fix(filters): fix the dynamic where clause deref and map its temporal values
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m3s
Build / Build-and-test-development (pull_request) Successful in 29m51s
Two findings from the review at 8f98ebe.

The IN branch read this.clauses.queryObj[groupInd][clauseInd].value, but the
entries sit at queryObj[groupInd].elements[clauseInd].value, so the forEach
dereferenced undefined and threw.  dynamicWhereClauseCreator runs inside
getValues / getValuesLoadMore, where the surrounding try/catch swallowed the
TypeError and the value list silently failed to load.  It now reads
element.value, like the BETWEEN and equality branches.

The BETWEEN branch composed element.value directly, so a temporal BETWEEN
picked from the value list sent the formatted strings into RAW_VALUE, which
mp_filtercheck rejects on a numeric column and which aborts the values load.
Both ends now go through the same formatted-to-unformatted lookup the IN branch
and createFilterQueryTable use, in one unformattedValue helper shared by the
two branches.

query.component gains a spec covering both branches.  581/581 unit tests pass.
2026-10-16 21:03:48 +00:00
dc 8f98ebe326 fix(filters): map an IN entry to its unformatted value before the literal
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m51s
Build / Build-and-test-development (pull_request) Successful in 30m50s
Finding from the review at be79f3a.

createFilterQueryTable's IN / NOT IN branch read element.val directly, so it
produced a temporal literal only because whereClauseCreator had already mutated
query.value[i].val from the formatted label to the unformatted count in place
before submit.  Any path reaching the branch without the preview having run
first would have put the formatted string into RAW_VALUE.

The entry now goes through the same unformattedValue() lookup as the BETWEEN
ends.  An already-unformatted value passes through unchanged, since it no longer
matches a formatted label.

Two unit cases added: an IN entry carrying the formatted label, and one already
carrying the internal count.  578/578 unit tests pass.
2026-10-16 20:21:38 +00:00
dc be79f3a1ea fix(filters): keep the range field readable and cover a BETWEEN in the rewrite
Build / Build-and-ng-test (pull_request) Successful in 5m10s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m27s
Build / Build-and-test-development (pull_request) Successful in 30m5s
Two findings from the review at 72a7963.

whereClauseCreator's guard could not match a BETWEEN: checkOperator composes
it as `<from> AND <to>` and the letters of AND are outside the character class,
so the temporal rewrite was skipped and the filter-panel preview kept the
internal day counts (`BETWEEN 42 AND 43`).  The separator is removed before the
test, so a BETWEEN pair is shown as literals like a single value or an IN list.

Binding the range dropdown to column.unformatted made the field render the
internal value: setOptionValue assigns the clicked option's value to the input's
ngModel, so after choosing 12FEB1960 the field read 42.  The option value is the
formatted label again, as the single-value list and the IN modal already are,
and createFilterQueryTable's BETWEEN branch maps each end back to its
unformatted value before building the literal - the same lookup
whereClauseCreator applies for the preview.

Three new unit cases pin the BETWEEN preview and the BETWEEN RAW_VALUE in both
directions.  576/576 unit tests pass.
2026-10-16 16:29:57 +00:00
dc 72a7963972 test(e2e): cover a temporal BETWEEN chosen from the value list
Build / Build-and-ng-test (pull_request) Successful in 5m56s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m18s
Build / Build-and-test-development (pull_request) Successful in 31m9s
With the pickers off, a temporal BETWEEN takes both of its values from a
soft-select over the column's values, so the clause is composed from whatever
that list binds.  The test chooses 12FEB1960 for both ends of SOME_DATE and
asserts the applied clause carries the literals and still selects the 5 rows
the internal value selected.

Typing into the range fields - which is what the existing helper does - does
not exercise the binding, so both values are picked from the datalist instead.

Also accepts the evaluation agreement in beforeAll when it is presented.  It
is per-origin localStorage, so a fresh browser profile lands on it and the nav
tree never renders; the accept is guarded, so it is a no-op once an earlier
spec in the run has accepted it.
2026-10-16 13:52:19 +00:00
dc a5f61d4a73 fix(filters): stop the temporal rewrite corrupting a typed value, and bind the range dropdown to the unformatted value
Build / Build-and-ng-test (pull_request) Successful in 5m51s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m11s
Build / Build-and-test-development (pull_request) Successful in 31m50s
Two findings from the review at f710d87.

whereClauseCreator's temporal rewrite is a regex pass over the composed value
string, so it rewrote every digit run in it.  A date typed by hand as
12/02/1960 composed to 12/02/1960 and the clause preview rendered
'13JAN1960'd/'03JAN1960'd/'14MAY1965'd.  The pass now runs only when the
composed value is nothing but the numbers the clause is built from - a single
value, a BETWEEN pair or an IN list - so a typed value is left alone while an
internal day count is still shown as a literal.

The range dropdown bound [value]="column.formatted", where the single-value
dropdown binds column.unformatted.  whereClauseCreator's BETWEEN mapping writes
the unformatted value to a local copy rather than back onto the emitted clause,
so the formatted strings reached createFilterQueryTable, sasTemporalLiteral
returned null for them, and RAW_VALUE came out as `12FEB1960 AND 12FEB1960` -
unquoted, which mp_filtercheck rejects for a numeric column and so aborts the
save.  The option value is now the unformatted value, with the formatted text
kept as the label, matching the single-value dropdown.

573/573 unit tests pass, including two new cases pinning the preview: a typed
value is left alone, and an internal day count is still shown as a literal.
2026-10-16 13:18:46 +00:00
dc f710d871ff fix(filters): do not turn a fractional temporal value into a literal
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m47s
Build / Build-and-test-development (pull_request) Successful in 32m9s
sasTemporalLiteral rendered the fraction with String(fraction).slice(1), which
emits raw IEEE-754 noise - 42.3 came out as '00:00:42.29999999999999716't and
1.1 as .10000000000000009.  Only the 0.5 case in the spec was clean.

SAS temporal values are whole days and whole seconds, so the value is rounded to
seconds first, which absorbs the noise a double picks up on the way here
(1.9999999999999 is simply the second 2).  A value that is genuinely fractional
does not land on a second, and rounding it would silently widen the filter, so
it now returns null and the caller sends the raw number instead.

sasTimeFromSeconds has no fraction left to render, so that branch is gone.

Also corrects the doc examples, which paired 34322 with '20APR2042'd - 34322
days after 1960-01-01 is '20DEC2053'd, which is what the spec already asserted.
Same fix in the mock's parser comment.

571/571 unit tests pass.
2026-10-16 13:07:52 +00:00
allan d6e0662b16 Merge pull request 'fix(editor): own the delete-record column client-side' (#343) from fix/getdata-delete-column-client-side into feat/release-7.18
Build / Build-and-ng-test (pull_request) Successful in 5m54s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m16s
Build / Build-and-test-development (pull_request) Successful in 32m5s
Reviewed-on: #343
2026-10-16 13:01:43 +00:00
allan f45b3259c7 Merge pull request 'fix(release): emit the release body as raw text, not a JSON string' (#342) from fix/release-notes-body-encoding into feat/release-7.18
Build / Build-and-ng-test (pull_request) Successful in 6m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m33s
Build / Build-and-test-development (pull_request) Successful in 31m11s
Reviewed-on: #342
2026-10-16 13:00:48 +00:00
dc b53f951307 chore: merge the 7.18 release branch into the filter date literals branch
Build / Build-and-ng-test (pull_request) Successful in 6m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m40s
Build / Build-and-test-development (pull_request) Successful in 32m53s
2026-10-16 12:42:46 +00:00
dc 9f8c71eb47 chore(deps): bump the sasjs dependencies for the 7.18 release
Build / Build-and-ng-test (pull_request) Successful in 6m11s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m12s
Build / Build-and-test-development (pull_request) Successful in 33m12s
sas/package.json
  @sasjs/cli    4.20.5 -> 4.20.9
  @sasjs/core   5.2.8  -> 5.5.2

client/package.json
  @sasjs/adapter  4.19.1 -> 5.1.0
  @sasjs/utils    ^3.6.0 -> ^4.0.0

Both client bumps are majors, and both are safe here:

- @sasjs/adapter 5.0.0 raised engines.node to >= 22.  The workflows run Node
  24.15.0 (24.5.0 in release.yaml), so the floor is satisfied.  5.1.0 also
  repackaged the adapter as compiled modules - DC imports only the package
  root (SASjsConfig, UploadFile, SASjsRequest), which index.d.ts still
  re-exports through ./types, and the package ships no exports map, so the
  imports DC does make are unaffected.
- @sasjs/utils 4.0.0 stopped re-exporting ./fs from the package root.  DC
  imports @sasjs/utils/input/validators and @sasjs/utils/types/serverType,
  neither of which is fs, and both still ship in 4.0.0.

@sasjs/adapter 5.1.0 depends on @sasjs/utils 4.0.0, so the two move together.

The shelljs override in sas/package.json is dropped: @sasjs/cli 4.20.9 no
longer depends on shelljs, so nothing in the tree resolves to it any more and
the pin was dead config.  The nanoid pin is untouched.

Lockfiles regenerated with npm install --package-lock-only --ignore-scripts,
then validated by enumerating every added, removed and version-changed entry
and tracing each one back to @sasjs/cli, @sasjs/core, @sasjs/adapter or
@sasjs/utils.  No entry was pruned as extraneous.

Verified by reproducing the Build-and-ng-test gates locally:

  npm audit --omit=dev      0 vulnerabilities in root, sas/ and client/
  npm run lint:check        pass
  npm run license-checker   pass
  npm run test:headless     563/563 SUCCESS
  npm run build             pass
2026-10-16 12:41:29 +00:00
dc 19fb032f0d fix(deps): bump @sasjs/core to 5.4.0
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m15s
Build / Build-and-test-development (pull_request) Successful in 31m29s
5.4.0 carries the mp_filtercheck change that accepts a SAS temporal literal
in RAW_VALUE, which is what the filter literal work needs.  Without it the
literal is rejected when the filter is stored, so the clause stays numeric on
a real estate even though the mocks (which do not run the SAS macro) pass.

Verified the build inlines it from the project copy, not the CLI's nested one:
sasjsbuild/server_compileTree.json resolves mp_filtercheck.sas from
node_modules/@sasjs/core, and the compiled server.sas carries the fix.
2026-10-16 11:16:33 +00:00
dc 662d00032c feat(filters): show dates as date literals in the filter query
Build / Build-and-ng-test (pull_request) Successful in 5m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m22s
Build / Build-and-test-development (pull_request) Successful in 31m9s
The clause shown above the grid (sasparams.FILTER_TEXT) is the resolved WHERE
clause, so a date filter read SOME_DATE = 34322 rather than a date.  The
client now sends a SAS literal for a DATE, DATETIME or TIME column -
'20APR2042'd, '01JAN1960:00:00:42'dt, '00:00:42't - which SAS accepts in the
same place, so the applied filter is unchanged and the clause is readable.

HelperService.sasTemporalLiteral builds the literal arithmetically in UTC, so
the day cannot shift with the browser timezone, and fractional seconds are
preserved.  It returns null for anything that is not a temporal value, and the
callers then send the raw value exactly as before.

Used by createFilterQueryTable (the applied filter), whereClauseCreator (the
clause preview in the filter panel), dynamicWhereClauseCreator (the filter
getcolvals applies to the value list) and the VA filter service.

Two fixes fall out of the same code paths:

- the dynamic where clause read the second BETWEEN value from
  queryObj[groupInd][clauseInd], which is undefined, so a BETWEEN clause threw;
- an IN over a temporal column sent the *formatted* value, which the backend
  refuses for a numeric column - the entries are now mapped back to the
  unformatted value before the literal is built.

The mock filter parser reads the literals too.  A time literal carries no date
part, so it needs its own pattern - the date pattern alone silently matched
nothing and a time filter returned no rows.

filtering.cy.ts asserts the new clause text and, for every temporal case, the
row count, so a clause that reads well but filters differently cannot pass.
2026-10-16 09:36:36 +00:00
dc b26f5571b6 fix(editor): surface the reserved-column refusal on every getdata path
Build / Build-and-ng-test (pull_request) Successful in 6m24s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m18s
Build / Build-and-test-development (pull_request) Successful in 32m50s
Review on #343:

- initSetup's retry branch called itself from a bare setTimeout, outside the
  callService promise chain, so the reserved-column throw on a retry never
  reached the .catch: getdataError was never set and the user was left on the
  loading spinner. Wrap the retry call and route it through the same failure
  handler the .catch uses (now a private handleGetdataFailure, shared by
  both).
- The multi-dataset catch logged and returned, leaving an empty table
  selection with the file still attached and only a console.warn. Surface it
  with eventService.showInfoModal, the pattern this file already uses for
  other upload problems.
2026-10-16 00:40:29 +00:00
dc 2b8a2fa103 fix(multi-dataset): surface the reserved-column guard, use the name constant
Lighthouse Checks / lighthouse (pull_request) Successful in 21m1s
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Build / Build-and-test-development (pull_request) Successful in 32m10s
Review on #343:

- buildDatasetsObjects() runs inside the Promise.allSettled(...).then(...)
  chain in onStartParsingFile, which had no .catch. prepareDeleteRecordColumn
  throws on a response carrying the reserved delete-record column, so in this
  flow that became an unhandled rejection: the loop aborted, no dataset was
  parsed and no state was shown. Wrap the call and drop the parse, the way
  fetchDataset's own catch handles a failed getdata.
- The rename-to-Delete? block still used the literal name while the same
  block in editor.component.ts uses DELETE_RECORD_COLUMN_NAME. Use the
  constant here too.
2026-10-15 23:46:22 +00:00
dc 56025cc052 fix(editor): own the delete-record column client-side
Build / Build-and-ng-test (pull_request) Successful in 5m36s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m15s
Build / Build-and-test-development (pull_request) Successful in 30m57s
getdata.sas synthesised _____DELETE__THIS__RECORD_____ itself - it added
the column to work.out, hoisted it in front of the PKs, set it to "No" on
every row, gave it a No/Yes dropdown in the metadata and forced MAXLEN=3
in maxvarlengths. In the editor that column is a constant: it is only ever
"No" on load, and its Yes/No rule is already injected client-side
(deleteRecordColumnRule.ts, since #253's vertical-array fix). So the
backend was sending a column the client already owns.

The editor now creates the column, the same way EDIT_STATUS / dc.row_status
is client-only:

- getdata.sas no longer emits it (data steps, the outdata hoist, the
  ctrlOptions dropdown, the maxvarlengths override).
- deleteRecordColumn.ts asserts the response does NOT carry the name, then
  prepends it to COLHEADERS. Everything downstream (DcValidator's rules,
  columnHeader, headerArray) is keyed off COLHEADERS order, so the delete
  column is still index 0 and the grid is unchanged.
- Every row is seeded to 'No' before the validator or HyperFormula sees it.
- A response that does carry the name now throws instead of rendering a
  table whose delete column would shadow a real column of that name. The
  throw lands in the existing getdata .catch, which sets getdataError and
  shows the error state. The name is reserved; the docs note is in the
  sibling docs PR.

The multi-dataset loader (multi-dataset.component.ts) builds its own
DcValidator from the same responses, so it goes through the same helper.

Mocks updated to the new contract (server + sas9 getdata); postdata,
getstagetable and makedata are untouched - they consume or seed the flag
legitimately, and the upload contract in excel.md / files.md is unchanged.

Verified: client unit tests 570/570, npm run build clean, sasjs lint clean
on the changed SAS, sasjs compile -t server succeeds, and against the
mocked estate coltype-delete-record.cy.ts 2/2, editor.cy.ts 55/55,
excel.cy.ts 23/23 and row-cell-limits.cy.ts 5/5.
2026-10-15 22:52:44 +00:00
dc a05554e67a fix(release): emit the release body as raw text, not a JSON string
Build / Build-and-ng-test (pull_request) Successful in 5m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m53s
Build / Build-and-test-development (pull_request) Successful in 30m52s
The "Upload assets to release" step builds the new body with
`jq -n --arg body ... '$body + "\n" + ...'` and no `-r`. jq then prints the
result as a JSON string literal - quoted, with `\n` escapes - which the
PATCH that follows encodes a second time, so the stored release body is a
quoted one-liner instead of markdown. On v7.17.0 this rendered the whole
release as a single escaped line in the markdown box.

Add `-r` so the concatenated notes are emitted as raw text. Verified by
running the step's jq invocation against sample notes: without `-r` the
stored body contains literal `\n`, with `-r` it contains real newlines.

release.yaml only runs on a push to main, so this cannot be exercised by
CI on the PR itself.
2026-10-15 20:50:01 +00:00
145 changed files with 13203 additions and 2770 deletions

No files matched your search

+2 -1
View File
@@ -1,10 +1,11 @@
# Testing
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory. For a deploy-and-run on the shared state see the `dc-estate-testing` skill (`.agents/skills/dc-estate-testing/SKILL.md`), which covers the credentials, the `makedata` seed the suite needs and the estate's group name.
## Commands
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
- The shared estate under its `server` target: `npm run server` (compile, deploy, seed) then `sasjs test -t server`.
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
+79
View File
@@ -0,0 +1,79 @@
---
name: dc-estate-testing
description: >
Use when deploying Data Controller to a shared SASjs Server estate (sas.4gl.io) or running the SAS
test suite against it. Covers the target and credential setup, the makedata seed the suite needs on
a fresh appLoc and the group it must name, how to read the results, and how to deploy the frontend
so the streamed app is the build you just made.
---
# Deploying and testing on the shared SASjs Server estate
The whole cycle is two npm scripts plus `sasjs test`. Everything below explains what those scripts do and the steps that go wrong.
cd sas && npm ci # macro dependencies
cd ../client && npm ci && npm run build # frontend only - see "Deploying the frontend"
cd ../sas && npm run server # cpfavicon + sasjs cbd -t server + makedata
npx sasjs test -t server
`npm run server` is `sasjs cbd -t server && npm run serverdata`, and the seed in the second half is not optional. `sasjs cbd` on its own deploys the services and the test programs but leaves the appLoc with no Data Controller configuration, and `testsetup` reads `<appLoc>/services/public/settings.sas` through the drive API. `makedata` is what creates that file, so without the seed `testsetup` gets a 404, aborts, and 59 of the 64 test programs fail to complete on setup alone - which reads like a broken deployment rather than a missing seed.
## The targets
`server` is the shared estate: `serverType: SASJS` on https://sas.4gl.io, `appLoc` `/Public/app/dc`, a real SAS 9 binary behind SASjs Server (`GET /SASjsApi/info` reports `mode: server`, `runTimes: ["sas"]`, `localLoginEnabled: true`). `4gl` is the same estate with a slightly different config, hence the older `npm run 4gl` / `4glmakedata` pair.
A run against `server` exercises the SASjs Server branch of every macro and the SAS 9 backend, and nothing else: no Viya branch, no SAS 9 STP branch. Say that when reporting, rather than implying the matrix is covered.
To deploy your own branch without replacing the app everyone else uses, add a target that copies `server` and changes `appLoc` and `streamServiceName` (for example `/Public/app/<you>/dc`), keep it out of the commit, and pass it with `-t`. `sasjs cbd` overwrites the whole appLoc it is pointed at.
## Credentials
The CLI logs in with a registered client, not a username and password: `.env.<target>` (gitignored, mode 600) needs `CLIENT`, `ACCESS_TOKEN` and `REFRESH_TOKEN`. With `CLIENT` alone it prints a `/SASjsLogon?client_id=...` URL and waits for an authorization code, which cannot be answered headless.
Mint the pair from a local login, with the password read from a file rather than argv:
1. `GET /` with a cookie jar - it sets `XSRF-TOKEN`, which every POST must echo in an `X-XSRF-TOKEN` header.
2. `POST /SASLogon/login` with `{"username","password"}`.
3. `GET /SASjsApi/client` on the same session - the registered client ids.
4. `POST /SASLogon/authorize` with `{"clientId"}` -> `{"code"}`.
5. `POST /SASjsApi/auth/token` with `{"clientId","code"}` -> `accessToken` / `refreshToken`.
An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check the content type; HTML means the route does not exist. Delete the token file when the run is over.
## The makedata seed
`npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
- **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
- **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
- **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.
## Reading the results
Results land in `sas/sasjsresults/`: `testResults.json` (the counts worth quoting: `tests_with_results`, `tests_that_pass`, `failed_to_complete`, `completed_with_failures`), `testResults.csv` (one row per assertion, lower-case headers `test_target`, `test_loc`, `sasjs_test_id`, `test_suite_result`, `test_description`, `test_url`), and one SAS log per program under `logs/`. The CLI's closing lines are not the source of truth when they disagree with the JSON.
A current green run is 64 of 64 test programs with 241 assertions passing and nothing failing to complete.
`ERROR` lines in the per-program logs are expected and are not a failed run: the specs that assert a refusal get `ERROR: Execution canceled by an %ABORT CANCEL statement`, and the ones that feed bad payloads get the rejection message they are asserting on (`Invalid libds:...`, `Invalid libref:...`). Judge the run by the JSON counts, then by whether the errors you find are the ones the tests asked for.
## Deploying the frontend
The streamed web app is a separate artefact and `sasjs cbd` only streams it if `client/dist` exists:
1. `client/libraries/sheet-crypto.tgz.gpg` is an encrypted dependency (`file:libraries/sheet-crypto.tgz` in `client/package.json`) that is not on the public registry. Decrypt it with the passphrase the maintainer holds: `printf '%s' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 --output ./libraries/sheet-crypto.tgz --decrypt ./libraries/sheet-crypto.tgz.gpg`.
2. `client/.npmrc` is gitignored and needs `legacy-peer-deps=true` (the committed lock is written with it) plus the `@sheet` registry lines.
3. `cd client && npm ci && npm run build`.
4. The build carries the repo default appLoc, so point it at the deployment before deploying: rewrite `apploc="/Public/app/dc"` in `dist/index.html` to the target appLoc.
5. `cd ../sas && npx sasjs cbd -t <target>` - with `client/dist` present the log says `Copying web app files` and `Web app is available at <url>`.
Without step 1 the deploy prints `ERROR An error has occurred when compiling web app services ... webSourcePath: .../client/dist doesn't exist` and then deploys the services anyway, so the deploy looks successful while the app streams nothing. The AppStream URL is session-protected: an anonymous `curl` gets a `401`, so verify it with a logged-in cookie jar and fetch the `main.<hash>.js` the page names.
Never commit `client/.npmrc` or the decrypted tarball; both are ignored, and `git status` staying clean is the check.
## Reading a failed run
- `ERROR An error occurred while executing the request. Job location: <appLoc>/tests/...` means the response was not JSON, so the program broke before it could write results. Open `sasjsresults/logs/<suite>.log` and read the first `ERROR`; a single compile break cascades into hundreds of `ERROR 180-322: Statement is not valid` lines and the tail is noise.
- `WARNING: Apparent symbolic reference TESTLOC not resolved` in `testsetup.log`, next to a `404 Not Found` for `services/public/settings.sas`, is the missing seed described above, not a macro problem.
- `sasjs test` runs the DEPLOYED program, so after editing a service, hook or test, run `sasjs cbd` again before testing or you will debug code that is no longer in the tree.
- A deployment can succeed and still be missing its web app, its data or its configuration - check the thing you changed rather than the exit code.
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env bash
# The Cypress specs CI runs, and the order they run in.
#
# ORDER MATTERS. These specs share one mock estate that is seeded once per job,
# so several of them depend on the state they find:
#
# - csv-limited asserts the free-tier "Locked Feature" modal, which an earlier
# spec can remove by planting a valid licence. It has always run first.
# - hook-programs asserts that an edit raises an approval row against its
# target table; it runs early, before other specs have added rows to the
# approval tables.
#
# Specs added later are appended rather than inserted, so the sequence that was
# known green keeps its shape. metanav is deliberately absent - see
# https://git.datacontroller.io/dc/dc/issues/395.
#
# download was held out while its selectors were stale - see
# https://git.datacontroller.io/dc/dc/issues/398. Its first case uploads a fixture
# to mpe_x_test and submits it, so it creates the approval it downloads instead of
# depending on what earlier specs leave behind - but the approval stays behind.
# It is appended because a spec added later goes on the end rather than reordering
# the sequence that was known green.
set -euo pipefail
SPECS="csv-limited hook-programs liveness editor excel-multi-load excel csv filtering filter-panel licensing viewer-labels full-table-search viewbox stage browser-info ddl-export row-cell-limits derived-variables startupservice-launcher coltype-delete-record submitted-details embed-va download"
# Specs that exist but are held out of CI, each with the reason. Listed here so
# the coverage check below can tell a deliberate exclusion from a forgotten file.
EXCLUDED="metanav"
case "${1:-}" in
all)
# comma separated, the form `cypress run --spec` takes
for name in $SPECS; do printf 'cypress/e2e/%s.cy.ts,' "$name"; done | sed 's/,$//'
;;
coverage)
on_disk=$(cd "$2/client" && ls cypress/e2e/*.cy.ts | xargs -n1 basename | sed 's/\.cy\.ts$//' | sort)
listed=$(printf '%s\n' $SPECS $EXCLUDED | sort)
missing=$(comm -23 <(echo "$on_disk") <(echo "$listed"))
if [ -n "$missing" ]; then
echo "Specs that are neither run nor explicitly excluded:"
echo "$missing"
echo "Add them to SPECS (or EXCLUDED, with a reason) in .gitea/scripts/cypress-specs.sh"
exit 1
fi
echo "All $(echo "$on_disk" | wc -l) specs are accounted for: $(echo $SPECS | wc -w) run, $(echo $EXCLUDED | wc -w) excluded."
;;
*)
echo "usage: $0 {all|coverage <repo-root>}" >&2
exit 2
;;
esac
+45 -5
View File
@@ -75,11 +75,33 @@ jobs:
cd client
npm run test:headless
- name: Angular Tests in other timezones
# The VA filter clauses are built from local calendar dates, so a
# UTC-only run cannot see a date that slips a day west of UTC or east of
# it. Only the VA specs carry that property, so both scripts run that
# subset rather than the whole suite a second and third time.
run: |
cd client
npm run test:tz-west
npm run test:tz-east
- name: Every Cypress spec is in the CI list
# The spec list used to be spelled out inline in --spec, and five files
# quietly fell out of it - among them the VA embed suite. Fails when a
# spec on disk is not listed.
run: bash .gitea/scripts/cypress-specs.sh coverage .
- name: Production Build
run: |
cd client
npm run build
# Cypress shard 1 of 4. The setup repeats in every shard because each
# one gets its own runner and its own SASjs server; that fixed cost is what
# limits how far sharding helps. The specs it runs come from
# .gitea/scripts/cypress-specs.sh.
Build-and-test-development:
runs-on: ubuntu-latest
needs: Build-and-ng-test
@@ -142,6 +164,7 @@ jobs:
echo NODE_PATH=node >> .env
echo CORS=enable >> .env
echo WHITELIST=http://localhost:4200 >> .env
echo HTTP_PORT=5000 >> .env
# The server refuses to run as root unless RUN_AS names an account
# (a security change in the recent releases). This job runs as root
# in a throwaway container, so opt in explicitly.
@@ -181,9 +204,20 @@ jobs:
sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
- name: Prepare and run frontend and cypress
timeout-minutes: 35
# One job, one SASjs server, one ng serve, one Chrome: four of each on one
# runner starved the app under test and every failure in that run was a
# timeout, not an assertion.
timeout-minutes: 50
run: |
cd ./client
# The spec list lives in one place, and an empty list is a broken
# workflow rather than a green run with no tests.
SPECS="$(bash ../.gitea/scripts/cypress-specs.sh all)"
if [ -z "$SPECS" ]; then
echo "the spec list resolved to nothing"
exit 1
fi
echo "specs: $SPECS"
mv ./cypress.env.example.json ./cypress.env.json
replace-in-files --regex='"username".*' --replacement='"username":"'${{ secrets.CYPRESS_USERNAME_SASJS }}'",' ./cypress.env.json
replace-in-files --regex='"password".*' --replacement='"password":"'${{ secrets.CYPRESS_PWD_SASJS }}'" ' ./cypress.env.json
@@ -196,18 +230,24 @@ jobs:
replace-in-files --regex='"hosturl".*' --replacement='hosturl:"http://localhost:4200",' ./cypress.config.ts
cat ./cypress.config.ts
# Start frontend and run cypress
# timeout 1800: SIGTERM after 30 min so Cypress can flush video/screenshots
# timeout 2700: SIGTERM after 45 min so Cypress can flush video/screenshots
# before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/hook-programs.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/filter-panel.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts,cypress/e2e/browser-info.cy.ts,cypress/e2e/ddl-export.cy.ts,cypress/e2e/row-cell-limits.cy.ts"
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --spec "$SPECS"
- name: Zip Cypress videos
if: always()
# Only on a failure: a green run has nothing to watch, and zipping and
# uploading every run's videos is the part of the video cost that lands
# on the clock.
if: failure()
run: |
mkdir -p ./client/cypress/videos
zip -r cypress-videos ./client/cypress/videos
- name: Add cypress videos artifacts
if: always()
if: failure()
# v3, not v4: this runner reports "GHESNotSupportedError:
# @actions/artifact v2.0.0+, upload-artifact@v4+ ... not supported on
# GHES", so the current major cannot upload here at all.
uses: actions/upload-artifact@v3
with:
name: cypress-videos.zip
+121
View File
@@ -0,0 +1,121 @@
name: Publish code documentation
run-name: Publishing typedoc and SAS docs
# The code documentation sites are published here rather than inline in the
# release job, so that they can be republished on their own - after a secret
# rotation, or for a release whose artefact upload failed (the release job
# cannot simply be re-run: semantic-release refuses once the tag exists).
on:
workflow_dispatch:
env:
NODE_VERSION: '24.5.0'
jobs:
typedoc:
runs-on: ubuntu-latest
# One deploy per site at a time: `surfer deploy` swaps the site atomically,
# so two runs racing is not corruption, but it is a wasted build and the
# last swap wins.
concurrency:
group: surfer-webdoc
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Write .npmrc file
run: |
printf '%s\n' "$NPMRC" >> client/.npmrc
echo "legacy-peer-deps=true" >> client/.npmrc
shell: bash
env:
NPMRC: ${{ secrets.NPMRC }}
- name: Install dependencies
run: |
cd client
# Decrypt and Install sheet
echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
shell: bash
env:
SHEET_PWD: ${{ secrets.SHEET_PWD }}
- name: Build typedoc
run: |
cd client
npm run compodoc:build
# Deploy tooling is pinned to an exact version: an unpinned `latest` would
# let a breaking or compromised upstream release publish straight to the
# live site. `deploy` (v7) replaces the site atomically, and it takes the
# Cloudron username plus an app password - the v6 `put --token` flow this
# replaces is gone.
#
# webdoc is a NAMED site on the app whose default site is the marketing
# site, so the name and its alias domain are both passed. Surfer picks
# the target only from `--site-name` (it travels as the Surfer-Deployment
# header); without it `deploy` replaces the app's DEFAULT site, which is
# what serves datacontroller.io - so an unnamed deploy published the
# Typedoc over the top of the marketing site instead of to webdoc.
- name: Deploy typedoc
env:
SURFER_PASSWORD: ${{ secrets.WEBDOC_DATACONTROLLER_IO }}
REF_NAME: ${{ gitea.ref_name }}
COMMIT_SHA: ${{ gitea.sha }}
run: |
set -euo pipefail
npx --yes @cloudron/surfer@7.2.0 deploy client/documentation \
--server webdoc.datacontroller.io \
--site-name webdoc \
--domain webdoc.datacontroller.io \
--username allan \
--password "$SURFER_PASSWORD" \
--message "Typedoc from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
sas-docs:
runs-on: ubuntu-latest
concurrency:
group: surfer-code-datacontroller
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Install dependencies
run: |
# `sasjs doc` shells out to doxygen
apt-get update
apt-get install -y doxygen
cd sas
npm ci
CLI_VERSION=$(node -p "require('./package.json').dependencies['@sasjs/cli']")
npm i -g "@sasjs/cli@${CLI_VERSION}"
- name: Build SAS docs
run: |
cd sas
sasjs doc
# No `--site-name` here on purpose: code.datacontroller.io is its own app
# and the SAS docs are its DEFAULT site, so naming one would create a
# separate site and leave the served copy stale. Do not copy the named
# form from the typedoc job above - that app's default site is the
# marketing site, which is why it must not be the target.
- name: Deploy SAS docs
env:
SURFER_PASSWORD: ${{ secrets.CODE_DATACONTROLLER_IO }}
REF_NAME: ${{ gitea.ref_name }}
COMMIT_SHA: ${{ gitea.sha }}
run: |
set -euo pipefail
npx --yes @cloudron/surfer@7.2.0 deploy sas/sasjsbuild/sasdocs \
--server code.datacontroller.io \
--username allan \
--password "$SURFER_PASSWORD" \
--message "SAS docs from ${REF_NAME} @ ${COMMIT_SHA:0:8}"
+1
View File
@@ -33,6 +33,7 @@ jobs:
echo NODE_PATH=node >> .env
echo CORS=enable >> .env
echo WHITELIST=http://localhost:4200 >> .env
echo HTTP_PORT=5000 >> .env
# The server refuses to run as root unless RUN_AS names an account
# (a security change in the recent releases). This job runs as root
# in a throwaway container, so opt in explicitly.
+209
View File
@@ -0,0 +1,209 @@
name: Release assets
run-name: Building and uploading the release assets
# Builds the downloadable artefacts for an existing release and uploads them.
#
# The release job cuts the tag before it builds, so a failure anywhere between
# the semantic-release step and the asset upload leaves a published release with
# no files - and that job cannot simply be re-run, because semantic-release
# refuses once the tag exists. This workflow rebuilds the artefacts from the tag
# and attaches them, so the release can be completed without cutting a new one.
on:
workflow_dispatch:
inputs:
tag:
description: 'Release tag to build and upload (blank = the latest release)'
required: false
default: ''
env:
NODE_VERSION: '24.5.0'
permissions:
contents: write
jobs:
assets:
runs-on: ubuntu-latest
# Two runs uploading the same asset names would race the delete-then-upload
# below.
concurrency:
group: release-assets
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Write .npmrc file
run: |
printf '%s\n' "$NPMRC" >> client/.npmrc
echo "legacy-peer-deps=true" >> client/.npmrc
shell: bash
env:
NPMRC: ${{ secrets.NPMRC }}
- name: Install system packages
# jq parses the release JSON below, so it is installed before the tag
# resolve.
run: |
apt-get update
apt-get install -y zip jq
- name: Resolve the release tag
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
REQUESTED_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
if [ -n "${REQUESTED_TAG}" ]; then
TAG="${REQUESTED_TAG}"
# Fail here rather than after a ten-minute build if the tag is not
# cut yet or has no release.
curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${TAG}" > /dev/null
else
TAG=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest" | jq -r '.tag_name')
fi
echo "Building assets for ${TAG}"
echo "TAG=${TAG}" >> "$GITHUB_ENV"
- name: Check out the tag
run: |
set -euo pipefail
git fetch --force --tags origin "+refs/tags/${TAG}:refs/tags/${TAG}"
git checkout --force "refs/tags/${TAG}"
- name: Install the SASjs CLI the tag pins
run: |
cd sas
CLI_VERSION=$(node -p "require('./package.json').dependencies['@sasjs/cli']")
npm i -g "@sasjs/cli@${CLI_VERSION}"
- name: Frontend build
run: |
cd client
# Decrypt and Install sheet
echo "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
npm i webpack
npm run build
shell: bash
env:
SHEET_PWD: ${{ secrets.SHEET_PWD }}
- name: Build SAS9 EBI Release
run: |
cd sas
npm i
sasjs c -t sas9
rm -rf sasjsbuild/tests
sasjs b -t sas9
cp sasjsbuild/mysas9deploy.sas ./demostream_sas9.sas
# remove streamed component and rebuild SAS 9 services
rm -rf sasjsbuild/services/web9
rm sasjsbuild/services/clickme.sas
sasjs b -t sas9
cp sasjsbuild/mysas9deploy.sas ./sas9.sas
- name: Build SASjs Server Release
run: |
cd sas
cp sasjs/utils/favicon.ico ../client/dist/favicon.ico
sasjs c -t server
rm -rf sasjsbuild/tests
server_apploc="/Public/app/dc"
sed -i "s|apploc=\"[^\"]*\"|apploc=\"${server_apploc}\"|g" sasjsbuild/services/web/index.html
sasjs b -t server
cp sasjsbuild/server.json.zip ./sasjs_server.json.zip
- name: Build Viya Release
run: |
cd sas
sasjs c -t viya
rm -rf sasjsbuild/tests
sed -i -e 's/servertype="SASJS"/servertype="SASVIYA"/g' sasjsbuild/services/DC.html
sasjs b -t viya
cp sasjsbuild/viya.sas ./viya.sas
# compile Viya Full deploy (without web)
rm -rf sasjsbuild/services/web
rm sasjsbuild/services/DC.html
sasjs b -t viya
cp sasjsbuild/viya.sas ./viya_noweb.sas
cp sasjsbuild/viya.json ./viya_noweb.json
- name: Zip Frontend (including viya.json for full viya deploy)
run: |
cd sas
cp sasjsbuild/viya.json ../client/dist/viya.json
# Zip from *inside* dist so the archive holds the frontend files at its
# root rather than under a client/dist/ prefix (see #147).
cd ../client/dist
zip -r ../../frontend.zip .
- name: Upload assets to the release
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
RELEASE_JSON=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${TAG}")
RELEASE_ID=$(echo "$RELEASE_JSON" | jq -r '.id')
RELEASE_BODY=$(echo "$RELEASE_JSON" | jq -r '.body')
ASSETS=(frontend.zip
sas/demostream_sas9.sas
sas/viya.sas
sas/sasjs_server.json.zip
sas/sas9.sas
sas/viya_noweb.sas
sas/viya_noweb.json)
: > SHA256SUMS
for f in "${ASSETS[@]}"; do
sum=$(sha256sum "$f" | cut -d' ' -f1)
printf '%s %s\n' "$sum" "${f##*/}" >> SHA256SUMS
done
cat SHA256SUMS
# Replacing the assets means removing the ones this workflow uploads,
# and a second run must not append the verification section twice. An
# asset the release carries from another source is left alone.
OWNED=$(printf '%s\n' "${ASSETS[@]##*/}" SHA256SUMS)
echo "$RELEASE_JSON" | jq -r '.assets[]? | "\(.id) \(.name)"' |
while read -r asset_id asset_name; do
if printf '%s\n' "$OWNED" | grep -qxF -- "$asset_name"; then
echo "Removing previous ${asset_name}"
curl --fail-with-body -sS -X DELETE -H "$AUTH_HEADER" "$BASE/releases/${RELEASE_ID}/assets/${asset_id}"
else
echo "Leaving ${asset_name} in place: this workflow did not upload it"
fi
done
if echo "$RELEASE_BODY" | grep -q "## Verifying the download"; then
echo "Release body already carries the verification section"
else
VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
'$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
curl --fail-with-body -sS -X PATCH \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg body "$NEW_BODY" '{draft:false, body:$body}')" \
"$BASE/releases/$RELEASE_ID"
fi
URL="$BASE/releases/${RELEASE_ID}/assets"
for f in "${ASSETS[@]}"; do
echo "Uploading $f ..."
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
done
echo "Uploading SHA256SUMS ..."
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
+80 -18
View File
@@ -107,6 +107,16 @@ jobs:
echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
- name: Build frontend for web streaming
# The server-ci target streams client/dist as the 'clickme' web app, so
# the production build has to exist before the mocked services are
# deployed. Without it `sasjs cbd` reports
# "webSourcePath: .../client/dist ... doesn't exist", streams no
# frontend and the deploy step fails.
run: |
cd client
npm run build
- name: Setup and start SASjs server
run: |
npm i -g pm2
@@ -117,6 +127,7 @@ jobs:
echo NODE_PATH=node >> .env
echo CORS=enable >> .env
echo WHITELIST=http://localhost:4200 >> .env
echo HTTP_PORT=5000 >> .env
# The server refuses to run as root unless RUN_AS names an account
# (a security change in the recent releases). This job runs as root
# in a throwaway container, so opt in explicitly.
@@ -193,11 +204,15 @@ jobs:
# and its permissions are derived from this block.
# contents: write -> push CHANGELOG.md / package.json commit + tag, and
# create the Release via the Gitea API.
# actions: write -> dispatch the Sites workflow for the new tag (that
# route is scoped to the actions unit, which contents
# does not grant).
# The TSDOC_TOKEN and CODE_DATACONTROLLER_IO secrets continue to be used
# because they target *different* servers (webdoc.datacontroller.io and
# code.datacontroller.io), not the Gitea API.
permissions:
contents: write
actions: write
steps:
- uses: actions/checkout@v4
@@ -266,6 +281,11 @@ jobs:
semantic-release
# The tag this job cuts, for the steps that publish against it: a
# `releases/latest` lookup would pick whichever release landed last
# if two overlap.
echo "RELEASE_TAG=v$(node -p "require('./package.json').version")" >> "$GITHUB_ENV"
- name: Frontend Build
description: Must be created AFTER the release as the version (git tag) is used in the interface
run: |
@@ -326,19 +346,6 @@ jobs:
cd ../client/dist
zip -r ../../frontend.zip .
- name: Release Typedoc
run: |
cd client
npm -g install cloudron-surfer
npm run compodoc:build
surfer put --token ${{ secrets.TSDOC_TOKEN }} --server webdoc.datacontroller.io documentation/* /
- name: Release code.datacontroller.io
run: |
cd sas
sasjs doc
surfer put --token ${{ secrets.CODE_DATACONTROLLER_IO }} --server code.datacontroller.io sasjsbuild/sasdocs/* /
- name: Upload assets to release
env:
# Use the same ephemeral per-job Gitea Actions token that the
@@ -353,7 +360,7 @@ jobs:
AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
RELEASE_JSON=$(curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/latest")
RELEASE_JSON=$(curl --fail-with-body -sS -H "$AUTH_HEADER" "$BASE/releases/tags/${RELEASE_TAG}")
RELEASE_ID=$(echo "$RELEASE_JSON" | jq -r '.id')
RELEASE_BODY=$(echo "$RELEASE_JSON" | jq -r '.body')
@@ -380,10 +387,14 @@ jobs:
# ahead of the installation footer. The section text lives in a
# dedicated script below (.gitea/scripts/verify-section.sh) so this
# YAML scalar stays clean.
# `jq -r` is required: without it jq emits the result as a JSON string
# literal (quoted, with \n escapes), and the PATCH below then encodes
# that literal a second time, so the release body renders as one
# escaped line instead of markdown.
VERIFY_SECTION=$(./.gitea/scripts/verify-section.sh)
NEW_BODY=$(jq -n --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
NEW_BODY=$(jq -nr --arg body "$RELEASE_BODY" --arg verify "$VERIFY_SECTION" \
'$body + "\n" + $verify + "\n\nFor installation instructions, please visit https://docs.datacontroller.io/"')
curl -k --fail-with-body -sS -X PATCH \
curl --fail-with-body -sS -X PATCH \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg body "$NEW_BODY" '{draft:false, body:$body}')" \
@@ -393,7 +404,58 @@ jobs:
URL="$BASE/releases/$RELEASE_ID/assets"
for f in "${ASSETS[@]}"; do
echo "Uploading $f ..."
curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@$f"
done
echo "Uploading SHA256SUMS ..."
curl -k --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
curl --fail-with-body -sS -H "$AUTH_HEADER" "$URL" -F "attachment=@SHA256SUMS"
- name: Publish the code documentation
# Typedoc (webdoc) and the SAS docs (code) are published by docs.yaml,
# not inline here. Two reasons: the v6 `surfer put --token` flow is gone
# in Surfer 7, which takes a Cloudron username plus an app password; and
# a dispatch means a stale or failed docs deploy can be republished
# without re-running this job, which semantic-release refuses once the
# tag exists.
#
# This runs AFTER the release assets are uploaded: the documentation is
# a side channel, and a docs problem must not cost a cut release its
# artefacts.
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token $GITEA_TOKEN"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
TAG="${RELEASE_TAG}"
echo "Dispatching code documentation for ${TAG}"
curl --fail-with-body -sS -X POST \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
"$BASE/actions/workflows/docs.yaml/dispatches"
- name: Publish the demo site for this release
# The tag is created by this job's semantic-release step, so it carries
# the Actions user as its author - and the forge suppresses ref events
# from that user, which means `on: push: tags` in sites.yaml never fires.
# Dispatch it instead: the API call is authenticated, and the run it
# starts checks out the tag the same way.
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
AUTH_HEADER="Authorization: token ${GITEA_TOKEN}"
BASE="${{ gitea.server_url }}/api/v1/repos/${{ gitea.repository }}"
TAG="${RELEASE_TAG}"
echo "Dispatching Sites for ${TAG}"
# the dispatch endpoint resolves a bare name as a branch; a tag needs
# the refs/tags/ prefix
curl --fail-with-body -sS -X POST \
-H "$AUTH_HEADER" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg ref "refs/tags/$TAG" '{ref: $ref}')" \
"$BASE/actions/workflows/sites.yaml/dispatches"
+183
View File
@@ -0,0 +1,183 @@
name: Sites
on:
push:
branches:
- 'release/**'
tags:
- 'v*'
paths-ignore:
- '**.md'
workflow_dispatch:
env:
NODE_VERSION: '24.15.0'
APP_LOC: /Public/app/dc
SURFER_HOST: https://demo.datacontroller.io
# A release branch publishes the staging site (a named site on the instance,
# served at its alias domain); a release tag publishes the demo, which is the
# default site and so needs no site name.
STAGING_SITE: staging
STAGING_DOMAIN: staging.datacontroller.io
# One deploy per target site: a newer push supersedes the one in flight, and two
# release branches (which share the staging site) cannot interleave their swaps.
# The demo and staging deploys are independent, so they do not queue.
concurrency:
group: surfer-${{ startsWith(gitea.ref, 'refs/tags/') && 'demo' || 'staging' }}
cancel-in-progress: true
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# The tag is what the client's build stamp describes (src/version.ts
# runs git-describe), so the demo build needs the tags in the clone.
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Write .npmrc file
run: |
touch client/.npmrc
echo '${{ secrets.NPMRC}}' > client/.npmrc
- name: Install client dependencies
run: |
cd client
# Decrypt and Install sheet
echo ${{ secrets.SHEET_PWD }} | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
npm ci
- name: Build frontend
# The site serves this build. `sasjs build` rewrites its index.html to
# point the adapter at the in-browser backend.
run: |
cd client
npm run postinstall
npm run build
- name: Install the SASjs CLI
# Pinned in sas/package.json - the browser build needs CLI >= 6.1.0
run: |
cd sas
npm ci
- name: Start SASjs server
# Needed only to run makedata: the drive the site ships with is captured
# from a real server (see the snapshot step below).
run: |
curl -L https://github.com/sasjs/server/releases/latest/download/linux.zip > linux.zip
unzip -q linux.zip
touch .env
echo RUN_TIMES=js >> .env
echo NODE_PATH=node >> .env
echo HTTP_PORT=5000 >> .env
# The server refuses to run as root unless RUN_AS names an account
echo RUN_AS=root >> .env
nohup ./api-linux > server.log 2>&1 &
for i in $(seq 1 60); do
if curl -sf -o /dev/null http://localhost:5000/; then
echo "SASjs server answered on attempt ${i}"
exit 0
fi
sleep 2
done
echo "SASjs server did not come up within 120s"
cat server.log
exit 1
- name: Deploy the mocked services and seed the database
run: |
cd sas/mocks/sasjs
# Start from a clean appLoc so the snapshot below is this build alone,
# not a merge with whatever a previous run left on the drive.
curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=$APP_LOC"
../../node_modules/.bin/sasjs cbd -t server-ci
# makedata replies with HTML (it is normally called as a URL redirect),
# which the CLI reports as "invalid Json string" - that exit is
# expected. A real failure must not be published as an empty site, so
# the drive is checked before the snapshot is taken.
../../node_modules/.bin/sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
DATA_DIR="../../../sasjs_root/drive/files$APP_LOC/data"
SEEDED=$(find "$DATA_DIR" -name '*.json' 2>/dev/null | wc -l)
echo "makedata seeded $SEEDED table(s) under $DATA_DIR"
if [ -z "$(find "$DATA_DIR" -name 'mpe_tables.json' 2>/dev/null | head -1)" ] || [ "$SEEDED" -lt 10 ]; then
echo "makedata did not seed the drive - refusing to publish a data-less site"
find ../../../sasjs_root/drive/files -maxdepth 3 2>/dev/null | head -20
exit 1
fi
- name: Build the browser-hosted site
run: |
# The published sites ship a working licence so they are usable beyond
# the free tier. The pair is committed in sas/mocks/mock-licence.json
# rather than minted here, so every deploy of both sites carries the
# same key; the CI estate stays unlicensed because that file is outside
# the deployed sync folder, so the specs still exercise the free tier.
# The pair must round trip through the same WebCrypto the app uses,
# because a committed pair can drift (one half regenerated, a value
# truncated on copy) and nothing else would notice before publishing.
# Identical values in the two fields is the HTTP-format marker the app
# refuses in a secure context, so refuse to publish one.
node -e "
const d = require('./sas/mocks/mock-licence.json')
const b = (s) => Buffer.from(String(s || ''), 'base64')
if (!d.licenceKey || !d.activationKey || d.licenceKey === d.activationKey) {
console.error('sas/mocks/mock-licence.json must hold an HTTPS-format key pair')
process.exit(1)
}
crypto.subtle
.importKey('pkcs8', b(d.activationKey), { name: 'RSA-OAEP', hash: 'SHA-256' }, false, ['decrypt'])
.then((key) => crypto.subtle.decrypt({ name: 'RSA-OAEP', hash: 'SHA-256' }, key, b(d.licenceKey)))
.then((plain) => {
const data = JSON.parse(new TextDecoder().decode(plain))
console.log('licence pair round trips: site', data.site_id, 'until', data.valid_until)
})
.catch((e) => {
console.error('sas/mocks/mock-licence.json does not decrypt:', e.message || e)
process.exit(1)
})
"
LICENCE_DIR="sasjs_root/drive/files$APP_LOC/mock-storage"
mkdir -p "$LICENCE_DIR"
cp sas/mocks/mock-licence.json "$LICENCE_DIR/licence.json"
# DC's data files exist only after makedata has run, so the drive is
# captured first and shipped as the site's starting drive. services/web9
# is the streamed frontend, which is already the site itself.
node sas/mocks/snapshot.js sasjs_root/drive "$APP_LOC" sas/mocks/sasjsbuild/mock-snapshot.json services/web9
cd sas/mocks/sasjs
../../node_modules/.bin/sasjs b -t server-ci
- name: Publish to Surfer
env:
SURFER_USER: allan
SURFER_PASSWORD: ${{ secrets.DEMO_DATACONTROLLER_IO }}
REF: ${{ gitea.ref }}
MESSAGE: Build from ${{ gitea.ref_name }} @ ${{ gitea.sha }}
run: |
# `deploy` uploads the folder as an archive and swaps the site in one
# step, so a failed upload leaves the current site serving.
if [ -z "$REF" ]; then
echo "REF is empty - cannot tell a branch push from a tag push"
exit 1
fi
SITE_ARGS=()
case "$REF" in
refs/tags/*) ;;
*) SITE_ARGS=(--site-name "$STAGING_SITE" --domain "$STAGING_DOMAIN") ;;
esac
npx --yes @cloudron/surfer@7.2.0 deploy \
--server "$SURFER_HOST" \
--username "$SURFER_USER" \
--password "$SURFER_PASSWORD" \
--message "$MESSAGE" \
"${SITE_ARGS[@]}" \
sas/mocks/sasjsbuild/mock
+5
View File
@@ -0,0 +1,5 @@
# sas/mocks/mock-licence.json holds a licence key pair for the mock estate: the
# activation key is RSA key material the browser demo decrypts with, not a
# credential for any service.
sas/mocks/mock-licence.json:generic-api-key:2
sas/mocks/mock-licence.json:generic-api-key:3
+3 -1
View File
@@ -32,6 +32,8 @@ Never consider a change complete until the relevant linters pass on the files yo
- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
`sas/.sasjslint` carries `ignoredMacros: ["webout", "mpeinit", "mpeterm"]`. `webout` is the linter's own default - `sasjs compile` injects `%webout` into every service - and an explicit list replaces that default, so it has to be restated there. `mpeinit` and `mpeterm` are listed because every service is compiled with a serviceInit program (`sasjs/utils/serviceinitsas9|viya|server.sas`) that declares both macros in its own header, so a service body may call them without declaring them. Add a macro to that list only when it reaches a file the same way - injected by the compiler, through serviceInit, testSetUp or another implicit provider - never to silence a genuinely missing declaration.
`noBannedStrings` flags the literal words `error ` and `warning ` anywhere in a file, comments included. Reword prose; where the text has to stay (a real `%put`, a doxygen `@warning` tag), keep the log output identical with `%put %str(WARN)ING:` rather than a per-file override, and use `@sasjslint {"noBannedStrings": false}` only when the word itself is the content.
## No external assets
@@ -41,7 +43,7 @@ Data Controller must run entirely locally (offline / on-prem, no internet access
Agent-related content lives in `.agents/`: technical/agent-facing documentation goes in `.agents/docs/` (not `docs/`), and skills in `.agents/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agents/docs/`.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
Skills installed via `npx skills add` are tracked in `skills-lock.json` and live under `.agents/skills/`. Repo-local custom skills (e.g. `dc-sas` for Data Controller SAS development, `dc-estate-testing` for estate deploys and test runs, `handsontable`, `hyperformula`) are kept here too. Avoid editing npx-managed skill files directly — use `npx skills update` instead.
## Code comments and test names
+188
View File
@@ -1,3 +1,191 @@
# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)
### Bug Fixes
* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
### Features
* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
### Bug Fixes
* **ci:** install doxygen for the SAS docs job ([f58925b](https://git.datacontroller.io/dc/dc/commit/f58925b29956f1f753f0a367fdcd0afe7f3f51a0))
* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](https://git.datacontroller.io/dc/dc/commit/27686d3aad5e34ace59ca1b735a17cca1a4025e1))
* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](https://git.datacontroller.io/dc/dc/commit/888873421c3aeaf312d5ec45e3c805317c66976a))
* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](https://git.datacontroller.io/dc/dc/commit/dd482c4069202a0c6b9983129c19efc09bd160a7))
* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](https://git.datacontroller.io/dc/dc/commit/25ffa71adddba5e334f4a8c166f284d1be498fcd))
* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](https://git.datacontroller.io/dc/dc/commit/c5808a16eba0d29562fcc76f13ac4a33ab8da0b7))
* **ci:** rebuild and upload release assets on demand ([6486c80](https://git.datacontroller.io/dc/dc/commit/6486c80ed4f368bbf116c0ecdecc0e5e5f02075f))
* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](https://git.datacontroller.io/dc/dc/commit/af1813f829861df41c89b512c824e8d3ebac7a4b))
* **ci:** verify the forge certificate on the release API calls ([cf7b21a](https://git.datacontroller.io/dc/dc/commit/cf7b21a8b61ec3bf7ebe21968ccf13ade264bc64))
* **cypress:** hold download out too, and retry once in CI ([c97e886](https://git.datacontroller.io/dc/dc/commit/c97e8866db9ee404b2118d24baf53fc7058debef)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395) [#398](https://git.datacontroller.io/dc/dc/issues/398)
* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](https://git.datacontroller.io/dc/dc/commit/c803773d3c36476d8263f01fc3b8344aa3503d44)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395)
* **deploy:** do not offer the deploy wizard inside a report ([80e8911](https://git.datacontroller.io/dc/dc/commit/80e8911af8857fa2c417f028cd59b64a87233b9e))
* **deploy:** keep the app query when opening the standalone URL ([e51953c](https://git.datacontroller.io/dc/dc/commit/e51953ca8a6734e1649bad68cf4ad2057d677334))
* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](https://git.datacontroller.io/dc/dc/commit/459ff2da371375e996839d69deb40f681c356f1d)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](https://git.datacontroller.io/dc/dc/commit/f7e753863b61d8545bd0777d80267661b3c58670))
* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](https://git.datacontroller.io/dc/dc/commit/7e3045a21b67053b76a81bfe836e31571ccbffbf))
* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](https://git.datacontroller.io/dc/dc/commit/2623a3fb4ba3bf19c00636b1717b53d5a2d54f3a))
* **embed:** log a parse-time mapping problem once ([0011f7e](https://git.datacontroller.io/dc/dc/commit/0011f7e933d85be8e33debe3a6ecb9000edaabe3))
* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](https://git.datacontroller.io/dc/dc/commit/541a61ba08bc7256a911c50de7078859cc46c84e))
* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](https://git.datacontroller.io/dc/dc/commit/e9216b08956d5599662259a2b05cc1524c458df5))
* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](https://git.datacontroller.io/dc/dc/commit/bd3c220af2678a2a3f8394fe3084ee03c23d0b15))
* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](https://git.datacontroller.io/dc/dc/commit/f83dc359e258f5d1cb9fc49f0eaab9748995d823))
* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](https://git.datacontroller.io/dc/dc/commit/aa91daf9f78c00cea486d0f885f8e634366bd1a0))
* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](https://git.datacontroller.io/dc/dc/commit/10157c11595685efdc206fe7b16a5cafa098f26d))
* **embed:** warn when a va_parmap entry is ignored ([411f6c3](https://git.datacontroller.io/dc/dc/commit/411f6c3354b7fbfec11fef3b174ae1f8066c92ba))
* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](https://git.datacontroller.io/dc/dc/commit/c0ea810766ebfda5ae5358ab14146fbcc07fcc8e))
### Features
* **embed:** map VA parameters to DC columns from the URL ([15a1f34](https://git.datacontroller.io/dc/dc/commit/15a1f34d5263503cac83b4fe4e52482777dc2f1f))
* **embed:** resolve va_parmap values through named date anchors ([88010c3](https://git.datacontroller.io/dc/dc/commit/88010c3980c2ea1c39bbb024ed32d63e495e61f0))
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
### Bug Fixes
* address the review findings on the harness and refresh changes ([36e4e88](https://git.datacontroller.io/dc/dc/commit/36e4e88eb134d3b48217d6dd185ea3c03f03fba5))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](https://git.datacontroller.io/dc/dc/commit/698dea6d6b7a8635313f11acfcc7a4f39a02bc99))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](https://git.datacontroller.io/dc/dc/commit/0d58be975d8eb2d7f1ab6a648f760151392c7973))
* **client:** open the SAS log page with noopener ([072a133](https://git.datacontroller.io/dc/dc/commit/072a1337020970b5b4ff8aa31719cc5ffec7701e))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](https://git.datacontroller.io/dc/dc/commit/1fb4818d215f538a0ac51772ab1f261aeb91f7e1))
* **cypress:** apply the licence key when the estate has none ([be8f445](https://git.datacontroller.io/dc/dc/commit/be8f445ae1051e132fb6eb5a69b45264255326b5)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](https://git.datacontroller.io/dc/dc/commit/fed43cb07ef7f2afc685612a124dbc16d7618e21))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](https://git.datacontroller.io/dc/dc/commit/f5aade59760cc376b0f96eb4c892c8bca82dde14))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](https://git.datacontroller.io/dc/dc/commit/1e224763ba80500a112812a983801968e3751959))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](https://git.datacontroller.io/dc/dc/commit/9b3cf50b9c3889786605f0f5d9032d84c50f7107))
* **derived:** address five review findings ([ccd468b](https://git.datacontroller.io/dc/dc/commit/ccd468b3358c1418ee6d8e198b088a0cf23b8c24))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](https://git.datacontroller.io/dc/dc/commit/be50cb5d082e788c6ae106bc1d02e73a8c7b9646))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](https://git.datacontroller.io/dc/dc/commit/9914790777f92128c74ecce67e6c67ee48824b45))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](https://git.datacontroller.io/dc/dc/commit/9ef813d439b48e7fefd35c426e62b99e5943002a))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](https://git.datacontroller.io/dc/dc/commit/422f766cd43db4ce740e1890fd5df30790eaa7e3))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](https://git.datacontroller.io/dc/dc/commit/7830b7bcfa9806dde840ce8dd3e6a1887d0b38be))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](https://git.datacontroller.io/dc/dc/commit/2d5de9a324a07a36edc0a4f74c8fad09fde07cbf))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](https://git.datacontroller.io/dc/dc/commit/6a1128a46352f77c95dd02df9309f0984a1db2d1))
* **derived:** eight review findings ([561d7fc](https://git.datacontroller.io/dc/dc/commit/561d7fc3ed2a460835619962cf4dcb29d3dc87c7))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](https://git.datacontroller.io/dc/dc/commit/1863076bc956b4ff94bf7b85b3dde0d6e3b54d7a)), closes [#366](https://git.datacontroller.io/dc/dc/issues/366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](https://git.datacontroller.io/dc/dc/commit/d3dec682aafeb143fe983d6546337cb6490f5620))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](https://git.datacontroller.io/dc/dc/commit/469229aecbaad304a835a5f229df111c37ff7c0d))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](https://git.datacontroller.io/dc/dc/commit/7fb981daa91bce29c2d62c246185d01dcad08d6b))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](https://git.datacontroller.io/dc/dc/commit/5f5d8681b9d78280a5200309917415e135c3ce5b))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](https://git.datacontroller.io/dc/dc/commit/5f443b0b16a2abd3c11d96caa7a043911c29ef01))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](https://git.datacontroller.io/dc/dc/commit/6b5bf3e6be89b22e5a633c34b9aecbc8c7d48ee2))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](https://git.datacontroller.io/dc/dc/commit/85c89ca2a4e02112ba496ac36a222f33675544da))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](https://git.datacontroller.io/dc/dc/commit/f8f65681e1489f4ce94d9127d5e4c0c82798e960))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](https://git.datacontroller.io/dc/dc/commit/7efbf31ea5ac234bd8f924fdf89a95d101b3aff1))
* **derived:** only the delete marker is a reserved name ([af31c36](https://git.datacontroller.io/dc/dc/commit/af31c36b670ec63a2a4ba056bf6c4b28f6096e08))
* **derived:** open the statements file with a long record length ([ee31782](https://git.datacontroller.io/dc/dc/commit/ee31782fbff909b540162b1bdfca6f307208331b))
* **derived:** pair the filterds %else with %do/%end ([88a38db](https://git.datacontroller.io/dc/dc/commit/88a38db877dba9b98ccf50f0cc7242674654586b))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](https://git.datacontroller.io/dc/dc/commit/27ddcfb5759d6eb8f9880adc7c271a408e7741ad))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](https://git.datacontroller.io/dc/dc/commit/7799a300df82339683eea21e4a89ce7a6471c575))
* **derived:** reject a dependency on a row this submission deletes ([8806504](https://git.datacontroller.io/dc/dc/commit/88065049674c2cc0141eef22704b759f5b708b0f))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](https://git.datacontroller.io/dc/dc/commit/9a047ad42ec86df7d1eaf5058c0edde56016e88f))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](https://git.datacontroller.io/dc/dc/commit/e0da9f1098db63185538398ca478696cc849da4b))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](https://git.datacontroller.io/dc/dc/commit/688550a28d9794ac6b878926a0e1e2ef8f6e9335))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](https://git.datacontroller.io/dc/dc/commit/1b5c4b3ecd962a28e08fc5671b57dbf7ae0b8d9f))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](https://git.datacontroller.io/dc/dc/commit/4d9ad38df1fa89b0d5a12214d703d9c1fe5295f2))
* **editor:** a blank truncation note is not a warning ([f182179](https://git.datacontroller.io/dc/dc/commit/f18217940365f27ae388fae087e6126931e1222c))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](https://git.datacontroller.io/dc/dc/commit/04ce1861e7821063b1be42c22f92f62aed509602))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](https://git.datacontroller.io/dc/dc/commit/4bcee706ffaf1929bdd14bb2fb15111174ca3916))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](https://git.datacontroller.io/dc/dc/commit/45dce7b11b7a4399c1d0dc9c3e7a283b1cf59df1))
* **getdata:** clear the globals it declares before use ([359b570](https://git.datacontroller.io/dc/dc/commit/359b570b85aa2f66507e81ea2977243be4590f83))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](https://git.datacontroller.io/dc/dc/commit/059cdf176a7ec556b45ea1b21d880c048eba14ca))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](https://git.datacontroller.io/dc/dc/commit/661ee19cfa5f996bf01f2b11817039f38dbb9985))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](https://git.datacontroller.io/dc/dc/commit/355c46a7b64994d0146a88657dc5635593249138))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](https://git.datacontroller.io/dc/dc/commit/d626cb8f2daf94fde7ff21d924b9a2ca4968d983))
* **mocks:** address the audit-pack review findings ([489819b](https://git.datacontroller.io/dc/dc/commit/489819b05111323ae1e34edda22a5446cdbc2d87))
* **mocks:** carry the zip flags into the central directory ([c4021a4](https://git.datacontroller.io/dc/dc/commit/c4021a42ffaed2af80c53aaa1455249190b339b8))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](https://git.datacontroller.io/dc/dc/commit/62f55db170185900063f7a431cd8f375ebabf930))
* **mocks:** copy the physical column list before deriving ([f411b7d](https://git.datacontroller.io/dc/dc/commit/f411b7d269381e3f7cab2d85b6d8f7d7d50ea4d1))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](https://git.datacontroller.io/dc/dc/commit/60ec470404e5c279b58ac6a01768dfc83413b197))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](https://git.datacontroller.io/dc/dc/commit/74060c0585fb8b217cb181ea747ad318f5238868))
* **mocks:** drop derived variables before staging ([73d25b1](https://git.datacontroller.io/dc/dc/commit/73d25b1220d574ef2547fbd9e6e3cf1ceda375ee))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](https://git.datacontroller.io/dc/dc/commit/347dbb53b01c47bbd562de14d5ccbae7d96407c6))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](https://git.datacontroller.io/dc/dc/commit/ee3c8787efef9cd716bdd2bf80f6cb6751461db9))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](https://git.datacontroller.io/dc/dc/commit/6cad23fce9ac0cf64c247c40f3b7eac875fd4cfb))
* **mocks:** search the physical columns only in the viewer ([293022d](https://git.datacontroller.io/dc/dc/commit/293022d46b27e096fd514cdda49f688612df2f5b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](https://git.datacontroller.io/dc/dc/commit/fb36379520028baaf4bc3f08131bfc88deab3ac3))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](https://git.datacontroller.io/dc/dc/commit/9c294c0ab5ca8fe30a235553a298c1c68a61442f))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](https://git.datacontroller.io/dc/dc/commit/0a52d20c5fc27b4adde5fc8ed6f39a58c48f62ff))
* **mpe_execute:** fail fast when the child left no status ([a036386](https://git.datacontroller.io/dc/dc/commit/a03638667f1968c5909f496d954eeaa13f316763))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](https://git.datacontroller.io/dc/dc/commit/220408538b26016bda45e48d619cdbc3e1cbcc59))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](https://git.datacontroller.io/dc/dc/commit/f2b1428ec3099b2b6e3c6da6627530ef81fab201))
* name the startupservice launcher button "Show log" ([3445107](https://git.datacontroller.io/dc/dc/commit/3445107d9a91642b67e114bcfcd81c3ce47f3f0f))
* offer the launcher on the globvars failure modal too ([ad2da45](https://git.datacontroller.io/dc/dc/commit/ad2da453896a51f2cc7e49cebb6c84f7b47d7395))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](https://git.datacontroller.io/dc/dc/commit/46bacdd6d9ea814bcd02a1ff8aa3f447baf02f64))
* **refresh:** one audit folder per refresh call ([56b6620](https://git.datacontroller.io/dc/dc/commit/56b6620ef402b288ce78a2a77662b651e310753e))
* **sites:** round-trip the committed licence before publishing ([4d462d3](https://git.datacontroller.io/dc/dc/commit/4d462d310f7df23e5d5229cb0702721b7ad41bbd))
* **test:** getdata test ([da30266](https://git.datacontroller.io/dc/dc/commit/da30266278c4fe27a72810cd5ed647cf3bd823af))
* **tests:** enable closeouts logging ([fdcd8fa](https://git.datacontroller.io/dc/dc/commit/fdcd8fa4b3e7a7a119c662974d70dc9eea80bdae))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](https://git.datacontroller.io/dc/dc/commit/033ffd302502c7530fc44a314364f5cdb3be6ad2))
* **ui:** one error modal at a time ([81ccf75](https://git.datacontroller.io/dc/dc/commit/81ccf758e9dab0c7632ab50bf469e88b8c4c4420))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](https://git.datacontroller.io/dc/dc/commit/42355f17bc07bf4d44ba858984d83b9ed72cc0ba))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](https://git.datacontroller.io/dc/dc/commit/337ea8e5e021c6f2451fa3ad8396148744118fc9))
### Features
* **dc_getlibs:** dump the library list when debug is on ([582523d](https://git.datacontroller.io/dc/dc/commit/582523dc87a253b71824359224070ccb4746a386))
* **derived:** allow a variable to shadow a physical column ([12beeff](https://git.datacontroller.io/dc/dc/commit/12beeff69518376a042fbbc58332acc40d89d132))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](https://git.datacontroller.io/dc/dc/commit/813bce685691ad252043eeab31eb5460de81b756))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](https://git.datacontroller.io/dc/dc/commit/7c9ce9a712137e9825a5f2ff52796d765423147b))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](https://git.datacontroller.io/dc/dc/commit/87e794f769fe420f214f43b141f0b41cf88eefee))
* **derived:** report the dependency cycle path ([cee1a95](https://git.datacontroller.io/dc/dc/commit/cee1a954ee97e17f297b7cf43ef0e5cfc6e393a6))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](https://git.datacontroller.io/dc/dc/commit/78e88a41128b047c0276ccf647f128229152604c))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](https://git.datacontroller.io/dc/dc/commit/0b010f2c65ddada70d62ea83c706943c6c45756f))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](https://git.datacontroller.io/dc/dc/commit/aa6d50e1ea5fc2fe6be8aa9fa859b92d0a1061c5))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](https://git.datacontroller.io/dc/dc/commit/33d043ea92f007dca70c95d5363329d4447eeba9))
* **editor:** derived variables in the EDIT screen ([b0fd354](https://git.datacontroller.io/dc/dc/commit/b0fd354ecdbd7bf29196ed983ff3cce691affeed))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](https://git.datacontroller.io/dc/dc/commit/6513d09f059de15c5304f99b3f8d898d46ea1be7))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](https://git.datacontroller.io/dc/dc/commit/34691dceecb63f952b23ad8acc5bc93753c6cfee))
* **editor:** show the derived column label in the header tooltip ([cfe6647](https://git.datacontroller.io/dc/dc/commit/cfe6647ca870c45a3d8abc2753ce7f070e3fdf4a))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](https://git.datacontroller.io/dc/dc/commit/752d944c2091ef6d2624a15ab2e2e7be35608798))
* **mocks:** implement derived variables in the JS services ([b4fae88](https://git.datacontroller.io/dc/dc/commit/b4fae88ae13658607e4209449577cd42849d3a99))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](https://git.datacontroller.io/dc/dc/commit/5d183d135085ea023f32e5b1d4992d997394a8fb))
* run startupservice in a new window from its error modal ([40f6133](https://git.datacontroller.io/dc/dc/commit/40f61331aad6bc8a52c551b924f0d1f1e6426cee))
* **sites:** mint a licence for the published sites ([7d9df71](https://git.datacontroller.io/dc/dc/commit/7d9df71fdb0ab265e745ed3b708038983e855a87))
### Reverts
* **derived:** no shadowing, and no self reference ([5af8e6f](https://git.datacontroller.io/dc/dc/commit/5af8e6fc359221586af74a685159dfd75a40ec45))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](https://git.datacontroller.io/dc/dc/commit/31aa8591538409fc1a3ef97d7474120094ce202d))
# [7.18.0](https://git.datacontroller.io/dc/dc/compare/v7.17.0...v7.18.0) (2026-10-19)
### Bug Fixes
* **deps:** bump @sasjs/core to 5.4.0 ([19fb032](https://git.datacontroller.io/dc/dc/commit/19fb032f0dbdd612651ca4ff3b55d1aefa4850d5))
* **deps:** pin @sasjs/utils and drop the unused jsrsasign ([05d2d1e](https://git.datacontroller.io/dc/dc/commit/05d2d1e08e2c3082f0dd5aa4629d91907c0401fc))
* **editor:** own the delete-record column client-side ([56025cc](https://git.datacontroller.io/dc/dc/commit/56025cc0524d34d3589992e9f1059c13fb5581ed))
* **editor:** surface the reserved-column refusal on every getdata path ([b26f557](https://git.datacontroller.io/dc/dc/commit/b26f5571b66b65166c1158817f2df3027ae45cdc))
* **filters:** do not turn a fractional temporal value into a literal ([f710d87](https://git.datacontroller.io/dc/dc/commit/f710d871fffc0fa451fd8cc4d037a26414725654))
* **filters:** fix the dynamic where clause deref and map its temporal values ([ece67d3](https://git.datacontroller.io/dc/dc/commit/ece67d37ba2c4e2037dea626b9730a692388f025))
* **filters:** keep the range field readable and cover a BETWEEN in the rewrite ([be79f3a](https://git.datacontroller.io/dc/dc/commit/be79f3a1ea5e4c4a157396d276f2c6654f615d39))
* **filters:** map an IN entry to its unformatted value before the literal ([8f98ebe](https://git.datacontroller.io/dc/dc/commit/8f98ebe326978b8da58d8ac895ec946e562b5de4))
* **filters:** stop the temporal rewrite corrupting a typed value, and bind the range dropdown to the unformatted value ([a5f61d4](https://git.datacontroller.io/dc/dc/commit/a5f61d4a73051237b671695f67122c237770c95a))
* **filters:** use the unformatted value in every IN branch ([cc01f54](https://git.datacontroller.io/dc/dc/commit/cc01f540c29d3d5f7dd86943b6eed82b332ce868))
* **makedata:** shorten the DC_MAXCELLS_WEBEDIT description to fit var_desc ([a51d9e5](https://git.datacontroller.io/dc/dc/commit/a51d9e5574383e5d1e1943c95261fbce63e8788d))
* **multi-dataset:** surface the reserved-column guard, use the name constant ([2b8a2fa](https://git.datacontroller.io/dc/dc/commit/2b8a2fa1039da3ab888af8c3ffae7e1b848b250a))
* **release:** emit the release body as raw text, not a JSON string ([a05554e](https://git.datacontroller.io/dc/dc/commit/a05554e67a8d44e32e2d9f5e4a959a9d0592be5e))
### Features
* **filters:** show dates as date literals in the filter query ([662d000](https://git.datacontroller.io/dc/dc/commit/662d00032c27634bbc2d82c392e7875a2e174eff))
# [7.17.0](https://git.datacontroller.io/dc/dc/compare/v7.16.0...v7.17.0) (2026-10-15)
+1 -1
View File
@@ -8,7 +8,7 @@ This file is the shared glossary and orientation for the repo. When your output
- `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
- `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).
## Roles
+6 -1
View File
@@ -86,6 +86,7 @@ RUN_TIMES=js
NODE_PATH=node
CORS=enable
WHITELIST=http://localhost:4200
HTTP_PORT=5000
```
`RUN_TIMES=js` means the server executes `.js` files as Stored Programs (no SAS binary needed). `CORS=enable` plus `WHITELIST` allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
@@ -94,7 +95,7 @@ WHITELIST=http://localhost:4200
./api-linux # or api-macos / api-win.exe depending on your platform
```
The server runs in desktop mode (no auth) on `http://localhost:5000` by default.
`HTTP_PORT` is not optional in SASjs Server 3.x: `PORT` and `PROTOCOL` are deprecated, and a server started without `HTTP_PORT` or `HTTPS_PORT` exits with "no listener is enabled". With it set, the server runs in desktop mode (no auth) on `http://localhost:5000`.
### 2. Deploy the mock services and streaming frontend
@@ -133,6 +134,10 @@ There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`)
After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sasjs/`. After changing the frontend, rebuild (`cd client && npm run build`) then re-deploy so the streaming app picks up the new `client/dist/`. If you are using `ng serve` (step 3), only the mock services need re-deploying; the frontend hot-reloads automatically.
### Demo and staging sites
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs `makedata` on a throwaway SASjs Server first and ships that drive as `sasjsbuild/mock-snapshot.json`; without it the site opens with no data. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
## GUI Elements
For documentation on the Clarity Design System, including a list of components and example usage, see [our website](https://vmware.github.io/clarity).
+22
View File
@@ -1,3 +1,4 @@
import fs from 'fs'
import { defineConfig } from 'cypress'
export default defineConfig({
@@ -34,7 +35,28 @@ export default defineConfig({
e2e: {
video: true,
// One retry in a CI run, none interactively. `editor.cy.ts` test 28
// intermittently fails on "app-loading ... continuously found" - a slow load,
// not an assertion - and a single retry costs a few seconds where a red
// pipeline costs forty minutes. Failures that are real still fail twice, and
// the retried attempt's output is in the run log.
retries: { runMode: 1, openMode: 0 },
setupNodeEvents(on, config) {
// `download.cy.ts` clears the downloads folder before each test through
// `cy.task('deleteFolder')`. That task was registered in
// cypress/plugins/index.js, which Cypress 10+ never loads unless it is
// required from here - so the task had no handler and the spec failed on
// its first hook.
on('task', {
deleteFolder(folderName: string) {
return new Promise((resolve, reject) => {
fs.rm(folderName, { recursive: true, force: true }, (err) =>
err ? reject(err) : resolve(null)
)
})
}
})
// Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
// render deterministically regardless of the runner's system locale.
on('before:browser:launch', (browser, launchOptions) => {
@@ -40,8 +40,8 @@ context('special missings clip (DEMO_01)', function () {
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
// The mock estate carries a valid licence key (mock-storage/licence.json),
// so the app activates directly with no free-tier banner in shot.
// Assumes the estate is already activated (the licensing spec registers a
// valid key); otherwise a licensing page would sit in shot.
cy.get('.nav-tree', { timeout: longerCommandTimeout }).should('exist')
})
@@ -25,11 +25,13 @@ const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
// 2. Ordinary columns (e.g. SOME_DROPDOWN) still get their rule via the
// new per-row cols[].COLTYPE path — test 2 proves the migration didn't
// silently drop real columns' rules either.
// Column positions below come from COLHEADERS in
// sas/mocks/sasjs/services/editors/getdata.js:
// "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,..."
// — childNodes[0] on a body <tr> is the row-header <th>, so childNodes[N+1]
// is the Nth (0-indexed) column in that list.
// Column positions below come from the mock's COLHEADERS
// (sas/mocks/sasjs/services/editors/getdata.js):
// "PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,..."
// — the client-owned delete column is prepended by the editor itself (see
// deleteRecordColumn.ts), so on screen it is still column 0. childNodes[0] on
// a body <tr> is the row-header <th>, so childNodes[N+1] is the Nth
// (0-indexed) column in that list.
context('coltype vertical-array regression tests: ', function () {
this.beforeAll(() => {
cy.visit(`${hostUrl}/SASLogon/logout`)
+249
View File
@@ -0,0 +1,249 @@
export {}
//
// Derived (computed) variables are configured per table in MPE_DERIVED and
// created at read time, ahead of the filter, so that they can be filtered on.
//
// TESTDATA.DERIVED_DEMO carries two of them, neither of which is a physical
// column of the table:
// REGION ifc(STATE in ('CA','OR','WA'),'West','East')
// LINE_VALUE QTY * PRICE
//
// The mock implements the derivation in getdata.js / getcolvals.js (see
// mpeDerived in dcMockUtils.js), so the EDIT screen can be exercised without a
// SAS server. The real evaluation is done by SAS.
const hostUrl = Cypress.env('hosturl')
const appLocation = Cypress.env('appLocation')
const longerCommandTimeout = Cypress.env('longerCommandTimeout')
context('derived variables: ', function () {
this.beforeAll(() => {
cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
cy.loginAndUpdateValidKey(true)
})
this.beforeEach(() => {
cy.viewport(1440, 900)
bootApp()
})
it('1 | derived columns are shown in the EDIT grid with computed values', () => {
openEditor('TESTDATA.DERIVED_DEMO')
getCellByHeaderAndRow(0, 'REGION').should('have.text', 'West')
getCellByHeaderAndRow(1, 'REGION').should('have.text', 'East')
getCellByHeaderAndRow(0, 'LINE_VALUE').should('have.text', '21')
getCellByHeaderAndRow(1, 'LINE_VALUE').should('have.text', '99')
// REGION_LABEL is built on REGION, so it only resolves if the derived
// statements run in VARIABLE_ORDER sequence
getCellByHeaderAndRow(0, 'REGION_LABEL').should('have.text', 'West-CA')
getCellByHeaderAndRow(1, 'REGION_LABEL').should('have.text', 'East-NY')
cy.screenshot('derived-1-grid', { capture: 'viewport' })
})
it('2 | filters on a character derived column', () => {
openEditor('TESTDATA.DERIVED_DEMO')
openFilterPopup()
setFilterVariable('REGION')
setFilterValue('West')
submitFilter()
cy.get('.infoBar-text').should('contain.text', "REGION = 'West'")
cy.get('.ht_master .htCore tbody tr', {
timeout: longerCommandTimeout
}).should('have.length', 2)
getCellByHeaderAndRow(1, 'REGION').should('have.text', 'West')
cy.screenshot('derived-2-filter-region', { capture: 'viewport' })
})
it('3 | filters on a numeric derived column', () => {
openEditor('TESTDATA.DERIVED_DEMO')
openFilterPopup()
setFilterVariable('LINE_VALUE')
setFilterOperator('>')
setFilterValue('50')
submitFilter()
cy.get('.ht_master .htCore tbody tr', {
timeout: longerCommandTimeout
}).should('have.length', 1)
getCellByHeaderAndRow(0, 'LINE_VALUE').should('have.text', '99')
cy.screenshot('derived-3-filter-value', { capture: 'viewport' })
})
it('4 | a derived column is marked read-only, and opens no editor', () => {
openEditor('TESTDATA.DERIVED_DEMO')
// a derived column is read-only AND computed, so the grid marks it apart
// from an ordinary read-only column - its own header/cell class, no
// readonlyCell dimming (editor.component.ts, afterGetColHeader /
// afterGetCellMeta). STATE is the control: same grid, a physical column.
headerCell('REGION').should('have.class', 'dc-derived-header')
headerCell('REGION').should('not.have.class', 'readonlyCell')
headerCell('STATE')
.should('not.have.class', 'dc-derived-header')
.and('not.have.class', 'readonlyCell')
getCellByHeaderAndRow(0, 'REGION').should('have.class', 'dc-derived-cell')
getCellByHeaderAndRow(0, 'STATE').should(
'not.have.class',
'dc-derived-cell'
)
// The computed block opens at whichever derived column comes first in the
// grid - MPE_DERIVED order, so LINE_VALUE here, not REGION. Assert the
// invariant rather than a name: exactly one derived header carries the
// marker, and it is the first one.
cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
const derived = [...$ths].filter((th) =>
th.classList.contains('dc-derived-header')
)
expect(derived.length, 'derived headers').to.be.greaterThan(0)
expect(
derived[0].classList.contains('dc-derived-first'),
'the first derived header opens the block'
).to.be.true
derived
.slice(1)
.forEach(
(th) =>
expect(
th.classList.contains('dc-derived-first'),
'only the first derived header carries the marker'
).to.be.false
)
})
// the hover text carries the label, which is what says what the column is:
// the configured VAR_LABEL where there is one, otherwise the generated
// name=expression
headerCell('REGION_LABEL')
.should('have.attr', 'title')
.and('contain', 'Region and state')
headerCell('REGION')
.should('have.attr', 'title')
.and('contain', 'REGION=ifc')
enterEditMode()
getCellByHeaderAndRow(0, 'REGION').dblclick({ force: true })
cy.get('.handsontableInput').should('not.exist')
getCellByHeaderAndRow(0, 'REGION').should('have.text', 'West')
})
/**
* The positive control for the case above: same screen, same edit mode, a
* physical column - so its silence is about the column, not the screen.
*/
it('5 | a physical column in the same grid does accept an edit', () => {
openEditor('TESTDATA.DERIVED_DEMO')
enterEditMode()
getCellByHeaderAndRow(0, 'STATE').dblclick({ force: true })
cy.get('.handsontableInput').should('exist')
})
})
/**
* Cypress clears cookies between tests, which drops the SASjs Server session
* and lands the app back on the evaluation agreement card. Accept it when it is
* shown - a no-op when the session survived.
*/
const bootApp = () => {
cy.visit(hostUrl, { timeout: longerCommandTimeout })
cy.get('body').then(($body: any) => {
if ($body.find('#TCS input[type="checkbox"]').length) {
cy.get('#TCS input[type="checkbox"]').check({ force: true })
cy.wait(4000)
}
})
}
/** Opens a table straight in the EDIT screen and waits for the grid. */
const openEditor = (libMem: string) => {
cy.visit(`${hostUrl}${appLocation}/#/editor/${libMem}`, {
timeout: longerCommandTimeout
})
cy.get('.ht_clone_top .htCore thead tr th', {
timeout: longerCommandTimeout
}).should('exist')
cy.wait(2500)
}
/** The EDIT screen opens read-only; this is the button that starts an edit. */
const enterEditMode = () => {
cy.get('.btnCtrl button.btn-primary', { timeout: longerCommandTimeout })
.click()
.then(() => cy.wait(2500))
}
/** The grid header cell for a column, by its rendered text. */
const headerCell = (headerText: string) =>
cy.get('.ht_clone_top .htCore thead tr th').then(($ths) => {
const th = [...$ths].find(
(candidate) => (candidate as HTMLElement).innerText.trim() === headerText
)
if (!th) throw new Error(`header not found: ${headerText}`)
return cy.wrap(th)
})
const openFilterPopup = () => {
cy.contains('button.btnView', 'Filter', {
timeout: longerCommandTimeout
}).click()
cy.get('.filter-modal', { timeout: longerCommandTimeout }).should(
'be.visible'
)
}
/**
* The soft-select inputs drop a transparent .overlay click-catcher over the
* modal while their suggestion list is open, so they need force.
*/
const setFilterVariable = (column: string) => {
cy.get('#vals_var_id0_0').clear({ force: true }).type(column, { force: true })
cy.get('#datalist_vals_var_id0_0 option').contains(column).click({
force: true
})
cy.get('#vals_var_id0_0').trigger('keyup', { key: 'Escape', force: true })
cy.wait(400)
}
const setFilterValue = (value: string) => {
cy.get('#vals_0_0').clear({ force: true }).type(value, { force: true })
cy.get('#vals_0_0').trigger('keyup', { key: 'Escape', force: true })
cy.wait(400)
}
const setFilterOperator = (operator: string) => {
cy.get('.filter-modal .operator-col select').first().select(operator)
cy.wait(500)
}
const submitFilter = () => {
cy.contains('.filter-modal button', 'Ok').click()
cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
cy.wait(2500)
}
const getCellByHeaderAndRow = (rowIndex: number, headerText: string) => {
return cy
.get('.ht_clone_top .htCore thead tr th')
.should(($ths) => {
const texts = [...$ths].map((th) => (th as HTMLElement).innerText.trim())
expect(texts).to.include(headerText)
})
.then(($ths) => {
const index = [...$ths].findIndex(
(th) => (th as HTMLElement).innerText.trim() === headerText
)
return cy
.get('.ht_master tbody tr')
.then((rows: any) => rows[rowIndex].childNodes[index])
.then((cell) => cy.get(cell))
})
}
+97 -27
View File
@@ -17,9 +17,20 @@ context('download files test: ', function () {
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
cy.get('input.username').type(username)
cy.get('input.password').type(password)
cy.get('.login-group button').click()
// beforeAll already logged in. Typing into the login form unconditionally
// failed with "the center of this element is hidden from view" whenever the
// session was live and the form was not on screen, so only fill it when it
// is actually shown.
cy.get('body').then(($body) => {
const usernameInput = $body.find('input.username')[0]
if (usernameInput && !Cypress.dom.isHidden(usernameInput)) {
cy.get('input.username').type(username)
cy.get('input.password').type(password)
cy.get('.login-group button').click()
}
})
visitPage('home')
})
@@ -29,18 +40,37 @@ context('download files test: ', function () {
})
it('1 | downloads audit file', (done) => {
visitPage('approve/toapprove')
/* The approve grid renders a download button per table that is waiting for
approval, and a freshly seeded estate has none, so create one: upload a
fixture to the test table and submit it from the stage screen. The button
carries the table id, which the stage URL also carries, so the assertion
names the row this test created rather than whichever row comes first.
The screen is at review/approve; approve/toapprove is a route from before
the review module was namespaced and renders the 404 page. */
openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
cy.get('.app-loading', { timeout: longerCommandTimeout })
.should('not.exist')
.then(() => {
cy.get('.btn.btn-success')
.should('be.visible')
.then((buttons) => {
buttons[0].click()
const id = buttons[0].id
attachExcelFile('regular_excel.xlsx')
checkForFileDownloaded(id, 'zip', () => done())
submitExcel()
cy.url({ timeout: longerCommandTimeout })
.should('include', '/stage/')
.then((url) => {
const tableId = url.split('/stage/')[1].split(/[?#]/)[0]
visitPage('review/approve')
cy.get('.app-loading', { timeout: longerCommandTimeout })
.should('not.exist')
.then(() => {
cy.get(`[aria-label="Download audit file"][id="${tableId}"]`, {
timeout: longerCommandTimeout
})
.should('have.length', 1)
.first()
.click()
checkForFileDownloaded(tableId, 'zip', () => done())
})
})
})
@@ -72,7 +102,7 @@ context('download files test: ', function () {
openDownloadModal(() => {
cy.get('select')
.select('Excel')
.select('EXCEL')
.then(() => {
cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
button.trigger('click')
@@ -92,7 +122,7 @@ context('download files test: ', function () {
openDownloadModal(() => {
cy.get('select')
.select('SAS Datalines')
.select('SAS')
.then(() => {
cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
button.trigger('click')
@@ -112,7 +142,7 @@ context('download files test: ', function () {
openDownloadModal(() => {
cy.get('select')
.select('SAS DDL')
.select('SAS_DDL')
.then(() => {
cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
button.trigger('click')
@@ -132,7 +162,7 @@ context('download files test: ', function () {
openDownloadModal(() => {
cy.get('select')
.select('TSQL DDL')
.select('TSQL_DDL')
.then(() => {
cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
button.trigger('click')
@@ -152,7 +182,7 @@ context('download files test: ', function () {
openDownloadModal(() => {
cy.get('select')
.select('PGSQL DDL')
.select('PGSQL_DDL')
.then(() => {
cy.get('.btn.btn-sm.btn-success-outline').then((button) => {
button.trigger('click')
@@ -198,18 +228,22 @@ const checkForFileDownloaded = (
}
const openDownloadModal = (callback?: any) => {
cy.get('.btn.btn-sm.btn-outline.filterSide.dropdown-toggle')
/* filterSide is the viewer's options dropdown; it no longer carries a
dropdown-toggle class - see filter-panel.cy.ts and row-cell-limits.cy.ts.
Its entries are div[clrDropdownItem], not buttons, so select Download by its
text the way filter-panel.cy.ts selects Filter. */
cy.get('.btn.btn-sm.btn-outline.filterSide', {
timeout: longerCommandTimeout
})
.click()
.then(() => {
cy.get('clr-dropdown-menu button').then((buttons) => {
for (let button of buttons) {
if (button.innerText.toLowerCase().includes('download')) {
button.click()
/* the click handler is on the clrDropdownItem element, not on the span
holding the text */
cy.contains('clr-dropdown-menu [clrDropdownItem]', 'Download', {
timeout: longerCommandTimeout
}).click()
if (callback) callback()
}
}
})
if (callback) callback()
})
}
@@ -248,3 +282,39 @@ const openTableFromTree = (libNameIncludes: string, tablename: string) => {
})
})
}
const fixturePath = 'excels'
/**
* Uploads a fixture to the open table. attachFile drives the editor's file
* input directly, so no click on the hidden control is needed and no positional
* selector is involved.
*/
const attachExcelFile = (excelFilename: string) => {
cy.get('input[type="file"]#file-upload')
.should('have.length', 1)
.attachFile(`/${fixturePath}/${excelFilename}`)
/* an upload raises a confirmation modal; a failure raises the abort modal
instead. Ask the body which one is there rather than waiting on a modal that
may never appear, and fail loudly on the abort rather than clicking through
the overlay. */
cy.get('body').then(($body) => {
const abortModal = $body.find('.clr-abort-modal').length > 0
if (abortModal) {
cy.get('.clr-abort-modal .modal-title')
.invoke('text')
.should('not.include', 'Abort Message')
return
}
cy.get('.modal-footer .btn.btn-primary').click()
})
}
const submitExcel = () => {
cy.get('.buttonBar button.preview-submit', {
timeout: longerCommandTimeout
}).click()
}
+437 -17
View File
@@ -6,44 +6,90 @@ const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
// SAS Visual Analytics data-driven content embed mode (`?embed=va`):
// - chrome is hidden (same as embed=true),
// - the editor opens in edit mode immediately (not read-only),
// - the only action button is a single Submit below the grid (stubbed in v1),
// - the top button bar and Add Record are hidden.
// - the editor opens READ-ONLY (the user clicks Edit, as in normal use),
// - the Filter button is hidden - VA filters replace the filter builder,
// - the Add Row / Add Record buttons are hidden.
//
// The detailed PK+label data-merge logic is covered by the VaMessagingService
// unit spec; here we only assert the deterministic UI layout and that pushing a
// VA postMessage does not break the page.
// unit spec; here we assert the deterministic UI layout, that pushing a VA
// postMessage does not break the page, and (case 3) that a `va_parmap` URL
// mapping filters the column it names end to end.
/**
* Assert on the row count the editor shows. The assertion runs inside a Cy
* `should`, so it retries until the grid has loaded: reading the text once would
* capture the count from before a filter is applied.
*/
const parseRows = (text: string): number => {
const match = /\(([\d,]+) rows?/.exec(text)
return match ? Number(match[1].replace(/,/g, '')) : NaN
}
const expectRows = (check: (rows: number) => void) =>
cy.get('h4', { timeout: longerCommandTimeout }).should(($h4) => {
check(parseRows($h4.text()))
})
/** Assert the grid shrank, without pinning a fixture-specific number. */
const expectFewerRowsThan = (baseline: number) =>
expectRows((rows) => {
expect(rows, 'filtered row count').to.be.lessThan(baseline)
})
/** Read the unfiltered row count from the app itself, retrying until it loads. */
const readRowCount = (): Cypress.Chainable<number> =>
expectRows((rows) => {
expect(rows, 'row count').to.be.greaterThan(0)
})
.get('h4')
.invoke('text')
.then(parseRows)
context('embed=va (VA data-driven content) tests: ', function () {
// How many rows the table has unfiltered. Captured once, from the app itself.
let unfilteredRows = 0
this.beforeAll(() => {
cy.visit(`${hostUrl}/SASLogon/logout`)
cy.loginAndUpdateValidKey()
// Baseline for the relative row-count assertions: open the table with no VA
// message and record what the editor itself reports.
cy.visit(`${hostUrl}${appLocation}/#/editor/DC_JSLIB.MPE_X_TEST`, {
timeout: 60000
})
readRowCount().then((count) => {
unfilteredRows = count
})
})
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
// A cy.visit that differs from the current URL only in the hash is a
// same-document navigation, so the app is not re-bootstrapped. Reload so
// each test starts from a fresh document (the previous test may have left
// ?embed=va in the URL, which the app reads once at init).
cy.reload()
visitPage('home')
})
it('1 | opens the editor in VA mode: chrome hidden, editable, single bottom Submit', () => {
it('1 | opens the editor in VA mode: chrome hidden, read-only, no Filter button', () => {
openEditorInVaMode('mpe_x_test', () => {
// Chrome hidden (same as embed=true)
cy.get('header.app-header').should('not.exist')
// Grid present and in edit mode (read-only Filter/Edit/Upload bar absent)
// Grid present, read-only: the user enters edit mode with Edit, as usual
cy.get('#hotTable', { timeout: longerCommandTimeout }).should('exist')
cy.get('.btnCtrl button').contains('Edit').should('not.exist')
cy.get('.btnCtrl button').contains('Edit').should('exist')
// The filter builder is replaced by VA filters
cy.get('.btnCtrl button').contains('Filter').should('not.exist')
// Top edit-mode buttons hidden in VA mode
// Not in edit mode, so no edit-mode controls
cy.get('.btnCtrl button.btn-outline-danger').should('not.exist') // Cancel
cy.get('.btnCtrl').contains('Add Row').should('not.exist')
cy.get('.btnCtrl').contains('Submit').should('not.exist')
// Bottom Add Record hidden
// Bottom Add Record hidden in VA mode
cy.contains('button', 'Add Record').should('not.exist')
// Exactly one Submit button, below the grid, disabled in v1
cy.get('button').filter(':contains("Submit")').should('have.length', 1)
cy.get('button').contains('Submit').should('be.disabled')
})
})
@@ -69,14 +115,382 @@ context('embed=va (VA data-driven content) tests: ', function () {
// Editor still loads in VA mode after the message (chrome hidden, grid up)
cy.get('#hotTable', { timeout: longerCommandTimeout }).should('exist')
cy.get('header.app-header').should('not.exist')
cy.get('button').contains('Submit').should('exist')
cy.get('.btnCtrl button').contains('Edit').should('exist')
})
})
it('3 | va_parmap filters a column from a parameter that is not its name (tab 1)', () => {
// Two DC instances over one table (one per report tab), each filtering the
// same DC column from its own report parameter. Without the URL mapping the
// parameter label is not a column name, so nothing would filter at all. The
// mapping here is written in lower case against an upper-case label and
// column, which pins the case-insensitivity of both sides end to end.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
// The applied filter names the MAPPED column, and the grid shrinks to
// the rows matching that value (Option 2 occurs twice in this table).
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
'SOME_DROPDOWN'
)
cy.get('.infoBar-text').should('contain.text', "'Option 2'")
expectFewerRowsThan(unfilteredRows)
},
'va_parmap=dropdown_tab1-some_dropdown'
)
})
it('4 | the other tab maps its own parameter to the same column', () => {
// Same table, same column, a different parameter label and value (Option 1
// occurs four times) - the per-instance URL mapping is what keeps the two
// tabs independent.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB2',
dataType: 'string',
value: 'Option 1'
})
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
'SOME_DROPDOWN'
)
cy.get('.infoBar-text').should('contain.text', "'Option 1'")
expectFewerRowsThan(unfilteredRows)
},
'va_parmap=DROPDOWN_TAB2-SOME_DROPDOWN'
)
})
it('5 | warns in the grid when a mapping names a column the table lacks', () => {
// Written by hand in the report's URL, so a typo must be visible to the
// report author: the parameter is skipped, and the reason is shown rather
// than leaving an unfiltered grid that looks like a working filter.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'VA parameter mapping problem'
)
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
cy.get('.va-parmap-warning').should('contain.text', 'NOT_A_COLUMN')
// VA mode hides the app header, so the banner must paint inside the
// viewport on its own - assert the real box, not just its presence.
cy.get('.va-parmap-warning').should(($alert) => {
const rect = $alert[0].getBoundingClientRect()
expect(rect.width, 'banner width').to.be.greaterThan(100)
expect(rect.height, 'banner height').to.be.greaterThan(20)
expect(rect.top, 'banner top').to.be.at.least(0)
expect(rect.bottom, 'banner bottom').to.be.at.most(
Cypress.config('viewportHeight')
)
})
cy.screenshot('va-parmap-banner', { capture: 'viewport' })
// Nothing was filtered: the row count matches the unfiltered baseline.
expectRows((rows) => expect(rows).to.equal(unfilteredRows))
},
'va_parmap=DROPDOWN_TAB1-NOT_A_COLUMN'
)
})
it('6 | warns straight away when a mapping entry cannot be parsed', () => {
// No separator, so the entry never becomes a mapping - reported before any
// VA message arrives.
openEditorInVaMode(
'mpe_x_test',
() => {
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'it needs a "-" between the VA parameter and the column name'
)
},
'va_parmap=DROPDOWN_TAB1'
)
})
it('7 | warns when a mapping names a parameter VA is not sending', () => {
// Parameter-side typo: the mapping never applies, so without the banner this
// is indistinguishable from a working filter over every row.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
// One message only tells us this object's parameter set is still being
// discovered; the warning waits for a message that introduces no new name.
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'not.exist'
)
pushVaMessage({
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'names VA parameter "dropdown_tab_1", which VA is not sending'
)
expectRows((rows) => expect(rows).to.equal(unfilteredRows))
},
'va_parmap=DROPDOWN_TAB_1-SOME_DROPDOWN'
)
})
it('8 | applies a mapping from the buffered message on the very first load', () => {
// VA posts once on iframe load, before the app bootstraps, so the first
// getdata is already filtered from that buffered message - with no column
// metadata available, the mapping is what names the column.
visitWithBufferedMessage(
'va_parmap=DROPDOWN_TAB1-SOME_DROPDOWN',
{
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
},
() => {
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
"SOME_DROPDOWN = 'Option 2'"
)
expectFewerRowsThan(unfilteredRows)
cy.url().should('match', /\/editor\/[^/]+\/\d+/)
}
)
})
it('9 | a bad mapping on the first load still ends at the banner', () => {
// The deferred first load cannot check the column against metadata, so its
// saveQuery may fail and fall back to an unfiltered load - the banner is
// what stops that reading as a working filter.
visitWithBufferedMessage(
'va_parmap=DROPDOWN_TAB1-NOT_A_COLUMN',
{
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 2'
},
() => {
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'NOT_A_COLUMN'
)
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
// Deliberately no row-count assertion: this path cannot validate the
// column (no metadata yet), so the outcome depends on the platform - on
// SAS, mp_filtercheck aborts the save and the load falls back to
// unfiltered; the JS mock stores the filter and returns 0 rows. The
// banner is the part that must hold either way, because both outcomes
// are otherwise indistinguishable from a working filter.
}
)
})
it('10 | warns when two parameters filter the same column (an empty grid)', () => {
// Clauses are AND-ed, so two different values on one column produce no rows
// at all - the last silent-empty-grid path. Reported rather than rewritten,
// so the author decides whether to use one parameter carrying both values.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage(
{
name: 'pr1',
label: 'DROPDOWN_TAB1',
dataType: 'string',
value: 'Option 1'
},
{
name: 'pr2',
label: 'DROPDOWN_TAB2',
dataType: 'string',
value: 'Option 2'
}
)
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'combined with AND'
)
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB1')
cy.get('.va-parmap-warning').should('contain.text', 'DROPDOWN_TAB2')
// The consequence: rows must match both clauses, and the two values
// differ, so none can match.
expectRows((rows) => expect(rows, 'matching rows').to.equal(0))
},
'va_parmap=DROPDOWN_TAB1-SOME_DROPDOWN&va_parmap=DROPDOWN_TAB2-SOME_DROPDOWN'
)
})
it('12 | a transform adds the previous quarter to the filter', () => {
// ADD(QUARTER_START,-1) is the reported requirement: one clause carrying the
// selected date and the same point one quarter earlier. 2026-10-01 = 24380,
// 2026-07-01 = 24288 in SAS days; nothing in the fixture matches either, so
// the assertion is on the clause rather than the row count.
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'SELECTED_DATE',
dataType: 'string',
value: '01OCT2026'
})
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
'SOME_DATE IN ('
)
cy.get('.infoBar-text').should('contain.text', '24380')
cy.get('.infoBar-text').should('contain.text', '24288')
cy.get('h4').should('contain.text', '(0 rows')
},
'va_parmap=SELECTED_DATE-SOME_DATE:ADD(QUARTER_START,-1)'
)
})
it('13 | warns when a transform lands on a column it cannot apply to', () => {
openEditorInVaMode(
'mpe_x_test',
() => {
pushVaMessage({
name: 'pr1',
label: 'SELECTED_DATE',
dataType: 'string',
value: 'x'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'is not a date or datetime column'
)
cy.get('h4').should('contain.text', '(7 rows')
},
'va_parmap=SELECTED_DATE-SOME_CHAR:ADD(QUARTER_START,-1)'
)
})
it('11 | logs a parse-time problem once, however many messages arrive', () => {
// The parse problem is logged at app init; the editor's own loop must not log
// it again. The second entry is valid-but-missing, so the loop definitely runs
// and logs its own line - which is what makes this a real assertion.
cy.visit(
`${hostUrl}${appLocation}/index.html#/editor/DC_JSLIB.MPE_X_TEST?embed=va&va_parmap=DROPDOWN_TAB1&va_parmap=DROPDOWN_TAB2-NOT_A_COLUMN`,
{
onBeforeLoad(win: any) {
win.__warns = []
const original = win.console.warn.bind(win.console)
win.console.warn = (...args: any[]) => {
win.__warns.push(args.join(' '))
original(...args)
}
}
}
)
cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
'not.exist'
)
pushVaMessage({
name: 'pr2',
label: 'DROPDOWN_TAB2',
dataType: 'string',
value: 'Option 2'
})
cy.get('.va-parmap-warning', { timeout: longerCommandTimeout }).should(
'contain.text',
'NOT_A_COLUMN'
)
cy.window().then((win: any) => {
const warns: string[] = win.__warns || []
const parseLines = warns.filter((w) =>
w.includes('va_parmap "DROPDOWN_TAB1"')
)
const runtimeLines = warns.filter((w) => w.includes('NOT_A_COLUMN'))
expect(parseLines.length, warns.join(' | ')).to.eq(1)
expect(runtimeLines.length, warns.join(' | ')).to.eq(1)
})
})
})
/**
* Loads the editor with a VA message already in the pre-bootstrap buffer, the
* way VA's own postMessage arrives before Angular starts. A distinct pathname is
* used so this is a real document load: a cy.visit that differs only in the hash
* is a same-document navigation and would not re-bootstrap the app.
*/
const visitWithBufferedMessage = (
extraParams: string,
parameter: any,
callback?: any
) => {
cy.visit(
`${hostUrl}${appLocation}/index.html#/editor/DC_JSLIB.MPE_X_TEST?embed=va&${extraParams}`,
{
onBeforeLoad(win: any) {
win.__vaLastMessage = {
t: Date.now(),
origin: win.location.origin,
data: {
version: '1',
resultName: 'dd1',
rowCount: 0,
availableRowCount: 0,
data: [],
columns: [],
parameters: [parameter]
}
}
}
}
)
cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
if (callback) callback()
}
/** Pushes one VA data-driven content message carrying the given parameters. */
const pushVaMessage = (...parameters: any[]) => {
cy.window().then((win) => {
win.postMessage(
{
version: '1',
resultName: 'dd1',
rowCount: 0,
availableRowCount: 0,
data: [],
columns: [],
parameters
},
'*'
)
})
}
// Opens a table from the tree (which routes to #/editor/LIB.TABLE), then
// re-visits the same route with ?embed=va so the app parses VA embed mode.
const openEditorInVaMode = (tablename: string, callback?: any) => {
// `extraParams` is appended to that query string (e.g. va_parmap entries).
const openEditorInVaMode = (
tablename: string,
callback?: any,
extraParams?: string
) => {
openTableFromTree(libraryToOpenIncludes, tablename)
cy.get('#hotTable', { timeout: longerCommandTimeout })
@@ -84,7 +498,13 @@ const openEditorInVaMode = (tablename: string, callback?: any) => {
.then(() => {
cy.url().then((url) => {
const separator = url.includes('?') ? '&' : '?'
cy.visit(`${url}${separator}embed=va`)
const extra = extraParams ? `&${extraParams}` : ''
cy.visit(`${url}${separator}embed=va${extra}`)
// A cy.visit that differs from the current URL only in the hash is a
// same-document navigation, so the app is not re-bootstrapped and the
// embed parameter (read once at app init) never takes effect. Reload to
// bootstrap it, as the VA iframe load does.
cy.reload()
cy.get('.app-loading', { timeout: longerCommandTimeout }).should(
'not.exist'
)
+33
View File
@@ -399,6 +399,26 @@ context('excel tests: ', function () {
})
})
it('25 | Keeps decimals when the upload preview becomes a manual edit', () => {
openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
// Entering manual edit mode discards the source file but keeps the values
// parsed from it. SOME_NUM holds 0.901045951946194 in the first data row
// of the fixture, so a truncated or zeroed cell fails the assertion.
attachExcelFile('regular_excel.xlsx', () => {
getFirstRowCells().then((previewCells) => {
expect(previewCells).to.include(decimalSomeNum)
cy.get('.btn-upload-preview.btn-primary').click()
cy.get('.modal-footer .btn-success-outline').click()
getFirstRowCells().then((editCells) => {
expect(editCells).to.include(decimalSomeNum)
})
})
})
})
// Large files break Cypress
// it ('? | Uploads Excel with size of 5MB', (done) => {
@@ -476,6 +496,19 @@ const attachExcelFile = (excelFilename: string, callback?: any) => {
})
}
const decimalSomeNum = '0.901045951946194'
// The row's cell texts, without depending on column positions: the preview
// grid renders row-status and delete cells in an order that varies.
const getFirstRowCells = () =>
cy
.get('#hotTable', { timeout: longerCommandTimeout })
.find('div.ht_master.handsontable tbody tr')
.eq(0)
.then(($row) =>
Array.from($row[0].children).map((cell: any) => cell.innerText.trim())
)
const submitExcel = (callback?: any) => {
cy.get('.buttonBar button.preview-submit', { timeout: longerCommandTimeout })
.click()
+144 -19
View File
@@ -11,6 +11,18 @@ context('filtering tests: ', function () {
this.beforeAll(() => {
cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
cy.loginAndUpdateValidKey()
/**
* The evaluation agreement is per-origin localStorage, so a fresh browser
* profile lands on it and the nav tree never renders. Guarded, so it is a
* no-op once an earlier spec in the run has accepted it.
*/
cy.get('body').then(($body) => {
if ($body.find('#TCS input[type="checkbox"]').length) {
cy.get('#TCS input[type="checkbox"]').check({ force: true })
cy.wait(4000)
}
})
})
this.beforeEach(() => {
@@ -51,9 +63,13 @@ context('filtering tests: ', function () {
openFilterPopup(() => {
setFilterWithValue('SOME_TIME', '00:00:42', 'time', () => {
checkInfoBarIncludes(`(( SOME_TIME = 42 ))`, (includes: boolean) => {
if (includes) done()
})
checkInfoBarIncludes(
`(( SOME_TIME = '00:00:42't ))`,
(includes: boolean) => {
if (!includes) return
assertGridRowCount(3, () => done())
}
)
})
})
})
@@ -63,9 +79,13 @@ context('filtering tests: ', function () {
openFilterPopup(() => {
setFilterWithValue('SOME_TIME', '42', 'value', () => {
checkInfoBarIncludes(`(( SOME_TIME = 42 ))`, (includes: boolean) => {
if (includes) done()
})
checkInfoBarIncludes(
`(( SOME_TIME = '00:00:42't ))`,
(includes: boolean) => {
if (!includes) return
assertGridRowCount(3, () => done())
}
)
})
}, false)
})
@@ -75,9 +95,13 @@ context('filtering tests: ', function () {
openFilterPopup(() => {
setFilterWithValue('SOME_DATE', '12/02/1960', 'date', () => {
checkInfoBarIncludes(`(( SOME_DATE = 42 ))`, (includes: boolean) => {
if (includes) done()
})
checkInfoBarIncludes(
`(( SOME_DATE = '12FEB1960'd ))`,
(includes: boolean) => {
if (!includes) return
assertGridRowCount(5, () => done())
}
)
})
})
})
@@ -87,9 +111,13 @@ context('filtering tests: ', function () {
openFilterPopup(() => {
setFilterWithValue('SOME_DATE', '42', 'value', () => {
checkInfoBarIncludes(`(( SOME_DATE = 42 ))`, (includes: boolean) => {
if (includes) done()
})
checkInfoBarIncludes(
`(( SOME_DATE = '12FEB1960'd ))`,
(includes: boolean) => {
if (!includes) return
assertGridRowCount(5, () => done())
}
)
})
}, false)
})
@@ -104,9 +132,10 @@ context('filtering tests: ', function () {
'datetime',
() => {
checkInfoBarIncludes(
`(( SOME_DATETIME = 42 ))`,
`(( SOME_DATETIME = '01JAN1960:00:00:42'dt ))`,
(includes: boolean) => {
if (includes) done()
if (!includes) return
assertGridRowCount(5, () => done())
}
)
}
@@ -120,9 +149,10 @@ context('filtering tests: ', function () {
openFilterPopup(() => {
setFilterWithValue('SOME_DATETIME', '42', 'value', () => {
checkInfoBarIncludes(
`(( SOME_DATETIME = 42 ))`,
`(( SOME_DATETIME = '01JAN1960:00:00:42'dt ))`,
(includes: boolean) => {
if (includes) done()
if (!includes) return
assertGridRowCount(5, () => done())
}
)
})
@@ -134,9 +164,13 @@ context('filtering tests: ', function () {
openFilterPopup(() => {
setFilterWithValue('SOME_DATE', '', 'in', () => {
checkInfoBarIncludes(`(( SOME_DATE IN (42) ))`, (includes: boolean) => {
if (includes) done()
})
checkInfoBarIncludes(
`(( SOME_DATE IN ('12FEB1960'd) ))`,
(includes: boolean) => {
if (!includes) return
assertGridRowCount(5, () => done())
}
)
})
})
})
@@ -156,6 +190,34 @@ context('filtering tests: ', function () {
// })
// })
// })
/**
* With the pickers off, a temporal BETWEEN takes both of its values from a
* soft-select over the column's values, so the value is chosen from that list
* rather than typed. The range dropdown binds the formatted label, as the
* single-value dropdown's list does, so the field reads the date and the
* clause composition maps it back to the internal day / second count before
* the literal is built - a display string that reached sasTemporalLiteral as
* a non-number would come back null and be passed through to RAW_VALUE
* unquoted, which mp_filtercheck rejects on a numeric column and aborts the
* save. The test asserts the applied clause and the row count, so it pins
* the mapping end to end.
*/
it('7 | Non picker - filter date field BETWEEN from the value list', (done) => {
openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
openFilterPopup(() => {
setTemporalRangeFromValueList('SOME_DATE', '12FEB1960', () => {
checkInfoBarIncludes(
`BETWEEN '12FEB1960'd AND '12FEB1960'd`,
(includes: boolean) => {
if (!includes) return
assertGridRowCount(5, () => done())
}
)
})
}, false)
})
})
const checkInfoBarIncludes = (text: string, callback: any) => {
@@ -166,6 +228,21 @@ const checkInfoBarIncludes = (text: string, callback: any) => {
})
}
/**
* The temporal filter tests assert the applied filter still selects the same
* rows as the internal value did - a readable clause that filtered differently
* would be a worse bug than the one being fixed.
*/
const assertGridRowCount = (expected: number, callback: any) => {
cy.get('#hotTable .ht_master tbody tr', {
timeout: longerCommandTimeout
})
.should('have.length', expected)
.then(() => {
if (callback) callback()
})
}
const openFilterPopup = (
callback?: any,
usePickers: boolean = true,
@@ -331,6 +408,54 @@ const setFilterWithValue = (
if (callback) callback()
}
/**
* Drives a temporal BETWEEN with the pickers off, choosing both values from the
* range soft-select's list rather than typing them - typing bypasses the
* binding this test exists to pin.
*/
const setTemporalRangeFromValueList = (
variableValue: string,
formatted: string,
callback?: any
) => {
cy.wait(600)
cy.focused().type(variableValue)
cy.wait(100)
cy.get('.variable-col .autocomplete-wrapper', { withinSubject: null })
.first()
.trigger('keydown', { key: 'ArrowDown' })
cy.get('.variable-col .autocomplete-wrapper', {
withinSubject: null
}).trigger('keydown', { key: 'Enter' })
cy.focused().tab()
cy.wait(100)
cy.focused().select('BETWEEN').trigger('change')
cy.wait(400)
const chooseFromList = (range: 'start' | 'end') => {
const inputId = `vals-${range}_0_0`
cy.get(`#${inputId}`)
.clear({ force: true })
.type(formatted, { force: true })
cy.get(`#datalist_${inputId} option`)
.contains(formatted)
.click({ force: true })
cy.get(`#${inputId}`).trigger('keyup', { key: 'Escape', force: true })
cy.wait(300)
}
chooseFromList('start')
chooseFromList('end')
cy.contains('.filter-modal button', 'Ok').click()
cy.get('.app-loading', { timeout: longerCommandTimeout }).should('not.exist')
if (callback) callback()
}
const openTableFromTree = (libNameIncludes: string, tablename: string) => {
cy.get('.app-loading', { timeout: longerCommandTimeout })
.should('not.exist')
+4 -5
View File
@@ -14,11 +14,10 @@ context('metanav tests: ', function () {
})
this.beforeEach(() => {
cy.visit(hostUrl + appLocation)
cy.get('input.username').type(username)
cy.get('input.password').type(password)
cy.get('.login-group button').click()
// beforeAll has already logged in. Typing into the login form here failed
// once the session existed - the form is not on screen, so Cypress reported
// "the center of this element is hidden from view" and the whole suite was
// skipped behind that hook.
visitPage('view/metadata')
})
+82 -51
View File
@@ -1,13 +1,15 @@
// The EDIT screen refuses a selection that is too large - DC_MAXOBS_WEBEDIT
// rows, or DC_MAXCELLS_WEBEDIT cells (rows x columns), whichever is reached
// first - and the VIEW screen caps at DC_MAXOBS_WEBVIEW rows.
// The EDIT screen is bounded by DC_MAXOBS_WEBEDIT rows and DC_MAXCELLS_WEBEDIT
// cells (rows x columns), whichever is reached first, and the VIEW screen caps
// at DC_MAXOBS_WEBVIEW rows.
//
// A selection over an EDIT limit is served TRUNCATED - the grid carries the
// first rows that fit, and a warning triangle against the table name says why.
// This spec pins those limits against the mock, which mirrors the shipped
// defaults (250 rows / 200000 cells for EDIT, 2000 rows for VIEW). It uses
// two existing fixtures rather than new ones:
// MPE_X_SEARCH - 1000 rows x 9 cols, so it trips the ROW limit
// MPE_X_WIDE - 200 rows x 1001 cols (200,400 cells), under the row limit
// but over the CELL limit
// MPE_X_WIDE - 200 rows x 1001 cols (200,200 cells), under the row limit
// but over the CELL limit, so the row cap is 199
export {}
const hostUrl = Cypress.env('hosturl')
@@ -60,27 +62,51 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
})
})
it('2 | EDIT refuses a table over the row limit', () => {
it('2 | EDIT serves a table over the row limit, truncated, with a warning', () => {
openTableFromTree(libraryToOpenIncludes, 'mpe_x_search')
openEdit()
cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
// 1000 rows x 9 columns is over the 250 row limit, so the grid carries the
// first 250 rows and the header says how many it is showing.
cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
'contain.text',
'Table is too big (1000 rows) - please filter and try again!'
'250 rows'
)
cy.get('#hotTable .ht_master .htCore tbody tr', {
timeout: longerCommandTimeout
}).should('have.length.at.least', 1)
cy.get('.truncation-warning', { timeout: longerCommandTimeout })
.should('be.visible')
.realHover()
cy.get('.editor-title clr-tooltip-content', {
timeout: longerCommandTimeout
})
.should('be.visible')
.should('contain.text', 'DC_MAXOBS_WEBEDIT')
.should(($el) => expect($el.css('opacity')).to.eq('1'))
})
it('3 | EDIT refuses a table over the cell limit', () => {
it('3 | EDIT serves a table over the cell limit, truncated, with a warning', () => {
openTableFromTree(libraryToOpenIncludes, 'mpe_x_wide')
openEdit()
// 200 rows x 1002 columns (the 1001 stored columns plus the housekeeping
// delete column) = 200,400 cells, over the 200,000 cell limit, while the
// row count is under the 250 row limit.
cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
// 200 rows x 1001 stored columns = 200,200 cells, over the 200,000 cell
// limit, while the row count is under the 250 row limit - so the cell
// limit sets the cap: floor(200000 / 1001) = 199 rows.
cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
'contain.text',
'Selection is too wide (200 rows x 1002 cols) - please filter and try again!'
'199 rows'
)
cy.get('.truncation-warning', { timeout: longerCommandTimeout })
.should('be.visible')
.realHover()
cy.get('.editor-title clr-tooltip-content', {
timeout: longerCommandTimeout
})
.should('be.visible')
.should('contain.text', 'DC_MAXCELLS_WEBEDIT')
.should(($el) => expect($el.css('opacity')).to.eq('1'))
})
it('4 | the EDIT limits follow MPE_CONFIG, not the shipped defaults', () => {
@@ -90,20 +116,29 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
// all. And MPE_CONFIG lives in the control library, so reading it from the
// table's own library would fail for this table in particular.
//
// The cell limit is the one to move: 7 rows x 18 columns is 126 cells, so
// at 110 the TESTDATA.MPE_X_NEW edit must be refused, where a hard-coded
// 200000 - or a config read that fails and falls back to it - lets it
// through. 110 is still above MPE_CONFIG's own 17 x 6 = 102, so the option
// can be put back afterwards.
// The cell limit is the one to move: 7 rows x 17 columns is 119 cells, so
// at 110 the TESTDATA.MPE_X_NEW edit must be truncated to
// floor(110 / 17) = 6 rows, where a hard-coded 200000 - or a config read
// that fails and falls back to it - lets it through. 110 is still above
// MPE_CONFIG's own 6 x 17 = 102, so the option can be put back afterwards.
setConfigValue('DC_MAXCELLS_WEBEDIT', '110', () => {
visitPage('view/data/TESTDATA.MPE_X_NEW')
closeAbortModalIfPresent()
openEdit()
cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
'contain.text',
'Selection is too wide (7 rows x 18 cols) - please filter and try again!'
'6 rows'
)
cy.get('.truncation-warning', { timeout: longerCommandTimeout })
.should('be.visible')
.realHover()
cy.get('.editor-title clr-tooltip-content', {
timeout: longerCommandTimeout
})
.should('be.visible')
.should('contain.text', 'DC_MAXCELLS_WEBEDIT')
.should(($el) => expect($el.css('opacity')).to.eq('1'))
})
// Put the option back, so the rest of the estate sees the shipped default.
@@ -112,28 +147,31 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
closeAbortModalIfPresent()
openEdit()
// 126 cells under a 200000 cell limit: the editor loads, no abort.
cy.get('#hotTable .ht_master .htCore tbody tr', {
timeout: longerCommandTimeout
}).should('exist')
// 119 cells under a 200000 cell limit: all 7 rows load, no warning.
cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
'contain.text',
'7 rows'
)
cy.get('.truncation-warning').should('not.exist')
})
})
it('5 | the filter picker still lists the variables after the abort', () => {
it('5 | the warning says to filter, and filtering narrows the grid', () => {
openTableFromTree(libraryToOpenIncludes, 'mpe_x_search')
openEdit()
cy.get('.abortMsg', { timeout: longerCommandTimeout }).should(
'contain.text',
'Table is too big (1000 rows) - please filter and try again!'
)
cy.get('.truncation-warning', { timeout: longerCommandTimeout })
.should('be.visible')
.realHover()
cy.get('.editor-title clr-tooltip-content', {
timeout: longerCommandTimeout
})
.should('be.visible')
.should('contain.text', 'Filter the table to work with the rest')
.should(($el) => expect($el.css('opacity')).to.eq('1'))
closeAbortModal()
// The abort returns no table, and the picker's variable list normally comes
// from the table's own columns - so without the public/getcols fallback the
// "please filter and try again" advice has nothing to pick from. Every
// column of MPE_X_SEARCH must be on offer.
// The advice has to be actionable: the picker must offer the table's
// columns and the filter must reload the grid.
openFilterPopup()
filterVariables().should(($options) => {
@@ -155,20 +193,20 @@ context('EDIT row and cell limits, VIEW row limit: ', function () {
])
})
// and the suggestion list is usable, not just populated
setFilterVariable('SITE_NAME')
// Submitting is what exercises TYPE. The picker keys its operator set and
// its value quoting on it, and the fallback list never passes through
// mergeColsRules, so without TYPE on this payload a character value would be
// submitted unquoted and the backend's %mp_filtercheck would reject it.
setFilterValue('Bristol')
// 100 of the 1000 rows: under the limit, so nothing is truncated.
setFilterValue('Nereid Trench')
submitFilter()
cy.get('.infoBar-text', { timeout: longerCommandTimeout }).should(
'contain.text',
"SITE_NAME = 'Bristol'"
"SITE_NAME = 'Nereid Trench'"
)
cy.get('.editor-title', { timeout: longerCommandTimeout }).should(
'contain.text',
'100 rows'
)
cy.get('.truncation-warning').should('not.exist')
})
})
@@ -185,13 +223,6 @@ const openEdit = () => {
}).click()
}
/** The abort modal offers Close, and the filter button sits behind it. */
const closeAbortModal = () => {
cy.get('app-info-modal', { timeout: longerCommandTimeout })
.contains('button', 'Close')
.click()
}
/**
* The EDIT screen's Filter button opens the same .filter-modal the viewer uses.
* It only renders once the editor is in its read-only (non-editing) state,
@@ -0,0 +1,81 @@
// Marks this file as an ES module so its top-level consts don't collide with
// other spec files under the TS type-checker.
export {}
//
// When the startup service fails without a usable response, the error modal
// offers a button that runs the same service in a new browser window, where the
// SAS log is returned instead. This spec forces a startup failure against the
// mock backend and checks the button and the URL it opens.
const hostUrl = Cypress.env('hosturl')
const STARTUP_PROGRAM = '/Public/app/dc/services/public/startupservice'
/** The body a Viya Compute task returns when the run outlives the
* execution-task timeout: a bare string, so the adapter has no webout and no
* log to report. */
const rawJobError = [
'Job error',
'The Compute service could not execute the task c74a707a-8b88-46a5-8f28-9a7694ad5e13 because the context is not reusable.',
'path: /compute/tasks',
'correlator: dc1a9fda-8f65-4f32-b4d4-2a70e02179b1;1bcb6a4c-8a16-4053-a790-292544c25a03'
].join('\n')
context('startupservice launcher: ', function () {
it('1 | runs the startupservice in a new window when its response is malformed', () => {
// Report an empty admin folder so the app loads the startup service: on a
// freshly seeded mock estate makedata is still present, which sends the app
// to the deploy screen instead.
cy.intercept('GET', '**/drive/folder**', (req) => {
// Guarded parse: a lone '%' in the URL would make decodeURIComponent
// throw, and an exception here aborts the request.
let url = req.url
try {
url = decodeURIComponent(req.url)
} catch (e) {
url = req.url
}
if (url.includes('services/admin')) {
req.reply({ files: [], folders: [] })
return
}
req.continue()
})
cy.intercept('**/stp/execute/**', (req) => {
if (req.url.includes('public/startupservice')) {
// The adapter cannot use a bare string as a response payload, so the
// startup request fails with nothing to show.
req.reply({
statusCode: 200,
headers: { 'content-type': 'application/json' },
body: JSON.stringify(rawJobError)
})
return
}
req.continue()
})
// The launcher must not actually open a tab under test.
cy.visit(hostUrl, {
timeout: 60000,
onBeforeLoad(win) {
cy.stub(win, 'open').as('windowOpen')
}
})
cy.get('.clr-abort-modal', { timeout: 60000 }).should('be.visible')
cy.get('.clr-abort-modal').last().should('contain.text', 'startupservice')
cy.get('.clr-abort-modal').last().contains('button', 'Show log').click()
cy.get('@windowOpen').should('have.been.calledOnce')
cy.get('@windowOpen').then((stub: any) => {
const url = stub.firstCall.args[0]
expect(url).to.contain(STARTUP_PROGRAM)
expect(url).to.contain('_debug=131')
// The browser route is what makes the log available: no execution task.
expect(url).not.to.contain('_executionTasks')
})
})
})
+36 -22
View File
@@ -79,14 +79,34 @@ Cypress.Commands.add(
.then(() => {
cy.wait(2000)
if ($body.find('.nav-tree').length > 0) {
/**
* A key that is already working needs no update. Ask the loaded
* page, not the tree: a free tier estate renders the nav tree as
* well, so the tree cannot tell a licensed estate from an
* unlicensed one - the licence banner is what does. Reading the
* body again here also matters on an estate that shows a login
* form, where the body captured above has no tree yet.
*/
cy.get('body').then(($loaded) => {
const licenceProblem = $loaded.find('#demo-banner').length > 0
if (!licenceProblem && $loaded.find('.nav-tree').length > 0) {
return logout(() => {
return
})
}
/**
* If licence key is already working, then skip rest of the function
* An estate with no key is left as it is unless the caller asked
* for a licensed one - the free tier specs are written against
* that state and must not have a key planted under them.
*/
return logout(() => {
return
})
} else {
if (!forceLicenceKey) {
return logout(() => {
return
})
}
const keyData = {
valid_until: moment().add(20, 'day').format('YYYY-MM-DD'),
number_of_users: 10,
@@ -106,27 +126,21 @@ Cypress.Commands.add(
cy.wait(20000)
}
if (!forceLicenceKey) {
return logout(() => {
return
})
} else {
return updateLicenseKeyQuick(keysGen, () => {
cy.wait(25000)
return acceptTermsIfPresented((result: boolean) => {
if (result) {
cy.wait(20000)
}
return logout(() => {
return
})
return updateLicenseKeyQuick(keysGen, () => {
cy.wait(25000)
return acceptTermsIfPresented((result: boolean) => {
if (result) {
cy.wait(20000)
}
return logout(() => {
return
})
})
}
})
})
}
)
}
})
})
})
}
+1 -1
View File
@@ -10,7 +10,7 @@ const check = (cwd) => {
onlyAllow:
'AFLv2.1;Apache 2.0;Apache-2.0;Apache*;Artistic-2.0;BlueOak-1.0.0;0BSD;BSD*;BSD-2-Clause;BSD-3-Clause;CC0-1.0;CC-BY-3.0;CC-BY-4.0;ISC;MIT;MPL-2.0;ODC-By-1.0;Python-2.0;Unlicense;',
excludePackages:
'@cds/city@1.1.0;@handsontable/angular-wrapper@16.0.1;@handsontable/angular-wrapper@17.1.0;@handsontable/angular-wrapper@18.0.0;@handsontable/angular-wrapper@18.1.1;handsontable@^16.0.1;handsontable@16.2.0;handsontable@17.1.0;handsontable@18.0.0;handsontable@18.1.1;hyperformula@2.7.1;hyperformula@3.0.0;hyperformula@3.1.0;hyperformula@3.2.0;hyperformula@3.3.0;jackspeak@3.4.3;path-scurry@1.11.1;package-json-from-dist@1.0.1;buffers@0.1.1'
'@cds/city@1.1.0;@handsontable/angular-wrapper@16.0.1;@handsontable/angular-wrapper@17.1.0;@handsontable/angular-wrapper@18.0.0;@handsontable/angular-wrapper@18.1.1;@handsontable/angular-wrapper@18.1.2;handsontable@^16.0.1;handsontable@16.2.0;handsontable@17.1.0;handsontable@18.0.0;handsontable@18.1.1;handsontable@18.1.2;hyperformula@2.7.1;hyperformula@3.0.0;hyperformula@3.1.0;hyperformula@3.2.0;hyperformula@3.3.0;jackspeak@3.4.3;path-scurry@1.11.1;package-json-from-dist@1.0.1;buffers@0.1.1'
},
(error, json) => {
if (error) {
+965 -1337
View File
File diff suppressed because it is too large. Load diff
+17 -16
View File
@@ -37,26 +37,28 @@
"compodoc:build-and-serve": "compodoc -p tsconfig.doc.json -s --name 'Data Controller Client'",
"compodoc:serve": "compodoc -s --name 'Data Controller Client'",
"lighthouse": "lhci autorun",
"ng": "ng"
"ng": "ng",
"test:tz-west": "TZ=America/New_York npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'",
"test:tz-east": "TZ=Pacific/Auckland npx ng test --no-watch --no-progress --browsers ChromeHeadlessCI --include='**/va-*.spec.ts'"
},
"private": true,
"dependencies": {
"@angular/animations": "20.3.33",
"@angular/animations": "20.3.34",
"@angular/cdk": "^20.2.14",
"@angular/common": "20.3.33",
"@angular/compiler": "20.3.33",
"@angular/core": "20.3.33",
"@angular/forms": "20.3.33",
"@angular/platform-browser": "20.3.33",
"@angular/platform-browser-dynamic": "20.3.33",
"@angular/router": "20.3.33",
"@angular/common": "20.3.34",
"@angular/compiler": "20.3.34",
"@angular/core": "20.3.34",
"@angular/forms": "20.3.34",
"@angular/platform-browser": "20.3.34",
"@angular/platform-browser-dynamic": "20.3.34",
"@angular/router": "20.3.34",
"@cds/core": "^6.15.1",
"@clr/angular": "file:libraries/clr-angular-17.9.0.tgz",
"@clr/icons": "^13.0.2",
"@clr/ui": "file:libraries/clr-ui-17.9.0.tgz",
"@handsontable/angular-wrapper": "18.1.1",
"@sasjs/adapter": "4.19.1",
"@sasjs/utils": "^3.6.0",
"@handsontable/angular-wrapper": "18.1.2",
"@sasjs/adapter": "5.1.2",
"@sasjs/utils": "4.0.0",
"@sheet/crypto": "file:libraries/sheet-crypto.tgz",
"@types/d3-graphviz": "^2.6.7",
"@types/text-encoding": "0.0.35",
@@ -69,12 +71,11 @@
"d3-graphviz": "^5.0.2",
"exceljs": "^4.4.0",
"fs-extra": "^7.0.1",
"handsontable": "18.1.1",
"handsontable": "18.1.2",
"https-browserify": "1.0.0",
"hyperformula": "^2.5.0",
"iconv-lite": "^0.5.0",
"jquery-datetimepicker": "^2.5.21",
"jsrsasign": "11.1.1",
"marked": "^5.0.0",
"moment": "^2.30.1",
"ngx-clipboard": "^16.0.0",
@@ -100,7 +101,7 @@
"@angular-eslint/schematics": "19.8.1",
"@angular-eslint/template-parser": "19.8.1",
"@angular/cli": "20.3.37",
"@angular/compiler-cli": "20.3.33",
"@angular/compiler-cli": "20.3.34",
"@babel/plugin-proposal-private-methods": "^7.18.6",
"@compodoc/compodoc": "^2.0.0",
"@cypress/webpack-preprocessor": "^5.17.1",
@@ -148,7 +149,7 @@
"ajv": "8.18.0",
"uuid": "11.1.1",
"lighthouse": "13.4.0",
"fast-uri": "3.1.8",
"fast-uri": "4.2.3",
"readdir-glob": {
"brace-expansion": "^5.0.9"
},
+18
View File
@@ -1,4 +1,5 @@
import { QueryClause } from './models/TableData'
import { VaParmapTarget } from './shared/utils/parse-va-parmap'
/**
* Filtering cache info, to be reused when filtering modal is re-open
@@ -61,6 +62,21 @@ export const globals: {
* Defaults to 'live'; toggled by the Auto-apply checkbox in the editor.
*/
vaApplyMode: 'live' | 'confirm'
/**
* Explicit VA parameter -> DC column mappings, parsed from the `va_parmap`
* URL parameter (see parseVaParmap). Empty unless configured; entries override
* VA's default same-name (parameter label -> column) resolution, so two DC
* instances over one table can filter the same column from differently-named
* VA parameters.
*/
vaParamMap: Map<string, VaParmapTarget>
/**
* `va_parmap` entries that could not be used at all (not a plain SAS column
* name, or missing a side of the `-`). Surfaced to the report author, because
* these are hand-written in the report's URL and a typo otherwise looks like
* a working, unfiltered view.
*/
vaParmapProblems: string[]
rootParam: string
dcLib: string
xlmaps: XLMapListItem[]
@@ -77,6 +93,8 @@ export const globals: {
} = {
embed: false,
vaApplyMode: 'live',
vaParamMap: new Map(),
vaParmapProblems: [],
rootParam: <string>'',
dcLib: '',
xlmaps: [],
+22 -3
View File
@@ -12,6 +12,7 @@ import '@clr/icons'
import '@clr/icons/shapes/all-shapes'
import { globals } from './_globals'
import { parseEmbedParam } from './shared/utils/parse-embed-param'
import { parseVaParmap } from './shared/utils/parse-va-parmap'
import moment from 'moment'
import { EventService } from './services/event.service'
import { AppService } from './services/app.service'
@@ -163,6 +164,21 @@ export class AppComponent {
globals.embed = embed
this.embed = embed
}
// VA parameter -> DC column mappings for this instance, held on globals
// (like `embed` and `vaApplyMode`) so in-app navigation that rebuilds the
// editor keeps them - e.g. returning from the stage / approve pages.
if (globals.embed === 'va') {
const parmap = parseVaParmap(hashQuery ?? '')
globals.vaParamMap = parmap.map
globals.vaParmapProblems = parmap.problems
// Visible in the browser console as well as the editor banner: the entry
// is hand-written in the report's URL, so a typo has to be findable when
// someone is looking at the report rather than at DC's UI.
for (const problem of parmap.problems) {
// eslint-disable-next-line no-console
console.warn('Data Controller: ' + problem)
}
}
this.subscribeToShowAbortModal()
this.subscribeToRequestsModal()
@@ -314,9 +330,12 @@ export class AppComponent {
*/
public subscribeToShowAbortModal() {
this.eventService.onShowAbortModal.subscribe((sasjsAbort: InfoModal) => {
let abortId = this.sasjsAborts.length + 1
sasjsAbort.id = abortId
this.sasjsAborts.push(sasjsAbort)
/* One message at a time. Pushing to the array stacked a second modal on
top of the one already open - two dismissals for one failure, and the
lower one was often a duplicate of the upper. Replacing keeps the
newest message, which is the one the user has to act on. */
sasjsAbort.id = Date.now()
this.sasjsAborts = [sasjsAbort]
this.cdr.detectChanges() //Changes were not triggered while hot is focused
})
}
+31 -2
View File
@@ -1,7 +1,36 @@
<main class="content-area position-relative">
<!-- An embedded instance (a report object) has no report author who can deploy:
showing the wizard there only produces a screen whose buttons cannot work,
because the deployment needs an authenticated, non-embedded session. Name
the state instead, and keep the wizard for standalone use. -->
@if (isEmbedded) {
<div class="clr-row">
<div id="not-configured" class="card">
<div class="card-header">Data Controller is not configured</div>
<div class="card-block">
<div class="card-text">
<p class="mt-0">
This Data Controller instance has not been deployed yet, so it
cannot be shown inside a report.
</p>
<p>
Deployment has to be done once, in its own browser tab, by someone
who can write to the application location:
<a [href]="deployUrl" target="_blank">{{ deployUrl }}</a>
</p>
<p>
When it is configured, reload the report and this object will show
the editor.
</p>
</div>
</div>
</div>
</div>
}
<div class="clr-row">
<!-- T&C section -->
@if (step === 0) {
@if (!isEmbedded && step === 0) {
<div id="TCS" class="card">
<div class="card-header">Terms and Conditions</div>
<div class="card-block">
@@ -35,7 +64,7 @@
</div>
<!-- T&C section end -->
@if (step > 0) {
@if (!isEmbedded && step > 0) {
@switch (true) {
@case (sasJsConfig.serverType === ServerType.SasViya) {
<div class="autodeploy-section card">
@@ -0,0 +1,229 @@
import { NO_ERRORS_SCHEMA } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { ServerType } from '@sasjs/utils/types/serverType'
import { DeployComponent } from './deploy.component'
import { standaloneUrl } from './standalone-url'
import { globals } from '../_globals'
import { AppStoreService } from '../services/app-store.service'
import { SasService } from '../services/sas.service'
/**
* An embedded instance cannot deploy itself: the deployment needs an
* authenticated, standalone session, and the person looking at a report object
* is not the person who installs the app. The component therefore withholds the
* wizard and names the state instead.
*
* These cases render the template, so they fail if the `!isEmbedded` guards in
* deploy.component.html are dropped - asserting the getter alone would not.
*/
describe('DeployComponent embed guard', () => {
let fixture: ComponentFixture<DeployComponent>
let component: DeployComponent
const originalEmbed = globals.embed
const configure = () =>
TestBed.configureTestingModule({
declarations: [DeployComponent],
providers: [
{
provide: AppStoreService,
useValue: {
getDcAdapterSettings: () => ({
appLoc: '/Public/app/dc',
dcPath: '/tmp/dc'
})
}
},
{
provide: SasService,
useValue: {
getSasjsInstance: () => ({}),
getSasjsConfig: () => ({ serverType: ServerType.Sasjs })
}
}
],
// The wizard renders child components this spec does not exercise.
schemas: [NO_ERRORS_SCHEMA]
}).compileComponents()
const render = () => {
fixture = TestBed.createComponent(DeployComponent)
component = fixture.componentInstance
fixture.detectChanges()
}
const tcsSection = () =>
fixture.nativeElement.querySelector('#TCS') as HTMLElement | null
const notConfigured = () =>
fixture.nativeElement.querySelector('#not-configured') as HTMLElement | null
const deploySection = () =>
fixture.nativeElement.querySelector(
'.autodeploy-section'
) as HTMLElement | null
afterEach(() => {
globals.embed = originalEmbed
TestBed.resetTestingModule()
})
it('offers the wizard when running in its own tab', () => {
globals.embed = false
configure()
render()
expect(tcsSection()).toBeTruthy()
expect(notConfigured()).toBeNull()
expect(component.isEmbedded).toBeFalse()
})
it('replaces the wizard with the notice for a VA embed', () => {
globals.embed = 'va'
configure()
render()
expect(tcsSection()).toBeNull()
expect(notConfigured()).toBeTruthy()
})
it('replaces the wizard with the notice for a plain embed too', () => {
globals.embed = true
configure()
render()
expect(tcsSection()).toBeNull()
expect(notConfigured()).toBeTruthy()
})
it('names the standalone URL in the embedded notice', () => {
globals.embed = 'va'
configure()
render()
const link = notConfigured()!.querySelector('a') as HTMLAnchorElement
expect(link.getAttribute('href')).toBe(component.deployUrl)
expect(link.textContent).toBe(component.deployUrl)
})
it('withholds the deploy section after the terms are accepted, when embedded', () => {
globals.embed = 'va'
configure()
render()
component.step = 1
fixture.detectChanges()
expect(deploySection()).toBeNull()
})
it('shows the deploy section after the terms are accepted, standalone', () => {
globals.embed = false
configure()
render()
component.step = 1
fixture.detectChanges()
expect(deploySection()).toBeTruthy()
})
})
describe('DeployComponent onNavigateToHome', () => {
const originalEmbed = globals.embed
const originalHref = window.location.href
afterEach(() => {
globals.embed = originalEmbed
history.replaceState(null, '', originalHref)
TestBed.resetTestingModule()
})
it('opens the standalone URL, merging an embed tab into the app URL', () => {
// Karma serves the runner with its own query, so the app URL is set here
// rather than read from window.location: the assertion has to see the
// app's query survive the embed parameters being dropped. The shape is
// the real SAS 9 one - the app's `_program` query in the search part,
// the embed switches in the hash query.
history.replaceState(
null,
'',
'/deploy-tc?_program=/Public/app/dc&_debug=131#/editor?embed=va&va_parmap=A-B'
)
const open = spyOn(window, 'open')
TestBed.configureTestingModule({
declarations: [DeployComponent],
providers: [
{
provide: AppStoreService,
useValue: { getDcAdapterSettings: () => ({ appLoc: '', dcPath: '' }) }
},
{
provide: SasService,
useValue: {
getSasjsInstance: () => ({}),
getSasjsConfig: () => ({ serverType: ServerType.Sasjs })
}
}
],
schemas: [NO_ERRORS_SCHEMA]
})
const fixture = TestBed.createComponent(DeployComponent)
fixture.componentInstance.onNavigateToHome()
const opened = open.calls.mostRecent().args[0] as string
expect(opened).toBe(
window.location.origin + '/deploy-tc?_program=/Public/app/dc&_debug=131'
)
expect(opened).not.toContain('#')
expect(opened).not.toContain('embed=')
})
})
describe('standaloneUrl', () => {
const origin = window.location.origin
it('keeps the path of a plain URL', () => {
expect(standaloneUrl(`${origin}/SASJobExecution/dc`)).toBe(
`${origin}/SASJobExecution/dc`
)
})
it('keeps a query the app needs, and drops the hash route', () => {
expect(standaloneUrl(`${origin}/path?x=1#/deploy`)).toBe(
`${origin}/path?x=1`
)
})
it('keeps `_program=`, which is how SAS 9 and Viya address the app', () => {
// The whole app URL is the query here: /SASJobExecution/?_program=... is
// the standalone app, so dropping the query would open a blank executor.
const sas9 =
`${origin}/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice` +
'&_debug=131'
expect(standaloneUrl(sas9)).toBe(sas9)
})
it('drops a bare embed query', () => {
expect(standaloneUrl(`${origin}/app?embed=va`)).toBe(`${origin}/app`)
})
it('drops the va_parmap entries along with the embed switch', () => {
expect(
standaloneUrl(
`${origin}/app?embed=va&va_parmap=A-B&va_parmap=C-D#/editor/x`
)
).toBe(`${origin}/app`)
})
it('keeps a query parameter that merely starts with va_', () => {
expect(standaloneUrl(`${origin}/app?embed=va&va_rowlimit=100`)).toBe(
`${origin}/app?va_rowlimit=100`
)
})
it('resolves a relative URL against the current origin', () => {
expect(standaloneUrl('/app/#/deploy')).toBe(`${origin}/app/`)
})
})
+19 -1
View File
@@ -4,6 +4,8 @@ import { SASjsConfig } from '@sasjs/adapter'
import { ServerType } from '@sasjs/utils/types/serverType'
import { AppStoreService } from '../services/app-store.service'
import { DcAdapterSettings } from '../models/DcAdapterSettings'
import { globals } from '../_globals'
import { standaloneUrl } from './standalone-url'
@Component({
selector: 'app-deploy',
@@ -44,6 +46,20 @@ export class DeployComponent implements OnInit {
this.setDeployDefaults()
}
/**
* True when DC is running inside a report rather than in its own tab. The
* deployment needs an authenticated, standalone session, so the wizard is not
* offered here - a report object cannot deploy the app that it is displaying.
*/
get isEmbedded(): boolean {
return !!globals.embed
}
/** The standalone URL an administrator deploys from, for the embedded notice. */
get deployUrl(): string {
return standaloneUrl(window.location.href)
}
/**
* Setting default values used for deploy request
*/
@@ -60,6 +76,8 @@ export class DeployComponent implements OnInit {
}
public onNavigateToHome() {
window.open(location.href.split('#')[0], '_blank')
// Same helper as deployUrl: splitting on '#' left the query string in
// place, so a tab opened from `?embed=va#/deploy` stayed embedded.
window.open(standaloneUrl(window.location.href), '_blank')
}
}
@@ -8,20 +8,6 @@
@if (autodeployDone) {
<p class="m-0 align-self-start">Done</p>
<hr class="w-100" />
@if (autoDeployStatus.deployServicePack !== null) {
<div class="deploy-status-row">
@if (autoDeployStatus.deployServicePack === true) {
<clr-icon
class="deploy-success"
shape="success-standard"
></clr-icon>
}
@if (!autoDeployStatus.deployServicePack === false) {
<clr-icon class="deploy-error" shape="times-circle"></clr-icon>
}
<p>Deploy SAS Jobs</p>
</div>
}
@if (autoDeployStatus.runMakeData !== null) {
<div class="deploy-status-row">
@if (autoDeployStatus.runMakeData) {
@@ -33,16 +19,13 @@
@if (autoDeployStatus.runMakeData === false) {
<clr-icon class="deploy-error" shape="times-circle"></clr-icon>
}
<p>Create database</p>
<p>Create control library</p>
</div>
}
<hr class="w-100" />
<div class="buttons">
<button (click)="navigateToHome()" class="btn btn-primary mt-15 mr-0">
@if (
autoDeployStatus.deployServicePack === false ||
autoDeployStatus.runMakeData === false
) {
@if (autoDeployStatus.runMakeData === false) {
<clr-icon class="deploy-error" shape="times-circle"></clr-icon>
}
LAUNCH
@@ -78,9 +61,9 @@
<h4 class="text-center my-15">Viya Deploy</h4>
<p class="deploy-intro">
Configure and deploy Data Controller to your Viya environment. Deployment
creates the backend SAS services and the control library used by the
application.
Configure Data Controller for your Viya environment. The frontend and services
are already in place; this runs the one-off job that creates the control
library and its configuration data, then removes itself.
</p>
<hr />
@@ -248,19 +231,8 @@
</div>
}
<!-- Keeping this for a reference in case future VIYA changes and starts allowing separate backend and frontend) -->
<!-- <clr-checkbox-wrapper>
<input
clrCheckbox
[(ngModel)]="recreateDatabase"
(click)="recreateDatabaseClicked($event)"
type="checkbox"
checked
/>
<label>Recreate database</label>
</clr-checkbox-wrapper> -->
<!-- Runs the one-off makedata job. This is the screen's only action: the
frontend and services are already deployed. -->
<hr />
<button
@@ -269,53 +241,3 @@
>
Deploy
</button>
<!-- Keeping this for a reference in case future VIYA changes and starts allowing separate backend and frontend) -->
<!-- <button
(click)="executeJson()"
class="btn-autodeploy btn btn-primary d-inline-block mr-10"
[disabled]="!jsonFile"
>
Deploy {{ !jsonFile ? '(json file is not available)' : '' }}
</button> -->
<!-- <button
(click)="uploadJsonAuto.click()"
class="btn-autodeploy btn btn-primary d-inline-block mr-10"
>
Upload different file to deploy
</button>
<input
#uploadJsonAuto
type="file"
hidden
(click)="clearUploadInput($event)"
(change)="onJsonFileChange($event)"
/> -->
<clr-modal [(clrModalOpen)]="recreateDatabaseModal" [clrModalClosable]="false">
<h3 class="modal-title">Warning</h3>
<div class="modal-body">
This action will recreate the database (if it exists). For an initial
deployment, this is expected. If this is a re-deployment, you will lose any
existing tables in <strong>{{ dcPath }}</strong
>.
</div>
<div class="modal-footer">
<button
type="button"
class="btn btn-sm btn-primary"
(click)="recreateDatabaseModal = false; recreateDatabase = false"
>
Cancel
</button>
<button
type="button"
class="btn btn-sm btn-primary"
(click)="recreateDatabaseModal = false; recreateDatabase = true"
>
Ok
</button>
</div>
</clr-modal>
@@ -0,0 +1,153 @@
import { SASjsConfig } from '@sasjs/adapter'
import { AutomaticComponent } from './automatic.component'
/**
* The Viya setup screen has one job: run `makedata` with the selections the
* form collected - the admin group, the DC path, and the compute context the
* job runs in. These tests pin that payload and the new-window URL, because a
* screen whose only action silently stopped calling the service would still
* compile, still render, and still look like it worked.
*/
describe('AutomaticComponent (Viya setup)', () => {
const build = () => {
const requests: any[][] = []
const sasJs: any = {
request: (...args: any[]) => {
requests.push(args)
return Promise.resolve({ result: [{}] })
}
}
const sasService: any = {
getSasjsConfig: () => new SASjsConfig(),
getExecutionPath: () => '/SASJobExecution/',
shouldLogin: { next: () => undefined },
viyaMakedataSuccessfull: () => Promise.resolve(true),
updateFileContent: () => Promise.resolve()
}
const component = new AutomaticComponent(
{
showAbortModal: () => undefined,
showInfoModal: () => undefined
} as any,
{ downloadFile: () => undefined } as any,
sasService,
{} as any,
{ error: () => undefined } as any,
{} as any
)
component.sasJs = sasJs
const config = {
contextName: 'Compute Reusable',
serverUrl: 'https://viya.example.com',
appLoc: '/Users/demo/dc'
} as any
component.sasJsConfig = config
// The new-window URL is built from the adapter config the component holds,
// while the request path reads the @Input one.
component.sasjsConfig = config
component.appLoc = '/Users/demo/dc'
component.dcPath = '/export/dc'
component.selectedAdminGroup = 'viyagroup18'
return { component, requests, sasService }
}
it('posts makedata with the admin group and the DC path', async () => {
const { component, requests } = build()
component.deployInNewWindow = false
await component.createDatabase()
expect(requests[0][0]).toBe('services/admin/makedata')
expect(requests[0][1]).toEqual({
fromjs: [{ ADMIN: 'viyagroup18', DCPATH: '/export/dc' }]
})
})
it('runs the job in the configured context, on the web method', async () => {
const { component, requests } = build()
component.deployInNewWindow = false
await component.createDatabase()
expect(requests[0][2]).toEqual({
useComputeApi: null,
contextName: 'Compute Reusable',
debug: true
})
})
it('prefers the context chosen in the form over the configured one', async () => {
const { component, requests } = build()
component.deployInNewWindow = false
component.computeContexts = [
{ id: 'ctx-1', name: 'Selected Context' } as any
]
component.selectedComputeContext = 'ctx-1'
await component.createDatabase()
expect(requests[0][2].contextName).toBe('Selected Context')
})
it('opens makedata in a new window with the selections as URL parameters', () => {
const { component } = build()
const opened: string[] = []
spyOn(window, 'open').and.callFake((url?: any) => {
opened.push(String(url))
return null
})
spyOn(component as any, 'pollMakedataCompletion')
component.deployInNewWindow = true
component.createDatabase()
expect(opened.length).toBe(1)
expect(opened[0]).toContain('/Users/demo/dc/services/admin/makedata')
expect(opened[0]).toContain('_contextname=Compute%20Reusable')
expect(opened[0]).toContain('admin=viyagroup18')
expect(opened[0]).toContain('dcpath=%2Fexport%2Fdc')
expect(opened[0]).toContain('_debug=131')
})
it('percent-encodes the values that need it in the new-window URL', () => {
// A context name or admin group can hold characters that would otherwise
// split the query string - an ampersand would start a new parameter and a
// space would arrive truncated.
const { component } = build()
const opened: string[] = []
spyOn(window, 'open').and.callFake((url?: any) => {
opened.push(String(url))
return null
})
component.runMakedataInNewWindow({
contextName: 'SAS Job Execution & co',
admin: 'group 1+2',
dcPath: '/export/dc data'
})
expect(opened[0]).toContain('_contextname=SAS%20Job%20Execution%20%26%20co')
expect(opened[0]).toContain('admin=group%201%2B2')
expect(opened[0]).toContain('dcpath=%2Fexport%2Fdc%20data')
// the parameter names themselves are not encoded
expect(opened[0]).toContain('&admin=group%201%2B2')
})
it('starts the job when deploy is pressed', () => {
// The screen's only button calls this; a version where it stopped doing so
// would leave the setup page unable to do anything at all.
const { component } = build()
const spy = spyOn(component, 'createDatabase').and.returnValue(
Promise.resolve() as any
)
component.runAutoDeploy()
expect(spy).toHaveBeenCalled()
})
})
@@ -51,17 +51,8 @@ export class AutomaticComponent implements OnInit {
public selectedComputeContext: string = ''
public makeDataResponse: string = ''
public jsonFile: any = null
public autodeploying: boolean = false
public autodeployDone: boolean = false
public recreateDatabaseModal: boolean = false
public isSubmittingJson: boolean = false
public isJsonSubmitted: boolean = false
/**
* Default was `false` when deploy was done with frontend and backend separately.
* Now we are using only streaming app, so we always want to recreate database (makedata)
*/
public recreateDatabase: boolean = true
public createDatabaseLoading: boolean = false
public adminGroupsLoading: boolean = false
public currentUserInfoLoading: boolean = false
@@ -86,18 +77,14 @@ export class AutomaticComponent implements OnInit {
public currentUserInfo: ViyaApiCurrentUser | null = null
public computeContexts: ComputeContextItem[] = []
/** autoDeployStatus
* This object presents the status for two steps that we have for deploy.
* `deployServicePack` - Creating services based on `viya.json`
* `runMakeData` - Running `makedata` service
* If any of them is `null` or `false` that means step failed
* and will be shown to user on deploy done modal.
/**
* Status of the one step this screen runs: `makedata`, which creates the
* control library and its configuration data and then deletes itself.
* `null` means it has not run yet, `false` that it failed.
*/
public autoDeployStatus: {
deployServicePack: any
runMakeData: any
} = {
deployServicePack: null,
runMakeData: null
}
@@ -420,61 +407,15 @@ export class AutomaticComponent implements OnInit {
}
/**
* Executes sas.json file to deploy the backend
* Method will first try to run the `auto deploy`
* If that fails the rest of the code is ignored.
* If request is successfull, method will continue to try
* to create database if checkbox is toggled on
* Runs the deployment. The control library and its configuration data are
* created by `makedata`, which self-deletes once it has run; the frontend and
* services arrived with the sasjs deploy, so there is nothing else to deploy
* from here.
*/
public async executeJson() {
this.isSubmittingJson = true
try {
let uploadJsonFile = await this.sasJs.deployServicePack(
this.jsonFile,
this.dcAdapterSettings?.appLoc,
undefined,
undefined,
true
)
this.autoDeployStatus.deployServicePack = true
this.isJsonSubmitted = true
} catch (ex: any) {
let textEx = ''
if (typeof ex.message !== 'string') {
textEx = JSON.stringify(ex).replace(/\\/gm, '')
} else {
textEx = ex.message
}
this.autoDeployStatus.deployServicePack = false
this.eventService.showInfoModal(
'Deploy error',
`Exception: \n ${textEx !== '' ? textEx : ex}`
)
this.autodeploying = false
this.autodeployDone = false
return
}
this.isSubmittingJson = false
}
public async runAutoDeploy(executeJson: boolean = false) {
public runAutoDeploy() {
if (!this.deployInNewWindow) this.autodeploying = true
if (executeJson) {
this.executeJson()
}
if (this.recreateDatabase) {
this.createDatabase()
} else {
if (!this.deployInNewWindow) this.autodeployDone = true
}
this.createDatabase()
}
/**
@@ -578,9 +519,9 @@ export class AutomaticComponent implements OnInit {
let serverUrl = this.sasjsConfig.serverUrl
let appLoc = this.sasjsConfig.appLoc
const execPath = this.sasService.getExecutionPath()
let contextname = `&_contextname=${params.contextName}`
let admin = `&admin=${params.admin}`
let dcPath = `&dcpath=${params.dcPath}`
let contextname = `&_contextname=${encodeURIComponent(params.contextName)}`
let admin = `&admin=${encodeURIComponent(params.admin)}`
let dcPath = `&dcpath=${encodeURIComponent(params.dcPath)}`
// Debug is ALWAYS enabled for makedata — it runs only once during
// deployment and the full log is needed to diagnose any issues.
let debug = '&_debug=131'
@@ -708,22 +649,6 @@ export class AutomaticComponent implements OnInit {
this.deployService.downloadFile(content, filename, extension)
}
public async onJsonFileChange(event: any) {
let file = event.target.files[0]
this.jsonFile = await this.deployService.readFile(file)
}
public recreateDatabaseClicked(event: Event) {
;(<HTMLInputElement>event.target).checked === true
? (this.recreateDatabaseModal = true)
: ''
}
public clearUploadInput(event: Event) {
this.deployService.clearUploadInput(event)
}
public openSasRequestsModal() {
this.eventService.openRequestsModal()
}
@@ -0,0 +1,88 @@
import { ServerType } from '@sasjs/utils/types/serverType'
import { SasjsConfiguratorComponent } from './sasjs-configurator.component'
/**
* The SAS9 / SASjs Server setup screen cannot deploy anything: it builds the
* one-off `makedata` URL and opens it, with the admin group and the directory
* the control library will live in. These tests pin the URL shape, because a
* wrong parameter name or a lost encode still produces a page that opens and
* merely fails inside SAS, where nobody is looking.
*/
describe('SasjsConfiguratorComponent (SAS9 / SASjs setup)', () => {
const build = (serverType: ServerType) => {
const sasService: any = {
getExecutionPath: () => '/SASStoredProcess/do',
sasjsMakedataChecking: () => Promise.resolve(false),
logout: () => undefined
}
const component = new SasjsConfiguratorComponent(
sasService,
{ getServerInfo: () => ({ subscribe: () => undefined }) } as any,
{ replaceState: () => undefined } as any
)
component.sasJsConfig = {
serverType,
serverUrl: 'https://sas9.example.com',
appLoc: '/User Folders/dc'
} as any
component.dcDirectory = '/tmp/DataController'
component.dcAdminGroup = 'DCADMINS'
const opened: string[] = []
spyOn(window, 'open').and.callFake((url?: any) => {
opened.push(String(url))
return null
})
return { component, opened, sasService }
}
it('opens makedata on SAS9 with the admin group and the directory', () => {
const { component, opened } = build(ServerType.Sas9)
component.makeData()
expect(opened.length).toBe(1)
expect(opened[0]).toContain(
'_program=/User Folders/dc/services/admin/makedata'
)
expect(opened[0]).toContain('admin=DCADMINS')
expect(opened[0]).toContain('path=%2Ftmp%2FDataController')
expect(opened[0]).toContain('_debug=131')
})
it('stays on the page origin for SAS9 rather than the configured server', () => {
const { component, opened } = build(ServerType.Sas9)
component.makeData()
// A SAS9 deployment is reached through the URL the user is already on; the
// configured serverUrl would point the browser at a host it may not reach.
expect(opened[0].startsWith(window.location.origin)).toBe(true)
expect(opened[0]).not.toContain('sas9.example.com')
})
it('keeps the directory readable for SASjs Server', () => {
const { component, opened } = build(ServerType.Sasjs)
component.makeData()
// The SASjs branch does not encode the path, so a plain directory stays as
// written; encoding it there would make makedata look for a folder with %2F
// in its name.
expect(opened[0]).toContain('path=/tmp/DataController')
})
it('checks the SASjs Server deployment before reloading the app', () => {
const { component, sasService } = build(ServerType.Sasjs)
const spy = spyOn(sasService, 'sasjsMakedataChecking').and.returnValue(
Promise.resolve(false)
)
component.makeData()
expect(spy).toHaveBeenCalled()
})
})
+29
View File
@@ -0,0 +1,29 @@
/**
* The URL a Data Controller instance runs at when it is not embedded: the same
* URL, without the parts that make it an embed - the `embed` switch, the
* `va_parmap` entries, and the hash route.
*
* The rest of the query is kept. On SAS 9 and Viya the app is addressed
* entirely through it (`/SASJobExecution/?_program=/.../startupservice`), so
* dropping the query would open a blank executor rather than the app, and any
* other parameter the deployment relies on would go the same way.
*
* Only these two parameters are embed parameters. A later parameter that
* happens to start with `va_` belongs to the app, not the embed, so it stays.
*
* Parsing with URL also handles the relative hrefs the adapter can hand back,
* which plain string splicing cannot.
*/
const EMBED_PARAMS = ['embed', 'va_parmap']
export const standaloneUrl = (href: string): string => {
const url = new URL(href, window.location.origin)
url.hash = ''
for (const key of Array.from(url.searchParams.keys())) {
if (EMBED_PARAMS.includes(key.toLowerCase())) {
url.searchParams.delete(key)
}
}
return url.toString()
}
@@ -245,6 +245,25 @@
}
</clr-tooltip>
@if (truncationNote) {
<clr-tooltip>
<clr-icon
clrTooltipTrigger
shape="warning"
aria-label="The selection was truncated"
class="color-orange truncation-warning"
size="20"
></clr-icon>
<clr-tooltip-content
clrPosition="right"
clrSize="lg"
*clrIfOpen
>
{{ truncationNote }}
</clr-tooltip-content>
</clr-tooltip>
}
@if (dataSource) {
@if (!zeroFilterRows) {
({{ dataSource.length | thousandSeparator: ',' }}
@@ -440,6 +459,43 @@
<!--<hr>-->
<!--{{ validationDone }}-->
<!-- VA embed: something about a `va_parmap` entry in this report
object's URL needs the report author's attention - it could not be
used, it was applied but AND-ed against another parameter, or it
names a parameter VA never sends. Shown as a banner, not just a
console warning, because
the entries are written by the report author and a typo otherwise
looks like a working filter over the whole table. -->
@if (isVaEmbed && vaMappingWarnings.length) {
<clr-alert
class="va-parmap-warning"
[clrAlertType]="'warning'"
[clrAlertClosable]="false"
>
<clr-alert-item>
<div class="alert-text">
<strong>VA parameter mapping problem</strong>
@for (warning of vaMappingWarnings; track warning) {
<div>{{ warning }}</div>
}
<!-- The reader is the report author, who can only fix this in
the report: name the parameter they edit, and say what a
correct entry looks like, so the message is actionable
without the docs. -->
<div>
Fix: edit the <code>va_parmap</code> entries in the URL of
this Data Controller object in the report. An entry is
<code>parameter-column</code>, optionally
<code>:ADD(anchor,offset)</code> or
<code>:RANGE(period,offset)</code> - for example
<code>va_parmap=SEL_DATE-D_DATE:ADD(QUARTER_START,-1)</code
>.
</div>
</div>
</clr-alert-item>
</clr-alert>
}
<div class="hot-wrapper clr-flex-1">
<hot-table
#hotInstance
@@ -27,3 +27,15 @@
font-style: italic;
opacity: 0.7;
}
/* VA embed: `va_parmap` diagnostic banner. Clarity's alert lays `.alert-title`
and `.alert-text` out on one row, which pushes a long message off the right
edge; this keeps the whole message in the flow and wraps it. */
.va-parmap-warning {
width: 100%;
.alert-text {
white-space: normal;
overflow-wrap: anywhere;
}
}
+249 -25
View File
@@ -73,6 +73,7 @@ import {
} from '../shared/dc-validator/utils/autoEscapedCellTracker'
import { HyperFormula } from 'hyperformula'
import { parseLabelsParam } from '../shared/utils/parse-labels-param'
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
import { getDisplayColHeaders } from '../shared/utils/display-col-headers'
import { buildColInfoHtml } from '../shared/utils/col-info-html'
import { globals } from '../_globals'
@@ -89,6 +90,11 @@ import { withoutEditStatus } from './utils/withoutEditStatus'
import { getEditStatusSymbol } from './utils/getEditStatusSymbol'
import { normalizeSortConfig } from './utils/normalizeSortConfig'
import { EDIT_STATUS_COLUMN_NAME } from '../shared/dc-validator/utils/editStatusColumnRule'
import { DELETE_RECORD_COLUMN_NAME } from '../shared/dc-validator/utils/deleteRecordColumnRule'
import {
prepareDeleteRecordColumn,
seedDeleteRecordColumn
} from '../shared/dc-validator/utils/deleteRecordColumn'
import {
errorRenderer,
noSpinnerRenderer,
@@ -155,6 +161,9 @@ export class EditorComponent
dsmeta: DSMeta[] = []
versions: Version[] = []
dsNote = ''
/** Set when getdata served a truncated selection - shown as a warning
* against the table name, with the reason in a tooltip. */
truncationNote = ''
viewboxes = false
@@ -477,6 +486,46 @@ export class EditorComponent
get isVaEmbed() {
return globals.embed === 'va'
}
/**
* Explicit VA parameter -> DC column mappings (`va_parmap`) for this instance.
* Parsed once at app init (see parseVaParmap) and held on globals so the
* editor reloads a filter apply performs keep using the same mapping.
*/
get vaParamMap(): Map<string, VaParmapTarget> {
return globals.vaParamMap
}
/** `va_parmap` entries that name a column this table does not have. */
public vaUnresolvedMappings: string[] = []
/** `va_parmap` keys that no VA message has named (a parameter-side typo). */
public vaUnmatchedMappings: string[] = []
/** Parameters in this message that filter the same column (an AND-ed pair). */
public vaSameColumnMappings: string[] = []
/** Transform mappings this message cannot apply (non-temporal column or value). */
public vaTransformProblems: string[] = []
/** The warning text last written to the console, so repeated VA messages log once. */
private vaWarningsReported = ''
/**
* Everything wrong with this instance's `va_parmap` entries, in the report
* author's terms: entries that could not be parsed at all, plus entries that
* name a column the open table does not have. Shown as a banner in the editor
* - a report developer is usually not going to read the browser console.
*/
get vaMappingWarnings(): string[] {
return [...globals.vaParmapProblems, ...this.vaRuntimeWarnings]
}
/**
* The warnings this instance discovers for itself, i.e. everything except the
* parse-time problems - those are logged once at app init (`app.component`),
* so logging them again here would double every line in the console.
*/
get vaRuntimeWarnings(): string[] {
return [
...this.vaUnresolvedMappings,
...this.vaUnmatchedMappings,
...this.vaSameColumnMappings,
...this.vaTransformProblems
]
}
/**
* VA filter mode. Live = auto-apply each VA change; confirm = stage the change
* and wait for the user to click Apply. Backed by globals (toggled by the
@@ -567,6 +616,8 @@ export class EditorComponent
public hasBaseDropZoneOver = false
public hasAnotherDropZoneOver = false
public headerPks: string[] = []
/** Derived (computed) columns from MPE_DERIVED - read-only, and marked as such */
public derivedVars: string[] = []
public columnLevelSecurityFlag = false
public dateTimeHeaders: string[] = []
public timeHeaders: string[] = []
@@ -1081,6 +1132,20 @@ export class EditorComponent
return colRules?.readOnly
}
/**
* A derived (computed) column - configured in MPE_DERIVED, created at read
* time. Read-only, and styled apart from an ordinary read-only column so the
* user can see that there is nothing to edit rather than that they may not.
*/
isDerivedCol(col: string | number) {
const name =
typeof col === 'number'
? (this.hotInstance?.colToProp(col) as string) || this.columnHeader[col]
: col
return !!name && this.derivedVars.includes(name.toUpperCase())
}
isColHeader(col: string) {
return this.headerArray.indexOf(col.toUpperCase()) > -1
}
@@ -1355,8 +1420,14 @@ export class EditorComponent
colSpecs.TYPE === 'num' &&
!colSpecs.DDTYPE.includes('TIME') &&
!colSpecs.DDTYPE.includes('DATE')
)
row[colKey] = parseInt(row[colKey])
) {
const numericValue = parseFloat(row[colKey])
// A cell the file parser could not read as a number stays as it
// was: the editor's numeric validation then flags it as text,
// where a NaN would render as an empty cell.
if (!Number.isNaN(numericValue)) row[colKey] = numericValue
}
}
}
}
@@ -2743,17 +2814,24 @@ export class EditorComponent
this.disableSubmit = true
this.submit = false
const errorText =
typeof err.adapterRespnse === 'string'
? err.adapterRespnse
: JSON.stringify(err.adapterRespnse)
/* sasService has already shown the SAS message when it rejected
(the abort, or the adapter's error), so reporting it again here
stacked a second modal on top of the first - and read
'undefined' because it looked for a property that does not
exist. Only speak up when nothing was shown. */
if (!err?.errorShown) {
const errorText =
typeof err?.adapterResponse?.error === 'string'
? err.adapterResponse.error
: JSON.stringify(err?.adapterResponse ?? err)
this.eventService.showAbortModal(
'editors/stagedata',
`Submit request failed\n${errorText}`,
null,
'Submit error'
)
this.eventService.showAbortModal(
'editors/stagedata',
`Submit request failed\n${errorText}`,
null,
'Submit error'
)
}
return false
})
@@ -3519,14 +3597,24 @@ export class EditorComponent
// A synchronous throw inside initSetup (called from the .then()
// above) lands here too, not just a network failure - log it, or
// the table just renders hidden with no clue why.
// eslint-disable-next-line no-console
console.error('editors/getdata failed:', err)
this.getdataError = true
this.tableTrue = true
this.handleGetdataFailure(err)
})
}
}
/**
* Puts the page into its getdata error state. Shared by the callService
* .catch and initSetup's own retry branch: a throw from the retry runs on a
* bare setTimeout, outside that promise chain, so it would otherwise never
* reach the .catch and the user would be left on the loading spinner.
*/
private handleGetdataFailure(err: any) {
// eslint-disable-next-line no-console
console.error('editors/getdata failed:', err)
this.getdataError = true
this.tableTrue = true
}
/**
* VA mode, initial (unfiltered) route only: briefly wait for VA's buffered
* first message and, if it already carries a filter, save that filter and
@@ -3555,11 +3643,20 @@ export class EditorComponent
const msg = this.vaMessaging.latestMessage()
if (!msg) return false
const clauses = this.vaFilter.buildInitialClauses(msg)
const clauses = this.vaFilter.buildInitialClauses(msg, this.vaParamMap)
if (clauses.length === 0) return false
try {
const res: any = await this.sasStoreService.saveQuery(libds, clauses)
// A `va_parmap` entry is a column the report author typed, so this
// request can legitimately fail: the table may not have that column, and
// validatefilter aborts on an unknown variable (dc/dc#404). Attempting it
// is still worth it - when the mapping is right the very first getdata is
// already filtered - but the failure must not stop the page. An abort
// modal here covers an editor that is about to load fine; the mapping
// banner names the bad entry once the real metadata arrives.
const res: any = await this.sasStoreService.saveQuery(libds, clauses, {
suppressErrorAbortModal: true
})
const id = res?.result?.[0]?.FILTER_RK
const table = res?.result?.[0]?.FILTER_TABLE
if (id === undefined || table === undefined) return false
@@ -3875,13 +3972,28 @@ export class EditorComponent
this.hotInstance = this.hotTableComponent?.hotInstance!
if (!this.hotInstance) {
// Retry init, don't permanently abort
// Retry init, don't permanently abort. A throw from the retry runs on a
// bare setTimeout rather than in the caller's promise chain, so it would
// never reach that .catch - surface it through the same handler here.
if (attempt < 60) {
setTimeout(() => this.initSetup(response, attempt + 1), 50)
setTimeout(() => {
try {
this.initSetup(response, attempt + 1)
} catch (err) {
this.handleGetdataFailure(err)
}
}, 50)
}
return
}
// The delete column is client-owned (see deleteRecordColumnRule.ts):
// this asserts getdata did not send it, then prepends it to COLHEADERS so
// it is the grid's first column. A throw here lands in the caller's
// .catch - or, on a retry, the branch above - which sets getdataError and
// shows the error state.
prepareDeleteRecordColumn(response)
this.cols = response.data.cols
this.dsmeta = response.data.dsmeta
this.versions = response.data.versions || []
@@ -3900,6 +4012,14 @@ export class EditorComponent
this.dsNote = ''
}
// getdata serves a selection over DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT
// truncated to the first rows within the limit, and carries the reason
// here. Empty means the whole selection is in the grid - and a blank
// character arrives as a single blank (mp_jsonout cannot emit a zero-length
// string, since SAS has none), so trim it: whitespace is "no note", not a
// warning with an empty tooltip.
this.truncationNote = (response.data.sasparams[0].TRUNC_MSG || '').trim()
const hot = this.hotInstance
const approvers: Approver[] = response.data.approvers
@@ -3920,6 +4040,12 @@ export class EditorComponent
this.headerColumns = response.data.sasparams[0].COLHEADERS.split(',')
this.headerPks = response.data.sasparams[0].PK.split(' ')
// Derived (computed) columns, named by getdata from MPE_DERIVED - read-only
// through a READONLY DQ rule, and marked in the grid so they read as
// computed rather than merely non-editable.
this.derivedVars = (response.data.sasparams[0].DERIVEDVARS || '')
.split(' ')
.filter((col: string) => !!col)
this.columnLevelSecurityFlag = !!response.data.sasparams[0].CLS_FLAG
if (this.columnLevelSecurityFlag)
@@ -3931,9 +4057,9 @@ export class EditorComponent
this.checkEmptyRowWhenFilter()
if (this.headerColumns.indexOf('_____DELETE__THIS__RECORD_____') !== -1) {
if (this.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME) !== -1) {
this.headerColumns[
this.headerColumns.indexOf('_____DELETE__THIS__RECORD_____')
this.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME)
] = 'Delete?'
}
@@ -3998,6 +4124,10 @@ export class EditorComponent
row[EDIT_STATUS_COLUMN_NAME] = 'U'
}
// The delete flag is client-owned too (see deleteRecordColumnRule.ts) -
// getdata no longer sends the column, so every row starts unmarked.
seedDeleteRecordColumn(this.dataSource)
// Seed HARDFORMULA/SOFTFORMULA columns with their computed formula
// string per row. dataSourceUnchanged isn't populated yet this early in
// a fresh load (only the excel-preview/add-row flows set it) - at this
@@ -4290,9 +4420,37 @@ export class EditorComponent
// header columns styling - primary keys
const isPKCol = column && this.isColPk(column)
const isReadonlyCol = column && this.isReadonlyCol(column)
const isDerivedCol = column && this.isDerivedCol(column)
if (isPKCol) th.classList.add('primaryKeyHeaderStyle')
if (isReadonlyCol && !isPKCol) th.classList.add('readonlyCell')
if (isReadonlyCol && !isPKCol && !isDerivedCol) {
th.classList.add('readonlyCell')
}
// a derived column is computed, not merely locked - give it its own
// hue and an 'fx' marker rather than the read-only dimming
if (isDerivedCol) {
th.classList.add('dc-derived-header')
// the label says what the column is - for a derived variable that
// is the configured VAR_LABEL, or the generated name=expression
// when the administrator left it blank. A label that merely
// repeats the name adds nothing, so it falls back to the rule.
const derivedLabel = (this.cols || [])
.filter((c: any) => c.NAME === column)
.map((c: any) => String(c.LABEL || '').trim())
.find(
(l: string) =>
l.length > 0 &&
l.toUpperCase() !== String(column).toUpperCase()
)
th.title = derivedLabel
? `${derivedLabel} - derived variable, computed at read time and cannot be edited`
: 'Derived variable - computed at read time and cannot be edited'
const firstDerivedCol = this.columnHeader.findIndex(
(name: string) => this.isDerivedCol(name)
)
if (col === firstDerivedCol) th.classList.add('dc-derived-first')
}
// Remove header arrow from Delete column
if (col === 0) {
@@ -4308,6 +4466,7 @@ export class EditorComponent
cellProperties: Handsontable.CellProperties
) => {
const isReadonlyCol = col && this.isReadonlyCol(col)
const isDerivedCol = col && this.isDerivedCol(col)
// Check if this cell should be marked as invalid due to duplicate primary key values
// Only applies to primary key columns (col 1 through readOnlyFields)
@@ -4331,7 +4490,26 @@ export class EditorComponent
}
// Add readonlyCell class for readonly columns to maintain original styling
if (isReadonlyCol && !classes.includes('readonlyCell')) {
if (isDerivedCol) {
// a derived cell carries its own tint; the read-only dimming would
// wash it out, and "greyed out" is a different message from
// "computed here"
classes = classes.filter((c: string) => c !== 'readonlyCell')
if (!classes.includes('dc-derived-cell')) {
classes.push('dc-derived-cell')
}
// derived columns are appended after the physical ones - mark the
// start of the block so it reads as one group
const firstDerivedCol = this.columnHeader.findIndex(
(name: string) => this.isDerivedCol(name)
)
if (
col === firstDerivedCol &&
!classes.includes('dc-derived-first')
) {
classes.push('dc-derived-first')
}
} else if (isReadonlyCol && !classes.includes('readonlyCell')) {
classes.push('readonlyCell')
}
@@ -4920,7 +5098,53 @@ export class EditorComponent
this.updateVaFilterText(msg)
const clauses = this.vaFilter.buildClauses(msg, labelToVar, this.cols)
// A `va_parmap` entry naming a column this table does not have is skipped by
// buildClauses. Report it (banner + console, once per distinct set) so the
// report author sees a misconfiguration instead of an unfiltered grid.
this.vaUnresolvedMappings = this.vaFilter.unresolvedMappings(
msg,
this.vaParamMap,
this.cols,
this.libds
)
// A key naming no parameter VA has ever sent is a typo on the parameter
// side: the mapping never applies. Only judged once parameters are actually
// arriving, since VA sends just the parameters its query uses.
this.vaUnmatchedMappings = this.vaFilter.unmatchedMappings(
this.vaParamMap,
this.vaMessaging.seenParamKeys,
this.vaMessaging.lastMessageParamsStable
)
// Two parameters filtering one column are AND-ed, so differing values leave
// an empty grid. Reported, not rewritten: turning that AND into an OR is a
// filter-semantics decision, taken in buildClauses and not in a diagnostic.
this.vaSameColumnMappings = this.vaFilter.sameColumnMappings(
msg,
this.vaParamMap,
this.cols
)
// A transform that cannot run here (char column, value that is not a date) is
// skipped by the clause builder, so the banner has to say why.
this.vaTransformProblems = this.vaFilter.unsupportedTransforms(
msg,
this.vaParamMap,
this.cols
)
const warnings = this.vaRuntimeWarnings.join(' | ')
if (warnings !== this.vaWarningsReported) {
this.vaWarningsReported = warnings
for (const warning of this.vaRuntimeWarnings) {
// eslint-disable-next-line no-console
console.warn('Data Controller: ' + warning)
}
}
const clauses = this.vaFilter.buildClauses(
msg,
labelToVar,
this.cols,
this.vaParamMap
)
const signature = this.vaFilter.signature(clauses)
if (signature === this.vaMessaging.filterSignature) {
+9
View File
@@ -47,6 +47,15 @@ export interface SASParam {
PKCNT: number
RK_FLAG: number
TMVARS: string
/** 1 when getdata cut the selection down to the row / cell limit */
TRUNCATED?: number
/** Reason shown in the tooltip when TRUNCATED is 1, else empty */
TRUNC_MSG?: string
/**
* Space separated list of the derived (computed) columns from MPE_DERIVED,
* upper case. Empty when the table has none.
*/
DERIVEDVARS?: string
}
export interface ExcelRule {
@@ -18,6 +18,11 @@ import {
import { globals } from '../_globals'
import { EditorsGetDataServiceResponse } from '../models/sas/editors-getdata.model'
import { DcValidator } from '../shared/dc-validator/dc-validator'
import {
prepareDeleteRecordColumn,
seedDeleteRecordColumn
} from '../shared/dc-validator/utils/deleteRecordColumn'
import { DELETE_RECORD_COLUMN_NAME } from '../shared/dc-validator/utils/deleteRecordColumnRule'
import { ExcelRule } from '../models/TableData'
import { HotTableInterface } from '../models/HotTable.interface'
import { Col } from '../shared/dc-validator/models/col.model'
@@ -360,7 +365,21 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
this.uploadLoading = false
const datasetObjects = this.buildDatasetsObjects(datasets)
// buildDatasetsObjects throws when a getdata response carries the
// reserved delete-record column (see deleteRecordColumn.ts). This chain
// has no .catch, so an uncaught throw here would become an unhandled
// rejection that aborts the loop with nothing shown. Surface it on
// screen, the way the editor flow refuses the same response, rather
// than leaving the user on an empty table selection with the file
// still attached.
let datasetObjects: DatasetsObject[] = []
try {
datasetObjects = this.buildDatasetsObjects(datasets)
} catch (err) {
console.warn('Error building dataset objects', err)
this.eventService.showInfoModal('Upload error', (err as Error).message)
return
}
datasetObjects.forEach((datasetObject) => {
this.parsedDatasets.push({
@@ -849,6 +868,21 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
return this.sasStoreService
.callService(myParams, 'SASControlTable', 'editors/getdata', libds)
.then((res: EditorsGetDataServiceResponse) => {
// getdata serves a selection over DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT
// truncated, rather than aborting the way it used to. This flow compares
// the whole table against the upload, so a partial read would compute
// the changes against the wrong starting state - refuse the table
// instead of submitting differences nobody asked for.
const sasParams = res?.data?.sasparams?.[0]
if (sasParams && Number(sasParams.TRUNCATED) === 1) {
this.eventService.showInfoModal(
'Upload error',
`${libds} holds more than the EDIT limit allows (${sasParams.TRUNC_MSG}), so it cannot be compared with the upload. Raise DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT, or submit the table without the multi-table upload.`
)
return undefined
}
return res
})
.catch((err: any) => {
@@ -1074,6 +1108,12 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
for (let response of responses) {
if (response.data) {
// The delete column is client-owned (see deleteRecordColumnRule.ts):
// asserts getdata did not send it, then prepends it to COLHEADERS so
// it is index 0 for everything below.
prepareDeleteRecordColumn(response)
seedDeleteRecordColumn(response.data.sasdata)
const datasetObject: DatasetsObject = {
...response,
hotTable: {
@@ -1097,14 +1137,10 @@ export class MultiDatasetComponent implements OnInit, AfterViewInit {
datasetObject.headerPks = response.data.sasparams[0].PK.split(' ')
if (
datasetObject.headerColumns.indexOf(
'_____DELETE__THIS__RECORD_____'
) !== -1
datasetObject.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME) !== -1
) {
datasetObject.headerColumns[
datasetObject.headerColumns.indexOf(
'_____DELETE__THIS__RECORD_____'
)
datasetObject.headerColumns.indexOf(DELETE_RECORD_COLUMN_NAME)
] = 'Delete?'
}
@@ -0,0 +1,110 @@
import { QueryComponent } from './query.component'
import { HelperService } from '../services/helper.service'
/**
* dynamicWhereClauseCreator builds the prefilter that getcolvals sends when the
* dynamic where clause is on, and it composes a temporal value into the same
* SAS literal the applied filter carries. The value it prefilters with comes
* from the *other* clauses in the query, so the clause under test sits at index
* 1 and index 0 is skipped.
*/
describe('QueryComponent - dynamic where clause', () => {
const buildComponent = (elements: any[]) => {
const component = new QueryComponent(
{} as any,
{ catchResponseError: () => undefined } as any,
new HelperService(null as any),
{ detectChanges: () => undefined } as any
)
component.clauses = {
queryObj: [{ clauseLogic: 'AND', elements }],
groupLogic: 'AND'
} as any
return component
}
const temporalClause = (over: any = {}) => ({
variable: 'SOME_DATE',
operator: '=',
value: '',
values: [],
type: 'num',
ddtype: 'DATE',
valueVariable: false,
...over
})
it('sends an IN entry as the literal for its unformatted value', () => {
const component = buildComponent([
temporalClause(),
temporalClause({
operator: 'IN',
value: [{ checked: true, val: '12FEB1960' }],
values: [{ formatted: '12FEB1960', unformatted: 42 }]
})
])
const table = component.dynamicWhereClauseCreator(0, 0)
expect(table.length).toBe(1)
expect(table[0].RAW_VALUE).toBe("('12FEB1960'd)")
})
it('maps both ends of a BETWEEN picked from the value list', () => {
const component = buildComponent([
temporalClause(),
temporalClause({
operator: 'BETWEEN',
value: ['12FEB1960', '13FEB1960'],
values: [
{ formatted: '12FEB1960', unformatted: 42 },
{ formatted: '13FEB1960', unformatted: 43 }
]
})
])
const table = component.dynamicWhereClauseCreator(0, 0)
expect(table.length).toBe(1)
expect(table[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
})
it('leaves an already-unformatted BETWEEN alone', () => {
const component = buildComponent([
temporalClause(),
temporalClause({
operator: 'BETWEEN',
value: [42, 43],
values: [
{ formatted: '12FEB1960', unformatted: 42 },
{ formatted: '13FEB1960', unformatted: 43 }
]
})
])
const table = component.dynamicWhereClauseCreator(0, 0)
expect(table[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
})
it('emits the unformatted value in the non-literal IN fallback', () => {
const component = buildComponent([
temporalClause(),
{
variable: 'SOME_CHAR',
operator: 'IN',
value: [{ checked: true, val: 'ABC ' }],
values: [{ formatted: 'ABC ', unformatted: 'ABC' }],
type: 'char',
ddtype: 'CHAR',
valueVariable: false
}
])
const table = component.dynamicWhereClauseCreator(0, 0)
expect(table[0].RAW_VALUE).toBe("('ABC')")
})
})
+49 -26
View File
@@ -425,35 +425,57 @@ export class QueryComponent
continue
let rawValue = ''
const operator =
this.clauses.queryObj[groupInd].elements[clauseInd].operator
const variable =
this.clauses.queryObj[groupInd].elements[clauseInd].variable
const element = this.clauses.queryObj[groupInd].elements[clauseInd]
const operator = element.operator
const variable = element.variable
/* a temporal value is sent as a SAS literal (eg '20APR2042'd), the
same way the applied filter carries it */
const temporalValue = (value: any) =>
element.valueVariable
? null
: this.helperService.sasTemporalLiteral(value, element.ddtype)
/* the dropdowns and the IN modal hold the *formatted* value (the
soft-select option value, and setINValues), so a picked value is
mapped back to the unformatted one the filter needs */
const unformattedValue = (value: any) =>
(element.values || []).find((v: any) => v.formatted === value)
?.unformatted ?? value
if (operator === 'BETWEEN') {
rawValue = `${this.clauses.queryObj[groupInd].elements[clauseInd].value[0]} AND ${this.clauses.queryObj[groupInd][clauseInd].value[1]}`
const from = unformattedValue(element.value[0])
const to = unformattedValue(element.value[1])
rawValue = `${temporalValue(from) ?? from} AND ${
temporalValue(to) ?? to
}`
} else if (operator === 'IN' || operator === 'NOT IN') {
this.clauses.queryObj[groupInd][clauseInd].value.forEach(
(element: any) => {
if (element.checked) {
let value = element.val
element.value.forEach((item: any) => {
if (item.checked) {
/* map once, and use the unformatted value in every branch - the
literal is built from it, and so are the fallbacks, so a
formatted label can never reach RAW_VALUE */
const value = unformattedValue(item.val)
const literal = temporalValue(value)
if (typeof value === 'string' && !isSpecialMissing(value)) {
if (rawValue.length > 0) {
rawValue += `,'${value}'`
} else {
rawValue = `('${value}'`
}
if (literal) {
rawValue += (rawValue.length > 0 ? ',' : '(') + literal
} else if (
typeof value === 'string' &&
!isSpecialMissing(value)
) {
if (rawValue.length > 0) {
rawValue += `,'${value}'`
} else {
if (rawValue.length > 0) {
rawValue += `,${value}`
} else {
rawValue = `(${value}`
}
rawValue = `('${value}'`
}
} else {
if (rawValue.length > 0) {
rawValue += `,${value}`
} else {
rawValue = `(${value}`
}
}
}
)
})
if (rawValue.length > 0) {
rawValue += ')'
@@ -470,11 +492,12 @@ export class QueryComponent
rawValue = '.'
}
} else {
if (
this.clauses.queryObj[groupInd].elements[clauseInd].type ===
'char'
) {
rawValue = `'${this.clauses.queryObj[groupInd].elements[clauseInd].value}'`
const literal = temporalValue(element.value)
if (literal) {
rawValue = literal
} else if (element.type === 'char') {
rawValue = `'${element.value}'`
}
}
+3 -1
View File
@@ -170,9 +170,11 @@ describe('AppService - startup retry', () => {
expect(deps.sasService.request).toHaveBeenCalledTimes(1)
expect(deps.eventService.showInfoModal).toHaveBeenCalledTimes(1)
const [, message] = deps.eventService.showInfoModal.calls.mostRecent().args
const [, message, sasService] =
deps.eventService.showInfoModal.calls.mostRecent().args
expect(message).toContain(rawJobError)
expect(message).not.toContain('Globvars, Sasdatasets, Saslibs, XLMaps')
expect(sasService).toBe('public/startupservice')
expect(deps.licenceService.isAppActivated.value).toBeFalse()
})
})
+9 -3
View File
@@ -107,7 +107,11 @@ export class AppService {
malformedMessage
)
this.startupCheckService.updateStep(2, 'error', 'Malformed response')
this.eventService.showInfoModal('Error', malformedMessage)
this.eventService.showInfoModal(
'Error',
malformedMessage,
'public/startupservice'
)
this.licenceService.isAppActivated.next(false)
return
@@ -139,7 +143,8 @@ export class AppService {
)
this.eventService.showInfoModal(
'Error',
`${missingProps.join(', ')} are not present in the startupservice`
`${missingProps.join(', ')} are not present in the startupservice`,
'public/startupservice'
)
this.licenceService.isAppActivated.next(false)
@@ -221,7 +226,8 @@ export class AppService {
this.startupCheckService.updateStep(2, 'error', 'Request failed')
this.eventService.showInfoModal(
'Error',
'There is an issue with startupservice response'
'There is an issue with startupservice response',
'public/startupservice'
)
this.licenceService.isAppActivated.next(false)
})
-18
View File
@@ -26,22 +26,4 @@ export class DeployService {
document.body.removeChild(downloadLink)
}
}
public readFile(file: File): Promise<any> {
return new Promise((resolve, reject) => {
let JsonFileReader = new FileReader()
JsonFileReader.onload = () => {
if (JsonFileReader.result) {
resolve(JSON.parse(JsonFileReader.result.toString()))
}
}
JsonFileReader.readAsText(file)
})
}
public clearUploadInput(event: Event) {
;(<HTMLInputElement>event.target).value = ''
}
}
+6 -2
View File
@@ -42,11 +42,15 @@ export class EventService {
this.onDemoLimitModalShow.emit(featureName)
}
public showInfoModal(modalTitle: string = 'Info', message: string) {
public showInfoModal(
modalTitle: string = 'Info',
message: string,
sasService: string | null = null
) {
let infoModal: InfoModal = {
modalTitle,
message,
sasService: '',
sasService,
details: null
}
@@ -46,3 +46,72 @@ describe('HelperService - convertArrayValues', () => {
expect(result.length).toBe(3)
})
})
describe('HelperService - sasTemporalLiteral', () => {
let service: HelperService
beforeEach(() => {
service = new HelperService(null as any)
})
it('renders a SAS date value as a date literal', () => {
expect(service.sasTemporalLiteral(42, 'DATE')).toBe("'12FEB1960'd")
expect(service.sasTemporalLiteral(0, 'DATE')).toBe("'01JAN1960'd")
expect(service.sasTemporalLiteral(34322, 'DATE')).toBe("'20DEC2053'd")
})
it('renders a date before 1960', () => {
expect(service.sasTemporalLiteral(-1, 'DATE')).toBe("'31DEC1959'd")
})
it('renders a SAS datetime value as a datetime literal', () => {
expect(service.sasTemporalLiteral(42, 'DATETIME')).toBe(
"'01JAN1960:00:00:42'dt"
)
expect(service.sasTemporalLiteral(86400, 'DATETIME')).toBe(
"'02JAN1960:00:00:00'dt"
)
expect(service.sasTemporalLiteral(31536001, 'DATETIME')).toBe(
"'31DEC1960:00:00:01'dt"
)
})
it('rounds double noise onto the second', () => {
expect(service.sasTemporalLiteral(1.9999999999999, 'TIME')).toBe(
"'00:00:02't"
)
expect(service.sasTemporalLiteral(42.000000000000004, 'DATETIME')).toBe(
"'01JAN1960:00:00:42'dt"
)
})
it('returns null rather than round a genuinely fractional value', () => {
expect(service.sasTemporalLiteral(42.5, 'DATETIME')).toBeNull()
expect(service.sasTemporalLiteral(42.5, 'TIME')).toBeNull()
expect(service.sasTemporalLiteral(42.3, 'TIME')).toBeNull()
})
it('renders a SAS time value as a time literal', () => {
expect(service.sasTemporalLiteral(0, 'TIME')).toBe("'00:00:00't")
expect(service.sasTemporalLiteral(42, 'TIME')).toBe("'00:00:42't")
expect(service.sasTemporalLiteral(3661, 'TIME')).toBe("'01:01:01't")
})
it('accepts a value that arrives as a string', () => {
expect(service.sasTemporalLiteral('42', 'DATE')).toBe("'12FEB1960'd")
})
it('returns null when the value cannot be a literal', () => {
// not temporal
expect(service.sasTemporalLiteral(42, 'N')).toBeNull()
expect(service.sasTemporalLiteral(42, 'C')).toBeNull()
expect(service.sasTemporalLiteral(42, '')).toBeNull()
// not a number
expect(service.sasTemporalLiteral('12FEB1960', 'DATE')).toBeNull()
expect(service.sasTemporalLiteral('', 'DATE')).toBeNull()
expect(service.sasTemporalLiteral(null, 'DATE')).toBeNull()
expect(service.sasTemporalLiteral(undefined, 'DATE')).toBeNull()
// a fractional number of days is not a SAS date
expect(service.sasTemporalLiteral(42.5, 'DATE')).toBeNull()
})
})
+80
View File
@@ -5,6 +5,22 @@ import { SasService } from './sas.service'
const librariesToShow = 50
/** Month names used by a SAS date literal, eg 20APR2042. */
const sasMonths = [
'JAN',
'FEB',
'MAR',
'APR',
'MAY',
'JUN',
'JUL',
'AUG',
'SEP',
'OCT',
'NOV',
'DEC'
]
@Injectable({
providedIn: 'root'
})
@@ -226,6 +242,70 @@ export class HelperService {
})
}
/**
* Formats a SAS internal date / datetime / time value as the SAS literal a
* WHERE clause can carry, eg:
* 34322 (DATE) -> '20DEC2053'd
* 42 (DATETIME) -> '01JAN1960:00:00:42'dt
* 42 (TIME) -> '00:00:42't
*
* The value is days since 1960-01-01 for a date, and seconds since
* 1960-01-01 (datetime) or since midnight (time). The date part is built
* arithmetically in UTC, so the day cannot shift with the browser timezone.
*
* SAS temporal values are whole days and whole seconds, so the value is
* rounded to seconds first - that kills the noise a double picks up on the
* way here, where 1.9999999999999 is simply the second 2. A value that is
* genuinely fractional does not land on a second, and rounding it would
* silently widen the filter, so it returns null instead.
*
* Returns null when the value is not a finite number, is not a whole day or
* second, or the ddtype is not temporal - callers then send the raw value.
*/
public sasTemporalLiteral(value: any, ddtype: string): string | null {
const kind = (ddtype ?? '').toString().toUpperCase()
if (kind !== 'DATE' && kind !== 'DATETIME' && kind !== 'TIME') return null
if (value === null || value === undefined || value === '') return null
const num = typeof value === 'number' ? value : Number(String(value).trim())
if (!isFinite(num)) return null
/* Within the precision of a double, is this a whole second? */
const seconds = Math.round(num * 1e8) / 1e8
if (!Number.isInteger(seconds)) return null
if (kind === 'TIME') return `'${this.sasTimeFromSeconds(seconds)}'t`
const days = kind === 'DATETIME' ? Math.floor(seconds / 86400) : seconds
const datePart = this.sasDateFromDays(days)
if (kind === 'DATE') return `'${datePart}'d`
return `'${datePart}:${this.sasTimeFromSeconds(seconds - days * 86400)}'dt`
}
/** `ddMMMyyyy` for a SAS date value (days since 1960-01-01). */
private sasDateFromDays(days: number): string {
const date = new Date(Date.UTC(1960, 0, 1) + days * 86400000)
return (
this.addLeadingZero(date.getUTCDate()) +
sasMonths[date.getUTCMonth()] +
date.getUTCFullYear()
)
}
/** `hh:mm:ss` for a SAS time value (whole seconds since midnight). */
private sasTimeFromSeconds(seconds: number): string {
return (
this.addLeadingZero(Math.floor(seconds / 3600)) +
':' +
this.addLeadingZero(Math.floor((seconds % 3600) / 60)) +
':' +
this.addLeadingZero(seconds % 60)
)
}
public addLeadingZero(value: string | number): string {
if (typeof value !== 'string') value = value.toString()
+2 -1
View File
@@ -142,7 +142,8 @@ export class LicenceService {
this.isAppActivated.next(false)
this.eventService.showInfoModal(
'Error',
'Some of the globvars are not present in the startupservice'
'Some of the globvars are not present in the startupservice',
'public/startupservice'
)
// this.router.navigateByUrl('/licensing/key?error=missing')
return
@@ -1,4 +1,5 @@
import { SasStoreService } from './sas-store.service'
import { HelperService } from './helper.service'
/**
* The filter picker keys its operator set, its variable highlight and its value
@@ -115,4 +116,245 @@ describe('SasStoreService', () => {
expect(seen.length).toBe(0)
})
})
describe('saveQuery', () => {
const capture = (adapterResponse: any = { result: [] }) => {
const calls: any[][] = []
const store = new SasStoreService(
{
request: (...args: any[]) => {
calls.push(args)
return Promise.resolve({ adapterResponse })
}
} as any,
{} as any,
{} as any
)
return { store, calls }
}
it('posts the filter to public/validatefilter', async () => {
const { store, calls } = capture()
await store.saveQuery('DC.MY_TABLE', [
{ VARIABLE_NM: 'A', OPERATOR_NM: '=', RAW_VALUE: "'x'" } as any
])
expect(calls[0][0]).toBe('public/validatefilter')
expect(calls[0][1].iwant[0].filter_table).toBe('DC.MY_TABLE')
expect(calls[0][1].filterquery.length).toBe(1)
})
it('passes wrapper options through, so a rejected filter can stay quiet', async () => {
// The VA deferred load attempts a filter built from a va_parmap entry the
// table may not have, and must not cover a page that loads fine with an
// abort modal (dc/dc#404).
const { store, calls } = capture()
await store.saveQuery('DC.MY_TABLE', [], {
suppressErrorAbortModal: true
})
expect(calls[0][3]).toEqual({ suppressErrorAbortModal: true })
})
it('sends no wrapper options when the caller gives none', async () => {
const { store, calls } = capture()
await store.saveQuery('DC.MY_TABLE', [])
expect(calls[0][3]).toBeUndefined()
})
})
})
/**
* The clause shown in the filter panel is composed by whereClauseCreator, and
* its temporal rewrite turns the internal day / second count into a SAS
* literal. It may only do that when the composed value really is nothing but
* those numbers - a value typed by hand is not, and rewriting the digit runs
* inside it produced clauses like '13JAN1960'd/'03JAN1960'd/'14MAY1965'd for
* a date typed as 12/02/1960.
*/
describe('SasStoreService - temporal clause preview', () => {
const buildStore = () =>
new SasStoreService({} as any, new HelperService(null as any), {} as any)
const clauseFor = (
value: any,
operator: string,
ddtype: string,
values: any[] = []
) => ({
queryObj: [
{
clauseLogic: '',
elements: [
{
variable: 'SOME_DATE',
operator,
value,
type: 'num',
ddtype,
valueVariable: false,
values
}
]
}
],
groupLogic: ''
})
it('leaves a typed temporal value alone when it is not a number', () => {
const store = buildStore()
const result = store.whereClauseCreator(
clauseFor('12/02/1960', '=', 'DATE'),
'',
'lib.ds'
)
expect(result.whereClause).toContain('12/02/1960')
expect(result.whereClause).not.toContain("'d")
})
it('still shows an internal day count as a date literal', () => {
const store = buildStore()
const result = store.whereClauseCreator(
clauseFor('42', '=', 'DATE'),
'',
'lib.ds'
)
expect(result.whereClause).toContain("'12FEB1960'd")
})
/**
* A BETWEEN pair is composed as `<from> AND <to>`, so the guard that decides
* whether the rewrite may run has to see past the separator - its letters are
* not part of the numbers the value is built from.
*/
it('shows a BETWEEN pair picked from the value list as literals', () => {
const store = buildStore()
const result = store.whereClauseCreator(
clauseFor(['12FEB1960', '13FEB1960'], 'BETWEEN', 'DATE', [
{ formatted: '12FEB1960', unformatted: '42' },
{ formatted: '13FEB1960', unformatted: '43' }
]),
'',
'lib.ds'
)
expect(result.whereClause).toContain(
"BETWEEN '12FEB1960'd AND '13FEB1960'd"
)
})
})
/**
* createFilterQueryTable builds the RAW_VALUE the backend validates and stores.
* The range dropdown binds the formatted label, so a BETWEEN has to map each
* end back to its unformatted value before the literal is built - otherwise
* sasTemporalLiteral returns null and an unquoted date reaches the backend.
*/
describe('SasStoreService - temporal RAW_VALUE', () => {
const buildStore = () =>
new SasStoreService({} as any, new HelperService(null as any), {} as any)
const betweenQuery = (value: any[], values: any[]) => ({
groupLogic: 'AND',
filterGroups: [
{
clauseLogic: 'AND',
filterClauses: [
{
ddtype: 'DATE',
invalidClause: false,
logic: '',
operator: 'BETWEEN',
operators: [],
type: 'num',
value,
valueVariable: false,
values,
variable: 'SOME_DATE'
}
]
}
]
})
it('maps a BETWEEN picked from the value list back to the internal value', () => {
const store = buildStore()
const clauses = store.createFilterQueryTable(
betweenQuery(
['12FEB1960', '13FEB1960'],
[
{ formatted: '12FEB1960', unformatted: 42 },
{ formatted: '13FEB1960', unformatted: 43 }
]
) as any
)
expect(clauses[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
})
it('still builds literals when the values are already internal counts', () => {
const store = buildStore()
const clauses = store.createFilterQueryTable(
betweenQuery(['42', '43'], []) as any
)
expect(clauses[0].RAW_VALUE).toBe("'12FEB1960'd AND '13FEB1960'd")
})
const inQuery = (value: any[], values: any[]) => ({
groupLogic: 'AND',
filterGroups: [
{
clauseLogic: 'AND',
filterClauses: [
{
ddtype: 'DATE',
invalidClause: false,
logic: '',
operator: 'IN',
operators: [],
type: 'num',
value,
valueVariable: false,
values,
variable: 'SOME_DATE'
}
]
}
]
})
it('maps an IN entry picked from the value list back to the internal value', () => {
const store = buildStore()
const clauses = store.createFilterQueryTable(
inQuery(
[{ checked: true, val: '12FEB1960' }],
[{ formatted: '12FEB1960', unformatted: 42 }]
) as any
)
expect(clauses[0].RAW_VALUE).toBe("('12FEB1960'd)")
})
it('passes an already-unformatted IN entry through unchanged', () => {
const store = buildStore()
const clauses = store.createFilterQueryTable(
inQuery(
[{ checked: true, val: 42 }],
[{ formatted: '12FEB1960', unformatted: 42 }]
) as any
)
expect(clauses[0].RAW_VALUE).toBe("('12FEB1960'd)")
})
})
+87 -6
View File
@@ -8,6 +8,7 @@ import { HelperService } from './helper.service'
import { QueryClause } from '../models/TableData'
import { globals } from '../_globals'
import { FilterClause, FilterGroup, FilterQuery } from '../models/FilterQuery'
import { RequestWrapperOptions } from '../models/request-wrapper/RequestWrapperOptions'
import {
$DataFormats,
EditorsGetDataSASResponse,
@@ -395,13 +396,30 @@ export class SasStoreService {
).adapterResponse
}
public async saveQuery(libds: string, filterQuery: QueryClause[]) {
/**
* Persists a filter and returns its FILTER_RK.
*
* `wrapperOptions` reaches the request wrapper, so a caller that can live with
* a rejected filter (the VA deferred load, whose `va_parmap` entry may name a
* column the table does not have) can suppress the abort modal that would
* otherwise cover a page which is about to load perfectly well.
*/
public async saveQuery(
libds: string,
filterQuery: QueryClause[],
wrapperOptions?: RequestWrapperOptions
) {
let tables = {
iwant: [{ filter_table: libds }],
filterquery: filterQuery
}
const res = await this.sasService.request('public/validatefilter', tables)
const res = await this.sasService.request(
'public/validatefilter',
tables,
undefined,
wrapperOptions
)
this.filter.next(res)
@@ -596,6 +614,27 @@ export class SasStoreService {
}
oper = this.checkOperator(oper, value, type).operator
/* A temporal value is shown as a SAS literal (eg '20DEC2053'd) rather
than the internal day / second count. Only rewrite when the composed
value is nothing but the numbers the clause is built from - a single
value, a BETWEEN pair or an IN list. The BETWEEN separator is
removed before the test because ' AND ' is not part of that character
set; a typed value that is not a number, such as a date entered by
hand, would otherwise have every digit run inside it rewritten into
an unrelated literal. */
if (
!isValueVariable &&
['DATE', 'DATETIME', 'TIME'].includes(query.ddtype) &&
/^[-0-9.,()\s]+$/.test(String(value).replace(/\s+AND\s+/gi, ' '))
) {
value = String(value).replace(
/-?\d+(?:\.\d+)?/g,
(match) =>
this.helperService.sasTemporalLiteral(match, query.ddtype) ??
match
)
}
if (type === 'char' && oper !== 'IN' && oper !== 'NOT IN') {
if (typeof value === 'undefined') {
value = ''
@@ -662,17 +701,53 @@ export class SasStoreService {
filterQuery.filterGroups.forEach(
(filterGroup: FilterGroup, groupIndex: number) => {
filterGroup.filterClauses.forEach((filterClause: FilterClause) => {
/* A temporal value is sent as a SAS literal (eg '20APR2042'd) rather
than the internal day / second count, so the resolved WHERE clause
- and the FILTER_TEXT shown above the grid - carries a date. */
const temporalValue = (value: any) =>
filterClause.valueVariable
? null
: this.helperService.sasTemporalLiteral(
value,
filterClause.ddtype
)
/* The range dropdown binds the formatted label, as the value list
and the IN modal do, so each end of a BETWEEN is mapped back to
its unformatted value before the literal is built. This is the
same lookup whereClauseCreator applies for the preview - without
it the formatted string reaches sasTemporalLiteral, comes back
null, and RAW_VALUE carries an unquoted date that the backend
rejects on a numeric column. */
const unformattedValue = (value: any) => {
const match = (filterClause.values || []).find(
(entry: any) => entry.formatted === value
)
return match ? match.unformatted : value
}
let rawValue = ''
if (filterClause.operator === 'BETWEEN') {
rawValue = `${filterClause.value[0]} AND ${filterClause.value[1]}`
const from = unformattedValue(filterClause.value[0])
const to = unformattedValue(filterClause.value[1])
rawValue = `${temporalValue(from) ?? from} AND ${
temporalValue(to) ?? to
}`
} else if (
filterClause.operator === 'IN' ||
filterClause.operator === 'NOT IN'
) {
filterClause.value.forEach((element: any) => {
if (element.checked) {
const value = element.val
if (
/* Mapped like the BETWEEN ends - whereClauseCreator normally
rewrites element.val to the unformatted count before submit,
so this is a pass-through for the usual path and a guard for
any path that reaches here without the preview having run. */
const value = unformattedValue(element.val)
const literal = temporalValue(value)
if (literal) {
rawValue += (rawValue.length > 0 ? ',' : '(') + literal
} else if (
typeof value === 'string' &&
(!isSpecialMissing(value) || filterClause.type === 'char')
) {
@@ -700,7 +775,13 @@ export class SasStoreService {
rawValue = '.'
}
} else {
if (filterClause.type === 'char' && !filterClause.valueVariable) {
const literal = temporalValue(filterClause.value)
if (literal) {
rawValue = literal
} else if (
filterClause.type === 'char' &&
!filterClause.valueVariable
) {
rawValue = `'${filterClause.value.replace(/'/g, "''")}'`
}
}
+43 -1
View File
@@ -14,7 +14,12 @@ const buildService = () => {
const sasjsAdapter: any = {
request: jasmine.createSpy('request').and.resolveTo({}),
getSasRequests: () => [],
setDebugState: jasmine.createSpy('setDebugState')
setDebugState: jasmine.createSpy('setDebugState'),
getSasjsConfig: () => ({
serverUrl: 'https://sas.example.com',
appLoc: '/Public/app/dc',
serverType: 'SASVIYA'
})
}
const loggerService: any = {
logRequestData: jasmine.createSpy('logRequestData')
@@ -163,3 +168,40 @@ describe('SasService debug state', () => {
expect(initService(true).getSasjsConfig().debug).toBe(false)
})
})
/**
* The URL shape the deploy configurator already uses for makedata: the service
* is run by the JES / STP web app in a browser tab, so the SAS log comes back
* as the page body instead of through the adapter.
*/
describe('SasService openServiceInNewWindow', () => {
it('opens the service URL built from the live adapter config', () => {
const { service } = buildService()
spyOn(service, 'getExecutionPath').and.returnValue('/SASJobExecution')
const openSpy = spyOn(window, 'open')
service.openServiceInNewWindow('public/startupservice', {
_contextname: 'ctx'
})
expect(openSpy).toHaveBeenCalledWith(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_contextname=ctx&_debug=131',
'_blank',
'noopener'
)
})
it('opens without parameters when none are given', () => {
const { service } = buildService()
spyOn(service, 'getExecutionPath').and.returnValue('/SASJobExecution')
const openSpy = spyOn(window, 'open')
service.openServiceInNewWindow('public/startupservice')
expect(openSpy).toHaveBeenCalledWith(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131',
'_blank',
'noopener'
)
})
})
+37 -2
View File
@@ -21,6 +21,7 @@ import { SasViyaService } from './sas-viya.service'
import { ViyaApiFolder } from '../viya-api-explorer/models/viya-api-folder.model'
import { ViyaApiFolderMembers } from '../viya-api-explorer/models/viya-api-folder-content.model'
import { parseUserAgent } from '../shared/utils/parse-user-agent'
import { buildServiceUrl } from '../shared/utils/service-url'
import { VERSION } from '../../environments/version'
/**
@@ -285,7 +286,10 @@ export class SasService {
adapterResponse: {
error: abortMsg
},
log: sasRequest?.logFile
log: sasRequest?.logFile,
/* the modal above already told the user: a caller that also
reports the failure would stack a second one on top */
errorShown: !wrapperOptions?.suppressSuccessAbortModal
})
}
@@ -328,7 +332,8 @@ export class SasService {
adapterResponse: {
error: errorMessage
},
log: sasRequest?.logFile
log: sasRequest?.logFile,
errorShown: !wrapperOptions?.suppressErrorAbortModal
})
}
@@ -842,6 +847,36 @@ export class SasService {
return this.sasjsAdapter
}
/**
* Runs a SAS service in a new browser window, bypassing the adapter.
*
* The adapter's request path can hide the backend log when a service fails:
* on Viya a long startup service may be submitted as an execution task and
* time out before it finishes, returning neither webout nor log. A browser
* request to the JES / STP web app runs the same `_program` with no task and
* returns the SAS log (`_debug=131`), so this is the diagnostic path offered
* when a failed request leaves nothing usable to display.
*/
public openServiceInNewWindow(
service: string,
params: { [name: string]: string | number | null | undefined } = {}
) {
const config = this.getSasjsConfig()
const url = buildServiceUrl({
serverUrl: config.serverUrl || '',
appLoc: config.appLoc || '',
executionPath: this.getExecutionPath() || '',
service,
params
})
// noopener: the SAS log page is on a different origin (Viya's serverUrl is
// not the app's), and without it the opened window keeps a handle on
// window.opener and can navigate this tab.
window.open(url, '_blank', 'noopener')
}
public getServerType(): string {
const sasjsConfig = this.sasjsAdapter.getSasjsConfig()
+743 -37
View File
@@ -1,27 +1,7 @@
import { VaFilterService } from './va-filter.service'
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
import { VaMessage } from './va-messaging.service'
/**
* Minimal stand-in for HelperService.convertJsDateToSasDate using the real
* epoch math, so date/datetime assertions use the actual SAS internal values
* without pulling in HelperService's dependency graph (SasService etc.).
*/
class HelperServiceStub {
convertJsDateToSasDate(jsDate: Date, unit = 'days'): number {
const ms = Date.UTC(
jsDate.getFullYear(),
jsDate.getMonth(),
jsDate.getDate(),
jsDate.getHours(),
jsDate.getMinutes(),
jsDate.getSeconds()
)
const sasMs = ms + 315619200000
return unit === 'days'
? Math.floor(Math.abs(sasMs / 86400000))
: sasMs / 1000
}
}
import { HelperService } from './helper.service'
function msg(parameters: any[]): VaMessage {
return {
@@ -35,6 +15,7 @@ function msg(parameters: any[]): VaMessage {
describe('VaFilterService', () => {
let service: VaFilterService
let helperService: HelperService
// DC column metadata used by the metadata path. cellValidation[0] is the
// delete column and must be skipped by buildLabelToVarMap.
@@ -63,7 +44,8 @@ describe('VaFilterService', () => {
]
beforeEach(() => {
service = new VaFilterService(new HelperServiceStub() as any)
helperService = new HelperService({} as any)
service = new VaFilterService(helperService)
})
const labelToVar = () =>
@@ -109,28 +91,30 @@ describe('VaFilterService', () => {
expect(build([{ label: 'SOME_NUM', value: 'abc' }])).toEqual([])
})
it('converts temporal values to SAS internal numbers', () => {
it('renders temporal values as SAS literals', () => {
expect(
build([{ label: 'SOME_TIME', value: '0:00:02' }])[0].RAW_VALUE
).toBe('2')
).toBe("'00:00:02't")
expect(
build([{ label: 'SOME_DATE', value: '01JUL1960' }])[0].RAW_VALUE
).toBe('182')
).toBe("'01JUL1960'd")
expect(
build([{ label: 'SOME_DATETIME', value: '01JAN1960:00:06:05' }])[0]
.RAW_VALUE
).toBe('365')
).toBe("'01JAN1960:00:06:05'dt")
})
it('passes an already-internal (bare number) temporal value through raw', () => {
it('renders an already-internal (bare number) temporal value as a literal', () => {
// VA sends the raw SAS internal value when the data item has no format.
expect(build([{ label: 'SOME_DATE', value: '182' }])[0].RAW_VALUE).toBe(
'182'
"'01JUL1960'd"
)
expect(
build([{ label: 'SOME_DATETIME', value: '365' }])[0].RAW_VALUE
).toBe('365')
expect(build([{ label: 'SOME_TIME', value: '2' }])[0].RAW_VALUE).toBe('2')
).toBe("'01JAN1960:00:06:05'dt")
expect(build([{ label: 'SOME_TIME', value: '2' }])[0].RAW_VALUE).toBe(
"'00:00:02't"
)
})
it('parses ISO date shapes identically to DATE9 (VA sends no format)', () => {
@@ -147,6 +131,41 @@ describe('VaFilterService', () => {
expect(build([{ label: 'SOME_DATE', value: '2025/03/31' }])).toHaveSize(1)
})
it('parses word dates, in either order and with an optional comma', () => {
const expected = "'01JAN2026'd"
// The reported shape: a date parameter displayed as WORDDATE.
expect(
build([{ label: 'SOME_DATE', value: 'January 1, 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: 'Jan 1, 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: 'Jan. 1, 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: '1 January 2026' }])[0].RAW_VALUE
).toBe(expected)
expect(
build([{ label: 'SOME_DATE', value: '1-Jan-2026' }])[0].RAW_VALUE
).toBe(expected)
})
it('matches a spelled-out month case-insensitively, and keeps DATE9 working', () => {
expect(
build([{ label: 'SOME_DATE', value: 'DECEMBER 31, 2025' }])[0].RAW_VALUE
).toBe("'31DEC2025'd")
expect(
build([{ label: 'SOME_DATE', value: '01AUG1962' }])[0].RAW_VALUE
).toBe("'01AUG1962'd")
})
it('still drops a month-name value that has no day', () => {
// A month and year alone is not a date - accepted shapes always carry a
// day, so this must not silently become the first of the month.
expect(build([{ label: 'SOME_DATE', value: 'January 2026' }])).toEqual([])
})
it('resolves ambiguous slash dates day-first (never US), disambiguating by value', () => {
const iso = build([{ label: 'SOME_DATE', value: '2025-03-31' }])[0]
.RAW_VALUE
@@ -218,7 +237,694 @@ describe('VaFilterService', () => {
})
})
describe('buildClauses with an explicit va_parmap', () => {
const build = (parameters: any[], paramMap: Map<string, VaParmapTarget>) =>
service.buildClauses(msg(parameters), labelToVar(), cols, paramMap)
it('overrides the same-name match, including the value formatting', () => {
// The VA label IS a DC column here (SOME_CHAR), but the mapping points at
// a different one - the numeric column - so the value is formatted numeric.
const map = new Map([['some_char', { column: 'SOME_NUM' }]])
expect(build([{ label: 'SOME_CHAR', value: '10' }], map)).toEqual([
{
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: 'SOME_NUM',
OPERATOR_NM: '=',
RAW_VALUE: '10'
}
])
})
it('lets two differently-named params filter the same column', () => {
// The multi-tab case: one DC instance per report tab, each with its own
// VA parameter, both mapping to the same DC column.
const map = new Map([
['region_tab1', { column: 'SOME_CHAR' }],
['region_tab2', { column: 'SOME_CHAR' }]
])
const clauses = build(
[
{ label: 'REGION_TAB1', value: 'north' },
{ label: 'REGION_TAB2', value: 'south' }
],
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['SOME_CHAR', "'north'"],
['SOME_CHAR', "'south'"]
])
})
it('matches the parameter label only, not its internal name', () => {
// The internal name is the report's own bookkeeping. Matching it as well
// meant one parameter answered to two keys, so an entry written against the
// name and one against the label silently fought over it.
const map = new Map([['pr273', { column: 'SOME_NUM' }]])
expect(
build([{ name: 'pr273', label: 'Tab_1_Value', value: '7' }], map)
).toEqual([])
// The label still matches, case-insensitively.
expect(
build(
[{ name: 'pr273', label: 'Tab_1_Value', value: 'x' }],
new Map([['tab_1_value', { column: 'SOME_CHAR' }]])
)[0].VARIABLE_NM
).toBe('SOME_CHAR')
})
it('uses the column case from the loaded metadata', () => {
const map = new Map([['region_tab1', { column: 'some_char' }]])
expect(
build([{ label: 'REGION_TAB1', value: 'x' }], map)[0].VARIABLE_NM
).toBe('SOME_CHAR')
})
it('leaves unmapped params on the same-name path', () => {
const map = new Map([['region_tab1', { column: 'SOME_NUM' }]])
const clauses = build(
[
{ label: 'REGION_TAB1', value: '10' },
{ label: 'SOME_CHAR', value: 'x' }
],
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['SOME_NUM', '10'],
['SOME_CHAR', "'x'"]
])
})
it('skips a mapping whose column is not in this table (no fallback)', () => {
// Falling back to the same-name match would silently apply the filter the
// mapping exists to override, so the param is dropped instead.
const map = new Map([['some_char', { column: 'NOT_IN_THIS_TABLE' }]])
expect(build([{ label: 'SOME_CHAR', value: 'x' }], map)).toEqual([])
})
it('ANDs two parameters that map to the same column - it does not union them', () => {
// What a report gets when one column's two values arrive in two
// parameters: every clause is AND-ed, so the result is the INTERSECTION.
// Two different dates therefore match no rows at all rather than both
// dates' rows - a single parameter carrying both values is the shape that
// ORs, via `IN (...)`.
const map = new Map([
['param_a', { column: 'SOME_DATE' }],
['param_b', { column: 'SOME_DATE' }]
])
const one = build([{ label: 'PARAM_A', value: '01JUL1960' }], map)[0]
const other = build([{ label: 'PARAM_B', value: '01JAN1961' }], map)[0]
const clauses = build(
[
{ label: 'PARAM_A', value: '01JUL1960' },
{ label: 'PARAM_B', value: '01JAN1961' }
],
map
)
expect(one.RAW_VALUE).not.toBe(other.RAW_VALUE)
expect(clauses).toHaveSize(2)
expect(clauses.map((c) => c.RAW_VALUE)).toEqual([
one.RAW_VALUE,
other.RAW_VALUE
])
for (const clause of clauses) {
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
expect(clause.OPERATOR_NM).toBe('=')
expect(clause.GROUP_LOGIC).toBe('AND')
expect(clause.SUBGROUP_LOGIC).toBe('AND')
expect(clause.SUBGROUP_ID).toBe(0)
}
})
it('applies a transform to the clause it builds', () => {
// ADD(QUARTER_START,-1) is the reported requirement: the selected row plus
// the row for the previous quarter, as one IN clause so the AND rule is
// untouched. 2026-10-01 = 24380, 2026-07-01 = 24288 in SAS days.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: '01OCT2026' }], map)
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
expect(clause.OPERATOR_NM).toBe('IN')
expect(clause.RAW_VALUE).toBe('(24380,24288)')
})
it('applies a RANGE as BETWEEN', () => {
// RANGE(QUARTER,-1) for a mid-quarter value: 2026-04-01..2026-06-30
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: '1962-08-15' }], map)
expect(clause.OPERATOR_NM).toBe('BETWEEN')
expect(clause.RAW_VALUE).toContain(' AND ')
expect(clause.RAW_VALUE.split(' AND ')).toHaveSize(2)
})
it('keeps equality when the value already sits on the anchor', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: 0
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: '01OCT2026' }], map)
expect(clause.OPERATOR_NM).toBe('=')
expect(clause.RAW_VALUE).toBe('24380')
})
it('drops a transform on a column it cannot apply to', () => {
// A character column has no dates to shift; the parameter is skipped rather
// than filtered on the raw value the author did not see.
const map = new Map([
[
'selected',
{
column: 'SOME_CHAR',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
expect(build([{ label: 'SELECTED', value: 'x' }], map)).toEqual([])
})
it('adds nothing when the value is unparseable as a date', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
expect(build([{ label: 'SELECTED', value: 'not a date' }], map)).toEqual(
[]
)
})
it('round-trips a bare SAS day number unchanged', () => {
// VA may send the SAS internal value rather than a formatted string. The
// Date a transform works from is local, so the number must come back out of
// the clause identical - a UTC-built Date shifts a day west of UTC.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const [clause] = build([{ label: 'SELECTED', value: 24380 }], map)
expect(clause.RAW_VALUE).toBe('(24380,24288)')
})
it('reports a range mapping fed more than one value', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
const problems = service.unsupportedTransforms(
msg([{ label: 'SELECTED', value: ['01APR2026', '01JUL2026'] }]),
map,
[{ NAME: 'SOME_DATE', DDTYPE: 'DATE', FMTNAME: 'DATE9' }]
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('is a range')
expect(problems[0]).toContain('2 values')
})
it('refuses a range fed more than one value, rather than picking one', () => {
// Two values, one of which does not parse: building a BETWEEN from the
// survivor would silently filter one period when VA selected two.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
expect(
build([{ label: 'SELECTED', value: ['01APR2026', 'not a date'] }], map)
).toEqual([])
})
it('never lets a non-finite date reach the clause', () => {
// convertJsDateToSasDate returning NaN would otherwise put a literal NaN in
// the WHERE clause.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
spyOn(helperService, 'convertJsDateToSasDate').and.returnValue(NaN as any)
expect(build([{ label: 'SELECTED', value: '01OCT2026' }], map)).toEqual(
[]
)
})
it('reports a partly unparseable value set instead of covering the rest', () => {
// One good value and one that is not a date: the clause still filters on
// the good one, so the author has to be told the other was dropped.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const problems = service.unsupportedTransforms(
msg([{ label: 'SELECTED', value: ['01OCT2026', 'not a date'] }]),
map,
cols
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('1 of its 2 values was not a date')
})
it('reports a range holding a value that does not parse', () => {
// Counted on what VA sent, not on what parsed: otherwise the second value
// vanishes and the range looks like a clean single-value case.
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'RANGE' as const,
period: 'QUARTER' as const,
offset: -1
}
}
]
])
const problems = service.unsupportedTransforms(
msg([{ label: 'SELECTED', value: ['01APR2026', 'not a date'] }]),
map,
cols
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('is a range')
expect(problems[0]).toContain('2 values')
})
it('applies a transform on the deferred initial load too', () => {
const map = new Map([
[
'selected',
{
column: 'SOME_DATE',
transform: {
op: 'ADD' as const,
anchor: 'QUARTER_START' as const,
offset: -1
}
}
]
])
const [clause] = service.buildInitialClauses(
msg([{ name: 'pr1', label: 'SELECTED', value: '01OCT2026' }]),
map
)
expect(clause.VARIABLE_NM).toBe('SOME_DATE')
expect(clause.RAW_VALUE).toBe('(24380,24288)')
})
it('behaves exactly as before with an empty map', () => {
expect(
service.buildClauses(
msg([{ label: 'SOME_CHAR', value: 'x' }]),
labelToVar(),
cols
)
).toEqual(build([{ label: 'SOME_CHAR', value: 'x' }], new Map()))
})
})
describe('unresolvedMappings (what the report author is warned about)', () => {
const report = (
parameters: any[],
map: Map<string, VaParmapTarget>,
libds?: string
) => service.unresolvedMappings(msg(parameters), map, cols, libds)
it('reports a mapping whose column is not in the loaded table', () => {
expect(
report(
[{ label: 'REGION_TAB1', value: 'x' }],
new Map([['region_tab1', { column: 'NOT_A_COLUMN' }]]),
'DC_JSLIB.MPE_X_TEST'
)
).toEqual([
'VA parameter "REGION_TAB1" is mapped to NOT_A_COLUMN, which is not a column of DC_JSLIB.MPE_X_TEST - the filter was ignored.'
])
})
it('says "this table" when the table name is not supplied', () => {
expect(
report(
[{ label: 'REGION_TAB1', value: 'x' }],
new Map([['region_tab1', { column: 'NOPE' }]])
)[0]
).toContain('is not a column of this table')
})
it('reports nothing when the mapping resolves, or the param is unmapped', () => {
expect(
report(
[{ label: 'REGION_TAB1', value: 'x' }],
new Map([['region_tab1', { column: 'some_char' }]])
)
).toEqual([])
expect(report([{ label: 'SOME_CHAR', value: 'x' }], new Map())).toEqual(
[]
)
// A mapping whose parameter this message does not carry is not an error.
expect(
report([], new Map([['region_tab1', { column: 'NOPE' }]]))
).toEqual([])
})
it('reports each mapping once, not once per message parameter', () => {
const problems = report(
[
{ label: 'REGION_TAB1', value: '1' },
{ name: 'pr99', label: 'REGION_TAB1', value: '2' }
],
new Map([['region_tab1', { column: 'NOPE' }]])
)
expect(problems).toHaveSize(1)
})
it('treats an unloaded column list as "not known yet", not as a bad mapping', () => {
// While the grid is still loading there is no metadata to judge against, so
// reporting would flash the banner on every load.
expect(
service.unresolvedMappings(
msg([{ label: 'REGION_TAB1', value: 'x' }]),
new Map([['region_tab1', { column: 'NOPE' }]]),
[],
'DC_JSLIB.MPE_X_TEST'
)
).toEqual([])
expect(
service.unresolvedMappings(
msg([{ label: 'REGION_TAB1', value: 'x' }]),
new Map([['region_tab1', { column: 'NOPE' }]]),
undefined as any,
'DC_JSLIB.MPE_X_TEST'
)
).toEqual([])
})
})
describe('sameColumnMappings (two parameters, one column)', () => {
it('reports an AND-ed pair, naming both parameters and the column', () => {
const map = new Map([
['dropdown_tab1', { column: 'SOME_DROPDOWN' }],
['dropdown_tab2', { column: 'SOME_DROPDOWN' }]
])
const problems = service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'Option 1' },
{ label: 'DROPDOWN_TAB2', value: 'Option 2' }
]),
map
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('"DROPDOWN_TAB1" and "DROPDOWN_TAB2"')
expect(problems[0]).toContain('all filter SOME_DROPDOWN')
expect(problems[0]).toContain('combined with AND')
})
it('says nothing for one parameter per column', () => {
const map = new Map([
['dropdown_tab1', { column: 'SOME_DROPDOWN' }],
['num_tab1', { column: 'SOME_NUM' }]
])
expect(
service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'Option 1' },
{ label: 'NUM_TAB1', value: '1' }
]),
map
)
).toEqual([])
expect(
service.sameColumnMappings(
msg([{ label: 'DROPDOWN_TAB1', value: 'Option 1' }]),
map
)
).toEqual([])
expect(service.sameColumnMappings(msg([]), map)).toEqual([])
})
it('says nothing when the AND-ed parameters carry the same value', () => {
// col = 'x' AND col = 'x' filters exactly as one clause does, so a banner
// claiming "different values will show no rows" would be wrong.
const map = new Map([
['dropdown_tab1', { column: 'SOME_CHAR' }],
['dropdown_tab2', { column: 'SOME_CHAR' }]
])
expect(
service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'x' },
{ label: 'DROPDOWN_TAB2', value: 'x' }
]),
map,
cols
)
).toEqual([])
// One differing value and the intersection can be empty: reported.
const problems = service.sameColumnMappings(
msg([
{ label: 'DROPDOWN_TAB1', value: 'x' },
{ label: 'DROPDOWN_TAB2', value: 'y' }
]),
map,
cols
)
expect(problems).toHaveSize(1)
expect(problems[0]).toContain('"DROPDOWN_TAB1" and "DROPDOWN_TAB2"')
})
it('leaves a column the table does not have to unresolvedMappings', () => {
// One cause, one line: the column is missing, so the AND story is moot.
const map = new Map([
['a', { column: 'NOPE' }],
['b', { column: 'NOPE' }]
])
const cols = [
{ LIBREF: 'DC_JSLIB', TABLE: 'MPE_X_TEST', VARIABLE: 'SOME_CHAR' }
]
expect(
service.sameColumnMappings(
msg([
{ label: 'A', value: '1' },
{ label: 'B', value: '2' }
]),
map,
cols
)
).toEqual([])
// Without metadata the pair is still reported: the AND semantics hold
// whatever the table turns out to be.
expect(
service.sameColumnMappings(
msg([
{ label: 'A', value: '1' },
{ label: 'B', value: '2' }
]),
map
)
).toHaveSize(1)
})
it('reports one line per contended column', () => {
const map = new Map([
['a', { column: 'SOME_DROPDOWN' }],
['b', { column: 'SOME_DROPDOWN' }],
['c', { column: 'SOME_NUM' }],
['d', { column: 'SOME_NUM' }]
])
const problems = service.sameColumnMappings(
msg([
{ label: 'A', value: '1' },
{ label: 'B', value: '2' },
{ label: 'C', value: '3' },
{ label: 'D', value: '4' }
]),
map
)
expect(problems).toHaveSize(2)
})
})
describe('unmatchedMappings (parameter-side typo)', () => {
const seen = (...keys: string[]) => new Set(keys)
it('reports a key that no message has named', () => {
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen('dropdown_tab_1', 'pr1'),
true
)
).toEqual([
'This object\'s va_parmap names VA parameter "dropdown_tab1", which VA is not sending - check the spelling.'
])
})
it('says nothing until a parameter has actually been seen', () => {
// Without evidence that this object receives parameters at all, an absent
// key proves nothing - VA sends only the parameters its query uses.
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen()
)
).toEqual([])
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
undefined
)
).toEqual([])
})
it('says nothing when every key has been seen', () => {
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen('dropdown_tab1'),
true
)
).toEqual([])
expect(
service.unmatchedMappings(new Map(), seen('anything'), true)
).toEqual([])
})
it('says nothing while the parameter set is still growing', () => {
// A message that introduces a new parameter name means this object's set
// has not settled, so an absent key may arrive in the next message.
expect(
service.unmatchedMappings(
new Map([['dropdown_tab1', { column: 'SOME_CHAR' }]]),
seen('pr1', 'pr2'),
false
)
).toEqual([])
})
})
describe('buildInitialClauses (no metadata, infer kind from value)', () => {
it('uses an explicit mapping instead of the label', () => {
// The deferred first load has no column metadata, so a mapped column is
// used as given and the SAS kind still comes from the value shape.
const map = new Map([
['region_tab1', { column: 'REGION' }],
['amt_tab1', { column: 'AMOUNT' }]
])
const clauses = service.buildInitialClauses(
msg([
{ name: 'pr1', label: 'REGION_TAB1', dataType: 'string', value: 'n' },
{ name: 'pr2', label: 'AMT_TAB1', dataType: 'number', value: '10' }
]),
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['REGION', "'n'"],
['AMOUNT', '10']
])
})
it('falls back to the label for params the map does not name', () => {
const map = new Map([['region_tab2', { column: 'REGION' }]])
const clauses = service.buildInitialClauses(
msg([
{ name: 'pr1', label: 'REGION_TAB1', value: 'n' },
{ name: 'pr2', label: 'REGION_TAB2', value: 's' }
]),
map
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['REGION_TAB1', "'n'"],
['REGION', "'s'"]
])
})
it('uses the VA label as the column and infers kinds by value shape', () => {
const clauses = service.buildInitialClauses(
msg([
@@ -232,9 +938,9 @@ describe('VaFilterService', () => {
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['SOME_CHAR', "'x'"],
['SOME_NUM', '10'],
['SOME_TIME', '2'],
['SOME_DATE', '182'],
['SOME_DATETIME', '365']
['SOME_TIME', "'00:00:02't"],
['SOME_DATE', "'01JUL1960'd"],
['SOME_DATETIME', "'01JAN1960:00:06:05'dt"]
])
})
@@ -247,9 +953,9 @@ describe('VaFilterService', () => {
])
)
expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
['SOME_DATE', '182'],
['SOME_DATE2', '182'],
['SOME_DTTM', '365']
['SOME_DATE', "'01JUL1960'd"],
['SOME_DATE2', "'01JUL1960'd"],
['SOME_DTTM', "'01JAN1960:00:06:05'dt"]
])
})
})
+494 -68
View File
@@ -2,6 +2,12 @@ import { Injectable } from '@angular/core'
import { HelperService } from './helper.service'
import { VaMessage } from './va-messaging.service'
import { QueryClause } from '../models/TableData'
import { VaParmapTarget } from '../shared/utils/parse-va-parmap'
import {
applyVaTransform,
isRangeTransform,
VaTransform
} from '../shared/utils/va-parmap-transform'
/** SAS data-type kind used to format a VA filter value into a DC RAW_VALUE. */
type VaColumnKind = 'char' | 'numeric' | 'time' | 'date' | 'datetime'
@@ -54,11 +60,19 @@ export class VaFilterService {
* Translates VA `parameters` into DC filter clauses. Each parameter is matched
* to a DC column by label; a single value becomes `col = value`, multiple
* values become `col IN (...)`. Values are formatted for the column's SAS kind.
*
* `paramMap` (from the URL `va_parmap` parameter, see parseVaParmap) overrides
* that same-name resolution for the parameters it names, so two DC instances
* over one table can filter the same column from differently-named VA
* parameters. A mapped column must exist in `cols` - if it does not the
* parameter is skipped rather than falling back to the same-name match, which
* is exactly the behaviour the mapping exists to override.
*/
buildClauses(
msg: VaMessage,
labelToVar: Map<string, string>,
cols: any[]
cols: any[],
paramMap?: Map<string, VaParmapTarget>
): QueryClause[] {
const clauses: QueryClause[] = []
for (const param of msg.parameters || []) {
@@ -67,60 +81,194 @@ export class VaFilterService {
.toString()
.trim()
.toLowerCase()
// A VA param whose label isn't a DC column (beyond case, which is handled)
// simply doesn't map — skip it.
const varName = labelToVar.get(key)
// An explicit URL mapping wins over the label->column lookup. The mapped
// column has to exist in THIS table, else the entry does not apply here
// (and the same-name fallback would be the behaviour it exists to
// override).
const mapped = this.mappedColumn(param, paramMap)
const mappedCol = mapped ? this.findCol(mapped, cols) : undefined
if (mapped && !mappedCol) continue
// The table's own case, so the clause matches what the metadata build
// produces for a same-name match.
const varName: string = mappedCol
? mappedCol.NAME
: (labelToVar.get(key) ?? '')
if (!varName) continue
const col = this.findCol(varName, cols)
const col = mappedCol ?? this.findCol(varName, cols)
const kind = this.columnKind(col)
const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
const rawValues = Array.isArray(param.value) ? param.value : [param.value]
// Format/validate every value for the column's SAS type. Numeric columns
// must receive a finite number (untrusted postMessage input is
// interpolated into a SAS WHERE clause, so a non-numeric value here would
// be an injection vector); temporal -> SAS internal number; char ->
// escaped. Invalid values are dropped.
const formatted = rawValues
.filter((v) => v !== undefined && v !== null && v !== '')
.map((v) => this.formatValueByKind(v, kind, order))
.filter((v): v is string => v !== null)
if (formatted.length === 0) continue
clauses.push(this.clause(varName, formatted))
const built = this.buildClauseValueSet(
param,
kind,
order,
this.mappedTarget(param, paramMap)?.transform
)
if (!built) continue
clauses.push({
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: varName,
OPERATOR_NM: built.operator,
RAW_VALUE: built.rawValue
})
}
return clauses
}
/**
* The operator and raw value for one parameter, after any transform.
*
* Values are formatted for the column's SAS kind: numeric columns must receive
* a finite number (untrusted postMessage input is interpolated into a WHERE
* clause, so anything else would be an injection vector); temporal values
* become SAS internal numbers; char values are escaped. Invalid values are
* dropped, and a transform that cannot be applied to the column's kind yields
* nothing at all - the caller reports that separately rather than filtering on
* a value the author did not choose.
*/
private buildClauseValueSet(
param: any,
kind: VaColumnKind,
order: 'YMD' | 'DMY' | 'MDY' | undefined,
transform?: VaTransform
): { operator: '=' | 'IN' | 'BETWEEN'; rawValue: string } | null {
const rawValues = (
Array.isArray(param.value) ? param.value : [param.value]
).filter((v: any) => v !== undefined && v !== null && v !== '')
if (rawValues.length === 0) return null
if (!transform) {
const formatted = rawValues
.map((v: any) => this.formatValueByKind(v, kind, order))
.filter((v: string | null): v is string => v !== null)
if (formatted.length === 0) return null
return formatted.length === 1
? { operator: '=', rawValue: formatted[0] }
: { operator: 'IN', rawValue: `(${formatted.join(',')})` }
}
// Transforms resolve to dates, so only temporal columns can take one.
if (kind !== 'date' && kind !== 'datetime') return null
const unit = kind === 'datetime' ? 'seconds' : 'days'
const values = new Set<string>()
for (const raw of rawValues) {
const resolved = this.valueAsDate(raw, kind, order)
if (!resolved) continue
for (const date of applyVaTransform(
resolved,
transform,
kind === 'datetime'
)) {
// A non-finite result (an Invalid Date reaching the helper) must never be
// interpolated into the clause: `NaN` is not a SAS value. Dropped here,
// and reported by unsupportedTransforms so the parameter is not silently
// filtered on nothing.
const sasDate = this.helperService.convertJsDateToSasDate(date, unit)
if (!Number.isFinite(Number(sasDate))) continue
values.add(`${sasDate}`)
}
}
const formatted = Array.from(values)
if (formatted.length === 0) return null
if (isRangeTransform(transform)) {
// One BETWEEN carries one period, so more than one value has no clause -
// dropping it here (rather than building a BETWEEN from whichever value
// parsed) is what unsupportedTransforms' count warning then describes.
if (rawValues.length > 1) return null
if (formatted.length !== 2) return null
return { operator: 'BETWEEN', rawValue: formatted.join(' AND ') }
}
return formatted.length === 1
? { operator: '=', rawValue: formatted[0] }
: { operator: 'IN', rawValue: `(${formatted.join(',')})` }
}
/**
* A VA value as a local Date, for a transform to work from. Formatted strings
* are parsed by shape; a bare number is already the SAS internal value, so it
* is converted back from days (date) or seconds (datetime) since 1960-01-01.
*
* Built with local calendar construction, matching `parseDate` and
* `HelperService.convertJsDateToSasDate`: a UTC instant read back through local
* accessors lands on the previous day west of UTC, and the round trip to a SAS
* day number would be one out.
*/
private valueAsDate(
value: any,
kind: VaColumnKind,
order?: 'YMD' | 'DMY' | 'MDY'
): Date | null {
const s = String(value).trim()
if (s === '') return null
if (this.isBareNumber(s)) {
const internal = Number(s)
if (kind !== 'datetime') return new Date(1960, 0, 1 + internal)
// Seconds since midnight 1960-01-01, split into whole days and a remainder
// so the constructor normalises the clock time locally.
const days = Math.floor(internal / 86400)
const seconds = internal - days * 86400
return new Date(1960, 0, 1 + days, 0, 0, seconds)
}
return kind === 'datetime'
? this.parseDatetime(s, order)
: this.parseDate(s, order)
}
/**
* Pre-getdata variant used on the deferred initial load, when DC column
* metadata isn't available. Uses the VA `label` as the column name and infers
* the SAS kind from the value shape, so the clause matches what the metadata
* build later produces (no reconcile reload).
*
* `paramMap` overrides the label-as-column rule for the parameters it names,
* so the very first (already filtered) load also honours an explicit mapping.
*/
buildInitialClauses(msg: VaMessage): QueryClause[] {
buildInitialClauses(
msg: VaMessage,
paramMap?: Map<string, VaParmapTarget>
): QueryClause[] {
const clauses: QueryClause[] = []
for (const param of msg.parameters || []) {
if (!param || param.value === undefined || param.value === null) continue
// The DC column name is the VA `label`; `name` is VA's internal id (pr###).
const varName = (param.label ?? param.name ?? '').toString().trim()
const target = this.mappedTarget(param, paramMap)
// Without a mapping the DC column name is the VA `label`; `name` is VA's
// internal id (pr###). A `va_parmap` entry wins over both.
const varName = (target?.column ?? param.label ?? param.name ?? '')
.toString()
.trim()
if (!varName) continue
const rawValues = Array.isArray(param.value) ? param.value : [param.value]
const rawValues = (
Array.isArray(param.value) ? param.value : [param.value]
) as any[]
const nonEmpty = rawValues.filter(
(v) => v !== undefined && v !== null && v !== ''
)
if (nonEmpty.length === 0) continue
const formatted = nonEmpty
.map((v) =>
this.formatValueByKind(
v,
this.inferKindFromValue(String(v).trim(), param.dataType)
)
)
.filter((v): v is string => v !== null)
if (formatted.length === 0) continue
clauses.push(this.clause(varName, formatted))
// A transform only ever produces dates, so it fixes the kind: inferring it
// from the value would see a formatted date either way, but with the
// transform the column is known to be temporal.
const kind = target?.transform
? ('date' as VaColumnKind)
: this.inferKindFromValue(String(nonEmpty[0]).trim(), param.dataType)
const built = this.buildClauseValueSet(
param,
kind,
undefined,
target?.transform
)
if (!built) continue
clauses.push({
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: varName,
OPERATOR_NM: built.operator,
RAW_VALUE: built.rawValue
})
}
return clauses
}
@@ -141,16 +289,254 @@ export class VaFilterService {
)
}
private clause(varName: string, formatted: string[]): QueryClause {
return {
GROUP_LOGIC: 'AND',
SUBGROUP_LOGIC: 'AND',
SUBGROUP_ID: 0,
VARIABLE_NM: varName,
OPERATOR_NM: formatted.length === 1 ? '=' : 'IN',
RAW_VALUE:
formatted.length === 1 ? formatted[0] : `(${formatted.join(',')})`
/**
* Transform mappings in this message that cannot be applied, one plain-English
* line each: the column is not temporal, or the value is not a date the browser
* can parse. `buildClauseValueSet` skips those parameters, so this is what stops
* the misconfiguration reading as a filter that simply matched nothing.
*
* A missing column is left to `unresolvedMappings`, which already reports it.
*/
unsupportedTransforms(
msg: VaMessage,
paramMap: Map<string, VaParmapTarget> | undefined,
cols: any[]
): string[] {
if (!paramMap || paramMap.size === 0) return []
if (!cols || cols.length === 0) return []
const problems = new Set<string>()
for (const param of msg.parameters || []) {
const target = this.mappedTarget(param, paramMap)
if (!target?.transform) continue
const transform = target.transform
const col = this.findCol(target.column, cols)
if (!col) continue
const kind = this.columnKind(col)
const paramName = (param.label ?? param.name ?? '').toString().trim()
const spec =
target.transform.op === 'ADD'
? `ADD(${target.transform.anchor},${target.transform.offset})`
: `RANGE(${target.transform.period},${target.transform.offset})`
if (kind !== 'date' && kind !== 'datetime') {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${target.column} is not a date or datetime column - the filter was ignored.`
)
continue
}
const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
const sent = (
Array.isArray(param.value) ? param.value : [param.value]
).filter((v: any) => v !== undefined && v !== null && v !== '')
const usable = sent.filter((v: any) => {
const date = this.valueAsDate(v, kind, order)
if (!date) return false
// Same test the clause builder applies, so this count matches what it uses.
return applyVaTransform(date, transform, kind === 'datetime').every(
(resolved) => {
const sasDate = this.helperService.convertJsDateToSasDate(
resolved,
kind === 'datetime' ? 'seconds' : 'days'
)
return Number.isFinite(Number(sasDate))
}
)
})
if (usable.length === 0) {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but its value was not a date - the filter was ignored.`
)
continue
}
// One BETWEEN can only express one period. A multi-select would have to
// become an OR of ranges, which the clause model does not carry, so say so
// rather than building a BETWEEN from whichever value happened to parse.
// Counted on what VA sent: a second value that does not parse must not make
// the single-value case look clean.
if (isRangeTransform(target.transform) && sent.length > 1) {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, which is a range, but the parameter has ${sent.length} values - a range needs one value, so the filter was ignored.`
)
continue
}
if (usable.length < sent.length) {
problems.add(
`VA parameter "${paramName}" maps to ${target.column}:${spec}, but ${sent.length - usable.length} of its ${sent.length} values was not a date - the filter covers only the rest.`
)
}
}
return Array.from(problems)
}
/**
* `va_parmap` entries that this message actually uses but that name a column
* the loaded table does not have, one plain-English line each. `buildClauses`
* skips those parameters (deliberately: falling back to the same-name match
* would silently apply the filter the mapping exists to override), so this is
* what makes the misconfiguration visible to the report author - otherwise the
* grid simply shows every row, which looks like a working filter.
*/
unresolvedMappings(
msg: VaMessage,
paramMap: Map<string, VaParmapTarget> | undefined,
cols: any[],
libds?: string
): string[] {
const problems = new Set<string>()
if (!paramMap || paramMap.size === 0) return []
// No metadata yet is not the same as "the table lacks this column": an empty
// `cols` means the grid is still loading, so there is nothing to judge yet.
// Reporting here would flash a misconfiguration banner during every load.
if (!cols || cols.length === 0) return []
for (const param of msg.parameters || []) {
const mapped = this.mappedColumn(param, paramMap)
if (!mapped || this.findCol(mapped, cols)) continue
const paramName = (param.label ?? param.name ?? '').toString().trim()
problems.add(
`VA parameter "${paramName}" is mapped to ${mapped}, which is not a column of ${
libds || 'this table'
} - the filter was ignored.`
)
}
return Array.from(problems)
}
/**
* `va_parmap` keys that no VA message has named, one plain-English line each.
* A key that matches nothing is a parameter-side typo: the mapping never
* applies and the grid stays unfiltered, which looks the same as a working
* filter.
*
* `seenKeys` must be every key observed across messages so far. `paramsStable`
* says the latest message introduced no new parameter name, i.e. this object's
* parameter set has settled - until then a missing name may simply not have
* arrived, because VA sends only the parameters its query uses.
*/
unmatchedMappings(
paramMap: Map<string, VaParmapTarget> | undefined,
seenKeys: Set<string> | undefined,
paramsStable = false
): string[] {
if (!paramMap || paramMap.size === 0) return []
if (!seenKeys || seenKeys.size === 0) return []
if (!paramsStable) return []
const problems: string[] = []
for (const key of Array.from(paramMap.keys())) {
if (!seenKeys.has(key)) {
problems.push(
`This object's va_parmap names VA parameter "${key}", which VA is not sending - check the spelling.`
)
}
}
return problems
}
/**
* Parameters in this message that filter the same column with *different*
* values, one plain-English line per column. Clauses are AND-ed, so two
* parameters on one column are an intersection: with different values the grid
* comes back empty, which looks like a working filter. Two parameters carrying
* the same value intersect cleanly and filter correctly, so they are not
* reported - a banner there would call a working configuration broken.
*
* Reported rather than rewritten - folding them into one `IN (...)` would
* silently turn an AND into an OR, which is a semantics decision, not a
* diagnostic.
*/
sameColumnMappings(
msg: VaMessage,
paramMap: Map<string, VaParmapTarget> | undefined,
cols?: any[]
): string[] {
if (!paramMap || paramMap.size === 0) return []
// column -> the parameters on it, each with the clause value it produces
const byColumn = new Map<string, { name: string; value: string }[]>()
for (const param of msg.parameters || []) {
const mapped = this.mappedColumn(param, paramMap)
if (!mapped) continue
// A column the table lacks is unresolvedMappings' finding; counting it here
// would report one cause as two symptoms.
const col =
cols && cols.length > 0 ? this.findCol(mapped, cols) : undefined
if (cols && cols.length > 0 && !col) continue
// A parameter that produces no clause cannot conflict with anything.
const value = this.valueSignature(
param,
col,
this.mappedTarget(param, paramMap)?.transform
)
if (value === null) continue
const column = mapped.toUpperCase()
const params = byColumn.get(column) || []
const paramName = (param.label ?? '').toString().trim()
if (paramName && !params.some((p) => p.name === paramName)) {
params.push({ name: paramName, value })
}
byColumn.set(column, params)
}
const problems: string[] = []
for (const [column, params] of Array.from(byColumn.entries())) {
if (params.length < 2) continue
// Same value on every parameter: the AND is a no-op, not an empty grid.
const distinct = new Set(params.map((p) => p.value))
if (distinct.size === 1) continue
problems.push(
`VA parameters ${params
.map((p) => `"${p.name}"`)
.join(
' and '
)} all filter ${column}. Parameters are combined with AND, so rows must match all of them - different values will show no rows. Use one parameter carrying both values to match either.`
)
}
return problems
}
/**
* The DC column an explicit URL mapping (`va_parmap`) assigns to this
* parameter, or undefined when the parameter is not mapped. The key is matched
* case-insensitively against the parameter's `label`.
*/
private mappedTarget(
param: any,
paramMap?: Map<string, VaParmapTarget>
): VaParmapTarget | undefined {
if (!paramMap || paramMap.size === 0) return undefined
// The label only. The internal name (pr273) is an implementation detail of
// the report, and matching both made one parameter answer to two keys: a
// mapping written against the name could never be distinguished from a typo
// once the label was also mapped, and the second entry silently lost.
const key = (param.label ?? '').toString().trim().toLowerCase()
if (!key) return undefined
return paramMap.get(key)
}
/**
* The clause value a parameter would filter its column with, as a comparable
* string, or null when it produces no clause at all. Used to tell an AND of two
* identical values (harmless) from an AND of two different ones (empty grid).
*
* Delegates to the clause builder so a transform's derived dates take part in
* the comparison: two parameters adding the same quarter are the same clause.
*/
private valueSignature(
param: any,
col: any,
transform?: VaTransform
): string | null {
const built = this.buildClauseValueSet(
param,
this.columnKind(col),
this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT),
transform
)
return built ? built.rawValue : null
}
/** The mapped column name, for callers that do not need the transform. */
private mappedColumn(
param: any,
paramMap?: Map<string, VaParmapTarget>
): string | undefined {
return this.mappedTarget(param, paramMap)?.column
}
/** The loaded column spec whose NAME matches varName (case-insensitive). */
@@ -223,15 +609,18 @@ export class VaFilterService {
/**
* Formats one VA filter value into a DC RAW_VALUE for the column kind, or null
* when invalid (dropped). Matches DC's native filters, which send the NUMERIC
* INTERNAL value (not SAS literals — those error out in the backend):
* - numeric -> finite number, raw (else null — injection guard)
* - time -> seconds since midnight (VA: "0:00:02" -> 2)
* - date -> days since 1960-01-01 (VA: "01JUL1960" -> 182)
* - datetime -> seconds since 1960-01-01 (VA: "01JAN1960:00:06:05" -> 365)
* when invalid (dropped). Matches DC's native filters, which send a SAS
* literal for a temporal column and the raw value otherwise:
* - numeric -> finite number, raw (else null - injection guard)
* - time -> seconds since midnight, as a SAS time literal
* (VA: "0:00:02" -> '00:00:02't)
* - date -> days since 1960-01-01, as a SAS date literal
* (VA: "01JUL1960" -> '01JUL1960'd)
* - datetime -> seconds since 1960-01-01, as a SAS datetime literal
* (VA: "01JAN1960:00:06:05" -> '01JAN1960:00:06:05'dt)
* - char -> single-quote escaped
* Temporal strings are regex-parsed (digits/letters/colons only), so a value
* either yields a finite number or is dropped — also the injection guard.
* either yields a finite number or is dropped - also the injection guard.
* Date/datetime reuse HelperService.convertJsDateToSasDate so the value is
* byte-identical to what DC's own pickers produce.
*
@@ -239,7 +628,7 @@ export class VaFilterService {
* itself (VA sends it unformatted when the data item carries no format / is
* used as a measure), so it is passed through raw. This is unambiguous because
* the only date formats in use (DATE9, ISO) always contain a letter or a
* separator — a plain integer is never a formatted date.
* separator - a plain integer is never a formatted date.
*/
private formatValueByKind(
value: any,
@@ -254,23 +643,32 @@ export class VaFilterService {
return Number.isFinite(num) ? `${num}` : null
}
case 'time': {
if (this.isBareNumber(s)) return `${Number(s)}`
const secs = this.timeToSeconds(s)
return secs === null ? null : `${secs}`
const secs = this.isBareNumber(s) ? Number(s) : this.timeToSeconds(s)
return secs === null
? null
: this.helperService.sasTemporalLiteral(secs, 'TIME')
}
case 'date': {
if (this.isBareNumber(s)) return `${Number(s)}`
const d = this.parseDate(s, order)
return d
? `${this.helperService.convertJsDateToSasDate(d, 'days')}`
: null
let days: number | null = null
if (this.isBareNumber(s)) days = Number(s)
else {
const d = this.parseDate(s, order)
if (d) days = this.helperService.convertJsDateToSasDate(d, 'days')
}
return days === null
? null
: this.helperService.sasTemporalLiteral(days, 'DATE')
}
case 'datetime': {
if (this.isBareNumber(s)) return `${Number(s)}`
const d = this.parseDatetime(s, order)
return d
? `${this.helperService.convertJsDateToSasDate(d, 'seconds')}`
: null
let secs: number | null = null
if (this.isBareNumber(s)) secs = Number(s)
else {
const d = this.parseDatetime(s, order)
if (d) secs = this.helperService.convertJsDateToSasDate(d, 'seconds')
}
return secs === null
? null
: this.helperService.sasTemporalLiteral(secs, 'DATETIME')
}
default:
return `'${s.replace(/'/g, "''")}'`
@@ -309,14 +707,42 @@ export class VaFilterService {
return this.parseMonthNameDate(value) ?? this.parseNumericDate(value, order)
}
/** DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62) -> local Date, or null. */
/**
* Month-name date shapes -> local Date, or null. Three families:
* - DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62);
* - word date with the month first, comma optional: "January 1, 2026",
* "Jan 1 2026", "Jan. 1, 2026";
* - word date with the day first: "1 January 2026", "1-Jan-2026".
*
* A date parameter's display format decides which of these the report author
* sees, and VA sends the formatted text, so a report using WORDDATE (or a
* long month name in any order) has to parse here rather than being dropped.
* The month name may be abbreviated or spelled out; only its first three
* letters are matched.
*/
private parseMonthNameDate(value: string): Date | null {
const m = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
if (!m) return null
const [, day, monthAbbr, year] = m
const monthIndex = VaFilterService.SAS_MONTHS.indexOf(
monthAbbr.toUpperCase()
)
const monthByName = (name: string): number =>
VaFilterService.SAS_MONTHS.indexOf(
name.replace(/\./g, '').slice(0, 3).toUpperCase()
)
const date9 = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
if (date9) {
const [, day, monthAbbr, year] = date9
const monthIndex = monthByName(monthAbbr)
if (monthIndex < 0) return null
return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
}
const monthFirst =
/^([A-Za-z]{3,9})\.?[\s-]+(\d{1,2}),?[\s-]+(\d{2}|\d{4})$/.exec(value)
const dayFirst =
/^(\d{1,2})[\s-]+([A-Za-z]{3,9})\.?,?[\s-]+(\d{2}|\d{4})$/.exec(value)
const name = monthFirst ? monthFirst[1] : dayFirst ? dayFirst[2] : null
if (!name) return null
const day = monthFirst ? monthFirst[2] : dayFirst![1]
const year = monthFirst ? monthFirst[3] : dayFirst![3]
const monthIndex = monthByName(name)
if (monthIndex < 0) return null
return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
}
@@ -1,4 +1,6 @@
import { VaMessagingService, VaMessage } from './va-messaging.service'
import { VaFilterService } from './va-filter.service'
import { HelperService } from './helper.service'
// Sample message taken from the official SAS DDC docs (brush column included).
const SAMPLE = {
@@ -51,6 +53,57 @@ describe('VaMessagingService', () => {
})
})
describe('parameter keys (what the va_parmap typo diagnostic is judged on)', () => {
const PARAMS = [
{ name: 'pr273', label: 'Tab_1_Date', value: '2025-03-31' },
{ name: 'pr274', label: 'Tab 2 Region', value: 'north' }
]
const withParams = () => ({ ...SAMPLE, parameters: PARAMS })
it('records each parameter label once, across messages', () => {
service.parseMessage({ data: withParams() } as MessageEvent)
service.parseMessage({ data: withParams() } as MessageEvent)
expect(Array.from(service.seenParamKeys)).toEqual([
'tab_1_date',
'tab 2 region'
])
// The second message introduced no new label, so the set has settled.
expect(service.lastMessageParamsStable).toBeTrue()
})
it('records the label only, so a mapping written against the internal name is reported', () => {
// `pr273` is VA's own bookkeeping. A va_parmap entry written against it
// never matches a parameter, and recording the name here would make the
// typo diagnostic believe VA is sending it - leaving the author with an
// unfiltered grid and no warning at all.
service.parseMessage({ data: withParams() } as MessageEvent)
service.parseMessage({ data: withParams() } as MessageEvent)
const filter = new VaFilterService(new HelperService({} as any))
const seen = service.seenParamKeys
const stable = service.lastMessageParamsStable
expect(
filter.unmatchedMappings(
new Map([['pr273', { column: 'D_DATE' }]]),
seen,
stable
)
).toEqual([
'This object\'s va_parmap names VA parameter "pr273", which VA is not sending - check the spelling.'
])
// The label is a real key, so the same map written that way is silent.
expect(
filter.unmatchedMappings(
new Map([['tab_1_date', { column: 'D_DATE' }]]),
seen,
stable
)
).toEqual([])
})
})
describe('dataColumns', () => {
it('excludes brush columns and keeps original indices', () => {
const msg = service.parseMessage({ data: SAMPLE } as MessageEvent)!
@@ -82,6 +82,26 @@ export class VaMessagingService {
/** DC variable names that should be visible (the rest are hidden). */
visibleColumns: string[] | null = null
/**
* Lower-cased label of every parameter any VA message has carried so far, kept
* across editor reloads. Lets the editor tell a `va_parmap` key that names no
* parameter in the report (a typo, so the filter would silently never apply)
* from one that is merely absent from the current message - VA only sends the
* parameters its query uses.
*
* Labels only: that is the key a mapping is matched on, and recording the
* internal name as well would make a name-keyed entry - which never applies -
* look like a parameter VA is sending.
*/
seenParamKeys = new Set<string>()
/**
* True when every parameter name in the latest message had already been seen.
* VA sends only the parameters a report object's query uses, and sends them per
* object, so a name that is absent from the first messages may simply not have
* arrived yet. A stable parameter set is what makes absence meaningful.
*/
lastMessageParamsStable = false
/** True once the pre-bootstrap early buffer has been drained (drain-once). */
private earlyDrained = false
@@ -187,6 +207,22 @@ export class VaMessagingService {
if (typeof data.resultName !== 'string') return null
if (!Array.isArray(data.columns) || !Array.isArray(data.data)) return null
const parameters = Array.isArray(data.parameters) ? data.parameters : []
const keys: string[] = []
for (const param of parameters) {
// The label only, which is the key a `va_parmap` entry is matched on. The
// internal name (`pr273`) is the report's own bookkeeping: recording it
// here would let the typo diagnostic below believe a name-keyed entry is
// being sent when no code path can ever apply it.
const key = (param?.label ?? '').toString().trim().toLowerCase()
if (key && !keys.includes(key)) keys.push(key)
}
// Judged before the keys are recorded: a message that introduces a new name
// means the set is still growing, so a missing name proves nothing yet.
this.lastMessageParamsStable =
keys.length > 0 && keys.every((key) => this.seenParamKeys.has(key))
for (const key of keys) this.seenParamKeys.add(key)
return {
version: data.version,
resultName: data.resultName,
@@ -194,7 +230,7 @@ export class VaMessagingService {
availableRowCount: data.availableRowCount,
data: data.data,
columns: data.columns,
parameters: Array.isArray(data.parameters) ? data.parameters : []
parameters
}
}
@@ -28,6 +28,16 @@
}
</div>
<div class="modal-footer">
@if (showServiceLauncher(data.sasService)) {
<button
type="button"
class="btn btn-primary"
(click)="openServiceInNewWindow()"
>
Show log
</button>
}
@if (showConfiguratorButton(data.sasService)) {
<button
type="button"
@@ -0,0 +1,71 @@
import { InfoModalComponent } from './info-modal.component'
describe('InfoModalComponent', () => {
let component: InfoModalComponent
let sasService: any
const buildComponent = (sasServiceConfig: any = {}) => {
sasService = {
getSasjsConfig: () => sasServiceConfig,
openServiceInNewWindow: jasmine.createSpy('openServiceInNewWindow')
}
component = new InfoModalComponent(
{} as any,
{} as any,
sasService,
{} as any
)
component.data = {
modalTitle: 'Error',
message: 'startupservice failed',
sasService: 'public/startupservice',
details: null
}
component.ngOnInit()
}
describe('showServiceLauncher', () => {
beforeEach(() => buildComponent())
it('is shown for the startupservice, on every server type', () => {
expect(component.showServiceLauncher('public/startupservice')).toBeTrue()
expect(
component.showServiceLauncher('services/public/startupservice')
).toBeTrue()
})
it('is not shown for other services or a missing service', () => {
expect(component.showServiceLauncher('public/getdata')).toBeFalse()
expect(component.showServiceLauncher('')).toBeFalse()
expect(component.showServiceLauncher(null)).toBeFalse()
})
})
describe('openServiceInNewWindow', () => {
it('runs the startupservice in a new window', () => {
buildComponent({ serverType: 'SASVIYA' })
component.openServiceInNewWindow()
expect(sasService.openServiceInNewWindow).toHaveBeenCalledWith(
'public/startupservice',
{}
)
})
it('passes the configured Viya context so the log comes from it', () => {
buildComponent({
serverType: 'SASVIYA',
contextName: 'SAS Job Execution compute context'
})
component.openServiceInNewWindow()
expect(sasService.openServiceInNewWindow).toHaveBeenCalledWith(
'public/startupservice',
{ _contextname: 'SAS Job Execution compute context' }
)
})
})
})
@@ -71,6 +71,34 @@ export class InfoModalComponent implements OnInit {
this.onConfirmModalClick.emit()
}
/**
* Whether to show the `Show log` button.
*
* The startupservice failure modal is a dead end otherwise: the adapter
* returned no log, so the only way to see the backend reason is to run the
* service through the JES / STP web app in a browser tab. Offered on every
* server type - SAS 9, Viya and SASjs Server all accept the `_program` URL.
*/
showServiceLauncher(sasService: string | null) {
return !!sasService?.includes('startupservice')
}
/**
* Runs the startupservice in a new browser window so the SAS log is
* available even when the adapter response carried none (e.g. a Viya
* execution task that timed out).
*/
openServiceInNewWindow() {
const sasjsConfig = this.sasService.getSasjsConfig()
const params: { [name: string]: string } = {}
if (sasjsConfig.contextName) {
params['_contextname'] = sasjsConfig.contextName
}
this.sasService.openServiceInNewWindow(this.data.sasService || '', params)
}
/**
* Only on SAS9, opening a backend configurator/deploy page
*/
@@ -61,13 +61,13 @@ export class DcValidator {
// injected separately — see its own doc comment for why.
//
// COLHEADERS is the authoritative grid order and the only safe key to
// build against: it is the variable list of the *output* dataset, where
// getdata.sas hoists the delete column and the PK to the front ("order
// delete var and pk fields at start of table"), whereas cols[].VARNUM
// describes the *source* table and cols[] itself can arrive sorted by
// NAME. Those orders diverge whenever the PK isn't already the source's
// first column, so rules are keyed by name and emitted in COLHEADERS
// order — never paired positionally against cols[] or VARNUM.
// build against: it is the variable list of the *output* dataset, and the
// editor prepends the client-owned delete column to it (see
// editor.component.ts's initSetup) before this constructor runs, whereas
// cols[].VARNUM describes the *source* table and cols[] itself can arrive
// sorted by NAME. Those orders diverge whenever the PK isn't already the
// source's first column, so rules are keyed by name and emitted in
// COLHEADERS order — never paired positionally against cols[] or VARNUM.
//
// A column with no parseable COLTYPE falls back to a plain text rule
// rather than being dropped: getRules() is consumed positionally against
@@ -0,0 +1,75 @@
import {
prepareDeleteRecordColumn,
seedDeleteRecordColumn
} from './deleteRecordColumn'
import { DELETE_RECORD_COLUMN_NAME } from './deleteRecordColumnRule'
const response = (overrides: any = {}) => ({
data: {
sasparams: [{ COLHEADERS: 'PRIMARY_KEY_FIELD,SOME_CHAR' }],
cols: [],
$sasdata: { vars: {} },
...overrides
}
})
describe('prepareDeleteRecordColumn', () => {
it('prepends the delete column to COLHEADERS', () => {
const res = response()
prepareDeleteRecordColumn(res)
expect(res.data.sasparams[0].COLHEADERS).toEqual(
`${DELETE_RECORD_COLUMN_NAME},PRIMARY_KEY_FIELD,SOME_CHAR`
)
})
it('throws when COLHEADERS already carries the delete column', () => {
const res = response({
sasparams: [{ COLHEADERS: `${DELETE_RECORD_COLUMN_NAME},SOME_CHAR` }]
})
expect(() => prepareDeleteRecordColumn(res)).toThrowError(/reserved column/)
})
it('throws when cols carries the delete column', () => {
const res = response({ cols: [{ NAME: DELETE_RECORD_COLUMN_NAME }] })
expect(() => prepareDeleteRecordColumn(res)).toThrowError(/reserved column/)
})
it('throws when $sasdata vars carries the delete column', () => {
const res = response({
$sasdata: { vars: { [DELETE_RECORD_COLUMN_NAME]: {} } }
})
expect(() => prepareDeleteRecordColumn(res)).toThrowError(/reserved column/)
})
it('leaves COLHEADERS untouched when it throws', () => {
const colHeaders = `${DELETE_RECORD_COLUMN_NAME},SOME_CHAR`
const res = response({ sasparams: [{ COLHEADERS: colHeaders }] })
expect(() => prepareDeleteRecordColumn(res)).toThrowError()
expect(res.data.sasparams[0].COLHEADERS).toEqual(colHeaders)
})
})
describe('seedDeleteRecordColumn', () => {
it("sets every row's delete flag to 'No'", () => {
const rows: any[] = [{ PRIMARY_KEY_FIELD: 1 }, { PRIMARY_KEY_FIELD: 2 }]
seedDeleteRecordColumn(rows)
expect(rows[0][DELETE_RECORD_COLUMN_NAME]).toEqual('No')
expect(rows[1][DELETE_RECORD_COLUMN_NAME]).toEqual('No')
})
it('overwrites a flag a row already carried', () => {
const rows: any[] = [{ [DELETE_RECORD_COLUMN_NAME]: 'Yes' }]
seedDeleteRecordColumn(rows)
expect(rows[0][DELETE_RECORD_COLUMN_NAME]).toEqual('No')
})
})
@@ -0,0 +1,64 @@
import { Col } from '../models/col.model'
import { DELETE_RECORD_COLUMN_NAME } from './deleteRecordColumnRule'
/**
* The parts of a getdata response this module touches. Declared structurally
* so the editor and the multi-dataset loader can both use it without either
* owning the other's response type.
*/
interface DeleteColumnResponse {
data: {
sasparams: { COLHEADERS: string }[]
cols?: Col[]
$sasdata?: { vars?: Record<string, unknown> }
}
}
/**
* Asserts that getdata did NOT send the client-owned delete column, then
* prepends it to COLHEADERS so the grid's first column is the delete flag.
*
* `_____DELETE__THIS__RECORD_____` is created by the editor itself (see
* deleteRecordColumnRule.ts) and getdata must never emit it. Its presence in a
* response can only mean a real table column of that name, or a hook that
* re-added it - either way it would shadow the delete flag, so refuse the load
* rather than render a table whose delete column cannot be trusted. Callers
* run this inside a promise chain whose `.catch` surfaces the error (the
* editor sets `getdataError`).
*
* Call it BEFORE constructing a DcValidator or deriving headerArray: rules,
* `columnHeader` and `headerArray` are all keyed off COLHEADERS order, and the
* delete column is always index 0.
*/
export const prepareDeleteRecordColumn = (response: DeleteColumnResponse) => {
const sasparams = response.data.sasparams[0]
const sentByBackend =
sasparams.COLHEADERS.split(',').includes(DELETE_RECORD_COLUMN_NAME) ||
(response.data.cols || []).some(
(col) => col.NAME === DELETE_RECORD_COLUMN_NAME
) ||
Object.prototype.hasOwnProperty.call(
response.data.$sasdata?.vars || {},
DELETE_RECORD_COLUMN_NAME
)
if (sentByBackend) {
throw new Error(
`getdata returned the reserved column ${DELETE_RECORD_COLUMN_NAME}. ` +
'That name is created by the editor itself and must not exist in the target table.'
)
}
sasparams.COLHEADERS = `${DELETE_RECORD_COLUMN_NAME},${sasparams.COLHEADERS}`
}
/**
* Seeds every row's delete flag to 'No' - getdata no longer sends the column,
* so the editor owns the value too (see deleteRecordColumnRule.ts).
*/
export const seedDeleteRecordColumn = (rows: any[]) => {
for (const row of rows) {
row[DELETE_RECORD_COLUMN_NAME] = 'No'
}
}
@@ -1,16 +1,21 @@
import { DcValidation } from '../models/dc-validation.model'
/**
* `_____DELETE__THIS__RECORD_____` is a client-side UI concept (the delete
* checkbox column) that getdata.sas synthesizes into vars3/vars4 for display
* purposes only — it's never a real column of the underlying dataset, so
* `%mp_getcols` (which builds `cols1`, the left side of the join into
* `cols`) never has a row for it, and its rule can no longer travel via
* `cols[].COLTYPE` (see issue #253's vertical-array fix). The rule is fixed
* and data-independent, so it's injected directly here instead.
* `_____DELETE__THIS__RECORD_____` is a purely client-owned column - the
* delete checkbox column. getdata.sas does NOT emit it: the editor adds the
* name to COLHEADERS itself (see editor.component.ts's initSetup) and seeds
* every row to 'No'. A response that carries it is refused, because the name
* can then only come from a real table column of that name (or a hook that
* re-added it), which would silently shadow the delete flag.
*
* Its rule can never travel via `cols[].COLTYPE` - `%mp_getcols` (which
* builds `cols1`, the left side of the join into `cols`) has no row for it -
* so it is injected directly, here and in DcValidator's constructor.
*/
export const DELETE_RECORD_COLUMN_NAME = '_____DELETE__THIS__RECORD_____'
export const DELETE_RECORD_COLUMN_RULE: DcValidation = {
data: '_____DELETE__THIS__RECORD_____',
data: DELETE_RECORD_COLUMN_NAME,
type: 'dropdown',
source: ['No', 'Yes']
}
@@ -0,0 +1,215 @@
import { parseVaParmap } from './parse-va-parmap'
describe('parseVaParmap', () => {
it('returns an empty map and no problems when the parameter is absent', () => {
for (const query of ['', 'embed=va', 'embed=va&labels=true']) {
const parsed = parseVaParmap(query)
expect(parsed.map.size).toBe(0)
expect(parsed.problems).toEqual([])
}
})
it('maps a VA parameter label to a DC column', () => {
const { map, problems } = parseVaParmap(
'embed=va&va_parmap=REGION_TAB1-REGION'
)
expect(map.get('region_tab1')?.column).toBe('REGION')
expect(map.size).toBe(1)
expect(problems).toEqual([])
})
it('collects repeated entries, so two parameters can map to one column', () => {
// The multi-tab case: each DC instance filters the same DC column from a
// differently-named VA parameter.
const { map } = parseVaParmap(
'embed=va&va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION'
)
expect(map.get('region_tab1')?.column).toBe('REGION')
expect(map.get('region_tab2')?.column).toBe('REGION')
expect(map.size).toBe(2)
})
it('is case-insensitive on the key and trims both sides', () => {
const { map } = parseVaParmap('va_parmap=%20Region_Tab1%20-%20region%20')
expect(map.get('region_tab1')?.column).toBe('region')
})
it('decodes a key containing a space (VA labels are free text)', () => {
expect(
parseVaParmap('va_parmap=My+Region-DEPT').map.get('my region')?.column
).toBe('DEPT')
})
it('splits at the LAST hyphen, so a label may contain hyphens', () => {
expect(
parseVaParmap('va_parmap=Sales-2024-DEPT').map.get('sales-2024')?.column
).toBe('DEPT')
})
describe('entries that cannot be used', () => {
it('reports an entry with no separator', () => {
const { map, problems } = parseVaParmap('va_parmap=DEPT')
expect(map.size).toBe(0)
expect(problems).toEqual([
'va_parmap "DEPT" was ignored: it needs a "-" between the VA parameter and the column name.'
])
})
it('reports an entry with no parameter name', () => {
const { map, problems } = parseVaParmap('va_parmap=-DEPT')
expect(map.size).toBe(0)
expect(problems).toEqual([
'va_parmap "-DEPT" was ignored: the VA parameter name is missing.'
])
})
it('reports an entry with no column name', () => {
const { map, problems } = parseVaParmap('va_parmap=REGION_TAB1-')
expect(map.size).toBe(0)
expect(problems).toEqual([
'va_parmap "REGION_TAB1-" was ignored: "" is not a valid SAS column name.'
])
})
it('reports a column that is not a plain SAS name (injection guard)', () => {
// The column is interpolated into a SAS WHERE clause, so anything beyond a
// SAS name is refused at parse time rather than carried into a clause.
for (const entry of [
'A-x;drop',
'A-REGION%20OR%201=1',
"A-RE'GION",
'A-1ABC'
]) {
const { map, problems } = parseVaParmap(`va_parmap=${entry}`)
expect(map.size).toBe(0)
expect(problems.length).toBe(1)
expect(problems[0]).toContain('is not a valid SAS column name')
}
expect(parseVaParmap('va_parmap=A-A_B9').map.get('a')?.column).toBe(
'A_B9'
)
})
it('keeps the usable entries and reports only the bad one', () => {
const { map, problems } = parseVaParmap(
'va_parmap=GOOD-REGION&va_parmap=BROKEN'
)
expect(map.get('good')?.column).toBe('REGION')
expect(problems.length).toBe(1)
expect(problems[0]).toContain('va_parmap "BROKEN"')
})
})
describe('transforms', () => {
it('parses an ADD anchor with an offset', () => {
const { map, problems } = parseVaParmap(
'va_parmap=SELECTED-D_DATE:ADD(QUARTER_START,-1)'
)
expect(problems).toEqual([])
expect(map.get('selected')?.column).toBe('D_DATE')
expect(map.get('selected')?.transform).toEqual({
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
})
it('defaults the offset to zero and accepts lower case', () => {
const { map, problems } = parseVaParmap(
'va_parmap=sel-d_date:add(quarter_start)'
)
expect(problems).toEqual([])
expect(map.get('sel')?.transform).toEqual({
op: 'ADD',
anchor: 'QUARTER_START',
offset: 0
})
})
it('parses a RANGE period', () => {
const { map, problems } = parseVaParmap(
'va_parmap=sel-D_DATE:RANGE(quarter,-1)'
)
expect(problems).toEqual([])
expect(map.get('sel')?.transform).toEqual({
op: 'RANGE',
period: 'QUARTER',
offset: -1
})
})
it('leaves a mapping without a transform as a plain column', () => {
const { map } = parseVaParmap('va_parmap=sel-D_DATE')
expect(map.get('sel')?.column).toBe('D_DATE')
expect(map.get('sel')?.transform).toBeUndefined()
})
it('reports an unknown operation, anchor, period or offset', () => {
const cases: [string, string][] = [
['sel-D_DATE:SHIFT(QUARTER_START,-1)', 'is not a known transform'],
['sel-D_DATE:ADD(FORTNIGHT_START,-1)', 'is not a known anchor'],
['sel-D_DATE:RANGE(DECADE,-1)', 'is not a known period'],
['sel-D_DATE:ADD(QUARTER_START,3.5)', 'is not a whole number'],
['sel-D_DATE:ADD', 'is not a transform']
]
for (const [entry, expected] of cases) {
const { map, problems } = parseVaParmap(`va_parmap=${entry}`)
expect(`${entry}: ${map.size}`).toBe(`${entry}: 0`)
expect(`${entry}: ${problems.length}`).toBe(`${entry}: 1`)
expect(problems[0]).toContain(expected)
}
})
it('keeps the parameter label free to contain a colon', () => {
// The key side is untouched by the transform grammar: only the column side
// is split, so a label with a colon still maps.
const { map, problems } = parseVaParmap(
'va_parmap=Sales:2026-D_DATE:ADD(QUARTER_START,-1)'
)
expect(problems).toEqual([])
expect(map.get('sales:2026')?.column).toBe('D_DATE')
expect(map.get('sales:2026')?.transform).toEqual({
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
})
})
describe('one parameter, one column', () => {
it('keeps the first mapping and reports a conflicting duplicate', () => {
const { map, problems } = parseVaParmap(
'va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER'
)
expect(map.get('dept')?.column).toBe('REGION')
expect(problems).toEqual([
'va_parmap "DEPT-OTHER" was ignored: VA parameter "dept" already maps to REGION, and a parameter can filter only one column.'
])
})
it('treats a duplicate key as a duplicate across case', () => {
const { map, problems } = parseVaParmap(
'va_parmap=Dept-REGION&va_parmap=DEPT-SOMETHING_ELSE'
)
expect(map.get('dept')?.column).toBe('REGION')
expect(problems.length).toBe(1)
})
it('accepts the same mapping written twice (no ambiguity)', () => {
const { map, problems } = parseVaParmap(
'va_parmap=DEPT-REGION&va_parmap=dept-region'
)
expect(map.get('dept')?.column).toBe('REGION')
expect(problems).toEqual([])
})
it('still allows two parameters to map to one column', () => {
// The multi-object case: different parameters, same DC column.
const { map, problems } = parseVaParmap(
'va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION'
)
expect(map.size).toBe(2)
expect(problems).toEqual([])
})
})
})
@@ -0,0 +1,192 @@
import {
VA_ANCHORS,
VA_PERIODS,
VaAnchor,
VaPeriod,
VaTransform
} from './va-parmap-transform'
/**
* What a VA parameter's mapping resolves to: the DC column, and optionally a
* transform that adds derived dates to the filter (e.g. the previous quarter's
* row alongside the selected one).
*/
export interface VaParmapTarget {
column: string
transform?: VaTransform
}
/** Result of `parseVaParmap` - the usable mappings, and what was ignored. */
export interface VaParmap {
/** Lower-cased VA parameter key -> DC column (and transform, if any). */
map: Map<string, VaParmapTarget>
/** One plain-English line per `va_parmap` entry that was ignored. */
problems: string[]
}
/**
* Parses the `va_parmap` entries out of a hash query string (the part after `?`
* in `window.location.hash`).
*
* Format - one mapping per occurrence, each `<VA parameter key>-<DC column>`:
*
* ?embed=va&va_parmap=REGION_TAB1-REGION&va_parmap=REGION_TAB2-REGION
*
* Repeating the parameter (rather than packing a comma-separated list into one
* value) keeps the grammar unambiguous: VA parameter labels are free text, so
* any list separator we picked could also appear inside a label.
*
* Each entry is split at its LAST `-`. A SAS column name can never contain a
* hyphen, so the final one always delimits the column, and a label that does
* contain hyphens (e.g. `Sales-2024`) still maps correctly.
*
* The key is matched case-insensitively, and trailing/leading blanks are
* trimmed, against a VA parameter's `label`, which is the same key the
* label->column match uses. The internal `name` (e.g. `pr273`) is not a key: it
* is the report's own bookkeeping, so an entry written against it never applies
* and is reported by `unmatchedMappings`.
*
* An entry that cannot be used is reported in `problems` rather than dropped in
* silence: these entries are hand-written in the report's DDC URL, usually by
* the report author, and a typo would otherwise degrade to "no filter" - which
* looks exactly like a working filter over the whole table.
*
* Returns an empty map when the parameter is absent, which leaves the default
* same-name resolution untouched (existing embeds are unaffected).
*
* Modeled on `parseEmbedParam` / `parseLabelsParam`, which read the same hash
* query.
*/
export function parseVaParmap(hashQuery: string): VaParmap {
const map = new Map<string, VaParmapTarget>()
const problems: string[] = []
if (!hashQuery) return { map, problems }
const entries = new URLSearchParams(hashQuery).getAll('va_parmap')
for (const entry of entries) {
// A transform sits at the end, in parentheses: `<param>-<column>:OP(args)`.
// It is lifted off FIRST, because its arguments may contain a hyphen (a
// negative offset) and the key/column split below uses the hyphen.
let body = entry ?? ''
let transformSpec = ''
if (body.trim().endsWith(')')) {
const open = body.lastIndexOf('(')
const colon = open > 0 ? body.lastIndexOf(':', open) : -1
if (colon > 0) {
transformSpec = body.slice(colon + 1).trim()
body = body.slice(0, colon)
}
}
// Last hyphen delimits the DC column (see above).
const separator = body.lastIndexOf('-')
if (separator < 0) {
problems.push(
`va_parmap "${entry}" was ignored: it needs a "-" between the VA parameter and the column name.`
)
continue
}
const key = body.slice(0, separator).trim().toLowerCase()
const column = body.slice(separator + 1).trim()
if (!key) {
problems.push(
`va_parmap "${entry}" was ignored: the VA parameter name is missing.`
)
continue
}
if (column.includes(':')) {
problems.push(
`va_parmap "${entry}" was ignored: "${column}" is not a transform - expected e.g. ADD(QUARTER_START,-1).`
)
continue
}
// The column name is interpolated into a SAS WHERE clause, so only a plain
// SAS name is accepted - anything else is refused rather than mapped.
if (!/^[A-Za-z_][A-Za-z0-9_]{0,31}$/.test(column)) {
problems.push(
`va_parmap "${entry}" was ignored: "${column}" is not a valid SAS column name.`
)
continue
}
const transform = transformSpec
? parseTransform(transformSpec, entry, problems)
: undefined
if (transformSpec && !transform) continue
// One VA parameter filters one column. A second entry for the same parameter
// is ambiguous, so the FIRST one wins and the conflict is reported - the
// alternative (last one wins, silently) would apply a filter the author
// never sees the origin of. Two parameters mapping to the same column is
// fine and is how two report objects filter one column from their own
// parameters.
const existing = map.get(key)
if (existing) {
if (existing.column.toUpperCase() !== column.toUpperCase()) {
problems.push(
`va_parmap "${entry}" was ignored: VA parameter "${key}" already maps to ${existing.column}, and a parameter can filter only one column.`
)
}
continue
}
map.set(key, { column, transform })
}
return { map, problems }
}
/**
* Parses `OP(arg[, arg])`, returning undefined (and pushing a problem line) when
* the operation or one of its arguments is not in the allow-list.
*
* ADD takes an anchor and an optional offset; RANGE takes a period and an
* optional offset. The offset is whole periods, negative for earlier.
*/
function parseTransform(
spec: string,
entry: string,
problems: string[]
): VaTransform | undefined {
const match = /^([A-Za-z]+)\s*\(([^)]*)\)$/.exec(spec)
if (!match) {
problems.push(
`va_parmap "${entry}" was ignored: "${spec}" is not a transform - expected e.g. ADD(QUARTER_START,-1).`
)
return undefined
}
const op = match[1].toUpperCase()
const args = match[2]
.split(',')
.map((arg) => arg.trim())
.filter((arg) => arg.length > 0)
const offsetArg = args[1] === undefined ? '0' : args[1]
if (!/^-?\d+$/.test(offsetArg)) {
problems.push(
`va_parmap "${entry}" was ignored: the offset "${offsetArg}" is not a whole number of periods.`
)
return undefined
}
const offset = Number(offsetArg)
if (op === 'ADD') {
const anchor = (args[0] || '').toUpperCase()
if (!VA_ANCHORS.includes(anchor as VaAnchor)) {
problems.push(
`va_parmap "${entry}" was ignored: "${args[0] || ''}" is not a known anchor (${VA_ANCHORS.join(', ')}).`
)
return undefined
}
return { op: 'ADD', anchor: anchor as VaAnchor, offset }
}
if (op === 'RANGE') {
const period = (args[0] || '').toUpperCase()
if (!VA_PERIODS.includes(period as VaPeriod)) {
problems.push(
`va_parmap "${entry}" was ignored: "${args[0] || ''}" is not a known period (${VA_PERIODS.join(', ')}).`
)
return undefined
}
return { op: 'RANGE', period: period as VaPeriod, offset }
}
problems.push(
`va_parmap "${entry}" was ignored: "${match[1]}" is not a known transform (ADD, RANGE).`
)
return undefined
}
@@ -0,0 +1,86 @@
import { buildServiceUrl } from './service-url'
describe('buildServiceUrl', () => {
const viyaConfig = {
serverUrl: 'https://sas.example.com',
appLoc: '/Public/app/dc',
executionPath: '/SASJobExecution',
service: 'public/startupservice'
}
it('builds the Viya JES URL for a service, with the log enabled', () => {
expect(buildServiceUrl(viyaConfig)).toBe(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
)
})
it('leaves serverUrl empty on SAS 9 so the browser resolves the origin', () => {
expect(
buildServiceUrl({
...viyaConfig,
serverUrl: '',
executionPath: '/SASStoredProcess/do'
})
).toBe(
'/SASStoredProcess/do/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
)
})
it('builds the SASjs Server URL from its execution path', () => {
expect(
buildServiceUrl({
...viyaConfig,
executionPath: '/SASjsApi/stp/execute'
})
).toBe(
'https://sas.example.com/SASjsApi/stp/execute/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
)
})
it('appends encoded parameters before the debug flag', () => {
expect(
buildServiceUrl({
...viyaConfig,
params: { _contextname: 'SAS Job Execution compute context' }
})
).toBe(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_contextname=SAS%20Job%20Execution%20compute%20context&_debug=131'
)
})
it('skips parameters with no value', () => {
expect(
buildServiceUrl({
...viyaConfig,
params: { _contextname: undefined, admin: null, dcpath: 'sas' }
})
).toBe(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&dcpath=sas&_debug=131'
)
})
it('omits the debug flag when debug is off', () => {
expect(buildServiceUrl({ ...viyaConfig, debug: false })).toBe(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice'
)
})
it('accepts a service with a leading slash', () => {
expect(
buildServiceUrl({ ...viyaConfig, service: '/public/startupservice' })
).toBe(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
)
})
it('accepts the services/ prefix the request wrapper reports', () => {
expect(
buildServiceUrl({
...viyaConfig,
service: 'services/public/startupservice'
})
).toBe(
'https://sas.example.com/SASJobExecution/?_program=/Public/app/dc/services/public/startupservice&_debug=131'
)
})
})
@@ -0,0 +1,46 @@
/**
* Builds the URL that runs a SAS service directly in a browser window.
*
* The adapter submits requests from JavaScript, and on Viya that can route the
* run through the Compute service (`_executionTasks=true`), where the SAS log
* is not returned to the caller and the request can time out before the
* service finishes. A browser request to the JES / STP web app runs the same
* `_program` without any task involved, and `_debug=131` returns the SAS log
* as the page body - which is what makes it useful when a service fails.
*
* `serverUrl` is left empty on SAS 9 when the app is served from the SAS
* server itself; the browser then resolves the path against the current
* origin.
*/
export interface ServiceUrlOptions {
serverUrl: string
appLoc: string
executionPath: string
service: string
params?: { [name: string]: string | number | null | undefined }
debug?: boolean
}
export const buildServiceUrl = ({
serverUrl,
appLoc,
executionPath,
service,
params = {},
debug = true
}: ServiceUrlOptions): string => {
// Callers pass the service as the app knows it: `public/getdata` in the
// request path, but the request wrapper reports `services/public/getdata`
// on a failed request. Both resolve to the same program.
const servicePath = service.replace(/^\/+/, '').replace(/^services\//, '')
const paramString = Object.entries(params)
.filter(([, value]) => value !== null && value !== undefined)
.map(
([name, value]) =>
`&${encodeURIComponent(name)}=${encodeURIComponent(String(value))}`
)
.join('')
const debugParam = debug ? '&_debug=131' : ''
return `${serverUrl}${executionPath}/?_program=${appLoc}/services/${servicePath}${paramString}${debugParam}`
}
@@ -0,0 +1,194 @@
import {
applyVaTransform,
isRangeTransform,
VA_ANCHORS,
VA_PERIODS,
VaTransform
} from './va-parmap-transform'
/** Local y-m-d, so assertions do not depend on the runner's timezone offset. */
const ymd = (d: Date) =>
`${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(
d.getDate()
).padStart(2, '0')}`
const date = (y: number, m: number, d: number) => new Date(y, m - 1, d)
describe('va_parmap transforms', () => {
it('exposes only allow-listed anchors and periods', () => {
expect(VA_ANCHORS).toContain('QUARTER_START')
expect(VA_PERIODS).toEqual(['QUARTER', 'MONTH', 'YEAR'])
})
describe('ADD - the value plus one anchor beside it', () => {
it('adds the previous quarter start, which is the reported requirement', () => {
// intnx('qtr', dt, -1, 'b') for a value on a quarter boundary
const dates = applyVaTransform(date(2026, 10, 1), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2026-10-01', '2026-07-01'])
})
it('resolves a mid-quarter value to its own value plus the previous quarter start', () => {
const dates = applyVaTransform(date(2026, 8, 15), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2026-08-15', '2026-04-01'])
})
it('de-duplicates when the value already sits on the anchor', () => {
// 2026-01-01 is a quarter start, so "0 quarters back" adds nothing
const dates = applyVaTransform(date(2026, 1, 1), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: 0
})
expect(dates.map(ymd)).toEqual(['2026-01-01'])
})
it('resolves the other anchors at their boundaries', () => {
const at = date(2026, 8, 15)
expect(
ymd(
applyVaTransform(at, {
op: 'ADD',
anchor: 'QUARTER_END',
offset: 0
})[1]
)
).toBe('2026-09-30')
expect(
ymd(
applyVaTransform(at, { op: 'ADD', anchor: 'MONTH_END', offset: 0 })[1]
)
).toBe('2026-08-31')
expect(
ymd(
applyVaTransform(at, {
op: 'ADD',
anchor: 'YEAR_START',
offset: -1
})[1]
)
).toBe('2025-01-01')
expect(
ymd(
applyVaTransform(at, { op: 'ADD', anchor: 'YEAR_END', offset: 0 })[1]
)
).toBe('2026-12-31')
})
it('handles February and leap years at month end', () => {
expect(
ymd(
applyVaTransform(date(2026, 3, 1), {
op: 'ADD',
anchor: 'MONTH_END',
offset: -1
})[1]
)
).toBe('2026-02-28')
expect(
ymd(
applyVaTransform(date(2024, 3, 15), {
op: 'ADD',
anchor: 'MONTH_END',
offset: -1
})[1]
)
).toBe('2024-02-29')
})
it('crosses a year boundary when the offset takes it out of the year', () => {
const dates = applyVaTransform(date(2027, 1, 1), {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2027-01-01', '2026-10-01'])
})
})
describe('RANGE - a closed period', () => {
it('covers the previous quarter', () => {
const dates = applyVaTransform(date(2026, 8, 15), {
op: 'RANGE',
period: 'QUARTER',
offset: -1
})
expect(dates.map(ymd)).toEqual(['2026-04-01', '2026-06-30'])
})
it('covers the current month and the previous year', () => {
expect(
applyVaTransform(date(2026, 8, 15), {
op: 'RANGE',
period: 'MONTH',
offset: 0
}).map(ymd)
).toEqual(['2026-08-01', '2026-08-31'])
expect(
applyVaTransform(date(2026, 8, 15), {
op: 'RANGE',
period: 'YEAR',
offset: -1
}).map(ymd)
).toEqual(['2025-01-01', '2025-12-31'])
})
it('reports itself as a range so the caller picks BETWEEN', () => {
expect(
isRangeTransform({ op: 'RANGE', period: 'QUARTER', offset: 0 })
).toBe(true)
expect(
isRangeTransform({
op: 'ADD',
anchor: 'QUARTER_START',
offset: 0
} as VaTransform)
).toBe(false)
})
})
describe('time of day', () => {
it('gives a datetime range whole days, not the value clock time', () => {
// A BETWEEN that starts at 13:45 on day one loses the earlier part of that
// day, and one that ends at 13:45 on the last day loses the rest of it.
const value = new Date(2026, 7, 15, 13, 45, 30)
const [start, end] = applyVaTransform(
value,
{ op: 'RANGE', period: 'QUARTER', offset: 0 },
true
)
expect([
start.getHours(),
start.getMinutes(),
start.getSeconds()
]).toEqual([0, 0, 0])
expect([end.getHours(), end.getMinutes(), end.getSeconds()]).toEqual([
23, 59, 59
])
expect(ymd(start)).toBe('2026-07-01')
expect(ymd(end)).toBe('2026-09-30')
})
it('keeps the time for datetime columns and drops it otherwise', () => {
const value = new Date(2026, 9, 1, 13, 45, 30)
const add: VaTransform = {
op: 'ADD',
anchor: 'QUARTER_START',
offset: -1
}
const kept = applyVaTransform(value, add, true)[1]
expect([kept.getHours(), kept.getMinutes(), kept.getSeconds()]).toEqual([
13, 45, 30
])
const dropped = applyVaTransform(value, add, false)[1]
expect([dropped.getHours(), dropped.getMinutes()]).toEqual([0, 0])
})
})
})
@@ -0,0 +1,151 @@
/**
* Anchors and periods a `va_parmap` transform can resolve a report value to.
*
* The names describe the result rather than mirroring SAS `intnx`, because the
* arithmetic runs in the browser: emulating `intnx`'s alignment letters (`b`,
* `e`, `s`, `m`, never mind week alignment and fiscal quarters) would invite
* assumptions we could not honour. Each anchor below is one small function, so
* adding one is a function plus a test row.
*/
export const VA_ANCHORS = [
'QUARTER_START',
'QUARTER_END',
'MONTH_START',
'MONTH_END',
'YEAR_START',
'YEAR_END'
] as const
export type VaAnchor = (typeof VA_ANCHORS)[number]
export const VA_PERIODS = ['QUARTER', 'MONTH', 'YEAR'] as const
export type VaPeriod = (typeof VA_PERIODS)[number]
/**
* A parsed transform: the operation plus its validated arguments.
*
* `ADD` yields the value and one anchor beside it (a list of dates), which is
* how "this period's row plus the previous quarter's" is expressed. `RANGE`
* yields a closed period (two dates) for a BETWEEN.
*/
export type VaTransform =
| { op: 'ADD'; anchor: VaAnchor; offset: number }
| { op: 'RANGE'; period: VaPeriod; offset: number }
/** True when the transform produces a closed range rather than a list. */
export function isRangeTransform(transform: VaTransform): boolean {
return transform.op === 'RANGE'
}
/** Local midnight on the first day of the month `offset` months from `date`. */
function monthStart(date: Date, offset: number): Date {
return new Date(date.getFullYear(), date.getMonth() + offset, 1)
}
/** Local midnight on the last day of the month `offset` months from `date`. */
function monthEnd(date: Date, offset: number): Date {
return new Date(date.getFullYear(), date.getMonth() + offset + 1, 0)
}
/** First day of the quarter `offset` quarters from the one containing `date`. */
function quarterStart(date: Date, offset: number): Date {
const firstMonthOfQuarter = 3 * Math.floor(date.getMonth() / 3)
return new Date(date.getFullYear(), firstMonthOfQuarter + offset * 3, 1)
}
/** Last day of the quarter `offset` quarters from the one containing `date`. */
function quarterEnd(date: Date, offset: number): Date {
// Day 0 of a month is the last day of the month before it: calendar
// construction, so a daylight-saving transition inside the quarter cannot
// shift the result by an hour the way subtracting 24 hours would.
const nextStart = quarterStart(date, offset + 1)
return new Date(nextStart.getFullYear(), nextStart.getMonth(), 0)
}
/** First day of the year `offset` years from the one containing `date`. */
function yearStart(date: Date, offset: number): Date {
return new Date(date.getFullYear() + offset, 0, 1)
}
/** Last day of the year `offset` years from the one containing `date`. */
function yearEnd(date: Date, offset: number): Date {
return new Date(date.getFullYear() + offset, 11, 31)
}
/** Resolve one anchor, keeping the time of day of `date` for datetime columns. */
function resolveAnchor(
anchor: VaAnchor,
date: Date,
offset: number,
keepTime: boolean
): Date {
let resolved: Date
switch (anchor) {
case 'QUARTER_START':
resolved = quarterStart(date, offset)
break
case 'QUARTER_END':
resolved = quarterEnd(date, offset)
break
case 'MONTH_START':
resolved = monthStart(date, offset)
break
case 'MONTH_END':
resolved = monthEnd(date, offset)
break
case 'YEAR_START':
resolved = yearStart(date, offset)
break
default:
resolved = yearEnd(date, offset)
break
}
if (keepTime) {
resolved.setHours(date.getHours(), date.getMinutes(), date.getSeconds(), 0)
}
return resolved
}
/**
* The dates a transform adds to a mapping, in order.
*
* `ADD` returns the report's own value plus the anchor beside it, de-duplicated -
* so a value that already sits on the anchor yields one date, and the clause
* stays an equality rather than an `IN` with a repeat. `RANGE` returns the
* period's first and last day.
*
* `keepTime` carries the time of day across for datetime columns, since a
* DATETIME filter compares to the second; date columns use midnight.
*/
export function applyVaTransform(
value: Date,
transform: VaTransform,
keepTime = false
): Date[] {
if (transform.op === 'RANGE') {
// A range covers whole days: the bounds are the period's own, never the
// value's clock time. Reusing keepTime here would start the range mid-morning
// on day one and end it mid-morning on the last day, dropping the rest of
// both boundary days from a BETWEEN.
const start = resolveAnchor(
(transform.period + '_START') as VaAnchor,
value,
transform.offset,
false
)
const end = resolveAnchor(
(transform.period + '_END') as VaAnchor,
value,
transform.offset,
false
)
if (keepTime) end.setHours(23, 59, 59, 0)
return [start, end]
}
const anchor = resolveAnchor(
transform.anchor,
value,
transform.offset,
keepTime
)
return anchor.getTime() === value.getTime() ? [value] : [value, anchor]
}
+51
View File
@@ -5019,6 +5019,57 @@ body[cds-theme='dark'] {
border: 1px solid var(--cds-alias-status-warning) !important;
}
/* Derived (computed) columns: read-only, but computed at read time rather than
locked by policy. Tinted and marked with an 'fx' glyph instead of the
read-only dimming, so the two do not read as the same thing. Blue is the
info family - the primary key green and the warning ochre keep their meaning. */
.handsontable th.dc-derived-header {
background: #d3e5f2 !important;
box-shadow: inset 0 -2px 0 #1a6ea3;
}
.handsontable th.dc-derived-header div::after {
content: 'fx';
margin-left: 4px;
font-size: 9px;
font-style: italic;
opacity: 0.7;
}
.handsontable td.dc-derived-cell {
background: #eef5fa !important;
font-style: italic;
}
/* the first derived column starts the computed block */
.handsontable th.dc-derived-first {
border-left: 2px solid #1a6ea3 !important;
}
.handsontable td.dc-derived-cell.dc-derived-first {
box-shadow: inset 2px 0 0 #1a6ea3;
}
body[cds-theme='dark'] {
.handsontable th.dc-derived-header {
background: #2b4a5c !important;
box-shadow: inset 0 -2px 0 #4b9cc9;
}
.handsontable td.dc-derived-cell {
background: #24343d !important;
color: #dbe7ee;
}
.handsontable th.dc-derived-first {
border-left-color: #4b9cc9 !important;
}
.handsontable td.dc-derived-cell.dc-derived-first {
box-shadow: inset 2px 0 0 #4b9cc9;
}
}
.handsontable .numericListbox {
text-align: right;
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "dcfrontend",
"version": "7.17.0",
"version": "7.21.0",
"description": "Data Controller",
"devDependencies": {
"@nogoo9/gitleaks": "8.30.1-post.2",
+1
View File
@@ -11,5 +11,6 @@
"noNestedMacros": true,
"hasMacroParentheses": true,
"strictMacroDefinition": true,
"ignoredMacros": ["webout", "mpeinit", "mpeterm"],
"ignoreList":["sasjs/scratch"]
}
+10
View File
@@ -0,0 +1,10 @@
{
"fromjs": [
{
"ADMIN": "all-users",
"DCPATH": "/tmp/dcdata",
"OUTFMT": "JSON",
"KEEPME": "Y"
}
]
}
+1 -1
View File
@@ -1,7 +1,7 @@
{
"fromjs": [
{
"ADMIN": "AllUsers",
"ADMIN": "all-users",
"DCPATH": "/tmp/dcdata",
"OUTFMT": "JSON"
}
+4
View File
@@ -0,0 +1,4 @@
{
"licenceKey": "G4H/DNTdEMkmCIxOrYkRYTQ1P4sWg0gs8Zk9rakxFNYDZcLyEPMJJo82YJQ/S24P2E/YPxaWsDZXgOrWKfeCDS7NrLcwsoFNa8KRF2WTlRgSMiG4sjBRX8zNEIvszj5DAq4LEgOUUAdfctNfpUcygbxT1dhqy6gU3hBu/xRuXGd36rhxQwfahaKkfaCg/rFplkrvqnjoJ6622BxWP/HG1dxZUE7HNJyT4tNaXzlC9U6u/aaJJ66+V+9511BwjM9ss4nyHuHoogVE1yrzLMlL94P3QoM1hwkGHnEHmFzg3zHbQG1f3pUs9YzK0ytsFKU7AK948gJrGbf37Zqa+Mz7EZf6XzqqXx0zQ4y9zAhLV9s4QG8JdhrByhfG5kefrs4L/BpgKq3ElgYL5EMkrLJUrlqWYI2ouxJuIGCQz41VCWXJ2S7pd39ljifc8Zj+t2Mn5EEkRVsrbdxjCkyoZOQ1ALofa1+auUI26Rm5rwEgrfhEocDmphJpO2F8lZgwDxbepSE6v/vEV/JQWBQTyrX10W+qOEu8OhvZrYJo+eIJMaO7q1pjfdrn3XhoH+BeB3Sy1sevGLXQZC+CzOtcvajetGGCDAabtTLc49ouWPUiUOAWPE3uYymsMoRZgpLIMxaUM86hoK0r9T6wSaP0bddK754sXN1jWwIDAHSGz5WO1ko=",
"activationKey": "MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQD2w8Ge6tMI82vaQ7x5ZdlwnTJycn0UFZoSgRwObaM0NC5wp9Y8lwkVzN8FUhg2oYSTpvyJpix+VGRj0AlNBwFAVcJG9VDPYh2LE3Us9ghRfkzFUiM2PqorHnzL5DwpailRtmFHoDQPGwwSmKncTo2QHql/K6tWe7g3Nv9uqQNks283hvDQ6PVGU4TaFTSRbjGX4j+JAR651J6wI6nY+9kNxOBwnL/yKTnJGmL8oqZ+oC45quZzgRW4dLIeTODcB4IZe2ruxZ9tvXpSuAESiOv5bBZyMJEKdKwsRdt6vk+bezkZBCzPq+ly892Ncq7x5u4/8dGcGxRxxp5TnIIx3hFoiXZ7svXJ7st6uHuKkuwYq+ebWh6bcvQAwH2wCP40twMjQgmU8nHHo6+pulxu1wVZBxtoyM4m3Lu0ZncTi6mF0duiPvpkNNAZzKOF98aUq3hxYp5PP28qrtWuPsgSdTNWJwnV2R288yAPx4WLUAGB9h889d/m67HUiKadmfG4oyUnChCU/mqeJ/5nQkK1n76ezRPGsTcMZIv2mmPLShA/j38lGMjR/3pzM2Ha6UFjJKpurpeDiyt5pUdmRBpQRoBUx/PHnbXBsO9PBXa0mjdq3UFbgb5iBD1Fz2YClZ/h0d4YlYUMSeharDrLfR15g3fGnzxrkR03zYeHkxMUi5vYHQIDAQABAoICAFxUg5pcH8lOF1FMHfXIY5MM5mbHKsC8knERCPfnEvVB/wxc6D60fow/I0Vg4blzNuFuTjiMykIcfpGQMiK1l8TEMdSODoYCHq6+gY3ootUnLj94i79MtCw9FJHPnKkcVUiDNqPvr/rk8l/06j2J0NMXffEc3FeYJCXYuLtlluTzdI4wueNY/wkoz+EvpQMzWXAMkhxHpIEAqr2aIn+TQDQ0MYxBQlqwDTSPYHFJxIVFlvYx2VdnecKSO2E1/76kOowhTzH3FL+hcWcI/TG7tJuXU1C3llV0sUVBgC1NjwjMkuEm3EN5/ffqgj/eJNJTR1r6tqSzxzMJD42WVmqlPhqEmmRE1yeaio6Ulm5EJLREYEY4/1EotdIO6PCXF1Xeo+10VJLcwi19vzMlz7vFB6vSJzJNYOaiwm4rgmF12+nNt18cHoFvegGPAoCJurUhXU+iV6LlKYB8J/+kpITleZwm4dEaa6EWxgFf4nZ4N+qRVqk4JfC+rN3DQgbPcII+z7axAb8xutq5rvzskGAmKcTEagMG+pnk9DI+0m1C2J6+mC/YruVlpSlPB7Ecu/OF++vJoSCLuDXVuxMTQ3tQXoicOG/S8d1aa3nq4fvLIgZTOZoffUvNvjCyy/Dlv6vaDVCjgqt2kFZVSZv3C6ttyrNVBSjhucASGTq/7q82JKVrAoIBAQD7e7SYqDM8SAjOfpRAHCbIVVSaHw1tul16B6JAF67OIs1uD//Y+maL4GHlpFxnib0tg6z3+sG10NMUhK7elB6SkJ8voT+Ztg/+on659vmaqwTlkGjgks+8DmIw1b0ZbXuIqhnkOeSsll2V9NWItIrbzNwvhtMEQsOtFvmrLlKQvx9Bo3lGmbfdU1/9XpwW4wiA5ONcFCl1xllX9nItB+ActPoAeQgnuejWAISQkITDTOFcGu0AjMjJ66HUlY/WDbf7wY/s2aMbOiVx7B6YzQnK+lZdjjNfoaL8nfN8Z8ZzJFAgmTQ9ztZCJ8fLTfZDsYD6XZnVEIG401+6de0Ea4MHAoIBAQD7Mlr++keNDxYy3k6kwTXDOSKMDfXIDverFy/tiNorcVU5nIpuhjowl4HIQVAhzlXhATp938LM/PvJIeu9HtgsY9BFNL52fr7iGEWknqZQVE+8Um35tzV3vtPH9NVXHIYmcRCAQQJ5a6oVQRJKAM3E1+htM2AeZKMaQSCC6VHGBXJG9nEltdMKTet09pd26Z5y/YIfTqbIekyjRomGdInyWBkpJLJ/uROVb9OGnsYWadDktn7y9+oH+VZoJxW5QwXV0OL7wMzymlrKhhBNMziJRBk4rVztclYzUShpwE8Xfjt5WYkOoieXtmkdpF7TMvRYy6cdflh39l8s7THq4U67AoIBAQDPShR+kIZK9gVsUCHbBllhYBSwRVBIVgmfAJPYyKyp9T3nrwBw1VWZjoajR2tzXLG2xobYkLUDLPIQQ53wkZ7xqIvOMte917gvZ7mVh+IOf1E1+Ga6VcktdZ2yKzwHrZrZ93wIB0BakkbpyDoKyOZ8bsw/0J43MOEh14JMe45LCX62GA1rCGHp6UOCEW4P32uu8vwFdgkGr0wNmXJbc9ZjUzmPEjSKqk8ltp+OmdEVOlW4GempPmJzLeri07UP95bjyZTc4jfzxWYAFWTTBo8/5EE0NHZcKVuzZm7A6LlHR3NUq3PITqmFNk0VY1nOSo3J0UVvvn9u3axWgcWwkv4ZAoIBAQD5GaB/RfZ6fcGHxEi8vRTxedE2jglz0NPuCFpw9ctUVVksfn4QyG8gDL6YtZwfShyDutIkTSdCXIXbmRXLP7Ejo/+/AWT24Q2t7F/ztZg8BL/lguCoR/RvwP8HRVXiwtiNeoXgiZdIjWnnFdZrKIzlbS6CS/Y/bZEI0hjJeVKsSm5ezLi7qqmTiF28aWUeeOG2KotFPJvMJuuk5qgue5boilC5KjkQy5oOEQz0bLMHOvv6MreFfi/IpIv5UPkL9NjuhezxtuoGkucB+xxjUi34gV15hQGoPkAZN5wZ8qj8JSQZXJ67sjYNDskCqiOW0Orau2oHp2lX3SvYw4dC4xdNAoIBAQDSXF8OnWu08smGDs4S57yfqe59N0ULIBDOZdEQcLXRaDeAoOwQj8szUf0pycXUZLvXxv29iF61FJg70iGGCp3lW+9Oe0P+dGsKvzphRGsQPZy4h/qZDoQ9dE667ejI4aDb/C1GaOZ8TT1QfpQcVKbgLjdC19IGQPhFljGM1wMQojU5lp4/iKmA3J3ahOuV4dvEUr6xkqzq+PhH405XdURXsVc1J6qYX2wQXRoB0oDsrCbqidZi14DaeE8UhZRA6m9haUOb/3Y+OkCs2QIW4MNZ+zg8XtHjiUhMxAaJ6zdEFltWPoVFsVfgoecquAHb0uDJJRz/+RfR0AiG0XG0NW+C"
}
@@ -14,7 +14,6 @@ function makeRows(n) {
const rows = []
for (let i = 0; i < n; i++) {
rows.push({
_____DELETE__THIS__RECORD_____: "No",
PRIMARY_KEY_FIELD: i,
SOME_CHAR: "dummy data row " + i,
SOME_DROPDOWN: dropdowns[i % dropdowns.length],
@@ -241,7 +240,6 @@ const data = {
{ ODS_TABLE: "ENGINEHOST", NAME: "File Size (bytes)", VALUE: "1048576" }
],
maxvarlengths: [
{ NAME: "_____DELETE__THIS__RECORD_____", MAXLEN: 3 },
{ NAME: "PRIMARY_KEY_FIELD", MAXLEN: 4 },
{ NAME: "some_char", MAXLEN: 591 },
{ NAME: "some_dropdown", MAXLEN: 8 },
@@ -261,7 +259,6 @@ const data = {
sasdata: makeRows(100),
$sasdata: {
vars: {
_____DELETE__THIS__RECORD_____: { format: "$3.", label: "_____DELETE__THIS__RECORD_____", length: "3", type: "char" },
PRIMARY_KEY_FIELD: { format: "best.", label: "PRIMARY_KEY_FIELD", length: "8", type: "num" },
SOME_CHAR: { format: "$32767.", label: "SOME_CHAR", length: "32767", type: "char" },
SOME_DROPDOWN: { format: "$128.", label: "SOME_DROPDOWN", length: "128", type: "char" },
@@ -280,14 +277,14 @@ const data = {
},
sasparams: [
{
COLHEADERS: "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,SOME_HARDSELECT,SOME_NUM,SOME_DATE,SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM,READONLY_COL,HIDDEN_COL,ROUND_COL,NUMFMT_COL",
COLHEADERS: "PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,SOME_HARDSELECT,SOME_NUM,SOME_DATE,SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM,READONLY_COL,HIDDEN_COL,ROUND_COL,NUMFMT_COL",
FILTER_TEXT: "",
PKCNT: 1,
PK: "PRIMARY_KEY_FIELD",
DTVARS: " SOME_DATE",
DTTMVARS: " SOME_DATETIME",
TMVARS: " SOME_TIME",
COLTYPE: "{\"data\":\"_____DELETE__THIS__RECORD_____\",\"type\":\"dropdown\",\"source\":[\"No\",\"Yes\"]},{\"data\":\"PRIMARY_KEY_FIELD\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_CHAR\"},{\"data\":\"SOME_DROPDOWN\"},{\"data\":\"SOME_HARDSELECT\"},{\"data\":\"SOME_NUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_DATE\",\"type\":\"date\"},{\"data\":\"SOME_DATETIME\",\"type\":\"datetime\"},{\"data\":\"SOME_TIME\",\"type\":\"time\"},{\"data\":\"SOME_SHORTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_BESTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"READONLY_COL\"},{\"data\":\"HIDDEN_COL\"},{\"data\":\"ROUND_COL\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"NUMFMT_COL\",\"type\":\"numeric\",\"format\":\"0\"}",
COLTYPE: "{\"data\":\"PRIMARY_KEY_FIELD\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_CHAR\"},{\"data\":\"SOME_DROPDOWN\"},{\"data\":\"SOME_HARDSELECT\"},{\"data\":\"SOME_NUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_DATE\",\"type\":\"date\"},{\"data\":\"SOME_DATETIME\",\"type\":\"datetime\"},{\"data\":\"SOME_TIME\",\"type\":\"time\"},{\"data\":\"SOME_SHORTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_BESTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"READONLY_COL\"},{\"data\":\"HIDDEN_COL\"},{\"data\":\"ROUND_COL\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"NUMFMT_COL\",\"type\":\"numeric\",\"format\":\"0\"}",
LOADTYPE: "UPDATE",
RK_FLAG: 0,
CLS_FLAG: 0
+3 -3
View File
@@ -1,4 +1,4 @@
{
"licenceKey": "invalid",
"activationKey": "invalid"
}
"licenceKey": "",
"activationKey": ""
}
+3 -1
View File
@@ -16,7 +16,9 @@
"streamWebFolder": "web9",
"webSourcePath": "../../client/dist",
"assetPaths": [],
"streamServiceName": "clickme"
"streamServiceName": "clickme",
"mockWeb": true,
"mockWebFolder": "sasjsbuild/mock"
}
},
{
+320 -2
View File
@@ -281,6 +281,53 @@ const tables = {
DD_SENSITIVITY: 'Low'
}
],
// Derived (computed) variables. Values are display-time only - the logic is
// applied by mpe_derived in the mock's getdata/getcolvals, and must use the
// expression subset that compileSasExpression supports (the real evaluation
// is done by SAS).
MPE_DERIVED: [
{
tx_from: 0,
tx_to: 253717919999,
libref: 'TESTDATA',
dsn: 'DERIVED_DEMO',
var_nm: 'REGION',
var_deps: 'STATE',
var_type: 'C',
var_len: 8,
var_fmt: '',
var_logic: "ifc(STATE in ('CA','OR','WA'),'West','East')",
active: 1
},
{
tx_from: 0,
tx_to: 253717919999,
libref: 'TESTDATA',
dsn: 'DERIVED_DEMO',
var_nm: 'LINE_VALUE',
var_deps: 'QTY PRICE',
var_type: 'N',
var_len: 8,
var_fmt: 'dollar12.2',
var_logic: 'QTY * PRICE',
active: 1
},
{
// built on REGION, so it proves the dependency order is applied
tx_from: 0,
tx_to: 253717919999,
libref: 'TESTDATA',
dsn: 'DERIVED_DEMO',
var_nm: 'REGION_LABEL',
var_deps: 'REGION STATE',
var_type: 'C',
var_len: 14,
var_fmt: '',
var_label: 'Region and state',
var_logic: "cats(REGION,'-',STATE)",
active: 1
}
],
MPE_DATALOADS: [],
MPE_EMAILS: [],
MPE_EXCEL_CONFIG: [],
@@ -1112,6 +1159,26 @@ const tables = {
selectbox_value: 'FORMULA',
selectbox_order: 1,
ver_to_dttm: 127490111999
},
{
selectbox_rk: 73,
ver_from_dttm: 0,
select_lib: dcLibref,
select_ds: 'MPE_DERIVED',
base_column: 'VAR_TYPE',
selectbox_value: 'C',
selectbox_order: 1,
ver_to_dttm: 127490111999
},
{
selectbox_rk: 74,
ver_from_dttm: 0,
select_lib: dcLibref,
select_ds: 'MPE_DERIVED',
base_column: 'VAR_TYPE',
selectbox_value: 'N',
selectbox_order: 2,
ver_to_dttm: 127490111999
}
],
MPE_SUBMIT: [],
@@ -1324,6 +1391,19 @@ const tables = {
'RLS_SCOPE RLS_GROUP RLS_LIBREF RLS_TABLE RLS_GROUP_LOGIC RLS_SUBGROUP_LOGIC RLS_SUBGROUP_ID RLS_VARIABLE_NM RLS_OPERATOR_NM RLS_RAW_VALUE',
post_edit_hook: 'services/hooks/mpe_row_level_security_postedit'
},
{
tx_from: 0,
tx_to: 127490111999,
libref: dcLibref,
dsn: 'MPE_DERIVED',
num_of_approvals_required: 1,
loadtype: 'TXTEMPORAL',
buskey: 'LIBREF DSN VAR_NM',
notes: 'Derived (computed) variables for the EDIT screen',
var_txfrom: 'TX_FROM',
var_txto: 'TX_TO',
post_edit_hook: 'services/hooks/mpe_derived_postedit'
},
{
tx_from: 0,
tx_to: 127490111999,
@@ -1955,6 +2035,206 @@ const tables = {
rule_value: 'services/validations/mpe_x_test.some_num',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'LIBREF',
rule_type: 'NOTNULL',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'LIBREF',
rule_type: 'CASE',
rule_value: 'UPCASE',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'DSN',
rule_type: 'NOTNULL',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'DSN',
rule_type: 'CASE',
rule_value: 'UPCASE',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_NM',
rule_type: 'NOTNULL',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_NM',
rule_type: 'CASE',
rule_value: 'UPCASE',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_NM',
rule_type: 'HARDREGEX',
rule_value: '^[A-Za-z_][A-Za-z0-9_]*$',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_DEPS',
rule_type: 'CASE',
rule_value: 'UPCASE',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_DEPS',
rule_type: 'HARDREGEX',
rule_value: '^[A-Za-z0-9_ ]*$',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_TYPE',
rule_type: 'NOTNULL',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_TYPE',
rule_type: 'CASE',
rule_value: 'UPCASE',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_TYPE',
rule_type: 'HARDSELECT',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_LEN',
rule_type: 'NOTNULL',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_LEN',
rule_type: 'MINVAL',
rule_value: '1',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_LEN',
rule_type: 'MAXVAL',
rule_value: '32767',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_FMT',
rule_type: 'CASE',
rule_value: 'UPCASE',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_FMT',
rule_type: 'HARDREGEX',
rule_value: '^(\$?[A-Za-z][A-Za-z0-9_]*(\.[0-9]+)?\.?|\$?[0-9]+(\.[0-9]+)?\.?)?$',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_LOGIC',
rule_type: 'NOTNULL',
rule_value: ' ',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_LABEL',
rule_type: 'HARDREGEX',
rule_value: '^[^\\x00-\\x1F\\x7F]*$',
rule_active: 1,
tx_to: 127490111999
},
{
tx_from: 0,
base_lib: dcLibref,
base_ds: 'MPE_DERIVED',
base_col: 'VAR_LOGIC',
rule_type: 'HARDREGEX',
rule_value: '^(?!.*(/\\*|\\*/))[^;&%]*$',
rule_active: 1,
tx_to: 127490111999
}
],
MPE_X_TEST: [
@@ -2226,7 +2506,7 @@ tables.MPE_TABLES.push({
// ─── MPE_X_WIDE: wide table for the EDIT cell limit ─────────────────────────
// 200 rows x 1001 columns (ROW_ID + 500 single-character + 500 numeric) is
// 200,400 cells - over DC_MAXCELLS_WEBEDIT (200000) while staying under
// 200,200 cells - over DC_MAXCELLS_WEBEDIT (200000) while staying under
// DC_MAXOBS_WEBEDIT (250), so it exercises the CELL limit on its own and the
// row limit spec can use MPE_X_SEARCH (1000 rows) instead. Values are one
// character / one digit on purpose: the point is width, and small values keep
@@ -3262,6 +3542,20 @@ const schema = {
notnull: false
}
],
MPE_DERIVED: [
{ name: 'TX_FROM', type: 'N', length: 8, format: 'datetime19.3', label: '', notnull: true },
{ name: 'LIBREF', type: 'C', length: 8, format: '', label: '', notnull: false },
{ name: 'DSN', type: 'C', length: 32, format: '', label: '', notnull: false },
{ name: 'VAR_NM', type: 'C', length: 32, format: '', label: '', notnull: false },
{ name: 'VAR_DEPS', type: 'C', length: 4000, format: '', label: '', notnull: false },
{ name: 'VAR_TYPE', type: 'C', length: 1, format: '', label: '', notnull: false },
{ name: 'VAR_LEN', type: 'N', length: 8, format: '', label: '', notnull: false },
{ name: 'VAR_FMT', type: 'C', length: 32, format: '', label: '', notnull: false },
{ name: 'VAR_LABEL', type: 'C', length: 256, format: '', label: '', notnull: false },
{ name: 'VAR_LOGIC', type: 'C', length: 4000, format: '', label: '', notnull: false },
{ name: 'ACTIVE', type: 'N', length: 8, format: '', label: '', notnull: false },
{ name: 'TX_TO', type: 'N', length: 8, format: 'datetime19.3', label: '', notnull: true }
],
MPE_DATALOADS: [
{
name: 'LIBREF',
@@ -5349,7 +5643,15 @@ const testTables = {
{ ORDER_ID: 1003, CUSTOMER: 'Cobalt GmbH', REGION: 'EU', AMOUNT: 620, STATUS: 'CLOSED' },
{ ORDER_ID: 1004, CUSTOMER: 'Delta Systems', REGION: 'US', AMOUNT: 4500, STATUS: 'NEW' }
],
DEMO_MIRROR: []
DEMO_MIRROR: [],
// Derived variable demo - REGION and LINE_VALUE are not columns of this
// table, they are computed at read time from MPE_DERIVED rows.
DERIVED_DEMO: [
{ ID: 1, STATE: 'CA', QTY: 2, PRICE: 10.5 },
{ ID: 2, STATE: 'NY', QTY: 1, PRICE: 99 },
{ ID: 3, STATE: 'OR', QTY: 5, PRICE: 3 },
{ ID: 4, STATE: 'TX', QTY: 3, PRICE: 7.25 }
]
}
// Schemas for the TESTDATA tables, matching the old inline getdata.js payload
@@ -5691,6 +5993,12 @@ const testSchema = {
label: 'Last reviewed',
notnull: false
}
],
DERIVED_DEMO: [
{ name: 'ID', type: 'N', length: 8, format: '', label: 'ID', notnull: true },
{ name: 'STATE', type: 'C', length: 2, format: '', label: 'State', notnull: false },
{ name: 'QTY', type: 'N', length: 8, format: '', label: 'Quantity', notnull: false },
{ name: 'PRICE', type: 'N', length: 8, format: 'dollar12.2', label: 'Price', notnull: false }
]
}
@@ -5739,6 +6047,16 @@ tables.MPE_TABLES.push(
buskey: 'ID',
notes: 'Compact rule demo table (eight columns)'
},
{
tx_from: 0,
tx_to: 127490111999,
libref: testDataLibref,
dsn: 'DERIVED_DEMO',
num_of_approvals_required: 1,
loadtype: 'UPDATE',
buskey: 'ID',
notes: 'Derived variable demo table'
},
{
tx_from: 0,
tx_to: 127490111999,
@@ -0,0 +1,104 @@
const nodePath = require('path')
let appLoc = nodePath.join(..._program.split('services')[0].split('/'))
const sasjsRoot = nodePath.resolve(weboutPath, '..', '..', '..')
const driveRoot = nodePath.resolve(sasjsRoot, 'drive')
// Load shared DC mock utilities
eval(
fs.readFileSync(
nodePath.resolve(driveRoot, 'files', appLoc, 'services', 'dcMockUtils.js'),
'utf8'
)
)
const dcConf = loadDcSettings(driveRoot, appLoc)
const mpeDataDir = dcConf.dataDir
const stagingDir = nodePath.resolve(mpeDataDir, 'staging')
const mpeLoadTableData = makeTableLoader(mpeDataDir)
// ─── Parse input ──────────────────────────────────────────────────────────────
// The submission id arrives as the `table` URL param, as in getauditfile.sas.
let tableId = ''
if (typeof table !== 'undefined' && table) tableId = String(table).trim()
// libds comes from MPE_SUBMIT, the same lookup the SAS service makes before it
// decides the request is valid.
let libds = ''
if (tableId) {
const submit = mpeLoadTableData('MPE_SUBMIT')
const row =
submit && submit.rows
? submit.rows.find((r) => r.TABLE_ID === tableId)
: null
if (row) {
libds = (String(row.BASE_LIB || '') + '.' + String(row.BASE_DS || '')).toUpperCase()
}
}
// ─── Respond ──────────────────────────────────────────────────────────────────
if (!tableId || !libds) {
// getauditfile.sas aborts with the same message when either is missing.
webOutOpen()
webOutObj([{ MSG: 'Missing: libds table', MAC: 'getauditfile' }], 'sasjsAbort')
webOutObj([{ STATUS: 'ERROR', DSID: '' }], 'sasparams')
webOutClose()
} else {
// The SAS service also checks EDIT or APPROVE access (mpe_accesscheck). The
// mock seeds MPE_SECURITY but no mock service enforces it, so access is not
// emulated here.
// The approval folder is zipped as it stands. getauditfile.sas lists it with
// mp_dirlist and adds each file to the package under contents/.
const stageFolder = nodePath.resolve(stagingDir, tableId)
const names = []
if (fs.existsSync(stageFolder)) {
for (const name of fs.readdirSync(stageFolder)) {
const absolute = nodePath.resolve(stageFolder, name)
let stat
try {
stat = fs.statSync(absolute)
} catch (err) {
continue
}
if (stat.isFile()) names.push(name)
}
}
names.sort()
const entries = names.map((name) => ({
name: 'contents/' + name,
content: fs
.readFileSync(nodePath.resolve(stageFolder, name), { encoding: 'utf8' })
.toString()
}))
const archive = mpZipPackage(entries)
const streamed = mpStreamfile({
filename: tableId + '.zip',
contentType: 'application/zip',
bytes: archive
})
if (!streamed) {
// The browser host returns a text body and ignores the streamed headers, so
// the archive cannot reach the browser as a download. The pack is described
// instead, which is what a reviewer opening the tab gets there.
_webout = JSON.stringify(
{
table: tableId,
libds: libds,
archive: tableId + '.zip',
note: 'The browser mock host streams text only, so this lists the archive contents.',
contents: entries.map((entry) => ({
name: entry.name,
bytes: entry.content.length
}))
},
null,
2
)
}
}
+516
View File
@@ -460,6 +460,60 @@ function sasVarsEntry(col) {
return { format: fmt, label: col.label || col.name, length: String(col.length), type: col.type === 'N' ? 'num' : 'char' }
}
/**
* Assigns the retained key (RK) to staged rows for a table registered with
* rk_underlying, mirroring %mp_retainedkey as %mpe_loader invokes it: any RK
* column in the input is dropped, and every row's RK is derived from its
* business key - the value that key already holds on the base table, or the
* next key value for a new one. Business keys are matched the way SAS compares
* them: case-sensitive, trailing blanks ignored.
*
* @param {Object} opts - { rows, tableReg, baseData }
* tableReg - the table's MPE_TABLES row (needs rk_underlying and buskey)
* baseData - { columns, rows } of the base table, for the key lookup
* @returns {String} the RK column name when the assignment ran, else ''
*/
function mpeAssignRetainedKey(opts) {
const { rows, tableReg, baseData } = opts
const rkUnderlying = tableReg && tableReg.rk_underlying
? String(tableReg.rk_underlying).trim()
: ''
const rkCol = tableReg && tableReg.buskey ? String(tableReg.buskey).trim() : ''
if (!rkUnderlying || !rkCol || !rows || rows.length === 0) return ''
const base = baseData || { columns: [], rows: [] }
const submitted = Object.keys(rows[0] || {}).find(
(k) => k.toUpperCase() === rkCol.toUpperCase()
)
const rkName = submitted ||
(base.columns.find((c) => c.name.toUpperCase() === rkCol.toUpperCase()) || {}).name ||
rkCol
const keys = rkUnderlying.split(/\s+/)
const keyValue = (value) => {
if (value === null || value === undefined) return ''
return typeof value === 'number' ? String(value) : String(value).replace(/\s+$/, '')
}
const businessKey = (row) => keys.map((k) => keyValue(colVal(row, k))).join('|')
const baseByKey = new Map()
let nextRk = 0
for (const row of base.rows) {
const rk = colVal(row, rkCol)
baseByKey.set(businessKey(row), rk)
const num = Number(rk)
if (isFinite(num) && num > nextRk) nextRk = num
}
for (const row of rows) {
Object.keys(row).forEach((k) => {
if (k.toUpperCase() === rkCol.toUpperCase()) delete row[k]
})
const existing = baseByKey.get(businessKey(row))
row[rkName] = existing === undefined || existing === null ? ++nextRk : existing
}
return rkName
}
/**
* Stages a submission: writes staged.json under staging/<tableId>/, copies the
* session audit items, and inserts the MPE_SUBMIT row. Shared by the
@@ -493,6 +547,12 @@ function stageSubmission(opts) {
// keep audit items from this Stored Program run alongside the staged data
copySessionAuditItems(stageFolder, 'submit')
// Files the SAS loader leaves in the approval folder, so the audit pack
// holds the same artefacts: the cleaned CSV and the uploaded file itself.
for (const file of opts.auditFiles || []) {
fs.writeFileSync(nodePath.resolve(stageFolder, file.name), file.content)
}
// Insert a record into MPE_SUBMIT (always in the control library /
// mpelib, regardless of which library the base table lives in)
const mpeDataDir = libDataDir('DC_JSLIB')
@@ -549,6 +609,146 @@ function copySessionAuditItems(stageFolder, prefix) {
}
}
// ─── %mp_streamfile + ODS package (zip) ───────────────────────────────────────
const ZIP_CRC_TABLE = (() => {
const table = new Array(256)
for (let i = 0; i < 256; i++) {
let c = i
for (let k = 0; k < 8; k++) c = (c & 1) ? (0xedb88320 ^ (c >>> 1)) : (c >>> 1)
table[i] = c >>> 0
}
return table
})()
function zipCrc32(bytes) {
let crc = -1
for (let i = 0; i < bytes.length; i++)
crc = (crc >>> 8) ^ ZIP_CRC_TABLE[(crc ^ bytes[i]) & 0xff]
return (crc ^ -1) >>> 0
}
function zipUtf8(value) {
const text = String(value)
if (typeof TextEncoder !== 'undefined') return new TextEncoder().encode(text)
const out = []
for (let i = 0; i < text.length; i++) {
const code = text.charCodeAt(i)
if (code < 0x80) out.push(code)
else if (code < 0x800) out.push(0xc0 | (code >> 6), 0x80 | (code & 0x3f))
else out.push(0xe0 | (code >> 12), 0x80 | ((code >> 6) & 0x3f), 0x80 | (code & 0x3f))
}
return Uint8Array.from(out)
}
/**
* Renders rows as CSV, the counterpart of %mp_ds2csv: the header row, then one
* line per row, with values quoted only when they carry a delimiter, quote or
* line break.
*
* @param {string[]} headers column names, in order
* @param {Array<Object>} rows objects keyed by column name
* @returns {string} CSV text with a trailing newline
*/
function toCsv(headers, rows) {
const cell = (value) => {
const text = value === undefined || value === null ? '' : String(value)
return /[",\n\r]/.test(text) ? '"' + text.replace(/"/g, '""') + '"' : text
}
const lines = [headers.map(cell).join(',')]
for (const row of rows) {
lines.push(headers.map((header) => cell(row[header])).join(','))
}
return lines.join('\n') + '\n'
}
/**
* Builds a zip archive in memory, entries stored rather than deflated - the JS
* counterpart of the ODS package getauditfile.sas publishes. The zip format
* leaves compression to the writer, so a stored archive opens anywhere.
*
* @param {Array<{name: string, content: string}>} entries archive entries
* @returns {Uint8Array} the archive
*/
function mpZipPackage(entries) {
const u16 = (n) => [n & 0xff, (n >>> 8) & 0xff]
const u32 = (n) => [n & 0xff, (n >>> 8) & 0xff, (n >>> 16) & 0xff, (n >>> 24) & 0xff]
const local = []
const central = []
let offset = 0
for (const entry of entries) {
const name = zipUtf8(entry.name)
const data = zipUtf8(entry.content)
const crc = zipCrc32(data)
// Bit 11 marks the name as UTF-8. Without it an extractor decodes a
// non-ASCII name as CP437, so the entry cannot be read back by name.
const flags = name.some((byte) => byte >= 0x80) ? 0x0800 : 0
// 1980-01-01 00:00, the earliest DOS timestamp: a stable value keeps
// the archive byte-identical for identical inputs.
const header = [].concat(
u32(0x04034b50), u16(20), u16(flags), u16(0), u16(0), u16(0x21),
u32(crc), u32(data.length), u32(data.length), u16(name.length), u16(0)
)
local.push(Uint8Array.from(header), name, data)
central.push({ name, crc, size: data.length, offset, flags })
offset += header.length + name.length + data.length
}
const directory = []
let directorySize = 0
for (const entry of central) {
const header = [].concat(
u32(0x02014b50), u16(20), u16(20), u16(entry.flags), u16(0), u16(0), u16(0x21),
u32(entry.crc), u32(entry.size), u32(entry.size),
u16(entry.name.length), u16(0), u16(0), u16(0), u16(0), u32(0),
u32(entry.offset)
)
directory.push(Uint8Array.from(header), entry.name)
directorySize += header.length + entry.name.length
}
const end = Uint8Array.from([].concat(
u32(0x06054b50), u16(0), u16(0),
u16(central.length), u16(central.length),
u32(directorySize), u32(offset), u16(0)
))
const chunks = local.concat(directory, [end])
const total = chunks.reduce((sum, chunk) => sum + chunk.length, 0)
const archive = new Uint8Array(total)
let at = 0
for (const chunk of chunks) {
archive.set(chunk, at)
at += chunk.length
}
return archive
}
/**
* Streams a file back as an attachment, the JS counterpart of %mp_streamfile on
* the SASJS platform: the headers name the content type and the body is the file
* itself. Returns false where that is not possible - the browser mock host has
* a text-only webout - so the caller can fall back to something textual.
*
* @param {{filename: string, contentType: string, bytes: Uint8Array}} opts
* @returns {boolean} whether the file was streamed
*/
function mpStreamfile(opts) {
if (typeof Buffer === 'undefined') return false
// A content-type header is what tells the SASjs Server to send webout.txt as
// binary rather than a string (api/src/controllers/internal/Execution.ts).
fs.writeFileSync(
_SASJS_WEBOUT_HEADERS,
`Content-type: ${opts.contentType} \n Content-Disposition: attachment;filename="${opts.filename}"`
)
fs.writeFileSync(weboutPath, Buffer.from(opts.bytes))
return true
}
/**
* Generates a DC table_id in the same format as the real backend
* (DC<yyyymmdd-hhmmss.sss>_<rand6>_<pid4>).
@@ -580,6 +780,7 @@ function makeTableId() {
* ('A','B') -> ["A","B"] (IN list)
* (1,2,3) -> [1,2,3] (IN list, numeric)
* 1 AND 5 -> [1, 5] (BETWEEN)
* '20DEC2053'd-> 34322 (date / datetime / time literal)
*/
function parseFilterRawValue(raw, operator) {
if (raw === undefined || raw === null || raw === '') return null
@@ -598,8 +799,40 @@ function parseFilterRawValue(raw, operator) {
return parseFilterScalar(s)
}
/**
* SAS date / datetime / time literal -> the SAS internal value, or null.
* 34322 <- '20DEC2053'd
* 42 <- '01JAN1960:00:00:42'dt
* 42 <- '00:00:42't
* The client builds these in canonical ddMMMyyyy form (HelperService
* .sasTemporalLiteral), so only that shape needs parsing here.
*/
const TEMPORAL_LITERAL_RE = /^'(\d{2})([A-Z]{3})(\d{4})(?::(\d{2}):(\d{2}):(\d{2})(\.\d+)?)?'(dt|d|t)$/i
/* A time literal carries no date part, so it is matched separately */
const TIME_LITERAL_RE = /^'(\d{2}):(\d{2}):(\d{2})(\.\d+)?'t$/i
function parseTemporalLiteral(s) {
const t = TIME_LITERAL_RE.exec(s)
if (t) {
return Number(t[1]) * 3600 + Number(t[2]) * 60 + Number(t[3]) +
(t[4] ? Number(t[4]) : 0)
}
const m = TEMPORAL_LITERAL_RE.exec(s)
if (!m) return null
const month = MONTHS_ABBR.indexOf(m[2].toUpperCase())
if (month < 0) return null
const days = (Date.UTC(Number(m[3]), month, Number(m[1])) - SAS_EPOCH_MS) / 86400000
const suffix = m[8].toLowerCase()
if (suffix === 'd') return days
const secs = Number(m[4]) * 3600 + Number(m[5]) * 60 + Number(m[6]) +
(m[7] ? Number(m[7]) : 0)
return suffix === 'dt' ? days * 86400 + secs : secs
}
function parseFilterScalar(s) {
s = s.trim()
const temporal = parseTemporalLiteral(s)
if (temporal !== null) return temporal
if (s.startsWith("'") && s.endsWith("'")) {
return s.slice(1, -1).replace(/''/g, "'")
}
@@ -1121,6 +1354,289 @@ function mpFilterStore(opts) {
}
}
// ─── mpe_derived.sas ────────────────────────────────────────────────────────
/**
* Compiles a SAS expression into a JS function - for the mock only.
*
* The real evaluation is done by SAS. This covers the subset used by the mock
* and demo configs, so the EDIT screen can be exercised (display, filtering,
* submission) without a SAS server:
*
* - column names, numbers, quoted strings
* - + - * / and comparisons (= ne < > <= >=)
* - AND OR, and `X IN (a, b, c)`
* - ifc(), cats(), catx(), upcase(), lowcase(), strip(), trim(), length(),
* substr(), index()
*
* Anything outside that subset throws when a row is derived, and the column is
* left missing for that row - the same way a failing SAS expression would.
*/
function compileSasExpression(expr, columnNames) {
// Tokenise first, so operators are only rewritten outside quoted strings
// (a literal such as 'OR' must never be read as an operator).
const tokens = []
const s = String(expr)
let i = 0
while (i < s.length) {
const c = s[i]
if (/\s/.test(c)) { i++; continue }
if (c === "'" || c === '"') {
const q = c
let j = i + 1
let val = ''
while (j < s.length) {
if (s[j] === q) {
if (s[j + 1] === q) { val += q; j += 2; continue }
break
}
val += s[j]
j++
}
tokens.push({ type: 'str', value: val })
i = j + 1
continue
}
if (/[0-9]/.test(c)) {
let j = i
while (j < s.length && /[0-9.]/.test(s[j])) j++
tokens.push({ type: 'num', value: s.slice(i, j) })
i = j
continue
}
if (/[A-Za-z_]/.test(c)) {
let j = i
while (j < s.length && /[A-Za-z0-9_]/.test(s[j])) j++
tokens.push({ type: 'word', value: s.slice(i, j) })
i = j
continue
}
const two = s.slice(i, i + 2)
if (['<=', '>=', '!=', '<>'].includes(two)) {
tokens.push({ type: 'punct', value: two })
i += 2
continue
}
// only the operators in the documented subset may reach the generated
// function - anything else is rejected rather than executed
if (!'+-*/(),<>='.includes(c)) {
throw new Error('Unsupported character in derived expression: ' + c)
}
tokens.push({ type: 'punct', value: c })
i++
}
// X IN (a, b, c) -> __in([a, b, c], X)
const out = []
for (let t = 0; t < tokens.length; t++) {
const tok = tokens[t]
if (
tok.type === 'word' &&
tok.value.toLowerCase() === 'in' &&
out.length > 0 &&
tokens[t + 1] &&
tokens[t + 1].value === '('
) {
let depth = 0
let j = t + 1
for (; j < tokens.length; j++) {
if (tokens[j].value === '(') depth++
else if (tokens[j].value === ')') {
depth--
if (depth === 0) break
}
}
if (j < tokens.length) {
const left = out.pop()
const inner = tokens.slice(t + 2, j)
out.push(
{ type: 'word', value: '__in' },
{ type: 'punct', value: '(' },
{ type: 'punct', value: '[' },
...inner,
{ type: 'punct', value: ']' },
{ type: 'punct', value: ',' },
left,
{ type: 'punct', value: ')' }
)
t = j
continue
}
}
out.push(tok)
}
const OPS = {
and: '&&', or: '||',
eq: '==', ne: '!=', lt: '<', gt: '>', le: '<=', ge: '>='
}
const js = out
.map(tok => {
if (tok.type === 'str') return JSON.stringify(tok.value)
if (tok.type === 'num') return tok.value
if (tok.type === 'word') return OPS[tok.value.toLowerCase()] || tok.value
if (tok.value === '=') return '=='
if (tok.value === '<>') return '!='
return tok.value
})
.join(' ')
const sasTrue = v => {
if (typeof v === 'boolean') return v
if (typeof v === 'number') return v !== 0 && !isNaN(v)
if (v === null || v === undefined) return false
const str = String(v).trim()
return str !== '' && str !== '0'
}
const toStr = v => {
if (v === null || v === undefined) return ''
if (typeof v === 'number') return isNaN(v) ? '' : String(v)
return String(v).trim()
}
const fns = {
ifc: (c, a, b) => (sasTrue(c) ? a : b),
cats: (...args) => args.map(toStr).join(''),
catx: (sep, ...args) => args.map(toStr).filter(v => v !== '').join(toStr(sep)),
upcase: v => toStr(v).toUpperCase(),
lowcase: v => toStr(v).toLowerCase(),
strip: v => toStr(v),
trim: v => toStr(v),
length: v => toStr(v).length,
substr: (v, p, n) => {
const str = toStr(v)
const start = Number(p) - 1
return n === undefined
? str.substring(start)
: str.substring(start, start + Number(n))
},
index: (v, sub) => toStr(v).indexOf(toStr(sub)) + 1,
__in: (list, v) => list.some(item => String(item) === String(v) || Number(item) === Number(v))
}
const names = Object.keys(fns)
const fn = new Function(...columnNames, ...names, 'return (' + js + ');')
return row => fn(...columnNames.map(n => row[n]), ...names.map(n => fns[n]))
}
/**
* Reads MPE_DERIVED for a table, mirroring %mpe_derived. Returns the active
* variables with a compiled row function, in dependency order (ties by name),
* the same sequence the real macro emits.
*
* @param {Object} opts
* @param {string} opts.libref - target libref
* @param {string} opts.table - target dataset
* @param {string} opts.dataDir - data directory of the DC control library
* @param {Array<Object>} opts.columns - physical columns of the target table
*/
function mpeDerived(opts) {
const { libref, table, dataDir, columns } = opts
const loadTableData = makeTableLoader(dataDir)
const cfg = loadTableData('MPE_DERIVED')
if (!cfg || !Array.isArray(cfg.rows) || cfg.rows.length === 0) return []
const now = Date.now() / 1000
const lib = String(libref || '').toUpperCase()
const dsn = String(table || '').toUpperCase()
const columnNames = (columns || []).map(c => c.name)
const vars = cfg.rows
.filter(r =>
String(colVal(r, 'LIBREF') || '').toUpperCase() === lib &&
String(colVal(r, 'DSN') || '').toUpperCase() === dsn &&
Number(colVal(r, 'ACTIVE')) === 1 &&
(!Number(colVal(r, 'TX_TO')) || Number(colVal(r, 'TX_TO')) > now)
)
.map(r => {
const logic = String(colVal(r, 'VAR_LOGIC') || '').trim()
const nm = String(colVal(r, 'VAR_NM') || '').toUpperCase()
return {
name: nm,
// every input the expression uses - columns and derived vars
deps: String(colVal(r, 'VAR_DEPS') || '')
.toUpperCase()
.split(/\s+/)
.filter(Boolean),
type: String(colVal(r, 'VAR_TYPE') || '').toUpperCase(),
length: Number(colVal(r, 'VAR_LEN')) || 8,
format: String(colVal(r, 'VAR_FMT') || ''),
logic,
label: String(colVal(r, 'VAR_LABEL') || '').trim() || nm + '=' + logic
}
})
.filter(d => d.name && (d.type === 'C' || d.type === 'N') && d.logic)
// dependency order: repeatedly take the variables whose derived inputs are
// already placed, alphabetically within a round
const byName = new Map(vars.map(v => [v.name, v]))
const placed = []
const done = new Set()
while (placed.length < vars.length) {
const ready = vars
.filter(v => !done.has(v.name))
.filter(v => v.deps.every(d => !byName.has(d) || done.has(d)))
.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0))
if (ready.length === 0) {
console.log('derived variables for ' + lib + '.' + dsn + ' contain a cycle')
return []
}
for (const v of ready) {
placed.push(v)
done.add(v.name)
}
}
vars.length = 0
vars.push.apply(vars, placed)
// compile each against the physical columns plus the variables defined
// earlier, so a derived variable can build on another one
const compileCols = columnNames.slice()
for (const d of vars) {
try {
// slice(): the compiled function captures the column list it was
// built with, not the array as it grows
d.fn = compileSasExpression(d.logic, compileCols.slice())
} catch (err) {
console.log('derived variable ' + d.name + ' could not be compiled: ' + err.message)
d.fn = () => (d.type === 'N' ? null : '')
}
compileCols.push(d.name)
}
return vars
}
/**
* Adds the derived variables to every row and to the column list, so that the
* filter (which runs afterwards) and the grid both see them.
*/
function mpeDeriveRows(tableData, derived) {
if (!derived || derived.length === 0) return tableData
const rows = tableData.rows.map(row => {
const copy = Object.assign({}, row)
for (const d of derived) {
try {
copy[d.name] = d.fn(copy)
} catch (err) {
copy[d.name] = d.type === 'N' ? null : ''
}
}
return copy
})
const derivedCols = derived.map(d => ({
name: d.name,
type: d.type,
length: d.length,
format: d.format,
label: d.label,
notnull: false
}))
return {
name: tableData.name,
columns: (tableData.columns || []).concat(derivedCols),
rows
}
}
// ─── DC settings (services/settings.js) ──────────────────────────────────────
/**
+118 -55
View File
@@ -77,6 +77,42 @@ if (mpeTablesData && mpeTablesData.rows) {
tableReg = mpeTablesData.rows.find(r => r.libref === requestedLibref && r.dsn === requestedTable)
}
// The variables exposed to the editor as the primary key, matching getdata.sas:
// pk = coalesce(rk_underlying, buskey). A table registered with rk_underlying
// is keyed on its business key in the editor - the RK itself is assigned by the
// backend loader, so it is neither shown nor sent by the user.
const rkUnderlying = tableReg && tableReg.rk_underlying ? tableReg.rk_underlying.trim() : ''
const buskey = tableReg && tableReg.buskey ? tableReg.buskey.trim() : ''
const pkFields = rkUnderlying || buskey
const pkColumns = pkFields ? pkFields.split(/\s+/) : []
// Order columns with the PK variables first, in PK order, then the remaining
// columns in schema order. getdata.sas does the same ("order delete var and pk
// fields at start of table") and the client reads the first PKCNT columns
// positionally when it detects duplicate keys.
function pkFirst(columns, keys) {
const rank = new Map(keys.map((name, i) => [name.toUpperCase(), i]))
const keyColumns = columns
.filter(col => rank.has(col.name.toUpperCase()))
.sort((a, b) => rank.get(a.name.toUpperCase()) - rank.get(b.name.toUpperCase()))
return keyColumns.concat(columns.filter(col => !rank.has(col.name.toUpperCase())))
}
// ─── Derived (computed) variables (mpe_derived.sas) ─────────────────────────
// Applied before the filter, so a filter can reference them. The derived
// columns are added to the schema too, which is what puts them in COLS and in
// the grid; they are held read-only by a READONLY rule below.
const derived = mpeDerived({
libref: requestedLibref,
table: requestedTable,
dataDir: mpeDataDir,
columns: tableData ? tableData.columns : []
})
if (tableData && derived.length > 0) {
tableData = mpeDeriveRows(tableData, derived)
}
// ─── DQ rules (from MPE_VALIDATIONS + NOTNULL from schema) ──────────────────
// Mirrors getdata.sas: dqrules = MPE_VALIDATIONS rows UNION dictionary.columns
// NOTNULL constraints. MPE_VALIDATIONS lives in the control library.
@@ -85,10 +121,7 @@ const validationsData = mpeLoadTableData('MPE_VALIDATIONS')
let dqrules = []
let dqdata = []
if (validationsData && validationsData.rows) {
const pkCols = new Set()
if (tableReg && tableReg.buskey) {
tableReg.buskey.trim().split(/\s+/).forEach(c => pkCols.add(c.toUpperCase()))
}
const pkCols = new Set(pkColumns.map(c => c.toUpperCase()))
dqrules = validationsData.rows
.filter(r => r.base_lib === requestedLibref && r.base_ds === requestedTable)
.map(r => ({
@@ -127,6 +160,16 @@ if (tableData && tableData.columns) {
}
}
// A derived variable is computed at read time, so the grid must not accept a
// value for it. COLS carries READ for the same reason (a derived column has no
// CLS rule row), but the client only applies COLS.CLS_RULE when CLS_FLAG is
// set, which column-level security drives - so the rule is declared here too,
// as getdata.sas does. dqrules is the request-local array the filter/map above
// built, so appending to it leaves the loaded MPE_VALIDATIONS rows untouched.
for (const v of derived) {
dqrules.push({ BASE_COL: v.name, RULE_TYPE: 'READONLY', RULE_VALUE: '' })
}
// ─── DQ dropdown data (from MPE_SELECTBOX + SOFTSELECT from table data) ──────
// MPE_SELECTBOX lives in the control library.
@@ -191,9 +234,8 @@ if (tableData && tableData.rows && tableData.columns) {
// ─── Build response ──────────────────────────────────────────────────────────
let response
// Set when the row or cell limit is exceeded - mirrors the %mp_abort in
// getdata.sas, and is emitted as sasjsAbort below.
let abortMsg = ''
// Set when the row or cell limit cut the selection down - mirrors the
// TRUNCATED / TRUNC_MSG in sasparams in getdata.sas.
if (tableData && tableData.columns && tableData.rows) {
// Exclude temporal columns (TX_FROM/TX_TO etc.) from the editor view
@@ -206,8 +248,17 @@ if (tableData && tableData.columns && tableData.rows) {
if (tableReg.var_busfrom) excludeCols.add(tableReg.var_busfrom.toUpperCase())
if (tableReg.var_busto) excludeCols.add(tableReg.var_busto.toUpperCase())
}
// getdata.sas drops the RK field from the editor view when the table is
// registered with rk_underlying (drop_rk=&buskey), and exposes the business
// key instead.
if (rkUnderlying && buskey) {
buskey.split(/\s+/).forEach(c => excludeCols.add(c.toUpperCase()))
}
let visibleColumns = tableData.columns.filter(c => !excludeCols.has(c.name.toUpperCase()))
let visibleColumns = pkFirst(
tableData.columns.filter(c => !excludeCols.has(c.name.toUpperCase())),
pkColumns
)
let visibleRows = tableData.rows
.filter(r => filterPredicate(r))
.map(r => {
@@ -236,15 +287,15 @@ if (tableData && tableData.columns && tableData.rows) {
}
// ─── Row and cell limits ─────────────────────────────────────────────────
// Mirrors getdata.sas: DC_MAXOBS_WEBEDIT rows, or DC_MAXCELLS_WEBEDIT
// cells (rows x columns), whichever is reached first. Both are read from
// MPE_CONFIG, the way the real service picks them up from the settings
// program it generates - so editing the option in DC changes what the mock
// allows, and an option that is absent (or switched off, var_active=0)
// falls back to the shipped default. The check runs after the
// PRE_EDIT_HOOK, as the single check in getdata.sas does - a hook may
// replace the data with a larger table. The cell count includes the
// housekeeping delete column, which is what getdata.sas counts.
// Mirrors getdata.sas: DC_MAXOBS_WEBEDIT rows or DC_MAXCELLS_WEBEDIT
// cells (rows x columns), whichever is reached first - the selection is
// cut to min(rows, cells/columns) and flagged in sasparams.
// Both options are read from MPE_CONFIG, the way the real service picks
// them up from the settings program it generates - so editing the option
// in DC changes what the mock serves, and an option that is absent (or
// switched off, var_active=0) falls back to the shipped default. The
// check runs after the PRE_EDIT_HOOK, as the single check in getdata.sas
// does - a hook may replace the data with a larger table.
// MPE_CONFIG lives in the control library, not in the table's own library,
// so this reads mpeDataDir - the same directory the MPE_TABLES lookup above
// uses. Passing the requested table's dataDir made the read fail for every
@@ -252,20 +303,32 @@ if (tableData && tableData.columns && tableData.rows) {
const dcCfg = loadDcConfig(driveRoot, appLoc, mpeDataDir)
const maxObsWebEdit = dcConfigNumber(dcCfg, 'DC_MAXOBS_WEBEDIT', 250)
const maxCellsWebEdit = dcConfigNumber(dcCfg, 'DC_MAXCELLS_WEBEDIT', 200000)
const editColCount = visibleColumns.length + 1
const cellCount = visibleRows.length * editColCount
if (visibleRows.length > maxObsWebEdit) {
abortMsg =
'Table is too big (' +
visibleRows.length +
' rows) - please filter and try again!'
} else if (cellCount > maxCellsWebEdit) {
abortMsg =
'Selection is too wide (' +
visibleRows.length +
' rows x ' +
editColCount +
' cols) - please filter and try again!'
// getdata.sas counts the columns of work.out, i.e. the table's own
// columns - the delete column is client-owned and not part of it.
const editColCount = visibleColumns.length
let maxRows = maxObsWebEdit
let limitVar = 'DC_MAXOBS_WEBEDIT'
if (editColCount > 0) {
const cellRows = Math.floor(maxCellsWebEdit / editColCount)
if (cellRows < maxRows) {
maxRows = cellRows
limitVar = 'DC_MAXCELLS_WEBEDIT'
}
}
if (maxRows < 1) maxRows = 1
let truncated = 0
let truncMsg = ''
if (visibleRows.length > maxRows) {
truncated = 1
// Same wording as the macro getdata.sas sets, so the spec can assert
// on the text the real service would send.
truncMsg =
'Only the first ' +
maxRows +
' rows are loaded - the ' +
limitVar +
' limit has been reached. Filter the table to work with the rest.'
visibleRows = visibleRows.slice(0, maxRows)
}
// Augment rows: add delete flag, normalise keys, convert temporal values to ISO
@@ -280,16 +343,16 @@ if (tableData && tableData.columns && tableData.rows) {
const targetKey = info ? info.name : key
normalised[targetKey] = info ? formatCellValue(r[key], info.ddtype) : r[key]
}
return { _____DELETE__THIS__RECORD_____: 'No', ...normalised }
return { ...normalised }
})
// $sasdata vars: temporal columns are char ($200.), others keep their SAS type
const vars = { _____DELETE__THIS__RECORD_____: { format: '$3.', label: '_____DELETE__THIS__RECORD_____', length: '3', type: 'char' } }
const vars = {}
for (const col of visibleColumns) {
vars[col.name] = sasVarsEntry(col)
}
const colHeaders = ['_____DELETE__THIS__RECORD_____', ...visibleColumns.map(c => c.name)]
const colHeaders = visibleColumns.map(c => c.name)
// Determine DTVARS, DTTMVARS, TMVARS from column formats
let dtVars = [], dttmVars = [], tmVars = []
@@ -330,7 +393,6 @@ if (tableData && tableData.columns && tableData.rows) {
// Build maxvarlengths
const maxvarlengths = [
{ NAME: '_____DELETE__THIS__RECORD_____', MAXLEN: 3 },
...visibleColumns.map(col => {
let maxLen = 0
const lcName = col.name.toLowerCase()
@@ -364,15 +426,21 @@ if (tableData && tableData.columns && tableData.rows) {
sasparams: [{
COLHEADERS: colHeaders.join(','),
FILTER_TEXT: filterText === '' ? ' ' : filterText,
PKCNT: tableReg && tableReg.buskey ? tableReg.buskey.trim().split(/\s+/).length : 0,
PK: tableReg ? tableReg.buskey || '' : '',
PKCNT: pkColumns.length,
PK: pkFields,
DTVARS: dtVars.length ? ' ' + dtVars.join(' ') : '',
DTTMVARS: dttmVars.length ? ' ' + dttmVars.join(' ') : '',
TMVARS: tmVars.length ? ' ' + tmVars.join(' ') : '',
LOADTYPE: tableReg ? tableReg.loadtype || 'UPDATE' : 'UPDATE',
RK_FLAG: tableReg && tableReg.rk_underlying ? 1 : 0,
CLS_FLAG: 0,
ISMAP: 0
ISMAP: 0,
TRUNCATED: truncated,
TRUNC_MSG: truncMsg,
// names the derived columns for the grid, as getdata.sas does - the
// READONLY rule above is why they cannot be edited, this is how the
// EDIT screen knows they are computed rather than locked
DERIVEDVARS: derived.map(d => d.name).join(' ')
}],
versions: [],
xl_rules: []
@@ -387,26 +455,21 @@ if (tableData && tableData.columns && tableData.rows) {
],
maxvarlengths: [], query: [], sasdata: [],
$sasdata: { vars: {} },
sasparams: [{ COLHEADERS: '', FILTER_TEXT: ' ', PKCNT: 0, PK: '', DTVARS: '', DTTMVARS: '', TMVARS: '', LOADTYPE: 'UPDATE', RK_FLAG: 0, CLS_FLAG: 0, ISMAP: 0 }],
sasparams: [{ COLHEADERS: '', FILTER_TEXT: ' ', PKCNT: 0, PK: '', DTVARS: '', DTTMVARS: '', TMVARS: '', LOADTYPE: 'UPDATE', RK_FLAG: 0, CLS_FLAG: 0, ISMAP: 0, TRUNCATED: 0, TRUNC_MSG: '', DERIVEDVARS: '' }],
versions: [], xl_rules: []
}
}
webOutOpen()
if (abortMsg) {
webOutObj([{ MSG: abortMsg, MAC: 'getdata' }], 'sasjsAbort')
webOutObj([{ STATUS: 'ERROR' }], 'sasparams')
} else {
webOutObj(response.approvers, 'approvers')
webOutObj(response.cols, 'cols')
webOutObj(response.dqdata, 'dqdata')
webOutObj(response.dqrules, 'dqrules')
webOutObj(response.dsmeta, 'dsmeta')
webOutObj(response.maxvarlengths, 'maxvarlengths')
webOutObj(response.query, 'query')
webOutObj(response.sasdata, 'sasdata', response.$sasdata)
webOutObj(response.sasparams, 'sasparams')
webOutObj(response.versions, 'versions')
webOutObj(response.xl_rules, 'xl_rules')
}
webOutObj(response.approvers, 'approvers')
webOutObj(response.cols, 'cols')
webOutObj(response.dqdata, 'dqdata')
webOutObj(response.dqrules, 'dqrules')
webOutObj(response.dsmeta, 'dsmeta')
webOutObj(response.maxvarlengths, 'maxvarlengths')
webOutObj(response.query, 'query')
webOutObj(response.sasdata, 'sasdata', response.$sasdata)
webOutObj(response.sasparams, 'sasparams')
webOutObj(response.versions, 'versions')
webOutObj(response.xl_rules, 'xl_rules')
webOutClose()
+43 -4
View File
@@ -59,6 +59,9 @@ for (let i = 1; i <= 10; i++) {
let libds = ''
if (typeof table !== 'undefined' && table) libds = String(table).toUpperCase()
// captured before any hook can re-point the submission: the cleaned CSV is
// named after the original table, as loadfile.sas names it
const origLibds = libds
let libref = libds.split('.')[0] || dcLibref
let dsn = libds.split('.')[1] || ''
@@ -147,6 +150,12 @@ if (!msg) {
if (tableEntry.var_txfrom) computedCols.push(tableEntry.var_txfrom.toUpperCase())
if (tableEntry.var_txto) computedCols.push(tableEntry.var_txto.toUpperCase())
if (tableEntry.var_processed) computedCols.push(tableEntry.var_processed.toUpperCase())
// loadfile.sas drops the RK from the comparison for a table
// with rk_underlying (the loader derives it), so an upload
// for one legitimately omits it.
if (tableEntry.rk_underlying && tableEntry.buskey) {
computedCols.push(String(tableEntry.buskey).toUpperCase())
}
}
}
const baseCols = schema.map((c) => c.name.toUpperCase())
@@ -172,6 +181,9 @@ if (!msg) {
// written, while LIBREF / DS are still ordinary macro variables, so it
// can re-point the changeset at a different table. The mock eval()s the
// hook's .js counterpart in this scope so it can reassign libref / dsn.
// A validating hook rejects the submission by setting hookErrmsg (see
// hooks/mpe_derived_postedit.js), which stops the staging below.
let hookErrmsg = ''
const hookTables = mpeLoadTableData('MPE_TABLES')
const hookReg = (hookTables && hookTables.rows)
? hookTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -179,19 +191,41 @@ if (!msg) {
if (hookReg && hookReg.post_edit_hook) {
const hookSrc = mockHookSource(hookReg.post_edit_hook)
if (hookSrc) {
// a hook reads and normalises the submitted rows as jsDataRows
// (stagedata.js names them that); here they are `rows`
const jsDataRows = rows
try {
eval(hookSrc)
} catch (err) {
console.log(
// a hook that throws has not validated anything, so the
// submission is refused rather than staged unchecked
hookErrmsg =
'post_edit_hook ' + hookReg.post_edit_hook + ' failed: ' + err.message
)
}
libds = libref + '.' + dsn
}
}
if (hookErrmsg) msg = hookErrmsg
tableId = makeTableId()
stageSubmission({
// ─── RK assignment (mpe_loader) ──────────────────────────────────────
// The uploaded file carries the business key and no RK (see the
// expected-columns check above), so the loader derives it before the
// rows are staged.
const finalTables = mpeLoadTableData('MPE_TABLES')
const finalReg =
finalTables && finalTables.rows
? finalTables.rows.find((r) => r.libref === libref && r.dsn === dsn)
: null
const rkName = mpeAssignRetainedKey({
rows,
tableReg: finalReg,
baseData: makeTableLoader(libDataDir(libref))(dsn)
})
if (rkName && !headers.some((h) => h.toUpperCase() === rkName.toUpperCase())) {
headers.push(rkName)
}
if (!msg) stageSubmission({
dataDir: libDataDir(libref),
stagingDir,
tableId,
@@ -200,7 +234,12 @@ if (!msg) {
dsn,
message: 'File upload: ' + csvName,
headers,
rows
rows,
// loadfile.sas leaves the cleaned CSV (%mp_cleancsv) in the approval
// folder, named after the table as the request found it. It copies
// the upload as BKP_<name> only when a second, non-CSV fileref came
// with the request, which the mock does not accept - so no BKP here.
auditFiles: [{ name: origLibds + '.csv', content: csvText }]
})
}
}
+64 -5
View File
@@ -22,6 +22,9 @@ let message = _sctRow.MESSAGE || ''
let libds = _sctRow.LIBDS || ''
let libref = libds.split('.')[0] || dcLibref
let dsn = libds.split('.')[1] || ''
// stagedata.sas names the CSV it stages after the table as the request found
// it, before a hook can re-point the submission
const origLibds = libds
// The base table lives in its own library (DC_JSLIB for control tables,
// TESTDATA for demo user tables); the staging dir stays in the control lib.
@@ -63,6 +66,10 @@ if (schema.length > 0) {
// eval()s the hook's .js counterpart in this scope so it can reassign
// libref / dsn / libds; the submit record then names the new base table, exactly
// as the SAS hook does via call symputx.
// A validating hook rejects the submission by setting hookErrmsg (see
// hooks/mpe_derived_postedit.js); the response below turns it into an abort,
// and nothing is staged.
let hookErrmsg = ''
const hookTables = makeTableLoader(mpeDataDir)('MPE_TABLES')
const hookReg = (hookTables && hookTables.rows)
? hookTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -73,14 +80,58 @@ if (hookReg && hookReg.post_edit_hook) {
try {
eval(hookSrc)
} catch (err) {
console.log(
// a hook that throws has not validated anything, so the submission
// is refused rather than staged unchecked
hookErrmsg =
'post_edit_hook ' + hookReg.post_edit_hook + ' failed: ' + err.message
)
}
libds = libref + '.' + dsn
}
}
// ─── RK assignment (mpe_loader) ──────────────────────────────────────────────
// A table registered with rk_underlying is edited on its business key and the
// RK column is withheld from the editor (see getdata.js), so the submitted rows
// carry no RK. The loader derives it, and MPE_SUBMIT records the staged width
// - which includes the derived column.
const finalTables = makeTableLoader(mpeDataDir)('MPE_TABLES')
const finalReg =
finalTables && finalTables.rows
? finalTables.rows.find((r) => r.libref === libref && r.dsn === dsn)
: null
const rkName = mpeAssignRetainedKey({
rows: jsDataRows,
tableReg: finalReg,
baseData: loadTableData(dsn)
})
if (rkName && !jsDataHeaders.some((h) => h.toUpperCase() === rkName.toUpperCase())) {
jsDataHeaders.push(rkName)
}
// ─── Derived variables ───────────────────────────────────────────────────────
// stagedata.sas drops the MPE_DERIVED droplist from the submitted data before
// anything is staged (they are display-only), and the approval CSV is written
// from that stripped dataset - so neither the staged rows nor the pack may
// carry them.
const derivedVars = mpeDerived({
libref,
table: dsn,
dataDir: mpeDataDir,
columns: schema
})
if (derivedVars.length > 0) {
const derivedNames = new Set(derivedVars.map((d) => d.name.toUpperCase()))
const keep = (key) => !derivedNames.has(key.toUpperCase())
jsDataHeaders = jsDataHeaders.filter(keep)
jsDataRows = jsDataRows.map((row) => {
const copy = {}
for (const key of Object.keys(row)) if (keep(key)) copy[key] = row[key]
return copy
})
}
// ─── Stage the data ──────────────────────────────────────────────────────────
// One subfolder per submission (DSID) under the staging directory, so that the
// staged data can sit alongside related artifacts (submit/approval logs,
@@ -88,7 +139,7 @@ if (hookReg && hookReg.post_edit_hook) {
const tableId = makeTableId()
stageSubmission({
if (!hookErrmsg) stageSubmission({
dataDir,
stagingDir,
tableId,
@@ -97,9 +148,17 @@ stageSubmission({
dsn,
message,
headers: jsDataHeaders,
rows: jsDataRows
rows: jsDataRows,
// stagedata.sas writes the staged rows as <orig_libds>.csv alongside them
// (%mp_ds2csv into the approval folder), so the audit pack carries it
auditFiles: [{ name: origLibds + '.csv', content: toCsv(jsDataHeaders, jsDataRows) }]
})
webOutOpen()
webOutObj([{ STATUS: 'SUCCESS', DSID: tableId, URL: '' }], 'sasparams')
if (hookErrmsg) {
webOutObj([{ MSG: hookErrmsg, MAC: 'stagedata' }], 'sasjsAbort')
webOutObj([{ STATUS: 'ERROR', DSID: '' }], 'sasparams')
} else {
webOutObj([{ STATUS: 'SUCCESS', DSID: tableId, URL: '' }], 'sasparams')
}
webOutClose()
@@ -0,0 +1,410 @@
// ─────────────────────────────────────────────────────────────────────────────
// Mock of services/hooks/mpe_derived_postedit
//
// Registered against DC_JSLIB.MPE_DERIVED (MPE_TABLES.post_edit_hook). This
// file is eval'd by editors/stagedata.js and editors/loadfile.js, mirroring how
// mpe_loader.sas %includes the hook program via %mpe_runhook, so it shares the
// caller's scope: jsDataRows (the submitted rows - work.staging_ds on the
// server), libref, dsn, mpeDataDir and the dcMockUtils helpers.
//
// A hook rejects a submission by setting hookErrmsg, which the caller turns
// into a sasjsAbort - the same outcome as the SAS hook's %mp_abort.
//
// The rules mirror services/hooks/mpe_derived_postedit.sas, in its order:
// normalise the incoming rows, reject a duplicate name and a bad field, then
// check the dependency graph that the resulting configuration would have (the
// incoming rows plus the current rows they do not replace). The field rules
// are core's mp_derivedvarcheck, which the server hook calls and the read-time
// engine runs too - this file spells them out a second time, so a change there
// has to be mirrored here. As on the server, the collision between VAR_NM and
// a physical column is left to the read-time engine, which runs with the target
// library assigned - the hook does not have it here either.
// ─────────────────────────────────────────────────────────────────────────────
;(function mpeDerivedPostedit() {
const DELETE_MARK = '_____DELETE__THIS__RECORD_____'
const NAME_CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
const FMT_CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.'
const text = (v) => (v === undefined || v === null ? '' : String(v))
const up = (v) => text(v).trim().toUpperCase()
const strip = (v) => text(v).trim()
/** Reads a field as the hook normalised it. */
const field = (row, name) => up(colVal(row, name))
/** Writes a field back under the key the row already carries. */
const setField = (row, name, value) => {
const key = Object.keys(row).find((k) => k.toUpperCase() === name)
if (key) row[key] = value
else row[name] = value
}
const rows = Array.isArray(jsDataRows) ? jsDataRows : []
const isDeleted = (row) => field(row, DELETE_MARK) === 'YES'
const isLive = (row) =>
!isDeleted(row) && Number(colVal(row, 'ACTIVE')) === 1
const keyOf = (row) =>
[field(row, 'LIBREF'), field(row, 'DSN'), field(row, 'VAR_NM')].join(
'.'
)
const reject = (msg) => {
hookErrmsg = msg
}
// ─── Normalise the incoming values ─────────────────────────────────────────
for (const row of rows) {
if (isDeleted(row)) continue
for (const name of [
'LIBREF',
'DSN',
'VAR_NM',
'VAR_TYPE',
'VAR_FMT',
'VAR_DEPS'
]) {
setField(row, name, up(colVal(row, name)))
}
for (const name of ['VAR_LABEL', 'VAR_LOGIC']) {
setField(row, name, strip(colVal(row, name)))
}
}
// ─── Duplicate names after normalisation ───────────────────────────────────
// mpe_loader compares the raw values, so a case variant reaches this point
const seen = new Set()
for (const row of rows) {
if (isDeleted(row)) continue
const key = keyOf(row)
if (seen.has(key)) {
return reject(
`MPE_DERIVED: ${field(row, 'VAR_NM')} is declared more than once for ${field(row, 'LIBREF')}.${field(row, 'DSN')}`
)
}
seen.add(key)
}
// ─── Field level checks ────────────────────────────────────────────────────
// Two passes, in the order the SAS hook runs them. The first is the fields
// this hook owns - the control table key, which the loader normalises but
// does not require, and ACTIVE, the picker pair the EDIT screen offers.
for (const row of rows) {
if (isDeleted(row)) continue
const name = field(row, 'VAR_NM')
let reason = ''
if (!field(row, 'LIBREF') || !field(row, 'DSN'))
reason = 'LIBREF and DSN are required'
// ACTIVE is a picker in the EDIT screen (HARDSELECT over the
// MPE_SELECTBOX pair) - the same pair the hook enforces
else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
reason = 'ACTIVE must be 1 or 0'
if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
}
// The second pass is the rules for the derived variable itself, which are
// core's (mp_derivedvarcheck): the engine runs the same macro at read time,
// so a definition is measured against one rule list wherever it is checked.
// C0 controls and DEL: what a label cannot carry once a line break has been
// flattened to a space, which is what the engine does with it
const CONTROL_CHARS = /[\u0000-\u001F\u007F]/
const flatten = (v) => v.replace(/[\r\n]+/g, ' ')
const AUTOMATIC_VARS = ['_N_', '_ERROR_', '_IORC_']
// the capabilities refused by name - a function name cannot be assembled at
// run time, so naming them is a control the expression cannot evade
const DENY_FUNCS = new RegExp(
'\\b(DOSUBL|JAVAOBJ|JAVAMETA|SYMGET|SYMPUTX?|SYSGET|GETOPTION' +
'|PATHNAME|FILEEXIST|FOPEN|FREAD|FGET|FCLOSE' +
'|DOPEN|DREAD|DNUM|MOPEN|SLEEP' +
'|FILENAME|FDELETE|FCOPY|DCREATE|DLCREATE)\\s*\\('
)
// quoted literals are data, not calls
const unquoted = (v) =>
v.replace(/'([^']|'')*'/g, ' ').replace(/"([^"]|"")*"/g, ' ')
// true when the value carries a character outside the allowed set
const hasInvalidChar = (value, chars) =>
[...value].some((c) => !chars.includes(c))
for (const row of rows) {
if (isDeleted(row)) continue
const name = field(row, 'VAR_NM')
const type = field(row, 'VAR_TYPE')
const len = Number(colVal(row, 'VAR_LEN'))
const fmt = field(row, 'VAR_FMT')
const label = strip(colVal(row, 'VAR_LABEL'))
const logic = strip(colVal(row, 'VAR_LOGIC'))
let reason = ''
if (!name) reason = 'VAR_NM is required'
else if (hasInvalidChar(name, NAME_CHARS))
reason = 'VAR_NM must be a valid SAS name'
else if ('0123456789'.includes(name[0]))
reason = 'VAR_NM must not start with a digit'
else if (AUTOMATIC_VARS.includes(name))
reason = 'VAR_NM is an automatic variable of the data step'
else if (name === DELETE_MARK) reason = 'VAR_NM is reserved'
else if (type !== 'C' && type !== 'N')
reason = 'VAR_TYPE must be C or N'
else if (!Number.isInteger(len) || len < 1)
reason = 'VAR_LEN must be a positive integer'
else if (type === 'C' && len > 32767)
reason = 'VAR_LEN exceeds 32767 for a character variable'
else if (type === 'N' && len > 8)
reason = 'VAR_LEN exceeds 8 for a numeric variable'
else if (
fmt.includes(';') ||
fmt.includes('&') ||
fmt.includes('%') ||
fmt.includes('/*') ||
fmt.includes('*/')
)
reason = 'VAR_FMT must be a format name, without ; & % or /* */'
else if (hasInvalidChar(fmt, FMT_CHARS))
reason = 'VAR_FMT must be a format name'
// free text, in any script - what is refused is a control character,
// and a value the engine would cut to 256
else if (CONTROL_CHARS.test(flatten(label)))
reason = 'VAR_LABEL must be plain text - no control characters'
else if ([...label].length > 256)
reason = 'VAR_LABEL exceeds 256 characters'
else if (!logic) reason = 'VAR_LOGIC is required'
else if (logic.length > 4000)
reason = 'VAR_LOGIC exceeds 4000 characters'
else if (logic.includes(';'))
reason = 'VAR_LOGIC must be a single expression - remove the ;'
else if (logic.includes('&') || logic.includes('%'))
reason = 'VAR_LOGIC must not contain macro triggers'
else if (logic.includes('/*') || logic.includes('*/'))
reason = 'VAR_LOGIC must not contain comments'
else if (DENY_FUNCS.test(unquoted(logic).toUpperCase()))
reason =
'VAR_LOGIC may not call a function that submits code or reaches the file system, the environment or the macro symbol table'
if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
}
// ─── The graph of the resulting configuration ──────────────────────────────
// The incoming live rows plus the current rows they do not replace. Only
// active rows are in it, because the engine reads nothing else - an inactive
// row can neither resolve a dependency nor be resolved by one.
const controlDir = mpeDataDir
const loadControl = makeTableLoader(controlDir)
const current = loadControl('MPE_DERIVED')
const nowSeconds = Date.now() / 1000
// mpeDerived reads a row as current when TX_TO is missing, zero or in the
// future (dcMockUtils.js, the read-time engine), so the same test is used
// here - a seeded row with no TX_TO is live for the engine, and a dependency
// on it has to validate
const isCurrent = (row) =>
!Number(colVal(row, 'TX_TO')) || Number(colVal(row, 'TX_TO')) > nowSeconds
const stagedKeys = new Set(rows.map(keyOf))
const stagedLive = rows.filter(isLive)
// the control-table arm of the SAS hook is limited to the tables this
// submission touches, so another table's configuration cannot block it
const stagedTables = new Set(
rows.map((row) => [field(row, 'LIBREF'), field(row, 'DSN')].join('.'))
)
/** libref.dsn.name - the key every graph lookup uses. */
const nodeKey = (n) => [n.libref, n.dsn, n.name].join('.')
const graph = new Map()
for (const row of stagedLive) {
graph.set(keyOf(row), {
libref: field(row, 'LIBREF'),
dsn: field(row, 'DSN'),
name: field(row, 'VAR_NM'),
deps: field(row, 'VAR_DEPS').split(/\s+/).filter(Boolean),
logic: strip(colVal(row, 'VAR_LOGIC')).toUpperCase()
})
}
const currentRows =
current && Array.isArray(current.rows)
? current.rows.filter(
(r) =>
isCurrent(r) &&
Number(colVal(r, 'ACTIVE')) === 1 &&
stagedTables.has(
[up(colVal(r, 'LIBREF')), up(colVal(r, 'DSN'))].join(
'.'
)
) &&
!stagedKeys.has(
[
up(colVal(r, 'LIBREF')),
up(colVal(r, 'DSN')),
up(colVal(r, 'VAR_NM'))
].join('.')
)
)
: []
for (const row of currentRows) {
graph.set(
[
up(colVal(row, 'LIBREF')),
up(colVal(row, 'DSN')),
up(colVal(row, 'VAR_NM'))
].join('.'),
{
libref: up(colVal(row, 'LIBREF')),
dsn: up(colVal(row, 'DSN')),
name: up(colVal(row, 'VAR_NM')),
deps: up(colVal(row, 'VAR_DEPS')).split(/\s+/).filter(Boolean),
logic: strip(colVal(row, 'VAR_LOGIC')).toUpperCase()
}
)
}
/** Names this load leaves inactive, which therefore cannot resolve a dep. */
const inactive = new Set()
const liveKeys = new Set(stagedLive.map(keyOf))
for (const row of rows) {
const key = keyOf(row)
if (!liveKeys.has(key)) inactive.add(key)
}
if (current && Array.isArray(current.rows)) {
for (const row of current.rows) {
const key = [
up(colVal(row, 'LIBREF')),
up(colVal(row, 'DSN')),
up(colVal(row, 'VAR_NM'))
].join('.')
if (
isCurrent(row) &&
Number(colVal(row, 'ACTIVE')) !== 1 &&
!liveKeys.has(key)
) {
inactive.add(key)
}
}
}
for (const node of graph.values()) {
for (const dep of node.deps) {
const key = [node.libref, node.dsn, dep].join('.')
if (inactive.has(key) && !graph.has(key)) {
return reject(
`MPE_DERIVED: ${node.name}: VAR_DEPS lists ${dep}, which is not active after this load - the engine cannot resolve it`
)
}
}
}
// ─── Self reference, undeclared and stale dependencies ─────────────────────
/** The identifiers an expression uses, with quoted runs removed. */
const identifiers = (logic) =>
logic
.replace(/'([^']|'')*'/g, ' ')
.replace(/"([^"]|"")*"/g, ' ')
.replace(/[^A-Za-z0-9_]/g, ' ')
.toUpperCase()
.split(/\s+/)
.filter(Boolean)
// scoped per table, as the server does: a name that is a derived variable
// on another table says nothing about this one
const graphKeys = new Set([...graph.values()].map((n) => nodeKey(n)))
// one check at a time, in the order the server runs them, so the message a
// row gets is the one it would get there
for (const node of graph.values()) {
if (node.deps.includes(node.name)) {
return reject(
`MPE_DERIVED: ${node.name}: VAR_DEPS may not list itself`
)
}
}
for (const node of graph.values()) {
for (const token of identifiers(node.logic)) {
if (
graphKeys.has([node.libref, node.dsn, token].join('.')) &&
token !== node.name &&
!node.deps.includes(token)
) {
return reject(
`MPE_DERIVED: ${node.name}: VAR_LOGIC references ${token}, which is not in VAR_DEPS`
)
}
}
}
for (const node of graph.values()) {
const tokens = identifiers(node.logic)
for (const dep of node.deps) {
if (!tokens.includes(dep)) {
return reject(
`MPE_DERIVED: ${node.name}: VAR_DEPS lists ${dep}, which VAR_LOGIC does not reference`
)
}
}
}
// ─── A derived variable may not be named by a row level security rule ──────
const rls = loadControl('MPE_ROW_LEVEL_SECURITY')
if (rls && Array.isArray(rls.rows)) {
for (const node of graph.values()) {
const match = rls.rows.find(
(r) =>
up(colVal(r, 'RLS_LIBREF')) === node.libref &&
up(colVal(r, 'RLS_TABLE')) === node.dsn &&
up(colVal(r, 'RLS_VARIABLE_NM')) === node.name &&
Number(colVal(r, 'RLS_ACTIVE')) === 1 &&
isCurrent(r)
)
if (match) {
return reject(
`MPE_DERIVED: ${node.name} is named by a row level security rule on ${node.libref}.${node.dsn} - derived variables cannot be used there`
)
}
}
}
// ─── Cycles ────────────────────────────────────────────────────────────────
// Repeatedly place the variables whose derived inputs are already placed; a
// round that places nothing leaves a cycle. The path is walked so the
// message names it, as the SAS hook does.
const nodeKeys = new Map(
[...graph.values()].map((n) => [n, [n.libref, n.dsn, n.name].join('.')])
)
const placed = new Set()
let remaining = [...graph.values()]
while (remaining.length > 0) {
const ready = remaining.filter((n) =>
n.deps.every((d) => {
const key = [n.libref, n.dsn, d].join('.')
return !graph.has(key) || placed.has(key)
})
)
if (ready.length === 0) {
// walk the unplaced edges so the message names the cycle
const start = remaining[0]
const path = [start.name]
const visited = new Set([nodeKeys.get(start)])
let cursor = start
for (let i = 0; i < remaining.length; i++) {
const parentKey = cursor.deps
.map((d) => [cursor.libref, cursor.dsn, d].join('.'))
.find((key) => graph.has(key) && !placed.has(key))
if (!parentKey) break
path.push(graph.get(parentKey).name)
if (visited.has(parentKey)) break
visited.add(parentKey)
cursor = graph.get(parentKey)
}
return reject(
`MPE_DERIVED: VAR_DEPS contains a cycle: ${path.join(' -> ')}`
)
}
for (const node of ready) placed.add(nodeKeys.get(node))
remaining = remaining.filter((n) => !placed.has(nodeKeys.get(n)))
}
})()
+17 -1
View File
@@ -21,9 +21,25 @@ const tableName = libds.split('.')[1] || ''
// ─── Load table data (searches all libref data folders) ──────────────────────
const _loaded = loadTableAnyLib(tableName) || {}
const tableData = _loaded.data
let tableData = _loaded.data
const tableLibref = _loaded.libref || libref
// ─── Derived (computed) variables (mpe_derived.sas) ─────────────────────────
// Present in the schema and in the rows before the filter is applied, so the
// filter may reference them and the values below are computed from the same
// row set the grid shows.
const controlDataDir = libDataDir(dcLibref)
const derived = mpeDerived({
libref: tableLibref,
table: tableName,
dataDir: controlDataDir,
columns: tableData ? tableData.columns : []
})
if (tableData && derived.length > 0) {
tableData = mpeDeriveRows(tableData, derived)
}
// ─── Resolve column metadata ─────────────────────────────────────────────────
let colMeta = null
+2 -1
View File
@@ -1,7 +1,8 @@
let mimeType = `application/octet-stream`
let filename = `dc-download`
if (libref) filename = `${libref}.ddl`
if (libref && ds) filename = `${libref}_${ds}.ddl`
else if (libref) filename = `${libref}.ddl`
fs.writeFile(_SASJS_WEBOUT_HEADERS, `Content-type: ${mimeType} \n Content-Disposition: attachment;filename=\"${filename}\"`, function (err) {
if (err) throw err;
+26 -6
View File
@@ -49,6 +49,23 @@ if (!tableData && dsn) {
if (found) tableData = found.data
}
// ─── Derived variables (mpe_derived.sas) ─────────────────────────────────────
// A stored filter can reference a derived column, so the derivation has to run
// before the filter. The columns themselves are dropped from the output - they
// belong to the EDIT screen, as viewdata.sas drops them.
// a copy: the derivation below returns a new column list, and every use here
// (cols, search, rows, vars) must keep looking at the physical columns only
const physicalColumns = [...(tableData && tableData.columns ? tableData.columns : [])]
const derived = mpeDerived({
libref: libref,
table: dsn,
dataDir: mpeDataDir,
columns: physicalColumns
})
if (tableData && derived.length > 0) {
tableData = mpeDeriveRows(tableData, derived)
}
// ─── Apply the stored filter (mpe_filtermaster, VIEW mode) ───────────────────
// Mirrors viewdata.sas: %mpe_filtermaster(VIEW,&libds,filter_rk=&filter_rk)
// Row Level Security and any saved filter are applied here, before the search.
@@ -103,7 +120,7 @@ let nobs = 0
if (tableData && tableData.columns) {
// mpGetcols, the JS counterpart of %mp_getcols.sas - the macro viewdata.sas
// calls for its own column list.
cols = mpGetcols(tableData.columns).map(c => {
cols = mpGetcols(physicalColumns).map(c => {
const ddtype = c.DDTYPE === 'CHARACTER' ? 'C' : c.DDTYPE === 'NUMERIC' ? 'N' : c.DDTYPE
const formatStr = c.TYPE === 'N' ? (c.FORMAT || '8.') : ('$' + c.LENGTH + '.')
return {
@@ -119,7 +136,10 @@ if (tableData && tableData.columns) {
let rows
if (searchType) {
const matches = filteredRows.filter(row => rowMatchesSearch(row, tableData.columns, searchType, searchVal))
// the derived columns are dropped before the search, so only the
// physical columns are searched (viewdata.sas drops them in the
// vwsearch data step mp_searchdata then reads)
const matches = filteredRows.filter(row => rowMatchesSearch(row, physicalColumns, searchType, searchVal))
// mp_searchdata caps its output at outobs=&DC_MAXOBS_WEBVIEW, and
// viewdata.sas then keeps only `if _n_<&DC_MAXOBS_WEBVIEW` of them.
nobs = Math.min(matches.length, maxObsWebView)
@@ -134,7 +154,7 @@ if (tableData && tableData.columns) {
// viewdata rows: all values as strings (formatted, SAS proc-print style)
viewdata = rows.map(r => {
const row = {}
for (const col of tableData.columns) {
for (const col of physicalColumns) {
const lcName = col.name.toLowerCase()
let val = r[col.name]
if (val === undefined) val = r[lcName]
@@ -174,15 +194,15 @@ if (tableData && tableData.columns) {
// (SAS missing), character columns get "".
// A search that matches nothing sends no rows at all (the client shows
// its "no data found" panel instead of an empty grid).
if (viewdata.length === 0 && !searchType && tableData.columns.length > 0) {
if (viewdata.length === 0 && !searchType && physicalColumns.length > 0) {
const emptyRow = {}
for (const col of tableData.columns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
for (const col of physicalColumns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
viewdata = [emptyRow]
}
// $viewdata vars: keep original SAS types (num stays num, char stays char)
vars = {}
for (const col of tableData.columns) {
for (const col of physicalColumns) {
vars[col.name] = sasVarsEntry(col)
}
}
+92
View File
@@ -0,0 +1,92 @@
// Collect the SASjs Server drive for an app into a mock-snapshot.json, which
// `sasjs build` ships as the browser build's starting drive (the alternative is
// a drive seeded from the compiled services alone, i.e. no DC data).
//
// Usage: node sas/mocks/snapshot.js <drive-root> <appLoc> <out-file>
// drive-root the SASjs Server drive folder (the `drive` folder of a server install)
// appLoc the deployed app location, e.g. /Public/app/dc
// out-file where to write the snapshot, e.g. sas/mocks/sasjsbuild/mock-snapshot.json
//
// Every file is written as a string because the browser runtime's drive holds
// text only; a file that does not survive a UTF-8 round trip is skipped rather
// than mangled.
const fs = require('fs')
const path = require('path')
const [driveRoot, appLoc, outFile, excludeArg] = process.argv.slice(2)
if (!driveRoot || !appLoc || !outFile) {
console.error(
'Usage: node sas/mocks/snapshot.js <drive-root> <appLoc> <out-file> [exclude,exclude]'
)
process.exit(1)
}
const app = '/' + appLoc.replace(/^\/+|\/+$/g, '')
const driveFolder = path.join(
driveRoot,
'files',
...app.split('/').filter(Boolean)
)
const driveKey = `/sasjs/drive/files${app}`
// Comma-separated paths relative to the app, e.g. "services/web9" - the
// streamed frontend is copied into the drive as a service folder, is already the
// site itself, and would otherwise duplicate the whole app inside the snapshot.
const excludes = (excludeArg || '')
.split(',')
.map((value) => value.trim().replace(/^\/+|\/+$/g, ''))
.filter(Boolean)
if (!fs.existsSync(driveFolder)) {
console.error('No drive folder at ' + driveFolder)
process.exit(1)
}
const files = {}
const skipped = []
const walk = (folder) => {
for (const entry of fs.readdirSync(folder, { withFileTypes: true })) {
const absolute = path.join(folder, entry.name)
if (entry.isDirectory()) {
walk(absolute)
continue
}
const relative = path
.relative(driveFolder, absolute)
.split(path.sep)
.join('/')
if (
excludes.some(
(prefix) => relative === prefix || relative.startsWith(prefix + '/')
)
) {
continue
}
const buffer = fs.readFileSync(absolute)
const text = buffer.toString('utf8')
if (!Buffer.from(text, 'utf8').equals(buffer)) {
skipped.push(relative)
continue
}
files[`${driveKey}/${relative}`] = text
}
}
walk(driveFolder)
fs.mkdirSync(path.dirname(path.resolve(outFile)), { recursive: true })
fs.writeFileSync(path.resolve(outFile), JSON.stringify({ files }, null, 2))
console.log(
`Snapshot: ${Object.keys(files).length} file(s) from ${driveFolder} -> ${outFile}` +
(skipped.length
? `, skipped ${skipped.length} non-text file(s): ${skipped.slice(0, 3).join(', ')}`
: '')
)
+174 -415
View File
@@ -6,8 +6,8 @@
"": {
"name": "dc-sas",
"dependencies": {
"@sasjs/cli": "4.20.5",
"@sasjs/core": "5.2.8"
"@sasjs/cli": "6.2.0",
"@sasjs/core": "7.0.1"
}
},
"node_modules/@asamuzakjp/css-color": {
@@ -140,56 +140,49 @@
}
},
"node_modules/@fast-csv/format": {
"version": "4.3.5",
"resolved": "https://registry.npmjs.org/@fast-csv/format/-/format-4.3.5.tgz",
"integrity": "sha512-8iRn6QF3I8Ak78lNAa+Gdl5MJJBM5vRHivFtMRUWINdevNo00K7OXxS2PshawLKTejVwieIlPmK5YlLu6w4u8A==",
"version": "5.0.7",
"resolved": "https://registry.npmjs.org/@fast-csv/format/-/format-5.0.7.tgz",
"integrity": "sha512-VdypoRxv7PF+LsyPouTMKdB0d76hync+gLpgdNqfqVK44MsgW4oiCJSdrki2FisWT7v2QGUYDHjp4L7w5oO6gw==",
"license": "MIT",
"dependencies": {
"@types/node": "^14.0.1",
"lodash.escaperegexp": "^4.1.2",
"lodash.isboolean": "^3.0.3",
"lodash.isequal": "^4.5.0",
"lodash.isfunction": "^3.0.9",
"lodash.isnil": "^4.0.0"
"lodash.escaperegexp": "^4.1.2"
}
},
"node_modules/@sasjs/adapter": {
"version": "4.19.1",
"resolved": "https://registry.npmjs.org/@sasjs/adapter/-/adapter-4.19.1.tgz",
"integrity": "sha512-IpN6UPSdqVb5cAWEBt+jG/i0rhgCubdyp5kxdouuX3ifL313oJW2ZPVUNUvs0+ZGzsXWcGDTNoEzzTrGKiqJlQ==",
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/@sasjs/adapter/-/adapter-5.1.1.tgz",
"integrity": "sha512-rso6+xRpZhFuuUbgHmGztznuse4xZjlu7+dOcYMUMJezAd5p/xdCt+wvv4eIKejsMWDcIteSeRPOkgKr0TlxWg==",
"license": "ISC",
"dependencies": {
"@sasjs/utils": "^3.6.0",
"@sasjs/utils": "4.0.1",
"axios": "1.20.0",
"axios-cookiejar-support": "5.0.5",
"form-data": "4.0.6",
"https": "1.0.0",
"tough-cookie": "4.1.3"
"tough-cookie": "5.1.2"
},
"engines": {
"node": ">=22"
}
},
"node_modules/@sasjs/cli": {
"version": "4.20.5",
"resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.5.tgz",
"integrity": "sha512-/HSpueCFS+SvXas9FwfP8qoD9skhdTDE5r7dT9Xlrs7HGCBjxDYvSiw6tW0GIjfw3/49Yf8TCVDFBsISang1hg==",
"version": "6.2.0",
"resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-6.2.0.tgz",
"integrity": "sha512-zovAOCmyPqcm+gooAZgwvgGH8n5cas3sY0lA80+hyFoFtmZ8VjW/bh19LM8OVwheT9dBCRCx4fnHwjy5XGpzvQ==",
"license": "ISC",
"dependencies": {
"@sasjs/adapter": "4.19.1",
"@sasjs/core": "5.2.10",
"@sasjs/lint": "4.1.0",
"@sasjs/utils": "3.6.2",
"@sasjs/adapter": "^5.1.1",
"@sasjs/core": "^6.0.0",
"@sasjs/utils": "^4.1.0",
"axios": "1.20.0",
"chalk": "4.1.2",
"dotenv": "17.4.2",
"find": "0.3.0",
"js-base64": "3.9.3",
"dotenv": "18.0.8",
"js-base64": "3.9.4",
"jsdom": "26.0.0",
"jwt-decode": "3.1.2",
"lodash.groupby": "4.6.0",
"lodash.uniqby": "4.7.0",
"node-graphviz": "0.1.1",
"node-powershell": "5.0.1",
"ora": "5.4.1",
"prompts": "2.4.2",
"shelljs": "0.10.0",
"ssl-root-cas": "1.3.1",
"xml": "1.0.1",
"yargs": "17.6.2",
@@ -198,65 +191,129 @@
},
"bin": {
"sasjs": "build/index.js"
},
"engines": {
"node": ">=22"
},
"peerDependencies": {
"@sasjs/lint": "^4.3.0"
}
},
"node_modules/@sasjs/cli/node_modules/@sasjs/core": {
"version": "5.2.10",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.10.tgz",
"integrity": "sha512-Lk/ENHVuXaeAl8M0+Mjh1rcBSdFt4tIM6OPxPPmk78ZD0gYFqAOUZreqIoV9EN+VwPDqYsUPDqXzh0crW6j8OA==",
"version": "6.1.4",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.4.tgz",
"integrity": "sha512-vIF2LIftxPWg4CvO3i5raEfrHFHY/6S+MqVGpjRR71dgjLRJEYNyPJgGqD6BCZwHv5V6/2nDnBi/BIIMiLXfzA==",
"license": "MIT"
},
"node_modules/@sasjs/cli/node_modules/shelljs": {
"version": "0.8.5",
"resolved": "https://registry.npmjs.org/shelljs/-/shelljs-0.8.5.tgz",
"integrity": "sha512-TiwcRcrkhHvbrZbnRcFYMLl30Dfov3HKqzp5tO5b4pt6G/SezKcYhmDg15zXVBswHmctSAQKznqNW2LO5tTDow==",
"license": "BSD-3-Clause",
"node_modules/@sasjs/cli/node_modules/@sasjs/utils": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.1.0.tgz",
"integrity": "sha512-aC5pOugg6/2jKQR7ZaYsLtbsFy/iVacsrN3/Kzh6BouJUszuVrr13HMrQUSKcWL+rJzxYof7ymmmCl+vE6YwAw==",
"license": "ISC",
"dependencies": {
"glob": "^7.0.0",
"interpret": "^1.0.0",
"rechoir": "^0.6.2"
},
"bin": {
"shjs": "bin/shjs"
"@fast-csv/format": "5.0.7",
"@types/fs-extra": "11.0.4",
"@types/prompts": "2.4.9",
"chalk": "4.1.1",
"consola": "3.4.2",
"fs-extra": "11.4.1",
"jwt-decode": "4.0.0",
"prompts": "2.4.2"
},
"engines": {
"node": ">=4"
"node": ">=22"
}
},
"node_modules/@sasjs/cli/node_modules/@sasjs/utils/node_modules/chalk": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
"integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
"license": "MIT",
"dependencies": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/@sasjs/core": {
"version": "5.2.8",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.8.tgz",
"integrity": "sha512-+K8tXTrsw52NR1y5MFHW0+Jf9euL+GukwqA79FIHHu4Oj+BhHBm1uZiyB5tPrvwl3V8mY1CQhMOT83//bV+jWg==",
"version": "7.0.1",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-7.0.1.tgz",
"integrity": "sha512-M75475GuOMTmbmrB8V/Hgcwm2j3PdIHlgmkbjtT88ALSQuzRPBAjEK2IDDYIUn6rsVJG9aZWvUpRV7fzrOjK/w==",
"license": "MIT"
},
"node_modules/@sasjs/lint": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-4.1.0.tgz",
"integrity": "sha512-Vpg1s0/zdPMFJ0hRXNXk53HhE3kfRr5gFRS0kvrse+bxgcbqih1qqVW7yZ2krhSCYylp9z13eBXLhXbRrj70VQ==",
"hasInstallScript": true,
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-4.3.0.tgz",
"integrity": "sha512-65cjy4IjA4PgSlicOruF0DI9N2HO3LJqtgphsZCg8g7N/StE2FWLJLJbD1N4OHxN6htrUfj5SEawJJiHo5T9Yw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@sasjs/utils": "3.6.2",
"@sasjs/utils": "4.0.0",
"ignore": "7.0.11"
},
"engines": {
"node": ">=22"
}
},
"node_modules/@sasjs/lint/node_modules/@sasjs/utils": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.0.0.tgz",
"integrity": "sha512-lARoCnRM4gSVf8ShoP5elHi7W28K/14MK037dQ0ABgMlP3+r6OuhAZYwK/a70tED47tKxyIoCAKLdP6jrG0ewQ==",
"license": "ISC",
"peer": true,
"dependencies": {
"@fast-csv/format": "5.0.7",
"@types/fs-extra": "11.0.4",
"@types/prompts": "2.4.9",
"chalk": "4.1.1",
"consola": "3.4.2",
"fs-extra": "11.4.1",
"jwt-decode": "4.0.0",
"prompts": "2.4.2"
},
"engines": {
"node": ">=22"
}
},
"node_modules/@sasjs/lint/node_modules/chalk": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
"integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
"license": "MIT",
"peer": true,
"dependencies": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/@sasjs/utils": {
"version": "3.6.2",
"resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-3.6.2.tgz",
"integrity": "sha512-AfC5OJuJx3NAbntKq2J8E8NNpRXthYfuqOyp2lOmOkTctL3mAfrA54U/cdPQvrlftXNxpK9P2U0RU10qNjlEzQ==",
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-4.0.1.tgz",
"integrity": "sha512-aNEbOJFZ2Ru71/XkHKtywexBQYdst8ovq0a9vuVO4subZuKMjRtYcn4kTRr9VVIz65p8xGtVUkFEf+6t9Vbfcg==",
"license": "ISC",
"dependencies": {
"@fast-csv/format": "4.3.5",
"@fast-csv/format": "5.0.7",
"@types/fs-extra": "11.0.4",
"@types/prompts": "2.0.13",
"@types/prompts": "2.4.9",
"chalk": "4.1.1",
"cli-table": "0.3.6",
"consola": "2.15.0",
"find": "0.3.0",
"fs-extra": "11.3.0",
"jwt-decode": "3.1.2",
"prompts": "2.4.1",
"valid-url": "1.0.9"
"consola": "3.4.2",
"fs-extra": "11.4.1",
"jwt-decode": "4.0.0",
"prompts": "2.4.2"
},
"engines": {
"node": ">=22"
}
},
"node_modules/@sasjs/utils/node_modules/chalk": {
@@ -275,19 +332,6 @@
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/@sasjs/utils/node_modules/prompts": {
"version": "2.4.1",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
"integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
"license": "MIT",
"dependencies": {
"kleur": "^3.0.3",
"sisteransi": "^1.0.5"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/@types/fs-extra": {
"version": "11.0.4",
"resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-11.0.4.tgz",
@@ -308,18 +352,22 @@
}
},
"node_modules/@types/node": {
"version": "14.18.63",
"resolved": "https://registry.npmjs.org/@types/node/-/node-14.18.63.tgz",
"integrity": "sha512-fAtCfv4jJg+ExtXhvCkCqUKZ+4ok/JQk01qDKhL5BDDoS3AxKXhV5/MAVUZyQnSEd2GT92fkgZl0pz0Q0AzcIQ==",
"license": "MIT"
},
"node_modules/@types/prompts": {
"version": "2.0.13",
"resolved": "https://registry.npmjs.org/@types/prompts/-/prompts-2.0.13.tgz",
"integrity": "sha512-jwMOIGy49VruR/gYehhJYgpVzB+EVpEE7t7j9m1oTo4HMpOe7KmsyqdBuoxAzA5B4caUgx0cKrWr7wUEqMXJ7Q==",
"version": "26.6.5",
"resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.5.tgz",
"integrity": "sha512-uMzsv9stDdofIZBx4UGh4Z9ZgXnGC9seRDscilBcTHAeOFk67MZg/k9bWVfO+bk4qZmG+l52hRnLwUiH2LnJTg==",
"license": "MIT",
"dependencies": {
"@types/node": "*"
"undici-types": "~8.9.0"
}
},
"node_modules/@types/prompts": {
"version": "2.4.9",
"resolved": "https://registry.npmjs.org/@types/prompts/-/prompts-2.4.9.tgz",
"integrity": "sha512-qTxFi6Buiu8+50/+3DGIWLHM6QuWsEKugJnnP6iv2Mc4ncxE4A/OJkjuVOA+5X0X1S/nq5VJRa8Lu+nwcvbrKA==",
"license": "MIT",
"dependencies": {
"@types/node": "*",
"kleur": "^3.0.3"
}
},
"node_modules/accumulate-stream": {
@@ -402,12 +450,6 @@
"tough-cookie": ">=4.0.0"
}
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"license": "MIT"
},
"node_modules/base64-js": {
"version": "1.5.1",
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
@@ -439,16 +481,6 @@
"readable-stream": "^3.4.0"
}
},
"node_modules/brace-expansion": {
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz",
"integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==",
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0",
"concat-map": "0.0.1"
}
},
"node_modules/buffer": {
"version": "5.7.1",
"resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
@@ -559,17 +591,6 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/cli-table": {
"version": "0.3.6",
"resolved": "https://registry.npmjs.org/cli-table/-/cli-table-0.3.6.tgz",
"integrity": "sha512-ZkNZbnZjKERTY5NwC2SeMeLeifSPq/pubeRoTpdr3WchLlnZg6hEgvHkK5zL7KNFdd9PmHN8lxrENUwI3cE8vQ==",
"dependencies": {
"colors": "1.0.3"
},
"engines": {
"node": ">= 0.2.0"
}
},
"node_modules/cliui": {
"version": "8.0.1",
"resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
@@ -611,15 +632,6 @@
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
"license": "MIT"
},
"node_modules/colors": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/colors/-/colors-1.0.3.tgz",
"integrity": "sha512-pFGrxThWcWQ2MsAz6RtgeWe4NK2kUE1WfsrvvlctdII745EW9I0yflqhe7++M5LEc7bV2c/9/5zc8sFcpL0Drw==",
"license": "MIT",
"engines": {
"node": ">=0.1.90"
}
},
"node_modules/combined-stream": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
@@ -632,17 +644,14 @@
"node": ">= 0.8"
}
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==",
"license": "MIT"
},
"node_modules/consola": {
"version": "2.15.0",
"resolved": "https://registry.npmjs.org/consola/-/consola-2.15.0.tgz",
"integrity": "sha512-vlcSGgdYS26mPf7qNi+dCisbhiyDnrN1zaRbw3CSuc2wGOMEGGPsp46PdRG5gqXwgtJfjxDkxRNAgRPr1B77vQ==",
"license": "MIT"
"version": "3.4.2",
"resolved": "https://registry.npmjs.org/consola/-/consola-3.4.2.tgz",
"integrity": "sha512-5IKcdX0nnYavi6G7TtOhwkYzyjfJlatbjMjuLSfE2kYT5pMDOilZ4OvMhi637CcDICTmz3wARPoyhqyX1Y+XvA==",
"license": "MIT",
"engines": {
"node": "^14.18.0 || >=16.10.0"
}
},
"node_modules/cssstyle": {
"version": "4.6.0",
@@ -715,10 +724,13 @@
}
},
"node_modules/dotenv": {
"version": "17.4.2",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
"integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
"version": "18.0.8",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-18.0.8.tgz",
"integrity": "sha512-2KDeoFoK/zRtcnbunCDlLG5VWVMcyqeQc6iK65o231nfNAcuGrKoqUQfJFUtBI+QlShk5HKIHxwebBtUl3rEfA==",
"license": "BSD-2-Clause",
"bin": {
"dotenv": "dist/index.cjs"
},
"engines": {
"node": ">=12"
},
@@ -818,15 +830,6 @@
"integrity": "sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==",
"license": "MIT"
},
"node_modules/find": {
"version": "0.3.0",
"resolved": "https://registry.npmjs.org/find/-/find-0.3.0.tgz",
"integrity": "sha512-iSd+O4OEYV/I36Zl8MdYJO0xD82wH528SaCieTVHhclgiYNe9y+yPKSwK+A7/WsmHL1EZ+pYUJBXWTL5qofksw==",
"license": "MIT",
"dependencies": {
"traverse-chain": "~0.1.0"
}
},
"node_modules/follow-redirects": {
"version": "1.16.1",
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.1.tgz",
@@ -864,9 +867,9 @@
}
},
"node_modules/fs-extra": {
"version": "11.3.0",
"resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.3.0.tgz",
"integrity": "sha512-Z4XaCL6dUDHfP/jT25jJKMmtxvuwbkrD1vNSMFlo9lNLY2c5FHYSQgHPRZUjAB26TpDEoW9HCOgplrdbaPV/ew==",
"version": "11.4.1",
"resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.4.1.tgz",
"integrity": "sha512-KYAb4c9BJQI6QqGKthV68OHe0badztdXJWKo0WtBA9IuCFPTKvE5ZdUBglP833aMjhaSPNO4A5j/EkzZtGlKjA==",
"license": "MIT",
"dependencies": {
"graceful-fs": "^4.2.0",
@@ -877,12 +880,6 @@
"node": ">=14.14"
}
},
"node_modules/fs.realpath": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
"integrity": "sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==",
"license": "ISC"
},
"node_modules/function-bind": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
@@ -938,27 +935,6 @@
"node": ">= 0.4"
}
},
"node_modules/glob": {
"version": "7.2.3",
"resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz",
"integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==",
"deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me",
"license": "ISC",
"dependencies": {
"fs.realpath": "^1.0.0",
"inflight": "^1.0.4",
"inherits": "2",
"minimatch": "^3.1.1",
"once": "^1.3.0",
"path-is-absolute": "^1.0.0"
},
"engines": {
"node": "*"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@@ -1074,12 +1050,6 @@
"node": ">= 14"
}
},
"node_modules/https": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/https/-/https-1.0.0.tgz",
"integrity": "sha512-4EC57ddXrkaF0x83Oj8sM6SLQHAWXw90Skqu2M4AEWENZ3F02dFJE/GARA8igO79tcgYqGrD7ae4f5L3um2lgg==",
"license": "ISC"
},
"node_modules/https-proxy-agent": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz",
@@ -1142,51 +1112,17 @@
"resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.11.tgz",
"integrity": "sha512-YChdK5txDjwGUvgR7oCJcLGkwi3LDh8Zx8tS7ndYciLEg/oOCL26VUBSUEtgS7EjiybcBqFxh5j0rAdDX7/bbg==",
"license": "MIT",
"peer": true,
"engines": {
"node": ">= 4"
}
},
"node_modules/inflight": {
"version": "1.0.6",
"resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
"integrity": "sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==",
"deprecated": "This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.",
"license": "ISC",
"dependencies": {
"once": "^1.3.0",
"wrappy": "1"
}
},
"node_modules/inherits": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"license": "ISC"
},
"node_modules/interpret": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/interpret/-/interpret-1.4.0.tgz",
"integrity": "sha512-agE4QfB2Lkp9uICn7BAqoscw4SZP9kTE2hxiFI3jBPmXJfdqiahTbUuKGsMoN2GtqL9AxhYioAcVvgsb1HvRbA==",
"license": "MIT",
"engines": {
"node": ">= 0.10"
}
},
"node_modules/is-core-module": {
"version": "2.17.0",
"resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.17.0.tgz",
"integrity": "sha512-J/vG0zBCbIKOQFfufSwyXdMrsohyJIUNkrnmo6WZGzoM7tr/lsbfW5b2BvisL6zsyMzK9UxV9L6c7AoFbyXHOA==",
"license": "MIT",
"dependencies": {
"hasown": "^2.0.4"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/is-docker": {
"version": "2.2.1",
"resolved": "https://registry.npmjs.org/is-docker/-/is-docker-2.2.1.tgz",
@@ -1251,9 +1187,9 @@
}
},
"node_modules/js-base64": {
"version": "3.9.3",
"resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.3.tgz",
"integrity": "sha512-uwYQp+VJ38FVvtim6qNbit6e9uT6dwWQ4Y1+H9TxhW5hcHjpHwoxlR0nMpqUmIFOmu4VqMxwdJA88gIVuZJQ/g==",
"version": "3.9.4",
"resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.4.tgz",
"integrity": "sha512-PtOMXpEGuP0RRiRXsjzHzl44dMHxSu2CPvAhinupR1tBa88me+1DPqsobl7eupn5UukjLkln1ZnAOAOXOrYG0Q==",
"license": "BSD-3-Clause"
},
"node_modules/jsdom": {
@@ -1309,18 +1245,6 @@
"node": ">= 14"
}
},
"node_modules/jsdom/node_modules/tough-cookie": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
"integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
"license": "BSD-3-Clause",
"dependencies": {
"tldts": "^6.1.32"
},
"engines": {
"node": ">=16"
}
},
"node_modules/jsonfile": {
"version": "6.2.1",
"resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.1.tgz",
@@ -1334,10 +1258,13 @@
}
},
"node_modules/jwt-decode": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/jwt-decode/-/jwt-decode-3.1.2.tgz",
"integrity": "sha512-UfpWE/VZn0iP50d8cz9NrZLM9lSWhcJ+0Gt/nm4by88UL+J1SiKN8/5dkjMmbEzwL2CAe+67GsegCbIKtbp75A==",
"license": "MIT"
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/jwt-decode/-/jwt-decode-4.0.0.tgz",
"integrity": "sha512-+KJGIyHgkGuIq3IEBNftfhW/LfWhXUIY6OmyVWjliu5KH1y0fw7VQ8YndE2O4qZdMSd9SqbnC8GOcZEy0Om7sA==",
"license": "MIT",
"engines": {
"node": ">=18"
}
},
"node_modules/kind-of": {
"version": "6.0.3",
@@ -1363,37 +1290,6 @@
"integrity": "sha512-TM9YBvyC84ZxE3rgfefxUWiQKLilstD6k7PTGt6wfbtXF8ixIJLOL3VYyV/z+ZiPLsVxAsKAFVwWlWeb2Y8Yyw==",
"license": "MIT"
},
"node_modules/lodash.groupby": {
"version": "4.6.0",
"resolved": "https://registry.npmjs.org/lodash.groupby/-/lodash.groupby-4.6.0.tgz",
"integrity": "sha512-5dcWxm23+VAoz+awKmBaiBvzox8+RqMgFhi7UvX9DHZr2HdxHXM/Wrf8cfKpsW37RNrvtPn6hSwNqurSILbmJw==",
"license": "MIT"
},
"node_modules/lodash.isboolean": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
"integrity": "sha512-Bz5mupy2SVbPHURB98VAcw+aHh4vRV5IPNhILUCsOzRmsTmSQ17jIuqopAentWoehktxGd9e/hbIXq980/1QJg==",
"license": "MIT"
},
"node_modules/lodash.isequal": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/lodash.isequal/-/lodash.isequal-4.5.0.tgz",
"integrity": "sha512-pDo3lu8Jhfjqls6GkMgpahsF9kCyayhgykjyLMNFTKWrpVdAQtYyB4muAMWozBB4ig/dtWAmsMxLEI8wuz+DYQ==",
"deprecated": "This package is deprecated. Use require('node:util').isDeepStrictEqual instead.",
"license": "MIT"
},
"node_modules/lodash.isfunction": {
"version": "3.0.9",
"resolved": "https://registry.npmjs.org/lodash.isfunction/-/lodash.isfunction-3.0.9.tgz",
"integrity": "sha512-AirXNj15uRIMMPihnkInB4i3NHeb4iBtNg9WRWuK2o31S+ePwwNmDPaTL3o7dTJ+VXNZim7rFs4rxN4YU1oUJw==",
"license": "MIT"
},
"node_modules/lodash.isnil": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/lodash.isnil/-/lodash.isnil-4.0.0.tgz",
"integrity": "sha512-up2Mzq3545mwVnMhTDMdfoG1OurpA/s5t88JmQX809eH3C8491iu2sfKhTfhQtKY78oPNhiaHJUpT/dUDAAtng==",
"license": "MIT"
},
"node_modules/lodash.uniqby": {
"version": "4.7.0",
"resolved": "https://registry.npmjs.org/lodash.uniqby/-/lodash.uniqby-4.7.0.tgz",
@@ -1461,18 +1357,6 @@
"node": ">=6"
}
},
"node_modules/minimatch": {
"version": "3.1.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz",
"integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==",
"license": "ISC",
"dependencies": {
"brace-expansion": "^1.1.7"
},
"engines": {
"node": "*"
}
},
"node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
@@ -1519,15 +1403,6 @@
"integrity": "sha512-gQPNF78qebCQ6tvVFBYrvJdBNOrYZm90ZlXgpIFm06p6qHDHq/XC4TnJftN6OMbxVE0UTBAoRgcsDeJBBooITw==",
"license": "MIT"
},
"node_modules/once": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
"license": "ISC",
"dependencies": {
"wrappy": "1"
}
},
"node_modules/onetime": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/onetime/-/onetime-5.1.2.tgz",
@@ -1624,21 +1499,6 @@
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/path-is-absolute": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
"integrity": "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/path-parse": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz",
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
"license": "MIT"
},
"node_modules/pend": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz",
@@ -1667,18 +1527,6 @@
"node": ">=10"
}
},
"node_modules/psl": {
"version": "1.15.0",
"resolved": "https://registry.npmjs.org/psl/-/psl-1.15.0.tgz",
"integrity": "sha512-JZd3gMVBAVQkSs6HdNZo9Sdo0LNcQeMNP3CozBJb3JYC/QUYZTnKxP+f8oWRX4rHP5EurWxqAHTSwUCjlNKa1w==",
"license": "MIT",
"dependencies": {
"punycode": "^2.3.1"
},
"funding": {
"url": "https://github.com/sponsors/lupomontero"
}
},
"node_modules/punycode": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz",
@@ -1688,12 +1536,6 @@
"node": ">=6"
}
},
"node_modules/querystringify": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/querystringify/-/querystringify-2.2.0.tgz",
"integrity": "sha512-FIqgj2EUvTa7R50u0rGsyTftzjYmv/a3hO345bZNrqabNqjtgiDMgmo4mkUjd+nzU5oF3dClKqFIPUKybUyqoQ==",
"license": "MIT"
},
"node_modules/readable-stream": {
"version": "3.6.2",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
@@ -1708,17 +1550,6 @@
"node": ">= 6"
}
},
"node_modules/rechoir": {
"version": "0.6.2",
"resolved": "https://registry.npmjs.org/rechoir/-/rechoir-0.6.2.tgz",
"integrity": "sha512-HFM8rkZ+i3zrV+4LQjwQ0W+ez98pApMGM3HUrN04j3CqzPOzl9nmP15Y8YXNm8QHGv/eacOVEjqhmWpkRV0NAw==",
"dependencies": {
"resolve": "^1.1.6"
},
"engines": {
"node": ">= 0.10"
}
},
"node_modules/require-directory": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
@@ -1728,33 +1559,6 @@
"node": ">=0.10.0"
}
},
"node_modules/requires-port": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
"integrity": "sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ==",
"license": "MIT"
},
"node_modules/resolve": {
"version": "1.22.12",
"resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.12.tgz",
"integrity": "sha512-TyeJ1zif53BPfHootBGwPRYT1RUt6oGWsaQr8UyZW/eAm9bKoijtvruSDEmZHm92CwS9nj7/fWttqPCgzep8CA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"is-core-module": "^2.16.1",
"path-parse": "^1.0.7",
"supports-preserve-symlinks-flag": "^1.0.0"
},
"bin": {
"resolve": "bin/resolve"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/restore-cursor": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/restore-cursor/-/restore-cursor-3.1.0.tgz",
@@ -1880,18 +1684,6 @@
"node": ">=8"
}
},
"node_modules/supports-preserve-symlinks-flag": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/supports-preserve-symlinks-flag/-/supports-preserve-symlinks-flag-1.0.0.tgz",
"integrity": "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/symbol-tree": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/symbol-tree/-/symbol-tree-3.2.4.tgz",
@@ -1917,27 +1709,15 @@
"license": "MIT"
},
"node_modules/tough-cookie": {
"version": "4.1.3",
"resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-4.1.3.tgz",
"integrity": "sha512-aX/y5pVRkfRnfmuX+OdbSdXvPe6ieKX/G2s7e98f4poJHnqH3281gDPm/metm6E/WRamfx7WC4HUqkWHfQHprw==",
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
"integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
"license": "BSD-3-Clause",
"dependencies": {
"psl": "^1.1.33",
"punycode": "^2.1.1",
"universalify": "^0.2.0",
"url-parse": "^1.5.3"
"tldts": "^6.1.32"
},
"engines": {
"node": ">=6"
}
},
"node_modules/tough-cookie/node_modules/universalify": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/universalify/-/universalify-0.2.0.tgz",
"integrity": "sha512-CJ1QgKmNg3CwvAv/kOFmtnEN05f0D/cn9QntgNOQlQF9dgvVTHj3t+8JPdjqawCHk7V/KA+fbUqzZ9XWhcqPUg==",
"license": "MIT",
"engines": {
"node": ">= 4.0.0"
"node": ">=16"
}
},
"node_modules/tr46": {
@@ -1952,18 +1732,18 @@
"node": ">=18"
}
},
"node_modules/traverse-chain": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/traverse-chain/-/traverse-chain-0.1.0.tgz",
"integrity": "sha512-up6Yvai4PYKhpNp5PkYtx50m3KbwQrqDwbuZP/ItyL64YEWHAvH6Md83LFLV/GRSk/BoUVwwgUzX6SOQSbsfAg==",
"license": "MIT"
},
"node_modules/trim-buffer": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/trim-buffer/-/trim-buffer-5.0.0.tgz",
"integrity": "sha512-PqdsRmhir3lUwraK/O6sdZ9QcmtUK0c6F1utX/WNquJsL4Jpdck+DSe9Mqy8Wk7uAQD/SlVn6cyGF5bj6VF33Q==",
"license": "MIT"
},
"node_modules/undici-types": {
"version": "8.9.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz",
"integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==",
"license": "MIT"
},
"node_modules/universalify": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.1.tgz",
@@ -1973,27 +1753,12 @@
"node": ">= 10.0.0"
}
},
"node_modules/url-parse": {
"version": "1.5.10",
"resolved": "https://registry.npmjs.org/url-parse/-/url-parse-1.5.10.tgz",
"integrity": "sha512-WypcfiRhfeUP9vvF0j6rw0J3hrWrw6iZv3+22h6iRMJ/8z1Tj6XfLP4DsUix5MhMPnXpiHDoKyoZ/bdCkwBCiQ==",
"license": "MIT",
"dependencies": {
"querystringify": "^2.1.1",
"requires-port": "^1.0.0"
}
},
"node_modules/util-deprecate": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
"license": "MIT"
},
"node_modules/valid-url": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/valid-url/-/valid-url-1.0.9.tgz",
"integrity": "sha512-QQDsV8OnSf5Uc30CKSwG9lnhMPe6exHtTXLRYX8uMwKENy640pU+2BgBL0LRbDh/eYRahNCS7aewCx0wf3NYVA=="
},
"node_modules/w3c-xmlserializer": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz",
@@ -2076,12 +1841,6 @@
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"license": "ISC"
},
"node_modules/ws": {
"version": "8.21.3",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
+7 -5
View File
@@ -26,15 +26,17 @@
"viyacbd": "sasjs folder delete /Public/app/viya && sasjs cbd -t viya && npm run viyadata",
"viyadata": "sasjs request services/admin/makedata -d deploy/makeDataViya.json -l sasjsresults/makedata.log -o sasjsresults/output.json -t viya",
"v4data": "sasjs request services/admin/makedata -d deploy/makeDataV4.json -l sasjsresults/makedatav4.log -o sasjsresults/outputv4.json -t v4",
"sasdocs": "sasjs doc && ./sasjs/utils/deploydocs.sh"
"sasdocs": "sasjs doc && ./sasjs/utils/deploydocs.sh",
"sas719setup": "sasjs cbd -t sas719 && npm run sas719data",
"sas719data": "sasjs request services/admin/makedata -d deploy/makeDataSas719.json -l sasjsresults/makedata_sas719.log -o sasjsresults/makedata_sas719.json -t sas719",
"sas719test": "sasjs test -t sas719"
},
"private": true,
"dependencies": {
"@sasjs/cli": "4.20.5",
"@sasjs/core": "5.2.8"
"@sasjs/cli": "6.2.0",
"@sasjs/core": "7.0.1"
},
"overrides": {
"nanoid": "3.3.18",
"shelljs": "0.8.5"
"nanoid": "3.3.18"
}
}
+32
View File
@@ -0,0 +1,32 @@
/**
@file mpe_derived.ddl
@brief ddl file
@details Defines the derived (computed) variable configuration table. This
drives MPE_DERIVED rows in MPE_TABLES, so that the table itself can be
maintained through the EDIT screen with the usual approve workflow.
Derived values are not persisted - the logic is applied at read time, before
the filter, so that the variables can be filtered on.
@version 9.2
@author 4GL Apps Ltd
@copyright 4GL Apps Ltd
**/
create table &curlib..MPE_DERIVED
(
TX_FROM num not null format=datetime19.3,
LIBREF varchar(8) not null,
DSN varchar(32) not null,
VAR_NM varchar(32) not null,
VAR_DEPS varchar(4000),
VAR_TYPE varchar(1) not null,
VAR_LEN int not null,
VAR_FMT varchar(32),
VAR_LABEL varchar(256),
VAR_LOGIC varchar(4000) not null,
ACTIVE int not null,
TX_TO num not null format=datetime19.3,
constraint pk_mpe_derived
primary key(tx_from, libref, dsn, var_nm));
+13
View File
@@ -118,3 +118,16 @@ insert into &mpelib..mpe_tables
,var_txfrom='TX_FROM'
,var_txto='TX_TO'
;
insert into &mpelib..mpe_tables
set tx_from=0
,tx_to='31DEC5999:23:59:59'dt
,libref="&mpelib"
,dsn='MPE_DERIVED'
,num_of_approvals_required=1
,loadtype='TXTEMPORAL'
,buskey='LIBREF DSN VAR_NM'
,post_edit_hook='services/hooks/mpe_derived_postedit'
,notes='Derived (computed) variables for the EDIT screen'
,var_txfrom='TX_FROM'
,var_txto='TX_TO'
;
@@ -97,6 +97,10 @@ quit;
set tx_to=&now
where var_name='DC_MAXCELLS_WEBEDIT' and var_active=1 and &now lt tx_to;
/* Why the cell limit exists, kept out of var_desc - MPE_CONFIG.VAR_DESC
is varchar(300), and a truncation WARNING aborts the build. A wide
table is expensive long before it is tall - 250 rows of a 1000 column
table is a quarter of a million cells. */
insert into &dclib..mpe_config
set tx_from=&now
,tx_to='31DEC9999:23:59:59'dt
@@ -106,10 +110,8 @@ quit;
,var_active=1
,var_desc='This sets the maximum number of CELLS (rows multiplied by'
!!' columns) that can be loaded into the browser for editing in the'
!!' EDIT screen. It exists because a wide table is expensive long'
!!' before it is tall - 250 rows of a 1000 column table is a quarter'
!!' of a million cells. Whichever of this and DC_MAXOBS_WEBEDIT is'
!!' reached first applies.';
!!' EDIT screen. Whichever of this and DC_MAXOBS_WEBEDIT is reached'
!!' first applies.';
quit;
%end;
%else %do;
@@ -0,0 +1,192 @@
/**
@file
@brief migration script to move to v7.19 of Data Controller
REQUIRED CHANGE - adds the MPE_DERIVED table, which holds derived (computed)
variable definitions for the EDIT screen, and registers it in MPE_TABLES so
that it can be maintained through the app with the usual approve workflow.
Replace YOURDCLIB and the library path below, then run the script. Any
MPE_DERIVED rows added before this script runs will be lost when the table
is created, so run it before configuring any derived variables.
**/
%let dclib=YOURDCLIB;
libname &dclib "/YOUR/DATACONTROLLER/LIBRARY/PATH";
proc sql;
create table &dclib..mpe_derived
(
tx_from num not null format=datetime19.3,
libref varchar(8) not null,
dsn varchar(32) not null,
var_nm varchar(32) not null,
var_deps varchar(4000),
var_type varchar(1) not null,
var_len num not null,
var_fmt varchar(32),
var_label varchar(256),
var_logic varchar(4000) not null,
active num not null,
tx_to num not null format=datetime19.3
);
quit;
proc datasets lib=&dclib noprint;
modify mpe_derived;
index create
pk_mpe_derived=(tx_from libref dsn var_nm)
/nomiss unique;
quit;
proc sql;
insert into &dclib..mpe_tables
set tx_from=0
,tx_to='31DEC5999:23:59:59'dt
,libref="&dclib"
,dsn='MPE_DERIVED'
,num_of_approvals_required=1
,loadtype='TXTEMPORAL'
,buskey='LIBREF DSN VAR_NM'
,post_edit_hook='services/hooks/mpe_derived_postedit'
,notes='Derived (computed) variables for the EDIT screen'
,var_txfrom='TX_FROM'
,var_txto='TX_TO'
;
quit;
/* Dropdown values for VAR_TYPE and ACTIVE. These rows, and the field rules
* further down, mirror the MPE_DERIVED rows in mpe_makedata.sas - keep the two
* in step, or an upgraded install and a freshly seeded one behave differently. */
proc sql noprint;
/* coalesce: an install with no selectbox rows yet returns a missing value,
and %eval(&maxrk+1) below would then fail */
select coalesce(max(selectbox_rk),0) into: maxrk from &dclib..mpe_selectbox;
quit;
%let rk1=%eval(&maxrk+1);
%let rk2=%eval(&maxrk+2);
%let rk3=%eval(&maxrk+3);
%let rk4=%eval(&maxrk+4);
proc sql;
insert into &dclib..mpe_selectbox
set selectbox_rk=&rk1
,ver_from_dttm=0
,select_lib="&dclib"
,select_ds="MPE_DERIVED"
,base_column="VAR_TYPE"
,selectbox_value='C'
,selectbox_order=1
,ver_to_dttm='31DEC5999:23:59:59'dt;
insert into &dclib..mpe_selectbox
set selectbox_rk=&rk2
,ver_from_dttm=0
,select_lib="&dclib"
,select_ds="MPE_DERIVED"
,base_column="VAR_TYPE"
,selectbox_value='N'
,selectbox_order=2
,ver_to_dttm='31DEC5999:23:59:59'dt;
/* ACTIVE - a fixed pair, not a lookup against another table */
insert into &dclib..mpe_selectbox
set selectbox_rk=&rk3
,ver_from_dttm=0
,select_lib="&dclib"
,select_ds="MPE_DERIVED"
,base_column="ACTIVE"
,selectbox_value='1'
,selectbox_order=1
,ver_to_dttm='31DEC5999:23:59:59'dt;
insert into &dclib..mpe_selectbox
set selectbox_rk=&rk4
,ver_from_dttm=0
,select_lib="&dclib"
,select_ds="MPE_DERIVED"
,base_column="ACTIVE"
,selectbox_value='0'
,selectbox_order=2
,ver_to_dttm='31DEC5999:23:59:59'dt;
quit;
/* Field rules for the EDIT screen. These are client side - the server side
* enforcement is the post edit hook registered above. */
proc sql;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="LIBREF"
,rule_type='CASE',rule_value='UPCASE',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
/* the libraries the user can edit, as MPE_VALIDATIONS.BASE_LIB does */
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="LIBREF"
,rule_type='SOFTSELECT_HOOK'
,rule_value="services/validations/libraries_editable"
,rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="DSN"
,rule_type='CASE',rule_value='UPCASE',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
/* tables in the LIBREF on the same row - the hook reads it from source_row */
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="DSN"
,rule_type='SOFTSELECT_HOOK'
,rule_value="services/validations/tables_editable"
,rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_NM"
,rule_type='CASE',rule_value='UPCASE',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_NM"
,rule_type='HARDREGEX',rule_value='^[A-Za-z_][A-Za-z0-9_]*$',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_DEPS"
,rule_type='CASE',rule_value='UPCASE',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_DEPS"
,rule_type='HARDREGEX',rule_value='^[A-Za-z0-9_ ]*$',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
,rule_type='CASE',rule_value='UPCASE',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_TYPE"
,rule_type='HARDSELECT',rule_value=' ',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
/* ACTIVE has no NOTNULL rule: the column is NOT NULL in the model, and the
* EDIT screen offers the MPE_SELECTBOX values for it. */
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
,rule_type='MINVAL',rule_value='1',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LEN"
,rule_type='MAXVAL',rule_value='32767',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_FMT"
,rule_type='CASE',rule_value='UPCASE',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_FMT"
,rule_type='HARDREGEX'
,rule_value='^(\$?[A-Za-z][A-Za-z0-9_]*(\.[0-9]+)?\.?|\$?[0-9]+(\.[0-9]+)?\.?)?$',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LABEL"
,rule_type='HARDREGEX'
,rule_value='^[^\x00-\x1F\x7F]*$',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
insert into &dclib..MPE_VALIDATIONS set
tx_from=0,base_lib="&dclib",base_ds="MPE_DERIVED",base_col="VAR_LOGIC"
,rule_type='HARDREGEX'
,rule_value='^(?!.*(/\*|\*/))[^;&%]*$',rule_active=1
,tx_to='31DEC5999:23:59:59'dt;
quit;
Loaded 100 of 145 files, more files were not shown because too many files have changed in this diff. Show more