Commit Graph
1546 Commits
Author SHA1 Message Date
dc 541a61ba08 fix(embed): make the va_parmap diagnostics say what they mean
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s
Build / Build-and-test-development (pull_request) Successful in 33m26s
Review findings, all about a warning that misleads:

- The banner heading said the mapping was ignored, but two of the three
  lists it renders are mappings that are applied (the AND-ed pair) or that
  were never applicable (a parameter VA does not send). Neutral heading.
- unmatchedMappings reported a key as unsent as soon as any parameter had
  been seen. VA sends the parameters a report object's query uses, so a
  name that has not arrived yet looked like a typo. A mapping is now only
  judged once a message introduces no new parameter name, i.e. once the
  object's parameter set has settled.
- sameColumnMappings counted a column the table does not have, so one
  mistake produced the unresolved-mapping line and the AND line. A column
  that is not in the table is now left to unresolvedMappings.
- The Cypress cases asserted exact row counts, which encode the demo
  table's contents. They now read the unfiltered count from the app and
  assert the filtered grid is smaller (or empty), so a fixture change
  cannot fail a behaviour test.
2026-10-28 12:20:37 +00:00
allan ad50acf647 Merge pull request 'fix(ci): publish the typedoc to the webdoc site, not the app default' (#385) from fix/webdoc-named-site into release/7.20
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #385
Reviewed-by: Allan <allan@4gl.io>
2026-10-28 12:10:16 +00:00
dc cd1ce4a7e2 test(embed): format the transform spec to satisfy prettier --check
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
2026-10-28 12:01:11 +00:00
dc 7e3045a21b fix(embed): build quarter ends from the calendar, not minus 24 hours
Build / Build-and-ng-test (pull_request) Failing after 2m2s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Canceled after 6m38s
Subtracting a fixed day from the next quarter's start moves by an hour
across a daylight-saving transition, which would place a DATETIME bound
off the intended midnight. Day 0 of the following month is the last day
of the month by construction.
2026-10-28 11:57:23 +00:00
dc 88010c3980 feat(embed): resolve va_parmap values through named date anchors
Build / Build-and-ng-test (pull_request) Failing after 1m55s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m22s
A VA parameter can now carry a transform on the column side of a mapping:

  va_parmap=SELECTED_DATE-D_DATE:ADD(QUARTER_START,-1)

which filters D_DATE on the report value AND the same point one quarter
earlier, as a single IN clause - so the AND rule, the clause signature,
saveQuery and the reload loop are untouched.

Transforms resolve to one clause with these operations:

- ADD(<ANCHOR>,<offset>)   the value plus one anchor beside it -> IN
- RANGE(<PERIOD>,<offset>) a closed period -> BETWEEN

Anchors: QUARTER_START/END, MONTH_START/END, YEAR_START/END.
Periods: QUARTER, MONTH, YEAR. Offset is a whole number of periods,
defaulting to 0.

The names describe the result rather than mirroring SAS intnx: the
arithmetic runs in the browser, so mirroring intnx would mean
reimplementing its alignment letters and their corners (week alignment,
's' and 'm') with no guarantee of matching SAS. A report author can read
ADD(QUARTER_START,-1) without knowing intnx.

Resolution is anchored on the parameter value, never on today, so a saved
report reproduces the same rows. A transform that cannot run where it is
used - a non-temporal column, a value that is not a date - is skipped and
reported in the existing warning banner rather than silently filtering on
the raw value.
2026-10-28 11:52:53 +00:00
dc c5808a16eb fix(ci): publish the typedoc to the webdoc site, not the app default
Build / Build-and-ng-test (pull_request) Successful in 5m42s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m39s
Build / Build-and-test-development (pull_request) Canceled after 19m50s
`surfer deploy` chooses its target only from `--site-name`; without one it
replaces the app's DEFAULT site. The typedoc job passed `--server
webdoc.datacontroller.io` and no site name, so the 7.19.0 docs were published
over the default site of that app - the marketing site served at
datacontroller.io - instead of to a webdoc site.

Name the site, and give it its alias domain so the first run creates it.
The SAS docs job is left unnamed on purpose: code.datacontroller.io is its own
app whose default site is the docs, and a comment now says so.
2026-10-28 11:49:25 +00:00
dc 0011f7e933 fix(embed): log a parse-time mapping problem once
Build / Build-and-ng-test (pull_request) Successful in 5m0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m41s
Build / Build-and-test-development (pull_request) Successful in 31m54s
The parse-time problems were logged at app init and then again by the
editor's warning loop, which spreads them into the banner's text. The loop
now logs only what the instance discovers for itself (unresolved column,
unmatched key, AND-ed pair); the parse problems stay in the banner and keep
their single init-time console line.

E2e case 11 pins it: one unparseable entry plus one valid-but-missing
mapping, so the loop certainly runs and logs its own line, and exactly one
line mentions the parse problem.
2026-10-28 11:13:10 +00:00
dc 4ed1e0684d test(embed): prove va_parmap case-insensitivity end to end
Build / Build-and-ng-test (pull_request) Successful in 5m31s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m42s
Build / Build-and-test-development (pull_request) Successful in 33m51s
2026-10-28 09:57:58 +00:00
dc 2623a3fb4b fix(embed): defer the column check until metadata loads, report AND-ed pairs
Build / Build-and-ng-test (pull_request) Successful in 5m50s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m50s
Build / Build-and-test-development (pull_request) Successful in 33m51s
Two review findings, both paths where a filter could look applied while
producing nothing.

- `unresolvedMappings` judged a mapping against an empty column list while the
  grid was still loading, so every entry was briefly reported as naming a
  column the table does not have. An empty or unset `cols` now means "not known
  yet" and returns nothing, so the banner appears only once metadata exists.
- Two parameters mapped to one column are AND-ed, so different values yield no
  rows at all, and nothing said so. `sameColumnMappings()` now reports the
  contended column and both parameters. The clauses are deliberately left as an
  intersection: turning that AND into an OR is a filter-semantics decision, not
  a diagnostic, so it is tracked separately.

Covered by unit tests for both (unloaded column list, one line per contended
column) and e2e case 10, which drives two parameters onto one column and
asserts the banner plus the empty grid.
2026-10-28 09:54:48 +00:00
dc 388f418bf4 test(embed): pin the semantics of two params on one column
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m39s
Build / Build-and-test-development (pull_request) Failing after 35m57s
Records that two parameters mapped to the same DC column are AND-ed, so the
result is the intersection: two different dates match no rows rather than
both dates' rows. Answers the 'filter one column on two dates' case from the
code rather than by assumption, and marks the single-parameter IN (...) shape
as the one that unions.
2026-10-28 09:35:36 +00:00
dc f83dc359e2 fix(embed): one parameter maps one column, and report typos on both sides
Build / Build-and-ng-test (pull_request) Successful in 5m42s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m20s
Build / Build-and-test-development (pull_request) Successful in 33m57s
Three follow-ups from review, all the same class of defect: an author error
that leaves an unfiltered grid looking like a working filter.

- A duplicate key in `va_parmap` silently kept whichever entry came last, so
  `va_parmap=DEPT-REGION&va_parmap=DEPT-OTHER` filtered OTHER with nothing to
  show for it. One VA parameter now maps one column: the first entry wins and
  a conflicting duplicate is reported. The same mapping written twice is not a
  conflict, and two different parameters mapping to one column stay valid -
  that is how two report objects filter the same column from their own
  parameters.
- A parameter-side typo (mapping `DROPDOWN_TAB1` when the report parameter is
  `DROPDOWN_TAB_1`) produced no clause, no banner and no console line.
  VaMessagingService now records every parameter key it has seen, across
  editor reloads, and the editor reports a mapping key that VA has never sent.
  Judged only once parameters are actually arriving, because VA sends just the
  parameters its query uses - an absent key is not on its own an error, and a
  key that turns up later stops being reported.
- The deferred first load cannot check a mapped column against metadata. On
  SAS `mp_filtercheck` aborts the save, so the load falls back to unfiltered;
  the banner now covers that path too, and an e2e case seeds the pre-bootstrap
  message buffer to prove it (and to prove a valid mapping filters the very
  first getdata).
2026-10-28 09:34:02 +00:00
dc 411f6c3354 fix(embed): warn when a va_parmap entry is ignored
Build / Build-and-ng-test (pull_request) Successful in 5m48s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m9s
Build / Build-and-test-development (pull_request) Successful in 33m7s
A `va_parmap` entry that cannot be used was dropped in silence, in both
places it can fail: at parse time (no `-` separator, a missing side, or a
column that is not a plain SAS name) and at apply time (the mapped column is
not in the table being opened). Either way the parameter is skipped, which
degrades to "no filter" - a typo in the report's URL then looks exactly like
a working filter over the whole table.

Both are now reported in terms the report author can act on:

- a warning banner in the editor (VA mode) naming the parameter, the column
  and the table, and pointing at the `va_parmap` entries in the URL
- console.warn for each distinct problem, so it is findable from the report's
  console too

The parser returns `{ map, problems }` rather than a bare Map, VaFilterService
gains `unresolvedMappings()` for the entries the loaded table cannot satisfy,
and the editor merges both sets for the banner (logging once per distinct set
rather than once per VA message).

The banner is covered by two e2e cases, one asserting its painted box inside
the viewport (VA hides the app header, so it has to stand on its own) and one
covering a parse-time failure before any VA message arrives. The alert markup
keeps the message inside the alert's text flow so it wraps, instead of being
laid out beside a title and pushed off the right edge.
2026-10-28 09:08:30 +00:00
allan f042ca82ea Merge pull request 'fix(ci): publish the docs with surfer 7, and rebuild release assets on demand' (#382) from fix/ci-surfer-app-password into release/7.20
Sites / deploy (push) Successful in 3m49s
Reviewed-on: #382
2026-10-28 08:49:33 +00:00
dc 15a1f34d52 feat(embed): map VA parameters to DC columns from the URL
Build / Build-and-ng-test (pull_request) Successful in 5m33s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m20s
Build / Build-and-test-development (pull_request) Successful in 32m46s
Two DC instances over the same table in one VA report could not filter the
same column from different parameters: parameter -> column matching is by
name, and two VA parameters cannot share a label.

The `va_parmap` URL parameter overrides that match per instance, one entry
per mapping (`va_parmap=<VA parameter>-<DC column>`, repeated for more):

  ...?embed=va&va_parmap=REGION_TAB1-REGION

The mapping is additive (unmapped parameters keep the name match), the
column is validated as a plain SAS name at parse time, and an entry only
applies where the named column exists in the table being opened - a miss
skips that parameter rather than falling back to the same-name match the
mapping exists to override. Entries split at the last hyphen, so a label
may itself contain hyphens, and the key matches the parameter label first,
then its internal name (pr###).

Parsed once at app init onto globals (like embed / vaApplyMode) so the
editor reloads a filter apply performs keep the mapping, and both filter
paths (the metadata build and the deferred initial load) honour it.

Also repairs embed-va.cy.ts: a cy.visit differing from the current URL
only in the hash is a same-document navigation, so ?embed=va never took
effect and both existing cases failed; their layout assertions also
described an earlier design (VA now opens read-only with an Edit button
and hides Filter). Cases 3 and 4 drive two mappings of the same column
through a real postMessage and assert the applied filter and row count.
2026-10-28 08:41:15 +00:00
dc 6486c80ed4 fix(ci): rebuild and upload release assets on demand
Build / Build-and-ng-test (pull_request) Successful in 5m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m33s
Build / Build-and-test-development (pull_request) Successful in 31m45s
2026-10-28 02:07:31 +00:00
dc dd482c4069 fix(ci): publish docs after the assets, with the webdoc app password 2026-10-28 02:05:10 +00:00
dc f58925b299 fix(ci): install doxygen for the SAS docs job 2026-10-28 02:02:57 +00:00
dc 25ffa71add fix(ci): publish the code documentation with surfer 7 and app passwords 2026-10-28 01:57:23 +00:00
semantic-release-bot 424b111737 chore(release): 7.19.0 [skip ci]
# [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)

### Bug Fixes

* address the review findings on the harness and refresh changes ([36e4e88](36e4e88eb1))
* **ci:** dispatch the demo deploy with a tag ref and the actions scope ([698dea6](698dea6d6b))
* **ci:** restore the gitleaks ignore entries for the mock licence pair ([0d58be9](0d58be975d))
* **client:** open the SAS log page with noopener ([072a133](072a133702))
* **client:** refuse a truncated table in the multi-table upload ([1fb4818](1fb4818d21))
* **cypress:** apply the licence key when the estate has none ([be8f445](be8f445ae1)), closes [input#file-upload](https://git.datacontroller.io/input/issues/file-upload)
* **dc_execute:** use = comparisons, not IN, in the fail-fast guard ([fed43cb](fed43cb07e))
* **dc_getlibs:** exclude the DC libref from the dictionary scan ([f5aade5](f5aade5976))
* **dc_refreshcatalog:** terminate the generated statements ([1e22476](1e224763ba))
* **derived:** a fileref with nothing written is a FAIL, not an error ([9b3cf50](9b3cf50b9c))
* **derived:** address five review findings ([ccd468b](ccd468b335))
* **derived:** carry the placement sequence in mpe_derived ([be50cb5](be50cb5d08))
* **derived:** close four gaps in the MPE_DERIVED hook ([9914790](9914790777))
* **derived:** close the RLS and search gaps on derived columns ([9ef813d](9ef813d439))
* **derived:** close the two gaps in the derived/RLS invariant ([422f766](422f766cd4))
* **derived:** compare the outref as text, prove the file is empty ([7830b7b](7830b7bcfa))
* **derived:** do not drop a shadowed column from staging or downloads ([2d5de9a](2d5de9a324))
* **derived:** do not duplicate a shadowed column in the grid payload ([6a1128a](6a1128a463))
* **derived:** eight review findings ([561d7fc](561d7fc3ed))
* **derived:** enforce the ACTIVE picker pair in the extracted check ([1863076](1863076bc9)), closes [#366](#366)
* **derived:** enforce the documented limits, and handle derived filters in VIEW ([d3dec68](d3dec682aa))
* **derived:** filtering a format catalog no longer aborts the service ([469229a](469229aecb))
* **derived:** guard the bitemporal cut-off, coalesce the selectbox seed ([7fb981d](7fb981daa9))
* **derived:** guard the empty physical column list in the dep check ([5f5d868](5f5d8681b9))
* **derived:** hold a derived column read-only whatever the table's security ([5f443b0](5f443b0b16))
* **derived:** ignore a re-added name when collecting inactive rows ([6b5bf3e](6b5bf3e6be))
* **derived:** ignore inactive rows and VIEW rules in the submit guards ([85c89ca](85c89ca2a4))
* **derived:** keep targetds for plain datasets, test the empty file ([f8f6568](f8f65681e1))
* **derived:** make the model enforce the required columns, and leave VAR_FMT optional ([7efbf31](7efbf31ea5))
* **derived:** only the delete marker is a reserved name ([af31c36](af31c36b67))
* **derived:** open the statements file with a long record length ([ee31782](ee31782fbf))
* **derived:** pair the filterds %else with %do/%end ([88a38db](88a38db877))
* **derived:** raise the VAR_ORDER ceiling to 10000 ([27ddcfb](27ddcfb575))
* **derived:** refuse dependencies on inactive rows, and keep the RLS scope guard coherent ([7799a30](7799a300df))
* **derived:** reject a dependency on a row this submission deletes ([8806504](8806504967))
* **derived:** repair the validation rows for MPE_DERIVED ([9a047ad](9a047ad42e))
* **derived:** review feedback - cover the empty response, assert the last row ([e0da9f1](e0da9f1098))
* **derived:** stop the hooks aborting on their own macro variables ([688550a](688550a28d))
* **derived:** truncate the derived label on an even quote boundary ([1b5c4b3](1b5c4b3ecd))
* **derived:** verify on stripped values, and label with the variable name ([4d9ad38](4d9ad38df1))
* **editor:** a blank truncation note is not a warning ([f182179](f182179403))
* **editor:** keep decimal values when an uploaded file is edited manually ([04ce186](04ce1861e7))
* **getcolvals:** report an invalid value, not missing inputs ([4bcee70](4bcee706ff))
* **getdata.test:** pin the derived filter to a libref, and spell the table as it is ([45dce7b](45dce7b11b))
* **getdata:** clear the globals it declares before use ([359b570](359b570b85))
* **getddl.test:** ask for a table, not a whole library ([059cdf1](059cdf176a))
* **getrawdata:** declare mp_ds2md so the Markdown download can run ([661ee19](661ee19cfa))
* **lint:** clear the sasjs lint warnings on the files 7.19 touched ([355c46a](355c46a7b6))
* **makedata:** read KEEPME from every input row, and keep it out of the Viya seed ([d626cb8](d626cb8f2d))
* **mocks:** address the audit-pack review findings ([489819b](489819b051))
* **mocks:** carry the zip flags into the central directory ([c4021a4](c4021a42ff))
* **mocks:** clear the licence for published sites, not for the estate ([62f55db](62f55db170))
* **mocks:** copy the physical column list before deriving ([f411b7d](f411b7d269))
* **mocks:** derive before filtering in the viewer, and drop the columns ([60ec470](60ec470404))
* **mocks:** derive the retained key wherever rows are staged ([74060c0](74060c0585))
* **mocks:** drop derived variables before staging ([73d25b1](73d25b1220))
* **mocks:** key rk_underlying tables on their business key ([347dbb5](347dbb53b0))
* **mocks:** register MPE_DERIVED and seed its field rules ([ee3c878](ee3c8787ef))
* **mocks:** scope the derived hook's graph to the submission ([6cad23f](6cad23fce9))
* **mocks:** search the physical columns only in the viewer ([293022d](293022d46b))
* **mocks:** stop the mock advertising invalid licence keys ([fb36379](fb36379520))
* **mpe_assert_abort:** drop the unused line variable ([9c294c0](9c294c0ab5))
* **mpe_assert_abort:** take the abort's own MSG, tolerate a missing response ([0a52d20](0a52d20c5f))
* **mpe_execute:** fail fast when the child left no status ([a036386](a03638667f))
* **mpe_execute:** no apostrophe in the SAS 9 status note ([2204085](220408538b))
* **mpe_execute:** read an unknown state as unmeasured, not failed ([f2b1428](f2b1428ec3))
* name the startupservice launcher button "Show log" ([3445107](3445107d9a))
* offer the launcher on the globvars failure modal too ([ad2da45](ad2da45389))
* **refresh:** name each catalogue load uniquely in the audit trail ([46bacdd](46bacdd6d9))
* **refresh:** one audit folder per refresh call ([56b6620](56b6620ef4))
* **sites:** round-trip the committed licence before publishing ([4d462d3](4d462d310f))
* **test:** getdata test ([da30266](da30266278))
* **tests:** enable closeouts logging ([fdcd8fa](fdcd8fa4b3))
* **tests:** read the work location from _SASJS_WORK ([033ffd3](033ffd3025))
* **ui:** one error modal at a time ([81ccf75](81ccf758e9))
* **validatefilter:** build targetds without macro triggers in comments ([42355f1](42355f17bc))
* **validatefilter:** drop the dead nobs assignment, keep targetds precedence ([337ea8e](337ea8e5e0))

### Features

* **dc_getlibs:** dump the library list when debug is on ([582523d](582523dc87))
* **derived:** allow a variable to shadow a physical column ([12beeff](12beeff695))
* **derived:** declare VAR_DEPENDENCIES instead of VAR_ORDER ([813bce6](813bce6856))
* **derived:** optional VAR_LABEL on MPE_DERIVED ([7c9ce9a](7c9ce9a712))
* **derived:** pickers, and an ACTIVE default, for MPE_DERIVED ([87e794f](87e794f769))
* **derived:** report the dependency cycle path ([cee1a95](cee1a954ee))
* **derived:** take any text in VAR_LABEL, but not a line break ([78e88a4](78e88a4112))
* **derived:** validate MPE_DERIVED at submit time ([0b010f2](0b010f2c65))
* **derived:** validation rules and a VAR_TYPE dropdown for MPE_DERIVED ([aa6d50e](aa6d50e1ea))
* **derived:** VARIABLE_ORDER, so a derived variable can build on another ([33d043e](33d043ea92))
* **editor:** derived variables in the EDIT screen ([b0fd354](b0fd354ecd))
* **editor:** mark derived variables as computed in the EDIT grid ([6513d09](6513d09f05))
* **editor:** serve a truncated selection with a warning instead of aborting ([34691dc](34691dceec))
* **editor:** show the derived column label in the header tooltip ([cfe6647](cfe6647ca8))
* **mocks:** add getauditfile and the audit-folder artefacts ([752d944](752d944c20))
* **mocks:** implement derived variables in the JS services ([b4fae88](b4fae88ae1))
* **mocks:** validate a submitted MPE_DERIVED row like the hook does ([5d183d1](5d183d1350))
* run startupservice in a new window from its error modal ([40f6133](40f61331aa))
* **sites:** mint a licence for the published sites ([7d9df71](7d9df71fdb))

### Reverts

* **derived:** no shadowing, and no self reference ([5af8e6f](5af8e6fc35))
* **stagedata.test.4:** back to the MPE_LOADS evidence for now ([31aa859](31aa859153))
v7.19.0
2026-10-28 01:42:51 +00:00
allan 849153d4a2 Merge pull request 'ci(release): build the frontend before deploying mocked services' (#381) from fix/ci-release-build-frontend into main
Release / Build-production-and-ng-test (push) Successful in 4m42s
Release / Build-and-test-development (push) Successful in 25m42s
Release / release (push) Failing after 6m56s
Reviewed-on: #381
2026-10-28 01:08:49 +00:00
dc bc9e0996ec ci(release): build the frontend before deploying mocked services
Build / Build-and-ng-test (pull_request) Canceled after 1m32s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Successful in 19m52s
2026-10-28 01:07:00 +00:00
allan 66039c91d7 Merge pull request 'feat: release 7.19 - derived variables, startupservice launcher, core v6' (#348) from release/7.19 into main
Release / Build-production-and-ng-test (push) Successful in 4m13s
Release / Build-and-test-development (push) Failing after 1m52s
Release / release (push) Skipped
Reviewed-on: #348
2026-10-28 00:52:14 +00:00
allan abb1ecbcab Merge pull request 'fix(validatefilter): drop the dead nobs assignment, keep targetds branches exclusive' (#380) from fix/validatefilter-branches into release/7.19
Sites / deploy (push) Successful in 4m2s
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m9s
Build / Build-and-test-development (pull_request) Successful in 33m26s
Reviewed-on: #380
2026-10-28 00:07:31 +00:00
dc 337ea8e5e0 fix(validatefilter): drop the dead nobs assignment, keep targetds precedence
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Review feedback on the merged fix:

- derived_nobs was assigned twice, the first one dead - the coalesced form below
  is the live value.  Removed, so there is one assignment to read.
- making the catalog test a plain %if after the derived-variables test let the
  catalog branch override it; the derived view won before and still should.  The
  catalog branch now requires derived_nobs=0, so the view keeps precedence and
  only a catalog with no derived configuration is blanked.

Two guarded if blocks, not %else %if: that construct is what the parser rejected
in this file before ('Expected DO not found'), and the first attempt at this
review point reproduced it - validatefilter and viewdata both failed to complete
again until it was replaced.

Verified: sasjs test -t new -> 64/64 (100%), TEST=0.
2026-10-28 00:05:19 +00:00
allan 709acfad73 Merge pull request 'fix(validatefilter): build targetds without macro triggers in comments' (#379) from fix/validatefilter-targetds into release/7.19
Sites / deploy (push) Successful in 3m50s
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 14m43s
Reviewed-on: #379
2026-10-27 23:46:46 +00:00
dc 42355f17bc fix(validatefilter): build targetds without macro triggers in comments
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m37s
Build / Build-and-test-development (pull_request) Successful in 33m19s
The suite failed to complete on the estate with

  ERROR: Expected %DO not found.
  ERROR: Skipping to next %END statement.

and, once the parser had lost sync, 'Apparent symbolic reference IS_FMTCAT not
resolved' - because the %let that defines it never ran.

Two causes, both in the code that decides which dataset the filter is validated
against:

- the comment explaining the catalog case contained the text '%if ...' and the
  macro processor resolves triggers even inside a comment, so the prose was
  executed as code and tripped the parser from there.  The comment now avoids
  macro triggers entirely.
- an empty value compared in a condition is a character operand, and
  %mf_nobs returns nothing when the dataset is missing.  It is wrapped in
  coalesce now, and the catalog flag is computed with %eval instead of a nested
  %if.

targetds is still blanked for a LIBREF.CATALOGNAME-FC name - which is what makes
mp_filterstore rewrite it to a cntlout dataset - and stays the plain libds
otherwise, as an empty targetds falls back to libds in the engine anyway.

The test now passes outref= and mdebug=&sasjs_mdebug rather than a hard-coded
1: the suite already owns that switch, and testinit turns it on by itself when
the run asks for _debug=2477 or 131, which is when the child's response and the
PROC HTTP trace are worth having.

Verified: sasjs test validatefilter -t new -> 3/3 (100%), six assertions, all
PASS, including the catalog and derived-variable cases.
2026-10-27 23:46:19 +00:00
dc 6ef2eec011 ci(sites): start the SASjs server with HTTP_PORT=5000
Sites / deploy (push) Successful in 3m45s
Build / Build-and-ng-test (pull_request) Successful in 5m5s
Build / Build-and-test-development (pull_request) Canceled after 5m11s
Lighthouse Checks / lighthouse (pull_request) Canceled after 7m59s
The Sites workflow was the one place that still started the server without a
listener, so it died with

  [error] Invalid environment variable(s) provided:
   - no listener is enabled
   - set HTTP_PORT and/or HTTPS_PORT (either may be used alone)

and the following step's 120s poll timed out. build.yaml, release.yaml and
lighthouse.yaml already carry the same line.
2026-10-27 23:26:26 +00:00
allan 1ef3e0a97b Merge pull request 'fix(refresh): one audit folder per refresh call' (#378) from fix/refresh-etlsource into release/7.19
Sites / deploy (push) Canceled after 2m14s
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 13m28s
Reviewed-on: #378
2026-10-27 23:12:33 +00:00
dc 56b6620ef4 fix(refresh): one audit folder per refresh call
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 18m57s
refreshcatalog.test and viewtables.test failed to complete with

  ERROR: could not create /tmp/DataController/<libref>/dc_staging/refreshcatalog:mpe_refreshcatalogs: + <stamp>

The staging folder a closeout writes its statements to is
&dc_staging_area/&etlsource, and etlsource defaulted to
'refreshcatalog:<step>:%mf_uid()'. A default is resolved when the macro is
compiled, not when it is called, so every refresh in a session reused the same
folder - the first step created it, the next one could not, and the abort took
the whole suite down. Whichever suite reached the staging area first decided
which others failed, which is why the count moved between runs.

etlsource now defaults to 0 and is set per call from
%mf_getuniquename(prefix=<step>): unique, a plain name rather than a path, and
resolved at the point of use.

Verified on the estate: refreshcatalog 4/4 (100%), viewtables 2/2 (100%).
2026-10-27 23:08:47 +00:00
allan fd272eec89 Merge pull request 'test(editor): assert the derived block marker, not a column name' (#377) from fix/derived-block-spec into release/7.19
Sites / deploy (push) Failing after 5m38s
Build / Build-and-ng-test (pull_request) Successful in 5m18s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 1m32s
Reviewed-on: #377
2026-10-27 22:25:30 +00:00
dc c463a4e03c test(editor): assert the derived block marker, not a column name
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m58s
Build / Build-and-test-development (pull_request) Canceled after 17m25s
derived-variables.cy.ts:97 expected dc-derived-first on the REGION header,
because the comment assumed REGION opens the computed block. The grid follows
MPE_DERIVED order, and that order is LINE_VALUE, REGION, REGION_LABEL - so the
marker was on LINE_VALUE and correct. The spec asserted the wrong column, and
the development pipeline has been red on it since.

Dumping the header row on a local mock estate (sasjs/server desktop mode,
RUN_TIMES=js) shows:

  6 | LINE_VALUE | dc-derived-header dc-derived-first
  7 | REGION | dc-derived-header
  8 | REGION_LABEL | dc-derived-header beforeHiddenColumn htLastVisibleHeader

and the same dump against the unmodified release/7.19 client, so this was never
a client defect.

The assertion now states the invariant instead: at least one derived header, the
first of them opens the block, and no other one carries the marker. That holds
whatever order the control table returns the variables in.
2026-10-27 22:21:52 +00:00
allan 23015ac41f Merge pull request 'fix(lint): clear the sasjs lint warnings on the files 7.19 touched' (#374) from fix/lint-719-touched-files into release/7.19
Sites / deploy (push) Failing after 5m39s
Build / Build-and-ng-test (pull_request) Successful in 5m26s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Build / Build-and-test-development (pull_request) Canceled after 20m56s
Reviewed-on: #374
2026-10-27 22:01:09 +00:00
dc a8f62bdfe2 docs(agents): say why webout is in ignoredMacros, and where the macros come from
Build / Build-and-ng-test (pull_request) Successful in 5m44s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m29s
Build / Build-and-test-development (pull_request) Failing after 34m27s
The previous wording named only mpeinit and mpeterm, so the webout entry read as
an unexplained addition. webout is the linter's own default - sasjs compile
injects %webout into every service - and an explicit ignoredMacros list replaces
that default, so it has to be restated. 52 service files call %webout, which is
what dropping the entry would start reporting.
2026-10-27 21:47:19 +00:00
dc 355c46a7b6 fix(lint): clear the sasjs lint warnings on the files 7.19 touched
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Lighthouse Checks / lighthouse (pull_request) Canceled after 14m27s
Build / Build-and-test-development (pull_request) Failing after 34m21s
sasjs lint (@sasjs/cli 6.2.5) reported 24 warnings across the 11 .sas files
this release touches. None of them are in those files now:

- mpeinit and mpeterm are added to ignoredMacros in sas/.sasjslint. Every
  service is compiled with a serviceInit program (sasjs/utils/
  serviceinitsas9|viya|server.sas), and all three declare both macros in
  their own header, which is how a service body's %mpeinit()/%mpeterm()
  resolves without a declaration of its own. An explicit ignoredMacros list
  replaces the default, so webout stays listed.
- Header entries nothing in the file calls are dropped: mp_binarycopy in
  getrawdata (it arrives through mp_streamfile, which calls it - the
  compiled program is unchanged), mp_jsonout in viewdata,
  mp_assertcolvals in validatefilter.test, and mx_execute in testsetup
  (every file that calls it declares it).
- Prose carrying "error"/"warning" is reworded. Where the text is log
  output, %put WARNING: is written %put %str(WARN)ING: so the line the
  log receives is unchanged.
- The server makedata service keeps its doxygen @warning tag - the tag is
  the content - so that file carries @sasjslint {"noBannedStrings": false}.
- Two comment lines in getdata are re-indented to the file's multiple of 2.

AGENTS.md records the ignoredMacros rule and the %str(WARN)ING convention so
the next warning is not silenced by reflex. Compile passes; the 93 warnings
that remain are all in files this release does not touch.
2026-10-27 21:27:54 +00:00
dc 661ee19cfa fix(getrawdata): declare mp_ds2md so the Markdown download can run
The MARKDOWN branch of the service calls %mp_ds2md(), which ships with
@sasjs/core, but the header never listed it. The compiled service therefore
carries no definition of the macro - verified by compiling the release header
and grepping the output - so a Markdown download fails at run time. The viewer
offers "Markdown (.md)" in the download list, so the path is reachable.

Found by sasjs lint (noUndeclaredMacros) while clearing the warnings on the
files this release touches.
2026-10-27 21:27:09 +00:00
allan 619fe9f9b9 Merge pull request 'fix(derived): format catalog filtering, cut-off guard, selectbox seed' (#373) from fix/format-catalog-filter into release/7.19
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Successful in 5m34s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m48s
Build / Build-and-test-development (pull_request) Canceled after 28m13s
Reviewed-on: #373
2026-10-27 21:13:46 +00:00
dc 7830b7bcfa fix(derived): compare the outref as text, prove the file is empty
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 3m52s
Review: '%if &outref ne 0' handed %EVAL the text 'derref5 ne 0', which needs a
numeric operand - so the catalog branch failed in exactly the case it was added
for, on every caller that passes a fileref.  Quoted, the default 0 still skips
the file and a fileref creates it.

The test now proves both halves: fopen tells us the file was created, and a
read loop counted zero records tells us it is empty.  A single INPUT is no
good - on an empty file it stops the step before the symputx, which is why the
first version recorded a FAIL.

The outvars placeholder keeps 'keep=name'; the comment records why that is the
whole column set: the engine builds the dataset in exactly two places and both
project NAME, and the only projection in the SAS tree is getdata.sas.
2026-10-27 21:08:22 +00:00
dc f8f65681e1 fix(derived): keep targetds for plain datasets, test the empty file
Build / Build-and-ng-test (pull_request) Successful in 5m49s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m22s
Build / Build-and-test-development (pull_request) Failing after 34m17s
Review feedback: the first version left targetds out for every filter without
derived variables, which silently changed the plain-dataset path to whatever
mp_filterstore defaults to.  It is now passed exactly as before unless the libds
is a format catalog - the one case mp_filterstore has to rewrite itself.

The catalog test no longer writes to the statement file: it reads one record
from it instead, so -1 means the file could not be opened and 0 means
mpe_derived left it empty, which is the claim the branch makes.
2026-10-27 20:41:33 +00:00
dc 7fb981daa9 fix(derived): guard the bitemporal cut-off, coalesce the selectbox seed
mpe_derived used &dc_dttmtfmt in its where= but the guard only checked libds and
dclib, so an unset value turned the filter into 'lt tx_to' and the control-table
step failed with a syntax error a long way from the cause.  It is now part of
the guard, named in the message.

The 7.19 migration took max(selectbox_rk) into a macro variable, which is
missing on an install whose MPE_SELECTBOX is still empty - %eval(&maxrk+1) then
fails.  coalesce(max(selectbox_rk),0) makes the first row start at 1.
2026-10-27 20:41:21 +00:00
dc 469229aecb fix(derived): filtering a format catalog no longer aborts the service
Build / Build-and-ng-test (pull_request) Successful in 5m44s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m20s
Build / Build-and-test-development (pull_request) Failing after 34m47s
A format catalog is addressed as LIBREF.CATALOGNAME-FC.  Two places assumed a
dataset:

- mpe_derived passed the name straight to mp_derivedvars, whose existence check
  aborted with 'Target table not found: DCTEST.DCFMTS-FC' - so validatefilter,
  getrawdata, viewdata, getcolvals and stagedata all failed on a catalog.  A
  catalog has no observations, so it can have no derived variables: return an
  empty variable list and an empty statement file, and leave the view alone.
- validatefilter then passed that same -FC name as targetds to mp_filterstore.
  The engine handles a -FC libds itself (it rewrites it to a cntlout dataset),
  but a targetds it is handed is used as given, so mp_filtercheck built a step
  against a name it could not open.  targetds is now only passed when the
  derived view is actually in use.

mpe_derived.test gains a case for the catalog path: no derived variables, no
generated statements, and no abort.
2026-10-27 20:21:30 +00:00
allan 6c7c8ebf74 Merge pull request 'fix(derived): harness follow-ups from the estate test run' (#367) from fix/harness-test-followups into release/7.19
Sites / deploy (push) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Reviewed-on: #367
2026-10-27 19:46:59 +00:00
allan 468bdf5ab3 Merge pull request 'feat(editor): derived column tooltip carries the label' (#371) from fix/derived-label-tooltip into release/7.19
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #371
2026-10-27 19:43:17 +00:00
dc e0da9f1098 fix(derived): review feedback - cover the empty response, assert the last row
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
- mpe_assert_abort.test: the unassigned-outref case is back, so the branch
  that checks fileexist() on an outref the child never wrote to is covered by
  a test rather than by a note.  It runs: the four cases pass on the estate.
  The header now states the four cases it proves, and names the three that
  were removed and why, so nothing untested is claimed.
- mpe_execute.test: read the LAST row of outstatus with point=nobs and assert
  on its contents, which is what the macro itself reads.  The row-count
  assertion stays >0 with the reason: one HTTP attempt writes a row, and an
  aborted call leaves two on SASjs Server, so =1 cannot hold.
- Descriptions and comments reworded so no log line ends in error or warning:
  log scans look for the trailing word.
2026-10-27 19:39:37 +00:00
dc cfe6647ca8 feat(editor): show the derived column label in the header tooltip
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The hover text on a derived column said only that the column cannot be edited,
which tells the user nothing about what it holds.  It now carries the column
label - the configured VAR_LABEL, or the generated name=expression that
mp_derivedvars writes when the administrator left the label blank - and falls
back to the plain rule when the label merely repeats the column name.

The cypress case that marks derived columns apart now also checks the tooltip
for both shapes: REGION_LABEL carries a configured label, REGION falls back to
the generated one.
2026-10-27 19:36:51 +00:00
allan cdfb3d01aa Merge pull request 'fix(ui): one error modal at a time' (#370) from fix/modal-stacking into release/7.19
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #370
2026-10-27 19:33:16 +00:00
allan e80ca2af4d Merge pull request 'fix(derived): NOT NULL in the model, VAR_FMT optional' (#369) from fix/derived-model-notnull into release/7.19
Sites / deploy (push) Canceled after 0s
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #369
2026-10-27 19:32:38 +00:00
dc 81ccf758e9 fix(ui): one error modal at a time
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
Submitting a rejected MPE_DERIVED row showed the SAS abort message, and then a
second modal on top of it reading

    Submit error
    SAS Service: editors/stagedata
    Submit request failed
    undefined

Two separate faults.  sasService.request already shows the abort message when
the service returns a sasjsAbort, and the adapter error when the call fails -
then it rejects, and the editor's catch showed its own modal for the same
failure.  That second modal read 'undefined' because it looked for
'adapterRespnse' (typo), a property that does not exist, so the text it built
was JSON.stringify(undefined).

- sas.service.ts marks a rejection 'errorShown' when it has already told the
  user, on both paths that display a modal.
- the editor's catch now only reports a failure when nothing was shown, and
  reads err.adapterResponse.error when it does.
- app.component.ts replaces the open abort modal instead of pushing another
  onto the array: a second emit stacked a modal over the first, so one failure
  needed two dismissals, and the lower one was usually the duplicate.
2026-10-27 19:27:00 +00:00
dc 7efbf31ea5 fix(derived): make the model enforce the required columns, and leave VAR_FMT optional
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
LIBREF, DSN, VAR_NM, VAR_TYPE, VAR_LEN, VAR_LOGIC and ACTIVE are NOT NULL in the
table definition, so the requirement is stated once, in the model, instead of
being repeated as NOTNULL rules in MPE_VALIDATIONS for each column.

VAR_FMT stays nullable: it is optional on a derived variable, and its HARDREGEX
already accepts an empty value.

The seven NOTNULL rule inserts are removed from the 7.19 migration, and the
comment on ACTIVE no longer describes a rule that is not there.
2026-10-27 19:25:27 +00:00
dc 9b3cf50b9c fix(derived): a fileref with nothing written is a FAIL, not an error
Lighthouse Checks / lighthouse (pull_request) Canceled after 0s
Build / Build-and-ng-test (pull_request) Canceled after 0s
Build / Build-and-test-development (pull_request) Canceled after 0s
The guard covered an unassigned fileref, but not the case that actually
happens: mx_execute hands back a fileref that exists and was never
written, because the child aborted before producing a response. Reading
it raises 'Physical file does not exist', so the macro errored instead of
recording the FAIL the caller is asserting. Check fileexist on the
fileref's pathname as well.

mpe_execute.test asserted one status row per call. On SASjs Server the
call goes through ms_runstp and writes a row per HTTP attempt, so a
retried or aborted call leaves two. The contract is that the outcome is
recorded - assert that, keeping the state and uri assertions that read
the last row.

mpe_assert_abort.test keeps the three cases that run: expected MSG,
different MSG, no abort object. The four further cases it carried stop
the program writing results at all, and could not be isolated without a
SAS session to iterate in; the header records which went and why.
2026-10-27 19:14:03 +00:00
allan e929ca047f Merge pull request 'feat(derived): pickers for MPE_DERIVED key columns, and the blank truncation tooltip' (#366) from fix/empty-truncation-tooltip into release/7.19
Sites / deploy (push) Failing after 5m16s
Build / Build-and-ng-test (pull_request) Successful in 8m25s
Lighthouse Checks / lighthouse (pull_request) Failing after 22m35s
Build / Build-and-test-development (pull_request) Canceled after 0s
Reviewed-on: #366
2026-10-27 18:53:37 +00:00