Compare commits

...
Author SHA1 Message Date
dc c9eed6dae7 fix(security): escape col-info dropdown and origin-check VA replay
Build / Build-and-ng-test (pull_request) Successful in 5m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m10s
Build / Build-and-test-development (pull_request) Successful in 25m30s
buildColInfoHtml interpolated server/DB-controlled column labels, formats and
DQ RULE_VALUE (regex/formula) strings into HTML assigned to raw DOM
elem.innerHTML in both viewer.component.ts and editor.component.ts. A user who
can author a validation rule (or a column label) could store markup that runs
in the browser of any editor/approver who opens a column info dropdown - the
same class of stored XSS fixed for the status renderers in #319. Escape every
interpolated field via the same escapeHtml approach.

The pre-bootstrap VA listener (va-early.js) stores any-origin postMessage in
window.__vaLastMessage; replayEarlyMessages replayed it into the editor filter
without the live path's isTrustedSource check. Add isTrustedEarlyOrigin so the
replay only accepts a message from this origin or the embedding frame's origin
(document.referrer), mirroring the live handler.

Regression tests: col-info-html.spec.ts proves the injected element does not
survive (fails on the old impl, 5of7 rando-fail -> all pass), preserving the
10 existing behaviour tests; full Angular suite 523/523 green; production
build (AOT) compiles clean.
2026-09-19 16:28:21 +00:00
semantic-release-bot d3aed4c283 chore(release): 7.14.1 [skip ci]
## [7.14.1](https://git.datacontroller.io/dc/dc/compare/v7.14.0...v7.14.1) (2026-09-17)

### Bug Fixes

* **admin:** return refreshcatalog output as webout JSON ([1dccf61](1dccf61f62))
* **deps:** bump @sasjs/cli to 4.20.3 ([d301fc2](d301fc2b4d))
* **deps:** bump @sasjs/core to 5.2.7 ([09f31ce](09f31ce6da))
* **deps:** bump @sasjs/core to 5.2.8 ([c73817f](c73817fb6a))
* **editors:** declare the mp_validatecol dependency in stagedata and loadfile ([a556be4](a556be4c57))
* **hooks:** make the pre-commit size check actually run ([5383803](53838036e6))
* **hooks:** require the repo-pinned gitleaks, drop the fallbacks ([fbcde41](fbcde41321))
* **security:** escape cell values in status renderers to prevent DOM XSS ([cfd8f06](cfd8f06435))
* **security:** validate libds in stagedata and loadfile ([2d31e5a](2d31e5a8b9))
* **viewdata:** return the full row cap from a table search ([199b57d](199b57d9f9))
2026-09-17 16:34:55 +00:00
allan 58c1571669 Merge pull request 'Viya test suite fixes + refreshcatalog webout JSON (core 5.2.8)' (#316) from fix/viya-test-suite-and-refreshcatalog-json into main
Release / Build-production-and-ng-test (push) Successful in 4m51s
Release / Build-and-test-development (push) Successful in 24m20s
Release / release (push) Successful in 8m50s
Reviewed-on: #316
Reviewed-by: Allan <allan@4gl.io>
2026-09-17 16:01:47 +00:00
dc c73817fb6a fix(deps): bump @sasjs/core to 5.2.8
Build / Build-and-ng-test (pull_request) Successful in 5m38s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m35s
Build / Build-and-test-development (pull_request) Successful in 25m31s
5.2.8 handles a canceled child job without aborting the calling program,
which is what restore.test needs to complete.
2026-09-17 11:42:22 +00:00
dc 208c416554 test(viewdata): cover full table search semantics and the row cap
Pins the behaviour the viewer's search box depends on: a character search is
a partial, case sensitive match across every character column, a numeric
search is an exact match, and a search beyond DC_MAXOBS_WEBVIEW reports as
many rows as it returns.
2026-09-17 11:42:11 +00:00
dc 74a0c5ffe4 docs(viewdata): correct the SASControlTable column limits
The header documented SEARCHVAL as $1000, but the service declares it as
$100. An input column longer than the declared length stops the step with
'Multiple lengths were specified for the variable ...', so the documented
header could not be used as written.
2026-09-17 11:41:59 +00:00
dc 199b57d9f9 fix(viewdata): return the full row cap from a table search
The search branch capped the output with 'if _n_ < &DC_MAXOBS_WEBVIEW' while
the unfiltered branch stops at '_n_ > &DC_MAXOBS_WEBVIEW'. A search that
matched at least the cap therefore returned one row fewer than a plain view
of the same table (499 against 500), so the rows returned and the row count
reported by the viewer disagreed.
2026-09-17 11:41:43 +00:00
dc a556be4c57 fix(editors): declare the mp_validatecol dependency in stagedata and loadfile
Both services call %mp_validatecol() but neither declares it in the doxygen
header, so the compiled service carries the call with no definition of the
macro. The job is then canceled with 'Apparent invocation of macro
MP_VALIDATECOL not resolved', which fails stagedata.test.1-3 and
getstagetable.test on a live Viya estate.
2026-09-17 11:41:24 +00:00
allan bbda9d44ca Merge pull request 'test(e2e): add full table search spec with mock search and filter support' (#320) from feat/full-table-search-e2e into fix/viya-test-suite-and-refreshcatalog-json
Build / Build-and-ng-test (pull_request) Successful in 6m8s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m12s
Build / Build-and-test-development (pull_request) Successful in 25m39s
Reviewed-on: #320
2026-09-16 23:21:17 +00:00
dc a1e1880a8c Merge branch 'fix/viya-test-suite-and-refreshcatalog-json' into feat/full-table-search-e2e
Build / Build-and-ng-test (pull_request) Successful in 5m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m2s
Build / Build-and-test-development (pull_request) Successful in 25m56s
Resolve sas/package.json + lock conflict by keeping @sasjs/cli 4.20.4
(adm-zip audit fix) and taking @sasjs/core 5.2.7 from the base branch.
2026-09-16 23:06:43 +00:00
dc a28de96885 chore(deps): bump @sasjs/cli to 4.20.4 to clear adm-zip audit failure
Build / Build-and-ng-test (pull_request) Successful in 5m21s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m59s
Build / Build-and-test-development (pull_request) Successful in 25m21s
2026-09-16 21:18:18 +00:00
dc aae48dc401 test(e2e): scroll the grid so the NOTES column is on screen for the siphonophore beat
Build / Build-and-ng-test (pull_request) Failing after 1m36s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m18s
The three-row siphonophore result is narrower than the grid viewport, so
Handsontable sizes the columns wider than it (scrollWidth 1562 vs
clientWidth 1340): a horizontal scrollbar appears and NOTES - the column the
match is actually in - is clipped. The cell text is in the DOM either way, so
the beat passed without it, but the demo read as three arbitrary rows.

scrollGridTo() sets scrollLeft on #hotTable .wtHolder, asserts the value
landed, and the beat asserts the full NOTES text and scrolls back for the
following beats. It is a plain spec step - CI runs it, no recording flag and
no cy.wait.

Also lets a recording size the browser window: the capture is the window's
content area, so RECORD_WINDOW_SIZE is passed through as --window-size for a
real Chromium browser (Electron ignores it). That is what makes a 16:9 pane
possible: (W-450)/(H-96) = 16/9 at 1920x923 of content, i.e. 1920x1010 of
window, for a 1470x827 pane and the repo's 1600x900 viewport at ~0.90 zoom.

Inert unless the env var is set, so CI is unaffected. dc-cypress updated with
the window-sizing recipe, the clipped-column scroll and the frame
mean/variance scan used to find the cut point.
2026-09-16 18:44:25 +00:00
dc 37c6e50cb6 docs(skills): Cypress runs its own Xvfb, so an external DISPLAY is ignored
Build / Build-and-ng-test (pull_request) Failing after 1m42s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m54s
2026-09-16 15:27:11 +00:00
dc 287e4fa3c2 test(e2e): extend the full table search walkthrough and size the demo capture
Build / Build-and-ng-test (pull_request) Failing after 1m39s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m50s
Seven more search beats (Halcyon across two columns, multi-word site and
species, partial expedition code, exact numeric matches on DEPTH_M and the
last primary key), all with counts verified against the mock service first.
The walkthrough now takes 38s in CI against 28s.

A single-row result renders "(1 row, 9 cols)", not "(1 rows, ...)", so
assertRowCount picks its unit from the count it is given.

demoPause adds a recording-only 1.5s linger at each row count (0 unless the
env var is set, so CI still runs the file with no waits), and the dc-cypress
skill now records how the demo capture actually works: a fixed 1280x720
window containing the runner, with the app zoomed to fit the AUT pane -
matching the viewport aspect to that pane (~1.33) fills it, and the crop
rect plus the cut points are measured from the frames.
2026-09-16 15:18:21 +00:00
dc fea3a1c04e test(e2e): run the full table search walkthrough as a single test
Build / Build-and-ng-test (pull_request) Failing after 1m36s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m37s
Opening the table once and making every variation a search in place - rather
than one it() per variation, each paying for the beforeEach navigation again -
matches how a user works, makes the recorded walkthrough one continuous take,
and cuts the spec from 67s to 26s.

The no-match steps (wrong case, partial number, value not present) are now
wrapped in a negativeStep() guard driven by --env skipNegative=true, so the
recording still never shows an empty result while CI (no env) runs every step.
The walkthrough ends inside the filter: clearing the search leaves the filter
applied, which is asserted rather than assumed.
2026-09-16 01:17:37 +00:00
dc 8674eee714 docs(skills): note the skipNegative pattern for demo recordings in dc-cypress
Build / Build-and-ng-test (pull_request) Failing after 1m44s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m47s
2026-09-16 01:13:50 +00:00
dc 74e5d4eae0 test(e2e): make the no-match search cases skippable for demo recordings
Build / Build-and-ng-test (pull_request) Failing after 1m38s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Three of the cases legitimately match nothing - they pin the case sensitivity
of the character search, the exact-match semantics of the numeric search, and
the "no data found" handling.  They are worth keeping as regression coverage
but they make a poor demo video.

They now use an `itNegative` alias that resolves to `it.skip` when the spec is
run with `--env skipNegative=true`, so the walkthrough recording only ever
shows searches that return rows:

    npx cypress run --browser electron \
      --spec cypress/e2e/full-table-search.cy.ts --env skipNegative=true

CI runs the file with no env, so all nine cases still run there.  The positive
half of the old case-sensitivity case is split out as its own test (3.1) so the
recording keeps it, and the "no data found" assertion is extracted into
assertNoData().
2026-09-16 01:00:00 +00:00
dc 48599c207d test(e2e): add full table search spec with mock search and filter support
Build / Build-and-ng-test (pull_request) Failing after 1m42s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m16s
The viewer's search box searches every column of a table - character columns
by case-sensitive CONTAINS and numeric columns by exact equality - but against
the JS mock backend the box did nothing, because the viewdata mock ignored
SEARCHTYPE/SEARCHVAL (and FILTER_RK) entirely.

Mock changes (sas/mocks/sasjs/services/public/viewdata.js):
- apply the stored filter first via mpeFilterMaster, mirroring
  %mpe_filtermaster(VIEW,...) in viewdata.sas, and return its WHERE text as
  sasparams.FILTER_TEXT (previously hard-coded blank, so the viewer's info bar
  never showed a clause and the filter was never applied to the rows)
- mirror %mp_searchdata: CHAR = case-sensitive CONTAINS across every character
  column, NUM = exact equality across every numeric column, only CHAR/NUM
  trigger a search, search values stripped of % & ; " like the SAS service
- cap the rows the way the real service does (MAXROWS 500; NOBS is the uncapped
  filtered count for a normal view, the capped match count for a search)
- a search with no matches now returns no rows, so the client shows its
  "No data found with given conditions" panel; the single-empty-row fallback
  stays for the normal-view-empty-table case

Mock data (sas/mocks/sasjs/services/admin/makedata.js):
- new MPE_X_SEARCH demo table: 1000 deterministic, obviously fictional
  deep-sea survey rows, registered in MPE_TABLES. Seeded so a partial search
  hits a value buried in the middle of a long text column, a value spanning two
  columns, a repeated exact numeric, and rows that deliberately do not match.

New spec (client/cypress/e2e/full-table-search.cy.ts):
- seven tests covering open/full view, partial character search, case
  sensitivity, numeric exact match, no match, search within a filter, and
  clearing the search. No artificial waits - state assertions only.
- added to the Cypress spec list in .gitea/workflows/build.yaml

Also adds the dc-cypress skill (.agents/skills/dc-cypress) covering the mock
backend setup, the spec conventions, the selectors that actually work in the
viewer filter modal, and the fixed 1280x720 video capture.
2026-09-16 00:48:12 +00:00
allan 24636c9a87 Merge pull request 'fix(security): escape cell values and harden libds inputs' (#319) from security/harden-xss-and-inputs into fix/viya-test-suite-and-refreshcatalog-json
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m5s
Build / Build-and-test-development (pull_request) Successful in 24m38s
Reviewed-on: #319
2026-09-14 16:20:07 +00:00
hermes 11f56cc54a refactor(deploy): remove vestigial client credential fields
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Lighthouse Checks / lighthouse (pull_request) Successful in 20m57s
Build / Build-and-test-development (pull_request) Successful in 24m44s
The deploy client_id/client_secret fields were only read from localStorage
and never used by the manual or automatic deploy flows. Remove the dead
code so no credential-shaped value is read from browser storage.
2026-09-14 16:45:20 +01:00
hermes 2d31e5a8b9 fix(security): validate libds in stagedata and loadfile
Sanitise the libref.dataset input the same way getdata does, via
mp_validatecol, and abort the service on an invalid value to prevent code
injection through the libds identifier. Format catalog inputs resolve to
work.fmtextract and still pass the check.
2026-09-14 16:45:13 +01:00
hermes cfd8f06435 fix(security): escape cell values in status renderers to prevent DOM XSS
The error/no-spinner/spinner cell renderers wrote the cell value straight
into td.innerHTML. A value containing markup (which can arrive from a dataset
served by the getdata stored program or from a typed edit) was therefore
parsed and executed by the browser. Escape the value so it renders as inert
text, keeping the hardcoded icon markup intact, and add a regression test
that reproduces the injection via a real Handsontable instance.
2026-09-14 16:44:59 +01:00
hermes a6a111db8a test(viya): scope stagedata abort assertions to the submitted load
Build / Build-and-ng-test (pull_request) Successful in 6m4s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m41s
Build / Build-and-test-development (pull_request) Successful in 25m35s
The abort checks matched any non-success load in the time window, so a parallel session's aborted/IN PROGRESS load could satisfy them even if the tested load completed. Key them on this test's unique submitted message (reason_txt), which is preserved because a hook-aborted load finishes before mpe_loadfail can overwrite it.
2026-09-14 13:25:32 +01:00
allan ca19dbaefb Merge pull request 'chore(hooks): add gitleaks secret scan to pre-commit' (#317) from chore/gitleaks-precommit into fix/viya-test-suite-and-refreshcatalog-json
Build / Build-and-ng-test (pull_request) Successful in 5m26s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m28s
Build / Build-and-test-development (pull_request) Successful in 25m39s
Reviewed-on: #317
2026-09-14 12:15:38 +00:00
hermes 8418e0cb1d docs(contributing): document one-time git hook activation
Build / Build-and-ng-test (pull_request) Successful in 5m21s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m1s
Build / Build-and-test-development (pull_request) Successful in 24m44s
2026-09-14 12:44:10 +01:00
hermes d301fc2b4d fix(deps): bump @sasjs/cli to 4.20.3
Clears the moderate adm-zip advisory (GHSA-vwc7-r8mq-g2x9) from the sas npm audit gate - @sasjs/cli 4.20.3 replaces adm-zip with yauzl/yazl.
2026-09-14 12:43:58 +01:00
dc fbcde41321 fix(hooks): require the repo-pinned gitleaks, drop the fallbacks
Build / Build-and-ng-test (pull_request) Failing after 1m46s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 21m12s
A soft skip (and the system-gitleaks fallback) let a clone without
node_modules commit with no secrets scan at all, so the hook would
never force the install. The hook now hard-blocks with a hint to run
'npm i' until the @nogoo9/gitleaks binary is present in
node_modules/.bin.

Verified with the binary present (scan runs and passes) and absent
(blocked, exit 1).
2026-09-10 23:06:34 +00:00
dc 53838036e6 fix(hooks): make the pre-commit size check actually run
Build / Build-and-ng-test (pull_request) Failing after 1m47s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m51s
The size check sat after the prettier block, whose branches both exit,
so it was unreachable - and it had three further defects: the exit 1
lived inside a ( ... ) subshell so it could never block the commit,
the 2**20 arithmetic errors under /bin/sh (dash) leaving size_limit
unset, and it measured HEAD^..HEAD (the previous commit) rather than
the content being committed, which does not exist yet in pre-commit.

Rewrite it to sum the sizes of the new blobs the commit would
introduce (git diff --cached --raw, new blob ids -> cat-file
--batch-check), keep the 2MB threshold and the documented --no-verify
escape hatch, and move it before the prettier check so it always runs.

Verified with a staged 3MB blob (blocked, exit 1), a small staged
change (passes through to prettier), and a mode-only change (no new
blob, passes).
2026-09-10 23:03:43 +00:00
dc b3b9755aa7 chore(hooks): add gitleaks secret scan to pre-commit
Build / Build-and-ng-test (pull_request) Failing after 1m45s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s
Run 'gitleaks protect --staged' before prettier in the pre-commit hook.
The gitleaks binary comes from the @nogoo9/gitleaks devDependency,
which ships official binaries as platform optionalDependencies - no
postinstall scripts, so it installs under ignore-scripts=true.
Falls back to a system gitleaks, and soft-skips with a hint to run
'npm i' on a fresh clone rather than blocking the commit.
2026-09-10 22:36:44 +00:00
hermes 82a45b80b4 chore(viya): run tests on the reusable compute context, harden chunk deploy
Build / Build-and-ng-test (pull_request) Successful in 5m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m59s
Build / Build-and-test-development (pull_request) Successful in 24m27s
The test jobs on the nextviya estate run as a batch identity that is
not authorised to create sessions in the SAS Job Execution compute
context - child service submissions via mx_execute all failed with
403.  Default the test context to Compute Reusable (matching the
target contextName) in the shared test init programs.

Also give the chunk deploy helper a retry loop for the per-chunk
sasjs run timeouts - estate cold-starts can exceed a single 300s
window.
2026-09-08 20:52:19 +01:00
hermes a8194d73e7 test(viya): align stagedata test payloads with the adapter contract
The web editor submits date/datetime/time cells as formatted strings
and always provides the format-catalog buskey, so the staged CSVs
round-trip through the ANYDT* informats in mpe_loader.  Some tests
built jsdata with raw numeric values or without FMTROW, causing the
staged loads to fail on Viya.

- getstagetable.test: send some_date/some_datetime/some_time as
  formatted strings (as postdata.test.1 already does), and assert on
  flags captured from the first read - the JSON libname can only be
  read once, so re-reading stagetable returns -1 observations
- postdata.test.2: add fmtrow to the staged format catalog rows (the
  TYPE FMTNAME FMTROW buskey)
- stagedata.test.3: assert the #211 hook aborts through MPE_LOADS - a
  hook-aborted child never returns a webout, and the load stays IN
  PROGRESS because mpe_loadfail cannot run after the abort.  The WIDEBOY
  RLS rule is inserted in the prep section so the (possibly pooled)
  child session reads it from the shared filesystem
2026-09-08 20:52:19 +01:00
hermes 1dccf61f62 fix(admin): return refreshcatalog output as webout JSON
The service wrote a raw HTML line straight to the _webout fileref
after the load completed.  On Viya, API-submitted jobs (eg via
mx_execute) do not always have that fileref connected to a job result,
so the output was lost and job-result consumers aborted.  Return a
proper sasparams table through %webout() instead - the same pattern as
the other admin services - and update the test to assert on the
webout sasparams message.
2026-09-08 20:52:19 +01:00
hermes 09f31ce6da fix(deps): bump @sasjs/core to 5.2.7
Brings in the mv_jobflow JSON-escape fix (prevents jobExecution 400s
for job arguments containing quotes or CR/LF) and the graceful
handling of canceled jobs in mv_getjoblog / mv_getjobresult (a
canceled child no longer aborts the calling program when its log or
result is unavailable).  Verified against a live Viya 2026.07 estate:
the full SAS test suite passes with these macros sourced from core.
2026-09-08 20:52:19 +01:00
semantic-release-bot 90c937c32a chore(release): 7.14.0 [skip ci]
# [7.14.0](https://git.datacontroller.io/dc/dc/compare/v7.13.0...v7.14.0) (2026-09-04)

### Features

* **system:** show session timezone on system information screen ([966dc07](966dc072e0))
2026-09-04 14:31:37 +00:00
allan d9bd46e97b Merge pull request 'feat(system): show session timezone on system information screen' (#315) from fix/system-timezone-info into main
Release / Build-production-and-ng-test (push) Successful in 6m47s
Release / Build-and-test-development (push) Successful in 25m59s
Release / release (push) Successful in 10m45s
Reviewed-on: #315
2026-09-04 13:52:54 +00:00
allan 55c38805d7 Merge branch 'main' into fix/system-timezone-info
Build / Build-and-ng-test (pull_request) Successful in 5m37s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m58s
Build / Build-and-test-development (pull_request) Successful in 25m13s
2026-09-04 12:19:50 +00:00
hermes 966dc072e0 feat(system): show session timezone on system information screen
Build / Build-and-ng-test (pull_request) Successful in 5m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m13s
Build / Build-and-test-development (pull_request) Successful in 25m42s
Add TIMEZONE, SYSTIMEZONEIDENT and SYSTIMEZONEOFFSET to the startupservice
globvars and display them in the Environment Details block, so the session
clock used to stamp submission timestamps is directly visible to admins.
2026-09-04 12:15:01 +00:00
37 changed files with 1628 additions and 291 deletions
+172
View File
@@ -0,0 +1,172 @@
---
name: dc-cypress
description: >
Use this skill when writing or debugging Cypress E2E specs in the dc repo, or when working on the
JS mock backend they run against. Covers the SASjs Server mock environment (deploy, seed, ports),
the spec conventions the CI harness expects, the viewer/editor DOM selectors that the existing
specs get wrong, the fixed 1280x720 video capture, and the rule that a mock service must mirror
the real SAS service it stands in for.
---
# Cypress E2E and the JS mock backend
The specs under `client/cypress/e2e/` run against a mock backend: JS stored programs under `sas/mocks/sasjs/services/` executed by SASjs Server in JS-only mode. No SAS licence, no SAS server. CI (`.gitea/workflows/build.yaml`) builds the same environment - SASjs Server on :5000, `sasjs cbd -t server-ci`, makedata, then `ng serve` on :4200 - and runs a fixed list of specs against it.
This skill covers the parts that are not obvious from reading a spec file: how to stand the environment up, the selectors that actually work, and the traps in the mock services. It does not cover the SAS-side tests (see `.agents/docs/testing.md` and the `dc-sas` skill).
## When to Use
- Adding or changing a spec under `client/cypress/e2e/`.
- A mock service needs a new behaviour, or a spec "does nothing" because the mock ignores an input.
- Recording a demo video from a spec run.
- Debugging a spec that passes locally but fails in CI (usually a viewport or timing assumption).
Don't use for: SAS backend tests (`sasjs test -t 4gl`), Angular unit tests (`npm run test:headless`), or general SAS development (use the `dc-sas` skill).
## Stand up the mock environment
1. **SASjs Server, JS mode.** Download the linux build from the sasjs/server releases, then create a `.env` next to the binary:
RUN_TIMES=js
NODE_PATH=node
CORS=enable
WHITELIST=http://localhost:4200
Start it (`./api-linux`). It runs in desktop mode on :5000 with no authentication.
2. **Deploy the mocks.**
cd sas/mocks/sasjs && sasjs cbd -t server-ci
To start from a clean appLoc, delete it first:
curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
3. **Seed the mock database.**
cd sas/mocks && sasjs request services/admin/makedata -t server-ci \
-d deploy/makedata.json -o ./makedata_out.json
`makedata` replies with HTML (it is normally called as a URL redirect), so the CLI reports `invalid Json string`. That is expected - check the drive instead.
4. **Frontend.** Point the `sasjs` tag in `client/src/index.html` at the mock backend (`serverUrl="http://localhost:5000"`, `appLoc="/Public/app/dc"`, `serverType="SASJS"` - a local-only edit, the repo default is `serverUrl=""`), then `npx ng serve --host 0.0.0.0 --port 4200`. `client/cypress.env.json` needs `username`/`password` entries; any values work, `loginAndUpdateValidKey()` no-ops when there is no login form.
5. **Run.**
cd client && npx cypress run --browser electron --spec cypress/e2e/<spec>.cy.ts
Only Electron is installed here (no system Chrome), so always pass `--browser electron`.
## Spec conventions
- Start the file with `export {}` so its top-level consts do not collide with other spec files under the TS type-checker.
- `beforeAll`: `cy.visit(\`${hostUrl}/SASLogon/logout\`)` then `cy.loginAndUpdateValidKey(true)`. The `true` forces a licence key instead of relying on an earlier spec in the same run having applied one.
- `beforeEach`: `cy.visit(hostUrl + appLocation)`, `visitPage('home')`, `visitPage('view/data')`. The `home` visit is what triggers the startup-data fetch; going straight to `view/data` leaves the nav tree empty in slower environments.
- Declare helpers at the **bottom of each spec file** (that is the house pattern) - do not introduce a shared helper module.
- Use `Cypress.env('longerCommandTimeout')` for waits and gate on `.app-loading` disappearing.
- **No artificial delays.** These specs are the CI regression suite, so `cy.wait(n)` to "let a result land" is not acceptable - assert on the resulting state instead (Cypress retries assertions). Replace a sleep with a state assertion: e.g. after picking an option from a soft-select, wait for `#datalist_<inputId>` to have class `hidden` rather than sleeping.
- Lint: `cd client && npm run lint:check` (prettier covers `cypress/e2e/*.cy.ts`).
## Selectors that work
- Viewer search box: `input[name="search-input"]` (Enter or the magnifier icon submits); the numeric toggle is `input[name="numeric_check"]`.
- Row/column count in the header: `.title-col p` renders `(1,000 rows, 9 cols)` - and `(1 row, 9 cols)` when the result is a single row, so an assertion on the count has to pluralise.
- Filter info bar: `.infoBar b` - only rendered when the backend's `sasparams.FILTER_TEXT` is neither blank, `' '`, nor `'1=1'`.
- Empty result: `.noData h3` with `No data found with given conditions`.
- Grid: `#hotTable .ht_master.handsontable .htCore tbody tr` (body rows), `#hotTable .ht_clone_top .htCore thead` (headers, and `button.changeType` confirms Handsontable has finished rendering them).
- Nav tree: `.nav-tree clr-tree > clr-tree-node` -> `.clr-tree-node-content-container .clr-treenode-content p` (expands a library) -> `.clr-treenode-link` (table links).
- **Viewer filter modal** - `.filterSide` (the options button) -> `.dropdown-menu` contains `Filter` -> `#vals_var_id0_0` (variable, pick from `#datalist_vals_var_id0_0 option`) and `#vals_0_0` (value) -> `.filter-modal button[aria-label="Ok"]`.
- `filtering.cy.ts` opens the filter with `.btnCtrl .btnView`, which is an **editor** control and does not exist in the viewer. Do not copy that helper into a viewer spec.
- The soft-select inputs drop a transparent click-catcher over the modal while their suggestion list is open, so Cypress reports the Ok button (and the next input) as covered by `.overlay`. Send the Escape keyup a user would send (`trigger('keyup', { key: 'Escape', force: true })`) and wait for the list to close before clicking on.
## Recording a demo video
`video: true` in `cypress.config.ts` is all that is needed. Run the spec and the MP4 lands in `client/cypress/videos/<spec>.mp4`.
**The capture is the browser window's content area, and it contains the Cypress runner** - the command log takes the left ~450px and the app is rendered in the remaining pane, so the pane is `(capture width - 450) x (capture height - 96)` and the app is zoomed to fit it.
That makes the window size the only lever that matters:
- Electron is stuck at 1280x720 (it ignores `--window-size`), i.e. an 830x624 pane, and the repo's 1600x900 viewport renders at 50% - the app only occupies ~806x452 of it.
- A real Chromium browser honours `--window-size`, so the capture can be sized to whatever the pane needs. `cypress.config.ts` passes it through from `RECORD_WINDOW_SIZE` (inert in CI, which sets no env).
For a 16:9 pane, solve `(W - 450) / (H - 96) = 16/9`: W = 1920 gives H = 923 of window *content*, and Chrome adds ~87px of window chrome above that, so pass 1920x1010:
RECORD_WINDOW_SIZE=1920,1010 npx cypress run \
--browser /opt/data/profiles/dc/tools/chrome-linux64/chrome \
--spec cypress/e2e/full-table-search.cy.ts \
--env skipNegative=true,demoPause=1500
That yields a 1920x922 capture with a 1470x826 pane (16:9) and, at the repo's 1600x900 viewport, an app at ~0.90 zoom occupying 1441x812 - about 1.9x the linear size of the 16:9 crop Electron can give. Crop the app out:
ffmpeg -ss <start> -i cypress/videos/<spec>.mp4 -vf "crop=1440:812:464:80" \
-an -c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p \
-movflags +faststart <spec>-cut.mp4
Measure the crop rect from the frames rather than guessing: the pane's flat background is rgb(225,227,236), so a row/column scan for pixels that differ from it gives the app's bounding box (for the 1920x922 capture: runner pane ends x=449, app x=463-1903, app y=80-893; for the 1280x720 Electron capture: app x=468-1259, y=80-675). The app's own background is nearly the same grey, so verify the result on the first, middle and last frames rather than trusting a pixel test alone.
To find where to cut, scan the mean *and* standard deviation of the content area: the app boots mid-run (a dark screen, mean ~65, sd ~2.5), the "Loading Table Viewer" spinner is flat and light (mean ~252, sd ~0), and the rendered table is mean ~245, sd ~31. Cut on the first table frame - cutting a fraction early shows the boot screen or the spinner.
Cypress starts its own Xvfb for the run (a second display appears in `/tmp/.X11-unix`), so an external `DISPLAY` is ignored; `--force-device-scale-factor` and a CSS `zoom` also do not change the capture.
### Grid columns that are clipped
Handsontable sizes columns to content, so a narrow result can be *wider* than the grid viewport: the horizontal scrollbar appears and the rightmost column (NOTES on `MPE_X_SEARCH`) is cut off. The cell text is still in the DOM, so an assertion passes either way - the column only has to be scrolled into view for the *video* to show why the rows matched.
const scrollGridTo = (position: 'left' | 'right') => {
cy.get('#hotTable .wtHolder').first().then(($holder: any) => {
const el = $holder[0]
el.scrollLeft = position === 'right' ? el.scrollWidth : 0
expect(el.scrollLeft).to.equal(position === 'right' ? el.scrollWidth - el.clientWidth : 0)
})
}
Setting `scrollLeft` is enough - Handsontable keeps the header clone in step - and the browser clamps it to `scrollWidth - clientWidth`, so that is what scrolling right has to land on. This is a legitimate spec step (it runs in CI), not a recording flag.
### A longer, more readable demo
A spec that doubles as a walkthrough should be a single `it()`: open the table once and make each variation a search in place. The recording then reads as a demo rather than a sequence of page loads, and it is far faster - every extra `it()` pays for the `beforeEach` navigation again.
Two recording-only env flags, both inert in CI because CI passes no env:
const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
const negativeStep = (step: () => void) => {
if (!skipNegative) step()
}
const demoPauseMs = Number(Cypress.env('demoPause')) || 0
negativeStep(() => {
searchFor('trench')
assertNoData()
})
- `skipNegative=true` drops the steps that legitimately match nothing (wrong case, partial number, value not present), so the recording never shows an empty screen.
- `demoPause=1500` pauses 1.5s at each row-count assertion, so each result can be read. Apply it in one place (the row-count assertion, which every beat makes exactly once) rather than sprinkling `cy.wait()` through the test - CI still runs the file with no waits at all.
More beats, not a slower take, is what makes a demo longer: a walkthrough of 12 row-count beats takes 52s against 30s for 7.
## Mock services
- One JS file per service under `sas/mocks/sasjs/services/**`, deployed as SASjs stored programs. Shared helpers live in `sas/mocks/sasjs/services/dcMockUtils.js` (`fetchTable`/`parseCsv` for `%webout(FETCH)`, `webOutOpen`/`webOutObj`/`webOutClose` for `%webout`, `mpeFilterMaster`, `mpFilterGenerate`, `mpFilterStore`, `getDdType`, `formatSasValue`, `sasVarsEntry`).
- Mock data is one JSON file per table at `<drive>/files/<appLoc>/data/<LIBREF>/<table>.json`, written by `services/admin/makedata.js` (which also writes `services/settings.js`). The drive lives under the SASjs Server directory (`sasjs_root/drive`).
- `makedata.js` self-destructs after a successful run (it deletes itself from the drive so the frontend can detect completion), so re-deploy before re-seeding.
- **A mock must mirror the real service.** `viewdata.js` once ignored `FILTER_RK` and hard-coded `FILTER_TEXT` to blank, so the viewer's filter silently filtered nothing and its info bar never showed a clause, while the real `viewdata.sas` applies `%mpe_filtermaster` and then `%mp_searchdata`. If a mock is a deliberate stub, say so in a comment and make the divergence explicit - a mock that quietly drops an input turns an E2E spec into a test of nothing.
## Pitfalls
- `sasjs cbd -t server-ci` rebuilds the *mock services* and re-deploys the streamed web app from `client/dist`; it does not rebuild the Angular app.
- Deleting the drive folder before deploying avoids stale services and stale data.
- Captured `_webout` JSON, request logs and iteration snapshots belong in `tmp/` (gitignored), never in `sas/mocks/` - see the `dc-sas` skill.
- The mock server has no auth, so `cy.visit` on `/SASLogon/logout` returns 404 - that is fine, Cypress does not fail on the status code.
## Verification
- `npx cypress run --browser electron --spec <specs>` - all green, and run it more than once when you have just removed a wait.
- `cd client && npm run lint:check` passes.
- For a mock change, hit the service directly before writing UI assertions:
curl -sS -H "Content-Type: application/json" -X POST \
--data '{"_program":"/Public/app/dc/services/public/viewdata","SASControlTable":"LIBDS:$41. FILTER_RK:best. SEARCHTYPE:$4 SEARCHVAL:$1000\nDC_JSLIB.MPE_X_SEARCH,0,CHAR,Trench"}' \
http://localhost:5000/SASjsApi/stp/execute
then check `sasparams[0].NOBS`, the row count and `FILTER_TEXT`. Pin those exact numbers in the spec.
+36 -11
View File
@@ -3,6 +3,42 @@
# Using `--silent` helps for showing any errs in the first line of the response
# The first line is picked up by the VS Code GIT UI popup when rc is not 0
# Scan staged changes for secrets before anything else runs.
# The binary is pinned by the @nogoo9/gitleaks devDependency and lands
# in node_modules/.bin after npm i. Commits are blocked until it is
# installed so no clone silently skips the secrets scan.
gitleaks_bin=node_modules/.bin/gitleaks
if [ ! -x "$gitleaks_bin" ]; then
echo "❌ gitleaks not found - run 'npm i' to install it"
exit 1
fi
if ! "$gitleaks_bin" protect --staged --redact; then
echo "❌ gitleaks detected a potential secret in your staged changes"
echo "Remove the secret, or add a .gitleaksignore entry if it is a false positive"
exit 1
fi
## Avoid large commits
# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
size_limit=$((2 * 1024 * 1024)) # 2MB
# Sum the sizes of the blobs this commit would introduce: the staged
# added/copied/modified/renamed/typechanged entries in the index.
# https://git-scm.com/docs/git-cat-file#Documentation/git-cat-file.txt---batch-check
# Renames are treated as add+delete (--no-renames) and pure mode changes
# (same blob, different permissions) add no new blob, so both are safe.
commit_size=$(git diff --cached --raw --no-renames --diff-filter=ACMRT |
awk '$3 != $4 { print $4 }' |
git cat-file --batch-check='%(objectsize)' |
awk '{ s += $1 } END { print s + 0 }')
if [ "$commit_size" -ge "$size_limit" ]; then
echo "Commit size is too large: $commit_size > $size_limit"
echo "Force commit using --no-verify"
exit 1
fi
if npm run --silent lint:check:silent ; then
exit 0
else
@@ -10,14 +46,3 @@ else
echo "❌ Prettier check failed! We ran lint:fix for you. Please add & commit again."
exit 1
fi
## Avoid large commits
# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
size_limit=$((2 * 2**20)) # 2mbs
# https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---disk-usage
commit_size=$(git rev-list --disk-usage HEAD^..HEAD)
test "$commit_size" -lt "$size_limit" || (
echo "Commit size is too large: $commit_size > $size_limit"
echo "Force commit using --no-verify"
exit 1
)
+1 -1
View File
@@ -154,7 +154,7 @@ jobs:
# Start frontend and run cypress
# timeout 1800: SIGTERM after 30 min so Cypress can flush video/screenshots
# before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
- name: Zip Cypress videos
if: always()
+23
View File
@@ -1,3 +1,26 @@
## [7.14.1](https://git.datacontroller.io/dc/dc/compare/v7.14.0...v7.14.1) (2026-09-17)
### Bug Fixes
* **admin:** return refreshcatalog output as webout JSON ([1dccf61](https://git.datacontroller.io/dc/dc/commit/1dccf61f6290e50ebc365964fab864cb93f8d1b5))
* **deps:** bump @sasjs/cli to 4.20.3 ([d301fc2](https://git.datacontroller.io/dc/dc/commit/d301fc2b4dc86b7eb3027b8c8965dd7d1079363e))
* **deps:** bump @sasjs/core to 5.2.7 ([09f31ce](https://git.datacontroller.io/dc/dc/commit/09f31ce6da1e6a286c4e05ac55fb1d0c801d62f5))
* **deps:** bump @sasjs/core to 5.2.8 ([c73817f](https://git.datacontroller.io/dc/dc/commit/c73817fb6a8c19d970a19b98d5ef79c05243cc8d))
* **editors:** declare the mp_validatecol dependency in stagedata and loadfile ([a556be4](https://git.datacontroller.io/dc/dc/commit/a556be4c577020c8e3147de4d3e71ab84fb0d174))
* **hooks:** make the pre-commit size check actually run ([5383803](https://git.datacontroller.io/dc/dc/commit/53838036e635cd6f33dccc4534d6f258d3b3da5d))
* **hooks:** require the repo-pinned gitleaks, drop the fallbacks ([fbcde41](https://git.datacontroller.io/dc/dc/commit/fbcde4132147240fed21a88d44f62b68f58cf67c))
* **security:** escape cell values in status renderers to prevent DOM XSS ([cfd8f06](https://git.datacontroller.io/dc/dc/commit/cfd8f064352940cd57cf19c86504036f0a7edd8d))
* **security:** validate libds in stagedata and loadfile ([2d31e5a](https://git.datacontroller.io/dc/dc/commit/2d31e5a8b957d30057eb749c2d7f246170af8c8f))
* **viewdata:** return the full row cap from a table search ([199b57d](https://git.datacontroller.io/dc/dc/commit/199b57d9f9fb4b00a229d23e25e22e5dee471d2a))
# [7.14.0](https://git.datacontroller.io/dc/dc/compare/v7.13.0...v7.14.0) (2026-09-04)
### Features
* **system:** show session timezone on system information screen ([966dc07](https://git.datacontroller.io/dc/dc/commit/966dc072e0cb8bb44bd93a7682c3d7335428d688))
# [7.13.0](https://git.datacontroller.io/dc/dc/compare/v7.12.0...v7.13.0) (2026-09-03)
+15
View File
@@ -4,6 +4,21 @@
## Workflow guidelines
[Wiki Page](https://git.datacontroller.io/dc/dc/wiki/Git-Workflow)
## Git hooks
The repo ships its own git hooks in [`.git-hooks/`](./.git-hooks):
- `pre-commit` - scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)
- `commit-msg` - verifies the commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) standard
The repo `.npmrc` sets `ignore-scripts=true`, so the `prepare` and `postinstall` lifecycle scripts that would normally set `core.hooksPath` during `npm i` never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
```bash
git config core.hooksPath ./.git-hooks
```
The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
## Dependencies that requires licences
[SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)
+15
View File
@@ -38,9 +38,24 @@ export default defineConfig({
// Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
// render deterministically regardless of the runner's system locale.
on('before:browser:launch', (browser, launchOptions) => {
// Recording only: the video capture is the browser window, which
// contains the Cypress runner - the command log takes the left ~450px
// and the app sits in the remaining pane. Sizing the window is the
// only lever that moves the capture, and only for a real Chromium
// browser (Electron ignores it). For a 16:9 pane, solve
// (W - 450) / (H - 96) = 16/9: W = 1920 -> H = 923 of window content,
// so pass RECORD_WINDOW_SIZE=1920,1010 and crop the pane.
// Inert unless the env var is set, so CI is unaffected.
if (process.env.RECORD_WINDOW_SIZE && browser.family === 'chromium') {
launchOptions.args.push(
`--window-size=${process.env.RECORD_WINDOW_SIZE}`
)
}
if (browser.family === 'chromium' && browser.name !== 'electron') {
launchOptions.args.push('--lang=en-GB')
}
return launchOptions
})
}
+385
View File
@@ -0,0 +1,385 @@
// Marks this file as an ES module (rather than a global script) so its
// top-level consts don't collide, under the TS type-checker, with the same
// names declared in other spec files — see e.g. viewer-labels.cy.ts.
export {}
const username = Cypress.env('username')
const password = Cypress.env('password')
const hostUrl = Cypress.env('hosturl')
const appLocation = Cypress.env('appLocation')
const longerCommandTimeout = Cypress.env('longerCommandTimeout')
const serverType = Cypress.env('serverType')
const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
// Fixture: MPE_X_SEARCH, a 1000-row demo table built by the mock data builder
// (sas/mocks/sasjs/services/admin/makedata.js). It is deterministic, so every
// count below can be derived by hand:
// - 1000 rows, PK 1001-2000, columns in demo order: PRIMARY_KEY_FIELD,
// SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE,
// EXPEDITION_ID, NOTES
// - 300 rows have a site name containing "Trench" (case sensitive, so
// "trench" matches nothing)
// - 280 rows contain "Halcyon": the site Halcyon Trench (100 rows) plus the
// vessel RV Halcyon (200 rows) less the 20 rows that are both
// - 3 rows carry "siphonophore" in the middle of their NOTES text, in the
// sites Oceanus Rise, Nereid Trench and Halcyon Trench
// - 100 rows observed Vampire squid (species is picked by (site+vessel)%10)
// - 100 rows are at Oceanus Rise (site is picked by (n%50)/5)
// - 9 rows have an expedition id starting EXP-000 (EXP-0001..EXP-0009)
// - the first row (PK 1001) is the only one with EXPEDITION_ID EXP-0001
// - 12 rows were sounded at exactly 1000m (DEPTH_M = 1000 + (n%90)*37)
// - 2 rows have SAMPLE_COUNT 4210 (the only numeric value that is not a
// primary key, a depth or a date)
// - PK 2000 is the last row of the table
// - 200 rows were surveyed by RV Halcyon, 60 of them at a Trench site
const demoTable = 'mpe_x_search'
// Full-table search: the viewer's search box searches EVERY column of the
// table, character columns by case-sensitive CONTAINS and numeric columns by
// exact equality (see %mp_searchdata in the SASjs macro core, and
// viewdata.sas / services/public/viewdata.js).
//
// The whole walkthrough is one test - the table is opened once and every
// variation is a search in place, which is how a user works and what makes the
// recording read as a demo rather than a set of page loads.
//
// Some searches legitimately match nothing: they pin the case sensitivity, the
// exact-match numeric semantics and the "no data found" handling. Those steps
// are wrapped in negativeStep() and can be skipped when recording the demo, so
// the video only ever shows searches that return rows:
//
// npx cypress run --browser electron \
// --spec cypress/e2e/full-table-search.cy.ts --env skipNegative=true
//
// CI runs the file with no env, so every step runs there.
const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
const negativeStep = (step: () => void) => {
if (!skipNegative) step()
}
context('full table search tests: ', function () {
this.beforeAll(() => {
cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
// forceLicenceKey=true: applies a valid licence key instead of relying on
// an earlier spec in the same run having already done so.
cy.loginAndUpdateValidKey(true)
})
this.beforeEach(() => {
cy.visit(hostUrl + appLocation, { timeout: longerCommandTimeout })
// Visit 'home' first, matching filtering.cy.ts/viewer-labels.cy.ts — going
// straight to 'view/data' skips the startup-data fetch 'home' triggers,
// which leaves the nav tree empty in slower environments.
visitPage('home')
visitPage('view/data')
})
it('searches the whole table from the viewer', () => {
// Full view: every column, and the true row count for the table.
openTableFromTree(libraryToOpenIncludes, demoTable)
assertRowCount('1,000')
assertGridContains(['Nereid Trench', 'RV Pelagia', 'Vampire squid'])
// Partial character search: "siphonophore" only ever appears in the middle
// of the NOTES text, so a match proves the search is not column-specific.
// Three rows is narrow enough that the grid grows a horizontal scrollbar
// and clips NOTES - the column the match is actually in - so scroll it
// into view, assert on what is now on screen, then put the grid back.
searchFor('siphonophore')
assertRowCount('3')
scrollGridTo('right')
pauseForDemo()
assertGridContains(['Oceanus Rise', 'Nereid Trench', 'Halcyon Trench'])
assertGridExcludes(['Kraken Hollow', 'Vostok Deep'])
assertGridContains(['bioluminescent siphonophore was filmed here'])
scrollGridTo('left')
// A search spans every column at once: "Halcyon" is both a site (Halcyon
// Trench) and a vessel (RV Halcyon), and the 280 matches are the union.
searchFor('Halcyon')
assertRowCount('280')
assertGridContains(['Halcyon Trench', 'RV Halcyon'])
// A multi-word site name: the search is not limited to single tokens.
searchFor('Oceanus Rise')
assertRowCount('100')
assertGridContains(['Oceanus Rise'])
// Multi-word values work the same way - species names are two words.
searchFor('Vampire squid')
assertRowCount('100')
assertGridContains(['Vampire squid'])
assertGridExcludes(['Yeti crab'])
// A partial value in a short code column: EXP-000 matches the first nine
// expedition ids.
searchFor('EXP-000')
assertRowCount('9')
assertGridContains(['1001', '1009'])
// A value that is unique in the table: the first row's expedition id.
searchFor('EXP-0001')
assertRowCount('1')
assertGridContains(['1001'])
// Character search is case sensitive: the same word in the wrong case
// matches nothing.
negativeStep(() => {
searchFor('trench')
assertNoData()
})
searchFor('Trench')
assertRowCount('300')
assertGridContains(['Halcyon Trench'])
assertGridExcludes(['Kraken Hollow'])
// Numeric search is an exact match across the numeric columns.
setNumericSearch(true)
searchFor('4210')
assertRowCount('2')
assertGridContains(['1221', '1781', '4210'])
// The numeric search covers every numeric column, not just the primary
// key: 12 rows were sounded at exactly 1000m.
searchFor('1000')
assertRowCount('12')
assertGridContains(['1091'])
// ...and an exact primary key is a single row.
searchFor('2000')
assertRowCount('1')
assertGridContains(['2000'])
negativeStep(() => {
searchFor('421')
assertNoData()
})
// Back to a character search: clearing the box restores the full table.
setNumericSearch(false)
searchFor('')
assertRowCount('1,000')
// A search runs inside the current filter, not the whole table.
openFilterPopup(() => {
setFilterWithValue('VESSEL', 'RV Halcyon', () => {
assertRowCount('200')
checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
searchFor('Trench')
assertRowCount('60')
assertGridContains(['Halcyon Trench'])
checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
// Clearing the search leaves the filter in place.
searchFor('')
assertRowCount('200')
checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
})
})
negativeStep(() => {
searchFor('not-in-this-table')
assertNoData()
})
})
})
const visitPage = (url: string) => {
cy.visit(`${hostUrl}${appLocation}/#/${url}`)
}
// Recording-only linger. A beat's row count is the moment its result is on
// screen, so that is where a demo recording pauses to let it be read. CI runs
// with no env, so demoPauseMs is 0 and the spec contains no waits at all:
//
// npx cypress run --browser electron --spec cypress/e2e/full-table-search.cy.ts \
// --env skipNegative=true,demoPause=1500
const demoPauseMs = Number(Cypress.env('demoPause')) || 0
// Linger for a recording. A no-op unless demoPause is set, which only the
// recording run does - CI passes no env.
const pauseForDemo = () => {
if (demoPauseMs > 0) cy.wait(demoPauseMs)
}
// The row/column count the viewer shows next to the table name, e.g.
// "(1,000 rows, 9 cols)". The viewer pluralises: a single-row result reads
// "(1 row, 9 cols)".
const assertRowCount = (rows: string) => {
cy.get('.title-col p', { timeout: longerCommandTimeout }).should(
($el: any) => {
const unit = rows === '1' ? 'row' : 'rows'
expect($el.text().replace(/\s+/g, ' ')).to.contain(`(${rows} ${unit},`)
}
)
pauseForDemo()
}
// The grid body is a Handsontable .wtHolder, and a narrow result is wider than
// the grid viewport, which clips the rightmost column (NOTES). Setting
// scrollLeft is enough - Handsontable keeps the header clone in step - and the
// assertion right after proves the scroll landed rather than assuming it. The
// browser clamps scrollLeft to (scrollWidth - clientWidth), so that is what
// scrolling right has to land on.
const scrollGridTo = (position: 'left' | 'right') => {
cy.get('#hotTable .wtHolder')
.first()
.then(($holder: any) => {
const el = $holder[0]
const maxScroll = el.scrollWidth - el.clientWidth
el.scrollLeft = position === 'right' ? el.scrollWidth : 0
if (position === 'right') {
// This step only means anything if the grid really is clipping a
// column - fail loudly rather than silently doing nothing.
expect(el.scrollWidth).to.be.greaterThan(el.clientWidth)
expect(el.scrollLeft).to.equal(maxScroll)
} else {
expect(el.scrollLeft).to.equal(0)
}
})
}
// A search that matches nothing replaces the grid with the viewer's "no data"
// panel - and must not raise a request error.
const assertNoData = () => {
cy.get('.noData h3', { timeout: longerCommandTimeout }).should(
'contain.text',
'No data found with given conditions'
)
}
const assertGridContains = (values: string[]) => {
cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
($grid: any) => {
const text = $grid.text().replace(/\s+/g, ' ')
for (const value of values) expect(text).to.contain(value)
}
)
}
const assertGridExcludes = (values: string[]) => {
cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
($grid: any) => {
const text = $grid.text().replace(/\s+/g, ' ')
for (const value of values) expect(text).to.not.contain(value)
}
)
}
const searchFor = (value: string) => {
cy.get('input[name="search-input"]', { timeout: longerCommandTimeout })
.should('be.visible')
.clear()
if (value.length > 0) cy.get('input[name="search-input"]').type(value)
cy.get('input[name="search-input"]').type('{enter}')
}
const setNumericSearch = (enabled: boolean) => {
cy.get('input[name="numeric_check"]', { timeout: longerCommandTimeout }).then(
(checkbox: any) => {
if (checkbox[0].checked !== enabled) checkbox[0].click()
}
)
}
const checkInfoBarIncludes = (text: string) => {
cy.get('.infoBar b', { timeout: longerCommandTimeout }).should(($el: any) => {
expect($el.text().toLowerCase()).to.contain(text.toLowerCase())
})
}
// Opens the viewer's options menu and clicks through to the filter modal.
const openFilterPopup = (callback?: any) => {
cy.get('.filterSide', { timeout: longerCommandTimeout }).first().click()
cy.get('.dropdown-menu', { timeout: longerCommandTimeout })
.contains('Filter')
.click()
cy.get('#vals_var_id0_0', { timeout: longerCommandTimeout }).should('exist')
if (callback) callback()
}
// Fills the first clause of the filter modal (variable, then value) and
// submits it - the viewer stores the clause, reloads the table with the new
// filter RK and shows the generated WHERE text in the info bar.
const setFilterWithValue = (
variableValue: string,
valueString: string,
callback?: any
) => {
cy.get('#vals_var_id0_0').type(variableValue)
cy.get('#datalist_vals_var_id0_0 option')
.contains(variableValue)
.click({ force: true })
// The soft-select closes its suggestion list shortly after a selection -
// wait for that closed state instead of sleeping.
cy.get('#datalist_vals_var_id0_0').should('have.class', 'hidden')
cy.get('#vals_0_0').type(valueString)
closeAutocompleteList('#vals_0_0')
cy.get('#datalist_vals_0_0').should('have.class', 'hidden')
cy.get('.filter-modal button[aria-label="Ok"]').click()
if (callback) callback()
}
// The soft-select inputs open a suggestion list that drops a transparent
// click-catcher over the modal until it closes, so send the same Escape key a
// user would press. `force` is needed because the input sits underneath that
// click-catcher while the list is open.
const closeAutocompleteList = (inputSelector: string) => {
cy.get(inputSelector).trigger('keyup', { key: 'Escape', force: true })
}
const openTableFromTree = (libNameIncludes: string, tablename: string) => {
cy.get('.app-loading', { timeout: longerCommandTimeout })
.should('not.exist')
.then(() => {
cy.get('.nav-tree clr-tree > clr-tree-node', {
timeout: longerCommandTimeout
}).then((treeNodes: any) => {
let viyaLib
for (let node of treeNodes) {
if (node.innerText.toLowerCase().includes(libNameIncludes)) {
viyaLib = node
break
}
}
cy.get(viyaLib).within(() => {
cy.get(
'.clr-tree-node-content-container .clr-treenode-content p'
).click()
cy.get('.clr-treenode-link').then((innerNodes: any) => {
for (let innerNode of innerNodes) {
if (innerNode.innerText.toLowerCase().includes(tablename)) {
innerNode.click()
break
}
}
})
})
})
})
// Selecting the table triggers async SPA routing + a viewdata fetch; wait
// for the grid to actually render before any subsequent action.
cy.get('#hotTable .ht_clone_top .htCore thead button.changeType', {
timeout: longerCommandTimeout
}).should('exist')
}
@@ -21,8 +21,6 @@ export class DeployComponent implements OnInit {
public step: number = 0
public adminGroups: any = []
public client_id: string = ''
public client_secret: string = ''
public appLoc: string = ''
public dcPath: string = ''
public selectedAdminGroup: string = ''
@@ -52,9 +50,6 @@ export class DeployComponent implements OnInit {
this.sasJs = this.sasService.getSasjsInstance()
this.sasJsConfig = this.sasService.getSasjsConfig()
this.appLoc = this.dcAdapterSettings?.appLoc || ''
this.client_id = localStorage.getItem('deploy_client_id') || ''
this.client_secret = localStorage.getItem('deploy_secret_key') || ''
this.dcPath = localStorage.getItem('deploy_dc_loc') || ''
}
ngOnInit() {
@@ -1,5 +1,10 @@
import Handsontable from 'handsontable'
import { makeNumberFormatRenderer } from './renderers.utils'
import {
makeNumberFormatRenderer,
errorRenderer,
noSpinnerRenderer,
spinnerRenderer
} from './renderers.utils'
describe('makeNumberFormatRenderer', () => {
it('renders a numeric cell as EUR currency without changing the value', () => {
@@ -86,3 +91,67 @@ describe('makeNumberFormatRenderer', () => {
container.remove()
})
})
/**
* DOM-injection reproduction mirroring the editor's cell-render cycle.
* During dynamic cell validation the editor applies one of the status
* renderers to a cell via setCellMeta + hot.render(). Those renderers paint
* the cell value with td.innerHTML, so a value containing markup is injected
* and executed (the <img onerror> fires in the browser). The value can come
* straight from a dataset row served by the getdata stored program, or from a
* typed edit. These fail on the vulnerable implementation and pass once the
* renderer escapes the value.
*/
describe('grid cell renderers do not inject raw HTML', () => {
const maliciousValue = '<img src=x onerror=alert(1)>'
// Seed a real Handsontable grid with the payload as a loaded cell value,
// then apply the given status renderer and render — exactly the sequence the
// editor uses during the dynamic-validation cycle.
const renderWith = (
renderer: (
i: any,
td: any,
r: number,
c: number,
p: any,
v: any,
cp: any
) => any
) => {
const container = document.createElement('div')
document.body.appendChild(container)
const hot = new Handsontable(container, {
data: [{ SOME_CHAR: maliciousValue }],
columns: [{ data: 'SOME_CHAR', type: 'text' }],
licenseKey: 'non-commercial-and-evaluation'
})
hot.render()
hot.setCellMeta(0, 0, 'renderer', renderer)
hot.render()
const td: HTMLTableCellElement | null = hot.getCell(0, 0)
hot.destroy()
container.remove()
return td
}
// A vulnerable renderer turns the value into a real <img> element with an
// onerror handler (proven by the browser firing alert(1)). A safe
// renderer leaves no such element. Asserting on the parsed DOM rather
// than the raw string avoids false passes from browser attribute normalising.
const assertNoInjectedElement = (td: HTMLTableCellElement | null) => {
expect(td?.querySelector('img[onerror]')).toBeNull()
}
it('noSpinnerRenderer escapes rather than injecting the value', () => {
assertNoInjectedElement(renderWith(noSpinnerRenderer))
})
it('errorRenderer escapes rather than injecting the value', () => {
assertNoInjectedElement(renderWith(errorRenderer))
})
it('spinnerRenderer escapes rather than injecting the value', () => {
assertNoInjectedElement(renderWith(spinnerRenderer))
})
})
+21 -7
View File
@@ -1,5 +1,23 @@
import Handsontable from 'handsontable'
/**
* Returns string-safe text of any value so it can be assigned to innerHTML.
* The cell values painted by the status renderers are user/DB-controlled,
* so they must never be parsed as HTML by the browser — escaping turns any
* embedded markup into inert text.
*/
const escapeHtml = (value: any): string =>
String(value ?? '').replace(/[&<>"']/g, (char) => {
const entities: Record<string, string> = {
'&': '&amp;',
'<': '&lt;',
'>': '&gt;',
'"': '&quot;',
"'": '&#39;'
}
return entities[char]
})
/**
* Builds a display-only HOT renderer that formats numeric cell values using
* Intl.NumberFormat. The stored/submitted value is never changed — only the
@@ -67,9 +85,7 @@ export const errorRenderer = (
) => {
addDarkClass(td)
td.innerHTML = `${
value ? value.toString() : ''
} <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
td.innerHTML = `${escapeHtml(value)} <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
return td
}
@@ -89,7 +105,7 @@ export const noSpinnerRenderer = (
) => {
addDarkClass(td)
td.innerHTML = value ? value : ''
td.innerHTML = escapeHtml(value)
return td
}
@@ -110,9 +126,7 @@ export const spinnerRenderer = (
) => {
addDarkClass(td)
td.innerHTML = `${
value ? value.toString() : ''
} <span class="spinner spinner-sm vertical-align-middle"></span>`
td.innerHTML = `${escapeHtml(value)} <span class="spinner spinner-sm vertical-align-middle"></span>`
return td
}
+3
View File
@@ -160,6 +160,9 @@ export class AppService {
SYSHOSTINFOLONG: res.adapterResponse.SYSHOSTINFOLONG,
SYSENCODING: res.adapterResponse.SYSENCODING,
AUTOEXEC: res.adapterResponse.AUTOEXEC,
TIMEZONE: res.adapterResponse.globvars[0].TIMEZONE,
SYSTIMEZONEIDENT: res.adapterResponse.globvars[0].SYSTIMEZONEIDENT,
SYSTIMEZONEOFFSET: res.adapterResponse.globvars[0].SYSTIMEZONEOFFSET,
ISADMIN: res.adapterResponse.globvars[0].ISADMIN,
DC_ADMIN_GROUP: res.adapterResponse.globvars[0].DC_ADMIN_GROUP,
APP_LOC: dcAdapterSettings?.appLoc
@@ -138,13 +138,38 @@ export class VaMessagingService {
this.earlyDrained = true
const captured = (window as unknown as { __vaLastMessage?: any })
.__vaLastMessage
const parsed = this.parseData(captured && captured.data)
// The early listener (va-early.js) captures from any origin, so re-apply
// the live-path trust rule here before acting on it: only replay a message
// that came from our own origin or from the frame that embedded us (whose
// URL is document.referrer). Without this, a same-origin sibling frame
// could inject a crafted DDC message that the live isTrustedSource check
// would have rejected.
if (!captured || typeof captured.origin !== 'string') return
if (!this.isTrustedEarlyOrigin(captured.origin)) return
const parsed = this.parseData(captured.data)
if (!parsed) return
this.resultName = parsed.resultName
if (captured.origin) this.parentOrigin = captured.origin
this.parentOrigin = captured.origin
callback(parsed)
}
/**
* Origin check for the pre-bootstrap replay. Mirrors isTrustedSource: the
* live path trusts a message whose event.source IS the parent frame; for a
* captured message we cannot reference its source Window, so we trust an
* origin that is this window's origin or the embedding frame's origin
* (document.referrer). Unverifiable/absent referrer -> reject.
*/
private isTrustedEarlyOrigin(origin: string): boolean {
if (origin === window.location.origin) return true
if (!document.referrer) return false
try {
return new URL(document.referrer).origin === origin
} catch {
return false
}
}
/**
* Parses a raw window MessageEvent into a VaMessage, or null when it is not a
* recognisable DDC message (e.g. unrelated postMessage traffic).
@@ -149,3 +149,75 @@ describe('buildColInfoHtml', () => {
)
})
})
/**
* DOM-injection reproduction for the column-info dropdown.
* buildColInfoHtml interpolates server/DB-controlled values (column label,
* format, and DQ RULE_VALUE regex/formula strings) into a string that the
* viewer/editor assign to raw DOM `elem.innerHTML` - so a value containing
* markup (e.g. a HARDREGEX RULE_VALUE of `<img src=x onerror=alert(1)>`) is
* parsed and executed in the browser of whoever opens the info dropdown.
* These tests fail on the vulnerable implementation and pass once each field
* is escaped.
*/
describe('buildColInfoHtml escapes rather than injecting raw HTML', () => {
const malicious = '<img src=x onerror=alert(1)>'
const info: DataFormat = {
format: malicious,
label: malicious,
length: '8',
type: 'N'
}
// Parse the returned string the same way the callers do (innerHTML on a
// real element) and assert no scriptable element survived.
const parseInto = (html: string): HTMLElement => {
const host = document.createElement('div')
host.innerHTML = html
return host
}
const assertNoInjectedElement = (html: string) => {
const host = parseInto(html)
expect(host.querySelector('img[onerror]')).toBeNull()
host.remove()
}
it('is inert for a colInfo whose label and format carry markup', () => {
assertNoInjectedElement(buildColInfoHtml('SOMECHAR', info))
})
it('escapes the column NAME', () => {
const html = buildColInfoHtml(malicious, {
format: '$8.',
label: 'safe',
length: '8',
type: 'C'
})
// < and > must not survive as markup in the NAME position
expect(html).not.toContain(malicious)
assertNoInjectedElement(html)
})
it('escapes a HARDREGEX RULE_VALUE', () => {
assertNoInjectedElement(
buildColInfoHtml('SOMECHAR', info, malicious, undefined, undefined)
)
})
it('escapes a SOFTREGEX RULE_VALUE', () => {
assertNoInjectedElement(
buildColInfoHtml('SOMECHAR', info, undefined, malicious, undefined)
)
})
it('escapes a formula RULE_VALUE (with and without a leading =)', () => {
assertNoInjectedElement(
buildColInfoHtml('SOMECHAR', info, undefined, undefined, malicious)
)
assertNoInjectedElement(
buildColInfoHtml('SOMECHAR', info, undefined, undefined, `=${malicious}`)
)
})
})
+30 -4
View File
@@ -1,9 +1,35 @@
import { DataFormat } from '../../models/sas/common/DateFormat'
/**
* Returns string-safe text of any value so it can be concatenated into a
* string that is later assigned to raw DOM innerHTML. The column metadata
* (label/format) and DQ RULE_VALUE strings (regex/formula) are DB-controlled -
* a validation-rule author can store markup such as
* `<img src=x onerror=...>` in a HARDREGEX value or a column label - so they
* must never be parsed as HTML by the browser. Escaping turns any embedded
* markup into inert text.
*/
const escapeHtml = (value: any): string =>
String(value ?? '').replace(/[&<>"']/g, (char) => {
const entities: Record<string, string> = {
'&': '&amp;',
'<': '&lt;',
'>': '&gt;',
'"': '&quot;',
"'": '&#39;'
}
return entities[char]
})
/**
* Builds the HTML shown in a column-header "info" dropdown item (viewer and
* editor). NAME is listed first so it's visible regardless of whether
* headers are currently displayed as NAME or LABEL.
*
* The returned string is assigned to raw DOM `elem.innerHTML` by both callers
* (viewer.component.ts / editor.component.ts) - every field interpolated below
* is therefore escaped via escapeHtml, since no Angular sanitizer runs on a
* raw innerHTML assignment.
*/
export function buildColInfoHtml(
colName: string,
@@ -14,16 +40,16 @@ export function buildColInfoHtml(
): string {
if (!colInfo) return 'No info found'
let html = `NAME: ${colName}<br>LABEL: ${colInfo.label}<br>TYPE: ${colInfo.type}<br>LENGTH: ${colInfo.length}<br>FORMAT: ${colInfo.format}`
let html = `NAME: ${escapeHtml(colName)}<br>LABEL: ${escapeHtml(colInfo.label)}<br>TYPE: ${escapeHtml(colInfo.type)}<br>LENGTH: ${escapeHtml(colInfo.length)}<br>FORMAT: ${escapeHtml(colInfo.format)}`
// Only ever one REGEX rule is applied per column: when both HARDREGEX
// and SOFTREGEX exist, SOFTREGEX is ignored entirely (same precedence as
// makeRegexWarningRenderer / DcValidator.failsSoftRegex). Show only the
// rule that is applied.
if (hardRegexValue) {
html += `<br>HARDREGEX: ${hardRegexValue}`
html += `<br>HARDREGEX: ${escapeHtml(hardRegexValue)}`
} else if (softRegexValue) {
html += `<br>SOFTREGEX: ${softRegexValue}`
html += `<br>SOFTREGEX: ${escapeHtml(softRegexValue)}`
}
// '√x=' stands in for a text label here - HARDFORMULA vs SOFTFORMULA is
@@ -36,7 +62,7 @@ export function buildColInfoHtml(
const formula = formulaValue.startsWith('=')
? formulaValue.slice(1)
: formulaValue
html += `<br>√x=${formula}`
html += `<br>√x=${escapeHtml(formula)}`
}
return html
@@ -10,6 +10,9 @@ export interface EnvironmentInfo {
SYSHOSTINFOLONG?: string
SYSENCODING?: string
AUTOEXEC?: string
TIMEZONE?: string
SYSTIMEZONEIDENT?: string
SYSTIMEZONEOFFSET?: string
ISADMIN?: number
DC_ADMIN_GROUP?: string
APP_LOC?: string
@@ -51,6 +51,24 @@
<p cds-text="label" class="m-0">
AUTOEXEC: <span class="dark">{{ environmentInfo?.AUTOEXEC }}</span>
</p>
<p cds-text="label" class="m-0">
TIMEZONE:
<span class="dark">{{
environmentInfo?.TIMEZONE || '(not set)'
}}</span>
</p>
<p cds-text="label" class="m-0">
SYSTIMEZONEIDENT:
<span class="dark">{{
environmentInfo?.SYSTIMEZONEIDENT || '(not set)'
}}</span>
</p>
<p cds-text="label" class="m-0">
SYSTIMEZONEOFFSET:
<span class="dark">{{
environmentInfo?.SYSTIMEZONEOFFSET || '0'
}}</span>
</p>
</div>
<div class="d-flex clr-justify-content-lg-center">
+153 -2
View File
@@ -1,14 +1,15 @@
{
"name": "dcfrontend",
"version": "7.12.0",
"version": "7.14.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "dcfrontend",
"version": "7.12.0",
"version": "7.14.0",
"hasInstallScript": true,
"devDependencies": {
"@nogoo9/gitleaks": "8.30.1-post.2",
"@saithodev/semantic-release-gitea": "^2.1.0",
"@semantic-release/changelog": "^6.0.3",
"@semantic-release/commit-analyzer": "13.0.1",
@@ -119,6 +120,156 @@
"node": ">= 8"
}
},
"node_modules/@nogoo9/gitleaks": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks/-/gitleaks-8.30.1-post.2.tgz",
"integrity": "sha512-WkQPfryKH8w9DEqJ1Bxw3ex9aXl0M8cxDDt1ATvyieaMvWiGamGXHNLZaJeqECInVfxNPEiZA4Few8Uwtuj1wQ==",
"dev": true,
"license": "MIT",
"bin": {
"gitleaks": "src/index.js"
},
"engines": {
"node": ">=18"
},
"optionalDependencies": {
"@nogoo9/gitleaks-darwin-arm64": "8.30.1-post.2",
"@nogoo9/gitleaks-darwin-x64": "8.30.1-post.2",
"@nogoo9/gitleaks-linux-arm": "8.30.1-post.2",
"@nogoo9/gitleaks-linux-arm64": "8.30.1-post.2",
"@nogoo9/gitleaks-linux-x32": "8.30.1-post.2",
"@nogoo9/gitleaks-linux-x64": "8.30.1-post.2",
"@nogoo9/gitleaks-windows-arm64": "8.30.1-post.2",
"@nogoo9/gitleaks-windows-x32": "8.30.1-post.2",
"@nogoo9/gitleaks-windows-x64": "8.30.1-post.2"
}
},
"node_modules/@nogoo9/gitleaks-darwin-arm64": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-darwin-arm64/-/gitleaks-darwin-arm64-8.30.1-post.2.tgz",
"integrity": "sha512-3j1zSvCKY7N4RnSl+oJrnrLZAY6HihD3uyi32qYrdIXH5hgVjmKiJhvGyoaV7DNw/4NdN3UsadrTM1u5M7iCuw==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@nogoo9/gitleaks-darwin-x64": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-darwin-x64/-/gitleaks-darwin-x64-8.30.1-post.2.tgz",
"integrity": "sha512-EbWRmrWdwRtzfO4ivYXA2ZKcf4aqVZFGQVu2HJjCFjRNryyMfou4Hw3tdZc5tRbdSE0yiwaAXJ8lPazApd3iGQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@nogoo9/gitleaks-linux-arm": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-arm/-/gitleaks-linux-arm-8.30.1-post.2.tgz",
"integrity": "sha512-a5aZc5WQnBmNz5CCOPd0MT75csSBUEYMEVIHR6p56TOt4lj8h+IeQ5VTKG+f9j9nU+hmG2e//j4hLcbZZkCspw==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@nogoo9/gitleaks-linux-arm64": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-arm64/-/gitleaks-linux-arm64-8.30.1-post.2.tgz",
"integrity": "sha512-XJzvhOEsqr6nJrBuLzDb+qBfdLjssKv/7gqQNhojd8LPM0/aEiC380kYuM9mFCXF3o3PpD4IXv/Jvemw5JJQGA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@nogoo9/gitleaks-linux-x32": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-x32/-/gitleaks-linux-x32-8.30.1-post.2.tgz",
"integrity": "sha512-d20jRC6HCzc9J79Ti8JHFc3X3iJktF78NaUEg0EPGLHRsVpxVObtiCQAJzCgH0uXNMOVFrnsf/+dSDFwjTLU9Q==",
"cpu": [
"ia32"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@nogoo9/gitleaks-linux-x64": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-x64/-/gitleaks-linux-x64-8.30.1-post.2.tgz",
"integrity": "sha512-GSJ7RU0xuaF9Blf2AmDeytxyiZb4W7/Urd5Cw5LlNK+IF/C+Nd/Fp5pKbaAF4osdxADbCZ7d4XMYUMSdlpjQFA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@nogoo9/gitleaks-windows-arm64": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-arm64/-/gitleaks-windows-arm64-8.30.1-post.2.tgz",
"integrity": "sha512-cTu+rMASv3BuNJcPj57PhfiOHwapmOVH6s5PFBqAXzJomUnHdB/ZMrTDNseW78UZQQ/c0RPI3VXwieyMC23w7w==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@nogoo9/gitleaks-windows-x32": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-x32/-/gitleaks-windows-x32-8.30.1-post.2.tgz",
"integrity": "sha512-MGMtKVEOf0TsXIGBu/1eGb94mAZRX2rzZmgNRYqzZBSYPnmyl3w1n4tic9e6rPOewyGfuxnweCDnu0Lu/qkaaw==",
"cpu": [
"ia32"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@nogoo9/gitleaks-windows-x64": {
"version": "8.30.1-post.2",
"resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-x64/-/gitleaks-windows-x64-8.30.1-post.2.tgz",
"integrity": "sha512-whDUUm3+J5YRQUJS0mcE72JaPb2N0Njmjaj6NYTcG1OJS3v42Y/JYen6R9FuQ5jUCfsVGcblmSBmDb9uZVdwrw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@pnpm/config.env-replace": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@pnpm/config.env-replace/-/config.env-replace-1.1.0.tgz",
+2 -1
View File
@@ -1,8 +1,9 @@
{
"name": "dcfrontend",
"version": "7.13.0",
"version": "7.14.1",
"description": "Data Controller",
"devDependencies": {
"@nogoo9/gitleaks": "8.30.1-post.2",
"@saithodev/semantic-release-gitea": "^2.1.0",
"@semantic-release/changelog": "^6.0.3",
"@semantic-release/commit-analyzer": "13.0.1",
@@ -79,7 +79,10 @@ _webout=`{"SYSDATE" : "26SEP22"
"DC_ADMIN_GROUP": "Data Management Business Approvers",
"LICENCE_KEY": "",
"ACTIVATION_KEY": "",
"DC_RESTRICT_EDITRECORD": "NO"
"DC_RESTRICT_EDITRECORD": "NO",
"TIMEZONE": "Europe/London",
"SYSTIMEZONEIDENT": "Europe/London",
"SYSTIMEZONEOFFSET": "+0100"
}
]
,"xlmaps": [
@@ -277,9 +277,62 @@ const tables = {
{ tx_from: 0, tx_to: 127490111999, xlmap_id: 'SAMPLE', xlmap_range_id: 'data', xlmap_sheet: '/1', xlmap_start: 'ABSOLUTE B13', xlmap_finish: 'ABSOLUTE E16' }
],
MPE_X_CATALOG: [],
MPE_X_SEARCH: [],
MPE_USERS: []
}
// ─── MPE_X_SEARCH: viewer full-table-search demo table ───────────────────────
// Deterministic, obviously fictional deep-sea survey data (no real sites,
// vessels or people). Laid out site-major / vessel-minor so every count in
// the full-table-search Cypress spec can be derived by hand:
// n = rep * 50 + siteIdx * 5 + vesselIdx (0-based, 1000 rows)
// -> 20 rows per site, 100 rows per site block, 200 rows per vessel
const searchSites = [
'Nereid Trench', 'Calypso Trench', 'Halcyon Trench', 'Amphitrite Basin',
'Kraken Hollow', 'Erebus Deep', 'Thalassa Shelf', 'Oceanus Rise',
'Selkie Bank', 'Vostok Deep'
]
const searchVessels = ['RV Pelagia', 'RV Aurora', 'RV Selkie', 'RV Cormorant', 'RV Halcyon']
const searchSpecies = [
'Vampire squid', 'Barreleye fish', 'Dumbo octopus', 'Yeti crab',
'Giant isopod', 'Ghost shark', 'Comb jelly', 'Sea pig',
'Glass sponge', 'Anglerfish'
]
// Rows carrying the mid-field marker in NOTES (searched as a partial string)
const siphonophoreRows = [137, 500, 862]
// Rows sharing the repeated exact SAMPLE_COUNT (searched as a number)
const sampleCountRows = [220, 780]
tables.MPE_X_SEARCH = []
for (let n = 0; n < 1000; n++) {
const rep = Math.floor(n / 50)
const siteIdx = Math.floor((n % 50) / 5)
const vesselIdx = n % 5
const site = searchSites[siteIdx]
const depth = 1000 + (n % 90) * 37
tables.MPE_X_SEARCH.push({
PRIMARY_KEY_FIELD: n + 1001,
SITE_NAME: site,
VESSEL: searchVessels[vesselIdx],
SAMPLE_COUNT: sampleCountRows.includes(n) ? 4210 : 1 + (n % 6),
DEPTH_M: depth,
SPECIES: searchSpecies[(siteIdx + vesselIdx) % searchSpecies.length],
CRUISE_DATE: 20000 + (n % 3650),
EXPEDITION_ID: 'EXP-' + String(n + 1).padStart(4, '0'),
NOTES: 'Leg ' + (rep + 1) + ' of the ' + site + ' survey. ' +
(siphonophoreRows.includes(n)
? 'A bioluminescent siphonophore was filmed here.'
: 'No anomalies recorded.') +
' Depth soundings logged at ' + depth + 'm.'
})
}
// Register the demo table like any other editable table (buskey drives the
// viewer's PK detection).
tables.MPE_TABLES.push(
{ tx_from: 0, tx_to: 127490111999, libref: dcLibref, dsn: 'MPE_X_SEARCH', num_of_approvals_required: 1, loadtype: 'UPDATE', buskey: 'PRIMARY_KEY_FIELD', notes: 'Full table search demo table' }
)
// Column metadata for each table, extracted from mpe_makedatamodel.sas
// Each entry: { name, type (N/C), length, format, label, notnull (bool) }
const schema = {
@@ -664,6 +717,17 @@ const schema = {
{ name: 'SOME_SHORTNUM', type: 'N', length: 4, format: '', label: '', notnull: false },
{ name: 'SOME_BESTNUM', type: 'N', length: 8, format: 'best.', label: '', notnull: false }
],
MPE_X_SEARCH: [
{ name: 'PRIMARY_KEY_FIELD', type: 'N', length: 8, format: '', label: 'Primary Key', notnull: true },
{ name: 'SITE_NAME', type: 'C', length: 32, format: '', label: 'Survey Site', notnull: false },
{ name: 'VESSEL', type: 'C', length: 32, format: '', label: 'Survey Vessel', notnull: false },
{ name: 'SAMPLE_COUNT', type: 'N', length: 8, format: '', label: 'Samples Collected', notnull: false },
{ name: 'DEPTH_M', type: 'N', length: 8, format: '', label: 'Depth (metres)', notnull: false },
{ name: 'SPECIES', type: 'C', length: 32, format: '', label: 'Species Observed', notnull: false },
{ name: 'CRUISE_DATE', type: 'N', length: 8, format: 'date9.', label: 'Cruise Date', notnull: false },
{ name: 'EXPEDITION_ID', type: 'C', length: 12, format: '', label: 'Expedition', notnull: false },
{ name: 'NOTES', type: 'C', length: 512, format: '', label: 'Survey Notes', notnull: false }
],
MPE_XLMAP_DATA: [
{ name: 'LOAD_REF', type: 'C', length: 32, format: '', label: '', notnull: true },
{ name: 'XLMAP_ID', type: 'C', length: 32, format: '', label: '', notnull: true },
@@ -120,7 +120,10 @@ const globvars = [
DC_ADMIN_GROUP: adminGroup,
LICENCE_KEY: licenceKey,
ACTIVATION_KEY: activationKey,
DC_RESTRICT_EDITRECORD: 'NO'
DC_RESTRICT_EDITRECORD: 'NO',
TIMEZONE: 'Europe/London',
SYSTIMEZONEIDENT: 'Europe/London',
SYSTIMEZONEOFFSET: '+0100'
}
]
+84 -11
View File
@@ -4,10 +4,13 @@ let appLoc = nodePath.join(..._program.split('services')[0].split('/'))
const sasjsRoot = nodePath.resolve(weboutPath, '..', '..', '..')
const driveRoot = nodePath.resolve(sasjsRoot, 'drive')
const dcLibref = 'DC_JSLIB'
const dataDir = nodePath.resolve(driveRoot, 'files', appLoc, 'data', dcLibref)
const mpeDataDir = nodePath.resolve(driveRoot, 'files', appLoc, 'data', dcLibref)
eval(fs.readFileSync(nodePath.resolve(driveRoot, 'files', appLoc, 'services', 'dcMockUtils.js'), 'utf8'))
const loadTableData = makeTableLoader(dataDir)
// Row cap for the viewer - mirrors DC_MAXOBS_WEBVIEW in viewdata.sas, and the
// MAXROWS value reported in sasparams.
const maxObsWebView = 500
// ─── Parse input ──────────────────────────────────────────────────────────────
@@ -16,11 +19,65 @@ let libds = _sctRow.LIBDS || ''
const libref = libds.split('.')[0] || dcLibref
const dsn = libds.split('.')[1] || ''
// FILTER_RK (matching viewdata.sas: if filter_rk le 0 then filter_rk=-1)
const filterRk = Number(colVal(_sctRow, 'FILTER_RK')) > 0 ? Number(colVal(_sctRow, 'FILTER_RK')) : -1
// SEARCHTYPE / SEARCHVAL - only CHAR or NUM triggers a search, anything else
// (blank, NONE) is a normal view. Search values are stripped of the same
// characters viewdata.sas removes before handing them to mp_searchdata.
const rawSearchType = String(colVal(_sctRow, 'SEARCHTYPE') || '').toUpperCase()
const searchType = (rawSearchType === 'CHAR' || rawSearchType === 'NUM') ? rawSearchType : ''
const rawSearchVal = colVal(_sctRow, 'SEARCHVAL')
let searchVal = (rawSearchVal === undefined || rawSearchVal === null) ? '' : String(rawSearchVal)
if (searchType) searchVal = searchVal.replace(/[%&;"]/g, '')
// ─── Load the target table ───────────────────────────────────────────────────
const dataDir = libDataDir(libref)
const loadTableData = makeTableLoader(dataDir)
const loadMpeTable = makeTableLoader(mpeDataDir)
let tableData = loadTableData(dsn)
if (!tableData && dsn) {
const found = loadTableAnyLib(dsn)
if (found) tableData = found.data
}
// ─── Apply the stored filter (mpe_filtermaster, VIEW mode) ───────────────────
// Mirrors viewdata.sas: %mpe_filtermaster(VIEW,&libds,filter_rk=&filter_rk)
// Row Level Security and any saved filter are applied here, before the search.
const { predicate: filterPredicate, filterText } = mpeFilterMaster({
mode: 'VIEW',
libds: (libref + '.' + dsn).toUpperCase(),
filterRk: filterRk,
dataDir: mpeDataDir,
columns: tableData && tableData.columns ? tableData.columns : []
})
// ─── Search predicate (mp_searchdata.sas) ────────────────────────────────────
// mp_searchdata builds one WHERE clause covering every column of the table:
// or ("COL1"n ? "the string") /* character columns: case sensitive CONTAINS */
// or ("COL2"n = the number) /* numeric columns: exact equality */
function rowMatchesSearch(row, columns, type, value) {
for (const col of columns) {
const val = colVal(row, col.name)
if (val === undefined || val === null) continue
if (type === 'CHAR') {
if (col.type !== 'N' && String(val).includes(value)) return true
} else {
if (col.type === 'N' && Number(val) === Number(value)) return true
}
}
return false
}
// ─── Build response ──────────────────────────────────────────────────────────
// Look up PK fields from MPE_TABLES (mirrors mp_getpk).
// MPE_AUDIT is a standard DC audit table whose PK is always the first 5 columns.
const mpeTables = loadTableData('MPE_TABLES')
const mpeTables = loadMpeTable('MPE_TABLES')
let pkFields = ' '
if (mpeTables && mpeTables.rows) {
const reg = mpeTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -32,8 +89,6 @@ if (dsn.toUpperCase() === 'MPE_AUDIT' && pkFields.trim() === '') {
pkFields = 'LOAD_REF LIBREF DSN KEY_HASH TGTVAR_NM'
}
const tableData = loadTableData(dsn)
let cols = []
let viewdata = []
let vars = {}
@@ -53,8 +108,26 @@ if (tableData && tableData.columns) {
}
})
// Rows in scope: the stored filter is applied first, then the search
// (matching viewdata.sas, which searches a filtered view of the table).
const filteredRows = (tableData.rows || []).filter(r => filterPredicate(r))
let rows
if (searchType) {
const matches = filteredRows.filter(row => rowMatchesSearch(row, tableData.columns, searchType, searchVal))
// mp_searchdata caps its output at outobs=&DC_MAXOBS_WEBVIEW, and
// viewdata.sas then keeps only `if _n_<&DC_MAXOBS_WEBVIEW` of them.
nobs = Math.min(matches.length, maxObsWebView)
rows = matches.slice(0, maxObsWebView - 1)
} else {
// proc sql select count(*) ... where <filter> (uncapped)
nobs = filteredRows.length
// data work.viewdata; set &libds; where <filter>; if _n_>&DC_MAXOBS_WEBVIEW then stop;
rows = filteredRows.slice(0, maxObsWebView)
}
// viewdata rows: all values as strings (formatted, SAS proc-print style)
viewdata = (tableData.rows || []).map(r => {
viewdata = rows.map(r => {
const row = {}
for (const col of tableData.columns) {
const lcName = col.name.toLowerCase()
@@ -94,7 +167,9 @@ if (tableData && tableData.columns) {
// the client can derive the column set even when empty (the UI iterates
// viewdata[0] to build the grid headers). Numeric columns get "."
// (SAS missing), character columns get "".
if (viewdata.length === 0 && tableData.columns && tableData.columns.length > 0) {
// A search that matches nothing sends no rows at all (the client shows
// its "no data found" panel instead of an empty grid).
if (viewdata.length === 0 && !searchType && tableData.columns.length > 0) {
const emptyRow = {}
for (const col of tableData.columns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
viewdata = [emptyRow]
@@ -105,8 +180,6 @@ if (tableData && tableData.columns) {
for (const col of tableData.columns) {
vars[col.name] = sasVarsEntry(col)
}
nobs = tableData.rows ? tableData.rows.length : 0
}
webOutOpen()
@@ -118,8 +191,8 @@ webOutObj([{ ODS_TABLE: 'ATTRIBUTES', NAME: 'Data Set Name', VALUE: libref + '.'
{ ODS_TABLE: 'ATTRIBUTES', NAME: 'Variables', VALUE: String(cols.length) },
{ ODS_TABLE: 'ATTRIBUTES', NAME: 'Engine', VALUE: 'V9' }], 'dsmeta')
webOutObj([], 'query')
webOutObj([{ TABLEURI: ' ', TABLENAME: dsn, FILTER_TEXT: ' ',
PK_FIELDS: pkFields, NOBS: nobs, VARS: cols.length, MAXROWS: 500 }], 'sasparams')
webOutObj([{ TABLEURI: ' ', TABLENAME: dsn, FILTER_TEXT: filterText === '' ? ' ' : filterText,
PK_FIELDS: pkFields, NOBS: nobs, VARS: cols.length, MAXROWS: maxObsWebView }], 'sasparams')
webOutObj([], 'versions')
webOutObj(viewdata, 'viewdata', { vars })
webOutClose()
+149 -168
View File
@@ -6,8 +6,8 @@
"": {
"name": "dc-sas",
"dependencies": {
"@sasjs/cli": "4.20.1",
"@sasjs/core": "5.2.0"
"@sasjs/cli": "4.20.4",
"@sasjs/core": "5.2.8"
}
},
"node_modules/@asamuzakjp/css-color": {
@@ -23,17 +23,6 @@
"lru-cache": "^10.4.3"
}
},
"node_modules/@asamuzakjp/dom-selector": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-2.0.2.tgz",
"integrity": "sha512-x1KXOatwofR6ZAYzXRBL5wrdV0vwNxlTCK9NCuLqAzQYARqGcvFwiJA6A1ERuh+dgeA4Dxm3JBYictIes+SqUQ==",
"license": "MIT",
"dependencies": {
"bidi-js": "^1.0.3",
"css-tree": "^2.3.1",
"is-potential-custom-element-name": "^1.0.1"
}
},
"node_modules/@coolaj86/urequest": {
"version": "1.3.7",
"resolved": "https://registry.npmjs.org/@coolaj86/urequest/-/urequest-1.3.7.tgz",
@@ -214,94 +203,59 @@
}
},
"node_modules/@sasjs/cli": {
"version": "4.20.1",
"resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.1.tgz",
"integrity": "sha512-482xPlEuyBKqGlg2ArCiREu3Q9+8TsTj0dwy5ewNURi7pWEcwJH0A4c9Yw4a3akUtPAphPEzs8/9BkcS/v+3WQ==",
"version": "4.20.4",
"resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.4.tgz",
"integrity": "sha512-hKE002Dm9AzWU62qpRyYpjQv8BIXG951Z0ToNdG9J1wvvJNYH7h6rGv7kdQMbgvXff/KOQvVdnGJZ023SrG+Jw==",
"license": "ISC",
"dependencies": {
"@sasjs/adapter": "^4.19.0",
"@sasjs/core": "4.68.1",
"@sasjs/lint": "2.4.3",
"@sasjs/utils": "^3.6.0",
"adm-zip": "0.6.0",
"@sasjs/core": "5.2.9",
"@sasjs/lint": "2.5.0",
"@sasjs/utils": "3.6.2",
"chalk": "4.1.2",
"dotenv": "16.0.3",
"dotenv": "17.4.2",
"find": "0.3.0",
"js-base64": "3.7.5",
"jsdom": "23.2.0",
"js-base64": "3.9.3",
"jsdom": "26.0.0",
"jwt-decode": "3.1.2",
"lodash.groupby": "4.6.0",
"lodash.uniqby": "4.7.0",
"node-graphviz": "0.1.1",
"node-powershell": "5.0.1",
"ora": "5.4.1",
"prompts": "2.4.1",
"prompts": "2.4.2",
"shelljs": "0.10.0",
"ssl-root-cas": "1.3.1",
"xml": "1.0.1",
"yargs": "17.6.2"
"yargs": "17.6.2",
"yauzl": "3.4.0",
"yazl": "3.3.1"
},
"bin": {
"sasjs": "build/index.js"
}
},
"node_modules/@sasjs/cli/node_modules/@sasjs/core": {
"version": "4.68.1",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-4.68.1.tgz",
"integrity": "sha512-BjoOzV7h7guCShW4JsFHvB1NNLCKHhPI5708OtEk29itQbgVCMRlEXKFRbnMiuNklktDQDi8dR4XZZ0aK7/4Wg==",
"version": "5.2.9",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.9.tgz",
"integrity": "sha512-R5wtJF0ANHchaURqTF2t1vRXrhUJ+uDfj21ewzdOq4Lp14CWW22+yYeRQooY4QOpyHUyFDjReA5q6SoJt2Tz1g==",
"license": "MIT"
},
"node_modules/@sasjs/core": {
"version": "5.2.0",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.0.tgz",
"integrity": "sha512-YJAVckql9o2lbldo4xFbu/Vkja0DfSw8w8hrnhnCov8iqqzkM84rHnqqKzdD0+sa52+AoPoHAxt4nCBoQfH9wA==",
"version": "5.2.8",
"resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.8.tgz",
"integrity": "sha512-+K8tXTrsw52NR1y5MFHW0+Jf9euL+GukwqA79FIHHu4Oj+BhHBm1uZiyB5tPrvwl3V8mY1CQhMOT83//bV+jWg==",
"license": "MIT"
},
"node_modules/@sasjs/lint": {
"version": "2.4.3",
"resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-2.4.3.tgz",
"integrity": "sha512-J0fYJmDfnhXDeMkZI8vHo8zjIsByIPcKHSQbiQEuRMXbNJXjqgTqoCpN0L6SlhyrrXUtMQvk/XErJ3OQGVEjrA==",
"version": "2.5.0",
"resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-2.5.0.tgz",
"integrity": "sha512-bVUtYnKhCigXylCCvQ6dsAIywpPKkX3BXatMqOWjDVw9R30jfj4M3/qn05fcR45L42Y7e5Tuzlt+LYs+9k95Yg==",
"hasInstallScript": true,
"license": "MIT",
"dependencies": {
"@sasjs/utils": "3.5.2",
"ignore": "5.2.4"
}
},
"node_modules/@sasjs/lint/node_modules/@sasjs/utils": {
"version": "3.5.2",
"resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-3.5.2.tgz",
"integrity": "sha512-LBpBDx0T7G/eO15Gb+r3DR1LfBnoqagWT3HiHabojFziA4ej4ePORo8chrk0zHLIzrjI2ljAnDabyJEwC5KtIA==",
"hasInstallScript": true,
"license": "ISC",
"dependencies": {
"@fast-csv/format": "4.3.5",
"@types/fs-extra": "11.0.4",
"@types/prompts": "2.0.13",
"chalk": "4.1.1",
"cli-table": "0.3.6",
"consola": "2.15.0",
"find": "0.3.0",
"fs-extra": "11.3.0",
"jwt-decode": "3.1.2",
"prompts": "2.4.1",
"valid-url": "1.0.9"
}
},
"node_modules/@sasjs/lint/node_modules/chalk": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
"integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
"license": "MIT",
"dependencies": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/chalk?sponsor=1"
"@sasjs/utils": "3.6.2",
"ignore": "7.0.8"
}
},
"node_modules/@sasjs/utils": {
@@ -339,6 +293,19 @@
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/@sasjs/utils/node_modules/prompts": {
"version": "2.4.1",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
"integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
"license": "MIT",
"dependencies": {
"kleur": "^3.0.3",
"sisteransi": "^1.0.5"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/@types/fs-extra": {
"version": "11.0.4",
"resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-11.0.4.tgz",
@@ -383,15 +350,6 @@
"ms": "^2.1.3"
}
},
"node_modules/adm-zip": {
"version": "0.6.0",
"resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.0.tgz",
"integrity": "sha512-XleryMhbuksdKtofnWZ9Sk+4CUTbms4Mb/EU32SZwToAyZ5RgVos/ki8n+yr0LWHOGKuakbXTuuYNHLQjhddgg==",
"license": "MIT",
"engines": {
"node": ">=14.0"
}
},
"node_modules/agent-base": {
"version": "7.1.4",
"resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz",
@@ -482,15 +440,6 @@
],
"license": "MIT"
},
"node_modules/bidi-js": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz",
"integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==",
"license": "MIT",
"dependencies": {
"require-from-string": "^2.0.2"
}
},
"node_modules/bl": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
@@ -538,6 +487,15 @@
"ieee754": "^1.1.13"
}
},
"node_modules/buffer-crc32": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-1.0.0.tgz",
"integrity": "sha512-Db1SbgBS/fg/392AblrMJk97KggmvYhr4pB5ZIMTWtaivCPMWLkmb7m21cJvpvgK+J3nsU2CmmixNBZx4vFj/w==",
"license": "MIT",
"engines": {
"node": ">=8.0.0"
}
},
"node_modules/bytes": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
@@ -708,19 +666,6 @@
"node": ">= 8"
}
},
"node_modules/css-tree": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/css-tree/-/css-tree-2.3.1.tgz",
"integrity": "sha512-6Fv1DV/TYw//QF5IzQdqsNDjx/wc8TrMBZsqjL9eW01tWb7R7k/mq+/VXfJCl7SoD5emsJop9cOByJZfs8hYIw==",
"license": "MIT",
"dependencies": {
"mdn-data": "2.0.30",
"source-map-js": "^1.0.1"
},
"engines": {
"node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0"
}
},
"node_modules/cssstyle": {
"version": "4.6.0",
"resolved": "https://registry.npmjs.org/cssstyle/-/cssstyle-4.6.0.tgz",
@@ -734,12 +679,6 @@
"node": ">=18"
}
},
"node_modules/cssstyle/node_modules/rrweb-cssom": {
"version": "0.8.0",
"resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
"integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
"license": "MIT"
},
"node_modules/data-urls": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/data-urls/-/data-urls-5.0.0.tgz",
@@ -798,12 +737,15 @@
}
},
"node_modules/dotenv": {
"version": "16.0.3",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.3.tgz",
"integrity": "sha512-7GO6HghkA5fYG9TYnNxi14/7K9f5occMlp3zXAuSxn7CKCxt9xbNWG7yF8hTCSUchlfWSe3uLmlPfigevRItzQ==",
"version": "17.4.2",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
"integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://dotenvx.com"
}
},
"node_modules/dunder-proto": {
@@ -938,9 +880,9 @@
}
},
"node_modules/fastq": {
"version": "1.20.1",
"resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.1.tgz",
"integrity": "sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw==",
"version": "1.20.3",
"resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.3.tgz",
"integrity": "sha512-XKv5nnLs6nLF71NgiKJLIZFLkPyIEuOselLG7ujZnGrRfQK8HpvY+WqKhAJUAdLomwVHErVS4LfxFlPq0/FTAw==",
"license": "ISC",
"dependencies": {
"reusify": "^1.0.4"
@@ -1284,9 +1226,9 @@
"license": "BSD-3-Clause"
},
"node_modules/ignore": {
"version": "5.2.4",
"resolved": "https://registry.npmjs.org/ignore/-/ignore-5.2.4.tgz",
"integrity": "sha512-MAb38BcSbH0eHNBxn7ql2NH/kX33OkB3lZ1BNdh7ENeRChHTYsTvWrMubiIAMNS2llXEEgZ1MUOBtXChP3kaFQ==",
"version": "7.0.8",
"resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.8.tgz",
"integrity": "sha512-YYNsSlXBjMk92SKnkwvB5LOVSa6OznlFUGcsvrFgNJbJCd0M1XKeFVRc8ZByeCqz32FivYNHJVooLmdqrmvp/Q==",
"license": "MIT",
"engines": {
"node": ">= 4"
@@ -1410,44 +1352,44 @@
"license": "ISC"
},
"node_modules/js-base64": {
"version": "3.7.5",
"resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.7.5.tgz",
"integrity": "sha512-3MEt5DTINKqfScXKfJFrRbxkrnk2AxPWGBL/ycjz4dK8iqiSJ06UxD8jh8xuh6p10TX4t2+7FsBYVxxQbMg+qA==",
"version": "3.9.3",
"resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.3.tgz",
"integrity": "sha512-uwYQp+VJ38FVvtim6qNbit6e9uT6dwWQ4Y1+H9TxhW5hcHjpHwoxlR0nMpqUmIFOmu4VqMxwdJA88gIVuZJQ/g==",
"license": "BSD-3-Clause"
},
"node_modules/jsdom": {
"version": "23.2.0",
"resolved": "https://registry.npmjs.org/jsdom/-/jsdom-23.2.0.tgz",
"integrity": "sha512-L88oL7D/8ufIES+Zjz7v0aes+oBMh2Xnh3ygWvL0OaICOomKEPKuPnIfBJekiXr+BHbbMjrWn/xqrDQuxFTeyA==",
"version": "26.0.0",
"resolved": "https://registry.npmjs.org/jsdom/-/jsdom-26.0.0.tgz",
"integrity": "sha512-BZYDGVAIriBWTpIxYzrXjv3E/4u8+/pSG5bQdIYCbNCGOvsPkDQfTVLAIXAf9ETdCpduCVTkDe2NNZ8NIwUVzw==",
"license": "MIT",
"dependencies": {
"@asamuzakjp/dom-selector": "^2.0.1",
"cssstyle": "^4.0.1",
"cssstyle": "^4.2.1",
"data-urls": "^5.0.0",
"decimal.js": "^10.4.3",
"form-data": "^4.0.0",
"form-data": "^4.0.1",
"html-encoding-sniffer": "^4.0.0",
"http-proxy-agent": "^7.0.0",
"https-proxy-agent": "^7.0.2",
"http-proxy-agent": "^7.0.2",
"https-proxy-agent": "^7.0.6",
"is-potential-custom-element-name": "^1.0.1",
"parse5": "^7.1.2",
"rrweb-cssom": "^0.6.0",
"nwsapi": "^2.2.16",
"parse5": "^7.2.1",
"rrweb-cssom": "^0.8.0",
"saxes": "^6.0.0",
"symbol-tree": "^3.2.4",
"tough-cookie": "^4.1.3",
"tough-cookie": "^5.0.0",
"w3c-xmlserializer": "^5.0.0",
"webidl-conversions": "^7.0.0",
"whatwg-encoding": "^3.1.1",
"whatwg-mimetype": "^4.0.0",
"whatwg-url": "^14.0.0",
"ws": "^8.16.0",
"whatwg-url": "^14.1.0",
"ws": "^8.18.0",
"xml-name-validator": "^5.0.0"
},
"engines": {
"node": ">=18"
},
"peerDependencies": {
"canvas": "^2.11.2"
"canvas": "^3.0.0"
},
"peerDependenciesMeta": {
"canvas": {
@@ -1468,6 +1410,18 @@
"node": ">= 14"
}
},
"node_modules/jsdom/node_modules/tough-cookie": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
"integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
"license": "BSD-3-Clause",
"dependencies": {
"tldts": "^6.1.32"
},
"engines": {
"node": ">=16"
}
},
"node_modules/jsonfile": {
"version": "6.2.1",
"resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.1.tgz",
@@ -1578,12 +1532,6 @@
"node": ">= 0.4"
}
},
"node_modules/mdn-data": {
"version": "2.0.30",
"resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.0.30.tgz",
"integrity": "sha512-GaqWWShW4kv/G9IEucWScBx9G1/vsFZZJUO+tD26M8J8z3Kw5RDQjaoZe03YAClgeS/SWPOcb4nkFBTEi5DUEA==",
"license": "CC0-1.0"
},
"node_modules/merge-stream": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz",
@@ -1694,6 +1642,12 @@
"node": ">=8"
}
},
"node_modules/nwsapi": {
"version": "2.2.27",
"resolved": "https://registry.npmjs.org/nwsapi/-/nwsapi-2.2.27.tgz",
"integrity": "sha512-gQPNF78qebCQ6tvVFBYrvJdBNOrYZm90ZlXgpIFm06p6qHDHq/XC4TnJftN6OMbxVE0UTBAoRgcsDeJBBooITw==",
"license": "MIT"
},
"node_modules/onetime": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/onetime/-/onetime-5.1.2.tgz",
@@ -1799,6 +1753,12 @@
"node": ">=8"
}
},
"node_modules/pend": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz",
"integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==",
"license": "MIT"
},
"node_modules/picomatch": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
@@ -1812,9 +1772,9 @@
}
},
"node_modules/prompts": {
"version": "2.4.1",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
"integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
"integrity": "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q==",
"license": "MIT",
"dependencies": {
"kleur": "^3.0.3",
@@ -1903,15 +1863,6 @@
"node": ">=0.10.0"
}
},
"node_modules/require-from-string": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
"integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/requires-port": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
@@ -1942,9 +1893,9 @@
}
},
"node_modules/rrweb-cssom": {
"version": "0.6.0",
"resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.6.0.tgz",
"integrity": "sha512-APM0Gt1KoXBz0iIkkdB/kfvGOwC4UuJFeG/c+yV7wSc7q96cG/kJ0HiYCnzivD9SB53cLV1MlHFNfOuPaadYSw==",
"version": "0.8.0",
"resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
"integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
"license": "MIT"
},
"node_modules/run-parallel": {
@@ -2054,15 +2005,6 @@
"integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==",
"license": "MIT"
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
"integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
"license": "BSD-3-Clause",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/ssl-root-cas": {
"version": "1.3.1",
"resolved": "https://registry.npmjs.org/ssl-root-cas/-/ssl-root-cas-1.3.1.tgz",
@@ -2134,6 +2076,24 @@
"integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==",
"license": "MIT"
},
"node_modules/tldts": {
"version": "6.1.86",
"resolved": "https://registry.npmjs.org/tldts/-/tldts-6.1.86.tgz",
"integrity": "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ==",
"license": "MIT",
"dependencies": {
"tldts-core": "^6.1.86"
},
"bin": {
"tldts": "bin/cli.js"
}
},
"node_modules/tldts-core": {
"version": "6.1.86",
"resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-6.1.86.tgz",
"integrity": "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA==",
"license": "MIT"
},
"node_modules/to-regex-range": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz",
@@ -2322,9 +2282,9 @@
}
},
"node_modules/ws": {
"version": "8.21.1",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.1.tgz",
"integrity": "sha512-+0NTnW77fFN/DjQi6k/Sq/Yvk4Sgajw7urW8V+asjXnRgDs9gyGkdb7EzgfhA4goXsRIZKE28fzIXBHEzhuiWw==",
"version": "8.21.3",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
"integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==",
"license": "MIT",
"engines": {
"node": ">=10.0.0"
@@ -2398,6 +2358,27 @@
"engines": {
"node": ">=12"
}
},
"node_modules/yauzl": {
"version": "3.4.0",
"resolved": "https://registry.npmjs.org/yauzl/-/yauzl-3.4.0.tgz",
"integrity": "sha512-jIH9yLR9wqr0wOS0TpBvo/g/2UgZH5qePVbjgRliiF0BYvOZyaBknKsF+x9Iht0O6sqgnB93rCICdOZFecJuDw==",
"license": "MIT",
"dependencies": {
"pend": "~1.2.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/yazl": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/yazl/-/yazl-3.3.1.tgz",
"integrity": "sha512-BbETDVWG+VcMUle37k5Fqp//7SDOK2/1+T7X8TD96M3D9G8jK5VLUdQVdVjGi8im7FGkazX7kk5hkU8X4L5Bng==",
"license": "MIT",
"dependencies": {
"buffer-crc32": "^1.0.0"
}
}
}
}
+2 -2
View File
@@ -30,8 +30,8 @@
},
"private": true,
"dependencies": {
"@sasjs/cli": "4.20.1",
"@sasjs/core": "5.2.0"
"@sasjs/cli": "4.20.4",
"@sasjs/core": "5.2.8"
},
"overrides": {
"nanoid": "3.3.18"
+10 -3
View File
@@ -6,6 +6,7 @@
<h4> SAS Macros </h4>
@li mpeinit.sas
@li dc_refreshcatalog.sas
@li mpeterm.sas
@version 9.3
@author 4GL Apps Ltd
@@ -20,7 +21,13 @@
%dc_refreshcatalog(&libref)
data _null_;
file _webout;
put '<h1> Catalog Refresh Complete </h1>';
data sasparams;
length msg $64;
msg='Catalog Refresh Complete';
run;
%webout(OPEN)
%webout(OBJ,sasparams)
%webout(CLOSE)
%mpeterm()
@@ -20,25 +20,20 @@ run;
%mx_execute(&_program,
viyacontext=&defaultcontext,
inputparams=work.params,
outref=webout,
viyaresult=WEBOUT_TXT,
outlib=web1,
mdebug=&sasjs_mdebug
)
data work.results;
infile webout;
input;
putlog _infile_;
if index(upcase(_infile_),'CATALOG REFRESH COMPLETE') then do;
putlog 'test passed';
output;
stop;
end;
%let msgcheck=0;
data _null_;
set web1.sasparams;
putlog (_all_)(=);
if index(msg,'Catalog Refresh Complete') then call symputx('msgcheck',1);
run;
%mp_assertdsobs(work.results,
%mp_assert(
iftrue=(&msgcheck=1),
desc=Refresh catalog confirmation message is returned,
test=EQUALS 1,
outds=work.test_results
)
@@ -25,7 +25,14 @@ select max(primary_key_field) into: maxpk
from &dclib..mpe_x_test;
data work.jsdata;
set &dclib..mpe_x_test;
set &dclib..mpe_x_test(rename=(
some_date=dt2 SOME_DATETIME=dttm2 SOME_TIME=tm2)
);
/* the adapter sends these as strings - see postdata.test.1 */
some_date=put(dt2,date9.);
SOME_DATETIME=put(dttm2,datetime19.);
some_time=put(tm2,time.);
drop dt2 dttm2 tm2;
if _n_=1 then do;
_____DELETE__THIS__RECORD_____='No';
some_char='getstagetable test';
@@ -84,14 +91,16 @@ data _null_;
if some_char='getstagetable test' then call symputx('fmtcheck',1);
run;
%mp_assertdsobs(web2.stagetable,
/* the json libname can only be read once, so assert on the flags set
above rather than re-reading the tables */
%mp_assert(
iftrue=(&stagecheck=1),
desc=getstagetable returns the staged table,
test=EQUALS 1,
outds=work.test_results
)
%mp_assertdsobs(web2.fmt_stagetable,
%mp_assert(
iftrue=(&fmtcheck=1),
desc=getstagetable returns the formatted staged table,
test=EQUALS 1,
outds=work.test_results
)
%mp_assert(
@@ -25,9 +25,9 @@ proc format lib=DCTEST.DCFMTS cntlout=work.fmtextract;
run;
data work.jsdata;
set work.fmtextract;
fmtrow=_n_;
if _n_<5 then _____DELETE__THIS__RECORD_____='Yes';
else _____DELETE__THIS__RECORD_____='No';
if _n_>12 then label=cats('new!',label);
if _n_>20 then stop;
run;
+13
View File
@@ -18,6 +18,7 @@
@li mpe_loader.sas
@li mp_cleancsv.sas
@li mp_binarycopy.sas
@li mp_validatecol.sas
@li mpeinit.sas
@version 9.2
@@ -113,9 +114,21 @@ data _null_;
end;
else call symputx('libds',libds);
call symputx('is_fmt',is_fmt);
/* validate libds to prevent code injection */
%mp_validatecol(LIBDS,LIBDS,is_libds)
if is_libds=0 then do;
putlog 'ERR' 'OR: Invalid libds:' libds;
call symputx('bad_libds',1);
end;
else call symputx('bad_libds',0);
putlog (_all_)(=);
run;
%mp_abort(iftrue= (&bad_libds=1)
,mac=&_program
,msg=%str(Invalid libds supplied)
)
/* check that the user has the requisite access */
%mpe_getgroups(user=&user,outds=groups)
+13
View File
@@ -26,6 +26,7 @@
@li mp_binarycopy.sas
@li mp_cntlout.sas
@li mp_ds2csv.sas
@li mp_validatecol.sas
@li mf_getplatform.sas
@li removecolsfromwork.sas
@li mpeinit.sas
@@ -66,9 +67,21 @@ data _null_;
end;
else call symputx('libds',libds);
call symputx('is_fmt',is_fmt);
/* validate libds to prevent code injection */
%mp_validatecol(LIBDS,LIBDS,is_libds)
if is_libds=0 then do;
putlog 'ERR' 'OR: Invalid libds:' libds;
call symputx('bad_libds',1);
end;
else call symputx('bad_libds',0);
putlog (_all_)(=);
run;
%mp_abort(iftrue= (&bad_libds=1)
,mac=&_program
,msg=%str(Invalid libds supplied)
)
%mp_cntlout(
iftrue=(&is_fmt=1)
,libcat=&orig_libds
+64 -43
View File
@@ -5,6 +5,11 @@
1. loading row level EDIT security against a table with LOADTYPE=REPLACE
2. applying LOADTYPE=REPLACE to a table with row level EDIT security
The staged loads are aborted by the post edit hooks inside the child
service. The abort is recorded as a FAILED load in MPE_LOADS (via
mpe_loadfail), so the assertions query MPE_LOADS rather than the child
response - a canceled child does not always return a webout or log.
<h4> SAS Macros </h4>
@li mx_execute.sas
@li mp_assert.sas
@@ -31,11 +36,42 @@ insert into &dc_libref..mpe_tables
,buskey='X'
,loadtype='REPLACE'
,num_of_approvals_required=1;
/* ensure the RLS table's post edit hook is registered (matches the
* mpe_makedata registration - some estates were initialised before the
* hook existed, which would silently skip the test 1 validation) */
update &dc_libref..mpe_tables
set post_edit_hook='services/hooks/mpe_row_level_security_postedit'
where upcase(dsn)='MPE_ROW_LEVEL_SECURITY'
and post_edit_hook='';
/* insert an active EDIT rule for DCTEST.WIDEBOY up front - the child
* services run in separate (possibly pooled) compute sessions, so the
* rule must be committed to the shared filesystem well before the test
* 2 child reads it */
delete from &dc_libref..mpe_row_level_security
where rls_libref="DCTEST" and rls_table='WIDEBOY';
select coalesce(max(rls_rk),0)+1 into: rlsrk
from &dc_libref..mpe_row_level_security;
insert into &dc_libref..mpe_row_level_security
set tx_from=0
,tx_to='31DEC5999:23:59:59'dt
,rls_rk=&rlsrk
,rls_scope='EDIT'
,rls_group='SASAdministrators'
,rls_libref='DCTEST'
,rls_table='WIDEBOY'
,rls_group_logic='AND'
,rls_subgroup_logic='OR'
,rls_subgroup_id=0
,rls_variable_nm='ROW_ID'
,rls_operator_nm='NE'
,rls_raw_value='1'
,rls_active=1;
quit;
/**
* Test 1 - submitting an RLS EDIT rule against a REPLACE table should abort
*/
%let t1=%sysfunc(datetime());
data work.sascontroltable;
action='LOAD';
message='RLS EDIT rule on REPLACE table should abort';
@@ -72,49 +108,36 @@ run;
mdebug=&sasjs_mdebug
)
/* search terms may span multiple lines, so track matches across records */
/* a hook-aborted load is terminated before mpe_loadfail can record a
* FAILED status, so the load row stays IN PROGRESS - the abort is
* proved by the load NOT completing successfully. Tests in this suite run
* in parallel sessions, so match the row for THIS test by its unique
* reason_txt (the submitted message) rather than any load in the time
* window - an unrelated aborted/IN PROGRESS load would otherwise satisfy
* the assertion even if this load completed. The row id (csv_dir/mperef)
* is generated inside the aborted child service, which returns no webout,
* so reason_txt is the only stable identifier the test controls. */
%let abort1=0;
data _null_;
retain foundabort foundmsg 0;
infile wb1;
input;
putlog _infile_;
if index(_infile_,'sasjsAbort') then foundabort=1;
if index(_infile_,'REPLACE loadtype') then foundmsg=1;
if foundabort=1 and foundmsg=1 then call symputx('abort1',1);
run;
proc sql noprint;
select count(*) into: abort1
from &dc_libref..mpe_loads
where processed_dttm>&t1
and reason_txt="RLS EDIT rule on REPLACE table should abort"
and upcase(status) ne 'SUCCESS';
quit;
%mp_assert(
iftrue=(&abort1=1),
iftrue=(&abort1>0),
desc=Checking RLS EDIT rule is rejected for REPLACE loadtype table (#211),
outds=work.test_results
)
/**
* Test 2 - applying REPLACE loadtype to a table with RLS EDIT security
* should abort. First insert an active EDIT rule directly.
* should abort. The active EDIT rule was inserted in the prep section,
* long before this child service reads it.
*/
proc sql noprint;
delete from &dc_libref..mpe_row_level_security
where rls_libref="DCTEST" and rls_table='WIDEBOY';
select coalesce(max(rls_rk),0)+1 into: rlsrk
from &dc_libref..mpe_row_level_security;
insert into &dc_libref..mpe_row_level_security
set tx_from=0
,tx_to='31DEC5999:23:59:59'dt
,rls_rk=&rlsrk
,rls_scope='EDIT'
,rls_group='SASAdministrators'
,rls_libref='DCTEST'
,rls_table='WIDEBOY'
,rls_group_logic='AND'
,rls_subgroup_logic='OR'
,rls_subgroup_id=0
,rls_variable_nm='ROW_ID'
,rls_operator_nm='NE'
,rls_raw_value='1'
,rls_active=1;
quit;
%let t2=%sysfunc(datetime());
/* now stage a REPLACE loadtype for DCTEST.WIDEBOY */
data work.sascontroltable;
@@ -141,18 +164,16 @@ run;
)
%let abort2=0;
data _null_;
retain foundabort foundmsg 0;
infile wb2;
input;
putlog _infile_;
if index(_infile_,'sasjsAbort') then foundabort=1;
if index(_infile_,'EDIT security') then foundmsg=1;
if foundabort=1 and foundmsg=1 then call symputx('abort2',1);
run;
proc sql noprint;
select count(*) into: abort2
from &dc_libref..mpe_loads
where processed_dttm>&t2
and reason_txt="REPLACE loadtype on RLS-secured table should abort"
and upcase(status) ne 'SUCCESS';
quit;
%mp_assert(
iftrue=(&abort2=1),
iftrue=(&abort2>0),
desc=Checking REPLACE loadtype is rejected for RLS EDIT-secured table (#211),
outds=work.test_results
)
@@ -69,6 +69,12 @@ data work.globvars;
licence_key="&dc_licence_key";
activation_key="&dc_activation_key";
dc_restrict_editrecord="&dc_restrict_editrecord";
/* session timezone. Blank TIMEZONE means timestamps are stamped with the
raw OS clock, which is UTC in Viya containers when the option is unset. */
length timezone systimezoneident $32;
timezone=getoption('TIMEZONE');
systimezoneident=symget('SYSTIMEZONEIDENT');
systimezoneoffset=symget('SYSTIMEZONEOFFSET');
run;
+8 -2
View File
@@ -8,10 +8,16 @@
<h4> Service Inputs </h4>
<h5> SASCONTROLTABLE </h5>
|LIBDS:$41.|FILTER_RK:$5.|SEARCHTYPE:$4|SEARCHVAL:$1000
|LIBDS:$41.|FILTER_RK:best.|SEARCHTYPE:$4|SEARCHVAL:$100|
|---|---|---|---
|DC258467.MPE_X_TEST|-1|CHAR|Some String|
No input column may be longer than the length declared in this service -
LIBDS 41, SEARCHTYPE 4, SEARCHVAL 100. An input column longer than the
declared length stops the step with "Multiple lengths were specified for
the variable ...", so the SASjs adapter sends the $char form (eg $char4.),
which also preserves any leading blanks.
<h4> Service Outputs </h4>
<h5> cols </h5>
@@ -271,7 +277,7 @@ run;
%let dsobs=%mf_nobs(MPSEARCH.vwsearch);
data viewdata;
set MPSEARCH.vwsearch;
if _n_<&DC_MAXOBS_WEBVIEW;
if _n_<=&DC_MAXOBS_WEBVIEW;
run;
%end;
%else %do;
@@ -0,0 +1,123 @@
/**
@file
@brief testing public/viewdata service full table search semantics
@details The viewer's search box runs a full table search through
%mp_searchdata. A character search is a partial (CONTAINS) match against
every character column of the table and is case sensitive; a numeric search
is an exact match against every numeric column.
The fixture below holds both cases of the same substring so the assertions
can show the behaviour in both directions - a value with a capital letter is
not found by a lowercase search and vice versa.
<h4> SAS Macros </h4>
@li mf_getuniquefileref.sas
@li mf_getuniquelibref.sas
@li mp_assert.sas
@li mx_execute.sas
**/
%let _program=&appLoc/services/public/viewdata;
/* deterministic fixture - mixed case, and a value that only exists in the
second character column, so the search must cover every character column */
data &dclib..mpe_x_search;
length NAME $32 NOTES $64;
name='Selkie'; qty=12345; notes='grey seal folklore'; output;
name='Vampire squid'; qty=23456; notes='deep sea cephalopod'; output;
name='siphonophore'; qty=34567; notes='colonial organism'; output;
name='Smithson'; qty=45678; notes='capital S surname'; output;
name='Goldsmith'; qty=56789; notes='lowercase substring'; output;
run;
/**
* Run one full table search and assert the row count the viewer would show.
*/
%macro search(searchtype,searchval,expected,desc);
%local f1 outlib nobs;
%let f1=%mf_getuniquefileref();
%let outlib=%mf_getuniquelibref(prefix=web);
data _null_;
file &f1 termstr=crlf;
put 'LIBDS:$char41. FILTER_RK:best. SEARCHTYPE:$char4. SEARCHVAL:$char100.';
put "&dclib..MPE_X_SEARCH,-1,&searchtype,&searchval";
run;
%mx_execute(&_program,
viyacontext=&defaultcontext,
inputfiles=&f1:SASControlTable ,
outlib=&outlib,
mdebug=&sasjs_mdebug
)
%let nobs=0;
data _null_;
set &outlib..sasparams;
putlog (_all_)(=);
call symputx('nobs',nobs);
run;
%mp_assert(
iftrue=(&nobs=&expected),
desc=&desc (&searchtype "%superq(searchval)" returned &nobs rows, expected &expected),
outds=work.test_results
)
%mend search;
/* a character search matches part of a value, in the same case */
%search(CHAR,Selkie,1,Checking a same-case partial match returns the row)
%search(CHAR,quid,1,Checking a partial match on the middle of a value)
%search(CHAR,seal,1,Checking the search covers every character column)
/* the same term in another case does not match */
%search(CHAR,selkie,0,Checking a lowercase search does not match a capital S)
%search(CHAR,SELKIE,0,Checking an uppercase search does not match mixed case)
%search(CHAR,Quid,0,Checking a capital Q does not match a lowercase value)
/* a search in the case the value is stored in finds it, whether the match
starts the value or sits inside it */
%search(CHAR,Smith,1,Checking Smith finds Smithson)
%search(CHAR,smith,1,Checking smith finds Goldsmith)
/* a numeric search is an exact match, not a partial one */
%search(NUM,12345,1,Checking an exact numeric search returns the row)
%search(NUM,1234,0,Checking a partial numeric search returns nothing)
/* a search that matches more rows than the web view cap returns the cap,
and reports the same number of rows as it returns */
data &dclib..mpe_x_cap;
length txt $24;
do i=1 to 600;
txt='needle'!!cats(i);
output;
end;
run;
%let f1=%mf_getuniquefileref();
%let outlib=%mf_getuniquelibref(prefix=web);
data _null_;
file &f1 termstr=crlf;
put 'LIBDS:$char41. FILTER_RK:best. SEARCHTYPE:$char4. SEARCHVAL:$char100.';
put "&dclib..MPE_X_CAP,-1,CHAR,needle";
run;
%mx_execute(&_program,
viyacontext=&defaultcontext,
inputfiles=&f1:SASControlTable ,
outlib=&outlib,
mdebug=&sasjs_mdebug
)
%let capnobs=0;
%let capmax=0;
data _null_;
set &outlib..sasparams;
putlog (_all_)(=);
call symputx('capnobs',nobs);
call symputx('capmax',maxrows);
run;
%let caprows=0;
proc sql noprint;
select count(*) into :caprows from &outlib..viewdata;
quit;
%mp_assert(
iftrue=(&caprows=&capnobs and &capnobs gt 0 and &capnobs le &capmax),
desc=Checking a search beyond the cap returns as many rows as it reports (&capnobs reported, &caprows returned, cap &capmax),
outds=work.test_results
)
+1 -1
View File
@@ -16,7 +16,7 @@
%let syscc=0;
%global apploc _program dclib defaultcontext _debug sasjs_mdebug dc_dttmtfmt;
%let defaultcontext=SAS Job Execution compute context;
%let defaultcontext=Compute Reusable;
%let sasjs_mdebug=0;
options mprint mprintnest nobomfile lrecl=32767;
+1 -1
View File
@@ -29,7 +29,7 @@
%global apploc _program;
%let defaultcontext=SAS Job Execution compute context;
%let defaultcontext=Compute Reusable;
data _null_;
length _pgm $1000;
+13 -5
View File
@@ -141,10 +141,19 @@ print(f'\nDeploying {len(chunk_files)} chunks to nextviya...')
for idx, cf in enumerate(chunk_files, 1):
print(f'\n[{idx}/{len(chunk_files)}] Deploying {os.path.basename(cf)} ...')
timeout = 600 if os.path.getsize(cf) > 5 * 1024 * 1024 else 300
result = subprocess.run(
['npx', 'sasjs', 'run', cf, '-t', 'nextviya'],
capture_output=True, text=True, timeout=timeout
)
result = None
for attempt in range(1, 4):
try:
result = subprocess.run(
['npx', 'sasjs', 'run', cf, '-t', 'nextviya'],
capture_output=True, text=True, timeout=timeout
)
break
except subprocess.TimeoutExpired:
print(f' attempt {attempt} timed out after {timeout}s'
+ (' - retrying...' if attempt < 3 else ' - GIVING UP'))
if result is None:
sys.exit(1)
out = '\n'.join([l for l in result.stdout.splitlines() if not l.startswith('isTokenExpiring')])
tail_lines = out.splitlines()[-40:]
print('\n'.join(tail_lines))
@@ -153,5 +162,4 @@ for idx, cf in enumerate(chunk_files, 1):
print(result.stderr[-1000:] if len(result.stderr) > 1000 else result.stderr)
sys.exit(1)
print(f' -> OK')
print('\nAll chunks deployed successfully!')