chore(release): 7.14.1 [skip ci]

## [7.14.1](https://git.datacontroller.io/dc/dc/compare/v7.14.0...v7.14.1) (2026-09-17)

### Bug Fixes

* **admin:** return refreshcatalog output as webout JSON ([1dccf61](1dccf61f62))
* **deps:** bump @sasjs/cli to 4.20.3 ([d301fc2](d301fc2b4d))
* **deps:** bump @sasjs/core to 5.2.7 ([09f31ce](09f31ce6da))
* **deps:** bump @sasjs/core to 5.2.8 ([c73817f](c73817fb6a))
* **editors:** declare the mp_validatecol dependency in stagedata and loadfile ([a556be4](a556be4c57))
* **hooks:** make the pre-commit size check actually run ([5383803](53838036e6))
* **hooks:** require the repo-pinned gitleaks, drop the fallbacks ([fbcde41](fbcde41321))
* **security:** escape cell values in status renderers to prevent DOM XSS ([cfd8f06](cfd8f06435))
* **security:** validate libds in stagedata and loadfile ([2d31e5a](2d31e5a8b9))
* **viewdata:** return the full row cap from a table search ([199b57d](199b57d9f9))

.agents/skills/dc-cypress/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: dc-cypress
+description: >
+  Use this skill when writing or debugging Cypress E2E specs in the dc repo, or when working on the
+  JS mock backend they run against. Covers the SASjs Server mock environment (deploy, seed, ports),
+  the spec conventions the CI harness expects, the viewer/editor DOM selectors that the existing
+  specs get wrong, the fixed 1280x720 video capture, and the rule that a mock service must mirror
+  the real SAS service it stands in for.
+---
+
+# Cypress E2E and the JS mock backend
+
+The specs under `client/cypress/e2e/` run against a mock backend: JS stored programs under `sas/mocks/sasjs/services/` executed by SASjs Server in JS-only mode. No SAS licence, no SAS server. CI (`.gitea/workflows/build.yaml`) builds the same environment - SASjs Server on :5000, `sasjs cbd -t server-ci`, makedata, then `ng serve` on :4200 - and runs a fixed list of specs against it.
+
+This skill covers the parts that are not obvious from reading a spec file: how to stand the environment up, the selectors that actually work, and the traps in the mock services. It does not cover the SAS-side tests (see `.agents/docs/testing.md` and the `dc-sas` skill).
+
+## When to Use
+
+- Adding or changing a spec under `client/cypress/e2e/`.
+- A mock service needs a new behaviour, or a spec "does nothing" because the mock ignores an input.
+- Recording a demo video from a spec run.
+- Debugging a spec that passes locally but fails in CI (usually a viewport or timing assumption).
+
+Don't use for: SAS backend tests (`sasjs test -t 4gl`), Angular unit tests (`npm run test:headless`), or general SAS development (use the `dc-sas` skill).
+
+## Stand up the mock environment
+
+1. **SASjs Server, JS mode.** Download the linux build from the sasjs/server releases, then create a `.env` next to the binary:
+
+       RUN_TIMES=js
+       NODE_PATH=node
+       CORS=enable
+       WHITELIST=http://localhost:4200
+
+   Start it (`./api-linux`). It runs in desktop mode on :5000 with no authentication.
+
+2. **Deploy the mocks.**
+
+       cd sas/mocks/sasjs && sasjs cbd -t server-ci
+
+   To start from a clean appLoc, delete it first:
+
+       curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
+
+3. **Seed the mock database.**
+
+       cd sas/mocks && sasjs request services/admin/makedata -t server-ci \
+           -d deploy/makedata.json -o ./makedata_out.json
+
+   `makedata` replies with HTML (it is normally called as a URL redirect), so the CLI reports `invalid Json string`. That is expected - check the drive instead.
+
+4. **Frontend.** Point the `sasjs` tag in `client/src/index.html` at the mock backend (`serverUrl="http://localhost:5000"`, `appLoc="/Public/app/dc"`, `serverType="SASJS"` - a local-only edit, the repo default is `serverUrl=""`), then `npx ng serve --host 0.0.0.0 --port 4200`. `client/cypress.env.json` needs `username`/`password` entries; any values work, `loginAndUpdateValidKey()` no-ops when there is no login form.
+
+5. **Run.**
+
+       cd client && npx cypress run --browser electron --spec cypress/e2e/<spec>.cy.ts
+
+   Only Electron is installed here (no system Chrome), so always pass `--browser electron`.
+
+## Spec conventions
+
+- Start the file with `export {}` so its top-level consts do not collide with other spec files under the TS type-checker.
+- `beforeAll`: `cy.visit(\`${hostUrl}/SASLogon/logout\`)` then `cy.loginAndUpdateValidKey(true)`. The `true` forces a licence key instead of relying on an earlier spec in the same run having applied one.
+- `beforeEach`: `cy.visit(hostUrl + appLocation)`, `visitPage('home')`, `visitPage('view/data')`. The `home` visit is what triggers the startup-data fetch; going straight to `view/data` leaves the nav tree empty in slower environments.
+- Declare helpers at the **bottom of each spec file** (that is the house pattern) - do not introduce a shared helper module.
+- Use `Cypress.env('longerCommandTimeout')` for waits and gate on `.app-loading` disappearing.
+- **No artificial delays.** These specs are the CI regression suite, so `cy.wait(n)` to "let a result land" is not acceptable - assert on the resulting state instead (Cypress retries assertions). Replace a sleep with a state assertion: e.g. after picking an option from a soft-select, wait for `#datalist_<inputId>` to have class `hidden` rather than sleeping.
+- Lint: `cd client && npm run lint:check` (prettier covers `cypress/e2e/*.cy.ts`).
+
+## Selectors that work
+
+- Viewer search box: `input[name="search-input"]` (Enter or the magnifier icon submits); the numeric toggle is `input[name="numeric_check"]`.
+- Row/column count in the header: `.title-col p` renders `(1,000 rows, 9 cols)` - and `(1 row, 9 cols)` when the result is a single row, so an assertion on the count has to pluralise.
+- Filter info bar: `.infoBar b` - only rendered when the backend's `sasparams.FILTER_TEXT` is neither blank, `' '`, nor `'1=1'`.
+- Empty result: `.noData h3` with `No data found with given conditions`.
+- Grid: `#hotTable .ht_master.handsontable .htCore tbody tr` (body rows), `#hotTable .ht_clone_top .htCore thead` (headers, and `button.changeType` confirms Handsontable has finished rendering them).
+- Nav tree: `.nav-tree clr-tree > clr-tree-node` -> `.clr-tree-node-content-container .clr-treenode-content p` (expands a library) -> `.clr-treenode-link` (table links).
+- **Viewer filter modal** - `.filterSide` (the options button) -> `.dropdown-menu` contains `Filter` -> `#vals_var_id0_0` (variable, pick from `#datalist_vals_var_id0_0 option`) and `#vals_0_0` (value) -> `.filter-modal button[aria-label="Ok"]`.
+  - `filtering.cy.ts` opens the filter with `.btnCtrl .btnView`, which is an **editor** control and does not exist in the viewer. Do not copy that helper into a viewer spec.
+  - The soft-select inputs drop a transparent click-catcher over the modal while their suggestion list is open, so Cypress reports the Ok button (and the next input) as covered by `.overlay`. Send the Escape keyup a user would send (`trigger('keyup', { key: 'Escape', force: true })`) and wait for the list to close before clicking on.
+
+## Recording a demo video
+
+`video: true` in `cypress.config.ts` is all that is needed. Run the spec and the MP4 lands in `client/cypress/videos/<spec>.mp4`.
+
+**The capture is the browser window's content area, and it contains the Cypress runner** - the command log takes the left ~450px and the app is rendered in the remaining pane, so the pane is `(capture width - 450) x (capture height - 96)` and the app is zoomed to fit it.
+
+That makes the window size the only lever that matters:
+
+- Electron is stuck at 1280x720 (it ignores `--window-size`), i.e. an 830x624 pane, and the repo's 1600x900 viewport renders at 50% - the app only occupies ~806x452 of it.
+- A real Chromium browser honours `--window-size`, so the capture can be sized to whatever the pane needs. `cypress.config.ts` passes it through from `RECORD_WINDOW_SIZE` (inert in CI, which sets no env).
+
+For a 16:9 pane, solve `(W - 450) / (H - 96) = 16/9`: W = 1920 gives H = 923 of window *content*, and Chrome adds ~87px of window chrome above that, so pass 1920x1010:
+
+    RECORD_WINDOW_SIZE=1920,1010 npx cypress run \
+      --browser /opt/data/profiles/dc/tools/chrome-linux64/chrome \
+      --spec cypress/e2e/full-table-search.cy.ts \
+      --env skipNegative=true,demoPause=1500
+
+That yields a 1920x922 capture with a 1470x826 pane (16:9) and, at the repo's 1600x900 viewport, an app at ~0.90 zoom occupying 1441x812 - about 1.9x the linear size of the 16:9 crop Electron can give. Crop the app out:
+
+    ffmpeg -ss <start> -i cypress/videos/<spec>.mp4 -vf "crop=1440:812:464:80" \
+      -an -c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p \
+      -movflags +faststart <spec>-cut.mp4
+
+Measure the crop rect from the frames rather than guessing: the pane's flat background is rgb(225,227,236), so a row/column scan for pixels that differ from it gives the app's bounding box (for the 1920x922 capture: runner pane ends x=449, app x=463-1903, app y=80-893; for the 1280x720 Electron capture: app x=468-1259, y=80-675). The app's own background is nearly the same grey, so verify the result on the first, middle and last frames rather than trusting a pixel test alone.
+
+To find where to cut, scan the mean *and* standard deviation of the content area: the app boots mid-run (a dark screen, mean ~65, sd ~2.5), the "Loading Table Viewer" spinner is flat and light (mean ~252, sd ~0), and the rendered table is mean ~245, sd ~31. Cut on the first table frame - cutting a fraction early shows the boot screen or the spinner.
+
+Cypress starts its own Xvfb for the run (a second display appears in `/tmp/.X11-unix`), so an external `DISPLAY` is ignored; `--force-device-scale-factor` and a CSS `zoom` also do not change the capture.
+
+### Grid columns that are clipped
+
+Handsontable sizes columns to content, so a narrow result can be *wider* than the grid viewport: the horizontal scrollbar appears and the rightmost column (NOTES on `MPE_X_SEARCH`) is cut off. The cell text is still in the DOM, so an assertion passes either way - the column only has to be scrolled into view for the *video* to show why the rows matched.
+
+    const scrollGridTo = (position: 'left' | 'right') => {
+      cy.get('#hotTable .wtHolder').first().then(($holder: any) => {
+        const el = $holder[0]
+        el.scrollLeft = position === 'right' ? el.scrollWidth : 0
+        expect(el.scrollLeft).to.equal(position === 'right' ? el.scrollWidth - el.clientWidth : 0)
+      })
+    }
+
+Setting `scrollLeft` is enough - Handsontable keeps the header clone in step - and the browser clamps it to `scrollWidth - clientWidth`, so that is what scrolling right has to land on. This is a legitimate spec step (it runs in CI), not a recording flag.
+
+### A longer, more readable demo
+
+A spec that doubles as a walkthrough should be a single `it()`: open the table once and make each variation a search in place. The recording then reads as a demo rather than a sequence of page loads, and it is far faster - every extra `it()` pays for the `beforeEach` navigation again.
+
+Two recording-only env flags, both inert in CI because CI passes no env:
+
+    const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
+    const negativeStep = (step: () => void) => {
+      if (!skipNegative) step()
+    }
+    const demoPauseMs = Number(Cypress.env('demoPause')) || 0
+
+    negativeStep(() => {
+      searchFor('trench')
+      assertNoData()
+    })
+
+- `skipNegative=true` drops the steps that legitimately match nothing (wrong case, partial number, value not present), so the recording never shows an empty screen.
+- `demoPause=1500` pauses 1.5s at each row-count assertion, so each result can be read. Apply it in one place (the row-count assertion, which every beat makes exactly once) rather than sprinkling `cy.wait()` through the test - CI still runs the file with no waits at all.
+
+More beats, not a slower take, is what makes a demo longer: a walkthrough of 12 row-count beats takes 52s against 30s for 7.
+
+## Mock services
+
+- One JS file per service under `sas/mocks/sasjs/services/**`, deployed as SASjs stored programs. Shared helpers live in `sas/mocks/sasjs/services/dcMockUtils.js` (`fetchTable`/`parseCsv` for `%webout(FETCH)`, `webOutOpen`/`webOutObj`/`webOutClose` for `%webout`, `mpeFilterMaster`, `mpFilterGenerate`, `mpFilterStore`, `getDdType`, `formatSasValue`, `sasVarsEntry`).
+- Mock data is one JSON file per table at `<drive>/files/<appLoc>/data/<LIBREF>/<table>.json`, written by `services/admin/makedata.js` (which also writes `services/settings.js`). The drive lives under the SASjs Server directory (`sasjs_root/drive`).
+- `makedata.js` self-destructs after a successful run (it deletes itself from the drive so the frontend can detect completion), so re-deploy before re-seeding.
+- **A mock must mirror the real service.** `viewdata.js` once ignored `FILTER_RK` and hard-coded `FILTER_TEXT` to blank, so the viewer's filter silently filtered nothing and its info bar never showed a clause, while the real `viewdata.sas` applies `%mpe_filtermaster` and then `%mp_searchdata`. If a mock is a deliberate stub, say so in a comment and make the divergence explicit - a mock that quietly drops an input turns an E2E spec into a test of nothing.
+
+## Pitfalls
+
+- `sasjs cbd -t server-ci` rebuilds the *mock services* and re-deploys the streamed web app from `client/dist`; it does not rebuild the Angular app.
+- Deleting the drive folder before deploying avoids stale services and stale data.
+- Captured `_webout` JSON, request logs and iteration snapshots belong in `tmp/` (gitignored), never in `sas/mocks/` - see the `dc-sas` skill.
+- The mock server has no auth, so `cy.visit` on `/SASLogon/logout` returns 404 - that is fine, Cypress does not fail on the status code.
+
+## Verification
+
+- `npx cypress run --browser electron --spec <specs>` - all green, and run it more than once when you have just removed a wait.
+- `cd client && npm run lint:check` passes.
+- For a mock change, hit the service directly before writing UI assertions:
+
+      curl -sS -H "Content-Type: application/json" -X POST \
+        --data '{"_program":"/Public/app/dc/services/public/viewdata","SASControlTable":"LIBDS:$41. FILTER_RK:best. SEARCHTYPE:$4 SEARCHVAL:$1000\nDC_JSLIB.MPE_X_SEARCH,0,CHAR,Trench"}' \
+        http://localhost:5000/SASjsApi/stp/execute
+
+  then check `sasparams[0].NOBS`, the row count and `FILTER_TEXT`. Pin those exact numbers in the spec.

.gitea/workflows/build.yaml

@@ -154,7 +154,7 @@ jobs:
           # Start frontend and run cypress
           # timeout 1800: SIGTERM after 30 min so Cypress can flush video/screenshots
           # before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
-          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
+          npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 1800 npx cypress run --browser chrome --spec "cypress/e2e/csv-limited.cy.ts,cypress/e2e/liveness.cy.ts,cypress/e2e/editor.cy.ts,cypress/e2e/excel-multi-load.cy.ts,cypress/e2e/excel.cy.ts,cypress/e2e/csv.cy.ts,cypress/e2e/filtering.cy.ts,cypress/e2e/licensing.cy.ts,cypress/e2e/viewer-labels.cy.ts,cypress/e2e/full-table-search.cy.ts,cypress/e2e/viewbox.cy.ts,cypress/e2e/stage.cy.ts"
 
       - name: Zip Cypress videos
         if: always()

CHANGELOG.md

@@ -1,3 +1,19 @@
+## [7.14.1](https://git.datacontroller.io/dc/dc/compare/v7.14.0...v7.14.1) (2026-09-17)
+
+
+### Bug Fixes
+
+* **admin:** return refreshcatalog output as webout JSON ([1dccf61](https://git.datacontroller.io/dc/dc/commit/1dccf61f6290e50ebc365964fab864cb93f8d1b5))
+* **deps:** bump @sasjs/cli to 4.20.3 ([d301fc2](https://git.datacontroller.io/dc/dc/commit/d301fc2b4dc86b7eb3027b8c8965dd7d1079363e))
+* **deps:** bump @sasjs/core to 5.2.7 ([09f31ce](https://git.datacontroller.io/dc/dc/commit/09f31ce6da1e6a286c4e05ac55fb1d0c801d62f5))
+* **deps:** bump @sasjs/core to 5.2.8 ([c73817f](https://git.datacontroller.io/dc/dc/commit/c73817fb6a8c19d970a19b98d5ef79c05243cc8d))
+* **editors:** declare the mp_validatecol dependency in stagedata and loadfile ([a556be4](https://git.datacontroller.io/dc/dc/commit/a556be4c577020c8e3147de4d3e71ab84fb0d174))
+* **hooks:** make the pre-commit size check actually run ([5383803](https://git.datacontroller.io/dc/dc/commit/53838036e635cd6f33dccc4534d6f258d3b3da5d))
+* **hooks:** require the repo-pinned gitleaks, drop the fallbacks ([fbcde41](https://git.datacontroller.io/dc/dc/commit/fbcde4132147240fed21a88d44f62b68f58cf67c))
+* **security:** escape cell values in status renderers to prevent DOM XSS ([cfd8f06](https://git.datacontroller.io/dc/dc/commit/cfd8f064352940cd57cf19c86504036f0a7edd8d))
+* **security:** validate libds in stagedata and loadfile ([2d31e5a](https://git.datacontroller.io/dc/dc/commit/2d31e5a8b957d30057eb749c2d7f246170af8c8f))
+* **viewdata:** return the full row cap from a table search ([199b57d](https://git.datacontroller.io/dc/dc/commit/199b57d9f9fb4b00a229d23e25e22e5dee471d2a))
+
 # [7.14.0](https://git.datacontroller.io/dc/dc/compare/v7.13.0...v7.14.0) (2026-09-04)
 
 

client/cypress.config.ts

@@ -38,9 +38,24 @@ export default defineConfig({
       // Pin the browser locale so locale-formatted cells (intl-date/time/datetime)
       // render deterministically regardless of the runner's system locale.
       on('before:browser:launch', (browser, launchOptions) => {
+        // Recording only: the video capture is the browser window, which
+        // contains the Cypress runner - the command log takes the left ~450px
+        // and the app sits in the remaining pane.  Sizing the window is the
+        // only lever that moves the capture, and only for a real Chromium
+        // browser (Electron ignores it).  For a 16:9 pane, solve
+        // (W - 450) / (H - 96) = 16/9: W = 1920 -> H = 923 of window content,
+        // so pass RECORD_WINDOW_SIZE=1920,1010 and crop the pane.
+        // Inert unless the env var is set, so CI is unaffected.
+        if (process.env.RECORD_WINDOW_SIZE && browser.family === 'chromium') {
+          launchOptions.args.push(
+            `--window-size=${process.env.RECORD_WINDOW_SIZE}`
+          )
+        }
+
         if (browser.family === 'chromium' && browser.name !== 'electron') {
           launchOptions.args.push('--lang=en-GB')
         }
+
         return launchOptions
       })
     }

client/cypress/e2e/full-table-search.cy.ts

@@ -0,0 +1,385 @@
+// Marks this file as an ES module (rather than a global script) so its
+// top-level consts don't collide, under the TS type-checker, with the same
+// names declared in other spec files — see e.g. viewer-labels.cy.ts.
+export {}
+
+const username = Cypress.env('username')
+const password = Cypress.env('password')
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+const serverType = Cypress.env('serverType')
+const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
+
+// Fixture: MPE_X_SEARCH, a 1000-row demo table built by the mock data builder
+// (sas/mocks/sasjs/services/admin/makedata.js).  It is deterministic, so every
+// count below can be derived by hand:
+//   - 1000 rows, PK 1001-2000, columns in demo order: PRIMARY_KEY_FIELD,
+//     SITE_NAME, VESSEL, SAMPLE_COUNT, DEPTH_M, SPECIES, CRUISE_DATE,
+//     EXPEDITION_ID, NOTES
+//   - 300 rows have a site name containing "Trench" (case sensitive, so
+//     "trench" matches nothing)
+//   - 280 rows contain "Halcyon": the site Halcyon Trench (100 rows) plus the
+//     vessel RV Halcyon (200 rows) less the 20 rows that are both
+//   - 3 rows carry "siphonophore" in the middle of their NOTES text, in the
+//     sites Oceanus Rise, Nereid Trench and Halcyon Trench
+//   - 100 rows observed Vampire squid (species is picked by (site+vessel)%10)
+//   - 100 rows are at Oceanus Rise (site is picked by (n%50)/5)
+//   - 9 rows have an expedition id starting EXP-000 (EXP-0001..EXP-0009)
+//   - the first row (PK 1001) is the only one with EXPEDITION_ID EXP-0001
+//   - 12 rows were sounded at exactly 1000m (DEPTH_M = 1000 + (n%90)*37)
+//   - 2 rows have SAMPLE_COUNT 4210 (the only numeric value that is not a
+//     primary key, a depth or a date)
+//   - PK 2000 is the last row of the table
+//   - 200 rows were surveyed by RV Halcyon, 60 of them at a Trench site
+const demoTable = 'mpe_x_search'
+
+// Full-table search: the viewer's search box searches EVERY column of the
+// table, character columns by case-sensitive CONTAINS and numeric columns by
+// exact equality (see %mp_searchdata in the SASjs macro core, and
+// viewdata.sas / services/public/viewdata.js).
+//
+// The whole walkthrough is one test - the table is opened once and every
+// variation is a search in place, which is how a user works and what makes the
+// recording read as a demo rather than a set of page loads.
+//
+// Some searches legitimately match nothing: they pin the case sensitivity, the
+// exact-match numeric semantics and the "no data found" handling.  Those steps
+// are wrapped in negativeStep() and can be skipped when recording the demo, so
+// the video only ever shows searches that return rows:
+//
+//   npx cypress run --browser electron \
+//     --spec cypress/e2e/full-table-search.cy.ts --env skipNegative=true
+//
+// CI runs the file with no env, so every step runs there.
+const skipNegative = `${Cypress.env('skipNegative')}` === 'true'
+const negativeStep = (step: () => void) => {
+  if (!skipNegative) step()
+}
+
+context('full table search tests: ', function () {
+  this.beforeAll(() => {
+    cy.visit(`${hostUrl}/SASLogon/logout`, { timeout: longerCommandTimeout })
+    // forceLicenceKey=true: applies a valid licence key instead of relying on
+    // an earlier spec in the same run having already done so.
+    cy.loginAndUpdateValidKey(true)
+  })
+
+  this.beforeEach(() => {
+    cy.visit(hostUrl + appLocation, { timeout: longerCommandTimeout })
+
+    // Visit 'home' first, matching filtering.cy.ts/viewer-labels.cy.ts — going
+    // straight to 'view/data' skips the startup-data fetch 'home' triggers,
+    // which leaves the nav tree empty in slower environments.
+    visitPage('home')
+    visitPage('view/data')
+  })
+
+  it('searches the whole table from the viewer', () => {
+    // Full view: every column, and the true row count for the table.
+    openTableFromTree(libraryToOpenIncludes, demoTable)
+    assertRowCount('1,000')
+    assertGridContains(['Nereid Trench', 'RV Pelagia', 'Vampire squid'])
+
+    // Partial character search: "siphonophore" only ever appears in the middle
+    // of the NOTES text, so a match proves the search is not column-specific.
+    // Three rows is narrow enough that the grid grows a horizontal scrollbar
+    // and clips NOTES - the column the match is actually in - so scroll it
+    // into view, assert on what is now on screen, then put the grid back.
+    searchFor('siphonophore')
+    assertRowCount('3')
+    scrollGridTo('right')
+    pauseForDemo()
+    assertGridContains(['Oceanus Rise', 'Nereid Trench', 'Halcyon Trench'])
+    assertGridExcludes(['Kraken Hollow', 'Vostok Deep'])
+    assertGridContains(['bioluminescent siphonophore was filmed here'])
+    scrollGridTo('left')
+
+    // A search spans every column at once: "Halcyon" is both a site (Halcyon
+    // Trench) and a vessel (RV Halcyon), and the 280 matches are the union.
+    searchFor('Halcyon')
+    assertRowCount('280')
+    assertGridContains(['Halcyon Trench', 'RV Halcyon'])
+
+    // A multi-word site name: the search is not limited to single tokens.
+    searchFor('Oceanus Rise')
+    assertRowCount('100')
+    assertGridContains(['Oceanus Rise'])
+
+    // Multi-word values work the same way - species names are two words.
+    searchFor('Vampire squid')
+    assertRowCount('100')
+    assertGridContains(['Vampire squid'])
+    assertGridExcludes(['Yeti crab'])
+
+    // A partial value in a short code column: EXP-000 matches the first nine
+    // expedition ids.
+    searchFor('EXP-000')
+    assertRowCount('9')
+    assertGridContains(['1001', '1009'])
+
+    // A value that is unique in the table: the first row's expedition id.
+    searchFor('EXP-0001')
+    assertRowCount('1')
+    assertGridContains(['1001'])
+
+    // Character search is case sensitive: the same word in the wrong case
+    // matches nothing.
+    negativeStep(() => {
+      searchFor('trench')
+      assertNoData()
+    })
+
+    searchFor('Trench')
+    assertRowCount('300')
+    assertGridContains(['Halcyon Trench'])
+    assertGridExcludes(['Kraken Hollow'])
+
+    // Numeric search is an exact match across the numeric columns.
+    setNumericSearch(true)
+    searchFor('4210')
+    assertRowCount('2')
+    assertGridContains(['1221', '1781', '4210'])
+
+    // The numeric search covers every numeric column, not just the primary
+    // key: 12 rows were sounded at exactly 1000m.
+    searchFor('1000')
+    assertRowCount('12')
+    assertGridContains(['1091'])
+
+    // ...and an exact primary key is a single row.
+    searchFor('2000')
+    assertRowCount('1')
+    assertGridContains(['2000'])
+
+    negativeStep(() => {
+      searchFor('421')
+      assertNoData()
+    })
+
+    // Back to a character search: clearing the box restores the full table.
+    setNumericSearch(false)
+    searchFor('')
+    assertRowCount('1,000')
+
+    // A search runs inside the current filter, not the whole table.
+    openFilterPopup(() => {
+      setFilterWithValue('VESSEL', 'RV Halcyon', () => {
+        assertRowCount('200')
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+
+        searchFor('Trench')
+        assertRowCount('60')
+        assertGridContains(['Halcyon Trench'])
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+
+        // Clearing the search leaves the filter in place.
+        searchFor('')
+        assertRowCount('200')
+        checkInfoBarIncludes(`(( VESSEL = 'RV Halcyon' ))`)
+      })
+    })
+
+    negativeStep(() => {
+      searchFor('not-in-this-table')
+      assertNoData()
+    })
+  })
+})
+
+const visitPage = (url: string) => {
+  cy.visit(`${hostUrl}${appLocation}/#/${url}`)
+}
+
+// Recording-only linger.  A beat's row count is the moment its result is on
+// screen, so that is where a demo recording pauses to let it be read.  CI runs
+// with no env, so demoPauseMs is 0 and the spec contains no waits at all:
+//
+//   npx cypress run --browser electron --spec cypress/e2e/full-table-search.cy.ts \
+//     --env skipNegative=true,demoPause=1500
+const demoPauseMs = Number(Cypress.env('demoPause')) || 0
+
+// Linger for a recording.  A no-op unless demoPause is set, which only the
+// recording run does - CI passes no env.
+const pauseForDemo = () => {
+  if (demoPauseMs > 0) cy.wait(demoPauseMs)
+}
+
+// The row/column count the viewer shows next to the table name, e.g.
+// "(1,000 rows, 9 cols)".  The viewer pluralises: a single-row result reads
+// "(1 row, 9 cols)".
+const assertRowCount = (rows: string) => {
+  cy.get('.title-col p', { timeout: longerCommandTimeout }).should(
+    ($el: any) => {
+      const unit = rows === '1' ? 'row' : 'rows'
+      expect($el.text().replace(/\s+/g, ' ')).to.contain(`(${rows} ${unit},`)
+    }
+  )
+
+  pauseForDemo()
+}
+
+// The grid body is a Handsontable .wtHolder, and a narrow result is wider than
+// the grid viewport, which clips the rightmost column (NOTES).  Setting
+// scrollLeft is enough - Handsontable keeps the header clone in step - and the
+// assertion right after proves the scroll landed rather than assuming it.  The
+// browser clamps scrollLeft to (scrollWidth - clientWidth), so that is what
+// scrolling right has to land on.
+const scrollGridTo = (position: 'left' | 'right') => {
+  cy.get('#hotTable .wtHolder')
+    .first()
+    .then(($holder: any) => {
+      const el = $holder[0]
+      const maxScroll = el.scrollWidth - el.clientWidth
+
+      el.scrollLeft = position === 'right' ? el.scrollWidth : 0
+
+      if (position === 'right') {
+        // This step only means anything if the grid really is clipping a
+        // column - fail loudly rather than silently doing nothing.
+        expect(el.scrollWidth).to.be.greaterThan(el.clientWidth)
+        expect(el.scrollLeft).to.equal(maxScroll)
+      } else {
+        expect(el.scrollLeft).to.equal(0)
+      }
+    })
+}
+
+// A search that matches nothing replaces the grid with the viewer's "no data"
+// panel - and must not raise a request error.
+const assertNoData = () => {
+  cy.get('.noData h3', { timeout: longerCommandTimeout }).should(
+    'contain.text',
+    'No data found with given conditions'
+  )
+}
+
+const assertGridContains = (values: string[]) => {
+  cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
+    ($grid: any) => {
+      const text = $grid.text().replace(/\s+/g, ' ')
+      for (const value of values) expect(text).to.contain(value)
+    }
+  )
+}
+
+const assertGridExcludes = (values: string[]) => {
+  cy.get('#hotTable', { timeout: longerCommandTimeout }).should(
+    ($grid: any) => {
+      const text = $grid.text().replace(/\s+/g, ' ')
+      for (const value of values) expect(text).to.not.contain(value)
+    }
+  )
+}
+
+const searchFor = (value: string) => {
+  cy.get('input[name="search-input"]', { timeout: longerCommandTimeout })
+    .should('be.visible')
+    .clear()
+
+  if (value.length > 0) cy.get('input[name="search-input"]').type(value)
+
+  cy.get('input[name="search-input"]').type('{enter}')
+}
+
+const setNumericSearch = (enabled: boolean) => {
+  cy.get('input[name="numeric_check"]', { timeout: longerCommandTimeout }).then(
+    (checkbox: any) => {
+      if (checkbox[0].checked !== enabled) checkbox[0].click()
+    }
+  )
+}
+
+const checkInfoBarIncludes = (text: string) => {
+  cy.get('.infoBar b', { timeout: longerCommandTimeout }).should(($el: any) => {
+    expect($el.text().toLowerCase()).to.contain(text.toLowerCase())
+  })
+}
+
+// Opens the viewer's options menu and clicks through to the filter modal.
+const openFilterPopup = (callback?: any) => {
+  cy.get('.filterSide', { timeout: longerCommandTimeout }).first().click()
+
+  cy.get('.dropdown-menu', { timeout: longerCommandTimeout })
+    .contains('Filter')
+    .click()
+
+  cy.get('#vals_var_id0_0', { timeout: longerCommandTimeout }).should('exist')
+
+  if (callback) callback()
+}
+
+// Fills the first clause of the filter modal (variable, then value) and
+// submits it - the viewer stores the clause, reloads the table with the new
+// filter RK and shows the generated WHERE text in the info bar.
+const setFilterWithValue = (
+  variableValue: string,
+  valueString: string,
+  callback?: any
+) => {
+  cy.get('#vals_var_id0_0').type(variableValue)
+  cy.get('#datalist_vals_var_id0_0 option')
+    .contains(variableValue)
+    .click({ force: true })
+
+  // The soft-select closes its suggestion list shortly after a selection -
+  // wait for that closed state instead of sleeping.
+  cy.get('#datalist_vals_var_id0_0').should('have.class', 'hidden')
+
+  cy.get('#vals_0_0').type(valueString)
+
+  closeAutocompleteList('#vals_0_0')
+  cy.get('#datalist_vals_0_0').should('have.class', 'hidden')
+
+  cy.get('.filter-modal button[aria-label="Ok"]').click()
+
+  if (callback) callback()
+}
+
+// The soft-select inputs open a suggestion list that drops a transparent
+// click-catcher over the modal until it closes, so send the same Escape key a
+// user would press.  `force` is needed because the input sits underneath that
+// click-catcher while the list is open.
+const closeAutocompleteList = (inputSelector: string) => {
+  cy.get(inputSelector).trigger('keyup', { key: 'Escape', force: true })
+}
+
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let viyaLib
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            viyaLib = node
+            break
+          }
+        }
+
+        cy.get(viyaLib).within(() => {
+          cy.get(
+            '.clr-tree-node-content-container .clr-treenode-content p'
+          ).click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+
+  // Selecting the table triggers async SPA routing + a viewdata fetch; wait
+  // for the grid to actually render before any subsequent action.
+  cy.get('#hotTable .ht_clone_top .htCore thead button.changeType', {
+    timeout: longerCommandTimeout
+  }).should('exist')
+}

client/src/app/deploy/deploy.component.ts

@@ -21,8 +21,6 @@ export class DeployComponent implements OnInit {
   public step: number = 0
   public adminGroups: any = []
 
-  public client_id: string = ''
-  public client_secret: string = ''
   public appLoc: string = ''
   public dcPath: string = ''
   public selectedAdminGroup: string = ''
@@ -52,9 +50,6 @@ export class DeployComponent implements OnInit {
     this.sasJs = this.sasService.getSasjsInstance()
     this.sasJsConfig = this.sasService.getSasjsConfig()
     this.appLoc = this.dcAdapterSettings?.appLoc || ''
-    this.client_id = localStorage.getItem('deploy_client_id') || ''
-    this.client_secret = localStorage.getItem('deploy_secret_key') || ''
-    this.dcPath = localStorage.getItem('deploy_dc_loc') || ''
   }
 
   ngOnInit() {

client/src/app/editor/utils/renderers.utils.spec.ts

@@ -1,5 +1,10 @@
 import Handsontable from 'handsontable'
-import { makeNumberFormatRenderer } from './renderers.utils'
+import {
+  makeNumberFormatRenderer,
+  errorRenderer,
+  noSpinnerRenderer,
+  spinnerRenderer
+} from './renderers.utils'
 
 describe('makeNumberFormatRenderer', () => {
   it('renders a numeric cell as EUR currency without changing the value', () => {
@@ -86,3 +91,67 @@ describe('makeNumberFormatRenderer', () => {
     container.remove()
   })
 })
+
+/**
+ * DOM-injection reproduction mirroring the editor's cell-render cycle.
+ * During dynamic cell validation the editor applies one of the status
+ * renderers to a cell via setCellMeta + hot.render().  Those renderers paint
+ * the cell value with td.innerHTML, so a value containing markup is injected
+ * and executed (the <img onerror> fires in the browser).  The value can come
+ * straight from a dataset row served by the getdata stored program, or from a
+ * typed edit.  These fail on the vulnerable implementation and pass once the
+ * renderer escapes the value.
+ */
+describe('grid cell renderers do not inject raw HTML', () => {
+  const maliciousValue = '<img src=x onerror=alert(1)>'
+
+  // Seed a real Handsontable grid with the payload as a loaded cell value,
+  // then apply the given status renderer and render — exactly the sequence the
+  // editor uses during the dynamic-validation cycle.
+  const renderWith = (
+    renderer: (
+      i: any,
+      td: any,
+      r: number,
+      c: number,
+      p: any,
+      v: any,
+      cp: any
+    ) => any
+  ) => {
+    const container = document.createElement('div')
+    document.body.appendChild(container)
+    const hot = new Handsontable(container, {
+      data: [{ SOME_CHAR: maliciousValue }],
+      columns: [{ data: 'SOME_CHAR', type: 'text' }],
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+    hot.setCellMeta(0, 0, 'renderer', renderer)
+    hot.render()
+    const td: HTMLTableCellElement | null = hot.getCell(0, 0)
+    hot.destroy()
+    container.remove()
+    return td
+  }
+
+  // A vulnerable renderer turns the value into a real <img> element with an
+  // onerror handler (proven by the browser firing alert(1)).  A safe
+  // renderer leaves no such element.  Asserting on the parsed DOM rather
+  // than the raw string avoids false passes from browser attribute normalising.
+  const assertNoInjectedElement = (td: HTMLTableCellElement | null) => {
+    expect(td?.querySelector('img[onerror]')).toBeNull()
+  }
+
+  it('noSpinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(noSpinnerRenderer))
+  })
+
+  it('errorRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(errorRenderer))
+  })
+
+  it('spinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(spinnerRenderer))
+  })
+})

client/src/app/editor/utils/renderers.utils.ts

@@ -1,5 +1,23 @@
 import Handsontable from 'handsontable'
 
+/**
+ * Returns string-safe text of any value so it can be assigned to innerHTML.
+ * The cell values painted by the status renderers are user/DB-controlled,
+ * so they must never be parsed as HTML by the browser — escaping turns any
+ * embedded markup into inert text.
+ */
+const escapeHtml = (value: any): string =>
+  String(value ?? '').replace(/[&<>"']/g, (char) => {
+    const entities: Record<string, string> = {
+      '&': '&amp;',
+      '<': '&lt;',
+      '>': '&gt;',
+      '"': '&quot;',
+      "'": '&#39;'
+    }
+    return entities[char]
+  })
+
 /**
  * Builds a display-only HOT renderer that formats numeric cell values using
  * Intl.NumberFormat. The stored/submitted value is never changed — only the
@@ -67,9 +85,7 @@ export const errorRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
+  td.innerHTML = `${escapeHtml(value)} <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
 
   return td
 }
@@ -89,7 +105,7 @@ export const noSpinnerRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = value ? value : ''
+  td.innerHTML = escapeHtml(value)
 
   return td
 }
@@ -110,9 +126,7 @@ export const spinnerRenderer = (
 ) => {
   addDarkClass(td)
 
-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <span class="spinner spinner-sm vertical-align-middle"></span>`
+  td.innerHTML = `${escapeHtml(value)} <span class="spinner spinner-sm vertical-align-middle"></span>`
 
   return td
 }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "dcfrontend",
-  "version": "7.14.0",
+  "version": "7.14.1",
   "description": "Data Controller",
   "devDependencies": {
     "@nogoo9/gitleaks": "8.30.1-post.2",

sas/mocks/sasjs/services/admin/makedata.js

@@ -277,9 +277,62 @@ const tables = {
       { tx_from: 0, tx_to: 127490111999, xlmap_id: 'SAMPLE', xlmap_range_id: 'data', xlmap_sheet: '/1', xlmap_start: 'ABSOLUTE B13', xlmap_finish: 'ABSOLUTE E16' }
     ],
     MPE_X_CATALOG: [],
+    MPE_X_SEARCH: [],
     MPE_USERS: []
 }
 
+// ─── MPE_X_SEARCH: viewer full-table-search demo table ───────────────────────
+// Deterministic, obviously fictional deep-sea survey data (no real sites,
+// vessels or people).  Laid out site-major / vessel-minor so every count in
+// the full-table-search Cypress spec can be derived by hand:
+//   n = rep * 50 + siteIdx * 5 + vesselIdx   (0-based, 1000 rows)
+//   -> 20 rows per site, 100 rows per site block, 200 rows per vessel
+const searchSites = [
+    'Nereid Trench', 'Calypso Trench', 'Halcyon Trench', 'Amphitrite Basin',
+    'Kraken Hollow', 'Erebus Deep', 'Thalassa Shelf', 'Oceanus Rise',
+    'Selkie Bank', 'Vostok Deep'
+]
+const searchVessels = ['RV Pelagia', 'RV Aurora', 'RV Selkie', 'RV Cormorant', 'RV Halcyon']
+const searchSpecies = [
+    'Vampire squid', 'Barreleye fish', 'Dumbo octopus', 'Yeti crab',
+    'Giant isopod', 'Ghost shark', 'Comb jelly', 'Sea pig',
+    'Glass sponge', 'Anglerfish'
+]
+// Rows carrying the mid-field marker in NOTES (searched as a partial string)
+const siphonophoreRows = [137, 500, 862]
+// Rows sharing the repeated exact SAMPLE_COUNT (searched as a number)
+const sampleCountRows = [220, 780]
+
+tables.MPE_X_SEARCH = []
+for (let n = 0; n < 1000; n++) {
+    const rep = Math.floor(n / 50)
+    const siteIdx = Math.floor((n % 50) / 5)
+    const vesselIdx = n % 5
+    const site = searchSites[siteIdx]
+    const depth = 1000 + (n % 90) * 37
+    tables.MPE_X_SEARCH.push({
+        PRIMARY_KEY_FIELD: n + 1001,
+        SITE_NAME: site,
+        VESSEL: searchVessels[vesselIdx],
+        SAMPLE_COUNT: sampleCountRows.includes(n) ? 4210 : 1 + (n % 6),
+        DEPTH_M: depth,
+        SPECIES: searchSpecies[(siteIdx + vesselIdx) % searchSpecies.length],
+        CRUISE_DATE: 20000 + (n % 3650),
+        EXPEDITION_ID: 'EXP-' + String(n + 1).padStart(4, '0'),
+        NOTES: 'Leg ' + (rep + 1) + ' of the ' + site + ' survey. ' +
+            (siphonophoreRows.includes(n)
+                ? 'A bioluminescent siphonophore was filmed here.'
+                : 'No anomalies recorded.') +
+            ' Depth soundings logged at ' + depth + 'm.'
+    })
+}
+
+// Register the demo table like any other editable table (buskey drives the
+// viewer's PK detection).
+tables.MPE_TABLES.push(
+    { tx_from: 0, tx_to: 127490111999, libref: dcLibref, dsn: 'MPE_X_SEARCH', num_of_approvals_required: 1, loadtype: 'UPDATE', buskey: 'PRIMARY_KEY_FIELD', notes: 'Full table search demo table' }
+)
+
 // Column metadata for each table, extracted from mpe_makedatamodel.sas
 // Each entry: { name, type (N/C), length, format, label, notnull (bool) }
 const schema = {
@@ -664,6 +717,17 @@ const schema = {
         { name: 'SOME_SHORTNUM', type: 'N', length: 4, format: '', label: '', notnull: false },
         { name: 'SOME_BESTNUM', type: 'N', length: 8, format: 'best.', label: '', notnull: false }
     ],
+    MPE_X_SEARCH: [
+        { name: 'PRIMARY_KEY_FIELD', type: 'N', length: 8, format: '', label: 'Primary Key', notnull: true },
+        { name: 'SITE_NAME', type: 'C', length: 32, format: '', label: 'Survey Site', notnull: false },
+        { name: 'VESSEL', type: 'C', length: 32, format: '', label: 'Survey Vessel', notnull: false },
+        { name: 'SAMPLE_COUNT', type: 'N', length: 8, format: '', label: 'Samples Collected', notnull: false },
+        { name: 'DEPTH_M', type: 'N', length: 8, format: '', label: 'Depth (metres)', notnull: false },
+        { name: 'SPECIES', type: 'C', length: 32, format: '', label: 'Species Observed', notnull: false },
+        { name: 'CRUISE_DATE', type: 'N', length: 8, format: 'date9.', label: 'Cruise Date', notnull: false },
+        { name: 'EXPEDITION_ID', type: 'C', length: 12, format: '', label: 'Expedition', notnull: false },
+        { name: 'NOTES', type: 'C', length: 512, format: '', label: 'Survey Notes', notnull: false }
+    ],
     MPE_XLMAP_DATA: [
         { name: 'LOAD_REF', type: 'C', length: 32, format: '', label: '', notnull: true },
         { name: 'XLMAP_ID', type: 'C', length: 32, format: '', label: '', notnull: true },

sas/mocks/sasjs/services/public/viewdata.js

@@ -4,10 +4,13 @@ let appLoc = nodePath.join(..._program.split('services')[0].split('/'))
 const sasjsRoot = nodePath.resolve(weboutPath, '..', '..', '..')
 const driveRoot = nodePath.resolve(sasjsRoot, 'drive')
 const dcLibref = 'DC_JSLIB'
-const dataDir = nodePath.resolve(driveRoot, 'files', appLoc, 'data', dcLibref)
+const mpeDataDir = nodePath.resolve(driveRoot, 'files', appLoc, 'data', dcLibref)
 
 eval(fs.readFileSync(nodePath.resolve(driveRoot, 'files', appLoc, 'services', 'dcMockUtils.js'), 'utf8'))
-const loadTableData = makeTableLoader(dataDir)
+
+// Row cap for the viewer - mirrors DC_MAXOBS_WEBVIEW in viewdata.sas, and the
+// MAXROWS value reported in sasparams.
+const maxObsWebView = 500
 
 // ─── Parse input ──────────────────────────────────────────────────────────────
 
@@ -16,11 +19,65 @@ let libds = _sctRow.LIBDS || ''
 const libref = libds.split('.')[0] || dcLibref
 const dsn = libds.split('.')[1] || ''
 
+// FILTER_RK (matching viewdata.sas: if filter_rk le 0 then filter_rk=-1)
+const filterRk = Number(colVal(_sctRow, 'FILTER_RK')) > 0 ? Number(colVal(_sctRow, 'FILTER_RK')) : -1
+
+// SEARCHTYPE / SEARCHVAL - only CHAR or NUM triggers a search, anything else
+// (blank, NONE) is a normal view.  Search values are stripped of the same
+// characters viewdata.sas removes before handing them to mp_searchdata.
+const rawSearchType = String(colVal(_sctRow, 'SEARCHTYPE') || '').toUpperCase()
+const searchType = (rawSearchType === 'CHAR' || rawSearchType === 'NUM') ? rawSearchType : ''
+const rawSearchVal = colVal(_sctRow, 'SEARCHVAL')
+let searchVal = (rawSearchVal === undefined || rawSearchVal === null) ? '' : String(rawSearchVal)
+if (searchType) searchVal = searchVal.replace(/[%&;"]/g, '')
+
+// ─── Load the target table ───────────────────────────────────────────────────
+
+const dataDir = libDataDir(libref)
+const loadTableData = makeTableLoader(dataDir)
+const loadMpeTable = makeTableLoader(mpeDataDir)
+
+let tableData = loadTableData(dsn)
+if (!tableData && dsn) {
+    const found = loadTableAnyLib(dsn)
+    if (found) tableData = found.data
+}
+
+// ─── Apply the stored filter (mpe_filtermaster, VIEW mode) ───────────────────
+// Mirrors viewdata.sas: %mpe_filtermaster(VIEW,&libds,filter_rk=&filter_rk)
+// Row Level Security and any saved filter are applied here, before the search.
+
+const { predicate: filterPredicate, filterText } = mpeFilterMaster({
+    mode: 'VIEW',
+    libds: (libref + '.' + dsn).toUpperCase(),
+    filterRk: filterRk,
+    dataDir: mpeDataDir,
+    columns: tableData && tableData.columns ? tableData.columns : []
+})
+
+// ─── Search predicate (mp_searchdata.sas) ────────────────────────────────────
+// mp_searchdata builds one WHERE clause covering every column of the table:
+//   or ("COL1"n ? "the string")   /* character columns: case sensitive CONTAINS */
+//   or ("COL2"n = the number)     /* numeric columns: exact equality           */
+
+function rowMatchesSearch(row, columns, type, value) {
+    for (const col of columns) {
+        const val = colVal(row, col.name)
+        if (val === undefined || val === null) continue
+        if (type === 'CHAR') {
+            if (col.type !== 'N' && String(val).includes(value)) return true
+        } else {
+            if (col.type === 'N' && Number(val) === Number(value)) return true
+        }
+    }
+    return false
+}
+
 // ─── Build response ──────────────────────────────────────────────────────────
 
 // Look up PK fields from MPE_TABLES (mirrors mp_getpk).
 // MPE_AUDIT is a standard DC audit table whose PK is always the first 5 columns.
-const mpeTables = loadTableData('MPE_TABLES')
+const mpeTables = loadMpeTable('MPE_TABLES')
 let pkFields = ' '
 if (mpeTables && mpeTables.rows) {
     const reg = mpeTables.rows.find(r => r.libref === libref && r.dsn === dsn)
@@ -32,8 +89,6 @@ if (dsn.toUpperCase() === 'MPE_AUDIT' && pkFields.trim() === '') {
     pkFields = 'LOAD_REF LIBREF DSN KEY_HASH TGTVAR_NM'
 }
 
-const tableData = loadTableData(dsn)
-
 let cols = []
 let viewdata = []
 let vars = {}
@@ -53,8 +108,26 @@ if (tableData && tableData.columns) {
         }
     })
 
+    // Rows in scope: the stored filter is applied first, then the search
+    // (matching viewdata.sas, which searches a filtered view of the table).
+    const filteredRows = (tableData.rows || []).filter(r => filterPredicate(r))
+
+    let rows
+    if (searchType) {
+        const matches = filteredRows.filter(row => rowMatchesSearch(row, tableData.columns, searchType, searchVal))
+        // mp_searchdata caps its output at outobs=&DC_MAXOBS_WEBVIEW, and
+        // viewdata.sas then keeps only `if _n_<&DC_MAXOBS_WEBVIEW` of them.
+        nobs = Math.min(matches.length, maxObsWebView)
+        rows = matches.slice(0, maxObsWebView - 1)
+    } else {
+        // proc sql select count(*) ... where <filter> (uncapped)
+        nobs = filteredRows.length
+        // data work.viewdata; set &libds; where <filter>; if _n_>&DC_MAXOBS_WEBVIEW then stop;
+        rows = filteredRows.slice(0, maxObsWebView)
+    }
+
     // viewdata rows: all values as strings (formatted, SAS proc-print style)
-    viewdata = (tableData.rows || []).map(r => {
+    viewdata = rows.map(r => {
         const row = {}
         for (const col of tableData.columns) {
             const lcName = col.name.toLowerCase()
@@ -94,7 +167,9 @@ if (tableData && tableData.columns) {
     // the client can derive the column set even when empty (the UI iterates
     // viewdata[0] to build the grid headers). Numeric columns get "."
     // (SAS missing), character columns get "".
-    if (viewdata.length === 0 && tableData.columns && tableData.columns.length > 0) {
+    // A search that matches nothing sends no rows at all (the client shows
+    // its "no data found" panel instead of an empty grid).
+    if (viewdata.length === 0 && !searchType && tableData.columns.length > 0) {
         const emptyRow = {}
         for (const col of tableData.columns) emptyRow[col.name] = col.type === 'N' ? '.' : ''
         viewdata = [emptyRow]
@@ -105,8 +180,6 @@ if (tableData && tableData.columns) {
     for (const col of tableData.columns) {
         vars[col.name] = sasVarsEntry(col)
     }
-
-    nobs = tableData.rows ? tableData.rows.length : 0
 }
 
 webOutOpen()
@@ -118,8 +191,8 @@ webOutObj([{ ODS_TABLE: 'ATTRIBUTES', NAME: 'Data Set Name', VALUE: libref + '.'
     { ODS_TABLE: 'ATTRIBUTES', NAME: 'Variables', VALUE: String(cols.length) },
     { ODS_TABLE: 'ATTRIBUTES', NAME: 'Engine', VALUE: 'V9' }], 'dsmeta')
 webOutObj([], 'query')
-webOutObj([{ TABLEURI: ' ', TABLENAME: dsn, FILTER_TEXT: ' ',
-    PK_FIELDS: pkFields, NOBS: nobs, VARS: cols.length, MAXROWS: 500 }], 'sasparams')
+webOutObj([{ TABLEURI: ' ', TABLENAME: dsn, FILTER_TEXT: filterText === '' ? ' ' : filterText,
+    PK_FIELDS: pkFields, NOBS: nobs, VARS: cols.length, MAXROWS: maxObsWebView }], 'sasparams')
 webOutObj([], 'versions')
 webOutObj(viewdata, 'viewdata', { vars })
 webOutClose()

sas/package-lock.json

@@ -6,8 +6,8 @@
     "": {
       "name": "dc-sas",
       "dependencies": {
-        "@sasjs/cli": "4.20.3",
-        "@sasjs/core": "5.2.7"
+        "@sasjs/cli": "4.20.4",
+        "@sasjs/core": "5.2.8"
       }
     },
     "node_modules/@asamuzakjp/css-color": {
@@ -203,13 +203,13 @@
       }
     },
     "node_modules/@sasjs/cli": {
-      "version": "4.20.3",
-      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.3.tgz",
-      "integrity": "sha512-ZI+7VoBi0bgjpGkmKTbS8f5p1pQKgE2sv4NKTtjQ9Qg3iltjdQK8IoUQXEn4IJI1pa3wCuaIrCoYjUmCS6VYMQ==",
+      "version": "4.20.4",
+      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.4.tgz",
+      "integrity": "sha512-hKE002Dm9AzWU62qpRyYpjQv8BIXG951Z0ToNdG9J1wvvJNYH7h6rGv7kdQMbgvXff/KOQvVdnGJZ023SrG+Jw==",
       "license": "ISC",
       "dependencies": {
         "@sasjs/adapter": "^4.19.0",
-        "@sasjs/core": "5.2.4",
+        "@sasjs/core": "5.2.9",
         "@sasjs/lint": "2.5.0",
         "@sasjs/utils": "3.6.2",
         "chalk": "4.1.2",
@@ -236,28 +236,15 @@
       }
     },
     "node_modules/@sasjs/cli/node_modules/@sasjs/core": {
-      "version": "5.2.4",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.4.tgz",
-      "integrity": "sha512-VAY62Zl8xkIfCr73jXFFgqGX4lab9htsZRmCIIttwbBR/T/LUD0N0g/mcfaVp/W+UvHHT8dqntNlm8SX0u11rQ==",
+      "version": "5.2.9",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.9.tgz",
+      "integrity": "sha512-R5wtJF0ANHchaURqTF2t1vRXrhUJ+uDfj21ewzdOq4Lp14CWW22+yYeRQooY4QOpyHUyFDjReA5q6SoJt2Tz1g==",
       "license": "MIT"
     },
-    "node_modules/@sasjs/cli/node_modules/prompts": {
-      "version": "2.4.2",
-      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
-      "integrity": "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q==",
-      "license": "MIT",
-      "dependencies": {
-        "kleur": "^3.0.3",
-        "sisteransi": "^1.0.5"
-      },
-      "engines": {
-        "node": ">= 6"
-      }
-    },
     "node_modules/@sasjs/core": {
-      "version": "5.2.7",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.7.tgz",
-      "integrity": "sha512-GH2k6pV/Ik2E1tYNmGH5UiPQIVfcQ04MrPL2UPTHMfcuIK5Sgo1Gb+7GhX0IwzQwymM1BU5wodSW5X/N2GLXZw==",
+      "version": "5.2.8",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.8.tgz",
+      "integrity": "sha512-+K8tXTrsw52NR1y5MFHW0+Jf9euL+GukwqA79FIHHu4Oj+BhHBm1uZiyB5tPrvwl3V8mY1CQhMOT83//bV+jWg==",
       "license": "MIT"
     },
     "node_modules/@sasjs/lint": {
@@ -306,6 +293,19 @@
         "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
+    "node_modules/@sasjs/utils/node_modules/prompts": {
+      "version": "2.4.1",
+      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
+      "integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
+      "license": "MIT",
+      "dependencies": {
+        "kleur": "^3.0.3",
+        "sisteransi": "^1.0.5"
+      },
+      "engines": {
+        "node": ">= 6"
+      }
+    },
     "node_modules/@types/fs-extra": {
       "version": "11.0.4",
       "resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-11.0.4.tgz",
@@ -880,9 +880,9 @@
       }
     },
     "node_modules/fastq": {
-      "version": "1.20.1",
-      "resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.1.tgz",
-      "integrity": "sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw==",
+      "version": "1.20.3",
+      "resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.3.tgz",
+      "integrity": "sha512-XKv5nnLs6nLF71NgiKJLIZFLkPyIEuOselLG7ujZnGrRfQK8HpvY+WqKhAJUAdLomwVHErVS4LfxFlPq0/FTAw==",
       "license": "ISC",
       "dependencies": {
         "reusify": "^1.0.4"
@@ -1772,9 +1772,9 @@
       }
     },
     "node_modules/prompts": {
-      "version": "2.4.1",
-      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.1.tgz",
-      "integrity": "sha512-EQyfIuO2hPDsX1L/blblV+H7I0knhgAd82cVneCwcdND9B8AuCDuRcBH6yIcG4dFzlOUqbazQqwGjx5xmsNLuQ==",
+      "version": "2.4.2",
+      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
+      "integrity": "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q==",
       "license": "MIT",
       "dependencies": {
         "kleur": "^3.0.3",

sas/package.json

@@ -30,8 +30,8 @@
   },
   "private": true,
   "dependencies": {
-    "@sasjs/cli": "4.20.3",
-    "@sasjs/core": "5.2.7"
+    "@sasjs/cli": "4.20.4",
+    "@sasjs/core": "5.2.8"
   },
   "overrides": {
     "nanoid": "3.3.18"

sas/sasjs/services/editors/loadfile.sas

@@ -18,6 +18,7 @@
   @li mpe_loader.sas
   @li mp_cleancsv.sas
   @li mp_binarycopy.sas
+  @li mp_validatecol.sas
   @li mpeinit.sas
 
   @version 9.2
@@ -113,9 +114,21 @@ data _null_;
   end;
   else call symputx('libds',libds);
   call symputx('is_fmt',is_fmt);
+  /* validate libds to prevent code injection */
+  %mp_validatecol(LIBDS,LIBDS,is_libds)
+  if is_libds=0 then do;
+    putlog 'ERR' 'OR: Invalid libds:' libds;
+    call symputx('bad_libds',1);
+  end;
+  else call symputx('bad_libds',0);
   putlog (_all_)(=);
 run;
 
+%mp_abort(iftrue= (&bad_libds=1)
+  ,mac=&_program
+  ,msg=%str(Invalid libds supplied)
+)
+
 /* check that the user has the requisite access */
 %mpe_getgroups(user=&user,outds=groups)
 

sas/sasjs/services/editors/stagedata.sas

@@ -26,6 +26,7 @@
   @li mp_binarycopy.sas
   @li mp_cntlout.sas
   @li mp_ds2csv.sas
+  @li mp_validatecol.sas
   @li mf_getplatform.sas
   @li removecolsfromwork.sas
   @li mpeinit.sas
@@ -66,9 +67,21 @@ data _null_;
   end;
   else call symputx('libds',libds);
   call symputx('is_fmt',is_fmt);
+  /* validate libds to prevent code injection */
+  %mp_validatecol(LIBDS,LIBDS,is_libds)
+  if is_libds=0 then do;
+    putlog 'ERR' 'OR: Invalid libds:' libds;
+    call symputx('bad_libds',1);
+  end;
+  else call symputx('bad_libds',0);
   putlog (_all_)(=);
 run;
 
+%mp_abort(iftrue= (&bad_libds=1)
+  ,mac=&_program
+  ,msg=%str(Invalid libds supplied)
+)
+
 %mp_cntlout(
   iftrue=(&is_fmt=1)
   ,libcat=&orig_libds

sas/sasjs/services/public/viewdata.sas

@@ -8,10 +8,16 @@
   <h4> Service Inputs </h4>
 
   <h5> SASCONTROLTABLE </h5>
-  |LIBDS:$41.|FILTER_RK:$5.|SEARCHTYPE:$4|SEARCHVAL:$1000
+  |LIBDS:$41.|FILTER_RK:best.|SEARCHTYPE:$4|SEARCHVAL:$100|
   |---|---|---|---
   |DC258467.MPE_X_TEST|-1|CHAR|Some String|
 
+  No input column may be longer than the length declared in this service -
+  LIBDS 41, SEARCHTYPE 4, SEARCHVAL 100.  An input column longer than the
+  declared length stops the step with "Multiple lengths were specified for
+  the variable ...", so the SASjs adapter sends the $char form (eg $char4.),
+  which also preserves any leading blanks.
+
   <h4> Service Outputs </h4>
 
   <h5> cols </h5>
@@ -271,7 +277,7 @@ run;
       %let dsobs=%mf_nobs(MPSEARCH.vwsearch);
       data viewdata;
         set MPSEARCH.vwsearch;
-        if _n_<&DC_MAXOBS_WEBVIEW;
+        if _n_<=&DC_MAXOBS_WEBVIEW;
       run;
     %end;
     %else %do;

sas/sasjs/services/public/viewdata.test.3.sas

@@ -0,0 +1,123 @@
+/**
+  @file
+  @brief testing public/viewdata service full table search semantics
+  @details The viewer's search box runs a full table search through
+  %mp_searchdata.  A character search is a partial (CONTAINS) match against
+  every character column of the table and is case sensitive; a numeric search
+  is an exact match against every numeric column.
+
+  The fixture below holds both cases of the same substring so the assertions
+  can show the behaviour in both directions - a value with a capital letter is
+  not found by a lowercase search and vice versa.
+
+  <h4> SAS Macros </h4>
+  @li mf_getuniquefileref.sas
+  @li mf_getuniquelibref.sas
+  @li mp_assert.sas
+  @li mx_execute.sas
+
+**/
+
+%let _program=&appLoc/services/public/viewdata;
+
+/* deterministic fixture - mixed case, and a value that only exists in the
+  second character column, so the search must cover every character column */
+data &dclib..mpe_x_search;
+  length NAME $32 NOTES $64;
+  name='Selkie';        qty=12345; notes='grey seal folklore';   output;
+  name='Vampire squid'; qty=23456; notes='deep sea cephalopod';  output;
+  name='siphonophore';  qty=34567; notes='colonial organism';    output;
+  name='Smithson';      qty=45678; notes='capital S surname';    output;
+  name='Goldsmith';     qty=56789; notes='lowercase substring';  output;
+run;
+
+/**
+  * Run one full table search and assert the row count the viewer would show.
+  */
+%macro search(searchtype,searchval,expected,desc);
+%local f1 outlib nobs;
+%let f1=%mf_getuniquefileref();
+%let outlib=%mf_getuniquelibref(prefix=web);
+data _null_;
+  file &f1 termstr=crlf;
+  put 'LIBDS:$char41. FILTER_RK:best. SEARCHTYPE:$char4. SEARCHVAL:$char100.';
+  put "&dclib..MPE_X_SEARCH,-1,&searchtype,&searchval";
+run;
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:SASControlTable ,
+  outlib=&outlib,
+  mdebug=&sasjs_mdebug
+)
+%let nobs=0;
+data _null_;
+  set &outlib..sasparams;
+  putlog (_all_)(=);
+  call symputx('nobs',nobs);
+run;
+%mp_assert(
+  iftrue=(&nobs=&expected),
+  desc=&desc (&searchtype "%superq(searchval)" returned &nobs rows, expected &expected),
+  outds=work.test_results
+)
+%mend search;
+
+/* a character search matches part of a value, in the same case */
+%search(CHAR,Selkie,1,Checking a same-case partial match returns the row)
+%search(CHAR,quid,1,Checking a partial match on the middle of a value)
+%search(CHAR,seal,1,Checking the search covers every character column)
+
+/* the same term in another case does not match */
+%search(CHAR,selkie,0,Checking a lowercase search does not match a capital S)
+%search(CHAR,SELKIE,0,Checking an uppercase search does not match mixed case)
+%search(CHAR,Quid,0,Checking a capital Q does not match a lowercase value)
+
+/* a search in the case the value is stored in finds it, whether the match
+  starts the value or sits inside it */
+%search(CHAR,Smith,1,Checking Smith finds Smithson)
+%search(CHAR,smith,1,Checking smith finds Goldsmith)
+
+/* a numeric search is an exact match, not a partial one */
+%search(NUM,12345,1,Checking an exact numeric search returns the row)
+%search(NUM,1234,0,Checking a partial numeric search returns nothing)
+
+/* a search that matches more rows than the web view cap returns the cap,
+  and reports the same number of rows as it returns */
+data &dclib..mpe_x_cap;
+  length txt $24;
+  do i=1 to 600;
+    txt='needle'!!cats(i);
+    output;
+  end;
+run;
+
+%let f1=%mf_getuniquefileref();
+%let outlib=%mf_getuniquelibref(prefix=web);
+data _null_;
+  file &f1 termstr=crlf;
+  put 'LIBDS:$char41. FILTER_RK:best. SEARCHTYPE:$char4. SEARCHVAL:$char100.';
+  put "&dclib..MPE_X_CAP,-1,CHAR,needle";
+run;
+%mx_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputfiles=&f1:SASControlTable ,
+  outlib=&outlib,
+  mdebug=&sasjs_mdebug
+)
+%let capnobs=0;
+%let capmax=0;
+data _null_;
+  set &outlib..sasparams;
+  putlog (_all_)(=);
+  call symputx('capnobs',nobs);
+  call symputx('capmax',maxrows);
+run;
+%let caprows=0;
+proc sql noprint;
+  select count(*) into :caprows from &outlib..viewdata;
+quit;
+%mp_assert(
+  iftrue=(&caprows=&capnobs and &capnobs gt 0 and &capnobs le &capmax),
+  desc=Checking a search beyond the cap returns as many rows as it reports (&capnobs reported, &caprows returned, cap &capmax),
+  outds=work.test_results
+)