Merge pull request 'fix(security): escape cell values and harden libds inputs' (#319 ) from security/harden-xss-and-inputs into fix/viya-test-suite-and-refreshcatalog-json

Reviewed-on: #319
refactor(deploy): remove vestigial client credential fields
2026-09-14 16:20:07 +00:00 · 2026-09-14 16:45:20 +01:00 · 2026-09-14 16:45:13 +01:00 · 2026-09-14 16:44:59 +01:00 · 2026-09-14 13:25:32 +01:00 · 2026-09-14 12:15:38 +00:00
19 changed files with 577 additions and 262 deletions
@@ -3,6 +3,42 @@
 # Using `--silent` helps for showing any errs in the first line of the response
 # The first line is picked up by the VS Code GIT UI popup when rc is not 0

+# Scan staged changes for secrets before anything else runs.
+# The binary is pinned by the @nogoo9/gitleaks devDependency and lands
+# in node_modules/.bin after npm i. Commits are blocked until it is
+# installed so no clone silently skips the secrets scan.
+gitleaks_bin=node_modules/.bin/gitleaks
+
+if [ ! -x "$gitleaks_bin" ]; then
+    echo "❌ gitleaks not found - run 'npm i' to install it"
+    exit 1
+fi
+
+if ! "$gitleaks_bin" protect --staged --redact; then
+    echo "❌ gitleaks detected a potential secret in your staged changes"
+    echo "Remove the secret, or add a .gitleaksignore entry if it is a false positive"
+    exit 1
+fi
+
+## Avoid large commits
+# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
+size_limit=$((2 * 1024 * 1024)) # 2MB
+# Sum the sizes of the blobs this commit would introduce: the staged
+# added/copied/modified/renamed/typechanged entries in the index.
+# https://git-scm.com/docs/git-cat-file#Documentation/git-cat-file.txt---batch-check
+# Renames are treated as add+delete (--no-renames) and pure mode changes
+# (same blob, different permissions) add no new blob, so both are safe.
+commit_size=$(git diff --cached --raw --no-renames --diff-filter=ACMRT |
+  awk '$3 != $4 { print $4 }' |
+  git cat-file --batch-check='%(objectsize)' |
+  awk '{ s += $1 } END { print s + 0 }')
+
+if [ "$commit_size" -ge "$size_limit" ]; then
+  echo "Commit size is too large: $commit_size > $size_limit"
+  echo "Force commit using --no-verify"
+  exit 1
+fi
+
 if npm run --silent lint:check:silent ; then
    exit 0
 else
@@ -10,14 +46,3 @@ else
    echo "❌ Prettier check failed! We ran lint:fix for you. Please add & commit again."
    exit 1
 fi
-
-## Avoid large commits
-# https://www.backblaze.com/blog/how-many-bytes-are-in-a-megabyte-really/
-size_limit=$((2 * 2**20)) # 2mbs
-# https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---disk-usage
-commit_size=$(git rev-list --disk-usage HEAD^..HEAD)
-test "$commit_size" -lt "$size_limit" || (
-  echo "Commit size is too large: $commit_size > $size_limit"
-  echo "Force commit using --no-verify"
-  exit 1
-)
@@ -4,6 +4,21 @@
 ## Workflow guidelines
 [Wiki Page](https://git.datacontroller.io/dc/dc/wiki/Git-Workflow)

+## Git hooks
+
+The repo ships its own git hooks in [`.git-hooks/`](./.git-hooks):
+
+- `pre-commit` - scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)
+- `commit-msg` - verifies the commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) standard
+
+The repo `.npmrc` sets `ignore-scripts=true`, so the `prepare` and `postinstall` lifecycle scripts that would normally set `core.hooksPath` during `npm i` never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
+
+```bash
+git config core.hooksPath ./.git-hooks
+```
+
+The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
+
 ## Dependencies that requires licences

 [SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)
@@ -21,8 +21,6 @@ export class DeployComponent implements OnInit {
  public step: number = 0
  public adminGroups: any = []

-  public client_id: string = ''
-  public client_secret: string = ''
  public appLoc: string = ''
  public dcPath: string = ''
  public selectedAdminGroup: string = ''
@@ -52,9 +50,6 @@ export class DeployComponent implements OnInit {
    this.sasJs = this.sasService.getSasjsInstance()
    this.sasJsConfig = this.sasService.getSasjsConfig()
    this.appLoc = this.dcAdapterSettings?.appLoc || ''
-    this.client_id = localStorage.getItem('deploy_client_id') || ''
-    this.client_secret = localStorage.getItem('deploy_secret_key') || ''
-    this.dcPath = localStorage.getItem('deploy_dc_loc') || ''
  }

  ngOnInit() {
@@ -1,5 +1,10 @@
 import Handsontable from 'handsontable'
-import { makeNumberFormatRenderer } from './renderers.utils'
+import {
+  makeNumberFormatRenderer,
+  errorRenderer,
+  noSpinnerRenderer,
+  spinnerRenderer
+} from './renderers.utils'

 describe('makeNumberFormatRenderer', () => {
  it('renders a numeric cell as EUR currency without changing the value', () => {
@@ -86,3 +91,67 @@ describe('makeNumberFormatRenderer', () => {
    container.remove()
  })
 })
+
+/**
+ * DOM-injection reproduction mirroring the editor's cell-render cycle.
+ * During dynamic cell validation the editor applies one of the status
+ * renderers to a cell via setCellMeta + hot.render().  Those renderers paint
+ * the cell value with td.innerHTML, so a value containing markup is injected
+ * and executed (the <img onerror> fires in the browser).  The value can come
+ * straight from a dataset row served by the getdata stored program, or from a
+ * typed edit.  These fail on the vulnerable implementation and pass once the
+ * renderer escapes the value.
+ */
+describe('grid cell renderers do not inject raw HTML', () => {
+  const maliciousValue = '<img src=x onerror=alert(1)>'
+
+  // Seed a real Handsontable grid with the payload as a loaded cell value,
+  // then apply the given status renderer and render — exactly the sequence the
+  // editor uses during the dynamic-validation cycle.
+  const renderWith = (
+    renderer: (
+      i: any,
+      td: any,
+      r: number,
+      c: number,
+      p: any,
+      v: any,
+      cp: any
+    ) => any
+  ) => {
+    const container = document.createElement('div')
+    document.body.appendChild(container)
+    const hot = new Handsontable(container, {
+      data: [{ SOME_CHAR: maliciousValue }],
+      columns: [{ data: 'SOME_CHAR', type: 'text' }],
+      licenseKey: 'non-commercial-and-evaluation'
+    })
+    hot.render()
+    hot.setCellMeta(0, 0, 'renderer', renderer)
+    hot.render()
+    const td: HTMLTableCellElement | null = hot.getCell(0, 0)
+    hot.destroy()
+    container.remove()
+    return td
+  }
+
+  // A vulnerable renderer turns the value into a real <img> element with an
+  // onerror handler (proven by the browser firing alert(1)).  A safe
+  // renderer leaves no such element.  Asserting on the parsed DOM rather
+  // than the raw string avoids false passes from browser attribute normalising.
+  const assertNoInjectedElement = (td: HTMLTableCellElement | null) => {
+    expect(td?.querySelector('img[onerror]')).toBeNull()
+  }
+
+  it('noSpinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(noSpinnerRenderer))
+  })
+
+  it('errorRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(errorRenderer))
+  })
+
+  it('spinnerRenderer escapes rather than injecting the value', () => {
+    assertNoInjectedElement(renderWith(spinnerRenderer))
+  })
+})
@@ -1,5 +1,23 @@
 import Handsontable from 'handsontable'

+/**
+ * Returns string-safe text of any value so it can be assigned to innerHTML.
+ * The cell values painted by the status renderers are user/DB-controlled,
+ * so they must never be parsed as HTML by the browser — escaping turns any
+ * embedded markup into inert text.
+ */
+const escapeHtml = (value: any): string =>
+  String(value ?? '').replace(/[&<>"']/g, (char) => {
+    const entities: Record<string, string> = {
+      '&': '&amp;',
+      '<': '&lt;',
+      '>': '&gt;',
+      '"': '&quot;',
+      "'": '&#39;'
+    }
+    return entities[char]
+  })
+
 /**
 * Builds a display-only HOT renderer that formats numeric cell values using
 * Intl.NumberFormat. The stored/submitted value is never changed — only the
@@ -67,9 +85,7 @@ export const errorRenderer = (
 ) => {
  addDarkClass(td)

-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`
+  td.innerHTML = `${escapeHtml(value)} <cds-icon shape="exclamation-triangle" status="warning"></cds-icon>`

  return td
 }
@@ -89,7 +105,7 @@ export const noSpinnerRenderer = (
 ) => {
  addDarkClass(td)

-  td.innerHTML = value ? value : ''
+  td.innerHTML = escapeHtml(value)

  return td
 }
@@ -110,9 +126,7 @@ export const spinnerRenderer = (
 ) => {
  addDarkClass(td)

-  td.innerHTML = `${
-    value ? value.toString() : ''
-  } <span class="spinner spinner-sm vertical-align-middle"></span>`
+  td.innerHTML = `${escapeHtml(value)} <span class="spinner spinner-sm vertical-align-middle"></span>`

  return td
 }
@@ -1,14 +1,15 @@
 {
  "name": "dcfrontend",
-  "version": "7.12.0",
+  "version": "7.14.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "dcfrontend",
-      "version": "7.12.0",
+      "version": "7.14.0",
      "hasInstallScript": true,
      "devDependencies": {
+        "@nogoo9/gitleaks": "8.30.1-post.2",
        "@saithodev/semantic-release-gitea": "^2.1.0",
        "@semantic-release/changelog": "^6.0.3",
        "@semantic-release/commit-analyzer": "13.0.1",
@@ -119,6 +120,156 @@
        "node": ">= 8"
      }
    },
+    "node_modules/@nogoo9/gitleaks": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks/-/gitleaks-8.30.1-post.2.tgz",
+      "integrity": "sha512-WkQPfryKH8w9DEqJ1Bxw3ex9aXl0M8cxDDt1ATvyieaMvWiGamGXHNLZaJeqECInVfxNPEiZA4Few8Uwtuj1wQ==",
+      "dev": true,
+      "license": "MIT",
+      "bin": {
+        "gitleaks": "src/index.js"
+      },
+      "engines": {
+        "node": ">=18"
+      },
+      "optionalDependencies": {
+        "@nogoo9/gitleaks-darwin-arm64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-darwin-x64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-arm": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-arm64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-x32": "8.30.1-post.2",
+        "@nogoo9/gitleaks-linux-x64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-windows-arm64": "8.30.1-post.2",
+        "@nogoo9/gitleaks-windows-x32": "8.30.1-post.2",
+        "@nogoo9/gitleaks-windows-x64": "8.30.1-post.2"
+      }
+    },
+    "node_modules/@nogoo9/gitleaks-darwin-arm64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-darwin-arm64/-/gitleaks-darwin-arm64-8.30.1-post.2.tgz",
+      "integrity": "sha512-3j1zSvCKY7N4RnSl+oJrnrLZAY6HihD3uyi32qYrdIXH5hgVjmKiJhvGyoaV7DNw/4NdN3UsadrTM1u5M7iCuw==",
+      "cpu": [
+        "arm64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-darwin-x64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-darwin-x64/-/gitleaks-darwin-x64-8.30.1-post.2.tgz",
+      "integrity": "sha512-EbWRmrWdwRtzfO4ivYXA2ZKcf4aqVZFGQVu2HJjCFjRNryyMfou4Hw3tdZc5tRbdSE0yiwaAXJ8lPazApd3iGQ==",
+      "cpu": [
+        "x64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-arm": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-arm/-/gitleaks-linux-arm-8.30.1-post.2.tgz",
+      "integrity": "sha512-a5aZc5WQnBmNz5CCOPd0MT75csSBUEYMEVIHR6p56TOt4lj8h+IeQ5VTKG+f9j9nU+hmG2e//j4hLcbZZkCspw==",
+      "cpu": [
+        "arm"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-arm64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-arm64/-/gitleaks-linux-arm64-8.30.1-post.2.tgz",
+      "integrity": "sha512-XJzvhOEsqr6nJrBuLzDb+qBfdLjssKv/7gqQNhojd8LPM0/aEiC380kYuM9mFCXF3o3PpD4IXv/Jvemw5JJQGA==",
+      "cpu": [
+        "arm64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-x32": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-x32/-/gitleaks-linux-x32-8.30.1-post.2.tgz",
+      "integrity": "sha512-d20jRC6HCzc9J79Ti8JHFc3X3iJktF78NaUEg0EPGLHRsVpxVObtiCQAJzCgH0uXNMOVFrnsf/+dSDFwjTLU9Q==",
+      "cpu": [
+        "ia32"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-linux-x64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-linux-x64/-/gitleaks-linux-x64-8.30.1-post.2.tgz",
+      "integrity": "sha512-GSJ7RU0xuaF9Blf2AmDeytxyiZb4W7/Urd5Cw5LlNK+IF/C+Nd/Fp5pKbaAF4osdxADbCZ7d4XMYUMSdlpjQFA==",
+      "cpu": [
+        "x64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-windows-arm64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-arm64/-/gitleaks-windows-arm64-8.30.1-post.2.tgz",
+      "integrity": "sha512-cTu+rMASv3BuNJcPj57PhfiOHwapmOVH6s5PFBqAXzJomUnHdB/ZMrTDNseW78UZQQ/c0RPI3VXwieyMC23w7w==",
+      "cpu": [
+        "arm64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-windows-x32": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-x32/-/gitleaks-windows-x32-8.30.1-post.2.tgz",
+      "integrity": "sha512-MGMtKVEOf0TsXIGBu/1eGb94mAZRX2rzZmgNRYqzZBSYPnmyl3w1n4tic9e6rPOewyGfuxnweCDnu0Lu/qkaaw==",
+      "cpu": [
+        "ia32"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
+    "node_modules/@nogoo9/gitleaks-windows-x64": {
+      "version": "8.30.1-post.2",
+      "resolved": "https://registry.npmjs.org/@nogoo9/gitleaks-windows-x64/-/gitleaks-windows-x64-8.30.1-post.2.tgz",
+      "integrity": "sha512-whDUUm3+J5YRQUJS0mcE72JaPb2N0Njmjaj6NYTcG1OJS3v42Y/JYen6R9FuQ5jUCfsVGcblmSBmDb9uZVdwrw==",
+      "cpu": [
+        "x64"
+      ],
+      "dev": true,
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
    "node_modules/@pnpm/config.env-replace": {
      "version": "1.1.0",
      "resolved": "https://registry.npmjs.org/@pnpm/config.env-replace/-/config.env-replace-1.1.0.tgz",
@@ -3,6 +3,7 @@
  "version": "7.14.0",
  "description": "Data Controller",
  "devDependencies": {
+    "@nogoo9/gitleaks": "8.30.1-post.2",
    "@saithodev/semantic-release-gitea": "^2.1.0",
    "@semantic-release/changelog": "^6.0.3",
    "@semantic-release/commit-analyzer": "13.0.1",
@@ -6,8 +6,8 @@
    "": {
      "name": "dc-sas",
      "dependencies": {
-        "@sasjs/cli": "4.20.1",
-        "@sasjs/core": "5.2.0"
+        "@sasjs/cli": "4.20.3",
+        "@sasjs/core": "5.2.7"
      }
    },
    "node_modules/@asamuzakjp/css-color": {
@@ -23,17 +23,6 @@
        "lru-cache": "^10.4.3"
      }
    },
-    "node_modules/@asamuzakjp/dom-selector": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-2.0.2.tgz",
-      "integrity": "sha512-x1KXOatwofR6ZAYzXRBL5wrdV0vwNxlTCK9NCuLqAzQYARqGcvFwiJA6A1ERuh+dgeA4Dxm3JBYictIes+SqUQ==",
-      "license": "MIT",
-      "dependencies": {
-        "bidi-js": "^1.0.3",
-        "css-tree": "^2.3.1",
-        "is-potential-custom-element-name": "^1.0.1"
-      }
-    },
    "node_modules/@coolaj86/urequest": {
      "version": "1.3.7",
      "resolved": "https://registry.npmjs.org/@coolaj86/urequest/-/urequest-1.3.7.tgz",
@@ -214,94 +203,72 @@
      }
    },
    "node_modules/@sasjs/cli": {
-      "version": "4.20.1",
-      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.1.tgz",
-      "integrity": "sha512-482xPlEuyBKqGlg2ArCiREu3Q9+8TsTj0dwy5ewNURi7pWEcwJH0A4c9Yw4a3akUtPAphPEzs8/9BkcS/v+3WQ==",
+      "version": "4.20.3",
+      "resolved": "https://registry.npmjs.org/@sasjs/cli/-/cli-4.20.3.tgz",
+      "integrity": "sha512-ZI+7VoBi0bgjpGkmKTbS8f5p1pQKgE2sv4NKTtjQ9Qg3iltjdQK8IoUQXEn4IJI1pa3wCuaIrCoYjUmCS6VYMQ==",
      "license": "ISC",
      "dependencies": {
        "@sasjs/adapter": "^4.19.0",
-        "@sasjs/core": "4.68.1",
-        "@sasjs/lint": "2.4.3",
-        "@sasjs/utils": "^3.6.0",
-        "adm-zip": "0.6.0",
+        "@sasjs/core": "5.2.4",
+        "@sasjs/lint": "2.5.0",
+        "@sasjs/utils": "3.6.2",
        "chalk": "4.1.2",
-        "dotenv": "16.0.3",
+        "dotenv": "17.4.2",
        "find": "0.3.0",
-        "js-base64": "3.7.5",
-        "jsdom": "23.2.0",
+        "js-base64": "3.9.3",
+        "jsdom": "26.0.0",
        "jwt-decode": "3.1.2",
        "lodash.groupby": "4.6.0",
        "lodash.uniqby": "4.7.0",
        "node-graphviz": "0.1.1",
        "node-powershell": "5.0.1",
        "ora": "5.4.1",
-        "prompts": "2.4.1",
+        "prompts": "2.4.2",
        "shelljs": "0.10.0",
        "ssl-root-cas": "1.3.1",
        "xml": "1.0.1",
-        "yargs": "17.6.2"
+        "yargs": "17.6.2",
+        "yauzl": "3.4.0",
+        "yazl": "3.3.1"
      },
      "bin": {
        "sasjs": "build/index.js"
      }
    },
    "node_modules/@sasjs/cli/node_modules/@sasjs/core": {
-      "version": "4.68.1",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-4.68.1.tgz",
-      "integrity": "sha512-BjoOzV7h7guCShW4JsFHvB1NNLCKHhPI5708OtEk29itQbgVCMRlEXKFRbnMiuNklktDQDi8dR4XZZ0aK7/4Wg==",
+      "version": "5.2.4",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.4.tgz",
+      "integrity": "sha512-VAY62Zl8xkIfCr73jXFFgqGX4lab9htsZRmCIIttwbBR/T/LUD0N0g/mcfaVp/W+UvHHT8dqntNlm8SX0u11rQ==",
      "license": "MIT"
    },
+    "node_modules/@sasjs/cli/node_modules/prompts": {
+      "version": "2.4.2",
+      "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
+      "integrity": "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q==",
+      "license": "MIT",
+      "dependencies": {
+        "kleur": "^3.0.3",
+        "sisteransi": "^1.0.5"
+      },
+      "engines": {
+        "node": ">= 6"
+      }
+    },
    "node_modules/@sasjs/core": {
-      "version": "5.2.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.0.tgz",
-      "integrity": "sha512-YJAVckql9o2lbldo4xFbu/Vkja0DfSw8w8hrnhnCov8iqqzkM84rHnqqKzdD0+sa52+AoPoHAxt4nCBoQfH9wA==",
+      "version": "5.2.7",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-5.2.7.tgz",
+      "integrity": "sha512-GH2k6pV/Ik2E1tYNmGH5UiPQIVfcQ04MrPL2UPTHMfcuIK5Sgo1Gb+7GhX0IwzQwymM1BU5wodSW5X/N2GLXZw==",
      "license": "MIT"
    },
    "node_modules/@sasjs/lint": {
-      "version": "2.4.3",
-      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-2.4.3.tgz",
-      "integrity": "sha512-J0fYJmDfnhXDeMkZI8vHo8zjIsByIPcKHSQbiQEuRMXbNJXjqgTqoCpN0L6SlhyrrXUtMQvk/XErJ3OQGVEjrA==",
+      "version": "2.5.0",
+      "resolved": "https://registry.npmjs.org/@sasjs/lint/-/lint-2.5.0.tgz",
+      "integrity": "sha512-bVUtYnKhCigXylCCvQ6dsAIywpPKkX3BXatMqOWjDVw9R30jfj4M3/qn05fcR45L42Y7e5Tuzlt+LYs+9k95Yg==",
      "hasInstallScript": true,
      "license": "MIT",
      "dependencies": {
-        "@sasjs/utils": "3.5.2",
-        "ignore": "5.2.4"
-      }
-    },
-    "node_modules/@sasjs/lint/node_modules/@sasjs/utils": {
-      "version": "3.5.2",
-      "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-3.5.2.tgz",
-      "integrity": "sha512-LBpBDx0T7G/eO15Gb+r3DR1LfBnoqagWT3HiHabojFziA4ej4ePORo8chrk0zHLIzrjI2ljAnDabyJEwC5KtIA==",
-      "hasInstallScript": true,
-      "license": "ISC",
-      "dependencies": {
-        "@fast-csv/format": "4.3.5",
-        "@types/fs-extra": "11.0.4",
-        "@types/prompts": "2.0.13",
-        "chalk": "4.1.1",
-        "cli-table": "0.3.6",
-        "consola": "2.15.0",
-        "find": "0.3.0",
-        "fs-extra": "11.3.0",
-        "jwt-decode": "3.1.2",
-        "prompts": "2.4.1",
-        "valid-url": "1.0.9"
-      }
-    },
-    "node_modules/@sasjs/lint/node_modules/chalk": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.1.tgz",
-      "integrity": "sha512-diHzdDKxcU+bAsUboHLPEDQiw0qEe0qd7SYUn3HgcFlWgbDcfLGswOHYeGrHKzG9z6UYf01d9VFMfZxPM1xZSg==",
-      "license": "MIT",
-      "dependencies": {
-        "ansi-styles": "^4.1.0",
-        "supports-color": "^7.1.0"
-      },
-      "engines": {
-        "node": ">=10"
-      },
-      "funding": {
-        "url": "https://github.com/chalk/chalk?sponsor=1"
+        "@sasjs/utils": "3.6.2",
+        "ignore": "7.0.8"
      }
    },
    "node_modules/@sasjs/utils": {
@@ -383,15 +350,6 @@
        "ms": "^2.1.3"
      }
    },
-    "node_modules/adm-zip": {
-      "version": "0.6.0",
-      "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.0.tgz",
-      "integrity": "sha512-XleryMhbuksdKtofnWZ9Sk+4CUTbms4Mb/EU32SZwToAyZ5RgVos/ki8n+yr0LWHOGKuakbXTuuYNHLQjhddgg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=14.0"
-      }
-    },
    "node_modules/agent-base": {
      "version": "7.1.4",
      "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz",
@@ -482,15 +440,6 @@
      ],
      "license": "MIT"
    },
-    "node_modules/bidi-js": {
-      "version": "1.0.3",
-      "resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz",
-      "integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==",
-      "license": "MIT",
-      "dependencies": {
-        "require-from-string": "^2.0.2"
-      }
-    },
    "node_modules/bl": {
      "version": "4.1.0",
      "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
@@ -538,6 +487,15 @@
        "ieee754": "^1.1.13"
      }
    },
+    "node_modules/buffer-crc32": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-1.0.0.tgz",
+      "integrity": "sha512-Db1SbgBS/fg/392AblrMJk97KggmvYhr4pB5ZIMTWtaivCPMWLkmb7m21cJvpvgK+J3nsU2CmmixNBZx4vFj/w==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=8.0.0"
+      }
+    },
    "node_modules/bytes": {
      "version": "3.1.2",
      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
@@ -708,19 +666,6 @@
        "node": ">= 8"
      }
    },
-    "node_modules/css-tree": {
-      "version": "2.3.1",
-      "resolved": "https://registry.npmjs.org/css-tree/-/css-tree-2.3.1.tgz",
-      "integrity": "sha512-6Fv1DV/TYw//QF5IzQdqsNDjx/wc8TrMBZsqjL9eW01tWb7R7k/mq+/VXfJCl7SoD5emsJop9cOByJZfs8hYIw==",
-      "license": "MIT",
-      "dependencies": {
-        "mdn-data": "2.0.30",
-        "source-map-js": "^1.0.1"
-      },
-      "engines": {
-        "node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0"
-      }
-    },
    "node_modules/cssstyle": {
      "version": "4.6.0",
      "resolved": "https://registry.npmjs.org/cssstyle/-/cssstyle-4.6.0.tgz",
@@ -734,12 +679,6 @@
        "node": ">=18"
      }
    },
-    "node_modules/cssstyle/node_modules/rrweb-cssom": {
-      "version": "0.8.0",
-      "resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
-      "integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
-      "license": "MIT"
-    },
    "node_modules/data-urls": {
      "version": "5.0.0",
      "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-5.0.0.tgz",
@@ -798,12 +737,15 @@
      }
    },
    "node_modules/dotenv": {
-      "version": "16.0.3",
-      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.3.tgz",
-      "integrity": "sha512-7GO6HghkA5fYG9TYnNxi14/7K9f5occMlp3zXAuSxn7CKCxt9xbNWG7yF8hTCSUchlfWSe3uLmlPfigevRItzQ==",
+      "version": "17.4.2",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
+      "integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
      "license": "BSD-2-Clause",
      "engines": {
        "node": ">=12"
+      },
+      "funding": {
+        "url": "https://dotenvx.com"
      }
    },
    "node_modules/dunder-proto": {
@@ -1284,9 +1226,9 @@
      "license": "BSD-3-Clause"
    },
    "node_modules/ignore": {
-      "version": "5.2.4",
-      "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.2.4.tgz",
-      "integrity": "sha512-MAb38BcSbH0eHNBxn7ql2NH/kX33OkB3lZ1BNdh7ENeRChHTYsTvWrMubiIAMNS2llXEEgZ1MUOBtXChP3kaFQ==",
+      "version": "7.0.8",
+      "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.8.tgz",
+      "integrity": "sha512-YYNsSlXBjMk92SKnkwvB5LOVSa6OznlFUGcsvrFgNJbJCd0M1XKeFVRc8ZByeCqz32FivYNHJVooLmdqrmvp/Q==",
      "license": "MIT",
      "engines": {
        "node": ">= 4"
@@ -1410,44 +1352,44 @@
      "license": "ISC"
    },
    "node_modules/js-base64": {
-      "version": "3.7.5",
-      "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.7.5.tgz",
-      "integrity": "sha512-3MEt5DTINKqfScXKfJFrRbxkrnk2AxPWGBL/ycjz4dK8iqiSJ06UxD8jh8xuh6p10TX4t2+7FsBYVxxQbMg+qA==",
+      "version": "3.9.3",
+      "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.9.3.tgz",
+      "integrity": "sha512-uwYQp+VJ38FVvtim6qNbit6e9uT6dwWQ4Y1+H9TxhW5hcHjpHwoxlR0nMpqUmIFOmu4VqMxwdJA88gIVuZJQ/g==",
      "license": "BSD-3-Clause"
    },
    "node_modules/jsdom": {
-      "version": "23.2.0",
-      "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-23.2.0.tgz",
-      "integrity": "sha512-L88oL7D/8ufIES+Zjz7v0aes+oBMh2Xnh3ygWvL0OaICOomKEPKuPnIfBJekiXr+BHbbMjrWn/xqrDQuxFTeyA==",
+      "version": "26.0.0",
+      "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-26.0.0.tgz",
+      "integrity": "sha512-BZYDGVAIriBWTpIxYzrXjv3E/4u8+/pSG5bQdIYCbNCGOvsPkDQfTVLAIXAf9ETdCpduCVTkDe2NNZ8NIwUVzw==",
      "license": "MIT",
      "dependencies": {
-        "@asamuzakjp/dom-selector": "^2.0.1",
-        "cssstyle": "^4.0.1",
+        "cssstyle": "^4.2.1",
        "data-urls": "^5.0.0",
        "decimal.js": "^10.4.3",
-        "form-data": "^4.0.0",
+        "form-data": "^4.0.1",
        "html-encoding-sniffer": "^4.0.0",
-        "http-proxy-agent": "^7.0.0",
-        "https-proxy-agent": "^7.0.2",
+        "http-proxy-agent": "^7.0.2",
+        "https-proxy-agent": "^7.0.6",
        "is-potential-custom-element-name": "^1.0.1",
-        "parse5": "^7.1.2",
-        "rrweb-cssom": "^0.6.0",
+        "nwsapi": "^2.2.16",
+        "parse5": "^7.2.1",
+        "rrweb-cssom": "^0.8.0",
        "saxes": "^6.0.0",
        "symbol-tree": "^3.2.4",
-        "tough-cookie": "^4.1.3",
+        "tough-cookie": "^5.0.0",
        "w3c-xmlserializer": "^5.0.0",
        "webidl-conversions": "^7.0.0",
        "whatwg-encoding": "^3.1.1",
        "whatwg-mimetype": "^4.0.0",
-        "whatwg-url": "^14.0.0",
-        "ws": "^8.16.0",
+        "whatwg-url": "^14.1.0",
+        "ws": "^8.18.0",
        "xml-name-validator": "^5.0.0"
      },
      "engines": {
        "node": ">=18"
      },
      "peerDependencies": {
-        "canvas": "^2.11.2"
+        "canvas": "^3.0.0"
      },
      "peerDependenciesMeta": {
        "canvas": {
@@ -1468,6 +1410,18 @@
        "node": ">= 14"
      }
    },
+    "node_modules/jsdom/node_modules/tough-cookie": {
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
+      "integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
+      "license": "BSD-3-Clause",
+      "dependencies": {
+        "tldts": "^6.1.32"
+      },
+      "engines": {
+        "node": ">=16"
+      }
+    },
    "node_modules/jsonfile": {
      "version": "6.2.1",
      "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.1.tgz",
@@ -1578,12 +1532,6 @@
        "node": ">= 0.4"
      }
    },
-    "node_modules/mdn-data": {
-      "version": "2.0.30",
-      "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.0.30.tgz",
-      "integrity": "sha512-GaqWWShW4kv/G9IEucWScBx9G1/vsFZZJUO+tD26M8J8z3Kw5RDQjaoZe03YAClgeS/SWPOcb4nkFBTEi5DUEA==",
-      "license": "CC0-1.0"
-    },
    "node_modules/merge-stream": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz",
@@ -1694,6 +1642,12 @@
        "node": ">=8"
      }
    },
+    "node_modules/nwsapi": {
+      "version": "2.2.27",
+      "resolved": "https://registry.npmjs.org/nwsapi/-/nwsapi-2.2.27.tgz",
+      "integrity": "sha512-gQPNF78qebCQ6tvVFBYrvJdBNOrYZm90ZlXgpIFm06p6qHDHq/XC4TnJftN6OMbxVE0UTBAoRgcsDeJBBooITw==",
+      "license": "MIT"
+    },
    "node_modules/onetime": {
      "version": "5.1.2",
      "resolved": "https://registry.npmjs.org/onetime/-/onetime-5.1.2.tgz",
@@ -1799,6 +1753,12 @@
        "node": ">=8"
      }
    },
+    "node_modules/pend": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz",
+      "integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==",
+      "license": "MIT"
+    },
    "node_modules/picomatch": {
      "version": "2.3.2",
      "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
@@ -1903,15 +1863,6 @@
        "node": ">=0.10.0"
      }
    },
-    "node_modules/require-from-string": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
-      "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
    "node_modules/requires-port": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
@@ -1942,9 +1893,9 @@
      }
    },
    "node_modules/rrweb-cssom": {
-      "version": "0.6.0",
-      "resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.6.0.tgz",
-      "integrity": "sha512-APM0Gt1KoXBz0iIkkdB/kfvGOwC4UuJFeG/c+yV7wSc7q96cG/kJ0HiYCnzivD9SB53cLV1MlHFNfOuPaadYSw==",
+      "version": "0.8.0",
+      "resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
+      "integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
      "license": "MIT"
    },
    "node_modules/run-parallel": {
@@ -2054,15 +2005,6 @@
      "integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==",
      "license": "MIT"
    },
-    "node_modules/source-map-js": {
-      "version": "1.2.1",
-      "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
-      "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
-      "license": "BSD-3-Clause",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
    "node_modules/ssl-root-cas": {
      "version": "1.3.1",
      "resolved": "https://registry.npmjs.org/ssl-root-cas/-/ssl-root-cas-1.3.1.tgz",
@@ -2134,6 +2076,24 @@
      "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==",
      "license": "MIT"
    },
+    "node_modules/tldts": {
+      "version": "6.1.86",
+      "resolved": "https://registry.npmjs.org/tldts/-/tldts-6.1.86.tgz",
+      "integrity": "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ==",
+      "license": "MIT",
+      "dependencies": {
+        "tldts-core": "^6.1.86"
+      },
+      "bin": {
+        "tldts": "bin/cli.js"
+      }
+    },
+    "node_modules/tldts-core": {
+      "version": "6.1.86",
+      "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-6.1.86.tgz",
+      "integrity": "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA==",
+      "license": "MIT"
+    },
    "node_modules/to-regex-range": {
      "version": "5.0.1",
      "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz",
@@ -2322,9 +2282,9 @@
      }
    },
    "node_modules/ws": {
-      "version": "8.21.1",
-      "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.1.tgz",
-      "integrity": "sha512-+0NTnW77fFN/DjQi6k/Sq/Yvk4Sgajw7urW8V+asjXnRgDs9gyGkdb7EzgfhA4goXsRIZKE28fzIXBHEzhuiWw==",
+      "version": "8.21.3",
+      "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
+      "integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==",
      "license": "MIT",
      "engines": {
        "node": ">=10.0.0"
@@ -2398,6 +2358,27 @@
      "engines": {
        "node": ">=12"
      }
+    },
+    "node_modules/yauzl": {
+      "version": "3.4.0",
+      "resolved": "https://registry.npmjs.org/yauzl/-/yauzl-3.4.0.tgz",
+      "integrity": "sha512-jIH9yLR9wqr0wOS0TpBvo/g/2UgZH5qePVbjgRliiF0BYvOZyaBknKsF+x9Iht0O6sqgnB93rCICdOZFecJuDw==",
+      "license": "MIT",
+      "dependencies": {
+        "pend": "~1.2.0"
+      },
+      "engines": {
+        "node": ">=12"
+      }
+    },
+    "node_modules/yazl": {
+      "version": "3.3.1",
+      "resolved": "https://registry.npmjs.org/yazl/-/yazl-3.3.1.tgz",
+      "integrity": "sha512-BbETDVWG+VcMUle37k5Fqp//7SDOK2/1+T7X8TD96M3D9G8jK5VLUdQVdVjGi8im7FGkazX7kk5hkU8X4L5Bng==",
+      "license": "MIT",
+      "dependencies": {
+        "buffer-crc32": "^1.0.0"
+      }
    }
  }
 }
@@ -30,8 +30,8 @@
  },
  "private": true,
  "dependencies": {
-    "@sasjs/cli": "4.20.1",
-    "@sasjs/core": "5.2.0"
+    "@sasjs/cli": "4.20.3",
+    "@sasjs/core": "5.2.7"
  },
  "overrides": {
    "nanoid": "3.3.18"
@@ -6,6 +6,7 @@
  <h4> SAS Macros </h4>
  @li mpeinit.sas
  @li dc_refreshcatalog.sas
+  @li mpeterm.sas

  @version 9.3
  @author 4GL Apps Ltd
@@ -20,7 +21,13 @@
 %dc_refreshcatalog(&libref)


-data _null_;
-  file _webout;
-  put '<h1> Catalog Refresh Complete </h1>';
+data sasparams;
+  length msg $64;
+  msg='Catalog Refresh Complete';
 run;
+
+%webout(OPEN)
+%webout(OBJ,sasparams)
+%webout(CLOSE)
+
+%mpeterm()
@@ -20,25 +20,20 @@ run;
 %mx_execute(&_program,
  viyacontext=&defaultcontext,
  inputparams=work.params,
-  outref=webout,
-  viyaresult=WEBOUT_TXT,
+  outlib=web1,
  mdebug=&sasjs_mdebug
 )

-data work.results;
-  infile webout;
-  input;
-  putlog _infile_;
-  if index(upcase(_infile_),'CATALOG REFRESH COMPLETE') then do;
-    putlog 'test passed';
-    output;
-    stop;
-  end;
+%let msgcheck=0;
+data _null_;
+  set web1.sasparams;
+  putlog (_all_)(=);
+  if index(msg,'Catalog Refresh Complete') then call symputx('msgcheck',1);
 run;

-%mp_assertdsobs(work.results,
+%mp_assert(
+  iftrue=(&msgcheck=1),
  desc=Refresh catalog confirmation message is returned,
-  test=EQUALS 1,
  outds=work.test_results
 )

@@ -25,7 +25,14 @@ select max(primary_key_field) into: maxpk
  from &dclib..mpe_x_test;

 data work.jsdata;
-  set &dclib..mpe_x_test;
+  set &dclib..mpe_x_test(rename=(
+    some_date=dt2 SOME_DATETIME=dttm2 SOME_TIME=tm2)
+  );
+  /* the adapter sends these as strings - see postdata.test.1 */
+  some_date=put(dt2,date9.);
+  SOME_DATETIME=put(dttm2,datetime19.);
+  some_time=put(tm2,time.);
+  drop dt2 dttm2 tm2;
  if _n_=1 then do;
    _____DELETE__THIS__RECORD_____='No';
    some_char='getstagetable test';
@@ -84,14 +91,16 @@ data _null_;
  if some_char='getstagetable test' then call symputx('fmtcheck',1);
 run;

-%mp_assertdsobs(web2.stagetable,
+/* the json libname can only be read once, so assert on the flags set
+   above rather than re-reading the tables */
+%mp_assert(
+  iftrue=(&stagecheck=1),
  desc=getstagetable returns the staged table,
-  test=EQUALS 1,
  outds=work.test_results
 )
-%mp_assertdsobs(web2.fmt_stagetable,
+%mp_assert(
+  iftrue=(&fmtcheck=1),
  desc=getstagetable returns the formatted staged table,
-  test=EQUALS 1,
  outds=work.test_results
 )
 %mp_assert(
@@ -25,9 +25,9 @@ proc format lib=DCTEST.DCFMTS cntlout=work.fmtextract;
 run;
 data work.jsdata;
  set work.fmtextract;
+  fmtrow=_n_;
  if _n_<5 then _____DELETE__THIS__RECORD_____='Yes';
  else _____DELETE__THIS__RECORD_____='No';
-  if _n_>12 then label=cats('new!',label);
  if _n_>20 then stop;
 run;

@@ -113,9 +113,21 @@ data _null_;
  end;
  else call symputx('libds',libds);
  call symputx('is_fmt',is_fmt);
+  /* validate libds to prevent code injection */
+  %mp_validatecol(LIBDS,LIBDS,is_libds)
+  if is_libds=0 then do;
+    putlog 'ERR' 'OR: Invalid libds:' libds;
+    call symputx('bad_libds',1);
+  end;
+  else call symputx('bad_libds',0);
  putlog (_all_)(=);
 run;

+%mp_abort(iftrue= (&bad_libds=1)
+  ,mac=&_program
+  ,msg=%str(Invalid libds supplied)
+)
+
 /* check that the user has the requisite access */
 %mpe_getgroups(user=&user,outds=groups)

@@ -66,9 +66,21 @@ data _null_;
  end;
  else call symputx('libds',libds);
  call symputx('is_fmt',is_fmt);
+  /* validate libds to prevent code injection */
+  %mp_validatecol(LIBDS,LIBDS,is_libds)
+  if is_libds=0 then do;
+    putlog 'ERR' 'OR: Invalid libds:' libds;
+    call symputx('bad_libds',1);
+  end;
+  else call symputx('bad_libds',0);
  putlog (_all_)(=);
 run;

+%mp_abort(iftrue= (&bad_libds=1)
+  ,mac=&_program
+  ,msg=%str(Invalid libds supplied)
+)
+
 %mp_cntlout(
  iftrue=(&is_fmt=1)
  ,libcat=&orig_libds
@@ -5,6 +5,11 @@
    1. loading row level EDIT security against a table with LOADTYPE=REPLACE
    2. applying LOADTYPE=REPLACE to a table with row level EDIT security

+  The staged loads are aborted by the post edit hooks inside the child
+  service.  The abort is recorded as a FAILED load in MPE_LOADS (via
+  mpe_loadfail), so the assertions query MPE_LOADS rather than the child
+  response - a canceled child does not always return a webout or log.
+
  <h4> SAS Macros </h4>
  @li mx_execute.sas
  @li mp_assert.sas
@@ -31,11 +36,42 @@ insert into &dc_libref..mpe_tables
    ,buskey='X'
    ,loadtype='REPLACE'
    ,num_of_approvals_required=1;
+/* ensure the RLS table's post edit hook is registered (matches the
+  * mpe_makedata registration - some estates were initialised before the
+  * hook existed, which would silently skip the test 1 validation) */
+update &dc_libref..mpe_tables
+  set post_edit_hook='services/hooks/mpe_row_level_security_postedit'
+  where upcase(dsn)='MPE_ROW_LEVEL_SECURITY'
+    and post_edit_hook='';
+/* insert an active EDIT rule for DCTEST.WIDEBOY up front - the child
+  * services run in separate (possibly pooled) compute sessions, so the
+  * rule must be committed to the shared filesystem well before the test
+  * 2 child reads it */
+delete from &dc_libref..mpe_row_level_security
+  where rls_libref="DCTEST" and rls_table='WIDEBOY';
+select coalesce(max(rls_rk),0)+1 into: rlsrk
+  from &dc_libref..mpe_row_level_security;
+insert into &dc_libref..mpe_row_level_security
+  set tx_from=0
+    ,tx_to='31DEC5999:23:59:59'dt
+    ,rls_rk=&rlsrk
+    ,rls_scope='EDIT'
+    ,rls_group='SASAdministrators'
+    ,rls_libref='DCTEST'
+    ,rls_table='WIDEBOY'
+    ,rls_group_logic='AND'
+    ,rls_subgroup_logic='OR'
+    ,rls_subgroup_id=0
+    ,rls_variable_nm='ROW_ID'
+    ,rls_operator_nm='NE'
+    ,rls_raw_value='1'
+    ,rls_active=1;
 quit;

 /**
  * Test 1 - submitting an RLS EDIT rule against a REPLACE table should abort
  */
+%let t1=%sysfunc(datetime());
 data work.sascontroltable;
  action='LOAD';
  message='RLS EDIT rule on REPLACE table should abort';
@@ -72,49 +108,36 @@ run;
  mdebug=&sasjs_mdebug
 )

-/* search terms may span multiple lines, so track matches across records */
+/* a hook-aborted load is terminated before mpe_loadfail can record a
+  * FAILED status, so the load row stays IN PROGRESS - the abort is
+  * proved by the load NOT completing successfully.  Tests in this suite run
+  * in parallel sessions, so match the row for THIS test by its unique
+  * reason_txt (the submitted message) rather than any load in the time
+  * window - an unrelated aborted/IN PROGRESS load would otherwise satisfy
+  * the assertion even if this load completed.  The row id (csv_dir/mperef)
+  * is generated inside the aborted child service, which returns no webout,
+  * so reason_txt is the only stable identifier the test controls. */
 %let abort1=0;
-data _null_;
-  retain foundabort foundmsg 0;
-  infile wb1;
-  input;
-  putlog _infile_;
-  if index(_infile_,'sasjsAbort') then foundabort=1;
-  if index(_infile_,'REPLACE loadtype') then foundmsg=1;
-  if foundabort=1 and foundmsg=1 then call symputx('abort1',1);
-run;
+proc sql noprint;
+select count(*) into: abort1
+  from &dc_libref..mpe_loads
+  where processed_dttm>&t1
+    and reason_txt="RLS EDIT rule on REPLACE table should abort"
+    and upcase(status) ne 'SUCCESS';
+quit;

 %mp_assert(
-  iftrue=(&abort1=1),
+  iftrue=(&abort1>0),
  desc=Checking RLS EDIT rule is rejected for REPLACE loadtype table (#211),
  outds=work.test_results
 )

 /**
  * Test 2 - applying REPLACE loadtype to a table with RLS EDIT security
-  * should abort.  First insert an active EDIT rule directly.
+  * should abort.  The active EDIT rule was inserted in the prep section,
+  * long before this child service reads it.
  */
-proc sql noprint;
-delete from &dc_libref..mpe_row_level_security
-  where rls_libref="DCTEST" and rls_table='WIDEBOY';
-select coalesce(max(rls_rk),0)+1 into: rlsrk
-  from &dc_libref..mpe_row_level_security;
-insert into &dc_libref..mpe_row_level_security
-  set tx_from=0
-    ,tx_to='31DEC5999:23:59:59'dt
-    ,rls_rk=&rlsrk
-    ,rls_scope='EDIT'
-    ,rls_group='SASAdministrators'
-    ,rls_libref='DCTEST'
-    ,rls_table='WIDEBOY'
-    ,rls_group_logic='AND'
-    ,rls_subgroup_logic='OR'
-    ,rls_subgroup_id=0
-    ,rls_variable_nm='ROW_ID'
-    ,rls_operator_nm='NE'
-    ,rls_raw_value='1'
-    ,rls_active=1;
-quit;
+%let t2=%sysfunc(datetime());

 /* now stage a REPLACE loadtype for DCTEST.WIDEBOY */
 data work.sascontroltable;
@@ -141,18 +164,16 @@ run;
 )

 %let abort2=0;
-data _null_;
-  retain foundabort foundmsg 0;
-  infile wb2;
-  input;
-  putlog _infile_;
-  if index(_infile_,'sasjsAbort') then foundabort=1;
-  if index(_infile_,'EDIT security') then foundmsg=1;
-  if foundabort=1 and foundmsg=1 then call symputx('abort2',1);
-run;
+proc sql noprint;
+select count(*) into: abort2
+  from &dc_libref..mpe_loads
+  where processed_dttm>&t2
+    and reason_txt="REPLACE loadtype on RLS-secured table should abort"
+    and upcase(status) ne 'SUCCESS';
+quit;

 %mp_assert(
-  iftrue=(&abort2=1),
+  iftrue=(&abort2>0),
  desc=Checking REPLACE loadtype is rejected for RLS EDIT-secured table (#211),
  outds=work.test_results
 )
@@ -16,7 +16,7 @@
 %let syscc=0;
 %global apploc _program dclib defaultcontext _debug sasjs_mdebug dc_dttmtfmt;

-%let defaultcontext=SAS Job Execution compute context;
+%let defaultcontext=Compute Reusable;
 %let sasjs_mdebug=0;

 options mprint mprintnest nobomfile lrecl=32767;
@@ -29,7 +29,7 @@

 %global apploc _program;

-%let defaultcontext=SAS Job Execution compute context;
+%let defaultcontext=Compute Reusable;

 data _null_;
  length _pgm $1000;
@@ -141,10 +141,19 @@ print(f'\nDeploying {len(chunk_files)} chunks to nextviya...')
 for idx, cf in enumerate(chunk_files, 1):
    print(f'\n[{idx}/{len(chunk_files)}] Deploying {os.path.basename(cf)} ...')
    timeout = 600 if os.path.getsize(cf) > 5 * 1024 * 1024 else 300
-    result = subprocess.run(
-        ['npx', 'sasjs', 'run', cf, '-t', 'nextviya'],
-        capture_output=True, text=True, timeout=timeout
-    )
+    result = None
+    for attempt in range(1, 4):
+        try:
+            result = subprocess.run(
+                ['npx', 'sasjs', 'run', cf, '-t', 'nextviya'],
+                capture_output=True, text=True, timeout=timeout
+            )
+            break
+        except subprocess.TimeoutExpired:
+            print(f'  attempt {attempt} timed out after {timeout}s'
+                  + (' - retrying...' if attempt < 3 else ' - GIVING UP'))
+    if result is None:
+        sys.exit(1)
    out = '\n'.join([l for l in result.stdout.splitlines() if not l.startswith('isTokenExpiring')])
    tail_lines = out.splitlines()[-40:]
    print('\n'.join(tail_lines))
@@ -153,5 +162,4 @@ for idx, cf in enumerate(chunk_files, 1):
        print(result.stderr[-1000:] if len(result.stderr) > 1000 else result.stderr)
        sys.exit(1)
    print(f'  -> OK')
-
 print('\nAll chunks deployed successfully!')
Author	SHA1	Message	Date
allan	24636c9a87	Merge pull request 'fix(security): escape cell values and harden libds inputs' (#319 ) from security/harden-xss-and-inputs into fix/viya-test-suite-and-refreshcatalog-json Build / Build-and-ng-test (pull_request) Successful in 5m28s Details Lighthouse Checks / lighthouse (pull_request) Successful in 21m5s Details Build / Build-and-test-development (pull_request) Successful in 24m38s Details Reviewed-on: #319	2026-09-14 16:20:07 +00:00
hermes	11f56cc54a	refactor(deploy): remove vestigial client credential fields Build / Build-and-ng-test (pull_request) Successful in 5m25s Details Lighthouse Checks / lighthouse (pull_request) Successful in 20m57s Details Build / Build-and-test-development (pull_request) Successful in 24m44s Details The deploy client_id/client_secret fields were only read from localStorage and never used by the manual or automatic deploy flows. Remove the dead code so no credential-shaped value is read from browser storage.	2026-09-14 16:45:20 +01:00
hermes	2d31e5a8b9	fix(security): validate libds in stagedata and loadfile Sanitise the libref.dataset input the same way getdata does, via mp_validatecol, and abort the service on an invalid value to prevent code injection through the libds identifier. Format catalog inputs resolve to work.fmtextract and still pass the check.	2026-09-14 16:45:13 +01:00
hermes	cfd8f06435	fix(security): escape cell values in status renderers to prevent DOM XSS The error/no-spinner/spinner cell renderers wrote the cell value straight into td.innerHTML. A value containing markup (which can arrive from a dataset served by the getdata stored program or from a typed edit) was therefore parsed and executed by the browser. Escape the value so it renders as inert text, keeping the hardcoded icon markup intact, and add a regression test that reproduces the injection via a real Handsontable instance.	2026-09-14 16:44:59 +01:00
hermes	a6a111db8a	test(viya): scope stagedata abort assertions to the submitted load Build / Build-and-ng-test (pull_request) Successful in 6m4s Details Lighthouse Checks / lighthouse (pull_request) Successful in 22m41s Details Build / Build-and-test-development (pull_request) Successful in 25m35s Details The abort checks matched any non-success load in the time window, so a parallel session's aborted/IN PROGRESS load could satisfy them even if the tested load completed. Key them on this test's unique submitted message (reason_txt), which is preserved because a hook-aborted load finishes before mpe_loadfail can overwrite it.	2026-09-14 13:25:32 +01:00
allan	ca19dbaefb	Merge pull request 'chore(hooks): add gitleaks secret scan to pre-commit' (#317 ) from chore/gitleaks-precommit into fix/viya-test-suite-and-refreshcatalog-json Build / Build-and-ng-test (pull_request) Successful in 5m26s Details Lighthouse Checks / lighthouse (pull_request) Successful in 21m28s Details Build / Build-and-test-development (pull_request) Successful in 25m39s Details Reviewed-on: #317	2026-09-14 12:15:38 +00:00
hermes	8418e0cb1d	docs(contributing): document one-time git hook activation Build / Build-and-ng-test (pull_request) Successful in 5m21s Details Lighthouse Checks / lighthouse (pull_request) Successful in 21m1s Details Build / Build-and-test-development (pull_request) Successful in 24m44s Details	2026-09-14 12:44:10 +01:00
hermes	d301fc2b4d	fix(deps): bump @sasjs/cli to 4.20.3 Clears the moderate adm-zip advisory (GHSA-vwc7-r8mq-g2x9) from the sas npm audit gate - @sasjs/cli 4.20.3 replaces adm-zip with yauzl/yazl.	2026-09-14 12:43:58 +01:00
dc	fbcde41321	fix(hooks): require the repo-pinned gitleaks, drop the fallbacks Build / Build-and-ng-test (pull_request) Failing after 1m46s Details Build / Build-and-test-development (pull_request) Skipped Details Lighthouse Checks / lighthouse (pull_request) Successful in 21m12s Details A soft skip (and the system-gitleaks fallback) let a clone without node_modules commit with no secrets scan at all, so the hook would never force the install. The hook now hard-blocks with a hint to run 'npm i' until the @nogoo9/gitleaks binary is present in node_modules/.bin. Verified with the binary present (scan runs and passes) and absent (blocked, exit 1).	2026-09-10 23:06:34 +00:00
dc	53838036e6	fix(hooks): make the pre-commit size check actually run Build / Build-and-ng-test (pull_request) Failing after 1m47s Details Build / Build-and-test-development (pull_request) Skipped Details Lighthouse Checks / lighthouse (pull_request) Successful in 20m51s Details The size check sat after the prettier block, whose branches both exit, so it was unreachable - and it had three further defects: the exit 1 lived inside a ( ... ) subshell so it could never block the commit, the 2**20 arithmetic errors under /bin/sh (dash) leaving size_limit unset, and it measured HEAD^..HEAD (the previous commit) rather than the content being committed, which does not exist yet in pre-commit. Rewrite it to sum the sizes of the new blobs the commit would introduce (git diff --cached --raw, new blob ids -> cat-file --batch-check), keep the 2MB threshold and the documented --no-verify escape hatch, and move it before the prettier check so it always runs. Verified with a staged 3MB blob (blocked, exit 1), a small staged change (passes through to prettier), and a mode-only change (no new blob, passes).	2026-09-10 23:03:43 +00:00
dc	b3b9755aa7	chore(hooks): add gitleaks secret scan to pre-commit Build / Build-and-ng-test (pull_request) Failing after 1m45s Details Build / Build-and-test-development (pull_request) Skipped Details Lighthouse Checks / lighthouse (pull_request) Successful in 20m14s Details Run 'gitleaks protect --staged' before prettier in the pre-commit hook. The gitleaks binary comes from the @nogoo9/gitleaks devDependency, which ships official binaries as platform optionalDependencies - no postinstall scripts, so it installs under ignore-scripts=true. Falls back to a system gitleaks, and soft-skips with a hint to run 'npm i' on a fresh clone rather than blocking the commit.	2026-09-10 22:36:44 +00:00
hermes	82a45b80b4	chore(viya): run tests on the reusable compute context, harden chunk deploy Build / Build-and-ng-test (pull_request) Successful in 5m20s Details Lighthouse Checks / lighthouse (pull_request) Successful in 21m59s Details Build / Build-and-test-development (pull_request) Successful in 24m27s Details The test jobs on the nextviya estate run as a batch identity that is not authorised to create sessions in the SAS Job Execution compute context - child service submissions via mx_execute all failed with 403. Default the test context to Compute Reusable (matching the target contextName) in the shared test init programs. Also give the chunk deploy helper a retry loop for the per-chunk sasjs run timeouts - estate cold-starts can exceed a single 300s window.	2026-09-08 20:52:19 +01:00
hermes	a8194d73e7	test(viya): align stagedata test payloads with the adapter contract The web editor submits date/datetime/time cells as formatted strings and always provides the format-catalog buskey, so the staged CSVs round-trip through the ANYDT* informats in mpe_loader. Some tests built jsdata with raw numeric values or without FMTROW, causing the staged loads to fail on Viya. - getstagetable.test: send some_date/some_datetime/some_time as formatted strings (as postdata.test.1 already does), and assert on flags captured from the first read - the JSON libname can only be read once, so re-reading stagetable returns -1 observations - postdata.test.2: add fmtrow to the staged format catalog rows (the TYPE FMTNAME FMTROW buskey) - stagedata.test.3: assert the #211 hook aborts through MPE_LOADS - a hook-aborted child never returns a webout, and the load stays IN PROGRESS because mpe_loadfail cannot run after the abort. The WIDEBOY RLS rule is inserted in the prep section so the (possibly pooled) child session reads it from the shared filesystem	2026-09-08 20:52:19 +01:00
hermes	1dccf61f62	fix(admin): return refreshcatalog output as webout JSON The service wrote a raw HTML line straight to the _webout fileref after the load completed. On Viya, API-submitted jobs (eg via mx_execute) do not always have that fileref connected to a job result, so the output was lost and job-result consumers aborted. Return a proper sasparams table through %webout() instead - the same pattern as the other admin services - and update the test to assert on the webout sasparams message.	2026-09-08 20:52:19 +01:00
hermes	09f31ce6da	fix(deps): bump @sasjs/core to 5.2.7 Brings in the mv_jobflow JSON-escape fix (prevents jobExecution 400s for job arguments containing quotes or CR/LF) and the graceful handling of canceled jobs in mv_getjoblog / mv_getjobresult (a canceled child no longer aborts the calling program when its log or result is unavailable). Verified against a live Viya 2026.07 estate: the full SAS test suite passes with these macros sourced from core.	2026-09-08 20:52:19 +01:00