chore(release): 7.21.0 [skip ci]

# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)

### Bug Fixes

* **client:** parse word-date values from VA parameters ([405b28d](405b28dccc))
* **client:** parse word-date values from VA parameters ([ce5b1aa](ce5b1aa961))
* **makedata:** name the sasjs server group in the estate seed ([eeffa92](eeffa92f9e))

### Features

* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](a4a0f9cb2f))

.agents/skills/dc-estate-testing/SKILL.md

@@ -44,7 +44,7 @@ An unmatched `/SASjsApi/...` path answers `200` with the Swagger page, so check
 
 `npm run serverdata` sends `deploy/makeDataServer.json`: one `fromjs` row with `ADMIN`, `DCPATH` and `OUTFMT`. Two things about it are worth knowing before you trust its result.
 
-- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `AllUsers`, which is not a group on the shared estate - the account belongs to `all-users` (check with a call to `services/public/getgroups`, or read the `MF_GETUSER` the response echoes back). With a group the account is not in, the seed still answers `SUCCESS` and every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Override `ADMIN` to `all-users` for the estate rather than editing the shipped JSON.
+- **`ADMIN` must name a group the calling account is actually in.** The shipped value is `all-users`, the SASjs Server group that every logged-in user is a member of, which is what the shared estate needs. A name the account is not in - a Viya-style group such as `AllUsers`, or one estate's own group carried into another - still answers `SUCCESS` from the seed, and then every later `getdata`/`stagedata` call aborts with `<user> is not authorised to edit <lib>.<ds> in the <lib>.MPE_SECURITY table`. Check what the account belongs to with a call to `services/public/getgroups` before changing the value.
 - **Each run creates a new library.** The response carries `STATUS`, `DCPATH` and `LIBREF` (for example `DC621835`), and the LIBREF is clock-derived, so a libref from an earlier run answers `dataset <lib>.<table> does not exist` - take the current one from the response.
 - **`KEEPME` controls whether the seed can be repeated.** Without it `makedata` deletes itself and the configurator after a successful run (deliberately, so it cannot be executed twice over real data), and a second seed fails with `The Program at (<appLoc>/services/admin/makedata) does not exist` until you run `sasjs cbd` again. A harness that reseeds between runs sets `KEEPME` in the input row.
 - **Seed once per deploy, not before every test run.** The suite builds its own `dctest` library from the seeded control tables; reseeding in between is wasted work, and there is no need to run `makedata` and `sasjs test` in the opposite order.

CHANGELOG.md

@@ -1,3 +1,55 @@
+# [7.21.0](https://git.datacontroller.io/dc/dc/compare/v7.20.0...v7.21.0) (2026-10-29)
+
+
+### Bug Fixes
+
+* **client:** parse word-date values from VA parameters ([405b28d](https://git.datacontroller.io/dc/dc/commit/405b28dcccfe5994521bf63f0270d4877f544294))
+* **client:** parse word-date values from VA parameters ([ce5b1aa](https://git.datacontroller.io/dc/dc/commit/ce5b1aa96175d0fd1f048fea6d127cbc743804f2))
+* **makedata:** name the sasjs server group in the estate seed ([eeffa92](https://git.datacontroller.io/dc/dc/commit/eeffa92f9e973b323171bbefe2ef1cbc27e3aee7))
+
+
+### Features
+
+* **derived:** apply the core field scan in the MPE_DERIVED hook ([a4a0f9c](https://git.datacontroller.io/dc/dc/commit/a4a0f9cb2f7c0b6252405b71df64aae912d1b075))
+
+# [7.20.0](https://git.datacontroller.io/dc/dc/compare/v7.19.0...v7.20.0) (2026-10-28)
+
+
+### Bug Fixes
+
+* **ci:** install doxygen for the SAS docs job ([f58925b](https://git.datacontroller.io/dc/dc/commit/f58925b29956f1f753f0a367fdcd0afe7f3f51a0))
+* **ci:** keep foreign release assets, install jq first, pin the CLI ([27686d3](https://git.datacontroller.io/dc/dc/commit/27686d3aad5e34ace59ca1b735a17cca1a4025e1))
+* **ci:** publish against the tag this job cuts, not releases/latest ([8888734](https://git.datacontroller.io/dc/dc/commit/888873421c3aeaf312d5ec45e3c805317c66976a))
+* **ci:** publish docs after the assets, with the webdoc app password ([dd482c4](https://git.datacontroller.io/dc/dc/commit/dd482c4069202a0c6b9983129c19efc09bd160a7))
+* **ci:** publish the code documentation with surfer 7 and app passwords ([25ffa71](https://git.datacontroller.io/dc/dc/commit/25ffa71adddba5e334f4a8c166f284d1be498fcd))
+* **ci:** publish the typedoc to the webdoc site, not the app default ([c5808a1](https://git.datacontroller.io/dc/dc/commit/c5808a16eba0d29562fcc76f13ac4a33ab8da0b7))
+* **ci:** rebuild and upload release assets on demand ([6486c80](https://git.datacontroller.io/dc/dc/commit/6486c80ed4f368bbf116c0ecdecc0e5e5f02075f))
+* **ci:** use the lockfile and the pinned CLI in the docs job ([af1813f](https://git.datacontroller.io/dc/dc/commit/af1813f829861df41c89b512c824e8d3ebac7a4b))
+* **ci:** verify the forge certificate on the release API calls ([cf7b21a](https://git.datacontroller.io/dc/dc/commit/cf7b21a8b61ec3bf7ebe21968ccf13ade264bc64))
+* **cypress:** hold download out too, and retry once in CI ([c97e886](https://git.datacontroller.io/dc/dc/commit/c97e8866db9ee404b2118d24baf53fc7058debef)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395) [#398](https://git.datacontroller.io/dc/dc/issues/398)
+* **cypress:** register the deleteFolder task, restore the spec order, hold metanav out ([c803773](https://git.datacontroller.io/dc/dc/commit/c803773d3c36476d8263f01fc3b8344aa3503d44)), closes [#395](https://git.datacontroller.io/dc/dc/issues/395)
+* **deploy:** do not offer the deploy wizard inside a report ([80e8911](https://git.datacontroller.io/dc/dc/commit/80e8911af8857fa2c417f028cd59b64a87233b9e))
+* **deploy:** keep the app query when opening the standalone URL ([e51953c](https://git.datacontroller.io/dc/dc/commit/e51953ca8a6734e1649bad68cf4ad2057d677334))
+* **deploy:** one definition of the standalone URL, and test the guards ([459ff2d](https://git.datacontroller.io/dc/dc/commit/459ff2da371375e996839d69deb40f681c356f1d)), closes [#TCS](https://git.datacontroller.io/dc/dc/issues/TCS)
+* **deploy:** strip the embed parameters by name, and scope the timezone runs to the VA specs ([f7e7538](https://git.datacontroller.io/dc/dc/commit/f7e753863b61d8545bd0777d80267661b3c58670))
+* **embed:** build quarter ends from the calendar, not minus 24 hours ([7e3045a](https://git.datacontroller.io/dc/dc/commit/7e3045a21b67053b76a81bfe836e31571ccbffbf))
+* **embed:** defer the column check until metadata loads, report AND-ed pairs ([2623a3f](https://git.datacontroller.io/dc/dc/commit/2623a3fb4ba3bf19c00636b1717b53d5a2d54f3a))
+* **embed:** log a parse-time mapping problem once ([0011f7e](https://git.datacontroller.io/dc/dc/commit/0011f7e933d85be8e33debe3a6ecb9000edaabe3))
+* **embed:** make the va_parmap diagnostics say what they mean ([541a61b](https://git.datacontroller.io/dc/dc/commit/541a61ba08bc7256a911c50de7078859cc46c84e))
+* **embed:** match a mapping on the parameter label only, and warn only when the values differ ([e9216b0](https://git.datacontroller.io/dc/dc/commit/e9216b08956d5599662259a2b05cc1524c458df5))
+* **embed:** never build a clause from a value that did not convert, and say so in the banner ([bd3c220](https://git.datacontroller.io/dc/dc/commit/bd3c220af2678a2a3f8394fe3084ee03c23d0b15))
+* **embed:** one parameter maps one column, and report typos on both sides ([f83dc35](https://git.datacontroller.io/dc/dc/commit/f83dc359e258f5d1cb9fc49f0eaab9748995d823))
+* **embed:** record the parameter label only, so a name-keyed va_parmap is reported ([aa91daf](https://git.datacontroller.io/dc/dc/commit/aa91daf9f78c00cea486d0f885f8e634366bd1a0))
+* **embed:** resolve transform values as local dates and give ranges whole days ([10157c1](https://git.datacontroller.io/dc/dc/commit/10157c11595685efdc206fe7b16a5cafa098f26d))
+* **embed:** warn when a va_parmap entry is ignored ([411f6c3](https://git.datacontroller.io/dc/dc/commit/411f6c3354b7fbfec11fef3b174ae1f8066c92ba))
+* **makedata:** pair the %else of the keepme block so it parses in open code ([c0ea810](https://git.datacontroller.io/dc/dc/commit/c0ea810766ebfda5ae5358ab14146fbcc07fcc8e))
+
+
+### Features
+
+* **embed:** map VA parameters to DC columns from the URL ([15a1f34](https://git.datacontroller.io/dc/dc/commit/15a1f34d5263503cac83b4fe4e52482777dc2f1f))
+* **embed:** resolve va_parmap values through named date anchors ([88010c3](https://git.datacontroller.io/dc/dc/commit/88010c3980c2ea1c39bbb024ed32d63e495e61f0))
+
 # [7.19.0](https://git.datacontroller.io/dc/dc/compare/v7.18.0...v7.19.0) (2026-10-28)
 
 

client/src/app/services/va-filter.service.spec.ts

@@ -131,6 +131,41 @@ describe('VaFilterService', () => {
       expect(build([{ label: 'SOME_DATE', value: '2025/03/31' }])).toHaveSize(1)
     })
 
+    it('parses word dates, in either order and with an optional comma', () => {
+      const expected = "'01JAN2026'd"
+      // The reported shape: a date parameter displayed as WORDDATE.
+      expect(
+        build([{ label: 'SOME_DATE', value: 'January 1, 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: 'Jan 1, 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: 'Jan. 1, 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: '1 January 2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+      expect(
+        build([{ label: 'SOME_DATE', value: '1-Jan-2026' }])[0].RAW_VALUE
+      ).toBe(expected)
+    })
+
+    it('matches a spelled-out month case-insensitively, and keeps DATE9 working', () => {
+      expect(
+        build([{ label: 'SOME_DATE', value: 'DECEMBER 31, 2025' }])[0].RAW_VALUE
+      ).toBe("'31DEC2025'd")
+      expect(
+        build([{ label: 'SOME_DATE', value: '01AUG1962' }])[0].RAW_VALUE
+      ).toBe("'01AUG1962'd")
+    })
+
+    it('still drops a month-name value that has no day', () => {
+      // A month and year alone is not a date - accepted shapes always carry a
+      // day, so this must not silently become the first of the month.
+      expect(build([{ label: 'SOME_DATE', value: 'January 2026' }])).toEqual([])
+    })
+
     it('resolves ambiguous slash dates day-first (never US), disambiguating by value', () => {
       const iso = build([{ label: 'SOME_DATE', value: '2025-03-31' }])[0]
         .RAW_VALUE

client/src/app/services/va-filter.service.ts

@@ -707,14 +707,42 @@ export class VaFilterService {
     return this.parseMonthNameDate(value) ?? this.parseNumericDate(value, order)
   }
 
-  /** DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62) -> local Date, or null. */
+  /**
+   * Month-name date shapes -> local Date, or null. Three families:
+   *  - DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62);
+   *  - word date with the month first, comma optional: "January 1, 2026",
+   *    "Jan 1 2026", "Jan. 1, 2026";
+   *  - word date with the day first: "1 January 2026", "1-Jan-2026".
+   *
+   * A date parameter's display format decides which of these the report author
+   * sees, and VA sends the formatted text, so a report using WORDDATE (or a
+   * long month name in any order) has to parse here rather than being dropped.
+   * The month name may be abbreviated or spelled out; only its first three
+   * letters are matched.
+   */
   private parseMonthNameDate(value: string): Date | null {
-    const m = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
-    if (!m) return null
-    const [, day, monthAbbr, year] = m
-    const monthIndex = VaFilterService.SAS_MONTHS.indexOf(
-      monthAbbr.toUpperCase()
-    )
+    const monthByName = (name: string): number =>
+      VaFilterService.SAS_MONTHS.indexOf(
+        name.replace(/\./g, '').slice(0, 3).toUpperCase()
+      )
+
+    const date9 = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
+    if (date9) {
+      const [, day, monthAbbr, year] = date9
+      const monthIndex = monthByName(monthAbbr)
+      if (monthIndex < 0) return null
+      return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
+    }
+
+    const monthFirst =
+      /^([A-Za-z]{3,9})\.?[\s-]+(\d{1,2}),?[\s-]+(\d{2}|\d{4})$/.exec(value)
+    const dayFirst =
+      /^(\d{1,2})[\s-]+([A-Za-z]{3,9})\.?,?[\s-]+(\d{2}|\d{4})$/.exec(value)
+    const name = monthFirst ? monthFirst[1] : dayFirst ? dayFirst[2] : null
+    if (!name) return null
+    const day = monthFirst ? monthFirst[2] : dayFirst![1]
+    const year = monthFirst ? monthFirst[3] : dayFirst![3]
+    const monthIndex = monthByName(name)
     if (monthIndex < 0) return null
     return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
   }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "dcfrontend",
-  "version": "7.19.0",
+  "version": "7.21.0",
   "description": "Data Controller",
   "devDependencies": {
     "@nogoo9/gitleaks": "8.30.1-post.2",

sas/deploy/makeDataServer.json

@@ -1,7 +1,7 @@
 {
   "fromjs": [
     {
-      "ADMIN": "AllUsers",
+      "ADMIN": "all-users",
       "DCPATH": "/tmp/dcdata",
       "OUTFMT": "JSON"
     }

sas/mocks/sasjs/services/hooks/mpe_derived_postedit.js

@@ -13,10 +13,12 @@
 // The rules mirror services/hooks/mpe_derived_postedit.sas, in its order:
 // normalise the incoming rows, reject a duplicate name and a bad field, then
 // check the dependency graph that the resulting configuration would have (the
-// incoming rows plus the current rows they do not replace).  As on the server,
-// the collision between VAR_NM and a physical column is left to the read-time
-// engine, which runs with the target library assigned - the hook does not have
-// it here either.
+// incoming rows plus the current rows they do not replace).  The field rules
+// are core's mp_derivedvarcheck, which the server hook calls and the read-time
+// engine runs too - this file spells them out a second time, so a change there
+// has to be mirrored here.  As on the server, the collision between VAR_NM and
+// a physical column is left to the read-time engine, which runs with the target
+// library assigned - the hook does not have it here either.
 // ─────────────────────────────────────────────────────────────────────────────
 
 ;(function mpeDerivedPostedit() {
@@ -83,9 +85,45 @@
         seen.add(key)
     }
 
-    // ─── Field level checks, the rules the engine applies at read time ─────────
-    // C0 controls and DEL: what a single-line label cannot carry
+    // ─── Field level checks ────────────────────────────────────────────────────
+    // Two passes, in the order the SAS hook runs them.  The first is the fields
+    // this hook owns - the control table key, which the loader normalises but
+    // does not require, and ACTIVE, the picker pair the EDIT screen offers.
+    for (const row of rows) {
+        if (isDeleted(row)) continue
+        const name = field(row, 'VAR_NM')
+        let reason = ''
+        if (!field(row, 'LIBREF') || !field(row, 'DSN'))
+            reason = 'LIBREF and DSN are required'
+        // ACTIVE is a picker in the EDIT screen (HARDSELECT over the
+        // MPE_SELECTBOX pair) - the same pair the hook enforces
+        else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
+            reason = 'ACTIVE must be 1 or 0'
+        if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
+    }
+
+    // The second pass is the rules for the derived variable itself, which are
+    // core's (mp_derivedvarcheck): the engine runs the same macro at read time,
+    // so a definition is measured against one rule list wherever it is checked.
+
+    // C0 controls and DEL: what a label cannot carry once a line break has been
+    // flattened to a space, which is what the engine does with it
     const CONTROL_CHARS = /[\u0000-\u001F\u007F]/
+    const flatten = (v) => v.replace(/[\r\n]+/g, ' ')
+    const AUTOMATIC_VARS = ['_N_', '_ERROR_', '_IORC_']
+
+    // the capabilities refused by name - a function name cannot be assembled at
+    // run time, so naming them is a control the expression cannot evade
+    const DENY_FUNCS = new RegExp(
+        '\\b(DOSUBL|JAVAOBJ|JAVAMETA|SYMGET|SYMPUTX?|SYSGET|GETOPTION' +
+            '|PATHNAME|FILEEXIST|FOPEN|FREAD|FGET|FCLOSE' +
+            '|DOPEN|DREAD|DNUM|MOPEN|SLEEP' +
+            '|FILENAME|FDELETE|FCOPY|DCREATE|DLCREATE)\\s*\\('
+    )
+
+    // quoted literals are data, not calls
+    const unquoted = (v) =>
+        v.replace(/'([^']|'')*'/g, ' ').replace(/"([^"]|"")*"/g, ' ')
 
     // true when the value carries a character outside the allowed set
     const hasInvalidChar = (value, chars) =>
@@ -93,8 +131,6 @@
 
     for (const row of rows) {
         if (isDeleted(row)) continue
-        const libref = field(row, 'LIBREF')
-        const dsn = field(row, 'DSN')
         const name = field(row, 'VAR_NM')
         const type = field(row, 'VAR_TYPE')
         const len = Number(colVal(row, 'VAR_LEN'))
@@ -103,19 +139,16 @@
         const logic = strip(colVal(row, 'VAR_LOGIC'))
 
         let reason = ''
-        if (!libref || !dsn) reason = 'LIBREF and DSN are required'
-        else if (!name) reason = 'VAR_NM is required'
+        if (!name) reason = 'VAR_NM is required'
         else if (hasInvalidChar(name, NAME_CHARS))
             reason = 'VAR_NM must be a valid SAS name'
         else if ('0123456789'.includes(name[0]))
             reason = 'VAR_NM must not start with a digit'
+        else if (AUTOMATIC_VARS.includes(name))
+            reason = 'VAR_NM is an automatic variable of the data step'
         else if (name === DELETE_MARK) reason = 'VAR_NM is reserved'
         else if (type !== 'C' && type !== 'N')
             reason = 'VAR_TYPE must be C or N'
-        // ACTIVE is a picker in the EDIT screen (HARDSELECT over the
-        // MPE_SELECTBOX pair) - the same pair mpe_derived_check.sas enforces
-        else if (!['0', '1'].includes(String(colVal(row, 'ACTIVE')).trim()))
-            reason = 'ACTIVE must be 1 or 0'
         else if (!Number.isInteger(len) || len < 1)
             reason = 'VAR_LEN must be a positive integer'
         else if (type === 'C' && len > 32767)
@@ -132,20 +165,24 @@
             reason = 'VAR_FMT must be a format name, without ; & % or /* */'
         else if (hasInvalidChar(fmt, FMT_CHARS))
             reason = 'VAR_FMT must be a format name'
-        // free text, in any script - what is refused is anything that cannot be
-        // a single-line column label, and a value the engine would cut to 256
-        else if (CONTROL_CHARS.test(label))
-            reason =
-                'VAR_LABEL must be plain text - no line breaks or control characters'
+        // free text, in any script - what is refused is a control character,
+        // and a value the engine would cut to 256
+        else if (CONTROL_CHARS.test(flatten(label)))
+            reason = 'VAR_LABEL must be plain text - no control characters'
         else if ([...label].length > 256)
             reason = 'VAR_LABEL exceeds 256 characters'
         else if (!logic) reason = 'VAR_LOGIC is required'
+        else if (logic.length > 4000)
+            reason = 'VAR_LOGIC exceeds 4000 characters'
         else if (logic.includes(';'))
             reason = 'VAR_LOGIC must be a single expression - remove the ;'
         else if (logic.includes('&') || logic.includes('%'))
             reason = 'VAR_LOGIC must not contain macro triggers'
         else if (logic.includes('/*') || logic.includes('*/'))
             reason = 'VAR_LOGIC must not contain comments'
+        else if (DENY_FUNCS.test(unquoted(logic).toUpperCase()))
+            reason =
+                'VAR_LOGIC may not call a function that submits code or reaches the file system, the environment or the macro symbol table'
 
         if (reason) return reject(`MPE_DERIVED: ${name}: ${reason}`)
     }

sas/package-lock.json

@@ -7,7 +7,7 @@
       "name": "dc-sas",
       "dependencies": {
         "@sasjs/cli": "6.2.0",
-        "@sasjs/core": "6.1.0"
+        "@sasjs/core": "6.1.4"
       }
     },
     "node_modules/@asamuzakjp/css-color": {
@@ -235,9 +235,9 @@
       }
     },
     "node_modules/@sasjs/core": {
-      "version": "6.1.0",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.0.tgz",
-      "integrity": "sha512-U0Ekq/TXVMquYV+AG6urBQSUgqUvV37yduufy68QQVgDj3PJiLUJDcOXKOGbhWGD+Nv877VuPCviGYqrqbjXvg==",
+      "version": "6.1.4",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-6.1.4.tgz",
+      "integrity": "sha512-vIF2LIftxPWg4CvO3i5raEfrHFHY/6S+MqVGpjRR71dgjLRJEYNyPJgGqD6BCZwHv5V6/2nDnBi/BIIMiLXfzA==",
       "license": "MIT"
     },
     "node_modules/@sasjs/lint": {

sas/package.json

@@ -34,7 +34,7 @@
   "private": true,
   "dependencies": {
     "@sasjs/cli": "6.2.0",
-    "@sasjs/core": "6.1.0"
+    "@sasjs/core": "6.1.4"
   },
   "overrides": {
     "nanoid": "3.3.18"

sas/sasjs/macros/mpe_derived.sas

@@ -49,9 +49,19 @@
   being maintained.  A cycle is rejected.
 
   Semicolons, macro triggers and comments are rejected, so a row can only ever
-  be an expression.  A derived variable may not reuse the name of a physical
-  column, and may not reference itself: an intra-row calculation would imply a
-  retained value, which depends on the order rows happen to arrive in.
+  be an expression, and an expression that would let the row submit code of its
+  own - DOSUBL, or a function reaching the file system, the environment or the
+  macro symbol table - is refused by name.
+
+  The field level rules themselves are core's: mp_derivedvarcheck scans the
+  definitions and mp_derivedvars calls it, which is the same macro the MPE_DERIVED
+  post edit hook runs at submit time (services/hooks/mpe_derived_postedit).  A
+  row is therefore measured against one rule list, whether it is being staged
+  or read.
+
+  A derived variable may not reuse the name of a physical column, and may not
+  reference itself: an intra-row calculation would imply a retained value,
+  which depends on the order rows happen to arrive in.
 
   <h4> SAS Macros </h4>
   @li mp_derivedvars.sas

sas/sasjs/macros/mpe_derived_check.sas

@@ -1,75 +0,0 @@
-/**
-  @file
-  @brief Field level checks for a staged MPE_DERIVED submission
-  @details Writes one row per staged row that breaks a rule, with the reason in
-  REASON, so a caller can report the first failure and abort.  Rows marked for
-  deletion are skipped - they are being removed, so their fields are not loaded.
-
-  The rules are the ones %mpe_derived applies at read time, so a submission
-  cannot stage a row the engine would then refuse.
-
-  @param inds the staged rows, already normalised by the caller
-  @param outds the dataset to create: one row per failing row, plus REASON
-
-  @author 4GL Apps Ltd
-  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
-    and may not be re-distributed or re-sold without the express permission of
-    4GL Apps Ltd.
-
-**/
-
-%macro mpe_derived_check(inds=work.staging_ds, outds=work.derived_check);
-data &outds;
-  set &inds;
-  length reason $200;
-  if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
-  call missing(reason);
-
-  if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
-  else if var_nm='' then reason='VAR_NM is required';
-  else if verify(strip(var_nm),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_')>0
-    then reason='VAR_NM must be a valid SAS name';
-  else if verify(substr(var_nm,1,1),'0123456789')=0
-    then reason='VAR_NM must not start with a digit';
-  else if var_nm='_____DELETE__THIS__RECORD_____'
-    then reason='VAR_NM is reserved';
-  else if var_type not in ('C','N')
-    then reason='VAR_TYPE must be C or N';
-  /* ACTIVE is a picker in the EDIT screen (HARDSELECT over the MPE_SELECTBOX
-    pair) - the same pair is enforced here, so a submission that bypasses the
-    grid cannot land a value the screen would never offer. */
-  else if missing(active) or active not in (0,1)
-    then reason='ACTIVE must be 1 or 0';
-  else if missing(var_len) or var_len ne int(var_len) or var_len<1
-    then reason='VAR_LEN must be a positive integer';
-  else if var_type='C' and var_len>32767
-    then reason='VAR_LEN exceeds 32767 for a character variable';
-  else if var_type='N' and var_len>8
-    then reason='VAR_LEN exceeds 8 for a numeric variable';
-  else if index(var_fmt,';')>0 or indexc(var_fmt,'&%')>0
-    or index(var_fmt,'/*')>0 or index(var_fmt,'*/')>0
-    then reason='VAR_FMT must be a format name, without ; & % or /* */';
-  else if verify(strip(var_fmt),'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$.')>0
-    then reason='VAR_FMT must be a format name';
-  /* VAR_LABEL is free text: any character is allowed, in any script, and the
-    quotes in it are escaped by %mp_derivedvars when the attrib statement is
-    built.  What is refused is anything that cannot be a single-line column
-    label - a control character or a line break - and a value long enough to
-    be cut down silently, since the engine truncates the label to 256. */
-  else if anycntrl(var_label)>0
-    then reason='VAR_LABEL must be plain text - no line breaks or control characters';
-  else if klength(var_label)>256
-    then reason='VAR_LABEL exceeds 256 characters';
-  else if missing(var_logic)
-    then reason='VAR_LOGIC is required';
-  else if index(var_logic,';')>0
-    then reason='VAR_LOGIC must be a single expression - remove the ;';
-  else if indexc(var_logic,'&%')>0
-    then reason='VAR_LOGIC must not contain macro triggers';
-  else if index(var_logic,'/*')>0 or index(var_logic,'*/')>0
-    then reason='VAR_LOGIC must not contain comments';
-
-  if not missing(reason) then output;
-run;
-
-%mend mpe_derived_check;

sas/sasjs/macros/mpe_derived_check.test.sas

@@ -1,194 +0,0 @@
-/**
-  @file
-  @brief testing the mpe_derived_check macro
-
-  @details Checks each field level rule the MPE_DERIVED post edit hook applies
-  to a staged submission.  The macro is exercised directly so the rules can be
-  probed with values a service cannot carry: VAR_LABEL is varchar(256) in the
-  table and a staging dataset declares the same length, so a longer label is
-  truncated before any check would see it.
-
-  <h4> SAS Macros </h4>
-  @li mf_nobs.sas
-  @li mp_assert.sas
-  @li mpe_derived_check.sas
-
-  @author 4GL Apps Ltd
-  @copyright 4GL Apps Ltd.  This code may only be used within Data Controller
-    and may not be re-distributed or re-sold without the express permission of
-    4GL Apps Ltd.
-
-**/
-
-/**
-  * Test 1 - one row per failing rule, and only failing rows
-  */
-data work.staging_ds;
-  length libref $8 dsn $32 var_nm $32 var_type $1 var_len 8
-    var_fmt $32 var_label $4000 var_deps $4000 var_logic $4000
-    active 8 _____DELETE__THIS__RECORD_____ $3;
-  libref='DCTEST';
-  dsn='DC_DERIV';
-  /* valid unless a row says otherwise - ACTIVE is mandatory in the screen and
-    the check enforces the same pair here */
-  active=1;
-  _____DELETE__THIS__RECORD_____='No';
-
-  /* meets every rule */
-  var_nm='GOOD';
-  var_type='N';
-  var_len=8;
-  var_fmt='';
-  var_label='a plain label';
-  var_deps='';
-  var_logic='1';
-  output;
-
-  /* delete marked rows are being removed, so their fields are not loaded */
-  var_nm='DELME';
-  var_label='first line'||'0A'x||'second line';
-  _____DELETE__THIS__RECORD_____='Yes';
-  output;
-  _____DELETE__THIS__RECORD_____='No';
-
-  /* free text is allowed, including quotes and punctuation */
-  var_nm='FREETXT';
-  var_label='It''s "quoted", plain text: 100 percent';
-  output;
-
-  /* a line break cannot be a single line column label */
-  var_nm='LBLBREAK';
-  var_label='first line'||'0A'x||'second line';
-  output;
-
-  /* longer than the table holds, so it would be cut down silently */
-  var_nm='LBLLONG';
-  var_label=repeat('x',300);
-  output;
-
-  /* a format name, not an expression */
-  var_nm='BADFMT';
-  var_fmt='DATE9;';
-  var_label='fmt';
-  output;
-
-  /* one expression, no statement separator, no macro triggers */
-  var_nm='BADLOGIC';
-  var_fmt='';
-  var_logic='x; y';
-  output;
-
-  /* C or N */
-  var_nm='BADTYPE';
-  var_logic='1';
-  var_type='X';
-  output;
-
-  /* ACTIVE is a picker in the screen: only 1 or 0 */
-  var_nm='BADACTIVE';
-  var_type='C';
-  var_logic='1';
-  active=2;
-  output;
-
-  stop;
-run;
-
-%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
-
-/**
-  * Test 2 - the reported rows, rule by rule
-  */
-data work.expected;
-  length var_nm $32 reason $200;
-  var_nm='LBLBREAK';
-  reason='VAR_LABEL must be plain text - no line breaks or control characters';
-  output;
-  var_nm='LBLLONG';
-  reason='VAR_LABEL exceeds 256 characters';
-  output;
-  var_nm='BADFMT';
-  reason='VAR_FMT must be a format name, without ; & % or /* */';
-  output;
-  var_nm='BADLOGIC';
-  reason='VAR_LOGIC must be a single expression - remove the ;';
-  output;
-  var_nm='BADTYPE';
-  reason='VAR_TYPE must be C or N';
-  output;
-  var_nm='BADACTIVE';
-  reason='ACTIVE must be 1 or 0';
-  output;
-run;
-
-proc sql;
-create table work.cmp as
-  select e.var_nm
-    ,case when a.reason=e.reason then 1 else 0 end as same
-  from work.expected e
-  left join work.derived_check a
-    on upcase(strip(a.var_nm))=upcase(e.var_nm);
-quit;
-
-data _null_;
-  set work.cmp;
-  call symputx(cats('rule_',var_nm),same,'l');
-run;
-
-%mp_assert(
-  iftrue=(%mf_nobs(work.derived_check)=6),
-  desc=%str(Checking only the six failing rows are reported),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_LBLBREAK=1),
-  desc=%str(Checking a label with a line break is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_LBLLONG=1),
-  desc=%str(Checking a label over 256 characters is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADFMT=1),
-  desc=%str(Checking a format carrying a statement separator is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADLOGIC=1),
-  desc=%str(Checking a logic value with a statement separator is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADTYPE=1),
-  desc=%str(Checking a VAR_TYPE other than C or N is rejected),
-  outds=work.test_results
-)
-
-%mp_assert(
-  iftrue=(&rule_BADACTIVE=1),
-  desc=%str(Checking an ACTIVE outside 1 or 0 is rejected),
-  outds=work.test_results
-)
-
-/**
-  * Test 3 - a row that meets every rule, a free text label, and a delete
-  * marked row are all left alone
-  */
-proc sql noprint;
-select count(*) into: notchecked
-  from work.derived_check
-  where upcase(strip(var_nm)) in ('GOOD','FREETXT','DELME');
-quit;
-
-%mp_assert(
-  iftrue=(&notchecked=0),
-  desc=%str(Checking a valid row, a free text label and a delete marked row are not reported),
-  outds=work.test_results
-)

sas/sasjs/macros/mpe_makedata.sas

@@ -2355,7 +2355,9 @@ insert into &lib..MPE_VALIDATIONS set
       ,some_char="&x bottles of beer on the wall"
       ,some_dropdown='Option 1'
       ,some_num=ranuni(0)
-      ,some_date=round(ranuni(0)*1000,1)
+      /* First of the month, cycling through two years: a randomly spread date
+      gave every row its own day, so a date filter never returned many rows. */
+      ,some_date=intnx('month','01JAN1960'd,mod(&x,24),'S')
       ,some_datetime=round(ranuni(0)*50000,1)
       ,some_time=round(ranuni(0)*100,1)
       ,some_shortnum=6+round(ranuni(0)*94,1)

sas/sasjs/services/editors/stagedata.test.4.sas

@@ -6,16 +6,19 @@
       table - the engine reads only active rows, so the dependency can never
       resolve at run time
     2. a row is staged with active=0 while a surviving active row depends on it
+    3. a staged row breaks a field rule of the shared scan (core's
+      mp_derivedvarcheck), or carries an ACTIVE value the EDIT screen's picker
+      cannot produce
 
   They should NOT abort when:
-    3. a new active row is added to a table that also holds an inactive row -
+    4. a new active row is added to a table that also holds an inactive row -
       the inactive row is not part of the dependency graph, so it cannot
       create a cycle or a stale reference
-    4. a delete-marked RLS rule naming a derived variable is submitted - the
+    5. a delete-marked RLS rule naming a derived variable is submitted - the
       rule is being removed, so its variables are not validated
-    5. a delete and re-add of the same variable is submitted together - the
+    6. a delete and re-add of the same variable is submitted together - the
       re-added active row resolves the dependency
-    6. an inactive row is set active again while another active row depends on
+    7. an inactive row is set active again while another active row depends on
       it - the re-activation is what makes the dependency resolvable
 
   The staged loads are aborted by the post edit hook. The assertion fetches the
@@ -553,6 +556,103 @@ run;
 )
 
 
+/**
+  * Test 8 - a field rule of the shared scan (mp_derivedvarcheck, core) is
+  * applied at submit time: an expression that could submit code of its own is
+  * refused by name, with the reason the scan gives. This is the same scan the
+  * engine runs at read time, so the case pins that the hook calls it rather
+  * than carrying a rule list of its own.
+  */
+data work.sascontroltable;
+  action='LOAD';
+  message='A refused function in VAR_LOGIC should abort';
+  libds="&dc_libref..MPE_DERIVED";
+  output;
+  stop;
+run;
+
+data work.jsdata;
+  length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
+    VAR_LOGIC $4000 VAR_FMT $32
+    _____DELETE__THIS__RECORD_____ $3;
+  LIBREF='DCTEST';
+  DSN='DC_DERIV';
+  VAR_NM='CALLER';
+  VAR_DEPS='';
+  VAR_TYPE='N';
+  VAR_LEN=8;
+  VAR_FMT='';
+  /* no semicolon in the payload: the single-expression rule is tested before
+    the function-name rule, so a quoted semicolon would abort on that first */
+  VAR_LOGIC='dosubl("data _null_")';
+  ACTIVE=1;
+  _____DELETE__THIS__RECORD_____='No';
+run;
+
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.sascontroltable work.jsdata,
+  outlib=web8,
+  outref=wb8,
+  mdebug=&sasjs_mdebug
+)
+
+/* the assertion compares the abort's MSG exactly, so the expected value is
+  assembled on one line: a literal wrapped across two source lines carries the
+  newline and the continuation indent with it */
+%let exp8=MPE_DERIVED: CALLER: VAR_LOGIC may not call a function that;
+%let exp8=&exp8 submits code or reaches the file system,;
+%let exp8=&exp8 the environment or the macro symbol table;
+
+%mpe_assert_abort(
+  desc=Checking a refused function in VAR_LOGIC is rejected,
+  outref=wb8,
+  msg=&exp8
+)
+
+/**
+  * Test 9 - the fields the hook owns rather than the scan: ACTIVE is a picker
+  * in the EDIT screen (HARDSELECT over the MPE_SELECTBOX pair), so a submission
+  * that bypasses the grid cannot land a value the screen would never offer.
+  */
+data work.sascontroltable;
+  action='LOAD';
+  message='An ACTIVE value the screen cannot offer should abort';
+  libds="&dc_libref..MPE_DERIVED";
+  output;
+  stop;
+run;
+
+data work.jsdata;
+  length LIBREF $8 DSN $32 VAR_NM $32 VAR_DEPS $4000 VAR_TYPE $1
+    VAR_LOGIC $4000 VAR_FMT $32
+    _____DELETE__THIS__RECORD_____ $3;
+  LIBREF='DCTEST';
+  DSN='DC_DERIV';
+  VAR_NM='BADACT';
+  VAR_DEPS='';
+  VAR_TYPE='N';
+  VAR_LEN=8;
+  VAR_FMT='';
+  VAR_LOGIC='1';
+  ACTIVE=2;
+  _____DELETE__THIS__RECORD_____='No';
+run;
+
+%mpe_execute(&_program,
+  viyacontext=&defaultcontext,
+  inputdatasets=work.sascontroltable work.jsdata,
+  outlib=web9,
+  outref=wb9,
+  mdebug=&sasjs_mdebug
+)
+
+%mpe_assert_abort(
+  desc=Checking an ACTIVE value outside the picker pair is rejected,
+  outref=wb9,
+  msg=MPE_DERIVED: BADACT: ACTIVE must be 1 or 0
+)
+
 /* tidy up so repeated runs start clean */
 proc sql noprint;
 delete from &dc_libref..mpe_derived

sas/sasjs/services/hooks/mpe_derived_postedit.sas

@@ -34,7 +34,7 @@
   <h4> SAS Macros </h4>
   @li mf_nobs.sas
   @li mp_abort.sas
-  @li mpe_derived_check.sas
+  @li mp_derivedvarcheck.sas
 
   <h4> Related Macros </h4>
   @li mpe_derived.sas
@@ -100,14 +100,43 @@ run;
 )
 
 /**
-  * Field level checks - the same rules %mpe_derived applies at read time
+  * Field level checks.  The rules for the derived variable itself - the name,
+  * the type, length, format, label and the expression - are core's, in
+  * mp_derivedvarcheck, so this hook and the read time engine (%mpe_derived ->
+  * mp_derivedvars) scan a row with the same rules and report the same reasons.
+  * What is added here are the two fields that belong to Data Controller rather
+  * than to the expression: the control table key, which the loader normalises
+  * but does not require, and ACTIVE, which the EDIT screen offers as a picker
+  * over the MPE_SELECTBOX pair - so a submission that bypasses the grid cannot
+  * land a value the screen would never offer.
   */
-%mpe_derived_check(inds=work.staging_ds, outds=work.derived_check)
+data work.derived_dcfield(keep=name reason);
+  set work.staging_ds;
+  if upcase(_____DELETE__THIS__RECORD_____)='YES' then delete;
+  length name $32 reason $200;
+  call missing(reason);
+  name=upcase(strip(var_nm));
+  if missing(libref) or missing(dsn) then reason='LIBREF and DSN are required';
+  else if missing(active) or active not in (0,1)
+    then reason='ACTIVE must be 1 or 0';
+  if not missing(reason) then output;
+run;
+
+%mp_derivedvarcheck(inds=work.staging_ds
+  ,outds=work.derived_field
+  ,reserved=%str(_____DELETE__THIS__RECORD_____)
+)
+
+/* the control table fields are reported before the expression's own rules, so
+  which of the two passes found a row does not change the message it gets */
+data work.derived_check;
+  set work.derived_dcfield work.derived_field;
+run;
 
 %if %mf_nobs(work.derived_check)>0 %then %do;
   data _null_;
     set work.derived_check(obs=1);
-    call symputx('der_errmsg',cats('MPE_DERIVED: ',var_nm,': ',reason));
+    call symputx('der_errmsg',cats('MPE_DERIVED: ',name,': ',reason));
     call symputx('der_errflag',1);
   run;
 %end;