From 4d53ced89c1e7f872e9d8b50bb5404cfef455a09 Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 07:50:06 +0000 Subject: [PATCH 1/7] docs(validations): document how each rule treats a special missing value New Special Missing Values section covering NOTNULL, MINVAL/MAXVAL, CASE, ROUND, the regex rules, the dropdowns and the formula rules, with the 'not supported' set called out. Behaviour verified against a real SAS estate and the deployed frontend validator. --- docs/dcc-validations.md | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index 9f7977d..d1bb4f1 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -23,12 +23,12 @@ It is possible to configure a number of other rules by updating the MPE_VALIDATI ## Configurable Checks -Check back frequently as we keep growing this list of checks. +Check back frequently as we keep growing this list of checks. For how each rule treats a special missing value (`.A`-`.Z`, `._`), see [Special Missing Values](#special-missing-values). |Rule Type|Example Value |Description| |---|---|---| |CASE|UPCASE|Will enforce the case of cell values. Valid values: UPCASE, LOWCASE, PROPCASE| -|NOTNULL|(defaultval)|Will prevent submission if null values are present. Optional - provide a default value.| +|NOTNULL|(defaultval)|Will prevent submission if null values are present - a special missing counts as null. Optional - provide a default value.| |MINVAL|1|Defines a minimum value for a numeric cell| |MAXVAL|1000000|Defines a maximum value for a numeric cell| |READONLY|(defaultval) |Renders the column read-only in the editor. The defaultval is used when rows are added. | @@ -45,6 +45,28 @@ Check back frequently as we keep growing this list of checks. |[SOFTSELECT_HOOK](/dynamic-cell-dropdown)|/physical/path/program.sas|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User-provided values may (or may not) be in this list. Cannot be used alongside a HARDSELECT_HOOK.| +## Special Missing Values + +A SAS numeric variable has 28 missing values: the regular `.` and 27 special ones (`.A`-`.Z` and `._`). Data Controller accepts them in a numeric cell as a bare letter - `a` to `z`, or `_`, with no period - and stores the matching special missing in SAS. + +The rules treat a special missing as a value for some checks and as a missing value for others: + +|Rule|What happens to a special missing| +|---|---| +|NOTNULL|Fails. A special missing is null to a NOT NULL (or primary key) constraint, and a real SAS NOT NULL constraint on the target table rejects it too.| +|MINVAL|Fails - a special missing sorts below every number, so it is below any minimum.| +|MAXVAL|Passes - but the rule is then pointless: a special missing is below every maximum, so the missings pass and the column's real numbers fail.| +|CASE|Not applicable - it is a character rule.| +|ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.| +|HARDREGEX / SOFTREGEX|Checked against the pattern like any other value. Unlike blanks and the plain `.`, special missings are **not** exempt. See [Regex Rules](#regex-rules).| +|SOFTSELECT|Passes - a soft dropdown never blocks a value.| +|HARDSELECT|Fails - the strict membership test cannot match it (the dropdown list for a numeric column holds numbers).| +|HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.| + +In short, a special missing is **not supported** in a column that carries NOTNULL, MINVAL, MAXVAL, HARDSELECT or a formula. It is fine alongside the regex rules, a soft dropdown, and ROUND. + +One related gotcha: a range rule's own value must be a number. Put a special missing in `MINVAL` and every cell in the column fails; put one in `MAXVAL` and every real number fails. + ## Formula Rules HARDFORMULA and SOFTFORMULA let you configure a column so that its value is automatically calculated from other columns in the same row - just like a spreadsheet formula. When a user opens the editor, the formula is evaluated live and the result is shown in each cell. From e7fdb3ab52b62b7a1e37aa81fac68c836230e999 Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 08:19:40 +0000 Subject: [PATCH 2/7] docs(validations): primary keys are NOT NULL; HARDSELECT can match a special missing - a primary key column rejects a blank and a special missing whether or not the table carries a constraint or MPE_VALIDATIONS a NOTNULL rule - HARDSELECT is now checked like any other value: it passes when the dropdown list contains the special missing - HARDSELECT drops out of the 'not supported' list --- docs/dcc-validations.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index d1bb4f1..e3053b8 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -60,10 +60,12 @@ The rules treat a special missing as a value for some checks and as a missing va |ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.| |HARDREGEX / SOFTREGEX|Checked against the pattern like any other value. Unlike blanks and the plain `.`, special missings are **not** exempt. See [Regex Rules](#regex-rules).| |SOFTSELECT|Passes - a soft dropdown never blocks a value.| -|HARDSELECT|Fails - the strict membership test cannot match it (the dropdown list for a numeric column holds numbers).| +|HARDSELECT|Checked against the dropdown list like any other value - it passes only if the list contains it.| |HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.| -In short, a special missing is **not supported** in a column that carries NOTNULL, MINVAL, MAXVAL, HARDSELECT or a formula. It is fine alongside the regex rules, a soft dropdown, and ROUND. +In short, a special missing is **not supported** in a column that carries NOTNULL, MINVAL, MAXVAL or a formula. It is fine alongside the regex rules, a soft dropdown, and ROUND. + +A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there. One related gotcha: a range rule's own value must be a number. Put a special missing in `MINVAL` and every cell in the column fails; put one in `MAXVAL` and every real number fails. From abf01d6375f255c59fe013fbe8ef6c76376f262e Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 08:37:12 +0000 Subject: [PATCH 3/7] docs(validations): the dropdowns support special missings Both SOFTSELECT and HARDSELECT support them, and the dropdown lists them alongside the ordinary values (no NaN entries for a numeric column). --- docs/dcc-validations.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index e3053b8..5921a01 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -59,11 +59,11 @@ The rules treat a special missing as a value for some checks and as a missing va |CASE|Not applicable - it is a character rule.| |ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.| |HARDREGEX / SOFTREGEX|Checked against the pattern like any other value. Unlike blanks and the plain `.`, special missings are **not** exempt. See [Regex Rules](#regex-rules).| -|SOFTSELECT|Passes - a soft dropdown never blocks a value.| -|HARDSELECT|Checked against the dropdown list like any other value - it passes only if the list contains it.| +|SOFTSELECT|Supports special missings - a soft dropdown never blocks a value.| +|HARDSELECT|Supports special missings - the value is matched against the dropdown list like any other value.| |HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.| -In short, a special missing is **not supported** in a column that carries NOTNULL, MINVAL, MAXVAL or a formula. It is fine alongside the regex rules, a soft dropdown, and ROUND. +In short, a special missing is **not supported** in a column that carries NOTNULL, MINVAL, MAXVAL or a formula. It is fine alongside the regex rules, the dropdowns, and ROUND. A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there. From 1a5cea9ebe9ce06cd601a4be11b80b16104c3311 Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 15:38:37 +0000 Subject: [PATCH 4/7] docs(validations): a date formatted column takes no special missing --- docs/dcc-validations.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index 5921a01..7408514 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -49,6 +49,8 @@ Check back frequently as we keep growing this list of checks. For how each rule A SAS numeric variable has 28 missing values: the regular `.` and 27 special ones (`.A`-`.Z` and `._`). Data Controller accepts them in a numeric cell as a bare letter - `a` to `z`, or `_`, with no period - and stores the matching special missing in SAS. +A numeric column that carries a date, datetime or time format is the exception. Those cells are edited through a date picker rather than the numeric editor, and the picker accepts only a date - so a special missing cannot be typed into one of them. + The rules treat a special missing as a value for some checks and as a missing value for others: |Rule|What happens to a special missing| From 66c48a4cc7377abed717c9918c4bc4e5581823e7 Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 21:01:11 +0000 Subject: [PATCH 5/7] docs(validations): a range rule ignores a missing value Three corrections to the special missing section: - the period is optional: a special missing is typed as `a` or `.a`, `_` or `._` - MINVAL and MAXVAL both accept a missing value, and both still reject a real number that is out of range. A minimum constrains a number, and a missing is not a number - NOTNULL is the rule for a column that must be populated - only NOTNULL and the formulas actually reject a special missing, so the "not supported" summary is replaced with what does reject it and what does not --- docs/dcc-validations.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index 7408514..29da41c 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -47,7 +47,7 @@ Check back frequently as we keep growing this list of checks. For how each rule ## Special Missing Values -A SAS numeric variable has 28 missing values: the regular `.` and 27 special ones (`.A`-`.Z` and `._`). Data Controller accepts them in a numeric cell as a bare letter - `a` to `z`, or `_`, with no period - and stores the matching special missing in SAS. +A SAS numeric variable has 28 missing values: the regular `.` and 27 special ones (`.A`-`.Z` and `._`). Data Controller accepts them in a numeric cell as a letter - `a` to `z` - or an underscore, with or without a leading period (`a`, `.a`, `_`, `._`), and stores the matching special missing in SAS. The period is optional, and a plain `.` stores the regular missing. A numeric column that carries a date, datetime or time format is the exception. Those cells are edited through a date picker rather than the numeric editor, and the picker accepts only a date - so a special missing cannot be typed into one of them. @@ -55,9 +55,9 @@ The rules treat a special missing as a value for some checks and as a missing va |Rule|What happens to a special missing| |---|---| -|NOTNULL|Fails. A special missing is null to a NOT NULL (or primary key) constraint, and a real SAS NOT NULL constraint on the target table rejects it too.| -|MINVAL|Fails - a special missing sorts below every number, so it is below any minimum.| -|MAXVAL|Passes - but the rule is then pointless: a special missing is below every maximum, so the missings pass and the column's real numbers fail.| +|NOTNULL|Fails. A special missing is a missing value, so a NOT NULL (or primary key) constraint rejects it, and a real SAS NOT NULL constraint on the target table rejects it too.| +|MINVAL|Passes. A missing value is not a number, so a minimum does not apply to it - use NOTNULL if the column must be populated. A real number below the minimum still fails.| +|MAXVAL|Passes, for the same reason. A real number above the maximum still fails.| |CASE|Not applicable - it is a character rule.| |ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.| |HARDREGEX / SOFTREGEX|Checked against the pattern like any other value. Unlike blanks and the plain `.`, special missings are **not** exempt. See [Regex Rules](#regex-rules).| @@ -65,11 +65,11 @@ The rules treat a special missing as a value for some checks and as a missing va |HARDSELECT|Supports special missings - the value is matched against the dropdown list like any other value.| |HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.| -In short, a special missing is **not supported** in a column that carries NOTNULL, MINVAL, MAXVAL or a formula. It is fine alongside the regex rules, the dropdowns, and ROUND. +In short, a special missing is rejected by NOTNULL (and by a primary key column), and a formula returns `#VALUE!` for it. It is fine in a column that carries a range rule, a regex, a dropdown, or ROUND. A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there. -One related gotcha: a range rule's own value must be a number. Put a special missing in `MINVAL` and every cell in the column fails; put one in `MAXVAL` and every real number fails. +One related gotcha: a range rule's own value must be a number. The rule value is compared as a number, so a special missing there leaves nothing for a real value to pass: put one in `MINVAL` and every number in the column fails; put one in `MAXVAL` and every number fails too. ## Formula Rules From 83641a204fd9ccb3e373a6d7a3efd488001300d2 Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 21:27:57 +0000 Subject: [PATCH 6/7] docs(validations): a range rule compares in SAS's own order A range rule keys both the cell value and its own value into the order SAS uses for a numeric variable: every missing below every non-missing value, and the missing values ordered ._ then the regular missing then .A through .Z. So a range can be written in special missings and mean what SAS means by it - MINVAL .A with MAXVAL .C accepts .B and rejects .D - and the numeric cases follow from the same order: a missing fails a numeric MINVAL and passes a numeric MAXVAL, and a number sits above every missing. Replaces the "a range rule ignores a missing value" wording, which described the previous take's behaviour rather than SAS's. --- docs/dcc-validations.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index 29da41c..e1d86d7 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -56,8 +56,8 @@ The rules treat a special missing as a value for some checks and as a missing va |Rule|What happens to a special missing| |---|---| |NOTNULL|Fails. A special missing is a missing value, so a NOT NULL (or primary key) constraint rejects it, and a real SAS NOT NULL constraint on the target table rejects it too.| -|MINVAL|Passes. A missing value is not a number, so a minimum does not apply to it - use NOTNULL if the column must be populated. A real number below the minimum still fails.| -|MAXVAL|Passes, for the same reason. A real number above the maximum still fails.| +|MINVAL|Fails if the minimum is a number - a missing sorts below every number, so it is below any numeric floor. A minimum that is itself a special missing is compared in the order the missing values have among themselves, and can accept one.| +|MAXVAL|Passes - a missing sorts below every number, so it is below any numeric ceiling. A ceiling that is itself a special missing is compared in the missing values' own order, and can reject one.| |CASE|Not applicable - it is a character rule.| |ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.| |HARDREGEX / SOFTREGEX|Checked against the pattern like any other value. Unlike blanks and the plain `.`, special missings are **not** exempt. See [Regex Rules](#regex-rules).| @@ -65,11 +65,13 @@ The rules treat a special missing as a value for some checks and as a missing va |HARDSELECT|Supports special missings - the value is matched against the dropdown list like any other value.| |HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.| -In short, a special missing is rejected by NOTNULL (and by a primary key column), and a formula returns `#VALUE!` for it. It is fine in a column that carries a range rule, a regex, a dropdown, or ROUND. +In short, a special missing is rejected by NOTNULL (and by a primary key column), by a MINVAL whose floor is a number, and a formula returns `#VALUE!` for it. It is fine in a column that carries a MAXVAL, a regex, a dropdown, or ROUND - and it can satisfy a MINVAL when the floor is expressed as a special missing. + +A range rule compares in the order SAS itself uses, so a range can be written in special missings and mean what SAS would mean by it. That order puts every missing below every non-missing value, and orders the missing values as `._`, then the regular missing, then `.A` through `.Z`. So `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails a floor of `.A` because it sorts below it. A number sorts above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`. A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there. -One related gotcha: a range rule's own value must be a number. The rule value is compared as a number, so a special missing there leaves nothing for a real value to pass: put one in `MINVAL` and every number in the column fails; put one in `MAXVAL` and every number fails too. +A range rule's own value is read as a number when it looks like one and as a missing value when it is a special missing. A value that is neither - a typo such as `..` or `AB` - satisfies nothing, so every cell in the column fails until the rule is corrected. ## Formula Rules From 10448de15bac806f2d8c4436588c1efcbea5737e Mon Sep 17 00:00:00 2001 From: dc Date: Wed, 23 Sep 2026 22:00:18 +0000 Subject: [PATCH 7/7] docs(validations): point the range rules at the special missing section, and tidy MINVAL and MAXVAL now say what a special missing does to them and that the bound itself may be a special missing, pointing at the Special Missing Values section, since that is the least obvious thing about them. Also in that section: the framing sentence covers the range rules' ordering, the summary sentence is rewritten (it read badly), and the arrow in the ROUND row is plain ASCII. --- docs/dcc-validations.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/dcc-validations.md b/docs/dcc-validations.md index e1d86d7..f89caf5 100644 --- a/docs/dcc-validations.md +++ b/docs/dcc-validations.md @@ -29,11 +29,11 @@ Check back frequently as we keep growing this list of checks. For how each rule |---|---|---| |CASE|UPCASE|Will enforce the case of cell values. Valid values: UPCASE, LOWCASE, PROPCASE| |NOTNULL|(defaultval)|Will prevent submission if null values are present - a special missing counts as null. Optional - provide a default value.| -|MINVAL|1|Defines a minimum value for a numeric cell| -|MAXVAL|1000000|Defines a maximum value for a numeric cell| +|MINVAL|1|Defines a minimum value for a numeric cell. A special missing value sorts below every number, so it fails a numeric floor, and the floor itself may be a special missing - see [Special Missing Values](#special-missing-values).| +|MAXVAL|1000000|Defines a maximum value for a numeric cell. A special missing value sorts below every number, so it passes a numeric ceiling, and the ceiling itself may be a special missing - see [Special Missing Values](#special-missing-values).| |READONLY|(defaultval) |Renders the column read-only in the editor. The defaultval is used when rows are added. | |HIDDEN|(defaultval) |Hides the column from the editor grid while still submitting its data. The defaultval is used when rows are added. | -|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc. Half-away-from-zero rounding is applied so `-0.5 → -1` and `2.5 → 3`. | +|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc. Half-away-from-zero rounding is applied so `-0.5 -> -1` and `2.5 -> 3`. | |NUMBER_FORMAT|`{"style":"currency","currency":"GBP"}` |Display-only [`Intl.NumberFormat` renderer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat). RULE_VALUE is the JSON options object passed straight to `Intl.NumberFormat`. Does not change the stored value. | |HARDFORMULA|`= PRICE * VOLUME`|The cell displays a value computed from a formula, using other columns in the same row. The column is read-only - the user cannot override the result. See [Formula Rules](#formula-rules) below.| |SOFTFORMULA|`= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )`|Like HARDFORMULA, but the user can override the computed value and type their own. See [Formula Rules](#formula-rules) below.| @@ -51,7 +51,7 @@ A SAS numeric variable has 28 missing values: the regular `.` and 27 special one A numeric column that carries a date, datetime or time format is the exception. Those cells are edited through a date picker rather than the numeric editor, and the picker accepts only a date - so a special missing cannot be typed into one of them. -The rules treat a special missing as a value for some checks and as a missing value for others: +The rules treat a special missing as a value for some checks and as a missing value for others - and the range rules compare it in the order SAS gives it, below every number but ordered among the other missing values: |Rule|What happens to a special missing| |---|---| @@ -65,9 +65,9 @@ The rules treat a special missing as a value for some checks and as a missing va |HARDSELECT|Supports special missings - the value is matched against the dropdown list like any other value.| |HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.| -In short, a special missing is rejected by NOTNULL (and by a primary key column), by a MINVAL whose floor is a number, and a formula returns `#VALUE!` for it. It is fine in a column that carries a MAXVAL, a regex, a dropdown, or ROUND - and it can satisfy a MINVAL when the floor is expressed as a special missing. +In short, a special missing is rejected by NOTNULL, by a primary key column, and by a MINVAL whose floor is a number; a formula returns `#VALUE!` for it. It is fine in a column that carries a MAXVAL, a regex, a dropdown, or ROUND - and it can satisfy a MINVAL when the floor is expressed as a special missing. -A range rule compares in the order SAS itself uses, so a range can be written in special missings and mean what SAS would mean by it. That order puts every missing below every non-missing value, and orders the missing values as `._`, then the regular missing, then `.A` through `.Z`. So `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails a floor of `.A` because it sorts below it. A number sorts above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`. +A range rule compares in the order SAS itself uses, so a range written in special missings means what SAS would mean by it. That order puts every missing below every non-missing value, and orders the missing values as `._`, then the regular missing, then `.A` through `.Z`. So `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails a floor of `.A` because it sorts below it. A number sorts above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`. A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there.