blog: NOTNULL now rejects special missings; range rules and formulas do not support them

The validator fix lands in dc/dc (PR #323), so the post describes the fixed
behaviour rather than the mismatch: a special missing fails NOTNULL, matching
a physical SAS NOT NULL / primary key constraint.

Also records what ROUND and SOFTSELECT/HARDSELECT do with one, and groups
MINVAL/MAXVAL/HARDFORMULA/SOFTFORMULA under 'not supported'.

content/feed/sas-special-missings/index.md

@@ -43,22 +43,24 @@ Once in SAS they are ordinary values, so they are what the approval DIFF screen
 
 ## What the Data Controller validation rules do with them
 
-These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit. A special missing is a value somebody deliberately set, so the rules treat it as a value - it is not blank, and it is not exempt:
+These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
 
-- `NOTNULL` - passes, because it is not null
-- `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum
-- `MAXVAL` - passes, for the same reason it is below any maximum
+- `NOTNULL` - fails. A special missing is NULL as far as a NOT NULL (or primary key) constraint is concerned, so the rule rejects it - and so does a physical SAS NOT NULL constraint on the target table
 - `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
 - `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
+- `SOFTSELECT` - never blocks, so a special missing passes; `HARDSELECT` is a strict membership test and rejects one
+- `ROUND` - harmless. It only rounds values that are numbers, so a special missing is left exactly as it was typed
+
+Special missings are **not supported** in a column that carries a range rule or a formula:
+
+- `MINVAL` / `MAXVAL` - a special missing sorts below every number, so `MINVAL` rejects it and `MAXVAL` accepts it while rejecting the column's real numbers. The rule's own value has to be a number too: put a special missing in `MINVAL` and every cell in the column fails, put one in `MAXVAL` and every real number fails
 - `HARDFORMULA` / `SOFTFORMULA` - a formula that reads a special-missing cell does not compute; the grid's spreadsheet engine returns `#VALUE!` for that row, where the plain `.` contributes 0
 
-One mismatch to be aware of: the `NOTNULL` rule above is the editor's, and it is more forgiving than the SAS constraint it mirrors. A real NOT NULL - or primary key - constraint on the target table rejects a special missing, so on a column carrying one the editor will let the value through and the load will then fail on the constraint.
-
-Two other sharp edges. A range rule's *value* has to be a number: put a special missing in `MINVAL` and every cell in the column fails, put one in `MAXVAL` and every real number fails. And although the regex pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
-
 `CASE` (`UPCASE` / `LOWCASE`) is a character rule and does not come into it: SAS hands the browser a special missing as an uppercase letter, so there is no case to enforce, and a `CASE` rule on a numeric column would reject the column's numbers rather than the missing.
 
-That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column.
+One thing worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
+
+In short, a special missing counts as a value for the pattern and dropdown rules, and as a missing value for everything that asks for a number.
 
 `video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
 
@@ -89,16 +91,20 @@ We solved that in the open source SASjs Adapter, and it has been in Data Control
 - where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
 - you type the letter on its own: a or A, no period, and case does not matter
 
-And they behave like values in Data Controller's validation rules, which is usually what you want:
+How they behave in Data Controller's validation rules:
 
-- NOTNULL passes - it is not null (though a real SAS NOT NULL constraint is stricter, and does reject a special missing)
-- MINVAL fails - it sorts below every number; MAXVAL passes for the same reason
+- NOTNULL fails - a special missing is null to a NOT NULL (or primary key) constraint, and a real SAS NOT NULL constraint rejects it too
 - HARDREGEX checks it against the pattern, unlike blanks and plain .
 - SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
-- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
-- CASE is a character rule, so it has no place on a numeric column
+- SOFTSELECT lets it through; HARDSELECT rejects it
+- ROUND leaves it alone, because it only rounds numbers
 
-So a value recorded as "not collected" cannot quietly satisfy a completeness or range rule.
+Special missings are not supported where a range rule or a formula is configured:
+
+- MINVAL rejects it; MAXVAL accepts it but rejects the column's real numbers
+- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
+
+In short: a value for the pattern and dropdown rules, a missing value for anything that wants a number.
 
 The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF.