docs(validations): document how each rule treats a special missing value #8

Merged
allan merged 7 commits from docs/special-missing-rule-behaviour into main 2026-09-23 22:30:42 +00:00
Collaborator

Adds a Special Missing Values section to the Data Validation page, covering what each rule does with a SAS special missing (.A-.Z, ._), plus a pointer from the rule table.

What was verified (against a real SAS estate, and against the deployed frontend validator in a real browser):

  • NOTNULL - fails. A special missing is null to a SAS NOT NULL / primary key constraint, and a physical constraint on the target table rejects it with an integrity constraint error.
  • MINVAL - fails if the minimum is a number: a missing sorts below every number, so it is below any numeric floor. A floor that is itself a special missing is compared in the order the missing values have among themselves, and can accept one.
  • MAXVAL - passes against a numeric ceiling, for the same reason. A ceiling that is itself a special missing is compared in the missing values' own order, and can reject one.
  • CASE - not applicable (character rule).
  • ROUND - ignored; it only rounds values that are numbers.
  • HARDREGEX / SOFTREGEX - checked like any other value; not exempt (the Regex Rules section already says this, now cross-referenced).
  • SOFTSELECT - passes; HARDSELECT - passes when the list holds the missing: the dropdown lists it as a bare letter alongside the ordinary values, and the rule accepts it like any other listed value (a value that is not in the list is still rejected).
  • HARDFORMULA / SOFTFORMULA - returns #VALUE! for the row rather than a number.

The section also documents the order itself, because a range rule compares in it: every missing below every non-missing value, and the missing values ordered ._, then the regular missing, then .A through .Z. So MINVAL .A with MAXVAL .C accepts .B and rejects .D. A rule value that is neither a number nor a special missing satisfies nothing.

The companion frontend change (the range rules comparing in SAS's order, and NOTNULL rejecting a special missing on a numeric column) is in dc/dc PR #323, which should merge first.

Adds a **Special Missing Values** section to the Data Validation page, covering what each rule does with a SAS special missing (`.A`-`.Z`, `._`), plus a pointer from the rule table. What was verified (against a real SAS estate, and against the deployed frontend validator in a real browser): - **NOTNULL** - fails. A special missing is null to a SAS NOT NULL / primary key constraint, and a physical constraint on the target table rejects it with an integrity constraint error. - **MINVAL** - fails if the minimum is a number: a missing sorts below every number, so it is below any numeric floor. A floor that is itself a special missing is compared in the order the missing values have among themselves, and can accept one. - **MAXVAL** - passes against a numeric ceiling, for the same reason. A ceiling that is itself a special missing is compared in the missing values' own order, and can reject one. - **CASE** - not applicable (character rule). - **ROUND** - ignored; it only rounds values that are numbers. - **HARDREGEX / SOFTREGEX** - checked like any other value; not exempt (the Regex Rules section already says this, now cross-referenced). - **SOFTSELECT** - passes; **HARDSELECT** - passes when the list holds the missing: the dropdown lists it as a bare letter alongside the ordinary values, and the rule accepts it like any other listed value (a value that is not in the list is still rejected). - **HARDFORMULA / SOFTFORMULA** - returns `#VALUE!` for the row rather than a number. The section also documents the order itself, because a range rule compares in it: every missing below every non-missing value, and the missing values ordered `._`, then the regular missing, then `.A` through `.Z`. So `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`. A rule value that is neither a number nor a special missing satisfies nothing. The companion frontend change (the range rules comparing in SAS's order, and NOTNULL rejecting a special missing on a numeric column) is in dc/dc PR #323, which should merge first.
hermes added 1 commit 2026-09-23 07:50:21 +00:00
New Special Missing Values section covering NOTNULL, MINVAL/MAXVAL, CASE,
ROUND, the regex rules, the dropdowns and the formula rules, with the
'not supported' set called out.  Behaviour verified against a real SAS
estate and the deployed frontend validator.
hermes added 1 commit 2026-09-23 08:19:42 +00:00
- a primary key column rejects a blank and a special missing whether or not
  the table carries a constraint or MPE_VALIDATIONS a NOTNULL rule
- HARDSELECT is now checked like any other value: it passes when the
  dropdown list contains the special missing
- HARDSELECT drops out of the 'not supported' list
hermes added 1 commit 2026-09-23 08:37:15 +00:00
Both SOFTSELECT and HARDSELECT support them, and the dropdown lists them
alongside the ordinary values (no NaN entries for a numeric column).
hermes added 1 commit 2026-09-23 15:38:43 +00:00
hermes added 1 commit 2026-09-23 21:01:12 +00:00
Three corrections to the special missing section:

- the period is optional: a special missing is typed as `a` or `.a`, `_` or `._`
- MINVAL and MAXVAL both accept a missing value, and both still reject a real
  number that is out of range. A minimum constrains a number, and a missing is
  not a number - NOTNULL is the rule for a column that must be populated
- only NOTNULL and the formulas actually reject a special missing, so the
  "not supported" summary is replaced with what does reject it and what does not
hermes added 1 commit 2026-09-23 21:27:58 +00:00
A range rule keys both the cell value and its own value into the order SAS uses
for a numeric variable: every missing below every non-missing value, and the
missing values ordered ._ then the regular missing then .A through .Z. So a range
can be written in special missings and mean what SAS means by it - MINVAL .A with
MAXVAL .C accepts .B and rejects .D - and the numeric cases follow from the same
order: a missing fails a numeric MINVAL and passes a numeric MAXVAL, and a number
sits above every missing.

Replaces the "a range rule ignores a missing value" wording, which described the
previous take's behaviour rather than SAS's.
hermes added 1 commit 2026-09-23 22:00:19 +00:00
MINVAL and MAXVAL now say what a special missing does to them and that the bound
itself may be a special missing, pointing at the Special Missing Values section,
since that is the least obvious thing about them.

Also in that section: the framing sentence covers the range rules' ordering, the
summary sentence is rewritten (it read badly), and the arrow in the ROUND row is
plain ASCII.
allan approved these changes 2026-09-23 22:30:36 +00:00
allan merged commit c21f413f06 into main 2026-09-23 22:30:42 +00:00
allan deleted branch docs/special-missing-rule-behaviour 2026-09-23 22:30:42 +00:00
Sign in to join this conversation.
No Reviewers
No labels
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dc/docs.datacontroller.io#8