feed: embed the special missings recording, and correct the copy

Embeds the "Special Missings in Data Controller" recording from vid.4gl.io as a
responsive PeerTube iframe, in place of the older YouTube embed, and drops the
stray `video: [...]` line that was sitting in the body as literal text.

Copy corrections, each verified against the running app rather than the source:

- the range rules compare in SAS's order, so MINVAL .A with MAXVAL .C takes .B
  and refuses .D, a missing fails a numeric MINVAL and passes a numeric MAXVAL,
  and a number sits above every missing. The old wording said the range rules
  step aside for a missing value
- a lone "." is another way of typing the regular missing; the previous claim
  that a literal "." is refused in favour of null is not what happens
- a strict dropdown does accept a special missing when its list holds one (the
  list shows it as a bare letter), and still rejects a value that is not listed
- NOTNULL rejects a special missing, and a primary key column is NOT NULL
  whether or not a rule is configured for it

The LinkedIn version of the post is kept in sync, and carries the new video URL.
This commit is contained in:
dc
2026-09-23 22:00:16 +00:00
parent 0b18658686
commit 66d5fecb0e
+22 -15
View File
@@ -37,7 +37,8 @@ The Data Controller frontend and the SAS backend exchange data as JSON, and JSON
- The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings. - The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
- `null` becomes `.` or an empty string, according to the type derived for that column. - `null` becomes `.` or an empty string, according to the type derived for that column.
- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend. - Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
- A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing. - A value that is neither a number nor a single valid character is refused rather than guessed - `aaaa`, or `!` in a numeric column
- A lone `.` is accepted as another way of typing the regular missing
There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker. There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker.
@@ -47,10 +48,10 @@ Once in SAS they are ordinary values, so they are what the approval DIFF screen
These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit. These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
- `NOTNULL` - works the same as SAS: cannot use special missings. A special missing is NULL to a NOT NULL (or primary key) constraint, so the rule rejects it, and so does a physical constraint on the target table - `NOTNULL` - rejects one. A special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it as well. A primary key column is treated as NOT NULL whether or not a rule is configured for it
- `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter - `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
- `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule - `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
- `SOFTSELECT` / `HARDSELECT` - both support special missings; a soft dropdown never blocks a value, and a hard one lists them alongside the ordinary values - `SOFTSELECT` / `HARDSELECT` - both support them. The dropdown lists a special missing as a bare letter alongside the ordinary values, and a hard rule then accepts it like any other listed value - it still rejects a value that is not in the list
- `ROUND` - harmless. It only rounds values that are numbers, so a special missing is left exactly as it was typed - `ROUND` - harmless. It only rounds values that are numbers, so a special missing is left exactly as it was typed
The range rules compare in the order SAS itself uses, which is what makes them interesting here. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. So a range can be written in special missings and mean exactly what SAS would mean by it: The range rules compare in the order SAS itself uses, which is what makes them interesting here. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. So a range can be written in special missings and mean exactly what SAS would mean by it:
@@ -68,16 +69,20 @@ A formula is the one case where a special missing genuinely does not work:
One thing worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match. One thing worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
In short, a special missing counts as a value for the pattern and dropdown rules, and as a missing value for everything that asks for a number. In short, a special missing counts as a value for the pattern and dropdown rules, and as a value with its own place in the order for the range rules: it sits below every number, so a numeric minimum rejects it and a numeric maximum accepts it, while a range written in special missings is decided among the missing values themselves.
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)` ## See it in action
The recording shows the full cycle: typing special missings into numeric cells, the rejections, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns. The recording below runs the whole cycle on one table: entering special missings, the rules that reject them, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on a date column.
It also shows the range rules doing what the section above describes. A special missing is refused by a numeric `MINVAL` and accepted by a numeric `MAXVAL`, and on a column carrying `MINVAL .A` with `MAXVAL .C`, `.B` is taken while `.D` is refused.
<div style="position: relative; padding-top: 56.25%; margin-bottom: 2rem;"><iframe title="Special Missings in Data Controller" width="100%" height="100%" src="https://vid.4gl.io/videos/embed/9UQZzCNBU3zPNQYxdzyV3A?peertubeLink=0" style="border: 0px; position: absolute; inset: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>
<!-- <!--
LinkedIn version of this post - publish it with the "Managing Special Missing LinkedIn version of this post - publish it with the "Special Missings in Data
Values with Data Controller for SAS" video attached. Kept in sync with the copy Controller" video attached. Kept in sync with the copy above: same points, same
above: same points, same claims, same order. claims, same order.
A SAS numeric missing is not a lone wolf. It is 28. A SAS numeric missing is not a lone wolf. It is 28.
@@ -102,22 +107,24 @@ We solved that in the open source SASjs Adapter, and it has been in Data Control
How they behave in Data Controller's validation rules: How they behave in Data Controller's validation rules:
- NOTNULL works the same as SAS - no special missings: one is null to a NOT NULL (or primary key) constraint - NOTNULL rejects one: a special missing is a missing value, so it fails the rule and a physical constraint. A primary key column is NOT NULL whether or not a rule is configured
- HARDREGEX checks it against the pattern, unlike blanks and plain . - HARDREGEX checks it against the pattern, unlike blanks and plain .
- SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX) - SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
- SOFTSELECT and HARDSELECT both support them - SOFTSELECT and HARDSELECT both support them - the dropdown lists the missing as a bare letter, and a hard rule accepts it like any other listed value
- ROUND leaves it alone, because it only rounds numbers - ROUND leaves it alone, because it only rounds numbers
The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number - a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
In short: a value for the pattern and dropdown rules, and a value with its own place in the order for the range rules. The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF. In short: a value for the pattern and dropdown rules, and a value with its own place in the order for the range rules - below every number, so a numeric minimum rejects it and a numeric maximum accepts it.
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF, including a range written in special missings.
#sas #datacapture #mdm #dataquality #sas #datacapture #mdm #dataquality
video: https://vid.4gl.io/w/9UQZzCNBU3zPNQYxdzyV3A
Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
dozen or more, spread wide and moving together across a vast snow plain at dusk, dozen or more, spread wide and moving together across a vast snow plain at dusk,
seen from a low wide angle. One wolf stands apart from the group on the left of seen from a low wide angle. One wolf stands apart from the group on the left of