feed: embed the special missings recording, and correct the copy
Embeds the "Special Missings in Data Controller" recording from vid.4gl.io as a responsive PeerTube iframe, in place of the older YouTube embed, and drops the stray `video: [...]` line that was sitting in the body as literal text. Copy corrections, each verified against the running app rather than the source: - the range rules compare in SAS's order, so MINVAL .A with MAXVAL .C takes .B and refuses .D, a missing fails a numeric MINVAL and passes a numeric MAXVAL, and a number sits above every missing. The old wording said the range rules step aside for a missing value - a lone "." is another way of typing the regular missing; the previous claim that a literal "." is refused in favour of null is not what happens - a strict dropdown does accept a special missing when its list holds one (the list shows it as a bare letter), and still rejects a value that is not listed - NOTNULL rejects a special missing, and a primary key column is NOT NULL whether or not a rule is configured for it The LinkedIn version of the post is kept in sync, and carries the new video URL.
This commit is contained in:
@@ -37,7 +37,8 @@ The Data Controller frontend and the SAS backend exchange data as JSON, and JSON
|
||||
- The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
|
||||
- `null` becomes `.` or an empty string, according to the type derived for that column.
|
||||
- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
|
||||
- A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing.
|
||||
- A value that is neither a number nor a single valid character is refused rather than guessed - `aaaa`, or `!` in a numeric column
|
||||
- A lone `.` is accepted as another way of typing the regular missing
|
||||
|
||||
There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker.
|
||||
|
||||
@@ -47,10 +48,10 @@ Once in SAS they are ordinary values, so they are what the approval DIFF screen
|
||||
|
||||
These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
|
||||
|
||||
- `NOTNULL` - works the same as SAS: cannot use special missings. A special missing is NULL to a NOT NULL (or primary key) constraint, so the rule rejects it, and so does a physical constraint on the target table
|
||||
- `NOTNULL` - rejects one. A special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it as well. A primary key column is treated as NOT NULL whether or not a rule is configured for it
|
||||
- `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
|
||||
- `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
|
||||
- `SOFTSELECT` / `HARDSELECT` - both support special missings; a soft dropdown never blocks a value, and a hard one lists them alongside the ordinary values
|
||||
- `SOFTSELECT` / `HARDSELECT` - both support them. The dropdown lists a special missing as a bare letter alongside the ordinary values, and a hard rule then accepts it like any other listed value - it still rejects a value that is not in the list
|
||||
- `ROUND` - harmless. It only rounds values that are numbers, so a special missing is left exactly as it was typed
|
||||
|
||||
The range rules compare in the order SAS itself uses, which is what makes them interesting here. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. So a range can be written in special missings and mean exactly what SAS would mean by it:
|
||||
@@ -68,16 +69,20 @@ A formula is the one case where a special missing genuinely does not work:
|
||||
|
||||
One thing worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
|
||||
|
||||
In short, a special missing counts as a value for the pattern and dropdown rules, and as a missing value for everything that asks for a number.
|
||||
In short, a special missing counts as a value for the pattern and dropdown rules, and as a value with its own place in the order for the range rules: it sits below every number, so a numeric minimum rejects it and a numeric maximum accepts it, while a range written in special missings is decided among the missing values themselves.
|
||||
|
||||
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
||||
## See it in action
|
||||
|
||||
The recording shows the full cycle: typing special missings into numeric cells, the rejections, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns.
|
||||
The recording below runs the whole cycle on one table: entering special missings, the rules that reject them, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on a date column.
|
||||
|
||||
It also shows the range rules doing what the section above describes. A special missing is refused by a numeric `MINVAL` and accepted by a numeric `MAXVAL`, and on a column carrying `MINVAL .A` with `MAXVAL .C`, `.B` is taken while `.D` is refused.
|
||||
|
||||
<div style="position: relative; padding-top: 56.25%; margin-bottom: 2rem;"><iframe title="Special Missings in Data Controller" width="100%" height="100%" src="https://vid.4gl.io/videos/embed/9UQZzCNBU3zPNQYxdzyV3A?peertubeLink=0" style="border: 0px; position: absolute; inset: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>
|
||||
|
||||
<!--
|
||||
LinkedIn version of this post - publish it with the "Managing Special Missing
|
||||
Values with Data Controller for SAS" video attached. Kept in sync with the copy
|
||||
above: same points, same claims, same order.
|
||||
LinkedIn version of this post - publish it with the "Special Missings in Data
|
||||
Controller" video attached. Kept in sync with the copy above: same points, same
|
||||
claims, same order.
|
||||
|
||||
A SAS numeric missing is not a lone wolf. It is 28.
|
||||
|
||||
@@ -102,22 +107,24 @@ We solved that in the open source SASjs Adapter, and it has been in Data Control
|
||||
|
||||
How they behave in Data Controller's validation rules:
|
||||
|
||||
- NOTNULL works the same as SAS - no special missings: one is null to a NOT NULL (or primary key) constraint
|
||||
- NOTNULL rejects one: a special missing is a missing value, so it fails the rule and a physical constraint. A primary key column is NOT NULL whether or not a rule is configured
|
||||
- HARDREGEX checks it against the pattern, unlike blanks and plain .
|
||||
- SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
|
||||
- SOFTSELECT and HARDSELECT both support them
|
||||
- SOFTSELECT and HARDSELECT both support them - the dropdown lists the missing as a bare letter, and a hard rule accepts it like any other listed value
|
||||
- ROUND leaves it alone, because it only rounds numbers
|
||||
|
||||
The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
|
||||
|
||||
- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
|
||||
|
||||
In short: a value for the pattern and dropdown rules, and a value with its own place in the order for the range rules.
|
||||
The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
|
||||
|
||||
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF.
|
||||
In short: a value for the pattern and dropdown rules, and a value with its own place in the order for the range rules - below every number, so a numeric minimum rejects it and a numeric maximum accepts it.
|
||||
|
||||
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF, including a range written in special missings.
|
||||
|
||||
#sas #datacapture #mdm #dataquality
|
||||
|
||||
video: https://vid.4gl.io/w/9UQZzCNBU3zPNQYxdzyV3A
|
||||
|
||||
Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
|
||||
dozen or more, spread wide and moving together across a vast snow plain at dusk,
|
||||
seen from a low wide angle. One wolf stands apart from the group on the left of
|
||||
|
||||
Reference in New Issue
Block a user