Merge pull request 'feed: add a special missings post' (#19) from blog/special-missings-education into main

Reviewed-on: #19
2026-09-23 22:42:31 +00:00
parent fcba390804 7d950cbdbe
commit 59ff2b13fc
3 changed files with 156 additions and 1 deletions
@@ -61,7 +61,7 @@ Did you know that, in addition to a regular missing value in SAS (`.`), there ar
 These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
-`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
+`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
 There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
@@ -0,0 +1,155 @@
 ---
 title: '28 Ways to Be Missing in SAS'
 description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has supported all of them since v4. How they work, and what the validation rules do with them.
 date: '2026-09-22 09:00:00'
 author: 'Allan Bowe'
 authorLink: https://www.linkedin.com/in/allanbowe/
 tags:
  - Special Missings
  - Data Quality
 previewImg: './cover.jpeg'
 ---
 # 28 Ways to Be Missing in SAS
 A SAS numeric missing is not a lone wolf: the ordinary missing (`.`) is one of **28** distinct missing values a numeric variable can hold. The other 27 are written with a single character - the letters `A` to `Z`, or an underscore (`._`).
 They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well-run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
 ## They are numbers, not text
 - Arithmetic on them yields missing - `.A + 1` is `.`
 - `PROC MEANS`, `PROC SUMMARY` and the other summarisation procedures exclude them, exactly as they exclude `.`
 - They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
 - `NMISS()` and `CMISS()` count them as missing
 - Converting one to text drops the period - `CATS(.A)` is the string `A`
 One display detail is worth knowing: a regular missing prints as `.`, unless `OPTIONS MISSING=` changes that character - set it to blank and a regular missing renders as an empty cell. The option affects only the regular missing; `._` and `.A`-`.Z` always print as their own letter. So a blank cell in a SAS listing is still unambiguously a regular missing, and a lone letter is still a special one. Data Controller itself is unaffected either way - it sends a regular missing to the browser as `null` and a special missing as its letter.
 In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type the letter or the underscore, with or without that period - `.a` and `a` are the same missing - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
 There is one cell where the letter cannot be typed at all. A numeric column that carries a date, datetime or time format is edited through a date picker rather than the numeric editor, and a picker accepts only a date.
 ## Carrying them between the browser and SAS
 The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
 - The adapter infers each column's SAS type from the values it is given. All numeric values mean a numeric column, all strings mean a character column, and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
 - `null` becomes `.` or an empty string, according to the type derived for that column.
 - Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
 - A value that is neither a number nor a single valid character is refused rather than guessed - `aaaa`, or `!` in a numeric column.
 - A lone `.` is accepted as another way of typing the regular missing.
 There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker.
 Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
 ## What the Data Controller validation rules do with them
 These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
 - `NOTNULL` - rejects one. A special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it as well. A primary key column is treated as NOT NULL whether or not a rule is configured for it
 - `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
 - `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
 - `SOFTSELECT` / `HARDSELECT` - both support them. The dropdown lists a special missing as a bare letter alongside the ordinary values, and a hard rule then accepts it like any other listed value - it still rejects a value that is not in the list
 - `ROUND` - no effect. It only rounds values that are numbers, so a special missing is left as it was typed
 The range rules compare in the order SAS itself uses, which is what lets a range be written in special missings. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. A range therefore means exactly what SAS would mean by it:
 - `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails that floor because it sorts below `.A`
 - a number sits above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`
 - against a numeric bound the ordering does the obvious thing: a missing sorts below every number, so it fails a `MINVAL` of 1 and passes a `MAXVAL` of 100. Use `NOTNULL` if the column has to be populated
 - a rule value that is neither a number nor a special missing - a typo such as `..` or `AB` - satisfies nothing, so the column fails until the rule is corrected
 A formula is the one case where a special missing genuinely does not work:
 - `HARDFORMULA` / `SOFTFORMULA` - a formula that reads a special-missing cell does not compute; the grid's spreadsheet engine returns `#VALUE!` for that row, where the plain `.` contributes 0
 `CASE` (`UPCASE` / `LOWCASE`) is a character rule, so it does not apply: SAS hands the browser a special missing as an uppercase letter, so there is no case to enforce, and a `CASE` rule on a numeric column would reject the column's numbers rather than the missing.
 Worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
 In short, a special missing counts as a value for the pattern and dropdown rules, and takes its own place in the order for the range rules: it sits below every number, so a numeric minimum rejects it and a numeric maximum accepts it, while a range written in special missings is decided among the missing values themselves.
 ## See it in action
 The recording below runs the whole cycle on one table: entering special missings, the rules that reject them, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on a date column.
 It also shows the range rules doing what the section above describes. A special missing is refused by a numeric `MINVAL` and accepted by a numeric `MAXVAL`, and on a column carrying `MINVAL .A` with `MAXVAL .C`, `.B` is taken while `.D` is refused.
 <div style="position: relative; padding-top: 56.25%; margin-bottom: 2rem;"><iframe title="Special Missings in Data Controller" width="100%" height="100%" src="https://vid.4gl.io/videos/embed/9UQZzCNBU3zPNQYxdzyV3A?peertubeLink=0" style="border: 0px; position: absolute; inset: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>
 <!--
 LinkedIn version of this post - publish it with the "Special Missings in Data
 Controller" video attached. Kept in sync with the copy above: same points, same
 claims, same order.
 A SAS numeric missing is not a lone wolf. There are 28 of them.
 The ordinary missing (.) is the best known of the 28. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
 They are real numeric values, not text:
 - .A + 1 is .
 - PROC MEANS excludes them, like any other missing
 - they sort below every number: ._ then . then .A to .Z
 - NMISS() and CMISS() count them as missing
 - a regular missing prints as . unless OPTIONS MISSING= changes it - and that option never touches a special missing
 The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
 We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
 - the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
 - where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
 - you type the letter or the underscore, with or without the period (a, .a, _, ._), and case does not matter
 - one exception: a date, datetime or time formatted numeric column edits through a date picker, which takes only a date
 How they behave in Data Controller's validation rules:
 - NOTNULL rejects one: a special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it too. A primary key column is NOT NULL whether or not a rule is configured
 - HARDREGEX checks it against the pattern, unlike blanks and plain .
 - SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
 - SOFTSELECT and HARDSELECT both support them - the dropdown lists the missing as a bare letter, and a hard rule accepts it like any other listed value
 - ROUND leaves it alone, because it only rounds numbers
 - a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
 The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
 In short: a value for the pattern and dropdown rules, and its own place in the order for the range rules - below every number, so a numeric minimum rejects it and a numeric maximum accepts it.
 The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF, including a range written in special missings.
 #sas #datacapture #mdm #dataquality
 video: https://vid.4gl.io/w/9UQZzCNBU3zPNQYxdzyV3A
 Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
 dozen or more, spread wide and moving together across a vast snow plain at dusk,
 seen from a low wide angle. One wolf stands apart from the group on the left of
 frame, turned back toward the pack. The pack is rendered in cool blue-grey and
 silver; the lone wolf catches the only warm light in the scene, a single low
 amber sun. Overcast dusk sky, faint falling snow, long soft shadows, generous
 empty sky and snow to the upper third so the wide crop breathes. Painterly
 digital illustration, muted desaturated palette, soft depth of field with the
 distant wolves falling out of focus, no text, no letters, no numbers, no logos,
 no watermark.
 Constraints: no text, no letters, no numbers, no digits, no symbols, no
 captions, no logos, no watermark, no signature, no border, no frame, no collar,
 no harness, no humans, no buildings, and do not ask for an exact head count of
 28 - a crowded pack reads worse than a dozen clear animals, and the number
 belongs in the headline, not the artwork. The no-text rule matters more than
 usual here: the subject is letters standing in for numbers, so a stray glyph
 anywhere undercuts the cover.
 Output: save as ./cover.jpeg at 1.91:1 (1200x627, matching the other feed
 covers and doubling as the LinkedIn share card), and set
 previewImg: './cover.jpeg' in the front matter above. Do not also embed the
 image in the markdown body. Full prompt, variants and rationale:
 https://paste.4gl.io/?513711f5bde3610e#9LZCA9xzQHDy3CycXmsWU5AoVNtjBZmPotUHHtHTAerF
 Fallback variant if the pack composition comes back muddled: a single wolf
 standing alone on a snow plain at dusk, its shadow stretching toward a distant
 pack reduced to small silhouettes on the horizon.
 -->