Merge pull request 'feed: add a special missings post' (#19) from blog/special-missings-education into main

Reviewed-on: #19

content/blog/v4-0-formats-special-missings/index.md

@@ -61,7 +61,7 @@ Did you know that, in addition to a regular missing value in SAS (`.`), there ar
 
 These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
 
-`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
+`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
 
 There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
 

content/feed/sas-special-missings/index.md

@@ -0,0 +1,155 @@
+---
+title: '28 Ways to Be Missing in SAS'
+description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has supported all of them since v4. How they work, and what the validation rules do with them.
+date: '2026-09-22 09:00:00'
+author: 'Allan Bowe'
+authorLink: https://www.linkedin.com/in/allanbowe/
+tags:
+  - Special Missings
+  - Data Quality
+previewImg: './cover.jpeg'
+---
+
+# 28 Ways to Be Missing in SAS
+
+A SAS numeric missing is not a lone wolf: the ordinary missing (`.`) is one of **28** distinct missing values a numeric variable can hold. The other 27 are written with a single character - the letters `A` to `Z`, or an underscore (`._`).
+
+They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well-run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
+
+## They are numbers, not text
+
+- Arithmetic on them yields missing - `.A + 1` is `.`
+- `PROC MEANS`, `PROC SUMMARY` and the other summarisation procedures exclude them, exactly as they exclude `.`
+- They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
+- `NMISS()` and `CMISS()` count them as missing
+- Converting one to text drops the period - `CATS(.A)` is the string `A`
+
+One display detail is worth knowing: a regular missing prints as `.`, unless `OPTIONS MISSING=` changes that character - set it to blank and a regular missing renders as an empty cell. The option affects only the regular missing; `._` and `.A`-`.Z` always print as their own letter. So a blank cell in a SAS listing is still unambiguously a regular missing, and a lone letter is still a special one. Data Controller itself is unaffected either way - it sends a regular missing to the browser as `null` and a special missing as its letter.
+
+In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type the letter or the underscore, with or without that period - `.a` and `a` are the same missing - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
+
+There is one cell where the letter cannot be typed at all. A numeric column that carries a date, datetime or time format is edited through a date picker rather than the numeric editor, and a picker accepts only a date.
+
+## Carrying them between the browser and SAS
+
+The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
+
+- The adapter infers each column's SAS type from the values it is given. All numeric values mean a numeric column, all strings mean a character column, and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
+- `null` becomes `.` or an empty string, according to the type derived for that column.
+- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
+- A value that is neither a number nor a single valid character is refused rather than guessed - `aaaa`, or `!` in a numeric column.
+- A lone `.` is accepted as another way of typing the regular missing.
+
+There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker.
+
+Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
+
+## What the Data Controller validation rules do with them
+
+These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
+
+- `NOTNULL` - rejects one. A special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it as well. A primary key column is treated as NOT NULL whether or not a rule is configured for it
+- `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
+- `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
+- `SOFTSELECT` / `HARDSELECT` - both support them. The dropdown lists a special missing as a bare letter alongside the ordinary values, and a hard rule then accepts it like any other listed value - it still rejects a value that is not in the list
+- `ROUND` - no effect. It only rounds values that are numbers, so a special missing is left as it was typed
+
+The range rules compare in the order SAS itself uses, which is what lets a range be written in special missings. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. A range therefore means exactly what SAS would mean by it:
+
+- `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails that floor because it sorts below `.A`
+- a number sits above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`
+- against a numeric bound the ordering does the obvious thing: a missing sorts below every number, so it fails a `MINVAL` of 1 and passes a `MAXVAL` of 100. Use `NOTNULL` if the column has to be populated
+- a rule value that is neither a number nor a special missing - a typo such as `..` or `AB` - satisfies nothing, so the column fails until the rule is corrected
+
+A formula is the one case where a special missing genuinely does not work:
+
+- `HARDFORMULA` / `SOFTFORMULA` - a formula that reads a special-missing cell does not compute; the grid's spreadsheet engine returns `#VALUE!` for that row, where the plain `.` contributes 0
+
+`CASE` (`UPCASE` / `LOWCASE`) is a character rule, so it does not apply: SAS hands the browser a special missing as an uppercase letter, so there is no case to enforce, and a `CASE` rule on a numeric column would reject the column's numbers rather than the missing.
+
+Worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
+
+In short, a special missing counts as a value for the pattern and dropdown rules, and takes its own place in the order for the range rules: it sits below every number, so a numeric minimum rejects it and a numeric maximum accepts it, while a range written in special missings is decided among the missing values themselves.
+
+## See it in action
+
+The recording below runs the whole cycle on one table: entering special missings, the rules that reject them, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on a date column.
+
+It also shows the range rules doing what the section above describes. A special missing is refused by a numeric `MINVAL` and accepted by a numeric `MAXVAL`, and on a column carrying `MINVAL .A` with `MAXVAL .C`, `.B` is taken while `.D` is refused.
+
+<div style="position: relative; padding-top: 56.25%; margin-bottom: 2rem;"><iframe title="Special Missings in Data Controller" width="100%" height="100%" src="https://vid.4gl.io/videos/embed/9UQZzCNBU3zPNQYxdzyV3A?peertubeLink=0" style="border: 0px; position: absolute; inset: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>
+
+<!--
+LinkedIn version of this post - publish it with the "Special Missings in Data
+Controller" video attached. Kept in sync with the copy above: same points, same
+claims, same order.
+
+A SAS numeric missing is not a lone wolf. There are 28 of them.
+
+The ordinary missing (.) is the best known of the 28. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
+
+They are real numeric values, not text:
+
+- .A + 1 is .
+- PROC MEANS excludes them, like any other missing
+- they sort below every number: ._ then . then .A to .Z
+- NMISS() and CMISS() count them as missing
+- a regular missing prints as . unless OPTIONS MISSING= changes it - and that option never touches a special missing
+
+The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
+
+We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
+
+- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
+- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
+- you type the letter or the underscore, with or without the period (a, .a, _, ._), and case does not matter
+- one exception: a date, datetime or time formatted numeric column edits through a date picker, which takes only a date
+
+How they behave in Data Controller's validation rules:
+
+- NOTNULL rejects one: a special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it too. A primary key column is NOT NULL whether or not a rule is configured
+- HARDREGEX checks it against the pattern, unlike blanks and plain .
+- SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
+- SOFTSELECT and HARDSELECT both support them - the dropdown lists the missing as a bare letter, and a hard rule accepts it like any other listed value
+- ROUND leaves it alone, because it only rounds numbers
+- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
+
+The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
+
+In short: a value for the pattern and dropdown rules, and its own place in the order for the range rules - below every number, so a numeric minimum rejects it and a numeric maximum accepts it.
+
+The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF, including a range written in special missings.
+
+#sas #datacapture #mdm #dataquality
+
+video: https://vid.4gl.io/w/9UQZzCNBU3zPNQYxdzyV3A
+
+Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
+dozen or more, spread wide and moving together across a vast snow plain at dusk,
+seen from a low wide angle. One wolf stands apart from the group on the left of
+frame, turned back toward the pack. The pack is rendered in cool blue-grey and
+silver; the lone wolf catches the only warm light in the scene, a single low
+amber sun. Overcast dusk sky, faint falling snow, long soft shadows, generous
+empty sky and snow to the upper third so the wide crop breathes. Painterly
+digital illustration, muted desaturated palette, soft depth of field with the
+distant wolves falling out of focus, no text, no letters, no numbers, no logos,
+no watermark.
+
+Constraints: no text, no letters, no numbers, no digits, no symbols, no
+captions, no logos, no watermark, no signature, no border, no frame, no collar,
+no harness, no humans, no buildings, and do not ask for an exact head count of
+28 - a crowded pack reads worse than a dozen clear animals, and the number
+belongs in the headline, not the artwork. The no-text rule matters more than
+usual here: the subject is letters standing in for numbers, so a stray glyph
+anywhere undercuts the cover.
+
+Output: save as ./cover.jpeg at 1.91:1 (1200x627, matching the other feed
+covers and doubling as the LinkedIn share card), and set
+previewImg: './cover.jpeg' in the front matter above. Do not also embed the
+image in the markdown body. Full prompt, variants and rationale:
+https://paste.4gl.io/?513711f5bde3610e#9LZCA9xzQHDy3CycXmsWU5AoVNtjBZmPotUHHtHTAerF
+
+Fallback variant if the pack composition comes back muddled: a single wolf
+standing alone on a snow plain at dusk, its shadow stretching toward a distant
+pack reduced to small silhouettes on the horizon.
+-->