blog: add formula, range-value and NOTNULL-constraint caveats to the special missings post

Verified on a real SAS estate:
- a physical NOT NULL (or primary key) constraint rejects a special missing,
  and getdata merges that constraint into a frontend NOTNULL rule - which
  passes a special missing, so the editor is more lenient than the constraint
- a special missing as a MINVAL rule value fails every cell; as a MAXVAL rule
  value it fails every real number
- a HARDFORMULA/SOFTFORMULA reading a special-missing cell returns #VALUE!
- PRX and the JS engine agree on the value; SAS pads the numeric-to-character
  conversion, so an anchored pattern re-used in SAS needs strip()

content/blog/v4-0-formats-special-missings/index.md

@@ -61,7 +61,7 @@ Did you know that, in addition to a regular missing value in SAS (`.`), there ar
 
 These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
 
-`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
+`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
 
 There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
 

content/feed/sas-special-missings/index.md

@@ -7,6 +7,7 @@ authorLink: https://www.linkedin.com/in/allanbowe/
 tags:
   - Special Missings
   - Data Quality
+previewImg: './cover.jpeg'
 ---
 
 # 28 Ways to Be Missing in SAS
@@ -23,6 +24,8 @@ They exist because "missing" is usually not the whole story. A survey question t
 - The `NMISS()` and `CMISS()` functions count them as missing
 - Converting one to text drops the period - `cats(.A)` is the string `A`
 
+A display gotcha worth knowing: a regular missing prints as `.`, unless `options MISSING=` changes that character - set it to blank and a regular missing renders as an empty cell. The option affects only the regular missing; `._` and `.A`-`.Z` always print as their own letter. So a blank cell in a SAS listing is still unambiguously a regular missing, and a lone letter is still a special one. Data Controller itself is unaffected either way - it sends a regular missing to the browser as `null` and a special missing as its letter.
+
 In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
 
 ## Carrying them between the browser and SAS
@@ -38,15 +41,22 @@ There is nothing to configure. Special missings are available by default, for nu
 
 Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
 
-## What the validation rules do with them
+## What the Data Controller validation rules do with them
 
-A special missing is a value somebody deliberately set, so the rules treat it as a value - it is not blank, and it is not exempt:
+These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit. A special missing is a value somebody deliberately set, so the rules treat it as a value - it is not blank, and it is not exempt:
 
 - `NOTNULL` - passes, because it is not null
 - `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum
 - `MAXVAL` - passes, for the same reason it is below any maximum
-- `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not
+- `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
-- `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
+- `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
+- `HARDFORMULA` / `SOFTFORMULA` - a formula that reads a special-missing cell does not compute; the grid's spreadsheet engine returns `#VALUE!` for that row, where the plain `.` contributes 0
+
+One mismatch to be aware of: the `NOTNULL` rule above is the editor's, and it is more forgiving than the SAS constraint it mirrors. A real NOT NULL - or primary key - constraint on the target table rejects a special missing, so on a column carrying one the editor will let the value through and the load will then fail on the constraint.
+
+Two other sharp edges. A range rule's *value* has to be a number: put a special missing in `MINVAL` and every cell in the column fails, put one in `MAXVAL` and every real number fails. And although the regex pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
+
+`CASE` (`UPCASE` / `LOWCASE`) is a character rule and does not come into it: SAS hands the browser a special missing as an uppercase letter, so there is no case to enforce, and a `CASE` rule on a numeric column would reject the column's numbers rather than the missing.
 
 That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column.
 
@@ -69,6 +79,7 @@ They are real numeric values, not text:
 - PROC MEANS excludes them, like any other missing
 - they sort below every number: ._ then . then .A to .Z
 - NMISS() and CMISS() count them as missing
+- a regular missing prints as . unless options MISSING changes it - and that option never touches a special missing
 
 The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
 
@@ -78,11 +89,14 @@ We solved that in the open source SASjs Adapter, and it has been in Data Control
 - where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
 - you type the letter on its own: a or A, no period, and case does not matter
 
-And they behave like values in DC's validation rules, which is usually what you want:
+And they behave like values in Data Controller's validation rules, which is usually what you want:
 
-- NOTNULL passes - it is not null
+- NOTNULL passes - it is not null (though a real SAS NOT NULL constraint is stricter, and does reject a special missing)
 - MINVAL fails - it sorts below every number; MAXVAL passes for the same reason
 - HARDREGEX checks it against the pattern, unlike blanks and plain .
+- SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
+- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
+- CASE is a character rule, so it has no place on a numeric column
 
 So a value recorded as "not collected" cannot quietly satisfy a completeness or range rule.
 
@@ -90,5 +104,32 @@ The video shows the whole cycle - entering them, the rejections, submit, approve
 
 #sas #datacapture #mdm #dataquality
 
-Image prompt: a pack of 28 wolves fanning out across a snowy plain at dusk, one wolf standing apart to the left, wide cinematic composition, cool blue-grey palette with a single warm amber accent on the lone wolf, soft depth of field, no text and no logos, 16:9, editorial cover art for a technical blog post about the 28 SAS missing values. Save the generated image as ./cover.jpeg and set previewImg: './cover.jpeg' in the front matter above.
+Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
+dozen or more, spread wide and moving together across a vast snow plain at dusk,
+seen from a low wide angle. One wolf stands apart from the group on the left of
+frame, turned back toward the pack. The pack is rendered in cool blue-grey and
+silver; the lone wolf catches the only warm light in the scene, a single low
+amber sun. Overcast dusk sky, faint falling snow, long soft shadows, generous
+empty sky and snow to the upper third so the wide crop breathes. Painterly
+digital illustration, muted desaturated palette, soft depth of field with the
+distant wolves falling out of focus, no text, no letters, no numbers, no logos,
+no watermark.
+
+Constraints: no text, no letters, no numbers, no digits, no symbols, no
+captions, no logos, no watermark, no signature, no border, no frame, no collar,
+no harness, no humans, no buildings, and do not ask for an exact head count of
+28 - a crowded pack reads worse than a dozen clear animals, and the number
+belongs in the headline, not the artwork. The no-text rule matters more than
+usual here: the subject is letters standing in for numbers, so a stray glyph
+anywhere undercuts the cover.
+
+Output: save as ./cover.jpeg at 1.91:1 (1200x627, matching the other feed
+covers and doubling as the LinkedIn share card), and set
+previewImg: './cover.jpeg' in the front matter above. Do not also embed the
+image in the markdown body. Full prompt, variants and rationale:
+https://paste.4gl.io/?513711f5bde3610e#9LZCA9xzQHDy3CycXmsWU5AoVNtjBZmPotUHHtHTAerF
+
+Fallback variant if the pack composition comes back muddled: a single wolf
+standing alone on a snow plain at dusk, its shadow stretching toward a distant
+pack reduced to small silhouettes on the horizon.
 -->