diff --git a/content/blog/v4-0-formats-special-missings/index.md b/content/blog/v4-0-formats-special-missings/index.md index 4caebfb..8bc2e27 100644 --- a/content/blog/v4-0-formats-special-missings/index.md +++ b/content/blog/v4-0-formats-special-missings/index.md @@ -61,7 +61,7 @@ Did you know that, in addition to a regular missing value in SAS (`.`), there ar These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types). -`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)` +`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)` There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells. diff --git a/content/feed/sas-special-missings/cover.jpeg b/content/feed/sas-special-missings/cover.jpeg new file mode 100644 index 0000000..4d64eee Binary files /dev/null and b/content/feed/sas-special-missings/cover.jpeg differ diff --git a/content/feed/sas-special-missings/index.md b/content/feed/sas-special-missings/index.md new file mode 100644 index 0000000..0ddcca3 --- /dev/null +++ b/content/feed/sas-special-missings/index.md @@ -0,0 +1,122 @@ +--- +title: '28 Ways to Be Missing in SAS' +description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has handled all of them since v4. How they work, and what the validation rules do with them. +date: '2026-09-22 09:00:00' +author: 'Allan Bowe' +authorLink: https://www.linkedin.com/in/allanbowe/ +tags: + - Special Missings + - Data Quality +previewImg: './cover.jpeg' +--- + +# 28 Ways to Be Missing in SAS + +A numeric missing value in SAS is not a lone wolf. The ordinary missing (`.`) is one of **28** distinct missing values available for a numeric variable - the other 27 are written with a single character, the letters `A` to `Z` or an underscore (`._`). + +They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing. + +## They are numbers, not text + +- Arithmetic on them yields missing - `.A + 1` is `.` +- `proc means`, `proc summary` and friends exclude them, exactly as they exclude `.` +- They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z` +- The `NMISS()` and `CMISS()` functions count them as missing +- Converting one to text drops the period - `cats(.A)` is the string `A` + +In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed. + +## Carrying them between the browser and SAS + +The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types): + +- The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings. +- `null` becomes `.` or an empty string, according to the type derived for that column. +- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend. +- A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing. + +There is nothing to configure. Special missings are available by default, for numeric cells. + +Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set. + +## What the validation rules do with them + +A special missing is a value somebody deliberately set, so the rules treat it as a value - it is not blank, and it is not exempt: + +- `NOTNULL` - passes, because it is not null +- `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum +- `MAXVAL` - passes, for the same reason it is below any maximum +- `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not +- `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter + +That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column. + +`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)` + +The recording shows the full cycle: typing special missings into numeric cells, the rejections, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns. + +