feed: add a special missings post #19
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: '28 Ways to Be Missing in SAS'
|
||||
description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has handled all of them since v4. How they work, and what the validation rules do with them.
|
||||
date: '2026-09-22 09:00:00'
|
||||
author: 'Allan Bowe'
|
||||
authorLink: https://www.linkedin.com/in/allanbowe/
|
||||
tags:
|
||||
- Special Missings
|
||||
- Data Quality
|
||||
---
|
||||
|
||||
# 28 Ways to Be Missing in SAS
|
||||
|
||||
A numeric missing value in SAS is not a lone wolf. The ordinary missing (`.`) is one of **28** distinct missing values available for a numeric variable - the other 27 are written with a single character, the letters `A` to `Z` or an underscore (`._`).
|
||||
|
||||
They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
|
||||
|
||||
## They are numbers, not text
|
||||
|
||||
- Arithmetic on them yields missing - `.A + 1` is `.`
|
||||
- `proc means`, `proc summary` and friends exclude them, exactly as they exclude `.`
|
||||
- They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
|
||||
- The `NMISS()` and `CMISS()` functions count them as missing
|
||||
- Converting one to text drops the period - `cats(.A)` is the string `A`
|
||||
|
||||
In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
|
||||
|
||||
## Carrying them between the browser and SAS
|
||||
|
||||
The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
|
||||
|
||||
- The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
|
||||
- `null` becomes `.` or an empty string, according to the type derived for that column.
|
||||
- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
|
||||
- A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing.
|
||||
|
||||
There is nothing to configure. Special missings are available by default, for numeric cells.
|
||||
|
||||
Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
|
||||
|
||||
## What the validation rules do with them
|
||||
|
||||
A special missing is a value somebody deliberately set, so the rules treat it as a value - it is not blank, and it is not exempt:
|
||||
|
||||
- `NOTNULL` - passes, because it is not null
|
||||
- `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum
|
||||
- `MAXVAL` - passes, for the same reason it is below any maximum
|
||||
- `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not
|
||||
- `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
|
||||
|
||||
That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column.
|
||||
|
||||
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
||||
|
||||
The recording shows the full cycle: typing special missings into numeric cells, the rejections, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns.
|
||||
|
||||
<!--
|
||||
LinkedIn version of this post - publish it with the "Managing Special Missing
|
||||
Values with Data Controller for SAS" video attached. Kept in sync with the copy
|
||||
above: same points, same claims, same order.
|
||||
|
||||
A SAS numeric missing is not a lone wolf. It is 28.
|
||||
|
||||
The ordinary missing (.) is just the most common of them. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
|
||||
|
||||
They are real numeric values, not text:
|
||||
|
||||
- .A + 1 is .
|
||||
- PROC MEANS excludes them, like any other missing
|
||||
- they sort below every number: ._ then . then .A to .Z
|
||||
- NMISS() and CMISS() count them as missing
|
||||
|
||||
The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
|
||||
|
||||
We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
|
||||
|
||||
- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
|
||||
- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
|
||||
- you type the letter on its own: a or A, no period, and case does not matter
|
||||
|
||||
And they behave like values in DC's validation rules, which is usually what you want:
|
||||
|
||||
- NOTNULL passes - it is not null
|
||||
- MINVAL fails - it sorts below every number; MAXVAL passes for the same reason
|
||||
- HARDREGEX checks it against the pattern, unlike blanks and plain .
|
||||
|
||||
So a value recorded as "not collected" cannot quietly satisfy a completeness or range rule.
|
||||
|
||||
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF.
|
||||
|
||||
#sas #datacapture #mdm #dataquality
|
||||
|
||||
Image prompt: a pack of 28 wolves fanning out across a snowy plain at dusk, one wolf standing apart to the left, wide cinematic composition, cool blue-grey palette with a single warm amber accent on the lone wolf, soft depth of field, no text and no logos, 16:9, editorial cover art for a technical blog post about the 28 SAS missing values. Save the generated image as ./cover.jpeg and set previewImg: './cover.jpeg' in the front matter above.
|
||||
-->
|
||||
Reference in New Issue
Block a user