Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6d554930be | ||
|
|
a56c2f8d48 | ||
|
|
27ff40b739 | ||
|
|
4937edd7f4 |
@@ -57,51 +57,13 @@ Information on configuration is available in the [documentation](https://docs.da
|
|||||||
|
|
||||||
## View & Edit SAS Special Missing Numerics
|
## View & Edit SAS Special Missing Numerics
|
||||||
|
|
||||||
### 28 ways to be missing
|
Did you know that, in addition to a regular missing value in SAS (`.`), there are 27 other types of missing? They are represented by the letters a-z and an underscore (`._`).
|
||||||
|
|
||||||
A numeric missing value in SAS is not a single thing. The ordinary missing (`.`) is one of **28** distinct missing values available for a numeric variable - the other 27 are written with a single character, the letters `A` to `Z` or an underscore (`._`).
|
These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
|
||||||
|
|
||||||
They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
|
|
||||||
|
|
||||||
They are legal numeric values, not strings:
|
|
||||||
|
|
||||||
* Arithmetic on them yields missing - `.A + 1` is `.`
|
|
||||||
* `proc means`, `proc summary` and friends exclude them, exactly as they exclude `.`
|
|
||||||
* They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
|
|
||||||
* The `NMISS()` and `CMISS()` functions count them as missing
|
|
||||||
* Converting one to text drops the period - `cats(.A)` is the string `A`, which is exactly the single character the Data Controller grid shows you
|
|
||||||
|
|
||||||
In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive.
|
|
||||||
|
|
||||||
### How Data Controller carries them between the browser and SAS
|
|
||||||
|
|
||||||
The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
|
|
||||||
|
|
||||||
* The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) **alongside** numeric values is numeric, with the lone characters written to SAS as special missings.
|
|
||||||
* `null` becomes `.` or an empty string, according to the type derived for that column.
|
|
||||||
* Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
|
|
||||||
* A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing.
|
|
||||||
|
|
||||||
There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
|
|
||||||
|
|
||||||
Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF screen's formatted / unformatted switch shows you either the formatted representation or the raw value - useful for confirming exactly which missing was set.
|
|
||||||
|
|
||||||
### Special missings and the other Data Controller rules
|
|
||||||
|
|
||||||
A special missing is a value somebody deliberately set, so the validation rules treat it as a value - it is not blank, and it is not exempt:
|
|
||||||
|
|
||||||
* `NOTNULL` - passes, because it is not null
|
|
||||||
* `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum
|
|
||||||
* `MAXVAL` - passes, for the same reason it is below any maximum
|
|
||||||
* `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not
|
|
||||||
* `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
|
|
||||||
|
|
||||||
That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column.
|
|
||||||
|
|
||||||
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
||||||
|
|
||||||
The recording shows the full cycle: typing special missings into numeric cells (a single letter is accepted, two letters or a number-and-letter combination are rejected), submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns.
|
There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
|
||||||
|
|
||||||
|
|
||||||
## Audit History Table
|
## Audit History Table
|
||||||
|
|
||||||
@@ -130,41 +92,3 @@ We continue to update and improve Data Controller. Upcoming features include:
|
|||||||
<hr>
|
<hr>
|
||||||
|
|
||||||
Did you know Data Controller Community Edition is free to use? [Contact us](/contact) for your copy!
|
Did you know Data Controller Community Edition is free to use? [Contact us](/contact) for your copy!
|
||||||
|
|
||||||
<!--
|
|
||||||
LinkedIn version of the special missings section above - post it with the
|
|
||||||
"Managing Special Missing Values with Data Controller for SAS" video attached.
|
|
||||||
Kept in sync with the copy above: same points, same claims, same order.
|
|
||||||
|
|
||||||
A SAS numeric missing is not one thing. It is 28.
|
|
||||||
|
|
||||||
The ordinary missing (.) is just the most common of them. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
|
|
||||||
|
|
||||||
They are real numeric values, not text:
|
|
||||||
|
|
||||||
- .A + 1 is .
|
|
||||||
- PROC MEANS excludes them, like any other missing
|
|
||||||
- they sort below every number: ._ then . then .A to .Z
|
|
||||||
- NMISS() and CMISS() count them as missing
|
|
||||||
|
|
||||||
The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
|
|
||||||
|
|
||||||
We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
|
|
||||||
|
|
||||||
- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
|
|
||||||
- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
|
|
||||||
- you type the letter on its own: a or A, no period, and case does not matter
|
|
||||||
|
|
||||||
And they behave like values in DC's validation rules, which is usually what you want:
|
|
||||||
|
|
||||||
- NOTNULL passes - it is not null
|
|
||||||
- MINVAL fails - it sorts below every number; MAXVAL passes for the same reason
|
|
||||||
- HARDREGEX checks it against the pattern, unlike blanks and plain .
|
|
||||||
|
|
||||||
So a value recorded as "not collected" cannot quietly satisfy a completeness or range rule.
|
|
||||||
|
|
||||||
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF.
|
|
||||||
|
|
||||||
#sas #datacapture #mdm #dataquality
|
|
||||||
-->
|
|
||||||
|
|
||||||
|
|||||||
Binary file not shown.
|
After Width: | Height: | Size: 148 KiB |
@@ -0,0 +1,122 @@
|
|||||||
|
---
|
||||||
|
title: '28 Ways to Be Missing in SAS'
|
||||||
|
description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has handled all of them since v4. How they work, and what the validation rules do with them.
|
||||||
|
date: '2026-09-22 09:00:00'
|
||||||
|
author: 'Allan Bowe'
|
||||||
|
authorLink: https://www.linkedin.com/in/allanbowe/
|
||||||
|
tags:
|
||||||
|
- Special Missings
|
||||||
|
- Data Quality
|
||||||
|
previewImg: './cover.jpeg'
|
||||||
|
---
|
||||||
|
|
||||||
|
# 28 Ways to Be Missing in SAS
|
||||||
|
|
||||||
|
A numeric missing value in SAS is not a lone wolf. The ordinary missing (`.`) is one of **28** distinct missing values available for a numeric variable - the other 27 are written with a single character, the letters `A` to `Z` or an underscore (`._`).
|
||||||
|
|
||||||
|
They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
|
||||||
|
|
||||||
|
## They are numbers, not text
|
||||||
|
|
||||||
|
- Arithmetic on them yields missing - `.A + 1` is `.`
|
||||||
|
- `proc means`, `proc summary` and friends exclude them, exactly as they exclude `.`
|
||||||
|
- They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
|
||||||
|
- The `NMISS()` and `CMISS()` functions count them as missing
|
||||||
|
- Converting one to text drops the period - `cats(.A)` is the string `A`
|
||||||
|
|
||||||
|
In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
|
||||||
|
|
||||||
|
## Carrying them between the browser and SAS
|
||||||
|
|
||||||
|
The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
|
||||||
|
|
||||||
|
- The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
|
||||||
|
- `null` becomes `.` or an empty string, according to the type derived for that column.
|
||||||
|
- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
|
||||||
|
- A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing.
|
||||||
|
|
||||||
|
There is nothing to configure. Special missings are available by default, for numeric cells.
|
||||||
|
|
||||||
|
Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
|
||||||
|
|
||||||
|
## What the validation rules do with them
|
||||||
|
|
||||||
|
A special missing is a value somebody deliberately set, so the rules treat it as a value - it is not blank, and it is not exempt:
|
||||||
|
|
||||||
|
- `NOTNULL` - passes, because it is not null
|
||||||
|
- `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum
|
||||||
|
- `MAXVAL` - passes, for the same reason it is below any maximum
|
||||||
|
- `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not
|
||||||
|
- `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
|
||||||
|
|
||||||
|
That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column.
|
||||||
|
|
||||||
|
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
||||||
|
|
||||||
|
The recording shows the full cycle: typing special missings into numeric cells, the rejections, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns.
|
||||||
|
|
||||||
|
<!--
|
||||||
|
LinkedIn version of this post - publish it with the "Managing Special Missing
|
||||||
|
Values with Data Controller for SAS" video attached. Kept in sync with the copy
|
||||||
|
above: same points, same claims, same order.
|
||||||
|
|
||||||
|
A SAS numeric missing is not a lone wolf. It is 28.
|
||||||
|
|
||||||
|
The ordinary missing (.) is just the most common of them. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
|
||||||
|
|
||||||
|
They are real numeric values, not text:
|
||||||
|
|
||||||
|
- .A + 1 is .
|
||||||
|
- PROC MEANS excludes them, like any other missing
|
||||||
|
- they sort below every number: ._ then . then .A to .Z
|
||||||
|
- NMISS() and CMISS() count them as missing
|
||||||
|
|
||||||
|
The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
|
||||||
|
|
||||||
|
We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
|
||||||
|
|
||||||
|
- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
|
||||||
|
- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
|
||||||
|
- you type the letter on its own: a or A, no period, and case does not matter
|
||||||
|
|
||||||
|
And they behave like values in DC's validation rules, which is usually what you want:
|
||||||
|
|
||||||
|
- NOTNULL passes - it is not null
|
||||||
|
- MINVAL fails - it sorts below every number; MAXVAL passes for the same reason
|
||||||
|
- HARDREGEX checks it against the pattern, unlike blanks and plain .
|
||||||
|
|
||||||
|
So a value recorded as "not collected" cannot quietly satisfy a completeness or range rule.
|
||||||
|
|
||||||
|
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF.
|
||||||
|
|
||||||
|
#sas #datacapture #mdm #dataquality
|
||||||
|
|
||||||
|
Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
|
||||||
|
dozen or more, spread wide and moving together across a vast snow plain at dusk,
|
||||||
|
seen from a low wide angle. One wolf stands apart from the group on the left of
|
||||||
|
frame, turned back toward the pack. The pack is rendered in cool blue-grey and
|
||||||
|
silver; the lone wolf catches the only warm light in the scene, a single low
|
||||||
|
amber sun. Overcast dusk sky, faint falling snow, long soft shadows, generous
|
||||||
|
empty sky and snow to the upper third so the wide crop breathes. Painterly
|
||||||
|
digital illustration, muted desaturated palette, soft depth of field with the
|
||||||
|
distant wolves falling out of focus, no text, no letters, no numbers, no logos,
|
||||||
|
no watermark.
|
||||||
|
|
||||||
|
Constraints: no text, no letters, no numbers, no digits, no symbols, no
|
||||||
|
captions, no logos, no watermark, no signature, no border, no frame, no collar,
|
||||||
|
no harness, no humans, no buildings, and do not ask for an exact head count of
|
||||||
|
28 - a crowded pack reads worse than a dozen clear animals, and the number
|
||||||
|
belongs in the headline, not the artwork. The no-text rule matters more than
|
||||||
|
usual here: the subject is letters standing in for numbers, so a stray glyph
|
||||||
|
anywhere undercuts the cover.
|
||||||
|
|
||||||
|
Output: save as ./cover.jpeg at 1.91:1 (1200x627, matching the other feed
|
||||||
|
covers and doubling as the LinkedIn share card), and set
|
||||||
|
previewImg: './cover.jpeg' in the front matter above. Do not also embed the
|
||||||
|
image in the markdown body. Full prompt, variants and rationale:
|
||||||
|
https://paste.4gl.io/?513711f5bde3610e#9LZCA9xzQHDy3CycXmsWU5AoVNtjBZmPotUHHtHTAerF
|
||||||
|
|
||||||
|
Fallback variant if the pack composition comes back muddled: a single wolf
|
||||||
|
standing alone on a snow plain at dusk, its shadow stretching toward a distant
|
||||||
|
pack reduced to small silhouettes on the horizon.
|
||||||
|
-->
|
||||||
Reference in New Issue
Block a user