diff --git a/content/blog/v4-0-formats-special-missings/index.md b/content/blog/v4-0-formats-special-missings/index.md index 4caebfb..7b2ff3c 100644 --- a/content/blog/v4-0-formats-special-missings/index.md +++ b/content/blog/v4-0-formats-special-missings/index.md @@ -57,14 +57,52 @@ Information on configuration is available in the [documentation](https://docs.da ## View & Edit SAS Special Missing Numerics -Did you know that, in addition to a regular missing value in SAS (`.`), there are 27 other types of missing? They are represented by the letters a-z and an underscore (`._`). +### 28 ways to be missing -These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types). +A numeric missing value in SAS is not a single thing. The ordinary missing (`.`) is one of **28** distinct missing values available for a numeric variable - the other 27 are written with a single character, the letters `A` to `Z` or an underscore (`._`). -`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)` +They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing. + +They are legal numeric values, not strings: + +* Arithmetic on them yields missing - `.A + 1` is `.` +* `proc means`, `proc summary` and friends exclude them, exactly as they exclude `.` +* They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z` +* The `NMISS()` and `CMISS()` functions count them as missing +* Converting one to text drops the period - `cats(.A)` is the string `A`, which is exactly the single character the Data Controller grid shows you + +In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive. + +### How Data Controller carries them between the browser and SAS + +The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types): + +* The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) **alongside** numeric values is numeric, with the lone characters written to SAS as special missings. +* `null` becomes `.` or an empty string, according to the type derived for that column. +* Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend. +* A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing. There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells. +Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF screen's formatted / unformatted switch shows you either the formatted representation or the raw value - useful for confirming exactly which missing was set. + +### Special missings and the other Data Controller rules + +A special missing is a value somebody deliberately set, so the validation rules treat it as a value - it is not blank, and it is not exempt: + +* `NOTNULL` - passes, because it is not null +* `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum +* `MAXVAL` - passes, for the same reason it is below any maximum +* `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not +* `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter + +That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column. + +`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)` + +The recording shows the full cycle: typing special missings into numeric cells (a single letter is accepted, two letters or a number-and-letter combination are rejected), submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns. + + ## Audit History Table Previously, transactional changes made to tables in Data Controller could only be tracked by means of individual CSV files. A user could (and still can) navigate to the HISTORY tab, find their change, and download a zip file containing all relevant information such as the original excel that was uploaded, SAS logs, the changed records (CSV) and the staging dataset. @@ -92,3 +130,41 @@ We continue to update and improve Data Controller. Upcoming features include:
Did you know Data Controller Community Edition is free to use? [Contact us](/contact) for your copy! + + +