dc
d059b72f6b
The special missings section of the v4.0 post was four lines: what they are, that the adapter supports them, and a video. It is now three sections. - What special missings are: the 28 missing values, why they exist, and how they behave as numeric values (arithmetic, PROC MEANS, sort order, NMISS/CMISS, cats()). Every claim was checked on a live SAS engine - the sort order I first wrote down was wrong, ._ sorts below the regular missing. - How they travel between the browser and SAS: the adapter's type inference, the null handling, the format override for a column holding only special missings, and the values it refuses. - How they behave in the other Data Controller rules: NOTNULL passes, MINVAL fails, MAXVAL passes, CASE compares them as text, and HARDREGEX/SOFTREGEX do not exempt them. The video embed was pointing at the right recording under the wrong label (a copy-paste of "Retain Formulas when Loading Excel to SAS" from other posts). It is "Managing Special Missing Values with Data Controller for SAS", and the post now describes what it shows. Also adds the LinkedIn version of the section as a comment, kept in sync with the copy above.
171 lines
10 KiB
Markdown
171 lines
10 KiB
Markdown
---
|
|
title: "v4.0 Release: Formats & Special Missings"
|
|
description: This release provides support for viewing/editing Format Catalogs, plus the ability to work with special SAS missing numerics (.a, .b etc)
|
|
date: '2022-03-07 09:00:00'
|
|
author: 'Allan Bowe'
|
|
authorLink: https://www.linkedin.com/in/allanbowe/
|
|
previewImg: './formats.png'
|
|
tags:
|
|
- Formats
|
|
- Releases
|
|
- Special Missings
|
|
---
|
|
|
|
We're excited to announce the new features in the v4 release:
|
|
|
|
* Ability to VIEW and EDIT values directly in Format Catalogs
|
|
* Ability to VIEW and EDIT special SAS missing numeric values (.a, .b etc)
|
|
* Audit table (single source for all data modifications in DC)
|
|
* Additional metadata on table viewer (Primary Key, labels, lengths, formats)
|
|
* Formatted / Unformatted switch on the DIFF screen
|
|
|
|
As well as the following fixes:
|
|
|
|
* Identical filter clauses on different tables being mis-assigned (this is the breaking change)
|
|
* Filter clauses applied to Excel / CSV / Datalines downloads
|
|
* Data values may now contain _leading_ blanks
|
|
* Customers may now have a single licence for multiple site ids
|
|
* Support for target table record deletion in Excel Uploads
|
|
|
|
We've also completely refactored the underlying engine for [Dynamic Cell Dropdowns](https://docs.datacontroller.io/dynamic-cell-dropdown) to make it faster, more robust, and easier to test.
|
|
|
|
We're also pleased to report that all the backend SAS Services now run with [strict mode](https://core.sasjs.io/mp__init_8sas.html) enabled, ensuring that SAS will 'fail fast' in case of data issues.
|
|
|
|
Some more background on our major features:
|
|
|
|
## View & Edit Format Catalogs
|
|
|
|
Data Controller allows you to view and edit formats _directly_ in the catalog. It does not require the maintenance of a secondary "cntlin" table! This ensures that the formats you are viewing / editing are indeed, the latest ones.
|
|
|
|
All of the usual Data Controller features are available for formats too, including:
|
|
|
|
* Configure HOOK scripts to run before / after a change or approval
|
|
* Locking mechanism to avoid issues with parallel updates
|
|
* Make changes in the web, or via Excel or CSV upload
|
|
* Edit individual format entries on dedicated screen
|
|
* Create complex filters (and save filtered views)
|
|
* Full-table (catalog) search for specific values
|
|
* Download in Excel, CSV or Datalines formats
|
|
* Export the DDL (in various flavours)
|
|
* Mark format entries for deletion
|
|
|
|
|
|
|
|
|
|
Information on configuration is available in the [documentation](https://docs.datacontroller.io/formats)
|
|
|
|
|
|
## View & Edit SAS Special Missing Numerics
|
|
|
|
### 28 ways to be missing
|
|
|
|
A numeric missing value in SAS is not a single thing. The ordinary missing (`.`) is one of **28** distinct missing values available for a numeric variable - the other 27 are written with a single character, the letters `A` to `Z` or an underscore (`._`).
|
|
|
|
They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
|
|
|
|
They are legal numeric values, not strings:
|
|
|
|
* Arithmetic on them yields missing - `.A + 1` is `.`
|
|
* `proc means`, `proc summary` and friends exclude them, exactly as they exclude `.`
|
|
* They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
|
|
* The `NMISS()` and `CMISS()` functions count them as missing
|
|
* Converting one to text drops the period - `cats(.A)` is the string `A`, which is exactly the single character the Data Controller grid shows you
|
|
|
|
In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type just the letter or the underscore - no period - and the letter is not case sensitive.
|
|
|
|
### How Data Controller carries them between the browser and SAS
|
|
|
|
The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
|
|
|
|
* The adapter infers each column's SAS type from the values it is given. All numeric values means numeric; all strings means character; and a column holding a single character (`a`-`z`, `_` or `.`) **alongside** numeric values is numeric, with the lone characters written to SAS as special missings.
|
|
* `null` becomes `.` or an empty string, according to the type derived for that column.
|
|
* Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
|
|
* A value that is neither a number nor a single valid character is refused rather than guessed (for example `aaaa`, or `!` in a numeric column), and a literal `.` is refused in favour of `null` for a regular missing.
|
|
|
|
There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
|
|
|
|
Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF screen's formatted / unformatted switch shows you either the formatted representation or the raw value - useful for confirming exactly which missing was set.
|
|
|
|
### Special missings and the other Data Controller rules
|
|
|
|
A special missing is a value somebody deliberately set, so the validation rules treat it as a value - it is not blank, and it is not exempt:
|
|
|
|
* `NOTNULL` - passes, because it is not null
|
|
* `MINVAL` - fails, because a special missing sorts below every number and so is below any minimum
|
|
* `MAXVAL` - passes, for the same reason it is below any maximum
|
|
* `CASE` (`UPCASE` / `LOWCASE`) - compared as text, so `A` passes `UPCASE` but `a` does not
|
|
* `HARDREGEX` / `SOFTREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
|
|
|
|
That is usually what you want - a value recorded as "not collected" should not quietly satisfy a completeness or range rule. It does mean a range rule on a column that uses special missings will warn on those rows, which is the signal to either accommodate them in the rule or not use special missings on that column.
|
|
|
|
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
|
|
|
The recording shows the full cycle: typing special missings into numeric cells (a single letter is accepted, two letters or a number-and-letter combination are rejected), submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on date and datetime columns.
|
|
|
|
|
|
## Audit History Table
|
|
|
|
Previously, transactional changes made to tables in Data Controller could only be tracked by means of individual CSV files. A user could (and still can) navigate to the HISTORY tab, find their change, and download a zip file containing all relevant information such as the original excel that was uploaded, SAS logs, the changed records (CSV) and the staging dataset.
|
|
|
|
The issue is that this information did not capture the original, unchanged records (necessary to support rollbacks), nor did it provide a central point for querying the entire change history of a particular variable / record over time.
|
|
|
|
These issues are now resolved with the introduction of the MPE_AUDIT table.
|
|
|
|
|
|
|
|
The macro used to load this table is open source and available [here](https://core.sasjs.io/mp__storediffs_8sas.html).
|
|
|
|
## Roadmap (7th March 2022)
|
|
|
|
We continue to update and improve Data Controller. Upcoming features include:
|
|
|
|
* PK highlighting for VIEW as well as EDIT tables
|
|
* Data Rollback
|
|
* Ability to configure individual audit tables (ie, 1 per EDIT table)
|
|
* Data Controller API
|
|
* Data Controller on Viya 4
|
|
* Data Controller on Desktop SAS (using [sasjs/server](https://github.com/sasjs/server))
|
|
|
|
|
|
<hr>
|
|
|
|
Did you know Data Controller Community Edition is free to use? [Contact us](/contact) for your copy!
|
|
|
|
<!--
|
|
LinkedIn version of the special missings section above - post it with the
|
|
"Managing Special Missing Values with Data Controller for SAS" video attached.
|
|
Kept in sync with the copy above: same points, same claims, same order.
|
|
|
|
A SAS numeric missing is not one thing. It is 28.
|
|
|
|
The ordinary missing (.) is just the most common of them. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
|
|
|
|
They are real numeric values, not text:
|
|
|
|
- .A + 1 is .
|
|
- PROC MEANS excludes them, like any other missing
|
|
- they sort below every number: ._ then . then .A to .Z
|
|
- NMISS() and CMISS() count them as missing
|
|
|
|
The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
|
|
|
|
We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
|
|
|
|
- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
|
|
- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
|
|
- you type the letter on its own: a or A, no period, and case does not matter
|
|
|
|
And they behave like values in DC's validation rules, which is usually what you want:
|
|
|
|
- NOTNULL passes - it is not null
|
|
- MINVAL fails - it sorts below every number; MAXVAL passes for the same reason
|
|
- HARDREGEX checks it against the pattern, unlike blanks and plain .
|
|
|
|
So a value recorded as "not collected" cannot quietly satisfy a completeness or range rule.
|
|
|
|
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF.
|
|
|
|
#sas #datacapture #mdm #dataquality
|
|
-->
|
|
|