Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
35c60941da | ||
|
|
59ff2b13fc | ||
|
|
7d950cbdbe | ||
|
|
66d5fecb0e | ||
|
|
0b18658686 | ||
|
|
bbd5793d8b | ||
|
|
0bda92b091 | ||
|
|
bf9fca1b9a | ||
|
|
d542ac442f | ||
|
|
0ee0b4ea7c | ||
|
|
2fe1bc2eea | ||
|
|
3e39339332 | ||
|
|
6d554930be | ||
|
|
a56c2f8d48 | ||
|
|
27ff40b739 | ||
|
|
4937edd7f4 | ||
|
|
fcba390804 | ||
|
|
8b6e3e60bc | ||
|
|
d244a34fc9 | ||
|
|
686a745f0b | ||
|
|
510947d173 | ||
|
|
589a745c5b | ||
|
|
d04c55c731 | ||
|
|
a6a6abf009 | ||
|
|
88368fa31d | ||
|
|
fdb1db216d | ||
|
|
6d833f9654 | ||
|
|
6bcd0adf81 | ||
|
|
93348ad604 | ||
|
|
7561bb0c39 | ||
|
|
959a9ffab4 | ||
|
|
2348575737 | ||
|
|
b2aac21b90 | ||
|
|
438b1e93d7 | ||
|
|
674737f840 | ||
|
|
150be89246 |
|
After Width: | Height: | Size: 50 KiB |
@@ -0,0 +1,112 @@
|
|||||||
|
---
|
||||||
|
title: 'One Table, Two Rule Sets'
|
||||||
|
description: 'A single physical table can carry only one set of Data Controller validation rules. Two librefs over the same data - or a pair of hook scripts - give you as many rule sets as you need.'
|
||||||
|
date: '2026-09-24 15:30:00'
|
||||||
|
author: 'Data Controller'
|
||||||
|
authorLink: https://www.linkedin.com/showcase/data-controller-for-sas
|
||||||
|
tags:
|
||||||
|
- Data Quality
|
||||||
|
- Configuration
|
||||||
|
previewImg: './rule-set-1.png'
|
||||||
|
---
|
||||||
|
|
||||||
|
# One Table, Two Rule Sets
|
||||||
|
|
||||||
|
Most Data Controller sites settle into an obvious mapping: one table, one edit screen, one set of validation rules. But that is not always what the business wants. A table of orders might be edited from a finance report that only tolerates small adjustments, and from an operations report where much larger ones are routine. Same table, same columns, same approvers - different rules.
|
||||||
|
|
||||||
|
Data Controller's validation rules are configured per table, so this takes a little thought. There are two ways to do it: the one we recommend, and the one to reach for when the first is not available.
|
||||||
|
|
||||||
|
## Why one table is one rule set
|
||||||
|
|
||||||
|
Two configuration tables decide this.
|
||||||
|
|
||||||
|
`MPE_TABLES` is the list of editable tables, and its primary key is `(tx_from, libref, dsn)`. One physical table is one editable table.
|
||||||
|
|
||||||
|
`MPE_VALIDATIONS` holds the rules, and its primary key is `(tx_from, base_lib, base_ds, base_col, rule_type)`. Rules hang off a physical `libref.dataset`. There is no per-menu or per-report scoping anywhere in the schema, and the editor is handed exactly the rules whose `base_lib` and `base_ds` match the table being opened.
|
||||||
|
|
||||||
|
So two rule sets on one table need two distinct `libref.dataset` identities. The question is how to get them without copying the data.
|
||||||
|
|
||||||
|
## Option 1 (recommended): two librefs over the same data
|
||||||
|
|
||||||
|
A libref is just a name pointing at a location. Nothing stops you assigning two of them to the same place, and Data Controller will treat the two as separate tables:
|
||||||
|
|
||||||
|
```sas
|
||||||
|
libname ORDERS_EU '/data/orders';
|
||||||
|
libname ORDERS_US '/data/orders';
|
||||||
|
```
|
||||||
|
|
||||||
|
`ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` are now the same physical file, but they are different rows in `MPE_TABLES` and can therefore carry different rows in `MPE_VALIDATIONS`. Register both, give each its own rules, and point each report at its own editor URL - `#/editor/ORDERS_EU.ORDERS` and `#/editor/ORDERS_US.ORDERS`.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Everything else works exactly as it always did. Filtering, search, the row cap, the approval diff and the audit trail all operate on the table as normal, because as far as Data Controller is concerned these are ordinary tables. The only difference is that the two names resolve to the same file, so an approval in either report updates the same data.
|
||||||
|
|
||||||
|
There is no copy to keep in sync, no hook to write and nothing to maintain. That is why it is the option we recommend.
|
||||||
|
|
||||||
|
### Things worth knowing
|
||||||
|
|
||||||
|
- Where the librefs are defined depends on your platform. On Viya, in the compute context's `autoexec.sas` (or `[DC Drive Path]/services/settings.sas`); on SAS 9, as metadata libraries or in the Data Controller Settings stored process; on SASjs Server, in `services/public/settings.sas`. The one requirement is that each library has a unique libref.
|
||||||
|
- `mp_lockanytable` keys on `libref.dataset`, so the two menus do not serialise against each other. Two people editing through different reports at the same moment can therefore collide at the database level. If that matters, add an explicit shared lock in a `PRE_APPROVE_HOOK`.
|
||||||
|
- The audit trail and approval queue record which libref a change came through, so `ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` stay distinguishable in history. For most people that is a feature - you can see which report a change originated from.
|
||||||
|
|
||||||
|
## Option 2: an empty mirror and a pair of hook scripts
|
||||||
|
|
||||||
|
Sometimes two librefs over one location are not available: a database library where the platform will not let you define the same object twice, or a site where adding a library definition is a change nobody wants to make. Then you can reach the same result with a mirror table and two hook scripts.
|
||||||
|
|
||||||
|
The idea is that the thing Data Controller edits is not the real table at all, but an empty table of the same shape, with hooks moving data in and out of it:
|
||||||
|
|
||||||
|
- a `PRE_EDIT_HOOK` fills the editor with the live rows of the real table, so the mirror never has to hold a copy
|
||||||
|
- a `POST_EDIT_HOOK` re-points the submitted changeset at the real table, so the approval is raised against the real table and the load writes there
|
||||||
|
|
||||||
|
The mirror exists purely to carry the rule set, and never stores any data.
|
||||||
|
|
||||||
|
### The pre-edit hook
|
||||||
|
|
||||||
|
`PRE_EDIT_HOOK` runs inside the `getdata` service, after the user's filter has been applied and the rows sorted, with the data in `work.OUT`. It may replace that dataset, which is all this needs:
|
||||||
|
|
||||||
|
```sas
|
||||||
|
data work.out;
|
||||||
|
set ORDERS.ORDERS;
|
||||||
|
run;
|
||||||
|
```
|
||||||
|
|
||||||
|
The registered table is `ORDERS.MIRROR`, which is empty - so without the hook the editor would show nothing at all. With it, the grid shows the live rows:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Note the title bar: the mirror really is empty. Everything on screen came from the hook, and the grid is validated against the mirror's own rules rather than the real table's.
|
||||||
|
|
||||||
|
### The post-edit hook
|
||||||
|
|
||||||
|
This is the part that surprises people. `POST_EDIT_HOOK` runs inside the `mpe_loader` macro at submit time, on the staged rows, before the submit record is written. It cannot choose the target table directly - but at that point `LIBREF` and `DS` are still ordinary macro variables, and the submit record is built from them. Reassigning them re-points the whole changeset:
|
||||||
|
|
||||||
|
```sas
|
||||||
|
data _null_;
|
||||||
|
call symputx('libref','ORDERS');
|
||||||
|
call symputx('ds','ORDERS');
|
||||||
|
run;
|
||||||
|
```
|
||||||
|
|
||||||
|
From that moment the changeset is an approval against `ORDERS.ORDERS`. The approver sees a diff against the real table, the load writes to the real table, and the mirror is never touched.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
### The detail that will bite you
|
||||||
|
|
||||||
|
Use `call symputx`, not `%let`. `LIBREF` and `DS` are not declared `%local` in `mpe_loader`, and the hook is included into that scope - so `call symputx` finds the existing variable and updates it, while a `%let` creates a new variable in the hook's own scope and is silently discarded. The hook runs, the log looks clean, and the changeset goes to the mirror anyway.
|
||||||
|
|
||||||
|
### Other things to watch
|
||||||
|
|
||||||
|
- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap will stop the edit screen with "Table is too big".
|
||||||
|
- The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load.
|
||||||
|
- The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table.
|
||||||
|
- The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical.
|
||||||
|
- At approval time the access checks run against the real table, so editors need `EDIT` on the mirror while approvers need `EDIT` and `APPROVE` on the real table.
|
||||||
|
|
||||||
|
## Which should you use?
|
||||||
|
|
||||||
|
If you can define two librefs over the same data, do that. It is configuration only, it leaves every other behaviour of the editor untouched, and there is nothing to maintain.
|
||||||
|
|
||||||
|
Reach for the hook scripts when the platform will not let you duplicate the library definition, or when you specifically want the rule set to be a property of the application rather than of the data.
|
||||||
|
After Width: | Height: | Size: 62 KiB |
|
After Width: | Height: | Size: 62 KiB |
|
After Width: | Height: | Size: 69 KiB |
@@ -61,7 +61,7 @@ Did you know that, in addition to a regular missing value in SAS (`.`), there ar
|
|||||||
|
|
||||||
These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
|
These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
|
||||||
|
|
||||||
`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
|
||||||
|
|
||||||
There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
|
There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
|
||||||
|
|
||||||
|
|||||||
|
After Width: | Height: | Size: 52 KiB |
@@ -0,0 +1,63 @@
|
|||||||
|
---
|
||||||
|
title: 'Full Table Search: Find Any Value in Any Table'
|
||||||
|
description: Type a value into the search box and Data Controller scans every column for it - no query, no WHERE clause.
|
||||||
|
date: '2026-09-15 17:00:00'
|
||||||
|
author: 'Data Controller'
|
||||||
|
authorLink: https://www.linkedin.com/showcase/data-controller-for-sas
|
||||||
|
tags:
|
||||||
|
- Announcements
|
||||||
|
previewImg: './full-table-search.jpeg'
|
||||||
|
---
|
||||||
|
|
||||||
|
# Full Table Search: Find Any Value in Any Table
|
||||||
|
|
||||||
|
Finding a value in a large table usually means writing a query first: guess which column it lives in, write a WHERE clause, run it, and try again when you guess wrong. Data Controller's **full table search** removes that step. Choose a library and table in the Viewer, type a value into the search box and press Enter - every column in the table is scanned for it, and the matching rows come straight back into the grid.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
The search box sits in the Viewer toolbar, next to a **Numeric** checkbox:
|
||||||
|
|
||||||
|
- **Text search** matches part of a value using the case sensitive SAS® `CONTAINS` operator, so `smith` finds `Goldsmith` but not `Smithson`.
|
||||||
|
- **Numeric search** (tick the box) matches the number exactly against every numeric column in the table.
|
||||||
|
|
||||||
|
Whichever you use, the results are ordinary rows in the Viewer, with the rest of the screen behaving exactly as it does for a normal view.
|
||||||
|
|
||||||
|
## Any database, not just SAS datasets
|
||||||
|
|
||||||
|
The scan runs inside SAS, against whichever libname engine the table is assigned to - so it is not limited to SAS datasets.
|
||||||
|
|
||||||
|
## Built on open source
|
||||||
|
|
||||||
|
This feature, like most of Data Controller, is built on the [SASjs Macro Core](https://github.com/sasjs/core) library, using the [`%mp_searchdata()`](https://core.sasjs.io/mp__searchdata_8sas.html) macro. The macro assembles a single DATA step with one `OR` clause per column - a `CONTAINS` test for character columns, an equality test for numeric ones - and writes out only the matching records. It is MIT licensed, so you can read, test and audit the code that is running against your data.
|
||||||
|
|
||||||
|
A few things worth knowing:
|
||||||
|
|
||||||
|
- The search respects your current filter, Row Level Security and Column Level Security - it scans the view you are entitled to see, not the raw table.
|
||||||
|
- Full table search is available in ViewBoxes as well, so the related tables lined up beside your main grid can be searched the same way.
|
||||||
|
- Nothing is shipped to an external index or search service: the scan happens on your own SAS platform.
|
||||||
|
|
||||||
|
See it in action:
|
||||||
|
|
||||||
|
<iframe title="Full Table Search" width="560" height="315" src="https://vid.4gl.io/videos/embed/qdEv4PP2oiPr5VXwLbN58B" style="border: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe>
|
||||||
|
|
||||||
|
More detail in the [Viewer documentation](https://docs.datacontroller.io/dcu-tableviewer/).
|
||||||
|
|
||||||
|
<!--
|
||||||
|
Source LinkedIn post:
|
||||||
|
|
||||||
|
Data Controller for SAS® has a plethora of features to make Data Discovery easier
|
||||||
|
|
||||||
|
Here we demonstrate "full table search". No need to formulate a query - just type a value and hit enter!
|
||||||
|
|
||||||
|
It works on all databases
|
||||||
|
|
||||||
|
This feature, like most others, is built on our open-source #SASjs library - using the `mp_searchdata()` macro (https://lnkd.in/gxZZq76j).
|
||||||
|
|
||||||
|
#sas #sasapps #sasviya
|
||||||
|
|
||||||
|
video: https://vid.4gl.io/w/qdEv4PP2oiPr5VXwLbN58B
|
||||||
|
-->
|
||||||
|
|
||||||
|
<!-- Image prompt:
|
||||||
|
Flat vector illustration on a dark slate (#314351) background with a faint dot grid: a stylised data table across the lower half (rounded header bar plus eight rows of rounded cells), with the fourth row highlighted in brand green (#90c445). A large magnifying glass with a thick green rim and diagonal handle overlaps the table on the right, and inside its dark lens the highlighted row appears magnified with two white value blocks. Headline "FULL TABLE SEARCH" in bold white uppercase, subheading "any value, every column - no query required" in brand green. 1200x627 landscape, no other text. Composed programmatically with PIL in the Data Controller brand palette rather than generated by an image model.
|
||||||
|
-->
|
||||||
|
After Width: | Height: | Size: 47 KiB |
|
After Width: | Height: | Size: 79 KiB |
|
After Width: | Height: | Size: 80 KiB |
|
After Width: | Height: | Size: 80 KiB |
|
After Width: | Height: | Size: 87 KiB |
|
After Width: | Height: | Size: 68 KiB |
|
After Width: | Height: | Size: 51 KiB |
@@ -0,0 +1,76 @@
|
|||||||
|
---
|
||||||
|
title: 'If: An Ode to Data Control'
|
||||||
|
description: Rudyard Kipling's 'If', reworked for the data governance era - the full text of a poem first published in 2022.
|
||||||
|
date: '2026-09-20 15:00:00'
|
||||||
|
author: 'Allan Bowe'
|
||||||
|
authorLink: https://www.linkedin.com/in/allanbowe/
|
||||||
|
tags:
|
||||||
|
- Announcements
|
||||||
|
previewImg: './if-an-ode-to-data-control.jpeg'
|
||||||
|
---
|
||||||
|
|
||||||
|
# If: An Ode to Data Control
|
||||||
|
|
||||||
|
Rudyard Kipling wrote "If" in 1895 and published it in his 1910 collection Rewards and Fairies. This ode to data control was first published in 2022. Here is the full text.
|
||||||
|
|
||||||
|
```poem
|
||||||
|
If you can keep your data when all about you
|
||||||
|
Are losing theirs and blaming it on you;
|
||||||
|
If you can trust your metrics when analysts doubt you,
|
||||||
|
But make allowance for their doubting too:
|
||||||
|
|
||||||
|
If you can query and not be tired by waiting,
|
||||||
|
Or building models, adjust for outliers,
|
||||||
|
Define ETL modules that are self-validating,
|
||||||
|
And require contracts from data suppliers;
|
||||||
|
|
||||||
|
|
||||||
|
If you can report - and not make KPIs your master;
|
||||||
|
If you can forecast - and not make compliance your aim,
|
||||||
|
If you can make the overnight batch faster
|
||||||
|
And ensure the end results are just the same:
|
||||||
|
|
||||||
|
If you can't bear two versions of truth spoken
|
||||||
|
Produced by silos to make a trap for fools,
|
||||||
|
Or watch systems you gave your life to, broken,
|
||||||
|
And stoop and build 'em up with off-the-shelf tools;
|
||||||
|
|
||||||
|
|
||||||
|
If you would make one lake with all your data
|
||||||
|
And risk it on one vendor's big bang plan,
|
||||||
|
And lose, and start again at invitation-to-tender
|
||||||
|
And salvage from the project, what you can:
|
||||||
|
|
||||||
|
If you can force your flat files and mainframe
|
||||||
|
To serve datamarts when key DBAs retire,
|
||||||
|
And so hold on when stakeholders proclaim
|
||||||
|
Why (oh why) is our Data Quality so dire?
|
||||||
|
|
||||||
|
|
||||||
|
If you can talk with architects & keep your virtue,
|
||||||
|
Or walk with CEOs - nor lose the accounting touch,
|
||||||
|
If neither UTF-8 nor PII can hurt you,
|
||||||
|
If all teams count with you, but none too much:
|
||||||
|
|
||||||
|
If you can keep Data Owners beholden
|
||||||
|
To uploads that are timely and accurate and whole,
|
||||||
|
Yours is the Earth and clean records (golden),
|
||||||
|
And - which is more - you'll have Data Control!
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*If you can't keep your data when all about you are losing theirs - [Data Controller for SAS](https://datacontroller.io) can help. Capture, review and approval for every change, with a full audit trail, on SAS Viya, SAS 9 EBI and SASjs Server.*
|
||||||
|
|
||||||
|
<!--
|
||||||
|
Source LinkedIn post:
|
||||||
|
|
||||||
|
If you can keep your data when all about you
|
||||||
|
are losing theirs, and blaming it on you...
|
||||||
|
|
||||||
|
#datagovernance #dataquality #masterdatamanagement
|
||||||
|
-->
|
||||||
|
|
||||||
|
<!-- Image prompt:
|
||||||
|
Classic book title-page cover on a dark slate (#314351) background with a faint vignette and dot grid. A centred cream paper card with a drop shadow, double rule and green spine hint holds the title page: "FIRST PUBLISHED 2022" in small caps, green ornament rules, "IF" in large bold serif, "An Ode to Data Control" in serif, "after Rudyard Kipling" in italic, and the closing couplet "And - which is more - you'll have Data Control!" above a green rule. 1200x627 landscape, composed programmatically with PIL in the Data Controller brand palette rather than generated by an image model.
|
||||||
|
-->
|
||||||
|
After Width: | Height: | Size: 148 KiB |
@@ -0,0 +1,155 @@
|
|||||||
|
---
|
||||||
|
title: '28 Ways to Be Missing in SAS'
|
||||||
|
description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has supported all of them since v4. How they work, and what the validation rules do with them.
|
||||||
|
date: '2026-09-22 09:00:00'
|
||||||
|
author: 'Allan Bowe'
|
||||||
|
authorLink: https://www.linkedin.com/in/allanbowe/
|
||||||
|
tags:
|
||||||
|
- Special Missings
|
||||||
|
- Data Quality
|
||||||
|
previewImg: './cover.jpeg'
|
||||||
|
---
|
||||||
|
|
||||||
|
# 28 Ways to Be Missing in SAS
|
||||||
|
|
||||||
|
A SAS numeric missing is not a lone wolf: the ordinary missing (`.`) is one of **28** distinct missing values a numeric variable can hold. The other 27 are written with a single character - the letters `A` to `Z`, or an underscore (`._`).
|
||||||
|
|
||||||
|
They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well-run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
|
||||||
|
|
||||||
|
## They are numbers, not text
|
||||||
|
|
||||||
|
- Arithmetic on them yields missing - `.A + 1` is `.`
|
||||||
|
- `PROC MEANS`, `PROC SUMMARY` and the other summarisation procedures exclude them, exactly as they exclude `.`
|
||||||
|
- They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
|
||||||
|
- `NMISS()` and `CMISS()` count them as missing
|
||||||
|
- Converting one to text drops the period - `CATS(.A)` is the string `A`
|
||||||
|
|
||||||
|
One display detail is worth knowing: a regular missing prints as `.`, unless `OPTIONS MISSING=` changes that character - set it to blank and a regular missing renders as an empty cell. The option affects only the regular missing; `._` and `.A`-`.Z` always print as their own letter. So a blank cell in a SAS listing is still unambiguously a regular missing, and a lone letter is still a special one. Data Controller itself is unaffected either way - it sends a regular missing to the browser as `null` and a special missing as its letter.
|
||||||
|
|
||||||
|
In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type the letter or the underscore, with or without that period - `.a` and `a` are the same missing - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
|
||||||
|
|
||||||
|
There is one cell where the letter cannot be typed at all. A numeric column that carries a date, datetime or time format is edited through a date picker rather than the numeric editor, and a picker accepts only a date.
|
||||||
|
|
||||||
|
## Carrying them between the browser and SAS
|
||||||
|
|
||||||
|
The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
|
||||||
|
|
||||||
|
- The adapter infers each column's SAS type from the values it is given. All numeric values mean a numeric column, all strings mean a character column, and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
|
||||||
|
- `null` becomes `.` or an empty string, according to the type derived for that column.
|
||||||
|
- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
|
||||||
|
- A value that is neither a number nor a single valid character is refused rather than guessed - `aaaa`, or `!` in a numeric column.
|
||||||
|
- A lone `.` is accepted as another way of typing the regular missing.
|
||||||
|
|
||||||
|
There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker.
|
||||||
|
|
||||||
|
Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
|
||||||
|
|
||||||
|
## What the Data Controller validation rules do with them
|
||||||
|
|
||||||
|
These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
|
||||||
|
|
||||||
|
- `NOTNULL` - rejects one. A special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it as well. A primary key column is treated as NOT NULL whether or not a rule is configured for it
|
||||||
|
- `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
|
||||||
|
- `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
|
||||||
|
- `SOFTSELECT` / `HARDSELECT` - both support them. The dropdown lists a special missing as a bare letter alongside the ordinary values, and a hard rule then accepts it like any other listed value - it still rejects a value that is not in the list
|
||||||
|
- `ROUND` - no effect. It only rounds values that are numbers, so a special missing is left as it was typed
|
||||||
|
|
||||||
|
The range rules compare in the order SAS itself uses, which is what lets a range be written in special missings. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. A range therefore means exactly what SAS would mean by it:
|
||||||
|
|
||||||
|
- `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails that floor because it sorts below `.A`
|
||||||
|
- a number sits above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`
|
||||||
|
- against a numeric bound the ordering does the obvious thing: a missing sorts below every number, so it fails a `MINVAL` of 1 and passes a `MAXVAL` of 100. Use `NOTNULL` if the column has to be populated
|
||||||
|
- a rule value that is neither a number nor a special missing - a typo such as `..` or `AB` - satisfies nothing, so the column fails until the rule is corrected
|
||||||
|
|
||||||
|
A formula is the one case where a special missing genuinely does not work:
|
||||||
|
|
||||||
|
- `HARDFORMULA` / `SOFTFORMULA` - a formula that reads a special-missing cell does not compute; the grid's spreadsheet engine returns `#VALUE!` for that row, where the plain `.` contributes 0
|
||||||
|
|
||||||
|
`CASE` (`UPCASE` / `LOWCASE`) is a character rule, so it does not apply: SAS hands the browser a special missing as an uppercase letter, so there is no case to enforce, and a `CASE` rule on a numeric column would reject the column's numbers rather than the missing.
|
||||||
|
|
||||||
|
Worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
|
||||||
|
|
||||||
|
In short, a special missing counts as a value for the pattern and dropdown rules, and takes its own place in the order for the range rules: it sits below every number, so a numeric minimum rejects it and a numeric maximum accepts it, while a range written in special missings is decided among the missing values themselves.
|
||||||
|
|
||||||
|
## See it in action
|
||||||
|
|
||||||
|
The recording below runs the whole cycle on one table: entering special missings, the rules that reject them, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on a date column.
|
||||||
|
|
||||||
|
It also shows the range rules doing what the section above describes. A special missing is refused by a numeric `MINVAL` and accepted by a numeric `MAXVAL`, and on a column carrying `MINVAL .A` with `MAXVAL .C`, `.B` is taken while `.D` is refused.
|
||||||
|
|
||||||
|
<div style="position: relative; padding-top: 56.25%; margin-bottom: 2rem;"><iframe title="Special Missings in Data Controller" width="100%" height="100%" src="https://vid.4gl.io/videos/embed/9UQZzCNBU3zPNQYxdzyV3A?peertubeLink=0" style="border: 0px; position: absolute; inset: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>
|
||||||
|
|
||||||
|
<!--
|
||||||
|
LinkedIn version of this post - publish it with the "Special Missings in Data
|
||||||
|
Controller" video attached. Kept in sync with the copy above: same points, same
|
||||||
|
claims, same order.
|
||||||
|
|
||||||
|
A SAS numeric missing is not a lone wolf. There are 28 of them.
|
||||||
|
|
||||||
|
The ordinary missing (.) is the best known of the 28. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
|
||||||
|
|
||||||
|
They are real numeric values, not text:
|
||||||
|
|
||||||
|
- .A + 1 is .
|
||||||
|
- PROC MEANS excludes them, like any other missing
|
||||||
|
- they sort below every number: ._ then . then .A to .Z
|
||||||
|
- NMISS() and CMISS() count them as missing
|
||||||
|
- a regular missing prints as . unless OPTIONS MISSING= changes it - and that option never touches a special missing
|
||||||
|
|
||||||
|
The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
|
||||||
|
|
||||||
|
We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
|
||||||
|
|
||||||
|
- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
|
||||||
|
- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
|
||||||
|
- you type the letter or the underscore, with or without the period (a, .a, _, ._), and case does not matter
|
||||||
|
- one exception: a date, datetime or time formatted numeric column edits through a date picker, which takes only a date
|
||||||
|
|
||||||
|
How they behave in Data Controller's validation rules:
|
||||||
|
|
||||||
|
- NOTNULL rejects one: a special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it too. A primary key column is NOT NULL whether or not a rule is configured
|
||||||
|
- HARDREGEX checks it against the pattern, unlike blanks and plain .
|
||||||
|
- SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
|
||||||
|
- SOFTSELECT and HARDSELECT both support them - the dropdown lists the missing as a bare letter, and a hard rule accepts it like any other listed value
|
||||||
|
- ROUND leaves it alone, because it only rounds numbers
|
||||||
|
- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
|
||||||
|
|
||||||
|
The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
|
||||||
|
|
||||||
|
In short: a value for the pattern and dropdown rules, and its own place in the order for the range rules - below every number, so a numeric minimum rejects it and a numeric maximum accepts it.
|
||||||
|
|
||||||
|
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF, including a range written in special missings.
|
||||||
|
|
||||||
|
#sas #datacapture #mdm #dataquality
|
||||||
|
|
||||||
|
video: https://vid.4gl.io/w/9UQZzCNBU3zPNQYxdzyV3A
|
||||||
|
|
||||||
|
Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
|
||||||
|
dozen or more, spread wide and moving together across a vast snow plain at dusk,
|
||||||
|
seen from a low wide angle. One wolf stands apart from the group on the left of
|
||||||
|
frame, turned back toward the pack. The pack is rendered in cool blue-grey and
|
||||||
|
silver; the lone wolf catches the only warm light in the scene, a single low
|
||||||
|
amber sun. Overcast dusk sky, faint falling snow, long soft shadows, generous
|
||||||
|
empty sky and snow to the upper third so the wide crop breathes. Painterly
|
||||||
|
digital illustration, muted desaturated palette, soft depth of field with the
|
||||||
|
distant wolves falling out of focus, no text, no letters, no numbers, no logos,
|
||||||
|
no watermark.
|
||||||
|
|
||||||
|
Constraints: no text, no letters, no numbers, no digits, no symbols, no
|
||||||
|
captions, no logos, no watermark, no signature, no border, no frame, no collar,
|
||||||
|
no harness, no humans, no buildings, and do not ask for an exact head count of
|
||||||
|
28 - a crowded pack reads worse than a dozen clear animals, and the number
|
||||||
|
belongs in the headline, not the artwork. The no-text rule matters more than
|
||||||
|
usual here: the subject is letters standing in for numbers, so a stray glyph
|
||||||
|
anywhere undercuts the cover.
|
||||||
|
|
||||||
|
Output: save as ./cover.jpeg at 1.91:1 (1200x627, matching the other feed
|
||||||
|
covers and doubling as the LinkedIn share card), and set
|
||||||
|
previewImg: './cover.jpeg' in the front matter above. Do not also embed the
|
||||||
|
image in the markdown body. Full prompt, variants and rationale:
|
||||||
|
https://paste.4gl.io/?513711f5bde3610e#9LZCA9xzQHDy3CycXmsWU5AoVNtjBZmPotUHHtHTAerF
|
||||||
|
|
||||||
|
Fallback variant if the pack composition comes back muddled: a single wolf
|
||||||
|
standing alone on a snow plain at dusk, its shadow stretching toward a distant
|
||||||
|
pack reduced to small silhouettes on the horizon.
|
||||||
|
-->
|
||||||
@@ -21,11 +21,6 @@ const Footer = () => (
|
|||||||
on <StyledAnchor href="https://sasapps.io">SAS Web Apps</StyledAnchor>
|
on <StyledAnchor href="https://sasapps.io">SAS Web Apps</StyledAnchor>
|
||||||
.
|
.
|
||||||
</StyledDesc>
|
</StyledDesc>
|
||||||
<StyledDesc>
|
|
||||||
<StyledAnchor href="https://sasapps.io/cyber-essentials-certified/">
|
|
||||||
Cyber Essentials Certified
|
|
||||||
</StyledAnchor>
|
|
||||||
</StyledDesc>
|
|
||||||
</div>
|
</div>
|
||||||
<div className="col-md-3">
|
<div className="col-md-3">
|
||||||
<StyledHeading>Source Code</StyledHeading>
|
<StyledHeading>Source Code</StyledHeading>
|
||||||
|
|||||||
@@ -1,8 +1,7 @@
|
|||||||
import React from 'react'
|
import React from 'react'
|
||||||
import { Link } from 'gatsby'
|
import { Link } from 'gatsby'
|
||||||
import { FaShieldAlt } from 'react-icons/fa'
|
|
||||||
|
|
||||||
import { Hero, HeroHeading, HeroDesc, HeroBadge } from './style'
|
import { Hero, HeroHeading, HeroDesc } from './style'
|
||||||
import { BottomSectionArrow, OutlineButton } from '../shared/styledComponents'
|
import { BottomSectionArrow, OutlineButton } from '../shared/styledComponents'
|
||||||
import { Container } from '../shared'
|
import { Container } from '../shared'
|
||||||
|
|
||||||
@@ -20,18 +19,9 @@ const HeroSection: React.FC<HeroSectionProps> = ({
|
|||||||
<HeroHeading>{heading}</HeroHeading>
|
<HeroHeading>{heading}</HeroHeading>
|
||||||
<HeroDesc>{desc}</HeroDesc>
|
<HeroDesc>{desc}</HeroDesc>
|
||||||
{location.pathname === pathPrefix + '/' && (
|
{location.pathname === pathPrefix + '/' && (
|
||||||
<>
|
<Link to="/contact/">
|
||||||
<HeroBadge
|
<OutlineButton>Try Data Controller</OutlineButton>
|
||||||
href="https://sasapps.io/cyber-essentials-certified/"
|
</Link>
|
||||||
target="_blank"
|
|
||||||
rel="noopener"
|
|
||||||
>
|
|
||||||
<FaShieldAlt /> Cyber Essentials Certified
|
|
||||||
</HeroBadge>
|
|
||||||
<Link to="/contact/">
|
|
||||||
<OutlineButton>Try Data Controller</OutlineButton>
|
|
||||||
</Link>
|
|
||||||
</>
|
|
||||||
)}
|
)}
|
||||||
</Container>
|
</Container>
|
||||||
<BottomSectionArrow />
|
<BottomSectionArrow />
|
||||||
|
|||||||
@@ -17,20 +17,3 @@ export const HeroHeading = styled.h1`
|
|||||||
export const HeroDesc = styled.p`
|
export const HeroDesc = styled.p`
|
||||||
opacity: 0.8;
|
opacity: 0.8;
|
||||||
`
|
`
|
||||||
|
|
||||||
export const HeroBadge = styled.a`
|
|
||||||
display: block;
|
|
||||||
width: fit-content;
|
|
||||||
margin: 0.75rem 0 0;
|
|
||||||
padding: 0.375rem 1rem;
|
|
||||||
font-size: 0.75rem;
|
|
||||||
border: 2px solid rgba(255, 255, 255, 0.5);
|
|
||||||
border-radius: 0.25rem;
|
|
||||||
color: white;
|
|
||||||
text-decoration: none;
|
|
||||||
&:hover {
|
|
||||||
color: white;
|
|
||||||
border-color: white;
|
|
||||||
opacity: 0.9;
|
|
||||||
}
|
|
||||||
`
|
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
import React from 'react'
|
||||||
|
import { FaShieldAlt } from 'react-icons/fa'
|
||||||
|
import styled from 'styled-components'
|
||||||
|
|
||||||
|
const StyledCertBadge = styled.a`
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.5rem;
|
||||||
|
padding: 0.375rem 1rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
border: 2px solid #314351;
|
||||||
|
border-radius: 0.25rem;
|
||||||
|
color: #314351;
|
||||||
|
text-decoration: none;
|
||||||
|
&:hover {
|
||||||
|
color: white;
|
||||||
|
background-color: #314351;
|
||||||
|
}
|
||||||
|
`
|
||||||
|
|
||||||
|
const CertBadge = () => (
|
||||||
|
<StyledCertBadge
|
||||||
|
href="https://sasapps.io/cyber-essentials-certified/"
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener"
|
||||||
|
>
|
||||||
|
<FaShieldAlt /> Cyber Essentials Certified
|
||||||
|
</StyledCertBadge>
|
||||||
|
)
|
||||||
|
|
||||||
|
export default CertBadge
|
||||||
@@ -5,6 +5,7 @@ import Layout from '../components/layout'
|
|||||||
import Seo from '../components/seo'
|
import Seo from '../components/seo'
|
||||||
|
|
||||||
import { Section } from '../components/shared'
|
import { Section } from '../components/shared'
|
||||||
|
import CertBadge from '../components/shared/certBadge'
|
||||||
import {
|
import {
|
||||||
SectionHeading,
|
SectionHeading,
|
||||||
SectionDesc
|
SectionDesc
|
||||||
@@ -37,6 +38,7 @@ const About: React.FC<PageProps<unknown>> = ({ location }) => {
|
|||||||
</a>
|
</a>
|
||||||
.
|
.
|
||||||
</SectionDesc>
|
</SectionDesc>
|
||||||
|
<CertBadge />
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</Section>
|
</Section>
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import Layout from '../components/layout'
|
|||||||
import Seo from '../components/seo'
|
import Seo from '../components/seo'
|
||||||
|
|
||||||
import { Section, ScheduleDemo } from '../components/shared'
|
import { Section, ScheduleDemo } from '../components/shared'
|
||||||
|
import CertBadge from '../components/shared/certBadge'
|
||||||
import {
|
import {
|
||||||
SectionHeading,
|
SectionHeading,
|
||||||
SectionDesc
|
SectionDesc
|
||||||
@@ -98,6 +99,7 @@ const Home: React.FC<PageProps<IndexPageData>> = ({ data, location }) => {
|
|||||||
perform manual data uploads into their preferred database, in
|
perform manual data uploads into their preferred database, in
|
||||||
real-time, with full validation, approval, security, and control.
|
real-time, with full validation, approval, security, and control.
|
||||||
</SectionDesc>
|
</SectionDesc>
|
||||||
|
<CertBadge />
|
||||||
</div>
|
</div>
|
||||||
<div className="col-md-3">
|
<div className="col-md-3">
|
||||||
<Art src={rightArt} info="Clinical Research Data" />
|
<Art src={rightArt} info="Clinical Research Data" />
|
||||||
|
|||||||
@@ -118,6 +118,22 @@ const StyledContent = styled.div`
|
|||||||
line-height: 2em;
|
line-height: 2em;
|
||||||
font-family: Monaco, 'Andale Mono', 'Courier New', Courier, monospace;
|
font-family: Monaco, 'Andale Mono', 'Courier New', Courier, monospace;
|
||||||
}
|
}
|
||||||
|
/* A fenced block tagged with the 'poem' language is verse, not code: drop
|
||||||
|
the code chrome and set it in the serif face, keeping the author's line
|
||||||
|
breaks and blank lines (which read as stanza breaks). */
|
||||||
|
pre:has(> code.language-poem) {
|
||||||
|
background-image: none;
|
||||||
|
border: none;
|
||||||
|
padding: 0;
|
||||||
|
margin: 1.8rem 0;
|
||||||
|
line-height: 1.75;
|
||||||
|
}
|
||||||
|
pre > code.language-poem {
|
||||||
|
font-family: Georgia, 'Times New Roman', serif;
|
||||||
|
font-size: 1.05rem;
|
||||||
|
color: #222222;
|
||||||
|
white-space: pre-wrap;
|
||||||
|
}
|
||||||
`
|
`
|
||||||
|
|
||||||
interface PostProps {
|
interface PostProps {
|
||||||
|
|||||||