Author SHA1 Message Date
dc 1ae06572c9 blog: add a post on running two rule sets against one table
Explains why a physical table carries one rule set in Data Controller
(MPE_TABLES and MPE_VALIDATIONS are both keyed on the physical libref.dataset),
then covers the two ways to get more than one:

- two librefs over the same data (recommended - configuration only, no hooks,
  no copies, filtering and every other editor behaviour unchanged)
- an empty mirror plus PRE_EDIT_HOOK (show the live rows) and POST_EDIT_HOOK
  (re-point the changeset at the real table with call symputx, not %let). The
  target table must itself be registered in MPE_TABLES.

Screenshots are from the demo estate.
2026-09-24 15:46:04 +00:00
allan 59ff2b13fc Merge pull request 'feed: add a special missings post' (#19) from blog/special-missings-education into main
publish / Build-and-publish (push) Successful in 4m13s
Reviewed-on: #19
2026-09-23 22:42:31 +00:00
dc 7d950cbdbe feed: copy-edit the special missings post
The LinkedIn opener did not parse - "A SAS numeric missing is not a lone wolf.
It is 28." - and reading the rest of the post and its LinkedIn version together
turned up a further set of style defects:

- the article opener made the missing value the wolf, and padded the count
- "All numeric values means numeric; all strings means character" was clumsy
- procedure and function names were lowercase in places (proc means, cats,
  options MISSING), and "and friends" was too casual for a list of procedures
- "gotcha" and "harmless" undersold the points they introduced
- "One thing worth knowing" opened two separate paragraphs
- "a value ... and as a value with its own place" repeated itself, in both the
  article and the LinkedIn version
- the adapter bullet list mixed trailing periods with none
- the LinkedIn list was split in two by a stray blank line, and "a physical
  constraint" did not say which constraint
- "the most common of them" echoed the "28 of them" in the line above

No claims changed - the range-rule behaviour, the period, the primary key and
the strict dropdown all read as they did.
2026-09-23 22:41:12 +00:00
dc 66d5fecb0e feed: embed the special missings recording, and correct the copy
Embeds the "Special Missings in Data Controller" recording from vid.4gl.io as a
responsive PeerTube iframe, in place of the older YouTube embed, and drops the
stray `video: [...]` line that was sitting in the body as literal text.

Copy corrections, each verified against the running app rather than the source:

- the range rules compare in SAS's order, so MINVAL .A with MAXVAL .C takes .B
  and refuses .D, a missing fails a numeric MINVAL and passes a numeric MAXVAL,
  and a number sits above every missing. The old wording said the range rules
  step aside for a missing value
- a lone "." is another way of typing the regular missing; the previous claim
  that a literal "." is refused in favour of null is not what happens
- a strict dropdown does accept a special missing when its list holds one (the
  list shows it as a bare letter), and still rejects a value that is not listed
- NOTNULL rejects a special missing, and a primary key column is NOT NULL
  whether or not a rule is configured for it

The LinkedIn version of the post is kept in sync, and carries the new video URL.
2026-09-23 22:00:16 +00:00
dc 0b18658686 fix(blog): the range rules compare in SAS's own order
A range rule keys both sides into the order SAS uses for a numeric variable, so a
range written in special missings means what SAS would mean by it: MINVAL .A with
MAXVAL .C accepts .B and rejects .D, a blank fails a floor of .A, and a number
sits above every missing - passing a floor of .A and failing a ceiling of .C.
Against a numeric bound the same order gives the obvious answer: a missing fails
MINVAL 1 and passes MAXVAL 100.

Replaces the "the range rules step aside for a missing value" wording.
2026-09-23 21:27:58 +00:00
dc bbd5793d8b fix(blog): the period is optional, and a range rule ignores a missing
Three corrections to the special missings post:

- a special missing is typed with or without its leading period - `.a` and `a`
  are the same missing
- MINVAL and MAXVAL both accept a special missing, and both still reject a real
  number that is out of range. A minimum constrains a number, and a missing is
  not a number; NOTNULL is the rule for a column that must be populated
- the formula rules are the one case where a special missing genuinely does not
  work, so the range rules are no longer grouped with them
2026-09-23 21:01:16 +00:00
dc 0bda92b091 blog: note the one cell that cannot take a special missing 2026-09-23 15:38:37 +00:00
dc bf9fca1b9a blog: the dropdowns support special missings, simply 2026-09-23 08:37:07 +00:00
dc d542ac442f blog: NOTNULL simply works the same as SAS (no special missings)
Also reflects the HARDSELECT fix: a strict dropdown now accepts a special
missing when the list contains it.
2026-09-23 08:19:34 +00:00
dc 0ee0b4ea7c blog: NOTNULL now rejects special missings; range rules and formulas do not support them
The validator fix lands in dc/dc (PR #323), so the post describes the fixed
behaviour rather than the mismatch: a special missing fails NOTNULL, matching
a physical SAS NOT NULL / primary key constraint.

Also records what ROUND and SOFTSELECT/HARDSELECT do with one, and groups
MINVAL/MAXVAL/HARDFORMULA/SOFTFORMULA under 'not supported'.
2026-09-23 07:48:44 +00:00
dc 2fe1bc2eea blog: add formula, range-value and NOTNULL-constraint caveats to the special missings post
Verified on a real SAS estate:
- a physical NOT NULL (or primary key) constraint rejects a special missing,
  and getdata merges that constraint into a frontend NOTNULL rule - which
  passes a special missing, so the editor is more lenient than the constraint
- a special missing as a MINVAL rule value fails every cell; as a MAXVAL rule
  value it fails every real number
- a HARDFORMULA/SOFTFORMULA reading a special-missing cell returns #VALUE!
- PRX and the JS engine agree on the value; SAS pads the numeric-to-character
  conversion, so an anchored pattern re-used in SAS needs strip()
2026-09-23 07:29:04 +00:00
dc 3e39339332 blog: scope the validation section to Data Controller rules and note the MISSING= gotcha
- heading + intro now say these are Data Controller's MPE_VALIDATIONS rules,
  applied in the browser
- new paragraph on options MISSING: a regular missing prints as . unless the
  option changes it (eg to blank); special missings are never affected
- CASE split out of the rule list: it is a character rule, and a special
  missing always reaches the browser as an uppercase letter
- HARDREGEX and SOFTREGEX separated (SOFTREGEX warns rather than blocks, and
  is ignored when the column also has a HARDREGEX)
- LinkedIn copy kept in sync

Behaviour confirmed against the deployed services on a real Viya estate.
2026-09-23 07:17:46 +00:00
dc 6d554930be docs(feed): add the cover image to the special missings post
Cover art for "28 Ways to Be Missing in SAS": a pack of wolves on a snow plain
at dusk with one animal standing apart, carrying the post's opening line - a
numeric missing is not a lone wolf, there are 28 of them.

Source image was 4:3, so it is cropped to 1.91:1 (1200x627) to match the other
feed covers and double as the LinkedIn share card. The crop was chosen to keep
both the lone wolf on the left and the full pack on the right in frame, losing
only sky above the clouds and foreground snow.

Sets previewImg in the front matter; the template renders it, so it is not
embedded in the body as well. Verified through gatsby build - the image
pipeline emits 300/600/1200-wide variants.
2026-09-22 21:36:39 +00:00
dc a56c2f8d48 docs(feed): record the final cover image prompt on the special missings post
Replaces the placeholder one-liner with the prompt actually intended for the
cover: the pack of wolves on a snow plain at dusk with one animal standing
apart, plus the negative constraints, the output spec (./cover.jpeg at
1200x627, previewImg in front matter) and a fallback single-subject variant.

Two deliberate choices are recorded so they are not lost on a regenerate:

- No text, letters or numbers anywhere in the image. The subject is letters
  standing in for numbers, so a stray glyph undercuts the cover.
- No exact head count of 28. Generators cannot count, and a crowded pack reads
  worse than a dozen clear animals; the number belongs in the headline.

Links to the full prompt and variants on paste.4gl.io.
2026-09-22 21:17:41 +00:00
dc 27ff40b739 blog: correct the video label on the v4.0 special missings post
The special missings video was labelled "Retain Formulas when Loading Excel
to SAS", which is a different video. Restores the one-line correction that
was in the earlier revision of this branch, kept separate from the feed post.
2026-09-22 18:10:28 +00:00
dc 4937edd7f4 feed: add a special missings post, and revert the blog edits
The special missings material that was added to the v4.0 blog post belongs in
its own post: that article is a release announcement for v4.0, not a teaching
piece. This reverts the blog change and publishes the content under /feed/.

The new post covers what SAS special missings are, how the SASjs Adapter
carries them between the browser and SAS, and how Data Controller's validation
rules treat them - NOTNULL passes, MINVAL fails, MAXVAL passes, and the regex
rules apply as they would to any other value.

The LinkedIn version of the copy, and the image prompt for the cover, are
recorded in a comment at the foot of the file.
2026-09-22 17:53:55 +00:00
allan fcba390804 Merge pull request 'feed: render poem fenced blocks as verse, not code' (#18) from feat/poem-format into main
publish / Build-and-publish (push) Successful in 4m13s
Reviewed-on: #18
2026-09-20 19:25:54 +00:00
allan 8b6e3e60bc Merge pull request 'feed: add the carousel assets (PDF + slides) to the poem post folder' (#17) from feat/if-ode-carousel into main
publish / Build-and-publish (push) Successful in 4m3s
Reviewed-on: #17
Reviewed-by: Allan <allan@4gl.io>
2026-09-20 19:25:17 +00:00
blog-dev d244a34fc9 feed: render poem fenced blocks as verse, not code 2026-09-20 19:18:11 +00:00
blog-dev 686a745f0b feed: break the carousel stanza slides between the two quatrains 2026-09-20 19:09:38 +00:00
blog-dev 510947d173 feed: add the LinkedIn carousel (PDF + slides) to the post folder 2026-09-20 18:55:18 +00:00
allan 589a745c5b Merge pull request 'Update content/feed/if-an-ode-to-data-control/index.md' (#16) from allan-patch-1 into main
publish / Build-and-publish (push) Successful in 3m53s
Reviewed-on: #16
2026-09-20 17:39:19 +00:00
allan d04c55c731 Update content/feed/if-an-ode-to-data-control/index.md 2026-09-20 17:39:07 +00:00
allan a6a6abf009 Merge pull request 'feat(feed): If - An Ode to Data Control' (#15) from feat/if-ode-to-data-control into main
publish / Build-and-publish (push) Successful in 4m0s
Reviewed-on: #15
Reviewed-by: Allan <allan@4gl.io>
2026-09-20 17:31:40 +00:00
blog-dev 88368fa31d feed: trim the boast and the repeated first-line echo from the intro 2026-09-20 17:21:19 +00:00
blog-dev fdb1db216d feed: rework the poem cover into a title-page design; note the 2022 publication year 2026-09-20 16:37:23 +00:00
blog-dev 6d833f9654 feat(feed): If - An Ode to Data Control 2026-09-20 15:54:56 +00:00
allan 6bcd0adf81 Merge pull request 'feed: correct the case sensitivity example in the full table search post' (#14) from fix/full-table-search-case-sensitivity into main
publish / Build-and-publish (push) Successful in 3m58s
Reviewed-on: #14
Reviewed-by: Allan <allan@4gl.io>
2026-09-17 07:47:31 +00:00
blog-dev 93348ad604 feed: tighten the case sensitivity note to a single sentence 2026-09-17 07:44:53 +00:00
dc 7561bb0c39 feed: correct the case sensitivity example in the full table search post
The post claimed `smith` finds `Smithson`, which only holds for a case
insensitive match. The comparison is case sensitive, so the search term has
to match the case as stored: `Smith` finds `Smithson`, and `smith` finds
`Goldsmith`, but `smith` will not find `Smithson`.
2026-09-16 23:42:45 +00:00
allan 959a9ffab4 Merge pull request 'Feed post: Full Table Search' (#13) from feat/full-table-search into main
publish / Build-and-publish (push) Successful in 4m34s
Reviewed-on: #13
Reviewed-by: Allan <allan@4gl.io>
2026-09-16 23:22:22 +00:00
blog-dev 2348575737 feed: full table search - new demo video, copy matches the recording
- embed the re-recorded 16:9 demo video (search walkthrough, 39s)
- drop the one million row CAS claim from the description, the body and the
  social copy: the recording demonstrates a table in the Viewer, so the
  text now says only what the video shows
2026-09-16 23:17:43 +00:00
blog-dev b2aac21b90 feed: full table search - any value, any column, no query 2026-09-15 17:19:07 +00:00
allan 438b1e93d7 Merge pull request 'Move Cyber Essentials badge to who-is section + about page' (#12) from fix/cert-badge-placement into main
publish / Build-and-publish (push) Successful in 3m56s
Reviewed-on: #12
2026-09-11 20:27:09 +00:00
blog-dev 674737f840 fix: move Cyber Essentials badge to who-is section + about page, drop hero/footer placements 2026-09-11 20:22:29 +00:00
hermes 150be89246 Merge pull request 'Surface Cyber Essentials certification on homepage hero and footer' (#11) from feat/cyber-essentials-badge into main
publish / Build-and-publish (push) Successful in 3m57s
2026-09-11 19:45:26 +00:00
26 changed files with 462 additions and 37 deletions
Binary file not shown.

After

Width:  |  Height:  |  Size: 139 KiB

@@ -0,0 +1,112 @@
---
title: 'One Table, Two Rule Sets'
description: 'A single physical table can carry only one set of Data Controller validation rules. Two librefs over the same data - or a pair of hook scripts - give you as many rule sets as you need.'
date: '2026-09-24 15:30:00'
author: 'Data Controller'
authorLink: https://www.linkedin.com/showcase/data-controller-for-sas
tags:
- Data Quality
- Configuration
previewImg: './rule-set-1.png'
---
# One Table, Two Rule Sets
Most Data Controller sites settle into an obvious mapping: one table, one edit screen, one set of validation rules. But that is not always what the business wants. A table of orders might be edited from a finance report that only tolerates small adjustments, and from an operations report where much larger ones are routine. Same table, same columns, same approvers - different rules.
Data Controller's validation rules are configured per table, so this takes a little thought. There are two ways to do it: the one we recommend, and the one to reach for when the first is not available.
## Why one table is one rule set
Two configuration tables decide this.
`MPE_TABLES` is the list of editable tables, and its primary key is `(tx_from, libref, dsn)`. One physical table is one editable table.
`MPE_VALIDATIONS` holds the rules, and its primary key is `(tx_from, base_lib, base_ds, base_col, rule_type)`. Rules hang off a physical `libref.dataset`. There is no per-menu or per-report scoping anywhere in the schema, and the editor is handed exactly the rules whose `base_lib` and `base_ds` match the table being opened.
So two rule sets on one table need two distinct `libref.dataset` identities. The question is how to get them without copying the data.
## Option 1 (recommended): two librefs over the same data
A libref is just a name pointing at a location. Nothing stops you assigning two of them to the same place, and Data Controller will treat the two as separate tables:
```sas
libname ORDERS_EU '/data/orders';
libname ORDERS_US '/data/orders';
```
`ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` are now the same physical file, but they are different rows in `MPE_TABLES` and can therefore carry different rows in `MPE_VALIDATIONS`. Register both, give each its own rules, and point each report at its own editor URL - `#/editor/ORDERS_EU.ORDERS` and `#/editor/ORDERS_US.ORDERS`.
![The first report rejects an amount of 5000](rule-set-1.png)
![The second report accepts the same value](rule-set-2.png)
Everything else works exactly as it always did. Filtering, search, the row cap, the approval diff and the audit trail all operate on the table as normal, because as far as Data Controller is concerned these are ordinary tables. The only difference is that the two names resolve to the same file, so an approval in either report updates the same data.
There is no copy to keep in sync, no hook to write and nothing to maintain. That is why it is the option we recommend.
### Things worth knowing
- Where the librefs are defined depends on your platform. On Viya, in the compute context's `autoexec.sas` (or `[DC Drive Path]/services/settings.sas`); on SAS 9, as metadata libraries or in the Data Controller Settings stored process; on SASjs Server, in `services/public/settings.sas`. The one requirement is that each library has a unique libref.
- `mp_lockanytable` keys on `libref.dataset`, so the two menus do not serialise against each other. Two people editing through different reports at the same moment can therefore collide at the database level. If that matters, add an explicit shared lock in a `PRE_APPROVE_HOOK`.
- The audit trail and approval queue record which libref a change came through, so `ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` stay distinguishable in history. For most people that is a feature - you can see which report a change originated from.
## Option 2: an empty mirror and a pair of hook scripts
Sometimes two librefs over one location are not available: a database library where the platform will not let you define the same object twice, or a site where adding a library definition is a change nobody wants to make. Then you can reach the same result with a mirror table and two hook scripts.
The idea is that the thing Data Controller edits is not the real table at all, but an empty table of the same shape, with hooks moving data in and out of it:
- a `PRE_EDIT_HOOK` fills the editor with the live rows of the real table, so the mirror never has to hold a copy
- a `POST_EDIT_HOOK` re-points the submitted changeset at the real table, so the approval is raised against the real table and the load writes there
The mirror exists purely to carry the rule set, and never stores any data.
### The pre-edit hook
`PRE_EDIT_HOOK` runs inside the `getdata` service, after the user's filter has been applied and the rows sorted, with the data in `work.OUT`. It may replace that dataset, which is all this needs:
```sas
data work.out;
set ORDERS.ORDERS;
run;
```
The registered table is `ORDERS.MIRROR`, which is empty - so without the hook the editor would show nothing at all. With it, the grid shows the live rows:
![An empty mirror displaying the live rows of the real table](mirror-live.png)
Note the title bar: the mirror really is empty. Everything on screen came from the hook, and the grid is validated against the mirror's own rules rather than the real table's.
### The post-edit hook
This is the part that surprises people. `POST_EDIT_HOOK` runs inside the `mpe_loader` macro at submit time, on the staged rows, before the submit record is written. It cannot choose the target table directly - but at that point `LIBREF` and `DS` are still ordinary macro variables, and the submit record is built from them. Reassigning them re-points the whole changeset:
```sas
data _null_;
call symputx('libref','ORDERS');
call symputx('ds','ORDERS');
run;
```
From that moment the changeset is an approval against `ORDERS.ORDERS`. The approver sees a diff against the real table, the load writes to the real table, and the mirror is never touched.
![A change submitted against the mirror, raised against the real table](approval-routed.png)
### The detail that will bite you
Use `call symputx`, not `%let`. `LIBREF` and `DS` are not declared `%local` in `mpe_loader`, and the hook is included into that scope - so `call symputx` finds the existing variable and updates it, while a `%let` creates a new variable in the hook's own scope and is silently discarded. The hook runs, the log looks clean, and the changeset goes to the mirror anyway.
### Other things to watch
- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap will stop the edit screen with "Table is too big".
- The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load.
- The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table.
- The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical.
- At approval time the access checks run against the real table, so editors need `EDIT` on the mirror while approvers need `EDIT` and `APPROVE` on the real table.
## Which should you use?
If you can define two librefs over the same data, do that. It is configuration only, it leaves every other behaviour of the editor untouched, and there is nothing to maintain.
Reach for the hook scripts when the platform will not let you duplicate the library definition, or when you specifically want the rule set to be a property of the application rather than of the data.
Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

@@ -61,7 +61,7 @@ Did you know that, in addition to a regular missing value in SAS (`.`), there ar
These values can now be both viewed and edited in Data Controller following an update to the [SASjs Adapter](https://github.com/sasjs/adapter#variable-types).
`video: [Retain Formulas when Loading Excel to SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
`video: [Managing Special Missing Values with Data Controller for SAS](https://www.youtube-nocookie.com/embed/ggrcNr23Jzw)`
There is nothing extra to configure for special SAS numerics - they are simply available by default, for numeric cells.
Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

+63
View File
@@ -0,0 +1,63 @@
---
title: 'Full Table Search: Find Any Value in Any Table'
description: Type a value into the search box and Data Controller scans every column for it - no query, no WHERE clause.
date: '2026-09-15 17:00:00'
author: 'Data Controller'
authorLink: https://www.linkedin.com/showcase/data-controller-for-sas
tags:
- Announcements
previewImg: './full-table-search.jpeg'
---
# Full Table Search: Find Any Value in Any Table
Finding a value in a large table usually means writing a query first: guess which column it lives in, write a WHERE clause, run it, and try again when you guess wrong. Data Controller's **full table search** removes that step. Choose a library and table in the Viewer, type a value into the search box and press Enter - every column in the table is scanned for it, and the matching rows come straight back into the grid.
## How it works
The search box sits in the Viewer toolbar, next to a **Numeric** checkbox:
- **Text search** matches part of a value using the case sensitive SAS® `CONTAINS` operator, so `smith` finds `Goldsmith` but not `Smithson`.
- **Numeric search** (tick the box) matches the number exactly against every numeric column in the table.
Whichever you use, the results are ordinary rows in the Viewer, with the rest of the screen behaving exactly as it does for a normal view.
## Any database, not just SAS datasets
The scan runs inside SAS, against whichever libname engine the table is assigned to - so it is not limited to SAS datasets.
## Built on open source
This feature, like most of Data Controller, is built on the [SASjs Macro Core](https://github.com/sasjs/core) library, using the [`%mp_searchdata()`](https://core.sasjs.io/mp__searchdata_8sas.html) macro. The macro assembles a single DATA step with one `OR` clause per column - a `CONTAINS` test for character columns, an equality test for numeric ones - and writes out only the matching records. It is MIT licensed, so you can read, test and audit the code that is running against your data.
A few things worth knowing:
- The search respects your current filter, Row Level Security and Column Level Security - it scans the view you are entitled to see, not the raw table.
- Full table search is available in ViewBoxes as well, so the related tables lined up beside your main grid can be searched the same way.
- Nothing is shipped to an external index or search service: the scan happens on your own SAS platform.
See it in action:
<iframe title="Full Table Search" width="560" height="315" src="https://vid.4gl.io/videos/embed/qdEv4PP2oiPr5VXwLbN58B" style="border: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe>
More detail in the [Viewer documentation](https://docs.datacontroller.io/dcu-tableviewer/).
<!--
Source LinkedIn post:
Data Controller for SAS® has a plethora of features to make Data Discovery easier
Here we demonstrate "full table search". No need to formulate a query - just type a value and hit enter!
It works on all databases
This feature, like most others, is built on our open-source #SASjs library - using the `mp_searchdata()` macro (https://lnkd.in/gxZZq76j).
#sas #sasapps #sasviya
video: https://vid.4gl.io/w/qdEv4PP2oiPr5VXwLbN58B
-->
<!-- Image prompt:
Flat vector illustration on a dark slate (#314351) background with a faint dot grid: a stylised data table across the lower half (rounded header bar plus eight rows of rounded cells), with the fourth row highlighted in brand green (#90c445). A large magnifying glass with a thick green rim and diagonal handle overlaps the table on the right, and inside its dark lens the highlighted row appears magnified with two white value blocks. Headline "FULL TABLE SEARCH" in bold white uppercase, subheading "any value, every column - no query required" in brand green. 1200x627 landscape, no other text. Composed programmatically with PIL in the Data Controller brand palette rather than generated by an image model.
-->
Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 87 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

@@ -0,0 +1,76 @@
---
title: 'If: An Ode to Data Control'
description: Rudyard Kipling's 'If', reworked for the data governance era - the full text of a poem first published in 2022.
date: '2026-09-20 15:00:00'
author: 'Allan Bowe'
authorLink: https://www.linkedin.com/in/allanbowe/
tags:
- Announcements
previewImg: './if-an-ode-to-data-control.jpeg'
---
# If: An Ode to Data Control
Rudyard Kipling wrote "If" in 1895 and published it in his 1910 collection Rewards and Fairies. This ode to data control was first published in 2022. Here is the full text.
```poem
If you can keep your data when all about you
Are losing theirs and blaming it on you;
If you can trust your metrics when analysts doubt you,
But make allowance for their doubting too:
If you can query and not be tired by waiting,
Or building models, adjust for outliers,
Define ETL modules that are self-validating,
And require contracts from data suppliers;
If you can report - and not make KPIs your master;
If you can forecast - and not make compliance your aim,
If you can make the overnight batch faster
And ensure the end results are just the same:
If you can't bear two versions of truth spoken
Produced by silos to make a trap for fools,
Or watch systems you gave your life to, broken,
And stoop and build 'em up with off-the-shelf tools;
If you would make one lake with all your data
And risk it on one vendor's big bang plan,
And lose, and start again at invitation-to-tender
And salvage from the project, what you can:
If you can force your flat files and mainframe
To serve datamarts when key DBAs retire,
And so hold on when stakeholders proclaim
Why (oh why) is our Data Quality so dire?
If you can talk with architects & keep your virtue,
Or walk with CEOs - nor lose the accounting touch,
If neither UTF-8 nor PII can hurt you,
If all teams count with you, but none too much:
If you can keep Data Owners beholden
To uploads that are timely and accurate and whole,
Yours is the Earth and clean records (golden),
And - which is more - you'll have Data Control!
```
---
*If you can't keep your data when all about you are losing theirs - [Data Controller for SAS](https://datacontroller.io) can help. Capture, review and approval for every change, with a full audit trail, on SAS Viya, SAS 9 EBI and SASjs Server.*
<!--
Source LinkedIn post:
If you can keep your data when all about you
are losing theirs, and blaming it on you...
#datagovernance #dataquality #masterdatamanagement
-->
<!-- Image prompt:
Classic book title-page cover on a dark slate (#314351) background with a faint vignette and dot grid. A centred cream paper card with a drop shadow, double rule and green spine hint holds the title page: "FIRST PUBLISHED 2022" in small caps, green ornament rules, "IF" in large bold serif, "An Ode to Data Control" in serif, "after Rudyard Kipling" in italic, and the closing couplet "And - which is more - you'll have Data Control!" above a green rule. 1200x627 landscape, composed programmatically with PIL in the Data Controller brand palette rather than generated by an image model.
-->
Binary file not shown.

After

Width:  |  Height:  |  Size: 148 KiB

+155
View File
@@ -0,0 +1,155 @@
---
title: '28 Ways to Be Missing in SAS'
description: A SAS numeric missing is not a lone wolf - there are 28 of them, and Data Controller has supported all of them since v4. How they work, and what the validation rules do with them.
date: '2026-09-22 09:00:00'
author: 'Allan Bowe'
authorLink: https://www.linkedin.com/in/allanbowe/
tags:
- Special Missings
- Data Quality
previewImg: './cover.jpeg'
---
# 28 Ways to Be Missing in SAS
A SAS numeric missing is not a lone wolf: the ordinary missing (`.`) is one of **28** distinct missing values a numeric variable can hold. The other 27 are written with a single character - the letters `A` to `Z`, or an underscore (`._`).
They exist because "missing" is usually not the whole story. A survey question that was never reached, a reading that was illegible, a value the respondent refused to give - in a well-run process those are different facts, and a lone `.` throws the difference away. Special missings record *why* the value is missing.
## They are numbers, not text
- Arithmetic on them yields missing - `.A + 1` is `.`
- `PROC MEANS`, `PROC SUMMARY` and the other summarisation procedures exclude them, exactly as they exclude `.`
- They sort below every non-missing number, in the order `._`, then `.`, then `.A` to `.Z`
- `NMISS()` and `CMISS()` count them as missing
- Converting one to text drops the period - `CATS(.A)` is the string `A`
One display detail is worth knowing: a regular missing prints as `.`, unless `OPTIONS MISSING=` changes that character - set it to blank and a regular missing renders as an empty cell. The option affects only the regular missing; `._` and `.A`-`.Z` always print as their own letter. So a blank cell in a SAS listing is still unambiguously a regular missing, and a lone letter is still a special one. Data Controller itself is unaffected either way - it sends a regular missing to the browser as `null` and a special missing as its letter.
In a SAS dataset they are written with a leading period (`.A`, `.B` ... `._`). In Data Controller you type the letter or the underscore, with or without that period - `.a` and `a` are the same missing - and the letter is not case sensitive. Two letters, or a letter mixed with a number, are refused rather than guessed.
There is one cell where the letter cannot be typed at all. A numeric column that carries a date, datetime or time format is edited through a date picker rather than the numeric editor, and a picker accepts only a date.
## Carrying them between the browser and SAS
The Data Controller frontend and the SAS backend exchange data as JSON, and JSON has no way to express a letter as a numeric value - `A` is a string. The conversion is handled in the open source [SASjs Adapter](https://github.com/sasjs/adapter#variable-types):
- The adapter infers each column's SAS type from the values it is given. All numeric values mean a numeric column, all strings mean a character column, and a column holding a single character (`a`-`z`, `_` or `.`) alongside numeric values is numeric, with the lone characters written to SAS as special missings.
- `null` becomes `.` or an empty string, according to the type derived for that column.
- Two cases cannot be inferred from the values alone: a numeric column containing *only* special missings looks like a single character column, and a character column containing only nulls looks numeric. For those, the adapter accepts an explicit format for the column - and Data Controller sends one automatically, because it already knows each column's SAS format from the metadata returned by the backend.
- A value that is neither a number nor a single valid character is refused rather than guessed - `aaaa`, or `!` in a numeric column.
- A lone `.` is accepted as another way of typing the regular missing.
There is nothing to configure. Special missings are available by default, for numeric cells - a date, datetime or time formatted column aside, since those edit through a date picker.
Once in SAS they are ordinary values, so they are what the approval DIFF screen compares, and the DIFF's formatted / unformatted switch shows either the formatted representation or the raw value - useful for confirming exactly which missing was set.
## What the Data Controller validation rules do with them
These are Data Controller's own rules, configured per column in the `MPE_VALIDATIONS` table and applied in the browser as you edit and submit.
- `NOTNULL` - rejects one. A special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it as well. A primary key column is treated as NOT NULL whether or not a rule is configured for it
- `HARDREGEX` - checked against the pattern like any other value; unlike blanks and the plain `.`, special missings are **not** exempt, so a numeric column that carries them needs a pattern which allows for a single letter
- `SOFTREGEX` - the same check, but a failure is only a warning rather than a block, and it is ignored entirely if the column also has a `HARDREGEX` rule
- `SOFTSELECT` / `HARDSELECT` - both support them. The dropdown lists a special missing as a bare letter alongside the ordinary values, and a hard rule then accepts it like any other listed value - it still rejects a value that is not in the list
- `ROUND` - no effect. It only rounds values that are numbers, so a special missing is left as it was typed
The range rules compare in the order SAS itself uses, which is what lets a range be written in special missings. SAS puts every missing below every non-missing value, and orders the missing values among themselves: `._` is the lowest, then the regular missing, then `.A` through `.Z`. A range therefore means exactly what SAS would mean by it:
- `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails that floor because it sorts below `.A`
- a number sits above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`
- against a numeric bound the ordering does the obvious thing: a missing sorts below every number, so it fails a `MINVAL` of 1 and passes a `MAXVAL` of 100. Use `NOTNULL` if the column has to be populated
- a rule value that is neither a number nor a special missing - a typo such as `..` or `AB` - satisfies nothing, so the column fails until the rule is corrected
A formula is the one case where a special missing genuinely does not work:
- `HARDFORMULA` / `SOFTFORMULA` - a formula that reads a special-missing cell does not compute; the grid's spreadsheet engine returns `#VALUE!` for that row, where the plain `.` contributes 0
`CASE` (`UPCASE` / `LOWCASE`) is a character rule, so it does not apply: SAS hands the browser a special missing as an uppercase letter, so there is no case to enforce, and a `CASE` rule on a numeric column would reject the column's numbers rather than the missing.
Worth knowing about the regex rules: although the pattern is written in SAS PRX syntax, the check itself runs in the browser - SAS only parses the pattern (`PRXPARSE`) when the rule is saved - so the two engines can disagree on exotic patterns, and SAS pads a numeric-to-character conversion, so a pattern re-used in SAS needs `strip()` for an anchored match.
In short, a special missing counts as a value for the pattern and dropdown rules, and takes its own place in the order for the range rules: it sits below every number, so a numeric minimum rejects it and a numeric maximum accepts it, while a range written in special missings is decided among the missing values themselves.
## See it in action
The recording below runs the whole cycle on one table: entering special missings, the rules that reject them, submitting the changes, approving them, and reviewing the DIFF - including a change from one special missing to another, and the formatted / unformatted switch on a date column.
It also shows the range rules doing what the section above describes. A special missing is refused by a numeric `MINVAL` and accepted by a numeric `MAXVAL`, and on a column carrying `MINVAL .A` with `MAXVAL .C`, `.B` is taken while `.D` is refused.
<div style="position: relative; padding-top: 56.25%; margin-bottom: 2rem;"><iframe title="Special Missings in Data Controller" width="100%" height="100%" src="https://vid.4gl.io/videos/embed/9UQZzCNBU3zPNQYxdzyV3A?peertubeLink=0" style="border: 0px; position: absolute; inset: 0px;" allow="fullscreen" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>
<!--
LinkedIn version of this post - publish it with the "Special Missings in Data
Controller" video attached. Kept in sync with the copy above: same points, same
claims, same order.
A SAS numeric missing is not a lone wolf. There are 28 of them.
The ordinary missing (.) is the best known of the 28. The other 27 are single characters - the letters A to Z, or an underscore (._) - and they let you record WHY a value is missing: the question was never reached, the reading was illegible, the respondent refused.
They are real numeric values, not text:
- .A + 1 is .
- PROC MEANS excludes them, like any other missing
- they sort below every number: ._ then . then .A to .Z
- NMISS() and CMISS() count them as missing
- a regular missing prints as . unless OPTIONS MISSING= changes it - and that option never touches a special missing
The hard part is every tool that is not SAS. JSON has no way to say "this number is a letter" - A is just a string. So a value that is perfectly legal in a SAS dataset quietly breaks in the browser, in Excel, in an API.
We solved that in the open source SASjs Adapter, and it has been in Data Controller for SAS since v4:
- the adapter infers the column type from the values, so a column of numbers that also holds a lone letter is written to SAS as numeric, with the letter as a special missing
- where the type cannot be inferred - a numeric column holding ONLY special missings - Data Controller passes the column format explicitly
- you type the letter or the underscore, with or without the period (a, .a, _, ._), and case does not matter
- one exception: a date, datetime or time formatted numeric column edits through a date picker, which takes only a date
How they behave in Data Controller's validation rules:
- NOTNULL rejects one: a special missing is a missing value, so it fails the rule, and a physical NOT NULL constraint on the target table rejects it too. A primary key column is NOT NULL whether or not a rule is configured
- HARDREGEX checks it against the pattern, unlike blanks and plain .
- SOFTREGEX warns instead of blocking (and is ignored if the column also has a HARDREGEX)
- SOFTSELECT and HARDSELECT both support them - the dropdown lists the missing as a bare letter, and a hard rule accepts it like any other listed value
- ROUND leaves it alone, because it only rounds numbers
- a HARDFORMULA/SOFTFORMULA that reads a special-missing cell returns #VALUE! rather than a number
The range rules compare in SAS's own order, so a range can be written in special missings: MINVAL .A with MAXVAL .C accepts .B and rejects .D. A missing sorts below every number, so it fails a numeric MINVAL and passes a numeric MAXVAL; a number sits above every missing.
In short: a value for the pattern and dropdown rules, and its own place in the order for the range rules - below every number, so a numeric minimum rejects it and a numeric maximum accepts it.
The video shows the whole cycle - entering them, the rejections, submit, approve, and the DIFF, including a range written in special missings.
#sas #datacapture #mdm #dataquality
video: https://vid.4gl.io/w/9UQZzCNBU3zPNQYxdzyV3A
Image prompt: Cinematic editorial cover illustration: a large pack of wolves, a
dozen or more, spread wide and moving together across a vast snow plain at dusk,
seen from a low wide angle. One wolf stands apart from the group on the left of
frame, turned back toward the pack. The pack is rendered in cool blue-grey and
silver; the lone wolf catches the only warm light in the scene, a single low
amber sun. Overcast dusk sky, faint falling snow, long soft shadows, generous
empty sky and snow to the upper third so the wide crop breathes. Painterly
digital illustration, muted desaturated palette, soft depth of field with the
distant wolves falling out of focus, no text, no letters, no numbers, no logos,
no watermark.
Constraints: no text, no letters, no numbers, no digits, no symbols, no
captions, no logos, no watermark, no signature, no border, no frame, no collar,
no harness, no humans, no buildings, and do not ask for an exact head count of
28 - a crowded pack reads worse than a dozen clear animals, and the number
belongs in the headline, not the artwork. The no-text rule matters more than
usual here: the subject is letters standing in for numbers, so a stray glyph
anywhere undercuts the cover.
Output: save as ./cover.jpeg at 1.91:1 (1200x627, matching the other feed
covers and doubling as the LinkedIn share card), and set
previewImg: './cover.jpeg' in the front matter above. Do not also embed the
image in the markdown body. Full prompt, variants and rationale:
https://paste.4gl.io/?513711f5bde3610e#9LZCA9xzQHDy3CycXmsWU5AoVNtjBZmPotUHHtHTAerF
Fallback variant if the pack composition comes back muddled: a single wolf
standing alone on a snow plain at dusk, its shadow stretching toward a distant
pack reduced to small silhouettes on the horizon.
-->
-5
View File
@@ -21,11 +21,6 @@ const Footer = () => (
on <StyledAnchor href="https://sasapps.io">SAS Web Apps</StyledAnchor>
.
</StyledDesc>
<StyledDesc>
<StyledAnchor href="https://sasapps.io/cyber-essentials-certified/">
Cyber Essentials Certified
</StyledAnchor>
</StyledDesc>
</div>
<div className="col-md-3">
<StyledHeading>Source Code</StyledHeading>
+4 -14
View File
@@ -1,8 +1,7 @@
import React from 'react'
import { Link } from 'gatsby'
import { FaShieldAlt } from 'react-icons/fa'
import { Hero, HeroHeading, HeroDesc, HeroBadge } from './style'
import { Hero, HeroHeading, HeroDesc } from './style'
import { BottomSectionArrow, OutlineButton } from '../shared/styledComponents'
import { Container } from '../shared'
@@ -20,18 +19,9 @@ const HeroSection: React.FC<HeroSectionProps> = ({
<HeroHeading>{heading}</HeroHeading>
<HeroDesc>{desc}</HeroDesc>
{location.pathname === pathPrefix + '/' && (
<>
<HeroBadge
href="https://sasapps.io/cyber-essentials-certified/"
target="_blank"
rel="noopener"
>
<FaShieldAlt /> Cyber Essentials Certified
</HeroBadge>
<Link to="/contact/">
<OutlineButton>Try Data Controller</OutlineButton>
</Link>
</>
<Link to="/contact/">
<OutlineButton>Try Data Controller</OutlineButton>
</Link>
)}
</Container>
<BottomSectionArrow />
-17
View File
@@ -17,20 +17,3 @@ export const HeroHeading = styled.h1`
export const HeroDesc = styled.p`
opacity: 0.8;
`
export const HeroBadge = styled.a`
display: block;
width: fit-content;
margin: 0.75rem 0 0;
padding: 0.375rem 1rem;
font-size: 0.75rem;
border: 2px solid rgba(255, 255, 255, 0.5);
border-radius: 0.25rem;
color: white;
text-decoration: none;
&:hover {
color: white;
border-color: white;
opacity: 0.9;
}
`
+31
View File
@@ -0,0 +1,31 @@
import React from 'react'
import { FaShieldAlt } from 'react-icons/fa'
import styled from 'styled-components'
const StyledCertBadge = styled.a`
display: inline-flex;
align-items: center;
gap: 0.5rem;
padding: 0.375rem 1rem;
font-size: 0.75rem;
border: 2px solid #314351;
border-radius: 0.25rem;
color: #314351;
text-decoration: none;
&:hover {
color: white;
background-color: #314351;
}
`
const CertBadge = () => (
<StyledCertBadge
href="https://sasapps.io/cyber-essentials-certified/"
target="_blank"
rel="noopener"
>
<FaShieldAlt /> Cyber Essentials Certified
</StyledCertBadge>
)
export default CertBadge
+2
View File
@@ -5,6 +5,7 @@ import Layout from '../components/layout'
import Seo from '../components/seo'
import { Section } from '../components/shared'
import CertBadge from '../components/shared/certBadge'
import {
SectionHeading,
SectionDesc
@@ -37,6 +38,7 @@ const About: React.FC<PageProps<unknown>> = ({ location }) => {
</a>
.
</SectionDesc>
<CertBadge />
</div>
</div>
</Section>
+2
View File
@@ -7,6 +7,7 @@ import Layout from '../components/layout'
import Seo from '../components/seo'
import { Section, ScheduleDemo } from '../components/shared'
import CertBadge from '../components/shared/certBadge'
import {
SectionHeading,
SectionDesc
@@ -98,6 +99,7 @@ const Home: React.FC<PageProps<IndexPageData>> = ({ data, location }) => {
perform manual data uploads into their preferred database, in
real-time, with full validation, approval, security, and control.
</SectionDesc>
<CertBadge />
</div>
<div className="col-md-3">
<Art src={rightArt} info="Clinical Research Data" />
+16
View File
@@ -118,6 +118,22 @@ const StyledContent = styled.div`
line-height: 2em;
font-family: Monaco, 'Andale Mono', 'Courier New', Courier, monospace;
}
/* A fenced block tagged with the 'poem' language is verse, not code: drop
the code chrome and set it in the serif face, keeping the author's line
breaks and blank lines (which read as stanza breaks). */
pre:has(> code.language-poem) {
background-image: none;
border: none;
padding: 0;
margin: 1.8rem 0;
line-height: 1.75;
}
pre > code.language-poem {
font-family: Georgia, 'Times New Roman', serif;
font-size: 1.05rem;
color: #222222;
white-space: pre-wrap;
}
`
interface PostProps {