Second review follow-up.
Audit gate: a flat `npm audit --audit-level=<x>` cannot work here at any
level - the tree reports 102 advisories (12 low, 38 moderate, 52 high) and
`--omit=dev` does not change the counts, because gatsby itself is a runtime
dependency of the build. Gating on the tree as a whole stops publishing
permanently; ignoring it lets a new advisory through. scripts/audit-gate.mjs
fails on any critical/high advisory that is not in scripts/audit-baseline.json,
which carries an owner and a reason, so today's set is a tracked exception and
anything new blocks the deploy. Verified both ways: exit 0 on the committed
baseline, exit 1 with an entry removed.
Lint and typecheck: the deleted GitHub workflow was the only place they ran, so
`npm run typecheck` and `npm run lint` are now steps in the publish workflow,
ahead of the build. Lint could not simply be switched on - prettier reported 8
pre-existing files on main, all single-line indentation - so those are fixed
here and the check is clean rather than baselined.
Verified locally in the order the workflow runs: audit gate 0, typecheck clean,
lint clean (8 files reformatted, indentation only), build 136 pages.
Review follow-up.
Lockfile: the previous commit pruned the surfer subtree by hand and left
orphaned nested entries behind (node_modules/cloudron-surfer/node_modules/*,
plus children of removed parents such as @cloudron/pankow's filesize and
express-session's cookie). Regenerated from the unmodified main lockfile with
`npm install --package-lock-only --legacy-peer-deps`, so the lock is npm's own
output rather than an edit. No dangling entries remain.
Publish workflow: added a blocking `npm audit --audit-level=critical` step
between install and build. Critical rather than the suggested default/low: the
tree carries 102 advisories at low/moderate/high that have no non-breaking fix
(mostly the Gatsby 5 build toolchain), so a low gate would fail every deploy
from the first run. The tree is now clear of criticals, having bumped the two
that were live, both within their parents' existing ranges:
- proxy-addr 2.0.7 -> 2.0.8 (GHSA-jqcg-44mw-7w3h, via express ~2.0.7)
- shell-quote 1.8.1 -> 1.12.0 (GHSA-w7jw-789q-3m8p, GHSA-395f-4hp3-45gv, via
react-dev-utils ^1.7.3)
.pre-commit-config.yaml: gitleaks hook pinned to v8.30.1 for staged-content
secret scanning. Developer-side only - `pre-commit install` per clone.
Verified: `npm ci --legacy-peer-deps` clean against the regenerated lock,
`npm audit --audit-level=critical` exits 0, build 136 pages, typecheck clean,
lint unchanged at the 8 pre-existing files.
The publish workflow now deploys with `npx --yes @cloudron/surfer@7.2.0`
(pinned) and the v6 `put` command it replaced no longer exists, so
`cloudron-surfer@6` in devDependencies was dead weight in every `npm ci`.
Removing it also drops 35 packages that only existed for its bundled admin
UI (monaco-editor, three, openid-client, ejs, webdav-server, ...). Only those
entries are removed from package-lock.json - no re-resolution, so the diff is
pure deletion.
`.github/workflows/publish.yml` is the pre-Cloudron deploy path: it pushes the
build to a `gh-pages` branch with a third-party action and a `DCSITE` secret.
The site is published by surfer from `.gitea/workflows/publish.yml`, so this
file describes a deploy that cannot run and only misleads a reader looking for
how the site goes live.
Verified locally: `npm ci --legacy-peer-deps` (lock still in sync),
`npm run build` (136 pages), `npm run typecheck` clean, `npm run lint` reports
the same 8 pre-existing files as main and no new ones.
The publish job used `npx surfer put --token` - the unscoped package, which
resolves to 0.0.2, and the v6 token flow, which Surfer no longer accepts. Move
to @cloudron/surfer 7.2.0, pinned: username allan plus the per-app password in
the SURFER_PASSWORD secret, app datacontroller.io.
`deploy` rather than `put`: it tars the build, extracts to a staging folder and
swaps it in, so a failed upload leaves the current site serving, and it records
the build's ref and commit in the site history. Also adds workflow_dispatch, so
a deploy can be run by hand without pushing to main.
The cover is the same capture as the article's grid still - 1024 CSS px at 2x,
clipped to the rows - so the derived columns are legible on the feed card rather
than a thumbnail of a mostly-empty page. It also stops promising the next
release, which is about to be stale.
Full-viewport 1920x900 captures leave most of the frame as empty grid canvas and
shrink the UI to a third of its CSS size when the post renders them at column
width - the thing is unreadable on a tablet. Re-taken at 1024 CSS px, captured
at 2x device pixels for retina, and clipped to just below the data rows.
Availability also stops promising the next release.
The feed post had grown to 800 words and retold everything the blog article
already covers. Feed is for short announcements, so this is now 180 words: the
request every reporting team makes, the computed-against-locked marking, one
line on why a derived column can be filtered, then a link to the article and the
docs. The inline screenshot goes with the detail, leaving the cover.
The opening was a second-hand request; it reads better as the request every
reporting team has. The post also ended on the docs link with nothing to do
next, so it closes with the same invitation the other recent posts use.
The four screenshots were taken on the mocked estate before the EDIT grid
marked derived columns apart from ordinary read-only ones. Re-taken against a
real SAS backend, so they show what ships: the fx badge, the tinted header and
cells, and the edge that opens the computed block.
The read-only sentence now says what the marking means, which is the one claim
the old images made ambiguously.
A short, opinionated post for the feed: the request that always sounds like five
minutes, the marking that says a column is computed rather than locked, and the
DATA step view that makes a derived column filterable. Links to the docs page
and to a discovery call.
Screenshots are from a real SAS backend, so the grid shown is the shipped one -
fx badge, tinted cells, and the edge that opens the computed block.
The feature post: why a computed column is not filterable in a single DATA
step, the view that makes it so, the MPE_DERIVED configuration, dependency
ordering by VAR_DEPS, and what the feature will not do. Three screenshots from
the EDIT screen - the grid, a filter on a character derived column, and a
filter on a numeric one.
The post has moved to the docs site as a Developer Guide page:
https://docs.datacontroller.io/developer-guide/two-rule-sets-one-table/
- content/blog/two-rule-sets-one-table/ is removed (the images moved to the
docs repo, where the page now lives).
- static/two-rule-sets-one-table/index.html keeps the old URL working with a
client-side redirect to the new page, plus a canonical link and noindex.
A static file at that path is served the same way the removed Gatsby page
was (both are index.html under a directory).
mp_lockanytable keys its control table on (lock_lib, lock_ds), and the
approve service takes that lock on the base table named in the submit
record. The MPE_DATALOADS staleness check is keyed the same way. So two
registrations of one physical file neither lock nor observe each other,
and a second approval can overwrite the first silently.
Spell that out, note that the hook route routes both mirrors to one base
table and so does not have the gap, and give the two ways to close it.
The LinkedIn opener did not parse - "A SAS numeric missing is not a lone wolf.
It is 28." - and reading the rest of the post and its LinkedIn version together
turned up a further set of style defects:
- the article opener made the missing value the wolf, and padded the count
- "All numeric values means numeric; all strings means character" was clumsy
- procedure and function names were lowercase in places (proc means, cats,
options MISSING), and "and friends" was too casual for a list of procedures
- "gotcha" and "harmless" undersold the points they introduced
- "One thing worth knowing" opened two separate paragraphs
- "a value ... and as a value with its own place" repeated itself, in both the
article and the LinkedIn version
- the adapter bullet list mixed trailing periods with none
- the LinkedIn list was split in two by a stray blank line, and "a physical
constraint" did not say which constraint
- "the most common of them" echoed the "28 of them" in the line above
No claims changed - the range-rule behaviour, the period, the primary key and
the strict dropdown all read as they did.
Embeds the "Special Missings in Data Controller" recording from vid.4gl.io as a
responsive PeerTube iframe, in place of the older YouTube embed, and drops the
stray `video: [...]` line that was sitting in the body as literal text.
Copy corrections, each verified against the running app rather than the source:
- the range rules compare in SAS's order, so MINVAL .A with MAXVAL .C takes .B
and refuses .D, a missing fails a numeric MINVAL and passes a numeric MAXVAL,
and a number sits above every missing. The old wording said the range rules
step aside for a missing value
- a lone "." is another way of typing the regular missing; the previous claim
that a literal "." is refused in favour of null is not what happens
- a strict dropdown does accept a special missing when its list holds one (the
list shows it as a bare letter), and still rejects a value that is not listed
- NOTNULL rejects a special missing, and a primary key column is NOT NULL
whether or not a rule is configured for it
The LinkedIn version of the post is kept in sync, and carries the new video URL.
A range rule keys both sides into the order SAS uses for a numeric variable, so a
range written in special missings means what SAS would mean by it: MINVAL .A with
MAXVAL .C accepts .B and rejects .D, a blank fails a floor of .A, and a number
sits above every missing - passing a floor of .A and failing a ceiling of .C.
Against a numeric bound the same order gives the obvious answer: a missing fails
MINVAL 1 and passes MAXVAL 100.
Replaces the "the range rules step aside for a missing value" wording.
Three corrections to the special missings post:
- a special missing is typed with or without its leading period - `.a` and `a`
are the same missing
- MINVAL and MAXVAL both accept a special missing, and both still reject a real
number that is out of range. A minimum constrains a number, and a missing is
not a number; NOTNULL is the rule for a column that must be populated
- the formula rules are the one case where a special missing genuinely does not
work, so the range rules are no longer grouped with them
The validator fix lands in dc/dc (PR #323), so the post describes the fixed
behaviour rather than the mismatch: a special missing fails NOTNULL, matching
a physical SAS NOT NULL / primary key constraint.
Also records what ROUND and SOFTSELECT/HARDSELECT do with one, and groups
MINVAL/MAXVAL/HARDFORMULA/SOFTFORMULA under 'not supported'.
Verified on a real SAS estate:
- a physical NOT NULL (or primary key) constraint rejects a special missing,
and getdata merges that constraint into a frontend NOTNULL rule - which
passes a special missing, so the editor is more lenient than the constraint
- a special missing as a MINVAL rule value fails every cell; as a MAXVAL rule
value it fails every real number
- a HARDFORMULA/SOFTFORMULA reading a special-missing cell returns #VALUE!
- PRX and the JS engine agree on the value; SAS pads the numeric-to-character
conversion, so an anchored pattern re-used in SAS needs strip()
- heading + intro now say these are Data Controller's MPE_VALIDATIONS rules,
applied in the browser
- new paragraph on options MISSING: a regular missing prints as . unless the
option changes it (eg to blank); special missings are never affected
- CASE split out of the rule list: it is a character rule, and a special
missing always reaches the browser as an uppercase letter
- HARDREGEX and SOFTREGEX separated (SOFTREGEX warns rather than blocks, and
is ignored when the column also has a HARDREGEX)
- LinkedIn copy kept in sync
Behaviour confirmed against the deployed services on a real Viya estate.
Cover art for "28 Ways to Be Missing in SAS": a pack of wolves on a snow plain
at dusk with one animal standing apart, carrying the post's opening line - a
numeric missing is not a lone wolf, there are 28 of them.
Source image was 4:3, so it is cropped to 1.91:1 (1200x627) to match the other
feed covers and double as the LinkedIn share card. The crop was chosen to keep
both the lone wolf on the left and the full pack on the right in frame, losing
only sky above the clouds and foreground snow.
Sets previewImg in the front matter; the template renders it, so it is not
embedded in the body as well. Verified through gatsby build - the image
pipeline emits 300/600/1200-wide variants.
Replaces the placeholder one-liner with the prompt actually intended for the
cover: the pack of wolves on a snow plain at dusk with one animal standing
apart, plus the negative constraints, the output spec (./cover.jpeg at
1200x627, previewImg in front matter) and a fallback single-subject variant.
Two deliberate choices are recorded so they are not lost on a regenerate:
- No text, letters or numbers anywhere in the image. The subject is letters
standing in for numbers, so a stray glyph undercuts the cover.
- No exact head count of 28. Generators cannot count, and a crowded pack reads
worse than a dozen clear animals; the number belongs in the headline.
Links to the full prompt and variants on paste.4gl.io.
The special missings video was labelled "Retain Formulas when Loading Excel
to SAS", which is a different video. Restores the one-line correction that
was in the earlier revision of this branch, kept separate from the feed post.
The special missings material that was added to the v4.0 blog post belongs in
its own post: that article is a release announcement for v4.0, not a teaching
piece. This reverts the blog change and publishes the content under /feed/.
The new post covers what SAS special missings are, how the SASjs Adapter
carries them between the browser and SAS, and how Data Controller's validation
rules treat them - NOTNULL passes, MINVAL fails, MAXVAL passes, and the regex
rules apply as they would to any other value.
The LinkedIn version of the copy, and the image prompt for the cover, are
recorded in a comment at the foot of the file.
The post claimed `smith` finds `Smithson`, which only holds for a case
insensitive match. The comparison is case sensitive, so the search term has
to match the case as stored: `Smith` finds `Smithson`, and `smith` finds
`Goldsmith`, but `smith` will not find `Smithson`.
dc