Three screenshots from the EDIT screen: the derived columns in the grid, a
filter on a character derived column, and a filter on a numeric one. Plus a
short 'how it works' explaining that the statements go into a DATA step view
because a WHERE cannot see a variable created in the same step.
Each row lists every input the expression uses, physical columns and derived
variables alike. No sequence number to maintain; statements are emitted in
dependency order and a cycle is rejected.
A derived variable can now be built on one with a lower VARIABLE_ORDER, so
the pages describe the order column, the ordered rule (replacing the
self-contained one) and a chained example.
Review follow-up: /dc-userguide/#editor (the heading is 'Editor'), the
reserved names include VAR_TXFROM/VAR_TXTO/VAR_PROCESSED, and derived
columns are not shown in VIEW or downloads though a saved filter on one
still applies there.
New Configuration page explaining derived (computed) variables: what they
are, how to configure them in MPE_DERIVED, how filtering works, the rules
and limits (expressions only, self-contained, read-only, not stored), and
the data-exposure and performance notes.
New MPE_DERIVED table page, registered in the Table Guide nav.
The clause shown above the grid is the resolved WHERE clause, so a date,
datetime or time filter reads as a SAS literal rather than the internal day or
second count. Note it on the filter page and on the MPE_FILTERSOURCE RAW_VALUE
column, which is where the value is stored.
The pre-edit hook example re-added _____DELETE__THIS__RECORD_____="No" when
it re-read the table, mirroring the data step getdata.sas used to run. That
column is created by the editor itself (it is the Delete? column) and is
never part of the dataset, so the example no longer adds it - and a response
that carries the name is refused by the editor, which the "What the hook
does" bullet now states.
Also notes the reserved name on the MPE_TABLES page, and says in the user
guide that the Delete? column is created by Data Controller rather than
being part of the table.
The post-edit hook section claimed a `%let` "creates a new variable in the
hook's own scope and is silently discarded" while `call symputx` updates the
existing one. That is not how SAS resolves it: LIBREF and DS already exist in
an enclosing symbol table (mpe_loader's local table), and a macro only gets a
new local when the name is absent from every enclosing table, so both forms
assign to the same variable. Reworded to state that, and to keep the practical
advice (call symputx is explicit and does not depend on the enclosing scope
still holding the name).
Raised in review of #16.
The publish workflow installed cloudron-surfer, the mkdocs pip set and
@marp-team/marp-cli with no version pins, so a breaking or compromised upstream
release would ship straight to the live site on the next merge to main.
- pin cloudron-surfer@6.4.1, mkdocs==1.6.1, mkdocs-material==9.7.7,
fontawesome_markdown==0.2.6, mkdocs-redirects==1.2.3 and
@marp-team/marp-cli@4.5.1 (the versions the site builds with today)
- add .npmrc with ignore-scripts=true and save-exact=true; neither npm package
ships an install/preinstall/postinstall hook, so nothing is skipped
- The re-read step now sorts work.out by &sortpk after rebuilding it, matching
getdata (which sorts before the hook runs and does not sort again), so the
grid keeps the order the user first saw. The bullet records why.
- Adds a .pre-commit-config.yaml with the gitleaks hook pinned to v8.30.1, so
staged content is scanned at commit time.
- Adds a "What it looks like" section to the pre-edit hook page: the editor
before (the user's own filter, one quarter of rows) and after (the hook's
widened filter, both quarters), captured from the mocked estate.
- Corrects the re-read step: it now drops &drop_rk like getdata does, and the
accompanying note records that a bitemporal table also needs the latest-record
dedup, so the example no longer claims to mirror getdata exactly while missing
both (raised in review of #16).
Two pages, both aimed at SAS developers rather than end users:
- Filter in a Pre-Edit Hook: what editors/getdata returns (sasparams.FILTER_TEXT
and the query clauses, never FILTER_RK), when PRE_EDIT_HOOK runs relative to
the filter, and a worked example that widens a date filter to the previous
quarter - re-reading the table so the rows match, and keeping Row Level
Security applied to both branches.
- One Table, Two Rule Sets: moved from the datacontroller.io blog, so the
developer-facing material lives in one place. Images moved to docs/img/.
DC_MAXOBS_WEBEDIT 100 -> 250, DC_MAXOBS_WEBVIEW 500 -> 2000, and a new
DC_MAXCELLS_WEBEDIT (200000) which is applied together with the EDIT
observation limit - whichever is reached first.
The docs site only offered Material's light scheme. This adds the dark
scheme and makes it the default.
- palette: two entries, slate first, each with a toggle. Material takes the
first entry as the default, so the site opens dark and a sun icon in the
header switches to light (and a moon switches back).
- dc-brand.css: the dark scheme's greys are derived from a hue variable, so
it is pointed at the brand slate (206deg) rather than Material's neutral
225deg, which gives the whole scheme the same cast as the header.
- dc-brand.css: the brand slate is unreadable on a dark background, so the
dark scheme uses a light tint of it (#9fb5c6) for body links. Green stays
the accent, which keeps link hover visible in both schemes.
Contrast, over the dark background rgb(30, 36, 41): body text 8.8:1, body
links 7.4:1, inline code 7.2:1, accent green 5.6:1, header text 10.2:1. The
light scheme is unchanged.
The docs site rendered as stock Material: indigo header, indigo links and
Open Sans. Neither colour nor type matched the marketing site
(datacontroller.io) or the application itself.
- Palette: the slate (#314351) and green (#79a843) already used by the
marketing navbar, hero and logo accent, wired up via a `custom` palette
and defined in the new docs/dc-brand.css.
- Type: Montserrat for text, matching the marketing site's headings.
- Logo: the application's own dc-logo.svg mark in the header.
The previous `palette:` mapping form (a dict of primary/accent) is no
longer honoured by the Material version this repo builds against, so the
theme fell through to Material's default indigo - the site was not in fact
rendering the White/Amber it asked for. The list form used here is the one
Material currently reads.
On SAS 9 and Viya the app is reached through the platform's job URL, so
work.browser_url_vars carries the platform's parameters (_FILE, _program,
_debug, _webout) alongside DC's. Say so, so a hook author does not read
_FILE as a Data Controller parameter.
The integrity section claimed SAS writes nothing to disk outside the
appLoc. The appLoc is the SAS Folder the stored processes or jobs are
deployed into (metadata), so it is not a disk location at all - the only
physical location the deployment programs write to is the Data Controller
location (dcLoc) configured at first launch.
The adapter sends _debug=128 (not 131) on the Viya web JES path when
runAsTask is enabled, which is what the frontend uses there. Document
that value alongside 131, and note that editors/loadfile is reached
through the multipart file upload, which carries no input tables.
The support diagnostics are now sent only where they can be used: the
startup service and the services that execute customer-provided code
(hook scripts, dynamic cell dropdown programs) - not with every service
call. The page's URL parameters arrive as a new browser_url_vars table,
one row per parameter, so a SAS developer reads a parameter by name
rather than parsing the url string.
Adds /downloads to the Installation section, listing what each release
asset is for and how to verify a download against the SHA256SUMS file
the release pipeline publishes (sha256sum -c, plus certutil for
Windows). The SAS 9 deployment page now links to it wherever it sends
the reader to the releases page.
mkdocs build passes; nav, sitemap and cross-links verified in the
rendered output.
Every service call carries a single-row browser_info input table, readable
by any service or hook script as work.browser_info and dumped to the job log
when debug is on. Document its columns and how to see it, so a support ticket
can be answered from the log without follow-up questions.
MINVAL and MAXVAL now say what a special missing does to them and that the bound
itself may be a special missing, pointing at the Special Missing Values section,
since that is the least obvious thing about them.
Also in that section: the framing sentence covers the range rules' ordering, the
summary sentence is rewritten (it read badly), and the arrow in the ROUND row is
plain ASCII.
A range rule keys both the cell value and its own value into the order SAS uses
for a numeric variable: every missing below every non-missing value, and the
missing values ordered ._ then the regular missing then .A through .Z. So a range
can be written in special missings and mean what SAS means by it - MINVAL .A with
MAXVAL .C accepts .B and rejects .D - and the numeric cases follow from the same
order: a missing fails a numeric MINVAL and passes a numeric MAXVAL, and a number
sits above every missing.
Replaces the "a range rule ignores a missing value" wording, which described the
previous take's behaviour rather than SAS's.
Three corrections to the special missing section:
- the period is optional: a special missing is typed as `a` or `.a`, `_` or `._`
- MINVAL and MAXVAL both accept a missing value, and both still reject a real
number that is out of range. A minimum constrains a number, and a missing is
not a number - NOTNULL is the rule for a column that must be populated
- only NOTNULL and the formulas actually reject a special missing, so the
"not supported" summary is replaced with what does reject it and what does not
- a primary key column rejects a blank and a special missing whether or not
the table carries a constraint or MPE_VALIDATIONS a NOTNULL rule
- HARDSELECT is now checked like any other value: it passes when the
dropdown list contains the special missing
- HARDSELECT drops out of the 'not supported' list
New Special Missing Values section covering NOTNULL, MINVAL/MAXVAL, CASE,
ROUND, the regex rules, the dropdowns and the formula rules, with the
'not supported' set called out. Behaviour verified against a real SAS
estate and the deployed frontend validator.
Follow-up to the previous commit on this branch, both wording fixes.
- The sentence this branch deleted was the page's only description of the row
count when no search is active - and that is the case where the count really
can exceed the rows displayed (viewdata reports count(*) for the filtered
view while the grid stops at DC_MAXOBS_WEBVIEW). State it, instead of leaving
the page silent on the path readers use most.
- "returns the first 500" claims an ordering the search does not apply: the
search is a WHERE clause with no ORDER BY, so which 500 come back is whatever
the engine hands over first. Say "returns 500 of them".
Checked against viewdata.sas and mp_searchdata.sas on main: the search path
passes outobs=DC_MAXOBS_WEBVIEW to mp_searchdata, which caps the result dataset
in the same data step that builds it, and viewdata takes its row count from that
dataset - so for a search the count and the rows are capped together.
frontend.zip now holds the frontend files at their root (dc/dc #147, released in
v7.14.2), so unzipping no longer produces a client/dist folder to navigate into.
Step 2 now says to create the app folder and unzip the archive into it, naming
the file that should end up there. Step 3's URL example matches the folder the
reader created, and the intro line drops "to the root of" - it contradicted the
subfolder instruction directly below it.
The row count next to the table name is the capped count, not the total
number of matches. A search that matches more rows than DC_MAXOBS_WEBVIEW
returns the first 500 and reports 500, so the count does not rise above the
cap even when many more rows match.
The example claimed `smith` finds `Smithson`, which only holds for a case
insensitive match. The search is a case sensitive CONTAINS match - DC passes
the search value straight to %mp_searchdata, which uses the SAS `?` operator -
so `smith` finds `Goldsmith` (a lowercase substring) and not `Smithson`.
The screenshot showed the siphonophore search returning 3 rows, but the
NOTES column - the only column the value appears in - was clipped off the
right edge, so the image did not show why those rows matched.
Re-captured at 1920x900 (all nine columns on screen, no clipping) from the
mocked instance, with the capture gated on an assertion that the matched
cell is inside the grid viewport.
The "Full Table Search" section said only that a search box exists. It now
states the behaviour that users actually hit:
- the search covers every column at once, character columns by case sensitive
CONTAINS and numeric columns by exact match
- the Numeric toggle switches to exact numeric matching
- an applied filter scopes the search
- results are capped by DC_MAXOBS_WEBVIEW (linked to the options page), and the
row count next to the table name reports the match count
- a search with no matches shows the "No data found with given conditions"
panel
Adds a screenshot of the search in action (docs/img/full-table-search.png),
captured from a running instance via the Cypress suite in the dc repo.
dc