dc/docs.datacontroller.io
6
1 Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: dc/docs.datacontroller.io:servername
dc/docs.datacontroller.io:main dc/docs.datacontroller.io:docs/7.19-release dc/docs.datacontroller.io:docs/v7.13-features dc/docs.datacontroller.io:servername dc/docs.datacontroller.io:gh-pages
dc/docs.datacontroller.io:v1.1.1 dc/docs.datacontroller.io:v1.1.0 dc/docs.datacontroller.io:v1.0.0
..
pull from: dc/docs.datacontroller.io:docs/7.19-release
dc/docs.datacontroller.io:main dc/docs.datacontroller.io:docs/7.19-release dc/docs.datacontroller.io:docs/v7.13-features dc/docs.datacontroller.io:servername dc/docs.datacontroller.io:gh-pages
dc/docs.datacontroller.io:v1.1.1 dc/docs.datacontroller.io:v1.1.0 dc/docs.datacontroller.io:v1.0.0
135 Commits
servername .. docs/7.19-release
Author SHA1 Message Date
dc 0e27464136 docs(7.19): publish derived variables - banners off, nav titles clean, roadmap entry 
Held until the release lands: the draft banner on both derived pages, the
(draft) suffix in the nav, and the Delivered Features entry they were waiting
for. Merge this with, or just after, dc/dc#348.
 2026-10-26 08:53:47 +00:00
allan 05daf64ba6 Merge pull request 'docs(config): derived variables in the EDIT screen' (#21) from docs/derived-variables into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 2m20s
Details 
Reviewed-on: #21
Reviewed-by: Allan <allan@4gl.io>
 2026-10-21 22:26:31 +00:00
dc 2076eddd1b docs(derived): say circular references rather than a cycle 2026-10-21 22:25:50 +00:00
dc 94bf510010 docs(derived): shorten the dependency rule 2026-10-21 22:24:55 +00:00
dc d6065826fb docs(derived): tighten the collision rule 2026-10-21 22:24:05 +00:00
dc e9d48594d7 docs(derived): shorten the collision rule to the rule itself 2026-10-21 22:22:24 +00:00
dc 64c3fb51aa docs(derived): correct the collision rule 
TX_FROM, TX_TO and PROCESSED_DTTM are the default names of the metadata
columns rather than reserved names - they are physical columns, refused by the
collision check whatever MPE_TABLES calls them.  The only name refused outright
is the delete marker.  The code now matches: the hard-coded list is gone.
 2026-10-21 22:20:04 +00:00
dc 3e4e741f75 docs(derived): drop the retired 'defined earlier' wording 
There is no earlier: dependencies are declared in VAR_DEPS and checked in both
directions, so the rejection is an undeclared reference. Both sentences now say
so.
 2026-10-21 22:15:07 +00:00
dc c25686c77b docs(derived): mark the page as draft so it can merge before the release 
A warning banner on both pages, and '(draft)' in the navigation title, so a
reader is not surprised when the feature is not in their installed version.
Remove both when the release carrying derived variables goes out.

mkdocs build passes; the banner and all four images render.
 2026-10-21 22:13:56 +00:00
dc 57edfe5478 docs(derived): include the MPE_DERIVED configuration, recapture at full page 
The screenshots now show the whole page rather than a cropped viewport, and the
configuration table is included so the three variables in the grid can be
traced to the three rows that define them.
 2026-10-21 15:10:17 +00:00
dc a70daf1789 docs(derived): show the feature, and why it needs a view 
Three screenshots from the EDIT screen: the derived columns in the grid, a
filter on a character derived column, and a filter on a numeric one.  Plus a
short 'how it works' explaining that the statements go into a DATA step view
because a WHERE cannot see a variable created in the same step.
 2026-10-21 14:36:51 +00:00
dc ac48b91bf4 docs(config): optional VAR_LABEL 2026-10-21 08:14:55 +00:00
dc 617007bbbd revert(config): a derived variable may not reuse a column name 2026-10-20 23:00:13 +00:00
dc d96453ad42 docs(config): a derived variable may shadow a physical column 2026-10-20 22:25:21 +00:00
dc cddf39e58c docs(config): VAR_DEPS 2026-10-20 18:33:30 +00:00
dc 710463ce06 docs(config): VAR_DEPENDENCIES replaces VAR_ORDER 
Each row lists every input the expression uses, physical columns and derived
variables alike.  No sequence number to maintain; statements are emitted in
dependency order and a cycle is rejected.
 2026-10-20 16:51:13 +00:00
dc 926299f2ee docs(config): the label is <VAR_NM>=<expression> 2026-10-20 11:52:03 +00:00
dc 98e41dbbd7 docs(config): duplicate names are caught by the loader, not the hook 2026-10-20 09:56:57 +00:00
dc 3d5090d49d docs(config): validation rules on MPE_DERIVED 2026-10-20 09:01:58 +00:00
dc 9d27f37ad2 docs(config): MPE_DERIVED is validated at submit time 2026-10-20 08:20:56 +00:00
dc b9f7300708 docs(config): VAR_* column names in MPE_DERIVED 
Matches dc/dc: VAR_NM, VAR_ORDER, VAR_TYPE, VAR_LEN, VAR_FMT, VAR_LOGIC.
 2026-10-20 07:57:59 +00:00
dc e233f508b6 docs(config): scope the configuration-error claim to what is actually reported 
Only the config checks name the variable; a syntax error or an unknown
column fails later in the view build and the abort names the table.
 2026-10-20 02:49:34 +00:00
dc 42ffc63009 docs(config): VARIABLE_ORDER and chained derived variables 
A derived variable can now be built on one with a lower VARIABLE_ORDER, so
the pages describe the order column, the ordered rule (replacing the
self-contained one) and a chained example.
 2026-10-20 02:09:46 +00:00
dc 2272f55162 docs(config): fix the editor anchor and the full reserved-name list 
Review follow-up: /dc-userguide/#editor (the heading is 'Editor'), the
reserved names include VAR_TXFROM/VAR_TXTO/VAR_PROCESSED, and derived
columns are not shown in VIEW or downloads though a saved filter on one
still applies there.
 2026-10-20 01:24:59 +00:00
dc 183037c4d3 docs(config): derived variables in the EDIT screen 
New Configuration page explaining derived (computed) variables: what they
are, how to configure them in MPE_DERIVED, how filtering works, the rules
and limits (expressions only, self-contained, read-only, not stored), and
the data-exposure and performance notes.

New MPE_DERIVED table page, registered in the Table Guide nav.
 2026-10-19 23:20:18 +00:00
allan b7794cef47 Merge pull request 'docs(filter): the applied clause carries SAS temporal literals' (#20) from docs/filter-date-literals into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m23s
Details 
Reviewed-on: #20
 2026-10-18 09:58:08 +00:00
allan 2b60c7443b Merge pull request 'docs(developer-guide): the delete-record column is client-owned' (#19) from docs/delete-record-column-client-side into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m27s
Details 
Reviewed-on: #19
 2026-10-18 09:53:47 +00:00
dc bef7a440ac docs(filter): the applied clause carries SAS temporal literals 
The clause shown above the grid is the resolved WHERE clause, so a date,
datetime or time filter reads as a SAS literal rather than the internal day or
second count.  Note it on the filter page and on the MPE_FILTERSOURCE RAW_VALUE
column, which is where the value is stored.
 2026-10-16 09:38:51 +00:00
dc 21fcf2d6fa docs(developer-guide): the delete-record column is client-owned 
The pre-edit hook example re-added _____DELETE__THIS__RECORD_____="No" when
it re-read the table, mirroring the data step getdata.sas used to run. That
column is created by the editor itself (it is the Delete? column) and is
never part of the dataset, so the example no longer adds it - and a response
that carries the name is refused by the editor, which the "What the hook
does" bullet now states.

Also notes the reserved name on the MPE_TABLES page, and says in the user
guide that the Delete? column is created by Data Controller rather than
being part of the table.
 2026-10-15 22:54:20 +00:00
allan 127f10fba9 Merge pull request 'docs(build): pin the deploy tooling, correct the hook macro-scope note' (#18) from docs/pin-deploy-and-hook-scope into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m29s
Details 
Reviewed-on: #18
 2026-10-15 18:56:57 +00:00
dc 8a22e28a46 docs(developer-guide): correct the LIBREF/DS macro-scope note 
The post-edit hook section claimed a `%let` "creates a new variable in the
hook's own scope and is silently discarded" while `call symputx` updates the
existing one. That is not how SAS resolves it: LIBREF and DS already exist in
an enclosing symbol table (mpe_loader's local table), and a macro only gets a
new local when the name is absent from every enclosing table, so both forms
assign to the same variable. Reworded to state that, and to keep the practical
advice (call symputx is explicit and does not depend on the enclosing scope
still holding the name).

Raised in review of #16.
 2026-10-15 18:28:36 +00:00
dc 8ed0f16004 docs(build): pin the deploy tooling versions and disable install scripts 
The publish workflow installed cloudron-surfer, the mkdocs pip set and
@marp-team/marp-cli with no version pins, so a breaking or compromised upstream
release would ship straight to the live site on the next merge to main.

- pin cloudron-surfer@6.4.1, mkdocs==1.6.1, mkdocs-material==9.7.7,
  fontawesome_markdown==0.2.6, mkdocs-redirects==1.2.3 and
  @marp-team/marp-cli@4.5.1 (the versions the site builds with today)
- add .npmrc with ignore-scripts=true and save-exact=true; neither npm package
  ships an install/preinstall/postinstall hook, so nothing is skipped
 2026-10-15 18:28:36 +00:00
allan cc2441fd2d Merge pull request 'docs(developer-guide): show the pre-edit hook result, fix the re-read drop list' (#17) from docs/pre-edit-hook-images into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m13s
Details 
Reviewed-on: #17
 2026-10-15 17:57:19 +00:00
dc 9d48240111 docs(developer-guide): address review on the pre-edit hook page 
- The re-read step now sorts work.out by &sortpk after rebuilding it, matching
  getdata (which sorts before the hook runs and does not sort again), so the
  grid keeps the order the user first saw. The bullet records why.
- Adds a .pre-commit-config.yaml with the gitleaks hook pinned to v8.30.1, so
  staged content is scanned at commit time.
 2026-10-15 17:49:01 +00:00
dc 0de0fbde4a docs(developer-guide): show the pre-edit hook result, fix the re-read drop list 
- Adds a "What it looks like" section to the pre-edit hook page: the editor
  before (the user's own filter, one quarter of rows) and after (the hook's
  widened filter, both quarters), captured from the mocked estate.
- Corrects the re-read step: it now drops &drop_rk like getdata does, and the
  accompanying note records that a bitemporal table also needs the latest-record
  dedup, so the example no longer claims to mirror getdata exactly while missing
  both (raised in review of #16).
 2026-10-15 17:11:55 +00:00
allan 10ca9547b3 Merge pull request 'docs(developer-guide): add a Developer Guide section' (#16) from docs/developer-guide into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m21s
Details 
Reviewed-on: #16
 2026-10-15 16:28:20 +00:00
dc cc91f8b1f9 docs(developer-guide): add a Developer Guide section 
Two pages, both aimed at SAS developers rather than end users:

- Filter in a Pre-Edit Hook: what editors/getdata returns (sasparams.FILTER_TEXT
  and the query clauses, never FILTER_RK), when PRE_EDIT_HOOK runs relative to
  the filter, and a worked example that widens a date filter to the previous
  quarter - re-reading the table so the rows match, and keeping Row Level
  Security applied to both branches.
- One Table, Two Rule Sets: moved from the datacontroller.io blog, so the
  developer-facing material lives in one place. Images moved to docs/img/.
 2026-10-15 16:16:15 +00:00
allan c5d6806d53 Merge pull request 'docs(videos): host the Customer Story video on PeerTube' (#15) from docs/customer-story-video into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m20s
Details 
Reviewed-on: #15
 2026-10-11 10:29:07 +00:00
allan b1ec8bec7c Merge pull request 'docs(options): update the EDIT and VIEW size limits' (#14) from docs/row-cell-limits into main
Publish to docs.datacontroller.io / Deploy docs (push) Failing after 1m14s
Details 
Reviewed-on: #14
 2026-10-11 10:28:48 +00:00
dc 029eee6c7b docs(videos): host the Customer Story video on PeerTube 2026-10-11 10:12:48 +00:00
dc e995626ff4 docs(options): update the EDIT and VIEW size limits 
DC_MAXOBS_WEBEDIT 100 -> 250, DC_MAXOBS_WEBVIEW 500 -> 2000, and a new
DC_MAXCELLS_WEBEDIT (200000) which is applied together with the EDIT
observation limit - whichever is reached first.
 2026-10-03 11:57:36 +00:00
allan c800145ead Merge pull request 'feat(theme): dark mode by default, with a light/dark toggle' (#13) from docs/dark-mode-toggle into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m17s
Details 
Reviewed-on: #13
 2026-09-28 23:44:54 +00:00
dc 504a3136b7 feat(theme): dark mode by default, with a light/dark toggle 
The docs site only offered Material's light scheme. This adds the dark
scheme and makes it the default.

- palette: two entries, slate first, each with a toggle. Material takes the
  first entry as the default, so the site opens dark and a sun icon in the
  header switches to light (and a moon switches back).
- dc-brand.css: the dark scheme's greys are derived from a hue variable, so
  it is pointed at the brand slate (206deg) rather than Material's neutral
  225deg, which gives the whole scheme the same cast as the header.
- dc-brand.css: the brand slate is unreadable on a dark background, so the
  dark scheme uses a light tint of it (#9fb5c6) for body links. Green stays
  the accent, which keeps link hover visible in both schemes.

Contrast, over the dark background rgb(30, 36, 41): body text 8.8:1, body
links 7.4:1, inline code 7.2:1, accent green 5.6:1, header text 10.2:1. The
light scheme is unchanged.
 2026-09-28 23:15:54 +00:00
allan 2e269341df Merge pull request 'style(theme): apply the Data Controller brand palette and logo' (#12) from docs/onbrand-styling into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m16s
Details 
Reviewed-on: #12
 2026-09-28 23:03:26 +00:00
dc 6d09a8e1d8 style(theme): apply the Data Controller brand palette and logo 
The docs site rendered as stock Material: indigo header, indigo links and
Open Sans. Neither colour nor type matched the marketing site
(datacontroller.io) or the application itself.

- Palette: the slate (#314351) and green (#79a843) already used by the
  marketing navbar, hero and logo accent, wired up via a `custom` palette
  and defined in the new docs/dc-brand.css.
- Type: Montserrat for text, matching the marketing site's headings.
- Logo: the application's own dc-logo.svg mark in the header.

The previous `palette:` mapping form (a dict of primary/accent) is no
longer honoured by the Material version this repo builds against, so the
theme fell through to Material's default indigo - the site was not in fact
rendering the White/Amber it asked for. The list form used here is the one
Material currently reads.
 2026-09-28 22:55:05 +00:00
allan 19e50fdf6a Merge pull request 'docs(troubleshooting): note the platform's own URL parameters' (#11) from docs/platform-params into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m13s
Details 
Reviewed-on: #11
 2026-09-28 00:17:27 +00:00
dc 6f4166dad3 docs(troubleshooting): note the platform's own URL parameters 
On SAS 9 and Viya the app is reached through the platform's job URL, so
work.browser_url_vars carries the platform's parameters (_FILE, _program,
_debug, _webout) alongside DC's. Say so, so a hook author does not read
_FILE as a Data Controller parameter.
 2026-09-27 22:42:51 +00:00
allan aff6806bb9 Merge pull request 'docs(troubleshooting): document the browser_info / browser_url_vars support diagnostics' (#9) from docs/support-diagnostics into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m21s
Details 
Reviewed-on: #9
 2026-09-27 20:51:20 +00:00
dc 13042876b8 docs(troubleshooting): a worked URL-parameter example for the hook read 2026-09-27 20:47:53 +00:00
dc 7f2a76986a docs(troubleshooting): correct the browser_url_vars collision, add hook read examples 2026-09-27 20:20:07 +00:00
allan 90b374a52e Merge pull request 'docs(downloads): new page for release assets and download verification' (#10) from docs/release-download-verification into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m17s
Details 
Reviewed-on: #10
 2026-09-25 19:35:08 +00:00
dc 4316f519c4 docs(downloads): appLoc is a metadata folder, not a disk location 
The integrity section claimed SAS writes nothing to disk outside the
appLoc. The appLoc is the SAS Folder the stored processes or jobs are
deployed into (metadata), so it is not a disk location at all - the only
physical location the deployment programs write to is the Data Controller
location (dcLoc) configured at first launch.
 2026-09-25 19:33:42 +00:00
dc be8d92f439 docs(troubleshooting): note the 128 debug value and the loadfile upload limitation 
The adapter sends _debug=128 (not 131) on the Viya web JES path when
runAsTask is enabled, which is what the frontend uses there. Document
that value alongside 131, and note that editors/loadfile is reached
through the multipart file upload, which carries no input tables.
 2026-09-25 07:55:49 +00:00
dc 7770737963 docs(troubleshooting): scope the diagnostics tables to their services, add browser_url_vars 
The support diagnostics are now sent only where they can be used: the
startup service and the services that execute customer-provided code
(hook scripts, dynamic cell dropdown programs) - not with every service
call. The page's URL parameters arrive as a new browser_url_vars table,
one row per parameter, so a SAS developer reads a parameter by name
rather than parsing the url string.
 2026-09-25 00:21:56 +00:00
dc fdb39a78a5 docs(downloads): new page for release assets and download verification 
Adds /downloads to the Installation section, listing what each release
asset is for and how to verify a download against the SHA256SUMS file
the release pipeline publishes (sha256sum -c, plus certutil for
Windows). The SAS 9 deployment page now links to it wherever it sends
the reader to the releases page.

mkdocs build passes; nav, sitemap and cross-links verified in the
rendered output.
 2026-09-24 22:30:38 +00:00
dc 2a65664a51 docs(troubleshooting): document the browser_info support diagnostics 
Every service call carries a single-row browser_info input table, readable
by any service or hook script as work.browser_info and dumped to the job log
when debug is on. Document its columns and how to see it, so a support ticket
can be answered from the log without follow-up questions.
 2026-09-24 20:29:30 +00:00
allan c21f413f06 Merge pull request 'docs(validations): document how each rule treats a special missing value' (#8) from docs/special-missing-rule-behaviour into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m20s
Details 
Reviewed-on: #8
Reviewed-by: Allan <allan@4gl.io>
 2026-09-23 22:30:41 +00:00
dc 10448de15b docs(validations): point the range rules at the special missing section, and tidy 
MINVAL and MAXVAL now say what a special missing does to them and that the bound
itself may be a special missing, pointing at the Special Missing Values section,
since that is the least obvious thing about them.

Also in that section: the framing sentence covers the range rules' ordering, the
summary sentence is rewritten (it read badly), and the arrow in the ROUND row is
plain ASCII.
 2026-09-23 22:00:18 +00:00
dc 83641a204f docs(validations): a range rule compares in SAS's own order 
A range rule keys both the cell value and its own value into the order SAS uses
for a numeric variable: every missing below every non-missing value, and the
missing values ordered ._ then the regular missing then .A through .Z. So a range
can be written in special missings and mean what SAS means by it - MINVAL .A with
MAXVAL .C accepts .B and rejects .D - and the numeric cases follow from the same
order: a missing fails a numeric MINVAL and passes a numeric MAXVAL, and a number
sits above every missing.

Replaces the "a range rule ignores a missing value" wording, which described the
previous take's behaviour rather than SAS's.
 2026-09-23 21:27:57 +00:00
dc 66c48a4cc7 docs(validations): a range rule ignores a missing value 
Three corrections to the special missing section:

- the period is optional: a special missing is typed as `a` or `.a`, `_` or `._`
- MINVAL and MAXVAL both accept a missing value, and both still reject a real
  number that is out of range. A minimum constrains a number, and a missing is
  not a number - NOTNULL is the rule for a column that must be populated
- only NOTNULL and the formulas actually reject a special missing, so the
  "not supported" summary is replaced with what does reject it and what does not
 2026-09-23 21:01:11 +00:00
dc 1a5cea9ebe docs(validations): a date formatted column takes no special missing 2026-09-23 15:38:37 +00:00
dc abf01d6375 docs(validations): the dropdowns support special missings 
Both SOFTSELECT and HARDSELECT support them, and the dropdown lists them
alongside the ordinary values (no NaN entries for a numeric column).
 2026-09-23 08:37:12 +00:00
dc e7fdb3ab52 docs(validations): primary keys are NOT NULL; HARDSELECT can match a special missing 
- a primary key column rejects a blank and a special missing whether or not
  the table carries a constraint or MPE_VALIDATIONS a NOTNULL rule
- HARDSELECT is now checked like any other value: it passes when the
  dropdown list contains the special missing
- HARDSELECT drops out of the 'not supported' list
 2026-09-23 08:19:40 +00:00
dc 4d53ced89c docs(validations): document how each rule treats a special missing value 
New Special Missing Values section covering NOTNULL, MINVAL/MAXVAL, CASE,
ROUND, the regex rules, the dropdowns and the formula rules, with the
'not supported' set called out.  Behaviour verified against a real SAS
estate and the deployed frontend validator.
 2026-09-23 07:50:06 +00:00
allan 7d0e349247 Merge pull request 'docs(viewer): correct the full table search row count claim' (#6) from docs/full-table-search-row-cap into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m12s
Details 
Reviewed-on: #6
 2026-09-22 16:38:19 +00:00
dc 99b2a7454e docs(viewer): state the row count rule for the unsearched view too 
Follow-up to the previous commit on this branch, both wording fixes.

- The sentence this branch deleted was the page's only description of the row
  count when no search is active - and that is the case where the count really
  can exceed the rows displayed (viewdata reports count(*) for the filtered
  view while the grid stops at DC_MAXOBS_WEBVIEW). State it, instead of leaving
  the page silent on the path readers use most.
- "returns the first 500" claims an ordering the search does not apply: the
  search is a WHERE clause with no ORDER BY, so which 500 come back is whatever
  the engine hands over first. Say "returns 500 of them".

Checked against viewdata.sas and mp_searchdata.sas on main: the search path
passes outobs=DC_MAXOBS_WEBVIEW to mp_searchdata, which caps the result dataset
in the same data step that builds it, and viewdata takes its row count from that
dataset - so for a search the count and the rows are capped together.
 2026-09-22 16:34:14 +00:00
allan 046f24d0b7 Merge pull request 'docs(sas9): unzip the frontend archive into its own folder' (#7) from docs/frontend-zip-flat-layout into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m21s
Details 
Reviewed-on: #7
Reviewed-by: Allan <allan@4gl.io>
 2026-09-22 16:31:29 +00:00
dc bd30a059ed docs(sas9): unzip the frontend archive into its own folder 
frontend.zip now holds the frontend files at their root (dc/dc #147, released in
v7.14.2), so unzipping no longer produces a client/dist folder to navigate into.

Step 2 now says to create the app folder and unzip the archive into it, naming
the file that should end up there. Step 3's URL example matches the folder the
reader created, and the intro line drops "to the root of" - it contradicted the
subfolder instruction directly below it.
 2026-09-22 16:28:10 +00:00
dc 206d88c6a9 docs(viewer): correct the full table search row count claim 
The row count next to the table name is the capped count, not the total
number of matches. A search that matches more rows than DC_MAXOBS_WEBVIEW
returns the first 500 and reports 500, so the count does not rise above the
cap even when many more rows match.
 2026-09-17 11:43:23 +00:00
allan cd47d0e442 Merge pull request 'docs(viewer): document full table search semantics with a screenshot' (#5) from docs/full-table-search into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m18s
Details 
Reviewed-on: #5
Reviewed-by: Allan <allan@4gl.io>
 2026-09-17 09:49:11 +00:00
dc 85f200c724 docs(viewer): correct the character search example to match the code 
The example claimed `smith` finds `Smithson`, which only holds for a case
insensitive match. The search is a case sensitive CONTAINS match - DC passes
the search value straight to %mp_searchdata, which uses the SAS `?` operator -
so `smith` finds `Goldsmith` (a lowercase substring) and not `Smithson`.
 2026-09-16 23:42:37 +00:00
dc 4a53499055 docs(viewer): show the matching column in the full table search screenshot 
The screenshot showed the siphonophore search returning 3 rows, but the
NOTES column - the only column the value appears in - was clipped off the
right edge, so the image did not show why those rows matched.

Re-captured at 1920x900 (all nine columns on screen, no clipping) from the
mocked instance, with the capture gated on an assertion that the matched
cell is inside the grid viewport.
 2026-09-16 23:38:14 +00:00
dc d95f440c4f docs(viewer): document full table search semantics with a screenshot 
The "Full Table Search" section said only that a search box exists. It now
states the behaviour that users actually hit:

- the search covers every column at once, character columns by case sensitive
  CONTAINS and numeric columns by exact match
- the Numeric toggle switches to exact numeric matching
- an applied filter scopes the search
- results are capped by DC_MAXOBS_WEBVIEW (linked to the options page), and the
  row count next to the table name reports the match count
- a search with no matches shows the "No data found with given conditions"
  panel

Adds a screenshot of the search in action (docs/img/full-table-search.png),
captured from a running instance via the Cypress suite in the dc repo.
 2026-09-16 00:52:12 +00:00
hermes 94b27628aa Merge pull request 'docs: remove customer reference and conversion-function warning from timezone FAQ' (#4) from fix/timezone-docs-redact into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m12s
Details
2026-09-04 11:34:05 +00:00
dc-bot 5236b74fee docs: remove customer reference and conversion-function warning from timezone FAQ 2026-09-04 11:33:49 +00:00
hermes 06df7e6e23 Merge pull request 'docs: remove customer-identifying details from timezone troubleshooting section' (#3) from fix/timezone-docs-redact into main
Publish to docs.datacontroller.io / Deploy docs (push) Failing after 13m6s
Details
2026-09-04 10:21:03 +00:00
dc-bot 7e49033aae docs: remove customer-identifying details from timezone troubleshooting section 2026-09-04 10:20:08 +00:00
dc-bot 76bdc90ed8 docs: add troubleshooting section for timestamps displayed in UTC (compute context TIMEZONE)
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 17m7s
Details
2026-09-04 10:00:58 +00:00
allan 6782b77e69 Merge pull request 'docs: document v7.13.0 features' (#2) from docs/v7.13-features into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m26s
Details 
Reviewed-on: #2
 2026-09-03 20:11:49 +00:00
dc-bot 6d9d29b786 docs(roadmap): remove duplicated delivered validation features from Requested section 2026-09-03 20:10:54 +00:00
dc-bot 6f176b35e1 docs(roadmap): use headings instead of bold text for design subsections 2026-09-03 20:07:09 +00:00
dc-bot d342e51241 docs(roadmap): fix wrong 'under consideration' heading - features are delivered 2026-09-03 19:53:03 +00:00
dc-bot 57e8a6e975 docs: improve formula examples, fix roadmap structure, add trailing newline 
- dcc-validations: add MATCH() example showing why spaces matter around column names; document that new rows are 'A' from creation
- roadmap: move delivered Frontend Formulae and Regex Rules to Delivered Features section; reword 'necessary' heading
- deploy-viya: add trailing newline
 2026-09-03 19:50:31 +00:00
dc-bot 9666ca3300 docs: remove non-DC and low-value sections from v7.13.0 docs bundle 
- deploy-viya: remove first-launch note, Deploy Checks section
- admin-services: remove Licence Key Screen section
- cas-tables: remove REPLACE Load Type subsection
 2026-09-03 19:32:49 +00:00
dc-bot 6fa1b4f6ba docs(viya): remove low-value login page note 2026-09-03 19:13:13 +00:00
dc-bot 1d0cfd6681 docs: document v7.13.0 features 
- Live formulas (HARDFORMULA/SOFTFORMULA) in dcc-validations with operator-friendly explanations
- MPE_VALIDATIONS table doc: add HARDFORMULA/SOFTFORMULA to RULE_TYPE values
- Roadmap: mark Frontend Formulae and Regex Rules as delivered
- SAS VA Embed: document Live vs Confirm filter modes
- ViewBoxes: document edge/corner drag resizing
- Licensing: combined-key paste, key preview, protocol warning
- CAS Tables: REPLACE load type support and temp table cleanup
- Editor: native date/time pickers, paste-validation overlay, row status indicators
- Stage page: Formatted/Unformatted toggle on the approvals screen
- Viya deploy: configurator improvements, deploy checks, login page UX
- Index page: add live formulas and VA embed to features list
 2026-09-03 17:25:40 +00:00
blog-dev 021db986e0 fix(docs): correct feed link to /rollback-data-changes/
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m13s
Details
2026-08-20 22:26:04 +01:00
blog-dev 11f298ee2a docs: drop version reference from data restore page
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m24s
Details
2026-08-20 21:50:26 +01:00
blog-dev 034700a33e docs(rewrite): expand data restore guide with detailed workflow, security, and limitations
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m17s
Details
2026-08-20 21:37:38 +01:00
4gl c875ce04b7 chore(docs): updating data catalog refresh process
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m16s
Details
2026-08-10 21:51:46 +01:00
4gl 2f30680e84 feat: adding all tables to the docs
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m17s
Details
2026-08-10 17:11:53 +01:00
4gl 9b68c55754 feat: regex docs
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m19s
Details
2026-07-28 12:22:09 +01:00
allan 0590191ff4 feat: updates for v7.10 release
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 2m31s
Details
2026-07-13 18:23:00 +01:00
allan 964434af0a fix: link
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 58s
Details
2026-07-01 13:14:29 +01:00
allan dec5f239de fix: embed va
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m4s
Details
2026-07-01 13:12:22 +01:00
allan 80cabc7312 fix: va page
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m46s
Details
2026-07-01 12:23:08 +01:00
allan ddd5866484 feat: 7.9 updates
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m48s
Details
2026-06-30 13:42:06 +01:00
4gl a0bb0c1e20 info about casuser lib
Publish to docs.datacontroller.io / Deploy docs (push) Failing after 13m12s
Details
2026-06-02 13:56:53 +01:00
4gl 1b590a6ee7 fix: docs link
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m3s
Details
2026-05-19 11:54:15 +01:00
4gl 874e045dff fix: deploy docs update for runastask
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m5s
Details
2026-05-15 13:27:48 +01:00
4gl 1ca32b4b71 fix: headings
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m4s
Details
2026-04-23 10:50:21 +01:00
4gl b880988887 fix: description
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m10s
Details
2026-04-23 10:48:10 +01:00
4gl 55ad7424fd fix: security settings
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m8s
Details
2026-04-23 10:39:57 +01:00
allan f3954fa046 feat: cas table proposal
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m17s
Details
2026-04-17 13:53:53 +01:00
allan 05a16acabb fix: title
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m1s
Details
2026-04-07 22:00:59 +00:00
allan ae6620b3db feat: viya deploy instructions
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 59s
Details
2026-04-07 21:55:09 +00:00
allan 447397236b fix: email link
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 59s
Details
2026-04-04 00:57:53 +01:00
allan 3bfd96f431 fix: typos
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m6s
Details
2026-04-04 00:56:10 +01:00
allan ffdbdb869c feat: email templates
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 2m20s
Details
2026-04-04 00:49:32 +01:00
allan 535937b586 chore: docs
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m1s
Details
2026-04-03 16:08:05 +01:00
allan ad5a566001 fix: doc improvements
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m3s
Details
2026-03-30 23:01:24 +00:00
allan 15c617f48e feat: snowflake support
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m2s
Details
2026-03-14 13:40:47 +00:00
_ 6a950ea839 fix: separate
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m1s
Details
2026-02-27 14:07:09 +00:00
_ 5dfbc2526e feat: updated viya flow
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m2s
Details
2026-02-27 14:01:01 +00:00
_ ca8499de90 roadmap
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m5s
Details
2026-02-16 14:22:15 +00:00
_ 102f59d2e8 fix: formula approach
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m3s
Details
2026-02-16 13:31:07 +00:00
_ 71af1df610 fix: roadmap improvements
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 54s
Details
2026-02-15 23:57:05 +00:00
_ f9daa7dfbb feat: new validations
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 57s
Details
2026-02-15 23:49:13 +00:00
allan 965efacf70 fix: typo
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 53s
Details
2026-02-08 00:57:47 +00:00
zver bd3addd619 fix: toc + copydate
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 55s
Details
2026-02-08 00:26:14 +00:00
zver 81ec07117d feat: mpe_validations table and a note about the pgmloc var in hook scripts
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m0s
Details
2026-02-08 00:22:29 +00:00
allan 45a46d6a0a fix: pipeline
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m5s
Details
2025-10-01 12:34:20 +01:00
allan 52c3101807 fix: guide
Publish to docs.datacontroller.io / Deploy docs (push) Failing after 2m19s
Details
2025-10-01 12:18:27 +01:00
allan 6a66f8439d feat: revert changes to a table
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m27s
Details
2025-07-15 16:41:27 +01:00
allan 38045f9ba6 fix: object catalog info
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m28s
Details
2025-07-15 15:59:35 +01:00
allan 6309e91272 feat: datastatus_cats
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m26s
Details
2025-07-15 15:35:23 +01:00
allan 807dd55bac fix: objects info
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m27s
Details
2025-07-15 15:07:40 +01:00
allan 2ec7d35342 feat: refresh catalog docs
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m31s
Details
2025-07-15 14:45:39 +01:00
allan 4c45779312 fix: mentioning DI in redeployment on sas 9
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 2m43s
Details
2025-07-02 16:22:22 +01:00
allan 92f2f4f6d2 feat: datacatalog
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m48s
Details
2025-06-11 23:34:34 +01:00
allan 5ec342cbc4 fix: typo
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m52s
Details
2025-06-05 13:55:06 +01:00
allan f090dc18ba fix: improvements2
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m52s
Details
2025-06-04 14:53:03 +01:00
allan 7c2cc2628b fix: numbering and additional details
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m50s
Details
2025-06-04 14:40:44 +01:00
allan c6ef23d54b feat: updated viya deploy instructions
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m53s
Details
2025-06-04 13:56:22 +01:00
allan 5715d17312 Merge pull request 'fix: servername' (#1) from servername into main
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 2m21s
Details 
Reviewed-on: #1
 2025-03-11 20:47:22 +00:00
87 changed files with 2033 additions and 401 deletions
Expand all files Collapse all files

No files matched your search

.gitea/workflows/publish.yml
+15 -16
View File
@@ -12,20 +12,17 @@ jobs:
- uses: actions/setup-node@v3
with:
node-version: 18
- name: Checkout master
uses: actions/checkout@v2
uses: actions/checkout@v4 # Updated to latest version
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.12' # Specify your desired version (e.g., 3.10 or 3.12)
env:
AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
RUNNER_TOOL_CACHE: /opt/hostedtoolcache
- name: Install pip3
run: |
apt-get update
apt-get install python3-pip -y
- name: Install Chrome
run: |
@@ -34,20 +31,22 @@ jobs:
apt install -y ./google-chrome*.deb;
export CHROME_BIN=/usr/bin/google-chrome
# Deploy tooling is pinned to exact versions: the workflow builds whatever
# is committed, and an unpinned `latest` would let a breaking or
# compromised upstream release ship straight to the live site.
- name: Install Surfer
run: |
npm -g install cloudron-surfer
npm -g install cloudron-surfer@6.4.1
- name: build site
run: |
pip3 install mkdocs
pip3 install mkdocs-material
pip3 install fontawesome_markdown
pip3 install mkdocs-redirects
python3 -m mkdocs build --clean
pip install mkdocs==1.6.1
pip install mkdocs-material==9.7.7
pip install fontawesome_markdown==0.2.6
pip install mkdocs-redirects==1.2.3
mkdocs build --clean
mkdir site/slides
npx @marp-team/marp-cli slides/innovation/innovation.md -o ./site/slides/innovation/index.html
npx @marp-team/marp-cli slides/if/if.md -o site/if.pdf --allow-local-files --html=true
npx @marp-team/marp-cli@4.5.1 slides/innovation/innovation.md -o ./site/slides/innovation/index.html
npx @marp-team/marp-cli@4.5.1 slides/if/if.md -o site/if.pdf --allow-local-files --html=true
- name: Deploy docs
run: surfer put --token ${{ secrets.SURFERKEY }} --server docs.datacontroller.io site/* /
.gitpod.yml
-8
View File
@@ -1,8 +0,0 @@
# This configuration file was automatically generated by Gitpod.
# Please adjust to your needs (see https://www.gitpod.io/docs/config-gitpod-file)
# and commit this file to your remote git repository to share the goodness with others.
tasks:
- init: npm install
.npmrc
+11
View File
@@ -0,0 +1,11 @@
# npm configuration for this repo's CI and local installs.
#
# ignore-scripts keeps package install scripts (an arbitrary-code execution
# point) from running during the deploy workflow. Both npm packages the
# workflow installs - cloudron-surfer and @marp-team/marp-cli - ship no
# install, preinstall or postinstall hook, so nothing is skipped.
#
# save-exact pins anything installed from here to an exact version rather than
# a caret range.
ignore-scripts=true
save-exact=true
.pre-commit-config.yaml
+11
View File
@@ -0,0 +1,11 @@
# Hooks for the pre-commit framework - https://pre-commit.com
#
# This repo carries no native git hooks, so this is the route for a secret scan
# at commit time: install the framework once with `pre-commit install`, and the
# scan then runs on every commit. The hook is pinned to an exact gitleaks
# release tag.
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.30.1
hooks:
- id: gitleaks
AGENTS.md
+51
View File
@@ -0,0 +1,51 @@
# Agent Instructions
This repository is the **user-facing documentation** for Data Controller for SAS®, published at [docs.datacontroller.io](https://docs.datacontroller.io). It is a [MkDocs](https://www.mkdocs.org/) site using the Material theme.
## Related repositories
Data Controller spans three sibling repositories (usually checked out side by side under the same parent directory):
- **`dc`** - the product source (Angular client + SAS backend). The behaviour documented here is implemented there. Deep technical notes live in `dc/.agent/docs/`.
- **`docs.datacontroller.io`** (this repo) - the user-facing product documentation.
- **`datacontroller.io`** - the marketing site, blog and feed (Gatsby).
When documenting a feature, the source of truth for behaviour is `dc`. When a doc page describes internals, prefer linking to the user-facing concept rather than duplicating implementation detail.
## Structure
- Pages are Markdown files in `docs/`.
- The navigation tree, site config, theme, plugins and redirects are all defined in `mkdocs.yml`. **A new page is not published until it is added to the `nav:` tree in `mkdocs.yml`.**
- `docs/tables/` documents the `MPE_*` control tables (see naming conventions below).
- `docs/img/` holds images; `docs/video/` holds video assets; `docs/marketing/` holds flyers/PDFs.
- `theme/` is the custom Material theme override; `slides/` and `slides.md` are the presentation deck.
## Page conventions
- Each page starts with YAML front matter: `layout: article`, `title`, `description`, and usually `og_image`. Match the style of existing pages.
- `description` is used for SEO and social cards - write a single, complete sentence.
- Reference images with root-relative paths (e.g. `/img/foo.png`) or relative paths consistent with neighbouring pages.
- This site uses these `markdown_extensions`: `admonition`, `pymdownx.superfences`, `codehilite`, `meta`, and `toc` (with permalinks). Use fenced code blocks with language hints (`sas`, `js`, `bash` are highlighted); use admonitions (`!!! note`) for callouts.
- Internal links use the page slug with a trailing slash (e.g. `/dcc-validations/`), matching existing cross-references.
## MPE table docs (`docs/tables/`)
Control tables are documented one file per table, named `mpe_<name>.md`, and registered under the "Table Guide" section of `nav:` in `mkdocs.yml`. Follow the existing pattern:
- Front matter with a `description` explaining what the table configures.
- A short intro paragraph, then a link to the relevant configuration guide.
- A `## Columns` list. Prefix primary-key / business-key columns with the 🔑 emoji, and give each column as `` `NAME type` ``: description. SCD2 tables carry `TX_FROM`/`TX_TO` as the first two columns.
## Writing style
Use regular dashes (`-`) in content, not em-dashes. Do not hard-wrap Markdown: each paragraph, list item and heading is a single logical line, regardless of length - let the renderer soft-wrap. This keeps diffs clean.
## Building
- `pip install mkdocs mkdocs-material mkdocs-redirects` (see `build.sh` / `mkdocs.yml` for the exact plugin list).
- `mkdocs serve` for a live-reloading local preview; `mkdocs build` (or `./build.sh`) to produce the static site.
- After adding or renaming a page, confirm it appears in the `nav:` tree and that `mkdocs build` reports no warnings about missing/orphaned files.
## Git
Do NOT auto-commit or push. Leave changes in the working tree for the user to review and commit.
CONTEXT.md
+33
View File
@@ -0,0 +1,33 @@
# Context: docs.datacontroller.io (product documentation)
The user-facing documentation for Data Controller for SAS®, published at [docs.datacontroller.io](https://docs.datacontroller.io) as a **MkDocs** (Material theme) static site.
## What the product is
Data Controller for SAS® is a web application that lets users safely add, modify and delete data in SAS datasets and databases. Every change is **staged** and **approved** before being applied to the **target table**, and the change history is retained. It runs on SAS Viya, SAS 9 EBI and SASjs Server. The product source lives in the sibling `dc` repo (see its `CONTEXT.md` for the full domain glossary); the marketing site is `datacontroller.io`.
## Domain vocabulary (used throughout the docs)
- **Roles**: Viewer, Editor, Approver, Auditor, Administrator.
- **Target table**: the physical SAS/database table a user changes; configured by an admin in `MPE_TABLES`.
- **Submission / staging / approval**: changes are staged and require approval before being applied.
- **Load types** (`MPE_TABLES.LOADTYPE`): `UPDATE`, `REPLACE`, `TXTEMPORAL`, `BITEMPORAL`, `FORMAT_CAT` - determine history behaviour (SCD2 / bitemporal / none).
- **MPE control tables** (`MPE_*`): configuration and state tables, each documented under `docs/tables/mpe_<name>.md`.
- **Validations** (`MPE_VALIDATIONS`): point-of-entry data-quality rules.
- **Row / Column Level Security** (RLS / CLS): server-side access control.
Use these terms consistently; match the casing used in the existing docs.
## Structure
- Pages are Markdown in `docs/`. **A page is not published until it is added to the `nav:` tree in `mkdocs.yml`.**
- `docs/tables/` documents the `MPE_*` control tables (one file per table, following the shared column-list pattern with 🔑 for key columns).
- `mkdocs.yml` defines nav, theme, plugins (search, redirects) and markdown extensions (`admonition`, `pymdownx.superfences`, `codehilite`, `meta`, `toc`).
## Conventions
- Front matter per page: `layout: article`, `title`, `description`, usually `og_image`.
- Use regular dashes, not em-dashes. Do not hard-wrap Markdown.
- Preview with `mkdocs serve`; build with `mkdocs build` (or `./build.sh`) and confirm no warnings.
See `AGENTS.md` for full page/table conventions and build instructions.
docs/admin-services.md
+14 -88
View File
@@ -1,98 +1,24 @@
---
layout: article
title: Admin Services
description: Data Controller contains a number of admin-only web services, such as DB Export, Lineage Generation, and Data Catalog refresh.
description: The Administrator Screen provides useful system information and buttons for various administrator actions
og_title: Administrator Screen
og_image: /img/admininfo.png
---
# Admin Services
## Administrator Screen
Several web services have been defined to provide additional functionality outside of the user interface. These somewhat-hidden services must be called directly, using a web browser.
The admin screen (under user profile / System) displays a number of useful system parameters as well as several buttons for executing administrator specific actions
In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs).
![](./img/admininfo.png)
The URL is made up of several components:
Button info as follows:
* `SERVERURL` -> the domain (and port) on which your SAS server resides
* `EXECUTOR` -> Either `SASStoredProcess` for SAS 9, else `SASJobExecution` for Viya
* `APPLOC` -> The root folder location in which the Data Controller backend services were deployed
* `SERVICE` -> The actual Data Controller service being described. May include additional parameters.
|Button|Description|
|---|---|
|Refresh Data Lineage|This is only displayed for SAS9 installs. Will refresh all TABLE level data lineage (impact analysis) using an efficient batch approach (proc metadata).|
|Refresh Data Catalog|Update Data Catalog for ALL libraries. More info [here](/dcu-datacatalog).|
|Download Configuration|This downloads a zip file containing the current database configuration - useful for migrating to a different data controller database instance.|
|Update Licence Key| Link to the screen for providing a new Data Controller licence key|
|Export DC Library DDL|COMING SOON!! <br>Exports the data controller control library in DB specific DDL (eg SAS, PGSQL, TSQL) and allows an optional schema name to be included|
To illustrate the above, consider the following URL:
[https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL](https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL
)
This is broken down into:
* `$SERVERURL` = `https://sas.analytium.co.uk`
* `$EXECUTOR` = `SASJobExecution`
* `$APPLOC` = `/Public/app/dc`
* `$SERVICE` = `services/admin/exportdb&flavour=PGSQL`
The below sections will only describe the `$SERVICE` component - you may construct this into a URL as follows:
* `$SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE`
## Export Config
This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance.
EXAMPLE:
* `services/admin/exportconfig`
## Export Database
Exports the data controller control library in DB specific DDL. The following URL parameters may be added:
* `&flavour=` (only PGSQL supported at this time)
* `&schema=` (optional, if target schema is needed)
EXAMPLES:
* `services/admin/exportdb&flavour=PGSQL&schema=DC`
* `services/admin/exportdb&flavour=PGSQL`
## Refresh Data Catalog
In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as:
- Library Properties (size, schema, path, number of tables)
- Table Properties (size, number of columns, primary keys)
- Variable Properties (presence in a primary key, constraints, position in the dataset)
The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared? Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
To run the refresh process, just trigger the stored process, eg below:
* `services/admin/refreshcatalog`
* `services/admin/refreshcatalog&libref=MYLIB`
The optional `&libref=` parameter allows you to run the process for a single library. Just provide the libref.
When doing a full scan, the following LIBREFS are ignored:
* 'CASUSER'
* 'MAPSGFK'
* 'SASUSER'
* 'SASWORK
* 'STPSAMP'
* 'TEMP'
* `WORK'
Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them. This can be useful where there are connection issues for a particular library.
Be aware that the scan process can take a long time if you have a lot of tables!
Output tables (all SCD2):
* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)
## Update Licence Key
Whenever navigating Data Controller, there is always a hash (`#`) in the URL. To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: `/licensing/update`.
If you are using https protocol, you will have 2 keys (licence key / activation key). In http mode, there is just one key (licence key) for both boxes.
docs/cas-tables.md
+56
View File
@@ -0,0 +1,56 @@
---
layout: article
title: CAS Tables
description: Dealing with CAS (in-memory) Tables in Data Controller
og_image: /img/SAS-Viya-and-CAS-300x249.png
---
!!! warning
Work in Progress!
# CAS Tables
CAS Tables require special consideration in Data Controller with regards to the following topics:
- System Account
- Loading
- Unloading
- Special Variables
## System Account
Despite having a shared SYSUSERID, the SPRE session will (by default) authenticate using the logged-in user credentials. To get around this, it is necessary to set up the CAS connection in the autoexec - ie, before the user takes over the session. The code snippet will be:
```sas
%let _CASHOST_ = <your-host>;
%let _CASPORT_ = 5570;
cas dcsession authdomain="<your-domain>" sessopts=(caslib=casuser);
```
The credentials need to be first placed in the viya credentials service as described [here](https://go.documentation.sas.com/doc/en/pgmsascdc/v_073/casref/n0z3r80fjqpobvn1lvegno9gefni.htm#p11ynzjbz96oq1n17rgt2utv6swj).
Another approach can be to use the `AUTHINFO="authentication-file" option, as described [here](https://go.documentation.sas.com/doc/en/pgmsascdc/v_073/casref/n0z3r80fjqpobvn1lvegno9gefni.htm#n174yddgn85756n1v5q4yb881xa0).
Note that since the CAS connection is using a shared account, the CASUSER library is never shown in the DC interface.
## Loading
It can happen that a CAS table is configured in Data Controller but not loaded into memory. In this case, when a user selects the table, it will be automatically loaded.
## Unloading
After an approval, the in-memory version of the CAS Table will be updated. To apply this to the underlying file on disk, the following code must be executed (eg in a POST APPROVE HOOK):
```sas
proc casutil;
save casdata="mytable" incaslib="mycaslib"
casout="mytable" outcaslib="mycaslib"
replace;
quit;
```
## Special Variables
Processing of data in Data Controller is performed in SPRE with SAS datasets - and as such, it is not possible to process character variables longer than 32k or other CAS specific data types.
docs/dc-brand.css
+45
View File
@@ -0,0 +1,45 @@
/*
* Data Controller brand overrides for the Material theme.
*
* The values below mirror the marketing site (https://datacontroller.io): the
* slate used for its navbar and hero, and the green used for its logo accent.
* They are wired up through the `custom` palette in mkdocs.yml, which leaves
* these CSS variables for us to define.
*/
/* Primary - the brand slate, used for the header, the mobile drawer title and body links. */
:root,
[data-md-color-primary='custom'] {
--md-primary-fg-color: #314351;
--md-primary-fg-color--light: #3d5468;
--md-primary-fg-color--dark: #263542;
--md-primary-bg-color: #fff;
--md-primary-bg-color--light: #ffffffb3;
--md-typeset-a-color: #314351;
}
/* Accent - the brand green, used for link hover, focus rings and the active search hit. */
[data-md-color-accent='custom'] {
--md-accent-fg-color: #79a843;
--md-accent-fg-color--transparent: #79a8431a;
--md-accent-bg-color: #fff;
--md-accent-bg-color--light: #ffffffb3;
}
/*
* Dark scheme. Material derives its dark greys from a hue variable, so pointing
* that at the brand slate (206deg) gives the whole scheme the same cast as the
* header instead of Material's neutral default.
*/
[data-md-color-scheme='slate'] {
--md-hue: 206deg;
}
/*
* The brand slate is only readable on a light background, so the dark scheme
* uses a light tint of it for body links. Green stays the accent, which keeps
* link hover visible in both schemes.
*/
[data-md-color-scheme='slate'][data-md-color-primary='custom'] {
--md-typeset-a-color: #9fb5c6;
}
docs/dc-userguide.md
+14 -2
View File
@@ -42,12 +42,24 @@ The Editor screen lets users who have been pre-authorised (via the `DATACTRL.MPE
2 - *Upload*. If you have a lot of data, you can [upload it directly](files). The changes are then approved in the usual way.
3 - *Edit*. This is the main interface, data is displayed in tabular format. The first column is always "Delete?", as this allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted.
3 - *Edit*. This is the main interface, data is displayed in tabular format. The first column is always "Delete?", as this allows you to mark rows for deletion. This column is created by Data Controller and is not part of your table. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted.
The next set of columns are the Primary Key, and are shaded grey. If the table has a surrogate / retained key, then it is the Business Key that is shown here (the RK field is calculated / updated at the backend). For SCD2 type tables, the 'validity' fields are not shown. It is assumed that the user is always working with the current version of the data, and the view is filtered as such.
After this, remaining columns are shown. Dates / datetime fields have appropriate datepickers. Other fields may also have dropdowns to ensure entry of standard values, these can be configured in the `DATACTRL.MPE_SELECTBOX` table.
New rows can be added using the right click context menu, or the 'Add Row' button. The data can also be sorted by clicking on the column headers.
#### Native Date and Time Pickers
Date, time, and datetime columns use native browser pickers when editing a cell. This provides a familiar calendar and time selector, and respects your browser's locale settings for date and time formats.
#### Paste Validation Overlay
When you paste data into the editor (or drag to autofill cells), a confirmation overlay appears so you can review the changes before they are applied. All pasted values are checked against your configured [validation rules](/dcc-validations/). For large pastes, a progress indicator shows how many cells have been validated.
#### Row Status Indicators
Each row's header cell is colour-coded to show its current status - modified, added, deleted, or unchanged. The "modified" indicator uses a `±` symbol so you can quickly spot which rows have changed.
When ready to submit, hit the SUBMIT button and enter a reason for the change. The owners of the data are now alerted (so long as their email addresses are in metadata) with a link to the approve screen.
If you are also an approver you can approve this change yourself.
@@ -60,7 +72,7 @@ Data Controller supports special missing numerics, ie - a single letter or under
The Data Controller only permits BiTemporal data uploads at a single point in time - so for convenience, when viewing data in the edit screen, only the most recent records are displayed. To edit earlier records, either use file upload, or apply a filter.
### Submitted
This page shows a list of the changes you have submitted (that are not yet approved).
This page shows a list of the changes you have submitted (that are not yet approved). When you open a submitted change for review, you can toggle between viewing the data with SAS formats applied (eg formatted dates and currency) or as raw underlying values. This is useful when you need to verify the exact value being submitted rather than its display representation.
### Approvals
This shows the list of changes that have been submitted to you (or your groups) for approval.
docs/dcc-dates.md
+8
View File
@@ -39,6 +39,14 @@ run;
!!! note
Data Controller does not support decimals when EDITING. For datetimes, this means that values must be rounded to 1 second (milliseconds are not supported).
In the LOAD screen these dates display as **ISO 8601** (`YYYY-MM-DD`, `HH:mm:ss`, `YYYY-MM-DDTHH:mm:ss`) regardless of the user's locale. This guarantees:
- Consistent CSV / Excel exports across geographies
- Predictable copy / paste between cells and into external tools
- No more `1/2/2026` vs `2/1/2026` ambiguity at edit time
If you need a locale-specific *display* (eg `DD/MM/YYYY`) on a per-column basis, use the new [`NUMBER_FORMAT`](/dcc-validations/) rule with an `Intl.DateTimeFormat`-compatible JSON value.
If you have other dates / datetimes / times you would like us to support, do [get in touch](https://datacontroller.io/contact)!
docs/dcc-options.md
+53 -10
View File
@@ -1,41 +1,52 @@
---
layout: article
title: DC Options
description: Options in Data Controller are set in the MPE_CONFIG table and apply to all users.
title: DC Configuration Options
description: Configuration Options in Data Controller are set in the MPE_CONFIG table and apply to all users.
og_title: Data Controller for SAS® Options
og_image: /img/mpe_config.png
---
# Data Controller for SAS® - Options
# Configuration Options
The [MPE_CONFIG](/tables/mpe_config/) table provides a number of system options, which apply to all users. The table may be re-purposed for other applications, so long as scopes beginning with "DC_" are avoided.
The [MPE_CONFIG](/tables/mpe_config/) table provides a number of system configuration options, which apply to all users. The table may be re-purposed for other applications, so long as scopes beginning with "DC_" are avoided.
Currently used scopes include:
* DC
* DC_CATALOG
* [DC](/dcc-options/#dc-scope)
* [DC_CATALOG](/dcc-options/#dc_catalog-scope)
* [DC_EMAIL](/dcc-options/#dc_email-scope)
## DC Scope
### DC_EMAIL_ALERTS
Set to YES or NO to enable email alerts. This requires email options to be preconfigured (mail server etc).
### DC_MAXCELLS_WEBEDIT
By default, a maximum of 200,000 cells (observations multiplied by variables) can be edited in the browser at one time. This limit is applied together with [DC_MAXOBS_WEBEDIT](#dc_maxobs_webedit) - whichever is reached first.
The cell count matters because the cost of an EDIT screen load tracks the number of cells, not the number of rows. A 10 column table is comfortable at 250 rows, but 250 rows of a 1000 column table is 250,000 cells.
### DC_MAXOBS_WEBEDIT
By default, a maximum of 100 observations can be edited in the browser at one time. This number can be increased, but note that the following factors will impact performance:
By default, a maximum of 250 observations can be edited in the browser at one time. This number can be increased, but note that the following factors will impact performance:
* Number of configured [Validations](/dcc-validations)
* Browser type and version (works best in Chrome)
* Number (and size) of columns
* Speed of client machine (laptop/desktop)
A wide table can reach the [DC_MAXCELLS_WEBEDIT](#dc_maxcells_webedit) limit well before this one, so increase both together.
### DC_MAXOBS_WEBVIEW
By default, a maximum of 2000 observations can be viewed in the browser at one time. Please see previous section for items to consider if increasing this value.
### DC_REQUEST_LOGS
On SASjs Server and SAS9 Server types, at the end of each DC SAS request, a record is added to the [MPE_REQUESTS](/tables/mpe_requests) table. In some situations this can cause table locks. To prevent this issue from occuring, the `DC_REQUEST_LOGS` option can be set to `NO` (Default is `YES`).
### DC_RESTRICT_EDITRECORD
Setting YES will prevent the EDIT RECORD dialog appearing in the EDIT screen by removing the "Edit Row" option in the right click menu, and the "ADD RECORD" button in the bottom left.
Setting YES will prevent the EDIT RECORD dialog appearing in the EDIT screen by removing the "Edit Row" option in the right click menu, and the "ADD RECORD" button in the bottom left.
Anything other than YES will mean that the modal _is_ available.
Default=NO
Default=NO
### DC_RESTRICT_VIEWER
Set to YES to restrict the list of libraries and tables in VIEWER to only those explicitly set to VIEW in the MPE_SECURITY table. The default is NO (users can see all tables they already have permission to see).
@@ -53,7 +64,7 @@ Default=SYSTEM.
!!! note
If you have clients in different geographies loading excel in local formats, you can also address this issue by ensuring the locale of the windows _user_ profile is not set to the default (eg `English (United States)`). When leaving the DC_LOCALE as SYSTEM, the locale settings in SAS are not added or modified.
## DC_CATALOG Scope
### DC_IGNORELIBS
@@ -66,4 +77,36 @@ When running the [Refresh Data Catalog](/admin-services/#refresh-data-catalog) s
Number of rows to return for each HISTORY page. Default - 100. Increasing this will increase for all users. Using very large numbers here can result in a sluggish page load time. If you need large amounts of HISTORY data, it is generally better to extract it directly from the [MPE_REVIEW](/tables/mpe_review/) table.
## DC_EMAIL Scope
This section allows more fine grained control over the email configuration. Be sure that `DC_EMAIL_ALERTS` is set to `YES` (above) for these to activate.
![](img/mpe_config_dc_email.png)
Embedded macro variables are resolved at runtime.
### APPROVED_TEMPLATE
Sent when a change is approved. Available variables:
- ALERT_LIB: Library of table being edited
- ALERT_DS: table being edited
- FROM_USER: the user who made the approval
### REJECTED_TEMPLATE
Sent when a change is rejected. Available variables:
- ALERT_LIB: Library of table being edited
- ALERT_DS: table being edited
- FROM_USER: the user who made the rejection
- REVIEW_REASON_TXT: The text provided by the user who made the rejection
### SUBMITTED_TEMPLATE
Sent when a change is submitted. Available variables:
- ALERT_LIB: Library of table being edited
- ALERT_DS: table being edited
- FROM_USER: the user who made the submission
- SUBMITTED_TXT: The text provided by the user who made the submission.
docs/dcc-tables.md
+5 -1
View File
@@ -16,6 +16,9 @@ Adding tables to the Data Controller is a matter of configuration, specifically
![screenshot](img/configtable.png)
!!! note
`_____DELETE__THIS__RECORD_____` is a reserved name. The editor creates that column itself - it is the **Delete?** column - and never reads it from the table, so a target table must not contain a column of that name. The editor refuses to open a table that does, rather than let the two collide.
## MPE_TABLES Configuration Details
Each table to be edited in the Data Controller is represented by one record in `DATACTRL.MPE_TABLES`. The fields should be populated as follows:
@@ -212,7 +215,6 @@ The code is simply `%include`'d at the relevant point during backend execution.
* Physical, ie the full path to a `.sas` program on the physical server directory
* Logical, ie a Viya Job (SAS Drive), SAS 9 Stored Process (Metadata Folder) or SASJS Stored Program (SASjs Drive).
If the entry ends in `".sas"` it is assumed to be a physical, filesystem file. Otherwise, the source code is extracted from SAS Drive or Metadata.
To illustrate:
@@ -220,5 +222,7 @@ To illustrate:
* Physical filesystem (ends in .sas): `/opt/sas/code/myprogram.sas`
* Logical filesystem: `/Shared Data/stored_processes/mydatavalidator`
You can access the path to hook script at runtime (ie, in the HOOK script SAS code) using the `PGMLOC` macro variable.
!!! warning
Do not place your hook scripts inside the Data Controller (logical) application folder, as they may be inadvertently lost during a deployment (eg in the case of a backup-and-deploy-new-instance approach).
docs/dcc-validations.md
+98 -4
View File
@@ -23,20 +23,114 @@ It is possible to configure a number of other rules by updating the MPE_VALIDATI
## Configurable Checks
Check back frequently as we plan to keep growing this list of checks.
Check back frequently as we keep growing this list of checks. For how each rule treats a special missing value (`.A`-`.Z`, `._`), see [Special Missing Values](#special-missing-values).
|Rule Type|Example Value |Description|
|---|---|---|
|CASE|UPCASE|Will enforce the case of cell values. Valid values: UPCASE, LOWCASE, PROPCASE|
|NOTNULL|(defaultval)|Will prevent submission if null values are present. Optional - provide a default value.|
|MINVAL|1|Defines a minimum value for a numeric cell|
|MAXVAL|1000000|Defines a maximum value for a numeric cell|
|NOTNULL|(defaultval)|Will prevent submission if null values are present - a special missing counts as null. Optional - provide a default value.|
|MINVAL|1|Defines a minimum value for a numeric cell. A special missing value sorts below every number, so it fails a numeric floor, and the floor itself may be a special missing - see [Special Missing Values](#special-missing-values).|
|MAXVAL|1000000|Defines a maximum value for a numeric cell. A special missing value sorts below every number, so it passes a numeric ceiling, and the ceiling itself may be a special missing - see [Special Missing Values](#special-missing-values).|
|READONLY|(defaultval) |Renders the column read-only in the editor. The defaultval is used when rows are added. |
|HIDDEN|(defaultval) |Hides the column from the editor grid while still submitting its data. The defaultval is used when rows are added. |
|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc. Half-away-from-zero rounding is applied so `-0.5 -> -1` and `2.5 -> 3`. |
|NUMBER_FORMAT|`{"style":"currency","currency":"GBP"}` |Display-only [`Intl.NumberFormat` renderer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat). RULE_VALUE is the JSON options object passed straight to `Intl.NumberFormat`. Does not change the stored value. |
|HARDFORMULA|`= PRICE * VOLUME`|The cell displays a value computed from a formula, using other columns in the same row. The column is read-only - the user cannot override the result. See [Formula Rules](#formula-rules) below.|
|SOFTFORMULA|`= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )`|Like HARDFORMULA, but the user can override the computed value and type their own. See [Formula Rules](#formula-rules) below.|
|HARDREGEX|`^[A-Z]{3}$`|The cell value **must** match the regex pattern, otherwise submission is blocked and the cell is highlighted red. See [Regex Rules](#regex-rules) below.|
|SOFTREGEX|`^[A-Z]{3}$`|A cell value that does not match the regex pattern is highlighted yellow as a warning, but submission is **not** blocked. See [Regex Rules](#regex-rules) below.|
|HARDSELECT|sashelp.class.name|A distinct list of values (max 1000) are taken from this library.member.column reference, and the value **must** be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table.|
|SOFTSELECT|dcdemo.mpe_tables.libref|A distinct list of values (max 1000) are taken from this library.member.column reference, and the user-provided value may (or may not) be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table.|
|[HARDSELECT_HOOK](/dynamic-cell-dropdown)|/logical/folder/stpname|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User provided values **must** be in this list. Cannot be used alongside a SOFTSELECT_HOOK.|
|[SOFTSELECT_HOOK](/dynamic-cell-dropdown)|/physical/path/program.sas|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User-provided values may (or may not) be in this list. Cannot be used alongside a HARDSELECT_HOOK.|
## Special Missing Values
A SAS numeric variable has 28 missing values: the regular `.` and 27 special ones (`.A`-`.Z` and `._`). Data Controller accepts them in a numeric cell as a letter - `a` to `z` - or an underscore, with or without a leading period (`a`, `.a`, `_`, `._`), and stores the matching special missing in SAS. The period is optional, and a plain `.` stores the regular missing.
A numeric column that carries a date, datetime or time format is the exception. Those cells are edited through a date picker rather than the numeric editor, and the picker accepts only a date - so a special missing cannot be typed into one of them.
The rules treat a special missing as a value for some checks and as a missing value for others - and the range rules compare it in the order SAS gives it, below every number but ordered among the other missing values:
|Rule|What happens to a special missing|
|---|---|
|NOTNULL|Fails. A special missing is a missing value, so a NOT NULL (or primary key) constraint rejects it, and a real SAS NOT NULL constraint on the target table rejects it too.|
|MINVAL|Fails if the minimum is a number - a missing sorts below every number, so it is below any numeric floor. A minimum that is itself a special missing is compared in the order the missing values have among themselves, and can accept one.|
|MAXVAL|Passes - a missing sorts below every number, so it is below any numeric ceiling. A ceiling that is itself a special missing is compared in the missing values' own order, and can reject one.|
|CASE|Not applicable - it is a character rule.|
|ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.|
|HARDREGEX / SOFTREGEX|Checked against the pattern like any other value. Unlike blanks and the plain `.`, special missings are **not** exempt. See [Regex Rules](#regex-rules).|
|SOFTSELECT|Supports special missings - a soft dropdown never blocks a value.|
|HARDSELECT|Supports special missings - the value is matched against the dropdown list like any other value.|
|HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.|
In short, a special missing is rejected by NOTNULL, by a primary key column, and by a MINVAL whose floor is a number; a formula returns `#VALUE!` for it. It is fine in a column that carries a MAXVAL, a regex, a dropdown, or ROUND - and it can satisfy a MINVAL when the floor is expressed as a special missing.
A range rule compares in the order SAS itself uses, so a range written in special missings means what SAS would mean by it. That order puts every missing below every non-missing value, and orders the missing values as `._`, then the regular missing, then `.A` through `.Z`. So `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails a floor of `.A` because it sorts below it. A number sorts above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`.
A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there.
A range rule's own value is read as a number when it looks like one and as a missing value when it is a special missing. A value that is neither - a typo such as `..` or `AB` - satisfies nothing, so every cell in the column fails until the rule is corrected.
## Formula Rules
HARDFORMULA and SOFTFORMULA let you configure a column so that its value is automatically calculated from other columns in the same row - just like a spreadsheet formula. When a user opens the editor, the formula is evaluated live and the result is shown in each cell.
### Writing a formula
Formulas use column names, not cell references, so there is no need to know the grid layout. For example, if you have PRICE and VOLUME columns, a REVENUE column formula would be:
```
= PRICE * VOLUME
```
Each row calculates its own result - the PRICE in row 1 is multiplied by the VOLUME in row 1, the PRICE in row 2 by the VOLUME in row 2, and so on.
!!! note
Each column name in the formula must be surrounded by spaces (eg ` PRICE ` not `PRICE`) so it is recognised as a column reference rather than part of a function name. For example, `=MATCH( PRICE )` resolves the column reference, but `=MATCH(PRICE)` does not - PRICE is left unrecognised and the formula will error rather than using the column's value. Text inside quotes is left as-is.
### HARDFORMULA vs SOFTFORMULA
* **HARDFORMULA** - the column is read-only. The formula result is always shown and submitted. The user cannot change it.
* **SOFTFORMULA** - the cell shows the formula result but the user can type a different value if needed. If they do, their value is submitted instead.
### Editor behaviour
* When you paste a formula into the grid, column names are automatically translated so the formula works in its new position.
* A cell that is overwritten by a formula is flagged so you can revert it.
* Formula-looking values pasted from Excel are treated as plain data (not evaluated), unless you explicitly choose "Apply as formula".
### Special values
Formulas can reference three special values that are resolved at runtime:
* `DC.ROW_STATUS` - the current state of the row: `M` (Modified), `A` (Added), `D` (Deleted), or `U` (Unchanged). A newly-added row is `A` from the moment it is created - there is no transient state before that.
* `DC.USER_NAME` - the logged-in user id.
* `DC.ORIG_VALUE` - the original cell value before the current edit.
Example - show the current user id if the row is changed, otherwise keep the original value:
```
= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )
```
## Regex Rules
HARDREGEX and SOFTREGEX validate cell values against a SAS (Perl-style) regular expression provided in `RULE_VALUE` - the same syntax accepted by [PRXPARSE](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lefunctionsref/p0s9ilagexmjl8n1u7e1t1jfnzlk.htm).
Things to be aware of:
* Patterns are validated with `PRXPARSE` when the rule is saved - a post-edit check on MPE_VALIDATIONS itself will reject invalid patterns and list the offending columns.
* The pattern is evaluated in the browser using the JavaScript regex engine, which shares the same core syntax (character classes, quantifiers, groups, alternation, `^`/`$` anchors, `\d \w \s` etc). Stick to that common subset: Perl-only constructs such as inline modifiers `(?i)`, `\A` / `\z` anchors, possessive quantifiers (`a++`) and atomic groups (`(?>...)`) will pass the SAS-side PRXPARSE check but fail (and be silently ignored) in the frontend.
* The pattern is used **as authored** - it is not auto-anchored. If you want to match the entire cell value, include `^` and `$` yourself (eg `^[A-Z]{3}$`).
* Blank values are exempt from pattern matching on any column type - use the NOTNULL rule if you also need to enforce populated values. On numeric columns the plain SAS missing (`.`) is also exempt. Special missings (`.A`-`.Z`, `._`) are **not** exempt - being deliberately-set values, they are validated against the pattern like any other value, so on numeric columns make sure your pattern accommodates them (or avoid special missings). On character columns there is no missing-value concept: even `.` is treated as real text.
* Only one regex rule is ever applied per column. If a column has both a HARDREGEX and a SOFTREGEX rule, the SOFTREGEX rule is ignored entirely - even for values that pass the HARDREGEX - so a dual-rule column behaves exactly like a HARDREGEX-only column. For the same reason, the column-header info dropdown shows only the rule that is applied (the HARDREGEX pattern when both exist).
* Cells in rows that are marked for deletion are not validated / warned (except primary key columns, which still validate).
* In the unlikely event a pattern that fails in the browser slips through (see above), the frontend treats it as always-valid (no blocking, no warning) rather than breaking the editor.
* `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.
## Dropdowns
There are now actually FIVE places where you can configure dropdowns!
docs/dci-deploysas9.md
+13 -10
View File
@@ -14,6 +14,8 @@ There are two ways to deploy Data Controller on SAS 9:
* Full Deployment (preferred)
* Streaming (for quick demos)
Both fetch a deployment program from the [releases page](https://git.datacontroller.io/dc/dc/releases) - see the [downloads page](/downloads) for what each asset contains and how to verify it.
### Full Deployment
#### 1 - Deploy Stored Processes
@@ -27,27 +29,27 @@ filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/sa
%inc dc;
```
If you don't have internet access from SAS, download `sas9.sas` from [here](https://git.datacontroller.io/dc/dc/releases), and change the initial `compiled_apploc` and `compiled_serverName` macro variable assignments as necessary.
If you don't have internet access from SAS, download `sas9.sas` from [here](https://git.datacontroller.io/dc/dc/releases), verify it against the release `SHA256SUMS` file ([downloads page](/downloads/#verifying-a-download)), and change the initial `compiled_apploc` and `compiled_serverName` macro variable assignments as necessary.
#### 2 - Deploy the Frontend
The Data Controller frontend comes pre-built, and ready to deploy to the root of the SAS Web Server (mid-tier).
The Data Controller frontend comes pre-built, and ready to deploy to the SAS Web Server (mid-tier).
Deploy as follows:
1. Download the `frontend.zip` file from: [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases)
2. Unzip and place in the [htdocs folder of your SAS Web Server](https://sasjs.io/frontend-deployment/#sas9-deploy) - typically a subdirectory of: `!SASCONFIG/LevX/Web/WebServer/htdocs`.
1. Download the `frontend.zip` file from: [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases) (verify it against the release `SHA256SUMS` file - see the [downloads page](/downloads/#verifying-a-download))
2. Create a folder for Data Controller in the [htdocs folder of your SAS Web Server](https://sasjs.io/frontend-deployment/#sas9-deploy) - typically a subdirectory of: `!SASCONFIG/LevX/Web/WebServer/htdocs` - and unzip the archive **into** that folder. The archive holds the frontend files at its root, so `index.html` sits directly inside your folder (alongside the `images` folder and the compiled asset files) - for example `htdocs/dc/index.html`.
3. Open the `index.html` file and update the values in the `<sasjs>` tag as follows:
* `appLoc` - same as per SAS code in the section above
* `serverType` - change this to`SAS9`
* `serverType` - change this to`SAS9`
* `serverUrl` - Provide only if your SAS Mid Tier is on a different domain than the web server (protocol://SASMIDTIERSERVER:port)
* `loginMechanism` - set to `Redirected` if using SSO or 2FA
* `debug` - set to `true` to debug issues on startup (otherwise it's faster to leave it off and turn on in the application itself when needed)
The remaining properties are not relevant for a SAS 9 deployment and can be **safely ignored**.
You can now open the app at `https://YOURWEBSERVER/unzippedfoldername` (step 2 above) and follow the configuration steps (DC Physical Location and Admin Group) to complete deployment.
You can now open the app at `https://YOURWEBSERVER/<yourfoldername>` (the folder created in step 2 above) and follow the configuration steps (DC Physical Location and Admin Group) to complete deployment.
#### 3 - Run the Configurator
@@ -93,7 +95,7 @@ filename dc url "https://git.datacontroller.io/dc/dc/releases/download/vX.X.X/de
%inc dc;
```
If you don't have internet access from your SAS environment, just download `demostream_sas9.sas` from [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases) and modify the `appLoc` on line 2, as follows:
If you don't have internet access from SAS, download `demostream_sas9.sas` from [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases) (verify it against the release `SHA256SUMS` file - see the [downloads page](/downloads/#verifying-a-download)) and modify the `appLoc` on line 2, as follows:
![](img/sas9_apploc.png)
@@ -169,14 +171,15 @@ The full redeployment process is as follows:
- To a new metadata folder
- To a new frontend folder (if full deploy)
* _Delete_ the **new** DC library (metadata + physical tables)
* _Move_ the **old** DC library (metadata only) to the new DC metadata folder
* _Move_ the **old** DC library (metadata only) to the new DC metadata folder. You will need to use DI Studio to do this (as you can't _move_ objects using SAS Management Console)
* Copy the _content_ of the old `services/public/Data_Controller_Settings` STP to the new one
- This will link the new DC instance to the old DC library / logs directory
- It will also re-apply any site-specific DC mods
* Run any/all DB migrations between the old and new DC version
- See [migrations](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/db/migrations) folder
- See [migrations](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/db/migrations) folder
- Update the metadata of the SAS Library, using DI Studio, to capture the model changes
* Test and make sure the new instance works as expected
* Delete (or rename) the **old** instance
* Delete (or rename) the **old** instance
- Metadata + frontend, NOT the underlying DC library data
* Rename the new instance so it is the same as the old
- Both frontend and metadata
docs/dci-deploysasviya.md
-185
View File
@@ -1,185 +0,0 @@
---
layout: article
title: DC SAS Viya Deployment
description: How to deploy Data Controller in a production SAS Viya environment
og_image: https://docs.datacontroller.io/img/dci_deploymentdiagramviya.png
---
# SAS Viya Deployment
## Overview
Data Controller for SAS Viya consists of a frontend, a set of Job Execution Services, a staging area, a Compute Context, and a database library. The library can be a SAS Base engine if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS.
A database that supports concurrent access is highly recommended.
## Prerequisites
### System Account
Data Controller makes use of a system account for performing backend data updates and writing to the staging area. This needs to be provisioned in advance using the Viya admin-cli. The process is well described here: [https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-3-5-Compute-Server-Service-Accounts/ta-p/620992](https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-3-5-Compute-Server-Service-Accounts/ta-p/620992)
### Database
Whilst we do recommend that Data Controller configuration tables are stored in a database for concurrency reasons, it is also possible to use a BASE engine library, which is adequate if you only have a few users.
To migrate the control library to a database, first perform a regular deployment, and afterwards you can generate the DDL and update the settings file..
Make sure the system account (see above) has full read / write access.
!!! note
"Modify schema" privileges are not required.
### Staging Directory
All deployments of Data Controller make use of a physical staging directory. This is used to store logs, as well as CSV and Excel files uploaded by end users. This directory should NOT be accessible by end users - only the SAS system account requires access to this directory.
A typical small deployment will grow by a 5-10 mb each month. A very large enterprise customer, with 100 or more editors, might generate up to 0.5 GB or so per month, depending on the size and frequency of the Excel EUCs and CSVs being uploaded. Web modifications are restricted only to modified rows, so are typically just a few kb in size.
## Deployment Diagram
The below areas of the SAS Viya platform are modified when deploying Data Controller:
<img src="/img/dci_deploymentdiagramviya.svg" height="350" style="border:3px solid black" >
## Deployment
Data Controller deployment is split between 3 deployment types:
* Demo version
* Full Version (manual deploy)
* Full Version (automated deploy)
<!--
## Full Version - Manual Deploy
-->
There are several parts to this proces:
1. Create the Compute Context
2. Deploy Frontend
4. Prepare the database and update settings (optional)
5. Update the Compute Context autoexec
### Create Compute Context
The Viya Compute context is used to spawn the Job Execution Services - such that those services may run under the specified system account, with a particular autoexec.
We strongly recommend a dedicated compute context for running Data Controller. The setup requires an Administrator account.
* Log onto SASEnvironment Manager, select Contexts, View Compute Contexts, and click the Create icon.
* In the New Compute Context dialog, enter the following attributes:
* Context Name
* Launcher Context
* Attribute pairs:
* reuseServerProcesses: true
* runServerAs: {{the account set up [earlier](#system-account)}}
* Save and exit
!!! note
XCMD is NOT required to use Data Controller.
### Deploy frontend
Unzip the frontend into your chosen directory (eg `/var/www/html/DataController`) on the SAS Web Server. Open `index.html` and update the following inside `dcAdapterSettings`:
- `appLoc` - this should point to the root folder on SAS Drive where you would like the Job Execution services to be created. This folder should initially, NOT exist (if it is found, the backend will not be deployed)
- `contextName` - here you should put the name of the compute context you created in the previous step.
- `dcPath` - the physical location on the filesystem to be used for staged data. This is only used at deployment time, it can be configured later in `$(appLoc)/services/settings.sas` or in the autoexec if used.
- `adminGroup` - the name of an existing group, which should have unrestricted access to Data Controller. This is only used at deployment time, it can be configured later in `$(appLoc)/services/settings.sas` or in the autoexec if used.
- `servertype` - should be SASVIYA
- `debug` - can stay as `false` for performance, but could be switched to `true` for debugging startup issues
- `useComputeApi` - use `true` for best performance.
![Updating index.html](img/viyadeployindexhtml.png)
Now, open https://YOURSERVER/DataController (using whichever subfolder you deployed to above) using an account that has the SAS privileges to write to the `appLoc` location.
You will be presented with a deployment screen like the one below. Be sure to check the "Recreate Database" option and then click the "Deploy" button.
![viya deploy](img/viyadeployauto.png)
Your services are deployed! And the app is operational, albeit still a little sluggish, as every single request is using the APIs to fetch the content of the `$(appLoc)/services/settings.sas` file.
To improve responsiveness by another 700ms we recommend you follow the steps in [Update Compute Context Autoexec](/dci-deploysasviya/#update-compute-context-autoexec) below.
### Deploy Database
If you have a lot of users, such that concurrency (locked datasets) becomes an issue, you might consider migrating the control library to a database.
The first part to this is generating the DDL (and inserts). For this, use the DDL exporter as described [here](/admin-services/#export-database). If you need a flavour of DDL that is not yet supported, [contact us](https://datacontroller.io/contact/).
Step 2 is simply to run this DDL in your preferred database.
Step 3 is to update the library definition in the `$(appLoc)/services/settings.sas` file using SAS Studio.
### Update Compute Context Autoexec
First, open the `$(appLoc)/services/settings.sas` file in SAS Studio, and copy the code.
Then, open SASEnvironment Manager, select Contexts, View Compute Contexts, and open the context we created earlier.
Switch to the Advanced tab and paste in the SAS code copied from SAS Studio above.
It will look similar to:
```
%let DC_LIBREF=DCDBVIYA;
%let DC_ADMIN_GROUP={{YOUR DC ADMIN GROUP}};
%let DC_STAGING_AREA={{YOUR DEDICATED FILE SYSTEM DRIVE}};
libname &dc_libref {{YOUR DC DATABASE}};
```
To explain each of these lines:
* `DC_LIBREF` can be any valid 8 character libref.
* `DC_ADMIN_GROUP` is the name of the group which will have unrestricted access to Data Controller
* `DC_STAGING_AREA` should point to the location on the filesystem where the staging files and logs are be stored
* The final libname statement can also be configured to point at a database instead of a BASE engine directory (contact us for DDL)
If you have additional libraries that you would like to use in Data Controller, they should also be defined here.
<!--
## Full Version - Automated Deploy
The automated deploy makes use of the SASjs CLI to create the dependent context and job execution services. In addition to the standard prerequisites (a registered viya system account and a prepared database) you will also need:
* a local copy of the [SASjs CLI](https://sasjs.io/sasjs-cli/#installation)
* a Client / Secret - with an administrator group in SCOPE, and an authorization_code GRANT_TYPE. The SASjs [Viya Token Generator](https://github.com/sasjs/viyatoken) may help with this.
### Prepare the Target and Token
To configure this part (one time, manual step), we need to run a single command:
```
sasjs add
```
A sequence of command line prompts will follow for defining the target. These prompts are described [here](https://sasjs.io/sasjs-cli-add/). Note that `appLoc` is the SAS Drive location in which the Data Controller jobs will be deployed.
### Prepare the Context JSON
This file describes the context that the CI/CD process will generate. Save this file, eg as `myContext.json`.
```
{
"name": "DataControllerContext",
"attributes": {
"reuseServerProcesses": true,
"runServerAs": "mycasaccount"
},
"environment": {
"autoExecLines": [
"%let DC_LIBREF=DCDBVIYA;",
"%let DC_ADMIN_GROUP={{YOUR DC ADMIN GROUP}};",
"%let DC_STAGING_AREA={{YOUR DEDICATED FILE SYSTEM DRIVE}};",
"libname &dc_libref {{YOUR DC DATABASE}};",
],
"options": []
},
"launchContext": {
"contextName": "SAS Job Execution launcher context"
},
"launchType": "service",
}
```
### Prepare Deployment Script
The deployment script will run on a build server (or local desktop) and execute as follows:
```
# Create the SAS Viya Target
sasjs context create --source myContext.json --target myTarget
```
-->
docs/dci-troubleshooting.md
+134 -1
View File
@@ -10,6 +10,16 @@ og_image: https://docs.datacontroller.io/img/cannotimport.png
## Overview
[Let us know](https://datacontroller.io/contact/) if you experience an installation problem that is not described here!
## max number of active processes has been reached for the user
On Viya versions 2025 or later you may get the following message in the network response:
`Unable to create compute server session. Unable to complete the launch request, max number of active processes has been reached for the user: user=USERNAME limit=10`
This limit should be set to at least 20 or 30 due to the way the [sasjs/adapter](https://github.com/sasjs/adapter) works (by prelaunching sessions to improve responsiveness).
A guide for making the configuration change is available [here](https://communities.sas.com/t5/SAS-Communities-Library/Limit-a-user-s-simultaneous-compute-server-processes-in-SAS-Viya/ta-p/761820).
## Internet Explorer - blank screen
If you have an older, or 'locked down' version of Internet Explorer you may get a blank / white screen when navigating to the Data Controller url. To fix this, click settings (cog icon in top right), *Compatibility View settings*, and **uncheck** *Display intranet sites in Compatibility view* as follows:
![menu](img/dci-trouble1.png)
@@ -98,8 +108,131 @@ This can happen if you enter the wrong `serverName` when deploying the SAS progr
The error may also be thrown due to an encoding issue - changing to a UTF-8 server has helped at least one customer.
## Displayed timestamps are in UTC (or the wrong timezone)
Data Controller records timestamps (such as the SUBMITTED column on the Submitted screen, and the audit history) using the SAS session clock, via the `datetime()` function. That function returns the time of the operating system, adjusted by the [TIMEZONE= system option](https://documentation.sas.com/doc/en/pgmsascdc/default/lesysoptsref/p15siqs0s00e50n1wuuvygzkr14r.htm) if it is set. The value is then displayed in the frontend exactly as stored, without conversion.
On Viya, the SAS compute sessions that run Data Controller jobs are started inside Kubernetes containers whose clock is UTC by default, and which inherit no timezone from the host machine. If the TIMEZONE= option is not set for the compute context used by Data Controller, every timestamp DC records and displays is UTC. Other SAS products can appear unaffected because clients such as SAS Studio create their own compute sessions and pass the browser timezone / locale, whereas Data Controller submits jobs to its own shared compute context, which gets the raw container clock.
To confirm the current state, run the following in a SAS session under the Data Controller compute context (or check the DC job log, where `_DEBUG` output shows the same values):
```sas
proc options option=timezone;
run;
%put &=SYSTIMEZONEIDENT &=SYSTIMEZONEOFFSET;
```
If TIMEZONE is blank (and SYSTIMEZONEOFFSET is 0), the session is on UTC. To fix it, ask your Viya administrator to set the timezone for the compute context used by Data Controller (in SAS Environment Manager, edit the context's Autoexec, or set the context's SAS options):
```sas
options timezone='Europe/Berlin';
```
Then recycle any existing compute sessions - hot sessions keep the old setting until they terminate.
!!! note
Use a region/area time zone ID such as `Europe/Berlin` rather than a fixed offset such as `GMT+2`. Fixed offsets do not follow daylight saving time, so the display would be 1 hour off in winter (CET = UTC+1).
!!! note
Changing the timezone affects new timestamps only. Previously recorded submissions keep the UTC values that were stored when they were created.
## Determining Application Version
The app version is bundled into the frontend during the release, and is visible by clicking your username in the top right.
You can also determine the app version (and SASjs Version, and build time) by opening browser Development Tools and running `appinfo()` in the console.
You can also determine the app version (and SASjs Version, and build time) by opening browser Development Tools and running `appinfo()` in the console.
## Support Diagnostics
The frontend sends two input tables with the startup service and with the services that execute customer-provided code - the [hook scripts](macros.md) (pre/post edit and approve hooks) and the dynamic cell dropdown programs. They are available in those services as `work.browser_info` and `work.browser_url_vars`, so a support ticket can be diagnosed from the job log without asking the user follow-up questions, and a hook can adapt to its context. Other services do not receive them.
The services that receive the tables are: `public/startupservice`, `editors/getdata`, `editors/getdynamiccolvals`, `editors/stagedata`, `editors/loadfile`, `editors/restore` and `auditors/postdata`.
`editors/loadfile` is the exception in practice: the app reaches it through the adapter's file-upload call (multipart), which carries no input tables, so the service - and the post edit hook it runs - does not receive them in normal use. It is on the list for completeness, and would receive them if the service were ever invoked through the ordinary request path.
### browser_info
A single-row table with these columns:
| Column | Description |
|---|---|
| `url` | The URL of the Data Controller page itself (the iframe), not the document embedding it. When the editor is embedded in a report, the embedding URL is visible here. |
| `referrer` | The embedding document, from `document.referrer`. For an embedded report this is the report URL, so the embedding report can be told apart from the editor URL. |
| `timezone` | The browser's IANA timezone name, e.g. `Europe/Berlin`. |
| `tz_offset` | The browser's UTC offset in minutes, as `Date.getTimezoneOffset()` returns it (positive west of UTC, so `Europe/Berlin` in summer is -120). |
| `locale` | The browser locale, e.g. `en-GB`. |
| `dc_version` | The Data Controller build, as shown by `appinfo()`. |
| `adapter_version` | The `@sasjs/adapter` version the client was built with. |
| `browser` | The browser family, parsed from the user agent. |
| `browser_version` | The browser version. |
| `platform` | The operating system, parsed from the user agent. |
| `user_agent` | The raw `navigator.userAgent` string. |
### browser_url_vars
The page URL parameters as one row per parameter, so a SAS program can read a parameter by name rather than parse the `url` string:
| Column | Description |
|---|---|
| `name` | The URL parameter name. |
| `value` | The URL parameter value. |
Parameters from both the search string and the hash query string (Angular routes carry them after the `#`) are included; on a name collision the search string value wins. The table is sent only when the page URL has at least one parameter.
On SAS 9 and Viya the app is reached through the platform's own job URL, so the platform's parameters (`_FILE`, `_program`, `_debug`, `_webout` and friends) are in the table alongside DC's. Read a parameter by name rather than assuming every row belongs to Data Controller - `embed` and `labels` are DC's, `_FILE` and `_program` are the platform's.
The values in both tables are supplied by the client, so treat them as diagnostics hints rather than a security boundary.
To see the tables, run any of the services above with debug on. The debug value depends on the path: `&_debug=131` on the Compute API and SAS 9, or `&_debug=128` on the Viya web (JES) path with `runAsTask` enabled - which is what the frontend sends there, so turning debug on in the app is enough. Either value makes the session initialisation write the tables to the job log:
```
NOTE: MPEINIT: work.browser_url_vars:
name=embed value=va
NOTE: MPEINIT: work.browser_info:
url=... referrer=... timezone=Europe/Berlin tz_offset=-120 locale=en-GB
dc_version=7.15.0 adapter_version=4.19.0 browser=Chrome browser_version=120.0
platform=Linux user_agent=...
```
A service called directly (for instance from a URL, or by a script), or by an older frontend, has no `work.browser_info` - the log then records that fact instead.
### Reading the tables from a hook
A [hook script](macros.md) is `%include`d into the running service, so it shares the service's WORK library and can read the tables directly. Take an embedded editor opened at:
```
https://datacontroller.io/#/editor/SASHELP.CLASS?labels=true&embed=va
```
`browser_url_vars` then holds a row per parameter - `labels=true` and `embed=va` - so a hook can pick up either by name:
```sas
%let labels=N;
%let embed=N;
%if %sysfunc(exist(work.browser_url_vars)) %then %do;
proc sql noprint;
select value into :labels
from work.browser_url_vars
where name = 'labels';
select value into :embed
from work.browser_url_vars
where name = 'embed';
quit;
%end;
```
To branch on what the client reports:
```sas
%if %sysfunc(exist(work.browser_info)) %then %do;
data _null_;
set work.browser_info;
call symputx('dc_timezone', timezone);
call symputx('dc_locale', locale);
run;
%end;
```
!!! warning
Guard with `%sysfunc(exist())` before reading. A service called directly, or by a frontend older than this feature, has no `work.browser_info` at all, so an unguarded `set` or `select` fails the job.
docs/dcu-datacatalog.md
+68 -4
View File
@@ -1,7 +1,25 @@
# Data Controller for SAS: Data Catalog
Data Controller collects information about the size and shape of the tables and columns. The Catalog does not contain information about the data content (values).
---
layout: article
title: DC Data Catalog
description: Catalog the Libraries, Tables, Columns, SAS Catalogs, and associated Objects in your SAS estate
og_title: DC Data Catalog Documentation
og_image: /img/catalog.png
---
The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
# DC Data Catalog
In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as:
- Library Properties (size, schema, path, number of tables)
- Table Properties (size, number of columns, primary keys)
- Variable Properties (presence in a primary key, constraints, position in the dataset)
- SAS Catalog Properties (number of entries, created / modified datetimes)
- SAS Catalog Object properties (entry name, type, description, created / modified datetimes)
The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared? Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
The Catalog does **not** contain information about the data content (values). It is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables. The rest is stored on the MPE_DATACATALOG_XXX tables.
@@ -20,14 +38,60 @@ Table attributes are split between those that change infrequently (eg PK_FIELDS)
Variable attributes come from dictionary tables with an extra PK indicator. A PK is identified by the fact the variable is within an index that is both UNIQUE and NOTNULL. Variable names are always uppercase.
### Catalogs & Objects
This info comes from the dictionary.catalogs table. The catalog created / modified time is considered to be the earliest created time / latest modified time of the underlying objects.
## Assumptions
The following assumptions are made:
* Data _Models_ (eg attributes) are not sensitive. If so the catalog tables should be disabled.
* Users can see all tables in the libraries they can access. The refresh process will close out any tables that are not found, if the user can see at least one table in a library.
* For a particular site, libraries are unique on LIBREF.
* For a particular site, libraries are unique on LIBREF.
If you have duplicate librefs, specific table security setups, or sensitive models - contact us.
## Refreshing the Data Catalog
The update process for INDIVIDUAL libraries can be run by any user, and is performed in the VIEW menu by expanding a library definition and clicking the refresh icon next to the library name.
![](./img/catalogrefresh.png)
Members of the admin group may run the refresh process for ALL libraries by clicking the REFRESH button on the System page.
Under the hood, the refresh is executed by three SAS macros:
- `mpe_refreshlibs` - library attributes (engine, paths, permissions, owners, schemas, metadata name / id) into [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs)
- `mpe_refreshtables` - table and column attributes (including primary key detection from constraints and unique not-null indexes) into the DATACATALOG_TABS / VARS tables, and sizes / row counts into the DATASTATUS tables
- `mpe_refreshcatalogs` - SAS Catalog and object attributes
These run inside the `refreshlibinfo` service (single library, any user) and the `refreshlibs` / `refreshcatalog` services (all libraries, admins only).
When doing a full scan, the following LIBREFS are ignored:
* 'CASUSER'
* 'MAPSGFK'
* 'SASUSER'
* 'SASWORK
* 'STPSAMP'
* 'TEMP'
* `WORK'
Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them. This can be useful where there are connection issues for a particular library.
Be aware that the scan process can take a long time if you have a lot of tables!
Output tables (all SCD2):
* [MPE_DATACATALOG_CATS](/tables/mpe_datacatalog_cats) - SAS Catalog list
* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
* [MPE_DATACATALOG_OBJS](/tables/mpe_datacatalog_objs) - SAS Catalog object attributes
* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
* [MPE_DATASTATUS_CATS](/tables/mpe_datastatus_cats) - Frequently changing catalog attributes (such as created / modified datetimes and number of entries)
* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
* [MPE_DATASTATUS_OBJS](/tables/mpe_datastatus_objs) - Frequently changing catalog object attributes (such as created / modified datetimes and library concatenation level)
* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)
docs/dcu-tableviewer.md
+15 -1
View File
@@ -1,7 +1,7 @@
# Data Controller for SAS: Viewer
The viewer screen provides a raw view of the underlying table.
Choose a library, then a table, and click view to see the first 5000 rows.
Choose a library, then a table, and click view to see the first 500 rows.
A filter option is provided should you wish to view a different section of rows.
The following libraries will be visible:
@@ -16,6 +16,18 @@ Row and Column level security can also be applied in VIEW mode, as can additiona
A single search box can be used to make a full table search on any character or numeric value, using this [macro](https://core.sasjs.io/mp__searchdata_8sas.html).
The search covers **every** column of the table at once, so you do not need to know which column holds the value you are looking for:
- **Character search** (the default) is a case sensitive CONTAINS match against every character column, so `Smith` finds `Smithson` and `smith` finds `Goldsmith`, but `smith` will not find `Smithson`.
- **Numeric search** (tick the Numeric box) is an exact match against every numeric column, so `42` will not find `420` or `4210`.
- If a filter is applied, the search runs within the filtered rows rather than the whole table.
- The grid is capped by the [`DC_MAXOBS_WEBVIEW`](/dcc-options/#dc_maxobs_webview) option (500 rows by default).
- For a search, the cap applies to the row count as well: a search that matches more rows than the cap returns 500 of them, and the count shown next to the table name is the number of rows returned, not the total number of matches.
- Without a search, the count shown next to the table name is the full number of rows in the filtered view, so it can be higher than the number of rows displayed - a table of 1,200 rows reports `1,200 rows` above a grid holding the first 500.
- A search that matches nothing shows a "No data found with given conditions" panel rather than an empty grid.
![Full table search](img/full-table-search.png)
<iframe width="560" height="315" src="https://www.youtube.com/embed/i27w-xq85WQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## Options
@@ -37,6 +49,8 @@ The Download button gives several options for obtaining the current view of data
Note - if the table is registered in Data Controller as being TXTEMPORAL (SCD2) then the download option will prefilter for the _current_ records and removes the valid from / valid to variables. This makes the CSV suitable for DC file upload, if desired.
Note that all the above items are exported from backend. There is also a _frontend_ context menu that can be used in the VIEW screen (right click menu) for quick export into CSV or Excel formats.
### Web Query URL
This option gives you a URL that can be used to import data directly into third party tools such as Power BI or Microsoft Excel (as a "web query"). You can set up a filter, eg for a particular month, and refresh the query on demand using client tooling such as VBA.
docs/deploy-viya.md
+178
View File
@@ -0,0 +1,178 @@
---
layout: article
title: DC SAS Viya Deployment
description: How to deploy Data Controller in a production SAS Viya environment
og_image: https://docs.datacontroller.io/img/dci_deploymentdiagramviya.png
---
# SAS Viya Deployment
## Overview
Data Controller for SAS Viya consists of a static web frontend, a set of Viya Jobs, a staging area (physical directory), a Compute Context, and a Library.
## Prerequisites
### System Account
Data Controller makes use of a system account for performing backend data updates and writing to the staging area. This needs to be provisioned in advance using the Viya admin-cli. The process is well described here: [https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-3-5-Compute-Server-Service-Accounts/ta-p/620992](https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-3-5-Compute-Server-Service-Accounts/ta-p/620992)
### Library
Currently, all of our customers are using the standard BASE engine library for the control tables. However it is possible to use a database instead. To migrate the control library to a database, first perform a regular deployment, and afterwards you can generate the DDL. For this, use the DDL exporter as described [here](/admin-services/#export-database). If you need a flavour of DDL that is not yet supported, [contact us](https://datacontroller.io/contact/).
Make sure the system account (see above) has full read / write access.
!!! note
"Modify schema" privileges are not required.
### Staging Directory
All deployments of Data Controller make use of a physical staging directory. This is used to store logs, as well as CSV and Excel files uploaded by end users. This directory should NOT be accessible by end users - only the SAS system account requires access to this directory.
A typical small deployment will grow by a 5-10 mb each month. A very large enterprise customer, with 100 or more editors, might generate up to 0.5 GB or so per month, depending on the size and frequency of the Excel EUCs and CSVs being uploaded. Web modifications are restricted only to modified rows, so are typically just a few kb in size.
## Deployment Diagram
The below areas of the SAS Viya platform are modified when deploying Data Controller:
<img src="/img/dci_deploymentdiagramviya.svg" height="350" style="border:3px solid black" >
!!! note
The "streaming" version of Viya uses the files API for web content, so there is no need for the web server component.
## Deployment
Data Controller deployment is split between 2 deployment types:
* Streaming (web content served from SAS Drive)
* Separated (web content served from dedicated web server)
For most customers, the streaming approach is preferred, as it makes the deployment much simpler.
There are several parts to this proces:
1. Create the Compute Context
2. Deploy Services
3. Configure Frontend
4. First Launch
5. Optmisation
### Create Shared Compute Context
We strongly recommend a dedicated compute context for running Data Controller. The setup requires an Administrator account.
* Log onto SASEnvironment Manager, select Contexts, View Compute Contexts, and click the Create icon.
* In the New Compute Context dialog, enter the following attributes:
* Context Name
* Launcher Context
* Attribute pairs:
* reuseServerProcesses: true
* runServerAs: {{the account set up [earlier](#system-account)}}
* Save and exit
!!! note
XCMD is NOT required to use Data Controller.
A group should be defined in Environment Manager that has the "create session" permission on this context.
To avoid giving users the ability to run code on that context (and shared system account) in SAS Studio and other apps, this permission should have the following condition attached: `clientId() == 'sas.jobExecution'`
### Deploy Services
Services are deployed by running a SAS program.
**Streaming Deploy (BACKEND + FRONTEND):**
Run the following in SAS Studio:
```sas
%let apploc=/Public/DataController; /* desired SAS Drive location */
filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/viya.sas";
%inc dc;
```
**Separated Deploy (BACKEND ONLY):**
Run the following in SAS Studio:
```sas
%let apploc=/Public/DataController; /* desired SAS Drive location */
filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/viya_noweb.sas";
%inc dc;
```
### Configure Frontend
**Streaming Deploy:**
At the end of the SAS log from Step 2, there will be a link (`YOURSAS.SERVER/SASJobExecution?_file=/YOUR/APPLOC/services/DC.html`). Open this in **SASJobExecution** (not SAS Studio) to perform the configuration (below).
**Separated Deploy:**
Unzip the frontend into your chosen directory (eg `/var/www/html/DataController`) on the SAS Web Server. Edit `index.html` to perform the configuration (below).
**index.html**
The following attributes may be updated in the index.html file. For streaming deploy, be sure to use the JobExecution app (not SAS Studio) for correct file type handling.
- `appLoc` - the root folder where the Jobs were deployed in step 2
- `contextName` - here you should put the name of the compute context you created in step 1
- `servertype` - should be SASVIYA
- `debug` - can stay as `false` for performance, but could be switched to `true` for debugging startup issues
- `useComputeApi` - Setting `true` will give the best performance due to the use of [hot sessions](https://github.com/sasjs/adapter#using-the-compute-api) created client-side by the SASjs adapter. This is great for demo purposes, or when running a single user DC instance, however for typical enterprise use we recommend `false` or `null` so that the compute context usage can be restricted as described [above](/deploy-viya/#create-shared-compute-context)
- `runAsTask` - setting `true` will trigger jobs as Viya Compute Tasks. Be sure to increase the default expiry to 30 seconds (it is 5 by default) and set `useComputeApi` to `null`. For scaling options, see [SAS docs](https://documentation.sas.com/doc/en/sasadmincdc/v_074/calsrvpgm/p059rp0q82tvpzn1hk9x26486do5.htm#p1ktrn1coq0rx1n1ux8bjqh8jx4h).
### First Launch
Now the services are deployed (including the service which creates the staging area) we can open the Data Controller web interface and make the necessary configurations:
* dcpath - physical path for deployment (will contain SAS datasets and a subfolder for staged content)
* Admin Group - the members of this group will have full access to Data Controller
* Compute Context - the context configured in Step 1
### Optimisation
At this point, every DC request will read the `services/public/settings.sas` file to get the DC library (and other) settings. To avoid these API calls (which will speed up the app) we can simply move this code to the autoexec. Steps as follows:
First, open the `$(appLoc)/services/settings.sas` file in SAS Studio, and copy the code.
Then, open SASEnvironment Manager, select Contexts, View Compute Contexts, and open the context we created earlier.
Switch to the Advanced tab and paste in the SAS code copied from SAS Studio above.
It will look similar to:
```
%let DC_LIBREF=DCDBVIYA;
%let DC_ADMIN_GROUP={{YOUR DC ADMIN GROUP}};
%let DC_STAGING_AREA={{YOUR DEDICATED FILE SYSTEM DRIVE}};
libname &dc_libref {{YOUR DC DATABASE}};
```
To explain each of these lines:
* `DC_LIBREF` can be any valid 8 character libref.
* `DC_ADMIN_GROUP` is the name of the group which will have unrestricted access to Data Controller
* `DC_STAGING_AREA` should point to the location on the filesystem where the staging files and logs are be stored
* The final libname statement can also be configured to point at a database instead of a BASE engine directory (contact us for DDL)
If you have additional libraries that you would like to use in Data Controller, they should also be defined here.
## Redeployment
To update DC, just deploy it as a fresh instance, then move the new config across, as follows:
1. Do a full deploy to a completely new location
2. Copy the contents of the old `$(appLoc)/services/settings.sas` file to the new SAS Folder location
3. Delete the new physical directory (just created) as it is now replaced with the old one (per step 2)
4. Either delete or rename the old SAS Folder location (appLoc), and rename the new SAS Folder location to equal the old one
5. If using a dedicated web frontend, backup/rename so that the new web server location matches the old one
6. Run any migrations relevant to the release, as defined [here](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/db/migrations)
## Visual Analytics
It is possible to embed a Data Controller table within SAS Visual Analytics by simply pasting the URL.
To make the portlet more visually appealing, the Data Controller title bar can be removed by adding `?embed=true` to the URL. When opening in a new window, the title bar will be gone.
For a deeper VA integration - where report row selections drive filters and column visibility in the Data Controller editor - append `?embed=va` instead. See the [SAS Visual Analytics Embed](/embed-va/) page for details on filter modes, configuration, and debugging.
docs/derived-variables.md
+131
View File
@@ -0,0 +1,131 @@
---
layout: article
title: Derived Variables
description: Derived variables are computed columns that appear in the EDIT screen and can be filtered on, without being stored in the target table.
---
# Derived Variables
A derived variable is a column that does not exist in the target table, but is computed when the table is loaded into the [EDIT screen](/dc-userguide/#editor). It is read-only, it appears after the physical columns, and it can be filtered on like any other column.
Derived variables are useful when users need to work with a value that is a function of the stored data - a banding, a concatenation, a calculation - without changing the table, the load process, or the ETL.
The values are **not stored**. They are computed on every read, and nothing derived is written back when a change is submitted or approved.
## What it looks like
The table below is shown in the EDIT screen. `LINE_VALUE` is `QTY * PRICE` and `REGION` is derived from `STATE`; neither exists in the table itself, and both are read-only.
![](./img/derived-grid.png)
Because the derived columns are created before the filter is applied, they can be filtered like any other column. Here `REGION = 'West'` selects the two western states, and the filter bar names the derived variable:
![](./img/derived-filter-region.png)
A numeric derived column works the same way - `LINE_VALUE > 50` leaves the one row that satisfies it:
![](./img/derived-filter-value.png)
## How it works
The statements are written into a DATA step view over the target table, and the filter is applied to that view rather than to the table:
```sas
data work.vw_derived / view=work.vw_derived;
set &libds;
attrib REGION length=$8 label='...' format=$8.;
REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
run;
data out;
set work.vw_derived;
where %inc filtref;;
run;
```
A `WHERE` statement applies to the data set named on the `SET` statement, so in a single step it cannot see a variable created later in the same step. Putting the derived variables on the view means they are on the input to the step that filters, which is what makes them filterable at all.
## Configuration
Derived variables are configured per table in the [MPE_DERIVED](/tables/mpe_derived/) control table, which is itself a Data Controller table - so it is maintained through the EDIT screen with the usual review and approve workflow. The three variables shown above, and nothing else, are the three rows below:
![](./img/derived-config.png)
Each row defines one variable:
- `LIBREF` / `DSN` - the target table
- `VAR_NM` - the name of the new column
- `VAR_DEPS` - every input the expression uses: the physical columns and any derived variables, space separated
- `VAR_TYPE` - `C` for character, `N` for numeric
- `VAR_LEN` - character length, or numeric length (1-8)
- `VAR_FMT` - optional, eg `DATE9.` or `DOLLAR12.2`
- `VAR_LABEL` - optional; when blank the label is `<VAR_NM>=<expression>`
- `VAR_LOGIC` - the SAS expression that produces the value
- `ACTIVE` - set to 1 to apply the variable
A simple example, deriving a region from a state:
```
VAR_NM REGION
VAR_DEPS STATE
VAR_TYPE C
VAR_LEN 8
VAR_FMT (blank)
VAR_LABEL (blank)
VAR_LOGIC ifc(STATE in ('CA','OR','WA'),'West','East')
ACTIVE 1
```
A second variable can build on it - its dependencies include the physical column and the derived one:
```
VAR_NM REGION_LABEL
VAR_DEPS REGION STATE
VAR_TYPE C
VAR_LEN 14
VAR_FMT (blank)
VAR_LABEL Region and state
VAR_LOGIC cats(REGION,'-',STATE)
ACTIVE 1
```
There is no sequence number to maintain. The statements are emitted in dependency order, so `REGION` is always assigned before `REGION_LABEL`, however the rows are ordered in the table.
For each row, Data Controller generates the following statements when the table is read:
```sas
attrib REGION length=$8 label='REGION=ifc(STATE in (''CA'',''OR'',''WA''),''West'',''East'')' format=$8.;
REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
```
The label is `VAR_LABEL` when one is supplied, otherwise the variable name, an equals sign and the expression - `REGION=ifc(...)` - so the column shows both what it is and where it comes from when labels are switched on. Either way the label is truncated to the maximum label length, on a boundary that cannot split a doubled quote.
The type and length are declared, not inferred, so a value is truncated exactly as you specify. If `VAR_FMT` is left blank, character variables default to `$<length>.` and numeric variables to `<length>.`; a date, time or datetime format makes the column render (and filter) as a date, time or datetime, the same way a stored column does.
## Filtering
The derived variable is created before the filter is applied, so it can be used in the [filter](/filter/) like any other column - including its list of values. A filter on a derived column is stored and re-applied in the normal way, and the value list for a derived column is built from the table under the rest of the filter, exactly as for a stored column.
Derived columns themselves are an EDIT screen feature: the VIEW screen and the download services do not show them. A saved filter that references a derived column still applies there, and is evaluated the same way.
## Rules and limits
- **Expressions only.** `VAR_LOGIC` must be a single SAS expression - no semicolons, macro triggers (`%`, `&`) or comments. Conditional logic is available through functions such as `ifc()`, `choosec()`, `cats()`, `catx()`, `put()`, `input()` and `intnx()`.
- **Dependencies declared.** `VAR_DEPS` lists every input the expression uses - physical columns and derived variables alike. Statements are generated in dependency order, so no sequence number is maintained, and the declaration is checked in both directions: every reference must be declared, and every declared name must be referenced and must exist. Circular references are rejected.
- **No collisions.** `VAR_NM` must not match an existing column of the target table, or `_____DELETE__THIS__RECORD_____`.
- **Read-only.** Derived columns cannot be edited in the grid, and any value submitted for them is discarded before the approval package is created.
- **Recomputed on every read.** The value always reflects the current data and the current definition, and there is no history of previous derived values.
A configuration error - a bad name, an invalid type or length, a name collision, or a reference to a derived variable that is not declared in `VAR_DEPS` - stops the EDIT screen with a message naming the variable. A syntax error or an unknown column in the expression itself is caught when the view is compiled, and stops the screen with a message naming the table.
Most of these rules are also applied when a row is submitted for approval, through a post edit hook on `MPE_DERIVED`. A bad name, type, length, format or expression - or a reference to a derived variable that is not declared in `VAR_DEPS` - is rejected at submit time, while the error can still be corrected, rather than breaking the EDIT screen afterwards. The collision between `VAR_NM` and a physical column of the target table is only checked at read time, as the hook does not have the target library assigned.
The simple rules are also configured as validation rules on the table itself, so they show up in the grid before submission: `VAR_TYPE` is a dropdown of `C` and `N`, `VAR_LEN` carries minimum and maximum values, and `VAR_NM`, `VAR_DEPS`, `VAR_FMT` and `VAR_LOGIC` are pattern checked. Validation rules are evaluated in the browser, so the post edit hook remains the authoritative check.
!!! warning "Derived values can expose hidden data"
An expression can reference a column that is hidden from the user by [Column Level Security](/column-level-security/), and the derived column will show the result. Configure derived variables with the same care as any other data exposure.
## Performance
The expression is evaluated for each row read, before the filter. The filter already requires a scan of the table, so this adds per-row arithmetic rather than a second pass - but keep expressions cheap (no lookups, no subqueries), and expect the cost to be paid on every load of the EDIT screen, including each request for filter values.
docs/developer-guide/filter-pre-edit-hook.md
+153
View File
@@ -0,0 +1,153 @@
---
layout: article
title: Updating the Filter in a Pre-Edit Hook
description: Read the filter a user applied, and hand back a modified one, from a Data Controller pre-edit hook - including changing the rows the editor shows.
---
# Updating the Filter in a Pre-Edit Hook
A [pre-edit hook](/dcc-tables/#pre_edit_hook) can read the filter a user applied and return a modified one. This is useful when the editor should widen a user's selection behind the scenes - for example, when a user filters for the current quarter and the grid should also show the previous one.
The hook runs inside the `editors/getdata` service, so both the applied filter and the filtered rows are in scope. This page describes what is available at that point and works through an example.
## What the editor receives
When a filtered editor loads, `editors/getdata` returns the filter to the client in two forms:
- `sasparams.FILTER_TEXT` - the resolved WHERE clause, which the editor shows above the grid.
- `query` - the filter's clauses as structured rows (`variable_nm`, `operator_nm`, `raw_value`), which the editor renders as the filter chips.
The response does not carry the filter key (`FILTER_RK`). That is an *input* to `getdata`, not an output - the editor already holds it in the URL (`/editor/<table>/<FILTER_RK>`), so there is nothing to return.
## When the hook runs
`getdata` builds the filter, applies it, and sorts the result, all before the pre-edit hook runs. In order:
1. the WHERE clause is written to a temporary fileref, `filtref`;
2. `work.OUT` is populated with `where %inc filtref` - the filter is applied here;
3. the pre-edit hook runs;
4. `FILTER_TEXT` is read back out of `filtref`, and the response is streamed.
So the hook sits between "the rows are filtered" and "the filter is returned". Two consequences follow:
- It can change what the editor is told the filter was, by rewriting `filtref` and, if the chips must match, `work.query`.
- It cannot change which rows came back *that* way, because the data step has already run. To return different rows, the hook must re-read the table itself.
## Take the value from the data
A hook that wants to extend a date filter needs the date the user is working with. It does not have to parse the filter text for it - the filtered rows are already in `work.OUT`, so the value can be read from there.
## Example: widen a date filter to the previous quarter
The hook below is registered in `MPE_TABLES.PRE_EDIT_HOOK` for a table with a date column `ORDER_DATE`. When a user opens the editor with a filter that includes `ORDER_DATE`, it extends the filter to also cover the previous quarter, re-reads the table so the grid shows both quarters, and returns the widened filter text.
```sas
/* PRE_EDIT_HOOK: widen the user's ORDER_DATE filter by one quarter */
%if &filter_rk > 0 %then %do;
/* the date the user is looking at, taken from the rows already returned */
%let refdt=;
proc sql noprint;
select max(ORDER_DATE) into :refdt from work.out;
quit;
%if &refdt ne %then %do;
/* the previous quarter, relative to that date */
%let pq_from=%sysfunc(putn(%sysfunc(intnx(quarter,&refdt,-1,b)),date9.));
%let pq_to =%sysfunc(putn(%sysfunc(intnx(quarter,&refdt,-1,e)),date9.));
/* the user's own clauses, as a standalone WHERE */
%mp_filtergenerate(work.query,outref=userref)
/* Data Controller's own clause - current-record validity, plus RLS */
%mpe_filtermaster(EDIT,&orig_libds,
dclib=&mpelib,
filter_rk=-1,
outref=ctlref,
outds=work.ctlq
)
/* read each clause (which may span several lines) into a macro variable */
%let usertext=;
data _null_;
infile userref end=eof;
input;
length t $32767;
retain t;
t=catx(' ',t,_infile_);
if eof then call symputx('usertext',t);
run;
%let ctltext=;
data _null_;
infile ctlref end=eof;
input;
length t $32767;
retain t;
t=catx(' ',t,_infile_);
if eof then call symputx('ctltext',t);
run;
/* ( user's filter OR previous quarter ) AND ( validity + RLS ) */
filename newfilt temp;
data _null_;
file newfilt lrecl=32767;
length f $32767;
f='(( '!!trim(symget('usertext'))!!' ) OR (ORDER_DATE between "&pq_from"d and "&pq_to"d))'
!!' AND ('!!trim(symget('ctltext'))!!' )';
put f;
run;
/* re-read the table, so the grid shows both quarters */
data work.out (drop=&var_txfrom &var_txto &var_processed &drop_rk);
set &orig_libds;
where %inc newfilt;
run;
/* keep the order the editor first showed */
proc sort data=work.out;
by &sortpk;
run;
/* hand the widened filter back to the editor */
data _null_;
file filtref lrecl=32767;
infile newfilt;
input;
put _infile_;
run;
%end;
%end;
```
### What the hook does
- **The reference date.** `work.OUT` holds the rows the user's filter returned, so the latest `ORDER_DATE` on screen is read straight from it. Nothing has to be parsed out of the filter text.
- **The previous quarter.** `INTNX` moves back one quarter from that date and returns its first and last day, formatted as SAS date literals (`01APR2026` and `30JUN2026`, for example).
- **The user's clause.** `work.QUERY` is the filter's clauses as structured rows, and `%mp_filtergenerate` turns them back into a WHERE clause. This is the user's filter on its own, with no Row Level Security or validity clause mixed in.
- **Data Controller's clause.** Calling `%mpe_filtermaster` with `filter_rk=-1` produces the part of the filter that is not the user's: the current-record validity clause and any Row Level Security, or `1=1` when there is neither.
- **The combined filter.** The two clauses are combined as `( user OR previous-quarter ) AND ( validity + RLS )`, so the previous quarter is added to the user's selection without widening anything the user is not entitled to see.
- **The rows.** `work.OUT` is rebuilt with the combined filter, using the same drop list as the data step `getdata` runs itself - the transaction and processing columns, plus `&drop_rk`, the surrogate key that `getdata` hides when a table is registered with `RK_UNDERLYING` - and sorted on `&sortpk`, because `getdata` sorts before the hook runs and does not sort again afterwards. On a bitemporal table `getdata` also deduplicates to the latest record per primary key (`by &pk &var_busfrom`); add that step too, or the re-read returns every temporal version rather than the current one. The delete column is **not** re-added: `_____DELETE__THIS__RECORD_____` is created by the editor itself and is never a column of the dataset, so a hook must not introduce it (a response that carries that name is refused by the editor).
- **The returned text.** `filtref` is overwritten with the combined filter, so the clause the editor shows is the one that was actually applied.
### What it looks like
The captures below are the editor for a table with an `ORDER_DATE` column, opened with a filter for the current quarter. In the first, no hook is configured, so the filter is the user's own and the grid holds one quarter of rows. In the second, the hook has run: the filter above the grid carries both quarters, and the rows have grown to match.
![The editor with the user's own filter, one quarter of rows](/img/pre-edit-hook-before.png)
![The same editor after the hook widened the filter to the previous quarter](/img/pre-edit-hook-after.png)
## Row Level Security
The user's clause and Data Controller's clause have to be kept apart, because Row Level Security lives in the second one. Combining them as `( user OR previous-quarter ) AND ( validity + RLS )` keeps RLS applied to both branches. Wrapping the whole existing filter instead - `( existing OR previous-quarter )` - would let the previous-quarter rows through without their RLS condition, so do not take that shortcut.
## The row cap
`getdata` refuses a result larger than `DC_MAXOBS_WEBEDIT` (500 by default) with *Table is too big*. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap. Check the count before handing the data back, or raise the option for that table in `MPE_CONFIG`.
## Matching the filter chips
The clause above the grid comes from `FILTER_TEXT`, so overwriting `filtref` is what updates it. The filter chips are rendered from the `query` output instead, so if they must reflect the widened filter too, amend `work.QUERY` in the same hook - append the extra clause as a new subgroup with `group_logic='OR'` before the response is built.
docs/developer-guide/two-rule-sets-one-table.md
+122
View File
@@ -0,0 +1,122 @@
---
layout: article
title: One Table, Two Rule Sets
description: A single physical table can carry only one set of Data Controller validation rules. Two librefs over the same data - or a pair of hook scripts - give you as many rule sets as you need.
---
# One Table, Two Rule Sets
Most Data Controller sites settle into an obvious mapping: one table, one edit screen, one set of validation rules. But that is not always what the business wants. A table of orders might be edited from a finance report that only tolerates small adjustments, and from an operations report where much larger ones are routine. Same table, same columns, same approvers - different rules.
Data Controller's validation rules are configured per table, so this takes a little thought. There are two ways to do it: the one we recommend, and the one to reach for when the first is not available.
## Why one table is one rule set
Two configuration tables decide this.
`MPE_TABLES` is the list of editable tables, and its primary key is `(tx_from, libref, dsn)`. One physical table is one editable table.
`MPE_VALIDATIONS` holds the rules, and its primary key is `(tx_from, base_lib, base_ds, base_col, rule_type)`. Rules hang off a physical `libref.dataset`. There is no per-menu or per-report scoping anywhere in the schema, and the editor is handed exactly the rules whose `base_lib` and `base_ds` match the table being opened.
So two rule sets on one table need two distinct `libref.dataset` identities. The question is how to get them without copying the data.
## Option 1 (recommended): two librefs over the same data
A libref is just a name pointing at a location. Nothing stops you assigning two of them to the same place, and Data Controller will treat the two as separate tables:
```sas
libname ORDERS_EU '/data/orders';
libname ORDERS_US '/data/orders';
```
`ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` are now the same physical file, but they are different rows in `MPE_TABLES` and can therefore carry different rows in `MPE_VALIDATIONS`. Register both, give each its own rules, and point each report at its own editor URL - `#/editor/ORDERS_EU.ORDERS` and `#/editor/ORDERS_US.ORDERS`.
![The first report rejects an amount of 5000](/img/rule-set-1.png)
![The second report accepts the same value](/img/rule-set-2.png)
Everything else works exactly as it always did. Filtering, search, the row cap, the approval diff and the audit trail all operate on the table as normal, because as far as Data Controller is concerned these are ordinary tables. The only difference is that the two names resolve to the same file, so an approval in either report updates the same data - and that one difference has a consequence for concurrency, which is covered below.
There is no copy to keep in sync, no hook to write and nothing to maintain. That is why it is the option we recommend.
### Things worth knowing
- Where the librefs are defined depends on your platform. On Viya, in the compute context's `autoexec.sas` (or `[DC Drive Path]/services/settings.sas`); on SAS 9, as metadata libraries or in the Data Controller Settings stored process; on SASjs Server, in `services/public/settings.sas`. The one requirement is that each library has a unique libref.
- `mp_lockanytable` keys on `libref.dataset`, so the two menus do not serialise against each other. That has consequences beyond a collision - see *The concurrency caveat* below.
- The audit trail and approval queue record which libref a change came through, so `ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` stay distinguishable in history. For most people that is a feature - you can see which report a change originated from.
### The concurrency caveat
Data Controller serialises its writes with `mp_lockanytable`, and the control table it uses, `MPE_LOCKANYTABLE`, has a primary key of `(lock_lib, lock_ds)`. That is the *registration*, not the physical file. At approve time, `postdata` takes the lock on the base table named in the submit record, and the same service runs its "has this table been updated since the diff screen was shown" check against `MPE_DATALOADS`, keyed on that same `libref` and `dsn`.
Two librefs over one location are two different `(libref, dsn)` pairs, so:
- two approvals arriving through different reports take two different locks, and neither one blocks the other;
- a load through one report writes its `MPE_DATALOADS` entry against its own name, so the other report's staleness check does not see it either.
So two people editing the same rows through different reports can both be approved, and the second write wins, silently. The lock is advisory to begin with - `mp_lockanytable` is, in its own words, "only useful if every update uses the macro" - so this is not a new class of risk, but registering one file twice is a new way to fall into it.
The hook route does not have this problem. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity.
If you do use two librefs, and concurrency matters to you, the cleanest fix is to keep the two registrations for the edit screen and give each one a `POST_EDIT_HOOK` that re-points the changeset at a single canonical registration - then every approval is keyed on the same table regardless of which report raised it. Register that canonical name as well. The alternative, an explicit shared lock taken in `PRE_APPROVE_HOOK` and released in `POST_APPROVE_HOOK`, works too, but a failed run leaves the sentinel locked and `MPE_LOCKANYTABLE` is then a table you have to unpick by hand.
## Option 2: an empty mirror and a pair of hook scripts
Sometimes two librefs over one location are not available: a database library where the platform will not let you define the same object twice, or a site where adding a library definition is a change nobody wants to make. Then you can reach the same result with a mirror table and two hook scripts.
The idea is that the thing Data Controller edits is not the real table at all, but an empty table of the same shape, with hooks moving data in and out of it:
- a `PRE_EDIT_HOOK` fills the editor with the live rows of the real table, so the mirror never has to hold a copy
- a `POST_EDIT_HOOK` re-points the submitted changeset at the real table, so the approval is raised against the real table and the load writes there
The mirror exists purely to carry the rule set, and never stores any data.
### The pre-edit hook
`PRE_EDIT_HOOK` runs inside the `getdata` service, after the user's filter has been applied and the rows sorted, with the data in `work.OUT`. It may replace that dataset, which is all this needs:
```sas
data work.out;
set ORDERS.ORDERS;
run;
```
The registered table is `ORDERS.MIRROR`, which is empty - so without the hook the editor would show nothing at all. With it, the grid shows the live rows:
![An empty mirror displaying the live rows of the real table](/img/mirror-live.png)
Note the title bar: the mirror really is empty. Everything on screen came from the hook, and the grid is validated against the mirror's own rules rather than the real table's.
### The post-edit hook
This is the part that surprises people. `POST_EDIT_HOOK` runs inside the `mpe_loader` macro at submit time, on the staged rows, before the submit record is written. It cannot choose the target table directly - but at that point `LIBREF` and `DS` are still ordinary macro variables, and the submit record is built from them. Reassigning them re-points the whole changeset:
```sas
data _null_;
call symputx('libref','ORDERS');
call symputx('ds','ORDERS');
run;
```
From that moment the changeset is an approval against `ORDERS.ORDERS`. The approver sees a diff against the real table, the load writes to the real table, and the mirror is never touched.
![A change submitted against the mirror, raised against the real table](/img/approval-routed.png)
### Reassigning LIBREF and DS
`LIBREF` and `DS` are ordinary macro variables inside `mpe_loader`, which creates them with `%let` and then updates them from the staged row. A `%let libref=ORDERS;` reassigns them just as well as the `call symputx` above: both statements assign to the *existing* variable, because it lives in an enclosing symbol table (`mpe_loader`'s local table) and SAS only creates a new local when the name is absent from every enclosing table. `call symputx` is used above because it is explicit about which variable is being set, and because it does not depend on the enclosing scope still holding the name - a wrapper macro that declared `%local libref` would turn a bare `%let` into an assignment to that shadow instead.
### Other things to watch
- Concurrency behaves correctly here. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity - unlike the two-libref route above.
- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap will stop the edit screen with "Table is too big". [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter.
- The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load.
- The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table.
- The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical.
- At approval time the access checks run against the real table, so editors need `EDIT` on the mirror while approvers need `EDIT` and `APPROVE` on the real table.
## Which should you use?
If you can define two librefs over the same data, do that. It is configuration only, it leaves every other behaviour of the editor untouched, and there is nothing to maintain.
Reach for the hook scripts when the platform will not let you duplicate the library definition, or when you specifically want the rule set to be a property of the application rather than of the data.
docs/downloads.md
+52
View File
@@ -0,0 +1,52 @@
---
layout: article
title: DC Downloads
description: Data Controller release assets and how to verify them
---
# Downloads
Data Controller releases are published on the [releases page](https://git.datacontroller.io/dc/dc/releases) of the source repository. Every release carries the same set of assets:
| Asset | Use |
|---|---|
| `frontend.zip` | The pre-built web frontend, for deployments that serve the frontend from a web server (SAS 9 full deploy, Viya separated deploy) |
| `sas9.sas` | SAS 9 full deployment program (stored processes + frontend registration) |
| `demostream_sas9.sas` | SAS 9 streaming deployment program (demos and evaluations) |
| `viya.sas` | Viya streaming deployment program (backend + frontend) |
| `viya_noweb.sas` | Viya separated deployment program (backend only) |
| `viya_noweb.json` | The Viya backend services as a SASjs Drive JSON deployment bundle |
| `sasjs_server.json.zip` | SASjs Server deployment bundle (all services, for [SASjs Server](https://server.sasjs.io) deployments) |
| `SHA256SUMS` | SHA-256 checksums for every asset above |
Which asset you need depends on the deployment route - see [SAS 9 Deployment](/dci-deploysas9) or [SAS Viya Deployment](/deploy-viya).
## Verifying a download
Every asset is covered by the `SHA256SUMS` file, which the release pipeline generates over the exact files it uploads. To verify a download:
1. Download the asset (for example `frontend.zip`) from the [releases page](https://git.datacontroller.io/dc/dc/releases).
2. Download `SHA256SUMS` from the same release into the same folder.
3. Run the check:
```bash
sha256sum --check SHA256SUMS --ignore-missing
```
Each asset that matches reports `OK`. The `--ignore-missing` flag means you only need the files you actually downloaded - the check skips assets you did not fetch.
On Windows, `certutil` can compute the hash of a single file for comparison against the value listed in `SHA256SUMS`:
```
certutil -hashfile frontend.zip SHA256
```
### What the checksum proves
A matching checksum confirms the asset is byte-for-byte the file the release pipeline uploaded. It catches corrupted and tampered downloads - interrupted transfers, proxies or mirrors that altered the file in transit.
It is not a cryptographic signature. The hashes travel in the same release as the files they cover, so they protect the download path, not the release itself: a compromise of the forge would allow both the asset and its checksum to be altered together. For download integrity - the typical concern when fetching deployment artifacts over the internet - it is the standard check.
## Integrity inside SAS
The SAS deployment programs embed the frontend and services as generated SAS code. Once `%inc`'d, their integrity is established by the check above at download time. They deploy the stored processes or jobs into the SAS Folder you nominate (`appLoc`) - a metadata location, not physical disk - and the only physical location they write to is the Data Controller location (`dcLoc`) you configure. The programs can be [reviewed in full](https://code.datacontroller.io) (or in the [source repository](https://git.datacontroller.io/dc/dc)) before you run them.
docs/dynamic-cell-dropdown.md
+20 -27
View File
@@ -49,15 +49,14 @@ The following tables should be created in the WORK library as outputs:
This output table can contain up to three columns:
* `display_index` (optional, mandatory if using `dynamic_extended_values`). Is a numeric key used to join the two tables.
* `display_value` - always character
* `raw_value` - unformatted character or numeric according to source data type
Example values:
|DISPLAY_INDEX:best.|DISPLAY_VALUE:$|RAW_VALUE|
|---|---|---|
|1|$77.43|77.43|
|2|$88.43|88.43|
|DISPLAY_INDEX:best.|RAW_VALUE|
|---|---|
|1|77.43|
|2|88.43|
### `WORK.DYNAMIC_EXTENDED_VALUES`
This output table is optional. If provided, it will map the DISPLAY_INDEX from the DYNAMIC_VALUES table to additional column/value pairs, that will be used to populate dropdowns for _other_ cells in the _same_ row.
@@ -66,7 +65,6 @@ The following columns should be provided:
* `display_index` - a numeric key joining each value to the `dynamic_values` table
* `extra_col_name` - the name of the additional variable(s) to contain the extra dropdown(s)
* `display_value` - the value to display in the dropdown. Always character.
* `display_type` - Either C or N depending on the raw value type
* `raw_value_num` - The unformatted value if numeric
* `raw_value_char` - The unformatted value if character
@@ -74,17 +72,17 @@ The following columns should be provided:
Example Values:
|DISPLAY_INDEX:best.|EXTRA_COL_NAME:$32|DISPLAY_VALUE:$|DISPLAY_TYPE:$1.|RAW_VALUE_NUM|RAW_VALUE_CHAR:$5000|FORCED_VALUE|
|---|---|---|---|---|---|---|
|1|DISCOUNT_RT|"50%"|N|0.5||.|
|1|DISCOUNT_RT|"40%"|N|0.4||0|
|1|DISCOUNT_RT|"30%"|N|0.3||1|
|1|CURRENCY_SYMBOL|"GBP"|C||"GBP"|.|
|1|CURRENCY_SYMBOL|"RSD"|C||"RSD"|.|
|2|DISCOUNT_RT|"50%"|N|0.5||.|
|2|DISCOUNT_RT|"40%"|N|0.4||1|
|2|CURRENCY_SYMBOL|"EUR"|C||"EUR"|.|
|2|CURRENCY_SYMBOL|"HKD"|C||"HKD"|1|
|DISPLAY_INDEX:best.|EXTRA_COL_NAME:$32|DISPLAY_TYPE:$1.|RAW_VALUE_NUM|RAW_VALUE_CHAR:$5000|FORCED_VALUE|
|---|---|---|---|---|---|
|1|DISCOUNT_RT|N|0.5||.|
|1|DISCOUNT_RT|N|0.4||0|
|1|DISCOUNT_RT|N|0.3||1|
|1|CURRENCY_SYMBOL|C||"GBP"|.|
|1|CURRENCY_SYMBOL|C||"RSD"|.|
|2|DISCOUNT_RT|N|0.5||.|
|2|DISCOUNT_RT|N|0.4||1|
|2|CURRENCY_SYMBOL|C||"EUR"|.|
|2|CURRENCY_SYMBOL|C||"HKD"|1|
### Code Examples
@@ -107,9 +105,9 @@ Simple dropdown
Output should be a single table called
"work.dynamic_values" in the format below.
|DISPLAY_VALUE:$|RAW_VALUE:??|
|---|---|
|$44.00|44|
|RAW_VALUE:??|
|---|
|44|
**/
@@ -133,37 +131,33 @@ create table work.source as
where tx_to > "%sysfunc(datetime(),E8601DT26.6)"dt
order by 1,2;
data work.DYNAMIC_VALUES (keep=display_index raw_value display_value);
data work.DYNAMIC_VALUES (keep=display_index raw_value);
set work.source end=last;
by libref;
if last.libref then do;
display_index+1;
raw_value=libref;
display_value=libref;
output;
end;
if last then do;
display_index+1;
raw_value='*ALL*';
display_value='*ALL*';
output;
end;
run;
data work.dynamic_extended_values(keep=display_index extra_col_name display_type
display_value RAW_VALUE_CHAR raw_value_num forced_value);
RAW_VALUE_CHAR raw_value_num forced_value);
set work.source end=last;
by libref dsn;
retain extra_col_name 'ALERT_DS';
retain display_type 'C';
retain raw_value_num .;
raw_value_char=dsn;
display_value=dsn;
forced_value=0;
if first.libref then display_index+1;
if last.libref then do;
display_value='*ALL*';
raw_value_char='*ALL*';
forced_value=1;
output;
@@ -171,7 +165,6 @@ data work.dynamic_extended_values(keep=display_index extra_col_name display_type
else output;
if last then do;
display_value='*ALL*';
raw_value_char='*ALL*';
forced_value=1;
output;
docs/emails.md
+18 -4
View File
@@ -1,5 +1,13 @@
Data Controller for SAS® - Emails
====================
---
layout: article
title: Data Controller Emails
description: Set up email alerts in Data Controller
og_title: Data Controller for SAS® emails
og_image: /img/mpe_config_dc_email.png
---
# Data Controller for SAS® - Emails
## Overview
Data Controller enables email alerts for users when tables are:
@@ -21,7 +29,7 @@ To switch it on, navigate to `DCXXXXXX.MPE_CONFIG` and set the value for `DC_EMA
![alerttable](img/mpe_alertconfig.png)
!!! tip
If your Stored Process session does not have the email options configured, then the appropriate options statement must be invoked. These options may need to be done at startup, or in the configuration file. See [documentation](https://documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.4&docsetId=lrcon&docsetTarget=n05iwqtqxzvtvun1eyw11nrd9i9r.htm&locale=en)
If your SAS 9 Stored Process or Viya Compute session does not have the email options configured, then the appropriate options statement must be invoked. These options may need to be done at startup, or in the configuration file. See [documentation](https://documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.4&docsetId=lrcon&docsetTarget=n05iwqtqxzvtvun1eyw11nrd9i9r.htm&locale=en)
## Configuration
The `DCXXXXXX.MPE_ALERTS` table must be updated with the following attributes:
@@ -31,4 +39,10 @@ The `DCXXXXXX.MPE_ALERTS` table must be updated with the following attributes:
* ALERT_DS - either `*ALL*` or the dataset name to be alerted on
* ALERT_USER - the metadata name (not displayname) of the user to be alerted
If your site does not put emails in metadata, then the user emails must instead be entered in `DCXXXXXX.MPE_EMAILS`.
If your site does not put emails in metadata (or have them available in the Viya identities service), then the user emails must instead be entered in `DCXXXXXX.MPE_EMAILS`.
## Templates
The wording of the emails can be easily modified by updating the template in the [MPE_CONFIG](/tables/mpe_config) table.
More information can be found in the [options configuration](https://docs.datacontroller.io/dcc-options/#dc_email-scope) page.
docs/embed-va.md
+60
View File
@@ -0,0 +1,60 @@
---
layout: article
title: SAS Visual Analytics Embed
description: Embed the Data Controller editor inside a SAS VA report as a data-driven content object. Row selections in the VA report drive filters and column visibility in the editor.
---
# Embedding inside SAS Visual Analytics
Data Controller can be embedded inside a SAS Visual Analytics report as a **data-driven content** (DDC) object. This unlocks scenarios where a user can safely modify the values in an underlying SAS table, and have the visualisation updated- all from inside the same report.
## URL
Open the editor as a DDC by appending `?embed=va` to the editor route:
```
`https://yourserver/DC/#/editor/<library>/<table>?embed=va`
```
The `?embed=` parameter accepts three values:
| Value | Behaviour |
|---|---|
| `true` | Chrome (header, back button, sub-nav) is hidden. Editor functions normally. |
| `va` | Same chrome-hiding as `true`, **plus** editor becomes "VA-aware" - any report filters are captured, and columns can be displayed / hidden using the Edit Report interface. Filter button is disabled (use VA filters instead)|
| `false` (or omitted) | Standard interactive UI. |
## How VA drives the editor
VA pushes a JSON payload to the iframe via `window.postMessage` whenever the selected rows change. Data Controller listens for these messages and:
1. Resolves each VA `parameter` to a DC column by matching on column label.
2. Builds a filter (using existing [Filter](/filter/) machinery) over the selected values.
3. Hides any VA columns marked as `brush` so the grid stays focused on the user-editable columns (except primary key cols which are always shown)
If VA sends an empty / unmatched message the editor falls back to the unfiltered view but stays in VA mode.
More logic available in [`va-messaging.service.ts'](https://git.datacontroller.io/dc/dc/src/branch/main/client/src/app/services/va-messaging.service.ts).
## Filter Modes
When running in `embed=va` mode, the editor provides two filter modes, controlled by the **Auto-apply** checkbox:
* **Live (default)** - the editor updates automatically as you select rows in the VA report, so the data you see always matches your current selection.
* **Confirm** - filter changes are held until you click the **Apply** button. This is useful when editing, where an automatic reload would discard unsaved changes.
A status indicator shows whether a filter change is pending, loading, or idle. In edit mode, a pending filter is held until you leave edit mode, so your unsaved edits are never lost.
## Configuration in VA
In the VA Report Designer, add a **Data-Driven Content** object and set the URL to the editor route shown above. Be sure that any report level filters have their corresponding parameters added to the DDC object itself.
## Debugging
The following snippet can be used in console to dump the values being provided to DC from VA:
```
console.log(JSON.stringify(window.__vaLastMessage?.data, null, 2))
```
docs/filter.md
+3 -1
View File
@@ -7,7 +7,9 @@ og_image: https://docs.datacontroller.io/img/filter_dynamic_on.png
# Filtering
Data Controller for SAS&reg; enables you to create complex table filters. The "dynamic" setting enables the dropdown values to be pre-filtered by previous filter clauses. Filtered views are shareable!
Data Controller for SAS&reg; enables you to create complex table filters. The "dynamic" setting enables the dropdown values to be pre-filtered by previous filter clauses. Filtered views are shareable! They are also used when [embedded in VA.](/embed-va)
The clause shown above the grid is the resolved WHERE clause that was applied, so it uses SAS syntax - a date, datetime or time filter reads as a SAS literal, eg `SOME_DATE = '20APR2042'd`, rather than the internal day or second count.
## Shared Filters
docs/img/admininfo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 150 KiB

docs/img/approval-routed.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

docs/img/cannotimport.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 46 KiB

docs/img/catalog.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 138 KiB

docs/img/catalogrefresh.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

docs/img/datacatalog_cats.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 126 KiB

docs/img/datacatalog_objs.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 178 KiB

docs/img/datastatus_cats.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 181 KiB

docs/img/datastatus_objs.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 195 KiB

docs/img/dc-logo.svg
+15
View File
@@ -0,0 +1,15 @@
<svg width="70" height="70" viewBox="0 0 70 70" fill="none" xmlns="http://www.w3.org/2000/svg">
<g clip-path="url(#clip0_101_8)">
<path d="M37.5993 16.6212C38.5916 16.7415 39.5538 16.8317 40.5461 16.9519C42.9216 17.2827 45.237 17.7337 47.402 18.8463C48.0334 19.1771 48.6348 19.5379 49.0859 20.1092C49.8075 20.9512 49.7774 21.8533 49.0558 22.6952C48.304 23.5371 47.3418 24.0183 46.3195 24.4092C43.944 25.3413 41.4482 25.7623 38.9224 25.9728C34.8029 26.3035 30.7435 26.1532 26.7142 25.191C25.3911 24.8903 24.0981 24.4693 22.9254 23.8078C22.4744 23.5672 22.0534 23.2665 21.6625 22.9057C20.58 21.8533 20.58 20.7407 21.6926 19.7484C22.8954 18.6659 24.3688 18.1547 25.8722 17.7337C27.9771 17.1324 30.112 16.8317 32.277 16.6813C32.4575 16.6813 32.6679 16.7415 32.8183 16.5911C34.3819 16.6212 36.0056 16.6212 37.5993 16.6212Z" fill="#E0E0E0"/>
<path d="M20.8507 41.3082C22.7751 43.7738 25.5115 44.6759 28.338 45.2773C33.9309 46.4801 39.5238 46.3297 44.9663 44.4053C46.6502 43.8039 48.2138 42.992 49.3264 41.4886C49.5068 41.248 49.597 41.3683 49.6872 41.5487C49.8677 41.9396 49.9579 42.3305 49.9579 42.7515C49.9579 44.2249 49.9579 45.7284 49.9579 47.2018C49.9579 48.164 49.5068 48.9157 48.8453 49.5772C47.4621 50.9604 45.7181 51.7122 43.8838 52.2835C40.1853 53.4261 36.3966 53.6667 32.5777 53.3961C29.7212 53.1856 26.9548 52.6443 24.3387 51.4716C23.2261 50.9604 22.2038 50.3591 21.3919 49.4269C20.7605 48.7052 20.4297 47.8934 20.4297 46.9311C20.4598 45.578 20.4297 44.2249 20.4297 42.8718C20.4297 42.3305 20.5199 41.8193 20.8507 41.3082Z" fill="#8EC63F"/>
<path d="M20.8807 32.0167C22.4744 34.1516 24.6996 35.0537 27.075 35.6852C32.7582 37.1886 38.4413 37.1586 44.0643 35.4747C45.9286 34.9334 47.6726 34.1216 49.0258 32.6482C49.2062 32.4677 49.3265 32.0468 49.5369 32.1069C49.8376 32.1971 49.8376 32.6181 49.9278 32.9488C50.0481 33.46 49.988 33.9712 49.988 34.4824C49.988 35.5047 49.9579 36.497 49.988 37.5194C50.0481 38.7823 49.5369 39.7746 48.6048 40.5865C47.1915 41.8494 45.4776 42.6011 43.6433 43.0822C37.4791 44.7361 31.3449 44.706 25.3009 42.6011C23.8275 42.09 22.4744 41.3683 21.3919 40.1956C20.7304 39.4739 20.3997 38.632 20.3997 37.6397C20.4297 36.3166 20.3997 34.9635 20.3997 33.6404C20.4297 33.0992 20.5199 32.5579 20.8807 32.0167Z" fill="#E0E0E0"/>
<path d="M20.8207 23.0861C21.2717 24.4091 22.3542 25.0105 23.4969 25.5517C25.5416 26.514 27.7366 26.965 29.9317 27.2357C34.2918 27.7769 38.6519 27.7168 42.9518 26.8147C44.6958 26.4539 46.3797 25.9727 47.9132 25.0105C48.7853 24.4693 49.2363 23.9581 49.537 23.1161C49.8377 23.5371 49.9279 23.9882 49.9279 24.4392C49.9279 25.9427 49.958 27.4161 49.9279 28.9195C49.8978 30.1825 49.1762 31.0845 48.244 31.8363C46.5 33.2195 44.4553 33.9411 42.3203 34.4223C36.6673 35.6551 31.0744 35.5649 25.5717 33.7006C24.1283 33.2195 22.7451 32.5279 21.6326 31.4454C20.7906 30.6335 20.3396 29.7014 20.3696 28.4986C20.3997 27.2056 20.3696 25.9126 20.3696 24.6196C20.4298 24.0483 20.52 23.5672 20.8207 23.0861Z" fill="#8EC63F"/>
<path d="M55.3403 63.4393C60.2417 63.7099 62.8577 57.696 59.3095 54.2981C55.5809 50.7199 49.4467 54.0576 50.409 59.1093C28.2478 73.2419 -0.258034 52.1031 7.86071 26.4238L2.38808 22.7553C-9.91033 55.4408 27.9471 83.0746 55.3403 63.4393ZM53.7767 56.3729C56.062 53.9974 59.7004 57.3051 57.5655 59.8009C55.3103 62.4169 51.3411 58.8988 53.7767 56.3729Z" fill="#E0E0E0"/>
<path d="M10.6272 15.2681C14.1453 18.6358 19.9487 15.8694 19.6179 11.0583C41.749 -3.19461 70.4052 17.9142 62.2865 43.6836L67.7591 47.3521C79.6967 15.5988 43.463 -12.9972 15.4383 6.21715C10.1761 5.13465 6.77827 11.5695 10.6272 15.2681ZM16.1599 13.1933C13.8145 15.6289 10.116 12.0807 12.4614 9.64507C14.8068 7.20944 18.5054 10.7576 16.1599 13.1933Z" fill="#8EC63F"/>
</g>
<defs>
<clipPath id="clip0_101_8">
<rect width="70" height="70" fill="white"/>
</clipPath>
</defs>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.7 KiB

docs/img/derived-config.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

docs/img/derived-filter-region.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

docs/img/derived-filter-value.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

docs/img/derived-grid.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

docs/img/full-table-search.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 157 KiB

docs/img/mirror-live.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

docs/img/mpe_config_dc_email.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 131 KiB

docs/img/mpe_emails.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

docs/img/mpe_validations.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 280 KiB

docs/img/pre-edit-hook-after.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 103 KiB

docs/img/pre-edit-hook-before.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 77 KiB

docs/img/revert.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 55 KiB

docs/img/rule-set-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

docs/img/rule-set-2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 65 KiB

docs/img/viyadeployauto.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 98 KiB

docs/index.md
+4 -3
View File
@@ -23,7 +23,7 @@ The following resources contain additional information on the Data Controller:
- Data Controller flyer ([front](/marketing/flyer-front.pdf) / [back](/marketing/flyer-back.pdf))
- Data Controller [videos](/videos)
- Data Controller [SAS Code](https://code.datacontroller.io)
- Data Controller [Download](https://4gl.uk/dcdeploy)
- Data Controller [Download](https://git.datacontroller.io/dc/dc/releases)
## Product Features
@@ -31,7 +31,7 @@ Data Controller is regularly updated with new features. If you see something th
* [Excel uploads](/excel) - drag & drop directly into SAS. All versions of excel supported.
* Data Lineage - at both table and column level, export as image or CSV
* Data Validation Rules - both automatic and user defined
* Data Validation Rules - both automatic and user defined, including [live formulas](/dcc-validations/#formula-rules)
* Data Dictionary - map data definitions and ownership
* Data Catalog - including primary key extraction
* DDL generator - in SAS, TSQL and PGSQL flavours
@@ -42,7 +42,8 @@ Data Controller is regularly updated with new features. If you see something th
* [Row Level Security](/row-level-security)
* Excel [formula support](excel)
* Dynamic [cell dropdown](/dynamic-cell-dropdown)
* Works on ALL flavours of SAS (Foundation, EBI, Viya)
* [SAS Visual Analytics embed](/embed-va/) - drive the editor from VA report selections
* Works on ALL flavours of SAS (Base, EBI, Viya)
docs/libraries.md
+3
View File
@@ -26,6 +26,9 @@ Library definitions should be added in the `autoexec.sas` of the designated Comp
If the above is not feasible, it is possible to insert code in the `[DC Drive Path]/services/settings.sas` file however - this will have a performance impact due to the additional API calls.
!!! note
The CASUSER library must be assigned, as it is used to store temporary in-memory tables during CAS data updates.
## SAS 9 EBI Libraries
In most cases, libname statements are NOT required so long as they are accessible in metadata.
docs/macros.md
+1
View File
@@ -26,6 +26,7 @@ Native pass through is also available for optimised data loads in the following
* Microsoft SQL SERVER
* Amazon REDSHIFT
* PostgreSQL
* Snowflake
The macros work dynamically, taking data types / lengths etc from the table metadata at runtime. Data Controller macros are available for unlimited (internal) use by licenced customers. They are currently in use, in production, in dozens of SAS environments globally and have been battle tested on large data volumes as well as some more esoteric gotchas such as:
docs/restore.md
+55 -13
View File
@@ -7,26 +7,68 @@ og_image: https://docs.datacontroller.io/img/restore.png
# Data Restore
For those tables which have [Audit Tracking](/dcc-tables/#audit_libds) enabled, it is possible to restore the data to an earlier state!
For tables that have [Audit Tracking](/dcc-tables/#audit_libds) enabled, it is possible to restore the data to an earlier state.
Simply open the submit to be reverted (via HISTORY or the table INFO/VERSIONS screen), and click the red **REVERT** button. This will generate a NEW submission, containing the necessary reversal entries. This new submission **must then be approved** in the usual fashion.
## How to restore data
Open the submit to be reverted (via the **History** tab, or the **Info / Versions** screen on the table viewer), and click the red **REVERT** button.
![](/img/restore.png)
This approach means that the audit history remains intact - there is simply a new entry, which reverts all the previous entries.
This will generate a **new submission** containing all the reversal entries needed to bring the table back to its previous state. This new submission **must then be approved** in the usual fashion - it goes through the same edit-stage-approve workflow as any other change.
## Caveats
This approach means that the audit history remains intact. There is simply a new entry in the audit trail which reverts all the previous entries.
Note that there are some caveats to this feature:
## How it works
- User must have EDIT permission
- Table must have TXTEMPORAL or UPDATE Load Type
- Changes **outside** of Data Controller cannot be reversed
- If there are COLUMN or ROW level security rules, the restore will abort
- If the model has changed (new / deleted) columns the restore will abort
Behind the scenes, rollback is not a silent undo. It is a **first-class approval workflow** just like any other edit. When you choose to restore a previous version, the backend reads the audit table and computes every difference between the current state and the version you want to go back to.
## Technical Information
The process is driven by the open-source [`%mp_stripdiffs`](https://core.sasjs.io/mp__stripdiffs_8sas.html) macro. It extracts all changes recorded in the audit table (the default is [`MPE_AUDIT`](/tables/mpe_audit/), or a custom table configured in `AUDIT_LIBDS`) from the selected version onwards, and left-joins them to the base table to build a staging dataset. Changes are then applied in **reverse chronological order**:
The restore works by undoing all the changes listed in the [MPE_AUDIT](/tables/mpe_audit/) table. The keys from this table (since and including the version to be restored) are left joined to the base table (to get current values) to create a staging dataset, and then the changes applied in reverse chronological order using [this macro](https://core.sasjs.io/mp__stripdiffs_8sas.html). This staging dataset is then submitted for approval, providing a final sense check before the new / reverted state is applied.
- **Deleted rows** are re-inserted with their original values.
- **Modified rows** are reverted to their previous values.
- **Added rows** are marked for deletion with the `_____DELETE__THIS__RECORD_____` flag.
Source code for the restore process is available [here](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/services/editors/restore.sas).
The computed differences are written to a new staging package in the approvals directory, complete with a CSV and a `macvars.sas` snapshot of the session context. A new `LOAD_REF` is generated, and the package is submitted via the standard [`mpe_loader`](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/macros/mpe_loader.sas) service. This means the rollback itself is **reviewable and approvable** - nothing is applied silently.
Because the rollback creates a new approved changeset rather than silently rewinding history, the full audit trail is maintained: the reversion appears as a new load reference in `MPE_SUBMIT`, `MPE_REVIEW`, `MPE_DATALOADS`, and `MPE_AUDIT`, just like any other submission.
## Security and access
Not everyone can restore everything. The [`mpe_checkrestore`](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/macros/mpe_checkrestore.sas) macro enforces a strict access check before the restore service will run.
### Who can restore
- **Admins** - members of the admin group (configured in `MPE_CONFIG`) can restore any table that has audit tracking enabled.
- **Editors** - non-admin users must have `EDIT` access to the target table, and must **not** be subject to Column Level Security or Row Level Security restrictions.
### What is checked
1. **The load must exist in the audit table.** Loads that were never applied, or tables without an audit table configured, cannot be restored.
2. **The user must have `EDIT` access.** This is checked via `MPE_SECURITY`.
3. **CLS and RLS restrictions block non-admin users.** If the user belongs to a group with active Column Level Security (`MPE_COLUMN_LEVEL_SECURITY`) or Row Level Security (`MPE_ROW_LEVEL_SECURITY`) rules scoped to `EDIT` or `ALL`, restore is denied. Admins bypass this check.
!!! note
The presence of CLS or RLS rules on a table does **not** prevent the table itself from being restorable. It only prevents non-admin users who are subject to those restrictions from initiating the restore. After a successful restore, the same CLS and RLS filters continue to apply when users view or edit the data.
If access is denied, the service aborts immediately with a clear reason - no opaque errors.
## Limitations
!!! warning
Be aware of the following caveats before attempting a restore:
- **Audit table required.** The table must have `AUDIT_LIBDS` configured in `MPE_TABLES` (it defaults to `MPE_AUDIT`). Without this, there is no change history to roll back from.
- **Only Data Controller changes can be reversed.** Changes made directly to the target table outside of Data Controller are not tracked in the audit table and cannot be reverted.
- **Supported load types.** Restore is supported for `UPDATE` and `TXTEMPORAL` load types. `REPLACE` loads do not maintain row-level audit history and cannot be restored. `BITEMPORAL` loads maintain audit history but restore behaviour may be constrained by validity windows.
- **Model consistency.** If the table structure has changed since the version being restored (for example, columns were added or removed), the restore will abort because the audit diffs no longer match the current schema.
- **SCD2 validity windows.** For `TXTEMPORAL` tables, the restore works on the current snapshot only. Historical validity windows are respected by the underlying [`%mp_stripdiffs`](https://core.sasjs.io/mp__stripdiffs_8sas.html) macro, but ensure your approvers understand that the reversion applies to the current state of the data.
- **No partial restore.** Rolling back a version reverts **all** changes from that version onwards, not just selected records. There is no way to cherry-pick individual rows from a previous version.
## Technical background
The restore process is implemented in the [`restore.sas`](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/services/editors/restore.sas) service. It delegates access control to [`mpe_checkrestore.sas`](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/macros/mpe_checkrestore.sas), diff computation to [`%mp_stripdiffs`](https://core.sasjs.io/mp__stripdiffs_8sas.html), and approval workflow submission to [`%mpe_loader`](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/macros/mpe_loader.sas).
Audit records are written by the open-source [`%mp_storediffs`](https://core.sasjs.io/mp__storediffs_8sas.html) macro during the normal approval workflow, which compares the pre-load snapshot with the applied changes and appends row-level entries to the audit table. This is why restore is only possible for tables that were loaded through Data Controller's tracked loaders.
For a lighter overview of this feature, see the [feed post on datacontroller.io](https://datacontroller.io/rollback-data-changes/).
docs/roadmap.md
+24 -3
View File
@@ -21,10 +21,8 @@ When features are requested, we will describe the work to be performed in the se
The following features are currently requested:
* Ability to set 'number of approvals' to zero, enabling instant updates (4 days)
* Ability to restore previous versions
* Ability to make automated submissions using an API
### Set Approvals to Zero
The following changes are necessary to implement this feature:
@@ -47,6 +45,10 @@ Described [here](https://docs.datacontroller.io/api/).
Below are some examples of Features that have been requested (and delivered) into Data Controller.
### Derived Variables
Columns that do not exist in the table - a banding, a concatenation, a calculation - can be configured per table and appear in the EDIT screen, read-only and filterable, without changing the table or the load process. Nothing derived is stored. See [docs](/derived-variables)
### Column Level Security
The columns that can be VIEWed or EDITed are now configurable. See [docs](/column-level-security)
@@ -160,4 +162,23 @@ Our customer was ingesting Basel III reports into SAS and needed an easy to use
We built an approach that allowed end users to define a series of rules for importing cells and ranges from anywhere within a workbook - based on absolute / relative positioning, or using search strings.
The changes we made to deliver this feature are described [here](https://git.datacontroller.io/dc/dc/issues/69) and the final documentation is [here](/excel).
The changes we made to deliver this feature are described [here](https://git.datacontroller.io/dc/dc/issues/69) and the final documentation is [here](/excel).
### Restore Previous Versions
It is now possible to restore any change by heading to the particular staged data screen and hitting the red REVERT button
![](./img/revert.png)
This will submit a NEW change (which must first be approved) that will revert the table to state it was in just prior to the selected upload.
Note that Data Controller can only track (and revert) changes that are made using the Data Controller tool itself! It does not / cannot track changes made externally to a table, by other tools.
### Frontend Formulae
Delivered in v7.13.0. Formula columns (HARDFORMULA / SOFTFORMULA) are computed live in the editor using HyperFormula, with variable-name references resolved to row-relative cell references. See [Formula Rules](/dcc-validations/#formula-rules).
### Regex Rules
Delivered. HARDREGEX (blocking) and SOFTREGEX (warning) rules validate cell values against a Perl-style regular expression. See [Regex Rules](/dcc-validations/#regex-rules).
docs/tables/mpe_datacatalog_cats.md
+24
View File
@@ -0,0 +1,24 @@
---
layout: article
title: MPE_DATACATALOG_CATS
description: The MPE_DATACATALOG_CATS table contains all the catalogs available in each library
og_title: MPE_DATACATALOG_CATS Table Documentation
og_image: /img/datacatalog_cats.png
---
# MPE_DATACATALOG_CATS
The `MPE_DATACATALOG_CATS` table contains the catalogs available in each library.
More frequently changing attributes are stored in [MPE_DATASTATUS_CATS](/tables/mpe_datastatus_cats).
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
![](/img/datacatalog_cats.png)
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `MEMNAME char(64)`: The catalog member name
docs/tables/mpe_datacatalog_libs.md
+19 -2
View File
@@ -10,11 +10,11 @@ The `MPE_DATACATALOG_LIBS` table catalogs library attributes such as engine, pat
More frequently changing attributes (such as size and number of tables) are stored in [MPE_DATASTATUS_LIBS](/mpe_datastatus_libs).
To ignore additional librefs, or to trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
To ignore additional librefs, or to trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
## Columns
- `TX_FROM num`: SCD2 open datetime
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- `ENGINE char(32)`: The engine used to connect to the library
@@ -25,5 +25,22 @@ To ignore additional librefs, or to trigger a scan, see the Refresh Data Catalog
- `SCHEMAS char(500)`: The library schema (DB engines)
- `LIBID char(17)`: The Library Id (from metadata if SAS 9)
## Refresh Process
The table is refreshed by the `mpe_refreshlibs` macro, which runs:
- When a user clicks the refresh icon next to a library in the VIEW menu (via the `refreshlibinfo` service)
- For ALL libraries when an administrator clicks REFRESH on the System page (via the `refreshlibs` service)
The refresh is driven primarily from `dictionary.libnames`, augmented with library name / id from metadata (SAS 9 only). Noteworthy behaviours:
- The `V9` engine is normalised to `BASE`
- Concatenated libraries produce multiple quoted entries in `PATHS`, with one comma-separated `PERMS` / `OWNERS` value per path
- `SCHEMAS` is populated for database engines
- On SAS 9 (metadata) deployments, invalid libraries are validated by attempting a META libname assignment and skipped on failure. If your environment has invalid libraries that cause exception errors, set the `DC_VIEWLIB_CHECK` config variable to `NO` in Data Controller Settings
- The following librefs are always excluded: `SASWORK`, `WORK`, `SASUSER`, `CASUSER`, `TEMP`, `STPSAMP`, `MAPSGFK`. Additional librefs can be ignored via `DC_IGNORELIBS` (see [Refreshing the Data Catalog](/dcu-datacatalog/#refreshing-the-data-catalog))
The load is TXTEMPORAL on the `LIBREF` key, so records are closed out (not deleted) when a library disappears.
docs/tables/mpe_datacatalog_objs.md
+28
View File
@@ -0,0 +1,28 @@
---
layout: article
title: MPE_DATACATALOG_OBJS
description: The MPE_DATACATALOG_OBJS table contains the objects inside every SAS Catalog
og_title: MPE_DATACATALOG_OBJS Table Documentation
og_image: /img/datacatalog_objs.png
---
# MPE_DATACATALOG_OBJS
The `MPE_DATACATALOG_OBJS` table contains a listing of all the objects available in each SAS Catalog.
More frequently changing attributes are stored in [MPE_DATASTATUS_OBJS](/mpe_datastatus_objs).
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
![](/img/datacatalog_objs.png)
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `MEMNAME char(64)`: The catalog member name
- 🔑 `OBJNAME char(32)`: The object name
- 🔑 `OBJTYPE char(8)`: The object type
- `OBJDESC char(256)`: The object description
- `ALIAS char(32)`: The object alias
docs/tables/mpe_datacatalog_tabs.md
+2 -2
View File
@@ -10,11 +10,11 @@ The `MPE_DATACATALOG_TABS` table catalogs attributes such as number of variables
More frequently changing attributes (such as size modification date and number of observations) are stored in [MPE_DATASTATUS_TABS](/mpe_datastatus_tabs).
To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
## Columns
- `TX_FROM num`: SCD2 open datetime
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `DSN char(64)`: The library member name
docs/tables/mpe_datacatalog_vars.md
+2 -2
View File
@@ -8,11 +8,11 @@ description: The MPE_DATACATALOG_VARS table catalogs variable attributes such as
The `MPE_DATACATALOG_VARS` table catalogs variable attributes such as primary key status, not null constraints and index usage.
To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
## Columns
- `TX_FROM num`: SCD2 open datetime
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `DSN char(64)`: The library member name
docs/tables/mpe_datadictionary.md
+21
View File
@@ -0,0 +1,21 @@
---
layout: article
title: MPE_DATADICTIONARY
description: The MPE_DATADICTIONARY table documents libraries, tables, columns and directories with descriptions, ownership and sensitivity information.
---
# MPE_DATADICTIONARY
The `MPE_DATADICTIONARY` table stores user-maintained documentation for libraries, tables, columns and directories. This content is surfaced in the Data Dictionary view of Data Controller.
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `DD_TYPE char(16)`: The type of item being documented (e.g. LIBRARY, TABLE, COLUMN, DIRECTORY)
- 🔑 `DD_SOURCE char(1024)`: The item being documented (e.g. `libref`, `libref.table`, `libref.table.column` or a directory path)
- `DD_SHORTDESC char(256)`: Short description
- `DD_LONGDESC char(32767)`: Long description (Markdown supported)
- `DD_OWNER char(128)`: Owner of the item
- `DD_RESPONSIBLE char(128)`: Responsible party for the item
- `DD_SENSITIVITY char(64)`: Sensitivity classification (e.g. Low)
- 🔑 `TX_TO num`: SCD2 close datetime
docs/tables/mpe_dataloads.md
+23
View File
@@ -0,0 +1,23 @@
---
layout: article
title: MPE_DATALOADS
description: The MPE_DATALOADS table is an audit trail of every load performed through Data Controller for SAS®, including record counts and duration.
---
# MPE_DATALOADS
The `MPE_DATALOADS` table records an audit entry for every load performed (via the frontend, or via the [bitemporal dataloader macros](/macros/)).
## Columns
- 🔑 `PROCESSED_DTTM num`: Datetime the load completed
- 🔑 `LIBREF char(8)`: SAS Libref of the target table
- 🔑 `DSN char(32)`: Target table name
- 🔑 `ETLSOURCE char(100)`: Source of the load (e.g. the submitting user / service)
- `LOADTYPE char(20)`: The load type applied (UPDATE, REPLACE, TXTEMPORAL, BITEMPORAL, FORMAT_CAT)
- `CHANGED_RECORDS num`: Number of records changed
- `NEW_RECORDS num`: Number of records added
- `DELETED_RECORDS num`: Number of records deleted
- `DURATION num`: Duration of the load (seconds)
- `USER_NM char(50)`: The user who performed the load
- `MAC_VER char(5)`: The version of the Data Controller macros used
docs/tables/mpe_datastatus_cats.md
+27
View File
@@ -0,0 +1,27 @@
---
layout: article
title: MPE_DATASTATUS_CATS
description: The MPE_DATASTATUS_CATS table captures frequently changing SAS catalog attributes such as created / modified datetimes and number of entries.
og_title: MPE_DATASTATUS_CATS Table Documentation
og_image: /img/datastatus_cats.png
---
# MPE_DATASTATUS_CATS
The `MPE_DATASTATUS_CATS` table captures frequently changing SAS table attributes such as created / modified datetimes and number of entries.
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
![](/img/datastatus_cats.png)
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `MEMNAME char(64)`: The catalog member name
- `NOBS num`: The number of catalog entries
- `CREATED num`: Creation datetime (based on earliest created object)
- `MODIFIED num`: Modified datetime (based on last modified object)
docs/tables/mpe_datastatus_libs.md
+12 -2
View File
@@ -8,13 +8,23 @@ description: The MPE_DATASTATUS_LIBS table captures frequently changing SAS libr
The `MPE_DATASTATUS_LIBS` table captures frequently changing SAS library attributes such as size (if filesystem based) and the number of tables.
To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
## Columns
- `TX_FROM num`: SCD2 open datetime
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- `LIBSIZE num`: The size of the library (in bytes), displayed with the SIZEKMG. format. Only applicable to BASE engine libraries.
- `TABLE_CNT num`: The number of tables in the library.
- `CATALOG_CNT num`: The number of SAS Catalogs in the library (from `dictionary.catalogs`).
## Refresh Process
This table is populated by the `mpe_refreshtables` macro when a FULL library refresh is performed (all tables). It does not change when a single table is refreshed.
- `LIBSIZE` is the sum of `filesize` across the library members in `dictionary.tables` - hence it is only applicable to BASE (filesystem) engines. For SQL Server libraries, filesize is not available and row counts are instead taken from `sys.partitions` via pass-through
- `TABLE_CNT` is the count of tables in the library
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
docs/tables/mpe_datastatus_objs.md
+28
View File
@@ -0,0 +1,28 @@
---
layout: article
title: MPE_DATASTATUS_OBJS
description: The MPE_DATASTATUS_OBJS table captures frequently changing SAS catalog object attributes such as created / modified datetimes and library concatenation level
og_title: MPE_DATASTATUS_OBJS Table Documentation
og_image: /img/datastatus_objs.png
---
# MPE_DATASTATUS_OBJS
The `MPE_DATASTATUS_OBJS` table captures frequently changing SAS catalog object attributes such as created / modified datetimes and library concatenation level
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
![](/img/datastatus_objs.png)
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `MEMNAME char(64)`: The catalog member name
- 🔑 `OBJNAME char(32)`: The object name
- 🔑 `OBJTYPE char(8)`: The object type
- `CREATED num`: Creation datetime (based on earliest created object)
- `MODIFIED num`: Modified datetime (based on last modified object)
- `LEVEL num`: Library concatenation level
docs/tables/mpe_datastatus_tabs.md
+2 -2
View File
@@ -8,11 +8,11 @@ description: The MPE_DATASTATUS_TABS table captures frequently changing SAS tabl
The `MPE_DATASTATUS_TABS` table captures frequently changing SAS table attributes such as size (if filesystem based), modification date, and the number of observations.
To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
## Columns
- `TX_FROM num`: SCD2 open datetime
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `DSN char(64)`: The library member name
docs/tables/mpe_derived.md
+26
View File
@@ -0,0 +1,26 @@
---
layout: article
title: MPE_DERIVED
description: The MPE_DERIVED table is used to configure computed columns that appear in the EDIT screen and can be filtered on without being stored in the table.
---
# MPE_DERIVED
The MPE_DERIVED table is used to configure derived (computed) variables - columns that are created when a table is loaded into the EDIT screen, so that they can be displayed and filtered on without being stored. More details are available in the [user guide](/derived-variables/)
The table is itself registered in [MPE_TABLES](/tables/mpe_tables/), so it is maintained through the EDIT screen with the usual review and approve workflow.
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: The libref of the target table
- 🔑 `DSN char(32)`: The target table
- 🔑 `VAR_NM char(32)`: The name of the derived column
- `VAR_DEPS char(4000)`: Every input the expression uses - physical columns and derived variables, space separated
- `VAR_TYPE char(1)`: `C` for character, `N` for numeric
- `VAR_LEN num`: Character length, or numeric length (1-8)
- `VAR_FMT char(32)`: Optional SAS format, eg `DATE9.`
- `VAR_LABEL char(256)`: Optional label; defaults to `<VAR_NM>=<expression>`
- `VAR_LOGIC char(4000)`: The SAS expression that produces the value
- `ACTIVE num`: Set to 1 to apply the variable
docs/tables/mpe_emails.md
+25
View File
@@ -0,0 +1,25 @@
---
layout: article
title: MPE_EMAILS
description: The MPE_EMAILS table is used to map email addresses to individual users
og_title: MPE_EMAILS Table Documentation
og_image: /img/mpe_emails.png
---
# MPE_EMAILS
The MPE_EMAILS table maps emails to user ids. This is helpful in situations where the email address is not automatically available (eg in SAS metadata or the Viya identities service)
![submits](../img/mpe_emails.png)
The table is SCD2 controlled for ease of rollback and version management.
For more information, see the [email config](/emails) page.
## Columns
- 🔑 `TX_FROM num`: SCD2 open datetime
- 🔑 `USER_NAME char(50)`: The system name of the user
- `USER_DISPLAYNAME char(100)`: The name by which the user should be addressed
- `USER_EMAIL char(100)`: The email address of the user
- `TX_TO num`: SCD2 close datetime
docs/tables/mpe_excel_config.md
+34
View File
@@ -0,0 +1,34 @@
---
layout: article
title: MPE_EXCEL_CONFIG
description: The MPE_EXCEL_CONFIG table configures column-level rules applied during Excel uploads in Data Controller for SAS®.
---
# MPE_EXCEL_CONFIG
The `MPE_EXCEL_CONFIG` table configures column-level rules that are applied when uploading data via Excel. See the [Excel](/excel/) guide for more details.
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `XL_LIBREF char(8)`: SAS Libref of the target table
- 🔑 `XL_TABLE char(32)`: Target table name
- 🔑 `XL_COLUMN char(32)`: Column to which the rule applies
- `XL_RULE char(32)`: The rule to apply. Currently the only supported rule is `FORMULA` - this extracts the underlying cell _formula_ (eg `=VLOOKUP(...)`) rather than the raw cell value during an Excel upload. The target column must be character, and wide enough to hold the longest formula.
- `XL_ACTIVE num`: Flag indicating whether the rule is active (1 = active)
- `TX_TO num`: SCD2 close datetime
## Example
The following entry (from the Data Controller sample data) causes the `DD_LONGDESC` column of `MPE_DATADICTIONARY` to be loaded as a formula rather than a raw value when uploading via Excel:
```sas
insert into &lib..MPE_EXCEL_CONFIG set
tx_from=0
,xl_libref="&lib"
,xl_table="MPE_DATADICTIONARY"
,xl_column="DD_LONGDESC"
,xl_rule="FORMULA"
,xl_active=1
,tx_to='31DEC5999:23:59:59'dt;
```
docs/tables/mpe_filteranytable.md
+18
View File
@@ -0,0 +1,18 @@
---
layout: article
title: MPE_FILTERANYTABLE
description: The MPE_FILTERANYTABLE table stores a record for each unique filter clause created in Data Controller for SAS®.
---
# MPE_FILTERANYTABLE
The `MPE_FILTERANYTABLE` table stores a record for each unique filter created via the FILTER menu. When a user submits a filter, the entire clause is hashed - if that hash already exists for the table, the existing `FILTER_RK` is reused, otherwise a new record is added. This means identical filters are only ever stored once, and the `FILTER_RK` can be safely embedded in the shareable URLs described in the [filter](/filter/) guide.
The individual lines of the filter clause itself are stored in [MPE_FILTERSOURCE](/tables/mpe_filtersource/).
## Columns
- 🔑 `FILTER_RK num`: Unique retained key for the filter, used to recall the filter (eg in shareable URLs)
- `FILTER_HASH char(32)`: Hash of the entire filter clause, used to detect duplicate filters and to join to [MPE_FILTERSOURCE](/tables/mpe_filtersource/)
- `FILTER_TABLE char(41)`: The table being filtered (in `libref.dataset` format)
- `PROCESSED_DTTM num`: Datetime the filter was first created
docs/tables/mpe_filtersource.md
+21
View File
@@ -0,0 +1,21 @@
---
layout: article
title: MPE_FILTERSOURCE
description: The MPE_FILTERSOURCE table stores the individual lines of each filter clause created in Data Controller for SAS®.
---
# MPE_FILTERSOURCE
The `MPE_FILTERSOURCE` table stores the individual query lines of each filter created via the FILTER menu, keyed by the hash stored in [MPE_FILTERANYTABLE](/tables/mpe_filteranytable/). See the [filter](/filter/) guide for more details.
## Columns
- 🔑 `FILTER_HASH char(32)`: Hash of the filter clause, joining to [MPE_FILTERANYTABLE](/tables/mpe_filteranytable/)
- 🔑 `FILTER_LINE num`: Line number within the filter clause
- `GROUP_LOGIC char(3)`: AND / OR logic applied between groups
- `SUBGROUP_LOGIC char(3)`: AND / OR logic applied within the subgroup
- `SUBGROUP_ID num`: Identifier of the subgroup to which this line belongs
- `VARIABLE_NM char(32)`: The variable being filtered
- `OPERATOR_NM char(12)`: The filter operator (e.g. `=`, `>`, `IN`)
- `RAW_VALUE char(4000)`: The filter value. A date, datetime or time is stored as a SAS literal (for example `'20APR2042'd`, `'01JAN1960:00:00:42'dt`, `'00:00:42't`), so the clause shown above the grid reads as a date rather than the internal day or second count.
- `PROCESSED_DTTM num`: Datetime the filter line was created
docs/tables/mpe_groups.md
+19
View File
@@ -0,0 +1,19 @@
---
layout: article
title: MPE_GROUPS
description: The MPE_GROUPS table defines optional groups and group membership used to secure access to tables in Data Controller for SAS®.
---
# MPE_GROUPS
The `MPE_GROUPS` table defines optional groups, and the members of those groups, used to secure access in Data Controller.
A more detailed breakdown is available in the [configuration](/dcc-groups/) section.
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `GROUP_NAME char(100)`: The name of the group
- `GROUP_DESC char(256)`: A description of the group
- 🔑 `USER_NAME char(50)`: The user (SAS identity name) who is a member of the group
- `TX_TO num`: SCD2 close datetime
docs/tables/mpe_lineage_cols.md
+27
View File
@@ -0,0 +1,27 @@
---
layout: article
title: MPE_LINEAGE_COLS
description: The MPE_LINEAGE_COLS table stores column-level lineage (forward and reverse) extracted by Data Controller for SAS®.
---
# MPE_LINEAGE_COLS
The `MPE_LINEAGE_COLS` table stores column-level lineage - the column mappings derived from jobs registered in SAS DI Studio. See the [lineage](/dcu-lineage/) guide for more details.
## Columns
- 🔑 `COL_ID char(32)`: Unique identifier of the lineage record
- 🔑 `DIRECTION char(1)`: Lineage direction (e.g. F for forward, R for reverse)
- `JOBNAME char(256)`: Name of the job in which the mapping was found
- `SOURCETABLENAME char(256)`: Name of the source table
- `SOURCECOLNAME char(256)`: Name of the source column
- 🔑 `SOURCECOLURI char(256)`: URI of the source column
- 🔑 `MAP_TYPE char(256)`: The type of mapping
- 🔑 `MAP_TRANSFORM char(256)`: The transformation applied in the mapping
- `TARGETTABLENAME char(256)`: Name of the target table
- `TARGETCOLNAME char(256)`: Name of the target column
- 🔑 `TARGETCOLURI char(256)`: URI of the target column
- `DERIVED_RULE char(500)`: The derivation rule applied
- `LEVEL num`: The depth of the mapping within the lineage tree
- `MODIFIED_DTTM num`: Datetime the record was last modified
- `MODIFIED_BY char(64)`: The user who last modified the record
docs/tables/mpe_lineage_tabs.md
+24
View File
@@ -0,0 +1,24 @@
---
layout: article
title: MPE_LINEAGE_TABS
description: The MPE_LINEAGE_TABS table stores table-level lineage (forward and reverse) extracted by Data Controller for SAS®.
---
# MPE_LINEAGE_TABS
The `MPE_LINEAGE_TABS` table stores table-level lineage - the table-to-table relationships derived from jobs registered in SAS DI Studio. See the [lineage](/dcu-lineage/) guide for more details.
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `JOBID char(17)`: Identifier of the job in which the relationship was found
- `JOBNAME char(128)`: Name of the job
- 🔑 `SRCTABLEID char(17)`: Identifier of the source table
- `SRCTABLETYPE char(16)`: Type of the source table
- `SRCTABLENAME char(64)`: Name of the source table
- `SRCLIBREF char(8)`: Libref of the source table
- 🔑 `TGTTABLEID char(17)`: Identifier of the target table
- `TGTTABLETYPE char(16)`: Type of the target table
- `TGTTABLENAME char(64)`: Name of the target table
- `TGTLIBREF char(8)`: Libref of the target table
docs/tables/mpe_loads.md
+19
View File
@@ -0,0 +1,19 @@
---
layout: article
title: MPE_LOADS
description: The MPE_LOADS table records the status of CSV file loads performed by the Data Controller for SAS® target loader.
---
# MPE_LOADS
The `MPE_LOADS` table tracks the status of CSV file loads processed by the target loader ([mpe_targetloader](/macros/) macro), including failures and their reasons.
## Columns
- 🔑 `CSV_DIR char(255)`: The staged folder reference (mperef) containing the CSV files being loaded
- `USER_NM char(50)`: The user who submitted the load
- `STATUS char(15)`: The status of the load (e.g. IN PROGRESS, SUCCESS, FAILED)
- `DURATION num`: Duration of the load (seconds)
- `PROCESSED_DTTM num`: Datetime the load was processed
- `REASON_TXT char(2048)`: The reason for failure (where applicable)
- `APPROVALS char(64)`: Approval information for the load
docs/tables/mpe_maxkeyvalues.md
+16
View File
@@ -0,0 +1,16 @@
---
layout: article
title: MPE_MAXKEYVALUES
description: The MPE_MAXKEYVALUES table stores the current maximum surrogate / retained key value for each keyed table in Data Controller for SAS®.
---
# MPE_MAXKEYVALUES
The `MPE_MAXKEYVALUES` table stores the current maximum surrogate / retained key value for each table configured with a retained key (see [RK_UNDERLYING](/dcc-tables/#rk_underlying)). It is used to generate new key values during loads.
## Columns
- 🔑 `KEYTABLE char(41)`: Base table in `libref.dataset` format
- `KEYCOLUMN char(32)`: The surrogate / retained key field containing the key values
- `MAX_KEY num`: Integer value representing the current max RK or SK value in the KEYTABLE
- `PROCESSED_DTTM num`: Datetime this value was last updated
docs/tables/mpe_security.md
+1 -1
View File
@@ -22,4 +22,4 @@ Edits to this table are covered by a (backend) post edit hook, which will automa
- 🔑 `DSN char(32)`: The name of the target dataset (or format catalog)
- 🔑 `ACCESS_LEVEL char(100)`: Either VIEW, EDIT or APPROVE
- 🔑 `SAS_GROUP char(100)`: The SAS Group to which to provide access
- `TX_TO num`: SCD2 close datetime
- `TX_TO num`: SCD2 close datetime
docs/tables/mpe_selectbox.md
+23
View File
@@ -0,0 +1,23 @@
---
layout: article
title: MPE_SELECTBOX
description: The MPE_SELECTBOX table configures the dropdown values available for columns of control tables in Data Controller for SAS®.
---
# MPE_SELECTBOX
The `MPE_SELECTBOX` table configures the values that appear in dropdowns when editing control tables (eg `LOADTYPE` in [MPE_TABLES](/tables/mpe_tables/) or `ACCESS_LEVEL` in [MPE_SECURITY](/tables/mpe_security/)).
A more detailed breakdown is available in the [configuration](/dcc-selectbox/) section.
## Columns
- `VER_FROM_DTTM num`: SCD2 open datetime
- 🔑 `SELECTBOX_RK num`: Surrogate key for the selectbox value
- `SELECT_LIB char(17)`: Libref of the table to which the dropdown applies
- `SELECT_DS char(32)`: Name of the table to which the dropdown applies
- `BASE_COLUMN char(36)`: The column against which the dropdown is applied
- `SELECTBOX_VALUE char(500)`: The dropdown value
- `SELECTBOX_ORDER num`: Optional ordering of the dropdown values (1 comes before 2)
- `SELECTBOX_TYPE char(32)`: Column type (blank for default, else `sas` or `js` to indicate relevant system functions)
- `VER_TO_DTTM num`: SCD2 close datetime
docs/tables/mpe_signoffs.md
+21
View File
@@ -0,0 +1,21 @@
---
layout: article
title: MPE_SIGNOFFS
description: The MPE_SIGNOFFS table records signoffs made against loaded data in Data Controller for SAS®.
---
# MPE_SIGNOFFS
The `MPE_SIGNOFFS` table is designed to record signoffs - the final approval step associated with SIGNOFF access (see [MPE_SECURITY](/tables/mpe_security/) and [SIGNOFF_COLS](/dcc-tables/#signoff_cols)).
!!! note
This table is created as part of the Data Controller data model but is not currently populated by any Data Controller service. It is reserved for custom signoff implementations.
## Columns
- 🔑 `TECH_FROM_DTTM num`: SCD2 open datetime
- 🔑 `SIGNOFF_TABLE char(50)`: The table being signed off
- 🔑 `SIGNOFF_SECTION_RK num`: The retained key of the section being signed off
- `SIGNOFF_VERSION_RK num`: The retained key of the version being signed off
- `SIGNOFF_NAME char(100)`: The name of the user performing the signoff
- `TECH_TO_DTTM num`: SCD2 close datetime
docs/tables/mpe_users.md
+15
View File
@@ -0,0 +1,15 @@
---
layout: article
title: MPE_USERS
description: The MPE_USERS table captures the users of Data Controller for SAS® and when they were last seen.
---
# MPE_USERS
The `MPE_USERS` table captures the actual users of the app - each user is registered on first login, and their last seen date is updated on subsequent activity.
## Columns
- 🔑 `USER_ID char(50)`: The user id
- `LAST_SEEN_DT num`: Date the user was last active
- `REGISTERED_DT num`: Date the user first registered
docs/tables/mpe_validations.md
+25
View File
@@ -0,0 +1,25 @@
---
layout: article
title: MPE_VALIDATIONS
description: The `MPE_VALIDATIONS` table enables a number of validations to be applied to the data at the point of entry - such as casing, min/max values, nullability, and the ability to generate dropdown values directly from source tables or dynamically from SAS programs.
og_image: /img/mpe_validations.png
---
# MPE_VALIDATIONS
The `MPE_VALIDATIONS` table enables a number of validations to be applied to the data at the point of entry - such as casing, min/max values, nullability, and the ability to generate dropdown values directly from source tables or dynamically from SAS programs.
A detailed breakdown is available in the [validations](/dcc-validations/) section.
![validations](/img/mpe_validations.png)
## Columns
- 🔑`TX_FROM num`: SCD2 open datetime
- `TX_TO num`: SCD2 close datetime
- 🔑 `BASE_LIB char(8)`: SAS Libref (8 chars)
- 🔑 `BASE_DS char(32)`: The library member name
- 🔑 `BASE_COL char(32)`: The column name
- 🔑 `RULE_TYPE char(32)`: The name of the rule to apply. Valid values include `CASE`, `NOTNULL`, `MINVAL`, `MAXVAL`, `READONLY`, `HIDDEN`, `ROUND`, `NUMBER_FORMAT`, `HARDFORMULA`, `SOFTFORMULA`, `HARDREGEX`, `SOFTREGEX`, `HARDSELECT`, `SOFTSELECT`, `HARDSELECT_HOOK` and `SOFTSELECT_HOOK`.
- `RULE_VALUE char(128)`: The value of the rule.
- `RULE_ACTIVE num`: Set to 1 for an active rule, or 0 to disable the rule.
docs/videos.md
+1 -1
View File
@@ -17,7 +17,7 @@ A collection of videos made in relation to Data Controller. Some of them were r
### Data Controller - a Customer Story
An illustration of the utility of Data Controller for Data & Analytics teams in regulated industries.
<iframe width="560" height="315" src="https://www.youtube.com/embed/M8hafkS4zY4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe title="Data Controller - a Customer Story" width="560" height="315" src="https://vid.4gl.io/videos/embed/kpSGoNE26FSif8X9A6pUfe?peertubeLink=0" frameborder="0" allowfullscreen="" sandbox="allow-same-origin allow-scripts allow-popups"></iframe>
### Use Case Video - Manual Overrides
docs/viewboxes.md
+1 -1
View File
@@ -12,7 +12,7 @@ Often when editing (or examining) raw data, it is helpful to see it alongside re
Each individual viewbox has the following features:
* Choose the columns to display (and which order)
* Resize individual boxes (or reset to original)
* Resize individual boxes by dragging any of the four edges or corners (or reset to original)
* Full filtering capability (complex clauses)
* Minimise / Restore all, or individually
* Reposition - manually, or snap to grid
mkdocs.yml
+46 -7
View File
@@ -14,27 +14,52 @@ nav:
- Table Viewer: dcu-tableviewer.md
- ViewBoxes: viewboxes.md
- Admin Services: admin-services.md
- Developer Guide:
- Filter in a Pre-Edit Hook: developer-guide/filter-pre-edit-hook.md
- One Table, Two Rule Sets: developer-guide/two-rule-sets-one-table.md
- Table Guide:
- MPE_AUDIT: tables/mpe_audit.md
- MPE_COLUMN_LEVEL_SECURITY: tables/mpe_column_level_security.md
- MPE_CONFIG: tables/mpe_config.md
- MPE_DATACATALOG_CATS: tables/mpe_datacatalog_cats.md
- MPE_DATACATALOG_LIBS: tables/mpe_datacatalog_libs.md
- MPE_DATACATALOG_OBJS: tables/mpe_datacatalog_objs.md
- MPE_DATACATALOG_TABS: tables/mpe_datacatalog_tabs.md
- MPE_DATACATALOG_VARS: tables/mpe_datacatalog_vars.md
- MPE_DATASTATUS_CATS: tables/mpe_datastatus_cats.md
- MPE_DATASTATUS_LIBS: tables/mpe_datastatus_libs.md
- MPE_DATASTATUS_OBJ: tables/mpe_datastatus_objs.md
- MPE_DATASTATUS_TABS: tables/mpe_datastatus_tabs.md
- MPE_DATADICTIONARY: tables/mpe_datadictionary.md
- MPE_DATALOADS: tables/mpe_dataloads.md
- MPE_DERIVED: tables/mpe_derived.md
- MPE_EMAILS: tables/mpe_emails.md
- MPE_EXCEL_CONFIG: tables/mpe_excel_config.md
- MPE_FILTERANYTABLE: tables/mpe_filteranytable.md
- MPE_FILTERSOURCE: tables/mpe_filtersource.md
- MPE_GROUPS: tables/mpe_groups.md
- MPE_LINEAGE_COLS: tables/mpe_lineage_cols.md
- MPE_LINEAGE_TABS: tables/mpe_lineage_tabs.md
- MPE_LOADS: tables/mpe_loads.md
- MPE_LOCKANYTABLE: tables/mpe_lockanytable.md
- MPE_MAXKEYVALUES: tables/mpe_maxkeyvalues.md
- MPE_REQUESTS: tables/mpe_requests.md
- MPE_REVIEW: tables/mpe_review.md
- MPE_SUBMIT: tables/mpe_submit.md
- MPE_SECURITY: tables/mpe_security.md
- MPE_SELECTBOX: tables/mpe_selectbox.md
- MPE_SIGNOFFS: tables/mpe_signoffs.md
- MPE_TABLES: tables/mpe_tables.md
- MPE_USERS: tables/mpe_users.md
- MPE_VALIDATIONS: tables/mpe_validations.md
- MPE_XLMAP_DATA: tables/mpe_xlmap_data.md
- MPE_XLMAP_INFO: tables/mpe_xlmap_info.md
- MPE_XLMAP_RULES: tables/mpe_xlmap_rules.md
- Configuration:
- CAS Tables: cas-tables.md
- Column Level Security: column-level-security.md
- Dates / Datetimes: dcc-dates.md
- Derived Variables: derived-variables.md
- Dynamic Cell Dropdown: dynamic-cell-dropdown.md
- Emails: emails.md
- Excel: excel.md
@@ -47,10 +72,12 @@ nav:
- Selectboxes: dcc-selectbox.md
- Tables: dcc-tables.md
- Validations: dcc-validations.md
- Visual Analytics: embed-va.md
- Macros: macros.md
- Installation:
- System Requirements: dci-requirements.md
- SAS Viya: dci-deploysasviya.md
- Downloads: downloads.md
- SAS Viya: deploy-viya.md
- SAS 9 EBI: dci-deploysas9.md
- SAS 9 STP Hardening: dci-stpinstance.md
- Troubleshooting: dci-troubleshooting.md
@@ -74,7 +101,9 @@ markdown_extensions:
extra:
manifest: manifest.webmanifest
extra_css: ['font-awesome.css']
extra_css:
- font-awesome.css
- dc-brand.css
plugins:
- search:
@@ -90,12 +119,22 @@ repo_url: 'https://git.datacontroller.io/dc/docs.datacontroller.io'
theme:
name: material
logo: 'img/favicon.ico'
logo: 'img/dc-logo.svg'
palette:
primary: 'White'
accent: 'Amber'
- scheme: slate
primary: custom
accent: custom
toggle:
icon: material/weather-sunny
name: Switch to light mode
- scheme: default
primary: custom
accent: custom
toggle:
icon: material/weather-night
name: Switch to dark mode
font:
text: 'Open Sans'
text: 'Montserrat'
code: 'Ubuntu Mono'
favicon: img/favicon.ico
custom_dir: 'theme'
@@ -105,4 +144,4 @@ theme:
- JavaScript
- Bash
copyright: All rights reserved &copy;2023 Bowe IO Ltd.
copyright: All rights reserved &copy;2026 Bowe IO Ltd.