docs(7.19): publish derived variables - banners off, nav titles clean, roadmap entry

Held until the release lands: the draft banner on both derived pages, the
(draft) suffix in the nav, and the Delivered Features entry they were waiting
for. Merge this with, or just after, dc/dc#348.

.gitea/workflows/publish.yml

@@ -31,19 +31,22 @@ jobs:
           apt install -y ./google-chrome*.deb;
           export CHROME_BIN=/usr/bin/google-chrome
 
+      # Deploy tooling is pinned to exact versions: the workflow builds whatever
+      # is committed, and an unpinned `latest` would let a breaking or
+      # compromised upstream release ship straight to the live site.
       - name: Install Surfer
         run: |
-          npm -g install cloudron-surfer
+          npm -g install cloudron-surfer@6.4.1
 
       - name: build site
         run: |
-          pip install mkdocs
-          pip install mkdocs-material
-          pip install fontawesome_markdown
-          pip install mkdocs-redirects
+          pip install mkdocs==1.6.1
+          pip install mkdocs-material==9.7.7
+          pip install fontawesome_markdown==0.2.6
+          pip install mkdocs-redirects==1.2.3
           mkdocs build --clean
           mkdir site/slides
-          npx @marp-team/marp-cli slides/innovation/innovation.md -o ./site/slides/innovation/index.html
-          npx @marp-team/marp-cli slides/if/if.md -o site/if.pdf --allow-local-files --html=true
+          npx @marp-team/marp-cli@4.5.1 slides/innovation/innovation.md -o ./site/slides/innovation/index.html
+          npx @marp-team/marp-cli@4.5.1 slides/if/if.md -o site/if.pdf --allow-local-files --html=true
       - name: Deploy docs
         run: surfer put --token ${{ secrets.SURFERKEY }} --server docs.datacontroller.io site/* /

.npmrc

@@ -0,0 +1,11 @@
+# npm configuration for this repo's CI and local installs.
+#
+# ignore-scripts keeps package install scripts (an arbitrary-code execution
+# point) from running during the deploy workflow. Both npm packages the
+# workflow installs - cloudron-surfer and @marp-team/marp-cli - ship no
+# install, preinstall or postinstall hook, so nothing is skipped.
+#
+# save-exact pins anything installed from here to an exact version rather than
+# a caret range.
+ignore-scripts=true
+save-exact=true

.pre-commit-config.yaml

@@ -0,0 +1,11 @@
+# Hooks for the pre-commit framework - https://pre-commit.com
+#
+# This repo carries no native git hooks, so this is the route for a secret scan
+# at commit time: install the framework once with `pre-commit install`, and the
+# scan then runs on every commit. The hook is pinned to an exact gitleaks
+# release tag.
+repos:
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.30.1
+    hooks:
+      - id: gitleaks

docs/admin-services.md

@@ -22,4 +22,3 @@ Button info as follows:
 |Update Licence Key| Link to the screen for providing a new Data Controller licence key|
 |Export DC Library DDL|COMING SOON!!  <br>Exports the data controller control library in DB specific DDL (eg SAS, PGSQL, TSQL) and allows an optional schema name to be included|
 
-

docs/dc-brand.css

@@ -0,0 +1,45 @@
+/*
+ * Data Controller brand overrides for the Material theme.
+ *
+ * The values below mirror the marketing site (https://datacontroller.io): the
+ * slate used for its navbar and hero, and the green used for its logo accent.
+ * They are wired up through the `custom` palette in mkdocs.yml, which leaves
+ * these CSS variables for us to define.
+ */
+
+/* Primary - the brand slate, used for the header, the mobile drawer title and body links. */
+:root,
+[data-md-color-primary='custom'] {
+  --md-primary-fg-color: #314351;
+  --md-primary-fg-color--light: #3d5468;
+  --md-primary-fg-color--dark: #263542;
+  --md-primary-bg-color: #fff;
+  --md-primary-bg-color--light: #ffffffb3;
+  --md-typeset-a-color: #314351;
+}
+
+/* Accent - the brand green, used for link hover, focus rings and the active search hit. */
+[data-md-color-accent='custom'] {
+  --md-accent-fg-color: #79a843;
+  --md-accent-fg-color--transparent: #79a8431a;
+  --md-accent-bg-color: #fff;
+  --md-accent-bg-color--light: #ffffffb3;
+}
+
+/*
+ * Dark scheme. Material derives its dark greys from a hue variable, so pointing
+ * that at the brand slate (206deg) gives the whole scheme the same cast as the
+ * header instead of Material's neutral default.
+ */
+[data-md-color-scheme='slate'] {
+  --md-hue: 206deg;
+}
+
+/*
+ * The brand slate is only readable on a light background, so the dark scheme
+ * uses a light tint of it for body links. Green stays the accent, which keeps
+ * link hover visible in both schemes.
+ */
+[data-md-color-scheme='slate'][data-md-color-primary='custom'] {
+  --md-typeset-a-color: #9fb5c6;
+}

docs/dc-userguide.md

@@ -42,12 +42,24 @@ The Editor screen lets users who have been pre-authorised (via the `DATACTRL.MPE
 
 2 - *Upload*.  If you have a lot of data, you can [upload it directly](files).  The changes are then approved in the usual way.
 
-3 - *Edit*.  This is the main interface, data is displayed in tabular format.  The first column is always "Delete?", as this allows you to mark rows for deletion.  Note that removing a row from display does not mark it for deletion!  It simply means that this row is not part of the changeset being submitted.
+3 - *Edit*.  This is the main interface, data is displayed in tabular format.  The first column is always "Delete?", as this allows you to mark rows for deletion.  This column is created by Data Controller and is not part of your table.  Note that removing a row from display does not mark it for deletion!  It simply means that this row is not part of the changeset being submitted.
 The next set of columns are the Primary Key, and are shaded grey.  If the table has a surrogate / retained key, then it is the Business Key that is shown here (the RK field is calculated / updated at the backend).  For SCD2 type tables, the 'validity' fields are not shown.  It is assumed that the user is always working with the current version of the data, and the view is filtered as such.
 After this, remaining columns are shown.  Dates / datetime fields have appropriate datepickers.  Other fields may also have dropdowns to ensure entry of standard values, these can be configured in the `DATACTRL.MPE_SELECTBOX` table.
 
 New rows can be added using the right click context menu, or the 'Add Row' button.  The data can also be sorted by clicking on the column headers.
 
+#### Native Date and Time Pickers
+
+Date, time, and datetime columns use native browser pickers when editing a cell.  This provides a familiar calendar and time selector, and respects your browser's locale settings for date and time formats.
+
+#### Paste Validation Overlay
+
+When you paste data into the editor (or drag to autofill cells), a confirmation overlay appears so you can review the changes before they are applied.  All pasted values are checked against your configured [validation rules](/dcc-validations/).  For large pastes, a progress indicator shows how many cells have been validated.
+
+#### Row Status Indicators
+
+Each row's header cell is colour-coded to show its current status - modified, added, deleted, or unchanged.  The "modified" indicator uses a `±` symbol so you can quickly spot which rows have changed.
+
 When ready to submit, hit the SUBMIT button and enter a reason for the change.  The owners of the data are now alerted (so long as their email addresses are in metadata) with a link to the approve screen.
 If you are also an approver you can approve this change yourself.
 
@@ -60,7 +72,7 @@ Data Controller supports special missing numerics, ie - a single letter or under
 The Data Controller only permits BiTemporal data uploads at a single point in time - so for convenience, when viewing data in the edit screen, only the most recent records are displayed.  To edit earlier records, either use file upload, or apply a filter.
 
 ### Submitted
-This page shows a list of the changes you have submitted (that are not yet approved).
+This page shows a list of the changes you have submitted (that are not yet approved).  When you open a submitted change for review, you can toggle between viewing the data with SAS formats applied (eg formatted dates and currency) or as raw underlying values.  This is useful when you need to verify the exact value being submitted rather than its display representation.
 
 ### Approvals
 This shows the list of changes that have been submitted to you (or your groups) for approval.

docs/dcc-options.md

@@ -20,16 +20,23 @@ Currently used scopes include:
 ### DC_EMAIL_ALERTS
 Set to YES or NO to enable email alerts. This requires email options to be preconfigured (mail server etc).
 
+### DC_MAXCELLS_WEBEDIT
+By default, a maximum of 200,000 cells (observations multiplied by variables) can be edited in the browser at one time.  This limit is applied together with [DC_MAXOBS_WEBEDIT](#dc_maxobs_webedit) - whichever is reached first.
+
+The cell count matters because the cost of an EDIT screen load tracks the number of cells, not the number of rows.  A 10 column table is comfortable at 250 rows, but 250 rows of a 1000 column table is 250,000 cells.
+
 ### DC_MAXOBS_WEBEDIT
-By default, a maximum of 100 observations can be edited in the browser at one time.  This number can be increased, but note that the following factors will impact performance:
+By default, a maximum of 250 observations can be edited in the browser at one time.  This number can be increased, but note that the following factors will impact performance:
 
 * Number of configured [Validations](/dcc-validations)
 * Browser type and version (works best in Chrome)
 * Number (and size) of columns
 * Speed of client machine (laptop/desktop)
 
+A wide table can reach the [DC_MAXCELLS_WEBEDIT](#dc_maxcells_webedit) limit well before this one, so increase both together.
+
 ### DC_MAXOBS_WEBVIEW
-By default, a maximum of 500 observations can be viewed in the browser at one time. Please see previous section for items to consider if increasing this value.
+By default, a maximum of 2000 observations can be viewed in the browser at one time. Please see previous section for items to consider if increasing this value.
 
 ### DC_REQUEST_LOGS
 On SASjs Server and SAS9 Server types, at the end of each DC SAS request, a record is added to the [MPE_REQUESTS](/tables/mpe_requests) table.  In some situations this can cause table locks.  To prevent this issue from occuring, the `DC_REQUEST_LOGS` option can be set to `NO` (Default is `YES`).

docs/dcc-tables.md

@@ -16,6 +16,9 @@ Adding tables to the Data Controller is a matter of configuration, specifically
 
 ![screenshot](img/configtable.png)
 
+!!! note
+    `_____DELETE__THIS__RECORD_____` is a reserved name. The editor creates that column itself - it is the **Delete?** column - and never reads it from the table, so a target table must not contain a column of that name. The editor refuses to open a table that does, rather than let the two collide.
+
 ## MPE_TABLES Configuration Details
 
 Each table to be edited in the Data Controller is represented by one record in `DATACTRL.MPE_TABLES`. The fields should be populated as follows:

docs/dcc-validations.md

@@ -23,18 +23,20 @@ It is possible to configure a number of other rules by updating the MPE_VALIDATI
 
 ## Configurable Checks
 
-Check back frequently as we plan to keep growing this list of checks.
+Check back frequently as we keep growing this list of checks.  For how each rule treats a special missing value (`.A`-`.Z`, `._`), see [Special Missing Values](#special-missing-values).
 
 |Rule Type|Example Value |Description|
 |---|---|---|
 |CASE|UPCASE|Will enforce the case of cell values.  Valid values: UPCASE, LOWCASE, PROPCASE|
-|NOTNULL|(defaultval)|Will prevent submission if null values are present.  Optional - provide a default value.|
-|MINVAL|1|Defines a minimum value for a numeric cell|
-|MAXVAL|1000000|Defines a maximum value for a numeric cell|
+|NOTNULL|(defaultval)|Will prevent submission if null values are present - a special missing counts as null.  Optional - provide a default value.|
+|MINVAL|1|Defines a minimum value for a numeric cell.  A special missing value sorts below every number, so it fails a numeric floor, and the floor itself may be a special missing - see [Special Missing Values](#special-missing-values).|
+|MAXVAL|1000000|Defines a maximum value for a numeric cell.  A special missing value sorts below every number, so it passes a numeric ceiling, and the ceiling itself may be a special missing - see [Special Missing Values](#special-missing-values).|
 |READONLY|(defaultval) |Renders the column read-only in the editor.  The defaultval is used when rows are added. |
 |HIDDEN|(defaultval) |Hides the column from the editor grid while still submitting its data. The defaultval is used when rows are added. |
-|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc.  Half-away-from-zero rounding is applied so `-0.5 → -1` and `2.5 → 3`. |
+|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc.  Half-away-from-zero rounding is applied so `-0.5 -> -1` and `2.5 -> 3`. |
 |NUMBER_FORMAT|`{"style":"currency","currency":"GBP"}` |Display-only [`Intl.NumberFormat` renderer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat).  RULE_VALUE is the JSON options object passed straight to `Intl.NumberFormat`.  Does not change the stored value. |
+|HARDFORMULA|`= PRICE * VOLUME`|The cell displays a value computed from a formula, using other columns in the same row.  The column is read-only - the user cannot override the result.  See [Formula Rules](#formula-rules) below.|
+|SOFTFORMULA|`= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )`|Like HARDFORMULA, but the user can override the computed value and type their own.  See [Formula Rules](#formula-rules) below.|
 |HARDREGEX|`^[A-Z]{3}$`|The cell value **must** match the regex pattern, otherwise submission is blocked and the cell is highlighted red.  See [Regex Rules](#regex-rules) below.|
 |SOFTREGEX|`^[A-Z]{3}$`|A cell value that does not match the regex pattern is highlighted yellow as a warning, but submission is **not** blocked.  See [Regex Rules](#regex-rules) below.|
 |HARDSELECT|sashelp.class.name|A distinct list of values (max 1000) are taken from this library.member.column reference, and the value **must** be in this list.  This list may be supplemented by entries in the MPE_SELECTBOX table.|
@@ -43,6 +45,77 @@ Check back frequently as we plan to keep growing this list of checks.
 |[SOFTSELECT_HOOK](/dynamic-cell-dropdown)|/physical/path/program.sas|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem.  User-provided values may (or may not) be in this list.  Cannot be used alongside a HARDSELECT_HOOK.|
 
 
+## Special Missing Values
+
+A SAS numeric variable has 28 missing values: the regular `.` and 27 special ones (`.A`-`.Z` and `._`).  Data Controller accepts them in a numeric cell as a letter - `a` to `z` - or an underscore, with or without a leading period (`a`, `.a`, `_`, `._`), and stores the matching special missing in SAS.  The period is optional, and a plain `.` stores the regular missing.
+
+A numeric column that carries a date, datetime or time format is the exception.  Those cells are edited through a date picker rather than the numeric editor, and the picker accepts only a date - so a special missing cannot be typed into one of them.
+
+The rules treat a special missing as a value for some checks and as a missing value for others - and the range rules compare it in the order SAS gives it, below every number but ordered among the other missing values:
+
+|Rule|What happens to a special missing|
+|---|---|
+|NOTNULL|Fails.  A special missing is a missing value, so a NOT NULL (or primary key) constraint rejects it, and a real SAS NOT NULL constraint on the target table rejects it too.|
+|MINVAL|Fails if the minimum is a number - a missing sorts below every number, so it is below any numeric floor.  A minimum that is itself a special missing is compared in the order the missing values have among themselves, and can accept one.|
+|MAXVAL|Passes - a missing sorts below every number, so it is below any numeric ceiling.  A ceiling that is itself a special missing is compared in the missing values' own order, and can reject one.|
+|CASE|Not applicable - it is a character rule.|
+|ROUND|Ignored - ROUND only rounds values that are numbers, so a special missing is left as typed.|
+|HARDREGEX / SOFTREGEX|Checked against the pattern like any other value.  Unlike blanks and the plain `.`, special missings are **not** exempt.  See [Regex Rules](#regex-rules).|
+|SOFTSELECT|Supports special missings - a soft dropdown never blocks a value.|
+|HARDSELECT|Supports special missings - the value is matched against the dropdown list like any other value.|
+|HARDFORMULA / SOFTFORMULA|The formula returns `#VALUE!` for that row instead of a number, and that is what gets submitted.|
+
+In short, a special missing is rejected by NOTNULL, by a primary key column, and by a MINVAL whose floor is a number; a formula returns `#VALUE!` for it.  It is fine in a column that carries a MAXVAL, a regex, a dropdown, or ROUND - and it can satisfy a MINVAL when the floor is expressed as a special missing.
+
+A range rule compares in the order SAS itself uses, so a range written in special missings means what SAS would mean by it.  That order puts every missing below every non-missing value, and orders the missing values as `._`, then the regular missing, then `.A` through `.Z`.  So `MINVAL .A` with `MAXVAL .C` accepts `.B` and rejects `.D`, and a blank - the regular missing - fails a floor of `.A` because it sorts below it.  A number sorts above every missing, so it passes a floor of `.A` and fails a ceiling of `.C`.
+
+A primary key column is treated as NOT NULL whether or not the target table carries a physical constraint, and whether or not MPE_VALIDATIONS has a NOTNULL rule for it - a key identifies the row, so a blank and a special missing are both rejected there.
+
+A range rule's own value is read as a number when it looks like one and as a missing value when it is a special missing.  A value that is neither - a typo such as `..` or `AB` - satisfies nothing, so every cell in the column fails until the rule is corrected.
+
+## Formula Rules
+
+HARDFORMULA and SOFTFORMULA let you configure a column so that its value is automatically calculated from other columns in the same row - just like a spreadsheet formula.  When a user opens the editor, the formula is evaluated live and the result is shown in each cell.
+
+### Writing a formula
+
+Formulas use column names, not cell references, so there is no need to know the grid layout.  For example, if you have PRICE and VOLUME columns, a REVENUE column formula would be:
+
+```
+= PRICE * VOLUME
+```
+
+Each row calculates its own result - the PRICE in row 1 is multiplied by the VOLUME in row 1, the PRICE in row 2 by the VOLUME in row 2, and so on.
+
+!!! note
+    Each column name in the formula must be surrounded by spaces (eg ` PRICE ` not `PRICE`) so it is recognised as a column reference rather than part of a function name.  For example, `=MATCH( PRICE )` resolves the column reference, but `=MATCH(PRICE)` does not - PRICE is left unrecognised and the formula will error rather than using the column's value.  Text inside quotes is left as-is.
+
+### HARDFORMULA vs SOFTFORMULA
+
+* **HARDFORMULA** - the column is read-only.  The formula result is always shown and submitted.  The user cannot change it.
+* **SOFTFORMULA** - the cell shows the formula result but the user can type a different value if needed.  If they do, their value is submitted instead.
+
+### Editor behaviour
+
+* When you paste a formula into the grid, column names are automatically translated so the formula works in its new position.
+* A cell that is overwritten by a formula is flagged so you can revert it.
+* Formula-looking values pasted from Excel are treated as plain data (not evaluated), unless you explicitly choose "Apply as formula".
+
+### Special values
+
+Formulas can reference three special values that are resolved at runtime:
+
+* `DC.ROW_STATUS` - the current state of the row: `M` (Modified), `A` (Added), `D` (Deleted), or `U` (Unchanged).  A newly-added row is `A` from the moment it is created - there is no transient state before that.
+* `DC.USER_NAME` - the logged-in user id.
+* `DC.ORIG_VALUE` - the original cell value before the current edit.
+
+Example - show the current user id if the row is changed, otherwise keep the original value:
+
+```
+= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )
+```
+
+
 ## Regex Rules
 
 HARDREGEX and SOFTREGEX validate cell values against a SAS (Perl-style) regular expression provided in `RULE_VALUE` - the same syntax accepted by [PRXPARSE](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lefunctionsref/p0s9ilagexmjl8n1u7e1t1jfnzlk.htm).

docs/dci-deploysas9.md

@@ -14,6 +14,8 @@ There are two ways to deploy Data Controller on SAS 9:
 * Full Deployment (preferred)
 * Streaming (for quick demos)
 
+Both fetch a deployment program from the [releases page](https://git.datacontroller.io/dc/dc/releases) - see the [downloads page](/downloads) for what each asset contains and how to verify it.
+
 ### Full Deployment
 
 #### 1 - Deploy Stored Processes
@@ -27,16 +29,16 @@ filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/sa
 %inc dc;
 ```
 
-If you don't have internet access from SAS, download `sas9.sas` from [here](https://git.datacontroller.io/dc/dc/releases), and change the initial `compiled_apploc` and `compiled_serverName` macro variable assignments as necessary.
+If you don't have internet access from SAS, download `sas9.sas` from [here](https://git.datacontroller.io/dc/dc/releases), verify it against the release `SHA256SUMS` file ([downloads page](/downloads/#verifying-a-download)), and change the initial `compiled_apploc` and `compiled_serverName` macro variable assignments as necessary.
 
 #### 2 - Deploy the Frontend
 
-The Data Controller frontend comes pre-built, and ready to deploy to the root of the SAS Web Server (mid-tier).
+The Data Controller frontend comes pre-built, and ready to deploy to the SAS Web Server (mid-tier).
 
 Deploy as follows:
 
-1.  Download the `frontend.zip` file from: [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases)
-2. Unzip and place in the [htdocs folder of your SAS Web Server](https://sasjs.io/frontend-deployment/#sas9-deploy) - typically a subdirectory of: `!SASCONFIG/LevX/Web/WebServer/htdocs`.
+1.  Download the `frontend.zip` file from: [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases) (verify it against the release `SHA256SUMS` file - see the [downloads page](/downloads/#verifying-a-download))
+2. Create a folder for Data Controller in the [htdocs folder of your SAS Web Server](https://sasjs.io/frontend-deployment/#sas9-deploy) - typically a subdirectory of: `!SASCONFIG/LevX/Web/WebServer/htdocs` - and unzip the archive **into** that folder. The archive holds the frontend files at its root, so `index.html` sits directly inside your folder (alongside the `images` folder and the compiled asset files) - for example `htdocs/dc/index.html`.
 3. Open the `index.html` file and update the values in the `<sasjs>` tag as follows:
 
   * `appLoc` - same as per SAS code in the section above
@@ -47,7 +49,7 @@ Deploy as follows:
 
 The remaining properties are not relevant for a SAS 9 deployment and can be **safely ignored**.
 
-You can now open the app at `https://YOURWEBSERVER/unzippedfoldername`  (step 2 above) and follow the configuration steps (DC Physical Location and Admin Group) to complete deployment.
+You can now open the app at `https://YOURWEBSERVER/<yourfoldername>`  (the folder created in step 2 above) and follow the configuration steps (DC Physical Location and Admin Group) to complete deployment.
 
 #### 3 - Run the Configurator
 
@@ -93,7 +95,7 @@ filename dc url "https://git.datacontroller.io/dc/dc/releases/download/vX.X.X/de
 %inc dc;
 ```
 
-If you don't have internet access from your SAS environment, just download `demostream_sas9.sas` from [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases) and modify the `appLoc` on line 2, as follows:
+If you don't have internet access from SAS, download `demostream_sas9.sas` from [https://git.datacontroller.io/dc/dc/releases](https://git.datacontroller.io/dc/dc/releases) (verify it against the release `SHA256SUMS` file - see the [downloads page](/downloads/#verifying-a-download)) and modify the `appLoc` on line 2, as follows:
 
 ![](img/sas9_apploc.png)
 

docs/dci-troubleshooting.md

@@ -108,8 +108,131 @@ This can happen if you enter the wrong `serverName` when deploying the SAS progr
 
 The error may also be thrown due to an encoding issue - changing to a UTF-8 server has helped at least one customer.
 
+## Displayed timestamps are in UTC (or the wrong timezone)
+
+Data Controller records timestamps (such as the SUBMITTED column on the Submitted screen, and the audit history) using the SAS session clock, via the `datetime()` function. That function returns the time of the operating system, adjusted by the [TIMEZONE= system option](https://documentation.sas.com/doc/en/pgmsascdc/default/lesysoptsref/p15siqs0s00e50n1wuuvygzkr14r.htm) if it is set. The value is then displayed in the frontend exactly as stored, without conversion.
+
+On Viya, the SAS compute sessions that run Data Controller jobs are started inside Kubernetes containers whose clock is UTC by default, and which inherit no timezone from the host machine. If the TIMEZONE= option is not set for the compute context used by Data Controller, every timestamp DC records and displays is UTC. Other SAS products can appear unaffected because clients such as SAS Studio create their own compute sessions and pass the browser timezone / locale, whereas Data Controller submits jobs to its own shared compute context, which gets the raw container clock.
+
+To confirm the current state, run the following in a SAS session under the Data Controller compute context (or check the DC job log, where `_DEBUG` output shows the same values):
+
+```sas
+proc options option=timezone;
+run;
+%put &=SYSTIMEZONEIDENT &=SYSTIMEZONEOFFSET;
+```
+
+If TIMEZONE is blank (and SYSTIMEZONEOFFSET is 0), the session is on UTC. To fix it, ask your Viya administrator to set the timezone for the compute context used by Data Controller (in SAS Environment Manager, edit the context's Autoexec, or set the context's SAS options):
+
+```sas
+options timezone='Europe/Berlin';
+```
+
+Then recycle any existing compute sessions - hot sessions keep the old setting until they terminate.
+
+!!! note
+    Use a region/area time zone ID such as `Europe/Berlin` rather than a fixed offset such as `GMT+2`. Fixed offsets do not follow daylight saving time, so the display would be 1 hour off in winter (CET = UTC+1).
+
+!!! note
+    Changing the timezone affects new timestamps only. Previously recorded submissions keep the UTC values that were stored when they were created.
+
 ## Determining Application Version
 
 The app version is bundled into the frontend during the release, and is visible by clicking your username in the top right.
 
-You can also determine the app version (and SASjs Version, and build time) by opening browser Development Tools and running `appinfo()` in the console.
+You can also determine the app version (and SASjs Version, and build time) by opening browser Development Tools and running `appinfo()` in the console.
+
+## Support Diagnostics
+
+The frontend sends two input tables with the startup service and with the services that execute customer-provided code - the [hook scripts](macros.md) (pre/post edit and approve hooks) and the dynamic cell dropdown programs. They are available in those services as `work.browser_info` and `work.browser_url_vars`, so a support ticket can be diagnosed from the job log without asking the user follow-up questions, and a hook can adapt to its context. Other services do not receive them.
+
+The services that receive the tables are: `public/startupservice`, `editors/getdata`, `editors/getdynamiccolvals`, `editors/stagedata`, `editors/loadfile`, `editors/restore` and `auditors/postdata`.
+
+`editors/loadfile` is the exception in practice: the app reaches it through the adapter's file-upload call (multipart), which carries no input tables, so the service - and the post edit hook it runs - does not receive them in normal use. It is on the list for completeness, and would receive them if the service were ever invoked through the ordinary request path.
+
+### browser_info
+
+A single-row table with these columns:
+
+| Column | Description |
+|---|---|
+| `url` | The URL of the Data Controller page itself (the iframe), not the document embedding it. When the editor is embedded in a report, the embedding URL is visible here. |
+| `referrer` | The embedding document, from `document.referrer`. For an embedded report this is the report URL, so the embedding report can be told apart from the editor URL. |
+| `timezone` | The browser's IANA timezone name, e.g. `Europe/Berlin`. |
+| `tz_offset` | The browser's UTC offset in minutes, as `Date.getTimezoneOffset()` returns it (positive west of UTC, so `Europe/Berlin` in summer is -120). |
+| `locale` | The browser locale, e.g. `en-GB`. |
+| `dc_version` | The Data Controller build, as shown by `appinfo()`. |
+| `adapter_version` | The `@sasjs/adapter` version the client was built with. |
+| `browser` | The browser family, parsed from the user agent. |
+| `browser_version` | The browser version. |
+| `platform` | The operating system, parsed from the user agent. |
+| `user_agent` | The raw `navigator.userAgent` string. |
+
+### browser_url_vars
+
+The page URL parameters as one row per parameter, so a SAS program can read a parameter by name rather than parse the `url` string:
+
+| Column | Description |
+|---|---|
+| `name` | The URL parameter name. |
+| `value` | The URL parameter value. |
+
+Parameters from both the search string and the hash query string (Angular routes carry them after the `#`) are included; on a name collision the search string value wins. The table is sent only when the page URL has at least one parameter.
+
+On SAS 9 and Viya the app is reached through the platform's own job URL, so the platform's parameters (`_FILE`, `_program`, `_debug`, `_webout` and friends) are in the table alongside DC's. Read a parameter by name rather than assuming every row belongs to Data Controller - `embed` and `labels` are DC's, `_FILE` and `_program` are the platform's.
+
+The values in both tables are supplied by the client, so treat them as diagnostics hints rather than a security boundary.
+
+To see the tables, run any of the services above with debug on. The debug value depends on the path: `&_debug=131` on the Compute API and SAS 9, or `&_debug=128` on the Viya web (JES) path with `runAsTask` enabled - which is what the frontend sends there, so turning debug on in the app is enough. Either value makes the session initialisation write the tables to the job log:
+
+```
+NOTE: MPEINIT: work.browser_url_vars:
+name=embed value=va
+NOTE: MPEINIT: work.browser_info:
+url=... referrer=... timezone=Europe/Berlin tz_offset=-120 locale=en-GB
+dc_version=7.15.0 adapter_version=4.19.0 browser=Chrome browser_version=120.0
+platform=Linux user_agent=...
+```
+
+A service called directly (for instance from a URL, or by a script), or by an older frontend, has no `work.browser_info` - the log then records that fact instead.
+
+### Reading the tables from a hook
+
+A [hook script](macros.md) is `%include`d into the running service, so it shares the service's WORK library and can read the tables directly. Take an embedded editor opened at:
+
+```
+https://datacontroller.io/#/editor/SASHELP.CLASS?labels=true&embed=va
+```
+
+`browser_url_vars` then holds a row per parameter - `labels=true` and `embed=va` - so a hook can pick up either by name:
+
+```sas
+%let labels=N;
+%let embed=N;
+
+%if %sysfunc(exist(work.browser_url_vars)) %then %do;
+  proc sql noprint;
+    select value into :labels
+    from work.browser_url_vars
+    where name = 'labels';
+    select value into :embed
+    from work.browser_url_vars
+    where name = 'embed';
+  quit;
+%end;
+```
+
+To branch on what the client reports:
+
+```sas
+%if %sysfunc(exist(work.browser_info)) %then %do;
+  data _null_;
+    set work.browser_info;
+    call symputx('dc_timezone', timezone);
+    call symputx('dc_locale', locale);
+  run;
+%end;
+```
+
+!!! warning
+    Guard with `%sysfunc(exist())` before reading. A service called directly, or by a frontend older than this feature, has no `work.browser_info` at all, so an unguarded `set` or `select` fails the job.

docs/dcu-tableviewer.md

@@ -16,6 +16,18 @@ Row and Column level security can also be applied in VIEW mode, as can additiona
 
 A single search box can be used to make a full table search on any character or numeric value, using this [macro](https://core.sasjs.io/mp__searchdata_8sas.html).
 
+The search covers **every** column of the table at once, so you do not need to know which column holds the value you are looking for:
+
+- **Character search** (the default) is a case sensitive CONTAINS match against every character column, so `Smith` finds `Smithson` and `smith` finds `Goldsmith`, but `smith` will not find `Smithson`.
+- **Numeric search** (tick the Numeric box) is an exact match against every numeric column, so `42` will not find `420` or `4210`.
+- If a filter is applied, the search runs within the filtered rows rather than the whole table.
+- The grid is capped by the [`DC_MAXOBS_WEBVIEW`](/dcc-options/#dc_maxobs_webview) option (500 rows by default).
+- For a search, the cap applies to the row count as well: a search that matches more rows than the cap returns 500 of them, and the count shown next to the table name is the number of rows returned, not the total number of matches.
+- Without a search, the count shown next to the table name is the full number of rows in the filtered view, so it can be higher than the number of rows displayed - a table of 1,200 rows reports `1,200 rows` above a grid holding the first 500.
+- A search that matches nothing shows a "No data found with given conditions" panel rather than an empty grid.
+
+![Full table search](img/full-table-search.png)
+
 <iframe width="560" height="315" src="https://www.youtube.com/embed/i27w-xq85WQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
 
 ## Options

docs/deploy-viya.md

@@ -130,7 +130,6 @@ Now the services are deployed (including the service which creates the staging a
 * Admin Group - the members of this group will have full access to Data Controller
 * Compute Context - the context configured in Step 1
 
-
 ### Optimisation
 
 At this point, every DC request will read the `services/public/settings.sas` file to get the DC library (and other) settings.  To avoid these API calls (which will speed up the app) we can simply move this code to the autoexec.  Steps as follows:
@@ -174,4 +173,6 @@ To update DC, just deploy it as a fresh instance, then move the new config acros
 
 It is possible to embed a Data Controller table within SAS Visual Analytics by simply pasting the URL.
 
-To make the portlet more visually appealing, the Data Controller title bar can be removed by adding `?embed=true` to the URL.  When opening in a new window, the title bar will be gone.
+To make the portlet more visually appealing, the Data Controller title bar can be removed by adding `?embed=true` to the URL.  When opening in a new window, the title bar will be gone.
+
+For a deeper VA integration - where report row selections drive filters and column visibility in the Data Controller editor - append `?embed=va` instead.  See the [SAS Visual Analytics Embed](/embed-va/) page for details on filter modes, configuration, and debugging.

docs/derived-variables.md

@@ -0,0 +1,131 @@
+---
+layout: article
+title: Derived Variables
+description: Derived variables are computed columns that appear in the EDIT screen and can be filtered on, without being stored in the target table.
+---
+
+# Derived Variables
+
+A derived variable is a column that does not exist in the target table, but is computed when the table is loaded into the [EDIT screen](/dc-userguide/#editor). It is read-only, it appears after the physical columns, and it can be filtered on like any other column.
+
+Derived variables are useful when users need to work with a value that is a function of the stored data - a banding, a concatenation, a calculation - without changing the table, the load process, or the ETL.
+
+The values are **not stored**. They are computed on every read, and nothing derived is written back when a change is submitted or approved.
+
+## What it looks like
+
+The table below is shown in the EDIT screen. `LINE_VALUE` is `QTY * PRICE` and `REGION` is derived from `STATE`; neither exists in the table itself, and both are read-only.
+
+![](./img/derived-grid.png)
+
+Because the derived columns are created before the filter is applied, they can be filtered like any other column. Here `REGION = 'West'` selects the two western states, and the filter bar names the derived variable:
+
+![](./img/derived-filter-region.png)
+
+A numeric derived column works the same way - `LINE_VALUE > 50` leaves the one row that satisfies it:
+
+![](./img/derived-filter-value.png)
+
+## How it works
+
+The statements are written into a DATA step view over the target table, and the filter is applied to that view rather than to the table:
+
+```sas
+data work.vw_derived / view=work.vw_derived;
+  set &libds;
+  attrib REGION length=$8 label='...' format=$8.;
+  REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
+run;
+
+data out;
+  set work.vw_derived;
+  where %inc filtref;;
+run;
+```
+
+A `WHERE` statement applies to the data set named on the `SET` statement, so in a single step it cannot see a variable created later in the same step. Putting the derived variables on the view means they are on the input to the step that filters, which is what makes them filterable at all.
+
+## Configuration
+
+Derived variables are configured per table in the [MPE_DERIVED](/tables/mpe_derived/) control table, which is itself a Data Controller table - so it is maintained through the EDIT screen with the usual review and approve workflow. The three variables shown above, and nothing else, are the three rows below:
+
+![](./img/derived-config.png)
+
+
+Each row defines one variable:
+
+ - `LIBREF` / `DSN` - the target table
+ - `VAR_NM` - the name of the new column
+ - `VAR_DEPS` - every input the expression uses: the physical columns and any derived variables, space separated
+ - `VAR_TYPE` - `C` for character, `N` for numeric
+ - `VAR_LEN` - character length, or numeric length (1-8)
+ - `VAR_FMT` - optional, eg `DATE9.` or `DOLLAR12.2`
+ - `VAR_LABEL` - optional; when blank the label is `<VAR_NM>=<expression>`
+ - `VAR_LOGIC` - the SAS expression that produces the value
+ - `ACTIVE` - set to 1 to apply the variable
+
+A simple example, deriving a region from a state:
+
+```
+VAR_NM      REGION
+VAR_DEPS    STATE
+VAR_TYPE    C
+VAR_LEN     8
+VAR_FMT     (blank)
+VAR_LABEL   (blank)
+VAR_LOGIC   ifc(STATE in ('CA','OR','WA'),'West','East')
+ACTIVE      1
+```
+
+A second variable can build on it - its dependencies include the physical column and the derived one:
+
+```
+VAR_NM      REGION_LABEL
+VAR_DEPS    REGION STATE
+VAR_TYPE    C
+VAR_LEN     14
+VAR_FMT     (blank)
+VAR_LABEL   Region and state
+VAR_LOGIC   cats(REGION,'-',STATE)
+ACTIVE      1
+```
+
+There is no sequence number to maintain. The statements are emitted in dependency order, so `REGION` is always assigned before `REGION_LABEL`, however the rows are ordered in the table.
+
+For each row, Data Controller generates the following statements when the table is read:
+
+```sas
+attrib REGION length=$8 label='REGION=ifc(STATE in (''CA'',''OR'',''WA''),''West'',''East'')' format=$8.;
+REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
+```
+
+The label is `VAR_LABEL` when one is supplied, otherwise the variable name, an equals sign and the expression - `REGION=ifc(...)` - so the column shows both what it is and where it comes from when labels are switched on. Either way the label is truncated to the maximum label length, on a boundary that cannot split a doubled quote.
+
+The type and length are declared, not inferred, so a value is truncated exactly as you specify. If `VAR_FMT` is left blank, character variables default to `$<length>.` and numeric variables to `<length>.`; a date, time or datetime format makes the column render (and filter) as a date, time or datetime, the same way a stored column does.
+
+## Filtering
+
+The derived variable is created before the filter is applied, so it can be used in the [filter](/filter/) like any other column - including its list of values. A filter on a derived column is stored and re-applied in the normal way, and the value list for a derived column is built from the table under the rest of the filter, exactly as for a stored column.
+
+Derived columns themselves are an EDIT screen feature: the VIEW screen and the download services do not show them. A saved filter that references a derived column still applies there, and is evaluated the same way.
+
+## Rules and limits
+
+ - **Expressions only.** `VAR_LOGIC` must be a single SAS expression - no semicolons, macro triggers (`%`, `&`) or comments. Conditional logic is available through functions such as `ifc()`, `choosec()`, `cats()`, `catx()`, `put()`, `input()` and `intnx()`.
+ - **Dependencies declared.** `VAR_DEPS` lists every input the expression uses - physical columns and derived variables alike. Statements are generated in dependency order, so no sequence number is maintained, and the declaration is checked in both directions: every reference must be declared, and every declared name must be referenced and must exist. Circular references are rejected.
+ - **No collisions.** `VAR_NM` must not match an existing column of the target table, or `_____DELETE__THIS__RECORD_____`.
+ - **Read-only.** Derived columns cannot be edited in the grid, and any value submitted for them is discarded before the approval package is created.
+ - **Recomputed on every read.** The value always reflects the current data and the current definition, and there is no history of previous derived values.
+
+A configuration error - a bad name, an invalid type or length, a name collision, or a reference to a derived variable that is not declared in `VAR_DEPS` - stops the EDIT screen with a message naming the variable. A syntax error or an unknown column in the expression itself is caught when the view is compiled, and stops the screen with a message naming the table.
+
+Most of these rules are also applied when a row is submitted for approval, through a post edit hook on `MPE_DERIVED`. A bad name, type, length, format or expression - or a reference to a derived variable that is not declared in `VAR_DEPS` - is rejected at submit time, while the error can still be corrected, rather than breaking the EDIT screen afterwards. The collision between `VAR_NM` and a physical column of the target table is only checked at read time, as the hook does not have the target library assigned.
+
+The simple rules are also configured as validation rules on the table itself, so they show up in the grid before submission: `VAR_TYPE` is a dropdown of `C` and `N`, `VAR_LEN` carries minimum and maximum values, and `VAR_NM`, `VAR_DEPS`, `VAR_FMT` and `VAR_LOGIC` are pattern checked. Validation rules are evaluated in the browser, so the post edit hook remains the authoritative check.
+
+!!! warning "Derived values can expose hidden data"
+    An expression can reference a column that is hidden from the user by [Column Level Security](/column-level-security/), and the derived column will show the result. Configure derived variables with the same care as any other data exposure.
+
+## Performance
+
+The expression is evaluated for each row read, before the filter. The filter already requires a scan of the table, so this adds per-row arithmetic rather than a second pass - but keep expressions cheap (no lookups, no subqueries), and expect the cost to be paid on every load of the EDIT screen, including each request for filter values.

docs/developer-guide/filter-pre-edit-hook.md

@@ -0,0 +1,153 @@
+---
+layout: article
+title: Updating the Filter in a Pre-Edit Hook
+description: Read the filter a user applied, and hand back a modified one, from a Data Controller pre-edit hook - including changing the rows the editor shows.
+---
+
+# Updating the Filter in a Pre-Edit Hook
+
+A [pre-edit hook](/dcc-tables/#pre_edit_hook) can read the filter a user applied and return a modified one. This is useful when the editor should widen a user's selection behind the scenes - for example, when a user filters for the current quarter and the grid should also show the previous one.
+
+The hook runs inside the `editors/getdata` service, so both the applied filter and the filtered rows are in scope. This page describes what is available at that point and works through an example.
+
+## What the editor receives
+
+When a filtered editor loads, `editors/getdata` returns the filter to the client in two forms:
+
+- `sasparams.FILTER_TEXT` - the resolved WHERE clause, which the editor shows above the grid.
+- `query` - the filter's clauses as structured rows (`variable_nm`, `operator_nm`, `raw_value`), which the editor renders as the filter chips.
+
+The response does not carry the filter key (`FILTER_RK`). That is an *input* to `getdata`, not an output - the editor already holds it in the URL (`/editor/<table>/<FILTER_RK>`), so there is nothing to return.
+
+## When the hook runs
+
+`getdata` builds the filter, applies it, and sorts the result, all before the pre-edit hook runs. In order:
+
+1. the WHERE clause is written to a temporary fileref, `filtref`;
+2. `work.OUT` is populated with `where %inc filtref` - the filter is applied here;
+3. the pre-edit hook runs;
+4. `FILTER_TEXT` is read back out of `filtref`, and the response is streamed.
+
+So the hook sits between "the rows are filtered" and "the filter is returned". Two consequences follow:
+
+- It can change what the editor is told the filter was, by rewriting `filtref` and, if the chips must match, `work.query`.
+- It cannot change which rows came back *that* way, because the data step has already run. To return different rows, the hook must re-read the table itself.
+
+## Take the value from the data
+
+A hook that wants to extend a date filter needs the date the user is working with. It does not have to parse the filter text for it - the filtered rows are already in `work.OUT`, so the value can be read from there.
+
+## Example: widen a date filter to the previous quarter
+
+The hook below is registered in `MPE_TABLES.PRE_EDIT_HOOK` for a table with a date column `ORDER_DATE`. When a user opens the editor with a filter that includes `ORDER_DATE`, it extends the filter to also cover the previous quarter, re-reads the table so the grid shows both quarters, and returns the widened filter text.
+
+```sas
+/* PRE_EDIT_HOOK: widen the user's ORDER_DATE filter by one quarter */
+
+%if &filter_rk > 0 %then %do;
+
+  /* the date the user is looking at, taken from the rows already returned */
+  %let refdt=;
+  proc sql noprint;
+    select max(ORDER_DATE) into :refdt from work.out;
+  quit;
+
+  %if &refdt ne %then %do;
+
+    /* the previous quarter, relative to that date */
+    %let pq_from=%sysfunc(putn(%sysfunc(intnx(quarter,&refdt,-1,b)),date9.));
+    %let pq_to  =%sysfunc(putn(%sysfunc(intnx(quarter,&refdt,-1,e)),date9.));
+
+    /* the user's own clauses, as a standalone WHERE */
+    %mp_filtergenerate(work.query,outref=userref)
+
+    /* Data Controller's own clause - current-record validity, plus RLS */
+    %mpe_filtermaster(EDIT,&orig_libds,
+      dclib=&mpelib,
+      filter_rk=-1,
+      outref=ctlref,
+      outds=work.ctlq
+    )
+
+    /* read each clause (which may span several lines) into a macro variable */
+    %let usertext=;
+    data _null_;
+      infile userref end=eof;
+      input;
+      length t $32767;
+      retain t;
+      t=catx(' ',t,_infile_);
+      if eof then call symputx('usertext',t);
+    run;
+    %let ctltext=;
+    data _null_;
+      infile ctlref end=eof;
+      input;
+      length t $32767;
+      retain t;
+      t=catx(' ',t,_infile_);
+      if eof then call symputx('ctltext',t);
+    run;
+
+    /* ( user's filter OR previous quarter ) AND ( validity + RLS ) */
+    filename newfilt temp;
+    data _null_;
+      file newfilt lrecl=32767;
+      length f $32767;
+      f='(( '!!trim(symget('usertext'))!!' ) OR (ORDER_DATE between "&pq_from"d and "&pq_to"d))'
+        !!' AND ('!!trim(symget('ctltext'))!!' )';
+      put f;
+    run;
+
+    /* re-read the table, so the grid shows both quarters */
+    data work.out (drop=&var_txfrom &var_txto &var_processed &drop_rk);
+      set &orig_libds;
+      where %inc newfilt;
+    run;
+
+    /* keep the order the editor first showed */
+    proc sort data=work.out;
+      by &sortpk;
+    run;
+
+    /* hand the widened filter back to the editor */
+    data _null_;
+      file filtref lrecl=32767;
+      infile newfilt;
+      input;
+      put _infile_;
+    run;
+
+  %end;
+%end;
+```
+
+### What the hook does
+
+- **The reference date.** `work.OUT` holds the rows the user's filter returned, so the latest `ORDER_DATE` on screen is read straight from it. Nothing has to be parsed out of the filter text.
+- **The previous quarter.** `INTNX` moves back one quarter from that date and returns its first and last day, formatted as SAS date literals (`01APR2026` and `30JUN2026`, for example).
+- **The user's clause.** `work.QUERY` is the filter's clauses as structured rows, and `%mp_filtergenerate` turns them back into a WHERE clause. This is the user's filter on its own, with no Row Level Security or validity clause mixed in.
+- **Data Controller's clause.** Calling `%mpe_filtermaster` with `filter_rk=-1` produces the part of the filter that is not the user's: the current-record validity clause and any Row Level Security, or `1=1` when there is neither.
+- **The combined filter.** The two clauses are combined as `( user OR previous-quarter ) AND ( validity + RLS )`, so the previous quarter is added to the user's selection without widening anything the user is not entitled to see.
+- **The rows.** `work.OUT` is rebuilt with the combined filter, using the same drop list as the data step `getdata` runs itself - the transaction and processing columns, plus `&drop_rk`, the surrogate key that `getdata` hides when a table is registered with `RK_UNDERLYING` - and sorted on `&sortpk`, because `getdata` sorts before the hook runs and does not sort again afterwards. On a bitemporal table `getdata` also deduplicates to the latest record per primary key (`by &pk &var_busfrom`); add that step too, or the re-read returns every temporal version rather than the current one. The delete column is **not** re-added: `_____DELETE__THIS__RECORD_____` is created by the editor itself and is never a column of the dataset, so a hook must not introduce it (a response that carries that name is refused by the editor).
+- **The returned text.** `filtref` is overwritten with the combined filter, so the clause the editor shows is the one that was actually applied.
+
+### What it looks like
+
+The captures below are the editor for a table with an `ORDER_DATE` column, opened with a filter for the current quarter. In the first, no hook is configured, so the filter is the user's own and the grid holds one quarter of rows. In the second, the hook has run: the filter above the grid carries both quarters, and the rows have grown to match.
+
+![The editor with the user's own filter, one quarter of rows](/img/pre-edit-hook-before.png)
+
+![The same editor after the hook widened the filter to the previous quarter](/img/pre-edit-hook-after.png)
+
+## Row Level Security
+
+The user's clause and Data Controller's clause have to be kept apart, because Row Level Security lives in the second one. Combining them as `( user OR previous-quarter ) AND ( validity + RLS )` keeps RLS applied to both branches. Wrapping the whole existing filter instead - `( existing OR previous-quarter )` - would let the previous-quarter rows through without their RLS condition, so do not take that shortcut.
+
+## The row cap
+
+`getdata` refuses a result larger than `DC_MAXOBS_WEBEDIT` (500 by default) with *Table is too big*. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap. Check the count before handing the data back, or raise the option for that table in `MPE_CONFIG`.
+
+## Matching the filter chips
+
+The clause above the grid comes from `FILTER_TEXT`, so overwriting `filtref` is what updates it. The filter chips are rendered from the `query` output instead, so if they must reflect the widened filter too, amend `work.QUERY` in the same hook - append the extra clause as a new subgroup with `group_logic='OR'` before the response is built.

docs/developer-guide/two-rule-sets-one-table.md

@@ -0,0 +1,122 @@
+---
+layout: article
+title: One Table, Two Rule Sets
+description: A single physical table can carry only one set of Data Controller validation rules. Two librefs over the same data - or a pair of hook scripts - give you as many rule sets as you need.
+---
+
+# One Table, Two Rule Sets
+
+Most Data Controller sites settle into an obvious mapping: one table, one edit screen, one set of validation rules. But that is not always what the business wants. A table of orders might be edited from a finance report that only tolerates small adjustments, and from an operations report where much larger ones are routine. Same table, same columns, same approvers - different rules.
+
+Data Controller's validation rules are configured per table, so this takes a little thought. There are two ways to do it: the one we recommend, and the one to reach for when the first is not available.
+
+## Why one table is one rule set
+
+Two configuration tables decide this.
+
+`MPE_TABLES` is the list of editable tables, and its primary key is `(tx_from, libref, dsn)`. One physical table is one editable table.
+
+`MPE_VALIDATIONS` holds the rules, and its primary key is `(tx_from, base_lib, base_ds, base_col, rule_type)`. Rules hang off a physical `libref.dataset`. There is no per-menu or per-report scoping anywhere in the schema, and the editor is handed exactly the rules whose `base_lib` and `base_ds` match the table being opened.
+
+So two rule sets on one table need two distinct `libref.dataset` identities. The question is how to get them without copying the data.
+
+## Option 1 (recommended): two librefs over the same data
+
+A libref is just a name pointing at a location. Nothing stops you assigning two of them to the same place, and Data Controller will treat the two as separate tables:
+
+```sas
+libname ORDERS_EU '/data/orders';
+libname ORDERS_US '/data/orders';
+```
+
+`ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` are now the same physical file, but they are different rows in `MPE_TABLES` and can therefore carry different rows in `MPE_VALIDATIONS`. Register both, give each its own rules, and point each report at its own editor URL - `#/editor/ORDERS_EU.ORDERS` and `#/editor/ORDERS_US.ORDERS`.
+
+![The first report rejects an amount of 5000](/img/rule-set-1.png)
+
+![The second report accepts the same value](/img/rule-set-2.png)
+
+Everything else works exactly as it always did. Filtering, search, the row cap, the approval diff and the audit trail all operate on the table as normal, because as far as Data Controller is concerned these are ordinary tables. The only difference is that the two names resolve to the same file, so an approval in either report updates the same data - and that one difference has a consequence for concurrency, which is covered below.
+
+There is no copy to keep in sync, no hook to write and nothing to maintain. That is why it is the option we recommend.
+
+### Things worth knowing
+
+- Where the librefs are defined depends on your platform. On Viya, in the compute context's `autoexec.sas` (or `[DC Drive Path]/services/settings.sas`); on SAS 9, as metadata libraries or in the Data Controller Settings stored process; on SASjs Server, in `services/public/settings.sas`. The one requirement is that each library has a unique libref.
+- `mp_lockanytable` keys on `libref.dataset`, so the two menus do not serialise against each other. That has consequences beyond a collision - see *The concurrency caveat* below.
+- The audit trail and approval queue record which libref a change came through, so `ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` stay distinguishable in history. For most people that is a feature - you can see which report a change originated from.
+
+### The concurrency caveat
+
+Data Controller serialises its writes with `mp_lockanytable`, and the control table it uses, `MPE_LOCKANYTABLE`, has a primary key of `(lock_lib, lock_ds)`. That is the *registration*, not the physical file. At approve time, `postdata` takes the lock on the base table named in the submit record, and the same service runs its "has this table been updated since the diff screen was shown" check against `MPE_DATALOADS`, keyed on that same `libref` and `dsn`.
+
+Two librefs over one location are two different `(libref, dsn)` pairs, so:
+
+- two approvals arriving through different reports take two different locks, and neither one blocks the other;
+- a load through one report writes its `MPE_DATALOADS` entry against its own name, so the other report's staleness check does not see it either.
+
+So two people editing the same rows through different reports can both be approved, and the second write wins, silently. The lock is advisory to begin with - `mp_lockanytable` is, in its own words, "only useful if every update uses the macro" - so this is not a new class of risk, but registering one file twice is a new way to fall into it.
+
+The hook route does not have this problem. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity.
+
+If you do use two librefs, and concurrency matters to you, the cleanest fix is to keep the two registrations for the edit screen and give each one a `POST_EDIT_HOOK` that re-points the changeset at a single canonical registration - then every approval is keyed on the same table regardless of which report raised it. Register that canonical name as well. The alternative, an explicit shared lock taken in `PRE_APPROVE_HOOK` and released in `POST_APPROVE_HOOK`, works too, but a failed run leaves the sentinel locked and `MPE_LOCKANYTABLE` is then a table you have to unpick by hand.
+
+## Option 2: an empty mirror and a pair of hook scripts
+
+Sometimes two librefs over one location are not available: a database library where the platform will not let you define the same object twice, or a site where adding a library definition is a change nobody wants to make. Then you can reach the same result with a mirror table and two hook scripts.
+
+The idea is that the thing Data Controller edits is not the real table at all, but an empty table of the same shape, with hooks moving data in and out of it:
+
+- a `PRE_EDIT_HOOK` fills the editor with the live rows of the real table, so the mirror never has to hold a copy
+- a `POST_EDIT_HOOK` re-points the submitted changeset at the real table, so the approval is raised against the real table and the load writes there
+
+The mirror exists purely to carry the rule set, and never stores any data.
+
+### The pre-edit hook
+
+`PRE_EDIT_HOOK` runs inside the `getdata` service, after the user's filter has been applied and the rows sorted, with the data in `work.OUT`. It may replace that dataset, which is all this needs:
+
+```sas
+data work.out;
+  set ORDERS.ORDERS;
+run;
+```
+
+The registered table is `ORDERS.MIRROR`, which is empty - so without the hook the editor would show nothing at all. With it, the grid shows the live rows:
+
+![An empty mirror displaying the live rows of the real table](/img/mirror-live.png)
+
+Note the title bar: the mirror really is empty. Everything on screen came from the hook, and the grid is validated against the mirror's own rules rather than the real table's.
+
+### The post-edit hook
+
+This is the part that surprises people. `POST_EDIT_HOOK` runs inside the `mpe_loader` macro at submit time, on the staged rows, before the submit record is written. It cannot choose the target table directly - but at that point `LIBREF` and `DS` are still ordinary macro variables, and the submit record is built from them. Reassigning them re-points the whole changeset:
+
+```sas
+data _null_;
+  call symputx('libref','ORDERS');
+  call symputx('ds','ORDERS');
+run;
+```
+
+From that moment the changeset is an approval against `ORDERS.ORDERS`. The approver sees a diff against the real table, the load writes to the real table, and the mirror is never touched.
+
+![A change submitted against the mirror, raised against the real table](/img/approval-routed.png)
+
+### Reassigning LIBREF and DS
+
+`LIBREF` and `DS` are ordinary macro variables inside `mpe_loader`, which creates them with `%let` and then updates them from the staged row. A `%let libref=ORDERS;` reassigns them just as well as the `call symputx` above: both statements assign to the *existing* variable, because it lives in an enclosing symbol table (`mpe_loader`'s local table) and SAS only creates a new local when the name is absent from every enclosing table. `call symputx` is used above because it is explicit about which variable is being set, and because it does not depend on the enclosing scope still holding the name - a wrapper macro that declared `%local libref` would turn a bare `%let` into an assignment to that shadow instead.
+
+### Other things to watch
+
+- Concurrency behaves correctly here. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity - unlike the two-libref route above.
+- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap will stop the edit screen with "Table is too big". [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter.
+- The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load.
+- The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table.
+- The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical.
+- At approval time the access checks run against the real table, so editors need `EDIT` on the mirror while approvers need `EDIT` and `APPROVE` on the real table.
+
+## Which should you use?
+
+If you can define two librefs over the same data, do that. It is configuration only, it leaves every other behaviour of the editor untouched, and there is nothing to maintain.
+
+Reach for the hook scripts when the platform will not let you duplicate the library definition, or when you specifically want the rule set to be a property of the application rather than of the data.

docs/downloads.md

@@ -0,0 +1,52 @@
+---
+layout: article
+title: DC Downloads
+description: Data Controller release assets and how to verify them
+---
+
+# Downloads
+
+Data Controller releases are published on the [releases page](https://git.datacontroller.io/dc/dc/releases) of the source repository. Every release carries the same set of assets:
+
+| Asset | Use |
+|---|---|
+| `frontend.zip` | The pre-built web frontend, for deployments that serve the frontend from a web server (SAS 9 full deploy, Viya separated deploy) |
+| `sas9.sas` | SAS 9 full deployment program (stored processes + frontend registration) |
+| `demostream_sas9.sas` | SAS 9 streaming deployment program (demos and evaluations) |
+| `viya.sas` | Viya streaming deployment program (backend + frontend) |
+| `viya_noweb.sas` | Viya separated deployment program (backend only) |
+| `viya_noweb.json` | The Viya backend services as a SASjs Drive JSON deployment bundle |
+| `sasjs_server.json.zip` | SASjs Server deployment bundle (all services, for [SASjs Server](https://server.sasjs.io) deployments) |
+| `SHA256SUMS` | SHA-256 checksums for every asset above |
+
+Which asset you need depends on the deployment route - see [SAS 9 Deployment](/dci-deploysas9) or [SAS Viya Deployment](/deploy-viya).
+
+## Verifying a download
+
+Every asset is covered by the `SHA256SUMS` file, which the release pipeline generates over the exact files it uploads. To verify a download:
+
+1. Download the asset (for example `frontend.zip`) from the [releases page](https://git.datacontroller.io/dc/dc/releases).
+2. Download `SHA256SUMS` from the same release into the same folder.
+3. Run the check:
+
+```bash
+sha256sum --check SHA256SUMS --ignore-missing
+```
+
+Each asset that matches reports `OK`. The `--ignore-missing` flag means you only need the files you actually downloaded - the check skips assets you did not fetch.
+
+On Windows, `certutil` can compute the hash of a single file for comparison against the value listed in `SHA256SUMS`:
+
+```
+certutil -hashfile frontend.zip SHA256
+```
+
+### What the checksum proves
+
+A matching checksum confirms the asset is byte-for-byte the file the release pipeline uploaded. It catches corrupted and tampered downloads - interrupted transfers, proxies or mirrors that altered the file in transit.
+
+It is not a cryptographic signature. The hashes travel in the same release as the files they cover, so they protect the download path, not the release itself: a compromise of the forge would allow both the asset and its checksum to be altered together. For download integrity - the typical concern when fetching deployment artifacts over the internet - it is the standard check.
+
+## Integrity inside SAS
+
+The SAS deployment programs embed the frontend and services as generated SAS code. Once `%inc`'d, their integrity is established by the check above at download time. They deploy the stored processes or jobs into the SAS Folder you nominate (`appLoc`) - a metadata location, not physical disk - and the only physical location they write to is the Data Controller location (`dcLoc`) you configure. The programs can be [reviewed in full](https://code.datacontroller.io) (or in the [source repository](https://git.datacontroller.io/dc/dc)) before you run them.

docs/embed-va.md

@@ -38,6 +38,15 @@ If VA sends an empty / unmatched message the editor falls back to the unfiltered
 
 More logic available in [`va-messaging.service.ts'](https://git.datacontroller.io/dc/dc/src/branch/main/client/src/app/services/va-messaging.service.ts).
 
+## Filter Modes
+
+When running in `embed=va` mode, the editor provides two filter modes, controlled by the **Auto-apply** checkbox:
+
+* **Live (default)** - the editor updates automatically as you select rows in the VA report, so the data you see always matches your current selection.
+* **Confirm** - filter changes are held until you click the **Apply** button.  This is useful when editing, where an automatic reload would discard unsaved changes.
+
+A status indicator shows whether a filter change is pending, loading, or idle.  In edit mode, a pending filter is held until you leave edit mode, so your unsaved edits are never lost.
+
 ## Configuration in VA
 
 In the VA Report Designer, add a **Data-Driven Content** object and set the URL to the editor route shown above.  Be sure that any report level filters have their corresponding parameters added to the DDC object itself.

docs/filter.md

@@ -9,6 +9,8 @@ og_image:  https://docs.datacontroller.io/img/filter_dynamic_on.png
 
 Data Controller for SAS® enables you to create complex table filters. The "dynamic" setting enables the dropdown values to be pre-filtered by previous filter clauses.  Filtered views are shareable!  They are also used when [embedded in VA.](/embed-va)
 
+The clause shown above the grid is the resolved WHERE clause that was applied, so it uses SAS syntax - a date, datetime or time filter reads as a SAS literal, eg `SOME_DATE = '20APR2042'd`, rather than the internal day or second count.
+
 ## Shared Filters
 
 When filters are submitted, the query is stored, and a unique URL is generated.  This means you can share the link to a filtered view of a table!  This can be used for VIEW, for EDIT and also for downloading data.

docs/img/dc-logo.svg

@@ -0,0 +1,15 @@
+<svg width="70" height="70" viewBox="0 0 70 70" fill="none" xmlns="http://www.w3.org/2000/svg">
+<g clip-path="url(#clip0_101_8)">
+<path d="M37.5993 16.6212C38.5916 16.7415 39.5538 16.8317 40.5461 16.9519C42.9216 17.2827 45.237 17.7337 47.402 18.8463C48.0334 19.1771 48.6348 19.5379 49.0859 20.1092C49.8075 20.9512 49.7774 21.8533 49.0558 22.6952C48.304 23.5371 47.3418 24.0183 46.3195 24.4092C43.944 25.3413 41.4482 25.7623 38.9224 25.9728C34.8029 26.3035 30.7435 26.1532 26.7142 25.191C25.3911 24.8903 24.0981 24.4693 22.9254 23.8078C22.4744 23.5672 22.0534 23.2665 21.6625 22.9057C20.58 21.8533 20.58 20.7407 21.6926 19.7484C22.8954 18.6659 24.3688 18.1547 25.8722 17.7337C27.9771 17.1324 30.112 16.8317 32.277 16.6813C32.4575 16.6813 32.6679 16.7415 32.8183 16.5911C34.3819 16.6212 36.0056 16.6212 37.5993 16.6212Z" fill="#E0E0E0"/>
+<path d="M20.8507 41.3082C22.7751 43.7738 25.5115 44.6759 28.338 45.2773C33.9309 46.4801 39.5238 46.3297 44.9663 44.4053C46.6502 43.8039 48.2138 42.992 49.3264 41.4886C49.5068 41.248 49.597 41.3683 49.6872 41.5487C49.8677 41.9396 49.9579 42.3305 49.9579 42.7515C49.9579 44.2249 49.9579 45.7284 49.9579 47.2018C49.9579 48.164 49.5068 48.9157 48.8453 49.5772C47.4621 50.9604 45.7181 51.7122 43.8838 52.2835C40.1853 53.4261 36.3966 53.6667 32.5777 53.3961C29.7212 53.1856 26.9548 52.6443 24.3387 51.4716C23.2261 50.9604 22.2038 50.3591 21.3919 49.4269C20.7605 48.7052 20.4297 47.8934 20.4297 46.9311C20.4598 45.578 20.4297 44.2249 20.4297 42.8718C20.4297 42.3305 20.5199 41.8193 20.8507 41.3082Z" fill="#8EC63F"/>
+<path d="M20.8807 32.0167C22.4744 34.1516 24.6996 35.0537 27.075 35.6852C32.7582 37.1886 38.4413 37.1586 44.0643 35.4747C45.9286 34.9334 47.6726 34.1216 49.0258 32.6482C49.2062 32.4677 49.3265 32.0468 49.5369 32.1069C49.8376 32.1971 49.8376 32.6181 49.9278 32.9488C50.0481 33.46 49.988 33.9712 49.988 34.4824C49.988 35.5047 49.9579 36.497 49.988 37.5194C50.0481 38.7823 49.5369 39.7746 48.6048 40.5865C47.1915 41.8494 45.4776 42.6011 43.6433 43.0822C37.4791 44.7361 31.3449 44.706 25.3009 42.6011C23.8275 42.09 22.4744 41.3683 21.3919 40.1956C20.7304 39.4739 20.3997 38.632 20.3997 37.6397C20.4297 36.3166 20.3997 34.9635 20.3997 33.6404C20.4297 33.0992 20.5199 32.5579 20.8807 32.0167Z" fill="#E0E0E0"/>
+<path d="M20.8207 23.0861C21.2717 24.4091 22.3542 25.0105 23.4969 25.5517C25.5416 26.514 27.7366 26.965 29.9317 27.2357C34.2918 27.7769 38.6519 27.7168 42.9518 26.8147C44.6958 26.4539 46.3797 25.9727 47.9132 25.0105C48.7853 24.4693 49.2363 23.9581 49.537 23.1161C49.8377 23.5371 49.9279 23.9882 49.9279 24.4392C49.9279 25.9427 49.958 27.4161 49.9279 28.9195C49.8978 30.1825 49.1762 31.0845 48.244 31.8363C46.5 33.2195 44.4553 33.9411 42.3203 34.4223C36.6673 35.6551 31.0744 35.5649 25.5717 33.7006C24.1283 33.2195 22.7451 32.5279 21.6326 31.4454C20.7906 30.6335 20.3396 29.7014 20.3696 28.4986C20.3997 27.2056 20.3696 25.9126 20.3696 24.6196C20.4298 24.0483 20.52 23.5672 20.8207 23.0861Z" fill="#8EC63F"/>
+<path d="M55.3403 63.4393C60.2417 63.7099 62.8577 57.696 59.3095 54.2981C55.5809 50.7199 49.4467 54.0576 50.409 59.1093C28.2478 73.2419 -0.258034 52.1031 7.86071 26.4238L2.38808 22.7553C-9.91033 55.4408 27.9471 83.0746 55.3403 63.4393ZM53.7767 56.3729C56.062 53.9974 59.7004 57.3051 57.5655 59.8009C55.3103 62.4169 51.3411 58.8988 53.7767 56.3729Z" fill="#E0E0E0"/>
+<path d="M10.6272 15.2681C14.1453 18.6358 19.9487 15.8694 19.6179 11.0583C41.749 -3.19461 70.4052 17.9142 62.2865 43.6836L67.7591 47.3521C79.6967 15.5988 43.463 -12.9972 15.4383 6.21715C10.1761 5.13465 6.77827 11.5695 10.6272 15.2681ZM16.1599 13.1933C13.8145 15.6289 10.116 12.0807 12.4614 9.64507C14.8068 7.20944 18.5054 10.7576 16.1599 13.1933Z" fill="#8EC63F"/>
+</g>
+<defs>
+<clipPath id="clip0_101_8">
+<rect width="70" height="70" fill="white"/>
+</clipPath>
+</defs>
+</svg>

docs/index.md

@@ -31,7 +31,7 @@ Data Controller is regularly updated with new features.  If you see something th
 
 * [Excel uploads](/excel) - drag & drop directly into SAS. All versions of excel supported.
 * Data Lineage - at both table and column level, export as image or CSV
-* Data Validation Rules - both automatic and user defined
+* Data Validation Rules - both automatic and user defined, including [live formulas](/dcc-validations/#formula-rules)
 * Data Dictionary - map data definitions and ownership
 * Data Catalog - including primary key extraction
 * DDL generator - in SAS, TSQL and PGSQL flavours
@@ -42,6 +42,7 @@ Data Controller is regularly updated with new features.  If you see something th
 * [Row Level Security](/row-level-security)
 * Excel [formula support](excel)
 * Dynamic [cell dropdown](/dynamic-cell-dropdown)
+* [SAS Visual Analytics embed](/embed-va/) - drive the editor from VA report selections
 * Works on ALL flavours of SAS (Base, EBI, Viya)
 
 

docs/roadmap.md

@@ -20,76 +20,9 @@ When features are requested, we will describe the work to be performed in the se
 
 The following features are currently requested:
 
-* Additional Validations
 * Ability to set 'number of approvals' to zero, enabling instant updates (4 days)
 * Ability to make automated submissions using an API
 
-### Additional Validations
-
-The following additional features are necessary:
-
-* Frontend Formulae
-* Regex Rules
-
-The above rules will apply only at frontend, and will be configurable in the MPE_VALIDATIONS table.  The values will appear as part of the [editors/getdata](https://code.datacontroller.io/getdata_8sas_source.html) service response in the `dqrules` object.
-
-**Frontend Formulae**
-
-The plan here will be to introduce [hyperformula](https://hyperformula.handsontable.com/guide/demo.html) into HandsOnTable.  It will allow a library of ~400 functions and advanced excel-like behaviour.  It integrates natively [as a plugin with HandsOnTable](https://handsontable.com/docs/javascript-data-grid/formula-calculation/#available-functions).
-
-The initial challenge will be that HyperFormula requires cell references to operate, eg:
-
-|ITEM|PRICE|VOLUME|REVENUE|
-|---|---|---|---|
-|PAPER|4.20|100|`= B1 * C1`|
-|PEN|61.02|1,971|`= B2 * C2`|
-
-Whereas end users cannot know these references ahead of time.  Therefore the references will be made using variable names, eg:
-
-
-|ITEM|PRICE|VOLUME|REVENUE|
-|---|---|---|---|
-|PAPER|4.20|100|`= PRICE * VOLUME`|
-|PEN|61.02|1,971|`= PRICE * VOLUME`|
-
-
-The frontend can then perform replacement of the variables for each Formula cell.  For instance, replacing ` PRICE ` with ` B1 ` and ` VOLUME ` with ` C1 ` in the first row (and with ` B2 ` and ` C2 ` in the second row respectively).
-
-To avoid clashes with names that match function names (eg, `MATCH()`) each named variable **must have a leading and trailing blank**, and we should be sure to ignore variables inside of single or double quoted strings - eg ` ITEM & " string ITEM "` (would resolve to `A1 & "string ITEM "`).
-
-There would be two types applied:
-
-* `HARDFORMULA` -> Column is readonly 
-* `SOFTFORMULA` -> Column can be changed by the user
-
-To enable conditional logic (eg, show the current user id if row is changed), we also need a new column in the EDIT grid, to show the edit status (Modified, Added, Deleted, Unchanged).  This should be the first column, and should NOT be submitted to backend.  We could display these values as icons, rather than letters.
-
-These properties can be accessed using the following literals:
-
-* DC.ROW_STATUS - replaced at runtime with the cell reference, eg A1 or A2.  Values would be M, A, D, or U.
-* DC.USER_NAME - replaced at runtime with the logged-in user id
-* DC.ORIG_VALUE - replaced at runtime with the original cell value
-
-An example of a rule value that intends to show the current user id if the row is changed:
-
-```
-RULE_VALUE= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )
-```
-
-Which would translate to the following formula, after the page is loaded:
-
-```
-=if(A1!='U',"sasdemo","sasinstaller")
-```
-
-**Regex Rules**
-
-There are two types of rule we can apply in the form of regular expressions:
-
-* `HARDREGEX` -> If the value fails the rule, the data cannot be submitted (turns red)
-* `SOFTREGEX` -> If the value fails the rule we change the cell colour to yellow (as a warning), but can still submit
-
-
 ### Set Approvals to Zero
 
 The following changes are necessary to implement this feature:
@@ -112,6 +45,10 @@ Described [here](https://docs.datacontroller.io/api/).
 
 Below are some examples of Features that have been requested (and delivered) into Data Controller.
 
+### Derived Variables
+
+Columns that do not exist in the table - a banding, a concatenation, a calculation - can be configured per table and appear in the EDIT screen, read-only and filterable, without changing the table or the load process. Nothing derived is stored.  See [docs](/derived-variables)
+
 ### Column Level Security
 
 The columns that can be VIEWed or EDITed are now configurable.  See [docs](/column-level-security)
@@ -236,4 +173,12 @@ It is now possible to restore any change by heading to the particular staged dat
 
 This will submit a NEW change (which must first be approved) that will revert the table to state it was in just prior to the selected upload.
 
-Note that Data Controller can only track (and revert) changes that are made using the Data Controller tool itself! It does not / cannot track changes made externally to a table, by other tools.
+Note that Data Controller can only track (and revert) changes that are made using the Data Controller tool itself! It does not / cannot track changes made externally to a table, by other tools.
+
+### Frontend Formulae
+
+Delivered in v7.13.0.  Formula columns (HARDFORMULA / SOFTFORMULA) are computed live in the editor using HyperFormula, with variable-name references resolved to row-relative cell references.  See [Formula Rules](/dcc-validations/#formula-rules).
+
+### Regex Rules
+
+Delivered.  HARDREGEX (blocking) and SOFTREGEX (warning) rules validate cell values against a Perl-style regular expression.  See [Regex Rules](/dcc-validations/#regex-rules).

docs/tables/mpe_derived.md

@@ -0,0 +1,26 @@
+---
+layout: article
+title: MPE_DERIVED
+description: The MPE_DERIVED table is used to configure computed columns that appear in the EDIT screen and can be filtered on without being stored in the table.
+---
+
+# MPE_DERIVED
+
+The MPE_DERIVED table is used to configure derived (computed) variables - columns that are created when a table is loaded into the EDIT screen, so that they can be displayed and filtered on without being stored.  More details are available in the [user guide](/derived-variables/)
+
+The table is itself registered in [MPE_TABLES](/tables/mpe_tables/), so it is maintained through the EDIT screen with the usual review and approve workflow.
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: The libref of the target table
+ - 🔑 `DSN char(32)`: The target table
+ - 🔑 `VAR_NM char(32)`: The name of the derived column
+ - `VAR_DEPS char(4000)`: Every input the expression uses - physical columns and derived variables, space separated
+ - `VAR_TYPE char(1)`: `C` for character, `N` for numeric
+ - `VAR_LEN num`: Character length, or numeric length (1-8)
+ - `VAR_FMT char(32)`: Optional SAS format, eg `DATE9.`
+ - `VAR_LABEL char(256)`: Optional label; defaults to `<VAR_NM>=<expression>`
+ - `VAR_LOGIC char(4000)`: The SAS expression that produces the value
+ - `ACTIVE num`: Set to 1 to apply the variable

docs/tables/mpe_filtersource.md

@@ -17,5 +17,5 @@ The `MPE_FILTERSOURCE` table stores the individual query lines of each filter cr
  - `SUBGROUP_ID num`: Identifier of the subgroup to which this line belongs
  - `VARIABLE_NM char(32)`: The variable being filtered
  - `OPERATOR_NM char(12)`: The filter operator (e.g. `=`, `>`, `IN`)
- - `RAW_VALUE char(4000)`: The filter value
+ - `RAW_VALUE char(4000)`: The filter value. A date, datetime or time is stored as a SAS literal (for example `'20APR2042'd`, `'01JAN1960:00:00:42'dt`, `'00:00:42't`), so the clause shown above the grid reads as a date rather than the internal day or second count.
  - `PROCESSED_DTTM num`: Datetime the filter line was created

docs/tables/mpe_validations.md

@@ -20,6 +20,6 @@ A detailed breakdown is available in the [validations](/dcc-validations/) sectio
  - 🔑 `BASE_LIB char(8)`: SAS Libref (8 chars)
  - 🔑 `BASE_DS char(32)`: The library member name
  - 🔑 `BASE_COL char(32)`: The column name
- - 🔑 `RULE_TYPE char(32)`: The name of the rule to apply.  Valid values include `CASE`, `NOTNULL`, `MINVAL`, `MAXVAL`, `READONLY`, `HIDDEN`, `ROUND`, `NUMBER_FORMAT`, `HARDREGEX`, `SOFTREGEX`, `HARDSELECT`, `SOFTSELECT`, `HARDSELECT_HOOK` and `SOFTSELECT_HOOK`.
+ - 🔑 `RULE_TYPE char(32)`: The name of the rule to apply.  Valid values include `CASE`, `NOTNULL`, `MINVAL`, `MAXVAL`, `READONLY`, `HIDDEN`, `ROUND`, `NUMBER_FORMAT`, `HARDFORMULA`, `SOFTFORMULA`, `HARDREGEX`, `SOFTREGEX`, `HARDSELECT`, `SOFTSELECT`, `HARDSELECT_HOOK` and `SOFTSELECT_HOOK`.
  - `RULE_VALUE char(128)`: The value of the rule.
  - `RULE_ACTIVE num`: Set to 1 for an active rule, or 0 to disable the rule.

docs/videos.md

@@ -17,7 +17,7 @@ A collection of videos made in relation to Data Controller.  Some of them were r
 ### Data Controller - a Customer Story
 An illustration of the utility of Data Controller for Data & Analytics teams in regulated industries.
 
-<iframe width="560" height="315" src="https://www.youtube.com/embed/M8hafkS4zY4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+<iframe title="Data Controller - a Customer Story" width="560" height="315" src="https://vid.4gl.io/videos/embed/kpSGoNE26FSif8X9A6pUfe?peertubeLink=0" frameborder="0" allowfullscreen="" sandbox="allow-same-origin allow-scripts allow-popups"></iframe>
 
 ### Use Case Video - Manual Overrides
 

docs/viewboxes.md

@@ -12,7 +12,7 @@ Often when editing (or examining) raw data, it is helpful to see it alongside re
 Each individual viewbox has the following features:
 
 * Choose the columns to display (and which order)
-* Resize individual boxes (or reset to original)
+* Resize individual boxes by dragging any of the four edges or corners (or reset to original)
 * Full filtering capability (complex clauses)
 * Minimise / Restore all, or individually
 * Reposition - manually, or snap to grid

mkdocs.yml

@@ -14,6 +14,9 @@ nav:
     - Table Viewer: dcu-tableviewer.md
     - ViewBoxes: viewboxes.md
     - Admin Services: admin-services.md
+- Developer Guide:
+    - Filter in a Pre-Edit Hook: developer-guide/filter-pre-edit-hook.md
+    - One Table, Two Rule Sets: developer-guide/two-rule-sets-one-table.md
 - Table Guide:
     - MPE_AUDIT: tables/mpe_audit.md
     - MPE_COLUMN_LEVEL_SECURITY: tables/mpe_column_level_security.md
@@ -29,6 +32,7 @@ nav:
     - MPE_DATASTATUS_TABS: tables/mpe_datastatus_tabs.md
     - MPE_DATADICTIONARY: tables/mpe_datadictionary.md
     - MPE_DATALOADS: tables/mpe_dataloads.md
+    - MPE_DERIVED: tables/mpe_derived.md
     - MPE_EMAILS: tables/mpe_emails.md
     - MPE_EXCEL_CONFIG: tables/mpe_excel_config.md
     - MPE_FILTERANYTABLE: tables/mpe_filteranytable.md
@@ -55,6 +59,7 @@ nav:
     - CAS Tables: cas-tables.md
     - Column Level Security: column-level-security.md
     - Dates / Datetimes: dcc-dates.md
+    - Derived Variables: derived-variables.md
     - Dynamic Cell Dropdown: dynamic-cell-dropdown.md
     - Emails: emails.md
     - Excel: excel.md
@@ -71,6 +76,7 @@ nav:
 - Macros: macros.md
 - Installation:
     - System Requirements: dci-requirements.md
+    - Downloads: downloads.md
     - SAS Viya: deploy-viya.md
     - SAS 9 EBI: dci-deploysas9.md
     - SAS 9 STP Hardening: dci-stpinstance.md
@@ -95,7 +101,9 @@ markdown_extensions:
 extra:
   manifest: manifest.webmanifest
 
-extra_css: ['font-awesome.css']
+extra_css:
+  - font-awesome.css
+  - dc-brand.css
 
 plugins:
   - search:
@@ -111,12 +119,22 @@ repo_url: 'https://git.datacontroller.io/dc/docs.datacontroller.io'
 
 theme:
   name: material
-  logo: 'img/favicon.ico'
+  logo: 'img/dc-logo.svg'
   palette:
-    primary: 'White'
-    accent: 'Amber'
+    - scheme: slate
+      primary: custom
+      accent: custom
+      toggle:
+        icon: material/weather-sunny
+        name: Switch to light mode
+    - scheme: default
+      primary: custom
+      accent: custom
+      toggle:
+        icon: material/weather-night
+        name: Switch to dark mode
   font:
-    text: 'Open Sans'
+    text: 'Montserrat'
     code: 'Ubuntu Mono'
   favicon: img/favicon.ico
   custom_dir: 'theme'