feat: adding all tables to the docs

AGENTS.md

@@ -0,0 +1,51 @@
+# Agent Instructions
+
+This repository is the **user-facing documentation** for Data Controller for SAS®, published at [docs.datacontroller.io](https://docs.datacontroller.io). It is a [MkDocs](https://www.mkdocs.org/) site using the Material theme.
+
+## Related repositories
+
+Data Controller spans three sibling repositories (usually checked out side by side under the same parent directory):
+
+- **`dc`** - the product source (Angular client + SAS backend). The behaviour documented here is implemented there. Deep technical notes live in `dc/.agent/docs/`.
+- **`docs.datacontroller.io`** (this repo) - the user-facing product documentation.
+- **`datacontroller.io`** - the marketing site, blog and feed (Gatsby).
+
+When documenting a feature, the source of truth for behaviour is `dc`. When a doc page describes internals, prefer linking to the user-facing concept rather than duplicating implementation detail.
+
+## Structure
+
+- Pages are Markdown files in `docs/`.
+- The navigation tree, site config, theme, plugins and redirects are all defined in `mkdocs.yml`. **A new page is not published until it is added to the `nav:` tree in `mkdocs.yml`.**
+- `docs/tables/` documents the `MPE_*` control tables (see naming conventions below).
+- `docs/img/` holds images; `docs/video/` holds video assets; `docs/marketing/` holds flyers/PDFs.
+- `theme/` is the custom Material theme override; `slides/` and `slides.md` are the presentation deck.
+
+## Page conventions
+
+- Each page starts with YAML front matter: `layout: article`, `title`, `description`, and usually `og_image`. Match the style of existing pages.
+- `description` is used for SEO and social cards - write a single, complete sentence.
+- Reference images with root-relative paths (e.g. `/img/foo.png`) or relative paths consistent with neighbouring pages.
+- This site uses these `markdown_extensions`: `admonition`, `pymdownx.superfences`, `codehilite`, `meta`, and `toc` (with permalinks). Use fenced code blocks with language hints (`sas`, `js`, `bash` are highlighted); use admonitions (`!!! note`) for callouts.
+- Internal links use the page slug with a trailing slash (e.g. `/dcc-validations/`), matching existing cross-references.
+
+## MPE table docs (`docs/tables/`)
+
+Control tables are documented one file per table, named `mpe_<name>.md`, and registered under the "Table Guide" section of `nav:` in `mkdocs.yml`. Follow the existing pattern:
+
+- Front matter with a `description` explaining what the table configures.
+- A short intro paragraph, then a link to the relevant configuration guide.
+- A `## Columns` list. Prefix primary-key / business-key columns with the 🔑 emoji, and give each column as `` `NAME type` ``: description. SCD2 tables carry `TX_FROM`/`TX_TO` as the first two columns.
+
+## Writing style
+
+Use regular dashes (`-`) in content, not em-dashes. Do not hard-wrap Markdown: each paragraph, list item and heading is a single logical line, regardless of length - let the renderer soft-wrap. This keeps diffs clean.
+
+## Building
+
+- `pip install mkdocs mkdocs-material mkdocs-redirects` (see `build.sh` / `mkdocs.yml` for the exact plugin list).
+- `mkdocs serve` for a live-reloading local preview; `mkdocs build` (or `./build.sh`) to produce the static site.
+- After adding or renaming a page, confirm it appears in the `nav:` tree and that `mkdocs build` reports no warnings about missing/orphaned files.
+
+## Git
+
+Do NOT auto-commit or push. Leave changes in the working tree for the user to review and commit.

CONTEXT.md

@@ -0,0 +1,33 @@
+# Context: docs.datacontroller.io (product documentation)
+
+The user-facing documentation for Data Controller for SAS®, published at [docs.datacontroller.io](https://docs.datacontroller.io) as a **MkDocs** (Material theme) static site.
+
+## What the product is
+
+Data Controller for SAS® is a web application that lets users safely add, modify and delete data in SAS datasets and databases. Every change is **staged** and **approved** before being applied to the **target table**, and the change history is retained. It runs on SAS Viya, SAS 9 EBI and SASjs Server. The product source lives in the sibling `dc` repo (see its `CONTEXT.md` for the full domain glossary); the marketing site is `datacontroller.io`.
+
+## Domain vocabulary (used throughout the docs)
+
+- **Roles**: Viewer, Editor, Approver, Auditor, Administrator.
+- **Target table**: the physical SAS/database table a user changes; configured by an admin in `MPE_TABLES`.
+- **Submission / staging / approval**: changes are staged and require approval before being applied.
+- **Load types** (`MPE_TABLES.LOADTYPE`): `UPDATE`, `REPLACE`, `TXTEMPORAL`, `BITEMPORAL`, `FORMAT_CAT` - determine history behaviour (SCD2 / bitemporal / none).
+- **MPE control tables** (`MPE_*`): configuration and state tables, each documented under `docs/tables/mpe_<name>.md`.
+- **Validations** (`MPE_VALIDATIONS`): point-of-entry data-quality rules.
+- **Row / Column Level Security** (RLS / CLS): server-side access control.
+
+Use these terms consistently; match the casing used in the existing docs.
+
+## Structure
+
+- Pages are Markdown in `docs/`. **A page is not published until it is added to the `nav:` tree in `mkdocs.yml`.**
+- `docs/tables/` documents the `MPE_*` control tables (one file per table, following the shared column-list pattern with 🔑 for key columns).
+- `mkdocs.yml` defines nav, theme, plugins (search, redirects) and markdown extensions (`admonition`, `pymdownx.superfences`, `codehilite`, `meta`, `toc`).
+
+## Conventions
+
+- Front matter per page: `layout: article`, `title`, `description`, usually `og_image`.
+- Use regular dashes, not em-dashes. Do not hard-wrap Markdown.
+- Preview with `mkdocs serve`; build with `mkdocs build` (or `./build.sh`) and confirm no warnings.
+
+See `AGENTS.md` for full page/table conventions and build instructions.

docs/tables/mpe_datadictionary.md

@@ -0,0 +1,21 @@
+---
+layout: article
+title: MPE_DATADICTIONARY
+description: The MPE_DATADICTIONARY table documents libraries, tables, columns and directories with descriptions, ownership and sensitivity information.
+---
+
+# MPE_DATADICTIONARY
+
+The `MPE_DATADICTIONARY` table stores user-maintained documentation for libraries, tables, columns and directories. This content is surfaced in the Data Dictionary view of Data Controller.
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `DD_TYPE char(16)`: The type of item being documented (e.g. LIBRARY, TABLE, COLUMN, DIRECTORY)
+ - 🔑 `DD_SOURCE char(1024)`: The item being documented (e.g. `libref`, `libref.table`, `libref.table.column` or a directory path)
+ - `DD_SHORTDESC char(256)`: Short description
+ - `DD_LONGDESC char(32767)`: Long description (Markdown supported)
+ - `DD_OWNER char(128)`: Owner of the item
+ - `DD_RESPONSIBLE char(128)`: Responsible party for the item
+ - `DD_SENSITIVITY char(64)`: Sensitivity classification (e.g. Low)
+ - 🔑 `TX_TO num`: SCD2 close datetime

docs/tables/mpe_dataloads.md

@@ -0,0 +1,23 @@
+---
+layout: article
+title: MPE_DATALOADS
+description: The MPE_DATALOADS table is an audit trail of every load performed through Data Controller for SAS®, including record counts and duration.
+---
+
+# MPE_DATALOADS
+
+The `MPE_DATALOADS` table records an audit entry for every load performed (via the frontend, or via the [bitemporal dataloader macros](/macros/)).
+
+## Columns
+
+ - 🔑 `PROCESSED_DTTM num`: Datetime the load completed
+ - 🔑 `LIBREF char(8)`: SAS Libref of the target table
+ - 🔑 `DSN char(32)`: Target table name
+ - 🔑 `ETLSOURCE char(100)`: Source of the load (e.g. the submitting user / service)
+ - `LOADTYPE char(20)`: The load type applied (UPDATE, REPLACE, TXTEMPORAL, BITEMPORAL, FORMAT_CAT)
+ - `CHANGED_RECORDS num`: Number of records changed
+ - `NEW_RECORDS num`: Number of records added
+ - `DELETED_RECORDS num`: Number of records deleted
+ - `DURATION num`: Duration of the load (seconds)
+ - `USER_NM char(50)`: The user who performed the load
+ - `MAC_VER char(5)`: The version of the Data Controller macros used

docs/tables/mpe_excel_config.md

@@ -0,0 +1,34 @@
+---
+layout: article
+title: MPE_EXCEL_CONFIG
+description: The MPE_EXCEL_CONFIG table configures column-level rules applied during Excel uploads in Data Controller for SAS®.
+---
+
+# MPE_EXCEL_CONFIG
+
+The `MPE_EXCEL_CONFIG` table configures column-level rules that are applied when uploading data via Excel. See the [Excel](/excel/) guide for more details.
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `XL_LIBREF char(8)`: SAS Libref of the target table
+ - 🔑 `XL_TABLE char(32)`: Target table name
+ - 🔑 `XL_COLUMN char(32)`: Column to which the rule applies
+ - `XL_RULE char(32)`: The rule to apply.  Currently the only supported rule is `FORMULA` - this extracts the underlying cell _formula_ (eg `=VLOOKUP(...)`) rather than the raw cell value during an Excel upload.  The target column must be character, and wide enough to hold the longest formula.
+ - `XL_ACTIVE num`: Flag indicating whether the rule is active (1 = active)
+ - `TX_TO num`: SCD2 close datetime
+
+## Example
+
+The following entry (from the Data Controller sample data) causes the `DD_LONGDESC` column of `MPE_DATADICTIONARY` to be loaded as a formula rather than a raw value when uploading via Excel:
+
+```sas
+insert into &lib..MPE_EXCEL_CONFIG set
+    tx_from=0
+    ,xl_libref="&lib"
+    ,xl_table="MPE_DATADICTIONARY"
+    ,xl_column="DD_LONGDESC"
+    ,xl_rule="FORMULA"
+    ,xl_active=1
+    ,tx_to='31DEC5999:23:59:59'dt;
+```

docs/tables/mpe_filteranytable.md

@@ -0,0 +1,18 @@
+---
+layout: article
+title: MPE_FILTERANYTABLE
+description: The MPE_FILTERANYTABLE table stores a record for each unique filter clause created in Data Controller for SAS®.
+---
+
+# MPE_FILTERANYTABLE
+
+The `MPE_FILTERANYTABLE` table stores a record for each unique filter created via the FILTER menu.  When a user submits a filter, the entire clause is hashed - if that hash already exists for the table, the existing `FILTER_RK` is reused, otherwise a new record is added.  This means identical filters are only ever stored once, and the `FILTER_RK` can be safely embedded in the shareable URLs described in the [filter](/filter/) guide.
+
+The individual lines of the filter clause itself are stored in [MPE_FILTERSOURCE](/tables/mpe_filtersource/).
+
+## Columns
+
+ - 🔑 `FILTER_RK num`: Unique retained key for the filter, used to recall the filter (eg in shareable URLs)
+ - `FILTER_HASH char(32)`: Hash of the entire filter clause, used to detect duplicate filters and to join to [MPE_FILTERSOURCE](/tables/mpe_filtersource/)
+ - `FILTER_TABLE char(41)`: The table being filtered (in `libref.dataset` format)
+ - `PROCESSED_DTTM num`: Datetime the filter was first created

docs/tables/mpe_filtersource.md

@@ -0,0 +1,21 @@
+---
+layout: article
+title: MPE_FILTERSOURCE
+description: The MPE_FILTERSOURCE table stores the individual lines of each filter clause created in Data Controller for SAS®.
+---
+
+# MPE_FILTERSOURCE
+
+The `MPE_FILTERSOURCE` table stores the individual query lines of each filter created via the FILTER menu, keyed by the hash stored in [MPE_FILTERANYTABLE](/tables/mpe_filteranytable/). See the [filter](/filter/) guide for more details.
+
+## Columns
+
+ - 🔑 `FILTER_HASH char(32)`: Hash of the filter clause, joining to [MPE_FILTERANYTABLE](/tables/mpe_filteranytable/)
+ - 🔑 `FILTER_LINE num`: Line number within the filter clause
+ - `GROUP_LOGIC char(3)`: AND / OR logic applied between groups
+ - `SUBGROUP_LOGIC char(3)`: AND / OR logic applied within the subgroup
+ - `SUBGROUP_ID num`: Identifier of the subgroup to which this line belongs
+ - `VARIABLE_NM char(32)`: The variable being filtered
+ - `OPERATOR_NM char(12)`: The filter operator (e.g. `=`, `>`, `IN`)
+ - `RAW_VALUE char(4000)`: The filter value
+ - `PROCESSED_DTTM num`: Datetime the filter line was created

docs/tables/mpe_groups.md

@@ -0,0 +1,19 @@
+---
+layout: article
+title: MPE_GROUPS
+description: The MPE_GROUPS table defines optional groups and group membership used to secure access to tables in Data Controller for SAS®.
+---
+
+# MPE_GROUPS
+
+The `MPE_GROUPS` table defines optional groups, and the members of those groups, used to secure access in Data Controller.
+
+A more detailed breakdown is available in the [configuration](/dcc-groups/) section.
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `GROUP_NAME char(100)`: The name of the group
+ - `GROUP_DESC char(256)`: A description of the group
+ - 🔑 `USER_NAME char(50)`: The user (SAS identity name) who is a member of the group
+ - `TX_TO num`: SCD2 close datetime

docs/tables/mpe_lineage_cols.md

@@ -0,0 +1,27 @@
+---
+layout: article
+title: MPE_LINEAGE_COLS
+description: The MPE_LINEAGE_COLS table stores column-level lineage (forward and reverse) extracted by Data Controller for SAS®.
+---
+
+# MPE_LINEAGE_COLS
+
+The `MPE_LINEAGE_COLS` table stores column-level lineage - the column mappings derived from jobs registered in SAS DI Studio. See the [lineage](/dcu-lineage/) guide for more details.
+
+## Columns
+
+ - 🔑 `COL_ID char(32)`: Unique identifier of the lineage record
+ - 🔑 `DIRECTION char(1)`: Lineage direction (e.g. F for forward, R for reverse)
+ - `JOBNAME char(256)`: Name of the job in which the mapping was found
+ - `SOURCETABLENAME char(256)`: Name of the source table
+ - `SOURCECOLNAME char(256)`: Name of the source column
+ - 🔑 `SOURCECOLURI char(256)`: URI of the source column
+ - 🔑 `MAP_TYPE char(256)`: The type of mapping
+ - 🔑 `MAP_TRANSFORM char(256)`: The transformation applied in the mapping
+ - `TARGETTABLENAME char(256)`: Name of the target table
+ - `TARGETCOLNAME char(256)`: Name of the target column
+ - 🔑 `TARGETCOLURI char(256)`: URI of the target column
+ - `DERIVED_RULE char(500)`: The derivation rule applied
+ - `LEVEL num`: The depth of the mapping within the lineage tree
+ - `MODIFIED_DTTM num`: Datetime the record was last modified
+ - `MODIFIED_BY char(64)`: The user who last modified the record

docs/tables/mpe_lineage_tabs.md

@@ -0,0 +1,24 @@
+---
+layout: article
+title: MPE_LINEAGE_TABS
+description: The MPE_LINEAGE_TABS table stores table-level lineage (forward and reverse) extracted by Data Controller for SAS®.
+---
+
+# MPE_LINEAGE_TABS
+
+The `MPE_LINEAGE_TABS` table stores table-level lineage - the table-to-table relationships derived from jobs registered in SAS DI Studio. See the [lineage](/dcu-lineage/) guide for more details.
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `JOBID char(17)`: Identifier of the job in which the relationship was found
+ - `JOBNAME char(128)`: Name of the job
+ - 🔑 `SRCTABLEID char(17)`: Identifier of the source table
+ - `SRCTABLETYPE char(16)`: Type of the source table
+ - `SRCTABLENAME char(64)`: Name of the source table
+ - `SRCLIBREF char(8)`: Libref of the source table
+ - 🔑 `TGTTABLEID char(17)`: Identifier of the target table
+ - `TGTTABLETYPE char(16)`: Type of the target table
+ - `TGTTABLENAME char(64)`: Name of the target table
+ - `TGTLIBREF char(8)`: Libref of the target table

docs/tables/mpe_loads.md

@@ -0,0 +1,19 @@
+---
+layout: article
+title: MPE_LOADS
+description: The MPE_LOADS table records the status of CSV file loads performed by the Data Controller for SAS® target loader.
+---
+
+# MPE_LOADS
+
+The `MPE_LOADS` table tracks the status of CSV file loads processed by the target loader ([mpe_targetloader](/macros/) macro), including failures and their reasons.
+
+## Columns
+
+ - 🔑 `CSV_DIR char(255)`: The staged folder reference (mperef) containing the CSV files being loaded
+ - `USER_NM char(50)`: The user who submitted the load
+ - `STATUS char(15)`: The status of the load (e.g. IN PROGRESS, SUCCESS, FAILED)
+ - `DURATION num`: Duration of the load (seconds)
+ - `PROCESSED_DTTM num`: Datetime the load was processed
+ - `REASON_TXT char(2048)`: The reason for failure (where applicable)
+ - `APPROVALS char(64)`: Approval information for the load

docs/tables/mpe_maxkeyvalues.md

@@ -0,0 +1,16 @@
+---
+layout: article
+title: MPE_MAXKEYVALUES
+description: The MPE_MAXKEYVALUES table stores the current maximum surrogate / retained key value for each keyed table in Data Controller for SAS®.
+---
+
+# MPE_MAXKEYVALUES
+
+The `MPE_MAXKEYVALUES` table stores the current maximum surrogate / retained key value for each table configured with a retained key (see [RK_UNDERLYING](/dcc-tables/#rk_underlying)). It is used to generate new key values during loads.
+
+## Columns
+
+ - 🔑 `KEYTABLE char(41)`: Base table in `libref.dataset` format
+ - `KEYCOLUMN char(32)`: The surrogate / retained key field containing the key values
+ - `MAX_KEY num`: Integer value representing the current max RK or SK value in the KEYTABLE
+ - `PROCESSED_DTTM num`: Datetime this value was last updated

docs/tables/mpe_selectbox.md

@@ -0,0 +1,23 @@
+---
+layout: article
+title: MPE_SELECTBOX
+description: The MPE_SELECTBOX table configures the dropdown values available for columns of control tables in Data Controller for SAS®.
+---
+
+# MPE_SELECTBOX
+
+The `MPE_SELECTBOX` table configures the values that appear in dropdowns when editing control tables (eg `LOADTYPE` in [MPE_TABLES](/tables/mpe_tables/) or `ACCESS_LEVEL` in [MPE_SECURITY](/tables/mpe_security/)).
+
+A more detailed breakdown is available in the [configuration](/dcc-selectbox/) section.
+
+## Columns
+
+ - `VER_FROM_DTTM num`: SCD2 open datetime
+ - 🔑 `SELECTBOX_RK num`: Surrogate key for the selectbox value
+ - `SELECT_LIB char(17)`: Libref of the table to which the dropdown applies
+ - `SELECT_DS char(32)`: Name of the table to which the dropdown applies
+ - `BASE_COLUMN char(36)`: The column against which the dropdown is applied
+ - `SELECTBOX_VALUE char(500)`: The dropdown value
+ - `SELECTBOX_ORDER num`: Optional ordering of the dropdown values (1 comes before 2)
+ - `SELECTBOX_TYPE char(32)`: Column type (blank for default, else `sas` or `js` to indicate relevant system functions)
+ - `VER_TO_DTTM num`: SCD2 close datetime

docs/tables/mpe_signoffs.md

@@ -0,0 +1,21 @@
+---
+layout: article
+title: MPE_SIGNOFFS
+description: The MPE_SIGNOFFS table records signoffs made against loaded data in Data Controller for SAS®.
+---
+
+# MPE_SIGNOFFS
+
+The `MPE_SIGNOFFS` table is designed to record signoffs - the final approval step associated with SIGNOFF access (see [MPE_SECURITY](/tables/mpe_security/) and [SIGNOFF_COLS](/dcc-tables/#signoff_cols)).
+
+!!! note
+    This table is created as part of the Data Controller data model but is not currently populated by any Data Controller service. It is reserved for custom signoff implementations.
+
+## Columns
+
+ - 🔑 `TECH_FROM_DTTM num`: SCD2 open datetime
+ - 🔑 `SIGNOFF_TABLE char(50)`: The table being signed off
+ - 🔑 `SIGNOFF_SECTION_RK num`: The retained key of the section being signed off
+ - `SIGNOFF_VERSION_RK num`: The retained key of the version being signed off
+ - `SIGNOFF_NAME char(100)`: The name of the user performing the signoff
+ - `TECH_TO_DTTM num`: SCD2 close datetime

docs/tables/mpe_users.md

@@ -0,0 +1,15 @@
+---
+layout: article
+title: MPE_USERS
+description: The MPE_USERS table captures the users of Data Controller for SAS® and when they were last seen.
+---
+
+# MPE_USERS
+
+The `MPE_USERS` table captures the actual users of the app - each user is registered on first login, and their last seen date is updated on subsequent activity.
+
+## Columns
+
+ - 🔑 `USER_ID char(50)`: The user id
+ - `LAST_SEEN_DT num`: Date the user was last active
+ - `REGISTERED_DT num`: Date the user first registered

mkdocs.yml

@@ -27,13 +27,26 @@ nav:
     - MPE_DATASTATUS_LIBS: tables/mpe_datastatus_libs.md
     - MPE_DATASTATUS_OBJ: tables/mpe_datastatus_objs.md
     - MPE_DATASTATUS_TABS: tables/mpe_datastatus_tabs.md
-    - MPE_EMAIL: tables/mpe_email.md
+    - MPE_DATADICTIONARY: tables/mpe_datadictionary.md
+    - MPE_DATALOADS: tables/mpe_dataloads.md
+    - MPE_EMAILS: tables/mpe_emails.md
+    - MPE_EXCEL_CONFIG: tables/mpe_excel_config.md
+    - MPE_FILTERANYTABLE: tables/mpe_filteranytable.md
+    - MPE_FILTERSOURCE: tables/mpe_filtersource.md
+    - MPE_GROUPS: tables/mpe_groups.md
+    - MPE_LINEAGE_COLS: tables/mpe_lineage_cols.md
+    - MPE_LINEAGE_TABS: tables/mpe_lineage_tabs.md
+    - MPE_LOADS: tables/mpe_loads.md
     - MPE_LOCKANYTABLE: tables/mpe_lockanytable.md
+    - MPE_MAXKEYVALUES: tables/mpe_maxkeyvalues.md
     - MPE_REQUESTS: tables/mpe_requests.md
     - MPE_REVIEW: tables/mpe_review.md
     - MPE_SUBMIT: tables/mpe_submit.md
     - MPE_SECURITY: tables/mpe_security.md
+    - MPE_SELECTBOX: tables/mpe_selectbox.md
+    - MPE_SIGNOFFS: tables/mpe_signoffs.md
     - MPE_TABLES: tables/mpe_tables.md
+    - MPE_USERS: tables/mpe_users.md
     - MPE_VALIDATIONS: tables/mpe_validations.md
     - MPE_XLMAP_DATA: tables/mpe_xlmap_data.md
     - MPE_XLMAP_INFO: tables/mpe_xlmap_info.md