dc/CONTEXT.md

# Context: Data Controller for SAS®

Data Controller for SAS® is a web application that lets users safely add, modify and delete data in SAS datasets and databases. Every change is **staged** and **approved** before being applied to the **target table**, and the full history of the change is retained. It runs on SAS Viya, SAS 9 EBI, and [SASjs Server](https://server.sasjs.io), and must run entirely **on-prem / offline** (no external assets or network calls at runtime).

This file is the shared glossary and orientation for the repo. When your output names a domain concept, use the term as defined here rather than a synonym. Related repos: `docs.datacontroller.io` (user-facing docs) and `datacontroller.io` (marketing site).

## Repository layout

- `client/` - the Angular frontend (TypeScript). Uses [Handsontable](.agents/skills/handsontable/SKILL.md) for the editable grid and [HyperFormula](.agents/skills/hyperformula/SKILL.md) for Excel-formula support. Lint with `npm run lint:check` from `client/`.
- `sas/` - the SAS backend. Services, macros, hooks and tests live under `sas/sasjs/**`; `sas/sasjsbuild/` is generated output (never hand-edit). Lint `.sas` files with `sasjs lint` from `sas/`.
- `.agents/docs/` - technical deep-dives (see below). `.agents/skills/` - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, dc-estate-testing for deploying to and testing on the shared estate, plus npx-installed sasjs/ sas skills).

## Roles

The five user roles, in increasing privilege:

- **Viewer** - explores/links to data without locking datasets.
- **Editor** - makes changes (add/modify/delete) and submits them for approval.
- **Approver** - accepts or rejects staged changes; on acceptance the change is applied to the target table.
- **Auditor** - reviews the history of changes to a table.
- **Administrator** - registers tables and configures security (at metadata group level).

Admins are listed in `&mpeadmins`; admin membership bypasses Row Level Security.

## Core concepts

- **Target table** - the physical table (SAS dataset or database table) a user is changing. Its attributes (primary key, load type, library, SCD/temporal variables) are predefined by an administrator in `MPE_TABLES`.
- **Submission** - the set of changed rows staged for approval. Submissions are *never* applied automatically; they always require one or more approvals. Three kinds: **Web submission** (only changed rows from the edited extract), **Excel submission** (whole Excel file staged with the file kept for audit; becomes a web submission if the rows are edited in the grid first), and **CSV submission** (all rows sent straight to staging, suitable for larger uploads).
- **Staging area** - the secure location where submitted changes wait for approval before being loaded.
- **Edit-Stage-Approve workflow** - the central flow: up to 500 rows edited in the web grid, staged, and applied to the target after approval.
- **Changeset / diff** - new / modified / deleted rows computed for a submission; approvers review the diff before accepting.

## Load types

Set per table in `MPE_TABLES.LOADTYPE`. Determines the loader and how history is kept (see [.agents/docs/bitemporal-dataloader.md](.agents/docs/bitemporal-dataloader.md)):

- **UPDATE** - no history; changed records deleted and re-appended (via `%bitemporal_dataloader`).
- **REPLACE** - no history; whole table wiped and reloaded. Implemented inline in `%mpe_targetloader`, deliberately *not* via `%bitemporal_dataloader` (see [.agents/docs/replace-load-type.md](.agents/docs/replace-load-type.md)).
- **TXTEMPORAL** - SCD2-style history on technical (transaction) time.
- **BITEMPORAL** - full two-dimensional history (business time + technical time).
- **FORMAT_CAT** - format-catalog load (via `%mp_loadformat`).

**Bitemporal** = two independent time dimensions: **business time** (`VAR_BUSFROM`/`VAR_BUSTO`, when a fact is true in the real world) and **technical/transaction time** (`VAR_TXFROM`/`VAR_TXTO`, when the system knew it). **SCD2** = slowly changing dimension, type 2 (row-versioned history via open/close datetimes). **BUSKEY** = the business/primary key (space-separated columns, excluding temporal columns).

## MPE control tables

Configuration and state live in `MPE_*` tables (the "control tables") in the DC control library (`&mpelib` / `&dclib`). Key ones:

- `MPE_TABLES` - registered target tables and their load config (`LOADTYPE`, `BUSKEY`, temporal vars, etc.).
- `MPE_VALIDATIONS` - point-of-entry data-quality rules (see below).
- `MPE_ROW_LEVEL_SECURITY` - Row Level Security rules.
- `MPE_COLUMN_LEVEL_SECURITY` - Column Level Security rules.
- `MPE_SUBMIT` / `MPE_REVIEW` / `MPE_REQUESTS` - the approval workflow tables.
- `MPE_AUDIT` - change history.
- `MPE_XLMAP_INFO` / `MPE_XLMAP_RULES` / `MPE_XLMAP_DATA` - Excel-map (XLMAP) definitions for structured Excel uploads.
- `MPE_CONFIG`, `MPE_GROUPS`, `MPE_SECURITY`, `MPE_EMAIL`, `MPE_LOCKANYTABLE`, and the `MPE_DATACATALOG_*` / `MPE_DATASTATUS_*` catalog tables.

Selectbox seed values for these tables are defined in `sas/sasjs/macros/mpe_makedata.sas`. User-facing documentation for each table is in the `docs.datacontroller.io` repo under `docs/tables/`.

## Validations

`MPE_VALIDATIONS` applies point-of-entry rules per `BASE_LIB`/`BASE_DS`/`BASE_COL` when `RULE_ACTIVE=1`. `RULE_TYPE` values include `CASE`, `NOTNULL`, `MINVAL`, `MAXVAL`, `READONLY`, `HIDDEN`, `ROUND`, `NUMBER_FORMAT`, `HARDREGEX`, `SOFTREGEX`, `HARDSELECT`, `SOFTSELECT`, `HARDSELECT_HOOK`, `SOFTSELECT_HOOK`.

- **HARDREGEX** - submission-blocking regex; failing cells are painted red and cannot be submitted.
- **SOFTREGEX** - display-only warning regex; failing cells are painted yellow but submission is allowed.

Regex rule values are authored in SAS PRX form `/pattern/flags` and limited to 128 chars. See [.agents/docs/regex-validations.md](.agents/docs/regex-validations.md).

## Security

- **Row Level Security (RLS)** - server-side `WHERE`-clause generation via `%mpe_filtermaster` / `%mp_filtergenerate`; no data leaves SAS without passing the filter. Works on any engine because it's expressed as a standard SAS `WHERE`. On write, the *inverse* filter is applied to reject out-of-scope rows. See [.agents/docs/row-level-security.md](.agents/docs/row-level-security.md).
- **Column Level Security (CLS)** - restricts visibility/editability at column level.

## Key macros / services

- `%mpe_targetloader` (`sas/sasjs/macros/mpe_targetloader.sas`) - the single load dispatch point; routes by `LOADTYPE`. Two-phase: `LOADTARGET=NO` builds diff tables for review, `LOADTARGET=YES` performs the destructive load.
- `%bitemporal_dataloader` - the temporal loader for UPDATE / TXTEMPORAL / BITEMPORAL.
- `%mpe_filtermaster` - builds the RLS filter for a request.

## Testing

Backend tests run with the sasjs CLI from `sas/`. `npm run 4gl` compiles+deploys+seeds then `sasjs test -t 4gl` runs tests. Assertions use `%mp_assert()`. See [.agents/docs/testing.md](.agents/docs/testing.md).

## Conventions

- Do not hard-wrap Markdown (one logical line per paragraph/list-item/heading).
- Use regular dashes, not em-dashes.
- No external assets - everything must be bundled and served locally.
- Comments and test names must be self-contained (no "original bug", "this PR", etc.); the only exception is a literal issue-tracker link.
- Do not auto-commit or push.
- Never edit `CHANGELOG.md` or bump `package.json` version by hand - the release pipeline generates both from Conventional Commit messages. See [.agents/docs/releases-and-changelog.md](.agents/docs/releases-and-changelog.md).

See `AGENTS.md` for the full set of enforced rules (git, linting, no-wrap, no-external-assets, `.agents/` layout).