dc/CONTEXT.md

Context: Data Controller for SAS®

Data Controller for SAS® is a web application that lets users safely add, modify and delete data in SAS datasets and databases. Every change is staged and approved before being applied to the target table, and the full history of the change is retained. It runs on SAS Viya, SAS 9 EBI, and SASjs Server, and must run entirely on-prem / offline (no external assets or network calls at runtime).

This file is the shared glossary and orientation for the repo. When your output names a domain concept, use the term as defined here rather than a synonym. Related repos: docs.datacontroller.io (user-facing docs) and datacontroller.io (marketing site).

Repository layout

• client/ - the Angular frontend (TypeScript). Uses Handsontable for the editable grid and HyperFormula for Excel-formula support. Lint with npm run lint:check from client/.
• sas/ - the SAS backend. Services, macros, hooks and tests live under sas/sasjs/**; sas/sasjsbuild/ is generated output (never hand-edit). Lint .sas files with sasjs lint from sas/.
• .agents/docs/ - technical deep-dives (see below). .agents/skills/ - task skills (handsontable, hyperformula, dc-sas for Data Controller SAS development, plus npx-installed sasjs/ sas skills).

Roles

The five user roles, in increasing privilege:

• Viewer - explores/links to data without locking datasets.
• Editor - makes changes (add/modify/delete) and submits them for approval.
• Approver - accepts or rejects staged changes; on acceptance the change is applied to the target table.
• Auditor - reviews the history of changes to a table.
• Administrator - registers tables and configures security (at metadata group level).

Admins are listed in &mpeadmins; admin membership bypasses Row Level Security.

Core concepts

• Target table - the physical table (SAS dataset or database table) a user is changing. Its attributes (primary key, load type, library, SCD/temporal variables) are predefined by an administrator in MPE_TABLES.
• Submission - the set of changed rows staged for approval. Submissions are never applied automatically; they always require one or more approvals. Three kinds: Web submission (only changed rows from the edited extract), Excel submission (whole Excel file staged with the file kept for audit; becomes a web submission if the rows are edited in the grid first), and CSV submission (all rows sent straight to staging, suitable for larger uploads).
• Staging area - the secure location where submitted changes wait for approval before being loaded.
• Edit-Stage-Approve workflow - the central flow: up to 500 rows edited in the web grid, staged, and applied to the target after approval.
• Changeset / diff - new / modified / deleted rows computed for a submission; approvers review the diff before accepting.

Load types

Set per table in MPE_TABLES.LOADTYPE. Determines the loader and how history is kept (see .agents/docs/bitemporal-dataloader.md):

• UPDATE - no history; changed records deleted and re-appended (via %bitemporal_dataloader).
• REPLACE - no history; whole table wiped and reloaded. Implemented inline in %mpe_targetloader, deliberately not via %bitemporal_dataloader (see .agents/docs/replace-load-type.md).
• TXTEMPORAL - SCD2-style history on technical (transaction) time.
• BITEMPORAL - full two-dimensional history (business time + technical time).
• FORMAT_CAT - format-catalog load (via %mp_loadformat).

Bitemporal = two independent time dimensions: business time (VAR_BUSFROM/VAR_BUSTO, when a fact is true in the real world) and technical/transaction time (VAR_TXFROM/VAR_TXTO, when the system knew it). SCD2 = slowly changing dimension, type 2 (row-versioned history via open/close datetimes). BUSKEY = the business/primary key (space-separated columns, excluding temporal columns).

MPE control tables

Configuration and state live in MPE_* tables (the "control tables") in the DC control library (&mpelib / &dclib). Key ones:

• MPE_TABLES - registered target tables and their load config (LOADTYPE, BUSKEY, temporal vars, etc.).
• MPE_VALIDATIONS - point-of-entry data-quality rules (see below).
• MPE_ROW_LEVEL_SECURITY - Row Level Security rules.
• MPE_COLUMN_LEVEL_SECURITY - Column Level Security rules.
• MPE_SUBMIT / MPE_REVIEW / MPE_REQUESTS - the approval workflow tables.
• MPE_AUDIT - change history.
• MPE_XLMAP_INFO / MPE_XLMAP_RULES / MPE_XLMAP_DATA - Excel-map (XLMAP) definitions for structured Excel uploads.
• MPE_CONFIG, MPE_GROUPS, MPE_SECURITY, MPE_EMAIL, MPE_LOCKANYTABLE, and the MPE_DATACATALOG_* / MPE_DATASTATUS_* catalog tables.

Selectbox seed values for these tables are defined in sas/sasjs/macros/mpe_makedata.sas. User-facing documentation for each table is in the docs.datacontroller.io repo under docs/tables/.

Validations

MPE_VALIDATIONS applies point-of-entry rules per BASE_LIB/BASE_DS/BASE_COL when RULE_ACTIVE=1. RULE_TYPE values include CASE, NOTNULL, MINVAL, MAXVAL, READONLY, HIDDEN, ROUND, NUMBER_FORMAT, HARDREGEX, SOFTREGEX, HARDSELECT, SOFTSELECT, HARDSELECT_HOOK, SOFTSELECT_HOOK.

• HARDREGEX - submission-blocking regex; failing cells are painted red and cannot be submitted.
• SOFTREGEX - display-only warning regex; failing cells are painted yellow but submission is allowed.

Regex rule values are authored in SAS PRX form /pattern/flags and limited to 128 chars. See .agents/docs/regex-validations.md.

Security

• Row Level Security (RLS) - server-side WHERE-clause generation via %mpe_filtermaster / %mp_filtergenerate; no data leaves SAS without passing the filter. Works on any engine because it's expressed as a standard SAS WHERE. On write, the inverse filter is applied to reject out-of-scope rows. See .agents/docs/row-level-security.md.
• Column Level Security (CLS) - restricts visibility/editability at column level.

Key macros / services

• %mpe_targetloader (sas/sasjs/macros/mpe_targetloader.sas) - the single load dispatch point; routes by LOADTYPE. Two-phase: LOADTARGET=NO builds diff tables for review, LOADTARGET=YES performs the destructive load.
• %bitemporal_dataloader - the temporal loader for UPDATE / TXTEMPORAL / BITEMPORAL.
• %mpe_filtermaster - builds the RLS filter for a request.

Testing

Backend tests run with the sasjs CLI from sas/. npm run 4gl compiles+deploys+seeds then sasjs test -t 4gl runs tests. Assertions use %mp_assert(). See .agents/docs/testing.md.

Conventions

• Do not hard-wrap Markdown (one logical line per paragraph/list-item/heading).
• Use regular dashes, not em-dashes.
• No external assets - everything must be bundled and served locally.
• Comments and test names must be self-contained (no "original bug", "this PR", etc.); the only exception is a literal issue-tracker link.
• Do not auto-commit or push.
• Never edit CHANGELOG.md or bump package.json version by hand - the release pipeline generates both from Conventional Commit messages. See .agents/docs/releases-and-changelog.md.

See AGENTS.md for the full set of enforced rules (git, linting, no-wrap, no-external-assets, .agents/ layout).