dc/.agents/docs/bitemporal-dataloader.md

Bitemporal Dataloader — Technical Deep Dive

This document explains how the Data Controller backend loads staged data into target tables: the load-type dispatch, the internals of the %bitemporal_dataloader macro, and a detailed description of the REPLACE load type. For user-facing load type documentation, see docs.datacontroller.io.

Overview

Every table registered for loading has a current record in &mpelib..MPE_TABLES with a LOADTYPE (selectbox values are seeded in mpe_makedata.sas):

LOADTYPE	Loader used	History kept
UPDATE	%bitemporal_dataloader (no temporal vars)	None — changed records are deleted and re-appended
REPLACE	Inline code in %mpe_targetloader (does not use %bitemporal_dataloader)	None — entire table wiped and reloaded
TXTEMPORAL	%bitemporal_dataloader (technical time only)	SCD2-style, technical (transaction) time
BITEMPORAL	%bitemporal_dataloader (business + technical time)	Full two-dimensional history
FORMAT_CAT	%mp_loadformat	Format catalog load (table suffix -FC)

The relevant MPE_TABLES columns read by the loader are: LOADTYPE, BUSKEY (primary key, space-separated, excluding temporal columns), VAR_TXFROM / VAR_TXTO (technical validity), VAR_BUSFROM / VAR_BUSTO (business validity), VAR_PROCESSED (processed timestamp column), RK_UNDERLYING (retained-key generation), CLOSE_VARS, and AUDIT_LIBDS (defaults to &dclib..MPE_AUDIT).

Request Flow

flowchart TD
    A[User submits changeset\neditors/stagedata.sas] --> B[Staging package written to\n&mpelocapprovals/&LOAD_REF\nCSV + jsdata]
    B --> C[Approval workflow\nMPE_SUBMIT / MPE_REVIEW]
    C --> D{auditors/postdata.sas}
    D -->|action=SHOW_DIFFS| E["%mpe_targetloader(LOADTARGET=NO)\nbuilds work.outds_add / outds_mod / outds_del\nfor the diff screen only"]
    D -->|action=APPROVE_TABLE| F["%mpe_targetloader(LOADTARGET=YES)\nactual load"]
    E --> G[Diff CSV + TEMPDIFFS stored\nin approval package]
    F --> H{LOADTYPE from\nMPE_TABLES}
    H -->|UPDATE / TXTEMPORAL / BITEMPORAL| I["%bitemporal_dataloader"]
    H -->|FORMAT_CAT| J["%mp_loadformat"]
    H -->|REPLACE| K[Inline delete-all + append\nin %mpe_targetloader]

%mpe_targetloader (sas/sasjs/macros/mpe_targetloader.sas) is the single dispatch point. It reads the current MPE_TABLES record (&dc_dttmtfmt. lt tx_to), aborts if the table is not registered (or has duplicate config records), and routes to the loader. Note the two-phase design: LOADTARGET=NO prepares the intermediate outds_* tables so the approver can review diffs; LOADTARGET=YES performs the destructive load. Both phases run the same preparation logic, so the reviewed diffs correspond to what is actually applied.

The Temporal Model

Bitemporal tables carry two independent time dimensions:

• Business time (bus_from / bus_to) — when the fact was true in the real world. Present on both staging and base tables.
• Technical time (tech_from / tech_to, a.k.a. transaction time) — when the record was known to the database. Present on the base table only; the loader stamps these itself.

All validity is expressed with half-open intervals (from <= t < to). Queries against bitemporal tables need two conditions and must not use BETWEEN or from LE t LE to (the latter excludes boundary records — see the macro header for background):

where &bus_from le [tstamp] lt &bus_to
  and &tx_from le [tstamp] lt &tx_to

"Current" records have tech_to set to the high date ('31DEC9999:23:59:59'dt when called from %mpe_targetloader). Closing out a record means setting tech_to = now — records are never physically deleted from a temporal table, they are superseded.

Key Components

Component	Location	Role
%mpe_targetloader	sas/sasjs/macros/mpe_targetloader.sas	Reads MPE_TABLES config and dispatches per LOADTYPE; implements REPLACE inline
%bitemporal_dataloader	sas/sasjs/macros/bitemporal_dataloader.sas	Generic loader for UPDATE / TXTEMPORAL / BITEMPORAL
%bitemporal_closeouts	sas/sasjs/macros/bitemporal_closeouts.sas	Closes out (sets tech_to=now) live records matching a key
%mp_retainedkey	SASjs core	Generates retained (surrogate) keys when RK_UNDERLYING is configured
%mp_rowhash	SASjs core	MD5 hash of non-temporal columns, used for change detection
%mp_storediffs	SASjs core	Writes row-level audit records to AUDIT_LIBDS
MPE_DATALOADS	&dclib	Load log (counts, duration, macro version, user) — written only by %bitemporal_dataloader
MPE_LOCKANYTABLE	&dclib	Lock control table used by %mp_lockanytable

%bitemporal_dataloader Execution Flow

The macro builds a series of work.bitemp* intermediate tables, then applies the changes in two target-table operations (closeout, then append) under a single lock.

1. Pre-checks and setup

• Early return if the staging table is empty; hard abort if &syscc > 0.
• CLOSE_VARS is not supported on REDSHIFT / POSTGRES / SNOWFLAKE engines (returns with a NOTE).
• A zero-row snapshot of the base table (&basecopy) is taken with data ... set base; stop; — this doubles as a lock check, since metadata functions fail against a locked table.
• proc contents on the base table feeds column lists. Columns are split into character and numeric lists for hashing (%mp_rowhash hashes the two types via separate arrays). Temporal columns, the processed column, and the delete-flag column are excluded from the hash.
• Table names containing ___TMP___ or _____ are rejected (they would collide with generated temp columns).

2. Locking

For LOADTARGET=YES, the base table (and audit table, if configured) is locked via %mp_lockanytable before any staging prep, because the load is a two-part update (closeouts + append) and must not interleave with another load.

3. Staging preparation (work.bitemp0_append)

• If RK_UNDERLYING is configured, %mp_retainedkey maps business keys to retained keys (filtering the base lookup to live records, &now < tech_to, for temporal types); otherwise the staging table is used as-is.
• bus_from / bus_to overrides are applied if provided; the processed column is stamped with the load timestamp (now). Even with processed=0, a column literally named PROCESSED_DTTM on the base table is used if present.
• The MD5 change-detection hash is computed for every staged record.
• If the staging table contains the delete-flag column (_____DELETE__THIS__RECORD_____), rows flagged "Yes" are diverted to &outds_del and closed out via %bitemporal_closeouts (PK is taken as bus_from + business key). The remaining rows continue as bitemp0_append.

4. CLOSE_VARS closeout

When CLOSE_VARS (a subset of the PK) is supplied, live base records whose CLOSE_VARS values appear in staging but whose full PK does not are closed out. This handles "this group was fully reloaded, so anything missing was removed" semantics without reloading the whole table.

5. Uniqueness check

If CHECK_UNIQUENESS=YES (the default) or business-date overrides are in play, the staging table is sorted nodupkey by the PK; a row-count mismatch aborts the load and releases the locks. The staging table must be a unique snapshot of the business key at one point in business time.

6. Base extract (work.bitemp0_base)

Only base records matching staged PKs are extracted — for temporal load types, only currently live records (now < tech_to). A left join from the staged keys to the base produces ___TMP___NEW_FLG to identify brand-new keys. This is engine-specific:

• OLEDB (SQL Server) — staged keys are pushed to a ##global temp table and joined via explicit pass-through.
• REDSHIFT / POSTGRES / SNOWFLAKE — an in-database temp table is created like the base, stripped to PK + an added md5 varchar(32) column, loaded with staged keys, and joined via pass-through. Snowflake uses transient tables; Redshift gets alter sortkey none plus any DCBL_REDSH config options from MPE_CONFIG.
• CAS — a FedSQL join against a CASUSER copy.
• BASE/other — plain PROC SQL in SAS.

7. Change classification

• &outds_add — staged records flagged as new (no base match). They get tech_from=now, tech_to=high_date.
• work.bitemp1_current — matched base records, re-hashed so hashes are comparable.
• Inserts (BITEMPORAL only) — a staged record whose business range falls strictly inside an existing record's range splits the existing record into a "before" and "after" segment (bitemp3_inserts / bitemp3a_inserts). The split segments replace the original in the comparison set (bitemp3b_newbase).
• Updates (bitemp4*) — staged records matching a base PK but with a different hash or different business dates. Base and staged versions are stacked, deduplicated, then aligned in two passes over the business timeline per key: a forward pass (carry bus_from forward across identical hashes; a staged record trims the preceding base record's range) and a reverse pass (carry bus_to back; records fully subsumed by the new version are deleted). Records that end up byte-identical to what is already stored are dropped via a hash lookup that includes the business dates (bitemp5a_lkp / bitemp5b_updates, BITEMPORAL only).

8. Closeout application

Changed records are closed out in the target before the new versions are appended. SAS SQL has no UPDATE-with-join, so a correlated EXISTS subquery against work.bitemp5d_subquery is used (pushed in-database as a temp table for OLEDB / REDSHIFT / POSTGRES / SNOWFLAKE):

• BITEMPORAL — per key, the closeout range is min(bus_from) / max(bus_to) of the changed records: update base set tech_to=now (, processed=now) where tech_from <= now < tech_to and exists (key match and base.bus_from >= min and base.bus_to <= max).
• TXTEMPORAL — same, on key only.
• UPDATE — changed records are physically deleted (they are re-appended in the next step). On CAS this uses table.deleteRows with a whereTable; temporal types are not supported on CAS and abort.
• BUSTEMPORAL — closeouts are not implemented; the macro aborts at this point ("BUSTEMPORAL NOT YET SUPPORTED").

9. Append, unlock, audit, log

• The union of modified + new records, deduplicated on all columns (bitemp6_unique), is appended to the base table (proc append ... force nowarn; CAS appends via a varchar-casting step; Redshift applies DCBL_REDSH options). Locks are then released.
• If outds_audit is set (always, from %mpe_targetloader — AUDIT_LIBDS defaulting to &dclib..MPE_AUDIT), %mp_storediffs compares the pre-load snapshot with the applied changes and appends row-level audit records; modified-row entries with no actual value change are removed (MOVE_TYPE="M" and IS_PK=0 and IS_DIFF=0).
• A summary row is inserted into &dclib..MPE_DATALOADS (libref, dsn, etlsource, loadtype, changed/new/deleted counts, duration, macro version, user, timestamp) unless LOG=0.

REPLACE Load Type

REPLACE is the simplest load type and is deliberately not routed through %bitemporal_dataloader — it is implemented inline in %mpe_targetloader and simply wipes the target table (delete * from) and re-appends the staging table verbatim, with no history, change detection, key matching, audit rows, or MPE_DATALOADS logging. See replace-load-type.md for the full deep dive.

Supporting Tables

Table	Role
MPE_TABLES	Per-table load configuration (loadtype, keys, temporal vars, audit target)
MPE_SELECTBOX	Dropdown values, including the LOADTYPE list (seeded in mpe_makedata.sas)
MPE_SUBMIT / MPE_REVIEW	Approval workflow state
MPE_LOADS	Submission-level log of CSV package loads
MPE_DATALOADS	Per-load statistics (temporal/UPDATE loads only)
MPE_AUDIT (or AUDIT_LIBDS)	Row-level change audit (temporal/UPDATE/FORMAT_CAT loads only)
MPE_LOCKANYTABLE	Lock control table
MPE_CONFIG	Engine-specific config (e.g. DCBL_REDSH scope for Redshift bulk options)

Testing

Unit tests for the loaders live next to the macros: sas/sasjs/macros/bitemporal_dataloader.test.[1-4].sas and sas/sasjs/macros/mpe_targetloader.test.sas (REPLACE loadtype), executed as sasjs tests. See testing.md for how to run them (npm run 4gl then sasjs test -t 4gl from the sas/ directory), and remember sasjs lint after touching any .sas files.