dc/.agent/docs/replace-load-type.md

REPLACE Load Type — Technical Deep Dive

This document describes the REPLACE load type in detail. For the overall loader architecture, the temporal model, and the other load types, see bitemporal-dataloader.md.

Overview

REPLACE is the simplest load type and is deliberately not routed through %bitemporal_dataloader. It is implemented inline in %mpe_targetloader (sas/sasjs/macros/mpe_targetloader.sas, search for &loadtype=REPLACE). There is no history, no change detection, no key matching — the target table is wiped and reloaded from the staging table verbatim.

Actual load (LOADTARGET=YES)

%mp_lockanytable(LOCK, lib=&lib, ds=&ds, ...)

data WORK.&STAGING_DS;
  set WORK.&STAGING_DS;
  /* only if the target contains the MPE_TABLES.VAR_PROCESSED variable: */
  &VAR_PROCESSED = &now;
  drop _____DELETE__THIS__RECORD_____;
run;

%if &engine_type=CAS %then %do;
  /* prep first: cast varchar columns in a CASUSER copy of staging */
  proc contents noprint data=&libds out=work.rpl_base_cols(keep=name type);
  proc contents noprint data=WORK.&STAGING_DS out=work.rpl_stag_cols(keep=name type);
  /* work.rpl_vchars = columns that are varchar in target AND in staging */
  data casuser.&tmpds;              /* temp copy with varchars casted   */
    length <varchar col> varchar(*); ...
    set WORK.&STAGING_DS (rename=(<varchar col>=<tmp> ...));
    <varchar col>=<tmp>; ... drop <tmp> ...;
  run;
  /* unlock + abort if any error so far - nothing destructive done yet */
  %mp_abort(iftrue= (&syscc>0) ...)
  /* destructive step, deliberately last before the append */
  proc cas;
    table.deleteRows / table={caslib="&lib",name="&ds",where="1=1"};
  quit;
  data &libds (append=yes) / sessref=dcsession;
    set casuser.&tmpds;
  run;
  proc sql; drop table CASUSER.&tmpds; quit;
%end;
%else %do;
  /* unlock + abort if any error so far - nothing destructive done yet */
  %mp_abort(iftrue= (&syscc>0) ...)
  proc sql;
    delete * from &libds;
  quit;
  proc append base=&libds data=WORK.&STAGING_DS force nowarn; run;
%end;

%mp_lockanytable(UNLOCK, lib=&lib, ds=&ds, ...)

Step by step:

1. Lock — the target is locked via %mp_lockanytable (control table &dclib..MPE_LOCKANYTABLE). This is the only concurrency guard for the whole operation.
2. Staging recopy — the staging dataset is rewritten in place: the processed-timestamp column is set to the approval timestamp only if the target table contains the variable named by MPE_TABLES.VAR_PROCESSED (unlike %bitemporal_dataloader, there is no PROCESSED_DTTM fallback). The delete-flag column _____DELETE__THIS__RECORD_____ is unconditionally dropped — per-record delete semantics do not exist in REPLACE (everything is deleted anyway), so a delete flag submitted by the user is silently discarded (a drop of a non-existent variable just produces a warning).
3. Prepare the load (CAS only) — fixed char variables cannot be appended to CAS varchar columns, so the staging table is first copied to a CASUSER temp table with every column that is varchar in the target (and present in staging) redeclared as varchar(*) via generated length / rename / assignment statements. This mirrors the CAS append in %bitemporal_dataloader.
4. Pre-destructive abort check — if &syscc > 0 after all preparation, the target is unlocked and the macro aborts via %mp_abort. Nothing destructive has happened at this point, so a failed prep leaves the target intact (and without a stale lock).
5. Delete all rows — every row is removed from the target while preserving structure, indexes and metadata, deliberately as the last step before the append (to minimise the time the target sits empty). On most engines this is proc sql; delete * from &libds; (passed through as a DELETE on database libraries). CAS tables do not support SQL deletes, so on the CAS engine the table is truncated instead via the table.deleteRows action with where="1=1" (same approach as %bitemporal_closeouts), with the libref passed as the caslib.
6. Append staged rows — the entire staging table is appended. On most engines this is proc append ... force nowarn (force allows the append to proceed despite attribute mismatches, so lengths/formats may be coerced or values truncated, and columns present in only one side are handled; nowarn suppresses the associated warnings). On CAS the pre-cast CASUSER temp table is appended with data &libds (append=yes) / sessref=dcsession and then dropped.
7. Unlock.

Diff screen (LOADTARGET=NO)

When the approver reviews a REPLACE submission, no real comparison is performed:

/* is full replace so treat all staged records as new in diff screen */
data work.outds_mod work.outds_add;
  set work.&staging_ds;
  output work.outds_add;   /* every staged record is "NEW" */
run;                       /* outds_mod stays empty        */

/* previous table will be considered fully deleted */
data work.outds_del;
  set &lib..&ds;           /* every existing record is "DELETED" */
run;

The approval screen therefore always shows full-table turnover: the entire current table as deleted and the entire staging table as added, with no "modified" records and no original/current comparison.

What REPLACE does not do

Because it bypasses %bitemporal_dataloader, none of the following apply to REPLACE:

• No row-level audit — nothing is written to AUDIT_LIBDS / MPE_AUDIT. The record of the change is the approval package itself (staged CSV and the TEMPDIFFS CSV stored under &mpelocapprovals/&LOAD_REF).
• No MPE_DATALOADS log entry — the load-log insert lives inside %bitemporal_dataloader, so REPLACE loads do not appear in load history. (Consequently the SHOW_DIFFS timestamp lookup in postdata.sas finds no MPE_DATALOADS row for a REPLACE load and falls back to the current datetime.)
• No PK usage — BUSKEY is ignored: no uniqueness check, no dedup, no join to existing data. Staged rows are loaded exactly as submitted, duplicates included.
• No temporal handling — VAR_TXFROM/VAR_TXTO/VAR_BUSFROM/VAR_BUSTO and CLOSE_VARS are ignored. A REPLACE table should not be treated as temporal; querying it with the usual validity filters makes no sense.
• No retained-key handling — RK_UNDERLYING is ignored.
• Minimal engine-specific handling — unlike the temporal loaders, there is no pass-through/temp-table optimisation for OLEDB, Redshift, Postgres or Snowflake. The only engine conditional is CAS, where rows are removed via the deleteRows action and appended via a varchar-casting CASUSER temp table and a data-step append (SQL deletes and fixed-char-to-varchar appends are not possible on CAS).
• Not atomic — the delete/truncate and append are separate steps with no transaction or rollback, so a session failure between them still leaves the target empty or partially loaded (recovery is a re-approval of the same or a previous staging package). The risk is mitigated by performing all preparation first, aborting on any error before the destructive step, and executing the delete/truncate immediately before the append.

Interaction with Row Level Security

REPLACE is incompatible with EDIT-scope RLS rules (a full-table wipe cannot honour row-level write restrictions); this is enforced at edit time in both directions — see row-level-security.md and issue #211. VIEW-scope rules remain compatible.