# REPLACE Load Type — Technical Deep Dive

This document describes the REPLACE load type in detail. For the overall loader architecture, the temporal model, and the other load types, see [bitemporal-dataloader.md](bitemporal-dataloader.md).

## Overview

REPLACE is the simplest load type and is deliberately **not** routed through `%bitemporal_dataloader`. It is implemented inline in `%mpe_targetloader` (`sas/sasjs/macros/mpe_targetloader.sas`, search for `&loadtype=REPLACE`). There is no history, no change detection, no key matching — the target table is wiped and reloaded from the staging table verbatim.

## Actual load (`LOADTARGET=YES`)

```sas
%mp_lockanytable(LOCK, lib=&lib, ds=&ds, ...)

data WORK.&STAGING_DS;
  set WORK.&STAGING_DS;
  /* only if the target contains the MPE_TABLES.VAR_PROCESSED variable: */
  &VAR_PROCESSED = &now;
  drop _____DELETE__THIS__RECORD_____;
run;

%if &engine_type=CAS %then %do;
  /* prep first: cast varchar columns in a CASUSER copy of staging */
  proc contents noprint data=&libds out=work.rpl_base_cols(keep=name type);
  proc contents noprint data=WORK.&STAGING_DS out=work.rpl_stag_cols(keep=name type);
  /* work.rpl_vchars = columns that are varchar in target AND in staging */
  data casuser.&tmpds;              /* temp copy with varchars casted   */
    length <varchar col> varchar(*); ...
    set WORK.&STAGING_DS (rename=(<varchar col>=<tmp> ...));
    <varchar col>=<tmp>; ... drop <tmp> ...;
  run;
  /* unlock + abort if any error so far - nothing destructive done yet */
  %mp_abort(iftrue= (&syscc>0) ...)
  /* destructive step, deliberately last before the append */
  proc cas;
    table.deleteRows / table={caslib="&lib",name="&ds",where="1=1"};
  quit;
  data &libds (append=yes) / sessref=dcsession;
    set casuser.&tmpds;
  run;
  proc sql; drop table CASUSER.&tmpds; quit;
%end;
%else %do;
  /* unlock + abort if any error so far - nothing destructive done yet */
  %mp_abort(iftrue= (&syscc>0) ...)
  proc sql;
    delete * from &libds;
  quit;
  proc append base=&libds data=WORK.&STAGING_DS force nowarn; run;
%end;

%mp_lockanytable(UNLOCK, lib=&lib, ds=&ds, ...)
```

Step by step:

1. **Lock** — the target is locked via `%mp_lockanytable` (control table `&dclib..MPE_LOCKANYTABLE`). This is the only concurrency guard for the whole operation.
2. **Staging recopy** — the staging dataset is rewritten in place: the processed-timestamp column is set to the approval timestamp **only if** the target table contains the variable named by `MPE_TABLES.VAR_PROCESSED` (unlike `%bitemporal_dataloader`, there is no `PROCESSED_DTTM` fallback). The delete-flag column `_____DELETE__THIS__RECORD_____` is unconditionally dropped — per-record delete semantics do not exist in REPLACE (everything is deleted anyway), so a delete flag submitted by the user is silently discarded (a `drop` of a non-existent variable just produces a warning).
3. **Prepare the load (CAS only)** — fixed char variables cannot be appended to CAS varchar columns, so the staging table is first copied to a CASUSER temp table with every column that is varchar in the target (and present in staging) redeclared as `varchar(*)` via generated `length` / `rename` / assignment statements. This mirrors the CAS append in `%bitemporal_dataloader`.
4. **Pre-destructive abort check** — if `&syscc > 0` after all preparation, the target is unlocked and the macro aborts via `%mp_abort`. Nothing destructive has happened at this point, so a failed prep leaves the target intact (and without a stale lock).
5. **Delete all rows** — every row is removed from the target while preserving structure, indexes and metadata, deliberately as the last step before the append (to minimise the time the target sits empty). On most engines this is `proc sql; delete * from &libds;` (passed through as a `DELETE` on database libraries). CAS tables do not support SQL deletes, so on the CAS engine the table is truncated instead via the `table.deleteRows` action with `where="1=1"` (same approach as `%bitemporal_closeouts`), with the libref passed as the caslib.
6. **Append staged rows** — the entire staging table is appended. On most engines this is `proc append ... force nowarn` (`force` allows the append to proceed despite attribute mismatches, so lengths/formats may be coerced or values truncated, and columns present in only one side are handled; `nowarn` suppresses the associated warnings). On CAS the pre-cast CASUSER temp table is appended with `data &libds (append=yes) / sessref=dcsession` and then dropped.
7. **Unlock**.

## Diff screen (`LOADTARGET=NO`)

When the approver reviews a REPLACE submission, no real comparison is performed:

```sas
/* is full replace so treat all staged records as new in diff screen */
data work.outds_mod work.outds_add;
  set work.&staging_ds;
  output work.outds_add;   /* every staged record is "NEW" */
run;                       /* outds_mod stays empty        */

/* previous table will be considered fully deleted */
data work.outds_del;
  set &lib..&ds;           /* every existing record is "DELETED" */
run;
```

The approval screen therefore always shows full-table turnover: the entire current table as deleted and the entire staging table as added, with no "modified" records and no original/current comparison.

## What REPLACE does not do

Because it bypasses `%bitemporal_dataloader`, none of the following apply to REPLACE:

* **No row-level audit** — nothing is written to `AUDIT_LIBDS` / `MPE_AUDIT`. The record of the change is the approval package itself (staged CSV and the `TEMPDIFFS` CSV stored under `&mpelocapprovals/&LOAD_REF`).
* **No `MPE_DATALOADS` log entry** — the load-log insert lives inside `%bitemporal_dataloader`, so REPLACE loads do not appear in load history. (Consequently the SHOW_DIFFS timestamp lookup in `postdata.sas` finds no `MPE_DATALOADS` row for a REPLACE load and falls back to the current datetime.)
* **No PK usage** — `BUSKEY` is ignored: no uniqueness check, no dedup, no join to existing data. Staged rows are loaded exactly as submitted, duplicates included.
* **No temporal handling** — `VAR_TXFROM`/`VAR_TXTO`/`VAR_BUSFROM`/`VAR_BUSTO` and `CLOSE_VARS` are ignored. A REPLACE table should not be treated as temporal; querying it with the usual validity filters makes no sense.
* **No retained-key handling** — `RK_UNDERLYING` is ignored.
* **Minimal engine-specific handling** — unlike the temporal loaders, there is no pass-through/temp-table optimisation for OLEDB, Redshift, Postgres or Snowflake. The only engine conditional is CAS, where rows are removed via the `deleteRows` action and appended via a varchar-casting CASUSER temp table and a data-step append (SQL deletes and fixed-char-to-varchar appends are not possible on CAS).
* **Not atomic** — the delete/truncate and append are separate steps with no transaction or rollback, so a session failure between them still leaves the target empty or partially loaded (recovery is a re-approval of the same or a previous staging package). The risk is mitigated by performing all preparation first, aborting on any error before the destructive step, and executing the delete/truncate immediately before the append.

## Interaction with Row Level Security

REPLACE is incompatible with `EDIT`-scope RLS rules (a full-table wipe cannot honour row-level write restrictions); this is enforced at edit time in both directions — see [row-level-security.md](row-level-security.md#incompatibility-with-replace-load-type) and [issue #211](https://git.datacontroller.io/dc/dc/issues/211). `VIEW`-scope rules remain compatible.