# Row Level Security — Technical Deep Dive

This document explains **how** Row Level Security (RLS) is implemented in the Data Controller backend. For the user-facing configuration guide, see [docs.datacontroller.io/row-level-security](https://docs.datacontroller.io/row-level-security/).

## Overview

RLS in Data Controller is implemented as **server-side WHERE clause generation**. No data leaves SAS without passing through a dynamically generated filter. The filter is built at runtime per request, based on:

1. The requesting user's group memberships (SAS metadata groups + DC groups)
2. The active rules in the `MPE_ROW_LEVEL_SECURITY` control table
3. The access mode (VIEW / EDIT / download / upload)

Because the filter is expressed as a standard SAS `WHERE` expression, it works against **any** engine — Base SAS datasets, database libraries (via implicit SQL pass-through pushdown), SPDE, CAS libnames, etc.

## Request Flow

```mermaid
flowchart TD
    A[Client request\nview / edit / download / upload] --> B{Service mode}
    B -->|VIEW / EDIT / DLOAD| C["%mpe_filtermaster(mode, libds)"]
    B -->|ULOAD stagedata.sas| C
    C --> D["%mpe_getgroups()\nmetadata groups + MPE_GROUPS"]
    D --> E{User in &mpeadmins?}
    E -->|Yes| F[No RLS filter\n1=1]
    E -->|No| G[Lookup active rules in\nMPE_ROW_LEVEL_SECURITY\nfor libref.table + user's groups]
    G --> H{Rules found?}
    H -->|No| F
    H -->|Yes| I["%mp_filtergenerate()\nper group, OR'd together"]
    I --> J[WHERE clause written\nto temp fileref]
    F --> J
    J --> K{Read or Write?}
    K -->|Read\nviewdata / getdata / getrawdata| L["where %inc filtref;;\nrows filtered server-side"]
    K -->|Write\nstagedata| M[Inverse filter\nwhere not( filtref )]
    M --> N{badrecords > 0?}
    N -->|Yes| O[Abort submission\nSecurity Problem]
    N -->|No| P[Staging proceeds\nto approval workflow]
```

## Key Components

| Component | Location | Role |
|---|---|---|
| `MPE_ROW_LEVEL_SECURITY` | `&mpelib` (DC control library) | The rule table (scope, group, libref, table, logic, subgroup, variable, operator, raw value, active flag) |
| `%mpe_filtermaster()` | `sas/sasjs/macros/mpe_filtermaster.sas` | Master macro that assembles the full WHERE clause for a request |
| `%mpe_getgroups()` | `sas/sasjs/macros/mpe_getgroups.sas` | Resolves group membership (metadata groups via `%dc_getusergroups` + `MPE_GROUPS` DC-internal groups) |
| `%mp_filtergenerate()` | SASjs core | Converts a query table (logic/subgroup/variable/operator/value rows) into WHERE clause text |
| `%mp_filtercheck()` | SASjs core | Validates rule syntax at *edit time* (defence against SAS code injection) |
| `mpe_row_level_security_postedit.sas` | `sas/sasjs/services/hooks/` | Post-edit hook that runs `%mp_filtercheck` whenever the RLS table itself is edited |

## The Modes

`%mpe_filtermaster` accepts a `mode` parameter, and every service that surfaces data calls it with the appropriate mode:

| Mode | Caller | Purpose |
|---|---|---|
| `VIEW` | `services/public/viewdata.sas`, `getchangeinfo.sas` | Read-only table viewer |
| `EDIT` | `services/editors/getdata.sas` | The EDIT grid (adds "current records only" validity filtering) |
| `DLOAD` | `services/public/getrawdata.sas` | Raw file downloads (RLS scope treated as `VIEW`) |
| `ULOAD` | `services/editors/stagedata.sas` | **Upload validation** (RLS scope treated as `EDIT`) |

Scope mapping: `DLOAD` requests match rules with `RLS_SCOPE in ('VIEW','ALL')`; `ULOAD` requests match rules with `RLS_SCOPE in ('EDIT','ALL')`.

## Execution Flow of `%mpe_filtermaster`

The macro writes the final WHERE expression to a temporary **fileref** (`outref`), line by line. (A fileref is used because a generated filter may exceed the 64k macro variable limit — and note that `%include` of a fileref is not allowed directly in a `proc sql` where clause, hence callers typically use it in a DATA step or data step view.)

### 1. User-supplied filter (FILTER_RK)

If the request includes a stored filter (`filter_rk > 0`), its clauses are read from `MPE_FILTERANYTABLE` / `MPE_FILTERSOURCE` and generated first via `%mp_filtergenerate`. RLS clauses are then **AND-ed on top** — a user filter can only ever narrow results, never widen them beyond RLS.

### 2. Validity-date filtering (EDIT / DLOAD only)

`MPE_TABLES` may define `VAR_TXFROM` / `VAR_TXTO` (SCD2-style validity variables) for the target table. Unless the user explicitly filtered on those variables, the macro appends:

```
("<current datetime>"dt < VAR_TXTO)
```

so that only current records are surfaced in the EDIT grid and downloads.

### 3. Group resolution and admin bypass

```sas
%mpe_getgroups(user=%mf_getuser(), outds=work.groups)
```

Groups come from two sources, concatenated:

* SAS metadata groups (Viya / EBI / Base-specific logic in `%dc_getusergroups`)
* The `MPE_GROUPS` DC table (group assignments managed inside Data Controller)

If the user is a member of the `&mpeadmins` group, **RLS is skipped entirely** — admins always see all rows.

### 4. Rule extraction

Non-admin users trigger a lookup of active, current rules:

```sas
create table work.&rlsds as
  select rls_group, rls_group_logic, rls_subgroup_logic, rls_subgroup_id,
         rls_variable_nm, rls_operator_nm, rls_raw_value
  from &mpelib..mpe_row_level_security
  where &dc_dttmtfmt. lt tx_to          /* only current (non-deleted) rules */
    and rls_scope in ("&scopeval",'ALL')
    and upcase(rls_group) in (select upcase(groupname) from work.groups)
    and rls_libref = "<libref>" and rls_table = "<dsname>"
    and rls_active = 1
  order by rls_group, rls_subgroup_id;
```

### 5. Clause assembly

If rules exist, they are appended to the fileref as `AND ( ... )`. Each **group** the user belongs to contributes one sub-filter, and the group-level sub-filters are joined with `OR`:

```
AND ( <group 1 filter> OR <group 2 filter> ... )
```

Each group's filter is produced by `%mp_filtergenerate`, which honours:

* `RLS_GROUP_LOGIC` — how subgroups (identified by `RLS_SUBGROUP_ID`) are joined
* `RLS_SUBGROUP_LOGIC` — how individual clauses within a subgroup are joined (AND/OR)

So membership in multiple groups is **permissive** (OR): the user sees the union of rows permitted by each of their groups.

### 6. Empty-filter fallback

If nothing was written to the fileref (no stored filter, no validity vars, no matching RLS rules), the macro writes a literal `1=1` so the fileref can always be consumed as a valid WHERE expression. Absence of rules therefore means **no restriction** — RLS is opt-in per table/group.

## How Callers Apply the Filter

### Read path (viewdata.sas)

```sas
%mpe_filtermaster(VIEW, &libds, dclib=&mpelib, filter_rk=&filter_rk,
  outref=filtref, outds=work.query)

data work.viewdata;
  set &libds;
  where %inc filtref;;
  if _n_ > &DC_MAXOBS_WEBVIEW then stop;
run;
```

The fileref is `%include`d directly inside the `where` statement — the filter never passes through client-visible state, so it cannot be tampered with. For database libraries the WHERE expression is pushed down to the database by the SAS engine.

### Write path (stagedata.sas) — inverse filter

Uploads cannot be filtered; instead the filter is **inverted** and any submitted row matching the inverse is rejected:

```sas
%mpe_filtermaster(ULOAD, &libds, dclib=&mpelib, outref=filtref, ...)

/* prepare inverse query */
data _null_;
  infile filtref end=eof;
  file &tempref;
  if _n_=1 then put 'where not(';
  input;  put _infile_;
  if eof then put ')';
run;

data work.badrecords;
  set work.jsdata;      /* rows submitted by the user */
  %inc &tempref/source2;;
run;

%mp_abort(iftrue=(%mf_nobs(work.badrecords)>0)
  ,msg=%str(Security Problem - N unauthorised records submitted))
```

If even one submitted row falls outside the user's permitted row set, the entire staging request is aborted before any approval/apply step.

## Incompatibility with REPLACE Load Type

RLS with `EDIT` scope is **incompatible** with tables configured with `LOAD_TYPE=REPLACE` in `MPE_TABLES`. A REPLACE load wipes and reloads the entire target table, so row-level filtering of submitted records cannot be enforced meaningfully (the rows a user is *not* allowed to see would also be deleted). Backend validations therefore abort in both directions (see [issue #211](https://git.datacontroller.io/dc/dc/issues/211)):

1. **`mpe_row_level_security_postedit.sas`** — aborts when activating a rule with `RLS_SCOPE in ('EDIT','ALL')` against a table whose current `MPE_TABLES` record has `LOADTYPE='REPLACE'`.
2. **`mpe_tables_postedit.sas`** — aborts when setting `LOADTYPE='REPLACE'` on a table that already has active, current `RLS_SCOPE in ('EDIT','ALL')` rules in `MPE_ROW_LEVEL_SECURITY`.

`VIEW`-scope rules remain compatible with REPLACE loads, since they only affect read paths.

## Edit-Time Validation (Injection Defence)

Because `RLS_RAW_VALUE` is free text that ends up inside a generated WHERE clause, it is a potential SAS code-injection vector. Mitigations:

1. **`mpe_row_level_security_postedit.sas`** — a post-edit hook on the `MPE_ROW_LEVEL_SECURITY` table itself. Every newly staged rule (with `rls_active=1`) is grouped by target `libref.table` and run through `%mp_filtercheck(targetds=..., abort=YES)`, which compiles/tests each clause against the real target table and rejects the whole submission on invalid syntax (see [mp_filtercheck](https://core.sasjs.io/mp__filtercheck_8sas.html)).
2. **Format rules** enforced by validation: character values must be single quoted, `IN`/`NOT IN` values must be bracketed, `BETWEEN` must contain `AND`.

Additionally, editing `MPE_ROW_LEVEL_SECURITY` is itself a Data Controller table edit, so it goes through the normal approval workflow, audit trail (`MPE_AUDIT`), and can be column-restricted via [Column Level Security](https://docs.datacontroller.io/column-level-security/).

## Worked Example

Given these active rules (all for `MYLIB.MYDS`, `RLS_ACTIVE=1`):

| SCOPE | GROUP | GROUP_LOGIC | SUBGRP_LOGIC | SUBGRP_ID | VAR | OP | VALUE |
|---|---|---|---|---|---|---|---|
| ALL | Group 1 | AND | AND | 1 | VAR_2 | IN | ('this','or') |
| ALL | Group 1 | AND | AND | 1 | VAR_3 | < | 42 |
| ALL | Group 2 | AND | AND | 1 | VAR_4 | CONTAINS | 'xyz' |

A non-admin user in **both** groups, opening the VIEW page, gets a fileref containing:

```
AND ( (VAR_2 IN ('this','or') AND VAR_3 < 42) OR (VAR_4 CONTAINS 'xyz') )
```

resulting in:

```sas
data work.viewdata;
  set mylib.myds;
  where (VAR_2 IN ('this','or') AND VAR_3 < 42) OR (VAR_4 CONTAINS 'xyz');
run;
```

The same user submitting an EDIT upload has the inverse applied to their staged rows; any row not matching the expression above aborts the submission.

## Summary of Security Properties

* **Enforced server-side** in every read service (VIEW/EDIT/DLOAD) and every write service (ULOAD via inverse filter).
* **Admin bypass** is explicit (`&mpeadmins` group membership check).
* **Multi-group semantics are OR** (union of permitted rows); rules within a group/subgroup are AND/OR per configuration.
* **Fail-open by design**: if no rules match a table, it is unrestricted — RLS must be opted into per table.
* **Injection-resistant**: values are validated with `%mp_filtercheck` at the time rules are edited, not at query time.