From 183037c4d3fdf5cb0a28c1dac773d8ea47875561 Mon Sep 17 00:00:00 2001 From: dc Date: Mon, 19 Oct 2026 23:20:18 +0000 Subject: [PATCH] docs(config): derived variables in the EDIT screen New Configuration page explaining derived (computed) variables: what they are, how to configure them in MPE_DERIVED, how filtering works, the rules and limits (expressions only, self-contained, read-only, not stored), and the data-exposure and performance notes. New MPE_DERIVED table page, registered in the Table Guide nav. --- docs/derived-variables.md | 70 ++++++++++++++++++++++++++++++++++++++ docs/tables/mpe_derived.md | 24 +++++++++++++ mkdocs.yml | 2 ++ 3 files changed, 96 insertions(+) create mode 100644 docs/derived-variables.md create mode 100644 docs/tables/mpe_derived.md diff --git a/docs/derived-variables.md b/docs/derived-variables.md new file mode 100644 index 0000000..be818e0 --- /dev/null +++ b/docs/derived-variables.md @@ -0,0 +1,70 @@ +--- +layout: article +title: Derived Variables +description: Derived variables are computed columns that appear in the EDIT screen and can be filtered on, without being stored in the target table. +--- + +# Derived Variables + +A derived variable is a column that does not exist in the target table, but is computed when the table is loaded into the [EDIT screen](/dc-userguide/#edit). It is read-only, it appears after the physical columns, and it can be filtered on like any other column. + +Derived variables are useful when users need to work with a value that is a function of the stored data - a banding, a concatenation, a calculation - without changing the table, the load process, or the ETL. + +The values are **not stored**. They are computed on every read, and nothing derived is written back when a change is submitted or approved. + +## Configuration + +Derived variables are configured per table in the [MPE_DERIVED](/tables/mpe_derived/) control table, which is itself a Data Controller table - so it is maintained through the EDIT screen with the usual review and approve workflow. + +Each row defines one variable: + + - `LIBREF` / `DSN` - the target table + - `VARIABLE_NM` - the name of the new column + - `VARIABLE_TYPE` - `C` for character, `N` for numeric + - `VARIABLE_LENGTH` - character length, or numeric length (1-8) + - `VARIABLE_FORMAT` - optional, eg `DATE9.` or `DOLLAR12.2` + - `VARIABLE_LOGIC` - the SAS expression that produces the value + - `ACTIVE` - set to 1 to apply the variable + +A simple example, deriving a region from a state: + +``` +VARIABLE_NM REGION +VARIABLE_TYPE C +VARIABLE_LENGTH 8 +VARIABLE_FORMAT (blank) +VARIABLE_LOGIC ifc(STATE in ('CA','OR','WA'),'West','East') +ACTIVE 1 +``` + +For each row, Data Controller generates the following statements when the table is read: + +```sas +attrib REGION length=$8 label='DERIVED: ifc(STATE in (''CA'',''OR'',''WA''),''West'',''East'')' format=$8.; +REGION = ifc(STATE in ('CA','OR','WA'),'West','East'); +``` + +The label defaults to the expression itself, prefixed with `DERIVED: `, so that computed columns are obvious when labels are switched on. The expression is truncated to the maximum label length. + +The type and length are declared, not inferred, so a value is truncated exactly as you specify. If `VARIABLE_FORMAT` is left blank, character variables default to `$.` and numeric variables to `.`; a date, time or datetime format makes the column render (and filter) as a date, time or datetime, the same way a stored column does. + +## Filtering + +The derived variable is created before the filter is applied, so it can be used in the [filter](/filter/) like any other column - including its list of values. A filter on a derived column is stored and re-applied in the normal way, and the value list for a derived column is built from the table under the rest of the filter, exactly as for a stored column. + +## Rules and limits + + - **Expressions only.** `VARIABLE_LOGIC` must be a single SAS expression - no semicolons, macro triggers (`%`, `&`) or comments. Conditional logic is available through functions such as `ifc()`, `choosec()`, `cats()`, `catx()`, `put()`, `input()` and `intnx()`. + - **Self-contained.** An expression may reference the columns of the target table only, not another derived variable. Derived variables are evaluated independently of each other. + - **No collisions.** `VARIABLE_NM` must not match an existing column of the target table, or one of the reserved names (`TX_FROM`, `TX_TO`, `PROCESSED_DTTM`, `_____DELETE__THIS__RECORD_____`). + - **Read-only.** Derived columns cannot be edited in the grid, and any value submitted for them is discarded before the approval package is created. + - **Recomputed on every read.** The value always reflects the current data and the current definition, and there is no history of previous derived values. + +A configuration error - a bad expression, an unknown column, a name collision - stops the EDIT screen with a message naming the variable, rather than silently dropping the column. + +!!! warning "Derived values can expose hidden data" + An expression can reference a column that is hidden from the user by [Column Level Security](/column-level-security/), and the derived column will show the result. Configure derived variables with the same care as any other data exposure. + +## Performance + +The expression is evaluated for each row read, before the filter. The filter already requires a scan of the table, so this adds per-row arithmetic rather than a second pass - but keep expressions cheap (no lookups, no subqueries), and expect the cost to be paid on every load of the EDIT screen, including each request for filter values. diff --git a/docs/tables/mpe_derived.md b/docs/tables/mpe_derived.md new file mode 100644 index 0000000..2560b01 --- /dev/null +++ b/docs/tables/mpe_derived.md @@ -0,0 +1,24 @@ +--- +layout: article +title: MPE_DERIVED +description: The MPE_DERIVED table is used to configure computed columns that appear in the EDIT screen and can be filtered on without being stored in the table. +--- + +# MPE_DERIVED + +The MPE_DERIVED table is used to configure derived (computed) variables - columns that are created when a table is loaded into the EDIT screen, so that they can be displayed and filtered on without being stored. More details are available in the [user guide](/derived-variables/) + +The table is itself registered in [MPE_TABLES](/tables/mpe_tables/), so it is maintained through the EDIT screen with the usual review and approve workflow. + +## Columns + + - `TX_FROM num`: SCD2 open datetime + - 🔑 `TX_TO num`: SCD2 close datetime + - 🔑 `LIBREF char(8)`: The libref of the target table + - 🔑 `DSN char(32)`: The target table + - 🔑 `VARIABLE_NM char(32)`: The name of the derived column + - `VARIABLE_TYPE char(1)`: `C` for character, `N` for numeric + - `VARIABLE_LENGTH num`: Character length, or numeric length (1-8) + - `VARIABLE_FORMAT char(32)`: Optional SAS format, eg `DATE9.` + - `VARIABLE_LOGIC char(4000)`: The SAS expression that produces the value + - `ACTIVE num`: Set to 1 to apply the variable diff --git a/mkdocs.yml b/mkdocs.yml index f8d5d59..453248e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -32,6 +32,7 @@ nav: - MPE_DATASTATUS_TABS: tables/mpe_datastatus_tabs.md - MPE_DATADICTIONARY: tables/mpe_datadictionary.md - MPE_DATALOADS: tables/mpe_dataloads.md + - MPE_DERIVED: tables/mpe_derived.md - MPE_EMAILS: tables/mpe_emails.md - MPE_EXCEL_CONFIG: tables/mpe_excel_config.md - MPE_FILTERANYTABLE: tables/mpe_filteranytable.md @@ -58,6 +59,7 @@ nav: - CAS Tables: cas-tables.md - Column Level Security: column-level-security.md - Dates / Datetimes: dcc-dates.md + - Derived Variables: derived-variables.md - Dynamic Cell Dropdown: dynamic-cell-dropdown.md - Emails: emails.md - Excel: excel.md