docs.datacontroller.io/docs/derived-variables.md

layout, title, description

layout	title	description
article	Derived Variables	Derived variables are computed columns that appear in the EDIT screen and can be filtered on, without being stored in the target table.

Derived Variables

A derived variable is a column that does not exist in the target table, but is computed when the table is loaded into the EDIT screen. It is read-only, it appears after the physical columns, and it can be filtered on like any other column.

Derived variables are useful when users need to work with a value that is a function of the stored data - a banding, a concatenation, a calculation - without changing the table, the load process, or the ETL.

The values are not stored. They are computed on every read, and nothing derived is written back when a change is submitted or approved.

Configuration

Derived variables are configured per table in the MPE_DERIVED control table, which is itself a Data Controller table - so it is maintained through the EDIT screen with the usual review and approve workflow.

Each row defines one variable:

• LIBREF / DSN - the target table
• VARIABLE_NM - the name of the new column
• VARIABLE_ORDER - evaluation order; a variable may reference one with a lower order
• VARIABLE_TYPE - C for character, N for numeric
• VARIABLE_LENGTH - character length, or numeric length (1-8)
• VARIABLE_FORMAT - optional, eg DATE9. or DOLLAR12.2
• VARIABLE_LOGIC - the SAS expression that produces the value
• ACTIVE - set to 1 to apply the variable

A simple example, deriving a region from a state:

VARIABLE_NM      REGION
VARIABLE_ORDER   10
VARIABLE_TYPE    C
VARIABLE_LENGTH  8
VARIABLE_FORMAT  (blank)
VARIABLE_LOGIC   ifc(STATE in ('CA','OR','WA'),'West','East')
ACTIVE           1

A later variable can build on it, provided its order is higher:

VARIABLE_NM      REGION_LABEL
VARIABLE_ORDER   20
VARIABLE_TYPE    C
VARIABLE_LENGTH  14
VARIABLE_FORMAT  (blank)
VARIABLE_LOGIC   cats(REGION,'-',STATE)
ACTIVE           1

For each row, Data Controller generates the following statements when the table is read:

attrib REGION length=$8 label='DERIVED: ifc(STATE in (''CA'',''OR'',''WA''),''West'',''East'')' format=$8.;
REGION = ifc(STATE in ('CA','OR','WA'),'West','East');

The label defaults to the expression itself, prefixed with DERIVED: , so that computed columns are obvious when labels are switched on. The expression is truncated to the maximum label length.

The type and length are declared, not inferred, so a value is truncated exactly as you specify. If VARIABLE_FORMAT is left blank, character variables default to $<length>. and numeric variables to <length>.; a date, time or datetime format makes the column render (and filter) as a date, time or datetime, the same way a stored column does.

Filtering

The derived variable is created before the filter is applied, so it can be used in the filter like any other column - including its list of values. A filter on a derived column is stored and re-applied in the normal way, and the value list for a derived column is built from the table under the rest of the filter, exactly as for a stored column.

Derived columns themselves are an EDIT screen feature: the VIEW screen and the download services do not show them. A saved filter that references a derived column still applies there, and is evaluated the same way.

Rules and limits

• Expressions only. VARIABLE_LOGIC must be a single SAS expression - no semicolons, macro triggers (%, &) or comments. Conditional logic is available through functions such as ifc(), choosec(), cats(), catx(), put(), input() and intnx().
• Ordered. Statements are generated in VARIABLE_ORDER sequence. An expression may reference the physical columns of the table, and any derived variable with a lower VARIABLE_ORDER. A reference to a variable that is not defined earlier is rejected, because SAS would otherwise return a missing value without an error. VARIABLE_ORDER is required, and must be a positive integer.
• No collisions. VARIABLE_NM must not match an existing column of the target table, or one of the reserved names (TX_FROM, TX_TO, PROCESSED_DTTM, VAR_TXFROM, VAR_TXTO, VAR_PROCESSED, _____DELETE__THIS__RECORD_____).
• Read-only. Derived columns cannot be edited in the grid, and any value submitted for them is discarded before the approval package is created.
• Recomputed on every read. The value always reflects the current data and the current definition, and there is no history of previous derived values.

A configuration error - a bad expression, an unknown column, a name collision - stops the EDIT screen with a message naming the variable, rather than silently dropping the column.

!!! warning "Derived values can expose hidden data" An expression can reference a column that is hidden from the user by Column Level Security, and the derived column will show the result. Configure derived variables with the same care as any other data exposure.

Performance

The expression is evaluated for each row read, before the filter. The filter already requires a scan of the table, so this adds per-row arithmetic rather than a second pass - but keep expressions cheap (no lookups, no subqueries), and expect the cost to be paid on every load of the EDIT screen, including each request for filter values.