blog(derived-variables): computed columns you can filter on #27

Open
hermes wants to merge 2 commits from site/derived-variables into main
pull from: site/derived-variables
5 changed files with 138 additions and 0 deletions

No files matched your search

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

+138
View File
@@ -0,0 +1,138 @@
---
title: 'Derived Variables: computed columns in the EDIT screen'
description: Data Controller can now compute columns that do not exist in the table - and, crucially, filter on them. Configured per table in MPE_DERIVED, computed on every read, never stored.
date: '2026-10-21 12:00:00'
author: 'Allan Bowe'
authorLink: https://www.linkedin.com/in/allanbowe/
previewImg: './derived-grid.png'
tags:
- Data Controller
- Features
---
# Derived variables: computed columns you can filter on
A customer asked for something simple and awkward: a column in the EDIT screen that is not in the table. A region worked out from a state code. A line value that is quantity times price. Not stored, not loaded, just *there* - and, they added, filterable.
The "not stored" part is easy. The "filterable" part is where the design lives, and it is worth writing down, because it explains every other decision in the feature.
## Why a computed column is not automatically filterable
You would write it in a DATA step:
```sas
data out;
set mylib.mytable;
REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
where REGION = 'West';
run;
```
That fails. A `WHERE` statement applies to the data set named on the `SET` statement - the *input* - and on that input `REGION` does not exist. SAS cannot see a variable that the same step creates later. You can put the assignment and the filter in separate steps, but then the filter runs after the step has read every row, which is precisely the work the filter exists to avoid - and for a large table that is the difference between a fast screen and a coffee break.
So the calculation has to happen somewhere the filter can see it. In Data Controller it happens in a **DATA step view**:
```sas
data work.vw_derived / view=work.vw_derived;
set mylib.mytable;
attrib REGION length=$8 label='REGION=ifc(...)' format=$8.;
REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
run;
data out;
set work.vw_derived;
where %inc filtref;;
run;
```
The view is not executed when it is defined. When the second step reads it, the assignment runs as part of the view's step, so by the time the `WHERE` is evaluated `REGION` is a column of the input. The filter is unchanged, the row-level security clause is unchanged, and nothing extra is read.
## What it looks like
The demo table below has four rows. `LINE_VALUE` is `QTY * PRICE`, `REGION` is derived from `STATE`, and `REGION_LABEL` is built on `REGION`. None of the three is a column of the table, and all three are read-only in the grid:
![](./derived-grid.png)
Because they are on the input to the filter, they filter like any other column. Here is `REGION = 'West'`, with the filter bar naming the derived variable:
![](./derived-filter-region.png)
and a numeric one, `LINE_VALUE > 50`:
![](./derived-filter-value.png)
The value list in the filter dialog is built the same way, from the table under the rest of the filter - so a derived column gets a dropdown of its distinct values, not a blank box.
## Configuration
`MPE_DERIVED` is itself a Data Controller table, so it is maintained through the EDIT screen with the usual submit / review / approve workflow. One row per variable - these are the three rows behind the screenshots above:
![](./derived-config.png)
```
VAR_NM REGION
VAR_DEPS STATE
VAR_TYPE C
VAR_LEN 8
VAR_FMT (blank)
VAR_LABEL (blank)
VAR_LOGIC ifc(STATE in ('CA','OR','WA'),'West','East')
ACTIVE 1
```
and the chained one:
```
VAR_NM REGION_LABEL
VAR_DEPS REGION STATE
VAR_TYPE C
VAR_LEN 14
VAR_FMT (blank)
VAR_LABEL Region and state
VAR_LOGIC cats(REGION,'-',STATE)
ACTIVE 1
```
Which produces, at read time:
```sas
attrib REGION length=$8 label='REGION=ifc(STATE in (''CA'',''OR'',''WA''),''West'',''East'')' format=$8.;
REGION = ifc(STATE in ('CA','OR','WA'),'West','East');
attrib REGION_LABEL length=$14 label='Region and state' format=$14.;
REGION_LABEL = cats(REGION,'-',STATE);
```
Type and length are declared rather than guessed, so a value is truncated exactly as specified. The format is optional; when it is left blank a character variable gets `$<length>.` and a numeric one `<length>.`, and a date, time or datetime format makes the column render - and filter - as a date, the same way a stored column does. The label is `VAR_LABEL` when you supply one, otherwise the name, an equals sign and the expression, so a user who switches labels on can see where the value came from.
## Dependencies, without a sequence number
`REGION_LABEL` needs `REGION` to exist first. Rather than maintain a sort order, each row declares its inputs in `VAR_DEPS`, and the statements are emitted in dependency order. Adding a variable in the middle of a chain means editing one row's `VAR_DEPS`, not renumbering the table.
`VAR_DEPS` lists *every* input - physical columns and derived variables alike - and the declaration is checked against the expression in both directions: a derived variable the expression references must be declared, and a declared name the expression does not reference is an error. A declared name must be a column of the table or a derived variable of the same table, and a cycle is rejected with the path that forms it:
```
VAR_DEPS contains a cycle: A -> B -> C -> A
```
## What it will not do
**Expressions only.** `VAR_LOGIC` must be a single SAS expression. Semicolons, macro triggers (`%`, `&`) and comments are rejected, so one row can never be more than an expression - which is the point: a config table is not a place to write programs. Conditional logic is available through functions such as `ifc()`, `choosec()`, `cats()`, `catx()`, `put()`, `input()` and `intnx()`.
**No shadowing, no self-reference.** A derived variable may not reuse the name of a column in the target table. Nor may it reference itself: `SUMMER = SUMMER + 1` implies a retained value, which depends on the order rows happen to arrive in.
**Never stored.** Derived values are computed on every read. Nothing derived is submitted, reviewed or written back - the approval package contains physical columns only, and a crafted submission that includes a derived column is discarded before the security check runs.
**EDIT screen only, for now.** The VIEW screen and the download services do not show derived columns. A saved filter that references one still applies there and is evaluated the same way.
**Not for row level security.** A rule in `MPE_ROW_LEVEL_SECURITY` cannot reference a derived column: the rule is evaluated against the submitted data, where the column does not exist. Saving a rule that names one is rejected with an explanation rather than left to fail at submit time.
## One warning
An expression can reference a column that is hidden from a user by [Column Level Security](https://docs.datacontroller.io/column-level-security/) - and the derived column will show the result. Configure derived variables with the same care as any other data exposure.
## Availability
Derived variables ship in the next release, on SAS 9, Viya and SASjs Server. The configuration table, the settings and the post-edit validation are created by the usual upgrade - there is nothing to install beyond the release itself.
The full reference, including every column of `MPE_DERIVED` and the validation rules applied at submit time, is in the [docs](https://docs.datacontroller.io/derived-variables/).