---
layout: article
title: Data Validation
description: Quality in, Quality out! Enforce data quality checks at the point of SAS data entry, both directly via the web interface and also via Excel uploads.
og_image: https://i.imgur.com/P64ijBB.png
---


# Data Controller for SAS® - DQ Validations

## Overview
Quality in, Quality out!  Data Controller lets you enforce quality checks at the point of data entry, both directly via the web interface and also via Excel uploads.

## Default Checks
By default, the following frontend rules are always applied:

* Length checking per target table variable lengths
* Type checking per target table datatypes (Character, Numeric, Date, Time, Datetime)
* Not Null check per target table constraints
* Primary Key checking per business key defined in MPE_TABLES

It is possible to configure a number of other rules by updating the MPE_VALIDATIONS table.  Simply set the `BASE_LIB`, `BASE_DS` and `BASE_COL` values, and ensure `RULE_ACTIVE=1` for it to be applied.

## Configurable Checks

Check back frequently as we keep growing this list of checks.

|Rule Type|Example Value |Description|
|---|---|---|
|CASE|UPCASE|Will enforce the case of cell values.  Valid values: UPCASE, LOWCASE, PROPCASE|
|NOTNULL|(defaultval)|Will prevent submission if null values are present.  Optional - provide a default value.|
|MINVAL|1|Defines a minimum value for a numeric cell|
|MAXVAL|1000000|Defines a maximum value for a numeric cell|
|READONLY|(defaultval) |Renders the column read-only in the editor.  The defaultval is used when rows are added. |
|HIDDEN|(defaultval) |Hides the column from the editor grid while still submitting its data. The defaultval is used when rows are added. |
|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc.  Half-away-from-zero rounding is applied so `-0.5 → -1` and `2.5 → 3`. |
|NUMBER_FORMAT|`{"style":"currency","currency":"GBP"}` |Display-only [`Intl.NumberFormat` renderer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat).  RULE_VALUE is the JSON options object passed straight to `Intl.NumberFormat`.  Does not change the stored value. |
|HARDFORMULA|`= PRICE * VOLUME`|The cell displays a value computed from a formula, using other columns in the same row.  The column is read-only - the user cannot override the result.  See [Formula Rules](#formula-rules) below.|
|SOFTFORMULA|`= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )`|Like HARDFORMULA, but the user can override the computed value and type their own.  See [Formula Rules](#formula-rules) below.|
|HARDREGEX|`^[A-Z]{3}$`|The cell value **must** match the regex pattern, otherwise submission is blocked and the cell is highlighted red.  See [Regex Rules](#regex-rules) below.|
|SOFTREGEX|`^[A-Z]{3}$`|A cell value that does not match the regex pattern is highlighted yellow as a warning, but submission is **not** blocked.  See [Regex Rules](#regex-rules) below.|
|HARDSELECT|sashelp.class.name|A distinct list of values (max 1000) are taken from this library.member.column reference, and the value **must** be in this list.  This list may be supplemented by entries in the MPE_SELECTBOX table.|
|SOFTSELECT|dcdemo.mpe_tables.libref|A distinct list of values (max 1000) are taken from this library.member.column reference, and the user-provided value may (or may not) be in this list.  This list may be supplemented by entries in the MPE_SELECTBOX table.|
|[HARDSELECT_HOOK](/dynamic-cell-dropdown)|/logical/folder/stpname|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem.  User provided values **must** be in this list.  Cannot be used alongside a SOFTSELECT_HOOK.|
|[SOFTSELECT_HOOK](/dynamic-cell-dropdown)|/physical/path/program.sas|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem.  User-provided values may (or may not) be in this list.  Cannot be used alongside a HARDSELECT_HOOK.|


## Formula Rules

HARDFORMULA and SOFTFORMULA let you configure a column so that its value is automatically calculated from other columns in the same row - just like a spreadsheet formula.  When a user opens the editor, the formula is evaluated live and the result is shown in each cell.

### Writing a formula

Formulas use column names, not cell references, so there is no need to know the grid layout.  For example, if you have PRICE and VOLUME columns, a REVENUE column formula would be:

```
= PRICE * VOLUME
```

Each row calculates its own result - the PRICE in row 1 is multiplied by the VOLUME in row 1, the PRICE in row 2 by the VOLUME in row 2, and so on.

!!! note
    Each column name in the formula must be surrounded by spaces (eg ` PRICE ` not `PRICE`) so it is recognised as a column reference rather than part of a function name.  For example, `=MATCH( PRICE )` resolves the column reference, but `=MATCH(PRICE)` does not - PRICE is left unrecognised and the formula will error rather than using the column's value.  Text inside quotes is left as-is.

### HARDFORMULA vs SOFTFORMULA

* **HARDFORMULA** - the column is read-only.  The formula result is always shown and submitted.  The user cannot change it.
* **SOFTFORMULA** - the cell shows the formula result but the user can type a different value if needed.  If they do, their value is submitted instead.

### Editor behaviour

* When you paste a formula into the grid, column names are automatically translated so the formula works in its new position.
* A cell that is overwritten by a formula is flagged so you can revert it.
* Formula-looking values pasted from Excel are treated as plain data (not evaluated), unless you explicitly choose "Apply as formula".

### Special values

Formulas can reference three special values that are resolved at runtime:

* `DC.ROW_STATUS` - the current state of the row: `M` (Modified), `A` (Added), `D` (Deleted), or `U` (Unchanged).  A newly-added row is `A` from the moment it is created - there is no transient state before that.
* `DC.USER_NAME` - the logged-in user id.
* `DC.ORIG_VALUE` - the original cell value before the current edit.

Example - show the current user id if the row is changed, otherwise keep the original value:

```
= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )
```


## Regex Rules

HARDREGEX and SOFTREGEX validate cell values against a SAS (Perl-style) regular expression provided in `RULE_VALUE` - the same syntax accepted by [PRXPARSE](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lefunctionsref/p0s9ilagexmjl8n1u7e1t1jfnzlk.htm).

Things to be aware of:

* Patterns are validated with `PRXPARSE` when the rule is saved - a post-edit check on MPE_VALIDATIONS itself will reject invalid patterns and list the offending columns.
* The pattern is evaluated in the browser using the JavaScript regex engine, which shares the same core syntax (character classes, quantifiers, groups, alternation, `^`/`$` anchors, `\d \w \s` etc).  Stick to that common subset: Perl-only constructs such as inline modifiers `(?i)`, `\A` / `\z` anchors, possessive quantifiers (`a++`) and atomic groups (`(?>...)`) will pass the SAS-side PRXPARSE check but fail (and be silently ignored) in the frontend.
* The pattern is used **as authored** - it is not auto-anchored.  If you want to match the entire cell value, include `^` and `$` yourself (eg `^[A-Z]{3}$`).
* Blank values are exempt from pattern matching on any column type - use the NOTNULL rule if you also need to enforce populated values.  On numeric columns the plain SAS missing (`.`) is also exempt.  Special missings (`.A`-`.Z`, `._`) are **not** exempt - being deliberately-set values, they are validated against the pattern like any other value, so on numeric columns make sure your pattern accommodates them (or avoid special missings).  On character columns there is no missing-value concept: even `.` is treated as real text.
* Only one regex rule is ever applied per column.  If a column has both a HARDREGEX and a SOFTREGEX rule, the SOFTREGEX rule is ignored entirely - even for values that pass the HARDREGEX - so a dual-rule column behaves exactly like a HARDREGEX-only column.  For the same reason, the column-header info dropdown shows only the rule that is applied (the HARDREGEX pattern when both exist).
* Cells in rows that are marked for deletion are not validated / warned (except primary key columns, which still validate).
* In the unlikely event a pattern that fails in the browser slips through (see above), the frontend treats it as always-valid (no blocking, no warning) rather than breaking the editor.
* `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.

## Dropdowns

There are now actually FIVE places where you can configure dropdowns!

1.  The [MPE_SELECTBOX](/dcc-selectbox/) table
2.  The HARDSELECT validation (library.member.column reference)
3.  The SOFTSELECT validation (library.member.column reference)
4.  The HARDSELECT_HOOK validation (SAS Program)
5.  The SOFTSELECT_HOOK validation (SAS Program)

How do these inter-operate?

Well - if you have values in MPE_SELECTBOX and/or HARDSELECT / SOFTSELECT tables, they will be merged together, and served in ADDITION to the values provided by any HOOK program.

Dropdowns are SOFT by default, unless a HARD rule is present.

Data Controller will not let you submit both a HARDSELECT_HOOK and a SOFTSELECT_HOOK on the same variable.