docs.datacontroller.io/docs/dcc-validations.md

---
layout: article
title: Data Validation
description: Quality in, Quality out! Enforce data quality checks at the point of SAS data entry, both directly via the web interface and also via Excel uploads.
og_image: https://i.imgur.com/P64ijBB.png
---


# Data Controller for SAS® - DQ Validations

## Overview
Quality in, Quality out!  Data Controller lets you enforce quality checks at the point of data entry, both directly via the web interface and also via Excel uploads.

## Default Checks
By default, the following frontend rules are always applied:

* Length checking per target table variable lengths
* Type checking per target table datatypes (Character, Numeric, Date, Time, Datetime)
* Not Null check per target table constraints
* Primary Key checking per business key defined in MPE_TABLES

It is possible to configure a number of other rules by updating the MPE_VALIDATIONS table.  Simply set the `BASE_LIB`, `BASE_DS` and `BASE_COL` values, and ensure `RULE_ACTIVE=1` for it to be applied.

## Configurable Checks

Check back frequently as we keep growing this list of checks.

|Rule Type|Example Value |Description|
|---|---|---|
|CASE|UPCASE|Will enforce the case of cell values.  Valid values: UPCASE, LOWCASE, PROPCASE|
|NOTNULL|(defaultval)|Will prevent submission if null values are present.  Optional - provide a default value.|
|MINVAL|1|Defines a minimum value for a numeric cell|
|MAXVAL|1000000|Defines a maximum value for a numeric cell|
|READONLY|(defaultval) |Renders the column read-only in the editor.  The defaultval is used when rows are added. |
|HIDDEN|(defaultval) |Hides the column from the editor grid while still submitting its data. The defaultval is used when rows are added. |
|ROUND|2 |Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc.  Half-away-from-zero rounding is applied so `-0.5 → -1` and `2.5 → 3`. |
|NUMBER_FORMAT|`{"style":"currency","currency":"GBP"}` |Display-only [`Intl.NumberFormat` renderer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat).  RULE_VALUE is the JSON options object passed straight to `Intl.NumberFormat`.  Does not change the stored value. |
|HARDFORMULA|`= PRICE * VOLUME`|Evaluates a live, spreadsheet-style formula in the editor using [HyperFormula](https://hyperformula.handsontable.com/).  The column is rendered read-only and the computed value (not the formula text) is submitted on save.  See [Formula Rules](#formula-rules) below.|
|SOFTFORMULA|`= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )`|Same as HARDFORMULA but the cell remains editable - the user can override the computed value.  See [Formula Rules](#formula-rules) below.|
|HARDREGEX|`^[A-Z]{3}$`|The cell value **must** match the regex pattern, otherwise submission is blocked and the cell is highlighted red.  See [Regex Rules](#regex-rules) below.|
|SOFTREGEX|`^[A-Z]{3}$`|A cell value that does not match the regex pattern is highlighted yellow as a warning, but submission is **not** blocked.  See [Regex Rules](#regex-rules) below.|
|HARDSELECT|sashelp.class.name|A distinct list of values (max 1000) are taken from this library.member.column reference, and the value **must** be in this list.  This list may be supplemented by entries in the MPE_SELECTBOX table.|
|SOFTSELECT|dcdemo.mpe_tables.libref|A distinct list of values (max 1000) are taken from this library.member.column reference, and the user-provided value may (or may not) be in this list.  This list may be supplemented by entries in the MPE_SELECTBOX table.|
|[HARDSELECT_HOOK](/dynamic-cell-dropdown)|/logical/folder/stpname|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem.  User provided values **must** be in this list.  Cannot be used alongside a SOFTSELECT_HOOK.|
|[SOFTSELECT_HOOK](/dynamic-cell-dropdown)|/physical/path/program.sas|A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem.  User-provided values may (or may not) be in this list.  Cannot be used alongside a HARDSELECT_HOOK.|


## Formula Rules

HARDFORMULA and SOFTFORMULA bring live, spreadsheet-style formula evaluation to the Data Editor using the [HyperFormula](https://hyperformula.handsontable.com/) engine (the same library that powers formula calculation in Handsontable).  When a column has either rule, the editor evaluates the formula in real time and displays the computed value in each cell.

### How formulas are written

Formulas are written using column names rather than cell references, so end users do not need to know the grid layout:

|ITEM|PRICE|VOLUME|REVENUE|
|---|---|---|---|
|PAPER|4.20|100|`= PRICE * VOLUME`|
|PEN|61.02|1,971|`= PRICE * VOLUME`|

At runtime the editor translates each column name to the correct cell reference for the current row (eg `PRICE` becomes `B1` in the first row, `B2` in the second, and so on).

!!! note
    Each named variable **must have a leading and trailing blank** so it does not clash with function names (eg `MATCH()`).  Variables inside single or double quoted strings are left untouched - eg `= "hello " & ITEM & " world"` resolves `ITEM` but not the word `ITEM` inside the string literal.

### HARDFORMULA vs SOFTFORMULA

* **HARDFORMULA** - the column is read-only.  The formula is always evaluated and the computed value is displayed and submitted.  The user cannot override it.
* **SOFTFORMULA** - the cell starts with the computed value but remains editable.  The user can type a different value, which is then submitted instead of the formula result.

### Formula-aware behaviour across the editor

Live formulas are evaluated for any character column, not just those with explicit HARDFORMULA / SOFTFORMULA rules.  Several editor behaviours have been extended to be formula-aware:

* When you paste a formula, column names are automatically translated to the correct cell references.
* A cell overwritten by a formula is flagged so you can revert it; cell revert has been generalised to any overwritten value, not just formulas.
* The primary key and numeric columns resolve a live formula to its computed value (not the raw formula text) when submitted.
* Formula-looking values pasted from Excel are auto-escaped so they are treated as data, not evaluated as formulas, unless you explicitly choose "Apply as formula".
* A dedicated `DC.ROW_STATUS` column is driven live by formulas, and has been hardened against collisions with real columns of that name.

### Special formula literals

The following literals are available for use in formula rules:

* `DC.ROW_STATUS` - replaced at runtime with the cell reference (eg `A1`).  Values are `M` (Modified), `A` (Added), `D` (Deleted), or `U` (Unchanged).
* `DC.USER_NAME` - replaced at runtime with the logged-in user id.
* `DC.ORIG_VALUE` - replaced at runtime with the original cell value before the current edit.

Example - show the current user id if the row has been changed, otherwise keep the original value:

```
RULE_VALUE = if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )
```


## Regex Rules

HARDREGEX and SOFTREGEX validate cell values against a SAS (Perl-style) regular expression provided in `RULE_VALUE` - the same syntax accepted by [PRXPARSE](https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lefunctionsref/p0s9ilagexmjl8n1u7e1t1jfnzlk.htm).

Things to be aware of:

* Patterns are validated with `PRXPARSE` when the rule is saved - a post-edit check on MPE_VALIDATIONS itself will reject invalid patterns and list the offending columns.
* The pattern is evaluated in the browser using the JavaScript regex engine, which shares the same core syntax (character classes, quantifiers, groups, alternation, `^`/`$` anchors, `\d \w \s` etc).  Stick to that common subset: Perl-only constructs such as inline modifiers `(?i)`, `\A` / `\z` anchors, possessive quantifiers (`a++`) and atomic groups (`(?>...)`) will pass the SAS-side PRXPARSE check but fail (and be silently ignored) in the frontend.
* The pattern is used **as authored** - it is not auto-anchored.  If you want to match the entire cell value, include `^` and `$` yourself (eg `^[A-Z]{3}$`).
* Blank values are exempt from pattern matching on any column type - use the NOTNULL rule if you also need to enforce populated values.  On numeric columns the plain SAS missing (`.`) is also exempt.  Special missings (`.A`-`.Z`, `._`) are **not** exempt - being deliberately-set values, they are validated against the pattern like any other value, so on numeric columns make sure your pattern accommodates them (or avoid special missings).  On character columns there is no missing-value concept: even `.` is treated as real text.
* Only one regex rule is ever applied per column.  If a column has both a HARDREGEX and a SOFTREGEX rule, the SOFTREGEX rule is ignored entirely - even for values that pass the HARDREGEX - so a dual-rule column behaves exactly like a HARDREGEX-only column.  For the same reason, the column-header info dropdown shows only the rule that is applied (the HARDREGEX pattern when both exist).
* Cells in rows that are marked for deletion are not validated / warned (except primary key columns, which still validate).
* In the unlikely event a pattern that fails in the browser slips through (see above), the frontend treats it as always-valid (no blocking, no warning) rather than breaking the editor.
* `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.

## Dropdowns

There are now actually FIVE places where you can configure dropdowns!

1.  The [MPE_SELECTBOX](/dcc-selectbox/) table
2.  The HARDSELECT validation (library.member.column reference)
3.  The SOFTSELECT validation (library.member.column reference)
4.  The HARDSELECT_HOOK validation (SAS Program)
5.  The SOFTSELECT_HOOK validation (SAS Program)

How do these inter-operate?

Well - if you have values in MPE_SELECTBOX and/or HARDSELECT / SOFTSELECT tables, they will be merged together, and served in ADDITION to the values provided by any HOOK program.

Dropdowns are SOFT by default, unless a HARD rule is present.

Data Controller will not let you submit both a HARDSELECT_HOOK and a SOFTSELECT_HOOK on the same variable.