docs.datacontroller.io/docs/dcc-validations.md

layout, title, description, og_image

layout	title	description	og_image
article	Data Validation	Quality in, Quality out! Enforce data quality checks at the point of SAS data entry, both directly via the web interface and also via Excel uploads.	https://i.imgur.com/P64ijBB.png

Data Controller for SAS® - DQ Validations

Overview

Quality in, Quality out! Data Controller lets you enforce quality checks at the point of data entry, both directly via the web interface and also via Excel uploads.

Default Checks

By default, the following frontend rules are always applied:

• Length checking per target table variable lengths
• Type checking per target table datatypes (Character, Numeric, Date, Time, Datetime)
• Not Null check per target table constraints
• Primary Key checking per business key defined in MPE_TABLES

It is possible to configure a number of other rules by updating the MPE_VALIDATIONS table. Simply set the BASE_LIB, BASE_DS and BASE_COL values, and ensure RULE_ACTIVE=1 for it to be applied.

Configurable Checks

Check back frequently as we keep growing this list of checks.

Rule Type	Example Value	Description
CASE	UPCASE	Will enforce the case of cell values. Valid values: UPCASE, LOWCASE, PROPCASE
NOTNULL	(defaultval)	Will prevent submission if null values are present. Optional - provide a default value.
MINVAL	1	Defines a minimum value for a numeric cell
MAXVAL	1000000	Defines a maximum value for a numeric cell
READONLY	(defaultval)	Renders the column read-only in the editor. The defaultval is used when rows are added.
HIDDEN	(defaultval)	Hides the column from the editor grid while still submitting its data. The defaultval is used when rows are added.
ROUND	2	Rounds numeric input on paste/edit. Positive digits round to number of decimal places (eg 2 rounds to 0.01) Negative digits round to the nearest ten/hundred etc. Half-away-from-zero rounding is applied so -0.5 → -1 and 2.5 → 3.
NUMBER_FORMAT	{"style":"currency","currency":"GBP"}	Display-only Intl.NumberFormat renderer. RULE_VALUE is the JSON options object passed straight to Intl.NumberFormat. Does not change the stored value.
HARDFORMULA	= PRICE * VOLUME	The cell displays a value computed from a formula, using other columns in the same row. The column is read-only - the user cannot override the result. See Formula Rules below.
SOFTFORMULA	= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )	Like HARDFORMULA, but the user can override the computed value and type their own. See Formula Rules below.
HARDREGEX	^[A-Z]{3}$	The cell value must match the regex pattern, otherwise submission is blocked and the cell is highlighted red. See Regex Rules below.
SOFTREGEX	^[A-Z]{3}$	A cell value that does not match the regex pattern is highlighted yellow as a warning, but submission is not blocked. See Regex Rules below.
HARDSELECT	sashelp.class.name	A distinct list of values (max 1000) are taken from this library.member.column reference, and the value must be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table.
SOFTSELECT	dcdemo.mpe_tables.libref	A distinct list of values (max 1000) are taken from this library.member.column reference, and the user-provided value may (or may not) be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table.
HARDSELECT_HOOK	/logical/folder/stpname	A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User provided values must be in this list. Cannot be used alongside a SOFTSELECT_HOOK.
SOFTSELECT_HOOK	/physical/path/program.sas	A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User-provided values may (or may not) be in this list. Cannot be used alongside a HARDSELECT_HOOK.

Formula Rules

HARDFORMULA and SOFTFORMULA let you configure a column so that its value is automatically calculated from other columns in the same row - just like a spreadsheet formula. When a user opens the editor, the formula is evaluated live and the result is shown in each cell.

Writing a formula

Formulas use column names, not cell references, so there is no need to know the grid layout. For example, if you have PRICE and VOLUME columns, a REVENUE column formula would be:

= PRICE * VOLUME

Each row calculates its own result - the PRICE in row 1 is multiplied by the VOLUME in row 1, the PRICE in row 2 by the VOLUME in row 2, and so on.

!!! note Each column name in the formula must be surrounded by spaces (eg PRICE not PRICE) so it is recognised as a column reference rather than part of a function name. Text inside quotes is left as-is.

HARDFORMULA vs SOFTFORMULA

• HARDFORMULA - the column is read-only. The formula result is always shown and submitted. The user cannot change it.
• SOFTFORMULA - the cell shows the formula result but the user can type a different value if needed. If they do, their value is submitted instead.

Editor behaviour

• When you paste a formula into the grid, column names are automatically translated so the formula works in its new position.
• A cell that has been overwritten by a formula is flagged so you can revert it.
• Formula-looking values pasted from Excel are treated as plain data (not evaluated), unless you explicitly choose "Apply as formula".

Special values

Formulas can reference three special values that are resolved at runtime:

• DC.ROW_STATUS - the current state of the row: M (Modified), A (Added), D (Deleted), or U (Unchanged).
• DC.USER_NAME - the logged-in user id.
• DC.ORIG_VALUE - the original cell value before the current edit.

Example - show the current user id if the row has been changed, otherwise keep the original value:

= if( DC.ROW_STATUS != 'U', DC.USER_NAME, DC.ORIG_VALUE )

Regex Rules

HARDREGEX and SOFTREGEX validate cell values against a SAS (Perl-style) regular expression provided in RULE_VALUE - the same syntax accepted by PRXPARSE.

Things to be aware of:

• Patterns are validated with PRXPARSE when the rule is saved - a post-edit check on MPE_VALIDATIONS itself will reject invalid patterns and list the offending columns.
• The pattern is evaluated in the browser using the JavaScript regex engine, which shares the same core syntax (character classes, quantifiers, groups, alternation, ^/$ anchors, \d \w \s etc). Stick to that common subset: Perl-only constructs such as inline modifiers (?i), \A / \z anchors, possessive quantifiers (a++) and atomic groups ((?>...)) will pass the SAS-side PRXPARSE check but fail (and be silently ignored) in the frontend.
• The pattern is used as authored - it is not auto-anchored. If you want to match the entire cell value, include ^ and $ yourself (eg ^[A-Z]{3}$).
• Blank values are exempt from pattern matching on any column type - use the NOTNULL rule if you also need to enforce populated values. On numeric columns the plain SAS missing (.) is also exempt. Special missings (.A-.Z, ._) are not exempt - being deliberately-set values, they are validated against the pattern like any other value, so on numeric columns make sure your pattern accommodates them (or avoid special missings). On character columns there is no missing-value concept: even . is treated as real text.
• Only one regex rule is ever applied per column. If a column has both a HARDREGEX and a SOFTREGEX rule, the SOFTREGEX rule is ignored entirely - even for values that pass the HARDREGEX - so a dual-rule column behaves exactly like a HARDREGEX-only column. For the same reason, the column-header info dropdown shows only the rule that is applied (the HARDREGEX pattern when both exist).
• Cells in rows that are marked for deletion are not validated / warned (except primary key columns, which still validate).
• In the unlikely event a pattern that fails in the browser slips through (see above), the frontend treats it as always-valid (no blocking, no warning) rather than breaking the editor.
• RULE_VALUE is limited to 128 characters, which constrains very long patterns.

Dropdowns

There are now actually FIVE places where you can configure dropdowns!

1. The MPE_SELECTBOX table
2. The HARDSELECT validation (library.member.column reference)
3. The SOFTSELECT validation (library.member.column reference)
4. The HARDSELECT_HOOK validation (SAS Program)
5. The SOFTSELECT_HOOK validation (SAS Program)

How do these inter-operate?

Well - if you have values in MPE_SELECTBOX and/or HARDSELECT / SOFTSELECT tables, they will be merged together, and served in ADDITION to the values provided by any HOOK program.

Dropdowns are SOFT by default, unless a HARD rule is present.

Data Controller will not let you submit both a HARDSELECT_HOOK and a SOFTSELECT_HOOK on the same variable.