# [7.12.0](https://git.datacontroller.io/dc/dc/compare/v7.11.0...v7.12.0) (2026-07-28)
### Bug Fixes
* adding REGEX validations to mpe_x_test ([bd798b4](bd798b424a))
* default value for label ([a3e46a9](a3e46a968e))
* **deps:** retarget Angular upgrade to 20, not 21 (CI install was broken) ([cac9244](cac9244f92))
* ensure only one REGEX applies at a time ([8fb58eb](8fb58eb36e))
* ensure that no assets (including og links) ever fetch from external sources ([33dcb98](33dcb989d3))
* include peer dependencies in package-lock for npm ci in pipeline ([b51c770](b51c770782))
* licensecheker ([f60bcef](f60bcef583))
* **lint:** remove redundant optional chaining ([d881290](d881290618))
* optimisation, renamed values for DDTYPE to save space ([cfb60e5](cfb60e5e4b))
* patch npm audit vulnerabilities in sas and client dependencies ([e22edf7](e22edf7ed3))
* **query:** isolate viewbox filter state from the base table's ([7a35cf4](7a35cf4a45))
* regenerate client lockfile to resolve Angular peer-dependency drift breaking npm ci ([05fe474](05fe4744d5))
* **regex:** special missing handling ([180c247](180c2477ed))
* removing low severity warning in npm audit ([2a771bb](2a771bb91a))
* removing thousand seperator from plain numerics in EDIT mode ([0392a81](0392a81cbd))
* **validations:** parse SAS PRX /pattern/flags syntax in HARDREGEX/SOFTREGEX ([7ed3730](7ed3730ae3))
### Features
* **docs:** adding agents.md and docs for RLS ([359d833](359d833406))
* **editor:** add HARDREGEX/SOFTREGEX validation rules ([17e4802](17e4802895))
* **editor:** evaluate HARDREGEX/SOFTREGEX independently instead of hard-wins precedence ([57db117](57db1179a9))
* **editor:** show applied HARDREGEX/SOFTREGEX pattern in column info dropdown ([39c8855](39c8855f37))
* **regex:** backend validations on regex strings ([d2c93a4](d2c93a46fa))
* using ALL libraries as validation in MPE_SECURITY. Closes [#279](#279) ([62ff0ae](62ff0aee4a))
* validation checks to prevent incompatible RLS rules (eg REPLACE load type). Closes [#211](#211) ([ea00c5a](ea00c5afad))
* validation on RLS for REPLACE, + docs + tests. Closes [#211](#211) ([7378f3b](7378f3ba30))
Both rules can now apply to one column: HARDREGEX still blocks submission
and takes its own tooltip, but SOFTREGEX is evaluated (and shown) whenever
HARDREGEX passes, instead of being silently suppressed whenever HARDREGEX
was merely present. Column-header info dropdown labels each rule separately
when a column has both.
Replace one-shot .then()/for-loop matching with retrying .should()/cy.contains()
lookups, drop stale cached DOM node references (viyaLib) in favor of live re-queries.
Verifies parseRegexRule against a corpus of RULE_VALUEs collected from an
existing MPE_VALIDATIONS table - none use PRX-only syntax, so behavior is
confirmed to match SAS PRX exactly, not just "doesn't throw". Pins two
data quirks found along the way (a range vs. literal-hyphen character
class, and a double-escaped lookahead that's a no-op in both engines)
rather than silently treating either as a bug to fix.
RULE_VALUE is authored in PRX delimiter form because prxparse() requires
it, but HARDREGEX, the SOFTREGEX grid renderer, and failsSoftRegex were
all passing that string straight into `new RegExp()`, so the delimiters
and flags were matched as literal characters instead of applied - making
these rules silently never match real data.
Adds parseRegexRule (extracted, tested independently) to strip the
delimiters, apply flags, hoist a leading (?i) modifier, and translate
\Q...\E and \A/\z to their JS equivalents. Atomic groups and possessive
quantifiers are left unfixed (documented, fail-safe) - translating them
risks renumbering the pattern's own capture groups.
Also switches the REGEX_HARD_COL/REGEX_SOFT_COL mock rules to the
delimited form so editor.cy.ts's existing e2e coverage actually exercises
this path.
Bumps @angular/* to 21 (passing through 20 as a transient step, since
vendored Clarity has no Angular-20-compatible release) and
@typescript-eslint to 8.65.0 for TS 5.9 compat. Applies the required
ng update migrations, including *ngIf/*ngFor/*ngSwitch -> control-flow
syntax across 37 templates.
Two new DQ rule types apply regular expressions to cell values:
HARDREGEX blocks submission on a non-matching value (same path as the
existing CASE/MINVAL/MAXVAL rules); SOFTREGEX is display-only — a
non-matching value gets a yellow warning cell but can still submit,
so it's wired as a grid renderer rather than a validator, and mirrored
in the edit-record modal (which has no grid renderer to hook into) via
DcValidator.failsSoftRegex. Both rules exempt blank and SAS special
missing values, and fail open on a malformed pattern rather than
blocking every submission on that column. HARDREGEX takes precedence
when both rules apply to the same column, so a failing value renders
red/blocked, never yellow.
An invalid cell no longer blocks submission if its row is marked for
delete, since those values are about to be removed anyway. Primary key
columns are exempt from the exemption: the key identifies which record
to delete, and a blank/invalid PK would otherwise also slip past the
separate duplicate-key check. Toggling the delete flag now re-validates
the row immediately so invalid highlights update without waiting for
submit.
Header matching in searchDataInExcel() was case-insensitive but not
whitespace-tolerant, so a header like " SOME_CHAR" (an easy defect to
pick up via copy-paste from another spreadsheet/system) was reported as
a missing column and aborted the whole upload.
- trim() the cell value before matching, alongside the existing
toLowerCase() normalization
- add spreadsheet-util.spec.ts (no prior spec file existed) covering
the trimmed match and a regression guard that a genuinely different
header is still correctly reported missing
- add a Cypress case + fixture in excel.cy.ts covering the same defect
end to end
getdata.sas now sends one COLTYPE JSON-object string per cols[] row (via
a LEFT JOIN on the real dataset's columns) instead of one comma-joined
sasparams.COLTYPE string, avoiding the 32K cats() buffer overflow on
wide tables (#253). Adapt the client accordingly:
- parseColTypeRow replaces parseColType; DcValidator builds validation
rules per cols[] entry instead of from sasparams.COLTYPE
- _____DELETE__THIS__RECORD_____'s rule is never present in cols[] (it's
a client-only synthetic column %mp_getcols doesn't know about), so
it's now hardcoded via deleteRecordColumnRule.ts
- cols[] isn't guaranteed to arrive in COLHEADERS/VARNUM order (backend
can serialize it alphabetically by NAME) but editor.component.ts pairs
rules with headers positionally, so DcValidator now sorts cols by
VARNUM before building rules
- update sas/mocks/sasjs getdata.js to the new per-column COLTYPE shape
- add Cypress coverage for the delete-record dropdown and an ordinary
dropdown column end to end
Brings in the SAS-side fixes for issue #253 (getdata.sas passing
variable formats via a vertical cols.COLTYPE array instead of a
single concatenated sasparams.COLTYPE string, which overflowed SAS's
32K buffer on wide datasets) plus related label/datadictionary fixes.
No client TypeScript files are touched by this merge.
Adds a URL-driven toggle (?labels=true, mirroring embed=va) that
swaps grid column headers between NAME and LABEL in both the Viewer
and Editor, with a "Show labels"/"Show names" item in each grid's
right-click context menu. The header-info popup now shows NAME first
regardless of display mode. Sorting/filtering/data binding stay
NAME-based throughout — only the displayed header text changes.
New shared, unit-tested utils (parse-labels-param, display-col-headers,
col-info-html) back both pages. Mock data (viewdata.js, getdata.js)
gains a couple of columns with LABEL != NAME plus a blank-LABEL case,
covered by a new viewer-labels.cy.ts e2e spec now wired into CI.
Also fixes a pre-existing gap in cypress/support/commands.ts: the
isLoggedIn/loginAndUpdateValidKey custom commands had no Cypress
Chainable type augmentation, and moment was imported as a namespace
import despite its `export =` typing making it uncallable that way.
Bump handsontable and @handsontable/angular-wrapper 17.1 -> 18.0 and
adapt to its stricter TypeScript-core types: moved handsontable/common
and deep walkontable imports to their new public export paths, added
explicit generics/casts where getData()/getCellMeta()/getSelected() are
now typed unknown/nullable instead of the loose v17 shapes, and updated
the AutocompleteEditor override to the new open() signature.
Also allowlists @handsontable/angular-wrapper@18.0.0 and
handsontable@18.0.0 in licenseChecker.js — both versions report the
same non-SPDX "SEE LICENSE IN LICENSE.txt" field that license-checker
mis-resolves as a disallowed custom license, same as prior versions.
BREAKING CHANGE: Introduction of 4 new tables for capturing information related to catalogs and their objects. Migration script prepared and available in the DB folder (usual place)
Whenever any `package.json` (root, `client/`, or `sas/`) or lockfile is modified, run the same checks CI runs before pushing:
1.**npm audit** (must be clean for prod deps):
```bash
npm audit --omit=dev # in repo root
cd sas && npm audit --omit=dev
cd ../client && npm audit --omit=dev
```
Fix with `npm audit fix`, targeted `overrides` in `package.json`, or version bumps — never `npm audit fix --force` blindly, as it can introduce breaking changes.
2. **License checker** (client only):
```bash
cd client && npm run license-checker
```
If a new dependency fails, either add its SPDX id to the `onlyAllow` list in `client/licenseChecker.js` (if the license is acceptable, e.g. permissive ones like `BlueOak-1.0.0`) or add the specific package to `excludePackages` with justification. Data Controller ships on-prem, so only OSI-approved permissive licenses are acceptable for production dependencies.
Both checks run in `.gitea/workflows/build.yaml` (`Check audit` and `Licence checker` steps) and will fail the build if skipped locally.
This document describes how the regex validation rules work internally. For user-facing documentation see `docs/dcc-validations.md` in the `docs.datacontroller.io` repo.
## Overview
Two validation rule types in `MPE_VALIDATIONS` validate cell values against a regular expression supplied in `RULE_VALUE`:
-`HARDREGEX` - submission-blocking. A failing value is rejected by the cell validator and painted red (HOT's own `htInvalid` class), so the row cannot be submitted.
-`SOFTREGEX` - display-only warning. A failing value is painted yellow (`dc-warning-cell` class) with a tooltip, but submission is not blocked.
Both rule types are selectable in the MPE_VALIDATIONS RULE_TYPE dropdown; they were added to the selectbox seed data in `sas/sasjs/macros/mpe_makedata.sas` and via the optional migration `sas/sasjs/db/migrations/20260720_v7.12_release.sas`. `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.
## Config-time validation (SAS side)
`sas/sasjs/services/hooks/mpe_validations_postedit.sas` runs `prxparse()` on any staged HARDREGEX/SOFTREGEX rule value and aborts the edit with a list of offending `libref.table.column` references if the pattern is invalid. This is a best-effort syntax check to catch typos at config time; an empty pattern is treated as valid (it matches everything in JS). Rows marked for delete are skipped.
Because patterns must pass `prxparse`, rule values are authored in the SAS PRX delimiter form `/pattern/flags` (although a bare pattern is also accepted for backwards compatibility).
## Frontend evaluation
All regex handling lives in the client; there is no server-side re-validation of data values. The per-cell decision flow:
```mermaid
flowchart TD
A[Cell value] --> B{Row marked for delete<br/>and not a PK column?}
B -- Yes --> Z[No validation / no warning]
B -- No --> C{isRegexRuleExempt?<br/>blank, or "." on a numeric column}
C -- Yes --> Z
C -- No --> D{HARDREGEX rule on column?}
D -- Yes --> E{Pattern matches?}
E -- No --> F[Invalid: submission blocked,<br/>red htInvalid + REGEX tooltip]
E -- Yes --> J[Valid]
D -- No --> G{SOFTREGEX rule on column?}
G -- Yes --> H{Pattern matches?}
H -- No --> I[Warning: yellow dc-warning-cell<br/>+ REGEX tooltip, submission allowed]
H -- Yes --> J[Valid]
G -- No --> J
```
A malformed pattern never reaches this flow: it is treated as always-valid (HARDREGEX) / never-warn (SOFTREGEX) with a `console.warn`, rather than breaking the editor.
Converts an authored SAS PRX pattern into a JavaScript `RegExp`:
1. If the value matches `/^\/(.*)\/([a-z]*)$/s`, the delimiters are stripped and the trailing flags are passed to `new RegExp(body, flags)`. Otherwise the value is used as-is (backwards compatibility with bare patterns).
2. Three mechanical Perl→JS translations are applied to the body:
- a leading `(?i)` inline modifier is removed and folded into the `i` flag;
-`\Q...\E` literal sequences are replaced with escaped literal text;
-`\A` → `^` and `\z` → `(?![\s\S])` (end-of-string anchor).
3. Perl-only constructs that would need a capture-group-renumbering rewrite (atomic groups `(?>...)`, possessive quantifiers `a++`) are deliberately NOT translated. They throw from `new RegExp`, and every caller treats a throw as "always valid / never warn" (see below) rather than breaking the editor.
Blank values (`undefined`, `null`, `''`) are exempt from pattern matching on any column type - use a separate NOTNULL rule if populated values must also be enforced. On numeric columns the plain SAS missing (`.`) is also exempt; special missings (`.A`-`.Z`, `._`, bare letters) are NOT exempt anywhere - being deliberately set, they are real values the pattern must match (and on a character column even `.` is real text). `isSpecialMissing` from `@sasjs/utils` is deliberately not used: its optional-dot regex would exempt any single-letter character value ("d", "z") before the regex ever ran. The check takes an `isNumeric` flag, passed by all callers (the dq validator via `dqValidate(rules, value, colType === 'numeric')`, the warning renderer via a `makeRegexWarningRenderer` argument, and `failsSoftRegex` via the column's HOT type).
### HARDREGEX - blocking validation
`HARDREGEX` is implemented as a cell validator in `client/src/app/shared/dc-validator/validations/dq-validation.ts`. It returns `true` (valid) for exempt values and for patterns that fail to compile (with a `console.warn`), and otherwise returns `parseRegexRule(ruleValue).test(value.toString())`. Returning `false` makes HOT mark the cell invalid, block submission, and paint it red via its standard `htInvalid` styling.
### SOFTREGEX - warning renderer
`client/src/app/editor/utils/regex-warning-renderer.ts` builds a display-only Handsontable renderer (registered per-column by `DcValidator.setupRules` in `client/src/app/shared/dc-validator/dc-validator.ts`). It never returns false; it only:
- adds a `REGEX: <pattern>` tooltip (`td.title`) when a rule fails;
- adds the yellow `dc-warning-cell` class when only SOFTREGEX fails.
`DcValidator.failsSoftRegex(col, value)` provides the same logic outside the grid (e.g. the edit-record screen).
### Precedence: HARD and SOFT on the same column
Only one regex ever runs per column. If a HARDREGEX rule exists, SOFTREGEX is ignored entirely - never compiled, never evaluated - regardless of whether individual cell values pass or fail the hard rule. A value failing HARDREGEX gets the red invalid styling (blocking submission) plus a `REGEX: <pattern>` tooltip; a SOFTREGEX-only column warns in yellow without blocking. This holds in both the renderer and `failsSoftRegex`.
### Other behaviour
- Rows marked for delete (`_____DELETE__THIS__RECORD_____ = 'Yes'`) are not warned/validated by the renderer (except primary key columns, which still validate).
- A malformed pattern never breaks the editor: the dq validator treats it as always-valid and the renderer disables the warning, logging to the console instead.
- The pattern is used as authored - it is NOT auto-anchored. Authors must include `^`/`$` to match the entire cell value.
- Column info: `client/src/app/shared/utils/col-info-html.ts` shows the applied pattern in the column-info dropdown - the HARDREGEX pattern if one exists (it is the rule actually applied when both are present), otherwise the SOFTREGEX pattern, otherwise nothing.
- SAS side: `sas/sasjs/services/editors/stagedata.test.3.sas`, plus seed data in `mpe_makedata.sas`: HARDREGEX "SOME_CHAR must contain 'the' or 'data'" (`/the|data/i`), SOFTREGEX "SOME_CHAR should contain the letter 't'", SOFTREGEX on PRIMARY_KEY_FIELD (`/^\d+$/` - yellow if the key contains a decimal), and HARDREGEX on SOME_SHORTNUM (`/^(??$).*/` - values 1-5 blocked; generated data starts at 6 so demos aren't blocked accidentally).
This document explains **how** Row Level Security (RLS) is implemented in the Data Controller backend. For the user-facing configuration guide, see [docs.datacontroller.io/row-level-security](https://docs.datacontroller.io/row-level-security/).
## Overview
RLS in Data Controller is implemented as **server-side WHERE clause generation**. No data leaves SAS without passing through a dynamically generated filter. The filter is built at runtime per request, based on:
1. The requesting user's group memberships (SAS metadata groups + DC groups)
2. The active rules in the `MPE_ROW_LEVEL_SECURITY` control table
Because the filter is expressed as a standard SAS `WHERE` expression, it works against **any** engine — Base SAS datasets, database libraries (via implicit SQL pass-through pushdown), SPDE, CAS libnames, etc.
Scope mapping: `DLOAD` requests match rules with `RLS_SCOPE in ('VIEW','ALL')`; `ULOAD` requests match rules with `RLS_SCOPE in ('EDIT','ALL')`.
## Execution Flow of `%mpe_filtermaster`
The macro writes the final WHERE expression to a temporary **fileref** (`outref`), line by line. (A fileref is used because a generated filter may exceed the 64k macro variable limit — and note that `%include` of a fileref is not allowed directly in a `proc sql` where clause, hence callers typically use it in a DATA step or data step view.)
### 1. User-supplied filter (FILTER_RK)
If the request includes a stored filter (`filter_rk > 0`), its clauses are read from `MPE_FILTERANYTABLE` / `MPE_FILTERSOURCE` and generated first via `%mp_filtergenerate`. RLS clauses are then **AND-ed on top** — a user filter can only ever narrow results, never widen them beyond RLS.
`MPE_TABLES` may define `VAR_TXFROM` / `VAR_TXTO` (SCD2-style validity variables) for the target table. Unless the user explicitly filtered on those variables, the macro appends:
```
("<current datetime>"dt < VAR_TXTO)
```
so that only current records are surfaced in the EDIT grid and downloads.
where&dc_dttmtfmt. lt tx_to /* only current (non-deleted) rules */
and rls_scope in ("&scopeval",'ALL')
andupcase(rls_group) in (selectupcase(groupname) from work.groups)
and rls_libref = "<libref>"and rls_table = "<dsname>"
and rls_active = 1
orderby rls_group, rls_subgroup_id;
```
### 5. Clause assembly
If rules exist, they are appended to the fileref as `AND ( ... )`. Each **group** the user belongs to contributes one sub-filter, and the group-level sub-filters are joined with `OR`:
```
AND ( <group 1 filter> OR <group 2 filter> ... )
```
Each group's filter is produced by `%mp_filtergenerate`, which honours:
*`RLS_GROUP_LOGIC` — how subgroups (identified by `RLS_SUBGROUP_ID`) are joined
*`RLS_SUBGROUP_LOGIC` — how individual clauses within a subgroup are joined (AND/OR)
So membership in multiple groups is **permissive** (OR): the user sees the union of rows permitted by each of their groups.
### 6. Empty-filter fallback
If nothing was written to the fileref (no stored filter, no validity vars, no matching RLS rules), the macro writes a literal `1=1` so the fileref can always be consumed as a valid WHERE expression. Absence of rules therefore means **no restriction** — RLS is opt-in per table/group.
The fileref is `%include`d directly inside the `where` statement — the filter never passes through client-visible state, so it cannot be tampered with. For database libraries the WHERE expression is pushed down to the database by the SAS engine.
### Write path (stagedata.sas) — inverse filter
Uploads cannot be filtered; instead the filter is **inverted** and any submitted row matching the inverse is rejected:
,msg=%str(Security Problem - N unauthorised records submitted))
```
If even one submitted row falls outside the user's permitted row set, the entire staging request is aborted before any approval/apply step.
## Incompatibility with REPLACE Load Type
RLS with `EDIT` scope is **incompatible** with tables configured with `LOAD_TYPE=REPLACE` in `MPE_TABLES`. A REPLACE load wipes and reloads the entire target table, so row-level filtering of submitted records cannot be enforced meaningfully (the rows a user is *not* allowed to see would also be deleted). Backend validations therefore abort in both directions (see [issue #211](https://git.datacontroller.io/dc/dc/issues/211)):
1.**`mpe_row_level_security_postedit.sas`** — aborts when activating a rule with `RLS_SCOPE in ('EDIT','ALL')` against a table whose current `MPE_TABLES` record has `LOADTYPE='REPLACE'`.
2.**`mpe_tables_postedit.sas`** — aborts when setting `LOADTYPE='REPLACE'` on a table that already has active, current `RLS_SCOPE in ('EDIT','ALL')` rules in `MPE_ROW_LEVEL_SECURITY`.
`VIEW`-scope rules remain compatible with REPLACE loads, since they only affect read paths.
## Edit-Time Validation (Injection Defence)
Because `RLS_RAW_VALUE` is free text that ends up inside a generated WHERE clause, it is a potential SAS code-injection vector. Mitigations:
1.**`mpe_row_level_security_postedit.sas`** — a post-edit hook on the `MPE_ROW_LEVEL_SECURITY` table itself. Every newly staged rule (with `rls_active=1`) is grouped by target `libref.table` and run through `%mp_filtercheck(targetds=..., abort=YES)`, which compiles/tests each clause against the real target table and rejects the whole submission on invalid syntax (see [mp_filtercheck](https://core.sasjs.io/mp__filtercheck_8sas.html)).
2.**Format rules** enforced by validation: character values must be single quoted, `IN`/`NOT IN` values must be bracketed, `BETWEEN` must contain `AND`.
Additionally, editing `MPE_ROW_LEVEL_SECURITY` is itself a Data Controller table edit, so it goes through the normal approval workflow, audit trail (`MPE_AUDIT`), and can be column-restricted via [Column Level Security](https://docs.datacontroller.io/column-level-security/).
## Worked Example
Given these active rules (all for `MYLIB.MYDS`, `RLS_ACTIVE=1`):
| SCOPE | GROUP | GROUP_LOGIC | SUBGRP_LOGIC | SUBGRP_ID | VAR | OP | VALUE |
|---|---|---|---|---|---|---|---|
| ALL | Group 1 | AND | AND | 1 | VAR_2 | IN | ('this','or') |
| ALL | Group 1 | AND | AND | 1 | VAR_3 | < | 42 |
| ALL | Group 2 | AND | AND | 1 | VAR_4 | CONTAINS | 'xyz' |
A non-admin user in **both** groups, opening the VIEW page, gets a fileref containing:
```
AND ( (VAR_2 IN ('this','or') AND VAR_3 < 42) OR (VAR_4 CONTAINS 'xyz') )
```
resulting in:
```sas
data work.viewdata;
set mylib.myds;
where (VAR_2 IN ('this','or') AND VAR_3 < 42) OR (VAR_4 CONTAINS 'xyz');
run;
```
The same user submitting an EDIT upload has the inverse applied to their staged rows; any row not matching the expression above aborts the submission.
## Summary of Security Properties
* **Enforced server-side** in every read service (VIEW/EDIT/DLOAD) and every write service (ULOAD via inverse filter).
* **Admin bypass** is explicit (`&mpeadmins` group membership check).
* **Multi-group semantics are OR** (union of permitted rows); rules within a group/subgroup are AND/OR per configuration.
* **Fail-open by design**: if no rules match a table, it is unrestricted — RLS must be opted into per table.
* **Injection-resistant**: values are validated with `%mp_filtercheck` at the time rules are edited, not at query time.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
## Commands
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
## Results
- Test results: `sas/sasjsresults/testResults.json` / `.csv` / `.xml`
- Individual test logs: `sas/sasjsresults/logs/<testsuite>.log` (e.g. `services_editors_stagedata.test.3.log`)
- Coverage: `sas/sasjsresults/coverage.lcov`
## Notes
- Test source files live in `sas/sasjs/**`; `sas/sasjsbuild/` is generated build output — do not hand-edit it.
-`sasjs test` executes tests against the deployed app, so run `npm run 4gl` first after changing any service/hook/test code.
- Assertions are made with `%mp_assert()`; results are written to `work.test_results`.
The Angular wrapper was modernized in Handsontable v17.1 to align with Angular 17–19, simplifying setup and reducing dependencies. Earlier Angular versions are no longer the target — upgrade Angular first if you're below v17.
HyperFormula version compatibility table: https://handsontable.com/docs/react-data-grid/formula-calculation/#hyperformula-version-support
---
## Server-side data with DataProvider (v17.1+)
The **DataProvider plugin** (new in v17.1) wires the grid up to a remote data source so rows are fetched, sorted, and mutated server-side instead of held in memory. Use it for datasets too large to load up front, or when the source of truth lives in a backend.
The `dataProvider` option is an object with five required keys: a row-id resolver, a paginated `fetchRows` callback, and three mutation callbacks. For a read-only grid, stub the mutation callbacks with `async () => {}` — they must still be present.
```jsx
<HotTable
colHeaders={['ID','Name','Email']}
pagination={{pageSize:25}}
columnSorting={true}
emptyDataState={true}
height={360}
licenseKey="non-commercial-and-evaluation"
dataProvider={{
// Required: how to extract a stable id from each row
rowId:'id',
// Required: paginated fetch. `sort` is null when no column is sorted.
// Second arg carries an AbortSignal — pass it to fetch() so the plugin
// can cancel superseded requests when the user pages/sorts quickly.
// Required. Stub with async () => {} if your grid is read-only.
// The plugin auto-refetches the current page after each callback resolves.
// See the plugin API ref for the exact payload shapes.
onRowsCreate:async()=>{},
onRowsUpdate:async()=>{},
onRowsRemove:async()=>{},
}}
/>
```
### Loading and error UI
The plugin fires three hooks you can wire up as `<HotTable>` props for loading indicators and error surfaces:
-`beforeDataProviderFetch(params)` — fires before each fetch. `params.skipLoading` is set when the plugin wants to suppress your loading indicator (e.g., during a quick refetch).
-`afterDataProviderFetch(result)` — fires after a successful fetch.
-`afterDataProviderFetchError(error)` — fires when `fetchRows` throws or returns a rejected promise.
### Companion options
The plugin is built to pair with `pagination` (paginates server-side), `columnSorting` (single-column server-side sort), and `emptyDataState` (loading + empty state UI). Enable all three when you use DataProvider.
The **Notification plugin** (new in v17.1) shows non-blocking toast notifications anchored to the grid — useful for confirming saves, surfacing validation errors, or signaling background sync state. Enable with `notifications: true` and trigger via `hot.getPlugin('notifications').showMessage(...)`. See the plugin guide for placement, severity levels, and auto-dismiss timing: https://handsontable.com/docs/react-data-grid/notification/
---
## Events & Hooks
Handsontable hooks are passed as props on `<HotTable>`:
```jsx
<HotTable
afterChange={(changes,source)=>{
if(source!=='loadData'){
console.log('Cell changed:',changes);
}
}}
beforeChange={(changes,source)=>{
// Return false to cancel the edit
}}
afterSelection={(row,col,row2,col2)=>{
console.log('Selected:',row,col,'to',row2,col2);
}}
/>
```
There are 100+ hooks available. Full reference: https://handsontable.com/docs/react-data-grid/api/hooks/
Handsontable v17 introduced a Recipes section in the docs — ready-made patterns for common use cases
(data validation workflows, dynamic column generation, etc.). Check the recipes index before building
something from scratch: https://handsontable.com/docs/react-data-grid/recipes/
---
## Common Pitfalls
- **Forgetting `height`**: The grid won't render without a `height` prop. Use `"auto"`, a pixel value, or a CSS string.
- **Not filtering `loadData` in `afterChange`**: The `afterChange` hook fires on initial data load with `source === 'loadData'`. Always check the source to avoid infinite loops when syncing changes back to state.
- **Using the old wrapper packages**: v17 removed `@handsontable/react` and `@handsontable/angular`. Use `@handsontable/react-wrapper` and `@handsontable/angular-wrapper`.
- **Using legacy CSS imports**: `handsontable.full.min.css` was removed in v17. Use `handsontable/styles/handsontable.min.css` plus a theme file.
- **Formulas with nested object data**: HyperFormula formulas don't work when `data` is an array of nested objects — use flat objects or arrays of arrays.
- **ExportFile `columnHeaders` renamed**: In v17.1 the ExportFile plugin's `columnHeaders` option was renamed to `colHeaders` to match the table-level option. Update any `exportAsString` / `exportAsBlob` / `downloadFile` calls that pass `columnHeaders: ...`.
---
## Version Awareness
Handsontable docs are versioned. The latest docs live at `/docs/react-data-grid/` (which redirects
to the current version). To link to a specific version, use `/docs/17.0/react-data-grid/`.
When helping a user, check which version they are on — breaking changes between major versions are
common. Point them to the relevant migration guide if they're upgrading.
For the full organized directory of documentation links, read `references/docs-map.md` in this
An agent skill for [**HyperFormula**](https://hyperformula.handsontable.com) — the headless, open-source TypeScript spreadsheet calculation engine. It gives any AI coding agent deep, task-oriented knowledge of HyperFormula so the agent can help you integrate, configure, and debug the engine faster.
This package is the npm distribution of the `hyperformula` skill maintained in [`handsontable/handsontable-skills`](https://github.com/handsontable/handsontable-skills). The package version tracks the HyperFormula product version the skill targets (e.g. `3.3.0` targets HyperFormula 3.3.0).
## What's in the package
A self-contained skill — plain markdown, no runtime dependencies:
```
SKILL.md ← entry point: concepts, routing, and a docs map (YAML frontmatter + body)
references/ ← task-oriented deep-dives the agent reads on demand
```
`SKILL.md` follows the [Agent Skills](https://claude.com/blog/skills) layout (a YAML frontmatter block with `name`/`description`, followed by markdown instructions). Any agent that can load a skill — or simply read a folder of markdown instructions — can use it.
## Install
```bash
npm install @handsontable/hyperformula-skill
```
The installed package lives at `node_modules/@handsontable/hyperformula-skill`. Resolve that path programmatically with:
Upload the package contents (`SKILL.md` + `references/`) to the [Skills API](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview).
### OpenAI Codex
Codex discovers [skills](https://developers.openai.com/codex/skills) as folders containing a `SKILL.md`, under `~/.agents/skills` (available in every repo) or `.agents/skills` in a repo (checked in for your team). Copy the package into one of those locations:
If your agent reads a single instruction file rather than a folder, point it at `SKILL.md`; it links the files under `references/` so the agent can pull them in as needed. For RAG / vector-store setups, ingest `SKILL.md` together with every file under `references/` (the repository also publishes a pre-flattened `hyperformula-rag.md` for this case).
## Other distribution formats
The same skill source is also shipped as a drag-and-drop `.zip` (Cowork / Claude.ai web), a flattened `-rag.md` doc (RAG / vector stores), and a Claude Code plugin marketplace entry. See the [repository README](https://github.com/handsontable/handsontable-skills#readme) for details.
## License
MIT — see the [repository](https://github.com/handsontable/handsontable-skills/blob/main/LICENSE.txt). HyperFormula itself is separately licensed (GPLv3 or commercial); see the [licensing guide](https://hyperformula.handsontable.com/docs/guide/licensing.html).
Use this skill when the user works with HyperFormula (HF) — a headless,
open-source TypeScript spreadsheet calculation engine. Use it when the user:
mentions HyperFormula or HF; wants to integrate Excel-like formulas into a
JS/TS app; evaluate formulas programmatically in a browser or Node.js;
simulate spreadsheet or Excel calculation behavior server-side; run
calculations over data parsed from XLSX files (e.g. via SheetJS) in code;
build a pricing engine, financial model, or what-if analysis in code; define
custom spreadsheet functions or named expressions outside of a UI context. Do
NOT use this skill for: general Excel questions unrelated to HF; general
Google Sheets questions; generic spreadsheet formula-syntax questions outside
an HF context; formulas used inside Handsontable (use the "handsontable"
skill instead).
---
# HyperFormula
HyperFormula is a **headless, open-source TypeScript spreadsheet calculation engine** for embedding spreadsheet logic in any JavaScript/TypeScript application (browser or Node.js). ~420 built-in functions, dependency graph, undo/redo, i18n (17 languages). Dual-licensed: GPLv3 or commercial.
Task-oriented references in `references/` (open the one that matches what the user is doing):
- [`getting-started.md`](references/getting-started.md) — how to install, create an instance (`buildFromArray` / `buildFromSheets` / `buildEmpty`), and work with cell addresses. Open this for onboarding questions and first-run errors.
- [`api-quickref.md`](references/api-quickref.md) — runnable examples for CRUD, rows/columns, sheets, exporting data, batching, named expressions, events, undo/redo, clipboard. Open this when implementing or debugging HF API calls.
- [`custom-functions.md`](references/custom-functions.md) — extending HyperFormula via `FunctionPlugin`: minimal plugin, argument types, volatile functions, range arguments, returning arrays, error handling, aliases, localized names. Open this when adding or debugging custom spreadsheet functions.
- [`configuration.md`](references/configuration.md) — `ConfigParams` options, Excel-compatibility preset, Google-Sheets preset, and config-related pitfalls (separator collisions, Node ICU, `precisionRounding` change in v3). Open this when tuning behavior or diagnosing locale issues.
- [`vue3.md`](references/vue3.md) — Vue 3 integration: `markRaw` requirement, Composition API, Pinia/Vuex stores, `destroy()` on unmount. Open this when embedding HF in a Vue 3 app.
- [`error-handling.md`](references/error-handling.md) — inspecting `CellError`, `ErrorType` enum, `getCellValueDetailedType`, common error causes (`#NAME?`, `#CYCLE!`, `#REF!`, …). Open this when a cell returns an error or you need to branch on result type.
- [`general-pitfalls.md`](references/general-pitfalls.md) — cross-cutting gotchas: `destroy()` lifecycle, forcing literal strings, Excel-parity caveats, hard limits. Open this when results look wrong or memory grows unboundedly.
Below this section is the **Documentation map** — the canonical directory of links to the official HyperFormula docs. Use it when pointing the user to authoritative material.
## Documentation map
All links resolve to `hyperformula.handsontable.com` unless noted.
"description":"Agent skill for HyperFormula — the headless, open-source TypeScript spreadsheet calculation engine. Teaches AI coding agents how to integrate, configure, and debug HyperFormula.",
Vue 3 requires wrapping the instance with `markRaw` — see [vue3.md](vue3.md) for the full integration guide (React / Angular / Svelte need no special handling).
### `functionArgSeparator` vs `thousandSeparator` collision
These two options **cannot share a character**. European locales that use `,` as thousand separator must change the argument separator:
```ts
consthf=HyperFormula.buildEmpty({
licenseKey:'gpl-v3',
thousandSeparator:',',
decimalSeparator:'.',
functionArgSeparator:';',// must differ from thousandSeparator
});
```
### Node.js must have full ICU
String comparisons (used in `MATCH`, `VLOOKUP`, sorting) can silently produce wrong results on Node.js < 13 or builds without full ICU. Verify at startup:
```ts
// Must report 'full' (or a detailed ICU version), not 'small'.
HyperFormula defers some structural transformations (row/column insertions, moves) and applies them lazily. In long-lived instances with heavy mutation throughput — bulk imports, frequent undo/redo, scripted batch edits — the pending queue can grow before cleanup. v3.3 fixed the unbounded-growth leak; this option lets you cap the queue explicitly. Lower values trade memory for more frequent flush work; the default is suitable for typical UI workloads.
### `precisionRounding` default changed in v3
Before v3 the default was `14`. It is now `10`. Calculations relying on the old precision need an explicit override:
Mark a function as volatile so it recalculates on every change (like `RAND` or `NOW`). Volatile functions are expensive in large sheets — use sparingly.
```ts
MyPlugin.implementedFunctions={
RAND_ID:{
method:'randId',
parameters:[],
isVolatile: true,
},
};
```
## Argument types
```ts
import{FunctionArgumentType}from'hyperformula';
// FunctionArgumentType.NUMBER
// FunctionArgumentType.STRING
// FunctionArgumentType.BOOLEAN
// FunctionArgumentType.NOERROR
// FunctionArgumentType.SCALAR
// FunctionArgumentType.RANGE // accepts a cell range
// FunctionArgumentType.ANY
```
Use `optionalArg: true`, `defaultValue`, and `minValue` / `maxValue` to validate arguments declaratively:
Custom functions can return 2D arrays. **PITFALL:** result arrays don't auto-resize when upstream dependencies change — the footprint is fixed at first evaluation.
```ts
classSplitPluginextendsFunctionPlugin{
splitToRow(ast,state){
returnthis.runFunction(
ast.args,
state,
this.metadata('SPLIT_ROW'),
(text,sep)=>[text.split(sep)]// 2D array: one row, N cols
// Usually: function not registered (check plugin registration or i18n language)
console.log('Unknown name:',value.message);
break;
default:
console.log('Error:',value.type,value.message);
}
}else{
console.log('Value:',value);
}
```
## `#CYCLE!` is HyperFormula-specific
Standard spreadsheet apps report cycles differently. HyperFormula's `IF` also reports cycles for all branches, even unreachable ones — this can produce `#CYCLE!` in formulas that Excel or Sheets would evaluate.
Use the detailed type when you need to distinguish e.g. number vs empty without touching the value.
## Returning errors from custom functions
Custom `FunctionPlugin` methods can return a `CellError` to surface a formula error. See [custom-functions.md](custom-functions.md) for the full pattern.
```ts
import{CellError,ErrorType}from'hyperformula';
returnnewCellError(ErrorType.DIV_BY_ZERO);
```
## Common causes
| Error | Common cause |
|---|---|
| `#NAME?` | Function not registered (check `registerFunctionPlugin` order or `language` config) |
| `#CYCLE!` | Circular reference — or an `IF` branch that *could* produce one |
| `#REF!` | Deleted row/column broke a formula reference |
| `#VALUE!` | Type mismatch — e.g. passing a string where a number is expected |
| `#DIV/0!` | Division by zero, including empty cells coerced to 0 |
| `#NUM!` | Out-of-range numeric result (e.g. `SQRT(-1)`) |
| `#N/A` | Lookup miss (`MATCH`, `VLOOKUP`) or propagated from an upstream `#N/A` |
- Runtime differences vs Excel/Sheets: https://hyperformula.handsontable.com/docs/guide/list-of-differences.html
## Error handling
See [error-handling.md](error-handling.md) — checking `CellError`, `ErrorType` enum, `getCellValueDetailedType`, common error causes.
## Always call `destroy()` in long-running apps
HyperFormula maintains internal data structures (dependency graph, address mapping) that are **not** garbage-collected until `destroy()` is called. Leaking instances in servers or SPAs accumulates memory.
```ts
hf.destroy();
// After destroy() the instance is unusable — create a new one if needed.
```
v3.3 fixed two longstanding leak sources inside live instances — pending lazy transformations and undo/redo history were not being trimmed. If you maintain very long-lived instances with heavy mutation throughput, also see `maxPendingLazyTransformations` in [configuration.md](configuration.md) to bound the lazy-transformation queue. `destroy()` is still mandatory at teardown.
## Force a string that looks like a formula
Prefix with `'` (apostrophe) to store the literal text instead of evaluating.
```ts
// Stored as the literal string "=SUM(1,2)", not a formula:
Every factory method (`buildFromArray`, `buildFromSheets`, `buildEmpty`) requires `licenseKey`. Use `'gpl-v3'` for open-source use or your commercial key.
## Known hard limits
- **Single workbook per instance** — no multi-workbook support.
- No 3D references, dynamic arrays, async functions, structured references ("Tables"), or relative named expressions.
-`IF` reports cycles for all branches, even unreachable ones.
- Custom function result arrays don't auto-resize when dependencies change.
Vue 3's Composition API wraps objects in a reactive Proxy that intercepts property access and **corrupts HyperFormula's internal state**, causing silent data corruption or crashes. React, Angular, and Svelte need no special handling.
```ts
import{markRaw}from'vue';
import{HyperFormula}from'hyperformula';
consthf=markRaw(
HyperFormula.buildEmpty({licenseKey:'gpl-v3'})
);
```
Use the same wrapper no matter where the instance is held — `ref`, `reactive`, component state, a store (Pinia/Vuex), or a composable. If the raw HyperFormula instance ever reaches Vue's reactivity system, it must be marked.
## Composition API example
```ts
import{onBeforeUnmount,ref}from'vue';
import{markRaw}from'vue';
import{HyperFormula}from'hyperformula';
exportfunctionuseHyperFormula() {
consthf=markRaw(
HyperFormula.buildEmpty({licenseKey:'gpl-v3'})
);
// Free internal data structures when the component unmounts.
# We do a semantic-release DRY RUN to make the job fail if there are no changes to release
GITEA_TOKEN=${{ secrets.RELEASE_TOKEN }} GITEA_URL=https://git.datacontroller.io semantic-release --dry-run | grep -q "There are no relevant changes, so no new version is released." && exit 1
echo "::error::semantic-release dry-run failed (exit $DRY_EXIT). See log above; common cause is the Gitea token lacking push permission on this repo (verifyConditions of @saithodev/semantic-release-gitea)."
exit "$DRY_EXIT"
fi
if echo "$DRY_OUTPUT" | grep -q "There are no relevant changes, so no new version is released."; then
echo "::error::No releasable changes since last tag - aborting release job."
exit 1
fi
semantic-release
- name:Frontend Build
description:Must be created AFTER the release as the version (git tag) is used in the interface
Do NOT auto-commit. Never run `git commit` (or `git push`) unless the user explicitly asks. Leave changes in the working tree for the user to review and commit themselves.
## Markdown files
Do NOT hard-wrap Markdown files (no fixed-width line wrapping / carriage returns inside paragraphs). Each paragraph, list item, and heading should be a single logical line, regardless of length. Let the editor/viewer soft-wrap. This includes AGENTS.md itself.
This is different from `.sas` files, where a maximum line length applies. The no-wrap rule applies to all `*.md` files in this repo (docs, READMEs, etc.).
Rationale: hard-wrapped prose produces noisy diffs when sentences are edited and reflowed, and Markdown renderers already handle wrapping.
## Linting (required before "done")
Never consider a change complete until the relevant linters pass on the files you touched — do not rely on the user's pre-commit hooks to catch it:
- **Client (TypeScript/HTML/etc.)**: run `npm run lint:check` from the `client/` directory (prettier). Fix any failures with `npm run lint:fix`.
- **SAS**: after creating or modifying any `.sas` files, run `sasjs lint` from the `sas/` directory and ensure the files you touched have no lint warnings (the repo currently has pre-existing warnings in other files, which can be ignored).
## No external assets
Data Controller must run entirely locally (offline / on-prem, no internet access). The built product must never fetch assets from remote servers — no external fonts, images, scripts, stylesheets, CDN links, or remote URLs in meta tags (e.g. `og:image`, `og:url`, `itemprop="image"`). All assets must be bundled and served locally.
## The .agent folder
Agent-related content lives in `.agent/`: technical/agent-facing documentation goes in `.agent/docs/` (not `docs/`), and skills in `.agent/skills/`. When writing explanatory or technical docs about the codebase, put them in `.agent/docs/`.
## Code comments and test names
Never reference items that are not active parts of the repository — no "the original bug", "regression from this fix", "this session/PR/commit", or similar ephemeral context. Comments and test names must be self-contained: describe the behaviour being asserted, not the history of how it was discovered. The one exception is a literal link to a ticket/issue tracker.
* adding REGEX validations to mpe_x_test ([bd798b4](https://git.datacontroller.io/dc/dc/commit/bd798b424a7c4b90d613425348d3f99eeb49b025))
* default value for label ([a3e46a9](https://git.datacontroller.io/dc/dc/commit/a3e46a968ef08295347b230235f1e79dfa51c5a2))
* **deps:** retarget Angular upgrade to 20, not 21 (CI install was broken) ([cac9244](https://git.datacontroller.io/dc/dc/commit/cac9244f92544f432078125c312d05c36ed4b855))
* ensure only one REGEX applies at a time ([8fb58eb](https://git.datacontroller.io/dc/dc/commit/8fb58eb36e05fdf7262459b05e53643b3ca672f7))
* ensure that no assets (including og links) ever fetch from external sources ([33dcb98](https://git.datacontroller.io/dc/dc/commit/33dcb989d3ae0c5ee0a13fe64195862a945c4348))
* include peer dependencies in package-lock for npm ci in pipeline ([b51c770](https://git.datacontroller.io/dc/dc/commit/b51c770782a8b62816e5145496dbea9fb83059e0))
* optimisation, renamed values for DDTYPE to save space ([cfb60e5](https://git.datacontroller.io/dc/dc/commit/cfb60e5e4bfd37dc79fa4e9a6c6649ab1a61f2e4))
* patch npm audit vulnerabilities in sas and client dependencies ([e22edf7](https://git.datacontroller.io/dc/dc/commit/e22edf7ed3aa54fc08411cf90d51df9ec30877c3))
* **query:** isolate viewbox filter state from the base table's ([7a35cf4](https://git.datacontroller.io/dc/dc/commit/7a35cf4a458791b26ec8141c0c84d97fcb555145))
* regenerate client lockfile to resolve Angular peer-dependency drift breaking npm ci ([05fe474](https://git.datacontroller.io/dc/dc/commit/05fe4744d58e7985ecc5ef5f9c1969dbab2d2efb))
* **regex:** special missing handling ([180c247](https://git.datacontroller.io/dc/dc/commit/180c2477ed9ae44b4e479e81ec642f40580bcbc8))
* removing low severity warning in npm audit ([2a771bb](https://git.datacontroller.io/dc/dc/commit/2a771bb91acaaa16aba44e2d7ddb51b8b7123c10))
* removing thousand seperator from plain numerics in EDIT mode ([0392a81](https://git.datacontroller.io/dc/dc/commit/0392a81cbd9a634590ca7a60a316cd12c690a5a5))
* **validations:** parse SAS PRX /pattern/flags syntax in HARDREGEX/SOFTREGEX ([7ed3730](https://git.datacontroller.io/dc/dc/commit/7ed3730ae3145dffd639ce3d2f831447d78f4e85))
### Features
* **docs:** adding agents.md and docs for RLS ([359d833](https://git.datacontroller.io/dc/dc/commit/359d833406ace79f96d03c08a5c72fbd3df29442))
* **editor:** evaluate HARDREGEX/SOFTREGEX independently instead of hard-wins precedence ([57db117](https://git.datacontroller.io/dc/dc/commit/57db1179a97973ecf0f711bda6057383de546dd3))
* **editor:** show applied HARDREGEX/SOFTREGEX pattern in column info dropdown ([39c8855](https://git.datacontroller.io/dc/dc/commit/39c8855f37477f06d8b8cf5c97e04543070e8385))
* **regex:** backend validations on regex strings ([d2c93a4](https://git.datacontroller.io/dc/dc/commit/d2c93a46facb386fd31661c02c0d3d34f38f6e43))
* using ALL libraries as validation in MPE_SECURITY. Closes [#279](https://git.datacontroller.io/dc/dc/issues/279) ([62ff0ae](https://git.datacontroller.io/dc/dc/commit/62ff0aee4a976184de65d87ea3c8b8b7cb777938))
* **editor:** ignore cell validation on rows marked for delete ([70c5805](https://git.datacontroller.io/dc/dc/commit/70c5805743c8c6e61bc216f37cbe8cd4f0614c98))
* explicit use of .sashdat in dc_casload ([6a95067](https://git.datacontroller.io/dc/dc/commit/6a95067618600d7b03b6af46bf3d74721e42c66a))
* refactor mf_wordsinstr1butnotstr2 and added a wide table to test suite ([be94c68](https://git.datacontroller.io/dc/dc/commit/be94c68f354d0f9b1eb52213807d1874a74d9b8a))
* retry the startup service request once before failing ([5391df8](https://git.datacontroller.io/dc/dc/commit/5391df8f04cf40205a600b23c207f97b6fdf5913))
* run dc_casload for dq tables in getdata. Closes [#264](https://git.datacontroller.io/dc/dc/issues/264). ([6b16f9b](https://git.datacontroller.io/dc/dc/commit/6b16f9b3647fca2e39d44544201ce972046e01a3))
* updating tests, moving mpe_rowhash to core library ([c5f0e4f](https://git.datacontroller.io/dc/dc/commit/c5f0e4f106aa4e6a7c2a2fd0ebb2e85d1e7bee7b))
### Features
* bumping core to get the mp_rowhash definition, [#269](https://git.datacontroller.io/dc/dc/issues/269) ([f73f5e4](https://git.datacontroller.io/dc/dc/commit/f73f5e47ac38a305bac4ebd7eac48d1a3edf4a83))
* **editor:** order validation rules by COLHEADERS, not cols[].VARNUM ([4c2aca3](https://git.datacontroller.io/dc/dc/commit/4c2aca34951d6d5c5cf2df69d5bf4256a9290d53))
* adapt column validation to vertical-array COLTYPE ([#253](https://git.datacontroller.io/dc/dc/issues/253)) ([3016750](https://git.datacontroller.io/dc/dc/commit/301675052fb5cbcb11f54c370ad7eda1d60461fd))
* backend updates to bring back labels under [#240](https://git.datacontroller.io/dc/dc/issues/240) ([fb94840](https://git.datacontroller.io/dc/dc/commit/fb94840016f067fa3523a3539dd0616d8970eed0))
* **editor:** normalise unpadded time strings on spreadsheet import ([7b4b4eb](https://git.datacontroller.io/dc/dc/commit/7b4b4ebeeb0e34ec5fcd5c485741d7bbddb43b7e))
* **hot:** row rendering on resize, resolve blank whitespace ([c7ba025](https://git.datacontroller.io/dc/dc/commit/c7ba025d39aa9681ad51f239503e884fb26613e2))
* including datadictionary descs in labels, [#240](https://git.datacontroller.io/dc/dc/issues/240) ([e7453a9](https://git.datacontroller.io/dc/dc/commit/e7453a9305263faf4d4f288d50977121353040c7))
* pass variable formats to front-end in a vertical array instead of horizontal list ([691d6f2](https://git.datacontroller.io/dc/dc/commit/691d6f277ea87051f539681b34da2f39cade8fc3))
* remove coltype from sasparams table, add it to cols table. ([8101400](https://git.datacontroller.io/dc/dc/commit/81014001ac4da6b8d262c72c65e0c657f186ec84))
* trim leading/trailing whitespace from Excel header cells on upload ([2728dac](https://git.datacontroller.io/dc/dc/commit/2728dac873ab8c9fc44413f873e4730515ca0e21))
* **va:** date filters refactor, support more dates ([df5c975](https://git.datacontroller.io/dc/dc/commit/df5c9758697d7481e67d20ed7ff81e9ce4e92ec0))
* **va:** enable readOnly and hold filtering while editing ([82a254d](https://git.datacontroller.io/dc/dc/commit/82a254d22cb7aaf75aae7b332f0b07fc8bda3e44))
* **va:** enable Upload button in va mode ([1beb3d4](https://git.datacontroller.io/dc/dc/commit/1beb3d490d0e7b8d5fa7985f121495ed62740cfe))
* **va:** guard deferred contextMenu toggle against destroyed instance ([b5e9b25](https://git.datacontroller.io/dc/dc/commit/b5e9b2531924bec1a23c67d3ad0ca3f512a75655))
* **ci:** bump Lighthouse to 13.4.0 ([47ba03e](https://git.datacontroller.io/dc/dc/commit/47ba03ec637113d8867372d64484927de9b645ff))
* **dc-validator:** display date/time/datetime cells as raw ISO, locale-independent ([9ca5719](https://git.datacontroller.io/dc/dc/commit/9ca57193b4aafa1e7bf216301cdf197cea6ec4c9))
* **editor:** await dynamic validation on paste; defer spinner ([ea03bde](https://git.datacontroller.io/dc/dc/commit/ea03bdecc5aaf917052add2c7e6fdc192b9974ce))
* **editor:** cancelEdit will reset cell's valid state ([51071b4](https://git.datacontroller.io/dc/dc/commit/51071b463bb311c6db48c301daa8cc7dfd92144a))
* **editor:** preserve commas in dropdown source values ([6af49cf](https://git.datacontroller.io/dc/dc/commit/6af49cf1fd8aa27cbf99f9514cca537339cdbf7e))
* **editor:** retry initSetup when the Handsontable instance isn't ready ([53b7ee0](https://git.datacontroller.io/dc/dc/commit/53b7ee0cb23329fa4f9d6211ffe6b102053fade9))
* **handsontable:** horizontal scrollbar in dropdown ([ac0bd10](https://git.datacontroller.io/dc/dc/commit/ac0bd102127ced6b6a908a7244f35002f22cb669))
* **handsontable:** restore dark mode for v17 ([9d97bf7](https://git.datacontroller.io/dc/dc/commit/9d97bf7ea13199ab9dab10c5fc47e07b4751fb9a))
* **handsontable:** update licenseChecker exclude buffers - exceljs dep ([12c7d30](https://git.datacontroller.io/dc/dc/commit/12c7d30894b5c3709b683189da6e52255ad0a3e7))
* incrementing rk val ([7444209](https://git.datacontroller.io/dc/dc/commit/74442096bac1481a4992e5c97dac47c8d277cc3f))
* **licensing:** add protocol info ([3668a74](https://git.datacontroller.io/dc/dc/commit/3668a7426fec078ad7939b5bc815d7ad920c0d62)), closes [#178](https://git.datacontroller.io/dc/dc/issues/178)
* migrate handsontables to v17 ([1b73e35](https://git.datacontroller.io/dc/dc/commit/1b73e355b78a0039aa48f7a137095eb5470c08e2))
* **mocks:** return dynamic cell values as OBJ to match new contract ([a15fdc4](https://git.datacontroller.io/dc/dc/commit/a15fdc401e6cdcf934a5d599f8b7c4d197e9d2d4))
* removing display_value in SAS code ([9b18a45](https://git.datacontroller.io/dc/dc/commit/9b18a45d354ad617e590d62758ef761b01b3095a))
* sending dynamic_values and dynamic_extended_values back as OBJ instead of ARR ([e859d33](https://git.datacontroller.io/dc/dc/commit/e859d3354e82f0faad53a0b8aa14ed66974cbffa))
* viewLibs now fires only once, libPromise shared between the calls ([dc4e07a](https://git.datacontroller.io/dc/dc/commit/dc4e07a692a5e839986abb69b17e51024c4cdc03))
### Features
* **edit-record:** use native date/time/datetime pickers ([2d29346](https://git.datacontroller.io/dc/dc/commit/2d29346cbf09ad057743e6f5ebea12f3257f246e))
* **editor:** migrate date/time/datetime to HOT 17 Intl cell types ([f70ea2f](https://git.datacontroller.io/dc/dc/commit/f70ea2fe712ff9a42bf47387f9f0aaa6d1a1f511))
* **editor:** paste-validation overlay with cancel and confirm ([609731f](https://git.datacontroller.io/dc/dc/commit/609731ff99185a162614df13bde7afb1a2c3bbca))
* **editor:** SAS VA data-driven content embed mode (?embed=va) ([ffa3ff9](https://git.datacontroller.io/dc/dc/commit/ffa3ff9c103d6d80677f4f67e7d53418f201f491))
* **editor:** VA embed filter modes with pending UX ([ebe6972](https://git.datacontroller.io/dc/dc/commit/ebe6972dd7d38c5742ed7eb446e0448d98b2c8a7))
* **editor:** validate autofilled cells; fix paste validation lag ([11ee49a](https://git.datacontroller.io/dc/dc/commit/11ee49a57a0bc7ccd59e91d5fe37bb5db1bf7d69))
* **handsontable:** enable export plugin, add export menu items ([29aaa72](https://git.datacontroller.io/dc/dc/commit/29aaa72c6047bb309d325887ab602a9369a1ad74))
* migration script for adding new validation rules under [#241](https://git.datacontroller.io/dc/dc/issues/241) ([9f84a58](https://git.datacontroller.io/dc/dc/commit/9f84a580fb762da8c1c796a35eb4f531df1caf13))
* new validation rules in mpe_selectbox. [#241](https://git.datacontroller.io/dc/dc/issues/241) ([2f5d3df](https://git.datacontroller.io/dc/dc/commit/2f5d3dfc387f02353601aaa844aaeabde3e9f3e5))
### Reverts
* eager-load feature modules instead of lazy-loading ([3429a7a](https://git.datacontroller.io/dc/dc/commit/3429a7a2a6ecfa6115394e1c842061cc68bf2be3))
* enabling DSN=*ALL* in MPE_SECURITY ([7d94cb2](https://git.datacontroller.io/dc/dc/commit/7d94cb2ae4a3f6c1fa1011ae0fced7083a2f2793))
* providing default values for RULE_ACTIVE on MPE_VALIDATIONS ([f031b4e](https://git.datacontroller.io/dc/dc/commit/f031b4eb8925397e60dcc739a721cfbbb6da8dff))
* switch away from api usage for CASLIB metadata ([ce921a0](https://git.datacontroller.io/dc/dc/commit/ce921a032a8970b8078a463a41da884e1fa71bc3))
* use correct debug param for runAsTask ([bb80476](https://git.datacontroller.io/dc/dc/commit/bb8047676749814d3b86eea666726dbe4bf5f270))
* move cas session assign to settings.sas and abort when lib is unassigned ([65f0b97](https://git.datacontroller.io/dc/dc/commit/65f0b979a401277b3e070d409659ae3fae2ff8c0))
* **client:** bundle Metropolis font locally to satisfy CSP ([9546fcd](https://git.datacontroller.io/dc/dc/commit/9546fcd6312f3e81f746ef6e32ef398810ed434a))
* **client:** clear angular build cache on font strip to avoid stale dist ([503cb08](https://git.datacontroller.io/dc/dc/commit/503cb08b2fa40397434189f9c20eff3358eb7010))
* **client:** postinstall removal of Metropolis [@font-face](https://git.datacontroller.io/font-face) from @clr/ui ([e6397ce](https://git.datacontroller.io/dc/dc/commit/e6397cecc13afe2a9238bdfb2b4b9b81f38d055c))
* **client:** serve text-security-disc font locally ([80ce80e](https://git.datacontroller.io/dc/dc/commit/80ce80ece40012e59c7cd0340b4aa9a9aca46443))
* **editor:** preserve numeric type for SAS num cols with static SOFTSELECT/HARDSELECT ([05a3289](https://git.datacontroller.io/dc/dc/commit/05a328976ea3d1d6ef7559850369aa580f0d067f))
* resolve outer promise in parseCsvFile for non-WLATIN1 path ([4ee15e1](https://git.datacontroller.io/dc/dc/commit/4ee15e1b6e83f27f279fc345e6998452a8f64d7e))
* use XLSX for CSV row truncation to handle new lines in values ([6d590c0](https://git.datacontroller.io/dc/dc/commit/6d590c050dcd593a73464fae5604f774f016b10d))
### Features
* add embed URL parameter to hide header and back button ([b0dc441](https://git.datacontroller.io/dc/dc/commit/b0dc441d681369e06eee58288dbdbb236f930bdc)), closes [#214](https://git.datacontroller.io/dc/dc/issues/214)
* support for SASIOSNF engine (SNOW alias) plus meta assignment ([7694d1b](https://git.datacontroller.io/dc/dc/commit/7694d1b0fb2bd0407c8598147fbae87a00d889a8))
* cli bump for mf_getscheme support ([a84ba41](https://git.datacontroller.io/dc/dc/commit/a84ba41ea9f0c97ae24f0a572b8cf5ec200f2132))
* missing upcase on SNOW section, plus local sasjs target ([dc20064](https://git.datacontroller.io/dc/dc/commit/dc200646f7df2fd1910841f392c314532aae7581))
### Features
* SAS code changes for snowflake support ([e273e87](https://git.datacontroller.io/dc/dc/commit/e273e870efbf7875db869b760f2c7b1f39d571ae))
* correctly applying deletes on viya, also ([46cdeb0](https://git.datacontroller.io/dc/dc/commit/46cdeb0babee6870553a41877cbe85204e7099c4))
* crypto module requirement for sheetjs/crypto package ([505d0af](https://git.datacontroller.io/dc/dc/commit/505d0af2b3b3c1c79c65045dcaffc263e0f8e796))
* disable parsing excel in web worker beacuse it breaks in the stream apps ([280bdee](https://git.datacontroller.io/dc/dc/commit/280bdeeb1b82f00689f46c68a3cde3f2d24bc18f))
* Display all contexts when installing DC on Viya ([d41f88f](https://git.datacontroller.io/dc/dc/commit/d41f88f8bf5bb2c725ee3edba085e8961ab8c727))
* **edit:** use cellValidation keys and hotDataSchema to fill in defaults on add row ([4957548](https://git.datacontroller.io/dc/dc/commit/495754816c0e757b8f8b1c0ad51246dc7b65d957))
* enabling closeouts for UPDATE in CAS tables ([8b8e8ae](https://git.datacontroller.io/dc/dc/commit/8b8e8aec159ff2f50cfa4683bcd7a25aabb75bf8))
* enabling rollback when the table has formatted values ([815d6e9](https://git.datacontroller.io/dc/dc/commit/815d6e97a8e304d79d48cc949ba126e02a318dc1))
* improvements to validations ([6ceb681](https://git.datacontroller.io/dc/dc/commit/6ceb6814633691b6d4ac2cb898cfb75e9d609102))
* remove IE checks and conditions ([ece6bd1](https://git.datacontroller.io/dc/dc/commit/ece6bd1d787d722531334fc4f1396a94cf6d92ec))
* updates to demodata to enable auto CAS promote ([7740d2a](https://git.datacontroller.io/dc/dc/commit/7740d2ac8694295b33b40a30603d8239818896f5))
* upgrade angular core and compiler ([aecd597](https://git.datacontroller.io/dc/dc/commit/aecd5976875a7c01189248c5f5aa3478b28c1ab2))
* using fcopy instead of binary copy for file upload, for Viya 2026 compatibility ([716ee6e](https://git.datacontroller.io/dc/dc/commit/716ee6eba0a28f4f0a7a96b0719caf15da9b6e78))
* adding demo data job ([8c2aeac](https://git.datacontroller.io/dc/dc/commit/8c2aeacc85da5c106c356709cefcb412ed0a71db))
* **dq rules:** notnull validation when invalid cell, will auto populate a default value ([96f2518](https://git.datacontroller.io/dc/dc/commit/96f2518af9e547956be5862a1322d9ab8e07369b))
* **ci:** cypress dependency package not available anymore ([26cdd73](https://git.datacontroller.io/dc/dc/commit/26cdd733315ef8babe9498ce93f6eb29c587dabd))
* **hot v16 migration:** multi dataset fixed issues, and cypress tests adapted ([712b384](https://git.datacontroller.io/dc/dc/commit/712b3848480a8769d149e00b0d2de91396022b66))
* **viewboxes:** hot v16 fails to load because of relative height `100%` ([672dd6d](https://git.datacontroller.io/dc/dc/commit/672dd6d4f1fda27e3706dd7caa42b45922319497))
* bumping CLI to 4.12.10 ([a08a717](https://git.datacontroller.io/dc/dc/commit/a08a717ca8d49e8a7d63f3fd91c6a7d42a1d6d8b))
* bumping sasjs/core and sasjs/cli ([63e9af4](https://git.datacontroller.io/dc/dc/commit/63e9af402ed65f6be4426e76ee1376a40e6ed097))
### Features
* improving accessibility score up to 100, hot update to v16.0.1 ([71c308d](https://git.datacontroller.io/dc/dc/commit/71c308d052400ecedc03f8020a5a69471ac6b116))
* **viya deploy:** run makedata in new window to ensure logs are available for the user ([0b4042a](https://git.datacontroller.io/dc/dc/commit/0b4042af6011fdc65cfaaa5d4b1d8f48cd67f3b3))
* ensuring apploc is not case sensitive. Closes [#171](https://git.datacontroller.io/dc/dc/issues/171) ([24545f2](https://git.datacontroller.io/dc/dc/commit/24545f2acdd5bd73cbe062526f2bd043269cc6a3))
* reload startupservice after user approves the MPE_TABLES page ([e5f8e50](https://git.datacontroller.io/dc/dc/commit/e5f8e500c125ee233c6f7af5ad0077c0ed6abfcb))
* showing catalog_cnt in libinfo ([e44a25d](https://git.datacontroller.io/dc/dc/commit/e44a25dcc39ba4b9714257c60da84c2dfa613a85)), closes [#160](https://git.datacontroller.io/dc/dc/issues/160)
### Features
* adding 4 new tables for catalogs ([e4dbab8](https://git.datacontroller.io/dc/dc/commit/e4dbab8b1654b24e610e4b0603d1cf2b02a451e2))
* capturing catalog specific information, closes [#159](https://git.datacontroller.io/dc/dc/issues/159) ([b4c586a](https://git.datacontroller.io/dc/dc/commit/b4c586a859929e0122cd46449e43d4ca597b8b2b))
* Introduction of 4 new tables for capturing information related to catalogs and their objects. Migration script prepared and available in the DB folder (usual place)
* bump core to ensure ff works on viya streaming deploy ([cbd69df](https://git.datacontroller.io/dc/dc/commit/cbd69df708edf3a8446115ca7315fac3557dcf97)), closes [#156](https://git.datacontroller.io/dc/dc/issues/156)
* viya deploy load data timing ([abdbb67](https://git.datacontroller.io/dc/dc/commit/abdbb674713796e5308eb4272197a5c253868a85))
### Features
* viya deploy, update the index.html contextname ([7223955](https://git.datacontroller.io/dc/dc/commit/72239558af2ee50cdfc71b7e185e6661ab568ba1))
@@ -23,10 +23,42 @@ _Problems with the above include:_
Data Controller for SAS® solves all these issues in a simple-to-install, user-friendly, secure, documented, battle-tested web application. Available on Viya, SAS 9 EBI, and [SASjs Server](https://server.sasjs.io).
For more information:
An individual Viya deploy can be done in just 2 lines of #SAS code!
```sas
filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/viya.sas";
%inc dc;
```
For a multi-user deploy, using a shared system account, please see [deploy docs](https://docs.datacontroller.io/deploy-viya/).
For further information:
* Main site: https://datacontroller.io
* Docs: https://docs.datacontroller.io
* Code: https://code.datacontroller.io
For support, contact support@4gl.io or reach out on [Matrix](https://matrix.to/#/#dc:4gl.io)!
For support, contact support@4gl.io or reach out on [Matrix](https://matrix.to/#/#dc:4gl.io)!
## Development
### Lighthouse CI
This project includes automated Lighthouse performance and accessibility checks that run on pull requests. The checks ensure:
- **Accessibility Score**: Minimum 1.0 (100%) median score across all tested pages
The Lighthouse CI workflow:
1. Sets up the development environment with SASjs server and mocked services
2. Builds and serves the Angular frontend
3. Installs Chrome and runs `lhci autorun` (Lighthouse CI) against key pages
4. Uploads results as artifacts for review
To run Lighthouse checks locally:
```bash
cd client
npm install
npm run lighthouse
```
Configuration is in `client/lighthouserc.js` (URL list, `desktop` preset, Chrome flags, assertions).