Add TIMEZONE, SYSTIMEZONEIDENT and SYSTIMEZONEOFFSET to the startupservice
globvars and display them in the Environment Details block, so the session
clock used to stamp submission timestamps is directly visible to admins.
# [7.13.0](https://git.datacontroller.io/dc/dc/compare/v7.12.0...v7.13.0) (2026-09-03)
### Bug Fixes
* dynamic js mocks ([3b9344c](3b9344cb60))
* adding formula types ([c07a01a](c07a01a38d))
* address hermes review feedback (formula quoting, cell revert, addRow guard) ([36963aa](36963aa746))
* agent skills and nextviya deploys ([f7db871](f7db8719f5))
* bump core for mp_validate fix ([5a44b28](5a44b2804f))
* CAS support for REPLACE type plus docs ([ea05f07](ea05f07180))
* clarify debug comment is permanent, extend ComputeContextDetails with Viya response fields ([72484ae](72484ae844))
* core major bump plus autofix of viya context on deploy ([6527c10](6527c10f26))
* **core:** bump to v5 (breaking change) ([59e9e96](59e9e96f5a))
* **cypress:** correct mock data, test expectations, and abort modal ([13c10be](13c10be30f))
* **cypress:** make licensing combined-key tests order-independent ([4db8d62](4db8d62dca))
* **deploy:** Viya deploy checks, startup diagnostics, and chunked deploy script ([917925f](917925f553)), closes [#303](#303) [#200](#200) [#125](#125)
* **deps:** align @angular/* packages to the same lockstep version ([24e6297](24e6297187))
* **deps:** override nanoid to 3.3.18 in sas/ ([d57ae03](d57ae03fc4))
* **deps:** pin @handsontable/angular-wrapper to 18.0.0 ([2d7df66](2d7df66fc2))
* **deps:** pin babel-loader to resolve an unresolvable peer conflict ([3bf3bf0](3bf3bf0dde))
* **deps:** regenerate lockfile with Node 24 and pin handsontable to 18.0.0 ([7da29bd](7da29bdb11))
* **deps:** regenerate lockfile with strict peer-dep resolution to fix npm ci in CI ([efd2e18](efd2e18c7b))
* **deps:** regenerate package-lock.json to resolve npm ci sync errors ([a501903](a501903e6d))
* **deps:** resolve npm audit findings and align @angular/* to the same lockstep version ([b9e4b27](b9e4b2733f))
* **deps:** resolve npm audit findings via direct lockfile patch, not regeneration ([4d1bfa6](4d1bfa6343))
* **deps:** resolve npm audit vulnerabilities in production dependencies ([257f69c](257f69ccc6))
* **deps:** scope the brace-expansion override to its actual vulnerable chain ([f9d061c](f9d061c489))
* dynamic getdynamiccolvals.js ([d59ef44](d59ef44226))
* **editor:** clear sort before reading cells to preserve on cancel ([4137ebd](4137ebdf6b))
* **editor:** escape formula-looking values in uploaded Excel data ([e69df3d](e69df3deb2))
* **editor:** fix formula $-substitution corruption and sorted-grid row/formula desync ([e139c37](e139c37fbf))
* **editor:** let the column info dropdown's text be selected and copied ([1e516f4](1e516f4012))
* **editor:** re-mark a reverted cell as auto-escaped so "Apply as formula" works again ([12091c4](12091c4044))
* **editor:** resolve a primary key's live formula to its computed value, not the raw formula text ([d72e19a](d72e19a308))
* **editor:** resolve computed value for a live formula in numeric columns too ([2cc8489](2cc8489892))
* **editor:** restore multi-column sort with one call, not a loop ([8fafeb7](8fafeb7923))
* **editor:** size table-header buttons and title to content, not fixed grid thirds ([d0a7561](d0a7561f1a))
* **editor:** stop dc.row_status from being submitted to the backend ([dc0f6a7](dc0f6a7baa))
* **editor:** stop turning missing columns into undefined, and fix column lookup for sparse rows ([a104f78](a104f78645))
* **editor:** submit a live formula's computed value, not its raw text ([b7d3106](b7d31066ae))
* ensure cleanup of casuser temp table in error condition ([c4123e5](c4123e5c96))
* ensuring formats arrive from backend in getstagetable.sas ([daacb49](daacb49c8a))
* escape regex metacharacters in context name before prxchange ([9023eb2](9023eb2a39))
* failing test ([f213d24](f213d24f09))
* fixed npm vuln ([26b55b1](26b55b1bde))
* **formulas:** avoid EDIT_STATUS colliding with a real column of that name ([acb97f4](acb97f4bfb))
* hardening following PR review feedback ([1f4aa6f](1f4aa6fcea))
* history mock ([0082828](0082828543))
* **licensing:** bridge async proceed() through cy.then and replace Response.arrayBuffer with manual stream reader ([8e51fde](8e51fde524))
* **mocks:** add fixture staged data and MPE_SUBMIT row for stage.cy.ts ([6e2548d](6e2548d78c))
* **mocks:** add LIBRARYNAME to startupservice, fix viewbox column order ([c0af86e](c0af86ef29))
* **mocks:** limit stage fixture to 1 row, fix column order ([f25ecd8](f25ecd8828))
* **mocks:** look up PK_FIELDS from MPE_TABLES in viewdata, fallback for MPE_AUDIT ([94d2a79](94d2a7970e))
* **mocks:** remove MPE_VALIDATIONS and MPE_ALERTS from DC996664 libref ([4f7b845](4f7b845390))
* **mocks:** remove shadowed saveTableData in rejection.js ([fed0396](fed0396737))
* **mocks:** remove SOME_SHORTNUM HARDREGEX from MPE_VALIDATIONS ([34bd660](34bd6607c5))
* **mocks:** rename saveTableData in postdata.js to avoid shadowing ([e4b04b1](e4b04b1d44))
* **mocks:** SAS datetime format in getchangeinfo, login in stage.cy.ts ([c180934](c180934167))
* more mock improvements ([a80927c](a80927cbb6))
* mp_execute dep ([023c29f](023c29f00f))
* pin fast-uri to 3.1.7 to close high-severity SSRF/host-confusion advisories ([f4325b2](f4325b2056)), closes [hi#severity](https://git.datacontroller.io/hi/issues/severity)
* spy for startup ([69b4155](69b41559f7))
* **stage:** submit data via editor flow, use dynamic date assertion ([490d013](490d0137b9))
* **startup:** show the real startupservice response text on a malformed reply ([9a1b7d0](9a1b7d0f52))
* tests and realistic getcolvals.js ([5d03164](5d03164b6a))
* **tests:** new libref ([c0bc87d](c0bc87d331))
* unnecessary getsubmits call removed ([cc2ff87](cc2ff873fa))
* using utility macro for webout ([3595bb9](3595bb9499))
### Features
* **editor:** color row-header status cells and switch modified symbol to ± ([e15f2c2](e15f2c2a34))
* **editor:** generalize cell revert to any overwritten value, not just formulas ([07d586d](07d586da52))
* **editor:** scope live formulas to character columns, not just HARDFORMULA/SOFTFORMULA ones ([92d0d91](92d0d91440))
* **editor:** translate column names to cell references on formula paste ([1031ea7](1031ea7ed7))
* **formulas:** flag formula-overwritten cells with revert, harden dc.row_status against SAS name collisions ([a8237b2](a8237b2881))
* **formulas:** live DC.ROW_STATUS, insert-row formula fixes, UserService singleton fix ([4a8c39b](4a8c39b4c0))
* **formulas:** wire HyperFormula into HARDFORMULA/SOFTFORMULA rules ([cbea04c](cbea04c8e1))
* functional JS mocks ([ee52bb8](ee52bb8967))
* **licensing:** support combined-key paste, object-format features, and a live key preview ([20ba633](20ba633087))
* **licensing:** warn and block applying a licence key generated for the wrong protocol ([41680e2](41680e2ecc))
* **stage:** add Formatted/Unformatted toggle to the staging page ([74c38e1](74c38e1641))
* updated configurator for Viya deploy ([df2027d](df2027dab6))
* **viewboxes:** add drag-to-resize on all four edges and corners ([e5a5bf2](e5a5bf2144))
multiColumnSorting's sort() replaces the entire sort state on every call rather than accumulating, so the restore loop after updateSettings()/editTable()/cancelEdit() silently dropped every sorted column but the last. Pass the captured configs as a single array instead. Adds a Karma integration test proving the loop
loses columns, and a Cypress test verifying cancelEdit() keeps a real shift-click multi-column sort intact end-to-end.
cancelEdit()'s getFormulaCellsToPreserveOnCancel callbacks passed a
physical rowIndex straight into visual-row-expecting Handsontable APIs
(getCommentAtCell, getDataAtRowProp) while the grid could still be
sorted, since clearSort() only ran later, right before updateSettings().
On a sorted grid this read the wrong row's comment/value, corrupting
which formula cells got preserved across cancel. Move clearSort() (via
getCurrentSortConfigs()) ahead of those callbacks instead.
String.replace interpreted $&/$`/$'/$<digit> specially in DC.ORIG_VALUE/DC.USER_NAME substitutions, corrupting formulas whose cell data happened to contain a literal $. Separately, afterChange indexed dataSource with Handsontable's visual row instead of translating to physical (as beforeChange already does), desyncing EDIT_STATUS/overwritten-comment writes on a sorted grid. Investigating that surfaced a third, unrelated Handsontable bug: updateSettings() corrupts formula cell references whenever a sort is active, regardless of what it changes - fixed by clearing the sort around every updateSettings() call.
Rewrite stage.cy.ts to follow the same pattern as excel.cy.ts:
open MPE_X_TEST, attach regular_excel.xlsx, submit to create staged
data, then test the formatted/unformatted toggle on the stage page.
The date assertion is now dynamic: reads the raw SOME_DATE value
from the unformatted view and computes the expected SAS date9.
formatted value, rather than hardcoding 12FEB1960.
Remove the fixture stageSubmission code from makedata.js — staged
data is created at runtime by stagedata.js, not by makedata.
The stage.cy.ts fixture had 7 rows of MPE_X_TEST data with wide string
values, causing Handsontable to auto-size columns wider so only 4 of
10 fit in the 1280px viewport. The test checks .ht_clone_top headers
which only render visible columns.
Limited to 1 row (matching the original sas9 mock) and moved
_____DELETE__THIS__RECORD_____ to the end of the column order.
getchangeinfo.js: use SAS datetime19. format (DDMMMYYYY:HH:MM:SS)
for SUBMITTED_ON_DTTM and REVIEWED_ON_DTTM, matching the real SAS
output. Use SUBMIT_STATUS_CD field name (not REVIEW_STATUS_ID)
to match the MPE_SUBMIT column name and getchangeinfo.sas output.
stage.cy.ts: add login step in beforeEach (the beforeAll logs out
after uploading the license, so each test needs to re-login).
Use loginAndUpdateValidKey(true) to force license upload.
stage.cy.ts navigates directly to /stage/DC20221007T122326121_612316_7259
and expects a hot table with MPE_X_TEST data. The mock had no fixture
staged.json or MPE_SUBMIT row for this table ID, so getstagetable
returned 0 rows and getchangeinfo returned empty jsparams.
Added a fixture staged.json with MPE_X_TEST rows (SOME_DATE=42) and
a matching MPE_SUBMIT row (SUBMITTED status) in makedata.js.
viewdata.js hardcoded PK_FIELDS as empty, so the viewbox grid treated
all columns as non-PK. removeAllColumns() could delete every column
including the ones the tests expect to be protected.
Now reads MPE_TABLES buskey (mirrors mp_getpk) and falls back to
LOAD_REF LIBREF DSN KEY_HASH TGTVAR_NM for MPE_AUDIT (which has a
known index-based PK in real SAS but isn't registered in MPE_TABLES).
All 12 viewbox.cy.ts tests now pass.
startupservice.js only returned LIBREF in saslibs, but the dc-tree
component renders LIBRARYNAME — without it the nav tree node text was
empty, so libraryToOpenIncludes could not match. Added LIBRARYNAME,
LIBRARYID, and ENGINE to match the viewlibs service format.
Changed libraryToOpenIncludes_SASJS from 'dc' to 'dc_jslib' to avoid
matching multiple libraries (DC_JSLIB and DC996664 both contain 'dc').
viewdata.js: removed alphabetical .sort() on cols (SAS preserves
physical column order via mp_getcols); added empty row with correct
SAS missing values ('.' for numeric, '' for character) when a table
has no data, so the viewbox grid can derive column headers.
The Excel fixture files (7th_tab, extra_column, duplicate_column,
duplicate_row, regular_xls, regular_with_delete, password, leading
whitespace) contain SOME_SHORTNUM values of 0, 1, and 2. The
HARDREGEX /^(??$).*/ blocks those values, aborting
the upload before it reaches the submit/approve flow. Removing the
rule allows all 23 excel.cy.ts tests and both liveness.cy.ts tests
to pass.
Same shadowing issue as rejection.js: postdata.js defined its own
saveTableData(tableName, tableData, tableLibref) which was shadowed
by dcMockUtils.saveTableData(dataDir, tableName, tableData) from the
eval. Renamed to saveTableToLib to eliminate the collision.
rejection.js defined its own local saveTableData(tableName, tableData)
with 2 args, but dcMockUtils.js (loaded via eval) also defines
saveTableData(dataDir, tableName, tableData) with 3 args. The
dcMockUtils version was shadowing the local one, causing
saveTableData('MPE_SUBMIT', submitData) to pass the table name as
dataDir and the data object as tableName, crashing with
'TypeError: tableName.toLowerCase is not a function'.
Removed the local saveTableData and updated calls to use the
dcMockUtils 3-arg signature: saveTableData(dataDir, tableName, data).
The multi-load fixture sheets for DC996664.MPE_VALIDATIONS had data
misaligned with the schema, corrupting the shared submit state. The
tests only need MPE_X_TEST and MPE_TABLES, so only register those two.
- editor.cy.ts: assert DC.USER_NAME is non-empty instead of hardcoded
'sasdemo' (CI runs as a different OS user)
- editor.cy.ts: use =SOME_BESTNUM * 0 + 1936 for a deterministic formula
result regardless of the column's actual value
- excel.cy.ts: expect computed value '2' instead of formula text '=1+1'
(no XL_RULE=FORMULA configured on that column)
- excel-multi-load.cy.ts: use DC996664 libref to match the actual fixture
sheet names
- makedata.js: add labels to DC_JSLIB.MPE_X_TEST schema (SOME_CHAR,
SOME_DATE, SOME_DATETIME, SOME_TIME) for the viewer-labels tests
- makedata.js: register DC996664 libref with MPE_X_TEST/MPE_TABLES/
MPE_VALIDATIONS/MPE_ALERTS and copy table files for multi-load fixtures
- makedata.js: change SOME_SHORTNUM HARDREGEX from [1-5] to [0-2]
- loadfile.js: exclude VAR_TXFROM/VAR_TXTO/VAR_PROCESSED columns from the
missing-cols check for TXTEMPORAL tables (computed at load time)
- InfoModal.ts: add _PROGRAM to AbortDetails
- info-modal.component.html: display _PROGRAM in the abort modal
- 4 call sites: pass abortRes._PROGRAM when building AbortDetails
- licensing.cy.ts: call proceed() unconditionally via cy.then() instead of conditionally chaining .then() after cy.wait(), so the command queue stays open across the async gap and enqueued commands always run
- combinedLicenceKey.ts: replace new Response(stream).arrayBuffer() with an explicit streamToArrayBuffer helper for broader browser support
- remove debug-license.cy.ts, an untracked throwaway repro for the promise-handling question
- Tests 7-10 only acted when already on the licensing page, silently
no-oping otherwise; now they navigate to licensing/update explicitly
when needed, chained via .then() to avoid Cypress conflating a raw
async function's promise with the preceding cy.visit()/cy.wait()
- Replace Response(readable).arrayBuffer() with a direct
ReadableStream reader loop in generateCombinedKey - the Response-based
read never settled under Cypress, likely due to its fetch/Response
patching for network interception
- Accept a single pasted/uploaded combined licence key (base64+gzip of
licence+activation key), auto-detected in either paste mode, alongside
the legacy two-field format
- Warn and block applying a key generated for the wrong protocol before
it ever reaches the backend
- Decode both the legacy positional features string and a new named
features object, so already-issued keys keep working unchanged
- Show a live "Key Details" preview (validity, users, site IDs, enabled
features) as a key is typed, pasted, or uploaded
- Fix cypress test setup broken by the paste-format default change, and
an unreliable FileReader-based base64 conversion in the test helpers
Writing back an unescaped cell's value unconditionally materialized a column the backend never populated as an explicit `undefined`, which the SASjs adapter rejects at submit time. dynamicCellValidation's column-name lookup (Object.keys(row)[column]) only worked by coincidence when every row had a full key set - fixed to use hot.colToProp instead.
saveTable() only resolved a live formula's computed value for character columns - coerceNumericRow (beforePaste/beforeAutofill) deliberately leaves a non-numeric, formula-looking value alone rather than forcing it to NaN, so a numeric column can hold a live formula too. Its raw `=...` text was reaching the backend, which a numeric SAS field can't parse.
saveTable() only resolved computed values for HARDFORMULA/SOFTFORMULA
columns - a formula typed, pasted, or "Apply as formula"'d into any
other character column still had its raw `=...` text submitted,
since HyperFormula recomputes the display without ever mutating the
underlying dataSource string.
- Add StartupCheckService to show deploy check progress on loading screen
with step-by-step status (appLoc check, Viya deploy, startup service)
- Add console logging throughout checkViyaDeploy and viyaMakedataSuccessfull
to diagnose deploy flow issues
- Fix missing return after resolve(false) in viyaMakedataSuccessfull when
folderId is undefined, which caused fall-through to getFolderMembers
with an undefined ID
- Fix Viya Folders API pagination: getFolderMembers now requests limit=500
so all members are returned (admin folder has 32 members, default page
size was hiding makedata)
- Fix error handler in viyaMakedataSuccessfull to resolve(false) instead of
reject() so the app falls back to setup screen on API errors
- Fix licence key whitespace: SAS makedata initialises keys as a single
space which is truthy in JS; trim keys before checking so empty keys
correctly fall back to free tier instead of triggering decryption errors
- Always set _debug=131 on makedata URLs (automatic and manual deploy)
so full debug output is available for the one-time setup service
- Add makedata completion polling after runMakedataInNewWindow so the app
auto-reloads when the makedata job self-deletes, instead of hanging
- Fix loading screen slider position from absolute to relative so it does
not obscure the startup check steps
- Add chunk_deploy.py to sasjs/utils for chunked Viya deploys, splitting
viya.sas by service and web file boundaries with correct %let path=
tracking per chunk
Closes#303, #200, #125
dataSourceRaw is now escaped the same way dataSource is, so Revert
restores the escaped text a user actually saw instead of the raw
backend string (which would otherwise evaluate live again).
beforeChange re-marks a cell revert_cells restores to its escaped
form, distinguishing that from a user typing a literal leading `'`
themselves via the write's source tag.
Uploaded file content is bulk external data, not user-typed input, so
a `=`-led value in a character column must be escaped the same way a
fresh backend load is - otherwise it silently evaluates as a live
formula now that formulas are enabled for every table.
Only backend-sourced `=`-led values are auto-escaped now; anything the
user types or pastes into a character column evaluates as a real
formula, and "Apply as formula" lets a user promote an auto-escaped
cell explicitly. Formulas are enabled on every table instead of only
ones with formula rules.
- Paste a formula using column names (e.g. "=A_COL * B_COL") into any
cell and have it translated to that row's cell references
("=B4 * C4") so HyperFormula can evaluate it - covers both a
grid-level paste and pasting directly into an open cell editor
- Only applies on tables that already have formulas enabled (an
existing HARDFORMULA/SOFTFORMULA column); otherwise the pasted text
is left untouched rather than becoming inert translated text
- New substituteColumnReferences() reuses parseFormulaRule's
boundary-matching helpers but deliberately skips DC.* variable
substitution, which only makes sense for admin-defined rule values
symget('_contextname') is concatenated directly into the PRX replacement
string in makedata.sas. A context name containing \ or $ would be
interpreted as a backreference or variable interpolation in the Perl
regex replacement. Escape both characters with a leading backslash in a
data step before embedding the value in the prxchange call.
- sasjs-configurator: rewrite _debug=131 comment to make explicit that
debug is ALWAYS enabled for the makedata service (runs once during
deployment, debug output always wanted for diagnostics). Remove the
commented-out debug-off line that caused reviewer confusion.
- viya-compute-context-details.model: extend Attributes with
allowXCMD, sessionInactiveTimeout (number), and make all attribute
fields optional. Add Environment interface with autoExecLines string
array. Add environment? to ComputeContextDetails (optional, not all
contexts expose it).
- automatic.component: rewrite extractRunAs to use
attributes.runServerAs directly per the actual Viya API response
shape. Remove all any casts and the non-existent runAsUserId /
environment.runAsUserId field probes.
- Escape embedded double quotes in DC.USER_NAME/DC.ORIG_VALUE formula
literals, preventing malformed HyperFormula expressions
- Extract cell-revert numeric parsing into resolveRevertedCellValue,
falling back to the raw text instead of writing NaN
- Guard addRow() against an empty dataSource before indexing into it
- Document the cross-repo assumption behind licence protocol-mismatch
detection
A key generated for one connection type (http/https) always fails to
decrypt on the other, surfacing only as a generic "invalid key" error
after a round trip to the backend. Detects the mismatch client-side
from the raw key text before the user submits, and disables the apply
button while it's showing.
dynamicCellValidation() built its editors/getdynamiccolvals payload
from a raw dataSource row clone, which always carries the client-only
EDIT_STATUS column - unlike saveTable()'s submit path, nothing
stripped it before the request went out. Extracted the strip logic
(previously private to classifyRow) into a shared withoutEditStatus
util and applied it here too.
@sasjs/cli's own nanoid override doesn't propagate to consumers - npm
only reads overrides from the root project being installed. Adding it
here directly, rather than a raw lockfile patch, so it survives future
npm install runs instead of silently reverting again.
Adds right, left, bottom, and bottom-left-corner resize handles
alongside the existing bottom-right corner, and raises resize handle
z-index above Handsontable's frozen header clones so clicks reach the
handle instead of the table underneath it.
A misconfigured Viya computeTasks deployment makes the Compute service
return a plain-text "Job error" body instead of JSON. @sasjs/adapter can't
parse it and resolves with that raw text as adapterResponse, so both
startup and manual deploy validation reported every expected field as
"not present" without ever showing the actual cause. Added
getMalformedAdapterResponseMessage to detect a non-object adapterResponse
and surface its real text instead.
Handsontable's Menu widget closes on any mouseup inside an item and
unconditionally preventDefault()s contextmenu, even for the info item,
which has no callback and exists purely to show read-only column details.
That made its text impossible to select or right-click-copy. Marked the
item isCommand: false and stop mousedown/mouseup/contextmenu/selectstart
from bubbling past its rendered content to the menu's own listeners.
The Filter/Edit/Upload buttons used btn-block (width: 100%), stretching
them wider than the Cancel/Add Row/Submit buttons shown during edit.
The back/viewboxes, title, and action-button columns were also locked to
equal 12-col grid thirds regardless of actual content, so the dataset
name/row-count wrapped even with visible free space on either side.
Switched the outer columns to size to their content (clr-col-*-auto) and
let the title column flex-grow into whatever space is left.
An unscoped "brace-expansion": "^5.0.9" override applies to every
occurrence in the tree, not just the vulnerable one - it was silently
replacing the 1.1.16 that glob's bundled minimatch@3.x actually needs,
breaking license-checker with "TypeError: expand is not a function".
Scoped it to the readdir-glob chain (exceljs -> archiver -> readdir-glob
-> minimatch@10.x), the same nested-override pattern already used here
for exceljs's own archiver/unzipper pins.
Fixed by patching only the 12 needed entries directly (9 @angular/*
packages to 20.3.27, brace-expansion to 5.0.9, fast-uri to 3.1.5, undici to 6.28.0) with exact registry resolved/integrity metadata, leaving the other 1925 packages untouched.
Incremental `npm install <pkg>@version` patches left the lockfile
internally inconsistent enough that CI's `npm ci` rejected it as out of
sync with package.json (a long list of unrelated nested deps reported
missing). Regenerated from scratch instead of patching further; verified
with a clean `npm ci`, matching the resolved versions from the prior fix.
- Angular packages (core/common/compiler/animations/forms/
platform-browser*/router/compiler-cli) all peer-depend on each other's
exact version - npm audit fix only bumped the three with a direct
advisory, leaving the rest mismatched; a plain `npm install` tolerated
it but CI's strict `npm ci` didn't. All now resolve to 20.3.27.
- undici/fast-uri updated via npm audit fix; brace-expansion pinned to
^5.0.9 via overrides (npm's resolver wasn't proactively bumping a
nested transitive dep that already satisfied its declared range)
npm audit fix only bumped @angular/core/common/compiler to 20.3.27 (the
packages with a direct advisory), leaving animations/forms/platform-browser*/
router at 20.3.26 - each peer-depends on the others' exact version, so the
mismatch passed a lenient `npm install` locally but failed CI's strict
`npm ci`. All seven now resolve to 20.3.27.
- Angular packages resolve within their existing ^20.3.26 range up to
20.3.27, which patches two advisories (HttpTransferCache cache-key
ambiguity, i18n XSS via event-handler attributes)
- brace-expansion (via exceljs -> archiver -> readdir-glob -> minimatch)
needed an override to ^5.0.9 - npm's resolver won't proactively bump a
nested transitive dep that already satisfies its declared semver range,
even when a patched version exists inside that same range
- Formulas' HyperFormula sync resolves a dotted `data` key differently
than getDataAtRowProp/datamap.get() - dataDotNotation: false is needed
for the renamed dc.row_status column to work as a live cell reference
- saveTable() now submits the live computed value for a formula cell, not
the raw '=...' string still sitting in dataSource
- Mirrors the review page's toggle: getstagetable.sas now also webouts
fmt_stagetable (SAS-formatted), selectFormattedRows() picks between it
and the raw stagetable, falling back to raw if fmt data is absent
- Adds stage.cy.ts e2e coverage and wires it into both build.yaml and
release.yaml's cypress --spec lists (previously out of sync with
each other on viewbox.cy.ts too)
- addRow()/insertRowAtPosition() use hot.alter() so HARDFORMULA/SOFTFORMULA
values realign on insert instead of going stale
- New hidden EDIT_STATUS column gives DC.ROW_STATUS a real, live cell
reference; row-header +/-/~ indicator now translates visual->physical
row index so it stays correct when the grid is sorted
- UserService is now providedIn: 'root' instead of module-scoped, fixing
DC.USER_NAME (and any other lazy-module reader) always seeing an unset user
- Column-info dropdown shows the applied formula ("√x=<formula>")
# [7.12.0](https://git.datacontroller.io/dc/dc/compare/v7.11.0...v7.12.0) (2026-07-28)
### Bug Fixes
* adding REGEX validations to mpe_x_test ([bd798b4](bd798b424a))
* default value for label ([a3e46a9](a3e46a968e))
* **deps:** retarget Angular upgrade to 20, not 21 (CI install was broken) ([cac9244](cac9244f92))
* ensure only one REGEX applies at a time ([8fb58eb](8fb58eb36e))
* ensure that no assets (including og links) ever fetch from external sources ([33dcb98](33dcb989d3))
* include peer dependencies in package-lock for npm ci in pipeline ([b51c770](b51c770782))
* licensecheker ([f60bcef](f60bcef583))
* **lint:** remove redundant optional chaining ([d881290](d881290618))
* optimisation, renamed values for DDTYPE to save space ([cfb60e5](cfb60e5e4b))
* patch npm audit vulnerabilities in sas and client dependencies ([e22edf7](e22edf7ed3))
* **query:** isolate viewbox filter state from the base table's ([7a35cf4](7a35cf4a45))
* regenerate client lockfile to resolve Angular peer-dependency drift breaking npm ci ([05fe474](05fe4744d5))
* **regex:** special missing handling ([180c247](180c2477ed))
* removing low severity warning in npm audit ([2a771bb](2a771bb91a))
* removing thousand seperator from plain numerics in EDIT mode ([0392a81](0392a81cbd))
* **validations:** parse SAS PRX /pattern/flags syntax in HARDREGEX/SOFTREGEX ([7ed3730](7ed3730ae3))
### Features
* **docs:** adding agents.md and docs for RLS ([359d833](359d833406))
* **editor:** add HARDREGEX/SOFTREGEX validation rules ([17e4802](17e4802895))
* **editor:** evaluate HARDREGEX/SOFTREGEX independently instead of hard-wins precedence ([57db117](57db1179a9))
* **editor:** show applied HARDREGEX/SOFTREGEX pattern in column info dropdown ([39c8855](39c8855f37))
* **regex:** backend validations on regex strings ([d2c93a4](d2c93a46fa))
* using ALL libraries as validation in MPE_SECURITY. Closes [#279](#279) ([62ff0ae](62ff0aee4a))
* validation checks to prevent incompatible RLS rules (eg REPLACE load type). Closes [#211](#211) ([ea00c5a](ea00c5afad))
* validation on RLS for REPLACE, + docs + tests. Closes [#211](#211) ([7378f3b](7378f3ba30))
Both rules can now apply to one column: HARDREGEX still blocks submission
and takes its own tooltip, but SOFTREGEX is evaluated (and shown) whenever
HARDREGEX passes, instead of being silently suppressed whenever HARDREGEX
was merely present. Column-header info dropdown labels each rule separately
when a column has both.
Replace one-shot .then()/for-loop matching with retrying .should()/cy.contains()
lookups, drop stale cached DOM node references (viyaLib) in favor of live re-queries.
Verifies parseRegexRule against a corpus of RULE_VALUEs collected from an
existing MPE_VALIDATIONS table - none use PRX-only syntax, so behavior is
confirmed to match SAS PRX exactly, not just "doesn't throw". Pins two
data quirks found along the way (a range vs. literal-hyphen character
class, and a double-escaped lookahead that's a no-op in both engines)
rather than silently treating either as a bug to fix.
Adds HARDFORMULA/SOFTFORMULA DQ rule types, backed by Handsontable's
formulas plugin (gated per-table via hasFormulaRules). parseFormulaRule
substitutes column names with row-relative cell references (quote-aware,
blank-boundary matching per spec) and resolves DC.USER_NAME/DC.ORIG_VALUE
to literal values; applyFormulaRules injects the computed formula per row,
reusing the existing READONLY mechanism for HARDFORMULA.
DC.ROW_STATUS/EDIT_STATUS deliberately deferred - would require prepending
COLHEADERS, which headerColumns has undocumented positional coupling to
elsewhere in editor.component.ts.
RULE_VALUE is authored in PRX delimiter form because prxparse() requires
it, but HARDREGEX, the SOFTREGEX grid renderer, and failsSoftRegex were
all passing that string straight into `new RegExp()`, so the delimiters
and flags were matched as literal characters instead of applied - making
these rules silently never match real data.
Adds parseRegexRule (extracted, tested independently) to strip the
delimiters, apply flags, hoist a leading (?i) modifier, and translate
\Q...\E and \A/\z to their JS equivalents. Atomic groups and possessive
quantifiers are left unfixed (documented, fail-safe) - translating them
risks renumbering the pattern's own capture groups.
Also switches the REGEX_HARD_COL/REGEX_SOFT_COL mock rules to the
delimited form so editor.cy.ts's existing e2e coverage actually exercises
this path.
Bumps @angular/* to 21 (passing through 20 as a transient step, since
vendored Clarity has no Angular-20-compatible release) and
@typescript-eslint to 8.65.0 for TS 5.9 compat. Applies the required
ng update migrations, including *ngIf/*ngFor/*ngSwitch -> control-flow
syntax across 37 templates.
Two new DQ rule types apply regular expressions to cell values:
HARDREGEX blocks submission on a non-matching value (same path as the
existing CASE/MINVAL/MAXVAL rules); SOFTREGEX is display-only — a
non-matching value gets a yellow warning cell but can still submit,
so it's wired as a grid renderer rather than a validator, and mirrored
in the edit-record modal (which has no grid renderer to hook into) via
DcValidator.failsSoftRegex. Both rules exempt blank and SAS special
missing values, and fail open on a malformed pattern rather than
blocking every submission on that column. HARDREGEX takes precedence
when both rules apply to the same column, so a failing value renders
red/blocked, never yellow.
An invalid cell no longer blocks submission if its row is marked for
delete, since those values are about to be removed anyway. Primary key
columns are exempt from the exemption: the key identifies which record
to delete, and a blank/invalid PK would otherwise also slip past the
separate duplicate-key check. Toggling the delete flag now re-validates
the row immediately so invalid highlights update without waiting for
submit.
Header matching in searchDataInExcel() was case-insensitive but not
whitespace-tolerant, so a header like " SOME_CHAR" (an easy defect to
pick up via copy-paste from another spreadsheet/system) was reported as
a missing column and aborted the whole upload.
- trim() the cell value before matching, alongside the existing
toLowerCase() normalization
- add spreadsheet-util.spec.ts (no prior spec file existed) covering
the trimmed match and a regression guard that a genuinely different
header is still correctly reported missing
- add a Cypress case + fixture in excel.cy.ts covering the same defect
end to end
getdata.sas now sends one COLTYPE JSON-object string per cols[] row (via
a LEFT JOIN on the real dataset's columns) instead of one comma-joined
sasparams.COLTYPE string, avoiding the 32K cats() buffer overflow on
wide tables (#253). Adapt the client accordingly:
- parseColTypeRow replaces parseColType; DcValidator builds validation
rules per cols[] entry instead of from sasparams.COLTYPE
- _____DELETE__THIS__RECORD_____'s rule is never present in cols[] (it's
a client-only synthetic column %mp_getcols doesn't know about), so
it's now hardcoded via deleteRecordColumnRule.ts
- cols[] isn't guaranteed to arrive in COLHEADERS/VARNUM order (backend
can serialize it alphabetically by NAME) but editor.component.ts pairs
rules with headers positionally, so DcValidator now sorts cols by
VARNUM before building rules
- update sas/mocks/sasjs getdata.js to the new per-column COLTYPE shape
- add Cypress coverage for the delete-record dropdown and an ordinary
dropdown column end to end
Brings in the SAS-side fixes for issue #253 (getdata.sas passing
variable formats via a vertical cols.COLTYPE array instead of a
single concatenated sasparams.COLTYPE string, which overflowed SAS's
32K buffer on wide datasets) plus related label/datadictionary fixes.
No client TypeScript files are touched by this merge.
Adds a URL-driven toggle (?labels=true, mirroring embed=va) that
swaps grid column headers between NAME and LABEL in both the Viewer
and Editor, with a "Show labels"/"Show names" item in each grid's
right-click context menu. The header-info popup now shows NAME first
regardless of display mode. Sorting/filtering/data binding stay
NAME-based throughout — only the displayed header text changes.
New shared, unit-tested utils (parse-labels-param, display-col-headers,
col-info-html) back both pages. Mock data (viewdata.js, getdata.js)
gains a couple of columns with LABEL != NAME plus a blank-LABEL case,
covered by a new viewer-labels.cy.ts e2e spec now wired into CI.
Also fixes a pre-existing gap in cypress/support/commands.ts: the
isLoggedIn/loginAndUpdateValidKey custom commands had no Cypress
Chainable type augmentation, and moment was imported as a namespace
import despite its `export =` typing making it uncallable that way.
Bump handsontable and @handsontable/angular-wrapper 17.1 -> 18.0 and
adapt to its stricter TypeScript-core types: moved handsontable/common
and deep walkontable imports to their new public export paths, added
explicit generics/casts where getData()/getCellMeta()/getSelected() are
now typed unknown/nullable instead of the loose v17 shapes, and updated
the AutocompleteEditor override to the new open() signature.
Also allowlists @handsontable/angular-wrapper@18.0.0 and
handsontable@18.0.0 in licenseChecker.js — both versions report the
same non-SPDX "SEE LICENSE IN LICENSE.txt" field that license-checker
mis-resolves as a disallowed custom license, same as prior versions.
This document explains how the Data Controller backend loads staged data into target tables: the load-type dispatch, the internals of the `%bitemporal_dataloader` macro, and a detailed description of the REPLACE load type. For user-facing load type documentation, see [docs.datacontroller.io](https://docs.datacontroller.io/).
## Overview
Every table registered for loading has a current record in `&mpelib..MPE_TABLES` with a `LOADTYPE` (selectbox values are seeded in `mpe_makedata.sas`):
| LOADTYPE | Loader used | History kept |
|---|---|---|
| `UPDATE` | `%bitemporal_dataloader` (no temporal vars) | None — changed records are deleted and re-appended |
| `REPLACE` | Inline code in `%mpe_targetloader` (does **not** use `%bitemporal_dataloader`) | None — entire table wiped and reloaded |
| `TXTEMPORAL` | `%bitemporal_dataloader` (technical time only) | SCD2-style, technical (transaction) time |
| `BITEMPORAL` | `%bitemporal_dataloader` (business + technical time) | Full two-dimensional history |
The relevant `MPE_TABLES` columns read by the loader are: `LOADTYPE`, `BUSKEY` (primary key, space-separated, excluding temporal columns), `VAR_TXFROM` / `VAR_TXTO` (technical validity), `VAR_BUSFROM` / `VAR_BUSTO` (business validity), `VAR_PROCESSED` (processed timestamp column), `RK_UNDERLYING` (retained-key generation), `CLOSE_VARS`, and `AUDIT_LIBDS` (defaults to `&dclib..MPE_AUDIT`).
## Request Flow
```mermaid
flowchart TD
A[User submits changeset\neditors/stagedata.sas] --> B[Staging package written to\n&mpelocapprovals/&LOAD_REF\nCSV + jsdata]
B --> C[Approval workflow\nMPE_SUBMIT / MPE_REVIEW]
C --> D{auditors/postdata.sas}
D -->|action=SHOW_DIFFS| E["%mpe_targetloader(LOADTARGET=NO)\nbuilds work.outds_add / outds_mod / outds_del\nfor the diff screen only"]
D -->|action=APPROVE_TABLE| F["%mpe_targetloader(LOADTARGET=YES)\nactual load"]
E --> G[Diff CSV + TEMPDIFFS stored\nin approval package]
F --> H{LOADTYPE from\nMPE_TABLES}
H -->|UPDATE / TXTEMPORAL / BITEMPORAL| I["%bitemporal_dataloader"]
H -->|FORMAT_CAT| J["%mp_loadformat"]
H -->|REPLACE| K[Inline delete-all + append\nin %mpe_targetloader]
```
`%mpe_targetloader` (`sas/sasjs/macros/mpe_targetloader.sas`) is the single dispatch point. It reads the current `MPE_TABLES` record (`&dc_dttmtfmt. lt tx_to`), aborts if the table is not registered (or has duplicate config records), and routes to the loader. Note the two-phase design: `LOADTARGET=NO` prepares the intermediate `outds_*` tables so the approver can review diffs; `LOADTARGET=YES` performs the destructive load. Both phases run the same preparation logic, so the reviewed diffs correspond to what is actually applied.
## The Temporal Model
Bitemporal tables carry two independent time dimensions:
* **Business time** (`bus_from` / `bus_to`) — when the fact was true in the real world. Present on **both** staging and base tables.
* **Technical time** (`tech_from` / `tech_to`, a.k.a. transaction time) — when the record was known to the database. Present on the **base table only**; the loader stamps these itself.
All validity is expressed with **half-open intervals** (`from <= t < to`). Queries against bitemporal tables need two conditions and must not use `BETWEEN` or `from LE t LE to` (the latter excludes boundary records — see the macro header for background):
```sas
where&bus_from le [tstamp] lt &bus_to
and&tx_from le [tstamp] lt &tx_to
```
"Current" records have `tech_to` set to the high date (`'31DEC9999:23:59:59'dt` when called from `%mpe_targetloader`). Closing out a record means setting `tech_to = now` — records are never physically deleted from a temporal table, they are superseded.
## Key Components
| Component | Location | Role |
|---|---|---|
| `%mpe_targetloader` | `sas/sasjs/macros/mpe_targetloader.sas` | Reads `MPE_TABLES` config and dispatches per LOADTYPE; implements REPLACE inline |
| `%bitemporal_closeouts` | `sas/sasjs/macros/bitemporal_closeouts.sas` | Closes out (sets `tech_to=now`) live records matching a key |
| `%mp_retainedkey` | SASjs core | Generates retained (surrogate) keys when `RK_UNDERLYING` is configured |
| `%mp_rowhash` | SASjs core | MD5 hash of non-temporal columns, used for change detection |
| `%mp_storediffs` | SASjs core | Writes row-level audit records to `AUDIT_LIBDS` |
| `MPE_DATALOADS` | `&dclib` | Load log (counts, duration, macro version, user) — written only by `%bitemporal_dataloader` |
| `MPE_LOCKANYTABLE` | `&dclib` | Lock control table used by `%mp_lockanytable` |
## `%bitemporal_dataloader` Execution Flow
The macro builds a series of `work.bitemp*` intermediate tables, then applies the changes in two target-table operations (closeout, then append) under a single lock.
### 1. Pre-checks and setup
* Early return if the staging table is empty; hard abort if `&syscc > 0`.
*`CLOSE_VARS` is not supported on REDSHIFT / POSTGRES / SNOWFLAKE engines (returns with a NOTE).
* A zero-row snapshot of the base table (`&basecopy`) is taken with `data ... set base; stop;` — this doubles as a lock check, since metadata functions fail against a locked table.
*`proc contents` on the base table feeds column lists. Columns are split into character and numeric lists for hashing (`%mp_rowhash` hashes the two types via separate arrays). Temporal columns, the processed column, and the delete-flag column are excluded from the hash.
* Table names containing `___TMP___` or `_____` are rejected (they would collide with generated temp columns).
### 2. Locking
For `LOADTARGET=YES`, the base table (and audit table, if configured) is locked via `%mp_lockanytable` before any staging prep, because the load is a two-part update (closeouts + append) and must not interleave with another load.
* If `RK_UNDERLYING` is configured, `%mp_retainedkey` maps business keys to retained keys (filtering the base lookup to live records, `&now < tech_to`, for temporal types); otherwise the staging table is used as-is.
*`bus_from` / `bus_to` overrides are applied if provided; the processed column is stamped with the load timestamp (`now`). Even with `processed=0`, a column literally named `PROCESSED_DTTM` on the base table is used if present.
* The MD5 change-detection hash is computed for every staged record.
* If the staging table contains the delete-flag column (`_____DELETE__THIS__RECORD_____`), rows flagged `"Yes"` are diverted to `&outds_del` and closed out via `%bitemporal_closeouts` (PK is taken as `bus_from` + business key). The remaining rows continue as `bitemp0_append`.
### 4. CLOSE_VARS closeout
When `CLOSE_VARS` (a subset of the PK) is supplied, live base records whose CLOSE_VARS values appear in staging but whose full PK does **not** are closed out. This handles "this group was fully reloaded, so anything missing was removed" semantics without reloading the whole table.
### 5. Uniqueness check
If `CHECK_UNIQUENESS=YES` (the default) or business-date overrides are in play, the staging table is sorted `nodupkey` by the PK; a row-count mismatch aborts the load and releases the locks. The staging table must be a unique snapshot of the business key at one point in business time.
### 6. Base extract (`work.bitemp0_base`)
Only base records matching staged PKs are extracted — for temporal load types, only **currently live** records (`now < tech_to`). A left join from the staged keys to the base produces `___TMP___NEW_FLG` to identify brand-new keys. This is engine-specific:
* **OLEDB (SQL Server)** — staged keys are pushed to a `##global` temp table and joined via explicit pass-through.
* **REDSHIFT / POSTGRES / SNOWFLAKE** — an in-database temp table is created `like` the base, stripped to PK + an added `md5 varchar(32)` column, loaded with staged keys, and joined via pass-through. Snowflake uses transient tables; Redshift gets `alter sortkey none` plus any `DCBL_REDSH` config options from `MPE_CONFIG`.
* **CAS** — a FedSQL join against a CASUSER copy.
* **BASE/other** — plain PROC SQL in SAS.
### 7. Change classification
* **`&outds_add`** — staged records flagged as new (no base match). They get `tech_from=now`, `tech_to=high_date`.
* **`work.bitemp1_current`** — matched base records, re-hashed so hashes are comparable.
* **Inserts (BITEMPORAL only)** — a staged record whose business range falls strictly *inside* an existing record's range splits the existing record into a "before" and "after" segment (`bitemp3_inserts` / `bitemp3a_inserts`). The split segments replace the original in the comparison set (`bitemp3b_newbase`).
* **Updates (`bitemp4*`)** — staged records matching a base PK but with a different hash or different business dates. Base and staged versions are stacked, deduplicated, then aligned in **two passes** over the business timeline per key: a forward pass (carry `bus_from` forward across identical hashes; a staged record trims the preceding base record's range) and a reverse pass (carry `bus_to` back; records fully subsumed by the new version are deleted). Records that end up byte-identical to what is already stored are dropped via a hash lookup that includes the business dates (`bitemp5a_lkp` / `bitemp5b_updates`, BITEMPORAL only).
### 8. Closeout application
Changed records are closed out in the target before the new versions are appended. SAS SQL has no UPDATE-with-join, so a correlated `EXISTS` subquery against `work.bitemp5d_subquery` is used (pushed in-database as a temp table for OLEDB / REDSHIFT / POSTGRES / SNOWFLAKE):
* **BITEMPORAL** — per key, the closeout range is `min(bus_from)` / `max(bus_to)` of the changed records: `update base set tech_to=now (, processed=now) where tech_from <= now < tech_to and exists (key match and base.bus_from >= min and base.bus_to <= max)`.
* **TXTEMPORAL** — same, on key only.
* **UPDATE** — changed records are physically `delete`d (they are re-appended in the next step). On CAS this uses `table.deleteRows` with a whereTable; temporal types are not supported on CAS and abort.
* **BUSTEMPORAL** — closeouts are not implemented; the macro aborts at this point ("BUSTEMPORAL NOT YET SUPPORTED").
### 9. Append, unlock, audit, log
* The union of modified + new records, deduplicated on all columns (`bitemp6_unique`), is appended to the base table (`proc append ... force nowarn`; CAS appends via a varchar-casting step; Redshift applies `DCBL_REDSH` options). Locks are then released.
* If `outds_audit` is set (always, from `%mpe_targetloader` — `AUDIT_LIBDS` defaulting to `&dclib..MPE_AUDIT`), `%mp_storediffs` compares the pre-load snapshot with the applied changes and appends row-level audit records; modified-row entries with no actual value change are removed (`MOVE_TYPE="M" and IS_PK=0 and IS_DIFF=0`).
* A summary row is inserted into `&dclib..MPE_DATALOADS` (libref, dsn, etlsource, loadtype, changed/new/deleted counts, duration, macro version, user, timestamp) unless `LOG=0`.
## REPLACE Load Type
REPLACE is the simplest load type and is deliberately **not** routed through `%bitemporal_dataloader` — it is implemented inline in `%mpe_targetloader` and simply wipes the target table (`delete * from`) and re-appends the staging table verbatim, with no history, change detection, key matching, audit rows, or `MPE_DATALOADS` logging. See [replace-load-type.md](replace-load-type.md) for the full deep dive.
Unit tests for the loaders live next to the macros: `sas/sasjs/macros/bitemporal_dataloader.test.[1-4].sas` and `sas/sasjs/macros/mpe_targetloader.test.sas` (REPLACE loadtype), executed as sasjs tests. See [testing.md](testing.md) for how to run them (`npm run 4gl` then `sasjs test -t 4gl` from the `sas/` directory), and remember `sasjs lint` after touching any `.sas` files.
Whenever any `package.json` (root, `client/`, or `sas/`) or lockfile is modified, run the same checks CI runs before pushing:
1.**npm audit** (must be clean for prod deps):
```bash
npm audit --omit=dev # in repo root
cd sas && npm audit --omit=dev
cd ../client && npm audit --omit=dev
```
Fix with `npm audit fix`, targeted `overrides` in `package.json`, or version bumps — never `npm audit fix --force` blindly, as it can introduce breaking changes.
2. **License checker** (client only):
```bash
cd client && npm run license-checker
```
If a new dependency fails, either add its SPDX id to the `onlyAllow` list in `client/licenseChecker.js` (if the license is acceptable, e.g. permissive ones like `BlueOak-1.0.0`) or add the specific package to `excludePackages` with justification. Data Controller ships on-prem, so only OSI-approved permissive licenses are acceptable for production dependencies.
Both checks run in `.gitea/workflows/build.yaml` (`Check audit` and `Licence checker` steps) and will fail the build if skipped locally.
This document describes how the regex validation rules work internally. For user-facing documentation see `docs/dcc-validations.md` in the `docs.datacontroller.io` repo.
## Overview
Two validation rule types in `MPE_VALIDATIONS` validate cell values against a regular expression supplied in `RULE_VALUE`:
-`HARDREGEX` - submission-blocking. A failing value is rejected by the cell validator and painted red (HOT's own `htInvalid` class), so the row cannot be submitted.
-`SOFTREGEX` - display-only warning. A failing value is painted yellow (`dc-warning-cell` class) with a tooltip, but submission is not blocked.
Both rule types are selectable in the MPE_VALIDATIONS RULE_TYPE dropdown; they were added to the selectbox seed data in `sas/sasjs/macros/mpe_makedata.sas` and via the optional migration `sas/sasjs/db/migrations/20260720_v7.12_release.sas`. `RULE_VALUE` is limited to 128 characters, which constrains very long patterns.
## Config-time validation (SAS side)
`sas/sasjs/services/hooks/mpe_validations_postedit.sas` runs `prxparse()` on any staged HARDREGEX/SOFTREGEX rule value and aborts the edit with a list of offending `libref.table.column` references if the pattern is invalid. This is a best-effort syntax check to catch typos at config time; an empty pattern is treated as valid (it matches everything in JS). Rows marked for delete are skipped.
Because patterns must pass `prxparse`, rule values are authored in the SAS PRX delimiter form `/pattern/flags` (although a bare pattern is also accepted for backwards compatibility).
## Frontend evaluation
All regex handling lives in the client; there is no server-side re-validation of data values. The per-cell decision flow:
```mermaid
flowchart TD
A[Cell value] --> B{Row marked for delete<br/>and not a PK column?}
B -- Yes --> Z[No validation / no warning]
B -- No --> C{isRegexRuleExempt?<br/>blank, or "." on a numeric column}
C -- Yes --> Z
C -- No --> D{HARDREGEX rule on column?}
D -- Yes --> E{Pattern matches?}
E -- No --> F[Invalid: submission blocked,<br/>red htInvalid + REGEX tooltip]
E -- Yes --> J[Valid]
D -- No --> G{SOFTREGEX rule on column?}
G -- Yes --> H{Pattern matches?}
H -- No --> I[Warning: yellow dc-warning-cell<br/>+ REGEX tooltip, submission allowed]
H -- Yes --> J[Valid]
G -- No --> J
```
A malformed pattern never reaches this flow: it is treated as always-valid (HARDREGEX) / never-warn (SOFTREGEX) with a `console.warn`, rather than breaking the editor.
Converts an authored SAS PRX pattern into a JavaScript `RegExp`:
1. If the value matches `/^\/(.*)\/([a-z]*)$/s`, the delimiters are stripped and the trailing flags are passed to `new RegExp(body, flags)`. Otherwise the value is used as-is (backwards compatibility with bare patterns).
2. Three mechanical Perl→JS translations are applied to the body:
- a leading `(?i)` inline modifier is removed and folded into the `i` flag;
-`\Q...\E` literal sequences are replaced with escaped literal text;
-`\A` → `^` and `\z` → `(?![\s\S])` (end-of-string anchor).
3. Perl-only constructs that would need a capture-group-renumbering rewrite (atomic groups `(?>...)`, possessive quantifiers `a++`) are deliberately NOT translated. They throw from `new RegExp`, and every caller treats a throw as "always valid / never warn" (see below) rather than breaking the editor.
Blank values (`undefined`, `null`, `''`) are exempt from pattern matching on any column type - use a separate NOTNULL rule if populated values must also be enforced. On numeric columns the plain SAS missing (`.`) is also exempt; special missings (`.A`-`.Z`, `._`, bare letters) are NOT exempt anywhere - being deliberately set, they are real values the pattern must match (and on a character column even `.` is real text). `isSpecialMissing` from `@sasjs/utils` is deliberately not used: its optional-dot regex would exempt any single-letter character value ("d", "z") before the regex ever ran. The check takes an `isNumeric` flag, passed by all callers (the dq validator via `dqValidate(rules, value, colType === 'numeric')`, the warning renderer via a `makeRegexWarningRenderer` argument, and `failsSoftRegex` via the column's HOT type).
### HARDREGEX - blocking validation
`HARDREGEX` is implemented as a cell validator in `client/src/app/shared/dc-validator/validations/dq-validation.ts`. It returns `true` (valid) for exempt values and for patterns that fail to compile (with a `console.warn`), and otherwise returns `parseRegexRule(ruleValue).test(value.toString())`. Returning `false` makes HOT mark the cell invalid, block submission, and paint it red via its standard `htInvalid` styling.
### SOFTREGEX - warning renderer
`client/src/app/editor/utils/regex-warning-renderer.ts` builds a display-only Handsontable renderer (registered per-column by `DcValidator.setupRules` in `client/src/app/shared/dc-validator/dc-validator.ts`). It never returns false; it only:
- adds a `REGEX: <pattern>` tooltip (`td.title`) when a rule fails;
- adds the yellow `dc-warning-cell` class when only SOFTREGEX fails.
`DcValidator.failsSoftRegex(col, value)` provides the same logic outside the grid (e.g. the edit-record screen).
### Precedence: HARD and SOFT on the same column
Only one regex ever runs per column. If a HARDREGEX rule exists, SOFTREGEX is ignored entirely - never compiled, never evaluated - regardless of whether individual cell values pass or fail the hard rule. A value failing HARDREGEX gets the red invalid styling (blocking submission) plus a `REGEX: <pattern>` tooltip; a SOFTREGEX-only column warns in yellow without blocking. This holds in both the renderer and `failsSoftRegex`.
### Other behaviour
- Rows marked for delete (`_____DELETE__THIS__RECORD_____ = 'Yes'`) are not warned/validated by the renderer (except primary key columns, which still validate).
- A malformed pattern never breaks the editor: the dq validator treats it as always-valid and the renderer disables the warning, logging to the console instead.
- The pattern is used as authored - it is NOT auto-anchored. Authors must include `^`/`$` to match the entire cell value.
- Column info: `client/src/app/shared/utils/col-info-html.ts` shows the applied pattern in the column-info dropdown - the HARDREGEX pattern if one exists (it is the rule actually applied when both are present), otherwise the SOFTREGEX pattern, otherwise nothing.
- SAS side: `sas/sasjs/services/editors/stagedata.test.3.sas`, plus seed data in `mpe_makedata.sas`: HARDREGEX "SOME_CHAR must contain 'the' or 'data'" (`/the|data/i`), SOFTREGEX "SOME_CHAR should contain the letter 't'", SOFTREGEX on PRIMARY_KEY_FIELD (`/^\d+$/` - yellow if the key contains a decimal), and HARDREGEX on SOME_SHORTNUM (`/^(??$).*/` - values 1-5 blocked; generated data starts at 6 so demos aren't blocked accidentally).
**Do not edit `CHANGELOG.md` by hand, and do not bump the version in `package.json` yourself.** Both are generated automatically by the release pipeline. Manual edits produce duplicate/colliding version sections and merge conflicts, and they are pointless because the next release overwrites them anyway.
## How releases work
Releases are driven by [semantic-release](https://semantic-release.gitbook.io/) from the `release` job in `.gitea/workflows/release.yaml`, which runs on every push to `main` (after the build/test jobs pass). The configuration lives in `.releaserc` at the repo root.
The plugin chain (`.releaserc``plugins`) does the following on each release:
1.`@semantic-release/commit-analyzer` - inspects Conventional Commit messages since the last tag to decide the next semantic version (`fix:` -> patch, `feat:` -> minor, breaking change -> major).
2.`@semantic-release/release-notes-generator` - builds the release notes from those commits.
3.`@semantic-release/changelog` - **writes the new section into `CHANGELOG.md`**.
4.`@semantic-release/npm` - updates the `version` in `package.json` (it does not publish; the package is `private`).
5.`@semantic-release/git` - commits `CHANGELOG.md` and `package.json` back to `main` as `chore(release): <version> [skip ci]` and tags it.
6.`@saithodev/semantic-release-gitea` - creates the Gitea release.
The build assets (frontend zip, SAS 9 / Viya / SASjs Server deployment files) are attached to the release afterwards by the workflow's own `Upload assets to release` step using the Gitea API (`curl`), not by the plugin. Before creating the release the workflow first runs `semantic-release --dry-run` and aborts the job if there are no releasable changes since the last tag.
Because the changelog entry and version bump are committed by the pipeline (step 5), they must **not** exist in your working tree beforehand. If they do, the release commit collides with them.
## What this means for you
- Never add, remove, or reorder entries in `CHANGELOG.md`.
- Never change `version` in `package.json`.
- Control what appears in the changelog through your **commit messages** (Conventional Commits: `fix:`, `feat:`, `feat!:`/`BREAKING CHANGE:`, plus scopes like `fix(editor): ...`). The scope and description become the changelog line.
- If you find hand-written entries in `CHANGELOG.md` in the working tree, revert them (`git checkout CHANGELOG.md`).
- CI does not use any local release script - the pipeline installs and runs `semantic-release` directly.
This document describes the REPLACE load type in detail. For the overall loader architecture, the temporal model, and the other load types, see [bitemporal-dataloader.md](bitemporal-dataloader.md).
## Overview
REPLACE is the simplest load type and is deliberately **not** routed through `%bitemporal_dataloader`. It is implemented inline in `%mpe_targetloader` (`sas/sasjs/macros/mpe_targetloader.sas`, search for `&loadtype=REPLACE`). There is no history, no change detection, no key matching — the target table is wiped and reloaded from the staging table verbatim.
## Actual load (`LOADTARGET=YES`)
```sas
%mp_lockanytable(LOCK, lib=&lib, ds=&ds, ...)
data WORK.&STAGING_DS;
set WORK.&STAGING_DS;
/* only if the target contains the MPE_TABLES.VAR_PROCESSED variable: */
&VAR_PROCESSED = &now;
drop _____DELETE__THIS__RECORD_____;
run;
%if&engine_type=CAS %then%do;
/* prep first: cast varchar columns in a CASUSER copy of staging */
/* unlock + abort if any error so far - nothing destructive done yet */
%mp_abort(iftrue= (&syscc>0) ...)
proc sql;
delete * from&libds;
quit;
proc append base=&libds data=WORK.&STAGING_DS force nowarn; run;
%end;
%mp_lockanytable(UNLOCK, lib=&lib, ds=&ds, ...)
```
Step by step:
1.**Lock** — the target is locked via `%mp_lockanytable` (control table `&dclib..MPE_LOCKANYTABLE`). This is the only concurrency guard for the whole operation.
2.**Staging recopy** — the staging dataset is rewritten in place: the processed-timestamp column is set to the approval timestamp **only if** the target table contains the variable named by `MPE_TABLES.VAR_PROCESSED` (unlike `%bitemporal_dataloader`, there is no `PROCESSED_DTTM` fallback). The delete-flag column `_____DELETE__THIS__RECORD_____` is unconditionally dropped — per-record delete semantics do not exist in REPLACE (everything is deleted anyway), so a delete flag submitted by the user is silently discarded (a `drop` of a non-existent variable just produces a warning).
3.**Prepare the load (CAS only)** — fixed char variables cannot be appended to CAS varchar columns, so the staging table is first copied to a CASUSER temp table with every column that is varchar in the target (and present in staging) redeclared as `varchar(*)` via generated `length` / `rename` / assignment statements. This mirrors the CAS append in `%bitemporal_dataloader`.
4.**Pre-destructive abort check** — if `&syscc > 0` after all preparation, the target is unlocked and the macro aborts via `%mp_abort`. Nothing destructive has happened at this point, so a failed prep leaves the target intact (and without a stale lock).
5.**Delete all rows** — every row is removed from the target while preserving structure, indexes and metadata, deliberately as the last step before the append (to minimise the time the target sits empty). On most engines this is `proc sql; delete * from &libds;` (passed through as a `DELETE` on database libraries). CAS tables do not support SQL deletes, so on the CAS engine the table is truncated instead via the `table.deleteRows` action with `where="1=1"` (same approach as `%bitemporal_closeouts`), with the libref passed as the caslib.
6.**Append staged rows** — the entire staging table is appended. On most engines this is `proc append ... force nowarn` (`force` allows the append to proceed despite attribute mismatches, so lengths/formats may be coerced or values truncated, and columns present in only one side are handled; `nowarn` suppresses the associated warnings). On CAS the pre-cast CASUSER temp table is appended with `data &libds (append=yes) / sessref=dcsession` and then dropped.
7.**Unlock**.
## Diff screen (`LOADTARGET=NO`)
When the approver reviews a REPLACE submission, no real comparison is performed:
```sas
/* is full replace so treat all staged records as new in diff screen */
data work.outds_mod work.outds_add;
set work.&staging_ds;
output work.outds_add; /* every staged record is "NEW" */
run;/* outds_mod stays empty */
/* previous table will be considered fully deleted */
data work.outds_del;
set&lib..&ds; /* every existing record is "DELETED" */
run;
```
The approval screen therefore always shows full-table turnover: the entire current table as deleted and the entire staging table as added, with no "modified" records and no original/current comparison.
## What REPLACE does not do
Because it bypasses `%bitemporal_dataloader`, none of the following apply to REPLACE:
* **No row-level audit** — nothing is written to `AUDIT_LIBDS` / `MPE_AUDIT`. The record of the change is the approval package itself (staged CSV and the `TEMPDIFFS` CSV stored under `&mpelocapprovals/&LOAD_REF`).
* **No `MPE_DATALOADS` log entry** — the load-log insert lives inside `%bitemporal_dataloader`, so REPLACE loads do not appear in load history. (Consequently the SHOW_DIFFS timestamp lookup in `postdata.sas` finds no `MPE_DATALOADS` row for a REPLACE load and falls back to the current datetime.)
* **No PK usage** — `BUSKEY` is ignored: no uniqueness check, no dedup, no join to existing data. Staged rows are loaded exactly as submitted, duplicates included.
* **No temporal handling** — `VAR_TXFROM`/`VAR_TXTO`/`VAR_BUSFROM`/`VAR_BUSTO` and `CLOSE_VARS` are ignored. A REPLACE table should not be treated as temporal; querying it with the usual validity filters makes no sense.
* **No retained-key handling** — `RK_UNDERLYING` is ignored.
* **Minimal engine-specific handling** — unlike the temporal loaders, there is no pass-through/temp-table optimisation for OLEDB, Redshift, Postgres or Snowflake. The only engine conditional is CAS, where rows are removed via the `deleteRows` action and appended via a varchar-casting CASUSER temp table and a data-step append (SQL deletes and fixed-char-to-varchar appends are not possible on CAS).
* **Not atomic** — the delete/truncate and append are separate steps with no transaction or rollback, so a session failure between them still leaves the target empty or partially loaded (recovery is a re-approval of the same or a previous staging package). The risk is mitigated by performing all preparation first, aborting on any error before the destructive step, and executing the delete/truncate immediately before the append.
## Interaction with Row Level Security
REPLACE is incompatible with `EDIT`-scope RLS rules (a full-table wipe cannot honour row-level write restrictions); this is enforced at edit time in both directions — see [row-level-security.md](row-level-security.md#incompatibility-with-replace-load-type) and [issue #211](https://git.datacontroller.io/dc/dc/issues/211). `VIEW`-scope rules remain compatible.
This document explains **how** Row Level Security (RLS) is implemented in the Data Controller backend. For the user-facing configuration guide, see [docs.datacontroller.io/row-level-security](https://docs.datacontroller.io/row-level-security/).
## Overview
RLS in Data Controller is implemented as **server-side WHERE clause generation**. No data leaves SAS without passing through a dynamically generated filter. The filter is built at runtime per request, based on:
1. The requesting user's group memberships (SAS metadata groups + DC groups)
2. The active rules in the `MPE_ROW_LEVEL_SECURITY` control table
Because the filter is expressed as a standard SAS `WHERE` expression, it works against **any** engine — Base SAS datasets, database libraries (via implicit SQL pass-through pushdown), SPDE, CAS libnames, etc.
Scope mapping: `DLOAD` requests match rules with `RLS_SCOPE in ('VIEW','ALL')`; `ULOAD` requests match rules with `RLS_SCOPE in ('EDIT','ALL')`.
## Execution Flow of `%mpe_filtermaster`
The macro writes the final WHERE expression to a temporary **fileref** (`outref`), line by line. (A fileref is used because a generated filter may exceed the 64k macro variable limit — and note that `%include` of a fileref is not allowed directly in a `proc sql` where clause, hence callers typically use it in a DATA step or data step view.)
### 1. User-supplied filter (FILTER_RK)
If the request includes a stored filter (`filter_rk > 0`), its clauses are read from `MPE_FILTERANYTABLE` / `MPE_FILTERSOURCE` and generated first via `%mp_filtergenerate`. RLS clauses are then **AND-ed on top** — a user filter can only ever narrow results, never widen them beyond RLS.
`MPE_TABLES` may define `VAR_TXFROM` / `VAR_TXTO` (SCD2-style validity variables) for the target table. Unless the user explicitly filtered on those variables, the macro appends:
```
("<current datetime>"dt < VAR_TXTO)
```
so that only current records are surfaced in the EDIT grid and downloads.
where&dc_dttmtfmt. lt tx_to /* only current (non-deleted) rules */
and rls_scope in ("&scopeval",'ALL')
andupcase(rls_group) in (selectupcase(groupname) from work.groups)
and rls_libref = "<libref>"and rls_table = "<dsname>"
and rls_active = 1
orderby rls_group, rls_subgroup_id;
```
### 5. Clause assembly
If rules exist, they are appended to the fileref as `AND ( ... )`. Each **group** the user belongs to contributes one sub-filter, and the group-level sub-filters are joined with `OR`:
```
AND ( <group 1 filter> OR <group 2 filter> ... )
```
Each group's filter is produced by `%mp_filtergenerate`, which honours:
*`RLS_GROUP_LOGIC` — how subgroups (identified by `RLS_SUBGROUP_ID`) are joined
*`RLS_SUBGROUP_LOGIC` — how individual clauses within a subgroup are joined (AND/OR)
So membership in multiple groups is **permissive** (OR): the user sees the union of rows permitted by each of their groups.
### 6. Empty-filter fallback
If nothing was written to the fileref (no stored filter, no validity vars, no matching RLS rules), the macro writes a literal `1=1` so the fileref can always be consumed as a valid WHERE expression. Absence of rules therefore means **no restriction** — RLS is opt-in per table/group.
The fileref is `%include`d directly inside the `where` statement — the filter never passes through client-visible state, so it cannot be tampered with. For database libraries the WHERE expression is pushed down to the database by the SAS engine.
### Write path (stagedata.sas) — inverse filter
Uploads cannot be filtered; instead the filter is **inverted** and any submitted row matching the inverse is rejected:
,msg=%str(Security Problem - N unauthorised records submitted))
```
If even one submitted row falls outside the user's permitted row set, the entire staging request is aborted before any approval/apply step.
## Incompatibility with REPLACE Load Type
RLS with `EDIT` scope is **incompatible** with tables configured with `LOAD_TYPE=REPLACE` in `MPE_TABLES`. A REPLACE load wipes and reloads the entire target table, so row-level filtering of submitted records cannot be enforced meaningfully (the rows a user is *not* allowed to see would also be deleted). Backend validations therefore abort in both directions (see [issue #211](https://git.datacontroller.io/dc/dc/issues/211)):
1.**`mpe_row_level_security_postedit.sas`** — aborts when activating a rule with `RLS_SCOPE in ('EDIT','ALL')` against a table whose current `MPE_TABLES` record has `LOADTYPE='REPLACE'`.
2.**`mpe_tables_postedit.sas`** — aborts when setting `LOADTYPE='REPLACE'` on a table that already has active, current `RLS_SCOPE in ('EDIT','ALL')` rules in `MPE_ROW_LEVEL_SECURITY`.
`VIEW`-scope rules remain compatible with REPLACE loads, since they only affect read paths.
## Edit-Time Validation (Injection Defence)
Because `RLS_RAW_VALUE` is free text that ends up inside a generated WHERE clause, it is a potential SAS code-injection vector. Mitigations:
1.**`mpe_row_level_security_postedit.sas`** — a post-edit hook on the `MPE_ROW_LEVEL_SECURITY` table itself. Every newly staged rule (with `rls_active=1`) is grouped by target `libref.table` and run through `%mp_filtercheck(targetds=..., abort=YES)`, which compiles/tests each clause against the real target table and rejects the whole submission on invalid syntax (see [mp_filtercheck](https://core.sasjs.io/mp__filtercheck_8sas.html)).
2.**Format rules** enforced by validation: character values must be single quoted, `IN`/`NOT IN` values must be bracketed, `BETWEEN` must contain `AND`.
Additionally, editing `MPE_ROW_LEVEL_SECURITY` is itself a Data Controller table edit, so it goes through the normal approval workflow, audit trail (`MPE_AUDIT`), and can be column-restricted via [Column Level Security](https://docs.datacontroller.io/column-level-security/).
## Worked Example
Given these active rules (all for `MYLIB.MYDS`, `RLS_ACTIVE=1`):
| SCOPE | GROUP | GROUP_LOGIC | SUBGRP_LOGIC | SUBGRP_ID | VAR | OP | VALUE |
|---|---|---|---|---|---|---|---|
| ALL | Group 1 | AND | AND | 1 | VAR_2 | IN | ('this','or') |
| ALL | Group 1 | AND | AND | 1 | VAR_3 | < | 42 |
| ALL | Group 2 | AND | AND | 1 | VAR_4 | CONTAINS | 'xyz' |
A non-admin user in **both** groups, opening the VIEW page, gets a fileref containing:
```
AND ( (VAR_2 IN ('this','or') AND VAR_3 < 42) OR (VAR_4 CONTAINS 'xyz') )
```
resulting in:
```sas
data work.viewdata;
set mylib.myds;
where (VAR_2 IN ('this','or') AND VAR_3 < 42) OR (VAR_4 CONTAINS 'xyz');
run;
```
The same user submitting an EDIT upload has the inverse applied to their staged rows; any row not matching the expression above aborts the submission.
## Summary of Security Properties
* **Enforced server-side** in every read service (VIEW/EDIT/DLOAD) and every write service (ULOAD via inverse filter).
* **Admin bypass** is explicit (`&mpeadmins` group membership check).
* **Multi-group semantics are OR** (union of permitted rows); rules within a group/subgroup are AND/OR per configuration.
* **Fail-open by design**: if no rules match a table, it is unrestricted — RLS must be opted into per table.
* **Injection-resistant**: values are validated with `%mp_filtercheck` at the time rules are edited, not at query time.
Backend (SAS) tests are run with the sasjs CLI from the `sas/` directory.
## Commands
- Full deploy + test cycle: `npm run 4gl` (compiles, deploys to the 4gl target, runs makedata) then `sasjs test -t 4gl`.
- Run tests only (no rebuild/redeploy): `sasjs test -t 4gl`.
- Run a subset of tests: `sasjs test -t 4gl SOMESTRING` — the positional argument filters to matching tests (e.g. `sasjs test -t 4gl stagedata.test.3`).
## Results
- Test results: `sas/sasjsresults/testResults.json` / `.csv` / `.xml`
- Individual test logs: `sas/sasjsresults/logs/<testsuite>.log` (e.g. `services_editors_stagedata.test.3.log`)
- Coverage: `sas/sasjsresults/coverage.lcov`
## Notes
- Test source files live in `sas/sasjs/**`; `sas/sasjsbuild/` is generated build output — do not hand-edit it.
-`sasjs test` executes tests against the deployed app, so run `npm run 4gl` first after changing any service/hook/test code.
- Assertions are made with `%mp_assert()`; results are written to `work.test_results`.
The Angular wrapper was modernized in Handsontable v17.1 to align with Angular 17–19, simplifying setup and reducing dependencies. Earlier Angular versions are no longer the target — upgrade Angular first if you're below v17.
HyperFormula version compatibility table: https://handsontable.com/docs/react-data-grid/formula-calculation/#hyperformula-version-support
---
## Server-side data with DataProvider (v17.1+)
The **DataProvider plugin** (new in v17.1) wires the grid up to a remote data source so rows are fetched, sorted, and mutated server-side instead of held in memory. Use it for datasets too large to load up front, or when the source of truth lives in a backend.
The `dataProvider` option is an object with five required keys: a row-id resolver, a paginated `fetchRows` callback, and three mutation callbacks. For a read-only grid, stub the mutation callbacks with `async () => {}` — they must still be present.
```jsx
<HotTable
colHeaders={['ID','Name','Email']}
pagination={{pageSize:25}}
columnSorting={true}
emptyDataState={true}
height={360}
licenseKey="non-commercial-and-evaluation"
dataProvider={{
// Required: how to extract a stable id from each row
rowId:'id',
// Required: paginated fetch. `sort` is null when no column is sorted.
// Second arg carries an AbortSignal — pass it to fetch() so the plugin
// can cancel superseded requests when the user pages/sorts quickly.
// Required. Stub with async () => {} if your grid is read-only.
// The plugin auto-refetches the current page after each callback resolves.
// See the plugin API ref for the exact payload shapes.
onRowsCreate:async()=>{},
onRowsUpdate:async()=>{},
onRowsRemove:async()=>{},
}}
/>
```
### Loading and error UI
The plugin fires three hooks you can wire up as `<HotTable>` props for loading indicators and error surfaces:
-`beforeDataProviderFetch(params)` — fires before each fetch. `params.skipLoading` is set when the plugin wants to suppress your loading indicator (e.g., during a quick refetch).
-`afterDataProviderFetch(result)` — fires after a successful fetch.
-`afterDataProviderFetchError(error)` — fires when `fetchRows` throws or returns a rejected promise.
### Companion options
The plugin is built to pair with `pagination` (paginates server-side), `columnSorting` (single-column server-side sort), and `emptyDataState` (loading + empty state UI). Enable all three when you use DataProvider.
The **Notification plugin** (new in v17.1) shows non-blocking toast notifications anchored to the grid — useful for confirming saves, surfacing validation errors, or signaling background sync state. Enable with `notifications: true` and trigger via `hot.getPlugin('notifications').showMessage(...)`. See the plugin guide for placement, severity levels, and auto-dismiss timing: https://handsontable.com/docs/react-data-grid/notification/
---
## Events & Hooks
Handsontable hooks are passed as props on `<HotTable>`:
```jsx
<HotTable
afterChange={(changes,source)=>{
if(source!=='loadData'){
console.log('Cell changed:',changes);
}
}}
beforeChange={(changes,source)=>{
// Return false to cancel the edit
}}
afterSelection={(row,col,row2,col2)=>{
console.log('Selected:',row,col,'to',row2,col2);
}}
/>
```
There are 100+ hooks available. Full reference: https://handsontable.com/docs/react-data-grid/api/hooks/
Handsontable v17 introduced a Recipes section in the docs — ready-made patterns for common use cases
(data validation workflows, dynamic column generation, etc.). Check the recipes index before building
something from scratch: https://handsontable.com/docs/react-data-grid/recipes/
---
## Common Pitfalls
- **Forgetting `height`**: The grid won't render without a `height` prop. Use `"auto"`, a pixel value, or a CSS string.
- **Not filtering `loadData` in `afterChange`**: The `afterChange` hook fires on initial data load with `source === 'loadData'`. Always check the source to avoid infinite loops when syncing changes back to state.
- **Using the old wrapper packages**: v17 removed `@handsontable/react` and `@handsontable/angular`. Use `@handsontable/react-wrapper` and `@handsontable/angular-wrapper`.
- **Using legacy CSS imports**: `handsontable.full.min.css` was removed in v17. Use `handsontable/styles/handsontable.min.css` plus a theme file.
- **Formulas with nested object data**: HyperFormula formulas don't work when `data` is an array of nested objects — use flat objects or arrays of arrays.
- **ExportFile `columnHeaders` renamed**: In v17.1 the ExportFile plugin's `columnHeaders` option was renamed to `colHeaders` to match the table-level option. Update any `exportAsString` / `exportAsBlob` / `downloadFile` calls that pass `columnHeaders: ...`.
---
## Version Awareness
Handsontable docs are versioned. The latest docs live at `/docs/react-data-grid/` (which redirects
to the current version). To link to a specific version, use `/docs/17.0/react-data-grid/`.
When helping a user, check which version they are on — breaking changes between major versions are
common. Point them to the relevant migration guide if they're upgrading.
For the full organized directory of documentation links, read `references/docs-map.md` in this
An agent skill for [**HyperFormula**](https://hyperformula.handsontable.com) — the headless, open-source TypeScript spreadsheet calculation engine. It gives any AI coding agent deep, task-oriented knowledge of HyperFormula so the agent can help you integrate, configure, and debug the engine faster.
This package is the npm distribution of the `hyperformula` skill maintained in [`handsontable/handsontable-skills`](https://github.com/handsontable/handsontable-skills). The package version tracks the HyperFormula product version the skill targets (e.g. `3.3.0` targets HyperFormula 3.3.0).
## What's in the package
A self-contained skill — plain markdown, no runtime dependencies:
```
SKILL.md ← entry point: concepts, routing, and a docs map (YAML frontmatter + body)
references/ ← task-oriented deep-dives the agent reads on demand
```
`SKILL.md` follows the [Agent Skills](https://claude.com/blog/skills) layout (a YAML frontmatter block with `name`/`description`, followed by markdown instructions). Any agent that can load a skill — or simply read a folder of markdown instructions — can use it.
## Install
```bash
npm install @handsontable/hyperformula-skill
```
The installed package lives at `node_modules/@handsontable/hyperformula-skill`. Resolve that path programmatically with:
Upload the package contents (`SKILL.md` + `references/`) to the [Skills API](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview).
### OpenAI Codex
Codex discovers [skills](https://developers.openai.com/codex/skills) as folders containing a `SKILL.md`, under `~/.agents/skills` (available in every repo) or `.agents/skills` in a repo (checked in for your team). Copy the package into one of those locations:
If your agent reads a single instruction file rather than a folder, point it at `SKILL.md`; it links the files under `references/` so the agent can pull them in as needed. For RAG / vector-store setups, ingest `SKILL.md` together with every file under `references/` (the repository also publishes a pre-flattened `hyperformula-rag.md` for this case).
## Other distribution formats
The same skill source is also shipped as a drag-and-drop `.zip` (Cowork / Claude.ai web), a flattened `-rag.md` doc (RAG / vector stores), and a Claude Code plugin marketplace entry. See the [repository README](https://github.com/handsontable/handsontable-skills#readme) for details.
## License
MIT — see the [repository](https://github.com/handsontable/handsontable-skills/blob/main/LICENSE.txt). HyperFormula itself is separately licensed (GPLv3 or commercial); see the [licensing guide](https://hyperformula.handsontable.com/docs/guide/licensing.html).
Use this skill when the user works with HyperFormula (HF) — a headless,
open-source TypeScript spreadsheet calculation engine. Use it when the user:
mentions HyperFormula or HF; wants to integrate Excel-like formulas into a
JS/TS app; evaluate formulas programmatically in a browser or Node.js;
simulate spreadsheet or Excel calculation behavior server-side; run
calculations over data parsed from XLSX files (e.g. via SheetJS) in code;
build a pricing engine, financial model, or what-if analysis in code; define
custom spreadsheet functions or named expressions outside of a UI context. Do
NOT use this skill for: general Excel questions unrelated to HF; general
Google Sheets questions; generic spreadsheet formula-syntax questions outside
an HF context; formulas used inside Handsontable (use the "handsontable"
skill instead).
---
# HyperFormula
HyperFormula is a **headless, open-source TypeScript spreadsheet calculation engine** for embedding spreadsheet logic in any JavaScript/TypeScript application (browser or Node.js). ~420 built-in functions, dependency graph, undo/redo, i18n (17 languages). Dual-licensed: GPLv3 or commercial.
Task-oriented references in `references/` (open the one that matches what the user is doing):
- [`getting-started.md`](references/getting-started.md) — how to install, create an instance (`buildFromArray` / `buildFromSheets` / `buildEmpty`), and work with cell addresses. Open this for onboarding questions and first-run errors.
- [`api-quickref.md`](references/api-quickref.md) — runnable examples for CRUD, rows/columns, sheets, exporting data, batching, named expressions, events, undo/redo, clipboard. Open this when implementing or debugging HF API calls.
- [`custom-functions.md`](references/custom-functions.md) — extending HyperFormula via `FunctionPlugin`: minimal plugin, argument types, volatile functions, range arguments, returning arrays, error handling, aliases, localized names. Open this when adding or debugging custom spreadsheet functions.
- [`configuration.md`](references/configuration.md) — `ConfigParams` options, Excel-compatibility preset, Google-Sheets preset, and config-related pitfalls (separator collisions, Node ICU, `precisionRounding` change in v3). Open this when tuning behavior or diagnosing locale issues.
- [`vue3.md`](references/vue3.md) — Vue 3 integration: `markRaw` requirement, Composition API, Pinia/Vuex stores, `destroy()` on unmount. Open this when embedding HF in a Vue 3 app.
- [`error-handling.md`](references/error-handling.md) — inspecting `CellError`, `ErrorType` enum, `getCellValueDetailedType`, common error causes (`#NAME?`, `#CYCLE!`, `#REF!`, …). Open this when a cell returns an error or you need to branch on result type.
- [`general-pitfalls.md`](references/general-pitfalls.md) — cross-cutting gotchas: `destroy()` lifecycle, forcing literal strings, Excel-parity caveats, hard limits. Open this when results look wrong or memory grows unboundedly.
Below this section is the **Documentation map** — the canonical directory of links to the official HyperFormula docs. Use it when pointing the user to authoritative material.
## Documentation map
All links resolve to `hyperformula.handsontable.com` unless noted.
"description":"Agent skill for HyperFormula — the headless, open-source TypeScript spreadsheet calculation engine. Teaches AI coding agents how to integrate, configure, and debug HyperFormula.",
Vue 3 requires wrapping the instance with `markRaw` — see [vue3.md](vue3.md) for the full integration guide (React / Angular / Svelte need no special handling).
### `functionArgSeparator` vs `thousandSeparator` collision
These two options **cannot share a character**. European locales that use `,` as thousand separator must change the argument separator:
```ts
consthf=HyperFormula.buildEmpty({
licenseKey:'gpl-v3',
thousandSeparator:',',
decimalSeparator:'.',
functionArgSeparator:';',// must differ from thousandSeparator
});
```
### Node.js must have full ICU
String comparisons (used in `MATCH`, `VLOOKUP`, sorting) can silently produce wrong results on Node.js < 13 or builds without full ICU. Verify at startup:
```ts
// Must report 'full' (or a detailed ICU version), not 'small'.
HyperFormula defers some structural transformations (row/column insertions, moves) and applies them lazily. In long-lived instances with heavy mutation throughput — bulk imports, frequent undo/redo, scripted batch edits — the pending queue can grow before cleanup. v3.3 fixed the unbounded-growth leak; this option lets you cap the queue explicitly. Lower values trade memory for more frequent flush work; the default is suitable for typical UI workloads.
### `precisionRounding` default changed in v3
Before v3 the default was `14`. It is now `10`. Calculations relying on the old precision need an explicit override:
Mark a function as volatile so it recalculates on every change (like `RAND` or `NOW`). Volatile functions are expensive in large sheets — use sparingly.
```ts
MyPlugin.implementedFunctions={
RAND_ID:{
method:'randId',
parameters:[],
isVolatile: true,
},
};
```
## Argument types
```ts
import{FunctionArgumentType}from'hyperformula';
// FunctionArgumentType.NUMBER
// FunctionArgumentType.STRING
// FunctionArgumentType.BOOLEAN
// FunctionArgumentType.NOERROR
// FunctionArgumentType.SCALAR
// FunctionArgumentType.RANGE // accepts a cell range
// FunctionArgumentType.ANY
```
Use `optionalArg: true`, `defaultValue`, and `minValue` / `maxValue` to validate arguments declaratively:
Custom functions can return 2D arrays. **PITFALL:** result arrays don't auto-resize when upstream dependencies change — the footprint is fixed at first evaluation.
```ts
classSplitPluginextendsFunctionPlugin{
splitToRow(ast,state){
returnthis.runFunction(
ast.args,
state,
this.metadata('SPLIT_ROW'),
(text,sep)=>[text.split(sep)]// 2D array: one row, N cols
// Usually: function not registered (check plugin registration or i18n language)
console.log('Unknown name:',value.message);
break;
default:
console.log('Error:',value.type,value.message);
}
}else{
console.log('Value:',value);
}
```
## `#CYCLE!` is HyperFormula-specific
Standard spreadsheet apps report cycles differently. HyperFormula's `IF` also reports cycles for all branches, even unreachable ones — this can produce `#CYCLE!` in formulas that Excel or Sheets would evaluate.
Use the detailed type when you need to distinguish e.g. number vs empty without touching the value.
## Returning errors from custom functions
Custom `FunctionPlugin` methods can return a `CellError` to surface a formula error. See [custom-functions.md](custom-functions.md) for the full pattern.
```ts
import{CellError,ErrorType}from'hyperformula';
returnnewCellError(ErrorType.DIV_BY_ZERO);
```
## Common causes
| Error | Common cause |
|---|---|
| `#NAME?` | Function not registered (check `registerFunctionPlugin` order or `language` config) |
| `#CYCLE!` | Circular reference — or an `IF` branch that *could* produce one |
| `#REF!` | Deleted row/column broke a formula reference |
| `#VALUE!` | Type mismatch — e.g. passing a string where a number is expected |
| `#DIV/0!` | Division by zero, including empty cells coerced to 0 |
| `#NUM!` | Out-of-range numeric result (e.g. `SQRT(-1)`) |
| `#N/A` | Lookup miss (`MATCH`, `VLOOKUP`) or propagated from an upstream `#N/A` |
- Runtime differences vs Excel/Sheets: https://hyperformula.handsontable.com/docs/guide/list-of-differences.html
## Error handling
See [error-handling.md](error-handling.md) — checking `CellError`, `ErrorType` enum, `getCellValueDetailedType`, common error causes.
## Always call `destroy()` in long-running apps
HyperFormula maintains internal data structures (dependency graph, address mapping) that are **not** garbage-collected until `destroy()` is called. Leaking instances in servers or SPAs accumulates memory.
```ts
hf.destroy();
// After destroy() the instance is unusable — create a new one if needed.
```
v3.3 fixed two longstanding leak sources inside live instances — pending lazy transformations and undo/redo history were not being trimmed. If you maintain very long-lived instances with heavy mutation throughput, also see `maxPendingLazyTransformations` in [configuration.md](configuration.md) to bound the lazy-transformation queue. `destroy()` is still mandatory at teardown.
## Force a string that looks like a formula
Prefix with `'` (apostrophe) to store the literal text instead of evaluating.
```ts
// Stored as the literal string "=SUM(1,2)", not a formula:
Every factory method (`buildFromArray`, `buildFromSheets`, `buildEmpty`) requires `licenseKey`. Use `'gpl-v3'` for open-source use or your commercial key.
## Known hard limits
- **Single workbook per instance** — no multi-workbook support.
- No 3D references, dynamic arrays, async functions, structured references ("Tables"), or relative named expressions.
-`IF` reports cycles for all branches, even unreachable ones.
- Custom function result arrays don't auto-resize when dependencies change.
Vue 3's Composition API wraps objects in a reactive Proxy that intercepts property access and **corrupts HyperFormula's internal state**, causing silent data corruption or crashes. React, Angular, and Svelte need no special handling.
```ts
import{markRaw}from'vue';
import{HyperFormula}from'hyperformula';
consthf=markRaw(
HyperFormula.buildEmpty({licenseKey:'gpl-v3'})
);
```
Use the same wrapper no matter where the instance is held — `ref`, `reactive`, component state, a store (Pinia/Vuex), or a composable. If the raw HyperFormula instance ever reaches Vue's reactivity system, it must be marked.
## Composition API example
```ts
import{onBeforeUnmount,ref}from'vue';
import{markRaw}from'vue';
import{HyperFormula}from'hyperformula';
exportfunctionuseHyperFormula() {
consthf=markRaw(
HyperFormula.buildEmpty({licenseKey:'gpl-v3'})
);
// Free internal data structures when the component unmounts.
description:Expert guidance for the SAS programming language — DATA step, PROC SQL, macro language, formats, ODS, and common procedures. Pure SAS syntax only, no SASjs-framework content. Use when writing, reviewing, or debugging .sas programs, or answering SAS language questions.
---
# SAS Language
Write idiomatic, production-quality SAS code. This skill covers the SAS language itself, independent of any framework.
## Scope
- DATA step programming (SET/MERGE/BY, RETAIN, arrays, DO loops, hash objects, first./last. processing)
- PROC SQL (joins, subqueries, views, pass-through with CONNECT TO)
Generated SAS code must run cleanly — **zero WARNINGs (and zero ERRORs) in the log**. Treat every `WARNING:` as a defect: uninitialized variables, implicit type conversions, truncation notes that should be warnings, "no observations", unresolved macro references, etc. If a warning is truly unavoidable, suppress it deliberately (e.g. an explicit option) and comment why. Likewise avoid superfluous NOTEs where reasonable.
## Style rules
These follow the @sasjs/core coding standards — apply them to all SAS code:
- One statement per line; indentation = 2 spaces, no tabs, no trailing whitespace
- Lines no longer than 80 characters; unix (LF) line endings; UTF-8
- Avoid non-ASCII / special characters entirely — maximum compatibility across SAS installations and encodings
- Always end steps with `run;`; for `proc sql` (and CAS-connected procs) `quit;` is essential to avoid `WARNING: You cannot disconnect or terminate session ...` on Viya
- All dataset references must be 2-level (`work.blah`, not `blah`) — protects against `DATASTMTCHK=ALLKEYWORDS` and an active `USER` library
- Explicit `length` / `attrib` for character variables rather than relying on defaults (avoids implicit length=8 truncation)
- Use literal suffixes for clarity (`'01JAN2020'd`, `'12:30't`)
- Prefer `proc sort` with `nodupkey` over manual dedup logic
- Macros:
- Define with parentheses, even with no parameters: `%macro x();` not `%macro x;`
- Closing `%mend;` must repeat the macro name
- Macro calls are not terminated with a semicolon: `%my_macro()` not `%my_macro();`
- Macro variables without trailing dot (`&var` not `&var.`) unless needed to prevent incorrect resolution
- Macro variable NAMES are case-insensitive: `&Foo`, `&FOO`, and `&foo` all resolve to the same symbol (same for `%symexist`/`%superq`/`symget` arguments, which take a NAME not a value). Don't chase case mismatches as a bug — it's never the cause.
- ALL macro variables must be `%local` unless deliberately global (globals should use an application prefix to avoid collisions); use `call symputx` (not `symput`) in DATA steps
- Comment with `/* */` inside macros (asterisk comments are compiled into the macro)
- Guard macro logic with `%length(&var)=0` checks rather than `&var=` (empty comparisons are unsafe)
- Avoid naming collisions: use `%sysfunc`-/`&syslast`-based work tables (e.g. `data &output; set &syslast; run;`) rather than hard-coded names
- No open (non-macro) conditional code: wrap platform-branching or conditionally-executed blocks (e.g. `%if %mf_getplatform()=VIYA %then %do; ... %end;`) in a `%macro ... %mend` and invoke the macro. Open `%if` at program level fails in some execution contexts (job/scheduler/test harnesses) and hides scope leaks.
## Portability awareness
- Note when code differs between SAS 9.4 and Viya (e.g. CAS actions vs procs, `proc casutil` for sashdat loading, no X command on locked-down servers)
- Avoid hard-coded physical paths and engine-specific options unless asked
- No open macro code with if/else logic: wrap branching blocks in `%macro ... %mend` and call them — open `%if`/`%else` does not behave as expected in all SAS environments.
- Macro timing issues: referencing `¯ovar` before it exists, `%if` evaluating data-step variables (use `if`/`symget` instead)
- Automatic macro variables (`&syscc`, `&syswarningtext`, `&syserrortext`, `&sysdate`, `&sysuserid`, etc.) are READ-ONLY — attempting to overwrite them (e.g. `%let syswarningtext=;` or `call symput('syscc',...)`) raises `ERROR: Unable to assign value to a macro variable that is read only` (or similar). Never try to "clear" them.
- Truncation from implicit length=8 on first assignment
description:Frontend/Node integration with SAS backends using @sasjs/adapter — configuring the SASjs class, authentication (SAS 9, Viya, SASjs server), requests with input/output tables, file upload, and session management. Use when writing TypeScript/JavaScript that calls SAS services or jobs.
---
# @sasjs/adapter
`@sasjs/adapter` is the TypeScript library for calling SAS services/jobs from browsers or Node, with a unified API across three server types: `SAS9`, `SASVIYA`, `SASJS`.
## Basic setup
```ts
importSASjsfrom'@sasjs/adapter'
constsasjs=newSASjs({
serverUrl:'https://sas.example.com',
serverType:'SASVIYA',// SAS9 | SASVIYA | SASJS
appLoc:'/Public/app/myapp',// root folder of deployed services
contextName:'SAS Job Execution compute context',// Viya only
mytable:[{col1:'value',col2: 42}]// input tables as JS arrays of objects
})
// response.result contains output tables sent back from SAS (_webout JSON)
```
- Input tables become SAS datasets via the `sasjs_tables` mechanism (work tables named after the JS keys).
- The SAS service must write JSON to `_webout` — conventionally with the `mp_jsonout` macro from @sasjs/core, wrapped in `proc stp`-style begin/end macros.
- Responses follow the `SASjsRequest`/`SASjsResponse` types; check `response.result` for tables and `response.log` where available.
## Authentication
- **SAS 9**: `sasjs.logIn(username, password)` (form-based against the stored process server). Session cookie is managed automatically.
- **Viya**: OAuth client/secret (client credentials grant) or authorization code flow; tokens are refreshed automatically. Configure via CLI (`sasjs add cred`) for Node usage.
- **SASJS server**: token-based auth against the sasjs/server API.
## Key classes / modules
-`SASjs` — main facade: `request()`, `logIn()/logOut()`, `uploadFile()`, `executeScript()`
-`file/` utilities — file upload to SAS (binary content handling)
## Tips
- Set `debug: true` to surface the SAS log in responses while developing.
- Always handle `response.status` / error responses — SAS-side errors (e.g. from `%mp_abort`) come back in the JSON, not necessarily as HTTP errors.
- For large payloads prefer CSV upload or streamed files over JSON input tables.
- Keep `appLoc` consistent with the `appLoc` in `sasjsconfig.json` used to deploy.
## Important: request() inputs are ALWAYS tables
Every key in the `data` object of `sasjs.request(path, data)` is serialized via the `sasjs_tables` CSV mechanism and arrives in SAS as a **work dataset named after the key** — even scalar values. You cannot pass ad-hoc macro variables this way; services must read inputs from the work table (e.g. `data _null_; set work.config; call symputx('rootdir', rootdir); run;`). Output column names in `response.result.<table>` come back UPPERCASE (SAS dataset semantics).
## Using the adapter without a bundler (zero-build / strict CSP frontends)
The package root `index.js` is a UMD bundle exposing a global `SASjs`. Pattern (from the minimal seed app):
1.`"prepare": "cp node_modules/@sasjs/adapter/index.js src/sasjs.js"` in package.json (runs on `npm i`).
2.`<script src="sasjs.js"></script>` before your app script.
3. Configure via a hidden custom element: `<sasjs serverType="SASJS" appLoc="/Public/app/myapp" debug="false"></sasjs>` and read attributes with `document.querySelector('sasjs')`. When the app is streamed by SAS itself, omit `serverUrl` — same-origin requests just work (CSP `default-src 'self'` safe).
description:Using the SASjs CLI (@sasjs/cli) to create, compile, build, deploy, run, and test SASjs projects against SAS 9, Viya, and SASjs server targets. Use for any `sasjs <command>` usage, CI/CD pipelines, target/auth config, sasjsconfig.json, service packs, or frontend streaming builds.
---
# @sasjs/cli
The SASjs CLI (`npm i -g @sasjs/cli`, invoked as `sasjs`) automates compiling, building, and deploying SAS projects. All commands support `-t <target>` to select a target from `sasjsconfig.json`.
## Targets and auth
- A **target** = `{ name, serverUrl, serverType, appLoc }`. `serverType` is one of `SAS9`, `SASVIYA`, `SASJS`.
- Credentials: `sasjs add cred` (or `.env` file). Viya uses client/secret **or**`sasjs auth login` (user/pass, no client/secret needed — see below); SAS 9 uses user/pass; SASJS server uses an access token.
-`sasjs context` manages Viya compute contexts; `sasjs add target` adds a new target.
## Core workflow
```
sasjs create myapp # scaffold a new app (templates available)
sasjs compile # gather macros/services/jobs into per-file build outputs (sasjsbuild/)
sasjs build # produce deployable artefacts: JSON + .sas per target
sasjs deploy # deploy compiled/built artefacts to the target server
sasjs cbd # compile + build + deploy in one step (-t viya etc.)
```
## Command reference
| Command | Purpose |
|---|---|
| `sasjs create / init` | Scaffold new app or add SASjs to existing repo |
| `sasjs compile` | Resolve dependencies (`<h4> SAS Macros </h4>` etc.) into `sasjsbuild/` |
| `sasjs build` | Create build JSON / service pack per target |
| `sasjs deploy` / `cbd` | Deploy to server (servicepack or direct) |
| `sasjs run <file.sas>` | Execute an arbitrary SAS file on the server, return log |
| `sasjs request <path>` | Execute a deployed service/job with input data (`-d`) |
| `sasjs job execute` | Run a deployed job |
| `sasjs flow execute` | Run a sequence of jobs with dependencies (CSV-defined flows) |
| `sasjs servicepack deploy` | Deploy from a JSON service pack |
| `sasjs web` | Build the frontend and stream it into the SAS web root (streamConfig) |
- Test coverage is generated only from a `sasjs compile` (or `sasjs c`). It accepts a target (`-t <target>`), but nothing is deployed to that target — compilation and coverage are fully local/offline, so no server needs to be available or reachable. Missing macro dependencies (e.g. `mp_ds2csv.sas`) mean `@sasjs/core` isn't installed — run `npm i` first.
- Dependencies are declared in doxygen headers: `<h4> SAS Macros </h4>`, `<h4> SAS Files </h4>`, `<h4> SAS Folders </h4>`, and `@li item` entries — the CLI builds the dependency tree from these.
-`sasjs compile` output goes to the `sasjsbuild/` folder (git-ignore it); `sasjsresults/` holds test/run outputs.
- CI/CD: `sasjs cbd -t viya` is the standard deploy step; combine with `sasjs servicepack deploy` for artefact-based releases.
- Exit codes are non-zero on failure — safe for pipelines.
## Gotchas
- Run `npm i` before `sasjs cb` — macro dependency resolution needs `node_modules/@sasjs/core` present, and `@sasjs/core` (and `@sasjs/adapter` if used) must be listed in `package.json`.
- Credentials files are per-target: `.env.<targetname>` (e.g. `.env.server`) with `CLIENT`, `ACCESS_TOKEN`, `REFRESH_TOKEN`. Never commit them — gitignore `.env*`.
## Viya auth without a client/secret (`sasjs auth login`)
`sasjs auth login -t <target>` authenticates with a regular SAS username/password via the OAuth2 password grant against the built-in, secret-less `sas.cli` public client. No admin-registered OAuth client is needed — the fastest way to get `sasjs run`/`deploy` working on dev/demo estates. The password is never stored; the minted ACCESS_TOKEN/REFRESH_TOKEN pair is persisted to `.env.<target>` (local) or `~/.sasjsrc` (global) and verified via `/identities/users/@currentUser` (`Logged in as <id> (<name>)`). Bare `sasjs auth` is still an alias for `sasjs add cred`.
- Token expiry: the CLI silently refreshes via the stored refresh token (works with and without a client/secret), and re-persists the rotated pair — Viya refresh tokens are **single-use/rotating**, so this persistence is what keeps later invocations working. If refresh fails, re-run `sasjs auth login`.
- Some estates give `sas.cli` a short access-token TTL (e.g. 1h); a refresh on most invocations is normal.
- Opaque (non-JWT) tokens are treated as usable — the server is the authority on expiry.
- Limitations: local/LDAP accounts only (no SSO/SAML/MFA estates); password grant must be enabled for `sas.cli` (default on Viya 3.5+/4); ROPC is deprecated in OAuth 2.1 — use a registered client/secret for CI/production.
-`sasjs run` 403 on session creation = the account isn't authorised for the configured compute context — set `contextName: "SAS Studio compute context"` on the target. First run on a cold estate can take many minutes (compute pod spin-up) and may appear to hang.
- Self-signed estates: use `--insecure` on `auth login`, or configure `httpsAgentOptions` on the target.
## Viya streaming apps (streamConfig.streamWeb)
With `streamWeb: true`, `sasjs web`/`cbd -t viya` deploys the frontend **into SAS Files Service**: a streaming job at `<appLoc>/services/<streamServiceName>.html` serves `index.html`, and assets land in `<appLoc>/services/web/...`. At build time every asset/script/css URL in the HTML is rewritten to `/SASJobExecution?_FILE=<appLoc>/services/web/...` and the adapter config element (`<sasjs apploc=...>`) is stamped with the target `appLoc`.
**Consequence: the app only works when served from the exact `appLoc` it was deployed against.** If the streaming HTML is placed anywhere else (e.g. manually uploaded to a user home folder like `/Users/<id>/myapp/...`), all rewritten `/Public/app/...` asset links 404 and the adapter calls the wrong service paths — the page renders with no CSS/JS and no backend. Fix by redeploying. The original build (`sasjs cb`) has the apploc from the sasjsconfig.json - when the app is deployed as a SAS program, the supplied `%let apploc = ` (runtime value) is swapped with the `compiled_apploc` (build time value) to allow apps to be dynamically deployed to a given apploc at deploy time.
### Verifying a Viya deployment headlessly (no browser)
1. Get a token (password grant works out of the box with the built-in `sas.ec` client, empty secret):
`curl -X POST <server>/SASLogon/oauth/token -u 'sas.ec:' -d 'grant_type=password&username=U&password=P'`
2. Fetch the app: `GET /SASJobExecution/?_FILE=<appLoc>/services/<name>.html` with `Authorization: Bearer` → expect `200` and the full `index.html`. `401` = auth, anything else = not deployed there.
3. Fetch each asset the HTML references. Response tells you what's wrong:
-`200` with file content → asset deployed correctly.
-`202` + a tiny plain-text body (e.g. `Parameter Error\nFile error`) → **file does not exist at that Drive path** (classic appLoc-mismatch symptom). Note `_FILE` responses can be async: add `&_action=wait` to get content synchronously.
- Services: `POST /SASJobExecution/?_program=<appLoc>/services/<svc>&_action=wait` → `Parameter Error / Unable to get job definition` means the service was never deployed as a JES job (files on Drive alone are not enough — only `sasjs deploy`/`cbd`/`servicepack deploy` registers them).
### Talking to SAS — ALWAYS use the adapter or the CLI
When executing a SAS service or job for any purpose (debugging, reproduction, CI), **always go through the `@sasjs/adapter` or `sasjs` CLI** — never hand-roll curl against `SASJobExecution`. The adapter and CLI handle the things that are trivial to get wrong by hand: the input-data CSV format (space-separated `name:$format.` headers, CRLF, double-quoted special values, `%nrstr(...)` wrapping), the execution-mode routing (`_executionTasks=true` reads `sasjs<N>data` as **macro variables**, not file uploads), the `_debug`/`>>weboutBEGIN<<`/`>>weboutEND<<` wrapper parsing, the token refresh, and the `_contextname` URL param. A hand-built comma-separated CSV will read as blanks, silently skip guarded macro blocks, and send you down a rabbit hole of phantom bugs.
- Build input data as a JSON file: `{ "<table>": [{ "<col>": "<val>", ... }] }`, e.g. `{ "config": [{ "rootdir": "/export/...", "runastask": "true", "usecomputeapi": "null", "contextname": "Compute Reusable" }] }`.
- Run: `sasjs request '<appLoc>/services/common/<svc>' -t viya -d data.json -l <path>.log -o <path>.json`. The CLI hardcodes `debug: true`, so the `-l` flag always captures the full MPRINT/NOTE/`&syscc` log; `-o` saves the parsed webout. **Always pass `-l`** — reproducing a bug without the log means re-running the whole thing.
- The Viya Folders/JES REST APIs ARE fine to hit directly with curl (token + `Authorization: Bearer`) for folder/file/member management and for `GET /SASJobExecution/?_FILE=...` asset checks — just not for *executing your own services with input data*.
#### Adapter execution-context and input-data facts
- The adapter automatically appends `_contextname=<value>` as a **URL parameter** to every Viya JES request (you do NOT need to pass it yourself). BUT JES request params are **NOT auto-promoted to SAS macro variables** — `%symexist(_contextname)` is false inside the service. If the service needs the chosen context name (e.g. to stamp it into the streamed HTML), pass it in the **input data table** (a `contextname` column) and read it with `call symputx` — that is the reliable channel.
- The target's `contextName` in `sasjsconfig.json` decides which compute context the service runs under (and thus the `runAs` identity). The adapter's URL `_contextname` param is the same value. To reproduce a service under a batch/reusable context via `sasjs request`, set the target's `contextName` to that reusable context (e.g. `Compute Reusable`, runAs=sasbatch) — otherwise it runs as your own identity and write-test steps to batch-owned folders fail with `User does not have appropriate authorization level`.
- Adapter CSV format (so you can read the `NOTE: The infile ... is:` RULE in the log correctly): row 1 is the header with `name:$informat.` pairs **space-separated**; data rows are **comma-separated** (CRLF), with values double-quoted only if they contain a special char (`,`, `"`, tab, newline). The sasjs/core webout reader reads it back with `dsd` + `firstobs=2` + an `input <name>:$informat.;` statement derived from the header.
-`_executionTasks=true` (runAsTask) changes how input data arrives: as `sasjs<N>data`**macro variables** (chunked into `sasjs<N>data0..N`), NOT as `_WEBIN_FILE` uploads. The core webout `mv_webout` macro handles both, but it branches on `_EXECUTIONTASKS` — be aware when reading logs.
### Redeploying cleanly on Viya (the 409 Conflict problem)
Re-running `sasjs cbd`/`sasjs run viya.sas` against an **existing** appLoc often fails mid-deploy with `409 Conflict` (and an `mp_abort` → `abort cancel`): `mv_createfile` DELETEs the old file id then tries to recreate it, but when an intermediate **folder** already exists (e.g. `<appLoc>/services/web/js`) the recreate step conflicts and the whole deploy aborts — leaving a half-deployed app (services present, some assets missing). The `?recursive=true` folder DELETE also returns `409 You cannot delete the folder because it is not empty`, and individual member/folder DELETEs can return `403` even as the owner, so you cannot easily tear the tree down by hand.
The reliable workaround is to **move the top appLoc folder out of the way** and redeploy to the original path — the deploy creates a fresh folder tree with no conflicts, and the old folder is retained as a backup. This is far faster and more reliable than fighting per-member deletes, and is the recommended pre-deploy step for any non-CI redeploy on Viya. Two ways to move it:
- **Rename in place** (simplest): `PATCH /folders/folders/{id}` with `{"name":"<old>.bak.<ts>","version":2}` — frees the original name, keeps the old folder as a sibling backup.
- **MOVE into a backup parent** (tidier — keeps all backups in one place): create (once) a backup parent folder e.g. `/Users/<id>/macrodash-backups`, then `POST /folders/folders/{backupParentId}/members/{appLocFolderId}?action=move` moves the whole tree (with contents) into it, freeing the original path for the fresh deploy.
(On sasjs/server just `sasjs fs delete` the appLoc first — Drive there supports clean recursive deletes.)
JES applies its own CSP header when streaming (includes `unsafe-inline`/`unsafe-eval`); a strict CSP meta tag in the app's HTML still applies and is the one that matters for the app code.
description:Standards and conventions for the @sasjs/core SAS macro library (mf_*, mp_*, mm*, ms_*, mv_* macros). Use when writing or editing SAS macros in a sasjs/core-style repo, picking an existing macro over reinventing one, or the sasjs/core build, lint, doxygen, and testing conventions.
---
# @sasjs/core — SAS Macro Library
@sasjs/core is an MIT-licensed library of production-quality SAS macros for SAS application development, portable across SAS 9 (meta), Viya, and SASjs server.
## Coding standards (mandatory)
- One macro per file; filename must match the macro name (lowercase, no spaces)
- Macro definitions must use parentheses: `%macro x();` not `%macro x;`
- Macro *calls* are NOT terminated with a semicolon: `%my_macro()` not `%my_macro();`
- All macro variables must be declared `%local` to prevent scope leakage
- Always use `mf_getuniquefileref` when assigning filerefs, and `mf_getuniquelibref` when assigning librefs (never hardcode or hand-roll unique references)
- 2-space indentation, no tabs, no trailing spaces, no invisible characters, max line length 300 (hard lint limit) but keep lines to 80 chars max where possible
Use `mf_` macros when the macro returns a value usable in an expression; use `mp_` for procedural macros that generate code/statements.
**Cross-suite rule:**`mp_` macros must never reference `mx_` macros. Platform dispatching (SAS 9 / Viya / SASjs server) belongs in the `mx_` suite, which delegates to `ms_`/`mv_`/PROC STP per platform. If an `mp_` macro seems to need platform-specific behaviour, the macro itself belongs in `xplatform/` as an `mx_` macro instead.
## Reuse before writing
Before writing a new macro, check the library for an existing one — common utilities already exist, e.g. `mp_abort` (the deprecated `mf_abort` is retained for backwards compatibility — don't use it in new code), `mf_existds`, `mf_existvar`, `mf_existfileref`, `mf_getuser`, `mp_jsonout` (SAS datasets → JSON for `_webout`), `mp_ds2ddl`, `mp_hashdataset`. Platform-specific variants exist under `meta/`, `viya/`, `server/` and are selected at compile time by the CLI.
## Aborting safely
Never invoke `%mp_abort` from inside an `%if/%else` block — as a procedural macro, the macro processor can continue executing statements after it before the abort takes effect. Use the `iftrue=` condition parameter instead:
```sas
%mp_abort(iftrue= (&syscc ne 0)
,mac=&_program
,msg=%str(Something went wrong)
)
```
When `%mp_abort` is called from within a `%include` block, SAS cannot exit cleanly (e.g. to `_webout`). Call `%mp_abort(mode=INCLUDE)` after the include (OUTSIDE any macro wrapper) — it checks `work.mp_abort_errds` for an abort status:
```sas
%mp_abort(mode=INCLUDE)
```
Note: `%include`s inside macros should be performed with `%mp_include()` so the `_SYSINCLUDEFILEDEVICE` indicator is set and the abort dataset (`work.mp_abort_errds`) is passed back to the calling program.
## Testing macros (mandatory conventions)
- **Always apply `%mp_assertscope` around the macro under test** to catch scope leakage (macro variables must stay `%local`):
```sas
%mp_assertscope(SNAPSHOT)
%mx_foo(args)
%mp_assertscope(COMPARE,
desc=Test 1: mx_foo does not leak scope,
outds=work.test_results
)
```
- Assertions go to `work.test_results` via `%mp_assert(iftrue=(...), desc=..., outds=work.test_results)`.
## Lint and build
- Run `sasjs lint` after every change; do not consider work done until it passes
- NEVER bump the version in `package.json` (semantic-release handles it)
- Do NOT edit generated files by hand: `all.sas`, `mc_*.sas`, the `lua/` wrappers, and `sasjsbuild/` outputs are produced by the CI build
- Markdown files: never hard-wrap; one paragraph per line
description:Building full SASjs applications — project structure, sasjsconfig.json, services/jobs/macros folders, multi-target (SAS 9 / Viya / SASjs server) configuration, streaming frontends, mocks and tests. Use when creating or modifying a SASjs app, editing sasjsconfig.json, or writing backend services returning JSON to a web frontend.
---
# SASjs Framework — Building SASjs Applications
A SASjs app = a web frontend (any framework: Angular, React, vanilla) + SAS backend code organised in a standard layout, compiled and deployed by `@sasjs/cli` to SAS 9, Viya, or SASjs server. Frontend talks to SAS via `@sasjs/adapter`; backend services return JSON via `_webout`.
mocks/ # mock responses for offline frontend dev (syncFolder)
doxy/ # extra doxygen content for `sasjs doc`
```
## sasjsconfig.json
Root config holds defaults; each entry in `targets[]` can override them. Key sections:
-`macroFolders`, `binaryFolders` — where the CLI finds macros/binaries
-`serviceConfig.serviceFolders` — service source folders; `initProgram` runs before every service (set up libnames, options)
-`jobConfig.jobFolders` — job source folders
-`programFolders` — programs compiled/deployed with the app
-`streamConfig` — `streamWeb: true` streams the built frontend into SAS so it is served by the platform itself (no separate web server needed); `webSourcePath` points at the frontend build output
-`syncFolder` — folder synced to the server (e.g. mocks)
-`testConfig` — init/term programs for `sasjs test`
The full JSON schema is bundled at `sasjsconfig-schema.json` next to this file — validate config changes against it. Reference it with `"$schema": "https://cli.sasjs.io/sasjsconfig-schema.json"`.
## Streamed frontend files on Viya (mime types)
When `streamWeb: true`, the CLI uploads the frontend (`index.html`, renamed per `streamServiceName`, plus css/js) to the Viya Files service using the `%mv_createfile` macro. That macro creates the file in a very particular way to ensure it streams correctly:
- POSTs to `/files/files` with the content type derived from the extension (`%mf_mimetype`)
- sets `typeDefName=file_html` (via `%mv_getViyaFileExtParms`) so the file is recognised as HTML
- sends `Content-Disposition`**without**`attachment` for HTML/SVG so it renders in the browser
**Never update a streamed frontend file in place** with a `filename filesrvc` fileref + data step rewrite — the Files service then treats it as a generic blob and the mime type is lost, so the app no longer streams (browser downloads it or shows raw text). To modify a streamed file at runtime (eg patching the compute `contextname` in the html), read it (a `filesrvc` fileref is fine for *reading*), write the modified content to a temp fileref, and **re-create the file with `%mv_createfile(path=..., name=..., inref=...)`** (it deletes the old file and re-POSTs with the correct mime type).
## Service contract (frontend ↔ SAS)
1. Adapter POSTs to `services/<folder>/<name>` with input tables (arrays of objects) → work datasets named after the JS keys.
2. Service SAS code runs after `initProgram`; it reads inputs, does work, and writes output JSON to `_webout`.
3. Conventional pattern using @sasjs/core macros:
```sas
/**
@file
@brief Example service returning data
<h4> SAS Macros </h4>
@li mp_jsonout.sas
@li mp_abort.sas
**/
/* validation / logic here */
%mp_jsonout(OPEN)
%mp_jsonout(OBJ,results,dslabel=results)
%mp_jsonout(CLOSE)
```
4. On error, abort cleanly with `%mp_abort(...)` (`mf_abort` is deprecated) so the adapter receives a structured error in the JSON, not a half-written response. Do **not** call `%mp_abort` inside an `%if/%else` block — the macro processor may keep executing beyond the abort. Use the conditional `iftrue=` parameter instead, e.g.:
```sas
%mp_abort(iftrue= (%mf_existds(work.results)=0)
,mac=&_program
,msg=%str(No results found)
)
```
If the abort happens inside a `%include` block, SAS cannot exit to `_webout` cleanly — after the include, call `%mp_abort(mode=INCLUDE)` (outside any macro wrapper), which checks `work.mp_abort_errds` for an abort status.
## Multi-target discipline
- Keep backend code platform-neutral in shared folders; put platform-specific shims in `targets/<name>/macros_*` folders and register them only on that target.
- Platform capability macros exist in @sasjs/core (`mm_*` metadata, `mv_*` Viya, `ms_*` server) — don't branch on server type by hand.
## Quality gates (follow the conventions of mature apps like Data Controller)
- Run `sasjs lint` after touching any `.sas` file; fix all warnings in files you touched.
- The linter enforces 2-space indentation everywhere, including continuation lines inside `/* ... */` block comments — never align comment text with 3+ spaces.
- Add tests and run `sasjs test` for backend logic changes. When testing macros, always wrap the macro under test with `%mp_assertscope(SNAPSHOT)` / `%mp_assertscope(COMPARE, ...)` to catch macro-variable scope leakage, and wrap any platform-branching code in `%macro` wrappers (no open conditional macro code in test programs).
- Provide mocks in `sasjs/mocks` so the frontend can be developed without a live SAS server.
- Never auto-commit or bump versions; releases are pipeline-driven (conventional commits).
- Markdown files: no hard wrapping — one paragraph per line.
- Apps must work offline/on-prem: no external CDN assets in the frontend bundle.
## Reference implementations
Look at existing apps for patterns: folder layouts, `sasjsconfig.json` multi-target setups, service structure, streaming builds, and test/mock conventions, eg:
"description":"These local folders are searched for Binary Files when running sasjs compile. Folders are relative to the sasjs/sasjsconfig.json file.",
"examples":[["binaries","../../more_binaries"]]
},
"sasjsBuildFolder":{
"$id":"#/properties/sasjsBuildFolder",
"type":"string",
"title":"sasjsBuildFolder",
"description":"The name of the folder containing the compiled output. The `sasjs build` command will take all of the subfolders here as inputs to create the build pack. By default this will be named `sasjsbuild`. In global, the default is `~/.sasjsbuild`.",
"default":"sasjsbuild",
"examples":["sasjsbuild",".sasjsbuild"]
},
"sasjsResultsFolder":{
"$id":"#/properties/sasjsResultsFolder",
"type":"string",
"title":"sasjsResultsFolder",
"description":"The name of the folder containing the output (eg logs, ODS output) from `sasjs run`. By default this will be named `sasjsresults`. In global, the default is `~/.sasjsresults`.",
"default":"sasjsresults",
"examples":["sasjsresults",".sasjsresults"]
},
"defaultTarget":{
"$id":"#/properties/defaultTarget",
"type":"string",
"title":"Default Target",
"description":"If a target is not specified, this target is used by default. The default target must exist in the (local) targets array.",
"default":"viya",
"examples":["viya"]
},
"docConfig":{
"$id":"#/properties/docConfig",
"type":"object",
"title":"The docConfig schema",
"description":"SASjs uses doxygen to auto-generate HTML documentation using the headers in your SAS programs, macros, services & jobs. For more info, see [https://cli.sasjs.io/doc](https://cli.sasjs.io/doc). Some properties are taken from package.json (such as the project Name).",
"description":"The CLI will autocompile macro dependencies that exist in the SASjs Macro Core library. These will also show in the documentation under 'node_modules'. If you'd prefer not to show these in the rendered docs, set this value to false.",
"description":"The location to which the generated HTML SAS documentation is written. If missing, or left blank, the files will be written to the `sasjsbuild/doc` directory (default behaviour).",
"description":"Provide the full URL to Data Controller so that `sasjs doc` can link the lineage diagram directly to the table viewer in [Data Controller](https://datacontroller.io).\nIf left blank, or undefined, no links will be generated.",
"description":"If you are having certificate errors connecting to SAS, that cannot be properly resolved, try setting this value to true. This option only has an effect if rejectUnauthorized is not present.",
"description":"Optionally override the trusted CA certificates. Default is to trust the well-known CAs curated by Mozilla. Mozilla's CAs are completely replaced when CAs are explicitly specified using this option.",
"description":"Private keys in PEM format. PEM allows the option of private keys being encrypted. Encrypted keys will be decrypted with options.passphrase. Multiple keys using different algorithms can be provided either as an array of unencrypted key strings or buffers, or an array of objects in the form {pem: <string|buffer>[, passphrase: <string>]}. The object form can only occur in an array. object.passphrase is optional. Encrypted keys will be decrypted with object.passphrase if provided, or options.passphrase if it is not.",
"description":"Cert chains in PEM format. One cert chain should be provided per private key. Each cert chain should consist of the PEM formatted certificate for a provided private key, followed by the PEM formatted intermediate certificates (if any), in order, and not including the root CA (the root CA must be pre-known to the peer, see ca). When providing multiple cert chains, they do not have to be in the same order as their private keys in key. If the intermediate certificates are not provided, the peer will not be able to validate the certificate, and the handshake will fail.",
"description":"If true the server will reject any connection which is not authorized with the list of supplied CAs. This option only has an effect if requestCert is true.",
"default":true,
"examples":[true,false]
}
}
},
"buildConfig":{
"$id":"#/properties/buildConfig",
"type":"object",
"title":"buildConfig",
"description":"Dictates which files get compiled into the build program (.sas), used to deploy services into SAS 9 or Viya environments (without a client/secret). You may use this config to include build specific macros, programs or macro variables - which is run a single time, on deployment - for things like database creation, or exporting a SAS 9 SPK after service creation.",
"description":"The name of the generated .sas program, which can be used to deploy the app using only SAS Studio. By default, this will be the name of the target.",
"description":"A series of name value pairs that will be turned into SAS macro variables in the build .sas program (generated when running `sasjs build`). The example provided will generate sas code as follows:\n```\n%let name=value;\n%let numvar=42;\n```",
"default":{},
"examples":[
{
"name":"value",
"numvar":"42"
}
]
}
}
},
"deployConfig":{
"$id":"#/properties/deployConfig",
"type":"object",
"title":"The deployConfig schema",
"description":"The deployConfig object enables settings that relate to the deployment of a SAS app - be that Viya, or SAS 9, or a pure Base environment.",
"description":"These scripts are executed when running `sasjs deploy`. If the file is a .sas file, it is executed on the SAS server (Viya only). Otherwise it is executed locally. These scripts are run AFTER the deployment of the servicepack, if `deployServicePack:true` (Viya only).",
"description":"If set to `true` the json pack produced by `sasjs build` will be auto-deployed to the `appLoc` of the specified target (creating all jobs and services in the SAS folder tree). Currently only Viya is supported for this flag.",
"default":false
}
}
},
"serviceConfig":{
"$id":"#/properties/serviceConfig",
"type":"object",
"title":"The serviceConfig schema",
"description":"The serviceConfig object defines how SASjs web services are compiled. Web services differ from jobs in that they include some fixed pre-code (eg the macros to stream out the result json).",
"description":"When running `sasjs compile`, all programs in the folders defined in this array are compiled and placed into same-named folders under `sasjsbuild/services`. They will be compiled as services (so, with the service pre-code). Folders can be absolute, or relative to the `sasjs` folder.",
"description":"The serviceConfig `initProgram` is a .sas file that is inserted at the start of every SAS service (after compiled macros and any `macroVars`, and before the service itself). ",
"description":"This object allows `sasjs compile` to insert specific macro variables at the start of every service. In this case, the code generated would be:\n```\n%let mac1=value;\n%let mac2=42;\n```",
"default":{},
"examples":[
{
"mac1":"value",
"mac2":"42"
}
]
}
}
},
"jobConfig":{
"$id":"#/properties/jobConfig",
"type":"object",
"title":"The jobConfig schema",
"description":"The jobConfig object defines how SASjs Jobs are compiled. ",
"description":"When running `sasjs compile`, all programs in the local folders defined in this array are compiled and placed into same-named folders under `sasjsbuild/jobs`. Folders can be absolute, or relative to the local project `/sasjs` folder.",
"description":"The jobConfig `initProgram` is a local .sas file that is inserted at the start of every SAS Job (after compiled macros and any `macroVars`, and before the Job itself). ",
"description":"This object allows `sasjs compile` to insert specific macro variables at the start of every Job. In this case, the code generated would be:\n```\n%let mac1=value;\n%let mac2=42;\n```",
"default":{},
"examples":[
{
"mac1":"value",
"mac2":"42"
}
]
}
}
},
"streamConfig":{
"$id":"#/properties/streamConfig",
"type":"object",
"title":"streamConfig",
"description":"SASjs allows a local web app to be compiled such that all html, css, javascript, and other assets such as png or mp4 are converted into web services and streamed directly from SAS.\n\nThis approach is convenient as it bypasses the need to deploy to a web server. ",
"description":"An array of local folders. All assets placed in these folders are converted into web services - example file types could be png, svg, mp3, mp4, excel - anything really.",
"description":"When set to `true`, frontend files saved in the `webSourcePath` will be converted to streaming services in the `streamWebFolder` in SAS.",
"description":"Active when `streamConfig` is `true`. Is the source (or build, or dist) LOCAL folder, relative to the `sasjs` folder, which contains the frontend to be deployed. All assets (PNG, JS, CSS, HTML etc) are taken from here and converted to streaming services in `streamWebFolder`. In Viya and SAS 9, any relative URLS will be modified such that the links still work and the assets still load.",
"description":"The name of the service containing the `index.html` for a streaming web app. Defaults to `clickme` and is always deployed under the appLoc/services SAS Folder.",
"default":"clickme.sas"
}
}
},
"testConfig":{
"$id":"#/properties/testConfig",
"type":"object",
"title":"testConfig",
"description":"Create tests for Macros, Services & Jobs by simply adding a '.test.sas' extension. ",
"description":"The testConfig `initProgram` is a local .sas file that is inserted at the start of every Test (after compiled macros and any `macroVars`, and before the Test itself). ",
"description":"This object allows `sasjs compile` to insert specific macro variables at the start of every Test. In this case, the code generated would be:\n```\n%let mac1=value;\n%let mac2=42;\n```",
"description":"This program is the first to execute as part of 'sasjs test'. It does not contain the testInit, testTerm or macroVariables. It IS compiled.",
"description":"The last program to execute as part of 'sasjs test'.",
"default":"sasjs/tests/testteardown.sas",
"examples":["sasjs/tests/testteardown.sas"]
}
}
},
"macroFolders":{
"$id":"#/properties/macroFolders",
"type":"array",
"title":"The macroFolders array",
"description":"These local folders are searched for SAS Macros when running `sasjs compile`. Folders are relative to the `sasjs/sasjsconfig.json` file.",
"default":[],
"examples":[["macros","../../more_macros"]]
},
"programFolders":{
"$id":"#/properties/programFolders",
"type":"array",
"title":"The programFolders array",
"description":"These local folders are searched for SAS Programs when running `sasjs compile`. Folders are relative to the `sasjs/sasjsconfig.json` file.",
"default":[],
"examples":[["programs","../../more_programs"]]
},
"syncFolder":{
"$id":"#/properties/syncFolder",
"type":"string",
"title":"Sync Folder",
"description":"The contents of this folder are simply copied to the sasjsbuild directory AFTER the rest of the project is compiled. Useful for synchronising random / generic content with SAS logical folders.",
"default":"sasjs/static_files"
},
"syncDirectories":{
"$id":"#/properties/syncDirectories",
"type":"array",
"title":"syncDirectories",
"description":"Maps the local filesystem to remote (SAS) physical directories.",
"examples":[
[
{
"local":"C:\\temp\\local\\fs1",
"remote":"/opt/data/fs1"
},
{
"local":"C:\\temp\\elsewhere",
"remote":"/opt/somewhere"
}
]
]
},
"targets":{
"$id":"#/properties/targets",
"type":"array",
"title":"The targets array",
"description":"A target is an alias for a deployment location, and includes at a minimum, the `serverUrl`, `serverType` and `appLoc`. This array allows multiple targets to be defined (eg dev / test / prod). Any properties defined here will override same-named properties in the sasjsconfig root.",
"description":"A target name can only contain alphanumeric characters and dashes. It cannot contain spaces. It is used as the alias when referencing the target using the `-t` attribute in many of the SASjs commands.",
"description":"The appLoc provides the root SAS folder location under which all jobs and services are deployed and executed. The SAS folder could be metadata in SAS 9, or SAS Drive in Viya. ",
"description":"The name of the compute context used to execute SAS code. The context determines the way in which the SAS session is spawned (eg user credentials, autoexec code, system options etc).\nContexts can be created / modified / deleted using the `sasjs context` command.",
"description":"The serverUrl is the location to which the app is deployed, and against which any server based operations are performed. If SAS is served from a particular port, that port should also be included here.",
description:Installing, configuring, and running @sasjs/server — the open-source NodeJS wrapper around the SAS binary that provides a REST API, filesystem (SASjs Drive), Stored Program execution, and web app streaming. Covers desktop vs server modes, runtimes (SAS/JS/Python/R), env vars, auth (tokens, LDAP), and mock servers. Use when deploying, troubleshooting, or developing against sasjs/server.
---
# @sasjs/server
SASjs Server is an open-source NodeJS wrapper for calling the SAS binary executable. It runs on a real SAS server or a local desktop and provides:
- A filesystem (SASjs Drive) for storing SAS programs and content
- Execution of Stored Programs from a URL (equivalent to SAS 9 Stored Processes / Viya Jobs)
- Web app streaming (serve frontend apps straight from SAS content)
- A REST API with Swagger docs
- Portability: apps built for SASjs Server deploy unchanged to SAS 9 / Viya via @sasjs/cli
## Modes
- **Desktop mode** (`MODE=desktop`, default): single-user, no authentication, no database. CORS enabled by default.
- **Server mode** (`MODE=server`): multi-user with authentication, requires a database (`DB_CONNECT`, `DB_TYPE=mongodb|cosmos_mongodb`). CORS disabled by default — configure `WHITELIST` if enabling.
## Installation
Download the relevant zip from GitHub releases and run the packaged executable (`api-linux`, etc.):
On first run it prompts (unless set as env vars) for the SAS executable path and the filesystem location for Stored Programs/temp files. Docker is also supported (`DockerfileApi`, docker-compose files in the repo).
## Configuration via environment variables
Set in `/etc/environment`, exported, prepended to the command, or in a `.env` file alongside the executable. Key variables:
| Variable | Purpose |
|---|---|
| `MODE` | `desktop` (default) or `server` |
| `SAS_PATH` | Path to `sas.exe` / `sas.sh` |
| `RUN_TIMES` | Comma-separated runtime priority, e.g. `sas,js,py` — options: `sas`, `js`, `py`, `r`. Each needs its path: `SAS_PATH`, `NODE_PATH`, `PYTHON_PATH`, `R_PATH` |
| `SASJS_ROOT` | Working directory: SAS WORK, staged files, drive, config |
| `DRIVE_LOCATION` | Location for files, sasjs packages, `appStreamConfig.json` |
| `PROTOCOL` / `PORT` | `http` (default) or `https` (needs `PRIVATE_KEY`, `CERT_CHAIN`, optional `CA_ROOT`); default port 5000 |
| `SAS_OPTIONS` / `SASV9_OPTIONS` | Extra SAS system options auto-applied to sessions (Windows vs Unix), e.g. `-NOXCMD` |
| `DB_CONNECT` / `DB_TYPE` | MongoDB connection string / type — required for server mode |
| `CORS` / `WHITELIST` | CORS is only applied when `CORS=enable`, and only origins in `WHITELIST` (space-separated) receive `Access-Control-Allow-Origin` — an empty whitelist means NO cross-origin calls work |
| `MOCK_SERVERTYPE` / `STATIC_MOCK_LOCATION` | Emulate `sas9`/`sasviya` API responses for frontend testing against a sasjs server (static canned files only — no logic) |
## Developing against the API
- Server type for @sasjs/adapter / CLI targets is `SASJS` (`serverType: 'SASJS'`); auth is token-based.
- The REST API is self-documented via Swagger on the running instance.
- Server-side execution uses `ms_*` macros from @sasjs/core (e.g. `ms_createfile`, `ms_adduser2group`) — services/jobs deployed by the CLI work as on other platforms.
## Mock services with the JS runtime (no SAS required)
With `RUN_TIMES=js` (and `NODE_PATH` set), any `.js` file on SASjs Drive is an executable Stored Program — this is how react-seed-app / Data Controller provide **mock backends** for frontend development. Desktop mode (`MODE=desktop`) has no auth, which makes local mocking trivial.
Writing a JS stored program (docs: https://server.sasjs.io/storedprograms/#js-programs):
- The runtime template predeclares `const fs = require('fs')`, `_program`, `weboutPath`, `_SASJS_TOKENFILE`, `_SASJS_WEBOUT_HEADERS`, `_SASJS_USERNAME` / `_SASJS_USERID` / `_SASJS_DISPLAYNAME`, `_METAPERSON`, `_METAUSER`, `SASJSPROCESSMODE`. **Do NOT redeclare `fs`** — `const fs = require('fs')` in your program crashes it with `Identifier 'fs' has already been declared`.
- Output: assign a JSON **string** to `_webout` (e.g. `_webout = JSON.stringify({...})`); it is written back only if non-empty. `console.log()` output is returned in the response `log` (like a SAS log). Custom response headers can be written as lines to the `_SASJS_WEBOUT_HEADERS` file.
- Mimic real services by including the standard SASjs automatic fields in the JSON: `_PROGRAM` (from `_program`), `SYSDATE` / `SYSTIME` (format `DDMMMYY` / `HH:mm`), `_METAUSER`, `SASJSPROCESSMODE`.
- URL/body parameters arrive as `const <name> = \`<value>\`` strings.
- Input tables (the `sasjs_tables` mechanism) arrive **either** as an inline CSV const **or** — when the adapter sends multipart — as an uploaded `<name>.csv` file in the session folder, referenced by generated module-scope consts (handle BOTH):
-`_WEBIN_FILE_COUNT` (always created), `_WEBIN_NAME<n>` (table/field name), `_WEBIN_FILENAME<n>` (original filename), `_WEBIN_FILEREF<n>` (file **contents**, a Buffer from `fs.readFileSync` — call `.toString('utf8')`)
- these consts are **not on `globalThis`** — look them up with `typeof` guards or direct `eval()` in module scope (server-side JS, no CSP)
- adapter CSV quirks: header row is **space-separated**`name:format.` entries (e.g. `rootdir:$char256.`) — strip the `:format` suffix; lines end CRLF; values containing special characters are wrapped in double quotes with `""` escaping
- Adapter response shape: `sasjs.request()` resolves with the webout JSON **already unwrapped** — output tables are arrays of row objects directly on the response (`res.mytable[0].COL`). A table named `result` is perfectly fine (`res.result` is then that array); do NOT add your own `res.result`-unwrapping layer, it breaks exactly that case.