feat: release 7.17 - DDL export, Handsontable 18.1.1, EDIT row and cell limits #333

Open

hermes wants to merge 14 commits from feat/export-dc-library-ddl into main

pull from: feat/export-dc-library-ddl

merge into: dc:main

dc:main

dc:feat/mock-config-and-ddl

dc:licensing

dc:lighthouse-pages

dc:fix/audit-20260403

dc:demodata

dc:css-refactor

dc:fix-nldatm-2

dc:development

Conversation 3 Commits 14 Files Changed 25

+1229 -929

hermes commented 2026-10-02 14:04:46 +00:00

Collaborator

Copy link

Copy source

Closes #179

What

The 7.17 release batch. Three changes that were reviewed as separate PRs have since been merged into this branch, so this PR against main now carries all of them:

change	merged as	effect
Export DC Library DDL	this PR	a fourth admin action on the System information screen
Handsontable 18.0.0 -> 18.1.1	#334	three upstream bug fixes, plus large-dataset performance
EDIT row and cell limits	#335	VIEW 500 -> 2000 rows; EDIT 100 -> 250 rows, plus a new 200,000 cell limit
Dependency and CI fixes	#336 and earlier commits	clears the audit gate, and gets the mocked-estate and Lighthouse jobs running again

semantic-release will cut 7.17.0 from the feat commits on merge - main is on 7.16.0. 25 files, +1229/-929.

---

1. Export DC Library DDL

Adds a fourth admin action to the System information screen - Export DC Library DDL - that downloads the whole Data Controller library as database-specific DDL.

Clicking EXPORT opens a secondary chooser for the flavour of the export:

• SAS (default)
• PGSQL
• TSQL

Choosing PGSQL or TSQL reveals an optional Schema box. The selection is passed to the existing services/admin/exportdb service, opened in a new window, as URL params - the same pattern the Download Configuration action already uses:

• .../services/admin/exportdb&flavour=SAS
• .../services/admin/exportdb&flavour=PGSQL&schema=DC

The backend service (sas/sasjs/services/admin/exportdb.sas) already accepted flavour and schema; this change wires it up in the UI and corrects the service's doc header, which claimed only PGSQL was supported. Inserts are generated for the SAS and PGSQL flavours; the TSQL export is DDL only, which is how mp_lib2inserts behaves.

Screenshots

All taken against the JS mock backend (mocked estate), 1600x900, licence applied.

The new admin action on the System information screen

The flavour chooser - defaults to SAS, no schema box

PGSQL reveals the optional schema box

TSQL, schema left blank

---

2. Handsontable 18.1.1 (#334)

handsontable and @handsontable/angular-wrapper 18.0.0 -> 18.1.1, plus the two repo-owned places that pin the version (client/licenseChecker.js excludePackages, and the generated client/src/_hot-icons.scss).

It fixes three upstream bugs, two of which this repo's own specs had pinned as "demonstrate the failure mode" tests. Those tests now pin the fixed behaviour instead:

• Sheet size limit exceeded (upstream #10672). The formulas plugin pushed the grid's maxRows into the HyperFormula engine as its sheet-size limit, so a licensing cap below an existing table's row count made the engine reject the whole table.
• Formula corruption when updateSettings runs while a sort is active - calling it even with a bare {} turned formula cells into #REF!.
• A crash in the filters plugin on the path DC actually uses (upstream #13480): editing a cell in a filtered column, or replacing the data with a filter active, after updateSettings was called with the filters option.

18.1 also tightened ColumnSettings['validator'] to its real type, which needed a small validatorOf() helper in dc-validator.spec.ts (mirroring the guard the production code already uses).

Measured, in the library - same page, same Chrome, 50,000 rows x 12 columns, best of 3:

operation (50k rows)	18.0.0	18.1.1	change
initial render	1,317 ms	1,232 ms	-6%
sort, one column	490 ms	21 ms	-96%
loadData with a filter active	1,699 ms	1,265 ms	-26%
updateSettings	1,042 ms	588 ms	-44%
200 scattered cell edits	108,519 ms	7,842 ms	-93%
scroll to the last row	540 ms	38 ms	-93%

Measured, end to end - mocked estate, ng serve, Cypress/Electron, timed click-to-first-row-paint, so service + transfer + parse + grid build are all inside the number. Narrow is the MPE_X_TEST column set (9 columns), wide is WIDEBOY (1,001 columns):

shape	mode	rows	cols	18.0.0	18.1.1	change
narrow	VIEW	2,000	9	1,552	1,048	-32%
narrow	EDIT	2,000	9	8,779	6,234	-29%
wide	VIEW	1,000	1,001	9,448	7,890	-16%
wide	EDIT	1,000	1,001	131,046	138,013	+5%

The wide EDIT figure is unchanged, and that is the point of the next section: that cost is DC's own per-cell work (the editor's cells function, validators, renderers) over up to a million cells, not Handsontable's render, so no grid upgrade can remove it.

---

3. EDIT row and cell limits (#335)

option	was	now
DC_MAXOBS_WEBVIEW (VIEW)	500	2000
DC_MAXOBS_WEBEDIT (EDIT)	100	250
DC_MAXCELLS_WEBEDIT (EDIT)	-	200000

The EDIT screen now refuses a selection that is too big on either count - rows or cells (rows x columns) - whichever is reached first. VIEW's 500 was doing very little work: a 9-column table is ~1 s to load at 2,000 rows (~420 KB). EDIT needed more than a bigger number, because the cost tracks cells, not rows: 100 rows of a 1,001-column table already takes ~17 s and 1,000 rows takes ~2.3 minutes, which a row cap cannot bound. 250 rows x 200,000 cells keeps both ends in proportion - a 9-column table hits the row limit at 250 rows / 2,250 cells, a 1,001-column table hits the cell limit at 200 rows.

The guard (getdata.sas) is a single check, run after the sort and after PRE_EDIT_HOOK, so the counts cover everything that could reach the browser: the rows the selection matched, plus anything a hook has added to work.out. The payload is capped at the row limit where work.outdata is built.

The mocks mirror it - viewdata.js cap, the row/cell guard in getdata.js (emitted as sasjsAbort), the three config rows, and a new MPE_X_WIDE fixture (200 rows x 1,001 columns = 200,400 cells).

Behaviour change: an existing install keeps 500 / 100 until the migration runs - see Deploy notes.

---

4. Dependency and CI fixes

The audit gate is strict (npm audit --omit=dev in root, sas and client, any advisory fails), and advisories were published after the last green build - they failed on main too:

• chore(sas): @sasjs/cli 4.20.4 -> 4.20.5, bringing @sasjs/adapter 4.19.1 and axios 1.20.0 into the SAS project.
• chore(client): @angular/* 20.3.30 -> 20.3.33, the fast-uri override 3.1.7 -> 3.1.8, and @sasjs/adapter ^4.18.0 -> ^4.19.1, plus npm audit fix.
• #336: braces (GHSA-vfj7-8cjw-p6xm) has no fixed version - it arrives via @sasjs/cli > shelljs > fast-glob > micromatch > braces, and npm's only remedy is downgrading @sasjs/cli to 4.13.1. shelljs is pinned to 0.8.5 instead (0.9.0 is what introduced fast-glob), which removes braces, micromatch and fast-glob from the tree. The CLI only uses shelljs for ls/cp/rm/exec, unchanged between the two versions. This is a workaround - the real fix belongs upstream in @sasjs/cli.

CI itself was failing with ECONNREFUSED :5000: recent sasjs/server releases refuse to run as root unless RUN_AS names an account, and the runners are root, so the server exited at startup and pm2 restart-looped. RUN_AS=root (the server's own documented override) is now set in build.yaml, lighthouse.yaml and release.yaml, and the workflows wait for the server to answer before deploying mocks.

---

Testing

• npm run build (production) passes; unit tests 548/548
• sasjs lint clean on the changed SAS files; sasjs compile -t server succeeds
• Cypress against the mocked estate - 138/138 across 17 specs, including the two new ones: ddl-export.cy.ts 3/3 and row-cell-limits.cy.ts 3/3. The same 138/138 passes on the Handsontable 18.1.1 combination (run 1861), so row-cell-limits.cy.ts - which depends on the grid's scroll/virtualisation behaviour - is verified on the version this branch ships
• all three CI checks green on the individual heads, and re-running on the batch head
• npm audit --omit=dev clean in both projects
• SAS test suite run on NEXTVIYA - 40/48 passing, the rest estate-side permission/guard limits, not code; the DDL export path was verified directly on that estate (SAS and PGSQL with inserts, TSQL DDL only, all 41 tables)

Deploy notes

• sas/sasjs/db/migrations/20261003_row_and_cell_limits.sas is optional, but an existing install needs it to move off 500 / 100 - the new defaults only apply to options that are absent. It closes the current MPE_CONFIG row and writes a new one, and it skips options a site has switched off (a current row with var_active=0), logging a note, rather than silently switching them back on.
• docs.datacontroller.io is a separate repo: PR #14 updates docs/dcc-options.md for the new defaults and the new option.

Closes #179 ## What The **7.17 release batch**. Three changes that were reviewed as separate PRs have since been merged into this branch, so this PR against `main` now carries all of them: | change | merged as | effect | |---|---|---| | Export DC Library DDL | this PR | a fourth admin action on the System information screen | | Handsontable 18.0.0 -> 18.1.1 | #334 | three upstream bug fixes, plus large-dataset performance | | EDIT row and cell limits | #335 | VIEW 500 -> 2000 rows; EDIT 100 -> 250 rows, plus a new 200,000 cell limit | | Dependency and CI fixes | #336 and earlier commits | clears the audit gate, and gets the mocked-estate and Lighthouse jobs running again | `semantic-release` will cut **7.17.0** from the `feat` commits on merge - `main` is on 7.16.0. 25 files, +1229/-929. --- ## 1. Export DC Library DDL Adds a fourth admin action to the **System information** screen - **Export DC Library DDL** - that downloads the whole Data Controller library as database-specific DDL. Clicking **EXPORT** opens a secondary chooser for the flavour of the export: - **SAS** (default) - **PGSQL** - **TSQL** Choosing PGSQL or TSQL reveals an optional **Schema** box. The selection is passed to the existing `services/admin/exportdb` service, opened in a new window, as URL params - the same pattern the **Download Configuration** action already uses: - `.../services/admin/exportdb&flavour=SAS` - `.../services/admin/exportdb&flavour=PGSQL&schema=DC` The backend service (`sas/sasjs/services/admin/exportdb.sas`) already accepted `flavour` and `schema`; this change wires it up in the UI and corrects the service's doc header, which claimed only PGSQL was supported. Inserts are generated for the SAS and PGSQL flavours; the TSQL export is DDL only, which is how `mp_lib2inserts` behaves. ### Screenshots All taken against the JS mock backend (mocked estate), 1600x900, licence applied. **The new admin action on the System information screen**  **The flavour chooser - defaults to SAS, no schema box**  **PGSQL reveals the optional schema box**  **TSQL, schema left blank**  --- ## 2. Handsontable 18.1.1 (#334) `handsontable` and `@handsontable/angular-wrapper` 18.0.0 -> 18.1.1, plus the two repo-owned places that pin the version (`client/licenseChecker.js` `excludePackages`, and the generated `client/src/_hot-icons.scss`). **It fixes three upstream bugs, two of which this repo's own specs had pinned as "demonstrate the failure mode" tests.** Those tests now pin the fixed behaviour instead: - `Sheet size limit exceeded` (upstream #10672). The formulas plugin pushed the grid's `maxRows` into the HyperFormula engine as its sheet-size limit, so a licensing cap below an existing table's row count made the engine reject the whole table. - Formula corruption when `updateSettings` runs while a sort is active - calling it even with a bare `{}` turned formula cells into `#REF!`. - A crash in the filters plugin on the path DC actually uses (upstream #13480): editing a cell in a filtered column, or replacing the data with a filter active, after `updateSettings` was called with the `filters` option. 18.1 also tightened `ColumnSettings['validator']` to its real type, which needed a small `validatorOf()` helper in `dc-validator.spec.ts` (mirroring the guard the production code already uses). **Measured, in the library** - same page, same Chrome, 50,000 rows x 12 columns, best of 3: | operation (50k rows) | 18.0.0 | 18.1.1 | change | |---|---|---|---| | initial render | 1,317 ms | 1,232 ms | -6% | | sort, one column | 490 ms | 21 ms | **-96%** | | loadData with a filter active | 1,699 ms | 1,265 ms | -26% | | updateSettings | 1,042 ms | 588 ms | -44% | | 200 scattered cell edits | 108,519 ms | 7,842 ms | **-93%** | | scroll to the last row | 540 ms | 38 ms | **-93%** | **Measured, end to end** - mocked estate, `ng serve`, Cypress/Electron, timed click-to-first-row-paint, so service + transfer + parse + grid build are all inside the number. Narrow is the `MPE_X_TEST` column set (9 columns), wide is `WIDEBOY` (1,001 columns): | shape | mode | rows | cols | 18.0.0 | 18.1.1 | change | |---|---|---|---|---|---|---| | narrow | VIEW | 2,000 | 9 | 1,552 | 1,048 | -32% | | narrow | EDIT | 2,000 | 9 | 8,779 | 6,234 | -29% | | wide | VIEW | 1,000 | 1,001 | 9,448 | 7,890 | -16% | | wide | EDIT | 1,000 | 1,001 | 131,046 | 138,013 | +5% | The wide EDIT figure is unchanged, and that is the point of the next section: that cost is DC's own per-cell work (the editor's `cells` function, validators, renderers) over up to a million cells, not Handsontable's render, so no grid upgrade can remove it. --- ## 3. EDIT row and cell limits (#335) | option | was | now | |---|---|---| | `DC_MAXOBS_WEBVIEW` (VIEW) | 500 | **2000** | | `DC_MAXOBS_WEBEDIT` (EDIT) | 100 | **250** | | `DC_MAXCELLS_WEBEDIT` (EDIT) | - | **200000** | The EDIT screen now refuses a selection that is too big on **either** count - rows *or* cells (rows x columns) - whichever is reached first. VIEW's 500 was doing very little work: a 9-column table is ~1 s to load at 2,000 rows (~420 KB). EDIT needed more than a bigger number, because the cost tracks cells, not rows: 100 rows of a 1,001-column table already takes ~17 s and 1,000 rows takes ~2.3 minutes, which a row cap cannot bound. 250 rows x 200,000 cells keeps both ends in proportion - a 9-column table hits the row limit at 250 rows / 2,250 cells, a 1,001-column table hits the cell limit at 200 rows. **The guard** (`getdata.sas`) is a single check, run after the sort and after `PRE_EDIT_HOOK`, so the counts cover everything that could reach the browser: the rows the selection matched, plus anything a hook has added to `work.out`. The payload is capped at the row limit where `work.outdata` is built. **The mocks mirror it** - `viewdata.js` cap, the row/cell guard in `getdata.js` (emitted as `sasjsAbort`), the three config rows, and a new `MPE_X_WIDE` fixture (200 rows x 1,001 columns = 200,400 cells). **Behaviour change:** an existing install keeps 500 / 100 until the migration runs - see Deploy notes. --- ## 4. Dependency and CI fixes The audit gate is strict (`npm audit --omit=dev` in root, `sas` and `client`, any advisory fails), and advisories were published after the last green build - they failed on `main` too: - `chore(sas)`: `@sasjs/cli` 4.20.4 -> 4.20.5, bringing `@sasjs/adapter` 4.19.1 and axios 1.20.0 into the SAS project. - `chore(client)`: `@angular/*` 20.3.30 -> 20.3.33, the `fast-uri` override 3.1.7 -> 3.1.8, and `@sasjs/adapter` ^4.18.0 -> ^4.19.1, plus `npm audit fix`. - `#336`: **`braces` (GHSA-vfj7-8cjw-p6xm) has no fixed version** - it arrives via `@sasjs/cli` > `shelljs` > `fast-glob` > `micromatch` > `braces`, and npm's only remedy is downgrading `@sasjs/cli` to 4.13.1. `shelljs` is pinned to 0.8.5 instead (0.9.0 is what introduced `fast-glob`), which removes `braces`, `micromatch` and `fast-glob` from the tree. The CLI only uses `shelljs` for `ls`/`cp`/`rm`/`exec`, unchanged between the two versions. This is a workaround - the real fix belongs upstream in `@sasjs/cli`. **CI itself** was failing with `ECONNREFUSED :5000`: recent `sasjs/server` releases refuse to run as root unless `RUN_AS` names an account, and the runners are root, so the server exited at startup and pm2 restart-looped. `RUN_AS=root` (the server's own documented override) is now set in `build.yaml`, `lighthouse.yaml` and `release.yaml`, and the workflows wait for the server to answer before deploying mocks. --- ## Testing - [x] `npm run build` (production) passes; unit tests 548/548 - [x] `sasjs lint` clean on the changed SAS files; `sasjs compile -t server` succeeds - [x] Cypress against the mocked estate - **138/138 across 17 specs**, including the two new ones: `ddl-export.cy.ts` 3/3 and `row-cell-limits.cy.ts` 3/3. The same 138/138 passes on the Handsontable 18.1.1 combination (run 1861), so `row-cell-limits.cy.ts` - which depends on the grid's scroll/virtualisation behaviour - is verified on the version this branch ships - [x] all three CI checks green on the individual heads, and re-running on the batch head - [x] `npm audit --omit=dev` clean in both projects - [x] SAS test suite run on NEXTVIYA - 40/48 passing, the rest estate-side permission/guard limits, not code; the DDL export path was verified directly on that estate (SAS and PGSQL with inserts, TSQL DDL only, all 41 tables) ## Deploy notes - **`sas/sasjs/db/migrations/20261003_row_and_cell_limits.sas`** is optional, but an existing install needs it to move off 500 / 100 - the new defaults only apply to options that are absent. It closes the current `MPE_CONFIG` row and writes a new one, and it **skips options a site has switched off** (a current row with `var_active=0`), logging a note, rather than silently switching them back on. - `docs.datacontroller.io` is a separate repo: **PR #14** updates `docs/dcc-options.md` for the new defaults and the new option.

01-system-export-row.png

125 KiB

02-ddl-modal-sas.png

118 KiB

03-ddl-modal-pgsql.png

114 KiB

04-ddl-modal-tsql.png

122 KiB

hermes added 1 commit 2026-10-02 14:04:47 +00:00

feat(system): export the DC library DDL from the system screen ... 05f3f49e0c

Adds a fourth admin action to the System information screen - Export DC
Library DDL - with a secondary flavour chooser (SAS, PGSQL, TSQL) and an
optional schema box for the DB flavours. The selection is passed to the
existing services/admin/exportdb service in a new window as URL params.

Also bumps @sasjs/adapter to ^4.19.1, adds a Cypress spec for the three
flavour states, and adds a TSQL case to the exportdb SAS test.

Closes #179

hermes added 1 commit 2026-10-02 14:18:02 +00:00

chore(sas): bump @sasjs/cli to 4.20.5 ... c019224e96

4.20.5 brings @sasjs/adapter 4.19.1, whose axios dependency is 1.20.0.
The previous CLI resolved axios 1.18.1, which the current npm audit
advisories flag (axios 1.0.0 - 1.19.0), failing the client-side
`npm audit --omit=dev` check in CI.

hermes added 1 commit 2026-10-02 14:33:37 +00:00

chore(client): clear newly-published npm audit advisories ... 644a7eb33d

The CI "Check audit" step (`npm audit --omit=dev`) now fails on the
client tree because of advisories published after the last green build:

- @angular/router 20.0.0 - 20.3.31 (SSR DoS) - bump @angular/* to 20.3.33
- fast-uri 3.0.0 - 3.1.7 - bump the override to 3.1.8
- brace-expansion, moment, undici - updated by `npm audit fix`

`npm audit --omit=dev` is clean after this. Verified with the production
build, the 548 unit tests and the Cypress suite.

hermes added 1 commit 2026-10-02 15:26:11 +00:00

ci: wait for the SASjs Server to answer before deploying mocks ... a0c9349e08

Both workflows start api-linux with `pm2 start api-linux --wait-ready`
and then hit http://localhost:5000 in the next step. pm2 can return
while the app is still 'launching' (it does not always block on the
ready signal), so the deploy step races the server and fails with
"Couldn't connect to server".

Wait up to 120s for the server to answer, and print `pm2 list` plus the
api-linux log if it never does, so a genuine startup failure is visible
in the job output instead of a bare ECONNREFUSED.

hermes added 1 commit 2026-10-02 15:31:31 +00:00

ci: set RUN_AS=root so the SASjs Server starts in the runner ... 1354fced46

Recent sasjs/server releases refuse to run as root unless RUN_AS names
an account (security hardening, "refuse to run as root unless RUN_AS
names an account"). The CI runners are root, so the server exited at
startup and pm2 restart-looped (103 restarts), leaving :5000
unreachable - which is what failed the mocked-estate deploy and the
Lighthouse job.

The server's own message offers RUN_AS=root as the deliberate override;
opt in explicitly for the throwaway CI container.

hermes added 1 commit 2026-10-02 16:16:38 +00:00

ci: apply the RUN_AS=root fix to the release workflow too ... e49e8e6710

release.yaml starts the SASjs Server the same way as build.yaml and
lighthouse.yaml, so the same root guard (sasjs/server v1.9.0) breaks it.
It runs on pushes to main and has not run since the guard shipped, so
the next release would have failed at the mock deploy.

hermes referenced this pull request 2026-10-02 17:55:09 +00:00

chore(deps): upgrade Handsontable to 18.1.1 #334

hermes reviewed 2026-10-03 10:08:53 +00:00

hermes left a comment

Author

Collaborator

Copy link

Copy source

Reviewed the full base...head diff (12 files: exportdb service and tests, system screen modal, new cypress spec, three CI workflows, client and sas lockfiles) plus repo hardening at this head.

• .pre-commit-config.yaml (missing) - the repo's gitleaks scan runs only from .git-hooks/pre-commit, which the root prepare script wires via git config core.hooksPath ./.git-hooks. The repo's own .npmrc sets ignore-scripts=true, which suppresses that prepare script, so a fresh clone installs no hook at all. Add a .pre-commit-config.yaml with the gitleaks hook pinned to an exact release tag (repo: https://github.com/gitleaks/gitleaks, rev: v8.30.1) so the scan does not depend on an npm script the repo itself disables - the same fallback sasjs/server carries for the same reason.
• client/package.json:44-52,58,103 - the bumped entries carry ^ ranges (^20.3.33, ^4.19.1) even though .npmrc sets save-exact=true. The lockfile pins the resolutions for CI, but the manifest drifts on every manual edit; pin the changed entries to exact versions. (The file-wide ^ style predates this PR, so this is the standing deviation surfacing again, not a new pattern.)

2 findings above for review.

Reviewed the full base...head diff (12 files: exportdb service and tests, system screen modal, new cypress spec, three CI workflows, client and sas lockfiles) plus repo hardening at this head. - .pre-commit-config.yaml (missing) - the repo's gitleaks scan runs only from .git-hooks/pre-commit, which the root `prepare` script wires via `git config core.hooksPath ./.git-hooks`. The repo's own .npmrc sets ignore-scripts=true, which suppresses that prepare script, so a fresh clone installs no hook at all. Add a .pre-commit-config.yaml with the gitleaks hook pinned to an exact release tag (repo: https://github.com/gitleaks/gitleaks, rev: v8.30.1) so the scan does not depend on an npm script the repo itself disables - the same fallback sasjs/server carries for the same reason. - client/package.json:44-52,58,103 - the bumped entries carry ^ ranges (^20.3.33, ^4.19.1) even though .npmrc sets save-exact=true. The lockfile pins the resolutions for CI, but the manifest drifts on every manual edit; pin the changed entries to exact versions. (The file-wide ^ style predates this PR, so this is the standing deviation surfacing again, not a new pattern.) 2 findings above for review.

hermes referenced this pull request 2026-10-03 12:40:43 +00:00

fix(deps): pin shelljs to clear the braces advisory in the sas audit #336

hermes referenced this pull request 2026-10-03 13:55:30 +00:00

feat(editor): raise the row limits and add a cell limit to the EDIT screen #335

allan added 2 commits 2026-10-03 17:26:06 +00:00

fix(deps): pin shelljs to clear the braces advisory in the sas audit ... 6e5093858b

braces (GHSA-vfj7-8cjw-p6xm) arrives via @sasjs/cli > shelljs > fast-glob
> micromatch > braces and has no fixed version - braces 3.0.3 is the
latest release, and npm's only remedy is downgrading @sasjs/cli to
4.13.1.  shelljs 0.9.0 is what introduced fast-glob, so pinning shelljs
to 0.8.5 removes braces, micromatch and fast-glob from the tree
entirely.  The CLI only uses shelljs for ls/cp/rm/exec, unchanged
between the two versions - sasjs lint and sasjs cb -t server both still
pass.

Merge pull request 'fix(deps): pin shelljs to clear the braces advisory in the sas audit' (#336) from fix/audit-braces into feat/export-dc-library-ddl ... b505e45c6f

Reviewed-on: #336

hermes reviewed 2026-10-03 17:48:53 +00:00

hermes left a comment

Author

Collaborator

Copy link

Copy source

Re-review at the new head b505e45; the delta since the prior review at e49e8e67 is the shelljs audit-pin merge (sas/package.json + lockfile), nothing new to raise there. Two findings carried over from the prior review, both still open at this head:

• .pre-commit-config.yaml (missing, repo root) - no gitleaks pre-commit hook in the repo. The scan in .git-hooks/pre-commit is wired by the "prepare" script, which the committed .npmrc (ignore-scripts=true) suppresses, so a fresh clone commits unscanned. Add a .pre-commit-config.yaml with the gitleaks hook pinned to an exact release tag (repo: https://github.com/gitleaks/gitleaks, rev: v8.30.1 or the current release), as sasjs/cli already carries.
• client/package.json:44-52,58,103 - the entries this PR changes carry ^ ranges ("@angular/animations": "^20.3.33", "@sasjs/adapter": "^4.19.1") despite the committed .npmrc setting save-exact=true. The lockfile pins the CI resolutions, but the manifest drifts on every manual edit; pin the changed entries to exact versions (the file-wide ^ style predates this PR, so this is the standing deviation surfacing again).

2 findings above for review.

Re-review at the new head b505e45; the delta since the prior review at e49e8e67 is the shelljs audit-pin merge (sas/package.json + lockfile), nothing new to raise there. Two findings carried over from the prior review, both still open at this head: - .pre-commit-config.yaml (missing, repo root) - no gitleaks pre-commit hook in the repo. The scan in .git-hooks/pre-commit is wired by the "prepare" script, which the committed .npmrc (ignore-scripts=true) suppresses, so a fresh clone commits unscanned. Add a .pre-commit-config.yaml with the gitleaks hook pinned to an exact release tag (repo: https://github.com/gitleaks/gitleaks, rev: v8.30.1 or the current release), as sasjs/cli already carries. - client/package.json:44-52,58,103 - the entries this PR changes carry ^ ranges ("@angular/animations": "^20.3.33", "@sasjs/adapter": "^4.19.1") despite the committed .npmrc setting save-exact=true. The lockfile pins the CI resolutions, but the manifest drifts on every manual edit; pin the changed entries to exact versions (the file-wide ^ style predates this PR, so this is the standing deviation surfacing again). 2 findings above for review.

allan added 6 commits 2026-10-03 18:49:35 +00:00

chore(deps): upgrade Handsontable to 18.1.1 ... b6b00ab96e

handsontable and @handsontable/angular-wrapper 18.0.0 -> 18.1.1, plus the
two repo-owned places that pin the version.

18.1 fixes two bugs this repo's specs had pinned as failure modes:

- "Sheet size limit exceeded" (upstream #10672): the formulas plugin used
  to push the grid's maxRows into the HyperFormula engine as its sheet
  size limit, so a licensing cap below an existing table's row count made
  the engine reject the whole table. initSetup works around it with
  Math.max(dataSource.length, editor_rows_allowed).
- Formula corruption on updateSettings while a sort is active: a bare {}
  settings object was enough to turn formula cells into #REF!.
  updateSettingsSortSafe clears and restores the sort around every call.

Both specs now assert the fixed behaviour instead, so a future regression
shows up. DC's own workarounds are untouched.

18.1.1 also fixes a filters plugin crash that lands on DC's usage exactly
(upstream #13480): editing a cell in a filtered column, or replacing the
data with a filter active, after updateSettings() with the filters option.
The Angular wrapper calls updateSettings on every update, and DC's viewer
and viewboxes run filters: true.

18.1.0 is mostly a performance release (viewport cell-meta release, index
translation, single-pass layout, bulk operations).

Also:
- licenseChecker.js: add the 18.1.1 handsontable packages to
  excludePackages, or the build stops at the license-checker step (the
  library's licence string is non-standard).
- _hot-icons.scss: regenerated by scripts/gen-hot-icons.mjs; drops the
  Pikaday rules because 18.1 removed the leftover Pikaday theme styles.
- dc-validator.spec.ts: 18.1 tightened ColumnSettings['validator'] to its
  real union (RemoveIndexSignature), so narrow it the way the production
  code already does.

feat(editor): raise the row limits and add a cell limit to the EDIT screen ... 3de6655f26

VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.

The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.

fix(editor): check the EDIT limits once, and skip switched-off options ... f08c4c231f

Review follow-ups on the row/cell limits:

- getdata.sas checked the limits twice - once before the sort that caps
  work.out, and again after PRE_EDIT_HOOK.  It is now a single check,
  after both, so the counts cover everything that could reach the
  browser: the rows the selection matched, plus anything a hook has
  added.  The payload cap moves to where work.outdata is built, which is
  the only place that needs it.  Each mp_abort also names its option once
  - the limit was repeated in the message text as well as the condition.

- The migration closed only var_active=1 rows but inserted active rows
  unconditionally, so an option a site had switched off was silently
  switched back on.  It now leaves those alone and logs a note.

- row-cell-limits.cy.ts test 1 matched a regex against the grid's text,
  which the fixture's own columns satisfy (DEPTH_M reaches 4293,
  SAMPLE_COUNT 4210), so it did not pin the raised VIEW cap.  It now
  reads the PK column - the first td of a .ht_master row - and requires
  values in the PK range 1001-2000 with a maximum above 1500, which the
  old 500 row cap could never render.

Merge pull request 'feat(editor): raise the row limits and add a cell limit to the EDIT screen' (#335) from feat/row-and-cell-limits into chore/handsontable-18.1.1 ... 831f0b27a9

Reviewed-on: #335

Merge branch 'feat/export-dc-library-ddl' into chore/handsontable-18.1.1 f2535ce335

Merge pull request 'chore(deps): upgrade Handsontable to 18.1.1' (#334) from chore/handsontable-18.1.1 into feat/export-dc-library-ddl ... 8169ef253d

Reviewed-on: #334

hermes changed title from feat(system): export the DC library DDL from the system screen to feat: release 7.17 - DDL export, Handsontable 18.1.1, EDIT row and cell limits 2026-10-03 18:58:32 +00:00

hermes reviewed 2026-10-03 20:00:59 +00:00

hermes left a comment

Author

Collaborator

Copy link

Copy source

Re-review at the new head 8169ef2 (the EDIT row/cell-limit landing and the Handsontable 18.1.1 merge since the prior review at b505e45); no new findings in that delta, two carried findings remain open at this head.

• .pre-commit-config.yaml (missing, repo root) - no gitleaks pre-commit hook in the repo. The scan in .git-hooks/pre-commit is wired by the "prepare" script, which the committed .npmrc (ignore-scripts=true) suppresses, so a fresh clone commits unscanned. Add a .pre-commit-config.yaml with the gitleaks hook pinned to an exact release tag (repo: https://github.com/gitleaks/gitleaks, rev: v8.30.1 or the current release), as sasjs/cli already carries.
• client/package.json:44-52,58,102-103 - the entries this PR changes carry ^ ranges ("@angular/animations": "^20.3.33", "@sasjs/adapter": "^4.19.1", "@angular/cli": "^20.3.32", "@angular/compiler-cli": "^20.3.33") despite the committed .npmrc setting save-exact=true. The lockfile pins the CI resolutions, but the manifest drifts on every manual edit; pin the changed entries to exact versions (the file-wide ^ style predates this PR, so this is the standing deviation surfacing again).

2 findings above for review.

Re-review at the new head 8169ef2 (the EDIT row/cell-limit landing and the Handsontable 18.1.1 merge since the prior review at b505e45); no new findings in that delta, two carried findings remain open at this head. - .pre-commit-config.yaml (missing, repo root) - no gitleaks pre-commit hook in the repo. The scan in .git-hooks/pre-commit is wired by the "prepare" script, which the committed .npmrc (ignore-scripts=true) suppresses, so a fresh clone commits unscanned. Add a .pre-commit-config.yaml with the gitleaks hook pinned to an exact release tag (repo: https://github.com/gitleaks/gitleaks, rev: v8.30.1 or the current release), as sasjs/cli already carries. - client/package.json:44-52,58,102-103 - the entries this PR changes carry ^ ranges ("@angular/animations": "^20.3.33", "@sasjs/adapter": "^4.19.1", "@angular/cli": "^20.3.32", "@angular/compiler-cli": "^20.3.33") despite the committed .npmrc setting save-exact=true. The lockfile pins the CI resolutions, but the manifest drifts on every manual edit; pin the changed entries to exact versions (the file-wide ^ style predates this PR, so this is the standing deviation surfacing again). 2 findings above for review.

All checks were successful

Hide all checks

Build / Build-and-ng-test (pull_request) Successful in 5m46s

Details

Lighthouse Checks / lighthouse (pull_request) Successful in 22m27s

Details

Build / Build-and-test-development (pull_request) Successful in 33m2s

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/export-dc-library-ddl:feat/export-dc-library-ddl

git checkout feat/export-dc-library-ddl

Sign in to join this conversation.

Reviewers

No Reviewers

hermes

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

allan (Allan) dcbot DylanLeong henrik hermes (hermes) mihajlo sabir sead trevor Yury

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dc/dc#333