Commit Graph
110 Commits
Author SHA1 Message Date
dc 1385741f2d fix(getcols): use %global and %mp_getcols, and merge the base branch
Build / Build-and-ng-test (pull_request) Successful in 5m30s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m21s
Build / Build-and-test-development (pull_request) Successful in 31m14s
Review feedback on this PR, plus the conflict with the base.

Conflict: row-cell-limits.cy.ts, where both branches added a case.  Both are
kept - the config-driven limits case is 4, the filter-picker-after-abort case
is 5 - with the helpers from each side.

- %local in open code is a macro error ("not valid in open code"), logged on
  every call, and the variables are never created - so a table with no
  mpe_tables row left the INTO variables unresolved.  It is %global now, the
  way getdata.sas declares the same three variables.
- The hand-rolled format-prefix test typed only DATETIME/DATE/TIME, so YYMMDD,
  MMDDYY, DDMMYY, MONYY, E8601DA, B8601DA, E8601DT and NLDATM columns reached
  the picker as numeric.  The proc contents step is replaced with
  %mp_getcols(&ds, outds=cols1) - the call getdata.sas already makes - so the
  fallback list is typed exactly as the normal path types it, with the
  temporal-column exclusion applied on top.
2026-10-04 12:00:41 +00:00
dc 05126ae73a fix(mocks): read MPE_CONFIG correctly, and pin the limits to it
Build / Build-and-ng-test (pull_request) Successful in 5m27s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m3s
Build / Build-and-test-development (pull_request) Successful in 31m9s
Review feedback on this PR.

- getdata.js read MPE_CONFIG from the requested table's own library, so the
  read failed for every non-control libref and both limits silently fell back
  to the shipped defaults.  It now reads mpeDataDir, as viewdata.js already
  did.  Reproduced first: with DC_MAXOBS_WEBEDIT=5, DC_JSLIB.MPE_X_TEST
  aborted while TESTDATA.DEMO_01 loaded untouched.
- exportdb.js mapped DATETIME to TIMESTAMP for TSQL as well, and T-SQL's
  TIMESTAMP is a deprecated synonym for rowversion, so datetime columns were
  declared as rowversions.  TSQL now gets DATETIME2; PGSQL still gets
  TIMESTAMP.  DOUBLE PRECISION is left alone - it is a T-SQL synonym for
  float, not an invalid type.
- loadDcConfig read the config row as direct properties, but a row that has
  been through the app's own edit-and-approve flow comes back from the adapter
  with upper-case keys, so the approved option was skipped and fell back to
  its default - the same silent-default failure by another route.  It now
  reads every field through colVal, like the rest of the mocks.
- row-cell-limits.cy.ts gains a case that moves DC_MAXCELLS_WEBEDIT to a
  non-default value through the editor, asserts that a non-control-libref
  table is then refused, and puts the option back.  It fails if the read is
  dropped: verified by reinstating the getdata.js bug and watching it fail.
2026-10-04 11:18:49 +00:00
dc 3a6a216b9d fix(filters): source the picker variables from public/getcols when the table is not returned
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m29s
Build / Build-and-test-development (pull_request) Successful in 30m40s
A limit aborting the table request left the filter picker with no variables,
so the "please filter and try again" advice could not be acted on: the
picker's list came from the columns that arrived with the table, and there
was no table.

The list now falls back to public/getcols, which reads the column metadata
without the rows. The normal path is unchanged - the service is only called
when the caller has no columns to hand - so nothing extra is requested for a
table that loaded.

- SasStoreService.getCols() calls public/getcols and returns cols.
- SasStoreService.setQueryVariablesFromTable() takes the columns the caller
  already has and reaches for the service only when they are empty.
- editor openQb() and viewboxes openFilter() use it (the viewer's filter is
  reachable after a VIEW abort, the editor's after an EDIT abort).
- public/getcols.sas keeps ddtype (already computed, dropped by the keep) and
  excludes the temporal columns exactly as the editor does, so the fallback
  list matches the table's own.
- New mock public/getcols.js mirrors the service.

Verified: the mock's list matches editors/getdata for every fixture except the
aborting one (MPE_X_SEARCH: getdata 0 cols, getcols 9), and a new Cypress case
in row-cell-limits.cy.ts aborts a too-big EDIT, opens the filter and asserts
all nine columns are offered. 89 Cypress tests green across the six affected
specs.
2026-10-04 09:30:33 +00:00
dc edac9a987a feat(mocks): read the DC_MAX* options from MPE_CONFIG, and mock the DDL export
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m5s
Build / Build-and-test-development (pull_request) Successful in 30m22s
The mocked estate could not exercise what 7.17 changed.

- getdata.js and viewdata.js hard-coded their limits, so editing
  DC_MAXOBS_WEBEDIT / DC_MAXCELLS_WEBEDIT / DC_MAXOBS_WEBVIEW in DC made
  no difference.  They now read them from MPE_CONFIG via a new
  loadDcConfig/dcConfigNumber pair in dcMockUtils, which mirrors the loop
  in the generated settings.sas: DC-scope rows only, var_active=1 only,
  current rows only (now < tx_to), latest tx_from wins.  An option that is
  absent or switched off falls back to the service's own default, which is
  what %symexist does in the real service.

- There was no mock for services/admin/exportdb, so the System screen's
  Export DC Library DDL action hit a service that did not exist.  The new
  mock renders every table in the DC libref as basic DDL - SAS, PGSQL and
  TSQL flavours, schema defaulting to the libref, a unique index from the
  MPE_TABLES buskey, and the download headers mp_streamfile produces.  DDL
  only: the real service appends inserts for SAS and PGSQL, but
  serialising the library's rows in a mock buys no test value.

ddl-export.cy.ts gains two cases that fetch the URL the action opens and
assert the DDL, so the spec covers the service and not just the URL.
2026-10-03 22:34:57 +00:00
dc f08c4c231f fix(editor): check the EDIT limits once, and skip switched-off options
Build / Build-and-ng-test (pull_request) Successful in 5m35s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m41s
Build / Build-and-test-development (pull_request) Successful in 30m42s
Review follow-ups on the row/cell limits:

- getdata.sas checked the limits twice - once before the sort that caps
  work.out, and again after PRE_EDIT_HOOK.  It is now a single check,
  after both, so the counts cover everything that could reach the
  browser: the rows the selection matched, plus anything a hook has
  added.  The payload cap moves to where work.outdata is built, which is
  the only place that needs it.  Each mp_abort also names its option once
  - the limit was repeated in the message text as well as the condition.

- The migration closed only var_active=1 rows but inserted active rows
  unconditionally, so an option a site had switched off was silently
  switched back on.  It now leaves those alone and logs a note.

- row-cell-limits.cy.ts test 1 matched a regex against the grid's text,
  which the fixture's own columns satisfy (DEPTH_M reaches 4293,
  SAMPLE_COUNT 4210), so it did not pin the raised VIEW cap.  It now
  reads the PK column - the first td of a .ht_master row - and requires
  values in the PK range 1001-2000 with a maximum above 1500, which the
  old 500 row cap could never render.
2026-10-03 18:00:46 +00:00
dc 3de6655f26 feat(editor): raise the row limits and add a cell limit to the EDIT screen
Build / Build-and-ng-test (pull_request) Successful in 5m22s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m58s
Build / Build-and-test-development (pull_request) Successful in 30m41s
VIEW default 500 -> 2000. EDIT default 100 -> 250, plus a new
DC_MAXCELLS_WEBEDIT (200000) so a wide selection is refused on cells as
well as rows, whichever is reached first. Measured end-to-end: 2000 rows
of a 9 column table is ~1s in VIEW / ~6s in EDIT, but 100 rows of a 1001
column table is already ~17s in EDIT - a row cap cannot bound that.

The EDIT guard also moves: it now checks the filtered row count before
the sort that caps work.out, and again after PRE_EDIT_HOOK, which can
replace work.out with a larger table.
2026-10-03 13:15:58 +00:00
dc 05f3f49e0c feat(system): export the DC library DDL from the system screen
Build / Build-and-ng-test (pull_request) Failing after 1m47s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Failing after 4m46s
Adds a fourth admin action to the System information screen - Export DC
Library DDL - with a secondary flavour chooser (SAS, PGSQL, TSQL) and an
optional schema box for the DB flavours. The selection is passed to the
existing services/admin/exportdb service in a new window as URL params.

Also bumps @sasjs/adapter to ^4.19.1, adds a Cypress spec for the three
flavour states, and adds a TSQL case to the exportdb SAS test.

Closes #179
2026-10-02 14:03:53 +00:00
dc 60ddb475d7 ci: run the hook-programs spec in the cypress job
Build / Build-and-ng-test (pull_request) Successful in 5m26s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m5s
Build / Build-and-test-development (pull_request) Successful in 29m17s
hook-programs.cy.ts was in no CI spec list, so it never ran - the only
end-to-end coverage of the new mock hook execution (mockHookSource plus
the PRE/POST_EDIT_HOOK blocks) and of the loader's target re-registration
check was dead weight. Add it to the development job's --spec list.

It goes second, not last: csv-limited.cy.ts asserts the free tier and so
must keep the first slot (before any spec applies a licence key), and test
2 reads the approval queue, which is paged oldest first - a changeset
submitted by an earlier spec pushes its own row off the first page. The
spec header records the constraint. Full list verified locally: 15 specs,
132 tests, all passing.
2026-09-27 23:28:42 +00:00
dc e926649787 test(hooks): anchor the approval assertion on the grid, not a row position
The post-edit-hook spec asserted on the last row of the approval queue.
The queue also holds changesets from earlier specs in the same run, so the
position is not stable - it failed on an estate with rows left over from a
previous run while passing on a fresh one. Search every row instead.
2026-09-27 22:44:54 +00:00
dc 5538e0c575 fix(diagnostics): give the startup service a body for browser_info
Build / Build-and-ng-test (pull_request) Successful in 6m6s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m55s
Build / Build-and-test-development (pull_request) Successful in 26m43s
Both startupservice call sites pass null, and the attach guard skipped a
null payload, so the service whose log a support ticket is read from
never received browser_info / browser_url_vars.

The e2e test named for the startup service only asserted on getdata, so
the gap was invisible. The name is corrected to what it asserts, its
sibling sweep now excludes the diagnostics services (it must, once the
startup service really does carry them), and the guard itself is covered
by a unit spec - red before the fix, green after.
2026-09-25 15:57:53 +00:00
allan 11b6c23000 Merge pull request 'feat(filters): make the applied-filter panel expandable' (#331) from feat/expandable-filter-panel into feat/browser-info
Build / Build-and-ng-test (pull_request) Successful in 6m8s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m56s
Build / Build-and-test-development (pull_request) Successful in 27m11s
Reviewed-on: #331
2026-09-25 15:16:00 +00:00
dc ba21ff00bc test(filters): wait for the editor before shooting its panel
Build / Build-and-ng-test (pull_request) Successful in 6m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m51s
Build / Build-and-test-development (pull_request) Successful in 28m0s
The editor screenshot was taken during the transition from the viewer, so it
captured the viewer's panel - byte-identical to the viewer shot. Wait for the
editor-only chrome (.editor-title, .btnCtrl) before asserting or shooting.
2026-09-25 14:58:14 +00:00
dc 65a96922f6 style(filters): set the filter clause in a monospace face, and shoot the editor panel
Build / Build-and-ng-test (pull_request) Successful in 6m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m11s
Build / Build-and-test-development (pull_request) Successful in 27m43s
The panel shows a query, so set it in the monospace face the app already uses
for code-like content (SAS logs, cell text) rather than the UI font.

The spec now also captures the editor's panel - collapsed and expanded - since
the editor is where the clause was previously clamped with an ellipsis.
2026-09-25 14:49:31 +00:00
dc fdaa249f44 feat(filters): make the applied-filter panel expandable
Build / Build-and-ng-test (pull_request) Successful in 5m45s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m28s
Build / Build-and-test-development (pull_request) Successful in 26m36s
A filter clause is shown in a panel above the grid. The panel was clamped to a
single line - in the editor with an ellipsis, and in the viewer by letting the
clause run the full width of the table - so a long or complex filter could not
be read on screen.

The panel is now collapsed to one line and carries a chevron that expands it to
show the whole clause, wrapped, and collapses it again. The chevron is rendered
only when the clause does not fit the collapsed line, so a short filter looks
exactly as it did before. Whether it fits is measured from the DOM rather than
guessed from the length of the text, since it depends on the rendered width; the
measurement is deferred out of the change-detection cycle, and re-run when the
clause, the available width, or the state changes.

This replaces the editor's hover-only reveal with a real control, which is a
button carrying aria-expanded and an accessible label, and gives the viewer the
same affordance. Both panels share the styling, which was previously duplicated
between the two component blocks.

Tested with `client/cypress/e2e/filter-panel.cy.ts`, at a laptop-sized viewport,
covering the three states:

1. no filter - the panel is not rendered
2. a short filter - shown in full, with no chevron
3. a long filter (an IN over every value of a free-text column, 646 characters)
   - collapsed with a chevron, expands to the whole clause over several lines,
   and collapses again
4. the editor panel behaves the same way for the same clause
2026-09-25 14:27:02 +00:00
dc 293636a5f8 refactor(client): scope browser_info to startup and hook services, add browser_url_vars
Build / Build-and-ng-test (pull_request) Successful in 5m24s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m15s
Build / Build-and-test-development (pull_request) Successful in 25m11s
The diagnostics tables now go only where they can be read: the startup
service, and the services that %include customer-provided code - the
hook scripts (getdata, stagedata, loadfile, restore, postdata) and the
dynamic cell dropdown programs (getdynamiccolvals). Every other service
is spared the payload, so the high-frequency calls stay lean.

The page's URL parameters now also travel as a browser_url_vars table,
one row per parameter (name, value), taken from both the search string
and the hash query string - easier for a SAS developer than parsing the
browser_info url column. The table is sent only when the URL has at
least one parameter, and only to the same services as browser_info.

mpeinit's debug dump covers browser_url_vars alongside browser_info.

The spec asserts both directions on the adapter interface: the viewer's
data services carry no browser_info, getdata does, and a ?labels=true
visit arrives as a browser_url_vars row. Each test boots from the app
root with a guarded evaluation-agreement acceptance, because Cypress
clears cookies between tests and the SASjs Server session drops (the
same flake hook-programs.cy.ts hits on this estate).
2026-09-25 00:26:16 +00:00
allan 9d740f9f47 Merge pull request 'fix(loader): abort when a post edit hook routes to an unregistered table' (#325) from fix/validate-post-edit-hook-target into feat/browser-info
Build / Build-and-ng-test (pull_request) Successful in 5m57s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m20s
Build / Build-and-test-development (pull_request) Successful in 25m54s
Reviewed-on: #325
2026-09-24 21:03:52 +00:00
dc a770f3695c feat(client): report the browser and both versions in browser_info
Build / Build-and-ng-test (pull_request) Successful in 5m15s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m12s
Build / Build-and-test-development (pull_request) Successful in 25m12s
Adds dc_version, adapter_version, browser, browser_version, platform and the
raw user_agent to the browser_info row, so a support ticket can be read off
the log and a hook can branch on the browser without parsing the user agent
itself. The summary comes from a new parse-user-agent util, following the
existing shared/utils convention.

Also adds a Cypress spec asserting the table reaches the adapter interface,
which is the only place the payload is observable end to end.
2026-09-24 19:31:13 +00:00
dc b5f228351b feat(mocks): run pre/post edit hook programs in the JS mock services
Build / Build-and-ng-test (pull_request) Successful in 5m41s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m52s
Build / Build-and-test-development (pull_request) Successful in 25m48s
The JS mock services carried MPE_TABLES.pre_edit_hook / post_edit_hook values
but never acted on them, so a table configured with hooks behaved differently
in the mock than in production.

Add mockHookSource() to dcMockUtils (resolving a hook program name to the
source of its .js counterpart on the Drive) and run the hook where the SAS
backend does:

- editors/getdata runs the pre-edit hook after filtering, letting it reassign
  visibleRows / visibleColumns (the work.OUT contract) - so an empty mirror can
  display the live rows of the table it points at
- editors/stagedata and editors/loadfile run the post-edit hook before the
  MPE_SUBMIT row is written, letting it reassign libref / dsn (the call
  symputx contract) - so a changeset submitted against a mirror is raised
  against the real table

Seed a demo pair in makedata (TESTDATA.DEMO_ORDERS plus an empty
TESTDATA.DEMO_MIRROR with both hooks) and cover it with a Cypress spec.
2026-09-24 15:17:07 +00:00
dc 283bf067c5 test(mocks): a missing range on the demo table, and the ordering beat in the clip
Build / Build-and-ng-test (pull_request) Successful in 6m8s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m33s
Build / Build-and-test-development (pull_request) Successful in 26m23s
DEMO_01 gains a GRADE column carrying MINVAL .A with MAXVAL .C, so the demo table
can show a range written in special missings: .B is inside it and .D is outside.
The clip's range-rule scene is rebuilt around the order - the same special missing
fails AMOUNT (MINVAL 1, below every number), passes SCORE (MAXVAL 100, below the
ceiling), and in GRADE .B is taken while .D is refused.

The captions from the previous take claimed a range rule steps aside for a
missing value, which the engine no longer does, so they are corrected along with
the scene.
2026-09-23 21:27:42 +00:00
dc 101087613e test(mocks): demo the corrected range-rule behaviour, and correct the captions
Build / Build-and-ng-test (pull_request) Successful in 5m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m12s
Build / Build-and-test-development (pull_request) Successful in 27m3s
DEMO_01 no longer carries a NOTNULL rule for ID: it is the table's buskey, and
Data Controller applies NOT NULL to a primary key by itself, so the demo now
shows the behaviour rather than a rule that duplicates it.

The clip's range-rule scene showed a special missing being rejected by MINVAL.
It now shows the missing accepted in both AMOUNT (MINVAL 1) and SCORE (MAXVAL
100), and a real number out of range - 0, then 200 - rejected.

Two captions were also wrong and are corrected: the period in a special missing
is optional (the demo's own row 2 holds .a), and ID is NOT NULL because it is the
primary key.
2026-09-23 21:00:45 +00:00
dc a36f0d5184 test(mocks): caption the clip from a track the spec writes as it runs
Build / Build-and-ng-test (pull_request) Successful in 5m11s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m20s
Build / Build-and-test-development (pull_request) Successful in 25m10s
Every beat now says what it shows and why the rule behaves as it does, so the
colour of a cell never has to be inferred. The spec appends a caption track as
it runs - a label, the text, and a wall-clock stamp per beat - and the encoder
turns consecutive entries into subtitle cues, calibrated against the one thing
measurable in the video (the amber SOFTREGEX warning cell appearing).

The timestamp is taken inside a cy.then(): Cypress evaluates a command's
arguments when the command is queued, so Date.now() passed to cy.writeFile
directly gives every mark the same value - the moment the spec body ran.

Two beats also gained room so their caption can be read: the opening grid hold,
and the DIFF hold before the staged screen. A mark with empty text clears the
caption while the clip moves between screens.
2026-09-23 16:55:36 +00:00
dc b6b0d3d343 test(mocks): record the clip as one editing session, with the staged screen
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m10s
Build / Build-and-test-development (pull_request) Successful in 25m10s
The clip reloaded the app halfway through, because the second half began with a
cy.visit - which reads as the table refreshing for no reason. It is now a single
session: the table is opened once, every rule scene is played on row 1 and each
value is put back, the real change is submitted, and the review screens are
reached through the app's own navigation. Nothing reloads.

Because the rule scenes are put back first, the DIFF carries only the two cells
the clip is about, and the spec asserts that: RATING and LAST_REVIEWED carry the
changed-cell marker and the other cells do not.

The staged-data screen gets a beat of its own (view staged data, held long
enough to read), since that is the row the approval acts on - and its assertion
checks text that only exists on that screen, because asserting 'Staged Data'
would have been satisfied by the button that was just clicked.

Also shortens the correction beats, which do not need a study pause.
2026-09-23 16:21:56 +00:00
dc 5cea685405 test(mocks): hover the clip's DIFF cells with the real mouse, and tighten the beats
Build / Build-and-ng-test (pull_request) Successful in 5m25s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m6s
Build / Build-and-test-development (pull_request) Successful in 25m8s
The clip hovered a changed DIFF cell with trigger('mouseover') and then asserted
contain.text on the tooltip. Clarity reveals a tooltip through CSS :hover, which
a synthetic event does not activate, so the tooltip stayed visibility: hidden
while the assertion passed anyway - the text is in the DOM either way. The
recording therefore showed no previous value at all.

hoverDiffCell now moves the real mouse (cypress-real-events, already a
dependency) and asserts the tooltip's computed visibility and opacity alongside
its text, so a take that does not actually display it fails. The first real
mouse move after another action can be swallowed, hence the repeat.

The holds are cut again as well - the recording is 82s where it was 101s - since
the finished clip is played at 1.43x on the way out.
2026-09-23 15:41:00 +00:00
dc b8f16a6382 fix(mocks): list a column's own special missing in its dropdown
Build / Build-and-ng-test (pull_request) Successful in 5m23s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m14s
Build / Build-and-test-development (pull_request) Successful in 25m16s
A SOFTSELECT/HARDSELECT whose rule value is a library.member.column reference
takes its list from that column, and getdata.sas builds it with cats() and then
orders it by the column itself. The mock sent the raw stored value instead, so a
numeric special missing reached the client in its period form (".a") rather than
as the bare letter cats() produces, and it sat in row order rather than below
every number. It now does both, so a dropdown sourced from a column reads the
way the real one does.

DEMO_01 gains a STATUS column carrying a SOFTSELECT over its own column, with
one row holding a special missing, so the dropdown can be demonstrated listing
that missing alongside the ordinary values.

The clip spec records that beat, hovers both changed cells on the review screen
so the value each replaced is on screen, and cuts the waits that were only
letting a page or a modal settle.
2026-09-23 15:11:27 +00:00
dc 12847cf071 fix(mocks): make the approval path work, and record the approval scene
Build / Build-and-ng-test (pull_request) Successful in 5m28s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m4s
Build / Build-and-test-development (pull_request) Successful in 25m22s
The mock's APPROVE_TABLE branch called loadTableData(), which is not defined in
that service's scope - the loader it defines is mpeLoadTableData(). Every ACCEPT
therefore failed with "No webout was returned by job", and it went unnoticed
because the demo clip stopped at the DIFF. With the loader corrected, ACCEPT
applies the load to the base table, writes the MPE_REVIEW / MPE_SUBMIT /
MPE_DATALOADS entries and lands on the history page.

DEMO_01 row 2 now carries a special missing (RATING .a) so a demo can show one
special missing changing into another - the DIFF compares the two rather than
treating either as a blank.

The clip spec gains the two beats the companion post describes: the approver
opens the submission from the approvals list, switches the DIFF between
formatted and unformatted (15JAN2026 / 24121 on the date9. column), accepts,
and the history shows the change APPROVED. Both DIFF beats scroll the table
right, because the changed columns sit off the edge of a 1280-wide frame. The
key entered in the NOTNULL scene is now a value that is not already a key, since
the editor checks the keys before it checks the rules and would report the
duplicate instead of the invalid value.
2026-09-23 13:50:15 +00:00
dc 6d85bce2a0 fix(validator): reject a special missing on a NOT NULL column
Build / Build-and-ng-test (pull_request) Successful in 5m50s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m41s
Build / Build-and-test-development (pull_request) Successful in 25m18s
A special missing is NULL as far as a SAS NOT NULL (or primary key)
constraint is concerned - an insert carrying one is rejected with an
integrity constraint error (_NM0001_ / _PK0001_) - but the frontend
NOTNULL rule passed it, so the editor accepted a value the target table
refuses.  getdata merges a physical NOT NULL constraint into a NOTNULL
rule, so the mismatch was reachable on any table with the constraint.

The rule now fails a special missing on a numeric column.  A single
letter stays valid on a character column, where there is no special
missing concept.

Also updates the DEMO_01 clip script: the NOTNULL scene now shows a
blank AND a special missing both rejected, with a number accepted.
2026-09-23 07:45:18 +00:00
dc 97cbf283bc test(mocks): make the clip beats assert the settled cell state
The recording spec paused for a fixed couple of seconds after each edit and
trusted that the rule engine had finished. It has not: the engine flags a cell
(htInvalid) shortly after the edit commits, so a blind pause can land before
the red appears. In the first take the blanked NOTNULL key never showed as
invalid on screen, which is the whole point of that scene.

Every beat now asserts the expected state - flagged for the rejections
(AB, 1a, a blank key, a special missing below MINVAL), unflagged for the
accepted ones - and only then holds. The hold therefore always rests on the
settled outcome, and the clip is self-checking: if the engine stops flagging
one of these states the recording fails instead of quietly showing the wrong
thing.

Verified against the app: blanking ID gives the cell htInvalid; AB and 1a in
RATING are flagged; a single letter in RATING, a special missing in ID and a
special missing under MAXVAL all settle clean.
2026-09-22 21:43:03 +00:00
dc d124ce3b36 fix(mocks): keep special missings in the DIFF, and add the clip recording spec
The mock's DIFF service normalised numeric values with Number(), so a SAS
special missing (".b") became NaN and was emitted as null - the value vanished
from the DIFF screen even though the staged data held it.

Real Data Controller writes the DIFF with missing=STRING, whose format maps ._
and .a-.z to a string and leaves a bare "." as null (see the bart format in
mp_jsonout.sas). The mock now mirrors that, so a special missing survives into
the DIFF. Special missings are numeric-only, so the guard sits in the numeric
branch of normVal.

Also adds cypress/clips/special-missings-clip.cy.ts - the recording script for
the companion demo clip. It lives outside cypress/e2e so the default spec
pattern does not pick it up in CI.
2026-09-22 21:13:08 +00:00
dc aae48dc401 test(e2e): scroll the grid so the NOTES column is on screen for the siphonophore beat
Build / Build-and-ng-test (pull_request) Failing after 1m36s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m18s
The three-row siphonophore result is narrower than the grid viewport, so
Handsontable sizes the columns wider than it (scrollWidth 1562 vs
clientWidth 1340): a horizontal scrollbar appears and NOTES - the column the
match is actually in - is clipped. The cell text is in the DOM either way, so
the beat passed without it, but the demo read as three arbitrary rows.

scrollGridTo() sets scrollLeft on #hotTable .wtHolder, asserts the value
landed, and the beat asserts the full NOTES text and scrolls back for the
following beats. It is a plain spec step - CI runs it, no recording flag and
no cy.wait.

Also lets a recording size the browser window: the capture is the window's
content area, so RECORD_WINDOW_SIZE is passed through as --window-size for a
real Chromium browser (Electron ignores it). That is what makes a 16:9 pane
possible: (W-450)/(H-96) = 16/9 at 1920x923 of content, i.e. 1920x1010 of
window, for a 1470x827 pane and the repo's 1600x900 viewport at ~0.90 zoom.

Inert unless the env var is set, so CI is unaffected. dc-cypress updated with
the window-sizing recipe, the clipped-column scroll and the frame
mean/variance scan used to find the cut point.
2026-09-16 18:44:25 +00:00
dc 287e4fa3c2 test(e2e): extend the full table search walkthrough and size the demo capture
Build / Build-and-ng-test (pull_request) Failing after 1m39s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m50s
Seven more search beats (Halcyon across two columns, multi-word site and
species, partial expedition code, exact numeric matches on DEPTH_M and the
last primary key), all with counts verified against the mock service first.
The walkthrough now takes 38s in CI against 28s.

A single-row result renders "(1 row, 9 cols)", not "(1 rows, ...)", so
assertRowCount picks its unit from the count it is given.

demoPause adds a recording-only 1.5s linger at each row count (0 unless the
env var is set, so CI still runs the file with no waits), and the dc-cypress
skill now records how the demo capture actually works: a fixed 1280x720
window containing the runner, with the app zoomed to fit the AUT pane -
matching the viewport aspect to that pane (~1.33) fills it, and the crop
rect plus the cut points are measured from the frames.
2026-09-16 15:18:21 +00:00
dc fea3a1c04e test(e2e): run the full table search walkthrough as a single test
Build / Build-and-ng-test (pull_request) Failing after 1m36s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m37s
Opening the table once and making every variation a search in place - rather
than one it() per variation, each paying for the beforeEach navigation again -
matches how a user works, makes the recorded walkthrough one continuous take,
and cuts the spec from 67s to 26s.

The no-match steps (wrong case, partial number, value not present) are now
wrapped in a negativeStep() guard driven by --env skipNegative=true, so the
recording still never shows an empty result while CI (no env) runs every step.
The walkthrough ends inside the filter: clearing the search leaves the filter
applied, which is asserted rather than assumed.
2026-09-16 01:17:37 +00:00
dc 74e5d4eae0 test(e2e): make the no-match search cases skippable for demo recordings
Build / Build-and-ng-test (pull_request) Failing after 1m38s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m43s
Three of the cases legitimately match nothing - they pin the case sensitivity
of the character search, the exact-match semantics of the numeric search, and
the "no data found" handling.  They are worth keeping as regression coverage
but they make a poor demo video.

They now use an `itNegative` alias that resolves to `it.skip` when the spec is
run with `--env skipNegative=true`, so the walkthrough recording only ever
shows searches that return rows:

    npx cypress run --browser electron \
      --spec cypress/e2e/full-table-search.cy.ts --env skipNegative=true

CI runs the file with no env, so all nine cases still run there.  The positive
half of the old case-sensitivity case is split out as its own test (3.1) so the
recording keeps it, and the "no data found" assertion is extracted into
assertNoData().
2026-09-16 01:00:00 +00:00
dc 48599c207d test(e2e): add full table search spec with mock search and filter support
Build / Build-and-ng-test (pull_request) Failing after 1m42s
Build / Build-and-test-development (pull_request) Skipped
Lighthouse Checks / lighthouse (pull_request) Successful in 20m16s
The viewer's search box searches every column of a table - character columns
by case-sensitive CONTAINS and numeric columns by exact equality - but against
the JS mock backend the box did nothing, because the viewdata mock ignored
SEARCHTYPE/SEARCHVAL (and FILTER_RK) entirely.

Mock changes (sas/mocks/sasjs/services/public/viewdata.js):
- apply the stored filter first via mpeFilterMaster, mirroring
  %mpe_filtermaster(VIEW,...) in viewdata.sas, and return its WHERE text as
  sasparams.FILTER_TEXT (previously hard-coded blank, so the viewer's info bar
  never showed a clause and the filter was never applied to the rows)
- mirror %mp_searchdata: CHAR = case-sensitive CONTAINS across every character
  column, NUM = exact equality across every numeric column, only CHAR/NUM
  trigger a search, search values stripped of % & ; " like the SAS service
- cap the rows the way the real service does (MAXROWS 500; NOBS is the uncapped
  filtered count for a normal view, the capped match count for a search)
- a search with no matches now returns no rows, so the client shows its
  "No data found with given conditions" panel; the single-empty-row fallback
  stays for the normal-view-empty-table case

Mock data (sas/mocks/sasjs/services/admin/makedata.js):
- new MPE_X_SEARCH demo table: 1000 deterministic, obviously fictional
  deep-sea survey rows, registered in MPE_TABLES. Seeded so a partial search
  hits a value buried in the middle of a long text column, a value spanning two
  columns, a repeated exact numeric, and rows that deliberately do not match.

New spec (client/cypress/e2e/full-table-search.cy.ts):
- seven tests covering open/full view, partial character search, case
  sensitivity, numeric exact match, no match, search within a filter, and
  clearing the search. No artificial waits - state assertions only.
- added to the Cypress spec list in .gitea/workflows/build.yaml

Also adds the dc-cypress skill (.agents/skills/dc-cypress) covering the mock
backend setup, the spec conventions, the selectors that actually work in the
viewer filter modal, and the fixed 1280x720 video capture.
2026-09-16 00:48:12 +00:00
YuryShkoda 8fafeb7923 fix(editor): restore multi-column sort with one call, not a loop
Build / Build-and-ng-test (pull_request) Successful in 5m23s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m1s
Build / Build-and-test-development (pull_request) Successful in 25m4s
multiColumnSorting's sort() replaces the entire sort state on every call rather than accumulating, so the restore loop after updateSettings()/editTable()/cancelEdit() silently dropped every sorted column but the last. Pass the captured configs as a single array instead. Adds a Karma integration test proving the loop
loses columns, and a Cypress test verifying cancelEdit() keeps a real shift-click multi-column sort intact end-to-end.
2026-09-03 12:46:31 +03:00
YuryShkoda e139c37fbf fix(editor): fix formula $-substitution corruption and sorted-grid row/formula desync
Build / Build-and-ng-test (pull_request) Successful in 5m23s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m6s
Build / Build-and-test-development (pull_request) Successful in 25m7s
String.replace interpreted $&/$`/$'/$<digit> specially in DC.ORIG_VALUE/DC.USER_NAME substitutions, corrupting formulas whose cell data happened to contain a literal $. Separately, afterChange indexed dataSource with Handsontable's visual row instead of translating to physical (as beforeChange already does), desyncing EDIT_STATUS/overwritten-comment writes on a sorted grid. Investigating that surfaced a third, unrelated Handsontable bug: updateSettings() corrupts formula cell references whenever a sort is active, regardless of what it changes - fixed by clearing the sort around every updateSettings() call.
2026-09-02 17:19:58 +03:00
hermes 490d0137b9 fix(stage): submit data via editor flow, use dynamic date assertion
Build / Build-and-ng-test (pull_request) Successful in 5m14s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m30s
Build / Build-and-test-development (pull_request) Successful in 24m11s
Rewrite stage.cy.ts to follow the same pattern as excel.cy.ts:
open MPE_X_TEST, attach regular_excel.xlsx, submit to create staged
data, then test the formatted/unformatted toggle on the stage page.

The date assertion is now dynamic: reads the raw SOME_DATE value
from the unformatted view and computes the expected SAS date9.
formatted value, rather than hardcoding 12FEB1960.

Remove the fixture stageSubmission code from makedata.js — staged
data is created at runtime by stagedata.js, not by makedata.
2026-08-30 22:29:40 +01:00
hermes c180934167 fix(mocks): SAS datetime format in getchangeinfo, login in stage.cy.ts
getchangeinfo.js: use SAS datetime19. format (DDMMMYYYY:HH:MM:SS)
for SUBMITTED_ON_DTTM and REVIEWED_ON_DTTM, matching the real SAS
output. Use SUBMIT_STATUS_CD field name (not REVIEW_STATUS_ID)
to match the MPE_SUBMIT column name and getchangeinfo.sas output.

stage.cy.ts: add login step in beforeEach (the beforeAll logs out
after uploading the license, so each test needs to re-login).
Use loginAndUpdateValidKey(true) to force license upload.
2026-08-30 20:10:52 +01:00
hermes 13c10be30f fix(cypress): correct mock data, test expectations, and abort modal
Build / Build-and-ng-test (pull_request) Successful in 5m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m39s
Build / Build-and-test-development (pull_request) Failing after 34m1s
- editor.cy.ts: assert DC.USER_NAME is non-empty instead of hardcoded
  'sasdemo' (CI runs as a different OS user)
- editor.cy.ts: use =SOME_BESTNUM * 0 + 1936 for a deterministic formula
  result regardless of the column's actual value
- excel.cy.ts: expect computed value '2' instead of formula text '=1+1'
  (no XL_RULE=FORMULA configured on that column)
- excel-multi-load.cy.ts: use DC996664 libref to match the actual fixture
  sheet names
- makedata.js: add labels to DC_JSLIB.MPE_X_TEST schema (SOME_CHAR,
  SOME_DATE, SOME_DATETIME, SOME_TIME) for the viewer-labels tests
- makedata.js: register DC996664 libref with MPE_X_TEST/MPE_TABLES/
  MPE_VALIDATIONS/MPE_ALERTS and copy table files for multi-load fixtures
- makedata.js: change SOME_SHORTNUM HARDREGEX from [1-5] to [0-2]
- loadfile.js: exclude VAR_TXFROM/VAR_TXTO/VAR_PROCESSED columns from the
  missing-cols check for TXTEMPORAL tables (computed at load time)
- InfoModal.ts: add _PROGRAM to AbortDetails
- info-modal.component.html: display _PROGRAM in the abort modal
- 4 call sites: pass abortRes._PROGRAM when building AbortDetails
2026-08-29 23:53:13 +01:00
hermes 8e51fde524 fix(licensing): bridge async proceed() through cy.then and replace Response.arrayBuffer with manual stream reader
Build / Build-and-ng-test (pull_request) Successful in 5m9s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m31s
Build / Build-and-test-development (pull_request) Failing after 33m53s
- licensing.cy.ts: call proceed() unconditionally via cy.then() instead of conditionally chaining .then() after cy.wait(), so the command queue stays open across the async gap and enqueued commands always run
- combinedLicenceKey.ts: replace new Response(stream).arrayBuffer() with an explicit streamToArrayBuffer helper for broader browser support
- remove debug-license.cy.ts, an untracked throwaway repro for the promise-handling question
2026-08-29 20:13:22 +01:00
YuryShkoda 4db8d62dca fix(cypress): make licensing combined-key tests order-independent
Build / Build-and-ng-test (pull_request) Successful in 5m16s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m48s
Build / Build-and-test-development (pull_request) Failing after 33m43s
- Tests 7-10 only acted when already on the licensing page, silently
  no-oping otherwise; now they navigate to licensing/update explicitly
  when needed, chained via .then() to avoid Cypress conflating a raw
  async function's promise with the preceding cy.visit()/cy.wait()
- Replace Response(readable).arrayBuffer() with a direct
  ReadableStream reader loop in generateCombinedKey - the Response-based
  read never settled under Cypress, likely due to its fetch/Response
  patching for network interception
2026-08-29 14:06:11 +01:00
hermes 5d03164b6a fix: tests and realistic getcolvals.js
Build / Build-and-ng-test (pull_request) Successful in 5m49s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m56s
Build / Build-and-test-development (pull_request) Failing after 33m48s
2026-08-29 13:14:18 +01:00
hermes c0bc87d331 fix(tests): new libref
Build / Build-and-ng-test (pull_request) Successful in 5m27s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m39s
Build / Build-and-test-development (pull_request) Failing after 33m47s
2026-08-29 00:17:13 +01:00
hermes f213d24f09 fix: failing test
Build / Build-and-ng-test (pull_request) Successful in 5m24s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m25s
Build / Build-and-test-development (pull_request) Failing after 33m33s
2026-08-28 21:29:15 +01:00
YuryShkoda 20ba633087 feat(licensing): support combined-key paste, object-format features, and a live key preview
- Accept a single pasted/uploaded combined licence key (base64+gzip of
  licence+activation key), auto-detected in either paste mode, alongside
  the legacy two-field format
- Warn and block applying a key generated for the wrong protocol before
  it ever reaches the backend
- Decode both the legacy positional features string and a new named
  features object, so already-issued keys keep working unchanged
- Show a live "Key Details" preview (validity, users, site IDs, enabled
  features) as a key is typed, pasted, or uploaded
- Fix cypress test setup broken by the paste-format default change, and
  an unreliable FileReader-based base64 conversion in the test helpers
2026-08-28 20:01:45 +01:00
hermes cc2ff873fa fix: unnecessary getsubmits call removed
Build / Build-and-ng-test (pull_request) Successful in 5m19s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m40s
Build / Build-and-test-development (pull_request) Failing after 33m35s
2026-08-28 14:02:52 +01:00
hermes 18a16fcb06 chore: lint fix
Build / Build-and-ng-test (pull_request) Successful in 5m53s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m3s
Build / Build-and-test-development (pull_request) Failing after 33m36s
2026-08-27 12:42:08 +01:00
YuryShkoda d72e19a308 fix(editor): resolve a primary key's live formula to its computed value, not the raw formula text
Build / Build-and-ng-test (pull_request) Successful in 5m12s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m40s
Build / Build-and-test-development (pull_request) Successful in 27m13s
2026-08-26 19:08:57 +03:00
YuryShkoda 2cc8489892 fix(editor): resolve computed value for a live formula in numeric columns too
Build / Build-and-ng-test (pull_request) Successful in 5m46s
Lighthouse Checks / lighthouse (pull_request) Successful in 21m55s
Build / Build-and-test-development (pull_request) Successful in 26m47s
saveTable() only resolved a live formula's computed value for character columns - coerceNumericRow (beforePaste/beforeAutofill) deliberately leaves a non-numeric, formula-looking value alone rather than forcing it to NaN, so a numeric column can hold a live formula too. Its raw `=...` text was reaching the backend, which a numeric SAS field can't parse.
2026-08-26 10:52:47 +03:00
YuryShkoda b7d31066ae fix(editor): submit a live formula's computed value, not its raw text
Build / Build-and-ng-test (pull_request) Successful in 5m47s
Lighthouse Checks / lighthouse (pull_request) Successful in 22m27s
Build / Build-and-test-development (pull_request) Successful in 27m6s
saveTable() only resolved computed values for HARDFORMULA/SOFTFORMULA
columns - a formula typed, pasted, or "Apply as formula"'d into any
other character column still had its raw `=...` text submitted,
since HyperFormula recomputes the display without ever mutating the
underlying dataSource string.
2026-08-25 18:15:58 +03:00
YuryShkoda 92d0d91440 feat(editor): scope live formulas to character columns, not just HARDFORMULA/SOFTFORMULA ones
Only backend-sourced `=`-led values are auto-escaped now; anything the
user types or pastes into a character column evaluates as a real
formula, and "Apply as formula" lets a user promote an auto-escaped
cell explicitly. Formulas are enabled on every table instead of only
ones with formula rules.
2026-08-20 18:44:47 +03:00