chore(release): 7.11.0 [skip ci]

# [7.11.0](https://git.datacontroller.io/dc/dc/compare/v7.10.1...v7.11.0) (2026-07-20)

### Bug Fixes

* audit-skipping syntax correction ([ad7efc9](ad7efc90bc))
* **editor:** ignore cell validation on rows marked for delete ([70c5805](70c5805743))
* explicit use of .sashdat in dc_casload ([6a95067](6a95067618))
* refactor mf_wordsinstr1butnotstr2 and added a wide table to test suite ([be94c68](be94c68f35))
* retry the startup service request once before failing ([5391df8](5391df8f04))
* run dc_casload for dq tables in getdata.  Closes [#264](#264). ([6b16f9b](6b16f9b364))
* updating tests, moving mpe_rowhash to core library ([c5f0e4f](c5f0e4f106))

### Features

* bumping core to get the mp_rowhash definition, [#269](#269) ([f73f5e4](f73f5e47ac))
* new hash algorithm - allows 1000+ cols.  Closes [#269](#269) ([08947f0](08947f0fc8))

CHANGELOG.md

@@ -1,3 +1,58 @@
+# [7.11.0](https://git.datacontroller.io/dc/dc/compare/v7.10.1...v7.11.0) (2026-07-20)
+
+
+### Bug Fixes
+
+* audit-skipping syntax correction ([ad7efc9](https://git.datacontroller.io/dc/dc/commit/ad7efc90bc33c93383e17359d14c0605cded69a9))
+* **editor:** ignore cell validation on rows marked for delete ([70c5805](https://git.datacontroller.io/dc/dc/commit/70c5805743c8c6e61bc216f37cbe8cd4f0614c98))
+* explicit use of .sashdat in dc_casload ([6a95067](https://git.datacontroller.io/dc/dc/commit/6a95067618600d7b03b6af46bf3d74721e42c66a))
+* refactor mf_wordsinstr1butnotstr2 and added a wide table to test suite ([be94c68](https://git.datacontroller.io/dc/dc/commit/be94c68f354d0f9b1eb52213807d1874a74d9b8a))
+* retry the startup service request once before failing ([5391df8](https://git.datacontroller.io/dc/dc/commit/5391df8f04cf40205a600b23c207f97b6fdf5913))
+* run dc_casload for dq tables in getdata.  Closes [#264](https://git.datacontroller.io/dc/dc/issues/264). ([6b16f9b](https://git.datacontroller.io/dc/dc/commit/6b16f9b3647fca2e39d44544201ce972046e01a3))
+* updating tests, moving mpe_rowhash to core library ([c5f0e4f](https://git.datacontroller.io/dc/dc/commit/c5f0e4f106aa4e6a7c2a2fd0ebb2e85d1e7bee7b))
+
+
+### Features
+
+* bumping core to get the mp_rowhash definition, [#269](https://git.datacontroller.io/dc/dc/issues/269) ([f73f5e4](https://git.datacontroller.io/dc/dc/commit/f73f5e47ac38a305bac4ebd7eac48d1a3edf4a83))
+* new hash algorithm - allows 1000+ cols.  Closes [#269](https://git.datacontroller.io/dc/dc/issues/269) ([08947f0](https://git.datacontroller.io/dc/dc/commit/08947f0fc8458be098e3aaf769b1da7fadf480cb))
+
+## [7.10.1](https://git.datacontroller.io/dc/dc/compare/v7.10.0...v7.10.1) (2026-07-15)
+
+
+### Bug Fixes
+
+* **editor:** order validation rules by COLHEADERS, not cols[].VARNUM ([4c2aca3](https://git.datacontroller.io/dc/dc/commit/4c2aca34951d6d5c5cf2df69d5bf4256a9290d53))
+
+# [7.10.0](https://git.datacontroller.io/dc/dc/compare/v7.9.1...v7.10.0) (2026-07-13)
+
+
+### Bug Fixes
+
+* adapt column validation to vertical-array COLTYPE ([#253](https://git.datacontroller.io/dc/dc/issues/253)) ([3016750](https://git.datacontroller.io/dc/dc/commit/301675052fb5cbcb11f54c370ad7eda1d60461fd))
+* backend updates to bring back labels under [#240](https://git.datacontroller.io/dc/dc/issues/240) ([fb94840](https://git.datacontroller.io/dc/dc/commit/fb94840016f067fa3523a3539dd0616d8970eed0))
+* **editor:** normalise unpadded time strings on spreadsheet import ([7b4b4eb](https://git.datacontroller.io/dc/dc/commit/7b4b4ebeeb0e34ec5fcd5c485741d7bbddb43b7e))
+* **hot:** row rendering on resize, resolve blank whitespace ([c7ba025](https://git.datacontroller.io/dc/dc/commit/c7ba025d39aa9681ad51f239503e884fb26613e2))
+* including datadictionary descs in labels, [#240](https://git.datacontroller.io/dc/dc/issues/240) ([e7453a9](https://git.datacontroller.io/dc/dc/commit/e7453a9305263faf4d4f288d50977121353040c7))
+* pass variable formats to front-end in a vertical array instead of horizontal list ([691d6f2](https://git.datacontroller.io/dc/dc/commit/691d6f277ea87051f539681b34da2f39cade8fc3))
+* remove coltype from sasparams table, add it to cols table. ([8101400](https://git.datacontroller.io/dc/dc/commit/81014001ac4da6b8d262c72c65e0c657f186ec84))
+* trim leading/trailing whitespace from Excel header cells on upload ([2728dac](https://git.datacontroller.io/dc/dc/commit/2728dac873ab8c9fc44413f873e4730515ca0e21))
+* **va:** date filters refactor, support more dates ([df5c975](https://git.datacontroller.io/dc/dc/commit/df5c9758697d7481e67d20ed7ff81e9ce4e92ec0))
+* **va:** enable readOnly and hold filtering while editing ([82a254d](https://git.datacontroller.io/dc/dc/commit/82a254d22cb7aaf75aae7b332f0b07fc8bda3e44))
+* **va:** enable Upload button in va mode ([1beb3d4](https://git.datacontroller.io/dc/dc/commit/1beb3d490d0e7b8d5fa7985f121495ed62740cfe))
+* **va:** guard deferred contextMenu toggle against destroyed instance ([b5e9b25](https://git.datacontroller.io/dc/dc/commit/b5e9b2531924bec1a23c67d3ad0ca3f512a75655))
+
+
+### Features
+
+* adding DC_MAXOBS_WEBVIEW config item, closes [#258](https://git.datacontroller.io/dc/dc/issues/258) ([ba7b610](https://git.datacontroller.io/dc/dc/commit/ba7b61082d39767f9539a84de6742a1704573e81))
+* viewer/editor column label display toggle (?labels=true) ([25c12f2](https://git.datacontroller.io/dc/dc/commit/25c12f2b18adda09e6edc93eba263bd5ea0f7409))
+
+
+### Reverts
+
+* enable upload button in va mode ([0d7fd34](https://git.datacontroller.io/dc/dc/commit/0d7fd342970744489a2e6c6fb08f239fe853cb81))
+
 ## [7.9.1](https://git.datacontroller.io/dc/dc/compare/v7.9.0...v7.9.1) (2026-06-30)
 
 

CONTRIBUTING.md

@@ -16,6 +16,21 @@ Add `client/.npmrc` file with following content:
 //pylon.sheetjs.com:54111/:_authToken="TOKEN-GOES-HERE"
 ```
 
+In addition to the registry auth, the `@sheet/crypto` package is installed from a local tarball
+(`client/libraries/sheet-crypto.tgz`) rather than the registry. This tarball is not committed to
+the repo directly - only an encrypted copy is: `client/libraries/sheet-crypto.tgz.gpg`.
+
+Before running `npm i`/`npm ci` in `client`, you must decrypt it using the passphrase
+(stored as the `SHEET_PWD` CI secret - ask a repo maintainer if you don't have it):
+```bash
+cd client
+echo "YOUR_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
+  --output ./libraries/sheet-crypto.tgz \
+  --decrypt ./libraries/sheet-crypto.tgz.gpg
+
+npm i
+```
+
 [Handsontable](https://www.npmjs.com/package/handsontable)
 
 Licence should be inserted in the `client/src/index.html` file:
@@ -66,6 +81,15 @@ To remove the release, you need to do it with repo administration over at [https
 
 # Troubleshooting
 
+## `npm i` fails with ENOENT on `sheet-crypto.tgz`
+
+If `npm i`/`npm ci` in `client` fails with:
+```
+npm error enoent ENOENT: no such file or directory, open '.../client/libraries/sheet-crypto.tgz'
+```
+it means the encrypted `sheet-crypto.tgz.gpg` hasn't been decrypted yet. See
+[Dependencies that requires licences](#dependencies-that-requires-licences) above for how to decrypt it.
+
 ## Makedata service "could not create directory" error
 
 The dcpath folder should have its permissions set so that the system account (SYSUSERID) can both read and write to it.

client/cypress/e2e/coltype-delete-record.cy.ts

@@ -0,0 +1,132 @@
+// Marks this file as an ES module (rather than a global script) so its
+// top-level consts don't collide, under the TS type-checker, with the same
+// names declared in other spec files — see viewer-labels.cy.ts for the same
+// pattern.
+export {}
+
+const username = Cypress.env('username')
+const password = Cypress.env('password')
+const hostUrl = Cypress.env('hosturl')
+const appLocation = Cypress.env('appLocation')
+const longerCommandTimeout = Cypress.env('longerCommandTimeout')
+const serverType = Cypress.env('serverType')
+const libraryToOpenIncludes = Cypress.env(`libraryToOpenIncludes_${serverType}`)
+
+// Regression coverage for issue #253's vertical-array COLTYPE fix — see
+// getdata.sas now sends one COLTYPE JSON-object string per cols[] row (via
+// a LEFT JOIN keyed off the real dataset's columns), instead of one
+// comma-joined sasparams.COLTYPE string covering every column. Two things
+// are specifically at risk from that change and aren't covered elsewhere:
+//   1. `_____DELETE__THIS__RECORD_____` (the delete checkbox column) is a
+//      client-side-only concept — %mp_getcols never has a row for it, so
+//      it can no longer travel via cols[].COLTYPE at all. Its Yes/No
+//      dropdown rule is now hardcoded client-side (deleteRecordColumnRule.ts)
+//      — test 1 proves that hardcode actually renders end to end.
+//   2. Ordinary columns (e.g. SOME_DROPDOWN) still get their rule via the
+//      new per-row cols[].COLTYPE path — test 2 proves the migration didn't
+//      silently drop real columns' rules either.
+// Column positions below come from COLHEADERS in
+// sas/mocks/sasjs/services/editors/getdata.js:
+// "_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,..."
+// — childNodes[0] on a body <tr> is the row-header <th>, so childNodes[N+1]
+// is the Nth (0-indexed) column in that list.
+context('coltype vertical-array regression tests: ', function () {
+  this.beforeAll(() => {
+    cy.visit(`${hostUrl}/SASLogon/logout`)
+    cy.loginAndUpdateValidKey(true)
+  })
+
+  this.beforeEach(() => {
+    cy.visit(hostUrl + appLocation)
+    visitPage('home')
+  })
+
+  it('1 | delete-record column renders its hardcoded Yes/No dropdown', () => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+    clickOnEdit()
+
+    cy.get('.ht_master tbody tr', { timeout: longerCommandTimeout }).then(
+      (rows: any) => {
+        // childNodes[1] = _____DELETE__THIS__RECORD_____ (1st column)
+        cy.get(rows[1].childNodes[1])
+          .click({ force: true })
+          .then(($td) => {
+            cy.get('.htAutocompleteArrow', { withinSubject: $td })
+              .should('exist')
+              .click({ force: true })
+
+            cy.get('.autocompleteEditor .htCore tbody td').should(
+              ($choices) => {
+                const texts = [...$choices].map((el) => el.innerText.trim())
+                expect(texts).to.deep.equal(['No', 'Yes'])
+              }
+            )
+          })
+      }
+    )
+  })
+
+  it('2 | an ordinary dropdown column (SOME_DROPDOWN) still gets its rule from cols[].COLTYPE', () => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+    clickOnEdit()
+
+    cy.get('.ht_master tbody tr', { timeout: longerCommandTimeout }).then(
+      (rows: any) => {
+        // childNodes[4] = SOME_DROPDOWN (4th column, 0-indexed as 3). This
+        // only holds if DcValidator emits rules in VARNUM order — see the
+        // "orders rules by VARNUM" regression test in dc-validator.spec.ts.
+        cy.get(rows[1].childNodes[4])
+          .click({ force: true })
+          .then(($td) => {
+            cy.get('.htAutocompleteArrow', { withinSubject: $td }).should(
+              'exist'
+            )
+          })
+      }
+    )
+  })
+})
+
+const clickOnEdit = (callback?: any) => {
+  cy.get('.btnCtrl button.btn-primary', { timeout: longerCommandTimeout })
+    .click()
+    .then(() => {
+      if (callback) callback()
+    })
+}
+
+const openTableFromTree = (libNameIncludes: string, tablename: string) => {
+  cy.get('.app-loading', { timeout: longerCommandTimeout })
+    .should('not.exist')
+    .then(() => {
+      cy.get('.nav-tree clr-tree > clr-tree-node', {
+        timeout: longerCommandTimeout
+      }).then((treeNodes: any) => {
+        let viyaLib
+
+        for (let node of treeNodes) {
+          if (node.innerText.toLowerCase().includes(libNameIncludes)) {
+            viyaLib = node
+            break
+          }
+        }
+
+        cy.get(viyaLib).within(() => {
+          cy.get('.clr-tree-node-content-container > button').click()
+
+          cy.get('.clr-treenode-link').then((innerNodes: any) => {
+            for (let innerNode of innerNodes) {
+              if (innerNode.innerText.toLowerCase().includes(tablename)) {
+                innerNode.click()
+                break
+              }
+            }
+          })
+        })
+      })
+    })
+}
+
+const visitPage = (url: string) => {
+  cy.visit(`${hostUrl}${appLocation}/#/${url}`)
+}

client/cypress/e2e/editor.cy.ts

@@ -115,8 +115,109 @@ context('editor tests: ', function () {
       })
     })
   })
+
+  // An invalid cell shouldn't block submission if its row
+  // is marked for delete
+  it('5 | Submits a delete-marked row with an invalid cell', (done) => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.get('.ht_master tbody tr').then((rows: any) => {
+          // childNodes[5] = SOME_NUM (numeric) — feed it a non-numeric value
+          // to trigger HOT's native invalid mark.
+          cy.get(rows[1].childNodes[5])
+            .dblclick({ force: true })
+            .then(() => {
+              cy.focused()
+                .clear()
+                .type('not a number{enter}')
+                .then(() => {
+                  // childNodes[1] = _____DELETE__THIS__RECORD_____
+                  setDeleteFlag(rows[1].childNodes[1], 'Yes', () => {
+                    submitTable(() => {
+                      // Validation passed despite the invalid SOME_NUM cell:
+                      // the confirm-submit modal's Submit button is enabled
+                      // (validationDone === 1), not the "invalid values" abort.
+                      cy.get('#submitBtn', {
+                        timeout: longerCommandTimeout
+                      })
+                        .should('exist')
+                        .should('not.be.disabled')
+                        .then(() => done())
+                    })
+                  })
+                })
+            })
+        })
+      })
+    })
+  })
+
+  // The delete exemption must not affect the uniqueness (primary key) check.
+  it('6 | Still blocks a duplicate primary key even on a delete-marked row', (done) => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    clickOnEdit(() => {
+      cy.get('.btn.btn-sm.btn-icon.btn-outline-danger', {
+        timeout: longerCommandTimeout
+      }).then(() => {
+        cy.get('.ht_master tbody tr').then((rows: any) => {
+          cy.get(rows[0].childNodes[2]).then((firstPk) => {
+            const duplicateValue = firstPk.text()
+
+            // childNodes[2] = PRIMARY_KEY_FIELD
+            cy.get(rows[1].childNodes[2])
+              .dblclick({ force: true })
+              .then(() => {
+                cy.focused()
+                  .clear()
+                  .type(`${duplicateValue}{enter}`)
+                  .then(() => {
+                    setDeleteFlag(rows[1].childNodes[1], 'Yes', () => {
+                      submitTable()
+
+                      cy.get('.duplicate-keys-modal', {
+                        timeout: longerCommandTimeout
+                      })
+                        .should('exist')
+                        .then(() => done())
+                    })
+                  })
+              })
+          })
+        })
+      })
+    })
+  })
 })
 
+// Opens the _____DELETE__THIS__RECORD_____ dropdown editor on the given cell
+// and picks the given choice ('Yes'/'No') — same technique as
+// coltype-delete-record.cy.ts (arrow -> autocompleteEditor choice list).
+const setDeleteFlag = (
+  deleteCell: any,
+  value: 'Yes' | 'No',
+  callback?: any
+) => {
+  cy.get(deleteCell)
+    .click({ force: true })
+    .then(($td) => {
+      cy.get('.htAutocompleteArrow', { withinSubject: $td })
+        .should('exist')
+        .click({ force: true })
+
+      cy.get('.autocompleteEditor .htCore tbody td')
+        .contains(value)
+        .click({ force: true })
+        .then(() => {
+          if (callback) callback()
+        })
+    })
+}
+
 const clickOnEdit = (callback?: any) => {
   cy.get('.btnCtrl button.btn-primary', { timeout: longerCommandTimeout })
     .click()

client/cypress/e2e/excel.cy.ts

@@ -386,6 +386,19 @@ context('excel tests: ', function () {
       })
   })
 
+  it('24 | Uploads Excel with leading whitespace in header row (should still succeed)', (done) => {
+    openTableFromTree(libraryToOpenIncludes, 'mpe_x_test')
+
+    // Every header except the first has a leading space, matching a
+    // real-world defect from copy-pasting between spreadsheets/systems.
+    // Header matching must tolerate
+    // this rather than reporting the columns missing and aborting.
+    attachExcelFile('leading_whitespace_header_excel.xlsx', () => {
+      submitExcel()
+      rejectExcel(done)
+    })
+  })
+
   // Large files break Cypress
 
   // it ('? | Uploads Excel with size of 5MB', (done) => {

client/cypress/e2e/liveness.cy.ts

@@ -36,6 +36,38 @@ context('liveness tests: ', function () {
     })
   })
 
+  // Multiple concurrent DC instances can cause the startupservice request
+  // to time out even though a retry succeeds. Fail only the first
+  // startupservice call (any later ones, e.g. from other tests/reloads,
+  // pass through untouched) and confirm the app still reaches the nav tree
+  // instead of showing the startup error modal.
+  it('2 | Recovers from a failed startupservice request via the single retry', () => {
+    let startupServiceCallCount = 0
+
+    cy.intercept('POST', '**/SASjsApi/stp/execute*', (req) => {
+      const isStartupService = JSON.stringify(req.body || '').includes(
+        'startupservice'
+      )
+
+      if (isStartupService) startupServiceCallCount++
+
+      if (isStartupService && startupServiceCallCount === 1) {
+        req.reply({ statusCode: 503, body: 'Service unavailable' })
+      } else {
+        req.continue()
+      }
+    }).as('stpExecute')
+
+    visitPage('home')
+
+    cy.get('.nav-tree clr-tree > clr-tree-node', {
+      // Longer than the retry's own 0-3s pause on top of the normal timeout.
+      timeout: longerCommandTimeout + 3000
+    }).should('exist')
+
+    cy.get('.abortMsg').should('not.exist')
+  })
+
   /**
    * Thist part will be needed if we add more tests in future
    */

client/src/app/editor/editor.component.html

@@ -257,6 +257,7 @@
         >
           <ng-container *ngIf="hotTable.readOnly && !uploadPreview">
             <button
+              *ngIf="!isVaEmbed"
               type="button"
               class="btnView btn icon-collapse btn-sm btn-icon btn-block btn-dimmed"
               (click)="openQb()"
@@ -285,9 +286,7 @@
             </button>
           </ng-container>
 
-          <ng-container
-            *ngIf="!hotTable.readOnly && !uploadPreview && !isVaEmbed"
-          >
+          <ng-container *ngIf="!hotTable.readOnly && !uploadPreview">
             <button
               type="button"
               class="btn btn-sm btn-icon btn-outline-danger"
@@ -471,51 +470,46 @@
               </clr-tooltip-content>
             </clr-tooltip>
 
-            <!-- VA data-driven content mode: single Submit below the grid.
-                 Routes through the same checkSave() pipeline as the normal
-                 editor (validation + approval modal + saveTable). -->
+            <!-- VA data-driven content mode: filter controls only. Edit/Submit
+                 use the normal top toolbar (same as embed=true). VA opens
+                 read-only and filters live with the report; filtering
+                 repopulates the grid, so it is held while editing (change shown
+                 pending) and resumes on return to read-only. -->
             <ng-container *ngIf="isVaEmbed">
-              <!-- Single VA action bar: Submit | Auto-apply toggle | filter
-                   status (pending/loading), all on one line. -->
               <div class="va-filter-controls w-100 mt-2-i">
-                <button
-                  type="button"
-                  class="btn btn-sm btn-primary"
-                  [disabled]="submitLoading"
-                  (click)="checkSave()"
-                  title="Submit changes for approval"
-                >
-                  <clr-icon
-                    aria-hidden="true"
-                    shape="check"
-                    size="20"
-                  ></clr-icon>
-                  Submit
-                </button>
+                <!-- Read-only: auto-apply toggle + manual Apply (confirm mode).
+                     Hidden while editing (filtering is held). -->
+                <ng-container *ngIf="hotTable.readOnly">
+                  <label class="va-toggle">
+                    <input
+                      type="checkbox"
+                      [checked]="vaAutoApply"
+                      (change)="toggleVaAutoApply()"
+                    />
+                    Auto-apply VA filters
+                  </label>
 
-                <label class="va-toggle">
-                  <input
-                    type="checkbox"
-                    [checked]="vaAutoApply"
-                    (change)="toggleVaAutoApply()"
-                  />
-                  Auto-apply VA filters
-                </label>
+                  <button
+                    *ngIf="!vaAutoApply"
+                    [disabled]="
+                      vaFilterStatus !== 'pending' &&
+                      vaFilterStatus !== 'loading'
+                    "
+                    type="button"
+                    class="btn btn-sm btn-primary"
+                    (click)="applyPendingVaFilter()"
+                  >
+                    Apply filter
+                  </button>
+                </ng-container>
 
-                <button
-                  *ngIf="!vaAutoApply"
-                  [disabled]="
-                    vaFilterStatus !== 'pending' && vaFilterStatus !== 'loading'
-                  "
-                  type="button"
-                  class="btn btn-sm btn-primary"
-                  (click)="applyPendingVaFilter()"
-                >
-                  Apply filter
-                </button>
+                <!-- Edit mode: filtering is held so edits aren't wiped. -->
+                <span *ngIf="!hotTable.readOnly" class="va-disabled-text">
+                  VA filtering paused while editing
+                </span>
 
-                <!-- Persistent live region so screen readers announce the
-                     pending/loading transitions as they happen. -->
+                <!-- Persistent live region (shown in both modes) so a filter
+                     change that arrives while editing is announced as pending. -->
                 <span class="va-filter-status" role="status" aria-live="polite">
                   <span
                     *ngIf="vaFilterStatus === 'pending'"

client/src/app/editor/editor.component.scss

@@ -1,4 +1,3 @@
-
 .va-filter-controls {
   display: flex;
   align-items: center;
@@ -23,3 +22,8 @@
   font-style: italic;
   color: #0072a3;
 }
+
+.va-disabled-text {
+  font-style: italic;
+  opacity: 0.7;
+}

client/src/app/editor/editor.component.ts

@@ -472,6 +472,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
 
   private ariaObserver: MutationObserver | undefined
   private ariaCheckInterval: any | undefined
+  private gridResizeObserver: ResizeObserver | undefined
 
   constructor(
     private licenceService: LicenceService,
@@ -1003,6 +1004,14 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     const hot = this.hotInstance
     if (!hot) return
 
+    // Entering edit mode: cancel any scheduled VA filter apply so it can't fire
+    // mid-edit and wipe the edits. A pending filter is re-applied on the return
+    // to read-only (cancelEdit).
+    if (this.vaDebounceTimer) {
+      clearTimeout(this.vaDebounceTimer)
+      this.vaDebounceTimer = undefined
+    }
+
     const columnSorting = hot.getPlugin('multiColumnSorting')
     const sortConfigs = this.getCurrentSortConfigs()
 
@@ -1098,6 +1107,16 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     }
 
     this.checkRowLimit()
+
+    // Back to read-only: apply any VA filter change that was held pending while
+    // editing (live mode). Confirm mode leaves it staged for the Apply button.
+    if (
+      this.isVaEmbed &&
+      this.vaAutoApply &&
+      this.vaFilterStatus === 'pending'
+    ) {
+      this.applyPendingVaFilter()
+    }
   }
 
   /**
@@ -2178,29 +2197,38 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
   toggleHotPlugin(pluginName: string, enable: boolean) {
     const hot = this.hotInstance
 
-    hot.batch(() => {
-      const contextMenuPlugin = hot.getPlugin<any>(pluginName)
+    const contextMenuPlugin = hot.getPlugin<any>(pluginName)
 
-      if (!contextMenuPlugin) {
-        console.warn(
-          'Toggle Hot Plugin failed - Plugin named: ' +
-            pluginName +
-            ' - could not be found.'
-        )
+    if (!contextMenuPlugin) {
+      console.warn(
+        'Toggle Hot Plugin failed - Plugin named: ' +
+          pluginName +
+          ' - could not be found.'
+      )
+      return
+    }
+
+    setTimeout(() => {
+      // The instance may be destroyed/rebuilt within this 100ms window (e.g. a
+      // VA filter reload repopulates the grid). A destroyed plugin has its `hot`
+      // reference deleted, so enablePlugin()/disablePlugin() would throw on
+      // `this.hot.getSettings()`. Bail if the instance is gone or was swapped.
+      if (
+        hot.isDestroyed ||
+        this.hotInstance !== hot ||
+        !contextMenuPlugin.hot
+      ) {
         return
       }
 
-      setTimeout(() => {
-        if (enable) {
-          contextMenuPlugin.enablePlugin()
-          return
-        }
-
+      if (enable) {
+        contextMenuPlugin.enablePlugin()
+      } else {
         contextMenuPlugin.disablePlugin()
-      }, 100)
+      }
 
       hot.render()
-    })
+    }, 100)
   }
 
   private dynamicCellValidationDisabled(row: number, col: number) {
@@ -2842,6 +2870,12 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
       this.ariaCheckInterval = undefined
     }
 
+    // Stop observing the grid container for resizes
+    if (this.gridResizeObserver) {
+      this.gridResizeObserver.disconnect()
+      this.gridResizeObserver = undefined
+    }
+
     // Cancel any pending debounced VA apply
     if (this.vaDebounceTimer) {
       clearTimeout(this.vaDebounceTimer)
@@ -2855,6 +2889,25 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     }
   }
 
+  /**
+   * Re-run height+render when the grid container settles.
+   */
+  private observeGridResize() {
+    const el = this.hotInstance?.rootElement?.parentElement
+    if (!el) return
+
+    this.gridResizeObserver?.disconnect()
+    this.gridResizeObserver = new ResizeObserver(() => {
+      requestAnimationFrame(() => {
+        const hot = this.hotInstance
+        if (!hot || hot.isDestroyed) return
+        hot.updateSettings({ height: this.hotTable.height }, false)
+        hot.render()
+      })
+    })
+    this.gridResizeObserver.observe(el)
+  }
+
   /**
    * Fixes ARIA accessibility issues in the Handsontable component
    * This addresses the accessibility report issues with treegrid and presentation roles
@@ -3150,7 +3203,7 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
         manualColumnResize: true,
         filters: false,
         manualRowResize: true,
-        viewportRowRenderingOffset: 50,
+        viewportRowRenderingOffset: 100,
         // show a bar on the left to enable users to select an entire row
         rowHeaders: (index: number) => {
           return ' '
@@ -3402,6 +3455,14 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
         if (cellMeta && cellMeta.filter === false)
           hot.setCellMeta(row, col, 'filter', true)
 
+        // Toggling Delete? changes every other cell's validation exemption
+        // in this row (see DcValidator.setDefaultValidator) — re-validate so
+        // invalid highlights clear/reappear immediately instead of only at
+        // submit time.
+        if (colProp === '_____DELETE__THIS__RECORD_____') {
+          hot.validateRows([row], () => hot.render())
+        }
+
         this.executeDynamicCellValidationIfApplicable(colProp, col, row)
       }
     })
@@ -3439,8 +3500,6 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     )
 
     hot.addHook('afterRender', (isForced: boolean) => {
-      this.eventService.dispatchEvent('resize')
-
       // Fix ARIA accessibility issues after each render
       this.fixAriaAccessibility()
     })
@@ -3571,16 +3630,18 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
 
     hot.render()
 
+    this.observeGridResize()
+
     // Fix ARIA accessibility issues after table initialization
     setTimeout(() => {
       this.fixAriaAccessibility()
     }, 500)
 
-    // SAS Visual Analytics data-driven content mode: open editable immediately,
-    // re-apply any column visibility chosen by VA before this (filter) reload,
-    // and start receiving VA messages over the postMessage interface.
+    // SAS Visual Analytics data-driven content mode: open in read-only view (NOT
+    // immediate edit — the user clicks Edit to make changes), re-apply any column
+    // visibility chosen by VA before this (filter) reload, and start receiving VA
+    // messages over the postMessage interface.
     if (this.isVaEmbed) {
-      this.editTable()
       if (this.vaMessaging.visibleColumns) {
         this.applyVaColumnVisibility(new Set(this.vaMessaging.visibleColumns))
         if (this.hotInstance) this.hotInstance.render()
@@ -3673,9 +3734,12 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
     this.vaPendingSignature = signature
     this.vaFilterStatus = 'pending'
 
-    // Live mode auto-advances after the debounce settles; confirm waits for the
-    // Apply button. The pending indicator stays visible throughout the debounce.
-    if (this.vaAutoApply) this.scheduleVaFilterApply()
+    // Filtering repopulates the grid, so it must never run while the user is
+    // editing (it would wipe their in-progress edits). While in edit mode we
+    // only surface the pending indicator; the filter is applied on return to
+    // read-only (see cancelEdit). Live mode auto-advances after the debounce
+    // settles; confirm waits for the Apply button.
+    if (this.vaAutoApply && this.hotTable.readOnly) this.scheduleVaFilterApply()
   }
 
   /** Live mode: debounce a burst of VA changes into a single apply of the latest. */
@@ -3738,6 +3802,9 @@ export class EditorComponent implements OnInit, AfterViewInit, OnDestroy {
   /** Apply the staged VA filter (the debounce in live mode, or the Apply button). */
   public applyPendingVaFilter(): void {
     if (!this.vaPendingClauses) return
+    // Never apply while editing — the reload would wipe in-progress edits. The
+    // pending filter stays staged and is applied when the user leaves edit mode.
+    if (!this.hotTable.readOnly) return
     const clauses = this.vaPendingClauses
     const signature = this.vaPendingSignature
     this.vaPendingClauses = null

client/src/app/editor/utils/date.utils.ts

@@ -18,6 +18,22 @@ export const dateToTime = (date: Date) => {
   return timeStr
 }
 
+/**
+ * Normalises a 24-hour time STRING to zero-padded `HH:mm:ss`.
+ *
+ * The `intl-time` cell type (validator + native `<input type=time>` editor) is
+ * strict and rejects `H:mm[:ss]`, so such a value paints red and the editor
+ * opens empty. Pad the hour and default the seconds so the value validates.
+ *
+ * Non time-like input is returned unchanged.
+ */
+export const padTimeString = (value: string): string => {
+  const match = value.match(/^(\d{1,2}):(\d{2})(?::(\d{2}))?$/)
+  if (!match) return value
+  const [, hours, minutes, seconds] = match
+  return `${hours.padStart(2, '0')}:${minutes}:${seconds ?? '00'}`
+}
+
 /**
  * Converts date object to the YYYY-MM-DD
  */

client/src/app/models/TableData.ts

@@ -37,7 +37,6 @@ export interface MaxVarLength {
 
 export interface SASParam {
   COLHEADERS: string
-  COLTYPE: string
   DTTMVARS: string
   DTVARS: string
   CLS_FLAG: number

client/src/app/services/app.service.spec.ts

@@ -0,0 +1,141 @@
+import { EventEmitter } from '@angular/core'
+import { BehaviorSubject, Subject } from 'rxjs'
+import { AppService } from './app.service'
+
+/**
+ * AppService's constructor subscribes to several emitters/subjects (see
+ * subscribe() and the persistSelectedTheme check) and startUpData() touches
+ * a handful of methods on each collaborator — these stubs cover exactly
+ * that surface, not the full real services (no TestBed/DI needed, same
+ * plain-instantiation-with-stubs precedent as va-filter.service.spec.ts).
+ */
+const buildDeps = () => {
+  const licenceService: any = {
+    isAppActivated: new BehaviorSubject<boolean | null>(null),
+    activation: jasmine.createSpy('activation').and.resolveTo(undefined)
+  }
+  const eventService: any = {
+    showInfoModal: jasmine.createSpy('showInfoModal'),
+    startupDataLoaded: jasmine.createSpy('startupDataLoaded'),
+    toggleDarkMode: jasmine.createSpy('toggleDarkMode')
+  }
+  const sasService: any = {
+    loadStartupServiceEmitter: new EventEmitter<any>(),
+    requestSiteIdEmitter: new EventEmitter<string>(),
+    request: jasmine.createSpy('request')
+  }
+  const loggerService: any = {
+    log: jasmine.createSpy('log')
+  }
+  const appSettingsService: any = {
+    settings: new BehaviorSubject({ persistSelectedTheme: false } as any)
+  }
+  const router: any = {
+    events: new Subject(),
+    url: '',
+    navigateByUrl: jasmine.createSpy('navigateByUrl')
+  }
+  const appStoreService: any = {
+    getDcAdapterSettings: () => undefined
+  }
+
+  return {
+    licenceService,
+    eventService,
+    sasService,
+    loggerService,
+    appSettingsService,
+    router,
+    appStoreService
+  }
+}
+
+const buildAppService = (deps: ReturnType<typeof buildDeps>) =>
+  new AppService(
+    deps.licenceService,
+    deps.eventService,
+    deps.sasService,
+    deps.loggerService,
+    deps.appSettingsService,
+    deps.router,
+    deps.appStoreService
+  )
+
+// Minimal valid startupservice payload — just enough to pass startUpData()'s
+// missing-props check (Globvars/Sasdatasets/Saslibs/XLMaps).
+const validStartupResponse = () => ({
+  adapterResponse: {
+    SYSSITE: 'SITE1',
+    globvars: [{ ISADMIN: false, DC_ADMIN_GROUP: '', DCLIB: 'DCLIB' }],
+    sasdatasets: [],
+    saslibs: {},
+    xlmaps: []
+  }
+})
+
+describe('AppService - startup retry', () => {
+  it('retries once and succeeds without showing the error modal when only the first attempt rejects', async () => {
+    const deps = buildDeps()
+    deps.sasService.request.and.returnValues(
+      Promise.reject('timeout'),
+      Promise.resolve(validStartupResponse())
+    )
+
+    const appService = buildAppService(deps)
+    // Keep the retry pause instant in tests — see STARTUP_RETRY_PLAN.md.
+    ;(appService as any).retryOptions = {
+      wait: () => Promise.resolve(),
+      random: () => 0
+    }
+
+    await appService.startUpData()
+
+    expect(deps.sasService.request).toHaveBeenCalledTimes(2)
+    expect(deps.eventService.showInfoModal).not.toHaveBeenCalled()
+    expect(deps.eventService.startupDataLoaded).toHaveBeenCalled()
+  })
+
+  it('shows the error modal once (not twice) when both attempts reject', async () => {
+    const deps = buildDeps()
+    deps.sasService.request.and.returnValues(
+      Promise.reject('timeout-1'),
+      Promise.reject('timeout-2')
+    )
+
+    const appService = buildAppService(deps)
+    ;(appService as any).retryOptions = {
+      wait: () => Promise.resolve(),
+      random: () => 0
+    }
+
+    await appService.startUpData()
+
+    expect(deps.sasService.request).toHaveBeenCalledTimes(2)
+    expect(deps.eventService.showInfoModal).toHaveBeenCalledTimes(1)
+    expect(deps.licenceService.isAppActivated.value).toBeFalse()
+  })
+
+  it('does not retry a response that arrives but is missing required properties', async () => {
+    const deps = buildDeps()
+    deps.sasService.request.and.resolveTo({
+      adapterResponse: {
+        SYSSITE: 'SITE1',
+        // globvars deliberately omitted
+        sasdatasets: [],
+        saslibs: {},
+        xlmaps: []
+      }
+    })
+
+    const appService = buildAppService(deps)
+    ;(appService as any).retryOptions = {
+      wait: () => Promise.resolve(),
+      random: () => 0
+    }
+
+    await appService.startUpData()
+
+    expect(deps.sasService.request).toHaveBeenCalledTimes(1)
+    expect(deps.eventService.showInfoModal).toHaveBeenCalledTimes(1)
+  })
+})

client/src/app/services/app.service.ts

@@ -11,11 +11,15 @@ import { AppSettingsService } from './app-settings.service'
 import { AppThemes } from '../models/AppSettings'
 import { RequestWrapperResponse } from '../models/request-wrapper/RequestWrapperResponse'
 import { AppStoreService } from './app-store.service'
+import { retryOnce, RetryOnceOptions } from '../shared/utils/retry-once'
 
 @Injectable()
 export class AppService {
   public syssite = new BehaviorSubject<string[] | null>(null)
   private environmentInfo: EnvironmentInfo = {}
+  // Overridable in tests to keep the retry pause instant — see
+  // retry-once.spec.ts and app.service.spec.ts.
+  private retryOptions: RetryOnceOptions = {}
 
   constructor(
     private licenceService: LicenceService,
@@ -82,8 +86,13 @@ export class AppService {
   public async startUpData() {
     let startupServiceError = false
 
-    await this.sasService
-      .request('public/startupservice', null)
+    // Retry once on failure — multiple concurrent DC instances (e.g. VA
+    // multi-page reports) can cause this request to time out even though a
+    // second attempt succeeds. See STARTUP_RETRY_PLAN.md.
+    await retryOnce(
+      () => this.sasService.request('public/startupservice', null),
+      this.retryOptions
+    )
       .then(async (res: RequestWrapperResponse) => {
         this.syssite.next([res.adapterResponse.SYSSITE])
 

client/src/app/services/va-filter.service.spec.ts

@@ -55,11 +55,11 @@ describe('VaFilterService', () => {
     'SOME_DATETIME'
   ]
   const cols = [
-    { NAME: 'SOME_CHAR', TYPE: 'char', DDTYPE: 'CHAR' },
-    { NAME: 'SOME_NUM', TYPE: 'num', DDTYPE: 'NUMERIC' },
-    { NAME: 'SOME_TIME', TYPE: 'num', DDTYPE: 'TIME' },
-    { NAME: 'SOME_DATE', TYPE: 'num', DDTYPE: 'DATE' },
-    { NAME: 'SOME_DATETIME', TYPE: 'num', DDTYPE: 'DATETIME' }
+    { NAME: 'SOME_CHAR', DDTYPE: 'CHARACTER' },
+    { NAME: 'SOME_NUM', DDTYPE: 'NUMERIC' },
+    { NAME: 'SOME_TIME', DDTYPE: 'TIME' },
+    { NAME: 'SOME_DATE', DDTYPE: 'DATE' },
+    { NAME: 'SOME_DATETIME', DDTYPE: 'DATETIME' }
   ]
 
   beforeEach(() => {
@@ -122,6 +122,90 @@ describe('VaFilterService', () => {
       ).toBe('365')
     })
 
+    it('passes an already-internal (bare number) temporal value through raw', () => {
+      // VA sends the raw SAS internal value when the data item has no format.
+      expect(build([{ label: 'SOME_DATE', value: '182' }])[0].RAW_VALUE).toBe(
+        '182'
+      )
+      expect(
+        build([{ label: 'SOME_DATETIME', value: '365' }])[0].RAW_VALUE
+      ).toBe('365')
+      expect(build([{ label: 'SOME_TIME', value: '2' }])[0].RAW_VALUE).toBe('2')
+    })
+
+    it('parses ISO date shapes identically to DATE9 (VA sends no format)', () => {
+      const date9 = build([{ label: 'SOME_DATE', value: '01AUG1962' }])[0]
+        .RAW_VALUE
+      // The production symptom: YYMMDDS "2025/03/31" style must parse, not drop.
+      expect(
+        build([{ label: 'SOME_DATE', value: '1962-08-01' }])[0].RAW_VALUE
+      ).toBe(date9)
+      expect(
+        build([{ label: 'SOME_DATE', value: '1962/08/01' }])[0].RAW_VALUE
+      ).toBe(date9)
+      // A real production-format value must yield a clause (not be dropped).
+      expect(build([{ label: 'SOME_DATE', value: '2025/03/31' }])).toHaveSize(1)
+    })
+
+    it('resolves ambiguous slash dates day-first (never US), disambiguating by value', () => {
+      const iso = build([{ label: 'SOME_DATE', value: '2025-03-31' }])[0]
+        .RAW_VALUE
+      // 31 can only be a day -> both orders resolve to 2025-03-31.
+      expect(
+        build([{ label: 'SOME_DATE', value: '31/03/2025' }])[0].RAW_VALUE
+      ).toBe(iso) // day-first
+      expect(
+        build([{ label: 'SOME_DATE', value: '03/31/2025' }])[0].RAW_VALUE
+      ).toBe(iso) // value forces month-first
+      // Genuinely ambiguous (both <= 12): default DAY-first => 4th March, NOT 3rd April.
+      expect(
+        build([{ label: 'SOME_DATE', value: '04/03/2025' }])[0].RAW_VALUE
+      ).toBe(build([{ label: 'SOME_DATE', value: '2025-03-04' }])[0].RAW_VALUE)
+      // Impossible date is dropped, not rolled over.
+      expect(build([{ label: 'SOME_DATE', value: '31/02/2025' }])).toEqual([])
+    })
+
+    it('uses the column FMTNAME to resolve ambiguous dates (MMDDYY reads US month-first)', () => {
+      const iso = (v: string) =>
+        service.buildClauses(
+          msg([{ label: 'D', value: v }]),
+          new Map([['d', 'D']]),
+          [{ NAME: 'D', DDTYPE: 'DATE' }]
+        )[0]?.RAW_VALUE
+      const withFmt = (fmt: string, v: string) =>
+        service.buildClauses(
+          msg([{ label: 'D', value: v }]),
+          new Map([['d', 'D']]),
+          [{ NAME: 'D', DDTYPE: 'DATE', FMTNAME: fmt }]
+        )[0]?.RAW_VALUE
+      // 04/03/2025, both <= 12 (ambiguous): FMTNAME decides.
+      expect(withFmt('MMDDYY', '04/03/2025')).toBe(iso('2025-04-03')) // 3 April
+      expect(withFmt('DDMMYY', '04/03/2025')).toBe(iso('2025-03-04')) // 4 March
+      // Value still overrides a wrong FMTNAME: 31 can only be the day.
+      expect(withFmt('MMDDYY', '31/03/2025')).toBe(iso('2025-03-31'))
+      // Falls back to the full FORMAT (e.g. MMDDYY10.) when FMTNAME is blank.
+      const withFormat = service.buildClauses(
+        msg([{ label: 'D', value: '04/03/2025' }]),
+        new Map([['d', 'D']]),
+        [{ NAME: 'D', DDTYPE: 'DATE', FMTNAME: '', FORMAT: 'MMDDYY10.' }]
+      )[0]?.RAW_VALUE
+      expect(withFormat).toBe(iso('2025-04-03')) // 3 April, from FORMAT
+    })
+
+    it('parses ISO datetime shapes identically to SAS DATETIME', () => {
+      const sas = build([
+        { label: 'SOME_DATETIME', value: '01JAN1960:00:06:05' }
+      ])[0].RAW_VALUE
+      expect(
+        build([{ label: 'SOME_DATETIME', value: '1960-01-01T00:06:05' }])[0]
+          .RAW_VALUE
+      ).toBe(sas)
+      expect(
+        build([{ label: 'SOME_DATETIME', value: '1960-01-01 00:06:05' }])[0]
+          .RAW_VALUE
+      ).toBe(sas)
+    })
+
     it('matches the column by label case-insensitively', () => {
       expect(build([{ label: 'some_char', value: 'x' }])[0].VARIABLE_NM).toBe(
         'SOME_CHAR'
@@ -153,6 +237,21 @@ describe('VaFilterService', () => {
         ['SOME_DATETIME', '365']
       ])
     })
+
+    it('infers ISO date/datetime shapes by value shape', () => {
+      const clauses = service.buildInitialClauses(
+        msg([
+          { name: 'pr1', label: 'SOME_DATE', value: '1960-07-01' },
+          { name: 'pr2', label: 'SOME_DATE2', value: '1960/07/01' },
+          { name: 'pr3', label: 'SOME_DTTM', value: '1960-01-01T00:06:05' }
+        ])
+      )
+      expect(clauses.map((c) => [c.VARIABLE_NM, c.RAW_VALUE])).toEqual([
+        ['SOME_DATE', '182'],
+        ['SOME_DATE2', '182'],
+        ['SOME_DTTM', '365']
+      ])
+    })
   })
 
   describe('signature', () => {

client/src/app/services/va-filter.service.ts

@@ -72,7 +72,9 @@ export class VaFilterService {
       const varName = labelToVar.get(key)
       if (!varName) continue
 
-      const kind = this.columnKind(varName, cols)
+      const col = this.findCol(varName, cols)
+      const kind = this.columnKind(col)
+      const order = this.dateOrderFromFmt(col?.FMTNAME || col?.FORMAT)
       const rawValues = Array.isArray(param.value) ? param.value : [param.value]
       // Format/validate every value for the column's SAS type. Numeric columns
       // must receive a finite number (untrusted postMessage input is
@@ -81,7 +83,7 @@ export class VaFilterService {
       // escaped. Invalid values are dropped.
       const formatted = rawValues
         .filter((v) => v !== undefined && v !== null && v !== '')
-        .map((v) => this.formatValueByKind(v, kind))
+        .map((v) => this.formatValueByKind(v, kind, order))
         .filter((v): v is string => v !== null)
       if (formatted.length === 0) continue
       clauses.push(this.clause(varName, formatted))
@@ -151,34 +153,63 @@ export class VaFilterService {
     }
   }
 
-  /**
-   * SAS data-type kind of a column, from the loaded column specs. DDTYPE carries
-   * TIME/DATE/DATETIME; checked DATETIME-before-DATE (substring).
-   */
-  private columnKind(varName: string, cols: any[]): VaColumnKind {
-    const col = (cols || []).find(
+  /** The loaded column spec whose NAME matches varName (case-insensitive). */
+  private findCol(varName: string, cols: any[]): any {
+    return (cols || []).find(
       (x: any) =>
         (x?.NAME ?? '').toString().toUpperCase() === varName.toUpperCase()
     )
+  }
+
+  /**
+   * SAS data-type kind of a column spec. DDTYPE carries
+   * TIME/DATE/DATETIME/NUMERIC/CHARACTER; checked DATETIME-before-DATE (substring).
+   */
+  private columnKind(col: any): VaColumnKind {
     const ddtype = (col?.DDTYPE ?? '').toString().toUpperCase()
     if (ddtype.includes('DATETIME')) return 'datetime'
     if (ddtype.includes('DATE')) return 'date'
     if (ddtype.includes('TIME')) return 'time'
-    if ((col?.TYPE ?? '') === 'num') return 'numeric'
+    if (ddtype.includes('NUMERIC') || (col?.TYPE ?? '') === 'num') {
+      return 'numeric'
+    }
     return 'char'
   }
 
+  /**
+   * Day/month/year order declared by a column's SAS format, used to resolve
+   * ambiguous numeric dates. Accepts either the base FMTNAME (e.g. `DDMMYY`) or
+   * the full FORMAT (e.g. `DDMMYY10.`) — the leading name is the same and is all
+   * that matters for order (width is irrelevant). This is the SOURCE column's
+   * format; the VA report can in principle display a different one, so
+   * value-based disambiguation (a group > 12 is the day) still takes precedence —
+   * this only decides the genuinely-ambiguous case. `undefined` when the format
+   * is absent, month-name (DATE*), or locale-dependent (NLDATE).
+   */
+  private dateOrderFromFmt(fmtname: any): 'YMD' | 'DMY' | 'MDY' | undefined {
+    const f = (fmtname ?? '').toString().toUpperCase()
+    if (/^MMDDYY/.test(f)) return 'MDY'
+    if (/^(DDMMYY|EURDF)/.test(f)) return 'DMY'
+    if (/^(YYMMDD|E8601DA|B8601DA)/.test(f)) return 'YMD'
+    return undefined
+  }
+
   /**
    * Infers a column's SAS kind from a VA value string (deferred load, no
-   * metadata). VA's temporal strings are distinctive — `ddMMMyyyy[:HH:MM:SS]`
-   * and `H:MM:SS` — so they're detected by shape; otherwise fall back to the VA
+   * metadata). VA's temporal strings are distinctive — DATE9 `ddMMMyyyy` and ISO
+   * `yyyy-MM-dd`/`yyyy/MM/dd`, optionally with a `[ T:]HH:MM[:SS]` time part, and
+   * `H:MM:SS` — so they're detected by shape; otherwise fall back to the VA
    * `dataType`.
    */
   private inferKindFromValue(value: string, dataType?: string): VaColumnKind {
-    if (/^\d{1,2}[A-Za-z]{3}\d{4}:\d{2}:\d{2}(:\d{2})?$/.test(value)) {
+    const monName = String.raw`\d{1,2}[A-Za-z]{3}\d{2,4}`
+    const ymd = String.raw`\d{4}[-/.]\d{1,2}[-/.]\d{1,2}` // 4-digit year first
+    const dmy = String.raw`\d{1,2}[-/.]\d{1,2}[-/.]\d{4}` // 4-digit year last
+    const date = `${monName}|${ymd}|${dmy}`
+    if (new RegExp(`^(?:${date})[ T:]\\d{1,2}:\\d{2}(:\\d{2})?$`).test(value)) {
       return 'datetime'
     }
-    if (/^\d{1,2}[A-Za-z]{3}\d{4}$/.test(value)) return 'date'
+    if (new RegExp(`^(?:${date})$`).test(value)) return 'date'
     if (/^\d{1,2}:\d{2}(:\d{2})?(\.\d+)?$/.test(value)) return 'time'
     if (dataType === 'number') return 'numeric'
     return 'char'
@@ -197,8 +228,18 @@ export class VaFilterService {
    * either yields a finite number or is dropped — also the injection guard.
    * Date/datetime reuse HelperService.convertJsDateToSasDate so the value is
    * byte-identical to what DC's own pickers produce.
+   *
+   * A temporal value that is already a bare number is the SAS internal value
+   * itself (VA sends it unformatted when the data item carries no format / is
+   * used as a measure), so it is passed through raw. This is unambiguous because
+   * the only date formats in use (DATE9, ISO) always contain a letter or a
+   * separator — a plain integer is never a formatted date.
    */
-  private formatValueByKind(value: any, kind: VaColumnKind): string | null {
+  private formatValueByKind(
+    value: any,
+    kind: VaColumnKind,
+    order?: 'YMD' | 'DMY' | 'MDY'
+  ): string | null {
     const s = String(value).trim()
     if (s === '') return null
     switch (kind) {
@@ -207,17 +248,20 @@ export class VaFilterService {
         return Number.isFinite(num) ? `${num}` : null
       }
       case 'time': {
+        if (this.isBareNumber(s)) return `${Number(s)}`
         const secs = this.timeToSeconds(s)
         return secs === null ? null : `${secs}`
       }
       case 'date': {
-        const d = this.sasDateToJsDate(s)
+        if (this.isBareNumber(s)) return `${Number(s)}`
+        const d = this.parseDate(s, order)
         return d
           ? `${this.helperService.convertJsDateToSasDate(d, 'days')}`
           : null
       }
       case 'datetime': {
-        const d = this.sasDatetimeToJsDate(s)
+        if (this.isBareNumber(s)) return `${Number(s)}`
+        const d = this.parseDatetime(s, order)
         return d
           ? `${this.helperService.convertJsDateToSasDate(d, 'seconds')}`
           : null
@@ -238,29 +282,140 @@ export class VaFilterService {
     return h * 3600 + m * 60 + sec
   }
 
-  /** SAS "ddMMMyyyy" (e.g. 01JUL1960) -> local JS Date, or null. */
-  private sasDateToJsDate(value: string): Date | null {
-    const m = /^(\d{1,2})([A-Za-z]{3})(\d{4})$/.exec(value)
-    if (!m) return null
-    const monIdx = VaFilterService.SAS_MONTHS.indexOf(m[2].toUpperCase())
-    if (monIdx < 0) return null
-    return new Date(Number(m[3]), monIdx, Number(m[1]))
+  /**
+   * VA date string -> local JS Date, or null. VA sends no format metadata, so the
+   * shape is sniffed. Accepted forms (separator may be `-`, `/`, or `.`):
+   *  - DATE9    "ddMMM[yy]yy" (e.g. 01AUG1962)
+   *  - ISO/YMD  "yyyy-MM-dd" (4-digit year first, e.g. 2025-03-31)
+   *  - year-last "dd/mm/yyyy" or "mm/dd/yyyy" (4-digit year last)
+   *
+   * Day/month order for the year-last form is resolved in priority order:
+   *   1. by value  — a group > 12 must be the day (ground truth);
+   *   2. by `order` — the column's declared FMTNAME order (DMY/MDY/YMD);
+   *   3. default DAY-FIRST — never US month-first — matching SAS's DDMMYY
+   *      default and non-US convention.
+   * `new Date(y, m-1, d)` is local, matching HelperService.convertJsDateToSasDate
+   * so the value is byte-identical to DC's own pickers. The constructed date is
+   * round-trip validated to reject impossible values (e.g. 31/02) rather than
+   * silently roll over.
+   */
+  private parseDate(value: string, order?: 'YMD' | 'DMY' | 'MDY'): Date | null {
+    return this.parseMonthNameDate(value) ?? this.parseNumericDate(value, order)
   }
 
-  /** SAS "ddMMMyyyy:HH:MM[:SS]" (e.g. 01JAN1960:00:06:05) -> local JS Date, or null. */
-  private sasDatetimeToJsDate(value: string): Date | null {
-    const m =
-      /^(\d{1,2})([A-Za-z]{3})(\d{4}):(\d{2}):(\d{2})(?::(\d{2}))?$/.exec(value)
+  /** DATE9 "ddMMM[yy]yy" (e.g. 01AUG1962, 01AUG62) -> local Date, or null. */
+  private parseMonthNameDate(value: string): Date | null {
+    const m = /^(\d{1,2})([A-Za-z]{3})(\d{2}|\d{4})$/.exec(value)
     if (!m) return null
-    const monIdx = VaFilterService.SAS_MONTHS.indexOf(m[2].toUpperCase())
-    if (monIdx < 0) return null
-    return new Date(
-      Number(m[3]),
-      monIdx,
-      Number(m[1]),
-      Number(m[4]),
-      Number(m[5]),
-      Number(m[6] || 0)
+    const [, day, monthAbbr, year] = m
+    const monthIndex = VaFilterService.SAS_MONTHS.indexOf(
+      monthAbbr.toUpperCase()
     )
+    if (monthIndex < 0) return null
+    return this.buildDate(this.fullYear(year), monthIndex + 1, Number(day))
+  }
+
+  /**
+   * Numeric date with `-`, `/`, or `.` separators -> local Date, or null:
+   *  - year-first "yyyy-MM-dd"  -> YMD directly;
+   *  - year-last  "dd/mm/yyyy" or "mm/dd/yyyy" -> day/month resolved by
+   *    resolveDayMonth (value, then column order, then day-first default).
+   * A 4-digit year (leading or trailing) is required to anchor the parse;
+   * an all-2-digit value (e.g. "03/04/05") is too ambiguous and is dropped.
+   */
+  private parseNumericDate(
+    value: string,
+    order?: 'YMD' | 'DMY' | 'MDY'
+  ): Date | null {
+    const m = /^(\d{1,4})[-/.](\d{1,2})[-/.](\d{1,4})$/.exec(value)
+    if (!m) return null
+    const [, first, middle, last] = m
+    const yearIsFirst = first.length === 4
+    const yearIsLast = last.length === 4
+
+    if (yearIsFirst) {
+      return this.buildDate(Number(first), Number(middle), Number(last))
+    }
+    if (!yearIsLast) return null
+
+    const { day, month } = this.resolveDayMonth(
+      Number(first),
+      Number(middle),
+      order
+    )
+    return this.buildDate(Number(last), month, day)
+  }
+
+  /**
+   * Resolves the day/month of a year-last date whose two leading groups (`a`,
+   * `b`) are in unknown order, in priority:
+   *   1. by value  — a group > 12 must be the day (ground truth);
+   *   2. by `order` — only MDY flips to month-first; DMY/YMD keep day-first;
+   *   3. default   — DAY-first (`a` is the day), never US month-first.
+   */
+  private resolveDayMonth(
+    a: number,
+    b: number,
+    order?: 'YMD' | 'DMY' | 'MDY'
+  ): { day: number; month: number } {
+    if (a > 12 && b <= 12) return { day: a, month: b }
+    if (b > 12 && a <= 12) return { day: b, month: a }
+    if (order === 'MDY') return { day: b, month: a }
+    return { day: a, month: b }
+  }
+
+  /** Local Date from 1-based month, round-trip validated (null if impossible). */
+  private buildDate(
+    year: number,
+    month: number,
+    day: number,
+    hours = 0,
+    minutes = 0,
+    seconds = 0
+  ): Date | null {
+    if (month < 1 || month > 12 || day < 1 || day > 31) return null
+    const d = new Date(year, month - 1, day, hours, minutes, seconds, 0)
+    // Reject rollovers (e.g. 31 Feb -> 3 Mar) so a bad value is dropped, not
+    // silently converted to the wrong SAS date.
+    if (
+      d.getFullYear() !== year ||
+      d.getMonth() !== month - 1 ||
+      d.getDate() !== day
+    ) {
+      return null
+    }
+    return d
+  }
+
+  /**
+   * VA datetime string -> local JS Date, or null. The date part is any shape
+   * accepted by parseDate; the time part is "HH:MM[:SS]". They are joined by the
+   * SAS ":" separator (e.g. 01JAN1960:05:13:03), an ISO "T", or a space
+   * (e.g. 2025-03-31T05:13:03, 2025-03-31 05:13:03).
+   */
+  private parseDatetime(
+    value: string,
+    order?: 'YMD' | 'DMY' | 'MDY'
+  ): Date | null {
+    const m = /^(.+?)[ T:](\d{1,2}:\d{2}(?::\d{2})?)$/.exec(value)
+    if (!m) return null
+    const date = this.parseDate(m[1], order)
+    if (!date) return null
+    const t = m[2].split(':')
+    date.setHours(Number(t[0]), Number(t[1]), Number(t[2] || 0), 0)
+    return date
+  }
+
+  /** True when the value is a bare (already-internal) SAS number: optional sign,
+   * digits, optional fraction. No letters/separators — so never a formatted date. */
+  private isBareNumber(value: string): boolean {
+    return /^-?\d+(\.\d+)?$/.test(value) && Number.isFinite(Number(value))
+  }
+
+  /** 2-digit SAS year -> 4-digit (SAS default cutoff: 00-19 -> 20xx, else 19xx). */
+  private fullYear(year: string): number {
+    if (year.length === 4) return Number(year)
+    const y = Number(year)
+    return y <= 19 ? 2000 + y : 1900 + y
   }
 }

client/src/app/shared/dc-validator/dc-validator.ts

@@ -17,7 +17,8 @@ import { DQRule, DQRuleTypes } from './models/dq-rules.model'
 import { getDqDataCols } from './utils/getDqDataCols'
 import { getNotNullDefault } from './utils/getNotNullDefault'
 import { mergeColsRules } from './utils/mergeColsRules'
-import { parseColType } from './utils/parseColType'
+import { parseColTypeRow } from './utils/parseColTypeRow'
+import { DELETE_RECORD_COLUMN_RULE } from './utils/deleteRecordColumnRule'
 import { dqValidate } from './validations/dq-validation'
 import {
   datetimeValidator,
@@ -50,7 +51,33 @@ export class DcValidator {
 
     this.sasparams = sasparams
     this.hotInstance = hotInstance
-    this.rules = parseColType(sasparams.COLTYPE)
+    // Each cols[i].COLTYPE is a single JSON-object string (one per column, as
+    // of the vertical-array getdata.sas fix — issue #253), not the old
+    // comma-joined sasparams.COLTYPE list. DELETE_RECORD_COLUMN_RULE is
+    // injected separately — see its own doc comment for why.
+    //
+    // COLHEADERS is the authoritative grid order and the only safe key to
+    // build against: it is the variable list of the *output* dataset, where
+    // getdata.sas hoists the delete column and the PK to the front ("order
+    // delete var and pk fields at start of table"), whereas cols[].VARNUM
+    // describes the *source* table and cols[] itself can arrive sorted by
+    // NAME. Those orders diverge whenever the PK isn't already the source's
+    // first column, so rules are keyed by name and emitted in COLHEADERS
+    // order — never paired positionally against cols[] or VARNUM.
+    //
+    // A column with no parseable COLTYPE falls back to a plain text rule
+    // rather than being dropped: getRules() is consumed positionally against
+    // COLHEADERS, so a missing entry would shift every later column.
+    const ruleByName = new Map<string, DcValidation>([
+      [DELETE_RECORD_COLUMN_RULE.data, { ...DELETE_RECORD_COLUMN_RULE }]
+    ])
+    for (const col of cols) {
+      const rule = parseColTypeRow(col.COLTYPE)
+      if (rule) ruleByName.set(col.NAME, rule)
+    }
+    this.rules = sasparams.COLHEADERS.split(',').map(
+      (name) => ruleByName.get(name) ?? { data: name }
+    )
     this.rules = mergeColsRules(cols, this.rules, $dataFormats)
     this.rules = applyNumericFormats(this.rules)
     this.rules = mapIntlCellTypes(this.rules)
@@ -389,6 +416,27 @@ export class DcValidator {
       const colType = self.rules[i].type || ''
       let handsontableValid = null
 
+      // A row marked for delete is exempt from cell-level validation — its
+      // values are about to be removed. Two deliberate exceptions:
+      // - PK columns still validate: the key identifies WHICH record to
+      //   delete, and blank/invalid PKs are also skipped by the separate
+      //   duplicate-key check (validatePrimaryKeys), so this is the only
+      //   net that would otherwise catch them.
+      // - Uniqueness checks are unaffected by construction: they run in a
+      //   separate pass (editor.component.ts validatePrimaryKeys) that
+      //   never goes through this validator.
+      const markedForDelete =
+        this.instance?.getDataAtRowProp?.(
+          this.row,
+          '_____DELETE__THIS__RECORD_____'
+        ) === 'Yes'
+
+      if (markedForDelete && !self.primaryKeys.includes(col || '')) {
+        callback(true)
+
+        return
+      }
+
       // We call handsontable predefined validators, if it returns false, we validate as FALSE
       // If return is true, we validate with our custom DQ validations
 
@@ -540,8 +588,11 @@ export class DcValidator {
 
     if (rule.clsRule === 'EDIT') {
       rule.readOnly = false
+      // Guard the -1: splice(-1, 1) drops the LAST hidden column rather than
+      // no-op'ing, so an EDIT column that was never hidden would un-hide an
+      // unrelated one (e.g. a column hidden by a HIDDEN DQ rule).
       const hiddenIndex = this.hiddenColumns.indexOf(index)
-      this.hiddenColumns.splice(hiddenIndex, 1)
+      if (hiddenIndex !== -1) this.hiddenColumns.splice(hiddenIndex, 1)
     }
   }
 }

client/src/app/shared/dc-validator/editors/numericAutocomplete.ts

@@ -1,9 +1,9 @@
-import Handsontable, { HotInstance } from 'handsontable'
+import Handsontable from 'handsontable'
 
 export class CustomAutocompleteEditor
   extends Handsontable.editors.AutocompleteEditor
 {
-  constructor(instance: HotInstance) {
+  constructor(instance: Handsontable.Core) {
     super(instance)
   }
 

client/src/app/shared/dc-validator/models/col.model.ts

@@ -9,4 +9,8 @@ export interface Col {
   MEMLABEL: string
   DESC: string
   LONGDESC: string
+  // Handsontable column-formatting spec (JSON-object string), one per column
+  // — see parseColTypeRow.ts. Optional: getdata.sas LEFT JOINs this in, so
+  // an unmatched row can leave it unset.
+  COLTYPE?: string
 }

client/src/app/shared/dc-validator/tests/dc-validator.spec.ts

@@ -23,11 +23,11 @@ describe('DC Validator', () => {
     // Check if COLS merged with FORMATS
     expect(cols[0].TYPE).toEqual('char')
 
-    // Get all
+    // Get all — one rule per cols[] entry, plus the injected
+    // DELETE_RECORD_COLUMN_RULE (never present in cols[], see its own
+    // doc comment)
     const validationRules = dcValidator.getRules()
-    expect(validationRules).toHaveSize(
-      JSON.parse(`[${example_COLTYPE}]`).length
-    )
+    expect(validationRules).toHaveSize(example_cols.length + 1)
 
     // Get col with notnull validation
     const someNumRule = dcValidator.getRule('SOME_NUM')
@@ -66,6 +66,15 @@ describe('DC Validator', () => {
     const someDropdownHardRule = dcValidator.getRule('SOME_DROPDOWN_HARD')
     // Check strict - it is hardselect so strict should be true
     expect(someDropdownHardRule?.strict).toBeTrue()
+
+    // _____DELETE__THIS__RECORD_____ never has a cols[] entry (%mp_getcols
+    // doesn't know about it — see DELETE_RECORD_COLUMN_RULE), but its rule
+    // must still exist with its fixed dropdown shape, independent of cols
+    const deleteRecordRule = dcValidator.getRule(
+      '_____DELETE__THIS__RECORD_____'
+    )
+    expect(deleteRecordRule?.type).toEqual('dropdown')
+    expect(deleteRecordRule?.source).toEqual(['No', 'Yes'])
   })
 
   it('should create an instance of validator and execute its functions', () => {
@@ -278,8 +287,261 @@ describe('DC Validator', () => {
     expect(dcValidator.getRoundDigits('SOME_SHORTNUM')).toEqual(2)
     expect(dcValidator.getRoundDigits('SOME_NUM')).toBeUndefined()
   })
+
+  it('5 | orders rules by COLHEADERS when cols[] arrives in a different order', () => {
+    // %mp_getcols/cols1 can serialize cols[] alphabetically by NAME rather
+    // than in COLHEADERS order (confirmed against a real getdata.sas
+    // response). getRules() is consumed positionally against headerColumns,
+    // so rule order must follow COLHEADERS regardless of cols[]'s own order.
+    const alphabeticalCols = [...example_cols].sort((a, b) =>
+      a.NAME.localeCompare(b.NAME)
+    )
+
+    const dcValidator: DcValidator = new DcValidator(
+      example_sasparams,
+      example_dataformats,
+      alphabeticalCols,
+      example_dqRules,
+      example_dqData
+    )
+
+    expect(dcValidator.getRules().map((rule) => rule.data)).toEqual(
+      example_sasparams.COLHEADERS.split(',')
+    )
+  })
+
+  it("6 | keeps rules aligned with COLHEADERS when the PK is not the source table's first column", () => {
+    // Regression test for the real-world misalignment: getdata.sas hoists the
+    // delete column and PK to the front of the *output* dataset ("order
+    // delete var and pk fields at start of table"), so COLHEADERS order and
+    // cols[].VARNUM order (which describes the *source* table) diverge
+    // whenever the PK isn't already the source's first column. Worst case (as
+    // seen in the wild): the PK is the source's LAST column, so pairing rules
+    // to headers by VARNUM shifts every single column by one.
+    const cols: Col[] = [
+      // deliberately alphabetical, as the backend serializes it
+      makeCol('PRIMARY_KEY_FIELD', 6), // the PK — last in the source table
+      makeCol('SOME_CHAR', 1),
+      makeCol('SOME_DATE', 2),
+      makeCol('SOME_DROPDOWN', 3),
+      makeCol('SOME_NUM', 4),
+      makeCol('SOME_TIME', 5)
+    ]
+    const sasparams: SASParam = {
+      ...example_sasparams,
+      // PK hoisted to the front by getdata.sas, ahead of VARNUM 1..5
+      COLHEADERS:
+        '_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,' +
+        'SOME_DATE,SOME_DROPDOWN,SOME_NUM,SOME_TIME',
+      PK: 'PRIMARY_KEY_FIELD'
+    } as SASParam
+
+    const dcValidator: DcValidator = new DcValidator(
+      sasparams,
+      { vars: {} } as $DataFormats,
+      cols,
+      [],
+      []
+    )
+
+    expect(dcValidator.getRules().map((rule) => rule.data)).toEqual(
+      sasparams.COLHEADERS.split(',')
+    )
+  })
+
+  it('7 | falls back to a text rule rather than dropping a column with an unparseable COLTYPE', () => {
+    // A dropped rule would shorten getRules() and shift every later column
+    // against headerColumns. getdata.sas LEFT JOINs COLTYPE in, so an
+    // unmatched row can legitimately leave it unset.
+    const cols: Col[] = [
+      makeCol('PRIMARY_KEY_FIELD', 1),
+      { ...makeCol('SOME_CHAR', 2), COLTYPE: undefined },
+      { ...makeCol('SOME_NUM', 3), COLTYPE: 'not valid json' }
+    ]
+    const sasparams: SASParam = {
+      ...example_sasparams,
+      COLHEADERS:
+        '_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_NUM',
+      PK: 'PRIMARY_KEY_FIELD'
+    } as SASParam
+
+    const dcValidator: DcValidator = new DcValidator(
+      sasparams,
+      { vars: {} } as $DataFormats,
+      cols,
+      [],
+      []
+    )
+
+    expect(dcValidator.getRules().map((rule) => rule.data)).toEqual(
+      sasparams.COLHEADERS.split(',')
+    )
+  })
+
+  it('8 | does not un-hide an unrelated column when a CLS EDIT column was never hidden', () => {
+    // hiddenColumns.indexOf() returns -1 for a column that isn't hidden, and
+    // splice(-1, 1) drops the LAST element instead of no-op'ing — so an EDIT
+    // column used to silently un-hide a column hidden by a HIDDEN DQ rule.
+    //
+    // The EDIT column must not be the PK: setColumnLevelSecurity forces PK
+    // columns to clsRule 'READ' before the EDIT branch is reached, which would
+    // skip the splice entirely and leave this test passing against the bug.
+    const cols: Col[] = [
+      { ...makeCol('PK_COL', 1), CLS_RULE: 'READ' },
+      { ...makeCol('SECRET_COL', 2), CLS_RULE: 'READ' },
+      { ...makeCol('OPEN_COL', 3), CLS_RULE: 'EDIT' }
+    ]
+    const sasparams: SASParam = {
+      ...example_sasparams,
+      CLS_FLAG: 1,
+      COLHEADERS: '_____DELETE__THIS__RECORD_____,PK_COL,SECRET_COL,OPEN_COL',
+      PK: 'PK_COL'
+    } as SASParam
+    const dqRules: DQRule[] = [
+      { BASE_COL: 'SECRET_COL', RULE_TYPE: 'HIDDEN', RULE_VALUE: '', X: 0 }
+    ]
+
+    const dcValidator: DcValidator = new DcValidator(
+      sasparams,
+      { vars: {} } as $DataFormats,
+      cols,
+      dqRules,
+      []
+    )
+
+    // SECRET_COL is rules[2]; CLS_FLAG also hides the delete column (index 0).
+    // OPEN_COL (rules[3]) is EDIT and was never hidden, so the splice guard is
+    // what keeps SECRET_COL hidden here.
+    expect(dcValidator.getHiddenColumns()).toContain(2)
+  })
+
+  describe('9 | ignores cell validation on rows marked for delete', () => {
+    // An invalid cell shouldn't block submission if its row
+    // is about to be deleted anyway. The PK is exempt from the exemption — it
+    // identifies WHICH record to delete, and a blank/invalid PK on a delete
+    // row would also slip past the separate uniqueness check (which skips
+    // blank PKs).
+    //
+    // The validator is a plain `function` invoked with cellProperties as
+    // `this` (row + instance), so a minimal fake stands in for a real HOT
+    // instance here — only `instance.getDataAtRowProp` is read by the guard.
+    const fakeCellProps = (markedForDelete: boolean): any => ({
+      row: 0,
+      editor: undefined,
+      instance: {
+        getDataAtRowProp: (_row: number, prop: string) =>
+          prop === '_____DELETE__THIS__RECORD_____'
+            ? markedForDelete
+              ? 'Yes'
+              : 'No'
+            : undefined
+      }
+    })
+
+    const buildValidator = () =>
+      new DcValidator(
+        example_sasparams,
+        example_dataformats,
+        example_cols,
+        example_dqRules,
+        example_dqData
+      )
+
+    it('exempts an invalid non-PK cell on a delete-marked row', () => {
+      const dcValidator = buildValidator()
+      const someNumRule = dcValidator.getRule('SOME_NUM')
+
+      someNumRule!.validator!.call(
+        fakeCellProps(true),
+        'not a number',
+        (valid: boolean) => {
+          expect(valid).toBeTrue()
+        }
+      )
+    })
+
+    it('exempts a NOTNULL-empty non-PK cell on a delete-marked row', () => {
+      const dcValidator = buildValidator()
+      const someNumRule = dcValidator.getRule('SOME_NUM')
+
+      someNumRule!.validator!.call(
+        fakeCellProps(true),
+        null,
+        (valid: boolean) => {
+          expect(valid).toBeTrue()
+        }
+      )
+    })
+
+    it('still rejects an invalid non-PK cell when the row is not marked for delete', () => {
+      const dcValidator = buildValidator()
+      const someNumRule = dcValidator.getRule('SOME_NUM')
+
+      someNumRule!.validator!.call(
+        fakeCellProps(false),
+        'not a number',
+        (valid: boolean) => {
+          expect(valid).toBeFalse()
+        }
+      )
+    })
+
+    it('still accepts a valid non-PK cell when the row is not marked for delete', () => {
+      const dcValidator = buildValidator()
+      const someNumRule = dcValidator.getRule('SOME_NUM')
+
+      someNumRule!.validator!.call(
+        fakeCellProps(false),
+        2,
+        (valid: boolean) => {
+          expect(valid).toBeTrue()
+        }
+      )
+    })
+
+    it('still rejects an invalid PK cell even when the row is marked for delete', () => {
+      const dcValidator = buildValidator()
+      const pkRule = dcValidator.getRule('PRIMARY_KEY_FIELD')
+
+      pkRule!.validator!.call(
+        fakeCellProps(true),
+        'not a number',
+        (valid: boolean) => {
+          expect(valid).toBeFalse()
+        }
+      )
+    })
+
+    it('treats the edit-record modal path (dummy HOT instance, no data) as not marked for delete', () => {
+      const dcValidator = buildValidator()
+      const someNumRule = dcValidator.getRule('SOME_NUM')
+
+      // executeHotValidator builds cellProperties against a real but
+      // data-less Handsontable instance — getDataAtRowProp() there returns
+      // undefined, never 'Yes', so the exemption never applies in the modal.
+      dcValidator.executeHotValidator(someNumRule!, null, (valid: boolean) => {
+        expect(valid).toBeFalse()
+      })
+    })
+  })
 })
 
+/** Minimal cols[] entry — only the fields rule ordering depends on. */
+const makeCol = (name: string, varnum: number): Col =>
+  ({
+    NAME: name,
+    VARNUM: varnum,
+    LABEL: name,
+    FMTNAME: '',
+    DDTYPE: 'CHARACTER',
+    TYPE: '',
+    CLS_RULE: 'READ',
+    MEMLABEL: '',
+    DESC: '',
+    LONGDESC: '',
+    COLTYPE: `{"data":"${name}"}`
+  }) as Col
+
 const example_dqData = [
   {
     BASE_COL: 'SOME_DROPDOWN',
@@ -383,6 +645,13 @@ const example_dqRules: any = [
   }
 ]
 
+// One row per column, each carrying its own COLTYPE JSON-object string — the
+// vertical-array shape from issue #253's getdata.sas fix. Order: SOME_DROPDOWN
+// then SOME_NUM first (the `cols[0].TYPE` assertion above is positional),
+// remaining columns in the same order the old concatenated
+// sasparams.COLTYPE used to list them. _____DELETE__THIS__RECORD_____ is
+// deliberately absent — DcValidator injects its rule directly (see
+// DELETE_RECORD_COLUMN_RULE), since %mp_getcols never has a row for it.
 const example_cols = [
   {
     CLS_RULE: 'READ',
@@ -394,7 +663,8 @@ const example_cols = [
     LONGDESC: '',
     MEMLABEL: '',
     NAME: 'SOME_DROPDOWN',
-    VARNUM: 3
+    VARNUM: 3,
+    COLTYPE: '{"data":"SOME_DROPDOWN"}'
   },
   {
     CLS_RULE: 'READ',
@@ -406,77 +676,157 @@ const example_cols = [
     LONGDESC: '',
     MEMLABEL: '',
     NAME: 'SOME_NUM',
-    VARNUM: 4
+    VARNUM: 4,
+    COLTYPE: '{"data":"SOME_NUM","type":"numeric","format":"0"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'NUMERIC',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'PRIMARY_KEY_FIELD',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'PRIMARY_KEY_FIELD',
+    VARNUM: 1,
+    COLTYPE: '{"data":"PRIMARY_KEY_FIELD","type":"numeric","format":"0"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'CHARACTER',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_CHAR',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_CHAR',
+    VARNUM: 2,
+    COLTYPE: '{"data":"SOME_CHAR"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'CHARACTER',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_CHAR_LOW',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_CHAR_LOW',
+    VARNUM: 5,
+    COLTYPE: '{"data":"SOME_CHAR_LOW"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'CHARACTER',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_CHAR_ANY',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_CHAR_ANY',
+    VARNUM: 6,
+    COLTYPE: '{"data":"SOME_CHAR_ANY"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'CHARACTER',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_DROPDOWN_HARD',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_DROPDOWN_HARD',
+    VARNUM: 7,
+    COLTYPE: '{"data":"SOME_DROPDOWN_HARD"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'DATE',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_DATE',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_DATE',
+    VARNUM: 8,
+    COLTYPE: '{"data":"SOME_DATE","type":"date"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'DATETIME',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_DATETIME',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_DATETIME',
+    VARNUM: 9,
+    COLTYPE: '{"data":"SOME_DATETIME","type":"datetime"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'TIME',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_TIME',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_TIME',
+    VARNUM: 10,
+    COLTYPE: '{"data":"SOME_TIME","type":"time"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'NUMERIC',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_SHORTNUM',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_SHORTNUM',
+    VARNUM: 11,
+    COLTYPE: '{"data":"SOME_SHORTNUM","type":"numeric","format":"0"}'
+  },
+  {
+    CLS_RULE: 'READ',
+    DDTYPE: 'NUMERIC',
+    DESC: '',
+    TYPE: '',
+    FMTNAME: '',
+    LABEL: 'SOME_BESTNUM',
+    LONGDESC: '',
+    MEMLABEL: '',
+    NAME: 'SOME_BESTNUM',
+    VARNUM: 12,
+    COLTYPE: '{"data":"SOME_BESTNUM","type":"numeric","format":"0"}'
   }
 ]
 
-const example_COLTYPE = `
-{
-    "data":"_____DELETE__THIS__RECORD_____",
-    "type":"dropdown",
-    "source":[
-        "No",
-        "Yes"
-    ]
-},
-{
-    "data":"PRIMARY_KEY_FIELD",
-    "type":"numeric",
-    "format":"0"
-},
-{
-    "data":"SOME_CHAR"
-},
-{
-	"data":"SOME_CHAR_LOW"
-},
-{
-	"data":"SOME_CHAR_ANY"
-},
-{
-    "data":"SOME_DROPDOWN"
-},
-{
-  "data":"SOME_DROPDOWN_HARD"
-},
-{
-    "data":"SOME_NUM",
-    "type":"numeric",
-    "format":"0"
-},
-{
-    "data":"SOME_DATE",
-    "type":"date"
-},
-{
-    "data":"SOME_DATETIME",
-    "type":"datetime"
-},
-{
-    "data":"SOME_TIME",
-    "type":"time"
-},
-{
-    "data":"SOME_SHORTNUM",
-    "type":"numeric",
-    "format":"0"
-},
-{
-    "data":"SOME_BESTNUM",
-    "type":"numeric",
-    "format":"0"
-}`
-
+// COLHEADERS mirrors a real getdata.sas response: the *output* dataset's
+// variable list, i.e. the delete column and PK hoisted to the front, then the
+// remaining source columns in VARNUM order. It drives rule order, so it must
+// list every example_cols entry.
 const example_sasparams = {
   CLS_FLAG: 0,
-  COLHEADERS: 'head1 head2',
-  COLTYPE: example_COLTYPE,
+  COLHEADERS:
+    '_____DELETE__THIS__RECORD_____,PRIMARY_KEY_FIELD,SOME_CHAR,SOME_DROPDOWN,' +
+    'SOME_NUM,SOME_CHAR_LOW,SOME_CHAR_ANY,SOME_DROPDOWN_HARD,SOME_DATE,' +
+    'SOME_DATETIME,SOME_TIME,SOME_SHORTNUM,SOME_BESTNUM',
   DTTMVARS: 'dttm vars',
   DTVARS: 'dt vars',
   FILTER_TEXT: 'filter text',
   LOADTYPE: 'load type',
-  PK: 'primary key',
-  PKCNT: 0,
+  PK: 'PRIMARY_KEY_FIELD',
+  PKCNT: 1,
   RK_FLAG: 0,
   TMVARS: 'string'
 }
@@ -507,6 +857,24 @@ const example_dataformats = {
       length: '128',
       type: 'char'
     },
+    SOME_CHAR_LOW: {
+      format: '$128.',
+      label: 'SOME_CHAR_LOW',
+      length: '128',
+      type: 'char'
+    },
+    SOME_CHAR_ANY: {
+      format: '$128.',
+      label: 'SOME_CHAR_ANY',
+      length: '128',
+      type: 'char'
+    },
+    SOME_DROPDOWN_HARD: {
+      format: '$128.',
+      label: 'SOME_DROPDOWN_HARD',
+      length: '128',
+      type: 'char'
+    },
     SOME_NUM: {
       format: 'best.',
       label: 'SOME_NUM',

client/src/app/shared/dc-validator/tests/parseColType.spec.ts

@@ -1,23 +0,0 @@
-import { HotColumnSettings } from '../models/dc-validation.model'
-import { parseColType } from '../utils/parseColType'
-
-describe('DC Validator - parse col type', () => {
-  it('should return array of parsed json', () => {
-    const colTypeString =
-      '{"data":"test","test2":"test2"}, {"data":"test3","test4":"test4"}'
-    const expected: HotColumnSettings[] = [
-      { data: 'test', test2: 'test2' },
-      { data: 'test3', test4: 'test4' }
-    ]
-
-    expect(parseColType(colTypeString)).toEqual(expected)
-  })
-
-  it('should return empty array for invalid json', () => {
-    const colTypeString =
-      '{"test":"test""test:"test2"}, {"test3":"test3","test4":"test4"}'
-    const expected: HotColumnSettings[] = []
-
-    expect(parseColType(colTypeString)).toEqual(expected)
-  })
-})

client/src/app/shared/dc-validator/tests/parseColTypeRow.spec.ts

@@ -0,0 +1,24 @@
+import { HotColumnSettings } from '../models/dc-validation.model'
+import { parseColTypeRow } from '../utils/parseColTypeRow'
+
+describe('DC Validator - parse col type row', () => {
+  it('parses a single JSON object', () => {
+    const expected: HotColumnSettings = { data: 'test', type: 'numeric' }
+
+    expect(parseColTypeRow('{"data":"test","type":"numeric"}')).toEqual(
+      expected
+    )
+  })
+
+  it('returns undefined for undefined input (LEFT JOIN can leave it unset)', () => {
+    expect(parseColTypeRow(undefined)).toBeUndefined()
+  })
+
+  it('returns undefined for an empty string', () => {
+    expect(parseColTypeRow('')).toBeUndefined()
+  })
+
+  it('returns undefined for malformed JSON, without throwing', () => {
+    expect(parseColTypeRow('{not valid')).toBeUndefined()
+  })
+})

client/src/app/shared/dc-validator/utils/deleteRecordColumnRule.ts

@@ -0,0 +1,16 @@
+import { DcValidation } from '../models/dc-validation.model'
+
+/**
+ * `_____DELETE__THIS__RECORD_____` is a client-side UI concept (the delete
+ * checkbox column) that getdata.sas synthesizes into vars3/vars4 for display
+ * purposes only — it's never a real column of the underlying dataset, so
+ * `%mp_getcols` (which builds `cols1`, the left side of the join into
+ * `cols`) never has a row for it, and its rule can no longer travel via
+ * `cols[].COLTYPE` (see issue #253's vertical-array fix). The rule is fixed
+ * and data-independent, so it's injected directly here instead.
+ */
+export const DELETE_RECORD_COLUMN_RULE: DcValidation = {
+  data: '_____DELETE__THIS__RECORD_____',
+  type: 'dropdown',
+  source: ['No', 'Yes']
+}

client/src/app/shared/dc-validator/utils/mergeColsRules.ts

@@ -27,7 +27,7 @@ export const mergeColsRules = (
         : colFormats.type
 
     if (rule && col.DESC) rule.desc = col.DESC
-    if (rule && colFormats.length) rule.length = parseInt(colFormats.length)
+    if (rule && colFormats?.length) rule.length = parseInt(colFormats.length)
     if (rule && col.CLS_RULE) rule.clsRule = col.CLS_RULE
     if (rule && colFormats?.type) rule.sasType = colFormats.type
   }

client/src/app/shared/dc-validator/utils/parseColType.ts

@@ -1,16 +0,0 @@
-import { HotColumnSettings } from '../models/dc-validation.model'
-
-/**
- * From sas we get a string instead of array of objects, in that string `[]` are missing so
- * before parsing JSON we need to add them.
- *
- * @param coltype string (objects) that comes from sas
- * @returns JSON Handsontable.ColumnSettings[]
- */
-export const parseColType = (coltype: string): HotColumnSettings[] => {
-  try {
-    return JSON.parse(`[${coltype}]`)
-  } catch (err: any) {
-    return []
-  }
-}

client/src/app/shared/dc-validator/utils/parseColTypeRow.ts

@@ -0,0 +1,23 @@
+import { HotColumnSettings } from '../models/dc-validation.model'
+
+/**
+ * Parses a single column's `COLTYPE` JSON-object string (one per `cols` row,
+ * as of the vertical-array getdata.sas fix — see issue #253). Unlike the
+ * previous comma-joined `sasparams.COLTYPE` format, each row is independent:
+ * a malformed or missing value here only drops that one column's rule,
+ * rather than every column's.
+ *
+ * @param coltype a single column's COLTYPE JSON string, or undefined (the
+ *   LEFT JOIN in getdata.sas can leave it unset for unmatched rows)
+ */
+export const parseColTypeRow = (
+  coltype: string | undefined
+): HotColumnSettings | undefined => {
+  if (!coltype) return undefined
+
+  try {
+    return JSON.parse(coltype)
+  } catch (err: any) {
+    return undefined
+  }
+}

client/src/app/shared/spreadsheet-util/spreadsheet-util.spec.ts

@@ -0,0 +1,151 @@
+import { BehaviorSubject } from 'rxjs'
+import * as XLSX from '@sheet/crypto'
+import { SpreadsheetUtil } from './spreadsheet-util'
+import { LicenceState } from 'src/app/models/LicenceState'
+import { ParseParams } from 'src/app/models/ParseParams.interface'
+import { SearchDataExcelResult } from 'src/app/models/SearchDataExcelResult.interface'
+
+describe('SpreadsheetUtil - header matching', () => {
+  // licenceState is unused by searchDataInExcel, so an empty stub is fine.
+  const buildSpreadsheetUtil = () =>
+    new SpreadsheetUtil({
+      licenceState: new BehaviorSubject<LicenceState>({} as LicenceState)
+    })
+
+  const buildWorkbook = (headerRow: string[], dataRow: any[]) => {
+    const ws = XLSX.utils.aoa_to_sheet([headerRow, dataRow])
+    const wb = XLSX.utils.book_new()
+
+    XLSX.utils.book_append_sheet(wb, ws, 'Sheet1')
+
+    return wb
+  }
+
+  // searchDataInExcel only reads headerArray/headerPks off ParseParams; the
+  // other required fields (dcValidator, etc.) are irrelevant here, so a
+  // minimal cast object stands in for a full ParseParams.
+  const buildParseParams = (): ParseParams =>
+    ({
+      headerArray: ['PRIMARY_KEY_FIELD', 'SOME_CHAR', 'SOME_NUM'],
+      headerPks: ['PRIMARY_KEY_FIELD']
+    }) as any as ParseParams
+
+  // searchDataInExcel is private; there's no existing precedent in this
+  // codebase for testing private methods, so this casts through `any` to
+  // reach it directly rather than exercising it via the public
+  // parseSpreadsheetFile (which would need a real File/FileReader).
+  const searchDataInExcel = (
+    spreadsheetUtil: SpreadsheetUtil,
+    wb: XLSX.WorkBook,
+    parseParams: ParseParams
+  ): SearchDataExcelResult =>
+    (spreadsheetUtil as any).searchDataInExcel(wb, parseParams)
+
+  it('matches headers with leading whitespace against the expected clean header names', () => {
+    const wb = buildWorkbook(
+      ['PRIMARY_KEY_FIELD', ' SOME_CHAR', ' SOME_NUM'],
+      [1, 'a value', 42]
+    )
+
+    const result = searchDataInExcel(
+      buildSpreadsheetUtil(),
+      wb,
+      buildParseParams()
+    )
+
+    expect(result.missing).toBeUndefined()
+    expect(result.found).toBeDefined()
+    expect(result.found?.headers).toContain('some_char')
+    expect(result.found?.headers).toContain('some_num')
+  })
+
+  it('still reports a genuinely different header as missing (no over-broadening)', () => {
+    const wb = buildWorkbook(
+      ['PRIMARY_KEY_FIELD', ' SOME_CHARX', ' SOME_NUM'],
+      [1, 'a value', 42]
+    )
+
+    const result = searchDataInExcel(
+      buildSpreadsheetUtil(),
+      wb,
+      buildParseParams()
+    )
+
+    expect(result.found).toBeUndefined()
+    expect(result.missing).toBeDefined()
+    expect(result.missing?.[0].missingHeaders).toContain('SOME_CHAR')
+  })
+})
+
+describe('SpreadsheetUtil - updateDateTimeCols time normalization', () => {
+  const buildSpreadsheetUtil = () =>
+    new SpreadsheetUtil({
+      licenceState: new BehaviorSubject<LicenceState>({} as LicenceState)
+    })
+
+  // updateDateTimeCols is private; cast through `any` (same precedent as above).
+  const updateDateTimeCols = (
+    spreadsheetUtil: SpreadsheetUtil,
+    headers: string[],
+    data: any[],
+    parseParams: ParseParams
+  ): any[] =>
+    (spreadsheetUtil as any).updateDateTimeCols(headers, data, parseParams)
+
+  const timeParseParams = (): ParseParams =>
+    ({
+      dateHeaders: [],
+      dateTimeHeaders: [],
+      timeHeaders: ['SOME_TIME']
+    }) as any as ParseParams
+
+  it('pads unpadded-hour time strings to canonical HH:mm:ss and leaves canonical values alone', () => {
+    // The core bug: SAS TIME8. / Excel render a single-digit hour (`0:00:42`),
+    // arriving as a plain string (not an Excel serial). Already-canonical values
+    // must pass through untouched.
+    const data = [[{ v: '0:00:42' }], [{ v: '9:05:03' }], [{ v: '23:00:00' }]]
+
+    const result = updateDateTimeCols(
+      buildSpreadsheetUtil(),
+      ['SOME_TIME'],
+      data,
+      timeParseParams()
+    )
+
+    expect(result.map((row) => row[0].v)).toEqual([
+      '00:00:42',
+      '09:05:03',
+      '23:00:00'
+    ])
+  })
+
+  it('reads a 2-part value as HOURS-first and defaults seconds (HHMM/TIME5.): 0:42 -> 00:42:00', () => {
+    // TIME5.-7. and HHMM. columns render `HH:mm`. Such values are never `mm:ss`
+    // — DC only classifies TIME/HHMM formats as time columns — so the hour is
+    // padded and minutes kept, with seconds defaulted to :00.
+    const data = [[{ v: '0:42' }], [{ v: '12:25' }]]
+
+    const result = updateDateTimeCols(
+      buildSpreadsheetUtil(),
+      ['SOME_TIME'],
+      data,
+      timeParseParams()
+    )
+
+    expect(result.map((row) => row[0].v)).toEqual(['00:42:00', '12:25:00'])
+  })
+
+  it('reconstructs an Excel serial time to zero-padded HH:mm:ss', () => {
+    // 0.0004861111 of a day = 42 seconds past midnight.
+    const data = [[{ v: 0.0004861111 }]]
+
+    const result = updateDateTimeCols(
+      buildSpreadsheetUtil(),
+      ['SOME_TIME'],
+      data,
+      timeParseParams()
+    )
+
+    expect(result[0][0].v).toBe('00:00:42')
+  })
+})

client/src/app/shared/spreadsheet-util/spreadsheet-util.ts

@@ -2,7 +2,8 @@ import { isSpecialMissing } from '@sasjs/utils/input/validators'
 import {
   dateFormat,
   dateToUtcTime,
-  dateToTime
+  dateToTime,
+  padTimeString
 } from 'src/app/editor/utils/date.utils'
 import {
   getMissingHeaders,
@@ -608,7 +609,11 @@ export class SpreadsheetUtil {
 
         // If the cell does not have `v` property we ignore it, those are metadata properties
         if (cellValue && typeof cellValue === 'string') {
-          const potentialHeader = cellValue.toLowerCase()
+          // .trim(): header cells can pick up incidental leading/trailing
+          // whitespace from copy-pasting between spreadsheets/systems — treat
+          // that the same as the existing case-insensitive matching below,
+          // not as a genuinely different (missing) column.
+          const potentialHeader = cellValue.trim().toLowerCase()
           const headerIndex = csvArrayHeadersLower.indexOf(potentialHeader)
 
           if (headerIndex > -1) {
@@ -852,11 +857,14 @@ export class SpreadsheetUtil {
           if (
             isStringNumber(obj.v) ||
             isStringDecimal(obj.v) ||
-            obj.v.includes('E-')
+            (typeof obj.v === 'string' && obj.v.includes('E-'))
           ) {
             const date = excelDateToJSDate(Number(obj.v))
 
             obj.v = dateToUtcTime(date)
+          } else if (typeof obj.v === 'string') {
+            // `0:00:42`— pad the hour for `intl-time` cell type
+            obj.v = padTimeString(obj.v)
           }
           row[element] = obj
         })

client/src/app/shared/utils/retry-once.spec.ts

@@ -0,0 +1,63 @@
+import { retryOnce } from './retry-once'
+
+describe('retryOnce', () => {
+  it("resolves with the first attempt's value and never waits when it succeeds", async () => {
+    const requestFn = jasmine.createSpy('requestFn').and.resolveTo('ok')
+    const wait = jasmine.createSpy('wait').and.resolveTo(undefined)
+
+    const result = await retryOnce(requestFn, { wait })
+
+    expect(result).toBe('ok')
+    expect(requestFn).toHaveBeenCalledTimes(1)
+    expect(wait).not.toHaveBeenCalled()
+  })
+
+  it('retries exactly once after a rejection and resolves with the retry value', async () => {
+    const requestFn = jasmine
+      .createSpy('requestFn')
+      .and.returnValues(Promise.reject('first-error'), Promise.resolve('ok'))
+    const wait = jasmine.createSpy('wait').and.resolveTo(undefined)
+
+    const result = await retryOnce(requestFn, { wait })
+
+    expect(result).toBe('ok')
+    expect(requestFn).toHaveBeenCalledTimes(2)
+    expect(wait).toHaveBeenCalledTimes(1)
+  })
+
+  it('derives the pre-retry pause from random within [0, maxDelayMs)', async () => {
+    const requestFn = jasmine
+      .createSpy('requestFn')
+      .and.returnValues(Promise.reject('err'), Promise.resolve('ok'))
+    const wait = jasmine.createSpy('wait').and.resolveTo(undefined)
+
+    await retryOnce(requestFn, { wait, random: () => 0.5 })
+    expect(wait).toHaveBeenCalledWith(1500)
+
+    wait.calls.reset()
+    requestFn.and.returnValues(Promise.reject('err'), Promise.resolve('ok'))
+    await retryOnce(requestFn, { wait, random: () => 0 })
+    expect(wait).toHaveBeenCalledWith(0)
+
+    wait.calls.reset()
+    requestFn.and.returnValues(Promise.reject('err'), Promise.resolve('ok'))
+    await retryOnce(requestFn, { wait, random: () => 0.999999 })
+    expect(wait).toHaveBeenCalledWith(jasmine.any(Number))
+    expect(wait.calls.mostRecent().args[0]).toBeLessThan(3000)
+  })
+
+  it('rejects with the second error when both attempts fail, never trying a third time', async () => {
+    const requestFn = jasmine
+      .createSpy('requestFn')
+      .and.returnValues(
+        Promise.reject('first-error'),
+        Promise.reject('second-error')
+      )
+    const wait = jasmine.createSpy('wait').and.resolveTo(undefined)
+
+    await expectAsync(retryOnce(requestFn, { wait })).toBeRejectedWith(
+      'second-error'
+    )
+    expect(requestFn).toHaveBeenCalledTimes(2)
+  })
+})

client/src/app/shared/utils/retry-once.ts

@@ -0,0 +1,33 @@
+export interface RetryOnceOptions {
+  /** Upper bound of the random pre-retry delay, in ms. Default 3000. */
+  maxDelayMs?: number
+  /** Injectable for tests. Default Math.random. */
+  random?: () => number
+  /** Injectable for tests. Default setTimeout-based wait. */
+  wait?: (ms: number) => Promise<void>
+}
+
+const defaultWait = (ms: number): Promise<void> =>
+  new Promise((resolve) => setTimeout(resolve, ms))
+
+/**
+ * Runs requestFn; if it rejects, waits a random 0..maxDelayMs and tries
+ * exactly once more. A second rejection propagates as-is. Multiple
+ * concurrent DC instances (e.g. VA multi-page reports) can cause the
+ * startup request to time out even though a second attempt succeeds.
+ */
+export const retryOnce = async <T>(
+  requestFn: () => Promise<T>,
+  options?: RetryOnceOptions
+): Promise<T> => {
+  const maxDelayMs = options?.maxDelayMs ?? 3000
+  const random = options?.random ?? Math.random
+  const wait = options?.wait ?? defaultWait
+
+  try {
+    return await requestFn()
+  } catch {
+    await wait(random() * maxDelayMs)
+    return requestFn()
+  }
+}

package.json

@@ -1,6 +1,6 @@
 {
   "name": "dcfrontend",
-  "version": "7.9.1",
+  "version": "7.11.0",
   "description": "Data Controller",
   "devDependencies": {
     "@saithodev/semantic-release-gitea": "^2.1.0",

sas/mocks/sasjs/services/editors/getdata.js

@@ -73,7 +73,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"PRIMARY_KEY_FIELD\",\"type\":\"numeric\",\"format\":\"0\"}"
             },
             {
                 NAME: "SOME_BESTNUM",
@@ -84,7 +85,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_BESTNUM\",\"type\":\"numeric\",\"format\":\"0\"}"
             },
             {
                 NAME: "SOME_CHAR",
@@ -95,7 +97,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_CHAR\"}"
             },
             {
                 NAME: "SOME_DATE",
@@ -106,7 +109,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_DATE\",\"type\":\"date\"}"
             },
             {
                 NAME: "SOME_DATETIME",
@@ -117,7 +121,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_DATETIME\",\"type\":\"datetime\"}"
             },
             {
                 NAME: "SOME_DROPDOWN",
@@ -128,7 +133,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_DROPDOWN\"}"
             },
             {
                 NAME: "SOME_HARDSELECT",
@@ -139,7 +145,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_HARDSELECT\"}"
             },
             {
                 NAME: "SOME_NUM",
@@ -150,7 +157,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_NUM\",\"type\":\"numeric\",\"format\":\"0\"}"
             },
             {
                 NAME: "SOME_SHORTNUM",
@@ -161,7 +169,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_SHORTNUM\",\"type\":\"numeric\",\"format\":\"0\"}"
             },
             {
                 NAME: "SOME_TIME",
@@ -172,7 +181,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SOME_TIME\",\"type\":\"time\"}"
             },
             {
                 NAME: "READONLY_COL",
@@ -183,7 +193,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Read-only: default value inserted on add-row, not editable",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"READONLY_COL\"}"
             },
             {
                 NAME: "HIDDEN_COL",
@@ -194,7 +205,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Hidden: invisible in grid but submitted; default on add-row",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"HIDDEN_COL\"}"
             },
             {
                 NAME: "ROUND_COL",
@@ -205,7 +217,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Round: edited values rounded Excel-style to 2 decimals",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"ROUND_COL\",\"type\":\"numeric\",\"format\":\"0\"}"
             },
             {
                 NAME: "NUMFMT_COL",
@@ -216,7 +229,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "Number format: displayed as EUR currency (value unchanged)",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"NUMFMT_COL\",\"type\":\"numeric\",\"format\":\"0\"}"
             }
         ],
         dqdata: [
@@ -323,7 +337,6 @@ let webouts = {
                 DTVARS: " SOME_DATE",
                 DTTMVARS: " SOME_DATETIME",
                 TMVARS: " SOME_TIME",
-                COLTYPE: "{\"data\":\"_____DELETE__THIS__RECORD_____\",\"type\":\"dropdown\",\"source\":[\"No\",\"Yes\"]},{\"data\":\"PRIMARY_KEY_FIELD\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_CHAR\"},{\"data\":\"SOME_DROPDOWN\"},{\"data\":\"SOME_HARDSELECT\"},{\"data\":\"SOME_NUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_DATE\",\"type\":\"date\"},{\"data\":\"SOME_DATETIME\",\"type\":\"datetime\"},{\"data\":\"SOME_TIME\",\"type\":\"time\"},{\"data\":\"SOME_SHORTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"SOME_BESTNUM\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"READONLY_COL\"},{\"data\":\"HIDDEN_COL\"},{\"data\":\"ROUND_COL\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"NUMFMT_COL\",\"type\":\"numeric\",\"format\":\"0\"}",
                 LOADTYPE: "UPDATE",
                 RK_FLAG: 0,
                 CLS_FLAG: 0
@@ -366,7 +379,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_LONGDESC\"}"
             },
             {
                 NAME: "DD_OWNER",
@@ -377,7 +391,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_OWNER\"}"
             },
             {
                 NAME: "DD_RESPONSIBLE",
@@ -388,7 +403,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_RESPONSIBLE\"}"
             },
             {
                 NAME: "DD_SENSITIVITY",
@@ -399,7 +415,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_SENSITIVITY\"}"
             },
             {
                 NAME: "DD_SHORTDESC",
@@ -410,7 +427,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_SHORTDESC\"}"
             },
             {
                 NAME: "DD_SOURCE",
@@ -421,7 +439,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_SOURCE\"}"
             },
             {
                 NAME: "DD_TYPE",
@@ -432,7 +451,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DD_TYPE\"}"
             },
             {
                 NAME: "TX_FROM",
@@ -593,7 +613,6 @@ let webouts = {
                 DTVARS: "",
                 DTTMVARS: "",
                 TMVARS: "",
-                COLTYPE: "{\"data\":\"_____DELETE__THIS__RECORD_____\",\"type\":\"dropdown\",\"source\":[\"No\",\"Yes\"]},{\"data\":\"DD_TYPE\"},{\"data\":\"DD_SOURCE\"},{\"data\":\"DD_SHORTDESC\"},{\"data\":\"DD_LONGDESC\"},{\"data\":\"DD_OWNER\"},{\"data\":\"DD_RESPONSIBLE\"},{\"data\":\"DD_SENSITIVITY\"}",
                 LOADTYPE: "TXTEMPORAL",
                 RK_FLAG: 0,
                 CLS_FLAG: 0
@@ -660,7 +679,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"USER_ID\"}"
             }
         ],
         dqdata: [],
@@ -722,7 +742,6 @@ let webouts = {
                 DTVARS: "",
                 DTTMVARS: "",
                 TMVARS: "",
-                COLTYPE: "{\"data\":\"_____DELETE__THIS__RECORD_____\",\"type\":\"dropdown\",\"source\":[\"No\",\"Yes\"]},{\"data\":\"USER_ID\"}",
                 LOADTYPE: "UPDATE",
                 RK_FLAG: 0,
                 CLS_FLAG: 0
@@ -765,7 +784,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"AUDIT_LIBDS\"}"
             },
             {
                 NAME: "BUSKEY",
@@ -776,7 +796,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"BUSKEY\"}"
             },
             {
                 NAME: "CLOSE_VARS",
@@ -787,7 +808,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"CLOSE_VARS\"}"
             },
             {
                 NAME: "DSN",
@@ -798,7 +820,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"DSN\"}"
             },
             {
                 NAME: "LIBREF",
@@ -809,7 +832,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"LIBREF\"}"
             },
             {
                 NAME: "LOADTYPE",
@@ -820,7 +844,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"LOADTYPE\"}"
             },
             {
                 NAME: "NOTES",
@@ -831,7 +856,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"NOTES\"}"
             },
             {
                 NAME: "NUM_OF_APPROVALS_REQUIRED",
@@ -842,7 +868,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"NUM_OF_APPROVALS_REQUIRED\",\"type\":\"numeric\",\"format\":\"0\"}"
             },
             {
                 NAME: "POST_APPROVE_HOOK",
@@ -853,7 +880,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"POST_APPROVE_HOOK\"}"
             },
             {
                 NAME: "POST_EDIT_HOOK",
@@ -864,7 +892,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"POST_EDIT_HOOK\"}"
             },
             {
                 NAME: "PRE_APPROVE_HOOK",
@@ -875,7 +904,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"PRE_APPROVE_HOOK\"}"
             },
             {
                 NAME: "PRE_EDIT_HOOK",
@@ -886,7 +916,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"PRE_EDIT_HOOK\"}"
             },
             {
                 NAME: "RK_UNDERLYING",
@@ -897,7 +928,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"RK_UNDERLYING\"}"
             },
             {
                 NAME: "SIGNOFF_COLS",
@@ -908,7 +940,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SIGNOFF_COLS\"}"
             },
             {
                 NAME: "SIGNOFF_HOOK",
@@ -919,7 +952,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"SIGNOFF_HOOK\"}"
             },
             {
                 NAME: "TX_FROM",
@@ -952,7 +986,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"VAR_BUSFROM\"}"
             },
             {
                 NAME: "VAR_BUSTO",
@@ -963,7 +998,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"VAR_BUSTO\"}"
             },
             {
                 NAME: "VAR_PROCESSED",
@@ -974,7 +1010,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"VAR_PROCESSED\"}"
             },
             {
                 NAME: "VAR_TXFROM",
@@ -985,7 +1022,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"VAR_TXFROM\"}"
             },
             {
                 NAME: "VAR_TXTO",
@@ -996,7 +1034,8 @@ let webouts = {
                 CLS_RULE: "READ",
                 MEMLABEL: "",
                 DESC: "",
-                LONGDESC: ""
+                LONGDESC: "",
+                COLTYPE: "{\"data\":\"VAR_TXTO\"}"
             }
         ],
         dqdata: [
@@ -1445,7 +1484,6 @@ let webouts = {
                 DTVARS: "",
                 DTTMVARS: "",
                 TMVARS: "",
-                COLTYPE: "{\"data\":\"_____DELETE__THIS__RECORD_____\",\"type\":\"dropdown\",\"source\":[\"No\",\"Yes\"]},{\"data\":\"LIBREF\"},{\"data\":\"DSN\"},{\"data\":\"NUM_OF_APPROVALS_REQUIRED\",\"type\":\"numeric\",\"format\":\"0\"},{\"data\":\"LOADTYPE\"},{\"data\":\"BUSKEY\"},{\"data\":\"VAR_TXFROM\"},{\"data\":\"VAR_TXTO\"},{\"data\":\"VAR_BUSFROM\"},{\"data\":\"VAR_BUSTO\"},{\"data\":\"VAR_PROCESSED\"},{\"data\":\"CLOSE_VARS\"},{\"data\":\"PRE_EDIT_HOOK\"},{\"data\":\"POST_EDIT_HOOK\"},{\"data\":\"PRE_APPROVE_HOOK\"},{\"data\":\"POST_APPROVE_HOOK\"},{\"data\":\"SIGNOFF_COLS\"},{\"data\":\"SIGNOFF_HOOK\"},{\"data\":\"NOTES\"},{\"data\":\"RK_UNDERLYING\"},{\"data\":\"AUDIT_LIBDS\"}",
                 LOADTYPE: "TXTEMPORAL",
                 RK_FLAG: 0,
                 CLS_FLAG: 0
@@ -1499,11 +1537,7 @@ function stripRuleCols(t) {
     })
     t.sasparams = t.sasparams.map((p) => {
         const headers = p.COLHEADERS.split(',').filter((h) => !uc.has(h))
-        const coltype = JSON.parse('[' + p.COLTYPE + ']')
-            .filter((o) => !uc.has(o.data))
-            .map((o) => JSON.stringify(o))
-            .join(',')
-        return { ...p, COLHEADERS: headers.join(','), COLTYPE: coltype }
+        return { ...p, COLHEADERS: headers.join(',') }
     })
     return t
 }

sas/package-lock.json

@@ -7,7 +7,7 @@
       "name": "dc-sas",
       "dependencies": {
         "@sasjs/cli": "4.18.0",
-        "@sasjs/core": "4.67.3"
+        "@sasjs/core": "4.68.1"
       }
     },
     "node_modules/@asamuzakjp/css-color": {
@@ -124,7 +124,6 @@
         }
       ],
       "license": "MIT",
-      "peer": true,
       "engines": {
         "node": ">=18"
       },
@@ -147,7 +146,6 @@
         }
       ],
       "license": "MIT",
-      "peer": true,
       "engines": {
         "node": ">=18"
       }
@@ -281,6 +279,12 @@
         "sasjs": "build/index.js"
       }
     },
+    "node_modules/@sasjs/cli/node_modules/@sasjs/core": {
+      "version": "4.67.3",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-4.67.3.tgz",
+      "integrity": "sha512-7s4UeVN4slF1URo1C5/UnNVv+TonUg+vBktFRiFhcbFYr3RnfXyClBPED6nQOF8AFd2xdXEss58hsCeIebr4lg==",
+      "license": "MIT"
+    },
     "node_modules/@sasjs/cli/node_modules/@sasjs/utils": {
       "version": "3.5.8",
       "resolved": "https://registry.npmjs.org/@sasjs/utils/-/utils-3.5.8.tgz",
@@ -317,9 +321,9 @@
       }
     },
     "node_modules/@sasjs/core": {
-      "version": "4.67.3",
-      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-4.67.3.tgz",
-      "integrity": "sha512-7s4UeVN4slF1URo1C5/UnNVv+TonUg+vBktFRiFhcbFYr3RnfXyClBPED6nQOF8AFd2xdXEss58hsCeIebr4lg==",
+      "version": "4.68.1",
+      "resolved": "https://registry.npmjs.org/@sasjs/core/-/core-4.68.1.tgz",
+      "integrity": "sha512-BjoOzV7h7guCShW4JsFHvB1NNLCKHhPI5708OtEk29itQbgVCMRlEXKFRbnMiuNklktDQDi8dR4XZZ0aK7/4Wg==",
       "license": "MIT"
     },
     "node_modules/@sasjs/lint": {
@@ -466,7 +470,6 @@
       "resolved": "https://registry.npmjs.org/axios/-/axios-1.16.0.tgz",
       "integrity": "sha512-6hp5CwvTPlN2A31g5dxnwAX0orzM7pmCRDLnZSX772mv8WDqICwFjowHuPs04Mc8deIld1+ejhtaMn5vp6b+1w==",
       "license": "MIT",
-      "peer": true,
       "dependencies": {
         "follow-redirects": "^1.16.0",
         "form-data": "^4.0.5",
@@ -2156,7 +2159,6 @@
       "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-4.1.3.tgz",
       "integrity": "sha512-aX/y5pVRkfRnfmuX+OdbSdXvPe6ieKX/G2s7e98f4poJHnqH3281gDPm/metm6E/WRamfx7WC4HUqkWHfQHprw==",
       "license": "BSD-3-Clause",
-      "peer": true,
       "dependencies": {
         "psl": "^1.1.33",
         "punycode": "^2.1.1",

sas/package.json

@@ -29,6 +29,6 @@
   "private": true,
   "dependencies": {
     "@sasjs/cli": "4.18.0",
-    "@sasjs/core": "4.67.3"
+    "@sasjs/core": "4.68.1"
   }
 }

sas/sasjs/db/migrations/20260710_v7.10_release.sas

@@ -0,0 +1,21 @@
+/**
+  @file
+  @brief migration script 
+
+**/
+
+%let dclib=YOURDCLIB;
+
+libname &dclib "/YOUR/DATACONTROLLER/LIBRARY/PATH";
+
+/* add new validation rules */
+proc sql noprint;
+insert into &dclib..mpe_config set
+    tx_from=%sysfunc(datetime())
+    ,tx_to='31DEC9999:23:59:59'dt
+    ,var_scope="DC"
+    ,var_name="DC_MAXOBS_WEBVIEW"
+    ,var_value='500'
+    ,var_active=1
+    ,var_desc='Default rows displayed for tables in VIEW mode.';
+

sas/sasjs/macros/bitemporal_closeouts.sas

@@ -35,6 +35,7 @@
   @li mf_getuniquename.sas
   @li mf_getuser.sas
   @li mf_getvartype.sas
+  @li mf_mkdir.sas
   @li mp_lockanytable.sas
   @li dc_assignlib.sas
 
@@ -136,6 +137,7 @@ run;
   libname &tmplib (work);
 %end;
 %else %do;
+  %mf_mkdir(&AUDITFOLDER)
   filename tmp "&AUDITFOLDER/deleterecords.sas" lrecl=10000;
   libname &tmplib "&AUDITFOLDER";
 %end;

sas/sasjs/macros/bitemporal_dataloader.sas

@@ -111,6 +111,7 @@ Areas for optimisation
   @li mp_lockfilecheck.sas
   @li mp_retainedkey.sas
   @li mp_storediffs.sas
+  @li mp_rowhash.sas
 
   @version 9.3
   @author 4GL Apps Ltd.
@@ -193,7 +194,7 @@ Areas for optimisation
 
 /* when changing this macro, update the version num here */
 %local ver;
-%let ver=32;
+%let ver=33;
 %put &sysmacroname entry vars:;
 %put _local_;
 
@@ -246,7 +247,7 @@ run;
 )
 
 
-%local cols idx_pk md5_col ;
+%local cols idx_pk md5_col hash_char_vars hash_num_vars ;
 %let md5_col=___TMP___md5;
 %let check_uniqueness=%upcase(&check_uniqueness);
 %let RK_UPDATE_MAXKEYTABLE=%upcase(&RK_UPDATE_MAXKEYTABLE);
@@ -321,14 +322,20 @@ select name into: cols separated by ','
     (%upcase("&bus_from","&bus_to"
             ,"&tech_from","&tech_to"
             ,"&processed","&delete_col")) ;
-select case when type in (2,6) then cats('put(md5(trim(',name,')),$hex32.)')
-        /* multiply by 1 to strip precision errors (eg 0 != 0) */
-        /* but ONLY if not missing, else will lose any special missing values */
-        else cats('put(md5(trim(put(ifn(missing('
-          ,name,'),',name,',',name,'*1),binary64.))),$hex32.)') end
-        into: stripcols separated by '||'
+/* Character variables are hashed separately from numerics so that the
+   iterative hash can be built with arrays rather than by concatenating
+   hundreds of 32-byte hex strings. */
+select name into: hash_char_vars separated by ' '
   from work.bitemp_cols
-  where upcase(name) not in
+  where type in (2,6)
+    and upcase(name) not in
+    (%upcase("&bus_from","&bus_to"
+            ,"&tech_from","&tech_to"
+            ,"&processed","&delete_col")) ;
+select name into: hash_num_vars separated by ' '
+  from work.bitemp_cols
+  where type not in (2,6)
+    and upcase(name) not in
     (%upcase("&bus_from","&bus_to"
             ,"&tech_from","&tech_to"
             ,"&processed","&delete_col")) ;
@@ -342,7 +349,8 @@ select case when type in (2,6) then cats('put(md5(trim(',name,')),$hex32.)')
 %let tech_to_fmt = format=datetime19.;
 
 
-%put &=stripcols;
+%put &=hash_char_vars;
+%put &=hash_num_vars;
 %put &=pk;
 
 data _null_;
@@ -368,8 +376,8 @@ data _null_;
     call symputx('processed_fmt',fmt,'L');
 run;
 
-%if %index(%quote(&cols),___TMP___) %then %do;
-  %let msg=%str(Table contains a variable name containing "___TMP___".%trim(
+%if %index(%quote(&cols),___TMP___) or %index(%quote(&cols),_____) %then %do;
+  %let msg=%str(Table contains a variable name containing "___TMP___" or "_____".%trim(
     ) This may conflict with temp variable generation!!);
   %mp_abort(msg=&msg,mac=bitemporal_dataloader);
   %let syscc=5;
@@ -485,7 +493,10 @@ data work.bitemp0_append &keepvars &outds_del(drop=&md5_col )
     &bus_to= %unquote(&bus_to_override) ;
   %end;
   length &md5_col $32;
-  &md5_col=put(md5(&stripcols),hex32.);
+  %mp_rowhash(md5_col=&md5_col
+    ,cvars=&hash_char_vars
+    ,nvars=&hash_num_vars
+  )
   %if %length(&processed)>0 %then %do;
     format &processed &processed_fmt;
     &processed=&now;
@@ -772,13 +783,17 @@ or &engine_type=SNOW or &engine_type=SASIOSNF
   ,msg=%str(syscc=&syscc sqlrc=&sqlrc)
 )
 
-%put hashcols2=&stripcols;
-proc sql;
-create table work.bitemp1_current(drop=___TMP___NEW_FLG) as
-  select *
-    , put(md5(&stripcols),$hex32.) as &md5_col
-  from work.bitemp0_base (drop=&md5_col)
-  where ___TMP___NEW_FLG=0;
+%put hash_char_vars=&hash_char_vars;
+%put hash_num_vars=&hash_num_vars;
+data work.bitemp1_current(drop=___TMP___NEW_FLG);
+  set work.bitemp0_base(drop=&md5_col);
+  if ___TMP___NEW_FLG=0;
+  length &md5_col $32;
+  %mp_rowhash(md5_col=&md5_col
+    ,cvars=&hash_char_vars
+    ,nvars=&hash_num_vars
+  )
+run;
 
 /**
 * NEW records were identified in ___TMP___NEW_FLG in bitemp0_base
@@ -1061,7 +1076,9 @@ run;
 */
 
 %if &loadtype=BITEMPORAL %then %do;
-  %let cat_string=catx('|' ,&bus_from,&bus_to);
+  /* For bitemporal we also include the business dates in the comparison hash.
+     The business dates are passed as prefix numerics so that both lookup and
+     update tables use exactly the same hashing order. */
 
   data work.bitemp5a_lkp (keep=&md5_col)
     %if "%substr(&sysver,1,1)" ne "4" & "%substr(&sysver,1,1)" ne "5" %then %do;
@@ -1069,8 +1086,10 @@ run;
     %end;
     ;
     set work.bitemp0_base;
-    /* for BITEMPORAL we need to compare business dates also */
-    &md5_col=put(md5(&cat_string!!'|'!!&stripcols),$hex32.);
+    %mp_rowhash(md5_col=&md5_col
+      ,cvars=&hash_char_vars
+      ,nvars=&bus_from &bus_to &hash_num_vars
+    )
   run;
 
   data bitemp5b_updates;
@@ -1081,7 +1100,10 @@ run;
       md5_lkp.definedone();
     end;
     /* drop old md5 col as will rebuild with new business dates */
-    &md5_col=put(md5(&cat_string!!'|'!!&stripcols),$hex32.) ;
+    %mp_rowhash(md5_col=&md5_col
+      ,cvars=&hash_char_vars
+      ,nvars=&bus_from &bus_to &hash_num_vars
+    )
     if md5_lkp.check()=0 then delete;
   run;
 

sas/sasjs/macros/bitemporal_dataloader.test.2.sas

@@ -1035,7 +1035,7 @@ run;
     set appendtable&testnum;
     /* delete even numbered observations */
     if mod(_n_,2)=0 and _n_ ne 2 then _____DELETE__THIS__RECORD_____='YES';
-    else _____DELETE__THIS__RECORD_____=0;
+    else _____DELETE__THIS__RECORD_____='NO';
   run;
 
   %bitemporal_dataloader(dclib=work

sas/sasjs/macros/mpe_checkrestore.sas

@@ -54,7 +54,7 @@
     */
   %local audtab;
   proc sql noprint;
-  select coalescec(audit_libds,"&dc_libref..MPE_AUDIT") into: audtab
+  select coalescec(audit_libds,"&dc_libref..MPE_AUDIT") into: audtab trimmed
   from &dc_libref..MPE_TABLES
   where &dc_dttmtfmt. lt tx_to
     and libref="%scan(&libds,1,.)" and dsn="%scan(&libds,2,.)";

sas/sasjs/macros/mpe_loader.sas

@@ -16,7 +16,6 @@
   @li dc_assignlib.sas
   @li mf_getattrn.sas
   @li mf_getuser.sas
-  @li mf_mkdir.sas
   @li mf_verifymacvars.sas
   @li mp_abort.sas
   @li mp_cntlout.sas

sas/sasjs/macros/mpe_refreshtables.test.sas

@@ -37,12 +37,12 @@ create table work.libinfo as
   from &mpelib..mpe_datacatalog_libs(where=(&dc_dttmtfmt. lt tx_to)) a
   left join &mpelib..mpe_datastatus_libs(where=(&dc_dttmtfmt. lt tx_to)) b
   on a.libref=b.libref
-  where a.libref="&libref";
+  where a.libref="FMTONLY";
 
 %let test1=0;
 data _null_;
   set work.libinfo;
-  call symputx('test1',table_cnt);
+  call symputx('test1',catalog_cnt);
 run;
 %mp_assert(
   iftrue=(&test1>0),

sas/sasjs/services/editors/getdata.sas

@@ -626,6 +626,7 @@ create table dqdata as
     %put &=source;
     %put &=lib;
     %dc_assignlib(READ,&lib)
+    %dc_casload(&lib..&ds)
     proc sql;
     create table dqdata&x as
       select distinct "&&base_col&x" as base_col length=32

sas/sasjs/services/editors/getdynamiccolvals.test.2.sas

@@ -57,7 +57,7 @@ data work.check;
   val='VAR_BUSFROM';output;
   val='VAR_BUSTO';output;
 run;
-%mp_assertcolvals(work.DYNAMIC_EXTENDED_VALUES.element2,
+%mp_assertcolvals(work.DYNAMIC_EXTENDED_VALUES.EXTRA_COL_NAME,
   checkvals=work.check.val,
   desc=Correct values found in EXTRA_COL_NAME response,
   test=ALLVALS

sas/sasjs/services/editors/restore.sas

@@ -7,6 +7,7 @@
   @li dc_assignlib.sas
   @li mcf_getfmttype.sas
   @li mf_nobs.sas
+  @li mf_mkdir.sas
   @li mp_abort.sas
   @li mp_applyformats.sas
   @li mp_ds2csv.sas

sas/sasjs/services/public/getrawdata.test.sas

@@ -30,11 +30,12 @@ run;
 data work.results;
   infile webout;
   input;
-  if _infile_=:'PRIMARY_KEY_FIELD,SOME_CHAR' then do;
+  if _infile_=:'PRIMARY_KEY_FIELD, SOME_CHAR' then do;
     output;
     output;
     stop;
   end;
+  else if _n_<5 then putlog _infile_;
 run;
 
 %mp_assertdsobs(work.results,

sas/sasjs/services/public/viewdata.sas

@@ -71,8 +71,6 @@
 /* configure macvars */
 %global LIBDS FILTER_RK SEARCHVAL SEARCHTYPE FMT_IND;
 
-%let maxrows=250;
-
 /* avoid code injection */
 %let FMT_IND=0;
 %let SEARCHTYPE=;
@@ -173,6 +171,13 @@ run;
 %global dsobs;
 %let dsobs=0;
 %macro x();
+%if not %symexist(DC_MAXOBS_WEBVIEW) %then %do;
+  %put NOTE:;%put NOTE- DC_MAXOBS_WEBVIEW not found!;
+  %put NOTE- Please add to &mpelib..MPE_CONFIG table;
+  %put NOTE-;%put NOTE-;
+  %global DC_MAXOBS_WEBVIEW;
+  %let DC_MAXOBS_WEBVIEW=500;
+%end;
 %if &existds>0 %then %do;
 
   %if &fmt_ind=1 %then %do;
@@ -234,7 +239,7 @@ run;
     data work.viewdata;
       set &libds;
       where %inc filtref;;
-      if _n_>&maxrows then stop;
+      if _n_>&DC_MAXOBS_WEBVIEW then stop;
     run;
     %if %mf_nobs(work.viewdata)=0 %then %do;
       data work.viewdata;
@@ -253,21 +258,21 @@ run;
       %mp_searchdata(lib=work
         ,ds=vwsearch
         ,string=%superq(searchval)
-        ,outobs=&maxrows
+        ,outobs=&DC_MAXOBS_WEBVIEW
       )
     %end;
     %else %if %upcase(&searchtype)=NUM %then %do;
       %mp_searchdata(lib=work
         ,ds=vwsearch
         ,numval=%superq(searchval)
-        ,outobs=&maxrows
+        ,outobs=&DC_MAXOBS_WEBVIEW
       )
     %end;
     %if %mf_existds(libds=MPSEARCH.vwsearch) %then %do;
       %let dsobs=%mf_nobs(MPSEARCH.vwsearch);
       data viewdata;
         set MPSEARCH.vwsearch;
-        if _n_<&maxrows;
+        if _n_<&DC_MAXOBS_WEBVIEW;
       run;
     %end;
     %else %do;
@@ -347,7 +352,7 @@ data work.sasparams;
   PK_FIELDS=symget('PK_FIELDS');
   nobs=&dsobs;
   vars=%mf_getvarcount(viewdata);
-  maxrows=&maxrows;
+  maxrows=&DC_MAXOBS_WEBVIEW;
 run;
 
 %mp_abort(iftrue= (&syscc ne 0)

sas/sasjs/targets/viya/macros_viya/dc_casload.sas

@@ -66,7 +66,7 @@
   %end;
 
   proc casutil;
-    load casdata="&ds"
+    load casdata="&ds..sashdat"
         incaslib="&caslib"
         casout="&ds"
         outcaslib="&caslib"

sas/sasjs/tests/testsetup.sas

@@ -64,6 +64,38 @@ data dctest.mpe_x_test;
   set &DC_LIBREF..mpe_x_test;
 run;
 
+data dctest.wideboy;
+  /* Create 500 character variables and 500 numeric variables */
+  array char_fields[500] $20 char1-char500;
+  array num_fields[500]       num1-num500;
+  length alphabet $62;
+  alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
+
+  /* Set a seed for reproducible random data */
+  call streaminit(12345);
+
+  do row_id = 1 to 1000;
+
+    /* Generate random 20-character alphanumeric values */
+    do field = 1 to dim(char_fields);
+      do position = 1 to 20;
+        substr(char_fields[field], position, 1) =
+          substr(alphabet,
+            floor(rand('uniform') * length(alphabet)) + 1,
+            1
+        );
+      end;
+    end;
+    /* Generate random numeric values from a normal distribution */
+    do field = 1 to dim(num_fields);
+      num_fields[field] = ceil(rand('normal', 0, 1));
+    end;
+    output;
+  end;
+  stop;
+run;
+
+
 /* test library with only one format catalog */
 %mf_mkdir(&testloc/fmtonly)
 libname fmtonly "&testloc/fmtonly";
@@ -90,8 +122,7 @@ proc format library=dctest.dcfmts;
   invalue indays (default=13) other=42;
 run;
 proc sql;
-delete from &DC_LIBREF..mpe_tables
-  where libref="DCTEST" and dsn='DCFMTS-FC';
+delete from &DC_LIBREF..mpe_tables where libref="DCTEST";
 data work.append;
   if 0 then set &DC_LIBREF..mpe_tables;
   tx_from=0;
@@ -102,6 +133,11 @@ data work.append;
   loadtype='FORMAT_CAT';
   num_of_approvals_required=1;
   output;
+  dsn='WIDEBOY';
+  buskey='ROW_ID CHAR1 NUM1';
+  loadtype='UPDATE';
+  output;
+  stop;
 run;
 proc append base=&dc_libref..mpe_tables data=work.append;
 run;