dc
761734016d
client/.npmrc is tracked now, so a token left in it from the old pattern is one `git add -A` from being committed. Prose in CONTRIBUTING is not a control: the pre-commit hook fails when the staged file holds _authToken, _auth or _password. `npm run check:archiver` runs the directory-walk guard, so it is reproducible before a push instead of only in CI, and the CI step calls the same script. CONTRIBUTING now spells out the per-project installs, since the root .npmrc has ignore-scripts=true and the composite install script therefore never runs.
11 KiB
Data Controller
Contributing
Workflow guidelines
Git hooks
The repo ships its own git hooks in .git-hooks/:
pre-commit- scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)commit-msg- verifies the commit message follows the Conventional Commits standard
The repo .npmrc sets ignore-scripts=true, so the prepare and postinstall lifecycle scripts that would normally set core.hooksPath during npm i never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
git config core.hooksPath ./.git-hooks
The pre-commit hook requires the gitleaks binary provided by the root @nogoo9/gitleaks devDependency, so make sure npm i has run in the repo root before your first commit.
The standard pre-commit route is available as well, and does not depend on the lifecycle scripts above: install it once with pre-commit install and the secret scan in .pre-commit-config.yaml runs on every commit. The same scan also runs in CI, so a leaked credential fails the build regardless of which local hooks are active.
Dependencies that requires licences
To auth for SheetJS Pro version you need to use their private registry
Add client/.npmrc file with following content:
@sheet:registry=https://pylon.sheetjs.com:54111/
//pylon.sheetjs.com:54111/:_authToken="TOKEN-GOES-HERE"
In addition to the registry auth, the @sheet/crypto package is installed from a local tarball
(client/libraries/sheet-crypto.tgz) rather than the registry. This tarball is not committed to
the repo directly - only an encrypted copy is: client/libraries/sheet-crypto.tgz.gpg.
Before running npm i/npm ci in client, you must decrypt it using the passphrase
(stored as the SHEET_PWD CI secret - ask a repo maintainer if you don't have it):
cd client
echo "YOUR_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output ./libraries/sheet-crypto.tgz \
--decrypt ./libraries/sheet-crypto.tgz.gpg
npm i
npm run postinstall
The root and sas/ .npmrc files set ignore-scripts=true too, so the composite install
script in the root package.json (which installs both subprojects) does not run on npm i at the
root. Install each part explicitly:
npm ci # root tooling: gitleaks, prettier, semantic-release
cd client && npm ci && npm run postinstall
cd ../sas && npm ci
The client .npmrc is now tracked and sets ignore-scripts=true, so dependency install hooks do not
run. Two of them matter here: the client's own postinstall - which generates
src/environments/version.ts, imported throughout the app - and cypress's binary download. Run
npm run postinstall after installing (and npx cypress install if you are running the Cypress
suite).
Registry credentials do not belong in client/.npmrc: it is a tracked policy file, and a token
written there is one git add -A away from being committed. Keep the token in ~/.npmrc, npm's
default user config, or point npm at a file outside the repo with
NPM_CONFIG_USERCONFIG=/path/to/npmrc. The pre-commit hook rejects a staged client/.npmrc that
holds _authToken, _auth or _password.
The licence key comes from the backend licence: the decrypted licence carries hot_license_key,
which is published on the hot_license_key subject and pushed into every Handsontable instance.
Until a licence is loaded the client falls back to the non-commercial key
(DEFAULT_HOT_LICENCE_KEY in client/src/app/services/licence.service.ts), so there is nothing to
configure in index.html for this.
Development
Update client/src/index.html so that it points to your SAS9, SASVIYA or SASJS backend.
Be aware that VIYA can be configured in such way that it would not work with cross origin frontend.
Follow this guide to disable CORS: https://sasjs.io/cors/ (NOTICE: Sometimes even this approach would fail to work, in such case it is imposible to set it up without reconfiguring the VIYA server)
Start dev server:
cd client
npm start
Mocked backend (frontend development without SAS)
Data Controller ships a full set of JS mock services under sas/mocks/sasjs/ that mimic the SAS backend using SASjs Server in JS-only mode. This lets you run the entire frontend (editor grid, approvals, viewer, admin, Excel maps, etc.) without a SAS licence or SAS server. The Cypress E2E tests and Lighthouse CI both run against this setup.
Prerequisites
- Node.js (same version used for the frontend build)
- The frontend dependencies installed (
cd client && npm ciafter decrypting sheet-crypto as described above)
1. Start SASjs Server
Download the latest Linux (or macOS/Windows) binary from SASjs Server releases, unzip it, and create a .env file alongside the executable:
RUN_TIMES=js
NODE_PATH=node
CORS=enable
WHITELIST=http://localhost:4200
HTTP_PORT=5000
RUN_TIMES=js means the server executes .js files as Stored Programs (no SAS binary needed). CORS=enable plus WHITELIST allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
./api-linux # or api-macos / api-win.exe depending on your platform
HTTP_PORT is not optional in SASjs Server 3.x: PORT and PROTOCOL are deprecated, and a server started without HTTP_PORT or HTTPS_PORT exits with "no listener is enabled". With it set, the server runs in desktop mode (no auth) on http://localhost:5000.
2. Deploy the mock services and streaming frontend
From sas/mocks/sasjs/:
npm ci
sasjs cbd -t server-ci
This compiles, builds, and deploys the JS mock services plus the Angular frontend (as a streaming app) to the SASjs Server. The server-ci target in sas/mocks/sasjs/sasjsconfig.json points to http://localhost:5000 with appLoc /Public/app/dc and stream service name clickme.
After deploy, the streamed app is available at: http://localhost:5000/AppStream/clickme/
3. Run the Angular dev server (optional, for hot reload)
For frontend development with live reload, point client/src/index.html at the local SASjs Server and start the dev server:
cd client
# Edit src/index.html: set serverUrl="http://localhost:5000", appLoc="/Public/app/dc", serverType="SASJS"
npm start
The app will be at http://localhost:4200. It calls the JS mock services on port 5000 via the SASjs adapter (CORS is enabled by the .env above).
How the mocks work
Each .js file in sas/mocks/sasjs/services/ corresponds to a real SAS service (e.g. public/startupservice.js mimics services/public/startupservice.sas). The JS runtime predeclares fs and _webout; the mock assigns a JSON string to _webout that matches the structure the real SAS service would return. Some mocks are stateful: they persist data to a mock-storage/ folder on the SASjs Drive (e.g. licence.json, users.json, filter.txt) using the predeclared fs module.
The mock data includes sample tables (DC996664.MPE_X_TEST, MPE_X_NEW, MPE_X_FORMULA_TEST, etc.) with realistic columns, validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT), and dropdown values. This is sufficient to exercise the full editor grid, approval workflow, viewer, and admin screens.
There is also a SAS 9 mock variant under sas/mocks/sas9/ (target sas9-mocks) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (/User Folders/sasdemo/).
Re-deploying after changes
After changing mock services, re-run sasjs cbd -t server-ci from sas/mocks/sasjs/. After changing the frontend, rebuild (cd client && npm run build) then re-deploy so the streaming app picks up the new client/dist/. If you are using ng serve (step 3), only the mock services need re-deploying; the frontend hot-reloads automatically.
Demo and staging sites
A push to release/** publishes staging.datacontroller.io; a release tag (v*, created by the release job after its tests pass) publishes demo.datacontroller.io - workflow file .gitea/workflows/sites.yaml, using the DEMO_DATACONTROLLER_IO Surfer app password (user allan). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, streamConfig.mockWeb), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs makedata on a throwaway SASjs Server first and ships that drive as sasjsbuild/mock-snapshot.json; without it the site opens with no data. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (admin/registerkey) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in sas/mocks/mock-licence.json: the two values are one RSA key and are regenerated together, and the .gitleaksignore entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
GUI Elements
For documentation on the Clarity Design System, including a list of components and example usage, see our website.
Code style
Run prettier fix:
npm run lint:fix
Generate docs
Typedoc is used for generating typescript documentation based on the code. That part is automated and beign done as a part of CI job.
Release
Release is automated as a part of CI job. Workflow file: .gitea/workflows/release.yaml.
It will run automatically when branch merged to the main branch.
IMPORTANT!
If release job fails, after it has been created empty release and a tag, we must not re-run the relase job until we removed the newly create GIT TAG and RELEASE.
To remove the git tag run:
git push -d origin vX.X.X
To remove the release, you need to do it with repo administration over at https://git.datacontroller.io/dc/dc
Troubleshooting
npm i fails with ENOENT on sheet-crypto.tgz
If npm i/npm ci in client fails with:
npm error enoent ENOENT: no such file or directory, open '.../client/libraries/sheet-crypto.tgz'
it means the encrypted sheet-crypto.tgz.gpg hasn't been decrypted yet. See
Dependencies that requires licences above for how to decrypt it.
Makedata service "could not create directory" error
The dcpath folder should have its permissions set so that the system account (SYSUSERID) can both read and write to it.
Example:
If dcpath is: '/tmp/dc'
Run:
chmod 777 /tmp/dc