dc
761734016d
client/.npmrc is tracked now, so a token left in it from the old pattern is one `git add -A` from being committed. Prose in CONTRIBUTING is not a control: the pre-commit hook fails when the staged file holds _authToken, _auth or _password. `npm run check:archiver` runs the directory-walk guard, so it is reproducible before a push instead of only in CI, and the CI step calls the same script. CONTRIBUTING now spells out the per-project installs, since the root .npmrc has ignore-scripts=true and the composite install script therefore never runs.
211 lines
11 KiB
Markdown
211 lines
11 KiB
Markdown
# Data Controller
|
|
# Contributing
|
|
|
|
## Workflow guidelines
|
|
[Wiki Page](https://git.datacontroller.io/dc/dc/wiki/Git-Workflow)
|
|
|
|
## Git hooks
|
|
|
|
The repo ships its own git hooks in [`.git-hooks/`](./.git-hooks):
|
|
|
|
- `pre-commit` - scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)
|
|
- `commit-msg` - verifies the commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) standard
|
|
|
|
The repo `.npmrc` sets `ignore-scripts=true`, so the `prepare` and `postinstall` lifecycle scripts that would normally set `core.hooksPath` during `npm i` never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
|
|
|
|
```bash
|
|
git config core.hooksPath ./.git-hooks
|
|
```
|
|
|
|
The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
|
|
|
|
The standard [pre-commit](https://pre-commit.com) route is available as well, and does not depend on the lifecycle scripts above: install it once with `pre-commit install` and the secret scan in [`.pre-commit-config.yaml`](./.pre-commit-config.yaml) runs on every commit. The same scan also runs in CI, so a leaked credential fails the build regardless of which local hooks are active.
|
|
|
|
## Dependencies that requires licences
|
|
|
|
[SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)
|
|
|
|
|
|
To auth for SheetJS Pro version you need to use their private registry
|
|
Add `client/.npmrc` file with following content:
|
|
```
|
|
@sheet:registry=https://pylon.sheetjs.com:54111/
|
|
//pylon.sheetjs.com:54111/:_authToken="TOKEN-GOES-HERE"
|
|
```
|
|
|
|
In addition to the registry auth, the `@sheet/crypto` package is installed from a local tarball
|
|
(`client/libraries/sheet-crypto.tgz`) rather than the registry. This tarball is not committed to
|
|
the repo directly - only an encrypted copy is: `client/libraries/sheet-crypto.tgz.gpg`.
|
|
|
|
Before running `npm i`/`npm ci` in `client`, you must decrypt it using the passphrase
|
|
(stored as the `SHEET_PWD` CI secret - ask a repo maintainer if you don't have it):
|
|
```bash
|
|
cd client
|
|
echo "YOUR_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
|
|
--output ./libraries/sheet-crypto.tgz \
|
|
--decrypt ./libraries/sheet-crypto.tgz.gpg
|
|
|
|
npm i
|
|
npm run postinstall
|
|
```
|
|
|
|
The root and `sas/` `.npmrc` files set `ignore-scripts=true` too, so the composite `install`
|
|
script in the root `package.json` (which installs both subprojects) does not run on `npm i` at the
|
|
root. Install each part explicitly:
|
|
|
|
```bash
|
|
npm ci # root tooling: gitleaks, prettier, semantic-release
|
|
cd client && npm ci && npm run postinstall
|
|
cd ../sas && npm ci
|
|
```
|
|
|
|
The client `.npmrc` is now tracked and sets `ignore-scripts=true`, so dependency install hooks do not
|
|
run. Two of them matter here: the client's own `postinstall` - which generates
|
|
`src/environments/version.ts`, imported throughout the app - and cypress's binary download. Run
|
|
`npm run postinstall` after installing (and `npx cypress install` if you are running the Cypress
|
|
suite).
|
|
|
|
Registry credentials do **not** belong in `client/.npmrc`: it is a tracked policy file, and a token
|
|
written there is one `git add -A` away from being committed. Keep the token in `~/.npmrc`, npm's
|
|
default user config, or point npm at a file outside the repo with
|
|
`NPM_CONFIG_USERCONFIG=/path/to/npmrc`. The pre-commit hook rejects a staged `client/.npmrc` that
|
|
holds `_authToken`, `_auth` or `_password`.
|
|
|
|
[Handsontable](https://www.npmjs.com/package/handsontable)
|
|
|
|
The licence key comes from the backend licence: the decrypted licence carries `hot_license_key`,
|
|
which is published on the `hot_license_key` subject and pushed into every Handsontable instance.
|
|
Until a licence is loaded the client falls back to the non-commercial key
|
|
(`DEFAULT_HOT_LICENCE_KEY` in `client/src/app/services/licence.service.ts`), so there is nothing to
|
|
configure in `index.html` for this.
|
|
|
|
## Development
|
|
Update `client/src/index.html` so that it points to your SAS9, SASVIYA or SASJS backend.
|
|
Be aware that VIYA can be configured in such way that it would not work with cross origin frontend.
|
|
Follow this guide to disable CORS: https://sasjs.io/cors/ (NOTICE: Sometimes even this approach would fail to work, in such case it is imposible to set it up without reconfiguring the VIYA server)
|
|
|
|
Start dev server:
|
|
```
|
|
cd client
|
|
|
|
npm start
|
|
```
|
|
|
|
## Mocked backend (frontend development without SAS)
|
|
|
|
Data Controller ships a full set of JS mock services under `sas/mocks/sasjs/` that mimic the SAS backend using [SASjs Server](https://server.sasjs.io) in JS-only mode. This lets you run the entire frontend (editor grid, approvals, viewer, admin, Excel maps, etc.) without a SAS licence or SAS server. The Cypress E2E tests and Lighthouse CI both run against this setup.
|
|
|
|
### Prerequisites
|
|
|
|
- Node.js (same version used for the frontend build)
|
|
- The frontend dependencies installed (`cd client && npm ci` after decrypting sheet-crypto as described above)
|
|
|
|
### 1. Start SASjs Server
|
|
|
|
Download the latest Linux (or macOS/Windows) binary from [SASjs Server releases](https://github.com/sasjs/server/releases), unzip it, and create a `.env` file alongside the executable:
|
|
|
|
```
|
|
RUN_TIMES=js
|
|
NODE_PATH=node
|
|
CORS=enable
|
|
WHITELIST=http://localhost:4200
|
|
HTTP_PORT=5000
|
|
```
|
|
|
|
`RUN_TIMES=js` means the server executes `.js` files as Stored Programs (no SAS binary needed). `CORS=enable` plus `WHITELIST` allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
|
|
|
|
```
|
|
./api-linux # or api-macos / api-win.exe depending on your platform
|
|
```
|
|
|
|
`HTTP_PORT` is not optional in SASjs Server 3.x: `PORT` and `PROTOCOL` are deprecated, and a server started without `HTTP_PORT` or `HTTPS_PORT` exits with "no listener is enabled". With it set, the server runs in desktop mode (no auth) on `http://localhost:5000`.
|
|
|
|
### 2. Deploy the mock services and streaming frontend
|
|
|
|
From `sas/mocks/sasjs/`:
|
|
|
|
```
|
|
npm ci
|
|
sasjs cbd -t server-ci
|
|
```
|
|
|
|
This compiles, builds, and deploys the JS mock services plus the Angular frontend (as a streaming app) to the SASjs Server. The `server-ci` target in `sas/mocks/sasjs/sasjsconfig.json` points to `http://localhost:5000` with appLoc `/Public/app/dc` and stream service name `clickme`.
|
|
|
|
After deploy, the streamed app is available at: `http://localhost:5000/AppStream/clickme/`
|
|
|
|
### 3. Run the Angular dev server (optional, for hot reload)
|
|
|
|
For frontend development with live reload, point `client/src/index.html` at the local SASjs Server and start the dev server:
|
|
|
|
```
|
|
cd client
|
|
# Edit src/index.html: set serverUrl="http://localhost:5000", appLoc="/Public/app/dc", serverType="SASJS"
|
|
npm start
|
|
```
|
|
|
|
The app will be at `http://localhost:4200`. It calls the JS mock services on port 5000 via the SASjs adapter (CORS is enabled by the `.env` above).
|
|
|
|
### How the mocks work
|
|
|
|
Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). The JS runtime predeclares `fs` and `_webout`; the mock assigns a JSON string to `_webout` that matches the structure the real SAS service would return. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
|
|
|
|
The mock data includes sample tables (`DC996664.MPE_X_TEST`, `MPE_X_NEW`, `MPE_X_FORMULA_TEST`, etc.) with realistic columns, validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT), and dropdown values. This is sufficient to exercise the full editor grid, approval workflow, viewer, and admin screens.
|
|
|
|
There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`).
|
|
|
|
### Re-deploying after changes
|
|
|
|
After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sasjs/`. After changing the frontend, rebuild (`cd client && npm run build`) then re-deploy so the streaming app picks up the new `client/dist/`. If you are using `ng serve` (step 3), only the mock services need re-deploying; the frontend hot-reloads automatically.
|
|
|
|
### Demo and staging sites
|
|
|
|
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a release tag (`v*`, created by the release job after its tests pass) publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs `makedata` on a throwaway SASjs Server first and ships that drive as `sasjsbuild/mock-snapshot.json`; without it the site opens with no data. The estate carries no licence key, so CI runs on the free tier until a key is registered through the licensing screen (`admin/registerkey`) - the Cypress setup does that for the specs that need one. The Sites job writes a licence into the published drive, so demo and staging are fully usable. The pair lives in `sas/mocks/mock-licence.json`: the two values are one RSA key and are regenerated together, and the `.gitleaksignore` entries for it are line-scoped, so inserting a line above the pair re-fires the scanner rather than revealing a new secret.
|
|
|
|
## GUI Elements
|
|
|
|
For documentation on the Clarity Design System, including a list of components and example usage, see [our website](https://vmware.github.io/clarity).
|
|
|
|
## Code style
|
|
Run prettier fix:
|
|
```bash
|
|
npm run lint:fix
|
|
```
|
|
|
|
## Generate docs
|
|
Typedoc is used for generating typescript documentation based on the code.
|
|
That part is automated and beign done as a part of CI job.
|
|
|
|
# Release
|
|
Release is automated as a part of CI job. Workflow file: `.gitea/workflows/release.yaml`.
|
|
It will run automatically when branch merged to the `main` branch.
|
|
IMPORTANT!
|
|
If release job fails, after it has been created empty release and a tag, we must not re-run the relase job until we removed the newly create GIT TAG and RELEASE.
|
|
To remove the git tag run:
|
|
```
|
|
git push -d origin vX.X.X
|
|
```
|
|
To remove the release, you need to do it with repo administration over at [https://git.datacontroller.io/dc/dc](https://git.datacontroller.io/dc/dc)
|
|
|
|
# Troubleshooting
|
|
|
|
## `npm i` fails with ENOENT on `sheet-crypto.tgz`
|
|
|
|
If `npm i`/`npm ci` in `client` fails with:
|
|
```
|
|
npm error enoent ENOENT: no such file or directory, open '.../client/libraries/sheet-crypto.tgz'
|
|
```
|
|
it means the encrypted `sheet-crypto.tgz.gpg` hasn't been decrypted yet. See
|
|
[Dependencies that requires licences](#dependencies-that-requires-licences) above for how to decrypt it.
|
|
|
|
## Makedata service "could not create directory" error
|
|
|
|
The dcpath folder should have its permissions set so that the system account (SYSUSERID) can both read and write to it.
|
|
|
|
Example:
|
|
|
|
If dcpath is: '/tmp/dc'
|
|
|
|
Run:
|
|
```
|
|
chmod 777 /tmp/dc
|
|
``` |