dc
bb8d007e2d
demo.datacontroller.io is now a multi-site Surfer instance: a release branch deploys the staging.datacontroller.io site, and a push to main deploys the demo (default site). Both run the same browser-hosted mock build.
187 lines
9.2 KiB
Markdown
187 lines
9.2 KiB
Markdown
# Data Controller
|
|
# Contributing
|
|
|
|
## Workflow guidelines
|
|
[Wiki Page](https://git.datacontroller.io/dc/dc/wiki/Git-Workflow)
|
|
|
|
## Git hooks
|
|
|
|
The repo ships its own git hooks in [`.git-hooks/`](./.git-hooks):
|
|
|
|
- `pre-commit` - scans staged changes for secrets with gitleaks, blocks commits that would add more than 2MB of new blobs, and runs the prettier check (auto-fixing on failure)
|
|
- `commit-msg` - verifies the commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) standard
|
|
|
|
The repo `.npmrc` sets `ignore-scripts=true`, so the `prepare` and `postinstall` lifecycle scripts that would normally set `core.hooksPath` during `npm i` never run. After cloning (or if your commits are not being checked), activate the hooks with a one-time command, run from the repo root:
|
|
|
|
```bash
|
|
git config core.hooksPath ./.git-hooks
|
|
```
|
|
|
|
The pre-commit hook requires the gitleaks binary provided by the root `@nogoo9/gitleaks` devDependency, so make sure `npm i` has run in the repo root before your first commit.
|
|
|
|
The standard [pre-commit](https://pre-commit.com) route is available as well, and does not depend on the lifecycle scripts above: install it once with `pre-commit install` and the secret scan in [`.pre-commit-config.yaml`](./.pre-commit-config.yaml) runs on every commit. The same scan also runs in CI, so a leaked credential fails the build regardless of which local hooks are active.
|
|
|
|
## Dependencies that requires licences
|
|
|
|
[SheetJS Pro Version](https://www.npmjs.com/package/sheetjs)
|
|
|
|
|
|
To auth for SheetJS Pro version you need to use their private registry
|
|
Add `client/.npmrc` file with following content:
|
|
```
|
|
@sheet:registry=https://pylon.sheetjs.com:54111/
|
|
//pylon.sheetjs.com:54111/:_authToken="TOKEN-GOES-HERE"
|
|
```
|
|
|
|
In addition to the registry auth, the `@sheet/crypto` package is installed from a local tarball
|
|
(`client/libraries/sheet-crypto.tgz`) rather than the registry. This tarball is not committed to
|
|
the repo directly - only an encrypted copy is: `client/libraries/sheet-crypto.tgz.gpg`.
|
|
|
|
Before running `npm i`/`npm ci` in `client`, you must decrypt it using the passphrase
|
|
(stored as the `SHEET_PWD` CI secret - ask a repo maintainer if you don't have it):
|
|
```bash
|
|
cd client
|
|
echo "YOUR_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
|
|
--output ./libraries/sheet-crypto.tgz \
|
|
--decrypt ./libraries/sheet-crypto.tgz.gpg
|
|
|
|
npm i
|
|
```
|
|
|
|
[Handsontable](https://www.npmjs.com/package/handsontable)
|
|
|
|
The licence key comes from the backend licence: the decrypted licence carries `hot_license_key`,
|
|
which is published on the `hot_license_key` subject and pushed into every Handsontable instance.
|
|
Until a licence is loaded the client falls back to the non-commercial key
|
|
(`DEFAULT_HOT_LICENCE_KEY` in `client/src/app/services/licence.service.ts`), so there is nothing to
|
|
configure in `index.html` for this.
|
|
|
|
## Development
|
|
Update `client/src/index.html` so that it points to your SAS9, SASVIYA or SASJS backend.
|
|
Be aware that VIYA can be configured in such way that it would not work with cross origin frontend.
|
|
Follow this guide to disable CORS: https://sasjs.io/cors/ (NOTICE: Sometimes even this approach would fail to work, in such case it is imposible to set it up without reconfiguring the VIYA server)
|
|
|
|
Start dev server:
|
|
```
|
|
cd client
|
|
|
|
npm start
|
|
```
|
|
|
|
## Mocked backend (frontend development without SAS)
|
|
|
|
Data Controller ships a full set of JS mock services under `sas/mocks/sasjs/` that mimic the SAS backend using [SASjs Server](https://server.sasjs.io) in JS-only mode. This lets you run the entire frontend (editor grid, approvals, viewer, admin, Excel maps, etc.) without a SAS licence or SAS server. The Cypress E2E tests and Lighthouse CI both run against this setup.
|
|
|
|
### Prerequisites
|
|
|
|
- Node.js (same version used for the frontend build)
|
|
- The frontend dependencies installed (`cd client && npm ci` after decrypting sheet-crypto as described above)
|
|
|
|
### 1. Start SASjs Server
|
|
|
|
Download the latest Linux (or macOS/Windows) binary from [SASjs Server releases](https://github.com/sasjs/server/releases), unzip it, and create a `.env` file alongside the executable:
|
|
|
|
```
|
|
RUN_TIMES=js
|
|
NODE_PATH=node
|
|
CORS=enable
|
|
WHITELIST=http://localhost:4200
|
|
```
|
|
|
|
`RUN_TIMES=js` means the server executes `.js` files as Stored Programs (no SAS binary needed). `CORS=enable` plus `WHITELIST` allows the Angular dev server on port 4200 to call the mock services on port 5000. Start the server:
|
|
|
|
```
|
|
./api-linux # or api-macos / api-win.exe depending on your platform
|
|
```
|
|
|
|
The server runs in desktop mode (no auth) on `http://localhost:5000` by default.
|
|
|
|
### 2. Deploy the mock services and streaming frontend
|
|
|
|
From `sas/mocks/sasjs/`:
|
|
|
|
```
|
|
npm ci
|
|
sasjs cbd -t server-ci
|
|
```
|
|
|
|
This compiles, builds, and deploys the JS mock services plus the Angular frontend (as a streaming app) to the SASjs Server. The `server-ci` target in `sas/mocks/sasjs/sasjsconfig.json` points to `http://localhost:5000` with appLoc `/Public/app/dc` and stream service name `clickme`.
|
|
|
|
After deploy, the streamed app is available at: `http://localhost:5000/AppStream/clickme/`
|
|
|
|
### 3. Run the Angular dev server (optional, for hot reload)
|
|
|
|
For frontend development with live reload, point `client/src/index.html` at the local SASjs Server and start the dev server:
|
|
|
|
```
|
|
cd client
|
|
# Edit src/index.html: set serverUrl="http://localhost:5000", appLoc="/Public/app/dc", serverType="SASJS"
|
|
npm start
|
|
```
|
|
|
|
The app will be at `http://localhost:4200`. It calls the JS mock services on port 5000 via the SASjs adapter (CORS is enabled by the `.env` above).
|
|
|
|
### How the mocks work
|
|
|
|
Each `.js` file in `sas/mocks/sasjs/services/` corresponds to a real SAS service (e.g. `public/startupservice.js` mimics `services/public/startupservice.sas`). The JS runtime predeclares `fs` and `_webout`; the mock assigns a JSON string to `_webout` that matches the structure the real SAS service would return. Some mocks are stateful: they persist data to a `mock-storage/` folder on the SASjs Drive (e.g. `licence.json`, `users.json`, `filter.txt`) using the predeclared `fs` module.
|
|
|
|
The mock data includes sample tables (`DC996664.MPE_X_TEST`, `MPE_X_NEW`, `MPE_X_FORMULA_TEST`, etc.) with realistic columns, validation rules (HARDREGEX, SOFTREGEX, HARDSELECT, READONLY, HIDDEN, ROUND, NUMBER_FORMAT), and dropdown values. This is sufficient to exercise the full editor grid, approval workflow, viewer, and admin screens.
|
|
|
|
There is also a SAS 9 mock variant under `sas/mocks/sas9/` (target `sas9-mocks`) that mimics SAS 9 stored processes with the same JS approach but a different appLoc (`/User Folders/sasdemo/`).
|
|
|
|
### Re-deploying after changes
|
|
|
|
After changing mock services, re-run `sasjs cbd -t server-ci` from `sas/mocks/sasjs/`. After changing the frontend, rebuild (`cd client && npm run build`) then re-deploy so the streaming app picks up the new `client/dist/`. If you are using `ng serve` (step 3), only the mock services need re-deploying; the frontend hot-reloads automatically.
|
|
|
|
### Demo and staging sites
|
|
|
|
A push to `release/**` publishes [staging.datacontroller.io](https://staging.datacontroller.io); a push to `main` publishes [demo.datacontroller.io](https://demo.datacontroller.io) - workflow file `.gitea/workflows/sites.yaml`, using the `DEMO_DATACONTROLLER_IO` Surfer app password (user `allan`). Both are sites on one multi-site Surfer instance; the demo is its default site. The job builds the mocks as a browser-hosted site (SASjs CLI >= 6.1, `streamConfig.mockWeb`), so the services run in a service worker and the drive lives in the visitor's IndexedDB - there is no server. It runs `makedata` on a throwaway SASjs Server first and ships that drive as `sasjsbuild/mock-snapshot.json`; without it the site opens with no data. The mock licence is minted for HTTP only, so both sites show the free-tier banner.
|
|
|
|
## GUI Elements
|
|
|
|
For documentation on the Clarity Design System, including a list of components and example usage, see [our website](https://vmware.github.io/clarity).
|
|
|
|
## Code style
|
|
Run prettier fix:
|
|
```bash
|
|
npm run lint:fix
|
|
```
|
|
|
|
## Generate docs
|
|
Typedoc is used for generating typescript documentation based on the code.
|
|
That part is automated and beign done as a part of CI job.
|
|
|
|
# Release
|
|
Release is automated as a part of CI job. Workflow file: `.gitea/workflows/release.yaml`.
|
|
It will run automatically when branch merged to the `main` branch.
|
|
IMPORTANT!
|
|
If release job fails, after it has been created empty release and a tag, we must not re-run the relase job until we removed the newly create GIT TAG and RELEASE.
|
|
To remove the git tag run:
|
|
```
|
|
git push -d origin vX.X.X
|
|
```
|
|
To remove the release, you need to do it with repo administration over at [https://git.datacontroller.io/dc/dc](https://git.datacontroller.io/dc/dc)
|
|
|
|
# Troubleshooting
|
|
|
|
## `npm i` fails with ENOENT on `sheet-crypto.tgz`
|
|
|
|
If `npm i`/`npm ci` in `client` fails with:
|
|
```
|
|
npm error enoent ENOENT: no such file or directory, open '.../client/libraries/sheet-crypto.tgz'
|
|
```
|
|
it means the encrypted `sheet-crypto.tgz.gpg` hasn't been decrypted yet. See
|
|
[Dependencies that requires licences](#dependencies-that-requires-licences) above for how to decrypt it.
|
|
|
|
## Makedata service "could not create directory" error
|
|
|
|
The dcpath folder should have its permissions set so that the system account (SYSUSERID) can both read and write to it.
|
|
|
|
Example:
|
|
|
|
If dcpath is: '/tmp/dc'
|
|
|
|
Run:
|
|
```
|
|
chmod 777 /tmp/dc
|
|
``` |