Files
dc/.gitea/workflows/build.yaml
T
dc c808c41333
Build / Build-and-ng-test (pull_request) Successful in 4m20s
Lighthouse Checks / lighthouse (pull_request) Successful in 8m36s
Build / Build-and-test-development (pull_request) Successful in 33m14s
fix(deploy): stop the makedata poll outliving its screen, and align the release gates
The release pipeline went red at 28157fae: Karma aborted with "Some of your
tests did a full page reload!" after test 753 of 861. The new-window spec calls
runMakedataInNewWindow, which starts the five-second makedata poll; the service
stub answers that makedata is gone, so the first tick assigns window.location.href
and the page reloads under the running spec. A local run finishes in under five
seconds, so the timer never fires there - only CI saw it.

- the poll is cleared in ngOnDestroy, so a timer cannot reload the page after
  the user has moved on; a spec covers that
- the spec that exercises the new-window URL no longer starts the real poll

Three open findings on #407, all taken:

- lighthouse.yaml: sas/package.json joins the change-detection pattern, because
  the job pins @sasjs/cli from that file
- release-assets.yaml: webpack is installed at the version the client lockfile
  resolves rather than whatever is current at release time
- build.yaml: the client audit runs with --audit-level=low, which is what the
  comment above it has always claimed
2026-11-05 17:09:52 +00:00

538 lines
26 KiB
YAML

name: Build
run-name: Running Lint Check and Licence checker on Pull Request
on: [pull_request]
# One run per pull request: a newer push supersedes the one in flight for the
# older head. Both runs test the same thing over the same twenty-six minute
# suite, the older head can no longer be what gets merged, and this workflow
# shares two runners with every other pull request in the repo - so leaving the
# old run queued only delays the head that matters. The group is the base build
# only: the release workflow is never cancelled by something it does not
# supersede.
concurrency:
group: build-pr-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
# the server the mocked services and app tests run against - bump deliberately.
# 3.12.0 is the floor: /ddc-host carries `timing=early` from there, which
# client/cypress/e2e/va-ddc-host.cy.ts needs. 3.12.2 also carries the
# packaged-executable fix for the same page, so the version the tests run
# against is the one that ships.
#
# 3.14.0 ships @sasjs/lint 5.0.0 (the studio reports OS commands) and is the
# release the estate runs. It still declares @sasjs/utils ^4.7.0, so the
# envelope *formats* (SYSCC as text, DATE7, TIME5, E8601DT26.6 from utils 4.8)
# are only guaranteed once the next server release raises that floor - a fresh
# install resolves 4.8.0 today, a cached tree need not. Bump again when it lands.
#
# 3.13.0 is the one that matters for licensing: @sasjs/utils 4.7 reports
# SYSSITE, which %webout takes from &syssite. Before it the prelude wrote an
# empty string, so the client - which routes expired and wrong-organisation
# licence keys on that value - never reached those pages on the mock estate.
#
# 3.12.3 is on top of that for a different reason: it is the first release whose
# package.json declares @sasjs/utils ^4.6.0, which is what gives webout() its
# options argument. getdata, viewdata and getcolvals pass { showmeta }, and a
# 4.5 install drops it without a word - so the pin is what makes the metadata
# members guaranteed here rather than whatever a fresh install happens to
# resolve. requireWeboutOptions in sasjs/mocks/sasjs/services/dcMockUtils.js
# refuses a runtime without it.
SASJS_SERVER_VERSION: '3.14.0'
NODE_VERSION: '24.15.0'
jobs:
Build-and-ng-test:
runs-on: ubuntu-latest
outputs:
# The Cypress job consumes this instead of deciding again: the base
# branch can move between the two jobs, and a second answer could say
# "run" after this job had already skipped the build and the upload,
# which would fail there as a missing artifact. Empty (the decision
# could not be made) means the Cypress job runs everything.
client: ${{ steps.changes.outputs.client }}
steps:
- uses: actions/checkout@v4
- name: Check runner size
# An undersized runner dies inside `ng build` (exitcode 137) or as a
# Karma disconnect twenty minutes in. Fail here instead, in seconds,
# with the measured numbers in the log - see the script.
run: bash .gitea/scripts/check-runner-memory.sh
- name: Detect the Cypress job's inputs
id: changes
# Decides whether the Cypress job will run at all, so the production
# build and its artifact upload can be skipped when nothing the suite
# exercises has changed. Same script the Cypress job itself calls, so
# the two jobs cannot disagree about which paths matter.
env:
BASE_REF: ${{ github.base_ref }}
run: bash .gitea/scripts/cypress-changes.sh
- name: Test the changed-path matcher
# The path list is the whole gate: a dropped alternative silently stops
# the suite running for a class of change, and the only symptom is a
# green job that finished sooner. The cases pin the list to what the
# suite reads.
run: bash .gitea/scripts/tests/cypress-changes.test.sh
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Scan for secrets (gitleaks)
# The local pre-commit hook is suppressed entirely when .npmrc sets
# ignore-scripts=true (the prepare script that would activate the
# hooks never runs), so CI is the enforcement point: a leaked
# credential must fail the build, not just a commit.
# Uses the same @nogoo9/gitleaks binary the pre-commit hook pins - the
# gitleaks-action requires a license key for organization repos.
# Runs before the registry config is written, so the scan cannot see
# the token this job injects.
# --no-git scans the working tree, which is what a pull request gate
# is about.
#
# The cache steps below deliberately run AFTER this one: the Angular
# cache holds webpack/babel JSON full of `{ token: ... }` fragments,
# and a restored copy in the tree is scanned as a pile of generic API
# keys (it failed the gate once, on run 3174).
run: |
npm ci
node_modules/.bin/gitleaks detect --no-git --source . --redact --no-banner -v
- name: Restore the npm download cache
# The runner's embedded cache server is reachable from job containers
# (they share a network with the runner), so these entries can actually
# come back. The key is the client lockfile: a dependency change
# invalidates it, and the prefix restore-keys keep the previous entries
# usable while the new ones are fetched.
#
# act_runner scopes cache entries to the REPOSITORY, not the ref (the
# handler stamps each entry with its repo and checks it on read), so
# the base branch is part of the key: a pull request into release/7.22
# cannot seed the entry a pull request into main restores. Entries
# still live on the runner that saved them - each act_runner runs its
# own cache server - so a box warms its cache from its own first run.
#
# An unreachable cache server degrades both the restore and the save to
# a warning - the action never fails the job - so a box on an older
# runner config costs a timeout, not a red build.
uses: actions/cache@v4
with:
path: ~/.npm
key: npm-client-${{ github.base_ref }}-${{ hashFiles('client/package-lock.json') }}
restore-keys: |
npm-client-${{ github.base_ref }}-
- name: Install Google Chrome
run: |
apt-get update
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install -y ./google-chrome*.deb
- name: Write .npmrc file
run: |
# the registry secret stays outside the repo: client/.npmrc is
# tracked now, and one git add -A would commit a live token
echo "$NPMRC" > "$RUNNER_TEMP/npmrc"
echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
shell: bash
env:
NPMRC: ${{ secrets.NPMRC}}
- name: Install dependencies
env:
SHEET_PWD: ${{ secrets.SHEET_PWD }}
run: |
cd client
# Decrypt and Install sheet
printf '%s\n' "$SHEET_PWD" | \
gpg --batch --yes --passphrase-fd 0 \
--output ./libraries/sheet-crypto.tgz \
--decrypt ./libraries/sheet-crypto.tgz.gpg
# dependency hooks stay off (client/.npmrc); the client's own
# generator runs explicitly, because the app imports the file it makes
npm ci --ignore-scripts
npm run postinstall
# cypress's binary is installed in the Cypress job, where it is
# actually used; nothing in this job starts it, so downloading and
# unzipping it here only cost time (11s, measured).
- name: Restore the Angular build cache
# `ng build` and both `ng test` runs use this directory, so a restored
# entry speeds up the production build and the Karma suites alike.
#
# This step MUST stay below "Install dependencies". postinstall runs
# scripts/strip-clr-base64-fonts.mjs, which edits Clarity's CSS inside
# node_modules and then deletes client/.angular/cache, because the
# builder treats node_modules as immutable and would otherwise serve
# styles built from the unstripped file (measured: a marker @font-face
# injected into clr-ui.min.css is absent from a warm build's CSS and
# present in a cold one). Every run strips before it builds, so every
# entry in this cache was produced from the stripped file - restoring
# after the clear is what keeps that true.
#
# Exact key only, no restore-keys: the builder cannot see a change
# inside node_modules, so an entry compiled against a different
# Clarity must not be restored. A lockfile bump misses and rebuilds,
# which is the correct outcome.
#
# Angular keeps the cache in memory when CI is set, so client/
# angular.json sets cli.cache.environment to "all" - without it this
# step has nothing to save ("Path Validation Error: Path(s) specified
# in the action for caching do(es) not exist").
#
# Saving is skipped for a key that already exists, so this refreshes
# when client/package-lock.json changes; bump the v1- prefix to force
# a reset by hand. The base ref keeps a pull request into main from
# seeding what a pull request into release/7.22 restores.
uses: actions/cache@v4
with:
path: client/.angular/cache
key: angular-v1-${{ github.base_ref }}-${{ hashFiles('client/package-lock.json') }}
- name: Check audit
# Root and sas fail on moderate and above (npm's default), client on
# low and above - the flags below are what makes that true, so they have
# to stay in step with this comment.
run: |
npm audit --omit=dev
cd ./sas
npm audit --omit=dev
cd ../client
npm audit --omit=dev --audit-level=low
- name: Check archiver directory walk
# the readdir-glob minimatch override in client/package.json is a major
# bump; archiver's directory() is the only runtime caller of that path,
# so exercise it rather than trusting that the install succeeded
run: |
cd client
npm run check:archiver
- name: Lint check
run: npm run lint:check
- name: SAS lint check
# sas/.sasjslint promotes each rule to "error" as its findings are
# cleared, and only errors change the exit code - so this gate fails on
# a regression in a cleared rule and passes while other rules still
# carry warnings.
run: |
cd sas
npm ci --ignore-scripts
npm run lint:sas
- name: Licence checker
run: |
cd client
npm run license-checker
- name: Mock estate service tests
# The deploy screen's questions (ACTION=DIRLIST / WRITECHECK) are
# answered by the mock makedata service on the browser-hosted sites.
# These run that service against a temporary estate, so a browse that
# seeds the site and deletes the program fails here instead of on the
# staging site, where it only shows as a wizard that cannot be used.
run: node --test sas/mocks/test/*.test.js
- name: Angular Tests
run: |
cd client
npm run test:headless
- name: Angular Tests in other timezones
# The VA filter clauses are built from local calendar dates, so a
# UTC-only run cannot see a date that slips a day west of UTC or east of
# it. Only the VA specs carry that property, so both scripts run that
# subset rather than the whole suite a second and third time.
run: |
cd client
npm run test:tz-west
npm run test:tz-east
- name: Every Cypress spec is in the CI list
# The spec list used to be spelled out inline in --spec, and five files
# quietly fell out of it - among them the VA embed suite. Fails when a
# spec on disk is not listed.
run: bash .gitea/scripts/cypress-specs.sh coverage .
- name: Production Build
# The build exists to give the Cypress job the client/dist it streams
# as the mock estate's frontend. When nothing that job exercises has
# changed, that job skips and nobody downloads the artifact, so the
# cold build (109s, measured on run 3132) would be spent on no one.
if: steps.changes.outputs.client != 'false'
run: |
cd client
npm run build
- name: Save the production build for the Cypress job
# Gated with the build it packages: when the Cypress job skips, this
# artifact has no consumer.
if: steps.changes.outputs.client != 'false'
# The Cypress job needs client/dist: the server-ci target streams it as
# the 'clickme' web app (see CONTRIBUTING.md). Handing it the build this
# job has already made saves a second cold `npm run build` (1.6 minutes,
# measured over thirteen runs) and guarantees both jobs test the same
# bytes.
#
# if-no-files-found: error, because an empty artifact is worse than a
# failed upload: the Cypress job would download nothing, deploy a service
# pack with no frontend, and the symptom would reappear there as the
# "webSourcePath ... doesn't exist" line this artefact exists to remove.
#
# v3, not v4: this runner reports "GHESNotSupportedError:
# @actions/artifact v2.0.0+, upload-artifact@v4+ ... not supported on
# GHES", so the current major cannot upload here at all - the download in
# Build-and-test-development is @v3 for the same reason.
uses: actions/upload-artifact@v3
with:
name: client-dist
path: client/dist
if-no-files-found: error
- name: Remove registry config
# written earlier in this job; nothing deletes it on the way out
if: always()
run: rm -f "$RUNNER_TEMP/npmrc"
Build-and-test-development:
runs-on: ubuntu-latest
needs: Build-and-ng-test
env:
CHROME_BIN: /usr/bin/google-chrome
# Pin OS locale
LANG: en_GB.UTF-8
LC_ALL: en_GB.UTF-8
steps:
- uses: actions/checkout@v4
- name: Take the changed-path decision from the build job
id: changes
# The build job made this decision before it built (or skipped) the
# client-dist this job downloads, and published it as a job output.
# Deciding again here could disagree if the base branch moved in
# between, and the disagreement would surface as a missing artifact.
# An empty value means that job could not decide, so the suite runs.
env:
CLIENT: ${{ needs['Build-and-ng-test'].outputs.client }}
run: |
case "${CLIENT:-}" in
true) echo "client=true" >> "$GITHUB_OUTPUT" ;;
false) echo "client=false" >> "$GITHUB_OUTPUT" ;;
*) echo "the build job published no decision - running the suite" ;;
esac
- name: Cypress suite not run - no client changes
# A named step rather than a run of greyed-out ones, so the job summary
# says why it took a minute: this job reports success having tested
# nothing, which is the intended behaviour for a pull request that
# changes no client code. If this job is a required check, "green" for
# such a pull request means "not applicable", and this step is the
# evidence for it.
if: steps.changes.outputs.client == 'false'
run: |
echo "No client, sas/mocks, sas/package.json or .gitea/scripts change in this pull request - the Cypress suite was skipped by design."
echo "The build job's 'Detect the Cypress job's inputs' step lists the files it saw, and warns instead if it could not decide (in which case the suite runs)."
- name: Check runner size
if: steps.changes.outputs.client != 'false'
# Each Cypress job compiles the frontend and runs a browser on top; a
# runner too small for that fails as a flaky-looking disconnect - see
# the script.
run: bash .gitea/scripts/check-runner-memory.sh
- uses: actions/setup-node@v4
if: steps.changes.outputs.client != 'false'
with:
node-version: ${{ env.NODE_VERSION }}
- name: Write .npmrc file
if: steps.changes.outputs.client != 'false'
run: |
# the secret arrives through the environment, not interpolated into
# this script: a quote or $( in the value would otherwise break the
# step or execute
printf '%s\n' "$NPMRC" > "$RUNNER_TEMP/npmrc"
echo "NPM_CONFIG_USERCONFIG=$RUNNER_TEMP/npmrc" >> "$GITHUB_ENV"
env:
NPMRC: ${{ secrets.NPMRC}}
- name: Install system dependencies
if: steps.changes.outputs.client != 'false'
run: |
apt-get update
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install -y ./google-chrome*.deb
apt-get -y install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2t64 libxtst6 xauth xvfb jq zip locales
# Generate the en_GB.UTF-8 locale referenced by LANG/LC_ALL
locale-gen en_GB.UTF-8
update-locale LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8
- name: Write cypress credentials
if: steps.changes.outputs.client != 'false'
run: echo "$CYPRESS_CREDS" > ./client/cypress.env.json
shell: bash
env:
CYPRESS_CREDS: ${{ secrets.CYPRESS_CREDS }}
- name: Install dependencies
if: steps.changes.outputs.client != 'false'
env:
SHEET_PWD: ${{ secrets.SHEET_PWD }}
run: |
cd client
# Decrypt and Install sheet
printf '%s\n' "$SHEET_PWD" | gpg --batch --yes --passphrase-fd 0 ./libraries/sheet-crypto.tgz.gpg
# dependency hooks stay off (client/.npmrc); the client's own
# generator runs explicitly, because the app imports the file it makes
npm ci --ignore-scripts
npm run postinstall
# cypress downloads its binary in a dependency install hook, which the
# policy skips, so it is installed explicitly here
npx cypress install
- name: Fetch the production build
if: steps.changes.outputs.client != 'false'
# The server-ci target streams client/dist as the 'clickme' web app (see
# CONTRIBUTING.md), so the production build has to exist before the mock
# services are deployed. Without it the deploy logs
# "webSourcePath: .../client/dist ... doesn't exist" and streams no
# frontend. It is the artefact Build-and-ng-test published, so this job
# does not compile the frontend a second time.
uses: actions/download-artifact@v3
with:
name: client-dist
path: client/dist
- name: Cache the npm downloads
if: steps.changes.outputs.client != 'false'
# The server is installed from npm at a pinned version, so the tarball is
# immutable and the key never needs invalidating by hand: a hit skips
# the 10.7MB download, a miss costs exactly what the zip cost before.
# continue-on-error, because a runner without cache support should still
# run the pipeline - this step only ever saves time.
continue-on-error: true
uses: actions/cache@v4
with:
path: ~/.npm
key: npm-sasjs-server-${{ env.SASJS_SERVER_VERSION }}
- name: Setup and start SASjs server
if: steps.changes.outputs.client != 'false'
run: |
npm i -g pm2@7.1.2
touch .env
echo RUN_TIMES=js >> .env
echo NODE_PATH=node >> .env
echo CORS=enable >> .env
echo WHITELIST=http://localhost:4200 >> .env
echo HTTP_PORT=5000 >> .env
# /ddc-host is a testing tool, so the server answers 404 for it unless
# this is set. va-ddc-host.cy.ts drives the app through that page with
# timing=early.
echo DDC_HOST_ENABLED=true >> .env
# The server refuses to run as root unless RUN_AS names an account
# (a security change in the recent releases). This job runs as root
# in a throwaway container, so opt in explicitly.
echo RUN_AS=root >> .env
cat .env
pm2 start npx --name sasjs-server -- --yes @sasjs/server@${{ env.SASJS_SERVER_VERSION }}
# `pm2 start` returns before the app is listening, and the next step
# hits :5000 immediately. Wait for the server to answer, and dump its
# log if it never does.
for i in $(seq 1 60); do
if curl -sf -o /dev/null http://localhost:5000/; then
echo "SASjs server answered on attempt ${i}"
exit 0
fi
sleep 2
done
echo "SASjs server did not come up within 120s"
pm2 list
pm2 logs sasjs-server --lines 100 --nostream
exit 1
- name: Deploy mocked services
if: steps.changes.outputs.client != 'false'
run: |
cd ./sas/mocks/sasjs
# The CLI comes from the version sas/package.json pins, as docs.yaml and
# release-assets.yaml already do: an unpinned global install runs
# whatever release is current, unreviewed.
CLI_VERSION=$(node -p "require('../../package.json').dependencies['@sasjs/cli']")
npm install -g "@sasjs/cli@${CLI_VERSION}"
npm install -g replace-in-files-cli@4.0.0
# Remove any previous deployment (drive folder delete API) so the
# deploy and makedata start from a clean appLoc
curl -sS -X DELETE "http://localhost:5000/SASjsApi/drive/folder/?_folderPath=/Public/app/dc"
sasjs cbd -t server-ci
# Seed the DC database (mock data files on the drive).
# makedata replies with HTML (it is normally called as a URL redirect),
# which the CLI reports as "invalid Json string" - ignore that.
# NOTE: -d paths resolve from the sasjs project root (sas/mocks),
# not the cwd (sas/mocks/sasjs).
sasjs request services/admin/makedata -t server-ci -d deploy/makedata.json -o ./makedata_out.json || true
- name: Prepare and run frontend and cypress
if: steps.changes.outputs.client != 'false'
# One job, one SASjs server, one ng serve, one Chrome: four of each on one
# runner starved the app under test and every failure in that run was a
# timeout, not an assertion.
timeout-minutes: 50
env:
CYPRESS_USERNAME: ${{ secrets.CYPRESS_USERNAME_SASJS }}
CYPRESS_PWD: ${{ secrets.CYPRESS_PWD_SASJS }}
run: |
cd ./client
# The spec list lives in one place, and an empty list is a broken
# workflow rather than a green run with no tests.
SPECS="$(bash ../.gitea/scripts/cypress-specs.sh all)"
if [ -z "$SPECS" ]; then
echo "the spec list resolved to nothing"
exit 1
fi
echo "specs: $SPECS"
mv ./cypress.env.example.json ./cypress.env.json
replace-in-files --regex='"username".*' --replacement='"username":"'"$CYPRESS_USERNAME"'",' ./cypress.env.json
replace-in-files --regex='"password".*' --replacement='"password":"'"$CYPRESS_PWD"'" ' ./cypress.env.json
cat ./cypress.env.json
npm run postinstall
# Prepare index.html to SASJS local
replace-in-files --regex='serverUrl=".*?"' --replacement='serverUrl="http://localhost:5000"' ./src/index.html
replace-in-files --regex='appLoc=".*?"' --replacement='appLoc="/Public/app/dc"' ./src/index.html
replace-in-files --regex='serverType=".*?"' --replacement='serverType="SASJS"' ./src/index.html
replace-in-files --regex='"hosturl".*' --replacement='hosturl:"http://localhost:4200",' ./cypress.config.ts
cat ./cypress.config.ts
# Start frontend and run cypress
# timeout 2700: SIGTERM after 45 min so Cypress can flush screenshots
# before the outer timeout-minutes hard-kills the step (avoids silent multi-hour hangs)
#
# video=false, screenshotOnRunFailure=true: recording twenty-four
# specs costs encode time on every run, and the recording is only ever
# opened after a failure. A screenshot of the failing step carries the
# same information for a fraction of the cost - and cypress.config.ts
# sets screenshotOnRunFailure false by default, so both are passed
# here, as CLI overrides, leaving local runs (and the demo recorder,
# which sets RECORD_WINDOW_SIZE) on video.
npx ng serve --host 0.0.0.0 --port 4200 & npx wait-on http://localhost:4200 && timeout 2700 npx cypress run --browser chrome --config video=false,screenshotOnRunFailure=true --spec "$SPECS"
- name: Zip Cypress failure evidence
# Only on a failure: a green run has nothing to look at.
if: failure() && steps.changes.outputs.client != 'false'
run: |
mkdir -p ./client/cypress/screenshots ./client/cypress/videos
zip -r cypress-artifacts.zip ./client/cypress/screenshots ./client/cypress/videos
- name: Add cypress failure evidence as an artifact
if: failure() && steps.changes.outputs.client != 'false'
# v3, not v4: this runner reports "GHESNotSupportedError:
# @actions/artifact v2.0.0+, upload-artifact@v4+ ... not supported on
# GHES", so the current major cannot upload here at all.
uses: actions/upload-artifact@v3
with:
name: cypress-artifacts.zip
path: cypress-artifacts.zip
- name: Remove registry config
# written earlier in this job; nothing deletes it on the way out
if: always()
run: rm -f "$RUNNER_TEMP/npmrc"