ci: fail a client job fast on a runner too small for it #429

2026-11-01T15:19:41Z

hermes commented

2026-11-01 15:19:41 +00:00

What

.gitea/scripts/check-runner-memory.sh runs as the first step of every job that compiles the frontend: the three review jobs - Build-and-ng-test, Build-and-test-development and lighthouse - and the release path - all three jobs in release.yaml, deploy in sites.yaml and assets in release-assets.yaml. It prints the runner's RAM, available RAM, vCPU count and any cgroup limit, and exits 1 before any install when the effective size is below the floor.

Why

An undersized runner fails in a way that reads as a flaky test rather than as an undersized runner:

exitcode '137' at Production Build / Build Frontend, right after - Generating browser application bundles (phase: setup)... - the kernel OOM killer.
Chrome Headless ... Disconnected, because no message in 30000 ms, mid-suite at Executed 250 of 704 or 644 of 704 - the same memory pressure reaching Chrome.

Both cost a full CI slot (20-30 minutes) and neither names the box.

The threshold

Measured on this repository's client, cold tree (no .angular/cache, which is what a fresh runner has), 4 vCPU:

configuration	peak RSS	wall clock
`ng build --configuration production`, cold	3773 MB	127 s
same, with `.angular/cache` present	1716 MB	32 s

The toolchain adds to that: npm ci (node_modules is 1.1 GB on disk), and for the development and Lighthouse jobs a dev server and a browser on top. 8 GB is the supported minimum - roughly 2x the peak of the build alone - and it is what the fleet runs now that the 4 GB boxes are out of the pool.

The floor in the script is 7000 MB, not 8192 MB, because an 8 GB box does not report 8 GB: this fleet's report 7751 MB total (measured by the step itself on the first CI run of this branch), the rest going to firmware and the kernel. A 4 GB box reports under 4000 MB, so 7000 separates the two sizes with room for a box that reserves a little more than these do.

A container may carry its own cgroup ceiling below the host's RAM, so memory.max / memory.limit_in_bytes is read too and the lower of the two sizes is what the check and the message use. MIN_TOTAL_MB (argument or environment) overrides the floor for a job that ever needs a different one.

Proof it bites

On this box (16 GB total, 16 GB cgroup limit):

$ .gitea/scripts/check-runner-memory.sh
runner memory: 16044 MB total, 6838 MB available, 4 vCPU, 16384 MB cgroup limit
$ echo $?
0

$ .gitea/scripts/check-runner-memory.sh 20000
runner memory: 16044 MB total, 6838 MB available, 4 vCPU, 16384 MB cgroup limit
::error::runner has 16044 MB of memory, a client job needs at least 20000 MB (8 GB is the supported minimum; a cold production build alone peaks at about 3.8 GB, and the Cypress and Lighthouse jobs add a dev server and a browser to that).
$ echo $?
1

The ::error:: line becomes an annotation on the run, so the failure is visible in the Actions list without opening the log.

A low MemAvailable only raises a ::warning::, not a failure: a second job landing on the same box is transient, whereas the box's size is not. That is the one case the check does not rule out - two client jobs on one box can OOM a build that the totals say fits - so the pool keeps one client job per box, and the script's header states the limit of the check.

## What `.gitea/scripts/check-runner-memory.sh` runs as the first step of every job that compiles the frontend: the three review jobs - `Build-and-ng-test`, `Build-and-test-development` and `lighthouse` - and the release path - all three jobs in `release.yaml`, `deploy` in `sites.yaml` and `assets` in `release-assets.yaml`. It prints the runner's RAM, available RAM, vCPU count and any cgroup limit, and exits 1 before any install when the effective size is below the floor. ## Why An undersized runner fails in a way that reads as a flaky test rather than as an undersized runner: - `exitcode '137'` at `Production Build` / `Build Frontend`, right after `- Generating browser application bundles (phase: setup)...` - the kernel OOM killer. - `Chrome Headless ... Disconnected, because no message in 30000 ms`, mid-suite at `Executed 250 of 704` or `644 of 704` - the same memory pressure reaching Chrome. Both cost a full CI slot (20-30 minutes) and neither names the box. ## The threshold Measured on this repository's client, cold tree (no `.angular/cache`, which is what a fresh runner has), 4 vCPU: | configuration | peak RSS | wall clock | |---|---|---| | `ng build --configuration production`, cold | 3773 MB | 127 s | | same, with `.angular/cache` present | 1716 MB | 32 s | The toolchain adds to that: `npm ci` (node_modules is 1.1 GB on disk), and for the development and Lighthouse jobs a dev server and a browser on top. 8 GB is the supported minimum - roughly 2x the peak of the build alone - and it is what the fleet runs now that the 4 GB boxes are out of the pool. The floor in the script is 7000 MB, not 8192 MB, because an 8 GB box does not report 8 GB: this fleet's report **7751 MB** total (measured by the step itself on the first CI run of this branch), the rest going to firmware and the kernel. A 4 GB box reports under 4000 MB, so 7000 separates the two sizes with room for a box that reserves a little more than these do. A container may carry its own cgroup ceiling below the host's RAM, so `memory.max` / `memory.limit_in_bytes` is read too and the lower of the two sizes is what the check and the message use. `MIN_TOTAL_MB` (argument or environment) overrides the floor for a job that ever needs a different one. ## Proof it bites On this box (16 GB total, 16 GB cgroup limit): ``` $ .gitea/scripts/check-runner-memory.sh runner memory: 16044 MB total, 6838 MB available, 4 vCPU, 16384 MB cgroup limit $ echo $? 0 $ .gitea/scripts/check-runner-memory.sh 20000 runner memory: 16044 MB total, 6838 MB available, 4 vCPU, 16384 MB cgroup limit ::error::runner has 16044 MB of memory, a client job needs at least 20000 MB (8 GB is the supported minimum; a cold production build alone peaks at about 3.8 GB, and the Cypress and Lighthouse jobs add a dev server and a browser to that). $ echo $? 1 ``` The `::error::` line becomes an annotation on the run, so the failure is visible in the Actions list without opening the log. A low `MemAvailable` only raises a `::warning::`, not a failure: a second job landing on the same box is transient, whereas the box's size is not. That is the one case the check does not rule out - two client jobs on one box can OOM a build that the totals say fits - so the pool keeps one client job per box, and the script's header states the limit of the check.

hermes force-pushed ci/runner-memory-guard from e4283bbd73 to f4113a3331

2026-11-01 15:47:38 +00:00

Compare

hermes added 1 commit 2026-11-01 15:47:50 +00:00

ci: fail a client job fast on a runner too small for it

Build / Build-and-ng-test (pull_request) Successful in 6m25s

Details

Lighthouse Checks / lighthouse (pull_request) Successful in 19m12s

Details

Build / Build-and-test-development (pull_request) Successful in 35m37s

Details

ab4f4c9b60

An undersized runner does not fail honestly: the kernel OOM killer takes
the build, so the job reports exitcode 137 at ng build, or Karma
disconnects mid-suite after twenty minutes. Both read as a flaky test
rather than as a box that cannot hold the job, and both cost a whole CI
slot.

The check runs first in the three client jobs (Build-and-ng-test,
Build-and-test-development, lighthouse), prints total and available RAM,
vCPU count and any cgroup limit, and exits 1 when the effective size is
under 8 GB - the supported minimum. A cold production build measures
3.8 GB peak on its own, so 8 GB leaves roughly 2x headroom.

hermes force-pushed ci/runner-memory-guard from f4113a3331 to ab4f4c9b60

2026-11-01 15:47:50 +00:00

Compare

hermes added 1 commit 2026-11-01 16:24:14 +00:00

ci: gate every job that builds the frontend, and scope the check in its header

Build / Build-and-ng-test (pull_request) Successful in 5m37s

Details

Lighthouse Checks / lighthouse (pull_request) Successful in 19m3s

Details

Build / Build-and-test-development (pull_request) Successful in 35m31s

Details

b1892ebb98

release.yaml (all three jobs), sites.yaml and release-assets.yaml run the same cold production build that the review jobs do, so they get the same pre-install check.

The header now says what the check rules out (an undersized box) and what it does not (two jobs sharing one box), so the MemAvailable warning is not read as a gate.

hermes commented

2026-11-01 16:24:25 +00:00

Both findings addressed in b1892eb.

The release path is gated. .gitea/workflows/release.yaml (all three jobs), sites.yaml's deploy and release-assets.yaml's assets now run the check as the first step after checkout, with a comment naming what dies in each. Every job in .gitea/workflows/*.yaml whose steps contain npm run build now carries the gate - 8 of 8, checked by parsing the workflows.
Availability is documented rather than switched. Two client jobs on one box is a real failure mode, but MemAvailable at job start cannot distinguish a busy box from a small one, and failing on it would kill runs that would have survived. So the script header now states what the check does rule out (a box below the floor) and what it does not (oversubscription), that the MemAvailable line is a warning, and that the pool keeps one client job per box; the PR body says the same. A default-off MIN_AVAILABLE_MB switch would be a path nothing sets and nothing exercises.

Both findings addressed in b1892eb. 1. The release path is gated. `.gitea/workflows/release.yaml` (all three jobs), `sites.yaml`'s `deploy` and `release-assets.yaml`'s `assets` now run the check as the first step after checkout, with a comment naming what dies in each. Every job in `.gitea/workflows/*.yaml` whose steps contain `npm run build` now carries the gate - 8 of 8, checked by parsing the workflows. 2. Availability is documented rather than switched. Two client jobs on one box is a real failure mode, but `MemAvailable` at job start cannot distinguish a busy box from a small one, and failing on it would kill runs that would have survived. So the script header now states what the check does rule out (a box below the floor) and what it does not (oversubscription), that the `MemAvailable` line is a warning, and that the pool keeps one client job per box; the PR body says the same. A default-off `MIN_AVAILABLE_MB` switch would be a path nothing sets and nothing exercises.

hermes added 1 commit 2026-11-01 18:06:36 +00:00

ci: make the memory check report its own input errors clearly

Build / Build-and-ng-test (pull_request) Successful in 5m45s

Details

Lighthouse Checks / lighthouse (pull_request) Successful in 20m47s

Details

Build / Build-and-test-development (pull_request) Successful in 31m22s

Details

f3d85d49bf

A MIN_TOTAL_MB typo reached the integer test under set -e and aborted with "integer expression expected"; it now prints usage and exits 2. A kernel without MemAvailable no longer reports 0 MB available in the summary or the warning, so the oversubscription warning stays meaningful.

hermes commented

2026-11-01 18:06:41 +00:00

Both fixed in f3d85d4.

MIN_TOTAL_MB is validated with a case guard before any arithmetic. 8GB, an empty value and a stray flag each print usage: check-runner-memory.sh [min_total_mb] - got '...' and exit 2, instead of reaching the integer test.
A missing MemAvailable is no longer rendered as 0: the summary prints available unknown, and the oversubscription warning is skipped - the same fail-open shape as the MemTotal path, so the warning only fires when the number is real.

Checked against fixture /proc/meminfo files: MemTotal only -> available unknown, exit 0; 976 MB available -> warning, exit 0; 8GB / abc / --min=8000 -> usage, exit 2. shellcheck clean.

Both fixed in f3d85d4. 1. `MIN_TOTAL_MB` is validated with a `case` guard before any arithmetic. `8GB`, an empty value and a stray flag each print `usage: check-runner-memory.sh [min_total_mb] - got '...'` and exit 2, instead of reaching the integer test. 2. A missing `MemAvailable` is no longer rendered as 0: the summary prints `available unknown`, and the oversubscription warning is skipped - the same fail-open shape as the `MemTotal` path, so the warning only fires when the number is real. Checked against fixture /proc/meminfo files: MemTotal only -> `available unknown`, exit 0; 976 MB available -> warning, exit 0; `8GB` / `abc` / `--min=8000` -> usage, exit 2. shellcheck clean.

hermes reviewed

2026-11-01 18:28:24 +00:00

hermes left a comment

REVIEWED ON 2026-11-01 18:28 UTC

Reviewed the full base...head diff: the runner-memory preflight script plus its invocation in the build, lighthouse, release, release-assets and sites workflows.

No blocking issues.

REVIEWED ON 2026-11-01 18:28 UTC Reviewed the full base...head diff: the runner-memory preflight script plus its invocation in the build, lighthouse, release, release-assets and sites workflows. No blocking issues.

allan merged commit 440ea9dc0f into release/7.22

2026-11-01 18:56:07 +00:00

allan deleted branch ci/runner-memory-guard

2026-11-01 18:56:07 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dc/dc#429