Compare commits

..
3 Commits
Author SHA1 Message Date
truecloud-patch bot 0d200f529d docs: refresh the TrueNAS compatibility matrix 2026-09-16 06:24:25 +00:00
flan 14ef25e3c6 Run CI as a single job so concurrent jobs cannot race
CI / ci (shell + python 3.11-3.13) (push) Successful in 46s
Two CI failures shared one cause: four jobs starting together on the
self-hosted runner.

act caches each action as one shared clone under /root/.cache/act/<hash> and
re-pulls it per job, so concurrent jobs fight over that directory and the
loser dies with "lstat .../<file>: no such file or directory" before any test
runs — a different victim each push. And the runner force-pulls its base image
per job, so four jobs meant four anonymous Docker Hub pulls per push; that hit
429 Too Many Requests and every job started failing before it began, including
the shell job nothing had touched.

Installing uv without an action only shrank the surface, since every job still
used actions/checkout. Concurrency is the ingredient, so this removes it: one
job cannot race itself whatever actions it uses, and one job is one pull. The
version sweep moves inside the job and still runs every version after one
fails, preserving what fail-fast: false bought.
2026-08-26 05:34:11 +00:00
flan 02ba653127 Install uv without an action so the matrix jobs stop racing
CI / python 3.12 (push) Failing after 1s
CI / shell (shellcheck + syntax) (push) Failing after 1s
CI / python 3.11 (push) Failing after 1s
CI / python 3.13 (push) Failing after 0s
act caches each action as one shared git clone under /root/.cache/act/<hash>
and re-pulls it per job. The three python matrix jobs start within the same
second on the self-hosted runner, race on that directory, and the loser dies
with "lstat /root/.cache/act/<hash>/.npmrc: no such file or directory" before
any test runs — a red main with zero suite output and a different victim each
push (3.12, then 3.11).

The action was only fetching a binary; the matrix interpreter is selected per
command by uvx --python. A run: step has no action-cache entry and cannot
race, and keeps the jobs parallel — serialising the matrix would cost 3x the
wall clock and still leave actions/checkout shared across four jobs. The uv
version is pinned under the same rule as ruff.

The accompanying test parses uses: directives rather than the raw text, so the
comment can still name the action it avoids, and uses re instead of PyYAML
because CI runs uvx pytest, whose environment holds pytest and nothing else.
2026-08-26 05:28:26 +00:00
4 changed files with 171 additions and 20 deletions
+63 -17
View File
@@ -9,9 +9,31 @@ on:
permissions: permissions:
contents: read contents: read
# ONE job, deliberately. This was four (shell + a 3-way python matrix) and they
# started within the same second on the self-hosted Gitea runner, which is what
# made CI unreliable in two separate ways:
#
# 1. The act action-cache race. `act` caches each ACTION as a single shared git
# clone under /root/.cache/act/<hash> and re-pulls it per job, so concurrent
# jobs using the same action fight over that directory and the loser dies
# with `lstat /root/.cache/act/<hash>/<file>: no such file or directory` --
# a red `main` with zero suite output, and a different victim each push
# (3.12 on one, 3.11 on the next). Dropping one action only shrank the
# surface: every job still used actions/checkout. Concurrency is the actual
# ingredient, so removing it removes the whole class -- a single job cannot
# race itself, no matter which actions it uses.
#
# 2. Docker Hub 429s. The runner force-pulls its base image per job, so four
# jobs meant four anonymous pulls per push. A few pushes and re-runs in an
# afternoon exhausted the anonymous limit and every job failed before it
# started -- including the shell job, which nothing had touched. One job is
# one pull.
#
# The cost is wall-clock parallelism, and this repo does not need it: the suite
# is ~1.5s, so container start and interpreter downloads dominate either way.
jobs: jobs:
shell: ci:
name: shell (shellcheck + syntax) name: ci (shell + python 3.11-3.13)
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -31,33 +53,57 @@ jobs:
env: env:
SHELLCHECK_OPTS: -S warning -e SC1091 SHELLCHECK_OPTS: -S warning -e SC1091
python:
name: python ${{ matrix.python }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
# TrueNAS SCALE middleware runs 3.11+; keep the patch importable across
# the versions it may be injected into.
python: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
# uv-managed interpreters instead of actions/setup-python: the prebuilt-CPython # uv-managed interpreters instead of actions/setup-python: the prebuilt-CPython
# download path setup-python relies on does not work on the self-hosted Gitea # download path setup-python relies on does not work on the self-hosted Gitea
# runner (all three matrix jobs failed at setup there while passing on GitHub); # runner (all three matrix jobs failed at setup there while passing on GitHub);
# uv works identically on both. # uv works identically on both.
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 #
# Installed by a plain `run:` step rather than astral-sh/setup-uv: one fewer
# action is one fewer thing to go wrong, and the action was only ever
# fetching a binary -- the interpreter is chosen per command by `uvx
# --python`, never by the action.
#
# Pinned for the same reason ruff is pinned below: an unpinned uv means any
# upstream release can turn main red with no code change here.
- name: install uv
env:
UV_VERSION: "0.11.21"
run: |
curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: ruff - name: ruff
# Pinned: an unpinned ruff means any upstream release can turn main red # Pinned: an unpinned ruff means any upstream release can turn main red
# with no code change. # with no code change.
run: uvx ruff@0.16.1 check patch tests tools run: uvx ruff@0.16.1 check patch tests tools
# TrueNAS SCALE middleware runs 3.11+; keep the patch importable across the
# versions it may be injected into. Every version runs even after one
# fails -- that is what `fail-fast: false` bought when this was a matrix,
# and losing it would mean a 3.11 break hides whether 3.12 and 3.13 are
# fine, which is exactly the information you want at that moment.
- name: pytest - name: pytest
run: uvx --python ${{ matrix.python }} pytest tests -v env:
PYTHONS: "3.11 3.12 3.13"
run: |
fail=0
for v in $PYTHONS; do
echo "::group::pytest on python $v"
uvx --python "$v" pytest tests -v \
|| { echo "::error::suite failed on python $v"; fail=1; }
echo "::endgroup::"
done
exit $fail
- name: verify injected middleware blocks compile - name: verify injected middleware blocks compile
# Belt-and-braces: the *_BLOCK strings are appended into live middlewared # Belt-and-braces: the *_BLOCK strings are appended into live middlewared
# modules. A syntax error there would break the box at boot. # modules. A syntax error there would break the box at boot.
run: uvx --python ${{ matrix.python }} pytest tests/test_apply_blocks.py -v env:
PYTHONS: "3.11 3.12 3.13"
run: |
fail=0
for v in $PYTHONS; do
uvx --python "$v" pytest tests/test_apply_blocks.py -v \
|| { echo "::error::injected blocks failed to compile on python $v"; fail=1; }
done
exit $fail
+28
View File
@@ -39,6 +39,34 @@ worse than no alert, because one day it carries a security fix.
### Fixed ### Fixed
- **CI turned `main` red on two of three pushes without running a single test,
then stopped running at all.** Two failures, one cause: four concurrent jobs on
one self-hosted runner.
`act`, the engine behind the Gitea runner, caches each *action* as one shared
git clone under `/root/.cache/act/<hash>` and re-pulls it per job. Jobs
starting in the same second fight over that directory, and the loser dies with
`lstat /root/.cache/act/<hash>/.npmrc: no such file or directory` — before any
suite output exists, with a different victim each push (3.12 on one, 3.11 on
the next). Separately, the runner force-pulls its base image per job, so four
jobs meant four anonymous Docker Hub pulls per push; a few pushes and re-runs
in one afternoon hit `429 Too Many Requests` and *every* job began failing
before it started — including the shell job, which nothing had touched.
CI is now a single job. Dropping one action (uv is installed by a `run:` step
rather than `astral-sh/setup-uv`, which was only ever fetching a binary) only
shrank the surface, because every job still used `actions/checkout`.
Concurrency is the actual ingredient, so removing it removes the whole class:
one job cannot race itself whatever actions it uses, and one job is one pull.
The Python sweep moved inside that job and still runs every version after one
fails — that is what `fail-fast: false` bought, and losing it would mean a 3.11
break hides whether 3.12 and 3.13 are fine. The cost is wall-clock
parallelism, which this repo does not need: the suite is ~1.5s, so container
start and interpreter downloads dominate either way.
A red gate that is usually noise is worse than no gate, because the one time
it means something, nobody looks.
- **The patch survived being applied and then silently stopped existing, because - **The patch survived being applied and then silently stopped existing, because
something else remounted `/usr` four seconds later.** On a box running something else remounted `/usr` four seconds later.** On a box running
TrueNAS 25.10.6 the boot of 2026-08-19 went: 16:41:56 `apply.sh` mounts its TrueNAS 25.10.6 the boot of 2026-08-19 went: 16:41:56 `apply.sh` mounts its
+2 -3
View File
@@ -61,10 +61,9 @@ If something is wrong, the reason is in `apply.log` — start at
| --- | --- | --- | --- | | --- | --- | --- | --- |
| 24.10.2.4 | ok | ok | — | | 24.10.2.4 | ok | ok | — |
| 25.04.2.6 | ok | ok | — | | 25.04.2.6 | ok | ok | — |
| 25.10.5 | ok | ok | — | | 25.10.7 | ok | ok | — |
| 24.10.2.5 _(unreleased)_ | ok | ok | — | | 24.10.2.5 _(unreleased)_ | ok | ok | — |
| 25.10.6 _(unreleased)_ | ok | ok | — | | 26.0.0-RC.1 _(unreleased)_ | ok | ok | — |
| 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — |
| master _(27-dev)_ | **BROKEN** | **BROKEN** | — | | master _(27-dev)_ | **BROKEN** | **BROKEN** | — |
| verdict | meaning | | verdict | meaning |
+78
View File
@@ -172,3 +172,81 @@ class TestCompatCannotSilentlyPass:
with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh: with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
src = fh.read() src = fh.read()
assert "steps.check.outputs.shipped_broken != '0'" in src assert "steps.check.outputs.shipped_broken != '0'" in src
class TestActionCacheRace:
"""CI must not run concurrent jobs on the self-hosted runner.
`act` caches each ACTION as one shared clone under /root/.cache/act/<hash>
and re-pulls it per job, so jobs starting together fight over that directory
and the loser dies with `lstat .../<file>: no such file or directory` before
any test runs -- a red `main` with zero suite output and a different victim
each push. The runner also force-pulls its base image per job, so job count
is also Docker Hub pull count, and four-per-push exhausted the anonymous
limit in an afternoon. Both problems have the same cure: one job.
"""
def _ci(self):
with open(os.path.join(WORKFLOWS, "ci.yml"), encoding="utf-8") as fh:
return fh.read()
def test_ci_runs_as_exactly_one_job(self):
"""The fix is the absence of concurrency, not the absence of one action.
Dropping astral-sh/setup-uv only shrank the surface -- every job still
used actions/checkout. A single job cannot race itself whatever actions
it uses, which is why this, and not the action count, is the invariant.
"""
ci = self._ci()
# Scope to the jobs: block -- `on:` has two-space keys of its own
# (push/pull_request/workflow_dispatch) that look identical otherwise.
body = ci[ci.index("\njobs:"):]
jobs = re.findall(r"^ (\w[\w-]*):$", body, re.M)
assert len(jobs) == 1, (
f"ci.yml defines {len(jobs)} jobs ({jobs}); concurrent jobs on the "
"self-hosted runner race on act's shared action cache and multiply "
"Docker Hub pulls. Keep CI to one job."
)
def test_no_matrix_reintroduces_parallel_jobs(self):
ci = self._ci()
assert "strategy:" not in ci and "matrix:" not in ci, (
"a matrix fans out into concurrent jobs again -- sweep versions "
"inside one job instead"
)
def test_every_python_version_still_runs_after_one_fails(self):
"""`fail-fast: false` is what the loop has to preserve.
A 3.11 break must not hide whether 3.12 and 3.13 are fine; that is
precisely the information you want at that moment.
"""
ci = self._ci()
assert 'PYTHONS: "3.11 3.12 3.13"' in ci
assert ci.count("fail=1") >= 2, "the sweeps must collect failures, not exit early"
def test_uv_is_installed_without_an_action(self):
"""Checks `uses:` directives, not prose.
The comment in ci.yml names the action it deliberately avoids, and that
explanation is the most useful thing in the file -- a test that greps the
raw text would forbid documenting the very lesson it enforces. Parsed
with a regex rather than PyYAML on purpose: CI runs `uvx pytest`, whose
environment holds pytest and nothing else, so a third-party import here
fails on the runner while passing locally.
"""
ci = self._ci()
used = re.findall(r"^\s*-?\s*uses:\s*(\S+)", ci, re.M)
assert not [u for u in used if "setup-uv" in u], (
"the action was only fetching a binary; a run: step does the same "
"with one less moving part"
)
assert "astral.sh/uv/" in ci
def test_the_uv_version_is_pinned(self):
ci = self._ci()
assert re.search(r'UV_VERSION:\s*"\d+\.\d+\.\d+"', ci), (
"an unpinned uv lets any upstream release turn main red with no "
"code change here -- the same rule ruff is pinned under"
)
assert "https://astral.sh/uv/${UV_VERSION}/install.sh" in ci