docs: refresh the TrueNAS compatibility matrix

Two CI failures shared one cause: four jobs starting together on the
self-hosted runner.

act caches each action as one shared clone under /root/.cache/act/<hash> and
re-pulls it per job, so concurrent jobs fight over that directory and the
loser dies with "lstat .../<file>: no such file or directory" before any test
runs — a different victim each push. And the runner force-pulls its base image
per job, so four jobs meant four anonymous Docker Hub pulls per push; that hit
429 Too Many Requests and every job started failing before it began, including
the shell job nothing had touched.

Installing uv without an action only shrank the surface, since every job still
used actions/checkout. Concurrency is the ingredient, so this removes it: one
job cannot race itself whatever actions it uses, and one job is one pull. The
version sweep moves inside the job and still runs every version after one
fails, preserving what fail-fast: false bought.

.github/workflows/ci.yml

@@ -9,9 +9,31 @@ on:
 permissions:
   contents: read
 
+# ONE job, deliberately. This was four (shell + a 3-way python matrix) and they
+# started within the same second on the self-hosted Gitea runner, which is what
+# made CI unreliable in two separate ways:
+#
+#   1. The act action-cache race. `act` caches each ACTION as a single shared git
+#      clone under /root/.cache/act/<hash> and re-pulls it per job, so concurrent
+#      jobs using the same action fight over that directory and the loser dies
+#      with `lstat /root/.cache/act/<hash>/<file>: no such file or directory` --
+#      a red `main` with zero suite output, and a different victim each push
+#      (3.12 on one, 3.11 on the next). Dropping one action only shrank the
+#      surface: every job still used actions/checkout. Concurrency is the actual
+#      ingredient, so removing it removes the whole class -- a single job cannot
+#      race itself, no matter which actions it uses.
+#
+#   2. Docker Hub 429s. The runner force-pulls its base image per job, so four
+#      jobs meant four anonymous pulls per push. A few pushes and re-runs in an
+#      afternoon exhausted the anonymous limit and every job failed before it
+#      started -- including the shell job, which nothing had touched. One job is
+#      one pull.
+#
+# The cost is wall-clock parallelism, and this repo does not need it: the suite
+# is ~1.5s, so container start and interpreter downloads dominate either way.
 jobs:
-  shell:
-    name: shell (shellcheck + syntax)
+  ci:
+    name: ci (shell + python 3.11-3.13)
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -31,33 +53,57 @@ jobs:
         env:
           SHELLCHECK_OPTS: -S warning -e SC1091
 
-  python:
-    name: python ${{ matrix.python }}
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        # TrueNAS SCALE middleware runs 3.11+; keep the patch importable across
-        # the versions it may be injected into.
-        python: ["3.11", "3.12", "3.13"]
-    steps:
-      - uses: actions/checkout@v4
-
       # uv-managed interpreters instead of actions/setup-python: the prebuilt-CPython
       # download path setup-python relies on does not work on the self-hosted Gitea
       # runner (all three matrix jobs failed at setup there while passing on GitHub);
       # uv works identically on both.
-      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+      #
+      # Installed by a plain `run:` step rather than astral-sh/setup-uv: one fewer
+      # action is one fewer thing to go wrong, and the action was only ever
+      # fetching a binary -- the interpreter is chosen per command by `uvx
+      # --python`, never by the action.
+      #
+      # Pinned for the same reason ruff is pinned below: an unpinned uv means any
+      # upstream release can turn main red with no code change here.
+      - name: install uv
+        env:
+          UV_VERSION: "0.11.21"
+        run: |
+          curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | sh
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
 
       - name: ruff
         # Pinned: an unpinned ruff means any upstream release can turn main red
         # with no code change.
         run: uvx ruff@0.16.1 check patch tests tools
 
+      # TrueNAS SCALE middleware runs 3.11+; keep the patch importable across the
+      # versions it may be injected into. Every version runs even after one
+      # fails -- that is what `fail-fast: false` bought when this was a matrix,
+      # and losing it would mean a 3.11 break hides whether 3.12 and 3.13 are
+      # fine, which is exactly the information you want at that moment.
       - name: pytest
-        run: uvx --python ${{ matrix.python }} pytest tests -v
+        env:
+          PYTHONS: "3.11 3.12 3.13"
+        run: |
+          fail=0
+          for v in $PYTHONS; do
+            echo "::group::pytest on python $v"
+            uvx --python "$v" pytest tests -v \
+              || { echo "::error::suite failed on python $v"; fail=1; }
+            echo "::endgroup::"
+          done
+          exit $fail
 
       - name: verify injected middleware blocks compile
         # Belt-and-braces: the *_BLOCK strings are appended into live middlewared
         # modules. A syntax error there would break the box at boot.
-        run: uvx --python ${{ matrix.python }} pytest tests/test_apply_blocks.py -v
+        env:
+          PYTHONS: "3.11 3.12 3.13"
+        run: |
+          fail=0
+          for v in $PYTHONS; do
+            uvx --python "$v" pytest tests/test_apply_blocks.py -v \
+              || { echo "::error::injected blocks failed to compile on python $v"; fail=1; }
+          done
+          exit $fail

CHANGELOG.md

@@ -39,6 +39,34 @@ worse than no alert, because one day it carries a security fix.
 
 ### Fixed
 
+- **CI turned `main` red on two of three pushes without running a single test,
+  then stopped running at all.** Two failures, one cause: four concurrent jobs on
+  one self-hosted runner.
+
+  `act`, the engine behind the Gitea runner, caches each *action* as one shared
+  git clone under `/root/.cache/act/<hash>` and re-pulls it per job. Jobs
+  starting in the same second fight over that directory, and the loser dies with
+  `lstat /root/.cache/act/<hash>/.npmrc: no such file or directory` — before any
+  suite output exists, with a different victim each push (3.12 on one, 3.11 on
+  the next). Separately, the runner force-pulls its base image per job, so four
+  jobs meant four anonymous Docker Hub pulls per push; a few pushes and re-runs
+  in one afternoon hit `429 Too Many Requests` and *every* job began failing
+  before it started — including the shell job, which nothing had touched.
+
+  CI is now a single job. Dropping one action (uv is installed by a `run:` step
+  rather than `astral-sh/setup-uv`, which was only ever fetching a binary) only
+  shrank the surface, because every job still used `actions/checkout`.
+  Concurrency is the actual ingredient, so removing it removes the whole class:
+  one job cannot race itself whatever actions it uses, and one job is one pull.
+  The Python sweep moved inside that job and still runs every version after one
+  fails — that is what `fail-fast: false` bought, and losing it would mean a 3.11
+  break hides whether 3.12 and 3.13 are fine. The cost is wall-clock
+  parallelism, which this repo does not need: the suite is ~1.5s, so container
+  start and interpreter downloads dominate either way.
+
+  A red gate that is usually noise is worse than no gate, because the one time
+  it means something, nobody looks.
+
 - **The patch survived being applied and then silently stopped existing, because
   something else remounted `/usr` four seconds later.** On a box running
   TrueNAS 25.10.6 the boot of 2026-08-19 went: 16:41:56 `apply.sh` mounts its
@@ -76,7 +104,14 @@ worse than no alert, because one day it carries a security fix.
   straight back into the same race. So the overlay is re-mounted for the benefit
   of the next restart, and the question of whether *this* middlewared actually
   holds the patch is left to the one thing that can answer it exactly — the
-  in-process alert below.
+  in-process alert below. That re-mount preserves `hook_status.json`'s
+  `patched_at`: `create_task.py verify` decides "loaded" by comparing
+  middlewared's start time against that stamp, so a re-apply running *after* the
+  restart would have made the stamp newer than the process which correctly
+  imported the patch, and `verify` would have reported FAIL forever on every
+  boot where the sysext merge detaches the overlay. Caught on hardware while
+  validating the candidate — a new lying status introduced by the fix for a
+  lying status.
 
   Two supporting fixes fell out of the same failure. `_ensure_writable` treated
   "one of our overlays is listed on this directory" as "already done" — but it

README.md

@@ -61,10 +61,9 @@ If something is wrong, the reason is in `apply.log` — start at
 | --- | --- | --- | --- |
 | 24.10.2.4 | ok | ok | — |
 | 25.04.2.6 | ok | ok | — |
-| 25.10.5 | ok | ok | — |
+| 25.10.7 | ok | ok | — |
 | 24.10.2.5 _(unreleased)_ | ok | ok | — |
-| 25.10.6 _(unreleased)_ | ok | ok | — |
-| 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — |
+| 26.0.0-RC.1 _(unreleased)_ | ok | ok | — |
 | master _(27-dev)_ | **BROKEN** | **BROKEN** | — |
 
 | verdict | meaning |

patch/wait_restart.sh

@@ -143,7 +143,26 @@ else
     _log "nvidia sysext merge follows it) -- re-mounting for the next restart."
     _log "Whether THIS middlewared loaded the patch is answered in-process by"
     _log "the 'installed but NOT loaded' alert, not by this check."
+
+    # Preserve hook_status.json's patched_at across this re-mount.
+    #
+    # create_task.py verify decides "loaded" by comparing middlewared's start
+    # time against patched_at. This re-apply restores the SAME patch the boot
+    # pass already applied, but it runs *after* the restart -- so letting it
+    # re-stamp would make patched_at newer than the process that correctly
+    # imported the patch, and verify would report FAIL forever, on every boot
+    # where docker's sysext merge detaches the overlay. That is precisely the
+    # lying-status failure this release exists to remove, so do not introduce a
+    # new one. The snapshot lives in /run, never in the repo: a leftover file
+    # there would leave the tree dirty and update.sh refuses to run over that.
+    _saved_status=/run/truecloud-hook_status.pre
+    cp -p "$PATCH_DIR/hook_status.json" "$_saved_status" 2>/dev/null
+
     TRUECLOUD_REAPPLY=1 /bin/bash "$PATCH_DIR/patch/apply.sh"
+
+    if [ -f "$_saved_status" ]; then
+        mv -f "$_saved_status" "$PATCH_DIR/hook_status.json" 2>/dev/null
+    fi
 fi
 
 _log "=== deferred restart complete ==="

tests/test_wait_restart.py

@@ -172,3 +172,28 @@ def test_mount_retries_on_a_private_workdir():
     body = src[start:end]
     assert body.count("mount -t overlay") == 2, "expected a retry mount"
     assert 'work="/run/truecloud-${tag}-work.$$"' in body
+
+
+def test_post_restart_remount_preserves_the_patched_at_stamp():
+    """create_task.py verify compares middlewared's start time to patched_at.
+
+    The post-restart re-mount restores the same patch the boot pass applied, so
+    letting apply.sh re-stamp would make patched_at newer than the process that
+    correctly imported it -- verify would then report FAIL forever on every boot
+    where docker's sysext merge detaches the overlay.
+    """
+    src = wait_restart_source()
+    tail = src[src.index("systemctl try-restart middlewared"):]
+    assert "hook_status.json" in tail
+    save = tail.index("/run/truecloud-hook_status.pre")
+    reapply = tail.index("TRUECLOUD_REAPPLY=1")
+    restore = tail.rindex("hook_status.json")
+    assert save < reapply < restore, "snapshot must bracket the re-apply"
+
+
+def test_status_snapshot_is_not_written_into_the_repo():
+    """A leftover file in the repo dir leaves the tree dirty, and update.sh
+    refuses to run over a dirty tree -- that once made the patch un-updatable."""
+    src = wait_restart_source()
+    assert "/run/truecloud-hook_status.pre" in src
+    assert '"$PATCH_DIR/hook_status.json.pre' not in src

tests/test_workflows.py

@@ -172,3 +172,81 @@ class TestCompatCannotSilentlyPass:
         with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
             src = fh.read()
         assert "steps.check.outputs.shipped_broken != '0'" in src
+
+
+class TestActionCacheRace:
+    """CI must not run concurrent jobs on the self-hosted runner.
+
+    `act` caches each ACTION as one shared clone under /root/.cache/act/<hash>
+    and re-pulls it per job, so jobs starting together fight over that directory
+    and the loser dies with `lstat .../<file>: no such file or directory` before
+    any test runs -- a red `main` with zero suite output and a different victim
+    each push. The runner also force-pulls its base image per job, so job count
+    is also Docker Hub pull count, and four-per-push exhausted the anonymous
+    limit in an afternoon. Both problems have the same cure: one job.
+    """
+
+    def _ci(self):
+        with open(os.path.join(WORKFLOWS, "ci.yml"), encoding="utf-8") as fh:
+            return fh.read()
+
+    def test_ci_runs_as_exactly_one_job(self):
+        """The fix is the absence of concurrency, not the absence of one action.
+
+        Dropping astral-sh/setup-uv only shrank the surface -- every job still
+        used actions/checkout. A single job cannot race itself whatever actions
+        it uses, which is why this, and not the action count, is the invariant.
+        """
+        ci = self._ci()
+        # Scope to the jobs: block -- `on:` has two-space keys of its own
+        # (push/pull_request/workflow_dispatch) that look identical otherwise.
+        body = ci[ci.index("\njobs:"):]
+        jobs = re.findall(r"^  (\w[\w-]*):$", body, re.M)
+        assert len(jobs) == 1, (
+            f"ci.yml defines {len(jobs)} jobs ({jobs}); concurrent jobs on the "
+            "self-hosted runner race on act's shared action cache and multiply "
+            "Docker Hub pulls. Keep CI to one job."
+        )
+
+    def test_no_matrix_reintroduces_parallel_jobs(self):
+        ci = self._ci()
+        assert "strategy:" not in ci and "matrix:" not in ci, (
+            "a matrix fans out into concurrent jobs again -- sweep versions "
+            "inside one job instead"
+        )
+
+    def test_every_python_version_still_runs_after_one_fails(self):
+        """`fail-fast: false` is what the loop has to preserve.
+
+        A 3.11 break must not hide whether 3.12 and 3.13 are fine; that is
+        precisely the information you want at that moment.
+        """
+        ci = self._ci()
+        assert 'PYTHONS: "3.11 3.12 3.13"' in ci
+        assert ci.count("fail=1") >= 2, "the sweeps must collect failures, not exit early"
+
+    def test_uv_is_installed_without_an_action(self):
+        """Checks `uses:` directives, not prose.
+
+        The comment in ci.yml names the action it deliberately avoids, and that
+        explanation is the most useful thing in the file -- a test that greps the
+        raw text would forbid documenting the very lesson it enforces. Parsed
+        with a regex rather than PyYAML on purpose: CI runs `uvx pytest`, whose
+        environment holds pytest and nothing else, so a third-party import here
+        fails on the runner while passing locally.
+        """
+        ci = self._ci()
+        used = re.findall(r"^\s*-?\s*uses:\s*(\S+)", ci, re.M)
+        assert not [u for u in used if "setup-uv" in u], (
+            "the action was only fetching a binary; a run: step does the same "
+            "with one less moving part"
+        )
+        assert "astral.sh/uv/" in ci
+
+    def test_the_uv_version_is_pinned(self):
+        ci = self._ci()
+        assert re.search(r'UV_VERSION:\s*"\d+\.\d+\.\d+"', ci), (
+            "an unpinned uv lets any upstream release turn main red with no "
+            "code change here -- the same rule ruff is pinned under"
+        )
+        assert "https://astral.sh/uv/${UV_VERSION}/install.sh" in ci