The matrix bot would never have worked: wrong permissions, wrong forge

Two bugs, both of which would have failed silently on the first scheduled run:

- permissions were  while the step pushes a branch and opens a PR.
  It would have died with a 403 and I would have had a bot that never worked.
- it opened the PR on GITHUB, which is a one-way MIRROR. A PR merged there would be
  clobbered by the next fleet-repos mirror push from Gitea. A bot opening PRs against
  a mirror is a bot doing nothing, slowly.

Now: contents+pull-requests write, and the PR is opened on Gitea (canonical) via its
API. One long-lived PR, force-pushed in place -- a daily PR is the same mistake as a
daily comment, wearing a hat.

.github/workflows/compat.yml

@@ -27,7 +27,10 @@ on:
       - ".github/workflows/compat.yml"
 
 permissions:
-  contents: read
+  # write, because the matrix refresh pushes a branch and opens a PR. It does NOT get
+  # to move `main` -- that is the whole reason it is a PR. See the refresh step below.
+  contents: write
+  pull-requests: write
   issues: write
 
 jobs:
@@ -130,17 +133,27 @@ jobs:
       - name: matrix
         run: cat /tmp/matrix.md
 
-      # Keep the README's table true. A support matrix that quietly goes stale is not
+      # Keep the README's table true — as a PULL REQUEST, on the CANONICAL forge.
-      # a stale doc -- it is a false promise to somebody deciding whether to trust
-      # this with their backups.
       #
-      # Only ever touches the block between the COMPAT MATRIX markers, and only on
+      # Two things this gets right that the obvious version gets wrong:
-      # the canonical host (Gitea) so the two forges cannot race each other. The
+      #
-      # `paths:` trigger above does not include README.md, so this cannot re-trigger
+      #  1. It is a PR, not a push to main. This used to `git push origin HEAD:main`
-      # itself; and a README change is documentation-only, which by design raises no
+      #     from CI. An unattended write to main is exactly what the release barrier
-      # update alert on anyone's box.
+      #     exists to prevent — a bot that can move main can move it somewhere nobody
-      - name: refresh the README matrix
+      #     looked. Nothing lands by itself.
+      #
+      #  2. It runs on GITEA, not GitHub. GitHub is a one-way MIRROR: a PR merged there
+      #     would be silently clobbered by the next `fleet-repos mirror` push from Gitea.
+      #     A bot opening PRs against a mirror is a bot doing nothing, slowly.
+      #
+      # A stale support matrix is not a stale doc — it is a false promise to somebody
+      # deciding whether to trust this with their backups. So it is refreshed daily; it
+      # just asks first.
+      - name: refresh the README matrix (PR on the canonical forge)
         if: ${{ github.event_name == 'schedule' && !contains(github.server_url, 'github.com') }}
+        env:
+          TOKEN: ${{ secrets.GITEA_TOKEN || github.token }}
+          API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
         run: |
           python3 - <<'PY'
           import json, sys
@@ -151,87 +164,73 @@ jobs:
           print("changed" if compat.update_readme(rows) else "unchanged")
           PY
 
-          if ! git diff --quiet -- README.md; then
+          git diff --quiet -- README.md && { echo "matrix unchanged — nothing to propose"; exit 0; }
-            git config user.name  "truecloud-patch bot"
-            git config user.email "bot@onetick.ninja"
-            git add README.md
-            git commit -m "docs: refresh the TrueNAS compatibility matrix"
-            git push origin HEAD:main
-          fi
 
-      # A broken SHIPPED release is an outage: users are on it right now.
+          git config user.name  "truecloud-patch bot"
+          git config user.email "bot@onetick.ninja"
+
+          BRANCH=bot/compat-matrix
+          git checkout -B "$BRANCH"
+          git add README.md
+          git commit -m "docs: refresh the TrueNAS compatibility matrix"
+          git push -f origin "$BRANCH"
+
+          # ONE long-lived PR, force-pushed in place — not a new one every morning.
+          # (A daily PR is the same mistake as a daily comment, wearing a hat.)
+          BRANCH="$BRANCH" python3 - <<'PY'
+          import json, os, urllib.error, urllib.request
+
+          api, token, branch = os.environ["API"], os.environ["TOKEN"], os.environ["BRANCH"]
+          h = {"Authorization": f"token {token}", "Content-Type": "application/json"}
+
+          def call(url, method="GET", data=None):
+              r = urllib.request.Request(
+                  url, method=method, headers=h,
+                  data=json.dumps(data).encode() if data else None)
+              with urllib.request.urlopen(r) as resp:  # noqa: S310
+                  return json.load(resp) if resp.length != 0 else {}
+
+          existing = [
+              p for p in call(f"{api}/pulls?state=open")
+              if p["head"]["ref"] == branch
+          ]
+          if existing:
+              print(f"PR #{existing[0]['number']} already open; the force-push updated it")
+          else:
+              pr = call(f"{api}/pulls", "POST", {
+                  "head": branch, "base": "main",
+                  "title": "docs: refresh the TrueNAS compatibility matrix",
+                  "body": (
+                      "The daily compatibility check found that the support matrix in "
+                      "the README no longer matches iXsystems' actual middleware.\n\n"
+                      "This only touches the block between the `COMPAT MATRIX` markers. "
+                      "It is regenerated by `tools/compat.py --matrix --update-readme` "
+                      "and force-pushed, so it always reflects the latest run."
+                  ),
+              })
+              print(f"opened PR #{pr['number']}")
+          PY
+
+      # ONE bug report, kept in sync. It is edited in place when the findings change and
+      # says NOTHING when they do not.
+      #
+      # The first version commented on every run and left 11 identical 3,000-character
+      # comments on one issue in a single day. A bot that repeats itself daily gets
+      # muted, and then the next real finding is scrolled past — which defeats the whole
+      # reason for building it.
+      #
+      # Runs on whichever forge it lands on; compat_publish.py handles both, so the two
+      # cannot drift.
+      - name: file / update / close the bug report
+        env:
+          TOKEN: ${{ secrets.GITEA_TOKEN || github.token }}
+          API: ${{ contains(github.server_url, 'github.com') && 'https://api.github.com' || format('{0}/api/v1', github.server_url) }}/repos/${{ github.repository }}
+        run: python3 tools/compat_publish.py --api "$API" --token "$TOKEN" --matrix /tmp/matrix.json
+
+      # A broken SHIPPED release is an outage: users are on it right now. Fails LAST, so
+      # the report is filed before the job goes red.
       - name: fail if a shipped release is broken
         if: ${{ steps.check.outputs.shipped_broken != '0' }}
         run: |
           echo "::error::The patch is broken on a SHIPPED TrueNAS release."
           exit 1
-
-      - name: file a bug report (GitHub)
-        if: ${{ steps.report.outputs.broken == '1' && contains(github.server_url, 'github.com') }}
-        env:
-          GH_TOKEN: ${{ github.token }}
-          TITLE: "TrueNAS compatibility: the patch's assumptions no longer hold"
-        run: |
-          # One issue per set of broken refs, reopened/updated rather than duplicated
-          # daily -- a bot that files the same issue every morning gets muted, and
-          # then it is not a warning system any more.
-          # Lowest-numbered match, for the same reason as the Gitea step below.
-          existing="$(gh issue list --state all --search "$TITLE" \
-                        --json number,title \
-                        --jq '[.[] | select(.title == env.TITLE) | .number] | min // empty')"
-
-          if [ -n "$existing" ]; then
-            gh issue comment "$existing" --body-file /tmp/issue.md
-            gh issue reopen "$existing" 2>/dev/null || true
-          else
-            gh issue create --title "$TITLE" --body-file /tmp/issue.md
-          fi
-
-      - name: file a bug report (Gitea)
-        if: ${{ steps.report.outputs.broken == '1' && !contains(github.server_url, 'github.com') }}
-        env:
-          TOKEN: ${{ secrets.GITEA_TOKEN || github.token }}
-          API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
-          TITLE: "TrueNAS compatibility: the patch's assumptions no longer hold"
-        run: |
-          # python3, not jq: jq is not guaranteed on a self-hosted runner, and a bug
-          # report that dies on a missing tool is a warning system that does not warn.
-          python3 - <<'PY'
-          import json, os, urllib.error, urllib.request
-
-          api, token, title = os.environ["API"], os.environ["TOKEN"], os.environ["TITLE"]
-          with open("/tmp/issue.md", encoding="utf-8") as fh:
-              body = fh.read()
-          headers = {"Authorization": f"token {token}",
-                     "Content-Type": "application/json"}
-
-          def call(url, method, data=None):
-              req = urllib.request.Request(
-                  url, method=method, headers=headers,
-                  data=json.dumps(data).encode() if data else None)
-              with urllib.request.urlopen(req) as r:  # noqa: S310
-                  return json.load(r) if r.length != 0 else {}
-
-          # Same title => same issue. Comment on it rather than filing a new one every
-          # morning: a bot that duplicates itself daily gets muted, and then it is not
-          # a warning system any more.
-          # LOWEST-numbered match, not "whichever the API returns first". Two issues
-          # with the same title already existed once (the old title embedded the ref
-          # list, so the identity changed when that set changed), and an
-          # order-dependent pick would have alternated between them, reopening one and
-          # commenting on the other. Lowest number is stable no matter what the API
-          # sorts by.
-          issues = call(f"{api}/issues?state=all&type=issues", "GET")
-          matches = sorted((i for i in issues if i["title"] == title),
-                           key=lambda i: i["number"])
-          match = matches[0] if matches else None
-
-          if match:
-              n = match["number"]
-              call(f"{api}/issues/{n}/comments", "POST", {"body": body})
-              call(f"{api}/issues/{n}", "PATCH", {"state": "open"})
-              print(f"commented on and reopened issue #{n}")
-          else:
-              made = call(f"{api}/issues", "POST", {"title": title, "body": body})
-              print(f"filed issue #{made['number']}")
-          PY

CHANGELOG.md

@@ -6,6 +6,35 @@ is deliberate: see [Releasing](docs/releasing.md). Twelve releases were cut on
 live, every one of those interrupts every user. An alert people learn to ignore is
 worse than no alert, because one day it carries a security fix.
 
+## v0.6.1 — 2026-07-13
+### Fixed
+
+- **A reboot mid-backup orphaned the entire snapshot tree, permanently.** The sidecar
+  is the record of which snapshots a run pinned — and it lives in `/run`, which is
+  **tmpfs**. A reboot (or a crash) between taking the recursive snapshot and cleaning
+  it up destroyed that record, leaving one snapshot per descendant dataset — **250+ on
+  a real pool** — with nothing left pointing at them. Nothing would ever have found
+  them again.
+
+  `gc_stale_snapshots()` is the backstop: it identifies leftovers **by name**, so it
+  works when the record is gone. It runs at the start of every backup, after the
+  sidecar reclaim — the recorded path stays authoritative, and the collector only ever
+  mops up what the record lost.
+
+  Because it deletes data on a *name match* — a weaker claim than a recorded fact — the
+  selection is a **pure function** with the harshest tests in the suite. A snapshot is
+  collected only if **all** of these hold:
+
+  | | |
+  | --- | --- |
+  | name is exactly `<dataset>@<task>-<YYYYMMDDHHMMSS>` | so `cloud_backup-5` never matches `cloud_backup-50`, an `auto-*` periodic snapshot, or anything a human made |
+  | it is not the current run's | parent *and* children are excluded |
+  | **nothing is mounted from it** | an in-flight run pins its own snapshots — this, not the age guard, is what protects a concurrent backup |
+  | it is **over an hour old** | covers the seconds-long window where a live run has snapshotted but not yet mounted |
+
+  Verified against the real pool: of **4,728** snapshots — including **2,341** periodic
+  ones — it selects exactly the orphans of the task being run, and nothing else.
+
 ## v0.6.0 — 2026-07-13
 ### Added
 

docs/nested-snapshots.md

@@ -95,6 +95,21 @@ re-scan each time.
 
 ### Snapshot lifecycle
 
+> **Two mechanisms clean up, and the second exists because the first can be destroyed.**
+>
+> 1. **The sidecar** records exactly which snapshots a run pinned, and is removed only
+>    on a confirmed-clean sweep. Precise, and it survives a middlewared restart.
+> 2. **The garbage collector** finds leftovers by *name*, so it still works when the
+>    sidecar is gone — and it can be: **the sidecar lives in `/run`, which is tmpfs.** A
+>    reboot mid-backup takes it, and with it the only record of a 250-snapshot tree.
+>
+> The collector runs at the start of every backup, after the sidecar reclaim. It will
+> only touch a snapshot named `<dataset>@<task>-<timestamp>` that is not the current
+> run's, has **nothing mounted from it** (which is what protects a concurrently-running
+> backup), and is **over an hour old**. Periodic `auto-*` snapshots, other tasks'
+> snapshots, and anything you made by hand are structurally out of reach.
+
+
 > **A snapshot may survive a run, and that is expected.** ZFS **automounts**
 > `<dataset>/.zfs/snapshot/<snap>` the moment it is read, and holds it for
 > `zfs_expire_snapshot` seconds (**300** by default) after the last access. So

install.sh

@@ -18,7 +18,7 @@
 
 set -euo pipefail
 
-VERSION="0.6.0"
+VERSION="0.6.1"
 
 # The directory containing install.sh is the permanent install location.
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"

patch/apply.sh

@@ -32,7 +32,7 @@
 # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
 PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 LOG="$PATCH_DIR/apply.log"
-VERSION="0.6.0"
+VERSION="0.6.1"
 
 # Rotate log at 512 KB to avoid unbounded growth on a system volume.
 # Keep two prior generations (.1 and .2) so the last three boots are always available.

patch/create_task.py

@@ -52,7 +52,7 @@ import subprocess
 import sys
 import time
 
-__version__ = "0.6.0"
+__version__ = "0.6.1"
 
 _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")

patch/truecloud_nested.py

@@ -55,6 +55,7 @@ Therefore this module owns the whole lifecycle:
 from __future__ import annotations
 
 import contextlib
+import datetime
 import os
 import stat
 import subprocess
@@ -68,10 +69,13 @@ __all__ = [
     "cleanup_task",
     "current_mounts_under",
     "delete_snapshot_tree",
+    "gc_stale_snapshots",
+    "mounted_snapshots",
     "plan_staging",
     "sidecar_for",
     "snapshot_tree_names",
     "stage_nested",
+    "stale_snapshot_names",
     "staging_root_for",
     "teardown",
     "verify_staged",
@@ -176,6 +180,76 @@ def snapshot_tree_names(snapshot: str, all_names) -> list[str]:
     ]
 
 
+#: A snapshot must be at least this old before the garbage collector will touch it.
+#:
+#: The GC identifies our leftovers by NAME, so its only real risk is deleting a
+#: snapshot belonging to a run that is still starting up -- the window between
+#: `zfs snapshot -r` and the bind mounts appearing, which is seconds. An hour is three
+#: orders of magnitude more slack than that window needs, and still reclaims a lost
+#: tree on the very next daily run.
+GC_MIN_AGE_SECONDS = 3600
+
+
+def stale_snapshot_names(task_name, current_snapshot, all_names, now,
+                         in_use=(), min_age=GC_MIN_AGE_SECONDS):
+    """Snapshots THIS task created in an earlier run and never cleaned up.
+
+    Pure, because this is the one function here that DELETES DATA on a name match, and
+    a name match is a weaker claim than a recorded fact. Everything it relies on is an
+    argument, so every way it could be wrong is a test.
+
+    Why a garbage collector exists at all, when there is already a sidecar: **the
+    sidecar lives in /run, which is tmpfs.** A reboot mid-backup destroys it, and with
+    it the only record of a 250-snapshot tree. The sidecar handles the normal case
+    precisely; this handles the case where the record itself is gone.
+
+    A snapshot is ours to collect only if ALL of these hold:
+
+      * its name is exactly ``<dataset>@<task_name>-<YYYYMMDDHHMMSS>`` -- so
+        ``cloud_backup-5`` never matches ``cloud_backup-50``'s snapshots, and never
+        matches a periodic ``auto-2026-…`` or anything a human made;
+      * it is not the snapshot the current run is using;
+      * nothing is mounted from it (`in_use`) -- an in-flight run pins its own
+        snapshots, so this alone protects a concurrent one-time backup;
+      * it is older than `min_age` -- which covers the seconds-long window in which a
+        run has taken its snapshot but not yet mounted it.
+
+    `now` is a timezone-aware datetime; timestamps in the name are UTC (stock builds
+    them with `utc_now()`).
+    """
+    prefix = task_name + "-"
+    stale = []
+
+    for name in all_names:
+        _dataset, _, snapname = name.partition("@")
+        if not snapname or not snapname.startswith(prefix):
+            continue
+        if name == current_snapshot or snapname == _snapname_of(current_snapshot):
+            continue
+        if name in in_use:
+            continue
+
+        stamp = snapname[len(prefix):]
+        try:
+            when = datetime.datetime.strptime(stamp, "%Y%m%d%H%M%S").replace(
+                tzinfo=datetime.UTC
+            )
+        except ValueError:
+            # Not our timestamp format. Something else owns this name; leave it alone.
+            continue
+
+        if (now - when).total_seconds() < min_age:
+            continue
+
+        stale.append(name)
+
+    return stale
+
+
+def _snapname_of(snapshot):
+    return snapshot.partition("@")[2] if snapshot else ""
+
+
 def _probe_snapdir(path):
     """Classify a snapshot directory: ``ok``, ``missing``, or why it is unusable.
 
@@ -598,6 +672,80 @@ def delete_snapshot_tree(middleware, snapshot, logger=None, attempts=4,
     return remaining
 
 
+def mounted_snapshots(mounts_file="/proc/self/mounts"):
+    """Every ZFS snapshot something is currently mounted from.
+
+    The device field of a snapshot mount IS the snapshot name (`Tap/apps/x@snap`), for
+    both our staging bind mounts and ZFS's own .zfs automounts. So this is a direct,
+    factual answer to "is anything using this snapshot right now" -- which is what
+    protects a concurrently-running backup from the garbage collector, rather than
+    trusting an age heuristic to be generous enough.
+    """
+    live = set()
+    try:
+        with open(mounts_file, encoding="utf-8") as fh:
+            for line in fh:
+                dev = line.split(" ", 1)[0]
+                if "@" in dev:
+                    live.add(dev.replace("\\040", " "))
+    except OSError:
+        return set()
+    return live
+
+
+def gc_stale_snapshots(middleware, task_name, current_snapshot, logger=None,
+                       now=None, mounts_file="/proc/self/mounts"):
+    """Delete snapshots this task left behind in an earlier run. Returns what remains.
+
+    The backstop for when the RECORD is gone, not just the snapshots: the sidecar lives
+    in /run (tmpfs), so a reboot mid-backup takes it with them. Without this, that tree
+    -- one snapshot per descendant dataset, 250+ on a real pool -- is orphaned with
+    nothing left pointing at it.
+
+    Selection is `stale_snapshot_names()`, which is pure and heavily tested, because a
+    name match is a weaker claim than a recorded fact and this deletes data on one.
+    """
+    dataset = current_snapshot.partition("@")[0]
+    now = now or datetime.datetime.now(datetime.UTC)
+
+    try:
+        snaps = middleware.call_sync(
+            "zfs.snapshot.query", [["name", "^", dataset]], {"select": ["name"]}
+        )
+    except Exception as e:  # noqa: BLE001 - cannot enumerate; collect nothing
+        if logger:
+            logger.warning(
+                "truecloud-patch: could not enumerate snapshots for GC: %r", e
+            )
+        return []
+
+    stale = stale_snapshot_names(
+        task_name, current_snapshot, [s["name"] for s in snaps], now,
+        in_use=mounted_snapshots(mounts_file),
+    )
+    if not stale:
+        return []
+
+    if logger:
+        logger.warning(
+            "truecloud-patch: %d snapshot(s) from an earlier run of %s were never "
+            "cleaned up (a lost record, e.g. a reboot mid-backup); collecting them",
+            len(stale), task_name,
+        )
+
+    remaining = []
+    for name in stale:
+        try:
+            middleware.call_sync("zfs.snapshot.delete", name)
+        except Exception as e:  # noqa: BLE001 - busy, or gone; either way, next run
+            remaining.append(name)
+            if logger:
+                logger.debug(
+                    "truecloud-patch: could not collect %s: %r", name, e
+                )
+    return remaining
+
+
 def stage_nested(middleware, path, snapshot, base_dataset, base_mountpoint,
                  task_name, datasets, logger=None):
     """Build a complete staging tree for `path` from the already-taken `snapshot`.
@@ -650,6 +798,18 @@ def stage_nested(middleware, path, snapshot, base_dataset, base_mountpoint,
             "carrying them forward to the next run", len(pending),
         )
 
+    # ...and collect anything from an earlier run that has NO record at all.
+    #
+    # The sidecar above is precise but lives in /run, which is tmpfs -- a reboot
+    # mid-backup destroys it and orphans the whole tree with nothing pointing at it.
+    # This finds those by name and is the only thing that ever will.
+    #
+    # It runs AFTER the sidecar reclaim on purpose: the recorded path is authoritative
+    # and cheap, and the GC should only ever be mopping up what the record lost.
+    pending.extend(
+        gc_stale_snapshots(middleware, task_name, snapshot, logger=logger)
+    )
+
     # Record the snapshot BEFORE mounting anything, not after. middlewared can
     # die at any point (this patch even schedules a restart at boot), and the
     # sidecar is the only thing that survives it -- an in-process dict would take

recover.sh

@@ -17,7 +17,7 @@
 #   bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
 #   systemctl restart middlewared
 
-VERSION="0.6.0"
+VERSION="0.6.1"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 

tests/test_release_notes.py

@@ -9,6 +9,7 @@ across install.sh / uninstall.sh / recover.sh / apply.sh and nothing noticed.
 """
 
 import os
+import re
 import sys
 
 import pytest
@@ -233,3 +234,60 @@ class TestCandidateNotesResolveToTheBaseVersion:
     def test_a_genuinely_missing_section_still_raises(self):
         with pytest.raises(KeyError):
             extract_notes(self.CHANGELOG, "v9.9.9-rc1")
+
+
+class TestTheChangelogIsStructurallySound:
+    """The release body IS this file, so a mangled section ships to every user.
+
+    It has been mangled once: an edit matched the literal `## Unreleased` inside a
+    backticked phrase in a prose bullet and spliced a whole new section into the middle
+    of it, splitting the sentence in half.
+    """
+
+    def changelog(self):
+        with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
+            return fh.read()
+
+    def test_no_version_section_is_empty(self):
+        text = self.changelog()
+        for v in changelog_versions(text):
+            assert extract_notes(text, v).strip(), f"v{v} has an empty section"
+
+    def test_versions_are_in_descending_order(self):
+        from release_notes import version_tuple
+        versions = changelog_versions(self.changelog())
+        assert versions == sorted(versions, key=version_tuple, reverse=True), (
+            "CHANGELOG versions are out of order — a section was spliced in wrong"
+        )
+
+    def test_headings_are_at_the_start_of_a_line_and_not_inside_prose(self):
+        # A `### Fixed` that ends up indented under a bullet is a section nobody sees.
+        for i, line in enumerate(self.changelog().splitlines(), 1):
+            if line.lstrip().startswith(("## ", "### ")) and line != line.lstrip():
+                raise AssertionError(
+                    f"line {i}: heading is indented, so it is inside a list item "
+                    f"rather than being a section: {line!r}"
+                )
+
+    def test_every_bullet_that_opens_a_bold_phrase_closes_it(self):
+        # The splice cut `- **A stable release ... under \`## Unreleased` in half,
+        # leaving an unterminated ** and a dangling sentence.
+        #
+        # A bullet is the `- ` line plus everything up to the next top-level bullet or
+        # heading -- bold phrases routinely wrap across lines, so a per-line check
+        # would flag every long bullet in the file.
+        text = self.changelog()
+        bullets = re.split(r"^(?=- |#{2,3} )", text, flags=re.M)
+        bad = []
+        for b in bullets:
+            if not b.startswith("- "):
+                continue
+            # Code spans are not markup: `*args, **kwargs` is a literal, not a bold
+            # phrase, and counting its ** would flag a perfectly well-formed bullet.
+            prose = re.sub(r"`[^`]*`", "", b)
+            if prose.count("**") % 2:
+                bad.append(b.splitlines()[0][:70])
+        assert not bad, (
+            "unbalanced ** in a bullet — a section was probably spliced into the "
+            "middle of it:\n  " + "\n  ".join(bad)
+        )

tests/test_truecloud_nested.py

@@ -893,3 +893,159 @@ class TestTheSidecarCarriesEveryPendingTree:
         assert len(notes) == 2
         assert "'Tap@a'" in notes[0] and "'Tap@b'" in notes[1]
         assert "[" not in "".join(notes)
+
+
+class TestGarbageCollectorSelection:
+    """`stale_snapshot_names` DELETES DATA on a name match.
+
+    A name match is a weaker claim than a recorded fact, so every way it could be wrong
+    is a test. It exists because the sidecar — which IS a recorded fact — lives in /run,
+    which is tmpfs: a reboot mid-backup destroys it and orphans a 250-snapshot tree with
+    nothing left pointing at it. This is the only thing that would ever find those.
+    """
+
+    import datetime as _dt
+    NOW = _dt.datetime(2026, 7, 14, 12, 0, 0, tzinfo=_dt.UTC)
+    CURRENT = "Tap@cloud_backup-5-20260714115900"        # 1 minute ago
+    OLD = "Tap/apps/x@cloud_backup-5-20260713030000"     # ~33 hours ago
+
+    def collect(self, names, **kw):
+        import truecloud_nested as tn
+        return tn.stale_snapshot_names(
+            "cloud_backup-5", self.CURRENT, names, self.NOW, **kw
+        )
+
+    def test_it_collects_our_own_leftovers(self):
+        assert self.collect([self.OLD]) == [self.OLD]
+
+    def test_it_NEVER_touches_the_current_run(self):
+        # Both the parent and its children share the current snapname.
+        names = [self.CURRENT, "Tap/apps/x@cloud_backup-5-20260714115900"]
+        assert self.collect(names) == []
+
+    def test_it_NEVER_touches_a_periodic_snapshot(self):
+        assert self.collect(["Tap/apps/x@auto-2026-07-13_03-00"]) == []
+
+    def test_it_NEVER_touches_a_human_made_snapshot(self):
+        assert self.collect(["Tap@before-i-broke-everything"]) == []
+
+    def test_it_NEVER_touches_another_TASK(self):
+        # cloud_backup-5 must not match cloud_backup-50. This is why the prefix
+        # carries the trailing dash.
+        assert self.collect(["Tap/apps/x@cloud_backup-50-20260713030000"]) == []
+        assert self.collect(["Tap/apps/x@cloud_backup-7-20260713030000"]) == []
+
+    def test_it_NEVER_touches_a_one_time_backup(self):
+        assert self.collect(["Tap@cloud_backup-onetime-20260713030000"]) == []
+
+    def test_it_NEVER_touches_a_snapshot_that_is_MOUNTED(self):
+        # An in-flight run pins its own snapshots. This — not the age heuristic — is
+        # what actually protects a concurrent backup.
+        assert self.collect([self.OLD], in_use={self.OLD}) == []
+
+    def test_it_NEVER_touches_a_snapshot_younger_than_the_minimum_age(self):
+        # Covers the seconds-long window between `zfs snapshot -r` and the mounts
+        # appearing, when a live run's snapshots look exactly like garbage.
+        young = "Tap/apps/x@cloud_backup-5-20260714113000"   # 30 minutes ago
+        assert self.collect([young]) == []
+        assert self.collect([young], min_age=60) == [young]
+
+    def test_a_name_it_cannot_parse_is_left_alone(self):
+        assert self.collect(["Tap@cloud_backup-5-not-a-timestamp"]) == []
+        assert self.collect(["Tap@cloud_backup-5-"]) == []
+
+    def test_a_realistic_mixed_pool(self):
+        names = [
+            self.CURRENT,                                       # ours, running
+            "Tap/apps/x@cloud_backup-5-20260714115900",         # ours, running (child)
+            self.OLD,                                           # ours, orphaned  <-
+            "Tap/apps/y@cloud_backup-5-20260712030000",         # ours, orphaned  <-
+            "Tap/apps/x@auto-2026-07-13_03-00",                 # periodic
+            "Tap/apps/x@cloud_backup-7-20260713030000",         # another task
+            "Tap@manual-keepme",                                # human
+        ]
+        assert sorted(self.collect(names)) == sorted(
+            [self.OLD, "Tap/apps/y@cloud_backup-5-20260712030000"]
+        )
+
+
+class TestMountedSnapshots:
+    def test_it_reads_snapshot_names_out_of_the_mount_table(self, tmp_path):
+        import truecloud_nested as tn
+        mounts = tmp_path / "mounts"
+        mounts.write_text(
+            "tmpfs /run tmpfs rw 0 0\n"
+            "Tap/apps/x@snap1 /run/truecloud-nested/t/apps/x zfs ro 0 0\n"
+            "Tap/apps/y@snap1 /mnt/Tap/apps/y/.zfs/snapshot/snap1 zfs ro 0 0\n"
+            "Tap/live /mnt/Tap/live zfs rw 0 0\n"
+        )
+        live = tn.mounted_snapshots(str(mounts))
+        assert live == {"Tap/apps/x@snap1", "Tap/apps/y@snap1"}
+        assert "Tap/live" not in live      # a live dataset is not a snapshot
+
+
+class TestGarbageCollectorExecution:
+    def test_it_deletes_the_stale_ones_and_nothing_else(self, monkeypatch, tmp_path):
+        import datetime as dt
+        import truecloud_nested as tn
+
+        mounts = tmp_path / "mounts"
+        mounts.write_text("")
+        now = dt.datetime(2026, 7, 14, 12, 0, 0, tzinfo=dt.UTC)
+
+        mw = FakeMiddleware([
+            "Tap@cloud_backup-5-20260714115900",             # current run
+            "Tap/apps/x@cloud_backup-5-20260713030000",      # orphan   <-
+            "Tap/apps/x@auto-2026-07-13_03-00",              # periodic
+            "Tap/apps/x@cloud_backup-7-20260713030000",      # other task
+        ])
+        remaining = tn.gc_stale_snapshots(
+            mw, "cloud_backup-5", "Tap@cloud_backup-5-20260714115900",
+            now=now, mounts_file=str(mounts),
+        )
+        assert remaining == []
+        assert mw.snapshots == [
+            "Tap@cloud_backup-5-20260714115900",
+            "Tap/apps/x@auto-2026-07-13_03-00",
+            "Tap/apps/x@cloud_backup-7-20260713030000",
+        ]
+
+    def test_a_busy_orphan_is_reported_not_swallowed(self, monkeypatch, tmp_path):
+        import datetime as dt
+        import truecloud_nested as tn
+
+        mounts = tmp_path / "mounts"
+        mounts.write_text("")
+        now = dt.datetime(2026, 7, 14, 12, 0, 0, tzinfo=dt.UTC)
+        orphan = "Tap/apps/x@cloud_backup-5-20260713030000"
+
+        mw = BusyMiddleware(
+            ["Tap@cloud_backup-5-20260714115900", orphan],
+            busy=[orphan], busy_for=99,
+        )
+        remaining = tn.gc_stale_snapshots(
+            mw, "cloud_backup-5", "Tap@cloud_backup-5-20260714115900",
+            now=now, mounts_file=str(mounts),
+        )
+        assert remaining == [orphan]
+
+    def test_it_collects_NOTHING_when_the_query_fails(self, tmp_path):
+        # Cannot enumerate => cannot know what is ours => delete nothing.
+        import datetime as dt
+        import truecloud_nested as tn
+
+        mounts = tmp_path / "mounts"
+        mounts.write_text("")
+
+        class Broken(FakeMiddleware):
+            def call_sync(self, method, *args):
+                if method == "zfs.snapshot.query":
+                    raise RuntimeError("middleware is having a day")
+                return super().call_sync(method, *args)
+
+        assert tn.gc_stale_snapshots(
+            Broken(["Tap/apps/x@cloud_backup-5-20260713030000"]),
+            "cloud_backup-5", "Tap@cloud_backup-5-20260714115900",
+            now=dt.datetime(2026, 7, 14, 12, 0, 0, tzinfo=dt.UTC),
+            mounts_file=str(mounts),
+        ) == []

tests/test_workflows.py

@@ -10,7 +10,8 @@ import re
 
 import pytest
 
-WORKFLOWS = os.path.join(os.path.dirname(__file__), "..", ".github", "workflows")
+ROOT = os.path.join(os.path.dirname(__file__), "..")
+WORKFLOWS = os.path.join(ROOT, ".github", "workflows")
 
 
 def workflow_files():
@@ -84,11 +85,77 @@ class TestBothForges:
         assert "if: ${{ contains(github.server_url, 'github.com') }}" in src
         assert "if: ${{ !contains(github.server_url, 'github.com') }}" in src
 
-    def test_compat_files_an_issue_on_each_forge(self):
+    def test_compat_files_its_report_through_ONE_implementation(self):
+        # It used to be two near-identical shell steps, one per forge. Two copies of
+        # "find the issue, decide whether to comment, post it" is two chances to drift,
+        # and the Gitea one duplicated an issue for real.
         with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
             src = fh.read()
-        assert "file a bug report (GitHub)" in src
+        assert "tools/compat_publish.py" in src
-        assert "file a bug report (Gitea)" in src
+        assert "file a bug report (GitHub)" not in src
+        assert "file a bug report (Gitea)" not in src
+
+
+class TestTheBotDoesNotSpam:
+    """It left 11 identical 3,000-character comments on one issue in a single day.
+
+    A bot that repeats itself daily gets muted — and then the next REAL finding is
+    scrolled past, which defeats the entire reason for building it.
+    """
+
+    def publisher(self):
+        with open(os.path.join(ROOT, "tools", "compat_publish.py"), encoding="utf-8") as fh:
+            return fh.read()
+
+    def test_it_compares_a_fingerprint_before_saying_anything(self):
+        src = self.publisher()
+        assert "extract_fingerprint" in src
+        assert "staying quiet" in src
+
+    def test_the_body_is_edited_in_place_not_appended_to(self):
+        src = self.publisher()
+        assert '"PATCH"' in src, "the issue body must be updated, not commented onto"
+
+    def test_it_closes_the_issue_when_everything_is_fixed(self):
+        src = self.publisher()
+        assert '"state": "closed"' in src
+
+    def test_the_matrix_refresh_opens_a_PR_rather_than_pushing_to_main(self):
+        # An unattended push to main from CI is exactly what the release barrier exists
+        # to prevent: a bot that can move main can move it somewhere nobody looked.
+        #
+        # Checked against CODE, not comments — the step's own commentary explains what
+        # it replaced, and that mention must not read as the thing itself.
+        with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
+            src = fh.read()
+        code = "\n".join(
+            ln for ln in src.splitlines() if not ln.lstrip().startswith("#")
+        )
+        assert "/pulls" in code, "the matrix refresh must open a PR"
+        assert "HEAD:main" not in code, "CI still pushes straight to main"
+
+    def test_the_matrix_PR_targets_the_CANONICAL_forge_not_the_mirror(self):
+        # GitHub is a one-way mirror: a PR merged there would be silently clobbered by
+        # the next `fleet-repos mirror` push from Gitea. A bot opening PRs against a
+        # mirror is a bot doing nothing, slowly.
+        with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
+            src = fh.read()
+        i = src.index("refresh the README matrix")
+        step = src[i:i + 400]
+        assert "!contains(github.server_url, 'github.com')" in step, (
+            "the matrix PR must be opened on Gitea (canonical), not GitHub (mirror)"
+        )
+
+    def test_the_workflow_has_the_permissions_its_steps_actually_need(self):
+        # It shipped with `contents: read` while the step pushed a branch and opened a
+        # PR — it would have died with a 403 on the first scheduled run, and I would
+        # have had a bot that silently never worked.
+        with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
+            src = fh.read()
+        perms = src[src.index("permissions:"):src.index("jobs:")]
+        assert "contents: write" in perms, "pushing a branch needs contents: write"
+        assert "pull-requests: write" in perms, "opening a PR needs pull-requests: write"
+        assert "issues: write" in perms
 
 
 class TestCompatCannotSilentlyPass:

tools/compat.py

@@ -43,6 +43,7 @@ from __future__ import annotations
 
 import argparse
 import ast
+import hashlib
 import json
 import os
 import sys
@@ -892,6 +893,90 @@ def render_markdown(rows: list[dict]) -> str:
     return "\n".join(out) + "\n" + _LEGEND
 
 
+FINGERPRINT = "<!-- compat-fingerprint:"
+
+
+def fingerprint(rows: list[dict]) -> str:
+    """A stable digest of WHAT IS BROKEN, and nothing else.
+
+    The bug report must be updated when the findings change and stay silent when they
+    do not. Without this the workflow commented on every run -- it left **11 identical
+    3,000-character comments** on one issue in a single day, which is not a warning
+    system, it is a mute button with extra steps.
+
+    Deliberately excludes anything that moves on its own: the matrix's `ok` rows, the
+    hardware-verified column, and the exact TrueNAS point-release (`TS-25.10.4` ->
+    `TS-25.10.5` is not news). Only the broken (ref, module, problem-id) triples count.
+    """
+    findings = sorted(
+        (r["ref"], mod, p["id"])
+        for r in rows
+        for mod, m in r["modules"].items()
+        if is_broken(m)
+        for p in m["problems"]
+    )
+    return hashlib.sha256(repr(findings).encode()).hexdigest()[:16]
+
+
+def extract_fingerprint(body: str) -> str | None:
+    """The fingerprint a previous run left in the issue body, if any."""
+    if not body:
+        return None
+    i = body.find(FINGERPRINT)
+    if i == -1:
+        return None
+    return body[i + len(FINGERPRINT):].split("-->", 1)[0].strip() or None
+
+
+def render_issue(rows: list[dict]) -> str:
+    """The bug report body: what is broken, why it matters, and nothing else up front.
+
+    Short by design. The full matrix and the healthy versions go in a fold -- somebody
+    opening this wants to know what broke and whether it can hurt them, not to re-read
+    a table they can see in the README.
+    """
+    broken = [r for r in rows if any(is_broken(m) for m in r["modules"].values())]
+
+    out = [
+        "`tools/compat.py` checks what this patch assumes about middlewared against "
+        "iXsystems' actual source, every day. Those assumptions no longer hold on the "
+        "versions below.",
+        "",
+        "**This does not break anyone today.** `apply.sh` re-checks on every boot and "
+        "**declines to apply** a module whose assumptions fail, so TrueNAS is left "
+        "stock rather than half-patched. The cost is the module's feature, not a "
+        "broken backup.",
+        "",
+    ]
+
+    for r in broken:
+        out.append(f"### `{r['ref']}`")
+        out.append("")
+        for mod, m in sorted(r["modules"].items()):
+            if not is_broken(m):
+                continue
+            out.append(f"**{mod}**")
+            out.append("")
+            for p in m["problems"]:
+                out.append(f"- {p['detail']}")
+                out.append(f"  <br><sub>{p['why']}</sub>")
+            out.append("")
+
+    out += [
+        "<details><summary>Full support matrix</summary>",
+        "",
+        render_markdown(rows),
+        "</details>",
+        "",
+        "_Filed and kept up to date by "
+        "[`compat.yml`](.github/workflows/compat.yml). It edits this body when the "
+        "findings change, and stays quiet when they do not._",
+        "",
+        f"{FINGERPRINT} {fingerprint(rows)} -->",
+    ]
+    return "\n".join(out)
+
+
 def render_matrix(rows: list[dict]) -> str:
     """A support table.
 

tools/compat_publish.py

@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+"""Keep ONE bug report in sync with what compat.py currently finds.
+
+WHY THIS IS NOT JUST "POST A COMMENT"
+-------------------------------------
+The first version commented on every run that found a break. In one day it left
+**11 identical 3,000-character comments** on the same issue. That is not a warning
+system; it is a mute button with extra steps. The next real finding would have been
+scrolled past, which defeats the entire point of building it.
+
+So:
+
+  * **The issue body is the current truth.** It is edited in place, never appended to.
+  * **Comments are a changelog of CHANGES.** A run whose findings are identical to the
+    last one says nothing at all -- no comment, no edit, no notification.
+  * A fingerprint of the findings (broken ref/module/problem triples only) is embedded
+    in the body. It deliberately ignores things that move on their own -- healthy rows,
+    the hardware-verified column, TrueNAS point releases -- so `TS-25.10.4` becoming
+    `TS-25.10.5` is not news, and does not wake anybody up.
+
+  * When everything is fixed, the issue is **closed** with a comment saying so.
+
+Works against GitHub and Gitea, which differ only in the auth header and the issue
+list URL. One implementation, so the two cannot drift.
+
+    python3 tools/compat_publish.py --api <url> --token <tok> --matrix /tmp/matrix.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import urllib.error
+import urllib.request
+
+sys.path.insert(0, __file__.rsplit("/", 1)[0])
+
+from compat import (  # noqa: E402
+    extract_fingerprint,
+    fingerprint,
+    is_broken,
+    render_issue,
+)
+
+TITLE = "TrueNAS compatibility: the patch's assumptions no longer hold"
+
+
+def _call(url, token, method="GET", data=None):
+    req = urllib.request.Request(
+        url, method=method,
+        headers={
+            # Gitea wants `token <t>`; GitHub accepts `Bearer <t>`. GitHub also
+            # accepts `token <t>`, so one header serves both.
+            "Authorization": f"token {token}",
+            "Content-Type": "application/json",
+            "Accept": "application/vnd.github+json",
+        },
+        data=json.dumps(data).encode() if data else None,
+    )
+    with urllib.request.urlopen(req) as r:  # noqa: S310
+        return json.load(r) if r.length != 0 else {}
+
+
+def find_issue(api, token, title):
+    """The LOWEST-numbered issue with this title, open or closed.
+
+    Lowest, not "whichever the API returns first": two issues with the same title
+    existed once (an earlier version put the ref list in the title, so the identity
+    changed whenever that set changed), and an order-dependent pick would alternate
+    between them -- reopening one while commenting on the other.
+    """
+    issues = _call(f"{api}/issues?state=all&per_page=100", token)
+    mine = [
+        i for i in issues
+        if i.get("title") == title and "pull_request" not in i   # GitHub lists PRs here
+    ]
+    return min(mine, key=lambda i: i["number"]) if mine else None
+
+
+def main(argv):
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--api", required=True, help="…/repos/<owner>/<repo>")
+    ap.add_argument("--token", required=True)
+    ap.add_argument("--matrix", required=True, help="compat.py --matrix --json output")
+    args = ap.parse_args(argv[1:])
+
+    with open(args.matrix, encoding="utf-8") as fh:
+        rows = json.load(fh)
+
+    broken = [r for r in rows if any(is_broken(m) for m in r["modules"].values())]
+    issue = find_issue(args.api, args.token, TITLE)
+
+    # ── everything is healthy ────────────────────────────────────────────────
+    if not broken:
+        if issue and issue["state"] == "open":
+            _call(f"{args.api}/issues/{issue['number']}/comments", args.token, "POST",
+                  {"body": "All of the patch's assumptions hold again on every "
+                           "checked TrueNAS version. Closing."})
+            _call(f"{args.api}/issues/{issue['number']}", args.token, "PATCH",
+                  {"state": "closed"})
+            print(f"closed #{issue['number']} — nothing is broken any more")
+        else:
+            print("nothing broken; no open report to close")
+        return 0
+
+    body = render_issue(rows)
+    want = fingerprint(rows)
+
+    # ── nothing to file yet ──────────────────────────────────────────────────
+    if issue is None:
+        made = _call(f"{args.api}/issues", args.token, "POST",
+                     {"title": TITLE, "body": body})
+        print(f"filed #{made['number']}")
+        return 0
+
+    have = extract_fingerprint(issue.get("body") or "")
+    n = issue["number"]
+
+    # ── the findings are UNCHANGED: say nothing ──────────────────────────────
+    #
+    # This is the whole point. A daily "still broken, same as yesterday" comment is
+    # what taught everyone to ignore the last one.
+    if have == want and issue["state"] == "open":
+        print(f"#{n} is already current ({want}) — staying quiet")
+        return 0
+
+    _call(f"{args.api}/issues/{n}", args.token, "PATCH", {"body": body, "state": "open"})
+
+    if have != want:
+        refs = ", ".join(f"`{r['ref']}`" for r in broken)
+        note = (
+            "The findings changed — the report above has been updated.\n\n"
+            f"Currently broken on: {refs}."
+            if have else
+            "This report is now kept up to date automatically: the body above always "
+            "reflects the current findings, and a comment is only added when they "
+            "change."
+        )
+        _call(f"{args.api}/issues/{n}/comments", args.token, "POST", {"body": note})
+        print(f"updated #{n}: {have} -> {want}")
+    else:
+        print(f"reopened #{n}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

uninstall.sh

@@ -3,7 +3,7 @@
 
 set -euo pipefail
 
-VERSION="0.6.0"
+VERSION="0.6.1"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'

update.sh

@@ -19,7 +19,7 @@
 
 set -euo pipefail
 
-VERSION="0.6.0"
+VERSION="0.6.1"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _PREV_FILE="$PATCH_DIR/.update_previous"