docs: TrueNAS 26 is supported; record what has actually been run

The README and how-it-works still said "TrueNAS 26: nested snapshots are not supported
yet" and "the third is not fixed, and is why 26 reports BROKEN". Both shipped in v0.7.0
and are now false — exactly the kind of stale claim that misleads somebody deciding
whether to trust this with their backups.

Adds docs/verification.md: what has ACTUALLY been run, as opposed to what the support
matrix proves. The matrix is static analysis — it shows the patch's assumptions still
hold, which is a strictly weaker claim than "a backup ran and a restore came back". The
new file records the three live tasks exercised on 25.10.4 (nested, nested+zvols,
non-nested), the md5 of the file that came back out of B2, the real orphan the collector
reclaimed from the pool, and what is NOT covered (24.10/25.04 unrun; master broken; no
reboot on v0.7.0). The README now points at it, next to the matrix it qualifies.

The how-it-works TrueNAS 26 section now explains the part that mattered: the public
pool.* queries are not like-for-like replacements for the deleted private zfs.* ones —
they apply a visibility policy hiding 84 of 270 datasets on a real pool, including live
app data — and the rule the module now follows (read the truth from ZFS, make changes
through middleware). Plus the divergence nothing warned about: 26 decides `recursive` by
a different rule than this patch decides `nested`, which orphaned one snapshot per zvol
on every run until ownership of the sweep was made unconditional.

README.md

@@ -59,7 +59,7 @@ If something is wrong, the reason is in `apply.log` — start at
 | --- | --- | --- | --- |
 | 24.10.2.4 | ok | ok | — |
 | 25.04.2.6 | ok | ok | — |
-| 25.10.4 | ok | ok | nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m |
+| 25.10.4 | ok | ok | v0.7.0: 3 live tasks — 191-dataset nested backup of /mnt/Tap, a 215-filesystem/2-zvol backup of /mnt/Tank/backups, and a non-nested one; 0 orphans, 0 leaked mounts, byte-identical restore; the collector also reclaimed a real orphan the pool had been carrying |
 | 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — |
 | master _(unreleased)_ | **BROKEN** | **BROKEN** | — |
 
@@ -77,18 +77,29 @@ source. It does not mean a human ran a backup on it — that is the
 The table is **regenerated daily by CI** against iXsystems' actual middleware source
 — it is not a claim somebody typed once and forgot.
 
-**On TrueNAS 26:** the patch was run on a real TrueNAS **26.0.0-BETA.1** install — a
-274-snapshot recursive backup of a 292-dataset pool, followed by a byte-identical
-restore of a four-level-deep child dataset. The *Hardware-verified* column tracks the
-newest beta iX has tagged (currently BETA.3), so it does not carry that mark: a build
-nobody has actually run a backup on does not get credit for one.
+It is also **static analysis**: it proves the patch's assumptions still hold, which is
+a weaker claim than "a backup ran and a restore came back". For what has actually been
+run — which tasks, on which hardware, and the md5 of the file that came back — see
+[docs/verification.md](docs/verification.md).
 
-**TrueNAS 26: nested snapshots are not supported yet, and upgrading will not break
-you.** 26 rewrites `cloud_backup` and deletes the ZFS methods this module calls. On
-26 `apply.sh` finds that the assumptions no longer hold and **does not apply the
-module**: TrueNAS is left stock, B2/S3 keeps working, nested datasets are simply not
-covered, and the reason is named in `apply.log`. A broken backup is worse than a
-missing feature. Details: [How it works](docs/how-it-works.md#truenas-26).
+**TrueNAS 26 is supported** as of v0.7.0, and was verified on a real
+**26.0.0-BETA.1** install: a 274-snapshot recursive backup of a 292-dataset pool, and a
+byte-identical restore of a four-level-deep child dataset. The *Hardware-verified*
+column tracks the newest beta iX has tagged (currently BETA.3), so it does not carry
+that mark — a build nobody has actually run a backup on does not get credit for one.
+
+26 rewrites `cloud_backup` from async to synchronous and deletes the private ZFS
+methods this module used to call, so getting there took real work: the patch now
+injects the wrapper flavour that matches the installed middleware, reads dataset and
+snapshot lists **from ZFS rather than middleware** (whose queries hide TrueNAS's own
+datasets — 84 of 270 on a real pool, including live app data), and owns the snapshot
+sweep even when it stages nothing (26 decides `recursive` by a rule this patch does not
+share, and would otherwise orphan one snapshot per zvol on every run).
+
+**And if a future TrueNAS breaks it, you get a missing feature, not a broken backup.**
+`apply.sh` re-checks the patch's assumptions at every boot and **refuses to apply a
+module whose assumptions no longer hold** — TrueNAS is left stock, and the reason is
+named in `apply.log`. Details: [How it works](docs/how-it-works.md#truenas-26).
 
 ---
 

docs/how-it-works.md

@@ -187,23 +187,71 @@ backup-breaking**, and none of them is visible from the `cloud_backup` files:
 | `get_dataset_recursive()` **deleted** from `plugins/cloud/snapshot.py` | `NameError` — the injected block called it out of the host module's namespace |
 | `plugins/zfs_/dataset.py` and `zfs_/snapshot.py` **deleted** | `zfs.dataset.query`, `zfs.snapshot.query` and `zfs.snapshot.delete` all vanish. 26 uses `filesystem.statfs` and `zfs.resource.*` |
 
-The first two are fixed: the patch reads which flavour of `cloud_backup` your box
-declares and injects the wrapper that matches (one implementation of the real logic,
-two thin wrappers), and it carries its own copy of the deleted helper.
+All three are fixed as of **v0.7.0**, and 26 is supported.
 
-The third is **not** fixed, and is why 26 reports BROKEN. Porting it means rewriting
-the module's ZFS calls onto 26's new API, and no single API spans 24.10 through 26 —
-so it needs a real 26 box to verify against, not a plausible-looking diff. Shipping a
-port nobody has run is exactly the failure this project exists to avoid.
+The first two were straightforward: the patch reads which flavour of `cloud_backup`
+your box declares and injects the wrapper that matches (one implementation of the real
+logic, two thin wrappers), and it carries its own copy of the deleted helper.
 
-It is also the row that would have hurt most. `zfs.snapshot.delete` is what sweeps the
-recursive snapshot; without it, **every run would orphan one snapshot per descendant
-dataset — 250 on a real pool — forever.** The compatibility check caught it only
-because it now asserts the middleware *methods the patch calls*, not just the symbols
-it wraps.
+The third was not, and it is the one that would have hurt most — `zfs.snapshot.delete`
+is what sweeps the recursive snapshot, and without it **every run would orphan one
+snapshot per descendant dataset (250 on a real pool), forever, while reporting
+success.**
 
-`master` (development after 26) reports BROKEN too: iXsystems are still reshaping
+The obvious port is to the public `pool.dataset.query` / `pool.snapshot.query`. Those
+methods exist, are documented, and are covered by iX's deprecation policy — and they
+are **not like-for-like replacements**. They apply a *visibility policy*: they hide the
+datasets TrueNAS considers its own (`ix-apps/*`, `.system/*`, `.ix-virt/*`). On a real
+pool that is **84 of 270 datasets, including live application data.** Staging from that
+view would have omitted every one of them from the backup — and the planner would never
+have seen them, so they would not have appeared in its "skipped" list either. A green
+backup, quietly missing data. The snapshot query lies the same way, so the sweep would
+have orphaned one snapshot per hidden dataset.
+
+No source analysis could have caught that. The methods are all present and correctly
+shaped. Only running it could, which is why it took a real 26 box.
+
+So the module now follows one rule:
+
+> **Read the truth from ZFS. Make changes through middleware.**
+
+Enumeration is `zfs list` — no policy can filter it, and it behaves identically on every
+release, which also means one code path instead of a version conditional. Mutation stays
+a middleware call, so TrueNAS's own bookkeeping stays consistent; an exact-name delete
+works fine even on a dataset the query hides. It is only enumeration that lies.
+
+The snapshot *delete* still needs a namespace, and no single one spans every release —
+24.10 and 25.04 have `zfs.snapshot`, 26 has only `pool.snapshot`, 25.10 has both. So it
+is resolved at runtime, by asking whether the namespace can actually delete. `tools/compat.py`
+asks the identical question against iX's source, and a test binds the two lists together,
+so what CI verifies and what runs cannot drift apart.
+
+### The one 26 changed that nothing warned about
+
+Stock decides whether to take a **recursive** snapshot by its own rule, and on 26 that
+rule stopped being ours:
+
+| | decides `recursive` by |
+| --- | --- |
+| stock ≤ 25.10 | `get_dataset_recursive()` — the same function this patch vendors |
+| **stock 26** | `filesystem.statfs`: `recursive = (path == the dataset's mountpoint)` |
+| this patch | `get_dataset_recursive()` — is a mounted *filesystem* child under the path? |
+
+Up to 25.10 those were the *same question*, so a snapshot the patch declined to stage
+provably had no children and stock's non-recursive delete was correct. On 26 they
+disagree: a dataset whose only descendants are **zvols** or **legacy-mountpoint**
+datasets gets a recursive snapshot, while the patch sees nothing to stage. Stock then
+destroys the parent only — and with no staging tree there was no sidecar, and the
+garbage collector only ever ran from the staging path. Nothing on the box would ever
+have found the children.
+
+It was reproduced on a 26 VM (one orphan per zvol, every run, backup green) and closed:
+**ownership of the sweep is no longer conditional on staging.**
+
+`master` (development after 26) **does** report BROKEN: iXsystems are still reshaping
 these functions there, renaming `middleware` → `context` and `cloud_backup` → `entry`
 and adding a required `credentials` parameter. That is a moving target and is
 deliberately not chased; the check keeps reporting it until it settles into a beta,
-which is when it becomes worth fixing.
+which is when it becomes worth fixing. Until then, a box running master would simply
+not get the modules — `apply.sh` refuses to apply a module whose assumptions no longer
+hold, and says why in `apply.log`.

docs/verification.md

@@ -0,0 +1,77 @@
+# What has actually been run
+
+The support matrix in the README is **static analysis**: it proves the patch's
+assumptions about middlewared still hold. That is a strictly weaker claim than "a
+backup ran and a restore came back". This file is the stronger claim, and it is
+maintained by hand, because the only way to fill it in is to do it.
+
+If you are deciding whether to trust this with your backups, read this file, not the
+matrix.
+
+---
+
+## v0.7.0 — TrueNAS 25.10.4 (production hardware)
+
+Six live TrueCloud tasks, all `snapshot = true`, backing up to Backblaze B2. Three were
+exercised end to end, chosen to cover the three shapes the code handles differently:
+
+| Task | Path | Shape | Result |
+| --- | --- | --- | --- |
+| 5 | `/mnt/Tap` | 191 nested datasets staged, 282-snapshot recursive tree | SUCCESS |
+| 7 | `/mnt/Tank/backups` | 215 filesystems **+ 2 zvols** | SUCCESS |
+| 9 | `/mnt/Tank/flan` | **no** nested filesystem children | SUCCESS |
+
+After every run: **0 orphaned snapshots, 0 leaked bind mounts, 0 stale sidecars.**
+
+**The restore.** `apps/vaultwarden/data/config.json` — a file inside a *child* dataset,
+which is exactly what stock TrueNAS cannot capture — was restored from B2 and compared
+against the live file:
+
+    live      f809df6ba231986b1ba824044228a03a   1808 bytes
+    restored  f809df6ba231986b1ba824044228a03a   1808 bytes
+    => byte-identical
+
+**The collector earned its keep on real data.** The pool was already carrying an orphan:
+`Tap/apps/prometheus@cloud_backup-5-20260713202355`, left behind by an earlier run when
+ZFS's automount held the snapshot busy past all four retries. The first v0.7.0 run found
+it by name, reclaimed it, and the pool's snapshot count went 2148 → 2147. That is the
+garbage collector doing the job it was written for, against a leak that was already
+there and that nothing else would ever have found.
+
+**Boot path.** `apply.sh` is registered as a PREINIT `initshutdownscript`; it was
+re-run against the live middleware and left exactly one `TRUECLOUD_PATCH` marker in
+each patched module (a second copy stacked into a live middlewared module would break
+the box at boot). It correctly detected the box as **async** (`cloud_backup is async
+(TrueNAS <= 25.10)`) and injected the matching wrappers.
+
+**Upgrade path.** `update.sh` was used to move the box from the release candidate to
+the stable tag, in detached HEAD at `v0.7.0`, which is how a user's box actually
+upgrades.
+
+## v0.7.0 — TrueNAS 26.0.0-BETA.1 (VM)
+
+A throwaway VM whose pool reproduces the production pool's *shape* — 292 datasets, 26
+`legacy` mountpoints, nesting five deep — because every bug found on the real box came
+from the shape of the pool, not the bytes in it. MinIO was not used; `rclone serve s3`
+(already on the box) provided the S3 target, so no real B2 credential ever entered the
+VM.
+
+* 274-snapshot recursive backup of the 292-dataset pool. 0 orphans, 0 leaked mounts.
+* Restored `ix-apps/app_mounts/vaultwarden/pgData` — **four levels deep, and a dataset
+  that middleware's own `pool.dataset.query` hides from itself** — byte-identical.
+* The zvol-orphan case was **reproduced with the fix disabled** (one orphan per zvol,
+  every run, backup green), then **closed with it enabled**. See the CHANGELOG entry
+  for why TrueNAS 26 decides `recursive` by a different rule than this patch decides
+  `nested`.
+
+## What is NOT covered
+
+* **24.10 and 25.04** are `ok` in the matrix — the assumptions hold, checked against
+  iX's source — but nobody has run a backup on them. The matrix says so.
+* **master** is BROKEN, and correctly reports so: iX renamed the leading parameters of
+  `get_restic_config` and `restic_backup`. It is not a shipped release; the daily
+  compatibility bot files it, and `apply.sh` would refuse to apply the modules on a box
+  running it.
+* A **reboot** of the production box has not been done on v0.7.0. `apply.sh` was
+  re-executed by hand against the live middleware, which exercises the same code path,
+  but the PREINIT ordering itself has only been proven on earlier versions.

tools/compat.py

@@ -941,10 +941,15 @@ def is_broken(r: dict) -> bool:
 #: is static analysis of iX's source, which proves the patch's assumptions hold --
 #: a strictly weaker claim than "a restore worked". Add a row only after doing it.
 HARDWARE_VERIFIED = {
-    "25.10.4": "nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m",
+    "25.10.4": (
+        "v0.7.0: 3 live tasks — 191-dataset nested backup of /mnt/Tap, a "
+        "215-filesystem/2-zvol backup of /mnt/Tank/backups, and a non-nested one; "
+        "0 orphans, 0 leaked mounts, byte-identical restore; the collector also "
+        "reclaimed a real orphan the pool had been carrying"
+    ),
     "26.0.0-BETA.1": (
-        "nested + providers; 274-snapshot recursive backup of a 292-dataset pool, "
-        "restored a 4-deep child dataset byte-identical"
+        "v0.7.0: 274-snapshot recursive backup of a 292-dataset pool; restored a "
+        "4-deep child dataset byte-identical; zvol-orphan case reproduced then closed"
     ),
 }