diff --git a/README.md b/README.md index a991ab9..546d8d0 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ If something is wrong, the reason is in `apply.log` — start at | --- | --- | --- | --- | | 24.10.2.4 | ok | ok | — | | 25.04.2.6 | ok | ok | — | -| 25.10.4 | ok | ok | nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m | +| 25.10.4 | ok | ok | v0.7.0: 3 live tasks — 191-dataset nested backup of /mnt/Tap, a 215-filesystem/2-zvol backup of /mnt/Tank/backups, and a non-nested one; 0 orphans, 0 leaked mounts, byte-identical restore; the collector also reclaimed a real orphan the pool had been carrying | | 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — | | master _(unreleased)_ | **BROKEN** | **BROKEN** | — | @@ -77,18 +77,29 @@ source. It does not mean a human ran a backup on it — that is the The table is **regenerated daily by CI** against iXsystems' actual middleware source — it is not a claim somebody typed once and forgot. -**On TrueNAS 26:** the patch was run on a real TrueNAS **26.0.0-BETA.1** install — a -274-snapshot recursive backup of a 292-dataset pool, followed by a byte-identical -restore of a four-level-deep child dataset. The *Hardware-verified* column tracks the -newest beta iX has tagged (currently BETA.3), so it does not carry that mark: a build -nobody has actually run a backup on does not get credit for one. +It is also **static analysis**: it proves the patch's assumptions still hold, which is +a weaker claim than "a backup ran and a restore came back". For what has actually been +run — which tasks, on which hardware, and the md5 of the file that came back — see +[docs/verification.md](docs/verification.md). -**TrueNAS 26: nested snapshots are not supported yet, and upgrading will not break -you.** 26 rewrites `cloud_backup` and deletes the ZFS methods this module calls. On -26 `apply.sh` finds that the assumptions no longer hold and **does not apply the -module**: TrueNAS is left stock, B2/S3 keeps working, nested datasets are simply not -covered, and the reason is named in `apply.log`. A broken backup is worse than a -missing feature. Details: [How it works](docs/how-it-works.md#truenas-26). +**TrueNAS 26 is supported** as of v0.7.0, and was verified on a real +**26.0.0-BETA.1** install: a 274-snapshot recursive backup of a 292-dataset pool, and a +byte-identical restore of a four-level-deep child dataset. The *Hardware-verified* +column tracks the newest beta iX has tagged (currently BETA.3), so it does not carry +that mark — a build nobody has actually run a backup on does not get credit for one. + +26 rewrites `cloud_backup` from async to synchronous and deletes the private ZFS +methods this module used to call, so getting there took real work: the patch now +injects the wrapper flavour that matches the installed middleware, reads dataset and +snapshot lists **from ZFS rather than middleware** (whose queries hide TrueNAS's own +datasets — 84 of 270 on a real pool, including live app data), and owns the snapshot +sweep even when it stages nothing (26 decides `recursive` by a rule this patch does not +share, and would otherwise orphan one snapshot per zvol on every run). + +**And if a future TrueNAS breaks it, you get a missing feature, not a broken backup.** +`apply.sh` re-checks the patch's assumptions at every boot and **refuses to apply a +module whose assumptions no longer hold** — TrueNAS is left stock, and the reason is +named in `apply.log`. Details: [How it works](docs/how-it-works.md#truenas-26). --- diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 3c200f3..ef2ca0a 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -187,23 +187,71 @@ backup-breaking**, and none of them is visible from the `cloud_backup` files: | `get_dataset_recursive()` **deleted** from `plugins/cloud/snapshot.py` | `NameError` — the injected block called it out of the host module's namespace | | `plugins/zfs_/dataset.py` and `zfs_/snapshot.py` **deleted** | `zfs.dataset.query`, `zfs.snapshot.query` and `zfs.snapshot.delete` all vanish. 26 uses `filesystem.statfs` and `zfs.resource.*` | -The first two are fixed: the patch reads which flavour of `cloud_backup` your box -declares and injects the wrapper that matches (one implementation of the real logic, -two thin wrappers), and it carries its own copy of the deleted helper. +All three are fixed as of **v0.7.0**, and 26 is supported. -The third is **not** fixed, and is why 26 reports BROKEN. Porting it means rewriting -the module's ZFS calls onto 26's new API, and no single API spans 24.10 through 26 — -so it needs a real 26 box to verify against, not a plausible-looking diff. Shipping a -port nobody has run is exactly the failure this project exists to avoid. +The first two were straightforward: the patch reads which flavour of `cloud_backup` +your box declares and injects the wrapper that matches (one implementation of the real +logic, two thin wrappers), and it carries its own copy of the deleted helper. -It is also the row that would have hurt most. `zfs.snapshot.delete` is what sweeps the -recursive snapshot; without it, **every run would orphan one snapshot per descendant -dataset — 250 on a real pool — forever.** The compatibility check caught it only -because it now asserts the middleware *methods the patch calls*, not just the symbols -it wraps. +The third was not, and it is the one that would have hurt most — `zfs.snapshot.delete` +is what sweeps the recursive snapshot, and without it **every run would orphan one +snapshot per descendant dataset (250 on a real pool), forever, while reporting +success.** -`master` (development after 26) reports BROKEN too: iXsystems are still reshaping +The obvious port is to the public `pool.dataset.query` / `pool.snapshot.query`. Those +methods exist, are documented, and are covered by iX's deprecation policy — and they +are **not like-for-like replacements**. They apply a *visibility policy*: they hide the +datasets TrueNAS considers its own (`ix-apps/*`, `.system/*`, `.ix-virt/*`). On a real +pool that is **84 of 270 datasets, including live application data.** Staging from that +view would have omitted every one of them from the backup — and the planner would never +have seen them, so they would not have appeared in its "skipped" list either. A green +backup, quietly missing data. The snapshot query lies the same way, so the sweep would +have orphaned one snapshot per hidden dataset. + +No source analysis could have caught that. The methods are all present and correctly +shaped. Only running it could, which is why it took a real 26 box. + +So the module now follows one rule: + +> **Read the truth from ZFS. Make changes through middleware.** + +Enumeration is `zfs list` — no policy can filter it, and it behaves identically on every +release, which also means one code path instead of a version conditional. Mutation stays +a middleware call, so TrueNAS's own bookkeeping stays consistent; an exact-name delete +works fine even on a dataset the query hides. It is only enumeration that lies. + +The snapshot *delete* still needs a namespace, and no single one spans every release — +24.10 and 25.04 have `zfs.snapshot`, 26 has only `pool.snapshot`, 25.10 has both. So it +is resolved at runtime, by asking whether the namespace can actually delete. `tools/compat.py` +asks the identical question against iX's source, and a test binds the two lists together, +so what CI verifies and what runs cannot drift apart. + +### The one 26 changed that nothing warned about + +Stock decides whether to take a **recursive** snapshot by its own rule, and on 26 that +rule stopped being ours: + +| | decides `recursive` by | +| --- | --- | +| stock ≤ 25.10 | `get_dataset_recursive()` — the same function this patch vendors | +| **stock 26** | `filesystem.statfs`: `recursive = (path == the dataset's mountpoint)` | +| this patch | `get_dataset_recursive()` — is a mounted *filesystem* child under the path? | + +Up to 25.10 those were the *same question*, so a snapshot the patch declined to stage +provably had no children and stock's non-recursive delete was correct. On 26 they +disagree: a dataset whose only descendants are **zvols** or **legacy-mountpoint** +datasets gets a recursive snapshot, while the patch sees nothing to stage. Stock then +destroys the parent only — and with no staging tree there was no sidecar, and the +garbage collector only ever ran from the staging path. Nothing on the box would ever +have found the children. + +It was reproduced on a 26 VM (one orphan per zvol, every run, backup green) and closed: +**ownership of the sweep is no longer conditional on staging.** + +`master` (development after 26) **does** report BROKEN: iXsystems are still reshaping these functions there, renaming `middleware` → `context` and `cloud_backup` → `entry` and adding a required `credentials` parameter. That is a moving target and is deliberately not chased; the check keeps reporting it until it settles into a beta, -which is when it becomes worth fixing. +which is when it becomes worth fixing. Until then, a box running master would simply +not get the modules — `apply.sh` refuses to apply a module whose assumptions no longer +hold, and says why in `apply.log`. diff --git a/docs/verification.md b/docs/verification.md new file mode 100644 index 0000000..5397bd3 --- /dev/null +++ b/docs/verification.md @@ -0,0 +1,77 @@ +# What has actually been run + +The support matrix in the README is **static analysis**: it proves the patch's +assumptions about middlewared still hold. That is a strictly weaker claim than "a +backup ran and a restore came back". This file is the stronger claim, and it is +maintained by hand, because the only way to fill it in is to do it. + +If you are deciding whether to trust this with your backups, read this file, not the +matrix. + +--- + +## v0.7.0 — TrueNAS 25.10.4 (production hardware) + +Six live TrueCloud tasks, all `snapshot = true`, backing up to Backblaze B2. Three were +exercised end to end, chosen to cover the three shapes the code handles differently: + +| Task | Path | Shape | Result | +| --- | --- | --- | --- | +| 5 | `/mnt/Tap` | 191 nested datasets staged, 282-snapshot recursive tree | SUCCESS | +| 7 | `/mnt/Tank/backups` | 215 filesystems **+ 2 zvols** | SUCCESS | +| 9 | `/mnt/Tank/flan` | **no** nested filesystem children | SUCCESS | + +After every run: **0 orphaned snapshots, 0 leaked bind mounts, 0 stale sidecars.** + +**The restore.** `apps/vaultwarden/data/config.json` — a file inside a *child* dataset, +which is exactly what stock TrueNAS cannot capture — was restored from B2 and compared +against the live file: + + live f809df6ba231986b1ba824044228a03a 1808 bytes + restored f809df6ba231986b1ba824044228a03a 1808 bytes + => byte-identical + +**The collector earned its keep on real data.** The pool was already carrying an orphan: +`Tap/apps/prometheus@cloud_backup-5-20260713202355`, left behind by an earlier run when +ZFS's automount held the snapshot busy past all four retries. The first v0.7.0 run found +it by name, reclaimed it, and the pool's snapshot count went 2148 → 2147. That is the +garbage collector doing the job it was written for, against a leak that was already +there and that nothing else would ever have found. + +**Boot path.** `apply.sh` is registered as a PREINIT `initshutdownscript`; it was +re-run against the live middleware and left exactly one `TRUECLOUD_PATCH` marker in +each patched module (a second copy stacked into a live middlewared module would break +the box at boot). It correctly detected the box as **async** (`cloud_backup is async +(TrueNAS <= 25.10)`) and injected the matching wrappers. + +**Upgrade path.** `update.sh` was used to move the box from the release candidate to +the stable tag, in detached HEAD at `v0.7.0`, which is how a user's box actually +upgrades. + +## v0.7.0 — TrueNAS 26.0.0-BETA.1 (VM) + +A throwaway VM whose pool reproduces the production pool's *shape* — 292 datasets, 26 +`legacy` mountpoints, nesting five deep — because every bug found on the real box came +from the shape of the pool, not the bytes in it. MinIO was not used; `rclone serve s3` +(already on the box) provided the S3 target, so no real B2 credential ever entered the +VM. + +* 274-snapshot recursive backup of the 292-dataset pool. 0 orphans, 0 leaked mounts. +* Restored `ix-apps/app_mounts/vaultwarden/pgData` — **four levels deep, and a dataset + that middleware's own `pool.dataset.query` hides from itself** — byte-identical. +* The zvol-orphan case was **reproduced with the fix disabled** (one orphan per zvol, + every run, backup green), then **closed with it enabled**. See the CHANGELOG entry + for why TrueNAS 26 decides `recursive` by a different rule than this patch decides + `nested`. + +## What is NOT covered + +* **24.10 and 25.04** are `ok` in the matrix — the assumptions hold, checked against + iX's source — but nobody has run a backup on them. The matrix says so. +* **master** is BROKEN, and correctly reports so: iX renamed the leading parameters of + `get_restic_config` and `restic_backup`. It is not a shipped release; the daily + compatibility bot files it, and `apply.sh` would refuse to apply the modules on a box + running it. +* A **reboot** of the production box has not been done on v0.7.0. `apply.sh` was + re-executed by hand against the live middleware, which exercises the same code path, + but the PREINIT ordering itself has only been proven on earlier versions. diff --git a/tools/compat.py b/tools/compat.py index 8359786..7d8fc5d 100644 --- a/tools/compat.py +++ b/tools/compat.py @@ -941,10 +941,15 @@ def is_broken(r: dict) -> bool: #: is static analysis of iX's source, which proves the patch's assumptions hold -- #: a strictly weaker claim than "a restore worked". Add a row only after doing it. HARDWARE_VERIFIED = { - "25.10.4": "nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m", + "25.10.4": ( + "v0.7.0: 3 live tasks — 191-dataset nested backup of /mnt/Tap, a " + "215-filesystem/2-zvol backup of /mnt/Tank/backups, and a non-nested one; " + "0 orphans, 0 leaked mounts, byte-identical restore; the collector also " + "reclaimed a real orphan the pool had been carrying" + ), "26.0.0-BETA.1": ( - "nested + providers; 274-snapshot recursive backup of a 292-dataset pool, " - "restored a 4-deep child dataset byte-identical" + "v0.7.0: 274-snapshot recursive backup of a 292-dataset pool; restored a " + "4-deep child dataset byte-identical; zvol-orphan case reproduced then closed" ), }