fix: own the snapshot sweep unconditionally; align the runtime and the manifest

Four audits of the TrueNAS 26 branch. The findings, in severity order.

1. TrueNAS 26 orphaned a snapshot on every run, with no backstop.

Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours.
<= 25.10 its create_snapshot called get_dataset_recursive() — the same function this
module vendors — so "stock went recursive" and "we have something to stage" were the
same question. 26 uses filesystem.statfs: recursive = (path == the dataset's
mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint
datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage.

The patch then handed the snapshot back to stock, which destroys the parent only. No
staging tree meant no sidecar, and the GC only ever ran from stage_nested — so
nothing on the box would ever have found the children. Reproduced on the VM: one
orphan per zvol, every run, forever, backup green.

Ownership of the sweep is no longer conditional on staging (own_snapshot()).

2. The runtime resolved a NAMESPACE; compat.py verified a METHOD.

get_service() only proves a namespace is registered. compat checks the namespace AND
that it defines delete/do_delete. So if iX guts the method but keeps the service —
which they have already done to pool.snapshot.do_update on master — compat falls
through to zfs.snapshot and reports the box healthy, while the runtime picks
pool.snapshot and fails every delete. Both sides now ask "can this namespace
delete?", and a test binds the two lists together.

3. query_filesystems() silently dropped malformed rows — the one remaining
silent-omission path, and a direct contradiction of the cardinal rule. It raises now.
A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed.

4. The retry loop discarded the delete error and reported every survivor as
"(still busy?)" — naming the one cause that is benign and hiding the ones that are
permanent. It keeps and reports the real error.

Also: the staging-failure handler could lose the original exception if its own sweep
raised; get_service is now a checked assumption; normalise_dataset and two dead
MiddlewareCall properties removed; stale comments corrected.

Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148
snapshots) and passed here only because this box has no zfs binary — they would have
gone red on the NAS, which is the one machine the release process requires them green
on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of
the five fixes above now fails the suite; before, all 293 passed.

Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset
backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4
(pool.snapshot.delete honours recursive=True).
This commit is contained in:
2026-07-13 23:30:48 +00:00
parent 605231b39f
commit 413cd60ed4
5 changed files with 566 additions and 153 deletions
+37 -12
View File
@@ -133,9 +133,43 @@ ASSUMPTIONS = [
params=["middleware", "job", "cloud_backup"],
why="SYNC_BLOCK wraps it to tear down bind mounts in a finally",
),
Assumption(
# Not a plugin method -- a method on the middleware OBJECT itself, which the
# manifest had no way to express and therefore never checked.
#
# The nested module calls `middleware.get_service(<ns>)` to decide whether to
# sweep snapshots through `pool.snapshot` or `zfs.snapshot` (see
# SNAPSHOT_SERVICES). If it ever disappears, `_can_delete()` catches the
# AttributeError, reports BOTH namespaces unusable, and every nested backup
# fails -- loudly, but only at RUN time, on a box the preflight had already
# declared healthy. Checking it costs one file read.
"get-service", NESTED, "utils/plugins.py",
"LoadPluginsMixin.get_service", kind="method",
params=["self", "name"],
why="snapshot_service() resolves the snapshot namespace through it; without "
"it the module cannot sweep the snapshot it just took",
),
]
def accepted_spellings(name):
"""The method names that satisfy a call to `<namespace>.<name>`.
A CRUDService exposes `create`/`update`/`delete` from methods NAMED
`do_create`/`do_update`/`do_delete`. Both are live across the matrix: 24.10 and
25.04 declare `do_delete`, 25.10 renamed it to `delete`, and all of them answer
to `<ns>.delete`. Accepting only the literal name reported working releases as
BROKEN and would have switched nested snapshots off on boxes where they work.
"""
return (name, f"do_{name}")
#: The spellings that satisfy `<ns>.delete`. A test binds this to the runtime's
#: `truecloud_nested.DELETE_METHODS`, so the checker and the patch cannot come to
#: disagree about what "can delete" means on the same box.
DELETE_NAMES = accepted_spellings("delete")
class MiddlewareCall:
"""A middlewared METHOD the injected code calls at runtime.
@@ -195,14 +229,6 @@ class MiddlewareCall:
def name_of(method):
return method.rsplit(".", 1)[1]
@property
def namespace(self):
return self.namespace_of(self.method)
@property
def name(self):
return self.name_of(self.method)
#: Every middlewared method the nested module calls at runtime.
#: The middleware methods the nested module CALLS.
@@ -220,9 +246,8 @@ class MiddlewareCall:
#: cannot be deleted from under us the way `zfs.*` just was.
#: * The same methods, in the same files, exist on 24.10 through 26. One code
#: path, no version conditionals.
#: * `pool.snapshot.delete` takes `recursive`, which the private call did not.
#: The old sweep had to enumerate ~250 snapshots and delete them one at a
#: time, and any it missed leaked forever.
#: * Both spellings take `recursive`, so ONE call sweeps the whole tree instead
#: of ~250 individual deletes, any of which could be missed.
MIDDLEWARE_CALLS = [
MiddlewareCall(
"call-snapshot-delete", NESTED, "pool.snapshot.delete",
@@ -303,7 +328,7 @@ def check_call(c: MiddlewareCall, src: str | None,
n.name for n in ast.walk(tree)
if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef)
}
if name not in defined and f"do_{name}" not in defined:
if not any(sp in defined for sp in accepted_spellings(name)):
return "broken", f"{path} no longer defines `{method}`"
return "ok", None