fix: own the snapshot sweep unconditionally; align the runtime and the manifest
Four audits of the TrueNAS 26 branch. The findings, in severity order. 1. TrueNAS 26 orphaned a snapshot on every run, with no backstop. Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours. <= 25.10 its create_snapshot called get_dataset_recursive() — the same function this module vendors — so "stock went recursive" and "we have something to stage" were the same question. 26 uses filesystem.statfs: recursive = (path == the dataset's mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage. The patch then handed the snapshot back to stock, which destroys the parent only. No staging tree meant no sidecar, and the GC only ever ran from stage_nested — so nothing on the box would ever have found the children. Reproduced on the VM: one orphan per zvol, every run, forever, backup green. Ownership of the sweep is no longer conditional on staging (own_snapshot()). 2. The runtime resolved a NAMESPACE; compat.py verified a METHOD. get_service() only proves a namespace is registered. compat checks the namespace AND that it defines delete/do_delete. So if iX guts the method but keeps the service — which they have already done to pool.snapshot.do_update on master — compat falls through to zfs.snapshot and reports the box healthy, while the runtime picks pool.snapshot and fails every delete. Both sides now ask "can this namespace delete?", and a test binds the two lists together. 3. query_filesystems() silently dropped malformed rows — the one remaining silent-omission path, and a direct contradiction of the cardinal rule. It raises now. A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed. 4. The retry loop discarded the delete error and reported every survivor as "(still busy?)" — naming the one cause that is benign and hiding the ones that are permanent. It keeps and reports the real error. Also: the staging-failure handler could lose the original exception if its own sweep raised; get_service is now a checked assumption; normalise_dataset and two dead MiddlewareCall properties removed; stale comments corrected. Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148 snapshots) and passed here only because this box has no zfs binary — they would have gone red on the NAS, which is the one machine the release process requires them green on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of the five fixes above now fails the suite; before, all 293 passed. Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4 (pool.snapshot.delete honours recursive=True).
This commit is contained in:
+25
-6
@@ -535,8 +535,8 @@ if _tc_nested is not None:
|
||||
# snapshot=true, not just ours. Two consequences, and the second is worse:
|
||||
#
|
||||
# * everything below is a NEW failure mode for tasks that worked before we
|
||||
# were installed. A `pool.dataset.query` that errors would break a
|
||||
# CloudSync job we have no business touching.
|
||||
# were installed. A `zfs list` that errors would break a CloudSync job
|
||||
# we have no business touching.
|
||||
# * if a CloudSync task ever were staged, nothing would ever tear it down:
|
||||
# the teardown is wired into cloud_backup's restic_backup finally, and
|
||||
# CRUD_BLOCK deliberately leaves CloudSync's nesting guard intact. The
|
||||
@@ -568,9 +568,16 @@ if _tc_nested is not None:
|
||||
dataset, nested = _tc_nested.get_dataset_recursive(datasets, path)
|
||||
|
||||
if not nested:
|
||||
# No children: stock behaviour, untouched. Stock's `finally` owns
|
||||
# the snapshot from here (its non-recursive delete is correct,
|
||||
# because a non-nested snapshot has no children).
|
||||
# Nothing to STAGE -- but we still own the SWEEP, and that is not a
|
||||
# formality. Stock decides `recursive` by its own rule, and on 26 that
|
||||
# rule is no longer ours: it snapshots recursively whenever the backup
|
||||
# path IS the dataset's mountpoint (filesystem.statfs), while
|
||||
# get_dataset_recursive() sees nothing to stage when the only
|
||||
# descendants are ZVOLs or legacy/none-mountpoint datasets. Stock then
|
||||
# deletes the PARENT ONLY. Without this, one snapshot per descendant is
|
||||
# orphaned on every run, forever, with no sidecar and no GC to find it --
|
||||
# and the backup still reports success.
|
||||
_tc_nested.own_snapshot(middleware, name, snapshot, logger=_logger)
|
||||
return snapshot, snap_path
|
||||
|
||||
staging_root = _tc_nested.stage_nested(
|
||||
@@ -584,7 +591,19 @@ if _tc_nested is not None:
|
||||
# stays None and its `finally` deletes NOTHING. Sweep the tree ourselves
|
||||
# or leak the parent plus one snapshot per descendant dataset (160+ here)
|
||||
# on every failed run.
|
||||
_tc_nested.delete_snapshot_tree(middleware, snapshot, logger=_logger)
|
||||
#
|
||||
# The sweep is itself wrapped: a cleanup that raises would REPLACE the
|
||||
# original exception with its own, hiding why the backup actually failed.
|
||||
# An error handler must not be able to lose the error.
|
||||
try:
|
||||
_tc_nested.delete_snapshot_tree(middleware, snapshot, logger=_logger)
|
||||
except Exception as _tc_sweep_err:
|
||||
if _logger:
|
||||
_logger.error(
|
||||
"truecloud-patch: could not sweep %s after a staging failure "
|
||||
"(%r) -- it is orphaned and must be deleted by hand",
|
||||
snapshot, _tc_sweep_err,
|
||||
)
|
||||
raise
|
||||
|
||||
return snapshot, staging_root
|
||||
|
||||
Reference in New Issue
Block a user