fix: second audit — the delete check did nothing on a real box, and five guards were untestable

The most important finding is that the FIRST audit's fix was wrong.

_can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines
`delete` on the BASE class and dispatches to self.do_delete at call time, so a bound
`delete` exists on every CRUDService subclass whether or not it still implements one.
The check was therefore answering "is this a CRUDService?" — precisely the weaker "is
the namespace registered?" question its own docstring said must never be asked. It
would still have picked a gutted pool.snapshot and failed every delete. It now walks
the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining
delete/do_delete counts. The test double was equally wrong: it modelled a gutted
service as object(), a shape middlewared cannot produce, so the test passed against a
fake it could never have caught in the field. It is now CRUDService-shaped.

Also:

- The recursive delete's fast path returned [] without confirming anything was
  destroyed. A delete that returns cleanly is not proof — iX has already gutted
  pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read
  "no survivors" as a clean sweep, dropped the sidecar (the only record), and would
  have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the
  by-name sweep trusts ZFS rather than the API's return value.

- When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not
  symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record
  clears), a lost record does not.

- _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist;
  failing to write it must never be invisible.

- stage_nested now refuses UP FRONT when middleware has no usable snapshot delete,
  rather than discovering it after restic has already run.

Tests. The autouse fixture added in the last commit did not work: `runner=_run`,
`mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__
at def time, so monkeypatching the module attribute never reached them. 19 tests were
still reading the real mount table — one matching name from running a real `umount` on
the NAS — and the retry loop really slept. All three are late-bound now; the suite
reads nothing outside tmp_path and runs in 1.1s.

Every mutation the audit reported as SURVIVING now fails the suite: the naive delete
check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager
service resolution, compat's method check, compat's unknown handling, a single-quoted
filtered query in apply.sh, and the get-service assumption.

Also: fingerprint() folded `unknown` problems into a broken module, so one transient
429 rewrote the bug report and the next clean run rewrote it back. Problems are
state-tagged; only definite breakage is digested.

Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep
child dataset.
This commit is contained in:
2026-07-14 00:10:10 +00:00
parent 928d0d1973
commit ce6998a935
5 changed files with 593 additions and 60 deletions
+15 -3
View File
@@ -494,7 +494,10 @@ def check_source(a: Assumption, src: str | None) -> tuple[str, str | None]:
breaks a box that was working.
"""
if src is None:
return "broken", f"{a.path} does not exist"
# Name the SYMBOL, not just the file. Whoever reads the bug report needs to
# know what the patch can no longer reach, and "utils/plugins.py does not
# exist" does not tell them that `get_service` is gone.
return "broken", f"{a.path} does not exist, so `{a.symbol}` is gone"
try:
tree = ast.parse(src)
@@ -629,6 +632,7 @@ def check(loader, modules=None) -> dict:
out[a.module]["unknown"] = True
out[a.module]["problems"].append({
"id": a.id, "detail": f"could not read {a.path}: {e}", "why": a.why,
"state": "unknown",
})
continue
@@ -636,12 +640,12 @@ def check(loader, modules=None) -> dict:
if status == "broken":
out[a.module]["ok"] = False
out[a.module]["problems"].append({
"id": a.id, "detail": detail, "why": a.why,
"id": a.id, "detail": detail, "why": a.why, "state": "broken",
})
elif status == "unknown":
out[a.module]["unknown"] = True
out[a.module]["problems"].append({
"id": a.id, "detail": detail, "why": a.why,
"id": a.id, "detail": detail, "why": a.why, "state": "unknown",
})
# The methods the injected code CALLS, not just the symbols it wraps.
@@ -680,11 +684,13 @@ def check(loader, modules=None) -> dict:
out[c.module]["unknown"] = True
out[c.module]["problems"].append({
"id": c.id, "detail": "; ".join(details), "why": c.why,
"state": "unknown",
})
else:
out[c.module]["ok"] = False
out[c.module]["problems"].append({
"id": c.id, "detail": "; ".join(details), "why": c.why,
"state": "broken",
})
for module, (path, phrase, native_when_present) in NATIVE_PROBES.items():
@@ -1026,6 +1032,12 @@ def fingerprint(rows: list[dict]) -> str:
for mod, m in r["modules"].items()
if is_broken(m)
for p in m["problems"]
# `unknown` problems are things we could not READ (a 429, an EACCES), not
# things iX changed. On a ref that is broken for some other reason they would
# otherwise join the digest, so one transient network blip rewrites the issue
# body and the next clean run rewrites it back. That is the daily-noise
# failure this fingerprint exists to prevent, wearing a different hat.
if p.get("state", "broken") == "broken"
)
return hashlib.sha256(repr(findings).encode()).hexdigest()[:16]