16 Commits

Include renames

Author	SHA1	Message	Date
flan	827c4358fd	Skip the chmod-based unreadable-sidecar tests when running as root ... The Gitea runner image executes jobs as root, and chmod(0) cannot make a file unreadable for root (CAP_DAC_OVERRIDE) — the two tests failed there while passing on GitHub's non-root runner. Skipping as root keeps the scenario exercised everywhere it is constructible.	2026-08-03 20:28:37 +00:00
flan	df412eeff7	test: make three vacuous tests actually test something; drop one duplicate ... The test suite is not bloated -- 3,701 lines of test code against 3,760 lines of product code, one duplicate pair in 356 tests, 8% single-assertion tests. The waste was not volume, it was four tests that looked like coverage and provided none: - test_parents_are_mounted_before_children: the fixture was already in depth order, so the sort it exists for was never exercised. Deleting `mounts.sort(key=_depth)` passed the whole suite. It now uses datasets whose NAME order differs from their MOUNTPOINT depth, which is the only case the sort is for. - test_it_NEVER_touches_the_current_run: the "current" snapshot was a minute old, so the age floor excluded it regardless and the same-snapname guard never ran. That guard only matters for a run that OUTLIVES the floor -- which a first full upload easily does, and where collecting it would yank the snapshot out from under a backup that is still reading from it. Now tested with a 12-hour-old current run. - the fingerprint/unknown test put the unreadable file in `providers`, which is not broken -- so fingerprint() skipped the whole module via is_broken() and the filter under test never executed. The blip has to land in the module that IS broken. - test_the_real_truenas_versions was a byte-identical copy of test_async_middleware_is_detected under a name promising more. Replaced with the fact actually worth pinning: the flavour probe must read STOCK source, because apply.sh re-runs on an already-patched overlay and our own SNAPSHOT_SYNC block is a plain `def create_snapshot` -- reading it would report a 25.10 box as synchronous and inject the wrong wrapper. All four now fail when the code they name is broken.	2026-07-14 01:41:25 +00:00
flan	086b20ed23	fix: fourth audit — two regressions from the last fix, and the boot preflight had no test ... Two of these were mine, from the previous round. - mounted_snapshots still swallowed OSError. I said I had fixed it and had not: the edit never matched, and I did not read it back. With the mount table unreadable the GC loses its in-use protection entirely and can destroy the snapshots of a backup that is still uploading (a first upload easily outlives the 1h age floor). It raises now, and both behaviours are tested. - The foreign-dataset check added last round had two bugs of its own. It ignored `mounted`, so a locked/encrypted dataset from a sibling tree turned a working nightly backup into a permanent failure — it belongs in `skipped`, exactly as an in-tree one does. And it tested `mp.startswith(path + "/")`, so a foreign dataset mounted EXACTLY at the backup path slipped through — the very hole the check was added to close, one character wide, and the worse case of the two because it SHADOWS the base dataset's own directory. - _read_sidecar's new raise broke cleanup_all, which is what recover.sh and uninstall.sh call — i.e. the code that must work when the box is ALREADY stuck. One unreadable sidecar aborted it before it unmounted anything, leaving the staging tree mounted, which pins the snapshots, which is the state recover.sh exists to escape. It now reports and carries on — and does not delete a record it could not read. - compat could report a FALSE OK: `defined` was collected by walking the whole file, so any function named `delete` anywhere in it — on an unrelated class, or nested inside another method — satisfied "this namespace defines delete". The runtime is stricter (a plugin class on the service's MRO), so the two could disagree in the ok direction. compat now looks in the class that declares the namespace. Same question on both sides, which is what pick_snapshot_service's docstring has been claiming all along. - apply.sh's compat preflight — the guard that refuses to patch a middleware whose assumptions no longer hold, on every boot, on a live NAS — had no test at all. It could be turned into a no-op eight different ways with the suite still green. The SHIPPED heredoc is now extracted and driven directly against fake verdicts. Also pinned: the Tap/Tap2 prefix collisions (a sweep that treats "Tap2/data@snap" as part of Tap's tree DESTROYS another pool's snapshot), and the GC's in_use wiring. 355 tests. Verified on TrueNAS 26.0.0-BETA.1: 292-dataset backup, 0 orphans, 0 leaked mounts, byte-identical restore of a 4-deep child dataset.	2026-07-14 01:34:13 +00:00
flan	8a41d7d7ef	fix: third audit — a cross-tree dataset was omitted silently, and the block tests passed on comments ... D1, the only cardinal-rule violation left. plan_staging scopes by dataset NAME, which is right (a dataset with no mountpoint cannot be scoped by path). But ZFS lets any dataset mount anywhere, so one from a DIFFERENT tree can sit inside the backup path: Tank/photos mountpoint=/mnt/Tap/apps/photos It holds data inside the path, and `zfs snapshot -r Tap@...` does NOT cover it — recursion follows the dataset tree, not the directory tree. It fell out of the name filter and vanished: not staged, not in `skipped`, no error. The backup reported SUCCESS with that data missing. Stock has the same blind spot but refuses the nested config outright; we are the ones relaxing that guard, so the hole is ours. It now raises. The test suite was the real weakness. apply.sh's injected blocks carry the highest-consequence logic in the project — the run_in_thread hop, the flavour selection, the finally-teardown, the re-raise — and were guarded only by substring greps. Two of them passed on COMMENTS: `assert "raise" in block` was satisfied by a comment reading "a cleanup that raises...", and `assert "cleanup_task" in block` by "cleanup_task gets logger=None". Deleting the actual re-raise (restic then backs up the UN-STAGED path — the silently-empty backup this module exists to prevent) and deleting the actual cleanup call from the finally (~250 orphans per run) both left the suite green. They are asserted structurally now, against the parsed block. Eleven regressions the audit found surviving now fail the suite, including: a swallowed staging failure, a missing teardown, an inverted flavour mapping, blocking work back on the asyncio event loop, the host's deleted get_dataset_recursive, query_filesystems quietly preferring the filtered middleware query, and a re-frozen `runner`/`sleep`/ `mounts_file` default (which would silently re-arm 19 tests reading the real mount table on the NAS). Also: _read_sidecar conflated "no sidecar" with "cannot read the sidecar", so cleanup_task took the empty branch and UNLINKED the only record of a tree it could not read. mounted_snapshots returned an empty set on error, silently switching off the GC's protection for snapshots a concurrent run is using. Both raise now. Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup 0 orphans / 0 leaked mounts, byte-identical restore of a 4-deep child dataset.	2026-07-14 00:38:45 +00:00
flan	ce6998a935	fix: second audit — the delete check did nothing on a real box, and five guards were untestable ... The most important finding is that the FIRST audit's fix was wrong. _can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines `delete` on the BASE class and dispatches to self.do_delete at call time, so a bound `delete` exists on every CRUDService subclass whether or not it still implements one. The check was therefore answering "is this a CRUDService?" — precisely the weaker "is the namespace registered?" question its own docstring said must never be asked. It would still have picked a gutted pool.snapshot and failed every delete. It now walks the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining delete/do_delete counts. The test double was equally wrong: it modelled a gutted service as object(), a shape middlewared cannot produce, so the test passed against a fake it could never have caught in the field. It is now CRUDService-shaped. Also: - The recursive delete's fast path returned [] without confirming anything was destroyed. A delete that returns cleanly is not proof — iX has already gutted pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read "no survivors" as a clean sweep, dropped the sidecar (the only record), and would have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the by-name sweep trusts ZFS rather than the API's return value. - When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record clears), a lost record does not. - _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist; failing to write it must never be invisible. - stage_nested now refuses UP FRONT when middleware has no usable snapshot delete, rather than discovering it after restic has already run. Tests. The autouse fixture added in the last commit did not work: `runner=_run`, `mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__ at def time, so monkeypatching the module attribute never reached them. 19 tests were still reading the real mount table — one matching name from running a real `umount` on the NAS — and the retry loop really slept. All three are late-bound now; the suite reads nothing outside tmp_path and runs in 1.1s. Every mutation the audit reported as SURVIVING now fails the suite: the naive delete check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager service resolution, compat's method check, compat's unknown handling, a single-quoted filtered query in apply.sh, and the get-service assumption. Also: fingerprint() folded `unknown` problems into a broken module, so one transient 429 rewrote the bug report and the next clean run rewrote it back. Problems are state-tagged; only definite breakage is digested. Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup 0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep child dataset.	2026-07-14 00:10:10 +00:00
flan	413cd60ed4	fix: own the snapshot sweep unconditionally; align the runtime and the manifest ... Four audits of the TrueNAS 26 branch. The findings, in severity order. 1. TrueNAS 26 orphaned a snapshot on every run, with no backstop. Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours. <= 25.10 its create_snapshot called get_dataset_recursive() — the same function this module vendors — so "stock went recursive" and "we have something to stage" were the same question. 26 uses filesystem.statfs: recursive = (path == the dataset's mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage. The patch then handed the snapshot back to stock, which destroys the parent only. No staging tree meant no sidecar, and the GC only ever ran from stage_nested — so nothing on the box would ever have found the children. Reproduced on the VM: one orphan per zvol, every run, forever, backup green. Ownership of the sweep is no longer conditional on staging (own_snapshot()). 2. The runtime resolved a NAMESPACE; compat.py verified a METHOD. get_service() only proves a namespace is registered. compat checks the namespace AND that it defines delete/do_delete. So if iX guts the method but keeps the service — which they have already done to pool.snapshot.do_update on master — compat falls through to zfs.snapshot and reports the box healthy, while the runtime picks pool.snapshot and fails every delete. Both sides now ask "can this namespace delete?", and a test binds the two lists together. 3. query_filesystems() silently dropped malformed rows — the one remaining silent-omission path, and a direct contradiction of the cardinal rule. It raises now. A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed. 4. The retry loop discarded the delete error and reported every survivor as "(still busy?)" — naming the one cause that is benign and hiding the ones that are permanent. It keeps and reports the real error. Also: the staging-failure handler could lose the original exception if its own sweep raised; get_service is now a checked assumption; normalise_dataset and two dead MiddlewareCall properties removed; stale comments corrected. Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148 snapshots) and passed here only because this box has no zfs binary — they would have gone red on the NAS, which is the one machine the release process requires them green on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of the five fixes above now fails the suite; before, all 293 passed. Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4 (pool.snapshot.delete honours recursive=True).	2026-07-13 23:30:48 +00:00
flan	605231b39f	feat: TrueNAS 26 support; enumerate datasets and snapshots from ZFS, not middleware ... TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query, zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested module's critical path, so nested snapshots were BROKEN on 26. Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10 and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported release. tools/compat.py checks the same list the runtime uses, so what CI verifies and what runs cannot drift apart. Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is the point of this commit. Those methods exist, are documented, and are covered by iX's deprecation policy — and they are not like-for-like replacements. They apply a visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets on a real pool, including live application data. Staging from that view omits them silently, and plan_staging never sees them, so they do not even reach the skipped list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one per hidden dataset on every run. So: read the truth from ZFS, make changes through middleware. zfs list cannot be filtered by policy and behaves identically on every release. A failing zfs list raises rather than returning an empty list — "no datasets" and "the command broke" must never look the same. No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query, which returns all 270 datasets. The bug existed only in this port. Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a 292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical restore of a four-level-deep child dataset that pool.dataset.query hides.	2026-07-13 22:43:20 +00:00
flan	0d04c2cd1c	Collect orphaned snapshots by name: the sidecar lives in tmpfs ... A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the recursive snapshot and its cleanup destroyed that record, leaving one snapshot per descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would ever have found them. gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and the collector only mops up what the record lost. It deletes data on a name match, which is a weaker claim than a recorded fact, so the selection is a pure function with the harshest tests here. A snapshot is collected only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a concurrent backup), and it is over an hour old. Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it selects exactly the orphans of the task being run and nothing else.	2026-07-13 19:56:32 +00:00
flan	2b8ef107f7	The sidecar must carry EVERY pending tree, not just the newest ... Found live, in the code written to prevent exactly this. The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of the survivor -- orphaning it permanently. Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46 reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it behind again, and then wrote its own snapshot over the record. Permanent orphan, created by the safety net. The sidecar is now a list. stage_nested carries forward whatever a reclaim could not delete; cleanup_task sweeps every pending tree and writes back only the survivors. cleanup_all reports them one per line instead of formatting a list into an f-string at the user during uninstall. Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots with no straggler.	2026-07-13 19:35:44 +00:00
flan	b50567e9a7	A few snapshots leaked on every nested run, forever ... Found on real hardware: a 256-snapshot backup of /mnt/Tap swept 253 and left 3 with 'dataset is busy'. ZFS AUTOMOUNTS <dataset>/.zfs/snapshot/<snap> when it is read, and keeps it mounted for zfs_expire_snapshot seconds (300 default) after the last access. teardown() unmounts OUR bind mounts but not the automount underneath, so zfs destroy refuses for exactly the datasets restic read most recently. cleanup_task() then removed the sidecar anyway -- destroying the only record those snapshots existed. Nothing would ever have reclaimed them. - release_snapdirs() unmounts ZFS's own automounts (deepest first) before deleting. - delete_snapshot_tree() retries the transient busy and RETURNS what it could not delete, instead of swallowing it. - The sidecar is removed only on a confirmed-clean sweep -- including on the staging-failure path, which used to remove it before the caller swept. A sidecar left behind when the tree is gone costs one no-op delete; a sidecar removed while the tree exists is unrecoverable.	2026-07-13 19:18:22 +00:00
flan	498b2690e1	TrueNAS 26 support: one sync implementation, two wrappers ... 26 rewrites cloud_backup from async to synchronous AND deletes get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's namespace. Either is a broken backup found at restore time. The nested module is now one synchronous implementation talking to middlewared via call_sync, behind two thin wrappers. apply.sh reads which flavour the installed middleware declares and injects the matching one: <= 25.10 reaches it through 'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls it directly. The snapshot/bind-mount/failure logic exists once -- an async twin would mean every future fix had to land twice. A middleware whose three wrapped functions disagree about asyncness is refused, not guessed at. get_dataset_recursive is vendored, removing the dependency on both versions rather than asserting it. master stays BROKEN on purpose: iX are still renaming middleware->context, cloud_backup->entry and adding a required credentials param there. Chasing a branch that moves daily is how you ship a patch nobody tested.	2026-07-13 18:18:28 +00:00
flan	8421a34d8d	Delete the snapshot tree atomically instead of 252 calls ... delete_snapshot_tree removed the parent and every child snapshot individually. On a real pool `zfs snapshot -r` creates one snapshot per descendant dataset -- 252 on Tap -- so cleanup was 252 sequential middleware calls. Slow, but the real problem is that it is not atomic: a job killed part-way through the sweep leaves behind exactly the orphaned snapshots this function exists to prevent. zfs.snapshot.delete accepts {"recursive": True}, which destroys the parent and all children in one call. Use that as the fast path and keep the name-by-name sweep as the fallback -- it is still needed when the parent is already gone (stock's finally can win the race once our mounts are released), which makes a recursive delete fail while the children survive. The test fake now emulates real `zfs destroy -r` semantics, so a test cannot pass while the shipped code deletes only the parent. 76 tests, ruff and shellcheck clean.	2026-07-13 14:35:04 +00:00
flan	24f1f2c648	Fix post-merge audit findings; unify staging teardown ... Audit ----- - create_task.py verify failed on a DEFAULT install. hook_status.json emitted a per-file entry for the nested module with ok:false whenever the feature was switched off -- the default -- so verify printed [FAIL] and exited 1, right after the README tells users to run it. Status is now per MODULE with an `active` flag, and verify renders an inactive module as [SKIP]. - A partial apply suppressed the middlewared restart. The exit code conflated "nothing applied" with "one module applied, one failed", so a failing providers patch would prevent the restart that a freshly-applied nested patch needs, leaving it on disk and never loaded. Exit 2 now means partial and the restart still fires. - The native-nested probe could never fire. It scanned crud.py for the guard message, but our own injected block quotes that message, so once applied the probe would always conclude the guard was still present. It now reads only the stock portion of the file. - recover.sh did not unmount staging trees, so an emergency recovery left bind mounts pinning ZFS snapshots that could then never be destroyed. - uninstall.sh deleted sidecar files without reading them. A sidecar is the only record that an interrupted run's snapshot tree is still on disk; both scripts now name the snapshot before clearing it. - Removed a dead branch in the restart gate (unreachable: the kill switch exits). Refactor -------- - Staging teardown had been copy-pasted into uninstall.sh and recover.sh -- two untested shell copies of the fiddly depth-ordering and lazy-umount logic. Both now call `python3 patch/truecloud_nested.py cleanup`, so there is exactly one implementation and it is the one under test. - Dropped the in-memory ACTIVE dict. The sidecar file was already the source of truth; a second in-process record could only desync -- and it is precisely the middlewared-restart case (which empties it) that must not orphan a snapshot tree. One record, on disk, or none. Not done: the overlay-unmount loop is duplicated across apply.sh/uninstall.sh/ recover.sh. It is pre-existing, and apply.sh runs at PREINIT under a tight timeout -- giving it a source dependency would trade 10 lines of duplication for a boot-time failure mode. 74 tests, ruff and shellcheck clean.	2026-07-12 22:29:31 +00:00
flan	a80de88078	Distinguish a missing snapshot from an unreadable one ... os.path.isdir() returns False both when a snapshot directory does not exist and when it cannot be stat'd. Staging aborted either way -- correct -- but reported every case as "has no snapshot", which sends you hunting for a snapshot that is sitting right there. Found while dry-running the planner against a real recursive snapshot of Tap: running as a non-root user, /mnt/Tap/apps/paperless/data is mode 0700 and the probe reported "has no snapshot" when `zfs list` showed the snapshot present. Middleware runs as root so this would not fire in production, but a backup system must not misreport why it failed. plan_staging now takes a probe() that classifies the path as ok / missing / unreadable, and the error names which. Dry-run results against Tap (250 datasets, real `zfs snapshot -r`): - 170 mounted filesystems under /mnt/Tap/, and the plan produces exactly 170 descendant mounts -- no omissions - 18 legacy-mountpoint datasets reported as skipped, never dropped silently - Tap/ix-apps mounts at /mnt/.ix-apps, correctly outside the backup path - parent snapshot exposes 0 entries under /apps; the staged sources expose 71, and lidarr/config resolves with lidarr.db present - snapshot_tree_names() identifies all 250; deleting only the parent (what stock does) leaves 249 orphans, and the sweep clears them	2026-07-12 22:04:56 +00:00
flan	bb26edf351	Make nested snapshots opt-in; fix snapshot leaks found in audit ... Opt-in ------ Nested-dataset snapshot support changes how backups read their source data, so it is now off by default and gated behind a marker file: install.sh --enable-nested-snapshots install.sh --disable-nested-snapshots With neither flag install.sh preserves the current setting, so a routine `git pull && bash install.sh` can never silently flip it. When disabled, apply.sh skips the patch entirely and the stock guard remains. uninstall.sh tears down staging mounts and removes the marker. Snapshot lifecycle ------------------ zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls it with no options. Stock is safe only because its validation means recursive is never True in the field. Enabling nested datasets makes recursive snapshots real: the parent then has one child snapshot per descendant dataset (160+ on an Apps pool), so stock's delete would orphan every child on EVERY successful run. The patch now owns the lifecycle end to end: - delete_snapshot_tree() sweeps the parent and all children, and is idempotent against stock's finally winning the race once our mounts are released - on a staging failure the tree is deleted here, because sync.py never completes `snapshot, local_path = await create_snapshot(...)` and so its finally deletes nothing at all - the snapshot is recorded in a sidecar file before anything is mounted, so a middlewared restart mid-backup cannot orphan it - a crashed run's snapshot tree is reclaimed on the next run instead of being overwritten and leaked Silent-omission fix ------------------- The dataset list is now enumerated AFTER the snapshot. Read beforehand it could miss a dataset created in the gap, which the recursive snapshot would capture but the staging plan would not -- silently omitting its data. Read afterwards, an unsnapshotted dataset trips the staging check and fails the run loudly. Also from the audit ------------------- - plan_staging scopes by dataset name, so skipped-dataset warnings no longer include every mountpoint-less dataset on the box, which buried the ones that matter - staging_root_for rejects "." / ".." components that would escape the staging base, and resolves STAGING_BASE at call time rather than freezing it into a default argument - uninstall.sh no longer `rm -rf`s a tree that may still contain live bind mounts, and unmounts by path depth rather than string length - apply_plan takes an injectable isdir; verify_staged drops an unused parameter - pin the shellcheck action instead of tracking @master 61 tests, ruff and shellcheck clean.	2026-07-12 21:52:09 +00:00
flan	a572eb2164	Support ZFS snapshots on datasets with child datasets ... TrueCloud Backup's "Take Snapshot" option is rejected on any path containing child datasets: This option is only available for datasets that have no further nesting That excludes every pool running Apps, where each app is its own dataset and often has config/pgdata children. Without the option the backup reads live files, so databases are captured mid-write and an app that continuously rewrites its files can stall a run as restic chases a moving target. The stock guard is correct and must not simply be removed. create_snapshot() already takes a recursive ZFS snapshot, but points the backup tool at the parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there: /mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data Deleting the check would make restic walk a near-empty tree, report success, and upload almost nothing. Implement the missing traversal instead. After the recursive snapshot is taken, each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a staging tree mirroring the original layout, and the backup tool is pointed at the staging root. The guard is relaxed only after that machinery is in place. Safety properties: - staging failure aborts the backup; a partial tree is never handed to restic - a post-mount pass asserts every target is a mountpoint and the root is non-empty, so this cannot regress into the empty backup it exists to prevent - apply.sh patches crud.py last, so a partial failure leaves the guard intact rather than exposing "guard removed, traversal missing" - every injected block no-ops when _truecloud_nested is absent - unmountable/locked datasets are skipped and reported, never dropped silently - scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays The staging root is stable per task, so restic can find its parent snapshot between runs; stock's timestamped .zfs path changes every run and forces a full re-scan. Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that compile the *_BLOCK strings, which are Python source appended to live middlewared modules and were previously unchecked. Also: sync stale version strings, untrack a committed .pyc, gitignore __pycache__.	2026-07-12 21:20:22 +00:00