Support ZFS snapshots on datasets with child datasets

TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:

  This option is only available for datasets that have no further nesting

That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.

The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:

  /mnt/Tap/.zfs/snapshot/<snap>/apps/                -> 0 entries
  /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/  -> the real data

Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.

Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.

Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
  non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
  rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays

The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.

Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.

Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
This commit is contained in:
flan
2026-07-12 21:20:22 +00:00
parent 4ded8cff3d
commit a572eb2164
14 changed files with 1113 additions and 15 deletions
+84
View File
@@ -1,5 +1,89 @@
# Changelog
## v0.3.0 — 2026-07-12
### Added
- **`snapshot = true` now works on datasets that have child datasets.**
Stock TrueNAS refuses this with *"This option is only available for datasets
that have no further nesting"*, which makes the snapshot option unusable for
the single most common case on any box running Apps — every app is its own
dataset, often with `config`/`pgdata` children of its own. Without it, the
backup reads **live** files: databases are captured mid-write, and a busy app
rewriting its files can stall a backup indefinitely as restic chases a moving
target.
The stock guard is **correct, and it is not an arbitrary limit.**
`plugins/cloud/snapshot.py` already takes a *recursive* ZFS snapshot, but it
then points the backup tool at the **parent** dataset's
`.zfs/snapshot/<snap>/` directory — and ZFS does not expose child datasets
through a parent's snapshot directory:
```
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries (children invisible)
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
```
So without the guard the backup tool would walk a near-empty tree, report
SUCCESS, and upload almost nothing. iX gate the config rather than ship a
backup that lies about succeeding.
This release implements the missing half. After the (already recursive)
snapshot is taken, every descendant dataset's own `.zfs/snapshot/<snap>` is
bind-mounted into a **staging tree** mirroring the original layout, and the
backup tool is pointed at the staging root — a complete, consistent,
point-in-time view of the whole subtree. Only then is the guard relaxed.
Safety properties, in order of importance:
- **Staging failure is loud.** If any descendant cannot be staged, the backup
fails. A silently-incomplete backup is the exact outcome the stock guard
exists to prevent, and it would be worse than not having the feature.
- **A post-mount verification pass** asserts every planned target is really a
mountpoint and the staging root is non-empty, so this can never regress into
the empty-backup failure it is meant to fix.
- **The guard is relaxed last.** `apply.sh` installs the traversal, patches
`snapshot.py`, then `sync.py`, and only then `crud.py`. A partial failure
leaves the guard intact and the option merely unavailable — never
"guard removed, traversal missing".
- **Every injected block no-ops** if `_truecloud_nested` is absent.
- Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
unmounted/locked, encrypted-and-locked) are skipped and **reported** —
never dropped silently.
- Scoped to `cloud_backup` only. Cloud Sync (rclone) shares the same
validation mixin but has no staging teardown wired in, so its guard is left
in place deliberately.
Side benefit: the staging root is a **stable** path per task, so restic can
find its parent snapshot between runs. Stock's
`.zfs/snapshot/<name>-<timestamp>/` path changes every run, which defeats
restic's parent detection and forces a full re-scan each time.
- **CI** (GitHub Actions): shellcheck + `bash -n` on every script, ruff, and
pytest on Python 3.11/3.12/3.13. Includes tests that `compile()` the
`*_BLOCK` strings — they are Python source appended to live middlewared
modules, so a syntax error there would break the box at boot, and nothing
previously checked them.
### Changed
- Version strings in `install.sh`, `uninstall.sh`, and `recover.sh` were stale
at `0.0.4`; all scripts now report the same version.
- `patch_ui.py`: replaced a `try`/`except`/`pass` with `contextlib.suppress`
(no behaviour change; satisfies the new lint gate).
### Removed
- `patch/__pycache__/create_task.cpython-314.pyc` was committed to the
repository; it is now untracked and `__pycache__/` is gitignored.
### Known issues
- Stock `restic_backup()` deletes the ZFS snapshot in its own `finally`, which
fails with `EBUSY` while the staging bind mounts pin it. It logs one benign
`Error deleting snapshot ...` warning per run; the patch then unmounts and
deletes the snapshot for real. The warning is expected and harmless.
## v0.2.1 — 2026-07-09
### Fixed