Support ZFS snapshots on datasets with child datasets
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing child datasets: This option is only available for datasets that have no further nesting That excludes every pool running Apps, where each app is its own dataset and often has config/pgdata children. Without the option the backup reads live files, so databases are captured mid-write and an app that continuously rewrites its files can stall a run as restic chases a moving target. The stock guard is correct and must not simply be removed. create_snapshot() already takes a recursive ZFS snapshot, but points the backup tool at the parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there: /mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data Deleting the check would make restic walk a near-empty tree, report success, and upload almost nothing. Implement the missing traversal instead. After the recursive snapshot is taken, each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a staging tree mirroring the original layout, and the backup tool is pointed at the staging root. The guard is relaxed only after that machinery is in place. Safety properties: - staging failure aborts the backup; a partial tree is never handed to restic - a post-mount pass asserts every target is a mountpoint and the root is non-empty, so this cannot regress into the empty backup it exists to prevent - apply.sh patches crud.py last, so a partial failure leaves the guard intact rather than exposing "guard removed, traversal missing" - every injected block no-ops when _truecloud_nested is absent - unmountable/locked datasets are skipped and reported, never dropped silently - scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays The staging root is stable per task, so restic can find its parent snapshot between runs; stock's timestamped .zfs path changes every run and forces a full re-scan. Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that compile the *_BLOCK strings, which are Python source appended to live middlewared modules and were previously unchecked. Also: sync stale version strings, untrack a committed .pyc, gitignore __pycache__.
This commit is contained in:
@@ -1,5 +1,89 @@
|
||||
# Changelog
|
||||
|
||||
## v0.3.0 — 2026-07-12
|
||||
|
||||
### Added
|
||||
|
||||
- **`snapshot = true` now works on datasets that have child datasets.**
|
||||
Stock TrueNAS refuses this with *"This option is only available for datasets
|
||||
that have no further nesting"*, which makes the snapshot option unusable for
|
||||
the single most common case on any box running Apps — every app is its own
|
||||
dataset, often with `config`/`pgdata` children of its own. Without it, the
|
||||
backup reads **live** files: databases are captured mid-write, and a busy app
|
||||
rewriting its files can stall a backup indefinitely as restic chases a moving
|
||||
target.
|
||||
|
||||
The stock guard is **correct, and it is not an arbitrary limit.**
|
||||
`plugins/cloud/snapshot.py` already takes a *recursive* ZFS snapshot, but it
|
||||
then points the backup tool at the **parent** dataset's
|
||||
`.zfs/snapshot/<snap>/` directory — and ZFS does not expose child datasets
|
||||
through a parent's snapshot directory:
|
||||
|
||||
```
|
||||
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries (children invisible)
|
||||
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
|
||||
```
|
||||
|
||||
So without the guard the backup tool would walk a near-empty tree, report
|
||||
SUCCESS, and upload almost nothing. iX gate the config rather than ship a
|
||||
backup that lies about succeeding.
|
||||
|
||||
This release implements the missing half. After the (already recursive)
|
||||
snapshot is taken, every descendant dataset's own `.zfs/snapshot/<snap>` is
|
||||
bind-mounted into a **staging tree** mirroring the original layout, and the
|
||||
backup tool is pointed at the staging root — a complete, consistent,
|
||||
point-in-time view of the whole subtree. Only then is the guard relaxed.
|
||||
|
||||
Safety properties, in order of importance:
|
||||
|
||||
- **Staging failure is loud.** If any descendant cannot be staged, the backup
|
||||
fails. A silently-incomplete backup is the exact outcome the stock guard
|
||||
exists to prevent, and it would be worse than not having the feature.
|
||||
- **A post-mount verification pass** asserts every planned target is really a
|
||||
mountpoint and the staging root is non-empty, so this can never regress into
|
||||
the empty-backup failure it is meant to fix.
|
||||
- **The guard is relaxed last.** `apply.sh` installs the traversal, patches
|
||||
`snapshot.py`, then `sync.py`, and only then `crud.py`. A partial failure
|
||||
leaves the guard intact and the option merely unavailable — never
|
||||
"guard removed, traversal missing".
|
||||
- **Every injected block no-ops** if `_truecloud_nested` is absent.
|
||||
- Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
|
||||
unmounted/locked, encrypted-and-locked) are skipped and **reported** —
|
||||
never dropped silently.
|
||||
- Scoped to `cloud_backup` only. Cloud Sync (rclone) shares the same
|
||||
validation mixin but has no staging teardown wired in, so its guard is left
|
||||
in place deliberately.
|
||||
|
||||
Side benefit: the staging root is a **stable** path per task, so restic can
|
||||
find its parent snapshot between runs. Stock's
|
||||
`.zfs/snapshot/<name>-<timestamp>/` path changes every run, which defeats
|
||||
restic's parent detection and forces a full re-scan each time.
|
||||
|
||||
- **CI** (GitHub Actions): shellcheck + `bash -n` on every script, ruff, and
|
||||
pytest on Python 3.11/3.12/3.13. Includes tests that `compile()` the
|
||||
`*_BLOCK` strings — they are Python source appended to live middlewared
|
||||
modules, so a syntax error there would break the box at boot, and nothing
|
||||
previously checked them.
|
||||
|
||||
### Changed
|
||||
|
||||
- Version strings in `install.sh`, `uninstall.sh`, and `recover.sh` were stale
|
||||
at `0.0.4`; all scripts now report the same version.
|
||||
- `patch_ui.py`: replaced a `try`/`except`/`pass` with `contextlib.suppress`
|
||||
(no behaviour change; satisfies the new lint gate).
|
||||
|
||||
### Removed
|
||||
|
||||
- `patch/__pycache__/create_task.cpython-314.pyc` was committed to the
|
||||
repository; it is now untracked and `__pycache__/` is gitignored.
|
||||
|
||||
### Known issues
|
||||
|
||||
- Stock `restic_backup()` deletes the ZFS snapshot in its own `finally`, which
|
||||
fails with `EBUSY` while the staging bind mounts pin it. It logs one benign
|
||||
`Error deleting snapshot ...` warning per run; the patch then unmounts and
|
||||
deletes the snapshot for real. The warning is expected and harmless.
|
||||
|
||||
## v0.2.1 — 2026-07-09
|
||||
|
||||
### Fixed
|
||||
|
||||
Reference in New Issue
Block a user