Support ZFS snapshots on datasets with child datasets

TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:

  This option is only available for datasets that have no further nesting

That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.

The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:

  /mnt/Tap/.zfs/snapshot/<snap>/apps/                -> 0 entries
  /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/  -> the real data

Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.

Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.

Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
  non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
  rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays

The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.

Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.

Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.

.github/workflows/ci.yml

@@ -0,0 +1,60 @@
+name: CI
+
+on:
+  push:
+    branches: [main, "feat/**", "fix/**"]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  shell:
+    name: shell (shellcheck + syntax)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: bash syntax check
+        run: |
+          fail=0
+          while IFS= read -r f; do
+            bash -n "$f" || { echo "::error file=$f::bash syntax error"; fail=1; }
+          done < <(find . -name '*.sh' -not -path './.git/*')
+          exit $fail
+
+      - name: shellcheck
+        uses: ludeeus/action-shellcheck@master
+        env:
+          SHELLCHECK_OPTS: -S warning -e SC1091
+
+  python:
+    name: python ${{ matrix.python }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # TrueNAS SCALE middleware runs 3.11+; keep the patch importable across
+        # the versions it may be injected into.
+        python: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: install dev deps
+        run: python -m pip install --upgrade pip pytest ruff
+
+      - name: ruff
+        run: ruff check patch tests
+
+      - name: pytest
+        run: pytest tests -v
+
+      - name: verify injected middleware blocks compile
+        # Belt-and-braces: the *_BLOCK strings are appended into live middlewared
+        # modules. A syntax error there would break the box at boot.
+        run: pytest tests/test_apply_blocks.py -v

.gitignore

@@ -4,3 +4,11 @@
 /apply.log.2
 /hook_status.json
 /disabled
+
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.venv/
+venv/

CHANGELOG.md

@@ -1,5 +1,89 @@
 # Changelog
 
+## v0.3.0 — 2026-07-12
+
+### Added
+
+- **`snapshot = true` now works on datasets that have child datasets.**
+  Stock TrueNAS refuses this with *"This option is only available for datasets
+  that have no further nesting"*, which makes the snapshot option unusable for
+  the single most common case on any box running Apps — every app is its own
+  dataset, often with `config`/`pgdata` children of its own. Without it, the
+  backup reads **live** files: databases are captured mid-write, and a busy app
+  rewriting its files can stall a backup indefinitely as restic chases a moving
+  target.
+
+  The stock guard is **correct, and it is not an arbitrary limit.**
+  `plugins/cloud/snapshot.py` already takes a *recursive* ZFS snapshot, but it
+  then points the backup tool at the **parent** dataset's
+  `.zfs/snapshot/<snap>/` directory — and ZFS does not expose child datasets
+  through a parent's snapshot directory:
+
+  ```
+  /mnt/Tap/.zfs/snapshot/<snap>/apps/               -> 0 entries (children invisible)
+  /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
+  ```
+
+  So without the guard the backup tool would walk a near-empty tree, report
+  SUCCESS, and upload almost nothing. iX gate the config rather than ship a
+  backup that lies about succeeding.
+
+  This release implements the missing half. After the (already recursive)
+  snapshot is taken, every descendant dataset's own `.zfs/snapshot/<snap>` is
+  bind-mounted into a **staging tree** mirroring the original layout, and the
+  backup tool is pointed at the staging root — a complete, consistent,
+  point-in-time view of the whole subtree. Only then is the guard relaxed.
+
+  Safety properties, in order of importance:
+
+  - **Staging failure is loud.** If any descendant cannot be staged, the backup
+    fails. A silently-incomplete backup is the exact outcome the stock guard
+    exists to prevent, and it would be worse than not having the feature.
+  - **A post-mount verification pass** asserts every planned target is really a
+    mountpoint and the staging root is non-empty, so this can never regress into
+    the empty-backup failure it is meant to fix.
+  - **The guard is relaxed last.** `apply.sh` installs the traversal, patches
+    `snapshot.py`, then `sync.py`, and only then `crud.py`. A partial failure
+    leaves the guard intact and the option merely unavailable — never
+    "guard removed, traversal missing".
+  - **Every injected block no-ops** if `_truecloud_nested` is absent.
+  - Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
+    unmounted/locked, encrypted-and-locked) are skipped and **reported** —
+    never dropped silently.
+  - Scoped to `cloud_backup` only. Cloud Sync (rclone) shares the same
+    validation mixin but has no staging teardown wired in, so its guard is left
+    in place deliberately.
+
+  Side benefit: the staging root is a **stable** path per task, so restic can
+  find its parent snapshot between runs. Stock's
+  `.zfs/snapshot/<name>-<timestamp>/` path changes every run, which defeats
+  restic's parent detection and forces a full re-scan each time.
+
+- **CI** (GitHub Actions): shellcheck + `bash -n` on every script, ruff, and
+  pytest on Python 3.11/3.12/3.13. Includes tests that `compile()` the
+  `*_BLOCK` strings — they are Python source appended to live middlewared
+  modules, so a syntax error there would break the box at boot, and nothing
+  previously checked them.
+
+### Changed
+
+- Version strings in `install.sh`, `uninstall.sh`, and `recover.sh` were stale
+  at `0.0.4`; all scripts now report the same version.
+- `patch_ui.py`: replaced a `try`/`except`/`pass` with `contextlib.suppress`
+  (no behaviour change; satisfies the new lint gate).
+
+### Removed
+
+- `patch/__pycache__/create_task.cpython-314.pyc` was committed to the
+  repository; it is now untracked and `__pycache__/` is gitignored.
+
+### Known issues
+
+- Stock `restic_backup()` deletes the ZFS snapshot in its own `finally`, which
+  fails with `EBUSY` while the staging bind mounts pin it. It logs one benign
+  `Error deleting snapshot ...` warning per run; the patch then unmounts and
+  deletes the snapshot for real. The warning is expected and harmless.
+
 ## v0.2.1 — 2026-07-09
 
 ### Fixed

README.md

@@ -1,7 +1,10 @@
 # truenas-truecloud-patch
 
-Extends TrueNAS SCALE's **TrueCloud Backup** feature to work with S3-compatible
+Extends TrueNAS SCALE's **TrueCloud Backup** feature to:
-providers and native Backblaze B2, instead of Storj only.
+
+- work with S3-compatible providers and native Backblaze B2, instead of Storj only;
+- take **consistent snapshots of datasets that have child datasets** — which is
+  every box running Apps (see [Nested-dataset snapshots](#nested-dataset-snapshots)).
 
 ---
 
@@ -77,12 +80,90 @@ not affected.
 | Layer | What changes | Technique |
 |---|---|---|
 | **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | File patch applied inside the overlayfs upper layer |
+| **Nested snapshots** | `_truecloud_nested.py` is installed into `plugins/cloud/`, and `plugins/cloud/{snapshot,crud}.py` + `plugins/cloud_backup/sync.py` are patched so `snapshot = true` works on a dataset that has child datasets. See [below](#nested-dataset-snapshots). | New module + file patches inside the overlayfs upper layer |
 | **UI** | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS chunk; original is backed up before patching |
 
-Both changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
+All changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
 restructured the relevant code), middlewared starts normally with Storj-only
 support and the reason is logged to `apply.log` in your repo root.
 
+## Nested-dataset snapshots
+
+TrueCloud Backup's **Take Snapshot** option makes restic read from a frozen ZFS
+snapshot instead of live files. Without it the backup reads data *while apps are
+writing to it* — databases get captured mid-write, and an app that rewrites its
+files continuously can stall a backup indefinitely as restic chases a moving
+target.
+
+Stock TrueNAS refuses to enable it on most real-world paths:
+
+```
+[EINVAL] cloud_backup_update.snapshot:
+  This option is only available for datasets that have no further nesting
+```
+
+That rules out **any pool running Apps** — every app is its own dataset, usually
+with `config`/`pgdata` children of its own. On a typical box that is 100+ nested
+datasets, so the feature is effectively unusable exactly where it matters most.
+
+### Why stock refuses — and why you must not simply delete the check
+
+The guard is **correct**. `plugins/cloud/snapshot.py` already takes a *recursive*
+ZFS snapshot — but it then points restic at the **parent** dataset's
+`.zfs/snapshot/<snap>/` directory, and ZFS does not expose child datasets
+through a parent's snapshot directory:
+
+```
+/mnt/Tap/.zfs/snapshot/<snap>/apps/                 ->  0 entries  (children invisible)
+/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/   ->  the real data
+```
+
+So if you just remove the validation, restic walks a **near-empty tree, reports
+SUCCESS, and uploads almost nothing.** You get a green backup job protecting no
+data. iX gate the config rather than ship a backup that lies about succeeding.
+
+> **⚠ If you are tempted to patch this yourself: deleting those four lines in
+> `plugins/cloud/crud.py` is the obvious move and it is catastrophic.** The
+> guard is load-bearing. It must be *replaced* with a working traversal, never
+> merely removed.
+
+### What this patch does instead
+
+After the (already recursive) snapshot is taken, every descendant dataset's own
+`.zfs/snapshot/<snap>` is bind-mounted into a **staging tree** that mirrors the
+original layout, and restic is pointed at the staging root — a complete,
+consistent, point-in-time view of the whole subtree. Only then is the guard
+relaxed.
+
+Safety properties, in order of importance:
+
+- **Staging failure is loud.** If any descendant cannot be staged, the backup
+  *fails*. A silently-incomplete backup is precisely what the stock guard exists
+  to prevent, and it would be worse than not having the feature at all.
+- **Post-mount verification** asserts every planned target really is a mountpoint
+  and the staging root is non-empty — so this cannot regress into the empty
+  backup it exists to fix.
+- **The guard is relaxed last.** `apply.sh` installs the traversal, patches
+  `snapshot.py`, then `sync.py`, and only then `crud.py`. Any partial failure
+  leaves the guard intact and the option merely unavailable — never
+  "guard removed, traversal missing".
+- Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
+  unmounted, locked/encrypted) are skipped and **reported to the log** — never
+  dropped silently.
+- Scoped to **cloud_backup only**. Cloud Sync (rclone) shares the same
+  validation mixin but has no staging teardown wired in, so its guard is
+  deliberately left in place.
+
+Side benefit: the staging root is a **stable path per task**, so restic can find
+its parent snapshot between runs. Stock's `.zfs/snapshot/<name>-<timestamp>/`
+path changes every run, defeating restic's parent detection and forcing a full
+re-scan each time.
+
+**Expected log noise:** stock `restic_backup()` deletes the ZFS snapshot in its
+own `finally`, which fails with `EBUSY` while the staging mounts pin it. You will
+see one benign `Error deleting snapshot ...` warning per run; the patch then
+unmounts and deletes the snapshot for real.
+
 ## Supported providers after patching
 
 | Provider | Credential type in TrueNAS |

install.sh

@@ -18,7 +18,7 @@
 
 set -euo pipefail
 
-VERSION="0.0.4"
+VERSION="0.3.0"
 
 # The directory containing install.sh is the permanent install location.
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"

patch/apply.sh

@@ -7,13 +7,21 @@
 # is already up and has already imported the stock modules — the on-disk
 # patch alone cannot reach the running process.
 #
-# TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
+# TrueNAS updates replace /usr/ entirely; this script re-applies three patches:
 #
 #   1. Backend — b2.py and restic.py are patched directly in the overlay.
 #      On a boot run, a single detached middlewared restart is scheduled
 #      (Step 3) so the patched modules actually get loaded.
 #
-#   2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
+#   2. Nested-dataset snapshots — installs _truecloud_nested.py and patches
+#      plugins/cloud/{snapshot,crud}.py + plugins/cloud_backup/sync.py so the
+#      "Take Snapshot" option works on a dataset that has child datasets.
+#      Stock middleware refuses that config, because it points the backup tool
+#      at the PARENT's .zfs/snapshot/ where children are invisible — it would
+#      silently back up a near-empty tree. We stage a complete tree of
+#      per-dataset bind mounts and only then relax the guard.
+#
+#   3. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
 #                          from Storj-only to include S3 and B2. Served from
 #                          disk per request, so no restart is needed for it.
 #
@@ -24,7 +32,7 @@
 # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
 PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 LOG="$PATCH_DIR/apply.log"
-VERSION="0.2.1"
+VERSION="0.3.0"
 
 # Rotate log at 512 KB to avoid unbounded growth on a system volume.
 # Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -175,12 +183,18 @@ elif [ -z "$_MW_DIR" ]; then
 else
     _B2_PY="$_MW_DIR/rclone/remote/b2.py"
     _RESTIC_PY="$_MW_DIR/plugins/cloud_backup/restic.py"
+    _CLOUD_DIR="$_MW_DIR/plugins/cloud"
+    _SYNC_PY="$_MW_DIR/plugins/cloud_backup/sync.py"
+    _NESTED_SRC="$PATCH_DIR/patch/truecloud_nested.py"
 
-        # ── patch b2.py + restic.py + hook_status.json (single subprocess) ──────
+        # ── patch b2.py + restic.py + nested-snapshot + hook_status.json ────────
-        if "$PYTHON" - "$_B2_PY" "$_RESTIC_PY" "$PATCH_DIR/hook_status.json" << 'PYEOF'
+        # (single subprocess: PREINIT has a tight timeout budget)
-import json, os, sys, time
+        if "$PYTHON" - "$_B2_PY" "$_RESTIC_PY" "$PATCH_DIR/hook_status.json" \
+                      "$_CLOUD_DIR" "$_SYNC_PY" "$_NESTED_SRC" << 'PYEOF'
+import json, os, shutil, sys, time
 
 b2_path, restic_path, status_path = sys.argv[1], sys.argv[2], sys.argv[3]
+cloud_dir, sync_path, nested_src = sys.argv[4], sys.argv[5], sys.argv[6]
 
 B2_BLOCK = """
 # TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
@@ -240,6 +254,116 @@ else:
     get_restic_config._truecloud_patched = True
 """
 
+# ── nested-dataset snapshot support ───────────────────────────────────────────
+# Stock middleware refuses `snapshot=true` on a path containing child datasets,
+# because it points the backup tool at the PARENT dataset's .zfs/snapshot/,
+# where child datasets are INVISIBLE -- it would silently back up a near-empty
+# tree. That guard is correct. We implement the missing traversal (a staging
+# tree of per-dataset bind mounts) and only then relax the guard.
+#
+# Fail-safe direction: if any of these three blocks fails to apply, the stock
+# guard remains and the option simply stays unavailable. We never end up with
+# the guard removed but the traversal missing -- that would be a silently empty
+# backup, the worst possible outcome.
+
+SNAPSHOT_BLOCK = """
+# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
+try:
+    from middlewared.plugins.cloud import _truecloud_nested as _tc_nested
+except ImportError:
+    _tc_nested = None
+
+if _tc_nested is not None:
+    _tc_orig_create_snapshot = create_snapshot
+
+    async def create_snapshot(middleware, path, name="cloud_task-onetime"):
+        # Determine nesting BEFORE delegating. We must never silently fall back
+        # to stock behaviour on a nested path: stock returns the parent's
+        # .zfs/snapshot/ path, where children are invisible, which is exactly
+        # the near-empty backup this feature exists to prevent. If we cannot
+        # tell, we raise -- loud failure beats a backup that lies.
+        datasets = await middleware.call("zfs.dataset.query", [["type", "=", "FILESYSTEM"]])
+        dataset, nested = get_dataset_recursive(datasets, path)
+
+        # Stock takes the (already recursive) snapshot; we only replace the PATH.
+        snapshot, snap_path = await _tc_orig_create_snapshot(middleware, path, name)
+
+        if not nested:
+            return snapshot, snap_path  # no children: stock behaviour, untouched
+
+        staging_root = await _tc_nested.stage_nested(
+            middleware, path, snapshot,
+            dataset["properties"]["mountpoint"]["value"], name,
+            logger=getattr(middleware, "logger", None),
+        )
+        return snapshot, staging_root
+
+    create_snapshot._truecloud_patched = True
+"""
+
+CRUD_BLOCK = """
+# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
+try:
+    from middlewared.plugins.cloud import _truecloud_nested as _tc_nested
+except ImportError:
+    _tc_nested = None
+
+if _tc_nested is not None:
+    _tc_orig_validate = CloudTaskServiceMixin._validate
+
+    async def _tc_validate(self, app, verrors, name, data):
+        await _tc_orig_validate(self, app, verrors, name, data)
+
+        # Only cloud_backup: staging teardown is wired into cloud_backup.sync's
+        # finally. cloudsync would leak bind mounts, so leave its guard intact.
+        if getattr(getattr(self, "_config", None), "namespace", "") != "cloud_backup":
+            return
+
+        # Drop ONLY the nested-dataset guard. If iX ever rewords the message the
+        # filter stops matching, the guard survives, and the option merely stays
+        # unavailable -- the safe direction to fail.
+        verrors.errors = [
+            e for e in verrors.errors
+            if not (
+                getattr(e, "attribute", "") == f"{name}.snapshot"
+                and "no further nesting" in getattr(e, "errmsg", "")
+            )
+        ]
+
+    CloudTaskServiceMixin._validate = _tc_validate
+    CloudTaskServiceMixin._validate._truecloud_patched = True
+"""
+
+SYNC_BLOCK = """
+# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
+try:
+    from middlewared.plugins.cloud import _truecloud_nested as _tc_nested
+except ImportError:
+    _tc_nested = None
+
+if _tc_nested is not None:
+    _tc_orig_restic_backup = restic_backup
+
+    async def restic_backup(middleware, job, cloud_backup, dry_run=False, rate_limit=None):
+        # Our bind mounts pin the ZFS snapshot, so stock's `finally` cannot
+        # destroy it (EBUSY) and logs one benign warning. We unmount here and
+        # then delete the snapshot for real.
+        try:
+            return await _tc_orig_restic_backup(middleware, job, cloud_backup, dry_run, rate_limit)
+        finally:
+            try:
+                await _tc_nested.cleanup_task(
+                    middleware,
+                    f"cloud_backup-{cloud_backup.get('id', 'onetime')}",
+                    logger=getattr(middleware, "logger", None),
+                )
+            except Exception as e:
+                middleware.logger.warning("truecloud-patch: staging cleanup failed: %r", e)
+
+    restic_backup._truecloud_patched = True
+"""
+
+
 def patch_file(path, block):
     with open(path, encoding="utf-8") as fh:
         content = fh.read()
@@ -250,6 +374,7 @@ def patch_file(path, block):
         fh.write(base.rstrip("\n") + "\n" + block)
 
 b2_ok = restic_ok = False
+nested_ok = False
 
 if os.path.exists(b2_path):
     try:
@@ -271,6 +396,35 @@ if os.path.exists(restic_path):
 else:
     print(f"WARNING: restic.py not found at {restic_path}")
 
+# ── nested-dataset snapshot support ───────────────────────────────────────────
+# Order matters: install the traversal machinery FIRST, relax the validation
+# guard LAST. If anything fails partway, the guard is still in place and the
+# option stays unavailable -- we never expose "guard removed, traversal missing".
+nested_detail = ''
+try:
+    snapshot_py = os.path.join(cloud_dir, 'snapshot.py')
+    crud_py = os.path.join(cloud_dir, 'crud.py')
+    nested_dst = os.path.join(cloud_dir, '_truecloud_nested.py')
+
+    missing = [p for p in (snapshot_py, crud_py, sync_path, nested_src) if not os.path.exists(p)]
+    if missing:
+        raise FileNotFoundError('missing: ' + ', '.join(missing))
+
+    shutil.copyfile(nested_src, nested_dst)   # 1. traversal implementation
+    patch_file(snapshot_py, SNAPSHOT_BLOCK)   # 2. build the staging tree
+    patch_file(sync_path, SYNC_BLOCK)         # 3. tear it down afterwards
+    patch_file(crud_py, CRUD_BLOCK)           # 4. ONLY NOW allow nested tasks
+
+    nested_ok = True
+    nested_detail = 'nested-dataset snapshots enabled (staging tree)'
+    print(f'OK: Installed nested-snapshot support → {nested_dst}')
+    print(f'OK: Patched snapshot.py, sync.py, crud.py → {cloud_dir}')
+except Exception as e:
+    nested_detail = f'not applied: {e}'
+    print(f'WARNING: Failed to apply nested-snapshot patch: {e}')
+    print('WARNING: Stock nesting guard remains; snapshot option stays unavailable')
+    print('WARNING: for nested datasets. Existing backups are unaffected.')
+
 patches = {
     'middlewared.rclone.remote.b2': {
         'ok': b2_ok,
@@ -280,6 +434,10 @@ patches = {
         'ok': restic_ok,
         'detail': 'patched on disk in overlay at boot' if restic_ok else 'restic.py not found or write failed',
     },
+    'middlewared.plugins.cloud.nested_snapshot': {
+        'ok': nested_ok,
+        'detail': nested_detail,
+    },
 }
 payload = {'patched_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), 'patches': patches}
 tmp = status_path + '.tmp'

patch/patch_ui.py

@@ -19,6 +19,7 @@ Safe to run multiple times — a marker string detects an already-patched file.
 Exits 0 in all cases (warnings are printed to stdout and logged by apply.sh).
 """
 
+import contextlib
 import os
 import re
 import shutil
@@ -131,10 +132,8 @@ def main():
         os.replace(tmp, path)
     except OSError as exc:
         print(f"[truecloud-patch] ERROR: Could not write {path}: {exc}")
-        try:
+        with contextlib.suppress(OSError):
             os.unlink(tmp)
-        except OSError:
-            pass
         return
 
     print(f"[truecloud-patch] UI bundle patched ({count} replacement(s)): {path}")

patch/truecloud_nested.py

@@ -0,0 +1,322 @@
+"""Nested-dataset snapshot support for TrueCloud Backup / Cloud Sync.
+
+Why this exists
+---------------
+Stock TrueNAS refuses ``snapshot = true`` when the backup path contains child
+datasets::
+
+    This option is only available for datasets that have no further nesting
+
+That guard is *correct* and it is not laziness. ``plugins/cloud/snapshot.py``
+already takes a **recursive** ZFS snapshot, but it then points the backup tool
+at the *parent* dataset's ``.zfs/snapshot/<snap>/`` directory -- and ZFS does
+not expose child datasets through a parent's snapshot directory::
+
+    /mnt/Tap/.zfs/snapshot/<snap>/apps/   ->  0 entries   (children invisible)
+    /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/  ->  the real data
+
+So without the guard, the backup tool would walk a near-empty tree, report
+SUCCESS, and upload almost nothing. A backup that lies about succeeding is the
+worst failure a backup system can have, so middleware refuses the config
+instead.
+
+This module implements the missing half: after the (already recursive) snapshot
+is taken, every descendant dataset's *own* ``.zfs/snapshot/<snap>`` directory is
+bind-mounted into a staging tree that mirrors the original layout. The backup
+tool is then pointed at the staging root, which is a complete, consistent,
+point-in-time view of the whole subtree.
+
+Cardinal safety rule
+--------------------
+**If the tree cannot be staged completely, fail loudly.** Never return a partial
+tree. Silently backing up an incomplete tree is precisely the failure this
+feature exists to prevent, and it would be worse than not having the feature.
+
+Notes
+-----
+* ZFS snapshots are immutable, so a plain ``mount --bind`` is inherently
+  read-only; no remount dance is needed.
+* Bind-mounting ``.zfs/snapshot/<snap>`` pins the snapshot, so ``zfs destroy``
+  of that snapshot returns EBUSY until we unmount. Stock ``restic_backup()``
+  deletes the snapshot in its ``finally``, which therefore logs one benign
+  "Error deleting snapshot ... busy" warning; :func:`cleanup_task` then unmounts
+  and deletes the snapshot for real. See ``patch/apply.sh``.
+* Staging roots live under a stable, per-task path so that the backup tool sees
+  the *same* path every run. Stock's ``.zfs/snapshot/<name>-<timestamp>/`` path
+  changes every run, which defeats restic's parent-snapshot detection; the
+  staging tree is an improvement on that.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import os
+import subprocess
+
+__all__ = [
+    "StagingError",
+    "STAGING_BASE",
+    "ACTIVE",
+    "staging_root_for",
+    "plan_staging",
+    "current_mounts_under",
+    "apply_plan",
+    "verify_staged",
+    "teardown",
+]
+
+#: Where staging trees are assembled. tmpfs; bind mounts consume no space.
+STAGING_BASE = "/run/truecloud-nested"
+
+#: staging_root -> zfs snapshot name ("pool/ds@snap"), for cleanup.
+ACTIVE: dict[str, str] = {}
+
+
+class StagingError(Exception):
+    """Staging could not produce a complete tree. The backup must not proceed."""
+
+
+def staging_root_for(name: str, base: str = STAGING_BASE) -> str:
+    """Stable staging root for a task name (e.g. ``cloud_backup-5``)."""
+    safe = "".join(c if (c.isalnum() or c in "-_.") else "_" for c in name) or "task"
+    return os.path.join(base, safe)
+
+
+def _depth(path: str) -> int:
+    return len([p for p in path.split("/") if p])
+
+
+def plan_staging(base_mountpoint, path, snapshot_name, datasets, staging_root,
+                 isdir=os.path.isdir):
+    """Compute the bind-mount plan for staging a nested tree. Pure function.
+
+    ``datasets`` is a list of dicts shaped like ``zfs.dataset.query`` results:
+    ``{"name": str, "properties": {"mountpoint": {"value": str},
+    "mounted": {"value": "yes"|"no"}}}``.
+
+    Returns ``(mounts, skipped)`` where ``mounts`` is an ordered list of
+    ``(source, target)`` pairs (parents before children) and ``skipped`` is a
+    list of ``(dataset_name, reason)``.
+
+    Raises StagingError if a descendant holds data we would silently omit.
+    """
+    def snapdir(mountpoint):
+        return os.path.join(mountpoint, ".zfs", "snapshot", snapshot_name)
+
+    # Root of the staging tree: the backup path as seen inside the base
+    # dataset's own snapshot.
+    rel = os.path.relpath(path, base_mountpoint)
+    root_src = snapdir(base_mountpoint)
+    if rel != ".":
+        root_src = os.path.join(root_src, rel)
+
+    mounts = [(root_src, staging_root)]
+    skipped = []
+
+    prefix = path.rstrip("/") + "/"
+    for ds in datasets:
+        props = ds.get("properties", {})
+        mp = props.get("mountpoint", {}).get("value", "")
+        name = ds.get("name", "?")
+
+        if not mp or mp in ("none", "legacy", "-"):
+            skipped.append((name, f"mountpoint is {mp or 'unset'}"))
+            continue
+        if not mp.startswith(prefix):
+            continue  # not a descendant of the backup path
+
+        mounted = props.get("mounted", {}).get("value", "yes")
+        if mounted == "no":
+            # An unmounted (e.g. locked/encrypted) dataset contributes nothing to
+            # the live tree either, so skipping matches stock semantics -- but it
+            # is a real gap and must be visible, never silent.
+            skipped.append((name, "dataset is not mounted (locked/encrypted?)"))
+            continue
+
+        src = snapdir(mp)
+        if not isdir(src):
+            # The recursive snapshot should have covered every descendant. If it
+            # did not, this dataset's data would be silently omitted. Refuse.
+            raise StagingError(
+                f"dataset {name!r} has no snapshot {snapshot_name!r} at {src!r}; "
+                f"refusing to back up an incomplete tree"
+            )
+
+        target = os.path.join(staging_root, os.path.relpath(mp, path))
+        mounts.append((src, target))
+
+    # Parents before children, so each mountpoint exists before we mount onto it.
+    mounts.sort(key=lambda m: _depth(m[1]))
+    return mounts, skipped
+
+
+def current_mounts_under(root, mounts_file="/proc/self/mounts"):
+    """Mountpoints at or under ``root``, deepest first. Used for teardown."""
+    found = []
+    try:
+        with open(mounts_file, encoding="utf-8") as fh:
+            for line in fh:
+                parts = line.split()
+                if len(parts) < 2:
+                    continue
+                mp = parts[1].replace("\\040", " ").replace("\\011", "\t")
+                if mp == root or mp.startswith(root.rstrip("/") + "/"):
+                    found.append(mp)
+    except OSError:
+        return []
+    found.sort(key=_depth, reverse=True)
+    return found
+
+
+def _run(cmd):
+    return subprocess.run(cmd, capture_output=True, text=True, check=False)
+
+
+def apply_plan(mounts, runner=_run):
+    """Execute the bind-mount plan. Blocking; call via ``run_in_thread``.
+
+    Raises StagingError on the first failure, after rolling back what was
+    mounted -- a half-built tree must never be handed to the backup tool.
+    """
+    if not mounts:
+        raise StagingError("empty staging plan")
+
+    staging_root = mounts[0][1]
+    done = []
+    try:
+        os.makedirs(staging_root, exist_ok=True)
+        for src, target in mounts:
+            if not os.path.isdir(target):
+                # Child mountpoint dirs come from the parent snapshot, which is
+                # read-only -- we cannot mkdir them. Only the root is ours.
+                raise StagingError(f"staging target {target!r} does not exist")
+            res = runner(["mount", "--bind", src, target])
+            if res.returncode != 0:
+                raise StagingError(
+                    f"bind-mount {src!r} -> {target!r} failed: "
+                    f"{(res.stderr or '').strip() or res.returncode}"
+                )
+            done.append(target)
+    except Exception:
+        for target in reversed(done):
+            runner(["umount", "-l", target])
+        with contextlib.suppress(OSError):
+            os.rmdir(staging_root)
+        raise
+    return staging_root
+
+
+def verify_staged(mounts, runner=_run, ismount=os.path.ismount, listdir=os.listdir):
+    """Assert the staged tree is real and complete. Raises StagingError if not.
+
+    This is the anti-regression guard: it is what stops this feature from ever
+    degrading back into the silently-empty backup that the stock validation
+    refuses to allow.
+    """
+    if not mounts:
+        raise StagingError("nothing was staged")
+
+    staging_root = mounts[0][1]
+    for _src, target in mounts:
+        if not ismount(target):
+            raise StagingError(f"staging target {target!r} is not a mountpoint")
+
+    try:
+        if not listdir(staging_root):
+            raise StagingError(f"staging root {staging_root!r} is empty")
+    except OSError as e:
+        raise StagingError(f"staging root {staging_root!r} unreadable: {e}") from e
+
+    return True
+
+
+def teardown(staging_root, runner=_run, mounts_file="/proc/self/mounts"):
+    """Unmount the staging tree (deepest first) and remove the root.
+
+    Idempotent, and does not depend on an in-memory plan -- so it also cleans up
+    leftovers from a crashed run.
+    """
+    errors = []
+    for mp in current_mounts_under(staging_root, mounts_file=mounts_file):
+        res = runner(["umount", mp])
+        if res.returncode != 0:
+            res = runner(["umount", "-l", mp])  # lazy: better than leaking
+            if res.returncode != 0:
+                errors.append(f"{mp}: {(res.stderr or '').strip()}")
+    with contextlib.suppress(OSError):
+        os.rmdir(staging_root)
+    return errors
+
+
+# ── async orchestration (middleware is duck-typed; no middlewared import) ─────
+
+
+async def stage_nested(middleware, path, snapshot, base_mountpoint, task_name, logger=None):
+    """Build a complete staging tree for `path` from the already-taken `snapshot`.
+
+    `snapshot` is a full ZFS snapshot name ("Tap@cloud_backup-5-2026...").
+    Returns the staging root to hand to the backup tool.
+
+    Raises StagingError if the tree cannot be staged completely -- the caller
+    must let that propagate so the backup fails instead of silently uploading a
+    partial tree.
+    """
+    snapshot_name = snapshot.split("@", 1)[1]
+    staging_root = staging_root_for(task_name)
+
+    # A previous run may have crashed mid-flight; never build on top of that.
+    await middleware.run_in_thread(teardown, staging_root)
+
+    datasets = await middleware.call("zfs.dataset.query", [["type", "=", "FILESYSTEM"]])
+
+    mounts, skipped = await middleware.run_in_thread(
+        plan_staging, base_mountpoint, path, snapshot_name, datasets, staging_root
+    )
+    if logger:
+        for name, reason in skipped:
+            logger.warning("truecloud-patch: not staging dataset %r: %s", name, reason)
+
+    await middleware.run_in_thread(apply_plan, mounts)
+    try:
+        await middleware.run_in_thread(verify_staged, mounts)
+    except Exception:
+        await middleware.run_in_thread(teardown, staging_root)
+        raise
+
+    ACTIVE[staging_root] = snapshot
+    if logger:
+        logger.info(
+            "truecloud-patch: staged %d dataset(s) from %s at %s",
+            len(mounts), snapshot, staging_root,
+        )
+    return staging_root
+
+
+async def cleanup_task(middleware, task_name, logger=None):
+    """Tear down a task's staging tree and delete the snapshot it pinned.
+
+    Safe to call unconditionally: a no-op when the task was never staged.
+
+    Stock `restic_backup()` deletes the snapshot in its own `finally`, which
+    fails with EBUSY while our bind mounts pin it (it logs a warning and moves
+    on). We unmount here and then delete the snapshot for real.
+    """
+    staging_root = staging_root_for(task_name)
+    snapshot = ACTIVE.pop(staging_root, None)
+
+    if snapshot is None and not os.path.isdir(staging_root):
+        return  # never staged; nothing to do
+
+    errors = await middleware.run_in_thread(teardown, staging_root)
+    if errors and logger:
+        for err in errors:
+            logger.warning("truecloud-patch: staging teardown: %s", err)
+
+    if snapshot is not None:
+        try:
+            await middleware.call("zfs.snapshot.delete", snapshot)
+        except Exception as e:  # noqa: BLE001 - cleanup must never mask the real error
+            if logger:
+                logger.warning(
+                    "truecloud-patch: could not delete snapshot %s: %r", snapshot, e
+                )

pyproject.toml

@@ -0,0 +1,15 @@
+# Tooling config only — this project is not a Python package. The patch modules
+# are copied into middlewared's site-packages by patch/apply.sh at boot.
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "B", "UP", "SIM"]
+ignore = [
+  "E501",  # long lines in explanatory comments are fine
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

recover.sh

@@ -17,7 +17,7 @@
 #   bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
 #   systemctl restart middlewared
 
-VERSION="0.0.4"
+VERSION="0.3.0"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 

tests/test_apply_blocks.py

@@ -0,0 +1,97 @@
+"""The *_BLOCK strings in apply.sh are Python source injected into middleware.
+
+A syntax error in one of them would be appended to a live middlewared module and
+break the box at boot. They are string literals, so nothing type-checks them --
+these tests do.
+"""
+
+import ast
+import os
+import re
+
+import pytest
+
+APPLY_SH = os.path.join(os.path.dirname(__file__), "..", "patch", "apply.sh")
+
+EXPECTED_BLOCKS = {
+    "B2_BLOCK",
+    "RESTIC_BLOCK",
+    "SNAPSHOT_BLOCK",
+    "CRUD_BLOCK",
+    "SYNC_BLOCK",
+}
+
+
+def heredoc_source():
+    with open(APPLY_SH, encoding="utf-8") as fh:
+        src = fh.read()
+    m = re.search(r"<< 'PYEOF'\n(.*?)\nPYEOF", src, re.S)
+    assert m, "could not find the PYEOF heredoc in apply.sh"
+    return m.group(1)
+
+
+def extract_blocks():
+    tree = ast.parse(heredoc_source())
+    blocks = {}
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Assign):
+            for tgt in node.targets:
+                if (
+                    isinstance(tgt, ast.Name)
+                    and tgt.id.endswith("_BLOCK")
+                    and isinstance(node.value, ast.Constant)
+                    and isinstance(node.value.value, str)
+                ):
+                    blocks[tgt.id] = node.value.value
+    return blocks
+
+
+def test_heredoc_itself_compiles():
+    compile(heredoc_source(), "apply.sh:PYEOF", "exec")
+
+
+def test_all_expected_blocks_present():
+    assert set(extract_blocks()) == EXPECTED_BLOCKS
+
+
+@pytest.mark.parametrize("name", sorted(EXPECTED_BLOCKS))
+def test_injected_block_is_valid_python(name):
+    block = extract_blocks()[name]
+    compile(block, f"apply.sh:{name}", "exec")
+
+
+@pytest.mark.parametrize("name", sorted(EXPECTED_BLOCKS))
+def test_injected_block_carries_the_idempotency_marker(name):
+    # patch_file() truncates each target file at "\n# TRUECLOUD_PATCH" before
+    # re-appending, so every block must start with that marker or repeated runs
+    # would stack duplicate copies into the middleware module.
+    assert extract_blocks()[name].lstrip("\n").startswith("# TRUECLOUD_PATCH")
+
+
+@pytest.mark.parametrize("name", ["SNAPSHOT_BLOCK", "CRUD_BLOCK", "SYNC_BLOCK"])
+def test_nested_blocks_degrade_safely_without_the_module(name):
+    # If _truecloud_nested failed to install, every nested block must no-op.
+    # Critically this includes CRUD_BLOCK: relaxing the guard without the
+    # traversal in place would mean silently-empty backups.
+    block = extract_blocks()[name]
+    assert "_tc_nested = None" in block
+    assert "if _tc_nested is not None:" in block
+
+
+def test_crud_block_is_scoped_to_cloud_backup():
+    # cloudsync has no staging teardown wired in, so its guard must stay.
+    assert '!= "cloud_backup"' in extract_blocks()["CRUD_BLOCK"]
+
+
+def test_guard_is_relaxed_only_after_traversal_is_installed():
+    # Ordering in apply.sh is a safety property: copy module -> patch snapshot.py
+    # -> patch sync.py -> patch crud.py. crud.py (which unlocks the feature) must
+    # come last, so a partial failure never leaves "guard removed, traversal gone".
+    src = heredoc_source()
+    order = [
+        src.index("shutil.copyfile(nested_src, nested_dst)"),
+        src.index("patch_file(snapshot_py, SNAPSHOT_BLOCK)"),
+        src.index("patch_file(sync_path, SYNC_BLOCK)"),
+        src.index("patch_file(crud_py, CRUD_BLOCK)"),
+    ]
+    assert order == sorted(order), "crud.py must be patched last"

tests/test_truecloud_nested.py

@@ -0,0 +1,274 @@
+"""Tests for nested-dataset snapshot staging.
+
+The cardinal rule under test: a tree that cannot be staged completely must fail
+LOUDLY. A silently-incomplete backup is the exact failure that stock TrueNAS's
+"no further nesting" guard exists to prevent, and it is the one regression this
+feature must never introduce.
+"""
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "patch"))
+
+from truecloud_nested import (  # noqa: E402
+    StagingError,
+    apply_plan,
+    current_mounts_under,
+    plan_staging,
+    staging_root_for,
+    teardown,
+    verify_staged,
+)
+
+SNAP = "cloud_backup-5-20260712030000"
+ROOT = "/run/truecloud-nested/cloud_backup-5"
+
+
+def ds(name, mountpoint, mounted="yes"):
+    return {
+        "name": name,
+        "properties": {
+            "mountpoint": {"value": mountpoint},
+            "mounted": {"value": mounted},
+        },
+    }
+
+
+# Mirrors the real layout: apps are datasets, several with their own children.
+DATASETS = [
+    ds("Tap", "/mnt/Tap"),
+    ds("Tap/apps", "/mnt/Tap/apps"),
+    ds("Tap/apps/lidarr", "/mnt/Tap/apps/lidarr"),
+    ds("Tap/apps/lidarr/config", "/mnt/Tap/apps/lidarr/config"),
+    ds("Tap/apps/immich", "/mnt/Tap/apps/immich"),
+    ds("Tap/apps/immich/pgdata", "/mnt/Tap/apps/immich/pgdata"),
+]
+
+ALL_DIRS_EXIST = lambda _p: True  # noqa: E731
+
+
+class TestPlanStaging:
+    def test_stages_every_descendant_dataset(self):
+        mounts, skipped = plan_staging(
+            "/mnt/Tap", "/mnt/Tap", SNAP, DATASETS, ROOT, isdir=ALL_DIRS_EXIST
+        )
+        assert skipped == []
+
+        # Root + all 5 descendants. The base dataset itself is the root, not a
+        # descendant, so it must not be double-mounted.
+        assert len(mounts) == 6
+        assert mounts[0] == (f"/mnt/Tap/.zfs/snapshot/{SNAP}", ROOT)
+
+        by_target = dict((t, s) for s, t in mounts)
+        assert by_target[f"{ROOT}/apps"] == f"/mnt/Tap/apps/.zfs/snapshot/{SNAP}"
+        assert by_target[f"{ROOT}/apps/immich/pgdata"] == (
+            f"/mnt/Tap/apps/immich/pgdata/.zfs/snapshot/{SNAP}"
+        )
+
+    def test_parents_are_mounted_before_children(self):
+        # A child's mountpoint dir only exists inside its parent's snapshot, so
+        # mounting a child first would fail.
+        mounts, _ = plan_staging(
+            "/mnt/Tap", "/mnt/Tap", SNAP, DATASETS, ROOT, isdir=ALL_DIRS_EXIST
+        )
+        seen = set()
+        for _src, target in mounts:
+            parent = os.path.dirname(target)
+            if target != ROOT:
+                assert parent in seen or parent == ROOT, f"{target} mounted before {parent}"
+            seen.add(target)
+
+    def test_backup_path_below_dataset_root(self):
+        mounts, _ = plan_staging(
+            "/mnt/Tap", "/mnt/Tap/apps", SNAP, DATASETS, ROOT, isdir=ALL_DIRS_EXIST
+        )
+        # Root source is the *subdirectory* inside the base dataset's snapshot.
+        assert mounts[0] == (f"/mnt/Tap/.zfs/snapshot/{SNAP}/apps", ROOT)
+        targets = [t for _s, t in mounts]
+        assert f"{ROOT}/lidarr" in targets       # relative to /mnt/Tap/apps
+        assert f"{ROOT}/apps/lidarr" not in targets
+
+    def test_base_dataset_is_not_a_descendant_of_itself(self):
+        mounts, _ = plan_staging(
+            "/mnt/Tap", "/mnt/Tap", SNAP, [ds("Tap", "/mnt/Tap")], ROOT, isdir=ALL_DIRS_EXIST
+        )
+        assert len(mounts) == 1  # just the root
+
+
+class TestSkipping:
+    @pytest.mark.parametrize("mp", ["none", "legacy", "-", ""])
+    def test_unmountable_mountpoints_are_skipped_and_reported(self, mp):
+        datasets = DATASETS + [ds("Tap/weird", mp)]
+        mounts, skipped = plan_staging(
+            "/mnt/Tap", "/mnt/Tap", SNAP, datasets, ROOT, isdir=ALL_DIRS_EXIST
+        )
+        assert len(mounts) == 6
+        assert any(name == "Tap/weird" for name, _reason in skipped)
+
+    def test_unmounted_dataset_is_skipped_but_never_silently(self):
+        # A locked/encrypted dataset contributes nothing to the live tree either,
+        # so skipping matches stock semantics -- but it MUST be reported.
+        datasets = DATASETS + [ds("Tap/apps/vault", "/mnt/Tap/apps/vault", mounted="no")]
+        mounts, skipped = plan_staging(
+            "/mnt/Tap", "/mnt/Tap", SNAP, datasets, ROOT, isdir=ALL_DIRS_EXIST
+        )
+        assert f"{ROOT}/apps/vault" not in [t for _s, t in mounts]
+        assert ("Tap/apps/vault", "dataset is not mounted (locked/encrypted?)") in skipped
+
+
+class TestSilentOmissionGuard:
+    """The whole point of the feature. These are the tests that matter."""
+
+    def test_missing_snapshot_on_descendant_raises(self):
+        # If the recursive snapshot somehow missed a dataset, staging it would
+        # silently omit its data. Refuse rather than upload an incomplete tree.
+        def isdir(path):
+            return "/mnt/Tap/apps/immich/pgdata/" not in path
+
+        with pytest.raises(StagingError, match="incomplete tree"):
+            plan_staging("/mnt/Tap", "/mnt/Tap", SNAP, DATASETS, ROOT, isdir=isdir)
+
+    def test_error_names_the_offending_dataset(self):
+        def isdir(path):
+            return "/mnt/Tap/apps/immich/pgdata/" not in path
+
+        with pytest.raises(StagingError, match="Tap/apps/immich/pgdata"):
+            plan_staging("/mnt/Tap", "/mnt/Tap", SNAP, DATASETS, ROOT, isdir=isdir)
+
+
+class TestVerifyStaged:
+    """Anti-regression guard: proves the staged tree is real before we back it up."""
+
+    def test_passes_when_every_target_is_mounted_and_root_non_empty(self):
+        mounts = [("/src", ROOT), ("/src/a", f"{ROOT}/a")]
+        assert verify_staged(
+            mounts, ismount=lambda p: True, listdir=lambda p: ["apps"]
+        )
+
+    def test_raises_when_a_target_is_not_actually_mounted(self):
+        # This is the case that would produce a silently-empty backup.
+        mounts = [("/src", ROOT), ("/src/a", f"{ROOT}/a")]
+        with pytest.raises(StagingError, match="not a mountpoint"):
+            verify_staged(
+                mounts, ismount=lambda p: p == ROOT, listdir=lambda p: ["apps"]
+            )
+
+    def test_raises_when_staging_root_is_empty(self):
+        mounts = [("/src", ROOT)]
+        with pytest.raises(StagingError, match="empty"):
+            verify_staged(mounts, ismount=lambda p: True, listdir=lambda p: [])
+
+    def test_raises_on_empty_plan(self):
+        with pytest.raises(StagingError):
+            verify_staged([])
+
+
+class FakeRunner:
+    def __init__(self, fail_on=None):
+        self.fail_on = fail_on
+        self.calls = []
+
+    def __call__(self, cmd):
+        self.calls.append(cmd)
+
+        class R:
+            returncode = 0
+            stderr = ""
+
+        if self.fail_on and cmd[0] == "mount" and cmd[2] == self.fail_on:
+            R.returncode = 32
+            R.stderr = "mount failed"
+        return R
+
+
+class TestApplyPlanRollback:
+    def test_rolls_back_mounts_when_one_fails(self, tmp_path, monkeypatch):
+        # A half-built tree must never reach the backup tool.
+        root = str(tmp_path / "root")
+        mounts = [("/src", root), ("/src/a", root + "/a"), ("/src/b", root + "/b")]
+        monkeypatch.setattr(os.path, "isdir", lambda p: True)
+
+        runner = FakeRunner(fail_on="/src/b")
+        with pytest.raises(StagingError, match="bind-mount"):
+            apply_plan(mounts, runner=runner)
+
+        umounts = [c for c in runner.calls if c[0] == "umount"]
+        # Everything successfully mounted before the failure is unmounted again.
+        assert [c[-1] for c in umounts] == [root + "/a", root]
+
+    def test_raises_when_target_missing(self, tmp_path, monkeypatch):
+        root = str(tmp_path / "root")
+        monkeypatch.setattr(os.path, "isdir", lambda p: p == root)
+        with pytest.raises(StagingError, match="does not exist"):
+            apply_plan([("/src", root), ("/src/a", root + "/a")], runner=FakeRunner())
+
+
+class TestTeardown:
+    def test_unmounts_deepest_first(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text(
+            f"tmpfs {ROOT} tmpfs rw 0 0\n"
+            f"tmpfs {ROOT}/apps tmpfs rw 0 0\n"
+            f"tmpfs {ROOT}/apps/lidarr/config tmpfs rw 0 0\n"
+            f"tmpfs {ROOT}/apps/lidarr tmpfs rw 0 0\n"
+            "tmpfs /somewhere/else tmpfs rw 0 0\n"
+        )
+        runner = FakeRunner()
+        teardown(ROOT, runner=runner, mounts_file=str(mounts_file))
+
+        order = [c[-1] for c in runner.calls if c[0] == "umount"]
+        assert order == [
+            f"{ROOT}/apps/lidarr/config",
+            f"{ROOT}/apps/lidarr",
+            f"{ROOT}/apps",
+            ROOT,
+        ]
+        assert "/somewhere/else" not in order  # never touch unrelated mounts
+
+    def test_is_idempotent_when_nothing_mounted(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text("tmpfs /somewhere/else tmpfs rw 0 0\n")
+        runner = FakeRunner()
+        assert teardown(ROOT, runner=runner, mounts_file=str(mounts_file)) == []
+        assert runner.calls == []
+
+    def test_falls_back_to_lazy_umount(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text(f"tmpfs {ROOT} tmpfs rw 0 0\n")
+
+        class Busy(FakeRunner):
+            def __call__(self, cmd):
+                self.calls.append(cmd)
+
+                class R:
+                    returncode = 0 if "-l" in cmd else 32
+                    stderr = "target is busy"
+
+                return R
+
+        runner = Busy()
+        assert teardown(ROOT, runner=runner, mounts_file=str(mounts_file)) == []
+        assert ["umount", "-l", ROOT] in runner.calls
+
+
+class TestCurrentMountsUnder:
+    def test_matches_only_the_staging_subtree(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        # "/run/truecloud-nested/cloud_backup-50" must NOT match "cloud_backup-5".
+        mounts_file.write_text(
+            f"tmpfs {ROOT} tmpfs rw 0 0\n"
+            "tmpfs /run/truecloud-nested/cloud_backup-50 tmpfs rw 0 0\n"
+        )
+        found = current_mounts_under(ROOT, mounts_file=str(mounts_file))
+        assert found == [ROOT]
+
+
+class TestStagingRootFor:
+    def test_stable_per_task(self):
+        assert staging_root_for("cloud_backup-5") == "/run/truecloud-nested/cloud_backup-5"
+
+    def test_sanitises_path_separators(self):
+        assert "/" not in staging_root_for("evil/../../etc").rsplit("/", 1)[-1]

uninstall.sh

@@ -3,7 +3,7 @@
 
 set -euo pipefail
 
-VERSION="0.0.4"
+VERSION="0.3.0"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'