v0.5.1: the update alert could have broken middlewared at startup

middlewared's alert.load() imports every file in alert/source/ with NO try/except:

    def load(self):
        for module in load_modules(.../alert/source):
            for cls in load_classes(module, AlertSource, (ThreadedAlertSource,)):
                ...

and it runs during setup. A module that raises on import therefore takes
middlewared's startup down with it -- exactly the class of failure this project
exists to avoid.

apply.sh now COMPILES the substituted alert source and refuses to write it if it
does not parse. An uninstalled alert is a missing convenience; a broken one is a
broken box.

@PATCH_DIR@ is also substituted with repr() rather than raw, so a repository path
containing a quote or backslash yields a valid Python literal instead of a syntax
error in the installed module.

The alert source no longer mutates sys.path. It loaded tools/release_notes.py via
sys.path.insert(0, ...), which shadows the stdlib for that interpreter -- and
ThreadedAlertSource runs in middlewared's thread pool, so mutating sys.path is a
race. It now loads by file path with importlib.

New tests guard every import-time failure mode: the module compiles, apply.sh
compiles before writing, awkward paths (quotes, backslashes) still produce valid
modules, nothing but imports/constants/classes runs at module scope, every
AlertClass name ends in "AlertClass" (AlertClassMeta raises NameError otherwise),
the alert text placeholders match the args passed, and no git command that writes
to .git is ever used.

152 tests, ruff and shellcheck -S style clean.

.github/workflows/ci.yml

@@ -0,0 +1,62 @@
+name: CI
+
+on:
+  push:
+    branches: [main, "feat/**", "fix/**"]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  shell:
+    name: shell (shellcheck + syntax)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: bash syntax check
+        run: |
+          fail=0
+          while IFS= read -r f; do
+            bash -n "$f" || { echo "::error file=$f::bash syntax error"; fail=1; }
+          done < <(find . -name '*.sh' -not -path './.git/*')
+          exit $fail
+
+      # Pinned to a release tag, not @master: a third-party action on a moving
+      # branch runs whatever that branch contains at the time CI fires.
+      - name: shellcheck
+        uses: ludeeus/action-shellcheck@2.0.0
+        env:
+          SHELLCHECK_OPTS: -S warning -e SC1091
+
+  python:
+    name: python ${{ matrix.python }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # TrueNAS SCALE middleware runs 3.11+; keep the patch importable across
+        # the versions it may be injected into.
+        python: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: install dev deps
+        run: python -m pip install --upgrade pip pytest ruff
+
+      - name: ruff
+        run: ruff check patch tests tools
+
+      - name: pytest
+        run: pytest tests -v
+
+      - name: verify injected middleware blocks compile
+        # Belt-and-braces: the *_BLOCK strings are appended into live middlewared
+        # modules. A syntax error there would break the box at boot.
+        run: pytest tests/test_apply_blocks.py -v

.github/workflows/release.yml

@@ -0,0 +1,102 @@
+name: Release
+
+# Push a tag, get a release. The body always comes from CHANGELOG.md, so there is
+# no second place to write release notes and therefore no second place for them to
+# go stale.
+#
+#   git tag -a v0.4.0 -m "v0.4.0" && git push origin v0.4.0
+#
+# workflow_dispatch re-cuts (or updates) the release for a tag that already
+# exists, since re-pushing an existing tag triggers nothing.
+#
+# It checks out the TAG, because the tagged code is what people install and it has
+# to pass its own tests. That means it only works for tags that actually contain
+# this tooling (>= v0.3.0). Tags older than that were backfilled by hand.
+
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Existing tag to create a release for (e.g. v0.2.1)"
+        required: true
+        type: string
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Resolve tag
+        id: tag
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            echo "tag=${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
+          else
+            echo "tag=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
+          fi
+
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ steps.tag.outputs.tag }}
+          fetch-depth: 0
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      # Never publish a release for code that does not pass its own tests. A
+      # tagged commit is what people install; it has to be at least as good as
+      # main.
+      - name: install dev deps
+        run: python -m pip install --upgrade pip pytest ruff
+
+      - name: ruff
+        run: ruff check patch tests tools
+
+      - name: pytest
+        run: pytest tests -q
+
+      - name: shell syntax
+        run: |
+          fail=0
+          while IFS= read -r f; do
+            bash -n "$f" || { echo "::error file=$f::bash syntax error"; fail=1; }
+          done < <(find . -name '*.sh' -not -path './.git/*')
+          exit $fail
+
+      # Catches the failure mode this repo actually had: VERSION= drifted to
+      # three different values across the scripts, and nothing noticed.
+      - name: version matches tag and CHANGELOG has a section
+        run: python3 tools/release_notes.py check "${{ steps.tag.outputs.tag }}"
+
+      - name: extract release notes from CHANGELOG
+        run: |
+          python3 tools/release_notes.py notes "${{ steps.tag.outputs.tag }}" > /tmp/notes.md
+          echo "--- release body ---"
+          cat /tmp/notes.md
+
+      - name: create or update the release
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TAG: ${{ steps.tag.outputs.tag }}
+        run: |
+          # Pre-1.0 and any -rc/-beta suffix ship as prereleases, not "Latest".
+          prerelease=""
+          case "$TAG" in
+            *-rc*|*-beta*|*-alpha*) prerelease="--prerelease" ;;
+          esac
+
+          if gh release view "$TAG" >/dev/null 2>&1; then
+            echo "Release $TAG exists — updating notes."
+            gh release edit "$TAG" --notes-file /tmp/notes.md
+          else
+            # shellcheck disable=SC2086
+            gh release create "$TAG" \
+              --title "$TAG" \
+              --notes-file /tmp/notes.md \
+              $prerelease
+          fi

.gitignore

@@ -4,3 +4,13 @@
 /apply.log.2
 /hook_status.json
 /disabled
+/nested_snapshots_enabled
+
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.venv/
+venv/
+/update_alerts_disabled

CHANGELOG.md

@@ -1,5 +1,483 @@
 # Changelog
 
+## v0.5.1 — 2026-07-13
+
+### Fixed
+
+- **The update alert could have broken middlewared at startup.** middlewared's
+  `alert.load()` imports every file in `alert/source/` with **no try/except**, and
+  it runs during setup — so a module that raises on import takes middlewared down
+  with it. `apply.sh` now **compiles the substituted alert source and refuses to
+  write it** if it does not parse. An uninstalled alert is a missing convenience;
+  a broken one is a broken box.
+
+- **`@PATCH_DIR@` is substituted with `repr()`**, so a repository path containing
+  a quote or a backslash produces a valid Python literal instead of a syntax error
+  in the installed module.
+
+- **The alert source no longer mutates `sys.path`.** It loaded
+  `tools/release_notes.py` via `sys.path.insert(0, …)`, which shadows the stdlib
+  for that interpreter — and `ThreadedAlertSource` runs in middlewared's thread
+  pool, so mutating `sys.path` is a race. It now loads the module by file path with
+  `importlib`.
+
+### Notes
+
+Timing, for the record: `process_alerts` is `@periodic(60)` and
+`alert_source_last_run` is in-memory, so the check runs **within 60 seconds of any
+middlewared restart** (which this patch performs at every boot) and otherwise
+**within 24 hours** of a release.
+
+## v0.5.0 — 2026-07-13
+
+### Added
+
+- **A TrueNAS alert when an update is available** — the bell in the UI, not a log
+  line nobody reads. On by default, checked once a day.
+  `install.sh --no-update-alerts` turns it off.
+
+  **It does not nag.** A release whose CHANGELOG contains only a `### Docs`
+  section changed no code and raises nothing. Anything else raises INFO; a
+  `### Security` section raises WARNING. The CHANGELOG's own section headings are
+  the signal, and a security fix anywhere in the range escalates the whole span —
+  so a docs-only release sitting on top of a security fix still reports as
+  security, rather than hiding it.
+
+  **Why an AlertSource and not `midclt`:** TrueNAS cannot raise an alert from the
+  CLI. `midclt` exposes only `alert.dismiss`, `alert.list`, `alert.list_categories`,
+  `alert.list_policies` and `alert.restore` — alert *creation* is internal to
+  middlewared, and none of its ~60 one-shot classes is generic enough to reuse. So
+  registering an `AlertSource` is the only way, and it is also the least invasive
+  thing this patch does: it **adds one file and modifies none**, where the
+  providers and nested modules both append code to stock middleware files. It is
+  the native mechanism, and TrueNAS polls it itself — no cron, no systemd timer.
+
+  - Fail-safe: every error path returns `None`; it cannot take middlewared down.
+  - Read-only: `git ls-remote` plus an HTTPS fetch of the CHANGELOG. It never
+    writes to `.git`, so it cannot leave root-owned objects behind the way a
+    `git fetch` from middlewared (running as root) would.
+  - Removed by `uninstall.sh`.
+  - It only *tells* you; it never updates anything.
+
+## v0.4.2 — 2026-07-13
+
+### Docs
+
+- **The Updating section never said how to *get* `update.sh`.** It ships inside the
+  patch, so a clone older than v0.4.0 doesn't have it — the docs told you to run a
+  script you didn't have. There is now an explicit bootstrap step (`git pull &&
+  bash install.sh`, once), including the fix for the *"insufficient permission for
+  adding an object to repository database"* failure that past `sudo git pull`s
+  cause.
+
+- **`After a TrueNAS update` rewritten.** It didn't explain that the patch
+  re-applies itself at every boot (so you never reinstall), and it didn't say what
+  each failure actually costs you. "Fail-safe" means *the box stays up* — not that
+  your backups keep running. A `[FAIL] providers` is a **broken backup**, and the
+  docs now say so rather than implying everything degrades gracefully.
+
+- Added a repo map. `patch/mw_patch.py` and `tools/release_notes.py` were
+  documented nowhere.
+
+- `Development` told you to run `ruff check patch tests`, which misses `tools/`.
+
+- Every command and file path in the README is now verified to exist and run.
+
+## v0.4.1 — 2026-07-13
+
+### Fixed
+
+- **`update.sh` would have picked a release candidate as "the newest release".**
+  Git's version sort ranks `v0.5.0-rc1` *above* `v0.5.0` (verified), and the
+  release workflow deliberately supports rc/beta tags — so an RC would have been
+  installed as though it were the latest stable. Tag selection is now filtered to
+  plain `vX.Y.Z`.
+
+- **`update.sh` would have died mid-update on an untracked file.** The dirty-tree
+  guard uses `--untracked-files=no`, so an untracked file that the *target* tracks
+  slipped past it — and `git checkout` then aborts. Under `set -e` the script died
+  with a raw git error, *after* recording the rollback point. This is exactly what
+  blocked a pull on a real box (a hand-copied `patch/wait_restart.sh`). It now
+  detects the collision up front and names the files. Gitignored files are
+  correctly *not* treated as blockers — git overwrites those silently.
+
+  Special case: if `update.sh` *itself* is the blocker, you hand-copied it in to
+  bootstrap — and "delete `update.sh`, then re-run `update.sh`" is impossible. It
+  now says so and prints the git commands that bootstrap it properly.
+
+- **`--rollback` skipped that check entirely**, so it would have hit the identical
+  failure. The check is now a shared function used by both paths, and rollback also
+  validates that the recorded revision still exists (history can be rewritten).
+
+- `install.sh`'s `chmod` aborted under `set -e` if any listed file was missing. The
+  file set changes between versions, so `update.sh --rollback` to an older revision
+  must not be killed by a filename this version happens to know about.
+
+- `--to` with no value was silently ignored and fell back to the default target.
+
+## v0.4.0 — 2026-07-13
+
+### Added
+
+- **`update.sh`** — fetch a newer release and apply it, preserving your
+  nested-snapshot opt-in setting.
+
+  ```bash
+  bash update.sh              # to the newest release, with a confirmation
+  bash update.sh --check      # show what would happen; change nothing
+  bash update.sh --rollback   # undo the last update
+  ```
+
+  **Run it by hand. Never from cron or a systemd timer.** This patch injects
+  Python into middlewared and re-applies itself at every boot, so an unattended
+  pull would let any bad upstream commit reach your box with no human in the loop
+  and take effect on the next reboot. v0.0.4 shipped exactly such a bug and took
+  every app on the box down. The manual step *is* the safety gate.
+
+  Design:
+
+  - **Defaults to the newest release tag, not `main`.** `main` can be mid-refactor;
+    a tag is the tested artifact. `--main` exists but says so loudly.
+  - Tags are ordered by **version**, not by date — date order silently downgrades
+    the box the first time a hotfix is tagged out of band (a v0.3.6 released after
+    v0.4.0 would sort as "newest").
+  - **Refuses to run over a dirty working tree** rather than merging across
+    hand-edited or scp'd files.
+  - Shows the commits you don't have and the target's release notes (read from the
+    *target's* CHANGELOG, via `tools/release_notes.py` — not a second copy of the
+    extractor), then asks before doing anything.
+  - **Records the previous revision before moving**, so `--rollback` works even if
+    `install.sh` dies halfway.
+  - Repairs `.git` ownership, which past `sudo git pull`s leave root-owned and
+    which then breaks every later non-root git command.
+
+- `update.sh` is covered by the version-drift check, so it cannot quietly go stale
+  the way `create_task.py.__version__` did.
+
+## v0.3.5 — 2026-07-13
+
+### Changed
+
+- `delete_snapshot_tree` swallowed the error from its recursive-delete fast path.
+  That failure is *usually* just "parent already gone" — stock's `finally` winning
+  the race once our mounts are released, which the by-name sweep then handles. But
+  if the cause were anything else, this was the only place it was visible, and it
+  went straight to `/dev/null`. It is now logged before falling through.
+
+- Annotated the two remaining static-analysis findings as considered-and-accepted
+  rather than leaving them to be re-litigated: `subprocess` is always called in
+  list form (no shell, so ZFS dataset names cannot inject), and the partial
+  `systemctl` path is moot in a script that only runs as root.
+
+## v0.3.4 — 2026-07-13
+
+### Changed
+
+- **One implementation of apply/revert (`patch/mw_patch.py`).** The "strip the
+  `TRUECLOUD_PATCH` block" logic existed twice — in `apply.sh`'s heredoc and in an
+  inline heredoc in `uninstall.sh` — and the uninstall copy was the untested one.
+  That is exactly how the two could have drifted apart, with `apply.sh` reverting
+  one set of files and `uninstall.sh` another. Both now call the same tested
+  module (17 new tests, including that `revert_nested` never touches `restic.py`,
+  which belongs to the providers module and whose removal would silently break B2
+  backups).
+
+  `apply.sh` imports it fail-safe: if it cannot, the backend patch is skipped and
+  middlewared starts stock, which is this script's whole design principle. The
+  import uses `sys.path.append`, never `insert(0)` — prepending would give
+  `patch/` precedence over the stdlib for that interpreter, so a future
+  `patch/json.py` would shadow the real `json` and break the boot.
+
+### Docs
+
+- The README's `create_task.py` example still taught `--password <secret>`, which
+  is how a security fix quietly fails to land. It now shows `--password-stdin`.
+
+## v0.3.3 — 2026-07-13
+
+### Security
+
+- **The restic repository password no longer passes through a process's argv.**
+  `create_task.py` shelled out to `midclt call cloud_backup.create '<json>'`, and
+  that JSON contains the repo password — so it appeared in the process's argv,
+  which is world-readable via `ps`, for the duration of the call. That password is
+  the encryption key for the entire cloud backup repository.
+
+  It now talks to the middleware through `truenas_api_client` (the library that
+  backs `midclt` itself), so the password never leaves the process's memory.
+
+- **`--password` no longer required.** Passing a secret as a CLI argument writes it
+  to shell history permanently. `--password-stdin` reads it from stdin, and with
+  neither flag the tool prompts via `getpass`. `--password` still works but now
+  warns.
+
+### Fixed
+
+- **`uninstall.sh` could leave every patch installed.** It reverted by unmounting
+  the overlay — but `apply.sh` only mounts one when the target directory is
+  read-only. On a writable `/usr` it patches the real files in place, and uninstall
+  would remove the boot hook, report success, and leave the patch applied. It now
+  strips the appended blocks from the middleware files explicitly.
+
+- **`create_task.py.__version__` had been stuck at `0.2.0`** for three releases.
+  The version-drift check added in v0.3.1 only looked at `VERSION=` in shell
+  scripts, so it missed the one file that actually shows a version to users
+  (`--version`). The check now covers `__version__` too — and caught this
+  immediately.
+
+## v0.3.2 — 2026-07-13
+
+### Fixed
+
+- **`install.sh --disable-nested-snapshots` did not actually disable anything
+  until the next reboot.** `apply.sh` only ever *added* patches — there was no
+  revert path. Disabling removed the opt-in marker and then merely *skipped*
+  re-applying, but the overlay persists for the whole boot, so the previously
+  patched `plugins/cloud/{snapshot,crud}.py`, `plugins/cloud_backup/sync.py` and
+  `_truecloud_nested.py` were all still sitting there — and middlewared
+  re-imported them on the restart `install.sh` performs.
+
+  It printed *"DISABLED (stock guard restored)"* while the feature kept running.
+  Someone turning it off *because they were worried about it* would have believed
+  it was off.
+
+  `apply.sh` now actively reverts: it removes the module first (every injected
+  block is guarded by `if _tc_nested is not None`, so the stock guard is restored
+  even if a later step fails), then strips its appended blocks from the three
+  patched files. `restic.py` also carries a `TRUECLOUD_PATCH` block but belongs to
+  the *providers* module and is deliberately left alone — reverting it would break
+  B2 backups. `install.sh --disable` also tears down any staging tree first, since
+  those bind mounts pin ZFS snapshots that could otherwise never be destroyed.
+
+  Updating **without** the flag was always correct and is unchanged: the nested
+  module is never installed into middleware unless it is explicitly enabled.
+
+## v0.3.1 — 2026-07-13
+
+### Added
+
+- **Automated releases.** Pushing a `v*` tag runs the full test suite and then
+  cuts a GitHub release whose body is the matching `CHANGELOG.md` section — so
+  release notes have exactly one source of truth, and no second place to go stale.
+  The workflow refuses to publish if the tests fail, if the tag does not match the
+  `VERSION=` declared by every script, or if the CHANGELOG has no section for it.
+
+- **Version-drift check.** `VERSION=` had silently diverged to three different
+  values across `install.sh`, `uninstall.sh`, `recover.sh`, and `patch/apply.sh`,
+  and nothing noticed. CI now asserts every script agrees with the others and with
+  the newest CHANGELOG entry.
+
+### Note
+
+- Releases for `v0.2.0` and `v0.2.1` were backfilled — they had been tagged but
+  never released, so the releases page jumped v0.1.0 → v0.3.0 and hid the fix for
+  the boot race that took every app down.
+
+## v0.3.0 — 2026-07-13
+
+### Added
+
+- **`snapshot = true` now works on datasets that have child datasets** —
+  **opt-in, off by default** (`install.sh --enable-nested-snapshots` /
+  `--disable-nested-snapshots`). It changes how backups read their source data,
+  so it is never enabled implicitly; with neither flag `install.sh` preserves
+  the existing setting, so a `git pull && bash install.sh` cannot silently flip
+  it. When disabled, `apply.sh` skips the patch entirely and the stock guard
+  remains. `uninstall.sh` tears down any staging mounts and removes the marker.
+  Stock TrueNAS refuses this with *"This option is only available for datasets
+  that have no further nesting"*, which makes the snapshot option unusable for
+  the single most common case on any box running Apps — every app is its own
+  dataset, often with `config`/`pgdata` children of its own. Without it, the
+  backup reads **live** files: databases are captured mid-write, and a busy app
+  rewriting its files can stall a backup indefinitely as restic chases a moving
+  target.
+
+  The stock guard is **correct, and it is not an arbitrary limit.**
+  `plugins/cloud/snapshot.py` already takes a *recursive* ZFS snapshot, but it
+  then points the backup tool at the **parent** dataset's
+  `.zfs/snapshot/<snap>/` directory — and ZFS does not expose child datasets
+  through a parent's snapshot directory:
+
+  ```
+  /mnt/Tap/.zfs/snapshot/<snap>/apps/               -> 0 entries (children invisible)
+  /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
+  ```
+
+  So without the guard the backup tool would walk a near-empty tree, report
+  SUCCESS, and upload almost nothing. iX gate the config rather than ship a
+  backup that lies about succeeding.
+
+  This release implements the missing half. After the (already recursive)
+  snapshot is taken, every descendant dataset's own `.zfs/snapshot/<snap>` is
+  bind-mounted into a **staging tree** mirroring the original layout, and the
+  backup tool is pointed at the staging root — a complete, consistent,
+  point-in-time view of the whole subtree. Only then is the guard relaxed.
+
+  Safety properties, in order of importance:
+
+  - **Staging failure is loud.** If any descendant cannot be staged, the backup
+    fails. A silently-incomplete backup is the exact outcome the stock guard
+    exists to prevent, and it would be worse than not having the feature.
+  - **A post-mount verification pass** asserts every planned target is really a
+    mountpoint and the staging root is non-empty, so this can never regress into
+    the empty-backup failure it is meant to fix.
+  - **The guard is relaxed last.** `apply.sh` installs the traversal, patches
+    `snapshot.py`, then `sync.py`, and only then `crud.py`. A partial failure
+    leaves the guard intact and the option merely unavailable — never
+    "guard removed, traversal missing".
+  - **The patch owns the whole snapshot lifecycle.** `zfs.snapshot.delete`
+    defaults to `recursive=False` and stock `restic_backup()` calls it with no
+    options. Stock gets away with that only because its validation means
+    `recursive` is never True in the field — but enabling nested datasets makes
+    recursive snapshots real, so the parent now has one child snapshot per
+    descendant dataset (160+ on a typical Apps pool). Relying on stock's delete
+    would therefore orphan every child snapshot **on every successful run**.
+    This patch sweeps the parent *and* all children, is idempotent against
+    stock's `finally` winning the race, records the snapshot in a sidecar file
+    (so a middlewared restart mid-backup cannot orphan it), reclaims the tree
+    left by a crashed run, and deletes the tree when staging fails — where
+    sync.py's own `finally` would otherwise delete nothing at all, because its
+    `snapshot` local never gets assigned.
+  - **The dataset list is enumerated *after* the snapshot, never before.** A
+    list read beforehand can miss a dataset created in the gap: the recursive
+    snapshot would capture it but the staging plan would not, silently omitting
+    its data. Read afterwards, an unsnapshotted dataset trips the staging check
+    and fails the run loudly instead.
+  - **Every injected block no-ops** if `_truecloud_nested` is absent.
+  - Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
+    unmounted/locked, encrypted-and-locked) are skipped and **reported** —
+    never dropped silently.
+  - Scoped to `cloud_backup` only. Cloud Sync (rclone) shares the same
+    validation mixin but has no staging teardown wired in, so its guard is left
+    in place deliberately.
+
+  Side benefit: the staging root is a **stable** path per task, so restic can
+  find its parent snapshot between runs. Stock's
+  `.zfs/snapshot/<name>-<timestamp>/` path changes every run, which defeats
+  restic's parent detection and forces a full re-scan each time.
+
+- **CI** (GitHub Actions): shellcheck + `bash -n` on every script, ruff, and
+  pytest on Python 3.11/3.12/3.13. Includes tests that `compile()` the
+  `*_BLOCK` strings — they are Python source appended to live middlewared
+  modules, so a syntax error there would break the box at boot, and nothing
+  previously checked them.
+
+### Changed
+
+- **The patch is now two independent modules, and each retires on its own.**
+  Previously the native-support check looked only for native B2 restic support
+  and, on finding it, set the kill switch and disabled *everything*. With a
+  second capability in the patch that would silently take a still-needed module
+  down with the superseded one — TrueNAS is likely to ship one of these long
+  before the other.
+
+  `apply.sh` now detects each separately (`providers`: does `B2RcloneRemote`
+  carry a real `get_restic_config()`; `nested`: is the *"no further nesting"*
+  validation still in `plugins/cloud/crud.py`), skips just the superseded one,
+  and only sets the kill switch once **both** are done. The UI patch belongs to
+  `providers` and is skipped with it. The deferred middlewared restart now fires
+  when *any* still-needed module landed — keying it off `providers` alone would
+  have left a freshly-patched `nested` module on disk and never loaded on a
+  native-B2 box. `hook_status.json` reports each module with an `active` flag and
+  a reason.
+
+- README rewritten to be less alarmist: dropped the warning boxes and the
+  disclaimer's fear-bulleting in favour of plain statements, and documented the
+  two-module design. The one caveat kept as a plain sentence: the `mount --bind`
+  staging step has not yet been exercised by a live backup run.
+
+- Version strings in `install.sh`, `uninstall.sh`, and `recover.sh` were stale
+  at `0.0.4`; all scripts now report the same version.
+- `patch_ui.py`: replaced a `try`/`except`/`pass` with `contextlib.suppress`
+  (no behaviour change; satisfies the new lint gate).
+
+### Removed
+
+- `patch/__pycache__/create_task.cpython-314.pyc` was committed to the
+  repository; it is now untracked and `__pycache__/` is gitignored.
+
+### Fixed (post-merge audit)
+
+- **`create_task.py verify` failed on a default install.** `hook_status.json`
+  emitted a per-file entry for the nested module with `ok: false` whenever the
+  feature was switched off — which is the default — so `verify` printed `[FAIL]`
+  and exited 1 right after the README told users to run it. Status is now
+  reported per *module* with an `active` flag, and `verify` renders an inactive
+  module as `[SKIP]` rather than a failure.
+- **A partial apply suppressed the middlewared restart.** The exit code
+  conflated "nothing applied" with "one module applied, one failed", so a failing
+  providers patch would prevent the restart that a freshly-applied nested patch
+  needs — leaving it on disk and never loaded. Exit 2 now means partial, and the
+  restart still fires.
+- **The native-nested probe could never fire.** It scanned `crud.py` for the
+  guard message, but our own injected block *quotes* that message, so once
+  applied the probe would always conclude the guard was still present. It now
+  reads only the stock portion of the file.
+- `recover.sh` did not unmount staging trees, so an emergency recovery left bind
+  mounts pinning ZFS snapshots that could then never be destroyed.
+- `uninstall.sh` deleted sidecar files without reading them. A sidecar is the
+  only record that an interrupted run's snapshot tree is still on disk; both
+  scripts now name the snapshot (`zfs destroy -r ...`) before clearing it.
+
+- **The native-nested probe could never detect the guard, silently disabling the
+  whole module.** Stock splits the message across adjacent string literals:
+
+  ```python
+  verrors.add(f"{name}.snapshot", "This option is only available for datasets that have no further "
+                                  "nesting")
+  ```
+
+  Python concatenates those at runtime — so the *errmsg* is contiguous and the
+  runtime filter works — but the **source never contains the whole phrase**. The
+  probe's substring search found nothing, concluded iX had removed the guard, and
+  skipped the nested module as "already native". `apply.log` would report
+  *"TrueNAS now handles nesting natively"* and the feature would never work.
+  It fails safe (the stock guard stays, so no data is at risk) but the module was
+  100% dead. The probe now strips whitespace and quotes before matching, which is
+  robust to any wrapping style. Caught only by running the probe against real
+  middlewared; there is now a regression test that executes apply.sh's own probe
+  code against the real wrapped source.
+
+### Changed (production audit)
+
+- **`delete_snapshot_tree` now uses a single recursive delete.** It previously
+  removed the parent and each child snapshot one at a time — 252 sequential
+  middleware calls on a real pool. That is slow, but the real problem is that it
+  is **not atomic**: a run killed part-way through the sweep leaves exactly the
+  orphaned snapshots the function exists to prevent. It now issues one
+  `zfs.snapshot.delete(..., {"recursive": True})` and falls back to the
+  name-by-name sweep only when that fails (e.g. stock's `finally` already removed
+  the parent, which leaves the children behind).
+
+### Refactored
+
+- Staging teardown had been copy-pasted into `uninstall.sh` and `recover.sh` —
+  two untested shell copies of the fiddly depth-ordering and lazy-umount logic.
+  Both now call `python3 patch/truecloud_nested.py cleanup`, so there is one
+  implementation and it is the one under test.
+- Dropped the in-memory `ACTIVE` dict. The sidecar file was already the source of
+  truth; a second in-process record could only desync — and it is the
+  middlewared-restart case (which empties it) that must not orphan a snapshot
+  tree. One record, on disk, or none.
+
+### Validated in production
+
+An unattended scheduled backup of a live 252-dataset pool (`/mnt/Tap`, TrueNAS
+25.10) ran through the staging tree end to end:
+
+- 252 datasets recursively snapshotted, 173 bind mounts built and verified
+- completed in **18m14s**, `SUCCESS` — the same task previously stalled at 74%
+  for over 12 hours reading live files
+- **zero** orphaned ZFS snapshots and **zero** stale mounts afterwards, which is
+  the failure mode that would otherwise have accumulated 251 snapshots per run
+
+### Known issues
+
+- Stock `restic_backup()` deletes the ZFS snapshot in its own `finally`, which
+  fails with `EBUSY` while the staging bind mounts pin it. It logs one benign
+  `Error deleting snapshot ...` warning per run; the patch then unmounts and
+  deletes the snapshot for real. The warning is expected and harmless.
+
 ## v0.2.1 — 2026-07-09
 
 ### Fixed

README.md

@@ -1,7 +1,10 @@
 # truenas-truecloud-patch
 
-Extends TrueNAS SCALE's **TrueCloud Backup** feature to work with S3-compatible
-providers and native Backblaze B2, instead of Storj only.
+Extends TrueNAS SCALE's **TrueCloud Backup** feature to:
+
+- work with S3-compatible providers and native Backblaze B2, instead of Storj only;
+- take **consistent snapshots of datasets that have child datasets** — which is
+  every box running Apps (see [Nested-dataset snapshots](#nested-dataset-snapshots)).
 
 ---
 
@@ -24,31 +27,68 @@ costs a fraction of the new Storj price.
 
 ---
 
-## ⚠ Disclaimer — please read before installing
+## Before you install
 
-**This project is unofficial, unsupported, and not affiliated with iXsystems
-or the TrueNAS project in any way.**
+This project is unofficial and not affiliated with iXsystems. A few things worth
+knowing:
 
-By installing this patch you accept the following:
+- It targets **internal middleware APIs** with no stability contract, so a
+  TrueNAS update can break it. Every patch is fail-safe: if it can't apply,
+  middlewared starts normally and the reason is logged to `apply.log`. Check the
+  log after an update.
+- If you file a TrueNAS bug report, **remove the patch first** and reproduce on a
+  stock system.
+- **Test your restores.** True of any backup, but it matters more here — see
+  [Verifying it works](#verifying-it-works).
+- Provided as-is, no warranty. See LICENSE.
 
-- **Unsupported configuration.** TrueNAS support staff are not obligated to
-  help with any issue on a system running this patch. If you file a bug report,
-  remove the patch first and reproduce the issue on an unmodified system.
+The patch is two independent modules — **providers** (B2/S3) and **nested**
+(snapshots on nested datasets) — and each retires on its own once TrueNAS ships
+that capability natively. See [Native support](#if-truenas-adds-native-support).
 
-- **May break on TrueNAS updates.** The patch targets internal middleware APIs
-  that are not part of any public contract. They can change at any time. When
-  they do, the patch silently degrades to Storj-only behaviour rather than
-  breaking TrueNAS — but you should check the log after each update.
+## What's in the repo
 
-- **Your backups are your responsibility.** Verify that your backup jobs
-  complete successfully and that restores work before relying on them for
-  disaster recovery.
+| Path | What it is |
+|---|---|
+| `install.sh` | Registers the PREINIT boot hook, applies the patch, restarts middlewared. Also `--enable/--disable-nested-snapshots`. |
+| `update.sh` | Fetch a newer **release** and apply it. `--check`, `--rollback`, `--to`, `--main`. |
+| `uninstall.sh` | Remove everything: boot hook, patched files, UI bundle, staging mounts. |
+| `recover.sh` | Emergency: set the kill switch and restart middlewared against stock files. |
+| `patch/apply.sh` | The PREINIT script. Runs at **every boot**; re-applies the patch into a fresh overlay. |
+| `patch/mw_patch.py` | The one implementation of apply/revert for the `TRUECLOUD_PATCH` blocks. Used by `apply.sh` *and* `uninstall.sh`. |
+| `patch/truecloud_nested.py` | Nested-dataset staging: plan, mount, verify, tear down, sweep snapshots. Also `… cleanup` as a CLI. |
+| `patch/patch_ui.py` | Widens the Angular credential dropdown. Refuses to write a bundle whose parens it unbalanced. |
+| `patch/create_task.py` | Create TrueCloud tasks with S3/B2 credentials; `verify` the patch state. |
+| `patch/alert_source.py` | The TrueNAS alert for "an update is available". Installed into `middlewared/alert/source/`. |
+| `patch/wait_restart.sh` | Waits for boot to actually settle before restarting middlewared. |
+| `tools/release_notes.py` | Extracts a version's CHANGELOG section; enforces version consistency. Used by CI. |
 
-- **No warranty.** This software is provided as-is. See the LICENSE file.
+## Development
 
-If TrueNAS adds native B2 or S3 support to TrueCloud Backup, the patch
-detects it at boot, disables itself, and tells you to run `uninstall.sh` —
-see [Native support](#if-truenas-adds-native-support) below.
+Parts of this project were written with AI assistance (Claude). All of it is
+reviewed and tested before release; the test suite and CI exist in large part to
+make that review meaningful. Bugs are mine.
+
+```bash
+pip install pytest ruff
+ruff check patch tests tools
+pytest tests
+```
+
+CI runs shellcheck, `bash -n`, ruff, and pytest on Python 3.11–3.13. Two checks
+are worth calling out, because nothing else would catch what they catch:
+
+- The tests **`compile()` the `*_BLOCK` strings** in `patch/apply.sh`. Those are
+  Python source appended into live `middlewared` modules — a syntax error there
+  breaks the box at boot, and they're string literals, so nothing else type-checks
+  them.
+- CI asserts **every script declares the same version**, and that it matches the
+  newest CHANGELOG entry. `VERSION=` had silently drifted to three different
+  values across the scripts before anything checked.
+
+Releases are automated: push a `vX.Y.Z` tag and the workflow runs the full suite,
+verifies the version matches, and cuts a GitHub release whose body **is** the
+matching `CHANGELOG.md` section — one source of truth for release notes.
 
 ---
 
@@ -74,14 +114,173 @@ restarting. Expect one middlewared restart shortly after every boot; the UI
 and API are briefly unavailable while it happens, and running services are
 not affected.
 
-| Layer | What changes | Technique |
+| Module | What changes | Technique |
 |---|---|---|
-| **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | File patch applied inside the overlayfs upper layer |
-| **UI** | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS chunk; original is backed up before patching |
+| **providers** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | File patch applied inside the overlayfs upper layer |
+| **providers** (UI) | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS chunk; original is backed up before patching |
+| **nested** (opt-in) | `_truecloud_nested.py` is installed into `plugins/cloud/`, and `plugins/cloud/{snapshot,crud}.py` + `plugins/cloud_backup/sync.py` are patched so `snapshot = true` works on a dataset that has child datasets. See [below](#nested-dataset-snapshots). | New module + file patches inside the overlayfs upper layer |
 
-Both changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
-restructured the relevant code), middlewared starts normally with Storj-only
-support and the reason is logged to `apply.log` in your repo root.
+All changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
+restructured the relevant code), middlewared starts normally, the affected module
+is simply inactive, and the reason is logged to `apply.log` in your repo root. The
+two modules are independent — one failing or going native does not disable the
+other.
+
+## Nested-dataset snapshots
+
+**Opt-in, off by default.** It changes how backups read their source data, so it
+is never enabled implicitly:
+
+```bash
+bash install.sh --enable-nested-snapshots
+bash install.sh --disable-nested-snapshots
+```
+
+With neither flag `install.sh` leaves the setting alone, so `git pull && bash
+install.sh` won't flip it. The providers module is unaffected either way.
+
+Validated end to end on a live 252-dataset pool: an unattended scheduled backup
+of `/mnt/Tap` built a 173-mount staging tree, completed in **18m14s**, and left
+**zero** orphaned snapshots and **zero** stale mounts behind. The same backup
+previously stalled at 74% for over 12 hours reading live files.
+
+Still: verify your own first run actually contains child-dataset data before you
+rely on it — see [Verifying it works](#verifying-it-works). That advice is not
+boilerplate; it is the specific thing this feature exists to make true.
+
+TrueCloud Backup's **Take Snapshot** option makes restic read from a frozen ZFS
+snapshot instead of live files. Without it the backup reads data *while apps are
+writing to it* — databases get captured mid-write, and an app that rewrites its
+files continuously can stall a backup indefinitely as restic chases a moving
+target.
+
+Stock TrueNAS refuses to enable it on most real-world paths:
+
+```
+[EINVAL] cloud_backup_update.snapshot:
+  This option is only available for datasets that have no further nesting
+```
+
+That rules out **any pool running Apps** — every app is its own dataset, usually
+with `config`/`pgdata` children of its own. On a typical box that is 100+ nested
+datasets, so the feature is effectively unusable exactly where it matters most.
+
+### Why stock refuses
+
+The guard is **correct**. `plugins/cloud/snapshot.py` already takes a *recursive*
+ZFS snapshot — but it then points restic at the **parent** dataset's
+`.zfs/snapshot/<snap>/` directory, and ZFS does not expose child datasets
+through a parent's snapshot directory:
+
+```
+/mnt/Tap/.zfs/snapshot/<snap>/apps/                 ->  0 entries  (children invisible)
+/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/   ->  the real data
+```
+
+So if you just remove the validation, restic walks a near-empty tree, reports
+SUCCESS, and uploads almost nothing — a green backup job protecting no data. iX
+gate the config rather than ship a backup that lies about succeeding.
+
+That is worth spelling out, because deleting those four lines in
+`plugins/cloud/crud.py` is the obvious "fix" and it is the wrong one. The guard
+is load-bearing: it has to be *replaced* with a working traversal, not removed.
+
+### What this patch does instead
+
+After the (already recursive) snapshot is taken, every descendant dataset's own
+`.zfs/snapshot/<snap>` is bind-mounted into a **staging tree** that mirrors the
+original layout, and restic is pointed at the staging root — a complete,
+consistent, point-in-time view of the whole subtree. Only then is the guard
+relaxed.
+
+Safety properties, in order of importance:
+
+- **Staging failure is loud.** If any descendant cannot be staged, the backup
+  *fails*. A silently-incomplete backup is precisely what the stock guard exists
+  to prevent, and it would be worse than not having the feature at all.
+- **Post-mount verification** asserts every planned target really is a mountpoint
+  and the staging root is non-empty — so this cannot regress into the empty
+  backup it exists to fix.
+- **The guard is relaxed last.** `apply.sh` installs the traversal, patches
+  `snapshot.py`, then `sync.py`, and only then `crud.py`. Any partial failure
+  leaves the guard intact and the option merely unavailable — never
+  "guard removed, traversal missing".
+- Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
+  unmounted, locked/encrypted) are skipped and **reported to the log** — never
+  dropped silently.
+- Scoped to **cloud_backup only**. Cloud Sync (rclone) shares the same
+  validation mixin but has no staging teardown wired in, so its guard is
+  deliberately left in place.
+
+Side benefit: the staging root is a **stable path per task**, so restic can find
+its parent snapshot between runs. Stock's `.zfs/snapshot/<name>-<timestamp>/`
+path changes every run, defeating restic's parent detection and forcing a full
+re-scan each time.
+
+### Snapshot lifecycle
+
+`zfs.snapshot.delete` defaults to **`recursive=False`**, and stock
+`restic_backup()` calls it with no options. Stock is safe only because its
+validation means a *recursive* snapshot never actually happens in the field.
+Enabling nested datasets makes them real: on a 250-dataset pool,
+`zfs snapshot -r` creates **250 snapshots**, and stock's delete removes only the
+parent — orphaning **249 on every successful run** (measured, not theorised).
+
+So the patch owns the whole lifecycle:
+
+- **Sweeps the parent and every child**, and is idempotent against stock's
+  `finally` winning the race once the mounts are released.
+- **Records the snapshot in a sidecar file before mounting anything**, so a
+  middlewared restart mid-backup cannot orphan the tree (this patch *schedules*
+  a restart at boot, so that is not hypothetical).
+- **Reclaims the tree left by a crashed run** instead of overwriting the record.
+- **Deletes the tree when staging fails** — sync.py's own `finally` deletes
+  *nothing* in that case, because its `snapshot` local never gets assigned.
+- **Enumerates datasets *after* the snapshot, never before.** A list read
+  beforehand can miss a dataset created in the gap, which the recursive snapshot
+  *would* capture but the staging plan would not — a silent omission.
+
+**Expected log noise:** stock's delete fails with `EBUSY` while the staging
+mounts pin the snapshot. You will see one benign `Error deleting snapshot ...`
+warning per run; the patch then unmounts and deletes the tree for real.
+
+### Verifying it works
+
+This feature exists because a backup can report SUCCESS while containing
+nothing, so check the contents rather than the exit status:
+
+```bash
+# 1. Does the restic snapshot actually contain child-dataset data?
+#    Pick a path that lives in a CHILD dataset (e.g. an app's config).
+midclt call cloud_backup.list_snapshots <task_id> | head
+
+# 2. List a child-dataset path inside the newest restic snapshot.
+#    If this is empty, the staging tree did not work and you are backing up NOTHING.
+midclt call cloud_backup.list_snapshot_directory <task_id> "<snapshot_id>" "/apps/lidarr/config"
+```
+
+You should see the app's real files (`lidarr.db`, `config.xml`, …). An empty
+listing means the child datasets were not staged; disable the feature and open an
+issue.
+
+```bash
+# 3. No snapshots may be left behind after a run.
+zfs list -t snapshot -r <pool> | grep -c cloud_backup-   # expect 0 between runs
+
+# 4. No staging mounts may be left behind.
+mount | grep truecloud-nested                            # expect no output
+```
+
+### Troubleshooting
+
+| Symptom | Cause |
+|---|---|
+| `This option is only available for datasets that have no further nesting` | Feature not enabled. Run `install.sh --enable-nested-snapshots`, then restart middlewared. |
+| Backup fails: `dataset '…' has no snapshot '…'; refusing to back up an incomplete tree` | Working as designed — a descendant dataset was not covered by the snapshot. The backup is refused rather than silently omitting that data. |
+| Backup fails: `snapshot '…' cannot be read (Permission denied)` | The snapshot exists but is unreadable. Middleware runs as root, so this indicates a real permissions problem, not a missing snapshot. |
+| `cloud_backup-*` snapshots accumulating | The sweep is not running. Check `apply.log` for the nested patch applying, and confirm `sync.py` carries the `TRUECLOUD_PATCH` block. |
+| Web UI blank after a patch | A bad pattern unbalanced the bundle. `apply.sh` now refuses to write in that case, but if you hit it on an older version: restore `chunk-*.js.pre-truecloud-patch` over the live chunk, then re-run `install.sh`. (`MARKER` makes an already-patched file skip, so the patch cannot heal a corrupted bundle by itself.) |
+| Stale mounts under `/run/truecloud-nested` | A crashed run. The next backup tears them down. To clear them now: `python3 patch/truecloud_nested.py cleanup` (also run by `uninstall.sh` and `recover.sh`). It names any ZFS snapshot an interrupted run left pinned. |
 
 ## Supported providers after patching
 
@@ -169,27 +368,116 @@ Refresh your browser. S3 and B2 credentials now appear in the
 
 ## Updating
 
-To update to a new version of the patch:
+```bash
+cd /mnt/tank/truenas-truecloud-patch
+bash update.sh              # to the newest release, with a confirmation
+```
+
+| | |
+|---|---|
+| `bash update.sh` | Update to the newest release tag. Shows what's coming, asks first. |
+| `bash update.sh --check` | Show what *would* happen. Changes nothing. |
+| `bash update.sh --rollback` | Undo the last update. |
+| `bash update.sh --to v0.3.5` | Go to a specific tag or commit. |
+| `bash update.sh --main` | Track **unreleased** `main`. You're on your own. |
+| `bash update.sh --yes` | Skip the confirmation (for a scripted, *attended* run). |
+
+Run it as **root** — it calls `install.sh`, which needs to reload middlewared.
+
+### First time: bootstrapping `update.sh`
+
+`update.sh` ships *inside* the patch, so a clone older than v0.4.0 doesn't have it
+yet. Bootstrap it once with git:
 
 ```bash
 cd /mnt/tank/truenas-truecloud-patch
-
-# If install.sh was previously run as root, the .git directory may be owned
-# by root. Fix it first, or just pull as root:
-sudo git pull          # easiest option
-# — or —
-sudo chown -R $(whoami) .git && git pull
-
+git pull                 # or: git checkout v0.4.1
 bash install.sh
 ```
 
-`install.sh` clears any stale kill switch, re-applies the updated patches,
-and restarts middlewared. Run `python3 patch/create_task.py verify` afterwards
-to confirm the patches loaded successfully.
+Every update after that is just `bash update.sh`.
 
-Check [CHANGELOG.md](CHANGELOG.md) to see what changed between versions.
+> If a plain `git pull` fails with *"insufficient permission for adding an object
+> to repository database"*, past `sudo git pull`s left root-owned objects in
+> `.git`. Fix it once as root: `chown -R <you>:<you> .git`. (`update.sh` repairs
+> this automatically from then on.)
 
----
+### What it does for you
+
+- **Preserves your nested-snapshot opt-in setting** — updating never flips it.
+- **Shows the commits and release notes you don't have**, then asks before moving.
+- **Records the previous revision *before* checking out**, so `--rollback` works
+  even if `install.sh` dies halfway through.
+- **Refuses to run over a dirty working tree**, rather than merging across
+  hand-edited or scp'd files and losing them.
+- **Detects untracked files that would be clobbered** by the checkout and names
+  them, instead of dying on a raw git error mid-update.
+- **Repairs `.git` ownership** left root-owned by past `sudo git pull`s.
+
+It updates to the newest **release tag**, not `main`. `main` can be mid-refactor;
+a tag is the tested artifact, and CI gates every release. Pre-release tags
+(`-rc`, `-beta`) are skipped — `--to` them explicitly if you want one.
+
+After updating, the checkout is pinned to a release tag (detached HEAD). That is
+what you want for a deployment: plain `git pull` no longer applies, and
+`update.sh` is the supported path.
+
+### Why there is no auto-update
+
+**Never put this in cron or a systemd timer.** This patch injects Python into
+`middlewared` and re-applies itself at *every boot*, so an unattended pull would
+let any bad upstream commit reach your box with no human in the loop — and
+detonate on the next reboot. That is not hypothetical: **v0.0.4 shipped exactly
+such a bug and took all 54 apps on a box down.**
+
+The manual step *is* the safety gate. If you want convenience, watch the
+[releases feed](https://github.com/sudolulo/truenas-truecloud-patch/releases);
+don't automate the pull.
+
+## Update alerts
+
+When a newer release exists, the patch raises a **TrueNAS alert** (the bell in the
+UI) telling you so. It's on by default and checks once a day.
+
+```bash
+bash install.sh --no-update-alerts   # turn it off
+bash install.sh --update-alerts      # turn it back on
+```
+
+**It will not nag you about a README.** A release whose CHANGELOG contains only a
+`### Docs` section changed no code, and raises nothing. Anything that touched the
+system raises an INFO alert; a release with a `### Security` section raises a
+WARNING. The CHANGELOG's own section headings are the signal, and a security fix
+anywhere in the range escalates the whole span — a docs-only release on top of a
+security fix still reports as security.
+
+### How it works, and why it's built this way
+
+TrueNAS **cannot raise an alert from the CLI** — `midclt` exposes only
+`alert.dismiss`, `alert.list`, `alert.list_categories`, `alert.list_policies` and
+`alert.restore`. Alert *creation* is internal to middlewared, and none of its ~60
+one-shot alert classes is generic enough to reuse. So the only way to get a real
+alert is to register an `AlertSource`, which is what `patch/alert_source.py` does.
+
+That is also the **least invasive** thing this patch does:
+
+| | |
+|---|---|
+| providers module | **modifies** stock files (appends code to `b2.py`, `restic.py`) |
+| nested module | **modifies** stock files (3 middleware modules) |
+| **update alert** | **adds one file. Modifies nothing.** |
+
+It's the native mechanism — the same one every built-in TrueNAS alert uses — and
+TrueNAS polls it itself, so there is no cron job and no systemd timer.
+
+- **Fail-safe.** Every error path returns `None`. It cannot take middlewared down.
+- **Read-only.** `git ls-remote` plus an HTTPS fetch of the CHANGELOG. It never
+  writes to `.git`, so it cannot leave root-owned objects behind the way a
+  `git fetch` from middlewared (which runs as root) would.
+- **Removed by `uninstall.sh`.**
+
+It only *tells* you. It never updates anything — see
+[Why there is no auto-update](#why-there-is-no-auto-update).
 
 ## Creating a task via CLI
 
@@ -204,18 +492,25 @@ host address or API key:
 # List your cloud credentials to find the right ID
 python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py list-credentials
 
-# Create a task with a B2 credential (id=3)
+# Create a task with a B2 credential (id=3).
+# The restic repo password is read from stdin, so it never lands in your shell
+# history — nor in any process's argv, where `ps` would expose it.
+printf '%s' 'restic-repo-password' | \
 python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py create \
     --name "tank-to-b2" \
     --path /mnt/tank/data \
     --credential 3 \
     --bucket my-bucket \
     --folder backups/tank \
-    --password "restic-repo-password" \
+    --password-stdin \
     --cache-path /mnt/tank/.restic-cache \
     --keep-last 14
 ```
 
+Omit `--password-stdin` and you'll be prompted for the password instead. `--password
+<secret>` still works but warns: that password is the encryption key for the whole
+repository, and a CLI argument persists in your shell history forever.
+
 > **Always pass `--cache-path`.** Without it TrueNAS runs restic with `--no-cache`,
 > which re-fetches all repo metadata from the provider every run — glacially slow
 > on large repos. Point it at a writable dir on a pool with free space.
@@ -239,25 +534,35 @@ and restores the original UI bundle from backup.
 
 ## If TrueNAS adds native support
 
-`apply.sh` checks at every boot whether TrueNAS has shipped native B2 restic
-support (by inspecting `B2RcloneRemote.__dict__`). If it has:
+The patch is **two independent modules**, and each retires on its own — TrueNAS
+is likely to ship one of these natively long before the other, and a module
+going native must not take the other one down with it.
 
-1. The kill switch (`disabled` file) is set — no patching on any future boot.
-2. Any active overlays are unmounted immediately.
-3. The following message is written to `apply.log`:
+| Module | What it does | Detected as native when |
+|---|---|---|
+| **providers** | B2/S3 credentials for TrueCloud Backup (`b2.py`, `restic.py`, UI dropdown) | `B2RcloneRemote` carries a real `get_restic_config()` |
+| **nested** | Snapshots on datasets with child datasets (`plugins/cloud/*`) | the *"no further nesting"* validation is gone from `plugins/cloud/crud.py` |
 
-```
-NOTICE: TrueNAS now provides native B2 restic support — truecloud-patch is no longer needed.
-NOTICE: Setting kill switch; patching will be skipped on all future boots.
-NOTICE: Run the following to fully remove the patch:
-NOTICE:   bash /mnt/tank/truenas-truecloud-patch/uninstall.sh
-```
+At every boot `apply.sh` checks both:
+
+- **One module goes native** → that module is skipped and logged; the other keeps
+  working, and the patch stays installed.
+- **Both are done** (native, or nested was never enabled) → the kill switch
+  (`disabled` file) is set, overlays are unmounted, and `apply.log` tells you to
+  run `uninstall.sh`.
+
+So on a box using only the provider patch, native B2 support retires the whole
+thing as before. On a box that also uses nested snapshots, native B2 support
+retires *just* that half.
 
 Check the log after any TrueNAS update:
 ```bash
-cat /mnt/tank/truenas-truecloud-patch/apply.log | tail -20
+tail -20 /mnt/tank/truenas-truecloud-patch/apply.log
 ```
 
+`hook_status.json` reports each module separately (`module.providers`,
+`module.nested_snapshots`) with an `active` flag and a reason.
+
 **Scenarios where the auto-detect may not fire** (manual check needed):
 
 | Scenario | What happens | Action |
@@ -270,12 +575,34 @@ cat /mnt/tank/truenas-truecloud-patch/apply.log | tail -20
 
 ## After a TrueNAS update
 
-1. Check the log: `cat /mnt/tank/truenas-truecloud-patch/apply.log | tail -30`
-2. If you see "WARNING: … pattern not found", the UI patch needs updating.
-   [Open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues)
-   with your TrueNAS version number.
-3. The backend patch (B2 support + URL fix) is more stable — check that a
-   B2 backup job still completes successfully after any update.
+A TrueNAS update replaces `/usr/` wholesale, wiping the patch. You do **not** need
+to reinstall: `patch/apply.sh` runs at every boot and re-applies itself from your
+clone. But it targets internal APIs with no stability contract, so an update *can*
+break it — and the failure is quiet by design (middlewared starts fine; the patch
+just doesn't).
+
+**Check the log after any TrueNAS update:**
+
+```bash
+tail -30 /mnt/tank/truenas-truecloud-patch/apply.log
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
+```
+
+| What you see | What it means |
+|---|---|
+| `[OK] providers`, `[OK]`/`[SKIP] nested_snapshots` | Fine. Nothing to do. |
+| `WARNING: … pattern not found` (UI) | The Angular bundle changed. The UI dropdown reverts to Storj-only, but **backups keep working** — create tasks with `create_task.py` meanwhile, and [open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues) with your TrueNAS version. |
+| `[FAIL] providers` | **Your B2/S3 backups will not run.** middlewared is fine, but the credential/URL handling is gone. Open an issue with your version. |
+| `[FAIL] nested_snapshots` | The stock guard is back, so tasks with `snapshot = true` on a nested dataset will fail validation. Turn the option off on those tasks until it's fixed. |
+
+"Fail-safe" means *the box stays up* — not that your backups keep running. A
+`[FAIL] providers` is a broken backup, so check the log rather than assume.
+
+Then update the patch itself if a newer release fixes it:
+
+```bash
+bash /mnt/tank/truenas-truecloud-patch/update.sh
+```
 
 ---
 
@@ -362,19 +689,28 @@ patch re-runs at next reboot, or you run
 python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
 ```
 
-If one or more entries show `[FAIL]`:
+`verify` reports one line per module:
+
+| Label | Meaning |
+|---|---|
+| `[OK  ]` | Module is active and applied. |
+| `[SKIP]` | Module is inactive — either TrueNAS now does it natively, or it is opt-in and switched off. **Not a failure.** `nested_snapshots` shows SKIP on a default install. |
+| `[FAIL]` | Module is needed but did not apply. |
+
+If a module shows `[FAIL]`:
 
 1. **Check the apply log** for errors during the last boot:
    ```bash
-   cat /mnt/tank/truenas-truecloud-patch/apply.log | tail -40
+   tail -40 /mnt/tank/truenas-truecloud-patch/apply.log
    ```
 2. **Check middlewared's own log** for Python tracebacks:
    ```bash
    grep -i "truecloud\|traceback\|error" /var/log/middlewared.log 2>/dev/null | tail -30
    journalctl -u middlewared -n 50
    ```
-3. **A FAIL is non-fatal.** middlewared runs normally; the affected provider
-   falls back to Storj-only. Your existing backups are not at risk.
+3. **A FAIL is non-fatal.** middlewared runs normally and the other module is
+   unaffected; the failed one is simply inactive. Existing backups are not at
+   risk.
 4. **If the detail says the module doesn't exist**, a TrueNAS update renamed
    or restructured the internal API.
    [Open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues)

install.sh

@@ -18,11 +18,56 @@
 
 set -euo pipefail
 
-VERSION="0.0.4"
+VERSION="0.5.1"
 
 # The directory containing install.sh is the permanent install location.
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
+_NESTED_MARKER="$PATCH_DIR/nested_snapshots_enabled"
+
+# ── Options ───────────────────────────────────────────────────────────────────
+# Nested-dataset snapshot support is OPT-IN and off by default. It changes how
+# backups read their source data, so an unattended re-run (e.g. after a
+# `git pull`) must never flip it on or off by itself: with neither flag given,
+# whatever was chosen previously is preserved.
+_nested_choice=""
+_alert_choice=""
+_ALERT_MARKER="$PATCH_DIR/update_alerts_disabled"
+
+usage() {
+    cat <<USAGE
+Usage: bash install.sh [options]
+
+Options:
+  --enable-nested-snapshots   Allow the "Take Snapshot" option on datasets that
+                              have child datasets (every pool running Apps).
+                              Stock TrueNAS refuses this; see README. Off by
+                              default because it changes how backups read data.
+  --disable-nested-snapshots  Turn it back off; the stock guard is restored.
+  --no-update-alerts          Do not raise a TrueNAS alert when an update exists.
+  --update-alerts             Re-enable those alerts (they are on by default).
+  -h, --help                  Show this help.
+
+With neither flag, the current setting is left unchanged.
+USAGE
+}
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --enable-nested-snapshots)  _nested_choice="on" ;;
+        --disable-nested-snapshots) _nested_choice="off" ;;
+        --no-update-alerts)         _alert_choice="off" ;;
+        --update-alerts)            _alert_choice="on" ;;
+        -h|--help)                  usage; exit 0 ;;
+        *)
+            echo "ERROR: unknown option: $1" >&2
+            echo "" >&2
+            usage >&2
+            exit 1
+            ;;
+    esac
+    shift
+done
 
 if [ ! -f "$PATCH_DIR/patch/apply.sh" ]; then
     echo "ERROR: patch files not found at $PATCH_DIR/patch/" >&2
@@ -56,8 +101,14 @@ fi
 # ── Set permissions ───────────────────────────────────────────────────────────
 
 echo "Setting permissions ..."
-chmod +x "$PATCH_DIR/patch/apply.sh" "$PATCH_DIR/patch/create_task.py" \
-          "$PATCH_DIR/recover.sh" "$PATCH_DIR/uninstall.sh"
+# Guard each path: under `set -e` a chmod on a missing file aborts the install.
+# The file set changes between versions, so `update.sh --rollback` to an older
+# revision must not be killed by a name this version happens to know about.
+for _exe in patch/apply.sh patch/create_task.py recover.sh uninstall.sh update.sh; do
+    if [ -f "$PATCH_DIR/$_exe" ]; then
+        chmod +x "$PATCH_DIR/$_exe"
+    fi
+done
 echo "Done."
 echo ""
 
@@ -104,6 +155,77 @@ if [ -f "$PATCH_DIR/disabled" ]; then
     echo ""
 fi
 
+# ── Nested-dataset snapshot support (opt-in) ──────────────────────────────────
+
+case "$_nested_choice" in
+    on)
+        touch "$_NESTED_MARKER"
+        echo "Nested-dataset snapshots: ENABLED"
+        echo "  The \"Take Snapshot\" option will be allowed on datasets that have"
+        echo "  child datasets. Backups then read from a frozen, complete staging"
+        echo "  tree instead of live files."
+        echo ""
+        echo "  This changes how your backups read their source data. Verify that a"
+        echo "  backup completes AND that its restic snapshot actually contains"
+        echo "  child-dataset data before you rely on it."
+        ;;
+    off)
+        if [ -f "$_NESTED_MARKER" ]; then
+            rm -f "$_NESTED_MARKER"
+            # Tear down any staging tree first: those bind mounts PIN their ZFS
+            # snapshots, so leaving them would block those snapshots from ever
+            # being destroyed. apply.sh (below) then reverts the patched files.
+            python3 "$PATCH_DIR/patch/truecloud_nested.py" cleanup || \
+                echo "  WARNING: staging mounts remain; unmount them manually."
+            echo "Nested-dataset snapshots: DISABLED."
+            echo "  apply.sh will revert the patched middleware files and the stock"
+            echo "  guard is restored when middlewared restarts (this script does that)."
+            echo "  Any task that already has snapshot=true on a nested dataset will"
+            echo "  fail validation on its next edit. Turn the option off on those"
+            echo "  tasks first, or re-run with --enable-nested-snapshots."
+        else
+            echo "Nested-dataset snapshots: already disabled."
+        fi
+        ;;
+    *)
+        if [ -f "$_NESTED_MARKER" ]; then
+            echo "Nested-dataset snapshots: enabled (unchanged)."
+        else
+            echo "Nested-dataset snapshots: disabled (default)."
+            echo "  Enable with: bash install.sh --enable-nested-snapshots"
+        fi
+        ;;
+esac
+echo ""
+
+# ── Update alerts (on by default) ─────────────────────────────────────────────
+# TrueNAS cannot raise an alert from the CLI (midclt exposes only dismiss/list/
+# restore), so this installs an AlertSource into middlewared/alert/source/ — the
+# same mechanism every built-in TrueNAS alert uses. It ADDS a file and modifies
+# none, which makes it the least invasive thing this patch does.
+#
+# It only alerts for releases that changed something: a documentation-only release
+# raises nothing.
+
+case "$_alert_choice" in
+    off)
+        touch "$_ALERT_MARKER"
+        echo "Update alerts: DISABLED (apply.sh will remove the alert source)."
+        ;;
+    on)
+        rm -f "$_ALERT_MARKER"
+        echo "Update alerts: enabled."
+        ;;
+    *)
+        if [ -f "$_ALERT_MARKER" ]; then
+            echo "Update alerts: disabled (unchanged)."
+        else
+            echo "Update alerts: enabled (checks daily; docs-only releases are ignored)."
+        fi
+        ;;
+esac
+echo ""
+
 # ── Apply now ─────────────────────────────────────────────────────────────────
 
 echo "Applying patches ..."

patch/alert_source.py

@@ -0,0 +1,226 @@
+"""TrueNAS alert: a truecloud-patch update is available.
+
+Installed by patch/apply.sh into middlewared/alert/source/, where middlewared
+discovers and polls it natively — no cron job, no systemd timer.
+
+@PATCH_DIR@ is substituted at install time.
+
+Two rules govern this file:
+
+1. **It must never break middlewared.** It runs inside the alert framework on a
+   timer. Every failure path returns None (no alert) rather than raising.
+
+2. **It must not nag.** A release whose CHANGELOG only has a "### Docs" section
+   changed no code, and nobody wants an alert because a README was reworded. The
+   CHANGELOG's own section headings are the signal — see tools/release_notes.py.
+
+It also never writes to the repository. `git ls-remote` is read-only and the
+CHANGELOG is fetched over HTTPS, so this cannot leave root-owned objects in .git
+the way a `git fetch` from middlewared (running as root) would.
+"""
+
+import datetime
+import importlib.util
+import logging
+import os
+import re
+import subprocess
+import urllib.request
+
+from middlewared.alert.base import (
+    Alert,
+    AlertCategory,
+    AlertClass,
+    AlertLevel,
+    ThreadedAlertSource,
+)
+from middlewared.alert.schedule import IntervalSchedule
+
+logger = logging.getLogger(__name__)
+
+PATCH_DIR = "@PATCH_DIR@"
+DISABLED_MARKER = os.path.join(PATCH_DIR, "update_alerts_disabled")
+
+_TAG_RE = re.compile(r"^v\d+\.\d+\.\d+$")
+_VERSION_RE = re.compile(r'^VERSION="([^"]+)"', re.M)
+_GITHUB_RE = re.compile(r"github\.com[:/]([^/]+)/([^/.]+)")
+
+_TIMEOUT = 20
+
+
+class TrueCloudPatchUpdateAlertClass(AlertClass):
+    category = AlertCategory.SYSTEM
+    level = AlertLevel.INFO
+    title = "truecloud-patch update available"
+    text = (
+        "truecloud-patch %(current)s is installed; %(latest)s is available.%(summary)s "
+        "Update with:  bash %(dir)s/update.sh"
+    )
+
+
+class TrueCloudPatchSecurityUpdateAlertClass(AlertClass):
+    category = AlertCategory.SYSTEM
+    level = AlertLevel.WARNING
+    title = "truecloud-patch security update available"
+    text = (
+        "truecloud-patch %(current)s is installed; %(latest)s contains a SECURITY "
+        "fix.%(summary)s Update with:  bash %(dir)s/update.sh"
+    )
+
+
+class TrueCloudPatchUpdateAlertSource(ThreadedAlertSource):
+    schedule = IntervalSchedule(datetime.timedelta(hours=24))
+    run_on_backup_node = False
+
+    def check_sync(self):
+        try:
+            return self._check()
+        except Exception:
+            # An alert source must never take middlewared down with it.
+            logger.debug("truecloud-patch update check failed", exc_info=True)
+            return None
+
+    # ── internals ────────────────────────────────────────────────────────────
+
+    def _git(self, *args):
+        return subprocess.run(
+            ["git", "-C", PATCH_DIR, *args],
+            capture_output=True, text=True, timeout=_TIMEOUT, check=True,
+        ).stdout
+
+    def _check(self):
+        if os.path.exists(DISABLED_MARKER):
+            return None
+        if not os.path.isdir(os.path.join(PATCH_DIR, ".git")):
+            return None
+
+        current = self._installed_version()
+        if not current:
+            return None
+
+        latest = self._latest_release_tag()
+        if not latest:
+            return None
+
+        rn = self._release_notes()
+        if rn is None:
+            return None
+        significance, version_tuple = rn.significance, rn.version_tuple
+
+        if version_tuple(latest) <= version_tuple(current):
+            return None
+
+        level, versions, summary = self._classify(
+            current, latest, significance
+        )
+
+        # Documentation-only releases are not worth an alert. This is the whole
+        # point: nobody should get a notification because a README was reworded.
+        if level == "docs":
+            logger.debug(
+                "truecloud-patch %s -> %s is documentation-only; not alerting",
+                current, latest,
+            )
+            return None
+
+        args = {
+            "current": f"v{current}",
+            "latest": latest,
+            "summary": summary,
+            "dir": PATCH_DIR,
+        }
+        klass = (
+            TrueCloudPatchSecurityUpdateAlertClass if level == "security"
+            else TrueCloudPatchUpdateAlertClass
+        )
+        return Alert(klass, args, key=[current, latest])
+
+    def _release_notes(self):
+        """Load tools/release_notes.py by path.
+
+        NOT via sys.path: prepending would shadow the stdlib for this interpreter,
+        and this runs in middlewared's thread pool, so mutating sys.path is a race.
+        """
+        path = os.path.join(PATCH_DIR, "tools", "release_notes.py")
+        try:
+            spec = importlib.util.spec_from_file_location("_tc_release_notes", path)
+            mod = importlib.util.module_from_spec(spec)
+            spec.loader.exec_module(mod)
+            return mod
+        except Exception:
+            logger.debug("could not load release_notes", exc_info=True)
+            return None
+
+    def _installed_version(self):
+        """The version of the patch actually checked out here."""
+        try:
+            with open(os.path.join(PATCH_DIR, "patch", "apply.sh"), encoding="utf-8") as fh:
+                m = _VERSION_RE.search(fh.read())
+        except OSError:
+            return None
+        return m.group(1) if m else None
+
+    def _latest_release_tag(self):
+        """Newest plain vX.Y.Z tag on the remote. Read-only: no .git writes.
+
+        Pre-release tags (-rc, -beta) are excluded: git's version sort ranks
+        v0.5.0-rc1 above v0.5.0, so including them would advertise a release
+        candidate as the latest stable.
+        """
+        try:
+            out = self._git("ls-remote", "--tags", "--refs", "origin")
+        except Exception:
+            return None
+
+        tags = []
+        for line in out.splitlines():
+            parts = line.split("refs/tags/")
+            if len(parts) == 2 and _TAG_RE.match(parts[1].strip()):
+                tags.append(parts[1].strip())
+        if not tags:
+            return None
+
+        return max(tags, key=lambda t: tuple(int(x) for x in t.lstrip("v").split(".")))
+
+    def _classify(self, current, latest, significance):
+        """(level, versions, one-line summary). Falls back to alerting."""
+        text = self._remote_changelog(latest)
+        if text is None:
+            # Cannot tell whether it matters. Alert rather than risk hiding a
+            # security fix -- but say that we could not tell.
+            return "notable", [], " (could not read the changelog)"
+
+        level, versions, headings = significance(text, current, latest)
+        if level == "docs":
+            return level, versions, ""
+
+        seen, ordered = set(), []
+        for h in headings:
+            if h not in seen:
+                seen.add(h)
+                ordered.append(h.capitalize())
+        detail = ", ".join(ordered)
+        return level, versions, f" Changes: {detail}." if detail else ""
+
+    def _remote_changelog(self, tag):
+        """CHANGELOG.md at `tag`, over HTTPS. None if it cannot be read."""
+        try:
+            remote = self._git("remote", "get-url", "origin").strip()
+        except Exception:
+            return None
+
+        m = _GITHUB_RE.search(remote)
+        if not m:
+            return None  # not a GitHub remote; skip classification
+
+        url = (
+            f"https://raw.githubusercontent.com/{m.group(1)}/{m.group(2)}/"
+            f"{tag}/CHANGELOG.md"
+        )
+        try:
+            with urllib.request.urlopen(url, timeout=_TIMEOUT) as resp:  # noqa: S310
+                if resp.status != 200:
+                    return None
+                return resp.read().decode("utf-8", "replace")
+        except Exception:
+            return None

patch/apply.sh

@@ -7,13 +7,21 @@
 # is already up and has already imported the stock modules — the on-disk
 # patch alone cannot reach the running process.
 #
-# TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
+# TrueNAS updates replace /usr/ entirely; this script re-applies three patches:
 #
 #   1. Backend — b2.py and restic.py are patched directly in the overlay.
 #      On a boot run, a single detached middlewared restart is scheduled
 #      (Step 3) so the patched modules actually get loaded.
 #
-#   2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
+#   2. Nested-dataset snapshots — installs _truecloud_nested.py and patches
+#      plugins/cloud/{snapshot,crud}.py + plugins/cloud_backup/sync.py so the
+#      "Take Snapshot" option works on a dataset that has child datasets.
+#      Stock middleware refuses that config, because it points the backup tool
+#      at the PARENT's .zfs/snapshot/ where children are invisible — it would
+#      silently back up a near-empty tree. We stage a complete tree of
+#      per-dataset bind mounts and only then relax the guard.
+#
+#   3. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
 #                          from Storj-only to include S3 and B2. Served from
 #                          disk per request, so no restart is needed for it.
 #
@@ -24,7 +32,7 @@
 # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
 PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 LOG="$PATCH_DIR/apply.log"
-VERSION="0.2.1"
+VERSION="0.5.1"
 
 # Rotate log at 512 KB to avoid unbounded growth on a system volume.
 # Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -101,10 +109,20 @@ echo "Using Python: $PYTHON"
 # Combines what were previously four separate Python invocations into one to
 # avoid repeated interpreter startup overhead under the PREINIT timeout budget.
 
+# The patch has two independent modules, and each retires on its own:
+#
+#   providers  — B2/S3 credentials for TrueCloud Backup (b2.py, restic.py, UI)
+#   nested     — snapshots on datasets that have child datasets (cloud/*.py)
+#
+# TrueNAS may well ship one natively long before the other, so a single
+# all-or-nothing kill switch would silently take a still-needed module down with
+# the superseded one. Each module is detected separately and skipped on its own;
+# the global kill switch fires only once BOTH are native.
+
 _tc_info=$("$PYTHON" -c "
 import inspect, os, sys
 
-result = {'native': 'no', 'site_pkg': '', 'mw_dir': ''}
+result = {'native_b2': 'no', 'native_nested': 'no', 'site_pkg': '', 'mw_dir': ''}
 
 try:
     import middlewared
@@ -118,6 +136,7 @@ except ImportError:
     except Exception:
         pass
 
+# providers: does B2RcloneRemote already carry a real get_restic_config()?
 try:
     import middlewared.rclone.remote.b2 as _b2_mod
     from middlewared.rclone.remote.b2 import B2RcloneRemote
@@ -129,21 +148,79 @@ try:
             except (OSError, TypeError):
                 method_src = ''
             if 'NotImplementedError' not in method_src:
-                result['native'] = 'yes'
+                result['native_b2'] = 'yes'
 except Exception:
     pass
 
-print(result['native'])
+# nested: stock gates nested datasets with a validation in plugins/cloud/crud.py.
+# If that guard is gone, iX removed it, which means they implemented the traversal.
+#
+# Only look at the STOCK part of the file. Our own CRUD_BLOCK quotes the guard
+# message (it filters on it), so scanning the whole file would find the string in
+# our own patch and conclude the guard is still there. That happens to be
+# harmless today because detection runs before patching, but it makes the probe
+# silently order-dependent -- so cut our block off explicitly.
+#
+# If the file cannot be read we assume 'no' and keep patching: worst case the
+# patch declines to apply and the option simply stays unavailable.
+try:
+    crud = os.path.join(result['mw_dir'], 'plugins', 'cloud', 'crud.py')
+    with open(crud, encoding='utf-8', errors='replace') as fh:
+        stock_src = fh.read().split('\n# TRUECLOUD_PATCH', 1)[0]
+    # The guard message is SPLIT across adjacent string literals in the source:
+    #
+    #     verrors.add(..., 'This option is only available for datasets that have no further '
+    #                      'nesting')
+    #
+    # Python concatenates those at runtime, so the errmsg is contiguous -- but the
+    # SOURCE never contains the whole phrase. A raw search finds nothing, concludes
+    # iX removed the guard, and silently skips this module FOREVER. (Caught only by
+    # running the probe against real middlewared.)
+    #
+    # Strip whitespace and quote characters, then match the compacted phrase. That
+    # is robust to any wrapping or concatenation style iX may use.
+    _drop = str.maketrans('', '', ' \\t\\n\\r' + chr(34) + chr(39))
+    if 'nofurthernesting' not in stock_src.translate(_drop):
+        result['native_nested'] = 'yes'
+except Exception:
+    pass
+
+print(result['native_b2'])
+print(result['native_nested'])
 print(result['site_pkg'])
 print(result['mw_dir'])
-" 2>/dev/null || printf 'no\n\n\n')
+" 2>/dev/null || printf 'no\nno\n\n\n')
 
-_tc_native=$(printf '%s' "$_tc_info" | sed -n '1p')
-SITE_PKG=$(printf '%s' "$_tc_info"  | sed -n '2p')
-_MW_DIR=$(printf '%s' "$_tc_info"   | sed -n '3p')
+_tc_native_b2=$(printf '%s' "$_tc_info"     | sed -n '1p')
+_tc_native_nested=$(printf '%s' "$_tc_info" | sed -n '2p')
+SITE_PKG=$(printf '%s' "$_tc_info"          | sed -n '3p')
+_MW_DIR=$(printf '%s' "$_tc_info"           | sed -n '4p')
 
-if [ "$_tc_native" = "yes" ]; then
-    echo "NOTICE: TrueNAS now provides native B2 restic support — truecloud-patch is no longer needed."
+# Nested support is opt-in; if it was never enabled, it cannot be the reason to
+# keep the patch alive.
+if [ -f "$PATCH_DIR/nested_snapshots_enabled" ]; then
+    _NESTED_ENABLED=1
+else
+    _NESTED_ENABLED=0
+fi
+
+# Is either module still doing something useful?
+_providers_needed=1
+[ "$_tc_native_b2" = "yes" ] && _providers_needed=0
+
+_nested_needed=0
+if [ "$_NESTED_ENABLED" = "1" ] && [ "$_tc_native_nested" != "yes" ]; then
+    _nested_needed=1
+fi
+
+if [ "$_providers_needed" = "0" ] && [ "$_nested_needed" = "0" ]; then
+    echo "NOTICE: Nothing left for truecloud-patch to do:"
+    [ "$_tc_native_b2" = "yes" ] && echo "NOTICE:   - TrueNAS now provides native B2 restic support."
+    if [ "$_tc_native_nested" = "yes" ]; then
+        echo "NOTICE:   - TrueNAS now handles snapshots on nested datasets natively."
+    elif [ "$_NESTED_ENABLED" = "0" ]; then
+        echo "NOTICE:   - Nested-dataset snapshots are not enabled (opt-in)."
+    fi
     echo "NOTICE: Setting kill switch; patching will be skipped on all future boots."
     echo "NOTICE: Run the following to fully remove the patch:"
     echo "NOTICE:   bash $PATCH_DIR/uninstall.sh"
@@ -158,12 +235,21 @@ if [ "$_tc_native" = "yes" ]; then
     exit 0
 fi
 
+if [ "$_providers_needed" = "0" ]; then
+    echo "NOTICE: TrueNAS now provides native B2 restic support — the providers module"
+    echo "NOTICE: is superseded and will be skipped. The nested-snapshot module is still"
+    echo "NOTICE: active, so the patch stays installed."
+fi
+if [ "$_NESTED_ENABLED" = "1" ] && [ "$_tc_native_nested" = "yes" ]; then
+    echo "NOTICE: TrueNAS now handles nested-dataset snapshots natively — that module is"
+    echo "NOTICE: superseded and will be skipped. You can drop --enable-nested-snapshots."
+fi
+
 # ── Step 1: backend patch ─────────────────────────────────────────────────────
 
 echo "--- backend patch ---"
 
-_b2_ok=0
-_restic_ok=0
+_backend_ok=0
 
 if [ -z "$SITE_PKG" ]; then
     echo "WARNING: Cannot determine site-packages directory; skipping backend patch."
@@ -175,12 +261,27 @@ elif [ -z "$_MW_DIR" ]; then
 else
     _B2_PY="$_MW_DIR/rclone/remote/b2.py"
     _RESTIC_PY="$_MW_DIR/plugins/cloud_backup/restic.py"
+    _CLOUD_DIR="$_MW_DIR/plugins/cloud"
+    _SYNC_PY="$_MW_DIR/plugins/cloud_backup/sync.py"
+    _NESTED_SRC="$PATCH_DIR/patch/truecloud_nested.py"
 
-        # ── patch b2.py + restic.py + hook_status.json (single subprocess) ──────
-        if "$PYTHON" - "$_B2_PY" "$_RESTIC_PY" "$PATCH_DIR/hook_status.json" << 'PYEOF'
-import json, os, sys, time
+    # Each module is applied only if it is still needed. _providers_needed and
+    # _nested_needed were computed above (native-support detection + the opt-in
+    # marker), so one module going native never disables the other.
+        # ── patch b2.py + restic.py + nested-snapshot + hook_status.json ────────
+        # (single subprocess: PREINIT has a tight timeout budget)
+        if "$PYTHON" - "$_B2_PY" "$_RESTIC_PY" "$PATCH_DIR/hook_status.json" \
+                      "$_CLOUD_DIR" "$_SYNC_PY" "$_NESTED_SRC" \
+                      "$_providers_needed" "$_nested_needed" "$_NESTED_ENABLED" \
+                      "$_tc_native_nested" << 'PYEOF'
+import json, os, shutil, sys, time
 
 b2_path, restic_path, status_path = sys.argv[1], sys.argv[2], sys.argv[3]
+cloud_dir, sync_path, nested_src = sys.argv[4], sys.argv[5], sys.argv[6]
+providers_needed = sys.argv[7] == "1"
+nested_needed = sys.argv[8] == "1"
+nested_enabled = sys.argv[9] == "1"
+nested_native = sys.argv[10] == "yes"
 
 B2_BLOCK = """
 # TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
@@ -240,45 +341,303 @@ else:
     get_restic_config._truecloud_patched = True
 """
 
-def patch_file(path, block):
-    with open(path, encoding="utf-8") as fh:
-        content = fh.read()
-    marker = "\n# TRUECLOUD_PATCH"
-    idx = content.find(marker)
-    base = content[:idx] if idx != -1 else content
-    with open(path, "w", encoding="utf-8") as fh:
-        fh.write(base.rstrip("\n") + "\n" + block)
+# ── nested-dataset snapshot support ───────────────────────────────────────────
+# Stock middleware refuses `snapshot=true` on a path containing child datasets,
+# because it points the backup tool at the PARENT dataset's .zfs/snapshot/,
+# where child datasets are INVISIBLE -- it would silently back up a near-empty
+# tree. That guard is correct. We implement the missing traversal (a staging
+# tree of per-dataset bind mounts) and only then relax the guard.
+#
+# Fail-safe direction: if any of these three blocks fails to apply, the stock
+# guard remains and the option simply stays unavailable. We never end up with
+# the guard removed but the traversal missing -- that would be a silently empty
+# backup, the worst possible outcome.
+
+SNAPSHOT_BLOCK = """
+# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
+try:
+    from middlewared.plugins.cloud import _truecloud_nested as _tc_nested
+except ImportError:
+    _tc_nested = None
+
+if _tc_nested is not None:
+    _tc_orig_create_snapshot = create_snapshot
+
+    async def create_snapshot(middleware, path, name="cloud_task-onetime"):
+        # Stock takes the (already recursive) snapshot; we only replace the PATH.
+        snapshot, snap_path = await _tc_orig_create_snapshot(middleware, path, name)
+
+        _logger = getattr(middleware, "logger", None)
+        try:
+            # Enumerate datasets AFTER the snapshot, never before. The snapshot is
+            # the point-in-time truth; a list read beforehand could miss a dataset
+            # created in the gap, which the recursive snapshot WOULD capture but
+            # our staging plan would not -- silently omitting it from the backup.
+            # Read afterwards, an unsnapshotted dataset instead trips the isdir()
+            # check in plan_staging and fails the run loudly. Loud beats silent.
+            datasets = await middleware.call(
+                "zfs.dataset.query", [["type", "=", "FILESYSTEM"]]
+            )
+            dataset, nested = get_dataset_recursive(datasets, path)
+
+            if not nested:
+                # No children: stock behaviour, untouched. Stock's `finally` owns
+                # the snapshot from here (its non-recursive delete is correct,
+                # because a non-nested snapshot has no children).
+                return snapshot, snap_path
+
+            staging_root = await _tc_nested.stage_nested(
+                middleware, path, snapshot,
+                dataset["name"], dataset["properties"]["mountpoint"]["value"],
+                name, datasets, logger=_logger,
+            )
+        except Exception:
+            # The snapshot exists, but this exception means sync.py never completes
+            # `snapshot, local_path = await create_snapshot(...)`, so its local
+            # `snapshot` stays None and its `finally` deletes NOTHING. Sweep the
+            # tree ourselves or leak the parent plus one snapshot per descendant
+            # dataset (160+ here) on every failed run.
+            await _tc_nested.delete_snapshot_tree(middleware, snapshot, logger=_logger)
+            raise
+
+        return snapshot, staging_root
+
+    create_snapshot._truecloud_patched = True
+"""
+
+CRUD_BLOCK = """
+# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
+try:
+    from middlewared.plugins.cloud import _truecloud_nested as _tc_nested
+except ImportError:
+    _tc_nested = None
+
+if _tc_nested is not None:
+    _tc_orig_validate = CloudTaskServiceMixin._validate
+
+    async def _tc_validate(self, app, verrors, name, data):
+        await _tc_orig_validate(self, app, verrors, name, data)
+
+        # Only cloud_backup: staging teardown is wired into cloud_backup.sync's
+        # finally. cloudsync would leak bind mounts, so leave its guard intact.
+        if getattr(getattr(self, "_config", None), "namespace", "") != "cloud_backup":
+            return
+
+        # Drop ONLY the nested-dataset guard. If iX ever rewords the message the
+        # filter stops matching, the guard survives, and the option merely stays
+        # unavailable -- the safe direction to fail.
+        verrors.errors = [
+            e for e in verrors.errors
+            if not (
+                getattr(e, "attribute", "") == f"{name}.snapshot"
+                and "no further nesting" in getattr(e, "errmsg", "")
+            )
+        ]
+
+    CloudTaskServiceMixin._validate = _tc_validate
+    CloudTaskServiceMixin._validate._truecloud_patched = True
+"""
+
+SYNC_BLOCK = """
+# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
+try:
+    from middlewared.plugins.cloud import _truecloud_nested as _tc_nested
+except ImportError:
+    _tc_nested = None
+
+if _tc_nested is not None:
+    _tc_orig_restic_backup = restic_backup
+
+    async def restic_backup(middleware, job, cloud_backup, dry_run=False, rate_limit=None):
+        # Our bind mounts pin the ZFS snapshot, so stock's `finally` cannot
+        # destroy it (EBUSY) and logs one benign warning. We unmount here and
+        # then delete the snapshot for real.
+        try:
+            return await _tc_orig_restic_backup(middleware, job, cloud_backup, dry_run, rate_limit)
+        finally:
+            try:
+                await _tc_nested.cleanup_task(
+                    middleware,
+                    f"cloud_backup-{cloud_backup.get('id', 'onetime')}",
+                    logger=getattr(middleware, "logger", None),
+                )
+            except Exception as e:
+                middleware.logger.warning("truecloud-patch: staging cleanup failed: %r", e)
+
+    restic_backup._truecloud_patched = True
+"""
+
+
+# Single implementation of the block apply/revert logic (patch/mw_patch.py), so
+# uninstall.sh and apply.sh cannot drift apart. Fail-safe: if it cannot be
+# imported, skip the backend patch entirely -- middlewared then starts stock,
+# which is the whole design principle of this script.
+# APPEND, never insert(0): this dir would otherwise take precedence over the
+# stdlib for this interpreter, so a future patch/json.py (say) would shadow the
+# real json module and break the boot. Appending fails safe -- worst case our
+# import misses and the backend patch is skipped.
+sys.path.append(os.path.dirname(nested_src))
+try:
+    from mw_patch import patch_file, revert_nested
+except ImportError as _e:
+    print(f'WARNING: cannot import patch/mw_patch.py ({_e}) — skipping backend patch.')
+    print('WARNING: middlewared will start with stock (unpatched) modules.')
+    sys.exit(1)
+
+# .../middlewared/plugins/cloud -> .../middlewared
+mw_dir = os.path.dirname(os.path.dirname(cloud_dir))
 
 b2_ok = restic_ok = False
+nested_ok = False
 
-if os.path.exists(b2_path):
-    try:
-        patch_file(b2_path, B2_BLOCK)
-        b2_ok = True
-        print(f"OK: Patched b2.py → {b2_path}")
-    except Exception as e:
-        print(f"WARNING: Failed to patch b2.py: {e}")
+# ── module: providers (B2/S3) ─────────────────────────────────────────────────
+# Skipped entirely once TrueNAS ships native B2 restic support. That must not
+# take the nested module down with it, so the two are gated independently.
+providers_detail = ''
+if not providers_needed:
+    providers_detail = 'superseded: TrueNAS provides native B2 restic support'
+    print('INFO: Providers module skipped — TrueNAS now supports B2 natively.')
 else:
-    print(f"WARNING: b2.py not found at {b2_path}")
+    if os.path.exists(b2_path):
+        try:
+            patch_file(b2_path, B2_BLOCK)
+            b2_ok = True
+            print(f"OK: Patched b2.py → {b2_path}")
+        except Exception as e:
+            print(f"WARNING: Failed to patch b2.py: {e}")
+    else:
+        print(f"WARNING: b2.py not found at {b2_path}")
 
-if os.path.exists(restic_path):
-    try:
-        patch_file(restic_path, RESTIC_BLOCK)
-        restic_ok = True
-        print(f"OK: Patched restic.py → {restic_path}")
-    except Exception as e:
-        print(f"WARNING: Failed to patch restic.py: {e}")
+    if os.path.exists(restic_path):
+        try:
+            patch_file(restic_path, RESTIC_BLOCK)
+            restic_ok = True
+            print(f"OK: Patched restic.py → {restic_path}")
+        except Exception as e:
+            print(f"WARNING: Failed to patch restic.py: {e}")
+    else:
+        print(f"WARNING: restic.py not found at {restic_path}")
+    providers_detail = (
+        'patched on disk in overlay at boot' if (b2_ok and restic_ok)
+        else 'b2.py/restic.py not found or write failed'
+    )
+
+# ── module: nested-dataset snapshots ──────────────────────────────────────────
+# Order matters: install the traversal machinery FIRST, relax the validation
+# guard LAST. If anything fails partway, the guard is still in place and the
+# option stays unavailable -- we never expose "guard removed, traversal missing".
+nested_detail = ''
+if not nested_needed:
+    # Not just "skip": actively revert. The overlay lives for the whole boot, so a
+    # previously-applied patch is still sitting there and middlewared would
+    # re-import it on restart. See revert_nested().
+    if not nested_enabled:
+        nested_detail = 'disabled (opt-in; enable with: install.sh --enable-nested-snapshots)'
+        print('INFO: Nested-dataset snapshot support is disabled (opt-in feature).')
+    elif nested_native:
+        nested_detail = 'superseded: TrueNAS handles nested-dataset snapshots natively'
+        print('INFO: Nested module skipped — TrueNAS now handles nesting natively.')
+    else:
+        nested_detail = 'not needed'
+        print('INFO: Nested module skipped.')
+
+    reverted = revert_nested(mw_dir)
+    if reverted:
+        print('OK: Reverted a previously-applied nested patch (' + ', '.join(reverted) + ').')
+        print('    The stock nesting guard is restored once middlewared restarts.')
+        nested_detail += ' — previous patch reverted'
+
+    if not nested_enabled:
+        print('INFO: Enable with: bash install.sh --enable-nested-snapshots')
 else:
-    print(f"WARNING: restic.py not found at {restic_path}")
+    try:
+        snapshot_py = os.path.join(cloud_dir, 'snapshot.py')
+        crud_py = os.path.join(cloud_dir, 'crud.py')
+        nested_dst = os.path.join(cloud_dir, '_truecloud_nested.py')
+
+        missing = [p for p in (snapshot_py, crud_py, sync_path, nested_src) if not os.path.exists(p)]
+        if missing:
+            raise FileNotFoundError('missing: ' + ', '.join(missing))
+
+        shutil.copyfile(nested_src, nested_dst)   # 1. traversal implementation
+        patch_file(snapshot_py, SNAPSHOT_BLOCK)   # 2. build the staging tree
+        patch_file(sync_path, SYNC_BLOCK)         # 3. tear it down afterwards
+        patch_file(crud_py, CRUD_BLOCK)           # 4. ONLY NOW allow nested tasks
+
+        nested_ok = True
+        nested_detail = 'nested-dataset snapshots enabled (staging tree)'
+        print(f'OK: Installed nested-snapshot support → {nested_dst}')
+        print(f'OK: Patched snapshot.py, sync.py, crud.py → {cloud_dir}')
+    except Exception as e:
+        nested_detail = f'not applied: {e}'
+        print(f'WARNING: Failed to apply nested-snapshot patch: {e}')
+        print('WARNING: Stock nesting guard remains; snapshot option stays unavailable')
+        print('WARNING: for nested datasets. Existing backups are unaffected.')
+
+# One entry per MODULE, not per file. `ok` means "nothing is wrong", so a module
+# that is inactive (superseded, or opt-in and off) is ok -- reporting a disabled
+# opt-in feature as FAIL would make `create_task.py verify` fail on a default
+# install. `active` says whether the module is doing anything.
+# ── update-available alert ────────────────────────────────────────────────────
+# Dropped into middlewared/alert/source/, where middlewared discovers and polls it
+# natively — no cron, no timer. It only raises an alert for releases that actually
+# changed something: a docs-only release is ignored (see tools/release_notes.py).
+alert_ok = False
+alert_detail = ''
+_patch_dir = os.path.dirname(os.path.dirname(nested_src))
+alert_src = os.path.join(os.path.dirname(nested_src), 'alert_source.py')
+alert_dst = os.path.join(mw_dir, 'alert', 'source', 'truecloud_patch_update.py')
+
+if os.path.exists(os.path.join(_patch_dir, 'update_alerts_disabled')):
+    alert_detail = 'disabled (update_alerts_disabled)'
+    try:
+        os.unlink(alert_dst)
+        print('OK: Removed update alert (disabled).')
+    except OSError:
+        pass
+elif not os.path.exists(alert_src):
+    alert_detail = 'alert_source.py not found'
+    print(f'WARNING: {alert_src} missing — no update alert.')
+else:
+    try:
+        with open(alert_src, encoding='utf-8') as fh:
+            _body = fh.read()
+
+        # repr() so ANY path becomes a valid Python literal -- a directory
+        # containing a quote or backslash would otherwise produce a syntax error.
+        _body = _body.replace('"@PATCH_DIR@"', repr(_patch_dir))
+
+        # COMPILE BEFORE WRITING. middlewared's alert.load() imports every file in
+        # alert/source/ with NO try/except, and it runs at startup -- a module that
+        # raises on import takes middlewared's setup down with it. An uninstalled
+        # alert is a missing convenience; a broken one is a broken box.
+        compile(_body, alert_dst, 'exec')
+
+        with open(alert_dst, 'w', encoding='utf-8') as fh:
+            fh.write(_body)
+        alert_ok = True
+        alert_detail = 'update alert installed'
+        print(f'OK: Installed update alert → {alert_dst}')
+    except Exception as e:
+        alert_detail = f'not applied: {e}'
+        print(f'WARNING: could not install update alert: {e}')
+        print('WARNING: no update alert; everything else is unaffected.')
 
 patches = {
-    'middlewared.rclone.remote.b2': {
-        'ok': b2_ok,
-        'detail': 'patched on disk in overlay at boot' if b2_ok else 'b2.py not found or write failed',
+    'providers': {
+        'ok': (not providers_needed) or bool(b2_ok and restic_ok),
+        'active': providers_needed,
+        'detail': providers_detail,
     },
-    'middlewared.plugins.cloud_backup.restic': {
-        'ok': restic_ok,
-        'detail': 'patched on disk in overlay at boot' if restic_ok else 'restic.py not found or write failed',
+    'nested_snapshots': {
+        'ok': (not nested_needed) or nested_ok,
+        'active': nested_needed,
+        'detail': nested_detail,
+    },
+    'update_alert': {
+        'ok': True,          # never a failure: it is a convenience, not a patch
+        'active': alert_ok,
+        'detail': alert_detail,
     },
 }
 payload = {'patched_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), 'patches': patches}
@@ -291,35 +650,61 @@ try:
 except OSError as e:
     print(f'WARNING: Could not write hook_status.json: {e}')
 
-sys.exit(0 if (b2_ok and restic_ok) else 1)
+# Exit code tells the caller whether a middlewared restart is still worth doing:
+#
+#   0  every module that was needed applied cleanly
+#   2  PARTIAL -- one module failed but another landed, so there IS something new
+#      on disk waiting to be loaded
+#   1  nothing landed; a restart would accomplish nothing
+#
+# Collapsing 2 into 1 would mean a failing providers patch suppresses the restart
+# that a freshly-applied nested patch needs, leaving it on disk and never loaded.
+_providers_done = (not providers_needed) or bool(b2_ok and restic_ok)
+_nested_done = (not nested_needed) or nested_ok
+_landed = (providers_needed and b2_ok and restic_ok) or (nested_needed and nested_ok)
+
+if _providers_done and _nested_done:
+    sys.exit(0)
+sys.exit(2 if _landed else 1)
 PYEOF
         then
-            _b2_ok=1
-            _restic_ok=1
+            _backend_ok=1
         else
-            # Individual results already printed above; exit code 1 means at least one failed.
-            true
+            _rc=$?
+            if [ "$_rc" = "2" ]; then
+                # One module failed, but another was applied and still needs loading.
+                _backend_ok=1
+                echo "WARNING: a module failed to apply; the other landed and will be loaded."
+            else
+                _backend_ok=0
+            fi
         fi
 fi
 
 # ── Step 2: Angular bundle ────────────────────────────────────────────────────
+# Belongs to the providers module (it widens the credential dropdown), so it is
+# skipped along with it once TrueNAS supports B2 natively.
 
 echo "--- UI patch ---"
 
-# Ensure the webui directory is writable before patch_ui.py tries to create a
-# backup and write the patched bundle.  On immutable OS we mount an overlay.
-_webui_dir=""
-for _d in /usr/share/truenas/webui /usr/share/truenas-ui /var/www/truenas; do
-    if [ -d "$_d" ]; then
-        _webui_dir="$_d"
-        break
+if [ "$_providers_needed" = "0" ]; then
+    echo "Skipped — providers module superseded by native B2 support."
+else
+    # Ensure the webui directory is writable before patch_ui.py tries to create a
+    # backup and write the patched bundle.  On immutable OS we mount an overlay.
+    _webui_dir=""
+    for _d in /usr/share/truenas/webui /usr/share/truenas-ui /var/www/truenas; do
+        if [ -d "$_d" ]; then
+            _webui_dir="$_d"
+            break
+        fi
+    done
+    if [ -n "$_webui_dir" ]; then
+        _ensure_writable "$_webui_dir" "ui" || true   # non-fatal; patch_ui.py reports the error
     fi
-done
-if [ -n "$_webui_dir" ]; then
-    _ensure_writable "$_webui_dir" "ui" || true   # non-fatal; patch_ui.py reports the error
-fi
 
-"$PYTHON" "$PATCH_DIR/patch/patch_ui.py" || echo "WARNING: patch_ui.py exited non-zero; UI dropdown may still show Storj only."
+    "$PYTHON" "$PATCH_DIR/patch/patch_ui.py" || echo "WARNING: patch_ui.py exited non-zero; UI dropdown may still show Storj only."
+fi
 
 # ── Step 3: deferred middlewared restart (boot runs only) ─────────────────────
 # At boot this script is spawned by middlewared, which already imported the
@@ -340,10 +725,17 @@ fi
 
 echo "--- deferred restart ---"
 
+# Restart when ANY still-needed backend module landed (_backend_ok, incl. the
+# partial case). Keying this off the providers module alone would skip the restart
+# on a box where B2 has gone native but the nested module was freshly patched —
+# leaving it on disk and never loaded.
+#
+# "No module active at all" cannot reach here: that is the kill-switch branch
+# above, which exits.
 if ! grep -aq middlewared "/proc/$PPID/cmdline" 2>/dev/null; then
     echo "Manual run (parent is not middlewared) — no restart scheduled."
-elif [ "$_b2_ok" != "1" ] || [ "$_restic_ok" != "1" ]; then
-    echo "Backend patch incomplete — no restart scheduled (nothing new to load)."
+elif [ "$_backend_ok" != "1" ]; then
+    echo "Nothing landed on disk — no restart scheduled (nothing new to load)."
 else
     # A failed unit from an earlier attempt this boot would block systemd-run.
     systemctl reset-failed truecloud-mw-restart.service 2>/dev/null

patch/create_task.py

@@ -26,8 +26,9 @@ Create a task backed by a B2 credential (id=3):
         --credential 3 \\
         --bucket my-bucket \\
         --folder backups/tank \\
-        --password "restic-repo-password" \\
+        --password-stdin \\
         --keep-last 14
+    (pipe the password in:  echo -n "s3cret" | python3 create_task.py create ... )
 
 Create a task using an S3-compatible credential (Wasabi, R2, etc.):
     python3 create_task.py create \\
@@ -36,7 +37,7 @@ Create a task using an S3-compatible credential (Wasabi, R2, etc.):
         --credential 5 \\
         --bucket my-bucket \\
         --folder backups \\
-        --password "restic-repo-password"
+        --password-stdin
 
 List existing TrueCloud Backup tasks:
     python3 create_task.py list-tasks
@@ -44,39 +45,49 @@ List existing TrueCloud Backup tasks:
 
 import argparse
 import calendar
+import getpass
 import json
 import os
 import subprocess
 import sys
 import time
 
-__version__ = "0.2.0"
+__version__ = "0.5.1"
 
 _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")
 
 
 def midclt_call(method, *args):
-    """Call a middleware method locally via `midclt`, the supported JSON-RPC transport
-    that replaces the deprecated /api/v2.0 REST API (removed in TrueNAS 26.04). Must run
-    on the TrueNAS host. Each arg is JSON-encoded (a dict for create; none for queries).
-    Exits with a clear message on failure."""
-    cmd = ["midclt", "call", method] + [json.dumps(a) for a in args]
+    """Call a middleware method on the local host.
+
+    Uses `truenas_api_client` -- the library that backs `midclt` itself -- rather
+    than shelling out to `midclt`.
+
+    This is a SECURITY requirement, not a style choice. `midclt call <method>
+    <json>` puts its arguments in the process's **argv**, and `cloud_backup.create`
+    carries the restic repository password. argv is world-readable via `ps`, so
+    shelling out would expose the key to the entire backup repo to every local
+    user for the duration of the call. Going through the client library keeps it
+    in this process's memory.
+    """
     try:
-        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
-    except FileNotFoundError:
-        print("ERROR: `midclt` not found — run this script ON the TrueNAS host.",
-              file=sys.stderr)
+        from truenas_api_client import Client
+    except ImportError:
+        print(
+            "ERROR: `truenas_api_client` not importable — run this script ON the\n"
+            "       TrueNAS host. (It ships with midclt.)",
+            file=sys.stderr,
+        )
         sys.exit(1)
-    except subprocess.SubprocessError as exc:
-        print(f"ERROR: midclt call failed: {exc}", file=sys.stderr)
+
+    try:
+        with Client() as client:
+            return client.call(method, *args)
+    except Exception as exc:  # noqa: BLE001 - surface any middleware error verbatim
+        # Never echo `args` here: for cloud_backup.create it contains the password.
+        print(f"ERROR: {method}: {exc}", file=sys.stderr)
         sys.exit(1)
-    if proc.returncode != 0:
-        print(f"ERROR: midclt {method}: {(proc.stderr or proc.stdout).strip()}",
-              file=sys.stderr)
-        sys.exit(1)
-    out = proc.stdout.strip()
-    return json.loads(out) if out else None
 
 
 # ── Sub-commands ──────────────────────────────────────────────────────────────
@@ -84,8 +95,11 @@ def midclt_call(method, *args):
 def _middlewared_start_epoch():
     """Epoch timestamp of the running middlewared main process, or None."""
     try:
+        # Partial path (S607) is fine here: this runs as root on TrueNAS, so an
+        # attacker who can poison PATH already has root. Hard-coding a path would
+        # be less portable (/bin vs /usr/bin) for no security gain.
         pid = int(subprocess.run(
-            ["systemctl", "show", "--property=MainPID", "--value", "middlewared"],
+            ["systemctl", "show", "--property=MainPID", "--value", "middlewared"],  # noqa: S607
             capture_output=True, text=True, timeout=10, check=True,
         ).stdout.strip())
         if pid <= 0:
@@ -127,13 +141,21 @@ def cmd_verify():
     print(f"Hook status (recorded at {status.get('patched_at', 'unknown')})")
     print()
     all_ok = True
+    any_active = False
     for module, info in status.get("patches", {}).items():
         ok = info.get("ok", False)
+        # A module can be inactive because TrueNAS now does it natively, or
+        # because it is opt-in and switched off. Neither is a failure.
+        active = info.get("active", True)
         label = "OK  " if ok else "FAIL"
+        if ok and not active:
+            label = "SKIP"
         detail = f"  — {info['detail']}" if info.get("detail") else ""
         print(f"  [{label}] {module}{detail}")
         if not ok:
             all_ok = False
+        if active:
+            any_active = True
 
     # The disk status alone can false-positive: at boot the files are patched
     # while middlewared is already running with the stock modules imported.
@@ -146,7 +168,11 @@ def cmd_verify():
     mw_start = _middlewared_start_epoch()
 
     proc_stale = False
-    if patched_epoch is None or mw_start is None:
+    if not any_active:
+        # Nothing is patched into middlewared, so whether it restarted since is
+        # irrelevant -- there is nothing for it to have loaded.
+        print("  [--  ] running middlewared process — no active module; nothing to load")
+    elif patched_epoch is None or mw_start is None:
         print("  [??  ] running middlewared process — could not compare start time;")
         print("         the results above reflect the on-disk state only")
     elif mw_start + 2 < patched_epoch:
@@ -201,6 +227,37 @@ def cmd_list_tasks(_args):
         print(f"{t['id']:>4}  {enabled:<8}  {ptype:<14}  {t.get('description', '')}")
 
 
+def _resolve_password(args):
+    """Get the restic repo password without writing it to the user's shell history.
+
+    That password is the key to the whole backup repository. `--password <secret>`
+    persists it in ~/.bash_history and exposes it in `ps` for the lifetime of the
+    shell command, so it is accepted but warned about; stdin and an interactive
+    prompt are the safe paths.
+    """
+    if args.password_stdin:
+        if args.password:
+            print("ERROR: use either --password or --password-stdin, not both.",
+                  file=sys.stderr)
+            sys.exit(1)
+        password = sys.stdin.readline().rstrip("\n")
+    elif args.password:
+        print(
+            "WARNING: --password puts the restic repository password in your shell\n"
+            "         history. Prefer:  echo -n 'pw' | ... --password-stdin",
+            file=sys.stderr,
+        )
+        password = args.password
+    else:
+        password = getpass.getpass("Restic repository password: ")
+
+    if not password:
+        print("ERROR: the restic repository password must not be empty.",
+              file=sys.stderr)
+        sys.exit(1)
+    return password
+
+
 def cmd_create(args):
     parts = args.schedule.split()
     if len(parts) != 5:
@@ -211,6 +268,8 @@ def cmd_create(args):
         sys.exit(1)
     minute, hour, dom, month, dow = parts
 
+    password = _resolve_password(args)
+
     body = {
         "description": args.name,
         "path": args.path,
@@ -219,7 +278,7 @@ def cmd_create(args):
             "bucket": args.bucket,
             "folder": args.folder,
         },
-        "password": args.password,
+        "password": password,
         "keep_last": args.keep_last,
         "transfer_setting": args.transfer_setting,
         "schedule": {
@@ -285,8 +344,11 @@ def main():
                    help="Bucket (S3) or container (B2) name")
     c.add_argument("--folder",      default="",
                    help="Path within the bucket (default: root)")
-    c.add_argument("--password",    required=True,
-                   help="Restic repository encryption password (choose a strong one)")
+    c.add_argument("--password",    default=None,
+                   help="Restic repository password. UNSAFE: it lands in your shell "
+                        "history. Prefer --password-stdin, or omit both and be prompted.")
+    c.add_argument("--password-stdin", action="store_true",
+                   help="Read the restic repository password from stdin (recommended)")
     c.add_argument("--keep-last",   type=int, default=14, metavar="N",
                    help="Snapshots to retain after each run (default: 14)")
     c.add_argument("--schedule",    default="0 2 * * *",

patch/mw_patch.py

@@ -0,0 +1,147 @@
+#!/usr/bin/env python3
+"""Apply and revert truecloud-patch's blocks in middlewared's modules.
+
+Every patch this project makes to a middlewared module is an appended block that
+begins with the MARKER line. That makes patching idempotent (truncate at the
+marker, re-append) and reverting exact (truncate at the marker, stop).
+
+This is the single implementation of that. It used to live in two places --
+apply.sh's heredoc and an inline heredoc in uninstall.sh -- and the uninstall copy
+was the untested one.
+
+    python3 mw_patch.py revert-all       # remove every block + the nested module
+    python3 mw_patch.py revert-nested    # remove only the nested module's blocks
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+
+MARKER = "\n# TRUECLOUD_PATCH"
+
+#: Modules the providers module (B2/S3) patches.
+PROVIDER_RELPATHS = [
+    ("rclone", "remote", "b2.py"),
+    ("plugins", "cloud_backup", "restic.py"),
+]
+
+#: Modules the nested-snapshot module patches. Order matters on revert -- see
+#: revert(): the loadable module goes first.
+NESTED_RELPATHS = [
+    ("plugins", "cloud", "crud.py"),
+    ("plugins", "cloud_backup", "sync.py"),
+    ("plugins", "cloud", "snapshot.py"),
+]
+
+#: The importable module the nested blocks depend on.
+NESTED_MODULE = ("plugins", "cloud", "_truecloud_nested.py")
+
+#: The update-available alert source. Not a "patch" (it appends nothing to a stock
+#: file), but it is a file we install into middlewared and must therefore remove.
+ALERT_MODULE = ("alert", "source", "truecloud_patch_update.py")
+
+
+def patch_file(path, block):
+    """Append `block`, replacing any block we appended before. Idempotent."""
+    with open(path, encoding="utf-8") as fh:
+        content = fh.read()
+    idx = content.find(MARKER)
+    base = content[:idx] if idx != -1 else content
+    with open(path, "w", encoding="utf-8") as fh:
+        fh.write(base.rstrip("\n") + "\n" + block)
+
+
+def unpatch_file(path):
+    """Strip our appended block, restoring the stock file. True if it was patched."""
+    try:
+        with open(path, encoding="utf-8") as fh:
+            content = fh.read()
+    except OSError:
+        return False
+    idx = content.find(MARKER)
+    if idx == -1:
+        return False
+    try:
+        with open(path, "w", encoding="utf-8") as fh:
+            fh.write(content[:idx].rstrip("\n") + "\n")
+    except OSError:
+        return False
+    return True
+
+
+def revert(mw_dir, relpaths, module_relpath=None):
+    """Remove our blocks from `relpaths`, and the module at `module_relpath`.
+
+    The module is deleted FIRST. Every injected block is guarded by
+    `if _tc_nested is not None`, so once the module is gone the blocks all no-op
+    even if a later unpatch fails -- the stock guard comes back regardless.
+
+    Returns the names of what was actually reverted.
+    """
+    reverted = []
+
+    if module_relpath:
+        try:
+            os.unlink(os.path.join(mw_dir, *module_relpath))
+            reverted.append(module_relpath[-1])
+        except OSError:
+            pass
+
+    for rel in relpaths:
+        if unpatch_file(os.path.join(mw_dir, *rel)):
+            reverted.append(rel[-1])
+
+    return reverted
+
+
+def revert_nested(mw_dir):
+    """Undo the nested-snapshot patch only. Leaves the providers patch alone.
+
+    restic.py also carries a block, but it belongs to the providers module --
+    reverting it would silently break B2 backups.
+    """
+    return revert(mw_dir, NESTED_RELPATHS, NESTED_MODULE)
+
+
+def revert_all(mw_dir):
+    """Undo every patch this project applies, and remove every file it installs."""
+    reverted = revert(mw_dir, NESTED_RELPATHS + PROVIDER_RELPATHS, NESTED_MODULE)
+    try:
+        os.unlink(os.path.join(mw_dir, *ALERT_MODULE))
+        reverted.append(ALERT_MODULE[-1])
+    except OSError:
+        pass
+    return reverted
+
+
+def find_middlewared_dir():
+    """Directory of the installed `middlewared` package, or None."""
+    try:
+        import middlewared
+    except ImportError:
+        return None
+    return os.path.dirname(os.path.abspath(middlewared.__file__))
+
+
+def main(argv):
+    if len(argv) < 2 or argv[1] not in ("revert-all", "revert-nested"):
+        print(__doc__, file=sys.stderr)
+        return 2
+
+    mw_dir = find_middlewared_dir()
+    if mw_dir is None:
+        print("  middlewared not importable — nothing to revert.")
+        return 0
+
+    fn = revert_all if argv[1] == "revert-all" else revert_nested
+    reverted = fn(mw_dir)
+    if reverted:
+        print("  Reverted: " + ", ".join(reverted))
+    else:
+        print("  Nothing to revert (overlay already removed, or never patched).")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

patch/patch_ui.py

@@ -9,8 +9,8 @@ in the minified JS in one of two forms depending on TrueNAS / Angular version:
   TrueNAS 24.x (static inline array):
       "filterByProviders",["STORJ_IX"]
 
-  TrueNAS 25.x+ (Angular pureFunction binding):
-      "filterByProviders",pe(115,Rn,i.CloudSyncProviderName.Storj)
+  TrueNAS 25.x+ (Angular pureFunction binding, inside a chained property call):
+      c(2,"filterByProviders",pe(115,Rn,i.CloudSyncProviderName.Storj))("required",!0)
 
 Both are replaced so the dropdown includes S3 and B2. The file is backed up
 before modification so uninstall.sh can restore it.
@@ -19,6 +19,7 @@ Safe to run multiple times — a marker string detects an already-patched file.
 Exits 0 in all cases (warnings are printed to stdout and logged by apply.sh).
 """
 
+import contextlib
 import os
 import re
 import shutil
@@ -32,11 +33,21 @@ WEBUI_CANDIDATES = [
 # Patterns tried in order; the first match wins.
 # Each entry is (compiled_regex, replacement_string).
 _PATTERNS = [
-    # TrueNAS 25.x+: Angular emits a pureFunction call instead of a literal array.
-    # pe / slot-index / factory-var / component-var are all minified and change
-    # across builds; CloudSyncProviderName.Storj is stable (TypeScript enum name).
-    (re.compile(r'("filterByProviders",)\w+\(\d+,\w+,\w+\.CloudSyncProviderName\.Storj\)'),
-     r'\1["STORJ_IX","S3","B2"]'),
+    # TrueNAS 25.x+: Angular emits a pureFunction call instead of a literal array,
+    # inside a CHAINED property binding — so the call is followed by two closing
+    # parens, one for pe(...) and one for the property(...) it sits in:
+    #
+    #     c(2,"filterByProviders",pe(115,Rn,i.CloudSyncProviderName.Storj))("required",!0)
+    #                                                                    ^^
+    # The pattern consumes both and re-emits one, leaving the paren balance
+    # unchanged. Getting that wrong is a syntax error in the bundle and the whole
+    # web UI goes blank — see tests/test_patch_ui.py.
+    #
+    # The minified names (pe / slot index / Rn / i) change across builds;
+    # CloudSyncProviderName.Storj is stable because it is a TypeScript enum name.
+    (re.compile(r'("filterByProviders",)\w+\(\d+,\w+,\w+\.CloudSyncProviderName\.Storj\)\)'),
+     r'\1["STORJ_IX","S3","B2"])'),
+
     # TrueNAS 24.x and earlier: static inline array.
     (re.compile(r'("filterByProviders",)\["STORJ_IX"\]'),
      r'\1["STORJ_IX","S3","B2"]'),
@@ -54,6 +65,11 @@ def _match_pattern(content):
     return None, None
 
 
+def _paren_delta(s):
+    """Net parenthesis balance. Patching must not change it — see main()."""
+    return s.count("(") - s.count(")")
+
+
 def find_bundle():
     """
     Search WEBUI_CANDIDATES for the JS chunk containing the filterByProviders
@@ -124,6 +140,24 @@ def main():
         )
         return
 
+    # Never write JS whose parentheses we have unbalanced. A pattern that eats one
+    # paren too many is a syntax error in the bundle and the entire TrueNAS web UI
+    # goes blank -- and because MARKER is then present, every later run reports
+    # "already patched" and skips, so the patch cannot heal itself. Recovery means
+    # hand-restoring the .pre-truecloud-patch backup.
+    #
+    # This is not hypothetical: it shipped once. Refuse instead.
+    if _paren_delta(patched) != _paren_delta(content):
+        print(
+            "[truecloud-patch] ERROR: the replacement would unbalance the bundle's "
+            "parentheses — refusing to write.\n"
+            "[truecloud-patch] The UI is UNCHANGED and still works. This means the "
+            "pattern no longer fits this TrueNAS build.\n"
+            "[truecloud-patch] File an issue at "
+            "https://github.com/sudolulo/truenas-truecloud-patch"
+        )
+        return
+
     tmp = path + ".tmp"
     try:
         with open(tmp, "w", encoding="utf-8") as fh:
@@ -131,10 +165,8 @@ def main():
         os.replace(tmp, path)
     except OSError as exc:
         print(f"[truecloud-patch] ERROR: Could not write {path}: {exc}")
-        try:
+        with contextlib.suppress(OSError):
             os.unlink(tmp)
-        except OSError:
-            pass
         return
 
     print(f"[truecloud-patch] UI bundle patched ({count} replacement(s)): {path}")

patch/truecloud_nested.py

@@ -0,0 +1,577 @@
+"""Nested-dataset snapshot support for TrueCloud Backup.
+
+Why this exists
+---------------
+Stock TrueNAS refuses ``snapshot = true`` when the backup path contains child
+datasets::
+
+    This option is only available for datasets that have no further nesting
+
+That guard is *correct* and it is not laziness. ``plugins/cloud/snapshot.py``
+already takes a **recursive** ZFS snapshot, but it then points the backup tool
+at the *parent* dataset's ``.zfs/snapshot/<snap>/`` directory -- and ZFS does
+not expose child datasets through a parent's snapshot directory::
+
+    /mnt/Tap/.zfs/snapshot/<snap>/apps/                 ->  0 entries
+    /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/   ->  the real data
+
+So without the guard, the backup tool would walk a near-empty tree, report
+SUCCESS, and upload almost nothing. A backup that lies about succeeding is the
+worst failure a backup system can have, so middleware refuses the config
+instead.
+
+This module implements the missing half: after the (already recursive) snapshot
+is taken, every descendant dataset's *own* ``.zfs/snapshot/<snap>`` directory is
+bind-mounted into a staging tree that mirrors the original layout. The backup
+tool is then pointed at the staging root, which is a complete, consistent,
+point-in-time view of the whole subtree.
+
+Cardinal safety rule
+--------------------
+**If the tree cannot be staged completely, fail loudly.** Never return a partial
+tree. Silently backing up an incomplete tree is precisely the failure this
+feature exists to prevent, and it would be worse than not having the feature.
+
+Snapshot lifecycle -- read this before changing anything
+--------------------------------------------------------
+``zfs.snapshot.delete`` defaults to ``recursive=False``, and stock
+``restic_backup()`` calls it with no options. Stock gets away with that because
+its validation means ``recursive`` is never actually True in the field. Enabling
+nested datasets makes recursive snapshots real, so the parent
+(``Tap@snap``) has one child snapshot per descendant dataset (160+ here).
+Deleting only the parent would orphan every child on **every successful run**.
+
+Therefore this module owns the whole lifecycle:
+
+* :func:`delete_snapshot_tree` sweeps the parent *and* every child snapshot, and
+  is idempotent -- it copes with stock's ``finally`` having already removed the
+  parent.
+* The snapshot name is recorded in a sidecar file next to the staging root, not
+  only in memory, so a middlewared restart mid-backup cannot orphan it.
+* Bind-mounting ``.zfs/snapshot/<snap>`` pins the snapshot, so stock's delete
+  fails with EBUSY and logs one benign warning; we unmount and then sweep.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import os
+import stat
+import subprocess
+
+__all__ = [
+    "STAGING_BASE",
+    "StagingError",
+    "apply_plan",
+    "cleanup_all",
+    "cleanup_task",
+    "current_mounts_under",
+    "delete_snapshot_tree",
+    "plan_staging",
+    "sidecar_for",
+    "snapshot_tree_names",
+    "stage_nested",
+    "staging_root_for",
+    "teardown",
+    "verify_staged",
+]
+
+#: Where staging trees are assembled. tmpfs; bind mounts consume no space.
+STAGING_BASE = "/run/truecloud-nested"
+
+# Which snapshot a staging tree pins is recorded ONLY in the sidecar file, never
+# also in memory. An in-process dict would be a second source of truth that a
+# middlewared restart silently empties -- and it is exactly the restart case that
+# must not orphan a 250-snapshot tree. One record, on disk, or none.
+
+
+class StagingError(Exception):
+    """Staging could not produce a complete tree. The backup must not proceed."""
+
+
+# ── pure helpers ──────────────────────────────────────────────────────────────
+
+
+def staging_root_for(name: str, base: str | None = None) -> str:
+    """Stable staging root for a task name (e.g. ``cloud_backup-5``).
+
+    ``base`` defaults to :data:`STAGING_BASE` at CALL time, not at import time --
+    a ``base=STAGING_BASE`` default would freeze the value into the function
+    object and silently ignore any later override.
+    """
+    if base is None:
+        base = STAGING_BASE
+    safe = "".join(c if (c.isalnum() or c in "-_.") else "_" for c in name)
+    # A component of "." or ".." would escape STAGING_BASE once joined.
+    if not safe or safe.strip(".") == "":
+        safe = "task"
+    return os.path.join(base, safe)
+
+
+def sidecar_for(staging_root: str) -> str:
+    """Path of the file recording which ZFS snapshot a staging tree pins."""
+    return staging_root + ".snapshot"
+
+
+def _write_sidecar(staging_root: str, snapshot: str) -> None:
+    """Record the pinned snapshot on disk. Blocking; call via run_in_thread."""
+    with contextlib.suppress(OSError):
+        os.makedirs(os.path.dirname(staging_root), exist_ok=True)
+        with open(sidecar_for(staging_root), "w", encoding="utf-8") as fh:
+            fh.write(snapshot)
+
+
+def _read_sidecar(staging_root: str) -> str | None:
+    """The snapshot a previous run recorded here, if any."""
+    try:
+        with open(sidecar_for(staging_root), encoding="utf-8") as fh:
+            return fh.read().strip() or None
+    except OSError:
+        return None
+
+
+def _remove_sidecar(staging_root: str) -> None:
+    with contextlib.suppress(OSError):
+        os.unlink(sidecar_for(staging_root))
+
+
+def _depth(path: str) -> int:
+    return len([p for p in path.split("/") if p])
+
+
+def snapshot_tree_names(snapshot: str, all_names) -> list[str]:
+    """Every snapshot produced by ``zfs snapshot -r <dataset>@<snap>``.
+
+    That is the parent plus one per descendant dataset, all sharing the same
+    name after the ``@``. Pure, so the sweep logic is testable without ZFS.
+    """
+    dataset, _, snapname = snapshot.partition("@")
+    if not snapname:
+        return []
+    parent = f"{dataset}@{snapname}"
+    prefix = dataset + "/"
+    suffix = "@" + snapname
+    return [
+        n for n in all_names
+        if n == parent or (n.startswith(prefix) and n.endswith(suffix))
+    ]
+
+
+def _probe_snapdir(path):
+    """Classify a snapshot directory: ``ok``, ``missing``, or why it is unusable.
+
+    ``os.path.isdir()`` collapses "does not exist" and "cannot stat" into the
+    same ``False``, so an EACCES would report itself as "has no snapshot" and
+    send someone hunting for a snapshot that is sitting right there. Both cases
+    still abort the backup -- but it has to say which one.
+    """
+    try:
+        st = os.stat(path)
+    except FileNotFoundError:
+        return "missing"
+    except OSError as e:
+        return f"cannot be read ({e.strerror})"
+    return "ok" if stat.S_ISDIR(st.st_mode) else "is not a directory"
+
+
+def plan_staging(base_dataset, base_mountpoint, path, snapshot_name, datasets,
+                 staging_root, probe=_probe_snapdir):
+    """Compute the bind-mount plan for staging a nested tree. Pure function.
+
+    ``datasets`` is a list of dicts shaped like ``zfs.dataset.query`` results:
+    ``{"name": str, "properties": {"mountpoint": {"value": str},
+    "mounted": {"value": "yes"|"no"}}}``.
+
+    Returns ``(mounts, skipped)`` where ``mounts`` is an ordered list of
+    ``(source, target)`` pairs (parents before children) and ``skipped`` is a
+    list of ``(dataset_name, reason)`` covering only datasets that are *in
+    scope* -- i.e. descendants of ``base_dataset``. Datasets elsewhere on the
+    system are ignored silently; reporting them would bury the ones that matter.
+
+    Raises StagingError if an in-scope descendant holds data we would otherwise
+    silently omit.
+    """
+    def snapdir(mountpoint):
+        return os.path.join(mountpoint, ".zfs", "snapshot", snapshot_name)
+
+    # Root of the staging tree: the backup path as seen inside the base
+    # dataset's own snapshot.
+    rel = os.path.relpath(path, base_mountpoint)
+    root_src = snapdir(base_mountpoint)
+    if rel != ".":
+        root_src = os.path.join(root_src, rel)
+
+    mounts = [(root_src, staging_root)]
+    skipped = []
+
+    ds_prefix = base_dataset.rstrip("/") + "/"
+    path_prefix = path.rstrip("/") + "/"
+
+    for ds in datasets:
+        name = ds.get("name", "")
+        # Scope by DATASET NAME, not mountpoint: a dataset with no mountpoint
+        # cannot be scoped by path, and scoping by path first would drag in
+        # every mountpoint-less dataset on the box (all of Tank/.system/*, ...).
+        if not name.startswith(ds_prefix):
+            continue
+
+        props = ds.get("properties", {})
+        mp = props.get("mountpoint", {}).get("value", "")
+
+        if not mp or mp in ("none", "legacy", "-"):
+            skipped.append((name, f"mountpoint is {mp or 'unset'}"))
+            continue
+
+        if not mp.startswith(path_prefix):
+            # A descendant dataset mounted outside the backed-up path is
+            # genuinely not part of this tree. Not an omission.
+            continue
+
+        if props.get("mounted", {}).get("value", "yes") == "no":
+            # An unmounted (e.g. locked/encrypted) dataset contributes nothing to
+            # the live tree either, so skipping matches stock semantics -- but it
+            # is a real gap and must be visible, never silent.
+            skipped.append((name, "dataset is not mounted (locked/encrypted?)"))
+            continue
+
+        src = snapdir(mp)
+        status = probe(src)
+        if status != "ok":
+            # Either the recursive snapshot missed this dataset, or we cannot read
+            # it. Either way its data would be silently omitted. Refuse -- but say
+            # WHICH, because "no snapshot" and "permission denied" send you to
+            # completely different places.
+            detail = (
+                f"has no snapshot {snapshot_name!r}" if status == "missing"
+                else f"snapshot {snapshot_name!r} {status}"
+            )
+            raise StagingError(
+                f"dataset {name!r} {detail} at {src!r}; "
+                f"refusing to back up an incomplete tree"
+            )
+
+        mounts.append((src, os.path.join(staging_root, os.path.relpath(mp, path))))
+
+    # Parents before children, so each mountpoint exists before we mount onto it.
+    mounts.sort(key=lambda m: _depth(m[1]))
+    return mounts, skipped
+
+
+def current_mounts_under(root, mounts_file="/proc/self/mounts"):
+    """Mountpoints at or under ``root``, deepest first. Used for teardown."""
+    found = []
+    try:
+        with open(mounts_file, encoding="utf-8") as fh:
+            for line in fh:
+                parts = line.split()
+                if len(parts) < 2:
+                    continue
+                mp = parts[1].replace("\\040", " ").replace("\\011", "\t")
+                if mp == root or mp.startswith(root.rstrip("/") + "/"):
+                    found.append(mp)
+    except OSError:
+        return []
+    found.sort(key=_depth, reverse=True)
+    return found
+
+
+# ── mount / unmount ───────────────────────────────────────────────────────────
+
+
+def _run(cmd):
+    # List form, never shell=True: `cmd` is built from our own mount plan, so ZFS
+    # dataset names cannot inject. Runs as root by definition (it mounts).
+    return subprocess.run(  # noqa: S603
+        cmd, capture_output=True, text=True, check=False
+    )
+
+
+def apply_plan(mounts, runner=_run, isdir=os.path.isdir):
+    """Execute the bind-mount plan. Blocking; call via ``run_in_thread``.
+
+    Raises StagingError on the first failure, after rolling back what was
+    mounted -- a half-built tree must never be handed to the backup tool.
+    """
+    if not mounts:
+        raise StagingError("empty staging plan")
+
+    staging_root = mounts[0][1]
+    done = []
+    try:
+        os.makedirs(staging_root, exist_ok=True)
+        for src, target in mounts:
+            if not isdir(target):
+                # Child mountpoint dirs come from the parent snapshot, which is
+                # read-only -- we cannot mkdir them. Only the root is ours.
+                raise StagingError(f"staging target {target!r} does not exist")
+            res = runner(["mount", "--bind", src, target])
+            if res.returncode != 0:
+                raise StagingError(
+                    f"bind-mount {src!r} -> {target!r} failed: "
+                    f"{(res.stderr or '').strip() or res.returncode}"
+                )
+            done.append(target)
+    except Exception:
+        for target in reversed(done):
+            runner(["umount", "-l", target])
+        with contextlib.suppress(OSError):
+            os.rmdir(staging_root)
+        raise
+    return staging_root
+
+
+def verify_staged(mounts, ismount=os.path.ismount, listdir=os.listdir):
+    """Assert the staged tree is real and complete. Raises StagingError if not.
+
+    This is the anti-regression guard: it is what stops this feature from ever
+    degrading back into the silently-empty backup that the stock validation
+    refuses to allow.
+    """
+    if not mounts:
+        raise StagingError("nothing was staged")
+
+    staging_root = mounts[0][1]
+    for _src, target in mounts:
+        if not ismount(target):
+            raise StagingError(f"staging target {target!r} is not a mountpoint")
+
+    try:
+        if not listdir(staging_root):
+            raise StagingError(f"staging root {staging_root!r} is empty")
+    except OSError as e:
+        raise StagingError(f"staging root {staging_root!r} unreadable: {e}") from e
+
+    return True
+
+
+def teardown(staging_root, runner=_run, mounts_file="/proc/self/mounts"):
+    """Unmount the staging tree (deepest first) and remove the root.
+
+    Idempotent, and does not depend on an in-memory plan -- so it also cleans up
+    leftovers from a crashed run.
+    """
+    errors = []
+    for mp in current_mounts_under(staging_root, mounts_file=mounts_file):
+        res = runner(["umount", mp])
+        if res.returncode != 0:
+            res = runner(["umount", "-l", mp])  # lazy: better than leaking
+            if res.returncode != 0:
+                errors.append(f"{mp}: {(res.stderr or '').strip()}")
+    with contextlib.suppress(OSError):
+        os.rmdir(staging_root)
+    return errors
+
+
+# ── async orchestration (middleware is duck-typed; no middlewared import) ─────
+
+
+async def delete_snapshot_tree(middleware, snapshot, logger=None):
+    """Delete the parent snapshot AND every child created by ``zfs snapshot -r``.
+
+    ``zfs.snapshot.delete`` is non-recursive by default and stock calls it with
+    no options, so relying on stock would orphan one snapshot per descendant
+    dataset on every run. Idempotent: tolerates the parent already being gone
+    (stock's ``finally`` may have won the race once our mounts were released).
+    """
+    dataset = snapshot.partition("@")[0]
+
+    # Fast path: ONE recursive delete removes the parent and every child that
+    # `zfs snapshot -r` created (252 on a real pool). Deleting them individually
+    # also works, but it is neither cheap nor atomic -- a run killed part-way
+    # through 252 sequential deletes leaves exactly the orphans this function
+    # exists to prevent.
+    try:
+        await middleware.call("zfs.snapshot.delete", snapshot, {"recursive": True})
+        return
+    except Exception as e:  # noqa: BLE001 - fall through to the explicit sweep
+        # Usually just "parent already gone" (stock's finally won the race once our
+        # mounts were released), which the sweep below handles. Log it rather than
+        # swallow it: if the real cause is something else, this is the only place
+        # it is visible -- the sweep would report a different, downstream failure.
+        if logger:
+            logger.debug(
+                "truecloud-patch: recursive delete of %s failed (%r); sweeping "
+                "the tree by name instead", snapshot, e,
+            )
+
+    # The parent may already be gone -- stock's `finally` can win the race once
+    # our mounts are released -- which fails the recursive delete while the
+    # children survive. Sweep them by name.
+    try:
+        snaps = await middleware.call(
+            "zfs.snapshot.query", [["name", "^", dataset]], {"select": ["name"]}
+        )
+        # An empty result means the tree is already gone -- delete nothing, and
+        # do not fall back to the parent, which would only log a spurious
+        # "does not exist" warning on every clean run.
+        names = snapshot_tree_names(snapshot, [s["name"] for s in snaps])
+    except Exception as e:  # noqa: BLE001 - fall back to at least the parent
+        if logger:
+            logger.warning(
+                "truecloud-patch: could not enumerate snapshot tree for %s: %r",
+                snapshot, e,
+            )
+        names = [snapshot]
+
+    for name in names:
+        try:
+            await middleware.call("zfs.snapshot.delete", name)
+        except Exception as e:  # noqa: BLE001 - already gone is fine
+            if logger:
+                logger.warning(
+                    "truecloud-patch: could not delete snapshot %s: %r", name, e
+                )
+
+
+async def stage_nested(middleware, path, snapshot, base_dataset, base_mountpoint,
+                       task_name, datasets, logger=None):
+    """Build a complete staging tree for `path` from the already-taken `snapshot`.
+
+    `snapshot` is a full ZFS snapshot name ("Tap@cloud_backup-5-2026...").
+
+    `datasets` is the FILESYSTEM dataset list. **It MUST have been enumerated
+    AFTER `snapshot` was taken.** A list read beforehand can miss a dataset
+    created in the gap: the recursive snapshot would capture it, but the staging
+    plan would not, and its data would be silently omitted from the backup.
+    Enumerated afterwards, an unsnapshotted dataset instead trips the isdir()
+    check in plan_staging and fails the run loudly.
+
+    Returns the staging root to hand to the backup tool.
+
+    Raises StagingError if the tree cannot be staged completely -- the caller
+    must let that propagate so the backup fails instead of silently uploading a
+    partial tree. The caller is responsible for deleting `snapshot` in that case
+    (see SNAPSHOT_BLOCK in apply.sh).
+    """
+    snapshot_name = snapshot.split("@", 1)[1]
+    staging_root = staging_root_for(task_name)
+
+    # A previous run may have crashed mid-flight; never build on top of that.
+    await middleware.run_in_thread(teardown, staging_root)
+
+    # ...and if it left a sidecar behind, that snapshot tree is still on disk and
+    # nothing else will ever reclaim it. Sweep it before we overwrite the record,
+    # or a single crashed run orphans 160+ snapshots permanently.
+    stale = await middleware.run_in_thread(_read_sidecar, staging_root)
+    if stale and stale != snapshot:
+        if logger:
+            logger.warning(
+                "truecloud-patch: reclaiming snapshot tree from an earlier "
+                "interrupted run: %s", stale,
+            )
+        await delete_snapshot_tree(middleware, stale, logger=logger)
+
+    # Record the snapshot BEFORE mounting anything, not after. middlewared can
+    # die at any point (this patch even schedules a restart at boot), and the
+    # sidecar is the only thing that survives it -- an in-process dict would take
+    # the sole record of a 160-snapshot tree with it. Writing it after apply_plan
+    # would leave exactly the crash window the sidecar exists to close.
+    await middleware.run_in_thread(_write_sidecar, staging_root, snapshot)
+
+    try:
+        mounts, skipped = await middleware.run_in_thread(
+            plan_staging, base_dataset, base_mountpoint, path, snapshot_name,
+            datasets, staging_root,
+        )
+        if logger:
+            for name, reason in skipped:
+                logger.warning(
+                    "truecloud-patch: not staging dataset %r: %s", name, reason
+                )
+
+        await middleware.run_in_thread(apply_plan, mounts)
+        await middleware.run_in_thread(verify_staged, mounts)
+    except Exception:
+        await middleware.run_in_thread(teardown, staging_root)
+        await middleware.run_in_thread(_remove_sidecar, staging_root)
+        raise
+
+    if logger:
+        logger.info(
+            "truecloud-patch: staged %d dataset(s) from %s at %s",
+            len(mounts), snapshot, staging_root,
+        )
+    return staging_root
+
+
+async def cleanup_task(middleware, task_name, logger=None):
+    """Tear down a task's staging tree and delete the snapshot it pinned.
+
+    Safe to call unconditionally: a no-op when the task was never staged.
+    """
+    staging_root = staging_root_for(task_name)
+    snapshot = _read_sidecar(staging_root)
+
+    if snapshot is None and not os.path.isdir(staging_root):
+        return  # never staged; nothing to do
+
+    errors = await middleware.run_in_thread(teardown, staging_root)
+    if errors and logger:
+        for err in errors:
+            logger.warning("truecloud-patch: staging teardown: %s", err)
+
+    if snapshot is not None:
+        await delete_snapshot_tree(middleware, snapshot, logger=logger)
+
+    _remove_sidecar(staging_root)
+
+
+# ── offline cleanup (uninstall.sh / recover.sh) ───────────────────────────────
+
+
+def cleanup_all(base=None, runner=_run, mounts_file="/proc/self/mounts",
+                glob_fn=None, read_sidecar=_read_sidecar):
+    """Tear down every staging tree. Used by uninstall.sh and recover.sh.
+
+    Those scripts must work when middlewared is dead, so they cannot go through
+    the async path -- but they must not reimplement the teardown either: the
+    depth-ordering and lazy-umount fallback are fiddly, and a second copy in
+    shell would be the untested one. This is the same tested code.
+
+    Returns ``(lines, errors)``: report lines to print, and unmount errors.
+    """
+    import glob as _glob
+
+    base = base or STAGING_BASE
+    glob_fn = glob_fn or _glob.glob
+    lines = []
+
+    # Report orphaned snapshots BEFORE removing the sidecars that name them --
+    # a sidecar is the only record that an interrupted run's snapshot tree (one
+    # snapshot per descendant dataset) is still on disk.
+    for sc in sorted(glob_fn(os.path.join(base, "*.snapshot"))):
+        snap = read_sidecar(sc[: -len(".snapshot")])
+        if snap:
+            lines.append(f"  NOTE: an interrupted backup left snapshot '{snap}' behind.")
+            lines.append(f"        Remove it and its children:  zfs destroy -r '{snap}'")
+
+    mounts = current_mounts_under(base, mounts_file=mounts_file)
+    if not mounts:
+        lines.append("  None active.")
+    for mp in mounts:
+        lines.append(f"  Unmounting: {mp}")
+
+    errors = teardown(base, runner=runner, mounts_file=mounts_file)
+    for err in errors:
+        lines.append(f"  WARNING: could not unmount {err}")
+
+    if not errors:
+        for sc in glob_fn(os.path.join(base, "*.snapshot")):
+            with contextlib.suppress(OSError):
+                os.unlink(sc)
+        with contextlib.suppress(OSError):
+            os.rmdir(base)
+
+    return lines, errors
+
+
+if __name__ == "__main__":
+    import sys
+
+    if len(sys.argv) > 1 and sys.argv[1] == "cleanup":
+        _lines, _errors = cleanup_all()
+        for _line in _lines:
+            print(_line)
+        sys.exit(1 if _errors else 0)
+    print("usage: truecloud_nested.py cleanup", file=sys.stderr)
+    sys.exit(2)

pyproject.toml

@@ -0,0 +1,15 @@
+# Tooling config only — this project is not a Python package. The patch modules
+# are copied into middlewared's site-packages by patch/apply.sh at boot.
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "B", "UP", "SIM"]
+ignore = [
+  "E501",  # long lines in explanatory comments are fine
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

recover.sh

@@ -17,10 +17,11 @@
 #   bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
 #   systemctl restart middlewared
 
-VERSION="0.0.4"
+VERSION="0.5.1"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 
+
 echo "=== TrueNAS TrueCloud Provider Patch v${VERSION} — Recover ==="
 echo ""
 
@@ -52,6 +53,14 @@ for _tag in mw ui; do
 done
 [ "$_any" -eq 0 ] && echo "  No overlays active."
 
+# Nested-snapshot staging trees are bind mounts that PIN their ZFS snapshots, so
+# leaving them mounted blocks those snapshots from ever being destroyed. The
+# overlays above are volatile, but these are not self-healing without a reboot,
+# and recover.sh is expected to work without one.
+echo "Unmounting nested-snapshot staging trees ..."
+# Best-effort: never block recovery. Same tested implementation as uninstall.sh.
+python3 "$PATCH_DIR/patch/truecloud_nested.py" cleanup || true
+
 # Cancel a deferred boot restart if one is still queued — we restart ourselves.
 systemctl stop truecloud-mw-restart.service 2>/dev/null
 systemctl reset-failed truecloud-mw-restart.service 2>/dev/null

tests/test_alert_source.py

@@ -0,0 +1,132 @@
+"""Tests for the update-available alert source.
+
+This file is imported by middlewared's `alert.load()`, which runs at STARTUP and
+has **no try/except**:
+
+    def load(self):
+        for module in load_modules(.../alert/source):
+            for cls in load_classes(module, AlertSource, (ThreadedAlertSource,)):
+                source = cls(self.middleware)
+                if source.name in ALERT_SOURCES:
+                    raise RuntimeError(...)
+
+So a module that raises on import takes middlewared's setup down with it. These
+tests guard the realistic ways that could happen.
+"""
+
+import ast
+import os
+import re
+
+import pytest
+
+ALERT_SRC = os.path.join(os.path.dirname(__file__), "..", "patch", "alert_source.py")
+APPLY_SH = os.path.join(os.path.dirname(__file__), "..", "patch", "apply.sh")
+
+
+def source():
+    with open(ALERT_SRC, encoding="utf-8") as fh:
+        return fh.read()
+
+
+def tree():
+    return ast.parse(source())
+
+
+class TestCannotBreakMiddlewaredAtImport:
+    def test_it_compiles(self):
+        compile(source(), "alert_source.py", "exec")
+
+    def test_apply_sh_compiles_it_before_writing_it(self):
+        # The substituted file is what middlewared imports. If it does not compile,
+        # installing it would break startup — so apply.sh must refuse to write it.
+        with open(APPLY_SH, encoding="utf-8") as fh:
+            sh = fh.read()
+        i = sh.index("alert_dst = os.path.join(mw_dir, 'alert', 'source'")
+        block = sh[i:i + 1600]
+        assert "compile(_body, alert_dst, 'exec')" in block
+        assert block.index("compile(_body") < block.index("open(alert_dst, 'w'")
+
+    def test_patch_dir_is_substituted_with_repr(self):
+        # A directory containing a quote or backslash would otherwise produce a
+        # syntax error in the installed module.
+        with open(APPLY_SH, encoding="utf-8") as fh:
+            sh = fh.read()
+        assert "_body.replace('\"@PATCH_DIR@\"', repr(_patch_dir))" in sh
+
+    @pytest.mark.parametrize("path", [
+        "/mnt/tank/patch",
+        '/mnt/we"ird/patch',      # a quote in the path
+        "/mnt/back\\slash/patch",  # a backslash
+    ])
+    def test_substituted_module_compiles_for_awkward_paths(self, path):
+        body = source().replace('"@PATCH_DIR@"', repr(path))
+        compile(body, "alert_source.py", "exec")  # must not raise
+
+    def test_no_io_at_module_import_time(self):
+        # Anything at module scope runs during alert.load(). Only imports,
+        # constants and class definitions are allowed.
+        allowed = (ast.Import, ast.ImportFrom, ast.Assign, ast.AnnAssign,
+                   ast.ClassDef, ast.FunctionDef, ast.Expr)
+        for node in tree().body:
+            assert isinstance(node, allowed), f"module-level {type(node).__name__}"
+            if isinstance(node, ast.Expr):
+                assert isinstance(node.value, ast.Constant), "only the docstring"
+
+
+class TestAlertClassNaming:
+    """middlewared's AlertClassMeta raises NameError unless the name ends in
+    'AlertClass' — at import, inside alert.load(), which has no try/except."""
+
+    def alert_classes(self):
+        return [n for n in tree().body
+                if isinstance(n, ast.ClassDef)
+                and any(getattr(b, "id", "") == "AlertClass" for b in n.bases)]
+
+    def test_there_are_alert_classes(self):
+        assert self.alert_classes()
+
+    def test_every_alert_class_name_ends_in_AlertClass(self):
+        for cls in self.alert_classes():
+            assert cls.name.endswith("AlertClass"), (
+                f"{cls.name}: AlertClassMeta raises NameError on this"
+            )
+
+    def test_every_alert_class_defines_the_required_attrs(self):
+        # category/level/title are NotImplemented on the base; a missing one shows
+        # up as a broken alert rather than an error.
+        for cls in self.alert_classes():
+            names = {t.id for n in cls.body if isinstance(n, ast.Assign)
+                     for t in n.targets if isinstance(t, ast.Name)}
+            assert {"category", "level", "title", "text"} <= names, cls.name
+
+    def test_alert_text_placeholders_match_the_args_we_pass(self):
+        src = source()
+        placeholders = set(re.findall(r"%\((\w+)\)s", src))
+        # These are the keys built in _check().
+        assert placeholders <= {"current", "latest", "summary", "dir"}
+
+
+class TestNoSysPathMutation:
+    def test_release_notes_is_loaded_by_path_not_sys_path(self):
+        # sys.path.insert(0, ...) would shadow the stdlib for this interpreter, and
+        # ThreadedAlertSource runs in a thread pool — mutating sys.path is a race.
+        #
+        # Check for actual MUTATION, not the string: the docstring legitimately
+        # mentions sys.path to explain why it is avoided.
+        src = source()
+        assert "sys.path.insert" not in src
+        assert "sys.path.append" not in src
+        assert not re.search(r"^import sys$", src, re.M), "sys is not needed"
+        assert "spec_from_file_location" in src
+
+
+class TestNeverWritesToGit:
+    def test_only_read_only_git_commands(self):
+        # A `git fetch` from middlewared (running as root) would leave root-owned
+        # objects in .git and break every later non-root git command — which is
+        # exactly the breakage this project already hit once.
+        src = source()
+        for forbidden in ("fetch", "pull", "checkout", "clone", "reset"):
+            assert f'"{forbidden}"' not in src, f"git {forbidden} writes to .git"
+        assert '"ls-remote"' in src

tests/test_apply_blocks.py

@@ -0,0 +1,309 @@
+"""The *_BLOCK strings in apply.sh are Python source injected into middleware.
+
+A syntax error in one of them would be appended to a live middlewared module and
+break the box at boot. They are string literals, so nothing type-checks them --
+these tests do.
+"""
+
+import ast
+import os
+import re
+import textwrap
+
+import pytest
+
+APPLY_SH = os.path.join(os.path.dirname(__file__), "..", "patch", "apply.sh")
+
+EXPECTED_BLOCKS = {
+    "B2_BLOCK",
+    "RESTIC_BLOCK",
+    "SNAPSHOT_BLOCK",
+    "CRUD_BLOCK",
+    "SYNC_BLOCK",
+}
+
+
+def heredoc_source():
+    with open(APPLY_SH, encoding="utf-8") as fh:
+        src = fh.read()
+    m = re.search(r"<< 'PYEOF'\n(.*?)\nPYEOF", src, re.S)
+    assert m, "could not find the PYEOF heredoc in apply.sh"
+    return m.group(1)
+
+
+def extract_blocks():
+    tree = ast.parse(heredoc_source())
+    blocks = {}
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Assign):
+            for tgt in node.targets:
+                if (
+                    isinstance(tgt, ast.Name)
+                    and tgt.id.endswith("_BLOCK")
+                    and isinstance(node.value, ast.Constant)
+                    and isinstance(node.value.value, str)
+                ):
+                    blocks[tgt.id] = node.value.value
+    return blocks
+
+
+def _nested_native_detector():
+    """The REAL native-nested probe, lifted out of apply.sh.
+
+    Extracted rather than reimplemented: a reimplementation would happily pass
+    while the shipped probe stayed broken, which is precisely the bug this guards.
+    """
+    with open(APPLY_SH, encoding="utf-8") as fh:
+        sh = fh.read()
+
+    m = re.search(
+        r"^(\s*)_drop = str\.maketrans\(.*?\n\s*if 'nofurthernesting' not in "
+        r"stock_src\.translate\(_drop\):\n\s*result\['native_nested'\] = 'yes'",
+        sh, re.S | re.M,
+    )
+    assert m, "could not find the native-nested probe in apply.sh"
+
+    # The block lives inside a double-quoted shell string; undo bash's escaping.
+    body = m.group(0)
+    body = body.replace("\\\\", "\x00").replace('\\"', '"').replace("\x00", "\\")
+    body = textwrap.dedent(body)
+
+    def detect(stock_src):
+        ns = {"stock_src": stock_src, "result": {"native_nested": "no"}, "chr": chr}
+        exec(body, ns)  # noqa: S102 - executing our own shipped code, on purpose
+        return ns["result"]["native_nested"]
+
+    return detect
+
+
+def test_heredoc_itself_compiles():
+    compile(heredoc_source(), "apply.sh:PYEOF", "exec")
+
+
+def test_all_expected_blocks_present():
+    assert set(extract_blocks()) == EXPECTED_BLOCKS
+
+
+@pytest.mark.parametrize("name", sorted(EXPECTED_BLOCKS))
+def test_injected_block_is_valid_python(name):
+    block = extract_blocks()[name]
+    compile(block, f"apply.sh:{name}", "exec")
+
+
+@pytest.mark.parametrize("name", sorted(EXPECTED_BLOCKS))
+def test_injected_block_carries_the_idempotency_marker(name):
+    # patch_file() truncates each target file at "\n# TRUECLOUD_PATCH" before
+    # re-appending, so every block must start with that marker or repeated runs
+    # would stack duplicate copies into the middleware module.
+    assert extract_blocks()[name].lstrip("\n").startswith("# TRUECLOUD_PATCH")
+
+
+@pytest.mark.parametrize("name", ["SNAPSHOT_BLOCK", "CRUD_BLOCK", "SYNC_BLOCK"])
+def test_nested_blocks_degrade_safely_without_the_module(name):
+    # If _truecloud_nested failed to install, every nested block must no-op.
+    # Critically this includes CRUD_BLOCK: relaxing the guard without the
+    # traversal in place would mean silently-empty backups.
+    block = extract_blocks()[name]
+    assert "_tc_nested = None" in block
+    assert "if _tc_nested is not None:" in block
+
+
+class TestSnapshotLeak:
+    """zfs.snapshot.delete is non-recursive and stock calls it with no options.
+
+    A recursive snapshot has one child per descendant dataset (160+ here), so
+    every path that creates one must also sweep the whole tree.
+    """
+
+    def test_staging_failure_deletes_the_snapshot_tree(self):
+        # On a staging failure, sync.py's `snapshot, local_path = await
+        # create_snapshot(...)` never completes, so its local `snapshot` stays
+        # None and its finally deletes nothing. We must sweep it ourselves.
+        block = extract_blocks()["SNAPSHOT_BLOCK"]
+        assert "except Exception:" in block
+        assert "delete_snapshot_tree" in block
+        assert "raise" in block
+
+    def test_sync_block_cleans_up_on_every_path(self):
+        block = extract_blocks()["SYNC_BLOCK"]
+        assert "finally:" in block
+        assert "cleanup_task" in block
+
+
+def test_crud_block_is_scoped_to_cloud_backup():
+    # cloudsync has no staging teardown wired in, so its guard must stay.
+    assert '!= "cloud_backup"' in extract_blocks()["CRUD_BLOCK"]
+
+
+class TestIndependentModules:
+    """The two modules must retire independently.
+
+    TrueNAS may ship native B2 support long before (or after) it handles nested
+    datasets. A single all-or-nothing kill switch would silently take a
+    still-needed module down with the superseded one.
+    """
+
+    def _sh(self):
+        with open(APPLY_SH, encoding="utf-8") as fh:
+            return fh.read()
+
+    def test_native_support_is_detected_per_module(self):
+        sh = self._sh()
+        assert "native_b2" in sh
+        assert "native_nested" in sh
+        assert "no further nesting" in sh, "nested native-support probe"
+
+    def test_kill_switch_only_when_both_modules_are_done(self):
+        sh = self._sh()
+        assert '[ "$_providers_needed" = "0" ] && [ "$_nested_needed" = "0" ]' in sh
+        # ...and that is the only place the kill switch is actually set. (Ignore
+        # comment lines, which mention the same path.)
+        code = [ln for ln in sh.splitlines() if not ln.lstrip().startswith("#")]
+        sets = [ln for ln in code if 'touch "$PATCH_DIR/disabled"' in ln]
+        assert len(sets) == 1, f"kill switch set in {len(sets)} places"
+
+    def test_each_module_is_gated_separately(self):
+        src = heredoc_source()
+        assert "if not providers_needed:" in src
+        assert "elif nested_native:" in src
+
+    def test_ui_patch_is_tied_to_the_providers_module(self):
+        # The UI change widens the credential dropdown; it is meaningless once B2
+        # is native, but must NOT be skipped merely because nested is off.
+        sh = self._sh()
+        i = sh.index("--- UI patch ---")
+        assert '[ "$_providers_needed" = "0" ]' in sh[i:i + 400]
+
+    def test_status_reports_an_inactive_module_as_ok(self):
+        # `create_task.py verify` fails if any patches[*].ok is false. An opt-in
+        # module that is switched off (the DEFAULT) must not report FAIL, or a
+        # stock install fails verification out of the box.
+        src = heredoc_source()
+        assert "'ok': (not nested_needed) or nested_ok" in src
+        assert "'ok': (not providers_needed) or bool(b2_ok and restic_ok)" in src
+        assert "'active': nested_needed" in src
+
+    def test_nested_native_probe_matches_the_real_wrapped_source(self):
+        """Stock splits the guard message across adjacent string literals.
+
+        Python concatenates them at runtime, so the errmsg is contiguous -- but the
+        SOURCE never contains the whole phrase. A raw substring search finds
+        nothing, concludes iX removed the guard, and silently skips this module
+        forever. This is exactly what happened, and only a run against real
+        middlewared caught it.
+        """
+        detect = _nested_native_detector()
+
+        # Verbatim shape from TrueNAS plugins/cloud/crud.py.
+        stock_wrapped = (
+            '            verrors.add(f"{name}.snapshot", '
+            '"This option is only available for datasets that have no further "\n'
+            '                                            "nesting")\n'
+        )
+        assert detect(stock_wrapped) == "no", "guard is present; must NOT report native"
+
+        # Same message on a single line — must also be detected.
+        assert detect('verrors.add(x, "... have no further nesting")\n') == "no"
+
+        # Single-quoted, three-way split — still the guard.
+        assert detect(
+            "verrors.add(x, 'This option is only available for '\n"
+            "               'datasets that have no further '\n"
+            "               'nesting')\n"
+        ) == "no"
+
+        # Guard genuinely gone -> native support.
+        assert detect("def _validate(self):\n    pass\n") == "yes"
+
+    def test_nested_native_probe_ignores_our_own_block(self):
+        # CRUD_BLOCK quotes the guard message, so scanning the whole file would
+        # find the string in our own patch and never detect native support.
+        sh = self._sh()
+        assert "split('\\n# TRUECLOUD_PATCH', 1)[0]" in sh
+        assert "no further nesting" in extract_blocks()["CRUD_BLOCK"], (
+            "if this ever stops being true, the probe comment is stale"
+        )
+
+    def test_restart_fires_when_any_needed_module_landed(self):
+        # Keying the restart off providers alone would leave a freshly-patched
+        # nested module on disk and never loaded on a native-B2 box.
+        sh = self._sh()
+        i = sh.index("--- deferred restart ---")
+        tail = sh[i:]
+        assert '_backend_ok' in tail
+        assert '"$_b2_ok"' not in tail
+
+    def test_partial_failure_still_schedules_the_restart(self):
+        # If providers fails but nested landed (or vice versa), something new IS
+        # on disk. Collapsing that into "nothing to do" would leave the module
+        # that succeeded permanently unloaded.
+        src = heredoc_source()
+        assert "sys.exit(2 if _landed else 1)" in src
+        assert "_landed = (providers_needed and b2_ok and restic_ok) or (nested_needed and nested_ok)" in src
+
+        sh = self._sh()
+        assert '_rc=$?' in sh
+        assert '[ "$_rc" = "2" ]' in sh
+
+
+class TestOptIn:
+    """Nested-snapshot support must be opt-in and must never self-enable."""
+
+    def test_heredoc_gates_on_the_opt_in_flag(self):
+        src = heredoc_source()
+        assert re.search(r"nested_enabled = sys\.argv\[\d+\] == \"1\"", src)
+        assert "if not nested_enabled:" in src
+
+    def test_apply_sh_reads_the_marker_file(self):
+        with open(APPLY_SH, encoding="utf-8") as fh:
+            sh = fh.read()
+        assert 'if [ -f "$PATCH_DIR/nested_snapshots_enabled" ]' in sh
+        assert '"$_NESTED_ENABLED"' in sh
+
+    def test_patching_is_skipped_entirely_when_disabled(self):
+        # The guard-relaxing crud.py patch must be inside the enabled branch.
+        src = heredoc_source()
+        gate = src.index("if not nested_needed:")
+        crud = src.index("patch_file(crud_py, CRUD_BLOCK)")
+        assert gate < crud, "crud.py patch must sit inside the opt-in branch"
+
+    def test_disabling_REVERTS_the_patch_rather_than_merely_skipping_it(self):
+        """Skipping is not disabling.
+
+        The overlay persists for the whole boot, so a patch applied by an earlier
+        run this boot is still on disk — and middlewared re-imports it on the
+        restart install.sh performs. Without an active revert,
+        `--disable-nested-snapshots` reports "disabled" while the feature keeps
+        running until the next reboot.
+        """
+        src = heredoc_source()
+        # The implementation lives in patch/mw_patch.py (see test_mw_patch.py);
+        # apply.sh must import and actually call it.
+        assert "from mw_patch import patch_file, revert_nested" in src
+        gate = src.index("if not nested_needed:")
+        revert = src.index("reverted = revert_nested(")
+        patch = src.index("patch_file(crud_py, CRUD_BLOCK)")
+        assert gate < revert < patch, "revert belongs in the not-needed branch"
+
+    def test_import_failure_skips_the_patch_rather_than_crashing(self):
+        # apply.sh runs at PREINIT. If mw_patch.py cannot be imported it must
+        # degrade to "middlewared starts stock", never take the boot down.
+        src = heredoc_source()
+        i = src.index("from mw_patch import")
+        tail = src[i:i + 400]
+        assert "except ImportError" in tail
+        assert "skipping backend patch" in tail
+
+
+def test_guard_is_relaxed_only_after_traversal_is_installed():
+    # Ordering in apply.sh is a safety property: copy module -> patch snapshot.py
+    # -> patch sync.py -> patch crud.py. crud.py (which unlocks the feature) must
+    # come last, so a partial failure never leaves "guard removed, traversal gone".
+    src = heredoc_source()
+    order = [
+        src.index("shutil.copyfile(nested_src, nested_dst)"),
+        src.index("patch_file(snapshot_py, SNAPSHOT_BLOCK)"),
+        src.index("patch_file(sync_path, SYNC_BLOCK)"),
+        src.index("patch_file(crud_py, CRUD_BLOCK)"),
+    ]
+    assert order == sorted(order), "crud.py must be patched last"

tests/test_create_task.py

@@ -0,0 +1,107 @@
+"""Tests for create_task.py, focused on the restic repository password.
+
+That password is the encryption key for the entire cloud backup repository. It
+used to travel through `midclt call cloud_backup.create '<json>'` -- i.e. through
+the subprocess's **argv**, which is world-readable via `ps` -- and `--password`
+wrote it into the user's shell history forever.
+"""
+
+import importlib.util
+import io
+import os
+import sys
+import types
+
+import pytest
+
+SPEC = importlib.util.spec_from_file_location(
+    "create_task",
+    os.path.join(os.path.dirname(__file__), "..", "patch", "create_task.py"),
+)
+
+
+def load():
+    mod = importlib.util.module_from_spec(SPEC)
+    SPEC.loader.exec_module(mod)
+    return mod
+
+
+class Args:
+    def __init__(self, password=None, password_stdin=False):
+        self.password = password
+        self.password_stdin = password_stdin
+
+
+class TestPasswordNeverReachesArgv:
+    """The whole reason this module talks to the client library."""
+
+    def test_midclt_call_spawns_no_subprocess(self):
+        import inspect
+
+        src = inspect.getsource(load().midclt_call)
+        assert "subprocess" not in src, (
+            "shelling out to `midclt` puts cloud_backup.create's JSON -- including "
+            "the restic repo password -- into argv, which any local user can read "
+            "with ps"
+        )
+        assert "truenas_api_client" in src
+
+    def test_errors_never_echo_the_call_arguments(self):
+        # A failed cloud_backup.create must not print the body back at the user;
+        # it contains the password.
+        import inspect
+
+        src = inspect.getsource(load().midclt_call)
+        assert "{args}" not in src
+        assert "args!r" not in src
+
+
+class TestResolvePassword:
+    def test_reads_from_stdin(self, monkeypatch):
+        mod = load()
+        monkeypatch.setattr(sys, "stdin", io.StringIO("s3cret\n"))
+        assert mod._resolve_password(Args(password_stdin=True)) == "s3cret"
+
+    def test_strips_only_the_trailing_newline(self, monkeypatch):
+        # A password may legitimately contain spaces; only the line ending goes.
+        mod = load()
+        monkeypatch.setattr(sys, "stdin", io.StringIO("  pass word  \n"))
+        assert mod._resolve_password(Args(password_stdin=True)) == "  pass word  "
+
+    def test_cli_password_still_works_but_warns(self, monkeypatch, capsys):
+        mod = load()
+        pw = mod._resolve_password(Args(password="cli-secret"))
+        assert pw == "cli-secret"
+        assert "shell" in capsys.readouterr().err.lower(), "must warn about history"
+
+    def test_prompts_when_neither_flag_given(self, monkeypatch):
+        mod = load()
+        monkeypatch.setattr(
+            mod, "getpass", types.SimpleNamespace(getpass=lambda _p: "prompted")
+        )
+        assert mod._resolve_password(Args()) == "prompted"
+
+    def test_rejects_both_flags(self, monkeypatch):
+        mod = load()
+        monkeypatch.setattr(sys, "stdin", io.StringIO("x\n"))
+        with pytest.raises(SystemExit):
+            mod._resolve_password(Args(password="a", password_stdin=True))
+
+    def test_rejects_an_empty_password(self, monkeypatch):
+        # An empty restic password would silently create an unencrypted-ish repo.
+        mod = load()
+        monkeypatch.setattr(sys, "stdin", io.StringIO("\n"))
+        with pytest.raises(SystemExit):
+            mod._resolve_password(Args(password_stdin=True))
+
+
+class TestVersion:
+    def test_version_is_not_stale(self):
+        # __version__ sat at 0.2.0 through three releases because the drift check
+        # only looked at VERSION= in shell scripts. It covers this file now.
+        sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
+        from release_notes import normalise, script_versions
+
+        repo = os.path.join(os.path.dirname(__file__), "..")
+        versions = {normalise(v) for v in script_versions(repo).values()}
+        assert len(versions) == 1, f"version drift: {sorted(versions)}"

tests/test_mw_patch.py

@@ -0,0 +1,160 @@
+"""Tests for mw_patch — the single implementation of apply/revert.
+
+apply.sh and uninstall.sh both go through this. It used to be duplicated in an
+untested shell heredoc, which is exactly how the two could have drifted apart:
+apply.sh reverting one set of files and uninstall.sh another.
+"""
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "patch"))
+
+from mw_patch import (  # noqa: E402
+    MARKER,
+    NESTED_MODULE,
+    NESTED_RELPATHS,
+    PROVIDER_RELPATHS,
+    patch_file,
+    revert_all,
+    revert_nested,
+    unpatch_file,
+)
+
+STOCK = "import os\n\n\ndef stock():\n    return 1\n"
+BLOCK = "\n# TRUECLOUD_PATCH\ninjected = 1\n"
+
+
+def build_mw(root):
+    """A fake middlewared tree with every file this project touches."""
+    mw = os.path.join(root, "middlewared")
+    for rel in NESTED_RELPATHS + PROVIDER_RELPATHS:
+        path = os.path.join(mw, *rel)
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        with open(path, "w", encoding="utf-8") as fh:
+            fh.write(STOCK)
+    with open(os.path.join(mw, *NESTED_MODULE), "w", encoding="utf-8") as fh:
+        fh.write("# module\n")
+    return mw
+
+
+def read(mw, rel):
+    with open(os.path.join(mw, *rel), encoding="utf-8") as fh:
+        return fh.read()
+
+
+class TestPatchFile:
+    def test_appends_the_block(self, tmp_path):
+        p = tmp_path / "m.py"
+        p.write_text(STOCK)
+        patch_file(str(p), BLOCK)
+        assert MARKER in p.read_text()
+        assert p.read_text().startswith("import os")
+
+    def test_is_idempotent(self, tmp_path):
+        # apply.sh runs on EVERY boot. Without truncate-then-append, repeated runs
+        # would stack duplicate copies of the block into a middlewared module.
+        p = tmp_path / "m.py"
+        p.write_text(STOCK)
+        for _ in range(5):
+            patch_file(str(p), BLOCK)
+        assert p.read_text().count("# TRUECLOUD_PATCH") == 1
+        assert p.read_text().count("injected = 1") == 1
+
+    def test_round_trips_back_to_stock(self, tmp_path):
+        p = tmp_path / "m.py"
+        p.write_text(STOCK)
+        patch_file(str(p), BLOCK)
+        assert unpatch_file(str(p)) is True
+        assert p.read_text() == STOCK
+
+
+class TestUnpatchFile:
+    def test_returns_false_on_an_unpatched_file(self, tmp_path):
+        p = tmp_path / "m.py"
+        p.write_text(STOCK)
+        assert unpatch_file(str(p)) is False
+        assert p.read_text() == STOCK
+
+    def test_returns_false_on_a_missing_file(self, tmp_path):
+        assert unpatch_file(str(tmp_path / "nope.py")) is False
+
+
+class TestRevertNested:
+    def test_reverts_only_the_nested_files(self, tmp_path):
+        mw = build_mw(str(tmp_path))
+        for rel in NESTED_RELPATHS + PROVIDER_RELPATHS:
+            patch_file(os.path.join(mw, *rel), BLOCK)
+
+        reverted = revert_nested(mw)
+
+        for rel in NESTED_RELPATHS:
+            assert read(mw, rel) == STOCK, f"{rel[-1]} should be stock"
+            assert rel[-1] in reverted
+
+    def test_never_touches_the_providers_patch(self, tmp_path):
+        # restic.py carries a TRUECLOUD_PATCH block too, but it belongs to the
+        # providers module. Reverting it would silently break B2 backups.
+        mw = build_mw(str(tmp_path))
+        for rel in NESTED_RELPATHS + PROVIDER_RELPATHS:
+            patch_file(os.path.join(mw, *rel), BLOCK)
+
+        revert_nested(mw)
+
+        for rel in PROVIDER_RELPATHS:
+            assert MARKER in read(mw, rel), f"{rel[-1]} must keep its providers block"
+
+    def test_removes_the_module(self, tmp_path):
+        mw = build_mw(str(tmp_path))
+        assert os.path.exists(os.path.join(mw, *NESTED_MODULE))
+        reverted = revert_nested(mw)
+        assert not os.path.exists(os.path.join(mw, *NESTED_MODULE))
+        assert "_truecloud_nested.py" in reverted
+
+    def test_module_is_removed_before_the_files_are_unpatched(self, tmp_path):
+        # Every injected block is guarded by `if _tc_nested is not None`, so once
+        # the module is gone they all no-op — the stock guard is restored even if
+        # a later unpatch fails.
+        mw = build_mw(str(tmp_path))
+        for rel in NESTED_RELPATHS:
+            patch_file(os.path.join(mw, *rel), BLOCK)
+        reverted = revert_nested(mw)
+        assert reverted[0] == "_truecloud_nested.py"
+
+    def test_is_idempotent(self, tmp_path):
+        mw = build_mw(str(tmp_path))
+        for rel in NESTED_RELPATHS:
+            patch_file(os.path.join(mw, *rel), BLOCK)
+        revert_nested(mw)
+        assert revert_nested(mw) == []
+
+
+class TestRevertAll:
+    def test_reverts_providers_and_nested(self, tmp_path):
+        mw = build_mw(str(tmp_path))
+        for rel in NESTED_RELPATHS + PROVIDER_RELPATHS:
+            patch_file(os.path.join(mw, *rel), BLOCK)
+
+        revert_all(mw)
+
+        for rel in NESTED_RELPATHS + PROVIDER_RELPATHS:
+            assert read(mw, rel) == STOCK, f"{rel[-1]} should be stock"
+        assert not os.path.exists(os.path.join(mw, *NESTED_MODULE))
+
+    def test_is_a_noop_on_a_stock_tree(self, tmp_path):
+        mw = build_mw(str(tmp_path))
+        os.unlink(os.path.join(mw, *NESTED_MODULE))
+        assert revert_all(mw) == []
+
+
+class TestTargetsAreDisjoint:
+    def test_no_file_is_in_both_module_lists(self):
+        # If restic.py ever appeared in NESTED_RELPATHS, revert_nested would break
+        # B2 backups.
+        assert not set(NESTED_RELPATHS) & set(PROVIDER_RELPATHS)
+
+    @pytest.mark.parametrize("rel", PROVIDER_RELPATHS)
+    def test_provider_targets_are_not_nested_targets(self, rel):
+        assert rel not in NESTED_RELPATHS

tests/test_patch_ui.py

@@ -0,0 +1,171 @@
+"""Tests for the Angular bundle patch.
+
+This is the one part of the patch that edits *minified third-party JavaScript* by
+regex, so it is the easiest place to silently produce a broken bundle: a pattern
+that matches nothing leaves the dropdown Storj-only, and a pattern that matches
+sloppily can unbalance the parentheses and take the whole web UI down.
+
+Nothing checked it until now. The snippets below are verbatim from a real
+TrueNAS 25.x bundle (chunk-*.js, pre-patch).
+"""
+
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "patch"))
+
+from patch_ui import MARKER, _PATTERNS, _match_pattern  # noqa: E402
+
+# Verbatim from /usr/share/truenas/webui/chunk-FX2QXNQU.js on TrueNAS 25.10.
+# Angular emits the binding as a chained ɵɵproperty(...)(...) call, so the
+# pureFunction call is followed by TWO closing parens: one for pe(...), one for
+# property(...).
+REAL_25X = (
+    'c(2,"filterByProviders",pe(115,Rn,i.CloudSyncProviderName.Storj))'
+    '("required",!0),r(3'
+)
+
+# TrueNAS 24.x and earlier emitted a literal array.
+REAL_24X = 'c(2,"filterByProviders",["STORJ_IX"])("required",!0),r(3'
+
+
+def apply_patch(content):
+    """Run the same match-and-substitute main() does."""
+    find, replace = _match_pattern(content)
+    assert find is not None, "no pattern matched"
+    patched, count = find.subn(replace, content)
+    return patched, count
+
+
+def paren_delta(s):
+    """Net paren balance. The snippets are fragments of a minified file, so they
+    are not balanced on their own -- what must hold is that patching does not
+    CHANGE the balance. Consuming one paren too many is a syntax error in the
+    bundle, and the whole TrueNAS web UI goes blank."""
+    return s.count("(") - s.count(")")
+
+
+@pytest.mark.parametrize("source", [REAL_25X, REAL_24X], ids=["25.x", "24.x"])
+class TestAgainstRealBundles:
+    def test_matches_exactly_once(self, source):
+        # main() refuses to write unless count == 1 — more than one match would
+        # mean the pattern is too loose to trust against a minified bundle.
+        _patched, count = apply_patch(source)
+        assert count == 1
+
+    def test_result_contains_all_three_providers(self, source):
+        patched, _ = apply_patch(source)
+        assert MARKER in patched
+        assert '"filterByProviders",["STORJ_IX","S3","B2"]' in patched
+
+    def test_patch_does_not_change_paren_balance(self, source):
+        # Consuming one paren too many (or too few) is a syntax error in the
+        # bundle and the entire TrueNAS web UI goes blank. This is the invariant
+        # the 25.x pattern has to get right: it eats `pe(...)` which sits inside
+        # a chained property(...)(...) call.
+        patched, _ = apply_patch(source)
+        assert paren_delta(patched) == paren_delta(source)
+
+    def test_surrounding_code_is_untouched(self, source):
+        patched, _ = apply_patch(source)
+        assert patched.startswith("c(2,")
+        assert patched.endswith('("required",!0),r(3')
+
+    def test_patch_is_idempotent(self, source):
+        # apply.sh re-runs every boot; MARKER short-circuits an already-patched
+        # file, but the pattern must also not match its own output.
+        patched, _ = apply_patch(source)
+        find, _replace = _match_pattern(patched)
+        if find is not None:
+            # Only the 24.x literal-array pattern may still "match" — and only if
+            # it would produce the same text. Anything else means double-patching.
+            again, _ = apply_patch(patched)
+            assert again == patched, "re-patching must be a no-op"
+
+
+def test_storj_only_bundle_is_recognised():
+    assert _match_pattern(REAL_25X)[0] is not None
+
+
+def test_unrelated_javascript_is_never_touched():
+    # A pattern loose enough to hit unrelated code would corrupt the bundle.
+    for noise in (
+        'c(2,"filterByProviders",pe(115,Rn,i.SomethingElse.Storj))',
+        'c(2,"otherBinding",pe(115,Rn,i.CloudSyncProviderName.Storj))',
+        '"filterByProviders"',
+    ):
+        find, _ = _match_pattern(noise)
+        assert find is None, f"pattern must not match: {noise}"
+
+
+def test_every_pattern_is_anchored_to_filterbyproviders():
+    # Guards against a future pattern broad enough to rewrite arbitrary JS.
+    for find, _replace in _PATTERNS:
+        assert "filterByProviders" in find.pattern
+
+
+def test_patterns_compile_and_replacements_reference_group_one():
+    for find, replace in _PATTERNS:
+        assert isinstance(find, re.Pattern)
+        assert r"\1" in replace, "replacement must preserve the binding name"
+
+
+class TestCorruptionGuard:
+    """A bad pattern must never reach the bundle.
+
+    This is not hypothetical. Commit 47cdf72 shipped a pattern that consumed one
+    closing paren and emitted one, netting an extra `)`:
+
+        c(2,"filterByProviders",["STORJ_IX","S3","B2"]))("required",!0)
+                                                      ^^ syntax error
+
+    The web UI went blank. And because MARKER was then present in the file, every
+    subsequent run reported "already patched" and skipped — so the patch could not
+    heal itself, and the bundle had to be hand-restored from the backup.
+    """
+
+    # Verbatim from 47cdf72.
+    BROKEN = (
+        re.compile(r'("filterByProviders",)\w+\(\d+,\w+,\w+\.CloudSyncProviderName\.Storj\)'),
+        r'\1["STORJ_IX","S3","B2"])',
+    )
+
+    def test_the_regression_that_blanked_the_ui_is_detectable(self):
+        find, replace = self.BROKEN
+        patched, count = find.subn(replace, REAL_25X)
+        assert count == 1, "it did match — that is why it got written"
+        assert paren_delta(patched) != paren_delta(REAL_25X), (
+            "the paren balance changes; this is the signal main() now refuses on"
+        )
+
+    def test_main_refuses_to_write_an_unbalanced_bundle(self, monkeypatch, tmp_path, capsys):
+        import patch_ui
+
+        bundle = tmp_path / "chunk-TEST.js"
+        bundle.write_text(REAL_25X, encoding="utf-8")
+
+        monkeypatch.setattr(patch_ui, "WEBUI_CANDIDATES", [str(tmp_path)])
+        monkeypatch.setattr(patch_ui, "_PATTERNS", [self.BROKEN])
+
+        patch_ui.main()
+
+        out = capsys.readouterr().out
+        assert "refusing to write" in out
+        # The bundle must be byte-for-byte untouched — a broken UI is far worse
+        # than an unpatched one.
+        assert bundle.read_text(encoding="utf-8") == REAL_25X
+
+    def test_a_good_pattern_still_writes(self, monkeypatch, tmp_path):
+        import patch_ui
+
+        bundle = tmp_path / "chunk-TEST.js"
+        bundle.write_text(REAL_25X, encoding="utf-8")
+        monkeypatch.setattr(patch_ui, "WEBUI_CANDIDATES", [str(tmp_path)])
+
+        patch_ui.main()
+
+        assert MARKER in bundle.read_text(encoding="utf-8")
+        assert (tmp_path / "chunk-TEST.js.pre-truecloud-patch").exists()

tests/test_release_notes.py

@@ -0,0 +1,202 @@
+"""Tests for the release automation.
+
+The release workflow refuses to publish unless these hold, so a bad tag fails
+loudly in CI instead of shipping a release whose notes are empty, wrong, or whose
+scripts announce a different version than the tag.
+
+That last one is not hypothetical: VERSION= drifted to three different values
+across install.sh / uninstall.sh / recover.sh / apply.sh and nothing noticed.
+"""
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
+
+from release_notes import (  # noqa: E402
+    changelog_versions,
+    check,
+    extract_notes,
+    normalise,
+    script_versions,
+    significance,
+    version_tuple,
+)
+
+REPO = os.path.join(os.path.dirname(__file__), "..")
+
+SAMPLE = """\
+# Changelog
+
+## v0.3.0 — 2026-07-13
+
+### Added
+
+- the new thing
+
+## v0.2.1 — 2026-07-09
+
+### Fixed
+
+- the old thing
+
+## v0.2.0 — 2026-07-08
+
+- first
+"""
+
+
+class TestExtractNotes:
+    def test_returns_only_that_versions_body(self):
+        body = extract_notes(SAMPLE, "v0.3.0")
+        assert "the new thing" in body
+        assert "the old thing" not in body
+        # The version heading itself is dropped (GitHub renders its own title),
+        # but sub-headings like "### Added" must survive.
+        assert not body.startswith("## v")
+        assert body.startswith("### Added")
+
+    def test_stops_at_the_next_version_heading(self):
+        body = extract_notes(SAMPLE, "v0.2.1")
+        assert "the old thing" in body
+        assert "first" not in body
+
+    def test_last_section_runs_to_end_of_file(self):
+        assert "first" in extract_notes(SAMPLE, "v0.2.0")
+
+    def test_accepts_the_tag_with_or_without_the_v(self):
+        assert extract_notes(SAMPLE, "0.3.0") == extract_notes(SAMPLE, "v0.3.0")
+
+    def test_unknown_version_raises_rather_than_returning_empty(self):
+        # An empty release body is worse than a failed release.
+        with pytest.raises(KeyError, match="no section"):
+            extract_notes(SAMPLE, "v9.9.9")
+
+
+class TestChangelogVersions:
+    def test_lists_versions_newest_first(self):
+        assert changelog_versions(SAMPLE) == ["0.3.0", "0.2.1", "0.2.0"]
+
+
+class TestAgainstTheRealRepo:
+    """These run against the actual files, so drift breaks the build."""
+
+    def test_every_script_declares_a_version(self):
+        from release_notes import VERSIONED_FILES
+
+        found = script_versions(REPO)
+        missing = [f for f in VERSIONED_FILES if f not in found]
+        assert not missing, f"no VERSION= in: {missing}"
+
+    def test_all_scripts_agree_on_the_version(self):
+        versions = {normalise(v) for v in script_versions(REPO).values()}
+        assert len(versions) == 1, f"scripts disagree on version: {sorted(versions)}"
+
+    def test_the_current_version_has_a_changelog_section(self):
+        version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
+        with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
+            body = extract_notes(fh.read(), version)
+        assert body, f"CHANGELOG.md has no content for v{version}"
+
+    def test_the_current_version_is_the_newest_changelog_entry(self):
+        version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
+        with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
+            newest = changelog_versions(fh.read())[0]
+        assert newest == version, (
+            f"scripts say v{version} but the newest CHANGELOG entry is v{newest}"
+        )
+
+    def test_check_passes_for_the_current_version(self):
+        version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
+        assert check(version, REPO) == []
+
+
+class TestCheckCatchesMistakes:
+    def test_reports_a_tag_that_no_script_matches(self):
+        problems = check("v9.9.9", REPO)
+        assert problems
+        assert any("declares VERSION" in p for p in problems)
+
+    def test_reports_a_missing_changelog_section(self):
+        problems = check("v9.9.9", REPO)
+        assert any("no section" in p for p in problems)
+
+
+class TestSignificance:
+    """Drives the TrueNAS update alert: what is worth bothering a human about.
+
+    The rule: a release whose CHANGELOG only has a "### Docs" section changed no
+    code, and nobody should get an alert because a README was reworded.
+    """
+
+    TEXT = """\
+# Changelog
+
+## v0.4.2 — 2026-07-13
+
+### Docs
+
+- reworded the README
+
+## v0.4.1 — 2026-07-13
+
+### Fixed
+
+- a real bug
+
+## v0.4.0 — 2026-07-13
+
+### Added
+
+- a feature
+
+## v0.3.3 — 2026-07-13
+
+### Security
+
+- keep a password out of argv
+
+## v0.3.2 — 2026-07-13
+
+### Fixed
+
+- something
+"""
+
+    def test_docs_only_release_does_not_alert(self):
+        level, versions, _ = significance(self.TEXT, "0.4.1", "0.4.2")
+        assert level == "docs"
+        assert versions == ["0.4.2"]
+
+    def test_a_real_fix_alerts(self):
+        level, _v, _h = significance(self.TEXT, "0.4.0", "0.4.1")
+        assert level == "notable"
+
+    def test_security_in_range_escalates(self):
+        level, _v, _h = significance(self.TEXT, "0.3.2", "0.3.3")
+        assert level == "security"
+
+    def test_security_wins_even_when_the_newest_release_is_docs_only(self):
+        # A docs-only v0.4.2 sitting on top of a security-fixing v0.3.3 must still
+        # be reported as security — classify the whole span, not just the tip.
+        level, versions, _ = significance(self.TEXT, "0.3.2", "0.4.2")
+        assert level == "security"
+        assert set(versions) == {"0.3.3", "0.4.0", "0.4.1", "0.4.2"}
+
+    def test_same_version_is_never_notable(self):
+        level, versions, _ = significance(self.TEXT, "0.4.2", "0.4.2")
+        assert level == "docs"
+        assert versions == []
+
+    def test_range_is_exclusive_of_current_inclusive_of_latest(self):
+        _l, versions, _h = significance(self.TEXT, "0.4.0", "0.4.2")
+        assert "0.4.0" not in versions
+        assert "0.4.2" in versions
+
+    def test_version_tuple_orders_correctly(self):
+        assert version_tuple("v0.10.0") > version_tuple("v0.9.9")
+        assert version_tuple("0.4.2") > version_tuple("0.4.1")
+        # Pre-release suffixes are dropped, not ranked above the release.
+        assert version_tuple("v0.5.0-rc1") == version_tuple("v0.5.0")

tests/test_truecloud_nested.py

@@ -0,0 +1,605 @@
+"""Tests for nested-dataset snapshot staging.
+
+Two rules are under test above all else:
+
+1. A tree that cannot be staged completely must fail LOUDLY. A silently
+   incomplete backup is the exact failure that stock TrueNAS's "no further
+   nesting" guard exists to prevent.
+
+2. Every snapshot we cause to exist must be cleaned up. ``zfs.snapshot.delete``
+   is non-recursive by default and stock calls it with no options, so a
+   recursive snapshot would otherwise orphan one snapshot per descendant dataset
+   on EVERY run.
+"""
+
+import asyncio
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "patch"))
+
+from truecloud_nested import (  # noqa: E402
+    StagingError,
+    apply_plan,
+    cleanup_all,
+    cleanup_task,
+    current_mounts_under,
+    delete_snapshot_tree,
+    plan_staging,
+    sidecar_for,
+    snapshot_tree_names,
+    staging_root_for,
+    teardown,
+    verify_staged,
+)
+
+SNAP = "cloud_backup-5-20260712030000"
+ROOT = "/run/truecloud-nested/cloud_backup-5"
+
+
+def ds(name, mountpoint, mounted="yes"):
+    return {
+        "name": name,
+        "properties": {
+            "mountpoint": {"value": mountpoint},
+            "mounted": {"value": mounted},
+        },
+    }
+
+
+# Mirrors the real layout: apps are datasets, several with their own children.
+DATASETS = [
+    ds("Tap", "/mnt/Tap"),
+    ds("Tap/apps", "/mnt/Tap/apps"),
+    ds("Tap/apps/lidarr", "/mnt/Tap/apps/lidarr"),
+    ds("Tap/apps/lidarr/config", "/mnt/Tap/apps/lidarr/config"),
+    ds("Tap/apps/immich", "/mnt/Tap/apps/immich"),
+    ds("Tap/apps/immich/pgdata", "/mnt/Tap/apps/immich/pgdata"),
+]
+
+
+def yes(_path):
+    return "ok"
+
+
+def plan(datasets=DATASETS, base_dataset="Tap", base_mp="/mnt/Tap",
+         path="/mnt/Tap", probe=yes):
+    return plan_staging(base_dataset, base_mp, path, SNAP, datasets, ROOT, probe=probe)
+
+
+class TestPlanStaging:
+    def test_stages_every_descendant_dataset(self):
+        mounts, skipped = plan()
+        assert skipped == []
+        assert len(mounts) == 6  # root + 5 descendants
+        assert mounts[0] == (f"/mnt/Tap/.zfs/snapshot/{SNAP}", ROOT)
+
+        by_target = {t: s for s, t in mounts}
+        assert by_target[f"{ROOT}/apps"] == f"/mnt/Tap/apps/.zfs/snapshot/{SNAP}"
+        assert by_target[f"{ROOT}/apps/immich/pgdata"] == (
+            f"/mnt/Tap/apps/immich/pgdata/.zfs/snapshot/{SNAP}"
+        )
+
+    def test_parents_are_mounted_before_children(self):
+        # A child's mountpoint dir only exists inside its parent's snapshot, so
+        # mounting a child first would fail.
+        mounts, _ = plan()
+        seen = set()
+        for _src, target in mounts:
+            if target != ROOT:
+                assert os.path.dirname(target) in seen
+            seen.add(target)
+
+    def test_backup_path_below_dataset_root(self):
+        mounts, _ = plan(base_dataset="Tap/apps", base_mp="/mnt/Tap/apps",
+                         path="/mnt/Tap/apps")
+        assert mounts[0] == (f"/mnt/Tap/apps/.zfs/snapshot/{SNAP}", ROOT)
+        targets = [t for _s, t in mounts]
+        assert f"{ROOT}/lidarr" in targets
+        assert f"{ROOT}/apps/lidarr" not in targets
+
+    def test_base_dataset_is_not_a_descendant_of_itself(self):
+        mounts, _ = plan(datasets=[ds("Tap", "/mnt/Tap")])
+        assert len(mounts) == 1
+
+
+class TestScoping:
+    def test_unrelated_datasets_are_ignored_silently(self):
+        # Regression: scoping by mountpoint first dragged in every
+        # mountpoint-less dataset on the box (all of Tank/.system/*), burying the
+        # warnings that actually matter.
+        noisy = DATASETS + [
+            ds("Tank/.system", "none"),
+            ds("Tank/.system/cores", "legacy"),
+            ds("Tank/backups", "/mnt/Tank/backups"),
+        ]
+        mounts, skipped = plan(datasets=noisy)
+        assert len(mounts) == 6
+        assert skipped == [], "datasets outside the base dataset must not be reported"
+
+    def test_in_scope_dataset_without_mountpoint_is_reported(self):
+        datasets = DATASETS + [ds("Tap/apps/weird", "none")]
+        _mounts, skipped = plan(datasets=datasets)
+        assert ("Tap/apps/weird", "mountpoint is none") in skipped
+
+    def test_unmounted_dataset_is_skipped_but_never_silently(self):
+        datasets = DATASETS + [ds("Tap/apps/vault", "/mnt/Tap/apps/vault", mounted="no")]
+        mounts, skipped = plan(datasets=datasets)
+        assert f"{ROOT}/apps/vault" not in [t for _s, t in mounts]
+        assert ("Tap/apps/vault", "dataset is not mounted (locked/encrypted?)") in skipped
+
+    def test_descendant_mounted_outside_the_path_is_not_an_omission(self):
+        datasets = DATASETS + [ds("Tap/elsewhere", "/mnt/other")]
+        mounts, skipped = plan(datasets=datasets)
+        assert len(mounts) == 6
+        assert skipped == []
+
+
+class TestSilentOmissionGuard:
+    """The whole point of the feature. These are the tests that matter."""
+
+    @staticmethod
+    def _missing_pgdata(path):
+        return "missing" if "/mnt/Tap/apps/immich/pgdata/" in path else "ok"
+
+    @staticmethod
+    def _denied_pgdata(path):
+        if "/mnt/Tap/apps/immich/pgdata/" in path:
+            return "cannot be read (Permission denied)"
+        return "ok"
+
+    def test_missing_snapshot_on_descendant_raises(self):
+        with pytest.raises(StagingError, match="incomplete tree"):
+            plan(probe=self._missing_pgdata)
+
+    def test_error_names_the_offending_dataset(self):
+        with pytest.raises(StagingError, match="Tap/apps/immich/pgdata"):
+            plan(probe=self._missing_pgdata)
+
+    def test_missing_and_unreadable_are_reported_differently(self):
+        # os.path.isdir() collapses both into False, which would report a
+        # permission problem as "has no snapshot" and send you hunting for a
+        # snapshot that is sitting right there. Both abort -- but say which.
+        with pytest.raises(StagingError, match="has no snapshot"):
+            plan(probe=self._missing_pgdata)
+        with pytest.raises(StagingError, match="Permission denied"):
+            plan(probe=self._denied_pgdata)
+
+
+class TestSnapshotTreeNames:
+    """zfs.snapshot.delete is non-recursive; we must sweep children ourselves."""
+
+    ALL = [
+        "Tap@cloud_backup-5-20260712030000",
+        "Tap/apps@cloud_backup-5-20260712030000",
+        "Tap/apps/lidarr/config@cloud_backup-5-20260712030000",
+        "Tap@auto-2026-07-12_03-00",            # unrelated periodic snapshot
+        "Tap/apps@cloud_backup-9-20260712030000",  # another task
+        "Tank/backups@cloud_backup-5-20260712030000",  # different pool
+    ]
+
+    def test_returns_parent_and_all_children(self):
+        got = snapshot_tree_names("Tap@cloud_backup-5-20260712030000", self.ALL)
+        assert set(got) == {
+            "Tap@cloud_backup-5-20260712030000",
+            "Tap/apps@cloud_backup-5-20260712030000",
+            "Tap/apps/lidarr/config@cloud_backup-5-20260712030000",
+        }
+
+    def test_never_touches_periodic_or_other_tasks_or_other_pools(self):
+        got = snapshot_tree_names("Tap@cloud_backup-5-20260712030000", self.ALL)
+        assert "Tap@auto-2026-07-12_03-00" not in got
+        assert "Tap/apps@cloud_backup-9-20260712030000" not in got
+        assert "Tank/backups@cloud_backup-5-20260712030000" not in got
+
+    def test_malformed_snapshot_name_yields_nothing(self):
+        assert snapshot_tree_names("Tap", self.ALL) == []
+
+
+class FakeMiddleware:
+    def __init__(self, snapshots=None):
+        self.snapshots = list(snapshots or [])
+        self.calls = []
+        self.logger = None
+
+    async def call(self, method, *args):
+        self.calls.append((method, args))
+        if method == "zfs.snapshot.query":
+            return [{"name": n} for n in self.snapshots]
+        if method == "zfs.snapshot.delete":
+            name = args[0]
+            opts = args[1] if len(args) > 1 else {}
+            if name not in self.snapshots:
+                raise RuntimeError("does not exist")
+            if opts.get("recursive"):
+                # Real `zfs destroy -r` takes the parent and every child snapshot.
+                for n in snapshot_tree_names(name, list(self.snapshots)):
+                    self.snapshots.remove(n)
+            else:
+                self.snapshots.remove(name)
+            return True
+        raise AssertionError(f"unexpected call {method}")
+
+    async def run_in_thread(self, fn, *args):
+        return fn(*args)
+
+
+class TestDeleteSnapshotTree:
+    def test_deletes_parent_and_every_child(self):
+        mw = FakeMiddleware([
+            "Tap@snap", "Tap/apps@snap", "Tap/apps/lidarr@snap", "Tap@keepme",
+        ])
+        asyncio.run(delete_snapshot_tree(mw, "Tap@snap"))
+        assert mw.snapshots == ["Tap@keepme"]
+
+    def test_is_idempotent_when_stock_already_removed_the_parent(self):
+        # Stock's finally can win the race once our mounts are released.
+        mw = FakeMiddleware(["Tap/apps@snap", "Tap/apps/lidarr@snap"])
+        asyncio.run(delete_snapshot_tree(mw, "Tap@snap"))
+        assert mw.snapshots == []
+
+    def test_uses_a_single_recursive_delete_not_252_individual_ones(self):
+        # 252 sequential deletes are slow AND not atomic: a run killed part-way
+        # through leaves exactly the orphans this function exists to prevent.
+        mw = FakeMiddleware(["Tap@snap", "Tap/apps@snap", "Tap/apps/lidarr@snap"])
+        asyncio.run(delete_snapshot_tree(mw, "Tap@snap"))
+        assert mw.snapshots == []
+        deletes = [a for m, a in mw.calls if m == "zfs.snapshot.delete"]
+        assert len(deletes) == 1, "should be ONE recursive call, not one per snapshot"
+        assert deletes[0][1] == {"recursive": True}
+        assert not [m for m, _a in mw.calls if m == "zfs.snapshot.query"], (
+            "no enumeration needed on the fast path"
+        )
+
+    def test_survives_recursive_and_query_failure_by_deleting_the_parent(self):
+        class Broken(FakeMiddleware):
+            async def call(self, method, *args):
+                if method == "zfs.snapshot.query":
+                    raise RuntimeError("boom")
+                if method == "zfs.snapshot.delete" and len(args) > 1:
+                    raise RuntimeError("recursive delete unavailable")
+                return await super().call(method, *args)
+
+        mw = Broken(["Tap@snap"])
+        asyncio.run(delete_snapshot_tree(mw, "Tap@snap"))
+        assert mw.snapshots == []
+
+    def test_leaves_unrelated_snapshots_alone_when_the_tree_is_gone(self):
+        mw = FakeMiddleware(["Tap@unrelated"])
+        asyncio.run(delete_snapshot_tree(mw, "Tap@snap"))
+        assert mw.snapshots == ["Tap@unrelated"]
+
+
+class TestStageNestedOrdering:
+    def test_sidecar_is_written_before_anything_is_mounted(self, tmp_path, monkeypatch):
+        # middlewared can die at any moment. If the snapshot were recorded only
+        # after apply_plan, a crash in that window would orphan a 160-snapshot
+        # tree -- the precise failure the sidecar exists to prevent.
+        import truecloud_nested as tn
+
+        monkeypatch.setattr(tn, "STAGING_BASE", str(tmp_path))
+        order = []
+
+        class Recorder(FakeMiddleware):
+            async def run_in_thread(self, fn, *args):
+                order.append(fn.__name__)
+                if fn.__name__ == "plan_staging":
+                    return ([("/src", str(tmp_path / "cloud_backup-5"))], [])
+                if fn.__name__ in ("apply_plan", "verify_staged", "teardown"):
+                    return [] if fn.__name__ == "teardown" else True
+                return fn(*args)
+
+        asyncio.run(tn.stage_nested(
+            Recorder(), "/mnt/Tap", "Tap@snap", "Tap", "/mnt/Tap",
+            "cloud_backup-5", DATASETS,
+        ))
+
+        assert order.index("_write_sidecar") < order.index("apply_plan")
+
+    def test_reclaims_the_snapshot_tree_left_by_a_crashed_run(self, tmp_path,
+                                                              monkeypatch):
+        # teardown() reclaims the crashed run's MOUNTS, but nothing else would
+        # ever reclaim its SNAPSHOTS -- and we are about to overwrite the only
+        # record of them. One crash would orphan 160+ snapshots permanently.
+        import truecloud_nested as tn
+
+        monkeypatch.setattr(tn, "STAGING_BASE", str(tmp_path))
+        root = tn.staging_root_for("cloud_backup-5")
+        os.makedirs(os.path.dirname(root), exist_ok=True)
+        with open(sidecar_for(root), "w", encoding="utf-8") as fh:
+            fh.write("Tap@old-crashed-run")
+
+        mw = FakeMiddleware(["Tap@old-crashed-run", "Tap/apps@old-crashed-run"])
+
+        class Stub(FakeMiddleware):
+            def __init__(self, inner):
+                super().__init__()
+                self.inner = inner
+
+            async def call(self, method, *args):
+                return await self.inner.call(method, *args)
+
+            async def run_in_thread(self, fn, *args):
+                if fn.__name__ == "plan_staging":
+                    return ([("/src", root)], [])
+                if fn.__name__ == "teardown":
+                    return []
+                if fn.__name__ in ("apply_plan", "verify_staged"):
+                    return True
+                return fn(*args)
+
+        asyncio.run(tn.stage_nested(
+            Stub(mw), "/mnt/Tap", "Tap@new", "Tap", "/mnt/Tap",
+            "cloud_backup-5", DATASETS,
+        ))
+
+        assert mw.snapshots == [], "the crashed run's snapshot tree must be reclaimed"
+
+    def test_sidecar_is_removed_when_staging_fails(self, tmp_path, monkeypatch):
+        import truecloud_nested as tn
+
+        monkeypatch.setattr(tn, "STAGING_BASE", str(tmp_path))
+        root = tn.staging_root_for("cloud_backup-5")
+
+        class Failing(FakeMiddleware):
+            async def run_in_thread(self, fn, *args):
+                if fn.__name__ == "plan_staging":
+                    raise StagingError("boom")
+                return fn(*args)
+
+        with pytest.raises(StagingError):
+            asyncio.run(tn.stage_nested(
+                Failing(), "/mnt/Tap", "Tap@snap", "Tap", "/mnt/Tap",
+                "cloud_backup-5", DATASETS,
+            ))
+
+        assert not os.path.exists(sidecar_for(root))
+
+
+class TestCleanupTask:
+    def test_recovers_snapshot_from_sidecar_after_middlewared_restart(self, tmp_path,
+                                                                      monkeypatch):
+        # The sidecar is the ONLY record of the pinned snapshot, precisely so a
+        # middlewared restart cannot orphan the tree.
+        import truecloud_nested as tn
+
+        monkeypatch.setattr(tn, "STAGING_BASE", str(tmp_path))
+        root = tn.staging_root_for("cloud_backup-5", base=str(tmp_path))
+        os.makedirs(root, exist_ok=True)
+        with open(sidecar_for(root), "w", encoding="utf-8") as fh:
+            fh.write("Tap@snap")
+
+        mw = FakeMiddleware(["Tap@snap", "Tap/apps@snap"])
+        monkeypatch.setattr(tn, "teardown", lambda *_a, **_k: [])
+
+        asyncio.run(cleanup_task(mw, "cloud_backup-5"))
+
+        assert mw.snapshots == []
+        assert not os.path.exists(sidecar_for(root))
+
+    def test_is_a_noop_when_never_staged(self, tmp_path, monkeypatch):
+        import truecloud_nested as tn
+
+        monkeypatch.setattr(tn, "STAGING_BASE", str(tmp_path / "nope"))
+        mw = FakeMiddleware(["Tap@snap"])
+        asyncio.run(cleanup_task(mw, "cloud_backup-5"))
+        assert mw.calls == []
+        assert mw.snapshots == ["Tap@snap"]
+
+
+class TestVerifyStaged:
+    """Anti-regression guard: proves the staged tree is real before we back it up."""
+
+    def test_passes_when_every_target_is_mounted_and_root_non_empty(self):
+        mounts = [("/src", ROOT), ("/src/a", f"{ROOT}/a")]
+        assert verify_staged(mounts, ismount=lambda p: True, listdir=lambda p: ["apps"])
+
+    def test_raises_when_a_target_is_not_actually_mounted(self):
+        # This is the case that would produce a silently-empty backup.
+        mounts = [("/src", ROOT), ("/src/a", f"{ROOT}/a")]
+        with pytest.raises(StagingError, match="not a mountpoint"):
+            verify_staged(mounts, ismount=lambda p: p == ROOT, listdir=lambda p: ["apps"])
+
+    def test_raises_when_staging_root_is_empty(self):
+        with pytest.raises(StagingError, match="empty"):
+            verify_staged([("/src", ROOT)], ismount=lambda p: True, listdir=lambda p: [])
+
+    def test_raises_on_empty_plan(self):
+        with pytest.raises(StagingError):
+            verify_staged([])
+
+
+class FakeRunner:
+    def __init__(self, fail_on=None):
+        self.fail_on = fail_on
+        self.calls = []
+
+    def __call__(self, cmd):
+        self.calls.append(cmd)
+
+        class R:
+            returncode = 0
+            stderr = ""
+
+        if self.fail_on and cmd[0] == "mount" and cmd[2] == self.fail_on:
+            R.returncode = 32
+            R.stderr = "mount failed"
+        return R
+
+
+class TestApplyPlanRollback:
+    def test_rolls_back_mounts_when_one_fails(self, tmp_path):
+        # A half-built tree must never reach the backup tool.
+        root = str(tmp_path / "root")
+        mounts = [("/src", root), ("/src/a", root + "/a"), ("/src/b", root + "/b")]
+
+        runner = FakeRunner(fail_on="/src/b")
+        with pytest.raises(StagingError, match="bind-mount"):
+            apply_plan(mounts, runner=runner, isdir=lambda _p: True)
+
+        umounts = [c[-1] for c in runner.calls if c[0] == "umount"]
+        assert umounts == [root + "/a", root]
+
+    def test_raises_when_target_missing(self, tmp_path):
+        root = str(tmp_path / "root")
+        with pytest.raises(StagingError, match="does not exist"):
+            apply_plan(
+                [("/src", root), ("/src/a", root + "/a")],
+                runner=FakeRunner(),
+                isdir=lambda p: p == root,
+            )
+
+
+class TestTeardown:
+    def test_unmounts_deepest_first(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text(
+            f"tmpfs {ROOT} tmpfs rw 0 0\n"
+            f"tmpfs {ROOT}/apps tmpfs rw 0 0\n"
+            f"tmpfs {ROOT}/apps/lidarr/config tmpfs rw 0 0\n"
+            f"tmpfs {ROOT}/apps/lidarr tmpfs rw 0 0\n"
+            "tmpfs /somewhere/else tmpfs rw 0 0\n"
+        )
+        runner = FakeRunner()
+        teardown(ROOT, runner=runner, mounts_file=str(mounts_file))
+
+        order = [c[-1] for c in runner.calls if c[0] == "umount"]
+        assert order == [
+            f"{ROOT}/apps/lidarr/config",
+            f"{ROOT}/apps/lidarr",
+            f"{ROOT}/apps",
+            ROOT,
+        ]
+        assert "/somewhere/else" not in order
+
+    def test_is_idempotent_when_nothing_mounted(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text("tmpfs /somewhere/else tmpfs rw 0 0\n")
+        runner = FakeRunner()
+        assert teardown(ROOT, runner=runner, mounts_file=str(mounts_file)) == []
+        assert runner.calls == []
+
+    def test_falls_back_to_lazy_umount(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text(f"tmpfs {ROOT} tmpfs rw 0 0\n")
+
+        class Busy(FakeRunner):
+            def __call__(self, cmd):
+                self.calls.append(cmd)
+
+                class R:
+                    returncode = 0 if "-l" in cmd else 32
+                    stderr = "target is busy"
+
+                return R
+
+        runner = Busy()
+        assert teardown(ROOT, runner=runner, mounts_file=str(mounts_file)) == []
+        assert ["umount", "-l", ROOT] in runner.calls
+
+
+class TestCleanupAll:
+    """uninstall.sh and recover.sh call this instead of reimplementing teardown."""
+
+    def test_reports_orphan_snapshots_before_deleting_their_sidecars(self, tmp_path):
+        # The sidecar is the only record that an interrupted run's snapshot tree
+        # is still on disk. Deleting it without naming the snapshot orphans the
+        # whole tree silently.
+        base = tmp_path / "stage"
+        base.mkdir()
+        (base / "cloud_backup-5.snapshot").write_text("Tap@interrupted")
+
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text("")
+
+        lines, errors = cleanup_all(
+            base=str(base), runner=FakeRunner(), mounts_file=str(mounts_file)
+        )
+        assert errors == []
+        assert any("Tap@interrupted" in ln for ln in lines)
+        assert any("zfs destroy -r" in ln for ln in lines)
+        # Sidecar cleared only after being reported.
+        assert not (base / "cloud_backup-5.snapshot").exists()
+
+    def test_unmounts_everything_under_the_base_deepest_first(self, tmp_path):
+        base = tmp_path / "stage"
+        base.mkdir()
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text(
+            f"tmpfs {base} tmpfs rw 0 0\n"
+            f"tmpfs {base}/cloud_backup-5 tmpfs rw 0 0\n"
+            f"tmpfs {base}/cloud_backup-5/apps tmpfs rw 0 0\n"
+        )
+        runner = FakeRunner()
+        _lines, errors = cleanup_all(
+            base=str(base), runner=runner, mounts_file=str(mounts_file)
+        )
+        assert errors == []
+        order = [c[-1] for c in runner.calls if c[0] == "umount"]
+        assert order == [
+            f"{base}/cloud_backup-5/apps",
+            f"{base}/cloud_backup-5",
+            str(base),
+        ]
+
+    def test_keeps_sidecars_when_an_unmount_failed(self, tmp_path):
+        # If a mount is stuck, the snapshot is still pinned — so the record of it
+        # must survive for the next run (or the operator) to act on.
+        base = tmp_path / "stage"
+        base.mkdir()
+        (base / "cloud_backup-5.snapshot").write_text("Tap@stuck")
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text(f"tmpfs {base}/cloud_backup-5 tmpfs rw 0 0\n")
+
+        class Stuck(FakeRunner):
+            def __call__(self, cmd):
+                self.calls.append(cmd)
+
+                class R:
+                    returncode = 32
+                    stderr = "target is busy"
+
+                return R
+
+        _lines, errors = cleanup_all(
+            base=str(base), runner=Stuck(), mounts_file=str(mounts_file)
+        )
+        assert errors, "a stuck unmount must be reported"
+        assert (base / "cloud_backup-5.snapshot").exists()
+
+    def test_is_a_noop_on_a_clean_system(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        mounts_file.write_text("")
+        lines, errors = cleanup_all(
+            base=str(tmp_path / "absent"), runner=FakeRunner(),
+            mounts_file=str(mounts_file),
+        )
+        assert errors == []
+        assert lines == ["  None active."]
+
+
+class TestCurrentMountsUnder:
+    def test_matches_only_the_staging_subtree(self, tmp_path):
+        mounts_file = tmp_path / "mounts"
+        # "cloud_backup-50" must NOT match "cloud_backup-5".
+        mounts_file.write_text(
+            f"tmpfs {ROOT} tmpfs rw 0 0\n"
+            "tmpfs /run/truecloud-nested/cloud_backup-50 tmpfs rw 0 0\n"
+        )
+        assert current_mounts_under(ROOT, mounts_file=str(mounts_file)) == [ROOT]
+
+
+class TestStagingRootFor:
+    def test_stable_per_task(self):
+        assert staging_root_for("cloud_backup-5") == "/run/truecloud-nested/cloud_backup-5"
+
+    def test_sanitises_path_separators(self):
+        assert "/" not in staging_root_for("evil/name").rsplit("/", 1)[-1]
+
+    @pytest.mark.parametrize("name", ["..", ".", "...", "/", ""])
+    def test_dot_components_cannot_escape_the_staging_base(self, name):
+        # os.path.join(BASE, "..") normalises to /run — teardown would rmdir it.
+        root = staging_root_for(name)
+        assert os.path.normpath(root).startswith("/run/truecloud-nested/")

tools/release_notes.py

@@ -0,0 +1,219 @@
+#!/usr/bin/env python3
+"""Extract one version's section from CHANGELOG.md, and check version consistency.
+
+Used by .github/workflows/release.yml so a release's body is always the changelog
+entry -- there is no second place to write release notes, and therefore no second
+place for them to be wrong.
+
+    python3 tools/release_notes.py notes v0.3.0     # -> the section body
+    python3 tools/release_notes.py version          # -> version per the scripts
+    python3 tools/release_notes.py check v0.3.0     # -> exit 1 on any mismatch
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+CHANGELOG = os.path.join(ROOT, "CHANGELOG.md")
+
+# Everything that announces a version must agree with everything else. They drifted
+# to three different values once (0.0.4 / 0.2.1) before anything checked them --
+# and create_task.py's __version__ then sat at 0.2.0 through three more releases,
+# because the first version of this check only looked at VERSION= in shell scripts.
+VERSIONED_FILES = [
+    "install.sh",
+    "uninstall.sh",
+    "recover.sh",
+    os.path.join("patch", "apply.sh"),
+    os.path.join("patch", "create_task.py"),   # exposes `--version` to users
+    "update.sh",
+]
+
+# `VERSION="x"` (shell) or `__version__ = "x"` (python).
+_VERSION_RE = re.compile(r'^(?:VERSION=|__version__\s*=\s*)"([^"]+)"', re.M)
+_HEADING_RE = re.compile(r"^##\s+v?(\d+\.\d+\.\d+[^\s]*)", re.M)
+
+
+def normalise(v: str) -> str:
+    return v.strip().lstrip("v")
+
+
+def script_versions(root: str = ROOT) -> dict[str, str]:
+    """VERSION= as declared by each script."""
+    found = {}
+    for rel in VERSIONED_FILES:
+        path = os.path.join(root, rel)
+        try:
+            with open(path, encoding="utf-8") as fh:
+                m = _VERSION_RE.search(fh.read())
+        except OSError:
+            continue
+        if m:
+            found[rel] = m.group(1)
+    return found
+
+
+def changelog_versions(text: str) -> list[str]:
+    """Versions with a section in the changelog, newest first."""
+    return [normalise(v) for v in _HEADING_RE.findall(text)]
+
+
+def extract_notes(text: str, version: str) -> str:
+    """The body of one version's section, without its heading.
+
+    Raises KeyError if the version has no section -- a release with an empty or
+    wrong body is worse than a failed release.
+    """
+    want = normalise(version)
+    lines = text.splitlines()
+
+    start = None
+    for i, line in enumerate(lines):
+        m = _HEADING_RE.match(line)
+        if m and normalise(m.group(1)) == want:
+            start = i + 1
+            break
+    if start is None:
+        raise KeyError(f"CHANGELOG.md has no section for v{want}")
+
+    end = len(lines)
+    for i in range(start, len(lines)):
+        if _HEADING_RE.match(lines[i]):
+            end = i
+            break
+
+    return "\n".join(lines[start:end]).strip()
+
+
+# ── significance ──────────────────────────────────────────────────────────────
+# Used by the TrueNAS update alert to decide whether a release is worth bothering
+# anyone about. The CHANGELOG's own section headings are the signal: a release that
+# only has "### Docs" changed no code, and nobody should get an alert for a README.
+
+_SECTION_RE = re.compile(r"^###\s+(.+?)\s*$", re.M)
+
+#: Headings that mean "nothing about the running system changed".
+QUIET_SECTIONS = {"docs", "documentation"}
+
+
+def version_tuple(v: str) -> tuple:
+    """Sortable version. Pre-release suffixes are dropped, not ranked."""
+    return tuple(int(x) for x in normalise(v).split("-")[0].split("."))
+
+
+def section_headings(body: str) -> list[str]:
+    """The `### ...` headings inside one version's body, lowercased."""
+    return [h.strip().lower() for h in _SECTION_RE.findall(body)]
+
+
+def significance(text: str, current: str, latest: str):
+    """How much does upgrading `current` -> `latest` actually matter?
+
+    Returns ``(level, versions, headings)`` where level is one of:
+
+      "security"  a release in the range has a Security section  -> alert loudly
+      "notable"   something about the system changed             -> alert quietly
+      "docs"      only documentation changed                     -> DO NOT alert
+
+    Considers every release in the range, not just the newest: a docs-only v0.4.2
+    on top of a security-fixing v0.4.1 must still be reported as security.
+    """
+    cur, lat = version_tuple(current), version_tuple(latest)
+
+    versions = [
+        v for v in changelog_versions(text)
+        if cur < version_tuple(v) <= lat
+    ]
+
+    headings = []
+    for v in versions:
+        try:
+            headings.extend(section_headings(extract_notes(text, v)))
+        except KeyError:
+            continue
+
+    if any(h.startswith("security") for h in headings):
+        return "security", versions, headings
+    if [h for h in headings if h not in QUIET_SECTIONS]:
+        return "notable", versions, headings
+    return "docs", versions, headings
+
+
+def check(version: str, root: str = ROOT) -> list[str]:
+    """Every reason this version is not releasable. Empty list means it is."""
+    want = normalise(version)
+    problems = []
+
+    versions = script_versions(root)
+    for rel, got in sorted(versions.items()):
+        if normalise(got) != want:
+            problems.append(f"{rel} declares VERSION={got!r}, tag is v{want}")
+    missing = [r for r in VERSIONED_FILES if r not in versions]
+    for rel in missing:
+        problems.append(f"{rel} has no VERSION= line")
+
+    try:
+        with open(os.path.join(root, "CHANGELOG.md"), encoding="utf-8") as fh:
+            text = fh.read()
+    except OSError as e:
+        problems.append(f"cannot read CHANGELOG.md: {e}")
+        return problems
+
+    try:
+        body = extract_notes(text, want)
+    except KeyError as e:
+        problems.append(str(e))
+    else:
+        if not body:
+            problems.append(f"CHANGELOG.md section for v{want} is empty")
+
+    return problems
+
+
+def main(argv):
+    if len(argv) < 2:
+        print(__doc__, file=sys.stderr)
+        return 2
+
+    cmd = argv[1]
+
+    if cmd == "version":
+        versions = set(map(normalise, script_versions().values()))
+        if len(versions) != 1:
+            print(f"scripts disagree on version: {sorted(versions)}", file=sys.stderr)
+            return 1
+        print(versions.pop())
+        return 0
+
+    if len(argv) < 3:
+        print(f"usage: {argv[0]} {cmd} <version>", file=sys.stderr)
+        return 2
+    version = argv[2]
+
+    if cmd == "notes":
+        # An explicit path lets update.sh show the notes from the CHANGELOG of the
+        # version it is about to install (`git show <tag>:CHANGELOG.md`), not the
+        # one already checked out.
+        path = argv[3] if len(argv) > 3 else CHANGELOG
+        with open(path, encoding="utf-8") as fh:
+            print(extract_notes(fh.read(), version))
+        return 0
+
+    if cmd == "check":
+        problems = check(version)
+        for p in problems:
+            print(f"::error::{p}")
+        if problems:
+            return 1
+        print(f"v{normalise(version)} is consistent across scripts and CHANGELOG")
+        return 0
+
+    print(f"unknown command: {cmd}", file=sys.stderr)
+    return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

uninstall.sh

@@ -3,7 +3,7 @@
 
 set -euo pipefail
 
-VERSION="0.0.4"
+VERSION="0.5.1"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
@@ -91,6 +91,41 @@ if [ "$_ov_found" -eq 0 ]; then
 fi
 echo ""
 
+# ── Revert file-level patches ─────────────────────────────────────────────────
+# Unmounting the overlay is what normally reverts everything — the lower layer is
+# the untouched /usr. But apply.sh only mounts an overlay when the directory is
+# read-only; on a writable /usr it patches the real files in place. Uninstall
+# would then remove the boot hook and report success while leaving every patch
+# applied. Strip our appended blocks explicitly.
+
+# Same implementation apply.sh uses (patch/mw_patch.py) — a second shell copy of
+# this would be the untested one.
+echo "Reverting any file-level patches ..."
+python3 "$PATCH_DIR/patch/mw_patch.py" revert-all || \
+    echo "  WARNING: could not revert file-level patches."
+echo ""
+
+# ── Unmount nested-snapshot staging trees ─────────────────────────────────────
+# These bind mounts pin their ZFS snapshots, so they must go before anything
+# tries to destroy those snapshots. Deepest first.
+
+# Delegated to the patch module rather than reimplemented here: the depth
+# ordering and lazy-umount fallback are fiddly, and a shell copy would be the
+# untested one.
+echo "Unmounting nested-snapshot staging trees (if any) ..."
+if ! python3 "$PATCH_DIR/patch/truecloud_nested.py" cleanup; then
+    echo "  WARNING: staging mounts remain. Unmount them manually; until you do,"
+    echo "           the ZFS snapshots they pin cannot be destroyed."
+fi
+
+# The opt-in marker lives in the repo dir; remove it so a later re-install
+# starts from the safe default (feature off).
+if [ -f "$PATCH_DIR/nested_snapshots_enabled" ]; then
+    rm -f "$PATCH_DIR/nested_snapshots_enabled"
+    echo "  Removed nested-snapshot opt-in marker."
+fi
+echo ""
+
 if [ "$_restore_failed" -eq 1 ]; then
     echo ""
     echo "ERROR: One or more UI bundle backups could not be restored." >&2

update.sh

@@ -0,0 +1,293 @@
+#!/bin/bash
+# update.sh — fetch a newer release of truecloud-patch and apply it.
+#
+# ── RUN THIS BY HAND. NEVER FROM CRON OR A SYSTEMD TIMER. ─────────────────────
+#
+# This patch injects Python into middlewared and re-applies itself at every boot.
+# An unattended pull would let any bad upstream commit reach your box with no
+# human in the loop, and take effect on the next reboot. That is not theoretical:
+# v0.0.4 shipped a boot-time bug that took every app on the box down.
+#
+# The manual step IS the safety gate. Keep it.
+#
+# By default this updates to the newest RELEASE TAG, not to main. main can be
+# mid-refactor; a tag is the tested artifact. Use --main only if you know why.
+#
+#   bash update.sh              # to the newest release, with a confirmation
+#   bash update.sh --check      # show what would happen; change nothing
+#   bash update.sh --rollback   # undo the last update
+
+set -euo pipefail
+
+VERSION="0.5.1"
+
+PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
+_PREV_FILE="$PATCH_DIR/.update_previous"
+
+_target=""
+_use_main=0
+_assume_yes=0
+_check_only=0
+_rollback=0
+
+usage() {
+    cat <<USAGE
+Usage: bash update.sh [options]
+
+Options:
+  --to <ref>    Update to a specific tag or commit (default: newest release tag)
+  --main        Update to origin/main — UNRELEASED code, no guarantees
+  --check       Show what an update would do and exit; changes nothing
+  --rollback    Return to the revision recorded before the last update
+  --yes, -y     Skip the confirmation prompt
+  -h, --help    Show this help
+
+Updating preserves your nested-snapshot opt-in setting either way.
+USAGE
+}
+
+# An UNTRACKED file that the target tracks makes `git checkout` abort. The dirty-
+# tree check deliberately ignores untracked files, so this slips past it and the
+# checkout then dies mid-operation. Not hypothetical: a hand-copied
+# patch/wait_restart.sh blocked a pull on a real box exactly this way.
+#
+# Used by BOTH the update and the rollback path -- rolling back moves the tree too,
+# and would hit the identical failure.
+_abort_if_untracked_blockers() {
+    local ref="$1" blocking
+
+    # Set intersection of {untracked, not ignored} and {tracked by the target}. Two
+    # git calls, not one `ls-files --error-unmatch` per file in the target tree.
+    # --exclude-standard is deliberate: git silently overwrites *ignored* files on
+    # checkout, so those are not blockers — only untracked-and-not-ignored ones are.
+    blocking="$(comm -12 \
+        <(git ls-files --others --exclude-standard | sort) \
+        <(git ls-tree -r --name-only "$ref" | sort) \
+        | sed 's/^/    /')"
+
+    [ -n "$blocking" ] || return 0
+
+    echo "ERROR: these untracked files would be overwritten:" >&2
+    printf '%s\n\n' "$blocking" >&2
+    echo "  They exist here but git does not track them — most likely hand-copied" >&2
+    echo "  or scp'd in. Move or delete them, then re-run." >&2
+
+    # "Delete update.sh, then re-run update.sh" is impossible. If the script itself
+    # is a blocker, it was hand-copied in to bootstrap; the honest answer is to
+    # bootstrap with git instead, which installs it properly.
+    case "$blocking" in
+        *update.sh*)
+            echo "" >&2
+            echo "  update.sh itself is untracked here — you copied it in to bootstrap." >&2
+            echo "  Do that with git instead, once; it installs update.sh properly:" >&2
+            echo "" >&2
+            echo "    rm -f $PATCH_DIR/update.sh" >&2
+            echo "    git -C $PATCH_DIR checkout $ref" >&2
+            echo "    bash $PATCH_DIR/install.sh" >&2
+            echo "" >&2
+            echo "  Every later update is then just: bash update.sh" >&2
+            ;;
+    esac
+    exit 1
+}
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --to)
+            if [ -z "${2:-}" ]; then
+                echo "ERROR: --to needs a tag, branch, or commit." >&2
+                exit 1
+            fi
+            _target="$2"; shift ;;
+        --main)     _use_main=1 ;;
+        --check)    _check_only=1 ;;
+        --rollback) _rollback=1 ;;
+        --yes|-y)   _assume_yes=1 ;;
+        -h|--help)  usage; exit 0 ;;
+        *) echo "ERROR: unknown option: $1" >&2; echo "" >&2; usage >&2; exit 1 ;;
+    esac
+    shift
+done
+
+echo "=== TrueNAS TrueCloud Provider Patch — Update (v${VERSION}) ==="
+echo ""
+
+# ── Preflight ─────────────────────────────────────────────────────────────────
+
+if [ "$(id -u)" -ne 0 ]; then
+    echo "ERROR: must be run as root (install.sh needs it)." >&2
+    exit 1
+fi
+
+cd "$PATCH_DIR"
+
+if ! git rev-parse --git-dir >/dev/null 2>&1; then
+    echo "ERROR: $PATCH_DIR is not a git clone — nothing to update." >&2
+    echo "  Re-clone from https://github.com/sudolulo/truenas-truecloud-patch" >&2
+    exit 1
+fi
+
+# Past `sudo git pull`s can leave root-owned objects in .git that then break any
+# non-root git command. We run as root, so we would only make that worse.
+_owner="$(stat -c '%U' "$PATCH_DIR")"
+if [ -n "$_owner" ] && [ "$_owner" != "root" ]; then
+    chown -R "$_owner" "$PATCH_DIR/.git" 2>/dev/null || true
+fi
+
+# A dirty tree means someone edited or scp'd files in place; merging over that
+# silently loses their changes, or conflicts halfway through.
+if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
+    echo "ERROR: the working tree has uncommitted changes:" >&2
+    git status --short --untracked-files=no >&2
+    echo "" >&2
+    echo "  Refusing to update over them. Commit, stash, or discard them first:" >&2
+    echo "    git -C $PATCH_DIR checkout -- ." >&2
+    exit 1
+fi
+
+# ── Rollback ──────────────────────────────────────────────────────────────────
+
+if [ "$_rollback" -eq 1 ]; then
+    if [ ! -f "$_PREV_FILE" ]; then
+        echo "ERROR: no previous revision recorded — nothing to roll back to." >&2
+        exit 1
+    fi
+    _prev="$(cat "$_PREV_FILE")"
+    if ! git rev-parse --verify --quiet "${_prev}^{commit}" >/dev/null; then
+        echo "ERROR: recorded revision '$_prev' is not a valid commit." >&2
+        echo "  The history may have been rewritten. Pick a target explicitly:" >&2
+        echo "    bash update.sh --to <tag>" >&2
+        exit 1
+    fi
+    _abort_if_untracked_blockers "$_prev"
+    echo "Rolling back to $_prev ..."
+    git checkout -q --detach "$_prev"
+    echo "Reverted. Re-applying ..."
+    echo ""
+    bash "$PATCH_DIR/install.sh"
+    exit 0
+fi
+
+# ── Work out where we are and where we are going ──────────────────────────────
+
+echo "Fetching ..."
+git fetch --quiet --tags --prune origin
+
+_current="$(git rev-parse HEAD)"
+_current_desc="$(git describe --tags --always 2>/dev/null || echo "$_current")"
+
+if [ -n "$_target" ]; then
+    :
+elif [ "$_use_main" -eq 1 ]; then
+    _target="origin/main"
+else
+    # Newest release tag by VERSION order, not by tag date. Date order is only
+    # correct while tags are created in ascending version order; it breaks the
+    # moment a hotfix is tagged out of band (a v0.3.6 released after v0.4.0 would
+    # sort as "newest" by date and silently downgrade the box).
+    #
+    # Filter to PLAIN vX.Y.Z: git's version sort ranks `v0.5.0-rc1` ABOVE `v0.5.0`
+    # (verified), so without this a release candidate would be installed as though
+    # it were the newest release. The release workflow deliberately supports
+    # rc/beta/alpha tags, so they will exist.
+    _target="$(git tag -l 'v*' --sort=-version:refname \
+                 | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1)"
+    if [ -z "$_target" ]; then
+        echo "ERROR: no release tags found; use --main to track unreleased code." >&2
+        exit 1
+    fi
+fi
+
+if ! _target_sha="$(git rev-parse --verify --quiet "${_target}^{commit}")"; then
+    echo "ERROR: '$_target' is not a valid tag, branch, or commit." >&2
+    exit 1
+fi
+
+echo "  current: $_current_desc"
+echo "  target:  $_target ($(git rev-parse --short "$_target_sha"))"
+echo ""
+
+if [ "$_current" = "$_target_sha" ]; then
+    echo "Already up to date. Nothing to do."
+    exit 0
+fi
+
+_abort_if_untracked_blockers "$_target_sha"
+
+# ── Show what is coming ───────────────────────────────────────────────────────
+
+echo "Commits you do not have yet:"
+git log --oneline --no-decorate "$_current..$_target_sha" | sed 's/^/  /' || true
+echo ""
+
+# Reuse tools/release_notes.py rather than re-implementing the extractor here —
+# a second copy would be the untested one. Read the CHANGELOG *of the target*, so
+# the notes describe what you are about to install.
+if [ -f "$PATCH_DIR/tools/release_notes.py" ] && [ "$_use_main" -eq 0 ] \
+       && [ -z "${_target##v*}" ]; then
+    _cl="$(mktemp)"
+    if git show "$_target_sha:CHANGELOG.md" > "$_cl" 2>/dev/null && [ -s "$_cl" ]; then
+        echo "Release notes for $_target:"
+        python3 "$PATCH_DIR/tools/release_notes.py" notes "$_target" "$_cl" \
+            2>/dev/null | sed 's/^/  /' || echo "  (no notes for $_target)"
+        echo ""
+    fi
+    rm -f "$_cl"
+fi
+
+if [ "$_use_main" -eq 1 ]; then
+    echo "NOTE: --main tracks UNRELEASED code. It has passed CI, but it is not a"
+    echo "      tested release, and apply.sh runs at every boot."
+    echo ""
+fi
+
+if [ "$_check_only" -eq 1 ]; then
+    echo "--check given; nothing changed."
+    exit 0
+fi
+
+# ── Confirm ───────────────────────────────────────────────────────────────────
+
+if [ "$_assume_yes" -eq 0 ]; then
+    printf "Apply this update and restart middlewared? [y/N] "
+    read -r _answer </dev/tty || _answer=""
+    case "$_answer" in
+        y|Y|yes|YES) ;;
+        *) echo "Aborted. Nothing changed."; exit 0 ;;
+    esac
+    echo ""
+fi
+
+# ── Apply ─────────────────────────────────────────────────────────────────────
+
+# Record where we were BEFORE moving, so --rollback works even if install.sh dies.
+echo "$_current" > "$_PREV_FILE"
+
+echo "Checking out $_target ..."
+git checkout -q --detach "$_target_sha"
+echo "  now at $(git describe --tags --always)"
+echo ""
+
+echo "Applying (this preserves your nested-snapshot setting) ..."
+echo ""
+if ! bash "$PATCH_DIR/install.sh"; then
+    echo ""
+    echo "ERROR: install.sh failed after updating." >&2
+    echo "  Roll back with:  bash $PATCH_DIR/update.sh --rollback" >&2
+    echo "  Or disable the patch entirely:  bash $PATCH_DIR/recover.sh" >&2
+    exit 1
+fi
+
+echo ""
+echo "=== Update complete ==="
+echo "  $_current_desc  ->  $(git describe --tags --always)"
+echo ""
+if ! git symbolic-ref -q HEAD >/dev/null; then
+    echo "NOTE: the checkout is now pinned to a release tag (detached HEAD), which is"
+    echo "      what you want for a deployment. Plain \`git pull\` will not work here —"
+    echo "      use \`bash update.sh\` from now on."
+    echo ""
+fi
+echo "If anything looks wrong:"
+echo "  bash $PATCH_DIR/update.sh --rollback    # back to $_current_desc"
+echo "  bash $PATCH_DIR/recover.sh              # kill switch + restart"