truenas-truecloud-patch/CHANGELOG.md

# Changelog

## v0.3.0 — 2026-07-12

### Added

- **`snapshot = true` now works on datasets that have child datasets** —
  **opt-in, off by default** (`install.sh --enable-nested-snapshots` /
  `--disable-nested-snapshots`). It changes how backups read their source data,
  so it is never enabled implicitly; with neither flag `install.sh` preserves
  the existing setting, so a `git pull && bash install.sh` cannot silently flip
  it. When disabled, `apply.sh` skips the patch entirely and the stock guard
  remains. `uninstall.sh` tears down any staging mounts and removes the marker.
  Stock TrueNAS refuses this with *"This option is only available for datasets
  that have no further nesting"*, which makes the snapshot option unusable for
  the single most common case on any box running Apps — every app is its own
  dataset, often with `config`/`pgdata` children of its own. Without it, the
  backup reads **live** files: databases are captured mid-write, and a busy app
  rewriting its files can stall a backup indefinitely as restic chases a moving
  target.

  The stock guard is **correct, and it is not an arbitrary limit.**
  `plugins/cloud/snapshot.py` already takes a *recursive* ZFS snapshot, but it
  then points the backup tool at the **parent** dataset's
  `.zfs/snapshot/<snap>/` directory — and ZFS does not expose child datasets
  through a parent's snapshot directory:

  ```
  /mnt/Tap/.zfs/snapshot/<snap>/apps/               -> 0 entries (children invisible)
  /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
  ```

  So without the guard the backup tool would walk a near-empty tree, report
  SUCCESS, and upload almost nothing. iX gate the config rather than ship a
  backup that lies about succeeding.

  This release implements the missing half. After the (already recursive)
  snapshot is taken, every descendant dataset's own `.zfs/snapshot/<snap>` is
  bind-mounted into a **staging tree** mirroring the original layout, and the
  backup tool is pointed at the staging root — a complete, consistent,
  point-in-time view of the whole subtree. Only then is the guard relaxed.

  Safety properties, in order of importance:

  - **Staging failure is loud.** If any descendant cannot be staged, the backup
    fails. A silently-incomplete backup is the exact outcome the stock guard
    exists to prevent, and it would be worse than not having the feature.
  - **A post-mount verification pass** asserts every planned target is really a
    mountpoint and the staging root is non-empty, so this can never regress into
    the empty-backup failure it is meant to fix.
  - **The guard is relaxed last.** `apply.sh` installs the traversal, patches
    `snapshot.py`, then `sync.py`, and only then `crud.py`. A partial failure
    leaves the guard intact and the option merely unavailable — never
    "guard removed, traversal missing".
  - **The patch owns the whole snapshot lifecycle.** `zfs.snapshot.delete`
    defaults to `recursive=False` and stock `restic_backup()` calls it with no
    options. Stock gets away with that only because its validation means
    `recursive` is never True in the field — but enabling nested datasets makes
    recursive snapshots real, so the parent now has one child snapshot per
    descendant dataset (160+ on a typical Apps pool). Relying on stock's delete
    would therefore orphan every child snapshot **on every successful run**.
    This patch sweeps the parent *and* all children, is idempotent against
    stock's `finally` winning the race, records the snapshot in a sidecar file
    (so a middlewared restart mid-backup cannot orphan it), reclaims the tree
    left by a crashed run, and deletes the tree when staging fails — where
    sync.py's own `finally` would otherwise delete nothing at all, because its
    `snapshot` local never gets assigned.
  - **The dataset list is enumerated *after* the snapshot, never before.** A
    list read beforehand can miss a dataset created in the gap: the recursive
    snapshot would capture it but the staging plan would not, silently omitting
    its data. Read afterwards, an unsnapshotted dataset trips the staging check
    and fails the run loudly instead.
  - **Every injected block no-ops** if `_truecloud_nested` is absent.
  - Datasets that cannot contribute to a file tree (`mountpoint=none|legacy`,
    unmounted/locked, encrypted-and-locked) are skipped and **reported** —
    never dropped silently.
  - Scoped to `cloud_backup` only. Cloud Sync (rclone) shares the same
    validation mixin but has no staging teardown wired in, so its guard is left
    in place deliberately.

  Side benefit: the staging root is a **stable** path per task, so restic can
  find its parent snapshot between runs. Stock's
  `.zfs/snapshot/<name>-<timestamp>/` path changes every run, which defeats
  restic's parent detection and forces a full re-scan each time.

- **CI** (GitHub Actions): shellcheck + `bash -n` on every script, ruff, and
  pytest on Python 3.11/3.12/3.13. Includes tests that `compile()` the
  `*_BLOCK` strings — they are Python source appended to live middlewared
  modules, so a syntax error there would break the box at boot, and nothing
  previously checked them.

### Changed

- **The patch is now two independent modules, and each retires on its own.**
  Previously the native-support check looked only for native B2 restic support
  and, on finding it, set the kill switch and disabled *everything*. With a
  second capability in the patch that would silently take a still-needed module
  down with the superseded one — TrueNAS is likely to ship one of these long
  before the other.

  `apply.sh` now detects each separately (`providers`: does `B2RcloneRemote`
  carry a real `get_restic_config()`; `nested`: is the *"no further nesting"*
  validation still in `plugins/cloud/crud.py`), skips just the superseded one,
  and only sets the kill switch once **both** are done. The UI patch belongs to
  `providers` and is skipped with it. The deferred middlewared restart now fires
  when *any* still-needed module landed — keying it off `providers` alone would
  have left a freshly-patched `nested` module on disk and never loaded on a
  native-B2 box. `hook_status.json` reports each module with an `active` flag and
  a reason.

- README rewritten to be less alarmist: dropped the warning boxes and the
  disclaimer's fear-bulleting in favour of plain statements, and documented the
  two-module design. The one caveat kept as a plain sentence: the `mount --bind`
  staging step has not yet been exercised by a live backup run.

- Version strings in `install.sh`, `uninstall.sh`, and `recover.sh` were stale
  at `0.0.4`; all scripts now report the same version.
- `patch_ui.py`: replaced a `try`/`except`/`pass` with `contextlib.suppress`
  (no behaviour change; satisfies the new lint gate).

### Removed

- `patch/__pycache__/create_task.cpython-314.pyc` was committed to the
  repository; it is now untracked and `__pycache__/` is gitignored.

### Fixed (post-merge audit)

- **`create_task.py verify` failed on a default install.** `hook_status.json`
  emitted a per-file entry for the nested module with `ok: false` whenever the
  feature was switched off — which is the default — so `verify` printed `[FAIL]`
  and exited 1 right after the README told users to run it. Status is now
  reported per *module* with an `active` flag, and `verify` renders an inactive
  module as `[SKIP]` rather than a failure.
- **A partial apply suppressed the middlewared restart.** The exit code
  conflated "nothing applied" with "one module applied, one failed", so a failing
  providers patch would prevent the restart that a freshly-applied nested patch
  needs — leaving it on disk and never loaded. Exit 2 now means partial, and the
  restart still fires.
- **The native-nested probe could never fire.** It scanned `crud.py` for the
  guard message, but our own injected block *quotes* that message, so once
  applied the probe would always conclude the guard was still present. It now
  reads only the stock portion of the file.
- `recover.sh` did not unmount staging trees, so an emergency recovery left bind
  mounts pinning ZFS snapshots that could then never be destroyed.
- `uninstall.sh` deleted sidecar files without reading them. A sidecar is the
  only record that an interrupted run's snapshot tree is still on disk; both
  scripts now name the snapshot (`zfs destroy -r ...`) before clearing it.

- **The native-nested probe could never detect the guard, silently disabling the
  whole module.** Stock splits the message across adjacent string literals:

  ```python
  verrors.add(f"{name}.snapshot", "This option is only available for datasets that have no further "
                                  "nesting")
  ```

  Python concatenates those at runtime — so the *errmsg* is contiguous and the
  runtime filter works — but the **source never contains the whole phrase**. The
  probe's substring search found nothing, concluded iX had removed the guard, and
  skipped the nested module as "already native". `apply.log` would report
  *"TrueNAS now handles nesting natively"* and the feature would never work.
  It fails safe (the stock guard stays, so no data is at risk) but the module was
  100% dead. The probe now strips whitespace and quotes before matching, which is
  robust to any wrapping style. Caught only by running the probe against real
  middlewared; there is now a regression test that executes apply.sh's own probe
  code against the real wrapped source.

### Refactored

- Staging teardown had been copy-pasted into `uninstall.sh` and `recover.sh` —
  two untested shell copies of the fiddly depth-ordering and lazy-umount logic.
  Both now call `python3 patch/truecloud_nested.py cleanup`, so there is one
  implementation and it is the one under test.
- Dropped the in-memory `ACTIVE` dict. The sidecar file was already the source of
  truth; a second in-process record could only desync — and it is the
  middlewared-restart case (which empties it) that must not orphan a snapshot
  tree. One record, on disk, or none.

### Known issues

- Stock `restic_backup()` deletes the ZFS snapshot in its own `finally`, which
  fails with `EBUSY` while the staging bind mounts pin it. It logs one benign
  `Error deleting snapshot ...` warning per run; the patch then unmounts and
  deletes the snapshot for real. The warning is expected and harmless.

## v0.2.1 — 2026-07-09

### Fixed

- **Deferred restart raced the rest of boot, leaving all apps and dashboard
  stats down.** The `truecloud-mw-restart` unit introduced in v0.0.4 relied
  on systemd ordering (`After=multi-user.target`, `After=ix-postinit.service`),
  which cannot see middlewared's *internal* boot work. Observed on 25.10.4:
  the restart fired two seconds into `ix-reporting.service`'s
  `midclt call reporting.start_service` and before the docker/apps startup
  task (created on middlewared's system-ready event) had run. Both were
  killed, and nothing retries them until the next boot — every app stayed
  down (`docker.status` FAILED, the apps dataset never mounted), netdata
  never started (no dashboard hardware stats), and the SMB middleware
  backend was left uninitialized.

  The transient unit now runs `patch/wait_restart.sh` instead of restarting
  directly: it waits for the systemd boot job queue to drain
  (`systemctl is-system-running --wait`, covering in-flight `ix-*` oneshots
  such as ix-reporting), then polls `midclt call docker.status` until the
  docker state machine leaves its transitional states, then allows a short
  grace period for middleware-internal tasks with no queryable state before
  issuing `systemctl try-restart middlewared`. The unit no longer sets
  `Type=oneshot` — a oneshot's start job stays in the very queue the script
  waits on and would deadlock on itself. All waits are bounded and fail
  open: worst case the restart still happens, just later.

  Recovery on a boot that already hit this (without rebooting):
  `midclt call reporting.start_service` and
  `midclt call docker.state.start_service true`.

## v0.2.0 — 2026-07-08

### Changed

- **`create_task.py` now uses the TrueNAS middleware via `midclt` instead of the
  deprecated `/api/v2.0` REST API**, which is removed in TrueNAS 26.04. Practical
  effects:
  - Run the script **on the TrueNAS host** — it uses the local middleware socket, so
    it no longer needs a host address or API key.
  - `--host`, `--api-key`, and `--insecure` are accepted but **ignored** (a deprecation
    note is printed); they will be removed in a future release.
  - `list-credentials` → `cloudsync.credentials.query`, `list-tasks` →
    `cloud_backup.query`, `create` → `cloud_backup.create`.
- Dropped the `ssl`/`urllib` HTTP client; no TLS certificate handling is needed anymore.

## v0.1.0 — 2026-07-08

### Added

- `create --cache-path PATH` — sets the restic cache directory on the task.
  Without a cache path, TrueNAS runs restic with `--no-cache`, which re-reads all
  repository metadata from the provider on every run and is glacially slow on
  large repos (a 564 GB dataset estimated **55 days** to a first backup). Tasks
  created without `--cache-path` now print a warning explaining the consequence.

## v0.0.4 — 2026-07-06

### Fixed

- **Backend patch inactive after every reboot.** PREINIT initshutdownscripts
  are executed by middlewared itself (`ix-preinit.service` runs
  `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after
  `ix-zfs.service` pool import). By the time `apply.sh` patched `b2.py` and
  `restic.py` in the overlay, the running middlewared had already imported the
  stock modules and never re-imports — so S3/B2 support silently reverted on
  every reboot until something restarted middlewared. `install.sh` masked the
  bug because it restarts middlewared explicitly.

  Fix: when `apply.sh` detects it was invoked by middlewared (boot context),
  it now schedules a single detached restart via a transient systemd unit
  (`truecloud-mw-restart`, ordered after `multi-user.target` and
  `ix-postinit.service`) so the patched modules are loaded once boot settles.
  The restart is never synchronous — `apply.sh` is a child of middlewared's
  own job runner, and later `ix-*` boot units still need midclt. Manual runs
  of `apply.sh` never trigger a restart.

- **`TypeError: string indices must be integers` when creating a B2 task on
  TrueNAS 24.10 (Electric Eel)** (#1). The credential schema differs between
  releases: on 24.10 `credentials["provider"]` is the type string (`"B2"`)
  with the account/key in `credentials["attributes"]`, while 25.04+ moved
  them into a provider dict. The injected `get_restic_config` only handled
  the 25.04+ shape. It now detects the schema and reads the credentials from
  the right place on both; `create_task.py list-credentials` and `list-tasks`
  got the same treatment.

- **`create_task.py verify` false-positive after reboot.** `verify` trusted
  `hook_status.json`, which only records that the files were patched on disk —
  not that the running process loaded them. `verify` now also compares the
  middlewared main-process start time against `patched_at` and reports FAIL
  (with recovery instructions) when the process predates the patch.

### Changed

- README and script comments no longer claim PREINIT runs "before middlewared
  starts"; the boot ordering and the deferred restart are now documented.
- `recover.sh` and `uninstall.sh` cancel a still-queued deferred restart
  before performing their own, and their re-enable instructions now include
  the required `systemctl restart middlewared`.

---

## v0.0.3 — 2026-06-22

### Fixed

- **`patch/apply.sh` silently killed by the 10-second PREINIT timeout.**
  TrueNAS PREINIT initshutdownscripts have a 10-second default timeout. The
  previous apply.sh ran approximately 8 Python subprocesses (each ~1-2 s), so it
  was routinely killed mid-run. Symptoms: patches not applied after reboot, but
  re-running `bash apply.sh` manually (no timeout) always succeeded.

  Fix: `install.sh` now registers the hook with `"timeout": 120`. Existing
  installations are updated to the new timeout on the next `bash install.sh` run.

  Additionally, `patch/apply.sh` consolidates its Python subprocess invocations
  from ~8 down to 2, reducing startup overhead from ~12-16 s to ~2-4 s — well
  within the new 120-second budget.

### Changed

- `find_mw_python` in `apply.sh` no longer spawns a separate Python process to
  verify the interpreter can import `middlewared`. Verification is now implicit in
  the combined path-discovery subprocess that follows.

---

## v0.0.2 — 2026-06-19

### Fixed

- **B2 backup failing with `NotImplementedError` after a TrueNAS update.**
  The patch block's guard (`if "get_restic_config" not in B2RcloneRemote.__dict__`)
  could misfire and silently skip injecting the method — most likely when a
  TrueNAS version adds a stub that raises `NotImplementedError`, causing the dict
  check to return `False`. The guard is removed; the assignment is now
  unconditional. This is safe because the native-support kill switch already
  prevents patching when TrueNAS ships a real, working implementation.

- **Native-support check falsely triggering kill switch on stubs.**
  The check now inspects the source of any pre-existing `get_restic_config`
  before concluding that TrueNAS has shipped native B2 support. If the method
  body contains `NotImplementedError` it is treated as a stub and patching
  continues; only a method that does not raise `NotImplementedError` triggers
  the kill switch and auto-disable.

---

## v0.0.1 — 2026-06-16

Initial public release. Extends TrueNAS SCALE's TrueCloud Backup feature to
work with S3-compatible providers and native Backblaze B2 in addition to Storj,
using volatile overlayfs patching of the TrueNAS middleware that persists across
system updates via a PREINIT initshutdownscript.

### Included

- `install.sh` — registers the PREINIT boot hook and applies patches immediately
- `patch/apply.sh` — PREINIT script; mounts writable overlays, patches `b2.py`
  and `restic.py`, patches the Angular UI bundle to widen the credential dropdown
- `patch/create_task.py` — CLI to create TrueCloud Backup tasks with S3 or B2
  credentials, bypassing the Storj-only restriction in the UI
- `recover.sh` — emergency recovery; sets the kill switch and restarts middlewared
- `uninstall.sh` — full removal of the patch and PREINIT hook
- Kill switch support (`disabled` file) for safe degradation
- Auto-disable when TrueNAS ships native B2 restic support
- `hook_status.json` written on each boot for `create_task.py verify`