Opt-in
------
Nested-dataset snapshot support changes how backups read their source data, so
it is now off by default and gated behind a marker file:
install.sh --enable-nested-snapshots
install.sh --disable-nested-snapshots
With neither flag install.sh preserves the current setting, so a routine
`git pull && bash install.sh` can never silently flip it. When disabled,
apply.sh skips the patch entirely and the stock guard remains. uninstall.sh
tears down staging mounts and removes the marker.
Snapshot lifecycle
------------------
zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls
it with no options. Stock is safe only because its validation means recursive is
never True in the field. Enabling nested datasets makes recursive snapshots real:
the parent then has one child snapshot per descendant dataset (160+ on an Apps
pool), so stock's delete would orphan every child on EVERY successful run.
The patch now owns the lifecycle end to end:
- delete_snapshot_tree() sweeps the parent and all children, and is idempotent
against stock's finally winning the race once our mounts are released
- on a staging failure the tree is deleted here, because sync.py never completes
`snapshot, local_path = await create_snapshot(...)` and so its finally deletes
nothing at all
- the snapshot is recorded in a sidecar file before anything is mounted, so a
middlewared restart mid-backup cannot orphan it
- a crashed run's snapshot tree is reclaimed on the next run instead of being
overwritten and leaked
Silent-omission fix
-------------------
The dataset list is now enumerated AFTER the snapshot. Read beforehand it could
miss a dataset created in the gap, which the recursive snapshot would capture but
the staging plan would not -- silently omitting its data. Read afterwards, an
unsnapshotted dataset trips the staging check and fails the run loudly.
Also from the audit
-------------------
- plan_staging scopes by dataset name, so skipped-dataset warnings no longer
include every mountpoint-less dataset on the box, which buried the ones that
matter
- staging_root_for rejects "." / ".." components that would escape the staging
base, and resolves STAGING_BASE at call time rather than freezing it into a
default argument
- uninstall.sh no longer `rm -rf`s a tree that may still contain live bind
mounts, and unmounts by path depth rather than string length
- apply_plan takes an injectable isdir; verify_staged drops an unused parameter
- pin the shellcheck action instead of tracking @master
61 tests, ruff and shellcheck clean.
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:
This option is only available for datasets that have no further nesting
That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.
The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.
Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.
Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays
The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.
Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.
Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds. Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.
Bumps all scripts to v0.0.3.
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.
- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
- Capture midclt output via $(...) instead of > /dev/null so that
failure detail (which midclt writes to stdout on TrueNAS) is
preserved and shown to the user on error rather than silently
discarded
- Expand update failure hint from a bare query command to an actionable
recovery path: show the midclt output, then print the exact delete
command with the known stale ID so the user can remove it and retry
- Extract hook comment string to _HOOK_COMMENT variable in both
install.sh and uninstall.sh; previously the literal string
'TrueCloud provider patch (S3/B2)' appeared three times across two
files with no shared constant — a silent mismatch on any divergence
would cause hook lookup to return empty with no error output
- Wrap midclt update and create calls with if/else error handlers;
previously a midclt failure under set -euo pipefail silently aborted
the script at "Updating path and enabling ..." with no diagnostic
or recovery guidance
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
substitution into sitecustomize.py; sed's & and | metacharacters
silently corrupt or truncate the output for paths containing those
chars; Python str.replace has no metacharacter issues; also write to
a tmp file and mv atomically so a failed substitution never leaves
an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
$PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
("TrueCloud provider patch (S3/B2)") instead of exact script path;
exact-path match breaks when the repo is moved after install —
uninstall leaves the stale hook registered (fires on every boot),
and reinstall creates a duplicate entry; install.sh now also updates
the script path on re-run so a moved repo self-corrects
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
Users who delete the cloned repo after install had no way to uninstall
without re-cloning. Now install.sh copies uninstall.sh to PATCH_DIR
alongside recover.sh, so the uninstall path is always the stable
/data/truecloud-patch/uninstall.sh. README updated to match.
- uninstall.sh: track _restore_failed separately from RESTORED so
"No backup files found" only prints when find returns nothing (not
when mv fails on existing backups); abort with exit 1 before rm -rf
when any restore fails, leaving PATCH_DIR and recover.sh intact
- install.sh: extend log-scan grep to catch ERROR: lines from
patch_ui.py (backup OSError was silently missed by WARNING:-only grep)
- install.sh: reword restart-failure message — hook IS already
registered and sitecustomize.py IS installed; patch activates on
next boot regardless
- apply.sh: replace `if $_can_install` with `[ "$_can_install" = true ]`
(explicit test, no implicit command lookup); drop 2>/dev/null on
install cp so OS error detail reaches the log
- install.sh: print 'verify' command immediately after successful restart
so users know to confirm the backend patch loaded before creating tasks
- README: correct --insecure description; it controls TLS to the TrueNAS
API (where the API key is transmitted), not the S3 endpoint — previous
wording implied it was safe to use for S3 self-signed certs
- create_task.py: add MITM risk warning to --insecure flag help text;
common home-user pattern (self-signed cert) exposes API key in transit
- install.sh: replace bare systemctl restart with explicit failure check
that prints a recovery hint when middlewared fails to start post-install
- patch_ui.py: wrap shutil.copy2 backup in try/except OSError so a
permission or read-only filesystem error prints a specific diagnostic
instead of crashing the script with a generic 'exited non-zero' message
- install.sh: add early guard that detects pipe-install (bash <(curl ...))
and exits with a clear error pointing to the git clone workflow
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return early and print 'UI already patched' instead of the misleading
'filterByProviders pattern not found' warning
- install.sh: scope warning grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: fix misleading 'before continuing' wording on warning banner
- install.sh: fix grep anchor (^WARNING: missed [truecloud-patch] WARNING: lines)
- uninstall.sh: add import-middlewared verification after Python detection,
matching apply.sh fallback logic
install.sh: running install.sh after a recover.sh left /data/truecloud-patch/disabled
in place, so apply.sh silently skipped all patching and middlewared restarted
without the patch. Clear the kill switch file before running apply.sh.
uninstall.sh: midclt initshutdownscript.delete was unguarded under set -euo pipefail,
so a delete failure (already-removed entry, transient API error) aborted the script
before sitecustomize.py was cleaned up or /data/truecloud-patch/ was removed. Now
guarded with an if/else that warns and continues.
sitecustomize.py: move importlib.machinery/.util imports inside the if block in
find_spec so they only execute when intercepting our two target modules, not on
every module import across the whole process. In _install(), import os before
importlib.util so the kill switch check (cheap) runs before the importlib import
(slightly heavier on first use). Remove the unused orig variable in _patch_restic.
patch_ui.py: add explicit encoding="utf-8" to both open() calls. errors="replace"
on read so malformed bytes in a bundle don't silently skip a candidate file.
If the patch ever prevents middlewared from starting, users now have a
clear escape hatch that requires no knowledge of Python internals:
bash /data/truecloud-patch/recover.sh
Or at a bare shell prompt:
touch /data/truecloud-patch/disabled
systemctl restart middlewared
sitecustomize.py checks for /data/truecloud-patch/disabled at Python
startup and skips the import hook entirely when the file exists.
apply.sh does the same so the PREINIT script also does nothing on reboot.
recover.sh is copied to /data/truecloud-patch/ by install.sh so it is
available even without the original repo directory.
README gains an "Emergency recovery" section above Troubleshooting.
Patches middlewared at runtime via sitecustomize.py (no file edits to /usr/)
and widens the UI credential dropdown from Storj-only to S3+B2+Storj.
Persists across TrueNAS updates via PREINIT initshutdownscript stored in DB.
flan
flan