The "strip the TRUECLOUD_PATCH block" logic existed twice -- in apply.sh's heredoc
and in an inline heredoc in uninstall.sh -- and the uninstall copy was the untested
one. That is precisely how the two could have drifted apart, with apply.sh
reverting one set of files and uninstall.sh another.
Both now call patch/mw_patch.py. 17 new tests cover it, including that
revert_nested never touches restic.py: that file carries a TRUECLOUD_PATCH block
too, but it belongs to the providers module, and removing it would silently break
B2 backups.
apply.sh imports it fail-safe -- on ImportError the backend patch is skipped and
middlewared starts stock, which is this script's whole design principle. The
import uses sys.path.append, never insert(0): prepending would give patch/
precedence over the stdlib for that interpreter, so a future patch/json.py would
shadow the real json module and break the boot.
Also: the README's create_task.py example still taught `--password <secret>`, which
is how a security fix quietly fails to land. It now shows --password-stdin.
132 tests, ruff and shellcheck clean.
Security
--------
create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that
JSON carries the restic repository password -- so it sat in the subprocess's argv,
which is world-readable via ps, for the duration of the call. That password is the
encryption key for the entire cloud backup repository.
It now talks to the middleware through truenas_api_client, the library that backs
midclt itself, so the password never leaves this process's memory. Verified on a
live box: list-tasks and list-credentials work through the new transport.
--password is also no longer required, because passing a secret as a CLI argument
writes it to shell history permanently. --password-stdin reads it from stdin, and
with neither flag the tool prompts via getpass. --password still works but warns.
Fixed
-----
uninstall.sh could leave every patch installed. It reverted by unmounting the
overlay -- but apply.sh only mounts one when the target directory is read-only. On
a writable /usr it patches the real files in place, and uninstall would remove the
boot hook, report success, and leave the patch applied. It now strips the appended
blocks from the middleware files explicitly. This also covers the case where the
overlay unmount fails.
create_task.py's __version__ had been stuck at 0.2.0 through three releases. The
version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it
missed the one file that actually shows a version to users (--version). The check
now covers __version__ too -- and caught this immediately.
118 tests, ruff and shellcheck clean.
apply.sh only ever ADDED patches; there was no revert path anywhere. Disabling
removed the opt-in marker and then merely skipped re-applying -- but the overlay
persists for the whole boot, so the previously patched cloud/{snapshot,crud}.py,
cloud_backup/sync.py and _truecloud_nested.py were all still on disk, and
middlewared re-imported them on the restart install.sh performs.
It printed "DISABLED (stock guard restored)" while the feature kept running until
the next reboot. Someone disabling it because they were worried about it would
have believed it was off.
apply.sh now reverts on every not-needed path (opt-out, or superseded by native
support): remove the module FIRST -- every injected block is guarded by
`if _tc_nested is not None`, so the stock guard comes back even if a later step
fails -- then strip the appended blocks from the three patched files.
restic.py also carries a TRUECLOUD_PATCH block but belongs to the providers
module; reverting it would silently break B2 backups, so it is explicitly
excluded. Verified: the three nested files restore byte-for-byte to stock, the
module is removed, and restic.py's block survives.
install.sh --disable also tears the staging tree down first, since those bind
mounts pin ZFS snapshots that could otherwise never be destroyed.
Updating WITHOUT the flag was always correct and is unchanged: the nested module
is never installed into middleware unless explicitly enabled.
109 tests, ruff and shellcheck clean.
The "Automated releases" entry was written under v0.3.0, but that commit landed
after the v0.3.0 tag -- the CHANGELOG was claiming the release contained something
it did not. Moved to its own version rather than left as a quiet inaccuracy.
Bumps VERSION to 0.3.1 across all four scripts, which the new consistency check
now enforces.
Audit
-----
- create_task.py verify failed on a DEFAULT install. hook_status.json emitted a
per-file entry for the nested module with ok:false whenever the feature was
switched off -- the default -- so verify printed [FAIL] and exited 1, right
after the README tells users to run it. Status is now per MODULE with an
`active` flag, and verify renders an inactive module as [SKIP].
- A partial apply suppressed the middlewared restart. The exit code conflated
"nothing applied" with "one module applied, one failed", so a failing providers
patch would prevent the restart that a freshly-applied nested patch needs,
leaving it on disk and never loaded. Exit 2 now means partial and the restart
still fires.
- The native-nested probe could never fire. It scanned crud.py for the guard
message, but our own injected block quotes that message, so once applied the
probe would always conclude the guard was still present. It now reads only the
stock portion of the file.
- recover.sh did not unmount staging trees, so an emergency recovery left bind
mounts pinning ZFS snapshots that could then never be destroyed.
- uninstall.sh deleted sidecar files without reading them. A sidecar is the only
record that an interrupted run's snapshot tree is still on disk; both scripts
now name the snapshot before clearing it.
- Removed a dead branch in the restart gate (unreachable: the kill switch exits).
Refactor
--------
- Staging teardown had been copy-pasted into uninstall.sh and recover.sh -- two
untested shell copies of the fiddly depth-ordering and lazy-umount logic. Both
now call `python3 patch/truecloud_nested.py cleanup`, so there is exactly one
implementation and it is the one under test.
- Dropped the in-memory ACTIVE dict. The sidecar file was already the source of
truth; a second in-process record could only desync -- and it is precisely the
middlewared-restart case (which empties it) that must not orphan a snapshot
tree. One record, on disk, or none.
Not done: the overlay-unmount loop is duplicated across apply.sh/uninstall.sh/
recover.sh. It is pre-existing, and apply.sh runs at PREINIT under a tight
timeout -- giving it a source dependency would trade 10 lines of duplication for
a boot-time failure mode.
74 tests, ruff and shellcheck clean.
Opt-in
------
Nested-dataset snapshot support changes how backups read their source data, so
it is now off by default and gated behind a marker file:
install.sh --enable-nested-snapshots
install.sh --disable-nested-snapshots
With neither flag install.sh preserves the current setting, so a routine
`git pull && bash install.sh` can never silently flip it. When disabled,
apply.sh skips the patch entirely and the stock guard remains. uninstall.sh
tears down staging mounts and removes the marker.
Snapshot lifecycle
------------------
zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls
it with no options. Stock is safe only because its validation means recursive is
never True in the field. Enabling nested datasets makes recursive snapshots real:
the parent then has one child snapshot per descendant dataset (160+ on an Apps
pool), so stock's delete would orphan every child on EVERY successful run.
The patch now owns the lifecycle end to end:
- delete_snapshot_tree() sweeps the parent and all children, and is idempotent
against stock's finally winning the race once our mounts are released
- on a staging failure the tree is deleted here, because sync.py never completes
`snapshot, local_path = await create_snapshot(...)` and so its finally deletes
nothing at all
- the snapshot is recorded in a sidecar file before anything is mounted, so a
middlewared restart mid-backup cannot orphan it
- a crashed run's snapshot tree is reclaimed on the next run instead of being
overwritten and leaked
Silent-omission fix
-------------------
The dataset list is now enumerated AFTER the snapshot. Read beforehand it could
miss a dataset created in the gap, which the recursive snapshot would capture but
the staging plan would not -- silently omitting its data. Read afterwards, an
unsnapshotted dataset trips the staging check and fails the run loudly.
Also from the audit
-------------------
- plan_staging scopes by dataset name, so skipped-dataset warnings no longer
include every mountpoint-less dataset on the box, which buried the ones that
matter
- staging_root_for rejects "." / ".." components that would escape the staging
base, and resolves STAGING_BASE at call time rather than freezing it into a
default argument
- uninstall.sh no longer `rm -rf`s a tree that may still contain live bind
mounts, and unmounts by path depth rather than string length
- apply_plan takes an injectable isdir; verify_staged drops an unused parameter
- pin the shellcheck action instead of tracking @master
61 tests, ruff and shellcheck clean.
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:
This option is only available for datasets that have no further nesting
That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.
The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.
Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.
Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays
The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.
Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.
Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds. Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.
Bumps all scripts to v0.0.3.
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.
- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
site.getsitepackages()[0] returns /usr/local/lib/python3.11/dist-packages/ on
TrueNAS 25.x but middlewared lives in /usr/lib/python3/dist-packages/.
sitecustomize.py was installed to the wrong directory and Python never loaded it.
Fix: derive SITE_PKG from middlewared.__file__ so the overlay and sitecustomize.py
land in the correct directory.
Also add direct patching of b2.py and restic.py in the overlay as the primary
backend approach — more reliable than an import hook since it works regardless
of Python's site initialisation configuration. apply.sh now also writes
hook_status.json at boot time so 'verify' shows OK without requiring a backup run.
Also fixes incorrect middlewared log path in README (/var/log/middlewared/middlewared.log
→ /var/log/middlewared.log) and simplifies the verify troubleshooting note.
TrueNAS 25.x mounts /usr as a read-only filesystem. Writing
sitecustomize.py to site-packages and patching the Angular bundle
both fail with EROFS.
Fix: mount a writable overlayfs on each target directory before
writing to it. Upper/work dirs live in /run (tmpfs), so overlays are
volatile per boot and are recreated by apply.sh on every PREINIT run
before middlewared starts.
apply.sh:
- Add _ensure_writable(dir, tag): probes writability; mounts overlay
in /run/truecloud-{tag}-{upper,work} if the directory is read-only;
detects if the overlay is already mounted (idempotent)
- Call _ensure_writable before site-packages writes (tag "sc")
- Detect webui dir with bash loop; call _ensure_writable before
patch_ui.py (tag "ui") — non-fatal if mount fails
uninstall.sh:
- Add overlay unmounting section after file restoration and before
rm -rf, so the lower layer's originals are exposed immediately
- Move _restore_failed exit 1 to after unmount so overlays are
cleaned up even on partial failure
- Update "no backup files" message for immutable OS context
- Extract hook comment string to _HOOK_COMMENT variable in both
install.sh and uninstall.sh; previously the literal string
'TrueCloud provider patch (S3/B2)' appeared three times across two
files with no shared constant — a silent mismatch on any divergence
would cause hook lookup to return empty with no error output
- Wrap midclt update and create calls with if/else error handlers;
previously a midclt failure under set -euo pipefail silently aborted
the script at "Updating path and enabling ..." with no diagnostic
or recovery guidance
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
substitution into sitecustomize.py; sed's & and | metacharacters
silently corrupt or truncate the output for paths containing those
chars; Python str.replace has no metacharacter issues; also write to
a tmp file and mv atomically so a failed substitution never leaves
an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
$PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
("TrueCloud provider patch (S3/B2)") instead of exact script path;
exact-path match breaks when the repo is moved after install —
uninstall leaves the stale hook registered (fires on every boot),
and reinstall creates a duplicate entry; install.sh now also updates
the script path on re-run so a moved repo self-corrects
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
- uninstall.sh: track _restore_failed separately from RESTORED so
"No backup files found" only prints when find returns nothing (not
when mv fails on existing backups); abort with exit 1 before rm -rf
when any restore fails, leaving PATCH_DIR and recover.sh intact
- install.sh: extend log-scan grep to catch ERROR: lines from
patch_ui.py (backup OSError was silently missed by WARNING:-only grep)
- install.sh: reword restart-failure message — hook IS already
registered and sitecustomize.py IS installed; patch activates on
next boot regardless
- apply.sh: replace `if $_can_install` with `[ "$_can_install" = true ]`
(explicit test, no implicit command lookup); drop 2>/dev/null on
install cp so OS error detail reaches the log
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp (disk full, read-only mount) previously fell through and
overwrote the vendor file with no recovery path
- create_task.py: handle unexpected 2xx response schema in cmd_create;
bare KeyError on result['id'] is replaced with a diagnostic print
- uninstall.sh: mv inside while loop had no error handling; under
set -euo pipefail a failed mv aborted the script before rm -rf PATCH_DIR,
leaving the system in partial-uninstall limbo
- apply.sh: correct comment from 'one prior generation' to 'two prior
generations (.1 and .2)' — rotation has always kept three log files
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
in patch/patch_ui.py, preventing silent drift if a new path is added
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return early and print 'UI already patched' instead of the misleading
'filterByProviders pattern not found' warning
- install.sh: scope warning grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: fix misleading 'before continuing' wording on warning banner
- install.sh: fix grep anchor (^WARNING: missed [truecloud-patch] WARNING: lines)
- uninstall.sh: add import-middlewared verification after Python detection,
matching apply.sh fallback logic
sitecustomize.py: when find_spec resolves real_spec as None (module absent
after a TrueNAS update), record a FAIL status and mark the module done so
hook_status.json is still written and cmd_verify shows a diagnostic FAIL
instead of the ambiguous "no status file found".
sitecustomize.py: the AttributeError fallback in the URL-fix wrapper now
writes a WARNING to stderr before returning the unmodified result, making
the unexpected ResticConfig type visible in journalctl.
apply.sh: after falling back to bare python3, verify that python3 can also
import middlewared; if not, emit a second warning so the operator knows the
backend patch may be installed in the wrong site-packages directory.
patch_ui.py: abort (return without writing) when FIND.subn produces a count
other than 1, instead of committing a doubly-patched bundle and having
subsequent runs silently accept it via the MARKER check.
uninstall.sh: when a vendor sitecustomize.py backup exists, use mv to
atomically overwrite our file rather than rm-then-mv; eliminates the window
where a read-only /usr causes rm to fail under set -e, aborting before the
backup is restored.
sitecustomize.py: exec_module now records a FAIL status when the underlying
module load raises, instead of leaving hook_status.json unwritten. cmd_verify
will now show a diagnostic FAIL rather than the misleading "no status file".
sitecustomize.py: URL-fix loop covers --repository and --repository= in
addition to --repo/--repo=/-r; these are documented restic synonyms.
sitecustomize.py: _replace fallback now catches AttributeError in addition
to TypeError so an unrecognised ResticConfig return type silently falls back
to returning the unmodified result rather than crashing the backup job.
patch_ui.py: warns when FIND.subn produces a count other than 1, making
unexpected multi-replacement visible in the apply log.
uninstall.sh: find for JS bundle restore now includes /usr/share/truenas-ui,
matching all three entries in patch_ui.py's WEBUI_CANDIDATES.
apply.sh: preserve two log generations (.1 and .2) on rotation so the
last two boots are always available for diagnosis.
patch_ui.py: find_bundle returns on the first matching JS file instead of
collecting all matches. The multi-match warning was dead weight — the Angular
Ivy compiler produces exactly one bundle and the WARNING path was unreachable
in practice.
uninstall.sh: restore an orphaned sitecustomize.py.pre-truecloud-patch when
sitecustomize.py itself has already been removed (e.g. manual deletion while
the backup survived). Prevents leaving ghost vendor files in site-packages.
patch_ui.py:
- Write Angular bundle atomically via tmp + os.replace, matching the pattern
already used by _record_status. Prevents a corrupt bundle if the write is
interrupted mid-boot.
sitecustomize.py:
- Tighten _record_status count barrier comment to name _Finder._targets as
the canonical count, making the coupling visible to future editors.
uninstall.sh:
- Verify middlewared restarted cleanly after uninstall, with journalctl
guidance on failure — matching recover.sh's existing pattern.
apply.sh:
- Change second line of site-packages error block from WARNING: prefix
(misleading for an instructional message) to a plain Run: hint.
create_task.py:
- Guard __doc__ against None in epilog extraction so -OO does not crash.
README.md:
- Split Emergency recovery into three named subsections: middlewared won't
start, web UI is blank or broken (corrupt bundle recovery), and backend
verify shows FAIL. Each gives direct commands and escalation steps.
- Add Restoring from a TrueCloud Backup section: finding the restic binary,
gathering credentials, provider-specific env var setup for B2 and S3,
listing and restoring snapshots, and operational notes on restore hygiene.
- Clarify that hook_status.json is written once both target modules have
loaded (not necessarily at the instant middlewared starts).
sitecustomize.py:
- Restore _record_status count barrier: write the status file only after all
patches have reported. middlewared.plugins.cloud_backup.restic is imported
lazily (only when a backup task runs), so without this barrier verify would
declare "all patches active" based solely on the B2 patch that fires at
startup. Barrier now gates on _Finder._targets rather than the removed
_PATCHES dict.
- Add comment in exec_module noting the if/elif must stay in sync with
_Finder._targets, making the coupling visible.
patch_ui.py:
- Merge find_webui into find_bundle: previously find_bundle(None) would crash
with os.walk(None) if main()'s guard were removed. Merged function returns
a 3-tuple (webui_dir, path, content); webui_dir=None means no candidate
directory found, path=None means directory found but pattern absent.
main() still produces distinct messages for each failure mode.
create_task.py:
- Split triple-chained .get() in cmd_list_tasks into two lines; the or {}
handling for None credentials was buried inside a one-liner.
uninstall.sh:
- Fix find loop: replace "for x in $(find ...)" with "while IFS= read -r"
to handle paths containing spaces or newlines.
- Add #!/usr/bin/env shebang form to Python detection, matching apply.sh.
Without this, uninstall on a system where middlewared uses the env form
would silently leave sitecustomize.py in the wrong site-packages.
install.sh: running install.sh after a recover.sh left /data/truecloud-patch/disabled
in place, so apply.sh silently skipped all patching and middlewared restarted
without the patch. Clear the kill switch file before running apply.sh.
uninstall.sh: midclt initshutdownscript.delete was unguarded under set -euo pipefail,
so a delete failure (already-removed entry, transient API error) aborted the script
before sitecustomize.py was cleaned up or /data/truecloud-patch/ was removed. Now
guarded with an if/else that warns and continues.
sitecustomize.py: move importlib.machinery/.util imports inside the if block in
find_spec so they only execute when intercepting our two target modules, not on
every module import across the whole process. In _install(), import os before
importlib.util so the kill switch check (cheap) runs before the importlib import
(slightly heavier on first use). Remove the unused orig variable in _patch_restic.
patch_ui.py: add explicit encoding="utf-8" to both open() calls. errors="replace"
on read so malformed bytes in a bundle don't silently skip a candidate file.
Patches middlewared at runtime via sitecustomize.py (no file edits to /usr/)
and widens the UI credential dropdown from Storj-only to S3+B2+Storj.
Persists across TrueNAS updates via PREINIT initshutdownscript stored in DB.
flan
flan