apply.sh only ever ADDED patches; there was no revert path anywhere. Disabling
removed the opt-in marker and then merely skipped re-applying -- but the overlay
persists for the whole boot, so the previously patched cloud/{snapshot,crud}.py,
cloud_backup/sync.py and _truecloud_nested.py were all still on disk, and
middlewared re-imported them on the restart install.sh performs.
It printed "DISABLED (stock guard restored)" while the feature kept running until
the next reboot. Someone disabling it because they were worried about it would
have believed it was off.
apply.sh now reverts on every not-needed path (opt-out, or superseded by native
support): remove the module FIRST -- every injected block is guarded by
`if _tc_nested is not None`, so the stock guard comes back even if a later step
fails -- then strip the appended blocks from the three patched files.
restic.py also carries a TRUECLOUD_PATCH block but belongs to the providers
module; reverting it would silently break B2 backups, so it is explicitly
excluded. Verified: the three nested files restore byte-for-byte to stock, the
module is removed, and restic.py's block survives.
install.sh --disable also tears the staging tree down first, since those bind
mounts pin ZFS snapshots that could otherwise never be destroyed.
Updating WITHOUT the flag was always correct and is unchanged: the nested module
is never installed into middleware unless explicitly enabled.
109 tests, ruff and shellcheck clean.
The "Automated releases" entry was written under v0.3.0, but that commit landed
after the v0.3.0 tag -- the CHANGELOG was claiming the release contained something
it did not. Moved to its own version rather than left as a quiet inaccuracy.
Bumps VERSION to 0.3.1 across all four scripts, which the new consistency check
now enforces.
Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never
released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the
incident that took every app down.
Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is
the matching CHANGELOG.md section -- one source of truth for release notes, so
there is no second place for them to be wrong.
The workflow refuses to publish when:
- the tests, ruff, or bash -n fail (a tagged commit is what people install; it
must be at least as good as main)
- the tag does not match the VERSION= declared by every script
- CHANGELOG.md has no section for the tag, or the section is empty
That version check is not theoretical: VERSION= had drifted to three different
values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing
noticed until this release. tests/test_release_notes.py now asserts the scripts
agree with each other and with the newest CHANGELOG entry, so the drift cannot
come back.
workflow_dispatch takes an existing tag, so releases can be backfilled for tags
that were pushed before this existed.
106 tests, ruff and shellcheck clean.
Commit 47cdf72 shipped a pattern that matched one closing paren and emitted one,
netting an extra `)` in the Angular bundle:
c(2,"filterByProviders",["STORJ_IX","S3","B2"]))("required",!0)
^^ syntax error
The TrueNAS web UI went blank. Worse, MARKER was now present in the file, so
every later run reported "already patched" and skipped -- the patch could not
heal itself, and the bundle had to be hand-restored from the .pre-truecloud-patch
backup.
patch_ui.py now compares the parenthesis balance before and after substitution and
refuses to write if it changed. A bundle we cannot patch correctly is left exactly
as it was: an unpatched UI is a missing dropdown entry, a corrupted one is a dead
web UI.
Tests cover the real regression (verbatim 47cdf72 pattern) end to end: it still
matches, the balance still shifts, main() refuses, and the file on disk is
byte-for-byte unchanged. README documents the manual recovery for anyone who
already hit it.
93 tests, ruff and shellcheck clean.
An unattended scheduled backup of a live 252-dataset pool ran through the staging
tree end to end:
task 5 /mnt/Tap SUCCESS 18m14s
- 252 datasets recursively snapshotted; 173 bind mounts built and verified
- zero orphaned ZFS snapshots and zero stale mounts afterwards -- the failure
that would otherwise have accumulated 251 snapshots on every single run
- the same task previously stalled at 74% for over 12 hours reading live files
The README said the mount --bind staging step had not been exercised by a live
backup run. That is no longer true, so it is removed rather than left to
understate the state of the code.
The advice to verify your own first backup actually contains child-dataset data
stays -- that one is not boilerplate.
patch_ui.py rewrites minified third-party JavaScript by regex and had no tests.
It is the easiest place in this project to do real damage: a pattern that matches
nothing silently leaves the dropdown Storj-only, and one that consumes a paren
too many is a syntax error in the bundle that blanks the entire TrueNAS web UI.
Tests run the real patterns against verbatim snippets from a TrueNAS 25.x
chunk-*.js (the chained property(...)(...) form) and a 24.x literal array, and
assert: exactly one match, all three providers present, the paren balance is
UNCHANGED, surrounding code untouched, re-patching is a no-op, unrelated JS is
never matched, and every pattern stays anchored to filterByProviders.
The paren-balance assertion is the load-bearing one -- a plausible-but-wrong
pattern that eats both parens and re-emits none shifts the delta from 1 to 2 and
is caught.
Also restore the comment explaining why the 25.x pattern is shaped the way it is,
and correct the module docstring, which showed the binding with a single closing
paren; the real bundle wraps it in a chained property call.
90 tests, ruff and shellcheck clean.
delete_snapshot_tree removed the parent and every child snapshot individually.
On a real pool `zfs snapshot -r` creates one snapshot per descendant dataset --
252 on Tap -- so cleanup was 252 sequential middleware calls.
Slow, but the real problem is that it is not atomic: a job killed part-way
through the sweep leaves behind exactly the orphaned snapshots this function
exists to prevent.
zfs.snapshot.delete accepts {"recursive": True}, which destroys the parent and
all children in one call. Use that as the fast path and keep the name-by-name
sweep as the fallback -- it is still needed when the parent is already gone
(stock's finally can win the race once our mounts are released), which makes a
recursive delete fail while the children survive.
The test fake now emulates real `zfs destroy -r` semantics, so a test cannot pass
while the shipped code deletes only the parent.
76 tests, ruff and shellcheck clean.
The probe searched plugins/cloud/crud.py for the contiguous string
"no further nesting". Stock does not contain it. The message is split across
adjacent string literals:
verrors.add(f"{name}.snapshot", "This option is only available for datasets that have no further "
"nesting")
Python concatenates those at runtime, so the errmsg IS contiguous and CRUD_BLOCK's
runtime filter matches correctly -- but the SOURCE never contains the whole
phrase. The probe therefore found nothing, concluded iX had removed the guard, and
skipped the nested module as "already native" on every boot. apply.log would say
"TrueNAS now handles nesting natively" and the feature would never work.
It fails safe -- the stock guard stays in place, so no backup could be
misconfigured and no data was at risk -- but the module was 100% dead.
Verified against real middlewared on a live box: the probe returned native=yes
(wrong) before this change and native=no (correct) after.
The probe now strips whitespace and quote characters before matching, which is
robust to any wrapping or concatenation style. Added a regression test that
EXECUTES apply.sh's own probe code (extracted, not reimplemented -- a
reimplementation would pass while the shipped probe stayed broken) against the
real wrapped source, a single-line variant, and a three-way split.
75 tests, ruff and shellcheck clean.
Audit
-----
- create_task.py verify failed on a DEFAULT install. hook_status.json emitted a
per-file entry for the nested module with ok:false whenever the feature was
switched off -- the default -- so verify printed [FAIL] and exited 1, right
after the README tells users to run it. Status is now per MODULE with an
`active` flag, and verify renders an inactive module as [SKIP].
- A partial apply suppressed the middlewared restart. The exit code conflated
"nothing applied" with "one module applied, one failed", so a failing providers
patch would prevent the restart that a freshly-applied nested patch needs,
leaving it on disk and never loaded. Exit 2 now means partial and the restart
still fires.
- The native-nested probe could never fire. It scanned crud.py for the guard
message, but our own injected block quotes that message, so once applied the
probe would always conclude the guard was still present. It now reads only the
stock portion of the file.
- recover.sh did not unmount staging trees, so an emergency recovery left bind
mounts pinning ZFS snapshots that could then never be destroyed.
- uninstall.sh deleted sidecar files without reading them. A sidecar is the only
record that an interrupted run's snapshot tree is still on disk; both scripts
now name the snapshot before clearing it.
- Removed a dead branch in the restart gate (unreachable: the kill switch exits).
Refactor
--------
- Staging teardown had been copy-pasted into uninstall.sh and recover.sh -- two
untested shell copies of the fiddly depth-ordering and lazy-umount logic. Both
now call `python3 patch/truecloud_nested.py cleanup`, so there is exactly one
implementation and it is the one under test.
- Dropped the in-memory ACTIVE dict. The sidecar file was already the source of
truth; a second in-process record could only desync -- and it is precisely the
middlewared-restart case (which empties it) that must not orphan a snapshot
tree. One record, on disk, or none.
Not done: the overlay-unmount loop is duplicated across apply.sh/uninstall.sh/
recover.sh. It is pre-existing, and apply.sh runs at PREINIT under a tight
timeout -- giving it a source dependency would trade 10 lines of duplication for
a boot-time failure mode.
74 tests, ruff and shellcheck clean.
Adds a second, opt-in module to the patch: TrueCloud Backup's "Take Snapshot"
option now works on datasets that have child datasets — i.e. every pool running
Apps, where each app is its own dataset and stock refuses the config with
"This option is only available for datasets that have no further nesting".
Stock's guard is correct and is not merely deleted. create_snapshot() already
takes a recursive ZFS snapshot but points restic at the PARENT dataset's
.zfs/snapshot/, where ZFS does not expose child datasets — so removing the check
would make restic walk a near-empty tree, report SUCCESS, and upload almost
nothing. The missing traversal is implemented instead: each descendant's own
.zfs/snapshot is bind-mounted into a staging tree and restic is pointed at that.
The guard is relaxed only after the machinery is in place.
The patch is now two modules (providers, nested) which retire independently as
TrueNAS ships each capability natively; the kill switch fires only when both are
done.
Also adds CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13; 67 tests),
including a pass that compile()s the Python source this patch appends into live
middlewared modules — previously unchecked, and a syntax error there breaks the
box at boot.
Nested snapshots are OFF by default (install.sh --enable-nested-snapshots).
The mount --bind staging step has not yet been exercised by a live backup run.
The native-support check looked only for native B2 restic support and, on
finding it, set the kill switch and disabled the whole patch. That was fine when
providers were the only thing here. With nested-dataset snapshots in the patch it
is wrong: TrueNAS is likely to ship one capability long before the other, and a
single all-or-nothing kill switch would silently take a still-needed module down
with the superseded one.
apply.sh now treats the patch as two independent modules:
providers b2.py + restic.py + the UI credential dropdown
native when B2RcloneRemote carries a real get_restic_config()
nested plugins/cloud/{snapshot,crud}.py + cloud_backup/sync.py (opt-in)
native when the "no further nesting" validation is gone from
plugins/cloud/crud.py
Each is detected and skipped on its own. The kill switch fires only once BOTH are
done (native, or nested was never enabled). The UI patch belongs to providers and
is skipped with it. The deferred middlewared restart now fires when ANY
still-needed module landed -- keying it off providers alone would have left a
freshly-patched nested module on disk and never loaded on a native-B2 box.
hook_status.json reports each module with an active flag and a reason.
README: dropped the warning boxes and the disclaimer's fear-bulleting in favour
of plain statements, documented the two-module design and the per-module
auto-disable, and added a Development section disclosing AI assistance. The one
caveat kept, as a plain sentence rather than a banner: the mount --bind staging
step has not yet been exercised by a live backup run.
67 tests, ruff and shellcheck clean.
os.path.isdir() returns False both when a snapshot directory does not exist and
when it cannot be stat'd. Staging aborted either way -- correct -- but reported
every case as "has no snapshot", which sends you hunting for a snapshot that is
sitting right there.
Found while dry-running the planner against a real recursive snapshot of Tap:
running as a non-root user, /mnt/Tap/apps/paperless/data is mode 0700 and the
probe reported "has no snapshot" when `zfs list` showed the snapshot present.
Middleware runs as root so this would not fire in production, but a backup
system must not misreport why it failed.
plan_staging now takes a probe() that classifies the path as ok / missing /
unreadable, and the error names which.
Dry-run results against Tap (250 datasets, real `zfs snapshot -r`):
- 170 mounted filesystems under /mnt/Tap/, and the plan produces exactly 170
descendant mounts -- no omissions
- 18 legacy-mountpoint datasets reported as skipped, never dropped silently
- Tap/ix-apps mounts at /mnt/.ix-apps, correctly outside the backup path
- parent snapshot exposes 0 entries under /apps; the staged sources expose 71,
and lidarr/config resolves with lidarr.db present
- snapshot_tree_names() identifies all 250; deleting only the parent (what stock
does) leaves 249 orphans, and the sweep clears them
Opt-in
------
Nested-dataset snapshot support changes how backups read their source data, so
it is now off by default and gated behind a marker file:
install.sh --enable-nested-snapshots
install.sh --disable-nested-snapshots
With neither flag install.sh preserves the current setting, so a routine
`git pull && bash install.sh` can never silently flip it. When disabled,
apply.sh skips the patch entirely and the stock guard remains. uninstall.sh
tears down staging mounts and removes the marker.
Snapshot lifecycle
------------------
zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls
it with no options. Stock is safe only because its validation means recursive is
never True in the field. Enabling nested datasets makes recursive snapshots real:
the parent then has one child snapshot per descendant dataset (160+ on an Apps
pool), so stock's delete would orphan every child on EVERY successful run.
The patch now owns the lifecycle end to end:
- delete_snapshot_tree() sweeps the parent and all children, and is idempotent
against stock's finally winning the race once our mounts are released
- on a staging failure the tree is deleted here, because sync.py never completes
`snapshot, local_path = await create_snapshot(...)` and so its finally deletes
nothing at all
- the snapshot is recorded in a sidecar file before anything is mounted, so a
middlewared restart mid-backup cannot orphan it
- a crashed run's snapshot tree is reclaimed on the next run instead of being
overwritten and leaked
Silent-omission fix
-------------------
The dataset list is now enumerated AFTER the snapshot. Read beforehand it could
miss a dataset created in the gap, which the recursive snapshot would capture but
the staging plan would not -- silently omitting its data. Read afterwards, an
unsnapshotted dataset trips the staging check and fails the run loudly.
Also from the audit
-------------------
- plan_staging scopes by dataset name, so skipped-dataset warnings no longer
include every mountpoint-less dataset on the box, which buried the ones that
matter
- staging_root_for rejects "." / ".." components that would escape the staging
base, and resolves STAGING_BASE at call time rather than freezing it into a
default argument
- uninstall.sh no longer `rm -rf`s a tree that may still contain live bind
mounts, and unmounts by path depth rather than string length
- apply_plan takes an injectable isdir; verify_staged drops an unused parameter
- pin the shellcheck action instead of tracking @master
61 tests, ruff and shellcheck clean.
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:
This option is only available for datasets that have no further nesting
That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.
The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.
Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.
Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays
The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.
Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.
Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
The tool created cloud_backup tasks via POST /api/v2.0/cloud_backup, which is deprecated
and removed in TrueNAS 26.04. It now calls the middleware directly with midclt
(cloudsync.credentials.query / cloud_backup.query / cloud_backup.create), so it runs on
the TrueNAS host with no host address or API key. --host/--api-key/--insecure are kept
accepted-but-ignored for compatibility. Dropped the ssl/urllib HTTP client. v0.2.0.
flan