truenas-truecloud-patch/docs/how-it-works.md

How it works

Part of truenas-truecloud-patch.

What is actually patched

Nothing in TrueNAS's persistent database or configuration is modified (other than the boot-hook entry itself). On every boot, patch/apply.sh runs as a PREINIT script. It mounts a writable overlayfs over the relevant directories in /usr/ (upper layer in /run tmpfs), then patches b2.py and restic.py inside that overlay. The overlay is volatile — it exists only for the current boot — but the PREINIT script recreates it automatically on every subsequent boot. Nothing in /usr/ is written to directly.

PREINIT scripts are executed by middlewared, which by then has already imported the stock modules — so after patching, apply.sh schedules a single detached middlewared restart (transient systemd unit truecloud-mw-restart running patch/wait_restart.sh) that loads the patched modules once boot has actually settled: the script waits for the systemd boot job queue to drain and for the docker/apps state machine to reach a terminal state before restarting. Expect one middlewared restart shortly after every boot; the UI and API are briefly unavailable while it happens, and running services are not affected.

Module	What changes	Technique
providers	B2RcloneRemote gains get_restic_config() — skipped automatically if TrueNAS already provides one on the class. restic.py URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (b2:bucket:path), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed.	File patch applied inside the overlayfs upper layer
providers (UI)	The Angular bundle's filterByProviders binding is widened from ["STORJ_IX"] to ["STORJ_IX","S3","B2"]	In-place text replacement in the compiled JS chunk; original is backed up before patching
nested (opt-in)	_truecloud_nested.py is installed into plugins/cloud/, and plugins/cloud/{snapshot,crud}.py + plugins/cloud_backup/sync.py are patched so snapshot = true works on a dataset that has child datasets. See below.	New module + file patches inside the overlayfs upper layer

All changes are fail-safe: if a patch cannot be applied (e.g. TrueNAS restructured the relevant code), middlewared starts normally, the affected module is simply inactive, and the reason is logged to apply.log in your repo root. The two modules are independent — one failing or going native does not disable the other.

How persistence works

Two different things must survive two different events:

Event	What would be lost	What makes it survive
Reboot	The overlay holding the patched files lives in /run (tmpfs) and vanishes	The PREINIT hook re-runs apply.sh on every boot and schedules one middlewared restart to load the result
TrueNAS update	/usr/ is replaced entirely; custom files in /etc/ are wiped with the new boot environment	This repo lives on your data pool, and the hook registration lives in the TrueNAS config database — both survive updates. The first boot after an update is just a normal boot

What happens on every boot

1. middlewared starts with the stock (unpatched) modules. This is unavoidable: PREINIT scripts are executed by middlewared (ix-preinit.service → midclt call initshutdownscript.execute_init_tasks), so nothing registered there can run before it.

2. Pools import (ix-zfs.service), making /mnt/<pool> — and this repository — available.

3. apply.sh runs (ix-preinit.service). Before it patches anything it runs the compatibility preflight (tools/compat.py) against the middlewared that is actually installed, and any module whose assumptions no longer hold is not applied — see TrueNAS compatibility. What survives that check gets applied: it mounts the writable overlay (upper layer in /run), patches b2.py and restic.py on disk inside it, patches the UI bundle, and writes apply.log and hook_status.json.

   An incompatible module is skipped for this boot only. It is not the kill switch: install a release that supports your TrueNAS and the patch re-applies itself on the next boot, with no manual step. (The kill switch is permanent and is set only when TrueNAS has made the patch unnecessary — a different situation, and the opposite conclusion.)

4. A deferred restart is scheduled. The middlewared that is running imported the stock modules in step 1 and never re-imports, so the on-disk patch alone is not enough. apply.sh detects it was invoked by middlewared and creates a transient systemd unit (truecloud-mw-restart, via systemd-run --no-block) running patch/wait_restart.sh — detached so it cannot disrupt the remainder of the boot sequence.

5. Once boot has settled, the patch is re-applied and middlewared restarts once, importing the patched modules from the overlay. wait_restart.sh holds the restart until the systemd boot job queue has drained (so in-flight ix-* units like ix-reporting finish first) and middlewared's docker/apps startup has reached a terminal state — plain unit ordering cannot see either, and restarting middlewared while they run kills apps and dashboard reporting for the whole boot. S3/B2 backup support is then active until the next reboot, when the cycle repeats.

   The re-apply in that sentence is load-bearing, not a safety blanket. The overlay from step 3 sits inside /usr, so anything that remounts that hierarchy detaches it, and two ordinary things do exactly that after our hook has finished: another PREINIT script running systemd-sysext merge/refresh over /usr (an out-of-tree nvidia driver, say), and middlewared's own docker.configure_nvidia when it brings docker up. Init scripts run sequentially in id order, so a hook registered after ours always wins — and ordering them differently would still not help, because docker.configure_nvidia fires at runtime. wait_restart.sh therefore re-runs apply.sh at the point where boot has settled and every such remount is behind it, re-mounting the overlay if it was torn off (same upper layer, so files patched in step 3 reappear intact), then verifies the patch is really on the live path, restarts, and verifies again — retrying once if it was lost in between.

   This is the failure that made it necessary: on 2026-08-19 the overlay was mounted at 16:41:56 and a sysext refresh unmerged and remerged /usr four seconds later. The restart at 16:47:24 loaded stock modules, and every B2 backup failed for nineteen hours while apply.log said OK — because apply.log can only report what was written to disk, never what the restart imported. That second question is now asked from inside middlewared by an hourly CRITICAL alert (see Update alerts).

What you will observe: one middlewared restart shortly after every boot (a brief web UI/API blip; running services are unaffected). Between steps 3 and 5 there is a short window — typically well under a minute — where the UI already shows S3/B2 (the JS bundle is read from disk per request) but the backend is still stock. A backup job that fires inside that window fails once with NotImplementedError and succeeds on its next run; see Troubleshooting if it persists beyond boot. If the backend is still stock an hour after boot, middlewared raises the "installed but NOT loaded" alert rather than leaving you to notice via a failed backup.

Manual runs of bash patch/apply.sh never trigger the restart — that only happens in boot context. install.sh and recover.sh perform their own explicit restarts instead, which is why a manual re-apply must be followed by systemctl restart middlewared.

---

If TrueNAS adds native support

The patch is two independent modules, and each retires on its own — TrueNAS is likely to ship one of these natively long before the other, and a module going native must not take the other one down with it.

Module	What it does	Detected as native when
providers	B2/S3 credentials for TrueCloud Backup (b2.py, restic.py, UI dropdown)	B2RcloneRemote carries a real get_restic_config()
nested	Snapshots on datasets with child datasets (plugins/cloud/*)	the "no further nesting" validation is gone from plugins/cloud/crud.py

At every boot apply.sh checks both:

• One module goes native → that module is skipped and logged; the other keeps working, and the patch stays installed.
• Both are done (native, or nested was never enabled) → the kill switch (disabled file) is set, overlays are unmounted, and apply.log tells you to run uninstall.sh.

So on a box using only the provider patch, native B2 support retires the whole thing as before. On a box that also uses nested snapshots, native B2 support retires just that half.

Check the log after any TrueNAS update:

tail -20 /mnt/tank/truenas-truecloud-patch/apply.log

hook_status.json reports each module separately (module.providers, module.nested_snapshots) with an active flag and a reason.

Scenarios where the auto-detect may not fire (manual check needed):

Scenario	What happens	Action
B2 support added to a base class (not B2RcloneRemote directly)	__dict__ check misses it; our method shadows native	Uninstall manually
B2 credential schema changed (e.g. provider["account"] renamed)	KeyError on first backup	Uninstall or update the patch
URL builder fixed but B2 class unchanged	URL wrapper becomes a no-op; no harm, but patch is dead weight	Uninstall at your convenience

---

After a TrueNAS update

A TrueNAS update replaces /usr/ wholesale, wiping the patch. You do not need to reinstall: patch/apply.sh runs at every boot and re-applies itself from your clone. But it targets internal APIs with no stability contract, so an update can break it — and the failure is quiet by design (middlewared starts fine; the patch just doesn't).

Check the log after any TrueNAS update:

tail -30 /mnt/tank/truenas-truecloud-patch/apply.log
python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify

What you see	What it means
[OK] providers, [OK]/[SKIP] nested_snapshots	Fine. Nothing to do.
WARNING: … pattern not found (UI)	The Angular bundle changed. The UI dropdown reverts to Storj-only, but backups keep working — create tasks with create_task.py meanwhile, and open an issue with your TrueNAS version.
WARNING: truecloud-patch is NOT COMPATIBLE with this TrueNAS version	This TrueNAS changed middleware underneath the patch, and the named module was deliberately not applied — see incompatible.json for exactly which assumption broke. TrueNAS is left stock, so nothing is half-patched. Check TrueNAS compatibility, then bash update.sh once a release supports your version; it re-applies itself on the next boot. This is not the kill switch and needs no manual reset.
[FAIL] providers	Your B2/S3 backups will not run. middlewared is fine, but the credential/URL handling is gone. Open an issue with your version.
[FAIL] nested_snapshots	The stock guard is back, so tasks with snapshot = true on a nested dataset will fail validation. Turn the option off on those tasks until it's fixed.

"Fail-safe" means the box stays up — not that your backups keep running. A [FAIL] providers is a broken backup, so check the log rather than assume.

Then update the patch itself if a newer release fixes it:

bash /mnt/tank/truenas-truecloud-patch/update.sh

---

TrueNAS 26

TrueNAS 26 changes three things underneath the nested module. Each one alone is backup-breaking, and none of them is visible from the cloud_backup files:

what changed	what it would do
cloud_backup rewritten async → synchronous	an async def wrapper hands sync.py a coroutine where it unpacks a tuple
get_dataset_recursive() deleted from plugins/cloud/snapshot.py	NameError — the injected block called it out of the host module's namespace
plugins/zfs_/dataset.py and zfs_/snapshot.py deleted	zfs.dataset.query, zfs.snapshot.query and zfs.snapshot.delete all vanish. 26 uses filesystem.statfs and zfs.resource.*

All three are fixed as of v0.7.0, and 26 is supported.

The first two were straightforward: the patch reads which flavour of cloud_backup your box declares and injects the wrapper that matches (one implementation of the real logic, two thin wrappers), and it carries its own copy of the deleted helper.

The third was not, and it is the one that would have hurt most — zfs.snapshot.delete is what sweeps the recursive snapshot, and without it every run would orphan one snapshot per descendant dataset (250 on a real pool), forever, while reporting success.

The obvious port is to the public pool.dataset.query / pool.snapshot.query. Those methods exist, are documented, and are covered by iX's deprecation policy — and they are not like-for-like replacements. They apply a visibility policy: they hide the datasets TrueNAS considers its own (ix-apps/*, .system/*, .ix-virt/*). On a real pool that is 84 of 270 datasets, including live application data. Staging from that view would have omitted every one of them from the backup — and the planner would never have seen them, so they would not have appeared in its "skipped" list either. A green backup, quietly missing data. The snapshot query lies the same way, so the sweep would have orphaned one snapshot per hidden dataset.

No source analysis could have caught that. The methods are all present and correctly shaped. Only running it could, which is why it took a real 26 box.

So the module now follows one rule:

Read the truth from ZFS. Make changes through middleware.

Enumeration is zfs list — no policy can filter it, and it behaves identically on every release, which also means one code path instead of a version conditional. Mutation stays a middleware call, so TrueNAS's own bookkeeping stays consistent; an exact-name delete works fine even on a dataset the query hides. It is only enumeration that lies.

The snapshot delete still needs a namespace, and no single one spans every release — 24.10 and 25.04 have zfs.snapshot, 26 has only pool.snapshot, 25.10 has both. So it is resolved at runtime, by asking whether the namespace can actually delete. tools/compat.py asks the identical question against iX's source, and a test binds the two lists together, so what CI verifies and what runs cannot drift apart.

The one 26 changed that nothing warned about

Stock decides whether to take a recursive snapshot by its own rule, and on 26 that rule stopped being ours:

	decides recursive by
stock ≤ 25.10	get_dataset_recursive() — the same function this patch vendors
stock 26	filesystem.statfs: recursive = (path == the dataset's mountpoint)
this patch	get_dataset_recursive() — is a mounted filesystem child under the path?

Up to 25.10 those were the same question, so a snapshot the patch declined to stage provably had no children and stock's non-recursive delete was correct. On 26 they disagree: a dataset whose only descendants are zvols or legacy-mountpoint datasets gets a recursive snapshot, while the patch sees nothing to stage. Stock then destroys the parent only — and with no staging tree there was no sidecar, and the garbage collector only ever ran from the staging path. Nothing on the box would ever have found the children.

It was reproduced on a 26 VM (one orphan per zvol, every run, backup green) and closed: ownership of the sweep is no longer conditional on staging.

master (development after 26) does report BROKEN: iXsystems are still reshaping these functions there, renaming middleware → context and cloud_backup → entry and adding a required credentials parameter. That is a moving target and is deliberately not chased; the check keeps reporting it until it settles into a beta, which is when it becomes worth fixing. Until then, a box running master would simply not get the modules — apply.sh refuses to apply a module whose assumptions no longer hold, and says why in apply.log.