flan
ce6998a935
The most important finding is that the FIRST audit's fix was wrong. _can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines `delete` on the BASE class and dispatches to self.do_delete at call time, so a bound `delete` exists on every CRUDService subclass whether or not it still implements one. The check was therefore answering "is this a CRUDService?" — precisely the weaker "is the namespace registered?" question its own docstring said must never be asked. It would still have picked a gutted pool.snapshot and failed every delete. It now walks the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining delete/do_delete counts. The test double was equally wrong: it modelled a gutted service as object(), a shape middlewared cannot produce, so the test passed against a fake it could never have caught in the field. It is now CRUDService-shaped. Also: - The recursive delete's fast path returned [] without confirming anything was destroyed. A delete that returns cleanly is not proof — iX has already gutted pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read "no survivors" as a clean sweep, dropped the sidecar (the only record), and would have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the by-name sweep trusts ZFS rather than the API's return value. - When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record clears), a lost record does not. - _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist; failing to write it must never be invisible. - stage_nested now refuses UP FRONT when middleware has no usable snapshot delete, rather than discovering it after restic has already run. Tests. The autouse fixture added in the last commit did not work: `runner=_run`, `mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__ at def time, so monkeypatching the module attribute never reached them. 19 tests were still reading the real mount table — one matching name from running a real `umount` on the NAS — and the retry loop really slept. All three are late-bound now; the suite reads nothing outside tmp_path and runs in 1.1s. Every mutation the audit reported as SURVIVING now fails the suite: the naive delete check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager service resolution, compat's method check, compat's unknown handling, a single-quoted filtered query in apply.sh, and the get-service assumption. Also: fingerprint() folded `unknown` problems into a broken module, so one transient 429 rewrote the bug report and the next clean run rewrote it back. Problems are state-tagged; only definite breakage is digested. Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup 0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep child dataset.
truenas-truecloud-patch
Extends TrueNAS SCALE's TrueCloud Backup to:
- back up to Backblaze B2 and any S3-compatible provider, not just Storj;
- snapshot datasets that have child datasets — which is every box running Apps.
Requires TrueNAS SCALE 24.10 or newer. TrueCloud Backup does not exist before
that, and install.sh will refuse.
Storj raised the price of their TrueNAS-integrated tier from $5/month to $50/month in 2026. TrueCloud Backup is the only native TrueNAS feature that gives you pre-backup ZFS snapshots, restic dedup, scheduled tasks with UI progress, and dataset-lock integration. Running restic by hand loses all of it. This gets the feature back with storage you already pay for.
Install
Clone it onto a pool (not the boot device — that is wiped on TrueNAS upgrades),
then run install.sh as root:
git clone https://github.com/sudolulo/truenas-truecloud-patch.git \
/mnt/tank/truenas-truecloud-patch # replace `tank` with your pool
cd /mnt/tank/truenas-truecloud-patch
sudo bash install.sh
That registers a PREINIT boot hook, patches middleware in a volatile overlay, and restarts middlewared. It survives TrueNAS updates — the patch is re-applied at every boot, never written to the system dataset.
Nested-dataset snapshots are opt-in:
sudo bash install.sh --enable-nested-snapshots
Then create a TrueCloud Backup task in the UI with a B2 or S3 credential, or from the CLI.
Check it worked:
sudo python3 patch/create_task.py verify
If something is wrong, the reason is in apply.log — start at
Recovery.
TrueNAS compatibility
| TrueNAS | B2/S3 providers | Nested snapshots | Hardware-verified |
|---|---|---|---|
| 24.10.2.4 | ok | ok | — |
| 25.04.2.6 | ok | ok | — |
| 25.10.4 | ok | ok | nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m |
| 26.0.0-BETA.3 (unreleased) | ok | ok | — |
| master (unreleased) | BROKEN | BROKEN | — |
| verdict | meaning |
|---|---|
| ok | Every assumption the patch makes about middleware still holds. |
| BROKEN | middleware changed underneath the patch. apply.sh refuses to apply that module on this version and leaves TrueNAS stock, so backups keep working — without the module's feature. |
| native | TrueNAS does this itself now. The module retires; it is not a failure. |
"ok" means the patch's assumptions hold, checked automatically against iX's source. It does not mean a human ran a backup on it — that is the Hardware-verified column, which is filled in by hand and only by doing it.
The table is regenerated daily by CI against iXsystems' actual middleware source — it is not a claim somebody typed once and forgot.
On TrueNAS 26: the patch was run on a real TrueNAS 26.0.0-BETA.1 install — a 274-snapshot recursive backup of a 292-dataset pool, followed by a byte-identical restore of a four-level-deep child dataset. The Hardware-verified column tracks the newest beta iX has tagged (currently BETA.3), so it does not carry that mark: a build nobody has actually run a backup on does not get credit for one.
TrueNAS 26: nested snapshots are not supported yet, and upgrading will not break
you. 26 rewrites cloud_backup and deletes the ZFS methods this module calls. On
26 apply.sh finds that the assumptions no longer hold and does not apply the
module: TrueNAS is left stock, B2/S3 keeps working, nested datasets are simply not
covered, and the reason is named in apply.log. A broken backup is worse than a
missing feature. Details: How it works.
Updating
cd /mnt/tank/truenas-truecloud-patch
sudo bash update.sh # newest release
sudo bash update.sh --check # what would change?
sudo bash update.sh --rollback # back to the previous version
update.sh refuses to run on a dirty checkout, pins you to a release tag, and keeps
the previous revision so a rollback is one command. There is no auto-update: this
patches system internals as root, and a bad commit reaching your box unattended would
detonate on the next reboot — v0.0.4 shipped exactly such a bug and took 54 apps
down. The manual step is the safety gate.
Update alerts
When a newer release exists, the patch raises a TrueNAS alert (the bell in the UI) telling you so. It's on by default and checks once a day.
bash install.sh --no-update-alerts # turn it off
bash install.sh --update-alerts # turn it back on
It will not nag you about a README. A release whose CHANGELOG contains only a
### Docs section changed no code, and raises nothing. Anything that touched the
system raises an INFO alert; a release with a ### Security section raises a
WARNING. The CHANGELOG's own section headings are the signal, and a security fix
anywhere in the range escalates the whole span — a docs-only release on top of a
security fix still reports as security.
Release candidates never alert. They are invisible to update.sh and to the
alert, which both take the newest plain vX.Y.Z tag. That is what lets debugging
happen in -rc tags instead of in your notification bell — see
Releasing.
The changelog is read from whichever forge origin points at, derived from the
remote rather than hard-coded. That is not cosmetic: when the changelog cannot be
read, the alert deliberately fires anyway rather than risk hiding a security
fix — so a wrong URL would not silence the alert, it would make it fire on every
release, including the documentation-only ones this section promises to suppress.
How it works, and why it's built this way
TrueNAS cannot raise an alert from the CLI — midclt exposes only
alert.dismiss, alert.list, alert.list_categories, alert.list_policies and
alert.restore. Alert creation is internal to middlewared, and none of its ~60
one-shot alert classes is generic enough to reuse. So the only way to get a real
alert is to register an AlertSource, which is what patch/alert_source.py does.
That is also the least invasive thing this patch does:
| providers module | modifies stock files (appends code to b2.py, restic.py) |
| nested module | modifies stock files (3 middleware modules) |
| update alert | adds one file. Modifies nothing. |
It's the native mechanism — the same one every built-in TrueNAS alert uses — and TrueNAS polls it itself, so there is no cron job and no systemd timer.
- Fail-safe. Every error path returns
None. It cannot take middlewared down. - Read-only.
git ls-remoteplus an HTTPS fetch of the CHANGELOG. It never writes to.git, so it cannot leave root-owned objects behind the way agit fetchfrom middlewared (which runs as root) would. - Removed by
uninstall.sh.
It only tells you. It never updates anything — see Updating.
Uninstall
bash /mnt/tank/truenas-truecloud-patch/uninstall.sh
Replace the path with your clone location. Removes the PREINIT hook, unmounts the overlay (restoring the original backend files immediately), and restores the original UI bundle from backup.
Documentation
| Nested-dataset snapshots | Why stock refuses, what this does instead, and how to verify your backups actually contain the data |
| How it works | What is patched, how it survives updates, the boot sequence, and what happens when TrueNAS goes native |
| Recovery | middlewared won't start, blank web UI, verify shows FAIL |
| CLI | Creating tasks with create_task.py |
| Development | Tests, CI, and the release process |
What's in the repo
| Path | What it is |
|---|---|
install.sh |
Register the boot hook, patch, restart middlewared. Also --enable/--disable-nested-snapshots. |
update.sh |
Fetch and apply a newer release. --check, --rollback, --to, --main. |
uninstall.sh |
Remove everything. |
recover.sh |
Emergency: kill switch + restart against stock files. |
patch/apply.sh |
The PREINIT script. Runs at every boot. |
patch/truecloud_nested.py |
Nested-dataset staging: plan, mount, verify, tear down, sweep snapshots. |
patch/create_task.py |
Create tasks with S3/B2 credentials; verify the patch state. |
tools/compat.py |
What the patch assumes about middlewared — and the checker. Run daily by CI and at every boot. |
release.sh |
Cut a release. Two stages, and the second is refused without the first. |
Before you install
- This is unofficial and not affiliated with iXsystems.
- It patches internal middleware APIs with no stability contract. Every patch is fail-safe: if it cannot apply, middlewared starts normally and the reason is logged.
- Test your restores. True of any backup; more so here. See Verifying it works.
- Filing a TrueNAS bug? Remove the patch first and reproduce on a stock system.
- Provided as-is, no warranty. See LICENSE.
Parts of this project were written with AI assistance (Claude); all of it is reviewed and tested before release. Bugs are mine.