Document the full boot sequence (stock start, pool import, PREINIT
patching, deferred restart via truecloud-mw-restart), the reboot vs
OS-update survival table, the short unpatched window after boot, and
why manual apply.sh runs require an explicit middlewared restart.
Add a troubleshooting entry for backups failing with
NotImplementedError after a reboot, with ordered diagnostic commands.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
apply.sh now inspects B2RcloneRemote.__dict__ before patching. If TrueNAS
has shipped get_restic_config natively, it sets the kill switch, unmounts
overlays, and logs a clear instruction to run uninstall.sh.
Also: drop all conditional 'if read-only' language — overlay is always
mounted unconditionally since /usr is always immutable on TrueNAS SCALE.
README updated with auto-disable behaviour and revised native-support table.
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.
- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
URL-fix logic now lives once in sitecustomize._tc_fix_restic_cmd.
apply.sh's restic.py BLOCK delegates to it instead of repeating
the ~40-line implementation.
Additional safety: the BLOCK now guards its get_restic_config
reference with try/except NameError, so a future TrueNAS that
restructures restic.py won't cause an import error.
README: updated Backend table and disclaimer to reflect graceful
degradation; added "If TrueNAS adds native support" section that
covers all five upgrade scenarios (safe pass-through, base-class
shadowing risk, schema-change risk, etc.).
restic 0.16.x changed the B2 URL format to use a colon between bucket and
path (b2:bucket:prefix) instead of a slash. The middlewared URL builder
produces b2:/bucket/path; restic then validates the full string after 'b2:'
as a bucket name, which fails because the slash is not in [a-z0-9-].
Fix the restic.py wrapper to strip the leading slash and replace the first
slash with a colon: b2:/bucket/path -> b2:bucket:path.
Also fix the hasattr bug in sitecustomize.py _patch_b2: hasattr() returns
True for methods inherited from the base class (which raises
NotImplementedError), causing the patch to be silently skipped. Use
'get_restic_config' not in cls.__dict__ instead.
site.getsitepackages()[0] returns /usr/local/lib/python3.11/dist-packages/ on
TrueNAS 25.x but middlewared lives in /usr/lib/python3/dist-packages/.
sitecustomize.py was installed to the wrong directory and Python never loaded it.
Fix: derive SITE_PKG from middlewared.__file__ so the overlay and sitecustomize.py
land in the correct directory.
Also add direct patching of b2.py and restic.py in the overlay as the primary
backend approach — more reliable than an import hook since it works regardless
of Python's site initialisation configuration. apply.sh now also writes
hook_status.json at boot time so 'verify' shows OK without requiring a backup run.
Also fixes incorrect middlewared log path in README (/var/log/middlewared/middlewared.log
→ /var/log/middlewared.log) and simplifies the verify troubleshooting note.
- create_task.py list-tasks: crash on null credentials.provider
(`creds.get("provider", {})` returns None when key exists but is null;
switch to `(creds.get("provider") or {})`)
- sitecustomize.py: write hook_status.json after each module, not only
when both have loaded; S3-only users (B2 module never imported) now
get a status file from verify instead of "No status file found"
- README: add filesystem find + sqlite3 DB query to the emergency
recovery section so users can locate their clone path when middlewared
is down and midclt is unavailable
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
Users who delete the cloned repo after install had no way to uninstall
without re-cloning. Now install.sh copies uninstall.sh to PATCH_DIR
alongside recover.sh, so the uninstall path is always the stable
/data/truecloud-patch/uninstall.sh. README updated to match.
- install.sh: print 'verify' command immediately after successful restart
so users know to confirm the backend patch loaded before creating tasks
- README: correct --insecure description; it controls TLS to the TrueNAS
API (where the API key is transmitted), not the S3 endpoint — previous
wording implied it was safe to use for S3 self-signed certs
patch_ui.py:
- Write Angular bundle atomically via tmp + os.replace, matching the pattern
already used by _record_status. Prevents a corrupt bundle if the write is
interrupted mid-boot.
sitecustomize.py:
- Tighten _record_status count barrier comment to name _Finder._targets as
the canonical count, making the coupling visible to future editors.
uninstall.sh:
- Verify middlewared restarted cleanly after uninstall, with journalctl
guidance on failure — matching recover.sh's existing pattern.
apply.sh:
- Change second line of site-packages error block from WARNING: prefix
(misleading for an instructional message) to a plain Run: hint.
create_task.py:
- Guard __doc__ against None in epilog extraction so -OO does not crash.
README.md:
- Split Emergency recovery into three named subsections: middlewared won't
start, web UI is blank or broken (corrupt bundle recovery), and backend
verify shows FAIL. Each gives direct commands and escalation steps.
- Add Restoring from a TrueCloud Backup section: finding the restic binary,
gathering credentials, provider-specific env var setup for B2 and S3,
listing and restoring snapshots, and operational notes on restore hygiene.
- Clarify that hook_status.json is written once both target modules have
loaded (not necessarily at the instant middlewared starts).
sitecustomize.py: _patch_restic no longer reimplements get_restic_config.
It now wraps the original: calls _orig(cloud_backup) to get a ResticConfig,
then post-processes only the -r argument to fix "b2:/bucket" → "b2:bucket"
when the URL contains a stray leading slash (the stock bug for empty-hostname
providers). Uses dataclasses.replace() to build the corrected result so new
ResticConfig fields added in future TrueNAS versions pass through unchanged.
This eliminates the transfer_setting gap, env dict mutation, and frozen-copy
drift that would occur over time.
Also adds a status file mechanism: sitecustomize.py writes
/data/truecloud-patch/hook_status.json atomically after both patches have
reported success or failure. This gives a machine-readable signal that the
hook fired correctly — without requiring log scraping.
create_task.py: new "verify" subcommand reads the status file and prints a
human-readable summary. Does not require --host or --api-key. --host and
--api-key are now optional at the parser level and validated only for
subcommands that actually need an API connection.
README: update troubleshooting to use "create_task.py verify" instead of
the manual Python introspection one-liner.
If the patch ever prevents middlewared from starting, users now have a
clear escape hatch that requires no knowledge of Python internals:
bash /data/truecloud-patch/recover.sh
Or at a bare shell prompt:
touch /data/truecloud-patch/disabled
systemctl restart middlewared
sitecustomize.py checks for /data/truecloud-patch/disabled at Python
startup and skips the import hook entirely when the file exists.
apply.sh does the same so the PREINIT script also does nothing on reboot.
recover.sh is copied to /data/truecloud-patch/ by install.sh so it is
available even without the original repo directory.
README gains an "Emergency recovery" section above Troubleshooting.
Patches middlewared at runtime via sitecustomize.py (no file edits to /usr/)
and widens the UI credential dropdown from Storj-only to S3+B2+Storj.
Persists across TrueNAS updates via PREINIT initshutdownscript stored in DB.
flan