On 24.10 (Electric Eel) credentials["provider"] is the type string with
account/key in credentials["attributes"]; 25.04+ moved them into a
provider dict. The injected get_restic_config only handled the newer
shape and raised TypeError on 24.10 at task creation (#1).
The method now detects the schema and reads credentials from the right
place on both. create_task.py list-credentials and list-tasks use the
same schema-agnostic lookup.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds. Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.
Bumps all scripts to v0.0.3.
The guard in the b2.py patch block skipped setting get_restic_config when
something caused B2RcloneRemote.__dict__ to already contain it at import
time (e.g. a TrueNAS version that adds a NotImplementedError stub). Remove
the guard and always assign, which is safe: the native-support kill switch
already prevents patching when TrueNAS ships a real implementation.
Also update the native-support check to distinguish a stub (source contains
NotImplementedError) from a working implementation, so a stub does not
trigger the kill switch and block all future patching.
apply.sh now inspects B2RcloneRemote.__dict__ before patching. If TrueNAS
has shipped get_restic_config natively, it sets the kill switch, unmounts
overlays, and logs a clear instruction to run uninstall.sh.
Also: drop all conditional 'if read-only' language — overlay is always
mounted unconditionally since /usr is always immutable on TrueNAS SCALE.
README updated with auto-disable behaviour and revised native-support table.
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.
- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
URL-fix logic now lives once in sitecustomize._tc_fix_restic_cmd.
apply.sh's restic.py BLOCK delegates to it instead of repeating
the ~40-line implementation.
Additional safety: the BLOCK now guards its get_restic_config
reference with try/except NameError, so a future TrueNAS that
restructures restic.py won't cause an import error.
README: updated Backend table and disclaimer to reflect graceful
degradation; added "If TrueNAS adds native support" section that
covers all five upgrade scenarios (safe pass-through, base-class
shadowing risk, schema-change risk, etc.).
restic 0.16.x changed the B2 URL format to use a colon between bucket and
path (b2:bucket:prefix) instead of a slash. The middlewared URL builder
produces b2:/bucket/path; restic then validates the full string after 'b2:'
as a bucket name, which fails because the slash is not in [a-z0-9-].
Fix the restic.py wrapper to strip the leading slash and replace the first
slash with a colon: b2:/bucket/path -> b2:bucket:path.
Also fix the hasattr bug in sitecustomize.py _patch_b2: hasattr() returns
True for methods inherited from the base class (which raises
NotImplementedError), causing the patch to be silently skipped. Use
'get_restic_config' not in cls.__dict__ instead.
Two bugs:
1. hasattr(B2RcloneRemote, "get_restic_config") returned True because the base
class defines the method (it just raises NotImplementedError). The method was
never added to B2RcloneRemote. Fixed: use __dict__ check instead.
2. "skip if TRUECLOUD_PATCH marker present" prevented a corrected patch block
from replacing a previously-written buggy one without clearing the overlay.
Fixed: always strip any existing TRUECLOUD_PATCH block and rewrite it fresh
using a Python heredoc. Each apply.sh run now self-corrects to the latest
version of the patch.
site.getsitepackages()[0] returns /usr/local/lib/python3.11/dist-packages/ on
TrueNAS 25.x but middlewared lives in /usr/lib/python3/dist-packages/.
sitecustomize.py was installed to the wrong directory and Python never loaded it.
Fix: derive SITE_PKG from middlewared.__file__ so the overlay and sitecustomize.py
land in the correct directory.
Also add direct patching of b2.py and restic.py in the overlay as the primary
backend approach — more reliable than an import hook since it works regardless
of Python's site initialisation configuration. apply.sh now also writes
hook_status.json at boot time so 'verify' shows OK without requiring a backup run.
Also fixes incorrect middlewared log path in README (/var/log/middlewared/middlewared.log
→ /var/log/middlewared.log) and simplifies the verify troubleshooting note.
TrueNAS 25.x mounts /usr as a read-only filesystem. Writing
sitecustomize.py to site-packages and patching the Angular bundle
both fail with EROFS.
Fix: mount a writable overlayfs on each target directory before
writing to it. Upper/work dirs live in /run (tmpfs), so overlays are
volatile per boot and are recreated by apply.sh on every PREINIT run
before middlewared starts.
apply.sh:
- Add _ensure_writable(dir, tag): probes writability; mounts overlay
in /run/truecloud-{tag}-{upper,work} if the directory is read-only;
detects if the overlay is already mounted (idempotent)
- Call _ensure_writable before site-packages writes (tag "sc")
- Detect webui dir with bash loop; call _ensure_writable before
patch_ui.py (tag "ui") — non-fatal if mount fails
uninstall.sh:
- Add overlay unmounting section after file restoration and before
rm -rf, so the lower layer's originals are exposed immediately
- Move _restore_failed exit 1 to after unmount so overlays are
cleaned up even on partial failure
- Update "no backup files" message for immutable OS context
TrueNAS 25.x changed how Angular emits the filterByProviders binding.
Previously a static inline array ("filterByProviders",["STORJ_IX"]),
it is now a pureFunction call:
pe(slot, factory, component.CloudSyncProviderName.Storj)
Add a _PATTERNS list tried in order, with a _match_pattern() helper.
The 25.x pureFunction pattern matches on the stable TypeScript enum
name (CloudSyncProviderName.Storj); the minified variable names and
slot index are matched with \w+ / \d+. Both patterns replace the
binding with ["STORJ_IX","S3","B2"]. MARKER and backup/restore logic
are unchanged.
- create_task.py list-tasks: crash on null credentials.provider
(`creds.get("provider", {})` returns None when key exists but is null;
switch to `(creds.get("provider") or {})`)
- sitecustomize.py: write hook_status.json after each module, not only
when both have loaded; S3-only users (B2 module never imported) now
get a status file from verify instead of "No status file found"
- README: add filesystem find + sqlite3 DB query to the emergency
recovery section so users can locate their clone path when middlewared
is down and midclt is unavailable
- Capture midclt output via $(...) instead of > /dev/null so that
failure detail (which midclt writes to stdout on TrueNAS) is
preserved and shown to the user on error rather than silently
discarded
- Expand update failure hint from a bare query command to an actionable
recovery path: show the midclt output, then print the exact delete
command with the known stale ID so the user can remove it and retry
- Extract hook comment string to _HOOK_COMMENT variable in both
install.sh and uninstall.sh; previously the literal string
'TrueCloud provider patch (S3/B2)' appeared three times across two
files with no shared constant — a silent mismatch on any divergence
would cause hook lookup to return empty with no error output
- Wrap midclt update and create calls with if/else error handlers;
previously a midclt failure under set -euo pipefail silently aborted
the script at "Updating path and enabling ..." with no diagnostic
or recovery guidance
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
substitution into sitecustomize.py; sed's & and | metacharacters
silently corrupt or truncate the output for paths containing those
chars; Python str.replace has no metacharacter issues; also write to
a tmp file and mv atomically so a failed substitution never leaves
an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
$PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
("TrueCloud provider patch (S3/B2)") instead of exact script path;
exact-path match breaks when the repo is moved after install —
uninstall leaves the stale hook registered (fires on every boot),
and reinstall creates a duplicate entry; install.sh now also updates
the script path on re-run so a moved repo self-corrects
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
Users who delete the cloned repo after install had no way to uninstall
without re-cloning. Now install.sh copies uninstall.sh to PATCH_DIR
alongside recover.sh, so the uninstall path is always the stable
/data/truecloud-patch/uninstall.sh. README updated to match.
- uninstall.sh: track _restore_failed separately from RESTORED so
"No backup files found" only prints when find returns nothing (not
when mv fails on existing backups); abort with exit 1 before rm -rf
when any restore fails, leaving PATCH_DIR and recover.sh intact
- install.sh: extend log-scan grep to catch ERROR: lines from
patch_ui.py (backup OSError was silently missed by WARNING:-only grep)
- install.sh: reword restart-failure message — hook IS already
registered and sitecustomize.py IS installed; patch activates on
next boot regardless
- apply.sh: replace `if $_can_install` with `[ "$_can_install" = true ]`
(explicit test, no implicit command lookup); drop 2>/dev/null on
install cp so OS error detail reaches the log
- install.sh: print 'verify' command immediately after successful restart
so users know to confirm the backend patch loaded before creating tasks
- README: correct --insecure description; it controls TLS to the TrueNAS
API (where the API key is transmitted), not the S3 endpoint — previous
wording implied it was safe to use for S3 self-signed certs
All findings from four consecutive full-codebase audit passes plus an
adversarial iXsystems-perspective audit. No new candidates surfaced in
the final clean-pass — branch declared complete.
Fixes:
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp previously fell through and could destroy the vendor file
- apply.sh: correct comment (keeps two prior log generations, not one)
- patch_ui.py: catch OSError on bundle backup with specific diagnostic
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return 'UI already patched' instead of misleading 'pattern not found'
- install.sh: early guard detects pipe-install (bash <(curl ...)) and
exits with a clear error pointing to the git clone workflow
- install.sh: scope WARNING grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: systemctl restart failure now surfaces a recovery hint
- create_task.py: add MITM risk warning to --insecure flag help text
- create_task.py: handle unexpected 2xx response schema in cmd_create
- uninstall.sh: add import-middlewared verification after Python detection
- uninstall.sh: mv failure in JS bundle restore loop no longer aborts
under set -e before cleanup; emits WARNING and continues
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp (disk full, read-only mount) previously fell through and
overwrote the vendor file with no recovery path
- create_task.py: handle unexpected 2xx response schema in cmd_create;
bare KeyError on result['id'] is replaced with a diagnostic print
- uninstall.sh: mv inside while loop had no error handling; under
set -euo pipefail a failed mv aborted the script before rm -rf PATCH_DIR,
leaving the system in partial-uninstall limbo
- create_task.py: add MITM risk warning to --insecure flag help text;
common home-user pattern (self-signed cert) exposes API key in transit
- install.sh: replace bare systemctl restart with explicit failure check
that prints a recovery hint when middlewared fails to start post-install
- patch_ui.py: wrap shutil.copy2 backup in try/except OSError so a
permission or read-only filesystem error prints a specific diagnostic
instead of crashing the script with a generic 'exited non-zero' message
- install.sh: add early guard that detects pipe-install (bash <(curl ...))
and exits with a clear error pointing to the git clone workflow
- apply.sh: correct comment from 'one prior generation' to 'two prior
generations (.1 and .2)' — rotation has always kept three log files
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
in patch/patch_ui.py, preventing silent drift if a new path is added
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return early and print 'UI already patched' instead of the misleading
'filterByProviders pattern not found' warning
- install.sh: scope warning grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: fix misleading 'before continuing' wording on warning banner
- install.sh: fix grep anchor (^WARNING: missed [truecloud-patch] WARNING: lines)
- uninstall.sh: add import-middlewared verification after Python detection,
matching apply.sh fallback logic
sitecustomize.py: when find_spec resolves real_spec as None (module absent
after a TrueNAS update), record a FAIL status and mark the module done so
hook_status.json is still written and cmd_verify shows a diagnostic FAIL
instead of the ambiguous "no status file found".
sitecustomize.py: the AttributeError fallback in the URL-fix wrapper now
writes a WARNING to stderr before returning the unmodified result, making
the unexpected ResticConfig type visible in journalctl.
apply.sh: after falling back to bare python3, verify that python3 can also
import middlewared; if not, emit a second warning so the operator knows the
backend patch may be installed in the wrong site-packages directory.
patch_ui.py: abort (return without writing) when FIND.subn produces a count
other than 1, instead of committing a doubly-patched bundle and having
subsequent runs silently accept it via the MARKER check.
uninstall.sh: when a vendor sitecustomize.py backup exists, use mv to
atomically overwrite our file rather than rm-then-mv; eliminates the window
where a read-only /usr causes rm to fail under set -e, aborting before the
backup is restored.
sitecustomize.py: exec_module now records a FAIL status when the underlying
module load raises, instead of leaving hook_status.json unwritten. cmd_verify
will now show a diagnostic FAIL rather than the misleading "no status file".
sitecustomize.py: URL-fix loop covers --repository and --repository= in
addition to --repo/--repo=/-r; these are documented restic synonyms.
sitecustomize.py: _replace fallback now catches AttributeError in addition
to TypeError so an unrecognised ResticConfig return type silently falls back
to returning the unmodified result rather than crashing the backup job.
patch_ui.py: warns when FIND.subn produces a count other than 1, making
unexpected multi-replacement visible in the apply log.
uninstall.sh: find for JS bundle restore now includes /usr/share/truenas-ui,
matching all three entries in patch_ui.py's WEBUI_CANDIDATES.
apply.sh: preserve two log generations (.1 and .2) on rotation so the
last two boots are always available for diagnosis.
patch_ui.py: find_bundle returns on the first matching JS file instead of
collecting all matches. The multi-match warning was dead weight — the Angular
Ivy compiler produces exactly one bundle and the WARNING path was unreachable
in practice.
uninstall.sh: restore an orphaned sitecustomize.py.pre-truecloud-patch when
sitecustomize.py itself has already been removed (e.g. manual deletion while
the backup survived). Prevents leaving ghost vendor files in site-packages.
sitecustomize.py — _install():
- Scope hook to middlewared service process only via sys.argv[0] check;
previously any tool in the same venv (midclt, debug scripts) would also
get its imports of the two target modules intercepted.
- Exec-chain a displaced sitecustomize.py: if apply.sh backed up a
pre-existing sitecustomize.py to .pre-truecloud-patch, run it in a
sandboxed namespace before installing our hook so any startup code
(path additions, codec registrations) still takes effect.
sitecustomize.py — _b2_restic_config():
- Validate expected credential fields ("account", "key") before accessing
them; raise a named KeyError listing what is missing and what is present
so a schema change produces an attributable error at backup time rather
than a bare KeyError with no indication this patch is involved.
sitecustomize.py — get_restic_config wrapper:
- Extend URL fix to cover all three flag forms restic accepts:
-r <url> (existing)
--repo <url> (long two-element form, now checked)
--repo=<url> (long single-element form, now handled)
Without this, a restic CLI change from -r to --repo would silently make
the fix a no-op while verify still reported the patch as OK.
patch_ui.py — find_bundle():
- Remove UnicodeDecodeError from except clause; errors="replace" in the
open() call means the exception can never be raised, and its presence
suggested the error parameter was not understood.
patch_ui.py:
- Write Angular bundle atomically via tmp + os.replace, matching the pattern
already used by _record_status. Prevents a corrupt bundle if the write is
interrupted mid-boot.
sitecustomize.py:
- Tighten _record_status count barrier comment to name _Finder._targets as
the canonical count, making the coupling visible to future editors.
uninstall.sh:
- Verify middlewared restarted cleanly after uninstall, with journalctl
guidance on failure — matching recover.sh's existing pattern.
apply.sh:
- Change second line of site-packages error block from WARNING: prefix
(misleading for an instructional message) to a plain Run: hint.
create_task.py:
- Guard __doc__ against None in epilog extraction so -OO does not crash.
README.md:
- Split Emergency recovery into three named subsections: middlewared won't
start, web UI is blank or broken (corrupt bundle recovery), and backend
verify shows FAIL. Each gives direct commands and escalation steps.
- Add Restoring from a TrueCloud Backup section: finding the restic binary,
gathering credentials, provider-specific env var setup for B2 and S3,
listing and restoring snapshots, and operational notes on restore hygiene.
- Clarify that hook_status.json is written once both target modules have
loaded (not necessarily at the instant middlewared starts).
sitecustomize.py:
- Restore _record_status count barrier: write the status file only after all
patches have reported. middlewared.plugins.cloud_backup.restic is imported
lazily (only when a backup task runs), so without this barrier verify would
declare "all patches active" based solely on the B2 patch that fires at
startup. Barrier now gates on _Finder._targets rather than the removed
_PATCHES dict.
- Add comment in exec_module noting the if/elif must stay in sync with
_Finder._targets, making the coupling visible.
patch_ui.py:
- Merge find_webui into find_bundle: previously find_bundle(None) would crash
with os.walk(None) if main()'s guard were removed. Merged function returns
a 3-tuple (webui_dir, path, content); webui_dir=None means no candidate
directory found, path=None means directory found but pattern absent.
main() still produces distinct messages for each failure mode.
create_task.py:
- Split triple-chained .get() in cmd_list_tasks into two lines; the or {}
handling for None credentials was buried inside a one-liner.
uninstall.sh:
- Fix find loop: replace "for x in $(find ...)" with "while IFS= read -r"
to handle paths containing spaces or newlines.
- Add #!/usr/bin/env shebang form to Python detection, matching apply.sh.
Without this, uninstall on a system where middlewared uses the env form
would silently leave sitecustomize.py in the wrong site-packages.
sitecustomize.py:
- Replace _PATCHES dispatch dict with if/elif in exec_module; removes
coupling between dispatch and _record_status's count barrier
- Drop _record_status count barrier entirely; both patches fire within
milliseconds during the same import sequence, write-on-every-call is safe
- Replace _broken_url regex with str.partition + startswith checks; same
semantics, no regex knowledge required to read
- Replace @staticmethod decorator inside plain function with explicit
staticmethod() assignment; decorator form creates a descriptor object,
not a callable, which confuses readers expecting class-body usage
apply.sh:
- Inline warn/ok helpers; each was one echo with a prefix, the indirection
cost more than the abstraction saved
- Collapse patch_ui.py if/else (whose if branch was a no-op comment) to
a single || fallback line
create_task.py:
- Remove vestigial (_client, _args) params from cmd_verify; it was pulled
out of dispatch, the params were never used
- Replace 3-entry dispatch dict with if/elif; dict implied a uniform calling
convention that verify already broke
create_task.py:
- Remove dead make_client() call before the verify branch; it was called
unconditionally with host=None/key=None, creating a broken client that was
immediately discarded or overwritten.
- Remove "verify" from the dispatch dict; it was never reached through dispatch
(the if/cmd==verify branch above it handled it). Dispatch now only contains
commands that actually use a client.
- Wrap json.load() in try/except (OSError, JSONDecodeError) so a corrupt or
partially-written status file produces a useful message instead of a traceback.
sitecustomize.py:
- Call _record_status() on the early-return paths in both _patch_b2 and
_patch_restic. Without this, if TrueNAS natively supports B2 or the patch
is already applied, the status file was never written and `verify` always
reported failure even when everything was fine.
- Add idempotency guard to _record_status(): first call wins; duplicate calls
for the same module are ignored so the entry count stays accurate.
- Make B2 get_restic_config a @staticmethod. The method never used self; the
noqa comment was suppressing the evidence of a design mismatch. Removing the
unused parameter makes the intent explicit.
- Add NamedTuple._replace() fallback after dataclasses.replace() in the restic
wrapper. If ResticConfig is ever refactored to a NamedTuple, the TypeError
from dataclasses.replace() would have surfaced as a backup job failure rather
than a graceful recovery.
sitecustomize.py: _patch_restic no longer reimplements get_restic_config.
It now wraps the original: calls _orig(cloud_backup) to get a ResticConfig,
then post-processes only the -r argument to fix "b2:/bucket" → "b2:bucket"
when the URL contains a stray leading slash (the stock bug for empty-hostname
providers). Uses dataclasses.replace() to build the corrected result so new
ResticConfig fields added in future TrueNAS versions pass through unchanged.
This eliminates the transfer_setting gap, env dict mutation, and frozen-copy
drift that would occur over time.
Also adds a status file mechanism: sitecustomize.py writes
/data/truecloud-patch/hook_status.json atomically after both patches have
reported success or failure. This gives a machine-readable signal that the
hook fired correctly — without requiring log scraping.
create_task.py: new "verify" subcommand reads the status file and prints a
human-readable summary. Does not require --host or --api-key. --host and
--api-key are now optional at the parser level and validated only for
subcommands that actually need an API connection.
README: update troubleshooting to use "create_task.py verify" instead of
the manual Python introspection one-liner.
install.sh: running install.sh after a recover.sh left /data/truecloud-patch/disabled
in place, so apply.sh silently skipped all patching and middlewared restarted
without the patch. Clear the kill switch file before running apply.sh.
uninstall.sh: midclt initshutdownscript.delete was unguarded under set -euo pipefail,
so a delete failure (already-removed entry, transient API error) aborted the script
before sitecustomize.py was cleaned up or /data/truecloud-patch/ was removed. Now
guarded with an if/else that warns and continues.
sitecustomize.py: move importlib.machinery/.util imports inside the if block in
find_spec so they only execute when intercepting our two target modules, not on
every module import across the whole process. In _install(), import os before
importlib.util so the kill switch check (cheap) runs before the importlib import
(slightly heavier on first use). Remove the unused orig variable in _patch_restic.
patch_ui.py: add explicit encoding="utf-8" to both open() calls. errors="replace"
on read so malformed bytes in a bundle don't silently skip a candidate file.
flan