restic 0.16.x changed the B2 URL format to use a colon between bucket and
path (b2:bucket:prefix) instead of a slash. The middlewared URL builder
produces b2:/bucket/path; restic then validates the full string after 'b2:'
as a bucket name, which fails because the slash is not in [a-z0-9-].
Fix the restic.py wrapper to strip the leading slash and replace the first
slash with a colon: b2:/bucket/path -> b2:bucket:path.
Also fix the hasattr bug in sitecustomize.py _patch_b2: hasattr() returns
True for methods inherited from the base class (which raises
NotImplementedError), causing the patch to be silently skipped. Use
'get_restic_config' not in cls.__dict__ instead.
Two bugs:
1. hasattr(B2RcloneRemote, "get_restic_config") returned True because the base
class defines the method (it just raises NotImplementedError). The method was
never added to B2RcloneRemote. Fixed: use __dict__ check instead.
2. "skip if TRUECLOUD_PATCH marker present" prevented a corrected patch block
from replacing a previously-written buggy one without clearing the overlay.
Fixed: always strip any existing TRUECLOUD_PATCH block and rewrite it fresh
using a Python heredoc. Each apply.sh run now self-corrects to the latest
version of the patch.
site.getsitepackages()[0] returns /usr/local/lib/python3.11/dist-packages/ on
TrueNAS 25.x but middlewared lives in /usr/lib/python3/dist-packages/.
sitecustomize.py was installed to the wrong directory and Python never loaded it.
Fix: derive SITE_PKG from middlewared.__file__ so the overlay and sitecustomize.py
land in the correct directory.
Also add direct patching of b2.py and restic.py in the overlay as the primary
backend approach — more reliable than an import hook since it works regardless
of Python's site initialisation configuration. apply.sh now also writes
hook_status.json at boot time so 'verify' shows OK without requiring a backup run.
Also fixes incorrect middlewared log path in README (/var/log/middlewared/middlewared.log
→ /var/log/middlewared.log) and simplifies the verify troubleshooting note.
TrueNAS 25.x mounts /usr as a read-only filesystem. Writing
sitecustomize.py to site-packages and patching the Angular bundle
both fail with EROFS.
Fix: mount a writable overlayfs on each target directory before
writing to it. Upper/work dirs live in /run (tmpfs), so overlays are
volatile per boot and are recreated by apply.sh on every PREINIT run
before middlewared starts.
apply.sh:
- Add _ensure_writable(dir, tag): probes writability; mounts overlay
in /run/truecloud-{tag}-{upper,work} if the directory is read-only;
detects if the overlay is already mounted (idempotent)
- Call _ensure_writable before site-packages writes (tag "sc")
- Detect webui dir with bash loop; call _ensure_writable before
patch_ui.py (tag "ui") — non-fatal if mount fails
uninstall.sh:
- Add overlay unmounting section after file restoration and before
rm -rf, so the lower layer's originals are exposed immediately
- Move _restore_failed exit 1 to after unmount so overlays are
cleaned up even on partial failure
- Update "no backup files" message for immutable OS context
TrueNAS 25.x changed how Angular emits the filterByProviders binding.
Previously a static inline array ("filterByProviders",["STORJ_IX"]),
it is now a pureFunction call:
pe(slot, factory, component.CloudSyncProviderName.Storj)
Add a _PATTERNS list tried in order, with a _match_pattern() helper.
The 25.x pureFunction pattern matches on the stable TypeScript enum
name (CloudSyncProviderName.Storj); the minified variable names and
slot index are matched with \w+ / \d+. Both patterns replace the
binding with ["STORJ_IX","S3","B2"]. MARKER and backup/restore logic
are unchanged.
- create_task.py list-tasks: crash on null credentials.provider
(`creds.get("provider", {})` returns None when key exists but is null;
switch to `(creds.get("provider") or {})`)
- sitecustomize.py: write hook_status.json after each module, not only
when both have loaded; S3-only users (B2 module never imported) now
get a status file from verify instead of "No status file found"
- README: add filesystem find + sqlite3 DB query to the emergency
recovery section so users can locate their clone path when middlewared
is down and midclt is unavailable
- Capture midclt output via $(...) instead of > /dev/null so that
failure detail (which midclt writes to stdout on TrueNAS) is
preserved and shown to the user on error rather than silently
discarded
- Expand update failure hint from a bare query command to an actionable
recovery path: show the midclt output, then print the exact delete
command with the known stale ID so the user can remove it and retry
- Extract hook comment string to _HOOK_COMMENT variable in both
install.sh and uninstall.sh; previously the literal string
'TrueCloud provider patch (S3/B2)' appeared three times across two
files with no shared constant — a silent mismatch on any divergence
would cause hook lookup to return empty with no error output
- Wrap midclt update and create calls with if/else error handlers;
previously a midclt failure under set -euo pipefail silently aborted
the script at "Updating path and enabling ..." with no diagnostic
or recovery guidance
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
substitution into sitecustomize.py; sed's & and | metacharacters
silently corrupt or truncate the output for paths containing those
chars; Python str.replace has no metacharacter issues; also write to
a tmp file and mv atomically so a failed substitution never leaves
an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
$PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
("TrueCloud provider patch (S3/B2)") instead of exact script path;
exact-path match breaks when the repo is moved after install —
uninstall leaves the stale hook registered (fires on every boot),
and reinstall creates a duplicate entry; install.sh now also updates
the script path on re-run so a moved repo self-corrects
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
Users who delete the cloned repo after install had no way to uninstall
without re-cloning. Now install.sh copies uninstall.sh to PATCH_DIR
alongside recover.sh, so the uninstall path is always the stable
/data/truecloud-patch/uninstall.sh. README updated to match.
- uninstall.sh: track _restore_failed separately from RESTORED so
"No backup files found" only prints when find returns nothing (not
when mv fails on existing backups); abort with exit 1 before rm -rf
when any restore fails, leaving PATCH_DIR and recover.sh intact
- install.sh: extend log-scan grep to catch ERROR: lines from
patch_ui.py (backup OSError was silently missed by WARNING:-only grep)
- install.sh: reword restart-failure message — hook IS already
registered and sitecustomize.py IS installed; patch activates on
next boot regardless
- apply.sh: replace `if $_can_install` with `[ "$_can_install" = true ]`
(explicit test, no implicit command lookup); drop 2>/dev/null on
install cp so OS error detail reaches the log
- install.sh: print 'verify' command immediately after successful restart
so users know to confirm the backend patch loaded before creating tasks
- README: correct --insecure description; it controls TLS to the TrueNAS
API (where the API key is transmitted), not the S3 endpoint — previous
wording implied it was safe to use for S3 self-signed certs
All findings from four consecutive full-codebase audit passes plus an
adversarial iXsystems-perspective audit. No new candidates surfaced in
the final clean-pass — branch declared complete.
Fixes:
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp previously fell through and could destroy the vendor file
- apply.sh: correct comment (keeps two prior log generations, not one)
- patch_ui.py: catch OSError on bundle backup with specific diagnostic
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return 'UI already patched' instead of misleading 'pattern not found'
- install.sh: early guard detects pipe-install (bash <(curl ...)) and
exits with a clear error pointing to the git clone workflow
- install.sh: scope WARNING grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: systemctl restart failure now surfaces a recovery hint
- create_task.py: add MITM risk warning to --insecure flag help text
- create_task.py: handle unexpected 2xx response schema in cmd_create
- uninstall.sh: add import-middlewared verification after Python detection
- uninstall.sh: mv failure in JS bundle restore loop no longer aborts
under set -e before cleanup; emits WARNING and continues
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp (disk full, read-only mount) previously fell through and
overwrote the vendor file with no recovery path
- create_task.py: handle unexpected 2xx response schema in cmd_create;
bare KeyError on result['id'] is replaced with a diagnostic print
- uninstall.sh: mv inside while loop had no error handling; under
set -euo pipefail a failed mv aborted the script before rm -rf PATCH_DIR,
leaving the system in partial-uninstall limbo
- create_task.py: add MITM risk warning to --insecure flag help text;
common home-user pattern (self-signed cert) exposes API key in transit
- install.sh: replace bare systemctl restart with explicit failure check
that prints a recovery hint when middlewared fails to start post-install
- patch_ui.py: wrap shutil.copy2 backup in try/except OSError so a
permission or read-only filesystem error prints a specific diagnostic
instead of crashing the script with a generic 'exited non-zero' message
- install.sh: add early guard that detects pipe-install (bash <(curl ...))
and exits with a clear error pointing to the git clone workflow
- apply.sh: correct comment from 'one prior generation' to 'two prior
generations (.1 and .2)' — rotation has always kept three log files
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
in patch/patch_ui.py, preventing silent drift if a new path is added
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return early and print 'UI already patched' instead of the misleading
'filterByProviders pattern not found' warning
- install.sh: scope warning grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: fix misleading 'before continuing' wording on warning banner
- install.sh: fix grep anchor (^WARNING: missed [truecloud-patch] WARNING: lines)
- uninstall.sh: add import-middlewared verification after Python detection,
matching apply.sh fallback logic
sitecustomize.py: when find_spec resolves real_spec as None (module absent
after a TrueNAS update), record a FAIL status and mark the module done so
hook_status.json is still written and cmd_verify shows a diagnostic FAIL
instead of the ambiguous "no status file found".
sitecustomize.py: the AttributeError fallback in the URL-fix wrapper now
writes a WARNING to stderr before returning the unmodified result, making
the unexpected ResticConfig type visible in journalctl.
apply.sh: after falling back to bare python3, verify that python3 can also
import middlewared; if not, emit a second warning so the operator knows the
backend patch may be installed in the wrong site-packages directory.
patch_ui.py: abort (return without writing) when FIND.subn produces a count
other than 1, instead of committing a doubly-patched bundle and having
subsequent runs silently accept it via the MARKER check.
uninstall.sh: when a vendor sitecustomize.py backup exists, use mv to
atomically overwrite our file rather than rm-then-mv; eliminates the window
where a read-only /usr causes rm to fail under set -e, aborting before the
backup is restored.
sitecustomize.py: exec_module now records a FAIL status when the underlying
module load raises, instead of leaving hook_status.json unwritten. cmd_verify
will now show a diagnostic FAIL rather than the misleading "no status file".
sitecustomize.py: URL-fix loop covers --repository and --repository= in
addition to --repo/--repo=/-r; these are documented restic synonyms.
sitecustomize.py: _replace fallback now catches AttributeError in addition
to TypeError so an unrecognised ResticConfig return type silently falls back
to returning the unmodified result rather than crashing the backup job.
patch_ui.py: warns when FIND.subn produces a count other than 1, making
unexpected multi-replacement visible in the apply log.
uninstall.sh: find for JS bundle restore now includes /usr/share/truenas-ui,
matching all three entries in patch_ui.py's WEBUI_CANDIDATES.
apply.sh: preserve two log generations (.1 and .2) on rotation so the
last two boots are always available for diagnosis.
patch_ui.py: find_bundle returns on the first matching JS file instead of
collecting all matches. The multi-match warning was dead weight — the Angular
Ivy compiler produces exactly one bundle and the WARNING path was unreachable
in practice.
uninstall.sh: restore an orphaned sitecustomize.py.pre-truecloud-patch when
sitecustomize.py itself has already been removed (e.g. manual deletion while
the backup survived). Prevents leaving ghost vendor files in site-packages.
sitecustomize.py — _install():
- Scope hook to middlewared service process only via sys.argv[0] check;
previously any tool in the same venv (midclt, debug scripts) would also
get its imports of the two target modules intercepted.
- Exec-chain a displaced sitecustomize.py: if apply.sh backed up a
pre-existing sitecustomize.py to .pre-truecloud-patch, run it in a
sandboxed namespace before installing our hook so any startup code
(path additions, codec registrations) still takes effect.
sitecustomize.py — _b2_restic_config():
- Validate expected credential fields ("account", "key") before accessing
them; raise a named KeyError listing what is missing and what is present
so a schema change produces an attributable error at backup time rather
than a bare KeyError with no indication this patch is involved.
sitecustomize.py — get_restic_config wrapper:
- Extend URL fix to cover all three flag forms restic accepts:
-r <url> (existing)
--repo <url> (long two-element form, now checked)
--repo=<url> (long single-element form, now handled)
Without this, a restic CLI change from -r to --repo would silently make
the fix a no-op while verify still reported the patch as OK.
patch_ui.py — find_bundle():
- Remove UnicodeDecodeError from except clause; errors="replace" in the
open() call means the exception can never be raised, and its presence
suggested the error parameter was not understood.
patch_ui.py:
- Write Angular bundle atomically via tmp + os.replace, matching the pattern
already used by _record_status. Prevents a corrupt bundle if the write is
interrupted mid-boot.
sitecustomize.py:
- Tighten _record_status count barrier comment to name _Finder._targets as
the canonical count, making the coupling visible to future editors.
uninstall.sh:
- Verify middlewared restarted cleanly after uninstall, with journalctl
guidance on failure — matching recover.sh's existing pattern.
apply.sh:
- Change second line of site-packages error block from WARNING: prefix
(misleading for an instructional message) to a plain Run: hint.
create_task.py:
- Guard __doc__ against None in epilog extraction so -OO does not crash.
README.md:
- Split Emergency recovery into three named subsections: middlewared won't
start, web UI is blank or broken (corrupt bundle recovery), and backend
verify shows FAIL. Each gives direct commands and escalation steps.
- Add Restoring from a TrueCloud Backup section: finding the restic binary,
gathering credentials, provider-specific env var setup for B2 and S3,
listing and restoring snapshots, and operational notes on restore hygiene.
- Clarify that hook_status.json is written once both target modules have
loaded (not necessarily at the instant middlewared starts).
sitecustomize.py:
- Restore _record_status count barrier: write the status file only after all
patches have reported. middlewared.plugins.cloud_backup.restic is imported
lazily (only when a backup task runs), so without this barrier verify would
declare "all patches active" based solely on the B2 patch that fires at
startup. Barrier now gates on _Finder._targets rather than the removed
_PATCHES dict.
- Add comment in exec_module noting the if/elif must stay in sync with
_Finder._targets, making the coupling visible.
patch_ui.py:
- Merge find_webui into find_bundle: previously find_bundle(None) would crash
with os.walk(None) if main()'s guard were removed. Merged function returns
a 3-tuple (webui_dir, path, content); webui_dir=None means no candidate
directory found, path=None means directory found but pattern absent.
main() still produces distinct messages for each failure mode.
create_task.py:
- Split triple-chained .get() in cmd_list_tasks into two lines; the or {}
handling for None credentials was buried inside a one-liner.
uninstall.sh:
- Fix find loop: replace "for x in $(find ...)" with "while IFS= read -r"
to handle paths containing spaces or newlines.
- Add #!/usr/bin/env shebang form to Python detection, matching apply.sh.
Without this, uninstall on a system where middlewared uses the env form
would silently leave sitecustomize.py in the wrong site-packages.
sitecustomize.py:
- Replace _PATCHES dispatch dict with if/elif in exec_module; removes
coupling between dispatch and _record_status's count barrier
- Drop _record_status count barrier entirely; both patches fire within
milliseconds during the same import sequence, write-on-every-call is safe
- Replace _broken_url regex with str.partition + startswith checks; same
semantics, no regex knowledge required to read
- Replace @staticmethod decorator inside plain function with explicit
staticmethod() assignment; decorator form creates a descriptor object,
not a callable, which confuses readers expecting class-body usage
apply.sh:
- Inline warn/ok helpers; each was one echo with a prefix, the indirection
cost more than the abstraction saved
- Collapse patch_ui.py if/else (whose if branch was a no-op comment) to
a single || fallback line
create_task.py:
- Remove vestigial (_client, _args) params from cmd_verify; it was pulled
out of dispatch, the params were never used
- Replace 3-entry dispatch dict with if/elif; dict implied a uniform calling
convention that verify already broke
create_task.py:
- Remove dead make_client() call before the verify branch; it was called
unconditionally with host=None/key=None, creating a broken client that was
immediately discarded or overwritten.
- Remove "verify" from the dispatch dict; it was never reached through dispatch
(the if/cmd==verify branch above it handled it). Dispatch now only contains
commands that actually use a client.
- Wrap json.load() in try/except (OSError, JSONDecodeError) so a corrupt or
partially-written status file produces a useful message instead of a traceback.
sitecustomize.py:
- Call _record_status() on the early-return paths in both _patch_b2 and
_patch_restic. Without this, if TrueNAS natively supports B2 or the patch
is already applied, the status file was never written and `verify` always
reported failure even when everything was fine.
- Add idempotency guard to _record_status(): first call wins; duplicate calls
for the same module are ignored so the entry count stays accurate.
- Make B2 get_restic_config a @staticmethod. The method never used self; the
noqa comment was suppressing the evidence of a design mismatch. Removing the
unused parameter makes the intent explicit.
- Add NamedTuple._replace() fallback after dataclasses.replace() in the restic
wrapper. If ResticConfig is ever refactored to a NamedTuple, the TypeError
from dataclasses.replace() would have surfaced as a backup job failure rather
than a graceful recovery.
sitecustomize.py: _patch_restic no longer reimplements get_restic_config.
It now wraps the original: calls _orig(cloud_backup) to get a ResticConfig,
then post-processes only the -r argument to fix "b2:/bucket" → "b2:bucket"
when the URL contains a stray leading slash (the stock bug for empty-hostname
providers). Uses dataclasses.replace() to build the corrected result so new
ResticConfig fields added in future TrueNAS versions pass through unchanged.
This eliminates the transfer_setting gap, env dict mutation, and frozen-copy
drift that would occur over time.
Also adds a status file mechanism: sitecustomize.py writes
/data/truecloud-patch/hook_status.json atomically after both patches have
reported success or failure. This gives a machine-readable signal that the
hook fired correctly — without requiring log scraping.
create_task.py: new "verify" subcommand reads the status file and prints a
human-readable summary. Does not require --host or --api-key. --host and
--api-key are now optional at the parser level and validated only for
subcommands that actually need an API connection.
README: update troubleshooting to use "create_task.py verify" instead of
the manual Python introspection one-liner.
install.sh: running install.sh after a recover.sh left /data/truecloud-patch/disabled
in place, so apply.sh silently skipped all patching and middlewared restarted
without the patch. Clear the kill switch file before running apply.sh.
uninstall.sh: midclt initshutdownscript.delete was unguarded under set -euo pipefail,
so a delete failure (already-removed entry, transient API error) aborted the script
before sitecustomize.py was cleaned up or /data/truecloud-patch/ was removed. Now
guarded with an if/else that warns and continues.
sitecustomize.py: move importlib.machinery/.util imports inside the if block in
find_spec so they only execute when intercepting our two target modules, not on
every module import across the whole process. In _install(), import os before
importlib.util so the kill switch check (cheap) runs before the importlib import
(slightly heavier on first use). Remove the unused orig variable in _patch_restic.
patch_ui.py: add explicit encoding="utf-8" to both open() calls. errors="replace"
on read so malformed bytes in a bundle don't silently skip a candidate file.
If the patch ever prevents middlewared from starting, users now have a
clear escape hatch that requires no knowledge of Python internals:
bash /data/truecloud-patch/recover.sh
Or at a bare shell prompt:
touch /data/truecloud-patch/disabled
systemctl restart middlewared
sitecustomize.py checks for /data/truecloud-patch/disabled at Python
startup and skips the import hook entirely when the file exists.
apply.sh does the same so the PREINIT script also does nothing on reboot.
recover.sh is copied to /data/truecloud-patch/ by install.sh so it is
available even without the original repo directory.
README gains an "Emergency recovery" section above Troubleshooting.
sitecustomize.py: exec_module was calling importlib.util.find_spec(fullname)
after Python had already added the module to sys.modules. find_spec
short-circuits to sys.modules[name].__spec__, which is our own wrapper spec,
so real_spec.loader.exec_module(module) immediately called exec_module again
— infinite recursion. Fix: resolve the real file spec in find_spec (before
sys.modules is populated) and store it on the Loader. exec_module now uses
the stored spec directly with no find_spec call.
Also move _mark_done into the finally block so a module that fails to load
doesn't trigger infinite retry loops on subsequent import attempts.
apply.sh: warn() writes to stdout. Inside $(find_mw_python), stdout is
captured by the command substitution, so any warn() call folded its text
into $PYTHON — causing every subsequent "$PYTHON" invocation to fail. Fix:
emit the fallback warning to stderr (>&2) so it goes to the log without
being captured.
Patches middlewared at runtime via sitecustomize.py (no file edits to /usr/)
and widens the UI credential dropdown from Storj-only to S3+B2+Storj.
Persists across TrueNAS updates via PREINIT initshutdownscript stored in DB.
flan