Security
--------
create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that
JSON carries the restic repository password -- so it sat in the subprocess's argv,
which is world-readable via ps, for the duration of the call. That password is the
encryption key for the entire cloud backup repository.
It now talks to the middleware through truenas_api_client, the library that backs
midclt itself, so the password never leaves this process's memory. Verified on a
live box: list-tasks and list-credentials work through the new transport.
--password is also no longer required, because passing a secret as a CLI argument
writes it to shell history permanently. --password-stdin reads it from stdin, and
with neither flag the tool prompts via getpass. --password still works but warns.
Fixed
-----
uninstall.sh could leave every patch installed. It reverted by unmounting the
overlay -- but apply.sh only mounts one when the target directory is read-only. On
a writable /usr it patches the real files in place, and uninstall would remove the
boot hook, report success, and leave the patch applied. It now strips the appended
blocks from the middleware files explicitly. This also covers the case where the
overlay unmount fails.
create_task.py's __version__ had been stuck at 0.2.0 through three releases. The
version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it
missed the one file that actually shows a version to users (--version). The check
now covers __version__ too -- and caught this immediately.
118 tests, ruff and shellcheck clean.
Audit
-----
- create_task.py verify failed on a DEFAULT install. hook_status.json emitted a
per-file entry for the nested module with ok:false whenever the feature was
switched off -- the default -- so verify printed [FAIL] and exited 1, right
after the README tells users to run it. Status is now per MODULE with an
`active` flag, and verify renders an inactive module as [SKIP].
- A partial apply suppressed the middlewared restart. The exit code conflated
"nothing applied" with "one module applied, one failed", so a failing providers
patch would prevent the restart that a freshly-applied nested patch needs,
leaving it on disk and never loaded. Exit 2 now means partial and the restart
still fires.
- The native-nested probe could never fire. It scanned crud.py for the guard
message, but our own injected block quotes that message, so once applied the
probe would always conclude the guard was still present. It now reads only the
stock portion of the file.
- recover.sh did not unmount staging trees, so an emergency recovery left bind
mounts pinning ZFS snapshots that could then never be destroyed.
- uninstall.sh deleted sidecar files without reading them. A sidecar is the only
record that an interrupted run's snapshot tree is still on disk; both scripts
now name the snapshot before clearing it.
- Removed a dead branch in the restart gate (unreachable: the kill switch exits).
Refactor
--------
- Staging teardown had been copy-pasted into uninstall.sh and recover.sh -- two
untested shell copies of the fiddly depth-ordering and lazy-umount logic. Both
now call `python3 patch/truecloud_nested.py cleanup`, so there is exactly one
implementation and it is the one under test.
- Dropped the in-memory ACTIVE dict. The sidecar file was already the source of
truth; a second in-process record could only desync -- and it is precisely the
middlewared-restart case (which empties it) that must not orphan a snapshot
tree. One record, on disk, or none.
Not done: the overlay-unmount loop is duplicated across apply.sh/uninstall.sh/
recover.sh. It is pre-existing, and apply.sh runs at PREINIT under a tight
timeout -- giving it a source dependency would trade 10 lines of duplication for
a boot-time failure mode.
74 tests, ruff and shellcheck clean.
The tool created cloud_backup tasks via POST /api/v2.0/cloud_backup, which is deprecated
and removed in TrueNAS 26.04. It now calls the middleware directly with midclt
(cloudsync.credentials.query / cloud_backup.query / cloud_backup.create), so it runs on
the TrueNAS host with no host address or API key. --host/--api-key/--insecure are kept
accepted-but-ignored for compatibility. Dropped the ssl/urllib HTTP client. v0.2.0.
On 24.10 (Electric Eel) credentials["provider"] is the type string with
account/key in credentials["attributes"]; 25.04+ moved them into a
provider dict. The injected get_restic_config only handled the newer
shape and raised TypeError on 24.10 at task creation (#1).
The method now detects the schema and reads credentials from the right
place on both. create_task.py list-credentials and list-tasks use the
same schema-agnostic lookup.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds. Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.
Bumps all scripts to v0.0.3.
- create_task.py list-tasks: crash on null credentials.provider
(`creds.get("provider", {})` returns None when key exists but is null;
switch to `(creds.get("provider") or {})`)
- sitecustomize.py: write hook_status.json after each module, not only
when both have loaded; S3-only users (B2 module never imported) now
get a status file from verify instead of "No status file found"
- README: add filesystem find + sqlite3 DB query to the emergency
recovery section so users can locate their clone path when middlewared
is down and midclt is unavailable
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp (disk full, read-only mount) previously fell through and
overwrote the vendor file with no recovery path
- create_task.py: handle unexpected 2xx response schema in cmd_create;
bare KeyError on result['id'] is replaced with a diagnostic print
- uninstall.sh: mv inside while loop had no error handling; under
set -euo pipefail a failed mv aborted the script before rm -rf PATCH_DIR,
leaving the system in partial-uninstall limbo
- create_task.py: add MITM risk warning to --insecure flag help text;
common home-user pattern (self-signed cert) exposes API key in transit
- install.sh: replace bare systemctl restart with explicit failure check
that prints a recovery hint when middlewared fails to start post-install
sitecustomize.py: when find_spec resolves real_spec as None (module absent
after a TrueNAS update), record a FAIL status and mark the module done so
hook_status.json is still written and cmd_verify shows a diagnostic FAIL
instead of the ambiguous "no status file found".
sitecustomize.py: the AttributeError fallback in the URL-fix wrapper now
writes a WARNING to stderr before returning the unmodified result, making
the unexpected ResticConfig type visible in journalctl.
apply.sh: after falling back to bare python3, verify that python3 can also
import middlewared; if not, emit a second warning so the operator knows the
backend patch may be installed in the wrong site-packages directory.
patch_ui.py: abort (return without writing) when FIND.subn produces a count
other than 1, instead of committing a doubly-patched bundle and having
subsequent runs silently accept it via the MARKER check.
uninstall.sh: when a vendor sitecustomize.py backup exists, use mv to
atomically overwrite our file rather than rm-then-mv; eliminates the window
where a read-only /usr causes rm to fail under set -e, aborting before the
backup is restored.
patch_ui.py:
- Write Angular bundle atomically via tmp + os.replace, matching the pattern
already used by _record_status. Prevents a corrupt bundle if the write is
interrupted mid-boot.
sitecustomize.py:
- Tighten _record_status count barrier comment to name _Finder._targets as
the canonical count, making the coupling visible to future editors.
uninstall.sh:
- Verify middlewared restarted cleanly after uninstall, with journalctl
guidance on failure — matching recover.sh's existing pattern.
apply.sh:
- Change second line of site-packages error block from WARNING: prefix
(misleading for an instructional message) to a plain Run: hint.
create_task.py:
- Guard __doc__ against None in epilog extraction so -OO does not crash.
README.md:
- Split Emergency recovery into three named subsections: middlewared won't
start, web UI is blank or broken (corrupt bundle recovery), and backend
verify shows FAIL. Each gives direct commands and escalation steps.
- Add Restoring from a TrueCloud Backup section: finding the restic binary,
gathering credentials, provider-specific env var setup for B2 and S3,
listing and restoring snapshots, and operational notes on restore hygiene.
- Clarify that hook_status.json is written once both target modules have
loaded (not necessarily at the instant middlewared starts).
sitecustomize.py:
- Restore _record_status count barrier: write the status file only after all
patches have reported. middlewared.plugins.cloud_backup.restic is imported
lazily (only when a backup task runs), so without this barrier verify would
declare "all patches active" based solely on the B2 patch that fires at
startup. Barrier now gates on _Finder._targets rather than the removed
_PATCHES dict.
- Add comment in exec_module noting the if/elif must stay in sync with
_Finder._targets, making the coupling visible.
patch_ui.py:
- Merge find_webui into find_bundle: previously find_bundle(None) would crash
with os.walk(None) if main()'s guard were removed. Merged function returns
a 3-tuple (webui_dir, path, content); webui_dir=None means no candidate
directory found, path=None means directory found but pattern absent.
main() still produces distinct messages for each failure mode.
create_task.py:
- Split triple-chained .get() in cmd_list_tasks into two lines; the or {}
handling for None credentials was buried inside a one-liner.
uninstall.sh:
- Fix find loop: replace "for x in $(find ...)" with "while IFS= read -r"
to handle paths containing spaces or newlines.
- Add #!/usr/bin/env shebang form to Python detection, matching apply.sh.
Without this, uninstall on a system where middlewared uses the env form
would silently leave sitecustomize.py in the wrong site-packages.
sitecustomize.py:
- Replace _PATCHES dispatch dict with if/elif in exec_module; removes
coupling between dispatch and _record_status's count barrier
- Drop _record_status count barrier entirely; both patches fire within
milliseconds during the same import sequence, write-on-every-call is safe
- Replace _broken_url regex with str.partition + startswith checks; same
semantics, no regex knowledge required to read
- Replace @staticmethod decorator inside plain function with explicit
staticmethod() assignment; decorator form creates a descriptor object,
not a callable, which confuses readers expecting class-body usage
apply.sh:
- Inline warn/ok helpers; each was one echo with a prefix, the indirection
cost more than the abstraction saved
- Collapse patch_ui.py if/else (whose if branch was a no-op comment) to
a single || fallback line
create_task.py:
- Remove vestigial (_client, _args) params from cmd_verify; it was pulled
out of dispatch, the params were never used
- Replace 3-entry dispatch dict with if/elif; dict implied a uniform calling
convention that verify already broke
create_task.py:
- Remove dead make_client() call before the verify branch; it was called
unconditionally with host=None/key=None, creating a broken client that was
immediately discarded or overwritten.
- Remove "verify" from the dispatch dict; it was never reached through dispatch
(the if/cmd==verify branch above it handled it). Dispatch now only contains
commands that actually use a client.
- Wrap json.load() in try/except (OSError, JSONDecodeError) so a corrupt or
partially-written status file produces a useful message instead of a traceback.
sitecustomize.py:
- Call _record_status() on the early-return paths in both _patch_b2 and
_patch_restic. Without this, if TrueNAS natively supports B2 or the patch
is already applied, the status file was never written and `verify` always
reported failure even when everything was fine.
- Add idempotency guard to _record_status(): first call wins; duplicate calls
for the same module are ignored so the entry count stays accurate.
- Make B2 get_restic_config a @staticmethod. The method never used self; the
noqa comment was suppressing the evidence of a design mismatch. Removing the
unused parameter makes the intent explicit.
- Add NamedTuple._replace() fallback after dataclasses.replace() in the restic
wrapper. If ResticConfig is ever refactored to a NamedTuple, the TypeError
from dataclasses.replace() would have surfaced as a backup job failure rather
than a graceful recovery.
sitecustomize.py: _patch_restic no longer reimplements get_restic_config.
It now wraps the original: calls _orig(cloud_backup) to get a ResticConfig,
then post-processes only the -r argument to fix "b2:/bucket" → "b2:bucket"
when the URL contains a stray leading slash (the stock bug for empty-hostname
providers). Uses dataclasses.replace() to build the corrected result so new
ResticConfig fields added in future TrueNAS versions pass through unchanged.
This eliminates the transfer_setting gap, env dict mutation, and frozen-copy
drift that would occur over time.
Also adds a status file mechanism: sitecustomize.py writes
/data/truecloud-patch/hook_status.json atomically after both patches have
reported success or failure. This gives a machine-readable signal that the
hook fired correctly — without requiring log scraping.
create_task.py: new "verify" subcommand reads the status file and prints a
human-readable summary. Does not require --host or --api-key. --host and
--api-key are now optional at the parser level and validated only for
subcommands that actually need an API connection.
README: update troubleshooting to use "create_task.py verify" instead of
the manual Python introspection one-liner.
Patches middlewared at runtime via sitecustomize.py (no file edits to /usr/)
and widens the UI credential dropdown from Storj-only to S3+B2+Storj.
Persists across TrueNAS updates via PREINIT initshutdownscript stored in DB.
flan
flan