Fix deferred restart racing boot: wait for boot to settle before restarting middlewared

The truecloud-mw-restart unit relied on After=multi-user.target /
After=ix-postinit.service, but systemd ordering cannot see middlewared's
internal boot work. On 25.10.4 the restart fired two seconds into
ix-reporting's reporting.start_service call and before the docker/apps
startup task ran, killing both for the whole boot: all apps down
(docker.status FAILED), no dashboard stats, SMB backend uninitialized.

The unit now runs patch/wait_restart.sh: drain the systemd boot job
queue (is-system-running --wait), poll docker.status until the state
machine leaves its transitional states, short grace period, then
try-restart. No Type=oneshot — a oneshot's start job sits in the very
queue the script waits on and would deadlock on itself. All waits are
bounded and fail open.

CHANGELOG.md

@@ -1,5 +1,132 @@
 # Changelog
 
+## v0.2.1 — 2026-07-09
+
+### Fixed
+
+- **Deferred restart raced the rest of boot, leaving all apps and dashboard
+  stats down.** The `truecloud-mw-restart` unit introduced in v0.0.4 relied
+  on systemd ordering (`After=multi-user.target`, `After=ix-postinit.service`),
+  which cannot see middlewared's *internal* boot work. Observed on 25.10.4:
+  the restart fired two seconds into `ix-reporting.service`'s
+  `midclt call reporting.start_service` and before the docker/apps startup
+  task (created on middlewared's system-ready event) had run. Both were
+  killed, and nothing retries them until the next boot — every app stayed
+  down (`docker.status` FAILED, the apps dataset never mounted), netdata
+  never started (no dashboard hardware stats), and the SMB middleware
+  backend was left uninitialized.
+
+  The transient unit now runs `patch/wait_restart.sh` instead of restarting
+  directly: it waits for the systemd boot job queue to drain
+  (`systemctl is-system-running --wait`, covering in-flight `ix-*` oneshots
+  such as ix-reporting), then polls `midclt call docker.status` until the
+  docker state machine leaves its transitional states, then allows a short
+  grace period for middleware-internal tasks with no queryable state before
+  issuing `systemctl try-restart middlewared`. The unit no longer sets
+  `Type=oneshot` — a oneshot's start job stays in the very queue the script
+  waits on and would deadlock on itself. All waits are bounded and fail
+  open: worst case the restart still happens, just later.
+
+  Recovery on a boot that already hit this (without rebooting):
+  `midclt call reporting.start_service` and
+  `midclt call docker.state.start_service true`.
+
+## v0.2.0 — 2026-07-08
+
+### Changed
+
+- **`create_task.py` now uses the TrueNAS middleware via `midclt` instead of the
+  deprecated `/api/v2.0` REST API**, which is removed in TrueNAS 26.04. Practical
+  effects:
+  - Run the script **on the TrueNAS host** — it uses the local middleware socket, so
+    it no longer needs a host address or API key.
+  - `--host`, `--api-key`, and `--insecure` are accepted but **ignored** (a deprecation
+    note is printed); they will be removed in a future release.
+  - `list-credentials` → `cloudsync.credentials.query`, `list-tasks` →
+    `cloud_backup.query`, `create` → `cloud_backup.create`.
+- Dropped the `ssl`/`urllib` HTTP client; no TLS certificate handling is needed anymore.
+
+## v0.1.0 — 2026-07-08
+
+### Added
+
+- `create --cache-path PATH` — sets the restic cache directory on the task.
+  Without a cache path, TrueNAS runs restic with `--no-cache`, which re-reads all
+  repository metadata from the provider on every run and is glacially slow on
+  large repos (a 564 GB dataset estimated **55 days** to a first backup). Tasks
+  created without `--cache-path` now print a warning explaining the consequence.
+
+## v0.0.4 — 2026-07-06
+
+### Fixed
+
+- **Backend patch inactive after every reboot.** PREINIT initshutdownscripts
+  are executed by middlewared itself (`ix-preinit.service` runs
+  `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after
+  `ix-zfs.service` pool import). By the time `apply.sh` patched `b2.py` and
+  `restic.py` in the overlay, the running middlewared had already imported the
+  stock modules and never re-imports — so S3/B2 support silently reverted on
+  every reboot until something restarted middlewared. `install.sh` masked the
+  bug because it restarts middlewared explicitly.
+
+  Fix: when `apply.sh` detects it was invoked by middlewared (boot context),
+  it now schedules a single detached restart via a transient systemd unit
+  (`truecloud-mw-restart`, ordered after `multi-user.target` and
+  `ix-postinit.service`) so the patched modules are loaded once boot settles.
+  The restart is never synchronous — `apply.sh` is a child of middlewared's
+  own job runner, and later `ix-*` boot units still need midclt. Manual runs
+  of `apply.sh` never trigger a restart.
+
+- **`TypeError: string indices must be integers` when creating a B2 task on
+  TrueNAS 24.10 (Electric Eel)** (#1). The credential schema differs between
+  releases: on 24.10 `credentials["provider"]` is the type string (`"B2"`)
+  with the account/key in `credentials["attributes"]`, while 25.04+ moved
+  them into a provider dict. The injected `get_restic_config` only handled
+  the 25.04+ shape. It now detects the schema and reads the credentials from
+  the right place on both; `create_task.py list-credentials` and `list-tasks`
+  got the same treatment.
+
+- **`create_task.py verify` false-positive after reboot.** `verify` trusted
+  `hook_status.json`, which only records that the files were patched on disk —
+  not that the running process loaded them. `verify` now also compares the
+  middlewared main-process start time against `patched_at` and reports FAIL
+  (with recovery instructions) when the process predates the patch.
+
+### Changed
+
+- README and script comments no longer claim PREINIT runs "before middlewared
+  starts"; the boot ordering and the deferred restart are now documented.
+- `recover.sh` and `uninstall.sh` cancel a still-queued deferred restart
+  before performing their own, and their re-enable instructions now include
+  the required `systemctl restart middlewared`.
+
+---
+
+## v0.0.3 — 2026-06-22
+
+### Fixed
+
+- **`patch/apply.sh` silently killed by the 10-second PREINIT timeout.**
+  TrueNAS PREINIT initshutdownscripts have a 10-second default timeout. The
+  previous apply.sh ran approximately 8 Python subprocesses (each ~1-2 s), so it
+  was routinely killed mid-run. Symptoms: patches not applied after reboot, but
+  re-running `bash apply.sh` manually (no timeout) always succeeded.
+
+  Fix: `install.sh` now registers the hook with `"timeout": 120`. Existing
+  installations are updated to the new timeout on the next `bash install.sh` run.
+
+  Additionally, `patch/apply.sh` consolidates its Python subprocess invocations
+  from ~8 down to 2, reducing startup overhead from ~12-16 s to ~2-4 s — well
+  within the new 120-second budget.
+
+### Changed
+
+- `find_mw_python` in `apply.sh` no longer spawns a separate Python process to
+  verify the interpreter can import `middlewared`. Verification is now implicit in
+  the combined path-discovery subprocess that follows.
+
+---
+
 ## v0.0.2 — 2026-06-19
 
 ### Fixed

README.md

@@ -54,9 +54,9 @@ see [Native support](#if-truenas-adds-native-support) below.
 
 ## What is actually patched
 
-**Nothing in TrueNAS's persistent database or configuration is modified.**
+**Nothing in TrueNAS's persistent database or configuration is modified**
-On every boot, `patch/apply.sh` runs as a PREINIT script before middlewared
+(other than the boot-hook entry itself). On every boot, `patch/apply.sh` runs
-starts. It mounts a writable
+as a PREINIT script. It mounts a writable
 [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) over the
 relevant directories in `/usr/` (upper layer in `/run` tmpfs), then patches
 `b2.py` and `restic.py` inside that overlay. The overlay is volatile — it
@@ -64,6 +64,16 @@ exists only for the current boot — but the PREINIT script recreates it
 automatically on every subsequent boot. Nothing in `/usr/` is written to
 directly.
 
+PREINIT scripts are executed *by* middlewared, which by then has already
+imported the stock modules — so after patching, `apply.sh` schedules a single
+detached middlewared restart (transient systemd unit `truecloud-mw-restart`
+running `patch/wait_restart.sh`) that loads the patched modules once boot has
+*actually* settled: the script waits for the systemd boot job queue to drain
+and for the docker/apps state machine to reach a terminal state before
+restarting. Expect one middlewared restart shortly after every boot; the UI
+and API are briefly unavailable while it happens, and running services are
+not affected.
+
 | Layer | What changes | Technique |
 |---|---|---|
 | **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | File patch applied inside the overlayfs upper layer |
@@ -83,14 +93,51 @@ support and the reason is logged to `apply.log` in your repo root.
 
 ## How persistence works
 
-TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by keeping
+Two different things must survive two different events:
-this repository on a **persistent ZFS pool** (your data pool, not `/tmp` or a
+
-system path) and registering a **PREINIT initshutdownscript** in the TrueNAS
+| Event | What would be lost | What makes it survive |
-database. On every boot, `patch/apply.sh` runs before `middlewared` starts. It
+|---|---|---|
-mounts a writable [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html)
+| **Reboot** | The overlay holding the patched files lives in `/run` (tmpfs) and vanishes | The PREINIT hook re-runs `apply.sh` on every boot and schedules one middlewared restart to load the result |
-over the relevant directories (upper layer in `/run`, recreated each boot), then
+| **TrueNAS update** | `/usr/` is replaced entirely; custom files in `/etc/` are wiped with the new boot environment | This repo lives on your **data pool**, and the hook registration lives in the **TrueNAS config database** — both survive updates. The first boot after an update is just a normal boot |
-patches `b2.py` and `restic.py` directly in that overlay and re-patches the UI
+
-bundle. No extra configuration is needed.
+### What happens on every boot
+
+1. **middlewared starts** with the stock (unpatched) modules. This is
+   unavoidable: PREINIT scripts are executed *by* middlewared
+   (`ix-preinit.service` → `midclt call initshutdownscript.execute_init_tasks`),
+   so nothing registered there can run before it.
+2. **Pools import** (`ix-zfs.service`), making `/mnt/<pool>` — and this
+   repository — available.
+3. **`apply.sh` runs** (`ix-preinit.service`): mounts the writable overlay
+   (upper layer in `/run`), patches `b2.py` and `restic.py` on disk inside it,
+   patches the UI bundle, and writes `apply.log` and `hook_status.json`.
+4. **A deferred restart is scheduled.** The middlewared that is running
+   imported the stock modules in step 1 and never re-imports, so the on-disk
+   patch alone is not enough. `apply.sh` detects it was invoked by middlewared
+   and creates a transient systemd unit (`truecloud-mw-restart`, via
+   `systemd-run --no-block`) running `patch/wait_restart.sh` — detached so it
+   cannot disrupt the remainder of the boot sequence.
+5. **Once boot has settled, middlewared restarts once** and imports the
+   patched modules from the overlay. `wait_restart.sh` holds the restart until
+   the systemd boot job queue has drained (so in-flight `ix-*` units like
+   `ix-reporting` finish first) *and* middlewared's docker/apps startup has
+   reached a terminal state — plain unit ordering cannot see either, and
+   restarting middlewared while they run kills apps and dashboard reporting
+   for the whole boot. S3/B2 backup support is then active until the next
+   reboot, when the cycle repeats.
+
+What you will observe: one middlewared restart shortly after every boot (a
+brief web UI/API blip; running services are unaffected). Between steps 3
+and 5 there is a short window — typically well under a minute — where the UI
+already shows S3/B2 (the JS bundle is read from disk per request) but the
+backend is still stock. A backup job that fires inside that window fails once
+with `NotImplementedError` and succeeds on its next run; see
+[Troubleshooting](#troubleshooting) if it persists beyond boot.
+
+Manual runs of `bash patch/apply.sh` never trigger the restart — that only
+happens in boot context. `install.sh` and `recover.sh` perform their own
+explicit restarts instead, which is why a manual re-apply must be followed by
+`systemctl restart middlewared`.
 
 ---
 
@@ -120,31 +167,61 @@ call that path on every boot.
 Refresh your browser. S3 and B2 credentials now appear in the
 **Data Protection → TrueCloud Backup → Add** credential dropdown.
 
+## Updating
+
+To update to a new version of the patch:
+
+```bash
+cd /mnt/tank/truenas-truecloud-patch
+
+# If install.sh was previously run as root, the .git directory may be owned
+# by root. Fix it first, or just pull as root:
+sudo git pull          # easiest option
+# — or —
+sudo chown -R $(whoami) .git && git pull
+
+bash install.sh
+```
+
+`install.sh` clears any stale kill switch, re-applies the updated patches,
+and restarts middlewared. Run `python3 patch/create_task.py verify` afterwards
+to confirm the patches loaded successfully.
+
+Check [CHANGELOG.md](CHANGELOG.md) to see what changed between versions.
+
+---
+
 ## Creating a task via CLI
 
 If the UI still shows only Storj after refreshing (e.g. the JS bundle pattern
-changed in a new TrueNAS version), create tasks directly via the REST API:
+changed in a new TrueNAS version), create tasks directly. Run this **on the
+TrueNAS host** — it talks to the local middleware via `midclt`, so it needs no
+host address or API key:
 
 ```bash
 # Replace /mnt/tank/truenas-truecloud-patch with your clone path
 
 # List your cloud credentials to find the right ID
-python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py \
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py list-credentials
-    --host 192.168.1.1 --api-key <key> list-credentials
 
 # Create a task with a B2 credential (id=3)
-python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py \
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py create \
-    --host 192.168.1.1 --api-key <key> create \
     --name "tank-to-b2" \
     --path /mnt/tank/data \
     --credential 3 \
     --bucket my-bucket \
     --folder backups/tank \
     --password "restic-repo-password" \
+    --cache-path /mnt/tank/.restic-cache \
     --keep-last 14
 ```
 
-Get an API key from **System → API Keys → Add**.
+> **Always pass `--cache-path`.** Without it TrueNAS runs restic with `--no-cache`,
+> which re-fetches all repo metadata from the provider every run — glacially slow
+> on large repos. Point it at a writable dir on a pool with free space.
+
+> Versions ≤ 0.1.0 used the `/api/v2.0` REST API with `--host`/`--api-key`; those
+> flags are now accepted-but-ignored (REST is removed in TrueNAS 26.04).
 
 ---
 
@@ -252,6 +329,7 @@ To re-enable the patch once you have investigated:
 ```bash
 rm /mnt/tank/truenas-truecloud-patch/disabled
 bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
+systemctl restart middlewared   # manual apply.sh runs never restart for you
 ```
 
 ---
@@ -306,6 +384,36 @@ If one or more entries show `[FAIL]`:
 
 ## Troubleshooting
 
+**Backups fail with `NotImplementedError` after a reboot**
+
+The traceback ends in `rclone/base.py` → `raise NotImplementedError` and
+contains no `_tc_` frames: the running middlewared is executing stock code.
+Either the deferred restart never fired, or the patch never landed on disk
+this boot. Diagnose in this order:
+
+```bash
+# Did apply.sh run this boot, at which version, and did it schedule the restart?
+tail -40 /mnt/tank/truenas-truecloud-patch/apply.log
+
+# Full check — compares the running process against the patch timestamp
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
+
+# Did the deferred restart unit run, fail, or never get created?
+systemctl status truecloud-mw-restart.service
+journalctl -u truecloud-mw-restart.service --no-pager | tail -20
+```
+
+- `verify` reports the process started **before** the patch → the restart
+  didn't happen. `systemctl restart middlewared` fixes it immediately; the
+  journal output above tells you why it was missed.
+- `apply.log` shows the kill switch is active → `rm .../disabled`, then
+  `bash install.sh`.
+- `apply.log` has no entry for this boot → the hook didn't run; re-run
+  `bash install.sh` to re-register it.
+- `apply.log` header shows `[v0.0.3]` or older → update:
+  `git pull && bash install.sh` (v0.0.4 fixed patches not loading after
+  reboot).
+
 **Apply log** (check after each reboot or install):
 ```bash
 cat /mnt/tank/truenas-truecloud-patch/apply.log
@@ -315,9 +423,10 @@ cat /mnt/tank/truenas-truecloud-patch/apply.log
 ```bash
 python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
 ```
-Reads `hook_status.json` written by `apply.sh` at boot. Reflects whether the
+Reads `hook_status.json` written by `apply.sh` at boot **and** checks that the
-overlay patches to `b2.py` and `restic.py` were applied successfully. Does not
+running middlewared process started *after* the patches were applied — an
-require `--host` or `--api-key`.
+on-disk patch that middlewared has not loaded yet is reported as FAIL with
+instructions. Does not require `--host` or `--api-key`.
 
 **Middlewared log:**
 ```bash
@@ -332,9 +441,7 @@ grep -c 'STORJ_IX.*S3.*B2' \
     | grep -v ':0'
 ```
 
-**`create_task.py` SSL error connecting to TrueNAS**
+**`create_task.py` — "midclt not found" or permission errors**
-`create_task.py` talks to the **TrueNAS API**, not your S3 endpoint, and
+`create_task.py` now talks to the local middleware via `midclt`, so run it **on
-verifies its TLS certificate. If your NAS uses a self-signed certificate,
+the TrueNAS host** (not remotely) as a user with middleware access (root). There
-pass `--insecure` — but be aware this disables certificate verification for
+is no HTTPS/API-key call anymore, so there is no TLS certificate to configure.
-the API call that transmits your TrueNAS API key. Adding your NAS certificate
-to your system's trust store is safer.

install.sh

@@ -10,12 +10,16 @@
 #
 # What this does:
 #   1. Registers a PREINIT initshutdownscript so patch/apply.sh re-runs on
-#      every boot before middlewared starts.
+#      every boot. At boot, apply.sh re-patches the overlay and schedules a
+#      one-time deferred middlewared restart to load the patched modules
+#      (PREINIT runs after middlewared starts, so a restart is required).
 #   2. Applies the patches immediately (no reboot required).
 #   3. Restarts middlewared so the backend change takes effect now.
 
 set -euo pipefail
 
+VERSION="0.0.4"
+
 # The directory containing install.sh is the permanent install location.
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
@@ -29,7 +33,7 @@ if [ ! -f "$PATCH_DIR/patch/apply.sh" ]; then
     exit 1
 fi
 
-echo "=== TrueNAS TrueCloud Provider Patch — Install ==="
+echo "=== TrueNAS TrueCloud Provider Patch v${VERSION} — Install ==="
 echo ""
 
 # ── Preflight ─────────────────────────────────────────────────────────────────
@@ -71,9 +75,9 @@ for s in json.load(sys.stdin):
 " 2>/dev/null || true)
 
 if [ -n "$EXISTING_ID" ]; then
-    echo "Already registered (id=$EXISTING_ID). Updating path and enabling ..."
+    echo "Already registered (id=$EXISTING_ID). Updating path, timeout, and enabling ..."
     if ! _midclt_out=$(midclt call initshutdownscript.update "$EXISTING_ID" \
-            "{\"enabled\": true, \"script\": \"$PATCH_DIR/patch/apply.sh\"}" 2>&1); then
+            "{\"enabled\": true, \"script\": \"$PATCH_DIR/patch/apply.sh\", \"timeout\": 120}" 2>&1); then
         echo "ERROR: Failed to update PREINIT hook (id=$EXISTING_ID)." >&2
         [ -n "$_midclt_out" ] && echo "  midclt: $_midclt_out" >&2
         echo "  To remove the stale entry and retry:" >&2
@@ -82,7 +86,7 @@ if [ -n "$EXISTING_ID" ]; then
     fi
 else
     if ! _midclt_out=$(midclt call initshutdownscript.create \
-            "{\"type\":\"SCRIPT\",\"script\":\"$PATCH_DIR/patch/apply.sh\",\"when\":\"PREINIT\",\"enabled\":true,\"comment\":\"$_HOOK_COMMENT\"}" \
+            "{\"type\":\"SCRIPT\",\"script\":\"$PATCH_DIR/patch/apply.sh\",\"when\":\"PREINIT\",\"enabled\":true,\"timeout\":120,\"comment\":\"$_HOOK_COMMENT\"}" \
             2>&1); then
         echo "ERROR: Failed to register PREINIT hook." >&2
         [ -n "$_midclt_out" ] && echo "  midclt: $_midclt_out" >&2

patch/apply.sh

@@ -1,15 +1,21 @@
 #!/bin/bash
 # patch/apply.sh — registered as a TrueNAS PREINIT initshutdownscript.
 #
-# Runs on every boot BEFORE middlewared starts, so patches land before
+# PREINIT scripts are executed BY middlewared itself (ix-preinit.service runs
-# the first Python process for middlewared is created.
+# `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after
+# ix-zfs.service pool import). So when this script runs at boot, middlewared
+# is already up and has already imported the stock modules — the on-disk
+# patch alone cannot reach the running process.
 #
 # TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
 #
 #   1. Backend — b2.py and restic.py are patched directly in the overlay.
+#      On a boot run, a single detached middlewared restart is scheduled
+#      (Step 3) so the patched modules actually get loaded.
 #
 #   2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
-#                          from Storj-only to include S3 and B2.
+#                          from Storj-only to include S3 and B2. Served from
+#                          disk per request, so no restart is needed for it.
 #
 # Design principle: every step is independently fail-safe.
 # A failed patch logs a warning and continues; middlewared always starts.
@@ -18,6 +24,7 @@
 # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
 PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 LOG="$PATCH_DIR/apply.log"
+VERSION="0.2.1"
 
 # Rotate log at 512 KB to avoid unbounded growth on a system volume.
 # Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -27,7 +34,7 @@ if [ -f "$LOG" ] && [ "$(wc -c < "$LOG")" -gt 524288 ]; then
 fi
 
 exec >> "$LOG" 2>&1
-echo "=== $(date -Iseconds) ==="
+echo "=== $(date -Iseconds) [v${VERSION}] ==="
 
 # Kill switch: if this file exists, skip all patching and exit cleanly.
 # Recovery: touch "$PATCH_DIR/disabled" (then reboot or restart middlewared).
@@ -40,7 +47,7 @@ fi
 
 # Mounts a writable overlayfs on $1 using /run (tmpfs) for the upper/work dirs
 # when the directory is read-only.  The overlay is volatile per boot; this
-# PREINIT script recreates it on every boot before middlewared starts.
+# PREINIT script recreates it on every boot.
 # Returns 0 if the directory is now writable, 1 if it could not be made so.
 _ensure_writable() {
     local dir="$1" tag="$2"
@@ -67,6 +74,8 @@ _ensure_writable() {
 # Find the Python interpreter that middlewared actually uses.
 # On TrueNAS SCALE, /usr/bin/middlewared is usually a Python entry-point script
 # with a shebang pointing at the right interpreter (system or venv).
+# The shebang is read with dd (no Python startup cost); import verification is
+# deferred to the combined subprocess below which handles failure gracefully.
 find_mw_python() {
     local py="python3"
     local shebang=""
@@ -82,47 +91,56 @@ find_mw_python() {
         fi
     fi
 
-    # Verify the chosen interpreter can actually import middlewared.
-    # Use >&2 so this message goes to stderr, not captured by $(...) substitution.
-    if ! "$py" -c "import middlewared" 2>/dev/null; then
-        echo "WARNING: '$py' cannot import middlewared; falling back to python3" >&2
-        py="python3"
-        if ! "$py" -c "import middlewared" 2>/dev/null; then
-            echo "WARNING: 'python3' also cannot import middlewared; backend patch will be skipped" >&2
-        fi
-    fi
-
     echo "$py"
 }
 
 PYTHON=$(find_mw_python)
 echo "Using Python: $PYTHON"
 
-# ── Native support check ──────────────────────────────────────────────────────
+# ── Discover paths + native support check (single Python subprocess) ──────────
-# If TrueNAS has shipped native B2 restic support, this patch is no longer
+# Combines what were previously four separate Python invocations into one to
-# needed. Set the kill switch and instruct the user to uninstall cleanly.
+# avoid repeated interpreter startup overhead under the PREINIT timeout budget.
+
+_tc_info=$("$PYTHON" -c "
+import inspect, os, sys
+
+result = {'native': 'no', 'site_pkg': '', 'mw_dir': ''}
+
+try:
+    import middlewared
+    mw_file = os.path.abspath(middlewared.__file__)
+    result['mw_dir']   = os.path.dirname(mw_file)
+    result['site_pkg'] = os.path.dirname(os.path.dirname(mw_file))
+except ImportError:
+    try:
+        import site
+        result['site_pkg'] = site.getsitepackages()[0]
+    except Exception:
+        pass
 
-_tc_native=$("$PYTHON" -c "
 try:
-    import inspect
     import middlewared.rclone.remote.b2 as _b2_mod
     from middlewared.rclone.remote.b2 import B2RcloneRemote
-    if 'get_restic_config' not in B2RcloneRemote.__dict__:
+    if 'get_restic_config' in B2RcloneRemote.__dict__:
-        print('no')
-    else:
         src = open(inspect.getfile(_b2_mod), encoding='utf-8', errors='replace').read()
-        if 'TRUECLOUD_PATCH' in src:
+        if 'TRUECLOUD_PATCH' not in src:
-            print('no')
-        else:
-            # Distinguish a real implementation from a stub that raises NotImplementedError.
             try:
                 method_src = inspect.getsource(B2RcloneRemote.get_restic_config)
             except (OSError, TypeError):
                 method_src = ''
-            print('no' if 'NotImplementedError' in method_src else 'yes')
+            if 'NotImplementedError' not in method_src:
+                result['native'] = 'yes'
 except Exception:
-    print('no')
+    pass
-" 2>/dev/null || echo "no")
+
+print(result['native'])
+print(result['site_pkg'])
+print(result['mw_dir'])
+" 2>/dev/null || printf 'no\n\n\n')
+
+_tc_native=$(printf '%s' "$_tc_info" | sed -n '1p')
+SITE_PKG=$(printf '%s' "$_tc_info"  | sed -n '2p')
+_MW_DIR=$(printf '%s' "$_tc_info"   | sed -n '3p')
 
 if [ "$_tc_native" = "yes" ]; then
     echo "NOTICE: TrueNAS now provides native B2 restic support — truecloud-patch is no longer needed."
@@ -144,29 +162,6 @@ fi
 
 echo "--- backend patch ---"
 
-# Derive site-packages from where middlewared actually lives.
-# getsitepackages()[0] may return the wrong directory; using middlewared.__file__
-# ensures we patch files in the directory Python will actually read.
-SITE_PKG=$("$PYTHON" -c "
-import os
-try:
-    import middlewared
-    print(os.path.dirname(os.path.dirname(os.path.abspath(middlewared.__file__))))
-except ImportError:
-    import site
-    print(site.getsitepackages()[0])
-" 2>/dev/null || true)
-
-# MW_DIR is the middlewared package directory itself (one level below SITE_PKG).
-_MW_DIR=$("$PYTHON" -c "
-import os
-try:
-    import middlewared
-    print(os.path.dirname(os.path.abspath(middlewared.__file__)))
-except ImportError:
-    pass
-" 2>/dev/null || true)
-
 _b2_ok=0
 _restic_ok=0
 
@@ -181,49 +176,28 @@ else
     _B2_PY="$_MW_DIR/rclone/remote/b2.py"
     _RESTIC_PY="$_MW_DIR/plugins/cloud_backup/restic.py"
 
-        # ── b2.py ─────────────────────────────────────────────────────────────
+        # ── patch b2.py + restic.py + hook_status.json (single subprocess) ──────
-        if [ -f "$_B2_PY" ]; then
+        if "$PYTHON" - "$_B2_PY" "$_RESTIC_PY" "$PATCH_DIR/hook_status.json" << 'PYEOF'
-            if "$PYTHON" - "$_B2_PY" << 'PYEOF'
+import json, os, sys, time
-import sys
 
-BLOCK = """
+b2_path, restic_path, status_path = sys.argv[1], sys.argv[2], sys.argv[3]
+
+B2_BLOCK = """
 # TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
 def _tc_get_restic_config(task):
     p = task["credentials"]["provider"]
+    if not isinstance(p, dict):
+        # TrueNAS <= 24.10: provider is the type string ("B2") and the
+        # account/key live in the credential's attributes dict. 25.04+
+        # moved them into a provider dict.
+        p = task["credentials"]["attributes"]
     return "", {"B2_ACCOUNT_ID": p["account"], "B2_ACCOUNT_KEY": p["key"]}
 
 B2RcloneRemote.get_restic_config = staticmethod(_tc_get_restic_config)
 B2RcloneRemote.restic = True
 """
 
-path = sys.argv[1]
+RESTIC_BLOCK = """
-with open(path, encoding="utf-8") as fh:
-    content = fh.read()
-
-marker = "\n# TRUECLOUD_PATCH"
-idx = content.find(marker)
-base = content[:idx] if idx != -1 else content
-patched = base.rstrip("\n") + "\n" + BLOCK
-
-with open(path, "w", encoding="utf-8") as fh:
-    fh.write(patched)
-PYEOF
-            then
-                echo "OK: Patched b2.py → $_B2_PY"
-                _b2_ok=1
-            else
-                echo "WARNING: Failed to patch b2.py"
-            fi
-        else
-            echo "WARNING: b2.py not found at $_B2_PY"
-        fi
-
-        # ── restic.py ─────────────────────────────────────────────────────────
-        if [ -f "$_RESTIC_PY" ]; then
-            if "$PYTHON" - "$_RESTIC_PY" << 'PYEOF'
-import sys
-
-BLOCK = """
 # TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
 try:
     _tc_orig_get_restic_config = get_restic_config
@@ -266,56 +240,66 @@ else:
     get_restic_config._truecloud_patched = True
 """
 
-path = sys.argv[1]
+def patch_file(path, block):
     with open(path, encoding="utf-8") as fh:
         content = fh.read()
-
     marker = "\n# TRUECLOUD_PATCH"
     idx = content.find(marker)
     base = content[:idx] if idx != -1 else content
-patched = base.rstrip("\n") + "\n" + BLOCK
-
     with open(path, "w", encoding="utf-8") as fh:
-    fh.write(patched)
+        fh.write(base.rstrip("\n") + "\n" + block)
-PYEOF
+
-            then
+b2_ok = restic_ok = False
-                echo "OK: Patched restic.py → $_RESTIC_PY"
+
-                _restic_ok=1
+if os.path.exists(b2_path):
-            else
+    try:
-                echo "WARNING: Failed to patch restic.py"
+        patch_file(b2_path, B2_BLOCK)
-            fi
+        b2_ok = True
-        else
+        print(f"OK: Patched b2.py → {b2_path}")
-            echo "WARNING: restic.py not found at $_RESTIC_PY"
+    except Exception as e:
-        fi
+        print(f"WARNING: Failed to patch b2.py: {e}")
-    # Write hook_status.json so 'verify' reflects the current patch state.
+else:
-    "$PYTHON" -c "
+    print(f"WARNING: b2.py not found at {b2_path}")
-import json, os, sys, time
+
-b2_ok = sys.argv[1] == '1'
+if os.path.exists(restic_path):
-restic_ok = sys.argv[2] == '1'
+    try:
+        patch_file(restic_path, RESTIC_BLOCK)
+        restic_ok = True
+        print(f"OK: Patched restic.py → {restic_path}")
+    except Exception as e:
+        print(f"WARNING: Failed to patch restic.py: {e}")
+else:
+    print(f"WARNING: restic.py not found at {restic_path}")
+
 patches = {
     'middlewared.rclone.remote.b2': {
         'ok': b2_ok,
-        'detail': ('patched on disk in overlay at boot' if b2_ok
+        'detail': 'patched on disk in overlay at boot' if b2_ok else 'b2.py not found or write failed',
-                   else 'b2.py not found or write failed'),
     },
     'middlewared.plugins.cloud_backup.restic': {
         'ok': restic_ok,
-        'detail': ('patched on disk in overlay at boot' if restic_ok
+        'detail': 'patched on disk in overlay at boot' if restic_ok else 'restic.py not found or write failed',
-                   else 'restic.py not found or write failed'),
     },
 }
-payload = {'patched_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
+payload = {'patched_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), 'patches': patches}
-           'patches': patches}
+tmp = status_path + '.tmp'
-sf = sys.argv[3]
-tmp = sf + '.tmp'
 try:
     with open(tmp, 'w') as f:
         json.dump(payload, f, indent=2)
-    os.replace(tmp, sf)
+    os.replace(tmp, status_path)
     print('OK: Wrote hook_status.json')
 except OSError as e:
     print(f'WARNING: Could not write hook_status.json: {e}')
-" "$_b2_ok" "$_restic_ok" "$PATCH_DIR/hook_status.json" || true
+
+sys.exit(0 if (b2_ok and restic_ok) else 1)
+PYEOF
+        then
+            _b2_ok=1
+            _restic_ok=1
+        else
+            # Individual results already printed above; exit code 1 means at least one failed.
+            true
+        fi
 fi
 
 # ── Step 2: Angular bundle ────────────────────────────────────────────────────
@@ -337,6 +321,43 @@ fi
 
 "$PYTHON" "$PATCH_DIR/patch/patch_ui.py" || echo "WARNING: patch_ui.py exited non-zero; UI dropdown may still show Storj only."
 
+# ── Step 3: deferred middlewared restart (boot runs only) ─────────────────────
+# At boot this script is spawned by middlewared, which already imported the
+# stock modules — the backend patch is on disk but not in the process. Schedule
+# ONE detached restart for after boot settles. Never restart synchronously
+# here: this script is a child of middlewared's own job runner, and the later
+# ix-* boot units still need midclt to answer.
+# Boot context is detected by the parent process being middlewared; manual
+# runs (install.sh, recovery) never trigger a restart.
+#
+# The unit runs wait_restart.sh, which blocks until boot has actually
+# settled (systemd job queue drained, docker/apps state terminal) before
+# restarting. systemd ordering alone (After=multi-user.target, ≤ v0.0.4)
+# fired while ix-reporting and the docker/apps startup were still in flight
+# and killed both — apps and dashboard stats stayed down until the next
+# boot. No Type=oneshot: a oneshot's start job would hold the boot queue
+# open against the `is-system-running --wait` inside the script.
+
+echo "--- deferred restart ---"
+
+if ! grep -aq middlewared "/proc/$PPID/cmdline" 2>/dev/null; then
+    echo "Manual run (parent is not middlewared) — no restart scheduled."
+elif [ "$_b2_ok" != "1" ] || [ "$_restic_ok" != "1" ]; then
+    echo "Backend patch incomplete — no restart scheduled (nothing new to load)."
+else
+    # A failed unit from an earlier attempt this boot would block systemd-run.
+    systemctl reset-failed truecloud-mw-restart.service 2>/dev/null
+    if systemd-run --no-block --collect --unit=truecloud-mw-restart \
+           /bin/bash "$PATCH_DIR/patch/wait_restart.sh"; then
+        echo "OK: Scheduled deferred middlewared restart (unit: truecloud-mw-restart)."
+        echo "    It waits for boot to fully settle (apps started, reporting up),"
+        echo "    then restarts middlewared so the backend patch actually loads."
+    else
+        echo "WARNING: Could not schedule deferred restart — backend patch is on disk but NOT loaded."
+        echo "  Activate manually: systemctl restart middlewared"
+    fi
+fi
+
 # ── Done ──────────────────────────────────────────────────────────────────────
 
 echo "=== done ==="

patch/create_task.py

@@ -3,22 +3,24 @@
 create_task.py — create TrueNAS TrueCloud Backup tasks with S3 or B2 credentials.
 
 The TrueNAS UI normally restricts the credential dropdown to Storj only.
-This script bypasses that restriction by calling the REST API directly.
+This script bypasses that restriction by talking to the TrueNAS middleware
+directly via `midclt` (the /api/v2.0 REST API is removed in TrueNAS 26.04).
 
 Compatible providers (after the truecloud-patch backend patch is applied):
   S3     — any S3-compatible endpoint (AWS, Wasabi, Cloudflare R2, MinIO, …)
   B2     — Backblaze B2 native API
   STORJ_IX — Storj (unchanged, always worked)
 
-Requires a TrueNAS API key: UI → System → API Keys → Add.
+Run this ON the TrueNAS host — it uses the local middleware socket via `midclt`,
+so no host address or API key is needed.
 
 Examples
 --------
 List available cloud credentials:
-    python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
+    python3 create_task.py list-credentials
 
 Create a task backed by a B2 credential (id=3):
-    python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
+    python3 create_task.py create \\
         --name "tank-to-b2" \\
         --path /mnt/tank/data \\
         --credential 3 \\
@@ -28,7 +30,7 @@ Create a task backed by a B2 credential (id=3):
         --keep-last 14
 
 Create a task using an S3-compatible credential (Wasabi, R2, etc.):
-    python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
+    python3 create_task.py create \\
         --name "tank-to-wasabi" \\
         --path /mnt/tank/data \\
         --credential 5 \\
@@ -37,53 +39,73 @@ Create a task using an S3-compatible credential (Wasabi, R2, etc.):
         --password "restic-repo-password"
 
 List existing TrueCloud Backup tasks:
-    python3 create_task.py --host 192.168.1.1 --api-key <key> list-tasks
+    python3 create_task.py list-tasks
 """
 
 import argparse
+import calendar
 import json
 import os
-import ssl
+import subprocess
 import sys
-import urllib.error
+import time
-import urllib.request
+
+__version__ = "0.2.0"
 
 _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")
 
 
-def make_client(host, api_key, insecure=False):
+def midclt_call(method, *args):
-    """Return a callable that makes authenticated REST API calls."""
+    """Call a middleware method locally via `midclt`, the supported JSON-RPC transport
-    base = f"https://{host}/api/v2.0"
+    that replaces the deprecated /api/v2.0 REST API (removed in TrueNAS 26.04). Must run
-    headers = {
+    on the TrueNAS host. Each arg is JSON-encoded (a dict for create; none for queries).
-        "Authorization": f"Bearer {api_key}",
+    Exits with a clear message on failure."""
-        "Content-Type": "application/json",
+    cmd = ["midclt", "call", method] + [json.dumps(a) for a in args]
-    }
-    ctx = ssl.create_default_context()
-    if insecure:
-        ctx.check_hostname = False
-        ctx.verify_mode = ssl.CERT_NONE
-
-    def call(method, path, body=None):
-        url = base + path
-        data = json.dumps(body).encode() if body is not None else None
-        req = urllib.request.Request(url, data=data, headers=headers, method=method)
     try:
-            with urllib.request.urlopen(req, context=ctx) as resp:
+        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
-                return json.loads(resp.read())
+    except FileNotFoundError:
-        except urllib.error.HTTPError as exc:
+        print("ERROR: `midclt` not found — run this script ON the TrueNAS host.",
-            detail = exc.read().decode(errors="replace")
+              file=sys.stderr)
-            print(f"HTTP {exc.code} {exc.reason}: {detail}", file=sys.stderr)
         sys.exit(1)
-        except urllib.error.URLError as exc:
+    except subprocess.SubprocessError as exc:
-            print(f"Connection error: {exc.reason}", file=sys.stderr)
+        print(f"ERROR: midclt call failed: {exc}", file=sys.stderr)
         sys.exit(1)
-
+    if proc.returncode != 0:
-    return call
+        print(f"ERROR: midclt {method}: {(proc.stderr or proc.stdout).strip()}",
+              file=sys.stderr)
+        sys.exit(1)
+    out = proc.stdout.strip()
+    return json.loads(out) if out else None
 
 
 # ── Sub-commands ──────────────────────────────────────────────────────────────
 
+def _middlewared_start_epoch():
+    """Epoch timestamp of the running middlewared main process, or None."""
+    try:
+        pid = int(subprocess.run(
+            ["systemctl", "show", "--property=MainPID", "--value", "middlewared"],
+            capture_output=True, text=True, timeout=10, check=True,
+        ).stdout.strip())
+        if pid <= 0:
+            return None
+        with open(f"/proc/{pid}/stat", encoding="ascii", errors="replace") as fh:
+            stat = fh.read()
+        # Field 22 (starttime, in clock ticks since boot); the comm field may
+        # contain spaces, so split after the closing paren.
+        start_ticks = float(stat.rsplit(")", 1)[1].split()[19])
+        # Base on /proc/stat btime, not uptime: starttime ticks count from the
+        # kernel boot, which uptime does not match inside containers.
+        with open("/proc/stat", encoding="ascii") as fh:
+            btime = next(float(line.split()[1]) for line in fh
+                         if line.startswith("btime "))
+        return btime + start_ticks / os.sysconf("SC_CLK_TCK")
+    except (OSError, ValueError, IndexError, StopIteration,
+            subprocess.SubprocessError):
+        return None
+
+
 def cmd_verify():
     """Print the hook status written by apply.sh at boot."""
     if not os.path.exists(_STATUS_FILE):
@@ -113,40 +135,73 @@ def cmd_verify():
         if not ok:
             all_ok = False
 
+    # The disk status alone can false-positive: at boot the files are patched
+    # while middlewared is already running with the stock modules imported.
+    # The running process only has the patch if it started AFTER patched_at.
+    try:
+        patched_epoch = calendar.timegm(
+            time.strptime(status.get("patched_at", ""), "%Y-%m-%dT%H:%M:%SZ"))
+    except ValueError:
+        patched_epoch = None
+    mw_start = _middlewared_start_epoch()
+
+    proc_stale = False
+    if patched_epoch is None or mw_start is None:
+        print("  [??  ] running middlewared process — could not compare start time;")
+        print("         the results above reflect the on-disk state only")
+    elif mw_start + 2 < patched_epoch:
+        proc_stale = True
+        print("  [FAIL] running middlewared process — started BEFORE the patch was applied,")
+        print("         so it is running the stock (unpatched) modules")
+    else:
+        print("  [OK  ] running middlewared process — started after the patch was applied")
+
     print()
-    if all_ok:
+    if all_ok and not proc_stale:
         print("All patches installed. Run a test backup to confirm end-to-end.")
+    elif all_ok:
+        print("The patch is on disk but not loaded. Right after boot, the deferred")
+        print("restart (unit truecloud-mw-restart) may still be pending — re-check in a")
+        print("minute. Otherwise run: systemctl restart middlewared")
+        sys.exit(1)
     else:
         print("One or more patches failed to apply.")
         print(f"Check {os.path.join(_PATCH_DIR, 'apply.log')} and journalctl -u middlewared")
         sys.exit(1)
 
-def cmd_list_credentials(client, _args):
+def _provider_type(cred):
-    creds = client("GET", "/cloudsync/credentials")
+    """Provider type string across schemas (<=24.10 plain str, >=25.04 dict)."""
+    p = (cred or {}).get("provider")
+    if isinstance(p, dict):
+        return p.get("type", "?")
+    return p or "?"
+
+
+def cmd_list_credentials(_args):
+    creds = midclt_call("cloudsync.credentials.query")
     if not creds:
         print("No cloud credentials configured.")
         return
     print(f"{'ID':>4}  {'Provider':<14}  Name")
     print("─" * 55)
     for c in sorted(creds, key=lambda x: x["id"]):
-        print(f"{c['id']:>4}  {c['provider']['type']:<14}  {c['name']}")
+        print(f"{c['id']:>4}  {_provider_type(c):<14}  {c['name']}")
 
 
-def cmd_list_tasks(client, _args):
+def cmd_list_tasks(_args):
-    tasks = client("GET", "/cloud_backup")
+    tasks = midclt_call("cloud_backup.query")
     if not tasks:
         print("No TrueCloud Backup tasks configured.")
         return
     print(f"{'ID':>4}  {'Enabled':<8}  {'Provider':<14}  Name")
     print("─" * 60)
     for t in sorted(tasks, key=lambda x: x["id"]):
-        creds = t.get("credentials") or {}
+        ptype = _provider_type(t.get("credentials"))
-        ptype = (creds.get("provider") or {}).get("type", "?")
         enabled = "yes" if t.get("enabled") else "no"
         print(f"{t['id']:>4}  {enabled:<8}  {ptype:<14}  {t.get('description', '')}")
 
 
-def cmd_create(client, args):
+def cmd_create(args):
     parts = args.schedule.split()
     if len(parts) != 5:
         print(
@@ -179,7 +234,18 @@ def cmd_create(client, args):
         "enabled": not args.disabled,
     }
 
-    result = client("POST", "/cloud_backup", body)
+    if args.cache_path:
+        body["cache_path"] = args.cache_path
+    else:
+        print(
+            "WARNING: no --cache-path given. TrueNAS will run restic with --no-cache, "
+            "which is very slow for large repositories (it re-reads all repo metadata "
+            "from the provider every run). Set --cache-path to a writable dir on a pool "
+            "with free space.",
+            file=sys.stderr,
+        )
+
+    result = midclt_call("cloud_backup.create", body)
     try:
         print(f"Created task id={result['id']}  name={result['description']!r}")
     except (KeyError, TypeError):
@@ -194,14 +260,13 @@ def main():
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog=__doc__.split("Examples")[1] if __doc__ and "Examples" in __doc__ else "",
     )
-    p.add_argument("--host", default=None, metavar="HOST",
+    p.add_argument("--version", "-V", action="version", version=f"truecloud-patch {__version__}")
-                   help="TrueNAS hostname or IP address (required except for verify)")
+    # Deprecated & ignored: the tool now uses the local middleware via `midclt` (the
-    p.add_argument("--api-key", default=None, metavar="KEY",
+    # /api/v2.0 REST API is removed in TrueNAS 26.04), so it must run ON the TrueNAS
-                   help="TrueNAS API key — System → API Keys (required except for verify)")
+    # host and needs no host/API key. Kept accepted-but-ignored for compatibility.
-    p.add_argument("--insecure", action="store_true",
+    p.add_argument("--host", default=None, help=argparse.SUPPRESS)
-                   help="Skip TLS certificate verification (self-signed certs). "
+    p.add_argument("--api-key", default=None, help=argparse.SUPPRESS)
-                        "WARNING: exposes your API key to network interception. "
+    p.add_argument("--insecure", action="store_true", help=argparse.SUPPRESS)
-                        "Prefer adding your cert to the trust store instead.")
 
     sub = p.add_subparsers(dest="cmd", required=True)
 
@@ -226,6 +291,11 @@ def main():
                    help="Snapshots to retain after each run (default: 14)")
     c.add_argument("--schedule",    default="0 2 * * *",
                    help="Cron schedule (default: '0 2 * * *' — daily at 02:00)")
+    c.add_argument("--cache-path",  default="", metavar="PATH",
+                   help="restic cache directory (e.g. /mnt/pool/.restic-cache). "
+                        "STRONGLY recommended: without it TrueNAS runs restic with "
+                        "--no-cache, which re-fetches all repo metadata from the "
+                        "provider every run and is extremely slow on large repos.")
     c.add_argument("--transfer-setting",
                    choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"],
                    default="DEFAULT",
@@ -243,16 +313,17 @@ def main():
         cmd_verify()
         return
 
-    if not args.host or not args.api_key:
+    if args.host or args.api_key or args.insecure:
-        p.error("--host and --api-key are required for this command")
+        print("NOTE: --host/--api-key/--insecure are deprecated and ignored; this tool "
+              "now uses the local middleware (midclt) and must run on the TrueNAS host.",
+              file=sys.stderr)
 
-    client = make_client(args.host, args.api_key, args.insecure)
     if args.cmd == "list-credentials":
-        cmd_list_credentials(client, args)
+        cmd_list_credentials(args)
     elif args.cmd == "list-tasks":
-        cmd_list_tasks(client, args)
+        cmd_list_tasks(args)
     elif args.cmd == "create":
-        cmd_create(client, args)
+        cmd_create(args)
 
 
 if __name__ == "__main__":

patch/wait_restart.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# patch/wait_restart.sh — payload of the transient `truecloud-mw-restart`
+# unit that apply.sh schedules in boot context (Step 3).
+#
+# Why not restart middlewared directly from the unit: systemd ordering
+# (`After=multi-user.target`, used up to v0.0.4) cannot see middlewared's
+# *internal* boot work. When the boot targets are reached, two things are
+# typically still in flight inside middlewared:
+#
+#   - ix-reporting.service's `midclt call reporting.start_service` (netdata,
+#     which feeds the dashboard hardware stats), and
+#   - the docker/apps startup task middlewared creates on its own
+#     system-ready event (`docker.state.start_service`).
+#
+# Restarting middlewared while those run kills them, and nothing retries
+# them until the next boot: every app stays down (`docker.status` FAILED),
+# the dashboard shows no stats, and middleware-internal service state (e.g.
+# the SMB backend) is left uninitialized. Observed on 25.10.4 with v0.0.4.
+#
+# So this script waits for both layers to settle before restarting. Every
+# wait is bounded and fails open: worst case the restart still happens, just
+# later — a restart on a settled system is harmless (docker, apps and
+# netdata are independent processes; only the middleware API blips).
+#
+# NOTE: the unit must NOT be Type=oneshot. A oneshot's start job stays in
+# the systemd job queue until the process exits, and `is-system-running
+# --wait` below waits for that same queue to drain — the unit would deadlock
+# on itself until the timeout. apply.sh schedules this with the default
+# service type, whose start job completes at fork.
+
+# 1. systemd layer: wait for the boot job queue to drain. This covers every
+#    ix-* oneshot still activating, including ix-reporting's in-flight midclt
+#    call. The exit code is irrelevant — a "degraded" boot (any unrelated
+#    failed unit) is still a finished boot. The timeout only guards against
+#    a boot that never settles (e.g. a unit stuck on a network wait).
+timeout 900 systemctl is-system-running --wait > /dev/null 2>&1
+
+# 2. middlewared layer: poll the docker state machine until it leaves the
+#    transitional states (PENDING/INITIALIZING/STOPPING/MIGRATING — see
+#    middlewared/plugins/docker/state_utils.py). An empty answer means
+#    midclt could not respond at all; keep waiting. Cap at 10 minutes.
+for _ in $(seq 1 120); do
+    _status=$(midclt call docker.status 2>/dev/null \
+                  | grep -oE '"status": "[A-Z_]+"' | cut -d'"' -f4)
+    case "$_status" in
+        RUNNING|STOPPED|UNCONFIGURED|FAILED|MIGRATION_FAILED) break ;;
+    esac
+    sleep 5
+done
+
+# 3. Grace period for middleware-internal ready-event tasks that expose no
+#    queryable state (smb.configure and friends). Bounded insurance.
+sleep 30
+
+exec systemctl try-restart middlewared

recover.sh

@@ -15,9 +15,15 @@
 # To re-enable the patch after investigating:
 #   rm /mnt/tank/truenas-truecloud-patch/disabled
 #   bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
+#   systemctl restart middlewared
+
+VERSION="0.0.4"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 
+echo "=== TrueNAS TrueCloud Provider Patch v${VERSION} — Recover ==="
+echo ""
+
 if [ "$(id -u)" -ne 0 ]; then
     echo "ERROR: must be run as root." >&2
     exit 1
@@ -46,6 +52,10 @@ for _tag in mw ui; do
 done
 [ "$_any" -eq 0 ] && echo "  No overlays active."
 
+# Cancel a deferred boot restart if one is still queued — we restart ourselves.
+systemctl stop truecloud-mw-restart.service 2>/dev/null
+systemctl reset-failed truecloud-mw-restart.service 2>/dev/null
+
 echo "Restarting middlewared ..."
 if systemctl restart middlewared; then
     echo ""
@@ -63,3 +73,4 @@ echo ""
 echo "To re-enable the patch once you have investigated:"
 echo "  rm $PATCH_DIR/disabled"
 echo "  bash $PATCH_DIR/patch/apply.sh"
+echo "  systemctl restart middlewared"

uninstall.sh

@@ -3,10 +3,12 @@
 
 set -euo pipefail
 
+VERSION="0.0.4"
+
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
 
-echo "=== TrueNAS TrueCloud Provider Patch — Uninstall ==="
+echo "=== TrueNAS TrueCloud Provider Patch v${VERSION} — Uninstall ==="
 echo ""
 
 if [ "$(id -u)" -ne 0 ]; then
@@ -97,6 +99,10 @@ if [ "$_restore_failed" -eq 1 ]; then
     exit 1
 fi
 
+# Cancel a deferred boot restart if one is still queued — we restart ourselves.
+systemctl stop truecloud-mw-restart.service 2>/dev/null || true
+systemctl reset-failed truecloud-mw-restart.service 2>/dev/null || true
+
 echo "Restarting middlewared ..."
 if systemctl restart middlewared; then
     echo ""