Fix backend patch never loading at boot: schedule deferred middlewared restart

PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.

apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.

create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.

recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.

CHANGELOG.md

@@ -1,5 +1,42 @@
 # Changelog
 
+## v0.0.4 — 2026-07-06
+
+### Fixed
+
+- **Backend patch inactive after every reboot.** PREINIT initshutdownscripts
+  are executed by middlewared itself (`ix-preinit.service` runs
+  `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after
+  `ix-zfs.service` pool import). By the time `apply.sh` patched `b2.py` and
+  `restic.py` in the overlay, the running middlewared had already imported the
+  stock modules and never re-imports — so S3/B2 support silently reverted on
+  every reboot until something restarted middlewared. `install.sh` masked the
+  bug because it restarts middlewared explicitly.
+
+  Fix: when `apply.sh` detects it was invoked by middlewared (boot context),
+  it now schedules a single detached restart via a transient systemd unit
+  (`truecloud-mw-restart`, ordered after `multi-user.target` and
+  `ix-postinit.service`) so the patched modules are loaded once boot settles.
+  The restart is never synchronous — `apply.sh` is a child of middlewared's
+  own job runner, and later `ix-*` boot units still need midclt. Manual runs
+  of `apply.sh` never trigger a restart.
+
+- **`create_task.py verify` false-positive after reboot.** `verify` trusted
+  `hook_status.json`, which only records that the files were patched on disk —
+  not that the running process loaded them. `verify` now also compares the
+  middlewared main-process start time against `patched_at` and reports FAIL
+  (with recovery instructions) when the process predates the patch.
+
+### Changed
+
+- README and script comments no longer claim PREINIT runs "before middlewared
+  starts"; the boot ordering and the deferred restart are now documented.
+- `recover.sh` and `uninstall.sh` cancel a still-queued deferred restart
+  before performing their own, and their re-enable instructions now include
+  the required `systemctl restart middlewared`.
+
+---
+
 ## v0.0.3 — 2026-06-22
 
 ### Fixed

README.md

@@ -54,9 +54,9 @@ see [Native support](#if-truenas-adds-native-support) below.
 
 ## What is actually patched
 
-**Nothing in TrueNAS's persistent database or configuration is modified.**
-On every boot, `patch/apply.sh` runs as a PREINIT script before middlewared
-starts. It mounts a writable
+**Nothing in TrueNAS's persistent database or configuration is modified**
+(other than the boot-hook entry itself). On every boot, `patch/apply.sh` runs
+as a PREINIT script. It mounts a writable
 [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) over the
 relevant directories in `/usr/` (upper layer in `/run` tmpfs), then patches
 `b2.py` and `restic.py` inside that overlay. The overlay is volatile — it
@@ -64,6 +64,14 @@ exists only for the current boot — but the PREINIT script recreates it
 automatically on every subsequent boot. Nothing in `/usr/` is written to
 directly.
 
+PREINIT scripts are executed *by* middlewared, which by then has already
+imported the stock modules — so after patching, `apply.sh` schedules a single
+detached middlewared restart (transient systemd unit `truecloud-mw-restart`,
+ordered after `multi-user.target`) that loads the patched modules once boot
+completes. Expect one middlewared restart shortly after every boot; the UI
+and API are briefly unavailable while it happens, and running services are
+not affected.
+
 | Layer | What changes | Technique |
 |---|---|---|
 | **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | File patch applied inside the overlayfs upper layer |
@@ -86,11 +94,14 @@ support and the reason is logged to `apply.log` in your repo root.
 TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by keeping
 this repository on a **persistent ZFS pool** (your data pool, not `/tmp` or a
 system path) and registering a **PREINIT initshutdownscript** in the TrueNAS
-database. On every boot, `patch/apply.sh` runs before `middlewared` starts. It
-mounts a writable [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html)
-over the relevant directories (upper layer in `/run`, recreated each boot), then
-patches `b2.py` and `restic.py` directly in that overlay and re-patches the UI
-bundle. No extra configuration is needed.
+database — the one piece of state that survives both reboots and OS updates.
+On every boot, `patch/apply.sh` runs (executed by middlewared after pools are
+imported), mounts a writable
+[overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) over the
+relevant directories (upper layer in `/run`, recreated each boot), patches
+`b2.py` and `restic.py` directly in that overlay, re-patches the UI bundle,
+and schedules the one-time deferred middlewared restart that loads the
+patched backend. No extra configuration is needed.
 
 ---
 
@@ -276,6 +287,7 @@ To re-enable the patch once you have investigated:
 ```bash
 rm /mnt/tank/truenas-truecloud-patch/disabled
 bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
+systemctl restart middlewared   # manual apply.sh runs never restart for you
 ```
 
 ---
@@ -339,9 +351,10 @@ cat /mnt/tank/truenas-truecloud-patch/apply.log
 ```bash
 python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
 ```
-Reads `hook_status.json` written by `apply.sh` at boot. Reflects whether the
-overlay patches to `b2.py` and `restic.py` were applied successfully. Does not
-require `--host` or `--api-key`.
+Reads `hook_status.json` written by `apply.sh` at boot **and** checks that the
+running middlewared process started *after* the patches were applied — an
+on-disk patch that middlewared has not loaded yet is reported as FAIL with
+instructions. Does not require `--host` or `--api-key`.
 
 **Middlewared log:**
 ```bash

install.sh

@@ -10,13 +10,15 @@
 #
 # What this does:
 #   1. Registers a PREINIT initshutdownscript so patch/apply.sh re-runs on
-#      every boot before middlewared starts.
+#      every boot. At boot, apply.sh re-patches the overlay and schedules a
+#      one-time deferred middlewared restart to load the patched modules
+#      (PREINIT runs after middlewared starts, so a restart is required).
 #   2. Applies the patches immediately (no reboot required).
 #   3. Restarts middlewared so the backend change takes effect now.
 
 set -euo pipefail
 
-VERSION="0.0.3"
+VERSION="0.0.4"
 
 # The directory containing install.sh is the permanent install location.
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"

patch/apply.sh

@@ -1,15 +1,21 @@
 #!/bin/bash
 # patch/apply.sh — registered as a TrueNAS PREINIT initshutdownscript.
 #
-# Runs on every boot BEFORE middlewared starts, so patches land before
-# the first Python process for middlewared is created.
+# PREINIT scripts are executed BY middlewared itself (ix-preinit.service runs
+# `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after
+# ix-zfs.service pool import). So when this script runs at boot, middlewared
+# is already up and has already imported the stock modules — the on-disk
+# patch alone cannot reach the running process.
 #
 # TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
 #
 #   1. Backend — b2.py and restic.py are patched directly in the overlay.
+#      On a boot run, a single detached middlewared restart is scheduled
+#      (Step 3) so the patched modules actually get loaded.
 #
 #   2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
-#                          from Storj-only to include S3 and B2.
+#                          from Storj-only to include S3 and B2. Served from
+#                          disk per request, so no restart is needed for it.
 #
 # Design principle: every step is independently fail-safe.
 # A failed patch logs a warning and continues; middlewared always starts.
@@ -18,7 +24,7 @@
 # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
 PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 LOG="$PATCH_DIR/apply.log"
-VERSION="0.0.3"
+VERSION="0.0.4"
 
 # Rotate log at 512 KB to avoid unbounded growth on a system volume.
 # Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -41,7 +47,7 @@ fi
 
 # Mounts a writable overlayfs on $1 using /run (tmpfs) for the upper/work dirs
 # when the directory is read-only.  The overlay is volatile per boot; this
-# PREINIT script recreates it on every boot before middlewared starts.
+# PREINIT script recreates it on every boot.
 # Returns 0 if the directory is now writable, 1 if it could not be made so.
 _ensure_writable() {
     local dir="$1" tag="$2"
@@ -310,6 +316,37 @@ fi
 
 "$PYTHON" "$PATCH_DIR/patch/patch_ui.py" || echo "WARNING: patch_ui.py exited non-zero; UI dropdown may still show Storj only."
 
+# ── Step 3: deferred middlewared restart (boot runs only) ─────────────────────
+# At boot this script is spawned by middlewared, which already imported the
+# stock modules — the backend patch is on disk but not in the process. Schedule
+# ONE detached restart for after boot settles. Never restart synchronously
+# here: this script is a child of middlewared's own job runner, and the later
+# ix-* boot units still need midclt to answer.
+# Boot context is detected by the parent process being middlewared; manual
+# runs (install.sh, recovery) never trigger a restart.
+
+echo "--- deferred restart ---"
+
+if ! grep -aq middlewared "/proc/$PPID/cmdline" 2>/dev/null; then
+    echo "Manual run (parent is not middlewared) — no restart scheduled."
+elif [ "$_b2_ok" != "1" ] || [ "$_restic_ok" != "1" ]; then
+    echo "Backend patch incomplete — no restart scheduled (nothing new to load)."
+else
+    # A failed unit from an earlier attempt this boot would block systemd-run.
+    systemctl reset-failed truecloud-mw-restart.service 2>/dev/null
+    if systemd-run --no-block --collect --unit=truecloud-mw-restart \
+           --property=Type=oneshot \
+           --property=After=multi-user.target \
+           --property=After=ix-postinit.service \
+           systemctl try-restart middlewared; then
+        echo "OK: Scheduled deferred middlewared restart (unit: truecloud-mw-restart)."
+        echo "    Backend patch becomes active once boot completes."
+    else
+        echo "WARNING: Could not schedule deferred restart — backend patch is on disk but NOT loaded."
+        echo "  Activate manually: systemctl restart middlewared"
+    fi
+fi
+
 # ── Done ──────────────────────────────────────────────────────────────────────
 
 echo "=== done ==="

patch/create_task.py

@@ -41,14 +41,17 @@ List existing TrueCloud Backup tasks:
 """
 
 import argparse
+import calendar
 import json
 import os
 import ssl
+import subprocess
 import sys
+import time
 import urllib.error
 import urllib.request
 
-__version__ = "0.0.3"
+__version__ = "0.0.4"
 
 _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")
@@ -86,6 +89,31 @@ def make_client(host, api_key, insecure=False):
 
 # ── Sub-commands ──────────────────────────────────────────────────────────────
 
+def _middlewared_start_epoch():
+    """Epoch timestamp of the running middlewared main process, or None."""
+    try:
+        pid = int(subprocess.run(
+            ["systemctl", "show", "--property=MainPID", "--value", "middlewared"],
+            capture_output=True, text=True, timeout=10, check=True,
+        ).stdout.strip())
+        if pid <= 0:
+            return None
+        with open(f"/proc/{pid}/stat", encoding="ascii", errors="replace") as fh:
+            stat = fh.read()
+        # Field 22 (starttime, in clock ticks since boot); the comm field may
+        # contain spaces, so split after the closing paren.
+        start_ticks = float(stat.rsplit(")", 1)[1].split()[19])
+        # Base on /proc/stat btime, not uptime: starttime ticks count from the
+        # kernel boot, which uptime does not match inside containers.
+        with open("/proc/stat", encoding="ascii") as fh:
+            btime = next(float(line.split()[1]) for line in fh
+                         if line.startswith("btime "))
+        return btime + start_ticks / os.sysconf("SC_CLK_TCK")
+    except (OSError, ValueError, IndexError, StopIteration,
+            subprocess.SubprocessError):
+        return None
+
+
 def cmd_verify():
     """Print the hook status written by apply.sh at boot."""
     if not os.path.exists(_STATUS_FILE):
@@ -115,9 +143,35 @@ def cmd_verify():
         if not ok:
             all_ok = False
 
+    # The disk status alone can false-positive: at boot the files are patched
+    # while middlewared is already running with the stock modules imported.
+    # The running process only has the patch if it started AFTER patched_at.
+    try:
+        patched_epoch = calendar.timegm(
+            time.strptime(status.get("patched_at", ""), "%Y-%m-%dT%H:%M:%SZ"))
+    except ValueError:
+        patched_epoch = None
+    mw_start = _middlewared_start_epoch()
+
+    proc_stale = False
+    if patched_epoch is None or mw_start is None:
+        print("  [??  ] running middlewared process — could not compare start time;")
+        print("         the results above reflect the on-disk state only")
+    elif mw_start + 2 < patched_epoch:
+        proc_stale = True
+        print("  [FAIL] running middlewared process — started BEFORE the patch was applied,")
+        print("         so it is running the stock (unpatched) modules")
+    else:
+        print("  [OK  ] running middlewared process — started after the patch was applied")
+
     print()
-    if all_ok:
+    if all_ok and not proc_stale:
         print("All patches installed. Run a test backup to confirm end-to-end.")
+    elif all_ok:
+        print("The patch is on disk but not loaded. Right after boot, the deferred")
+        print("restart (unit truecloud-mw-restart) may still be pending — re-check in a")
+        print("minute. Otherwise run: systemctl restart middlewared")
+        sys.exit(1)
     else:
         print("One or more patches failed to apply.")
         print(f"Check {os.path.join(_PATCH_DIR, 'apply.log')} and journalctl -u middlewared")

recover.sh

@@ -15,8 +15,9 @@
 # To re-enable the patch after investigating:
 #   rm /mnt/tank/truenas-truecloud-patch/disabled
 #   bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
+#   systemctl restart middlewared
 
-VERSION="0.0.3"
+VERSION="0.0.4"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 
@@ -51,6 +52,10 @@ for _tag in mw ui; do
 done
 [ "$_any" -eq 0 ] && echo "  No overlays active."
 
+# Cancel a deferred boot restart if one is still queued — we restart ourselves.
+systemctl stop truecloud-mw-restart.service 2>/dev/null
+systemctl reset-failed truecloud-mw-restart.service 2>/dev/null
+
 echo "Restarting middlewared ..."
 if systemctl restart middlewared; then
     echo ""
@@ -68,3 +73,4 @@ echo ""
 echo "To re-enable the patch once you have investigated:"
 echo "  rm $PATCH_DIR/disabled"
 echo "  bash $PATCH_DIR/patch/apply.sh"
+echo "  systemctl restart middlewared"

uninstall.sh

@@ -3,7 +3,7 @@
 
 set -euo pipefail
 
-VERSION="0.0.3"
+VERSION="0.0.4"
 
 PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
 _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
@@ -99,6 +99,10 @@ if [ "$_restore_failed" -eq 1 ]; then
     exit 1
 fi
 
+# Cancel a deferred boot restart if one is still queued — we restart ourselves.
+systemctl stop truecloud-mw-restart.service 2>/dev/null || true
+systemctl reset-failed truecloud-mw-restart.service 2>/dev/null || true
+
 echo "Restarting middlewared ..."
 if systemctl restart middlewared; then
     echo ""