diff --git a/CHANGELOG.md b/CHANGELOG.md index 012a1eb..872d71e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,42 @@ # Changelog +## v0.0.4 — 2026-07-06 + +### Fixed + +- **Backend patch inactive after every reboot.** PREINIT initshutdownscripts + are executed by middlewared itself (`ix-preinit.service` runs + `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after + `ix-zfs.service` pool import). By the time `apply.sh` patched `b2.py` and + `restic.py` in the overlay, the running middlewared had already imported the + stock modules and never re-imports — so S3/B2 support silently reverted on + every reboot until something restarted middlewared. `install.sh` masked the + bug because it restarts middlewared explicitly. + + Fix: when `apply.sh` detects it was invoked by middlewared (boot context), + it now schedules a single detached restart via a transient systemd unit + (`truecloud-mw-restart`, ordered after `multi-user.target` and + `ix-postinit.service`) so the patched modules are loaded once boot settles. + The restart is never synchronous — `apply.sh` is a child of middlewared's + own job runner, and later `ix-*` boot units still need midclt. Manual runs + of `apply.sh` never trigger a restart. + +- **`create_task.py verify` false-positive after reboot.** `verify` trusted + `hook_status.json`, which only records that the files were patched on disk — + not that the running process loaded them. `verify` now also compares the + middlewared main-process start time against `patched_at` and reports FAIL + (with recovery instructions) when the process predates the patch. + +### Changed + +- README and script comments no longer claim PREINIT runs "before middlewared + starts"; the boot ordering and the deferred restart are now documented. +- `recover.sh` and `uninstall.sh` cancel a still-queued deferred restart + before performing their own, and their re-enable instructions now include + the required `systemctl restart middlewared`. + +--- + ## v0.0.3 — 2026-06-22 ### Fixed diff --git a/README.md b/README.md index 91adb19..72bb753 100644 --- a/README.md +++ b/README.md @@ -54,9 +54,9 @@ see [Native support](#if-truenas-adds-native-support) below. ## What is actually patched -**Nothing in TrueNAS's persistent database or configuration is modified.** -On every boot, `patch/apply.sh` runs as a PREINIT script before middlewared -starts. It mounts a writable +**Nothing in TrueNAS's persistent database or configuration is modified** +(other than the boot-hook entry itself). On every boot, `patch/apply.sh` runs +as a PREINIT script. It mounts a writable [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) over the relevant directories in `/usr/` (upper layer in `/run` tmpfs), then patches `b2.py` and `restic.py` inside that overlay. The overlay is volatile — it @@ -64,6 +64,14 @@ exists only for the current boot — but the PREINIT script recreates it automatically on every subsequent boot. Nothing in `/usr/` is written to directly. +PREINIT scripts are executed *by* middlewared, which by then has already +imported the stock modules — so after patching, `apply.sh` schedules a single +detached middlewared restart (transient systemd unit `truecloud-mw-restart`, +ordered after `multi-user.target`) that loads the patched modules once boot +completes. Expect one middlewared restart shortly after every boot; the UI +and API are briefly unavailable while it happens, and running services are +not affected. + | Layer | What changes | Technique | |---|---|---| | **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | File patch applied inside the overlayfs upper layer | @@ -86,11 +94,14 @@ support and the reason is logged to `apply.log` in your repo root. TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by keeping this repository on a **persistent ZFS pool** (your data pool, not `/tmp` or a system path) and registering a **PREINIT initshutdownscript** in the TrueNAS -database. On every boot, `patch/apply.sh` runs before `middlewared` starts. It -mounts a writable [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) -over the relevant directories (upper layer in `/run`, recreated each boot), then -patches `b2.py` and `restic.py` directly in that overlay and re-patches the UI -bundle. No extra configuration is needed. +database — the one piece of state that survives both reboots and OS updates. +On every boot, `patch/apply.sh` runs (executed by middlewared after pools are +imported), mounts a writable +[overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) over the +relevant directories (upper layer in `/run`, recreated each boot), patches +`b2.py` and `restic.py` directly in that overlay, re-patches the UI bundle, +and schedules the one-time deferred middlewared restart that loads the +patched backend. No extra configuration is needed. --- @@ -276,6 +287,7 @@ To re-enable the patch once you have investigated: ```bash rm /mnt/tank/truenas-truecloud-patch/disabled bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh +systemctl restart middlewared # manual apply.sh runs never restart for you ``` --- @@ -339,9 +351,10 @@ cat /mnt/tank/truenas-truecloud-patch/apply.log ```bash python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify ``` -Reads `hook_status.json` written by `apply.sh` at boot. Reflects whether the -overlay patches to `b2.py` and `restic.py` were applied successfully. Does not -require `--host` or `--api-key`. +Reads `hook_status.json` written by `apply.sh` at boot **and** checks that the +running middlewared process started *after* the patches were applied — an +on-disk patch that middlewared has not loaded yet is reported as FAIL with +instructions. Does not require `--host` or `--api-key`. **Middlewared log:** ```bash diff --git a/install.sh b/install.sh index c591f23..5c635ff 100755 --- a/install.sh +++ b/install.sh @@ -10,13 +10,15 @@ # # What this does: # 1. Registers a PREINIT initshutdownscript so patch/apply.sh re-runs on -# every boot before middlewared starts. +# every boot. At boot, apply.sh re-patches the overlay and schedules a +# one-time deferred middlewared restart to load the patched modules +# (PREINIT runs after middlewared starts, so a restart is required). # 2. Applies the patches immediately (no reboot required). # 3. Restarts middlewared so the backend change takes effect now. set -euo pipefail -VERSION="0.0.3" +VERSION="0.0.4" # The directory containing install.sh is the permanent install location. PATCH_DIR="$(cd "$(dirname "$0")" && pwd)" diff --git a/patch/apply.sh b/patch/apply.sh index e60ca8b..9b0e1e4 100755 --- a/patch/apply.sh +++ b/patch/apply.sh @@ -1,15 +1,21 @@ #!/bin/bash # patch/apply.sh — registered as a TrueNAS PREINIT initshutdownscript. # -# Runs on every boot BEFORE middlewared starts, so patches land before -# the first Python process for middlewared is created. +# PREINIT scripts are executed BY middlewared itself (ix-preinit.service runs +# `midclt call initshutdownscript.execute_init_tasks PREINIT`, ordered after +# ix-zfs.service pool import). So when this script runs at boot, middlewared +# is already up and has already imported the stock modules — the on-disk +# patch alone cannot reach the running process. # # TrueNAS updates replace /usr/ entirely; this script re-applies two patches: # # 1. Backend — b2.py and restic.py are patched directly in the overlay. +# On a boot run, a single detached middlewared restart is scheduled +# (Step 3) so the patched modules actually get loaded. # # 2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown -# from Storj-only to include S3 and B2. +# from Storj-only to include S3 and B2. Served from +# disk per request, so no restart is needed for it. # # Design principle: every step is independently fail-safe. # A failed patch logs a warning and continues; middlewared always starts. @@ -18,7 +24,7 @@ # Derive PATCH_DIR from this script's location (parent of the patch/ directory). PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)" LOG="$PATCH_DIR/apply.log" -VERSION="0.0.3" +VERSION="0.0.4" # Rotate log at 512 KB to avoid unbounded growth on a system volume. # Keep two prior generations (.1 and .2) so the last three boots are always available. @@ -41,7 +47,7 @@ fi # Mounts a writable overlayfs on $1 using /run (tmpfs) for the upper/work dirs # when the directory is read-only. The overlay is volatile per boot; this -# PREINIT script recreates it on every boot before middlewared starts. +# PREINIT script recreates it on every boot. # Returns 0 if the directory is now writable, 1 if it could not be made so. _ensure_writable() { local dir="$1" tag="$2" @@ -310,6 +316,37 @@ fi "$PYTHON" "$PATCH_DIR/patch/patch_ui.py" || echo "WARNING: patch_ui.py exited non-zero; UI dropdown may still show Storj only." +# ── Step 3: deferred middlewared restart (boot runs only) ───────────────────── +# At boot this script is spawned by middlewared, which already imported the +# stock modules — the backend patch is on disk but not in the process. Schedule +# ONE detached restart for after boot settles. Never restart synchronously +# here: this script is a child of middlewared's own job runner, and the later +# ix-* boot units still need midclt to answer. +# Boot context is detected by the parent process being middlewared; manual +# runs (install.sh, recovery) never trigger a restart. + +echo "--- deferred restart ---" + +if ! grep -aq middlewared "/proc/$PPID/cmdline" 2>/dev/null; then + echo "Manual run (parent is not middlewared) — no restart scheduled." +elif [ "$_b2_ok" != "1" ] || [ "$_restic_ok" != "1" ]; then + echo "Backend patch incomplete — no restart scheduled (nothing new to load)." +else + # A failed unit from an earlier attempt this boot would block systemd-run. + systemctl reset-failed truecloud-mw-restart.service 2>/dev/null + if systemd-run --no-block --collect --unit=truecloud-mw-restart \ + --property=Type=oneshot \ + --property=After=multi-user.target \ + --property=After=ix-postinit.service \ + systemctl try-restart middlewared; then + echo "OK: Scheduled deferred middlewared restart (unit: truecloud-mw-restart)." + echo " Backend patch becomes active once boot completes." + else + echo "WARNING: Could not schedule deferred restart — backend patch is on disk but NOT loaded." + echo " Activate manually: systemctl restart middlewared" + fi +fi + # ── Done ────────────────────────────────────────────────────────────────────── echo "=== done ===" diff --git a/patch/create_task.py b/patch/create_task.py index f87341a..d05e135 100755 --- a/patch/create_task.py +++ b/patch/create_task.py @@ -41,14 +41,17 @@ List existing TrueCloud Backup tasks: """ import argparse +import calendar import json import os import ssl +import subprocess import sys +import time import urllib.error import urllib.request -__version__ = "0.0.3" +__version__ = "0.0.4" _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json") @@ -86,6 +89,31 @@ def make_client(host, api_key, insecure=False): # ── Sub-commands ────────────────────────────────────────────────────────────── +def _middlewared_start_epoch(): + """Epoch timestamp of the running middlewared main process, or None.""" + try: + pid = int(subprocess.run( + ["systemctl", "show", "--property=MainPID", "--value", "middlewared"], + capture_output=True, text=True, timeout=10, check=True, + ).stdout.strip()) + if pid <= 0: + return None + with open(f"/proc/{pid}/stat", encoding="ascii", errors="replace") as fh: + stat = fh.read() + # Field 22 (starttime, in clock ticks since boot); the comm field may + # contain spaces, so split after the closing paren. + start_ticks = float(stat.rsplit(")", 1)[1].split()[19]) + # Base on /proc/stat btime, not uptime: starttime ticks count from the + # kernel boot, which uptime does not match inside containers. + with open("/proc/stat", encoding="ascii") as fh: + btime = next(float(line.split()[1]) for line in fh + if line.startswith("btime ")) + return btime + start_ticks / os.sysconf("SC_CLK_TCK") + except (OSError, ValueError, IndexError, StopIteration, + subprocess.SubprocessError): + return None + + def cmd_verify(): """Print the hook status written by apply.sh at boot.""" if not os.path.exists(_STATUS_FILE): @@ -115,9 +143,35 @@ def cmd_verify(): if not ok: all_ok = False + # The disk status alone can false-positive: at boot the files are patched + # while middlewared is already running with the stock modules imported. + # The running process only has the patch if it started AFTER patched_at. + try: + patched_epoch = calendar.timegm( + time.strptime(status.get("patched_at", ""), "%Y-%m-%dT%H:%M:%SZ")) + except ValueError: + patched_epoch = None + mw_start = _middlewared_start_epoch() + + proc_stale = False + if patched_epoch is None or mw_start is None: + print(" [?? ] running middlewared process — could not compare start time;") + print(" the results above reflect the on-disk state only") + elif mw_start + 2 < patched_epoch: + proc_stale = True + print(" [FAIL] running middlewared process — started BEFORE the patch was applied,") + print(" so it is running the stock (unpatched) modules") + else: + print(" [OK ] running middlewared process — started after the patch was applied") + print() - if all_ok: + if all_ok and not proc_stale: print("All patches installed. Run a test backup to confirm end-to-end.") + elif all_ok: + print("The patch is on disk but not loaded. Right after boot, the deferred") + print("restart (unit truecloud-mw-restart) may still be pending — re-check in a") + print("minute. Otherwise run: systemctl restart middlewared") + sys.exit(1) else: print("One or more patches failed to apply.") print(f"Check {os.path.join(_PATCH_DIR, 'apply.log')} and journalctl -u middlewared") diff --git a/recover.sh b/recover.sh index bbf28cd..37cf741 100755 --- a/recover.sh +++ b/recover.sh @@ -15,8 +15,9 @@ # To re-enable the patch after investigating: # rm /mnt/tank/truenas-truecloud-patch/disabled # bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh +# systemctl restart middlewared -VERSION="0.0.3" +VERSION="0.0.4" PATCH_DIR="$(cd "$(dirname "$0")" && pwd)" @@ -51,6 +52,10 @@ for _tag in mw ui; do done [ "$_any" -eq 0 ] && echo " No overlays active." +# Cancel a deferred boot restart if one is still queued — we restart ourselves. +systemctl stop truecloud-mw-restart.service 2>/dev/null +systemctl reset-failed truecloud-mw-restart.service 2>/dev/null + echo "Restarting middlewared ..." if systemctl restart middlewared; then echo "" @@ -68,3 +73,4 @@ echo "" echo "To re-enable the patch once you have investigated:" echo " rm $PATCH_DIR/disabled" echo " bash $PATCH_DIR/patch/apply.sh" +echo " systemctl restart middlewared" diff --git a/uninstall.sh b/uninstall.sh index 54c1a87..34a640e 100755 --- a/uninstall.sh +++ b/uninstall.sh @@ -3,7 +3,7 @@ set -euo pipefail -VERSION="0.0.3" +VERSION="0.0.4" PATCH_DIR="$(cd "$(dirname "$0")" && pwd)" _HOOK_COMMENT='TrueCloud provider patch (S3/B2)' @@ -99,6 +99,10 @@ if [ "$_restore_failed" -eq 1 ]; then exit 1 fi +# Cancel a deferred boot restart if one is still queued — we restart ourselves. +systemctl stop truecloud-mw-restart.service 2>/dev/null || true +systemctl reset-failed truecloud-mw-restart.service 2>/dev/null || true + echo "Restarting middlewared ..." if systemctl restart middlewared; then echo ""