Re-apply and verify the patch before the deferred restart

Patching at PREINIT and restarting minutes later is only sound while the
patched files are still on the live path when middlewared re-imports them,
and PREINIT cannot guarantee that. The overlay sits inside /usr, so anything
that remounts that hierarchy detaches it — a systemd-sysext merge/refresh
from another PREINIT hook, or middlewared's own docker.configure_nvidia at
runtime. Init scripts run sequentially in id order, so a hook registered
after this one always wins, and reordering them would not help because
docker.configure_nvidia fires long after PREINIT is done.

Observed on 25.10.6: the overlay was mounted at 16:41:56, a sysext refresh
unmerged and remerged /usr four seconds later, and the deferred restart at
16:47:24 loaded stock modules. Every B2 cloud_backup task then failed with
NotImplementedError for nineteen hours across four scheduled runs while
apply.log and hook_status.json both reported the patch active.

wait_restart.sh now re-applies immediately before restarting — after boot has
settled, which is also after every sysext merge and docker nvidia
configuration — verifies the marker is on the live path, restarts, and
verifies again, retrying once. It is no longer exec'd, so something can run
after the restart to find out what it loaded. apply.sh records the resolved
middlewared directory in .mw_dir for that check, and honours TRUECLOUD_REAPPLY
so the re-apply pass does not schedule a second restart.

_ensure_writable treated "one of our overlays is listed here" as "already
done", but it only reaches that check when the directory is not writable, and
a live overlay of ours always is — a shadowed overlay was indistinguishable
from a healthy one. It is now detached and re-mounted, reusing the upperdir so
files patched earlier in the boot survive, with a fresh workdir and a retry on
a private one, since overlayfs refuses a workdir a detached mount still holds.

Add a CRITICAL hourly alert for the case none of this can prevent: the patch
being on disk but not in the running process. apply.log can only report the
first. The alert asks the second question from inside middlewared, where the
patch's own stamps make it exact, and checks both halves since either can go
missing alone. It stays quiet when the kill switch is set or the providers
module has been retired as native, and is not muted by update_alerts_disabled.

wait_restart.sh also logs to apply.log now: journald retention on a busy box
is easily shorter than the interval between reboots, and the boot that caused
this had already rotated away by the time it was investigated.

patch/wait_restart.sh

@@ -27,6 +27,50 @@
 # --wait` below waits for that same queue to drain — the unit would deadlock
 # on itself until the timeout. apply.sh schedules this with the default
 # service type, whose start job completes at fork.
+#
+# THE RE-APPLY PASS (added 2026-08-26). Applying the patch at PREINIT and
+# restarting later is only sound if the patched files are still on the live
+# path at the moment middlewared re-imports them. They may not be: our patch
+# lives in an overlay mounted *inside* /usr, and anything that remounts the
+# hierarchy above it detaches or buries that overlay. Two things on a normal
+# TrueNAS box do exactly that, both AFTER our PREINIT hook has run:
+#
+#   - `systemd-sysext merge/refresh` over /usr (an nvidia sysext, for
+#     instance) — `Unmerged '/usr'` then `Merged extensions into '/usr'`;
+#   - middlewared's own `docker.configure_nvidia`, which merges the stock
+#     nvidia sysext over /usr when it brings docker up.
+#
+# PREINIT scripts run sequentially in id order, so a hook registered after
+# ours always wins the race, silently. Observed 2026-08-19: our overlay was
+# mounted at 16:41:56 and a sysext refresh tore /usr down four seconds later;
+# the restart at 16:47:24 then loaded stock modules and every B2 cloud_backup
+# job failed for the next nineteen hours while apply.log said "OK".
+#
+# Ordering the hooks cannot fix this — docker.configure_nvidia re-merges at
+# runtime, long after every PREINIT hook is done. So instead of trusting the
+# PREINIT pass, re-apply immediately before the restart (apply.sh is
+# idempotent and re-mounts a lost overlay, keeping the same upperdir so
+# already-patched files survive), verify the marker is really on the live
+# path, and verify again afterwards.
+
+PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+LOG="$PATCH_DIR/apply.log"
+
+_log() { echo "[wait_restart] $*" >> "$LOG" 2>/dev/null; }
+
+# Is the providers patch visible on the live filesystem path -- i.e. would a
+# middlewared starting right now import it? Reads the marker apply.sh leaves
+# in restic.py. Returns 0 when patched, 1 when stock, 2 when we cannot tell
+# (no recorded middlewared dir yet, or the file is gone).
+_patch_visible() {
+    local mw_dir restic_py
+    mw_dir=$(cat "$PATCH_DIR/.mw_dir" 2>/dev/null)
+    [ -n "$mw_dir" ] || return 2
+    restic_py="$mw_dir/plugins/cloud_backup/restic.py"
+    [ -f "$restic_py" ] || return 2
+    grep -q "TRUECLOUD_PATCH" "$restic_py" 2>/dev/null && return 0
+    return 1
+}
 
 # 1. systemd layer: wait for the boot job queue to drain. This covers every
 #    ix-* oneshot still activating, including ix-reporting's in-flight midclt
@@ -39,6 +83,9 @@ timeout 900 systemctl is-system-running --wait > /dev/null 2>&1
 #    transitional states (PENDING/INITIALIZING/STOPPING/MIGRATING — see
 #    middlewared/plugins/docker/state_utils.py). An empty answer means
 #    midclt could not respond at all; keep waiting. Cap at 10 minutes.
+#    This also covers docker.configure_nvidia, the runtime /usr re-merge:
+#    waiting for docker to reach a terminal state means the merge that would
+#    bury our overlay has already happened by the time we re-apply below.
 for _ in $(seq 1 120); do
     _status=$(midclt call docker.status 2>/dev/null \
                   | grep -oE '"status": "[A-Z_]+"' | cut -d'"' -f4)
@@ -52,4 +99,45 @@ done
 #    queryable state (smb.configure and friends). Bounded insurance.
 sleep 30
 
-exec systemctl try-restart middlewared
+# 4. Re-apply pass. Boot has settled, so every sysext merge and docker nvidia
+#    configuration that could bury our overlay is behind us. Re-running
+#    apply.sh is cheap and idempotent: it re-mounts the overlay if it was
+#    detached (same upperdir, so files patched at PREINIT reappear intact)
+#    and re-patches anything that reverted to stock.
+_patch_visible
+case $? in
+    0) _log "providers patch still visible on the live path before restart" ;;
+    1) _log "PATCH LOST since PREINIT (something remounted /usr) — re-applying" ;;
+    *) _log "cannot confirm patch state before restart — re-applying anyway" ;;
+esac
+
+TRUECLOUD_REAPPLY=1 /bin/bash "$PATCH_DIR/patch/apply.sh"
+
+if ! _patch_visible; then
+    _log "WARNING: patch is STILL not on the live path after the re-apply pass;"
+    _log "WARNING: restarting anyway, but middlewared will load stock modules."
+fi
+
+# 5. The restart itself.
+systemctl try-restart middlewared
+
+# 6. Verify what the restart actually loaded, and retry once if the patch was
+#    torn off in the window between the re-apply and the restart. A silent
+#    "on disk but never loaded" is the exact failure this whole script exists
+#    to prevent, so it must never pass unreported.
+if _patch_visible; then
+    _log "OK: providers patch present on the live path across the restart"
+else
+    _log "patch missing again after the restart — one more re-apply and restart"
+    TRUECLOUD_REAPPLY=1 /bin/bash "$PATCH_DIR/patch/apply.sh"
+    systemctl try-restart middlewared
+    if _patch_visible; then
+        _log "OK: providers patch loaded after the second attempt"
+    else
+        _log "ERROR: the patch could not be kept on the live path. TrueNAS is"
+        _log "ERROR: running STOCK cloud_backup — B2/S3 backup tasks will fail."
+        _log "ERROR: middlewared raises the 'not loaded' alert for this."
+    fi
+fi
+
+_log "=== deferred restart complete ==="