Re-apply and verify the patch before the deferred restart

Patching at PREINIT and restarting minutes later is only sound while the
patched files are still on the live path when middlewared re-imports them,
and PREINIT cannot guarantee that. The overlay sits inside /usr, so anything
that remounts that hierarchy detaches it — a systemd-sysext merge/refresh
from another PREINIT hook, or middlewared's own docker.configure_nvidia at
runtime. Init scripts run sequentially in id order, so a hook registered
after this one always wins, and reordering them would not help because
docker.configure_nvidia fires long after PREINIT is done.

Observed on 25.10.6: the overlay was mounted at 16:41:56, a sysext refresh
unmerged and remerged /usr four seconds later, and the deferred restart at
16:47:24 loaded stock modules. Every B2 cloud_backup task then failed with
NotImplementedError for nineteen hours across four scheduled runs while
apply.log and hook_status.json both reported the patch active.

wait_restart.sh now re-applies immediately before restarting — after boot has
settled, which is also after every sysext merge and docker nvidia
configuration — verifies the marker is on the live path, restarts, and
verifies again, retrying once. It is no longer exec'd, so something can run
after the restart to find out what it loaded. apply.sh records the resolved
middlewared directory in .mw_dir for that check, and honours TRUECLOUD_REAPPLY
so the re-apply pass does not schedule a second restart.

_ensure_writable treated "one of our overlays is listed here" as "already
done", but it only reaches that check when the directory is not writable, and
a live overlay of ours always is — a shadowed overlay was indistinguishable
from a healthy one. It is now detached and re-mounted, reusing the upperdir so
files patched earlier in the boot survive, with a fresh workdir and a retry on
a private one, since overlayfs refuses a workdir a detached mount still holds.

Add a CRITICAL hourly alert for the case none of this can prevent: the patch
being on disk but not in the running process. apply.log can only report the
first. The alert asks the second question from inside middlewared, where the
patch's own stamps make it exact, and checks both halves since either can go
missing alone. It stays quiet when the kill switch is set or the providers
module has been retired as native, and is not muted by update_alerts_disabled.

wait_restart.sh also logs to apply.log now: journald retention on a busy box
is easily shorter than the interval between reboots, and the boot that caused
this had already rotated away by the time it was investigated.

docs/how-it-works.md

@@ -74,22 +74,46 @@ Two different things must survive two different events:
    and creates a transient systemd unit (`truecloud-mw-restart`, via
    `systemd-run --no-block`) running `patch/wait_restart.sh` — detached so it
    cannot disrupt the remainder of the boot sequence.
-5. **Once boot has settled, middlewared restarts once** and imports the
-   patched modules from the overlay. `wait_restart.sh` holds the restart until
-   the systemd boot job queue has drained (so in-flight `ix-*` units like
-   `ix-reporting` finish first) *and* middlewared's docker/apps startup has
-   reached a terminal state — plain unit ordering cannot see either, and
-   restarting middlewared while they run kills apps and dashboard reporting
+5. **Once boot has settled, the patch is re-applied and middlewared restarts
+   once**, importing the patched modules from the overlay. `wait_restart.sh`
+   holds the restart until the systemd boot job queue has drained (so in-flight
+   `ix-*` units like `ix-reporting` finish first) *and* middlewared's docker/apps
+   startup has reached a terminal state — plain unit ordering cannot see either,
+   and restarting middlewared while they run kills apps and dashboard reporting
    for the whole boot. S3/B2 backup support is then active until the next
    reboot, when the cycle repeats.
 
+   The **re-apply** in that sentence is load-bearing, not a safety blanket. The
+   overlay from step 3 sits *inside* `/usr`, so anything that remounts that
+   hierarchy detaches it, and two ordinary things do exactly that after our hook
+   has finished: another PREINIT script running `systemd-sysext merge`/`refresh`
+   over `/usr` (an out-of-tree nvidia driver, say), and middlewared's own
+   `docker.configure_nvidia` when it brings docker up. Init scripts run
+   sequentially in id order, so a hook registered after ours always wins — and
+   ordering them differently would still not help, because `docker.configure_nvidia`
+   fires at runtime. `wait_restart.sh` therefore re-runs `apply.sh` at the point
+   where boot has settled and every such remount is behind it, re-mounting the
+   overlay if it was torn off (same upper layer, so files patched in step 3
+   reappear intact), then verifies the patch is really on the live path, restarts,
+   and verifies again — retrying once if it was lost in between.
+
+   This is the failure that made it necessary: on 2026-08-19 the overlay was
+   mounted at 16:41:56 and a sysext refresh unmerged and remerged `/usr` four
+   seconds later. The restart at 16:47:24 loaded stock modules, and every B2
+   backup failed for nineteen hours while `apply.log` said `OK` — because
+   `apply.log` can only report what was written to disk, never what the restart
+   imported. That second question is now asked from inside middlewared by an
+   hourly CRITICAL alert (see [Update alerts](../README.md#update-alerts)).
+
 What you will observe: one middlewared restart shortly after every boot (a
 brief web UI/API blip; running services are unaffected). Between steps 3
 and 5 there is a short window — typically well under a minute — where the UI
 already shows S3/B2 (the JS bundle is read from disk per request) but the
 backend is still stock. A backup job that fires inside that window fails once
 with `NotImplementedError` and succeeds on its next run; see
-[Troubleshooting](recovery.md) if it persists beyond boot.
+[Troubleshooting](recovery.md) if it persists beyond boot. If the backend is
+still stock an hour after boot, middlewared raises the "installed but NOT
+loaded" alert rather than leaving you to notice via a failed backup.
 
 Manual runs of `bash patch/apply.sh` never trigger the restart — that only
 happens in boot context. `install.sh` and `recover.sh` perform their own

docs/recovery.md

@@ -120,8 +120,9 @@ If a module shows `[FAIL]`:
 
 The traceback ends in `rclone/base.py` → `raise NotImplementedError` and
 contains no `_tc_` frames: the running middlewared is executing stock code.
-Either the deferred restart never fired, or the patch never landed on disk
-this boot. Diagnose in this order:
+Either the deferred restart never fired, the patch never landed on disk this
+boot, or it landed and was then torn off before the restart. Diagnose in this
+order:
 
 ```bash
 # Did apply.sh run this boot, at which version, and did it schedule the restart?
@@ -130,11 +131,21 @@ tail -40 /mnt/tank/truenas-truecloud-patch/apply.log
 # Full check — compares the running process against the patch timestamp
 python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
 
-# Did the deferred restart unit run, fail, or never get created?
-systemctl status truecloud-mw-restart.service
+# What the deferred restart did -- re-apply, restart, and what it verified.
+# apply.log is the durable record; journald retention on a busy box is often
+# shorter than the gap between reboots, so the journal may have nothing left.
+grep wait_restart /mnt/tank/truenas-truecloud-patch/apply.log | tail -20
 journalctl -u truecloud-mw-restart.service --no-pager | tail -20
+
+# Did something remount /usr and detach the patch overlay?
+systemd-sysext status
+findmnt -o TARGET,SOURCE /usr/lib/python3/dist-packages
 ```
 
+`systemctl status truecloud-mw-restart.service` reporting *"could not be
+found"* is **normal** — the unit is transient and is collected once it exits.
+It is not evidence that the restart was skipped.
+
 - `verify` reports the process started **before** the patch → the restart
   didn't happen. `systemctl restart middlewared` fixes it immediately; the
   journal output above tells you why it was missed.
@@ -145,6 +156,22 @@ journalctl -u truecloud-mw-restart.service --no-pager | tail -20
 - `apply.log` header shows `[v0.0.3]` or older → update:
   `git pull && bash install.sh` (v0.0.4 fixed patches not loading after
   reboot).
+- `apply.log` says the patch applied, but `findmnt` shows no `truecloud-mw`
+  overlay on the dist-packages path → something remounted `/usr` after our
+  PREINIT hook and detached it. `systemd-sysext status` names the culprit if it
+  is a sysext (the `SINCE` column will sit a few seconds *after* the `apply.log`
+  timestamp). Releases from 2026-08-26 on re-apply and verify immediately before
+  the restart, so this should self-heal; if you are seeing it, update first.
+
+**TrueNAS raises "truecloud-patch is installed but NOT loaded"**
+
+The definitive symptom, and it does not depend on a backup failing first: the
+running middlewared has stock cloud_backup modules even though the patch is
+installed and its providers module is meant to be active. `bash install.sh`
+re-applies and restarts. The alert clears within the hour. It is silent when the
+kill switch is set or the providers module has been retired as native, and it is
+deliberately not muted by `update_alerts_disabled` — that silences release
+notifications, not a broken backup path.
 
 **Apply log** (check after each reboot or install):
 ```bash