Re-apply and verify the patch before the deferred restart
Patching at PREINIT and restarting minutes later is only sound while the patched files are still on the live path when middlewared re-imports them, and PREINIT cannot guarantee that. The overlay sits inside /usr, so anything that remounts that hierarchy detaches it — a systemd-sysext merge/refresh from another PREINIT hook, or middlewared's own docker.configure_nvidia at runtime. Init scripts run sequentially in id order, so a hook registered after this one always wins, and reordering them would not help because docker.configure_nvidia fires long after PREINIT is done. Observed on 25.10.6: the overlay was mounted at 16:41:56, a sysext refresh unmerged and remerged /usr four seconds later, and the deferred restart at 16:47:24 loaded stock modules. Every B2 cloud_backup task then failed with NotImplementedError for nineteen hours across four scheduled runs while apply.log and hook_status.json both reported the patch active. wait_restart.sh now re-applies immediately before restarting — after boot has settled, which is also after every sysext merge and docker nvidia configuration — verifies the marker is on the live path, restarts, and verifies again, retrying once. It is no longer exec'd, so something can run after the restart to find out what it loaded. apply.sh records the resolved middlewared directory in .mw_dir for that check, and honours TRUECLOUD_REAPPLY so the re-apply pass does not schedule a second restart. _ensure_writable treated "one of our overlays is listed here" as "already done", but it only reaches that check when the directory is not writable, and a live overlay of ours always is — a shadowed overlay was indistinguishable from a healthy one. It is now detached and re-mounted, reusing the upperdir so files patched earlier in the boot survive, with a fresh workdir and a retry on a private one, since overlayfs refuses a workdir a detached mount still holds. Add a CRITICAL hourly alert for the case none of this can prevent: the patch being on disk but not in the running process. apply.log can only report the first. The alert asks the second question from inside middlewared, where the patch's own stamps make it exact, and checks both halves since either can go missing alone. It stays quiet when the kill switch is set or the providers module has been retired as native, and is not muted by update_alerts_disabled. wait_restart.sh also logs to apply.log now: journald retention on a busy box is easily shorter than the interval between reboots, and the boot that caused this had already rotated away by the time it was investigated.
This commit is contained in:
@@ -156,6 +156,17 @@ alert, which both take the newest plain `vX.Y.Z` tag. That is what lets debuggin
|
||||
happen in `-rc` tags instead of in your notification bell — see
|
||||
[Releasing](docs/releasing.md).
|
||||
|
||||
**A second alert reports the patch not being loaded**, and this one you cannot
|
||||
turn off with `--no-update-alerts` — it is CRITICAL, hourly, and it means B2/S3
|
||||
backup tasks are about to fail. Being patched *on disk* and being patched *in the
|
||||
running middlewared* are different facts, and only middlewared can answer the
|
||||
second one: the patch stamps the objects it replaces, so a missing stamp means
|
||||
the process imported stock code. It fires if something detaches the patch overlay
|
||||
(a `systemd-sysext` merge over `/usr`, for instance) and the self-healing re-apply
|
||||
in the deferred restart could not put it back. `bash install.sh` clears it. It
|
||||
stays quiet when the kill switch is set, or when the providers module has been
|
||||
retired because TrueNAS went native.
|
||||
|
||||
The changelog is read from whichever forge `origin` points at, derived from the
|
||||
remote rather than hard-coded. That is not cosmetic: when the changelog cannot be
|
||||
read, the alert deliberately fires **anyway** rather than risk hiding a security
|
||||
|
||||
Reference in New Issue
Block a user