Fix deferred restart racing boot: wait for boot to settle before restarting middlewared

The truecloud-mw-restart unit relied on After=multi-user.target /
After=ix-postinit.service, but systemd ordering cannot see middlewared's
internal boot work. On 25.10.4 the restart fired two seconds into
ix-reporting's reporting.start_service call and before the docker/apps
startup task ran, killing both for the whole boot: all apps down
(docker.status FAILED), no dashboard stats, SMB backend uninitialized.

The unit now runs patch/wait_restart.sh: drain the systemd boot job
queue (is-system-running --wait), poll docker.status until the state
machine leaves its transitional states, short grace period, then
try-restart. No Type=oneshot — a oneshot's start job sits in the very
queue the script waits on and would deadlock on itself. All waits are
bounded and fail open.

CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## v0.2.1 — 2026-07-09
+
+### Fixed
+
+- **Deferred restart raced the rest of boot, leaving all apps and dashboard
+  stats down.** The `truecloud-mw-restart` unit introduced in v0.0.4 relied
+  on systemd ordering (`After=multi-user.target`, `After=ix-postinit.service`),
+  which cannot see middlewared's *internal* boot work. Observed on 25.10.4:
+  the restart fired two seconds into `ix-reporting.service`'s
+  `midclt call reporting.start_service` and before the docker/apps startup
+  task (created on middlewared's system-ready event) had run. Both were
+  killed, and nothing retries them until the next boot — every app stayed
+  down (`docker.status` FAILED, the apps dataset never mounted), netdata
+  never started (no dashboard hardware stats), and the SMB middleware
+  backend was left uninitialized.
+
+  The transient unit now runs `patch/wait_restart.sh` instead of restarting
+  directly: it waits for the systemd boot job queue to drain
+  (`systemctl is-system-running --wait`, covering in-flight `ix-*` oneshots
+  such as ix-reporting), then polls `midclt call docker.status` until the
+  docker state machine leaves its transitional states, then allows a short
+  grace period for middleware-internal tasks with no queryable state before
+  issuing `systemctl try-restart middlewared`. The unit no longer sets
+  `Type=oneshot` — a oneshot's start job stays in the very queue the script
+  waits on and would deadlock on itself. All waits are bounded and fail
+  open: worst case the restart still happens, just later.
+
+  Recovery on a boot that already hit this (without rebooting):
+  `midclt call reporting.start_service` and
+  `midclt call docker.state.start_service true`.
+
 ## v0.2.0 — 2026-07-08
 
 ### Changed

README.md

@@ -66,9 +66,11 @@ directly.
 
 PREINIT scripts are executed *by* middlewared, which by then has already
 imported the stock modules — so after patching, `apply.sh` schedules a single
-detached middlewared restart (transient systemd unit `truecloud-mw-restart`,
-ordered after `multi-user.target`) that loads the patched modules once boot
-completes. Expect one middlewared restart shortly after every boot; the UI
+detached middlewared restart (transient systemd unit `truecloud-mw-restart`
+running `patch/wait_restart.sh`) that loads the patched modules once boot has
+*actually* settled: the script waits for the systemd boot job queue to drain
+and for the docker/apps state machine to reach a terminal state before
+restarting. Expect one middlewared restart shortly after every boot; the UI
 and API are briefly unavailable while it happens, and running services are
 not affected.
 
@@ -113,10 +115,15 @@ Two different things must survive two different events:
    imported the stock modules in step 1 and never re-imports, so the on-disk
    patch alone is not enough. `apply.sh` detects it was invoked by middlewared
    and creates a transient systemd unit (`truecloud-mw-restart`, via
-   `systemd-run --no-block`, ordered after `multi-user.target`) — detached and
-   deferred so it cannot disrupt the remainder of the boot sequence.
-5. **Once boot completes, middlewared restarts once** and imports the patched
-   modules from the overlay. S3/B2 backup support is now active until the next
+   `systemd-run --no-block`) running `patch/wait_restart.sh` — detached so it
+   cannot disrupt the remainder of the boot sequence.
+5. **Once boot has settled, middlewared restarts once** and imports the
+   patched modules from the overlay. `wait_restart.sh` holds the restart until
+   the systemd boot job queue has drained (so in-flight `ix-*` units like
+   `ix-reporting` finish first) *and* middlewared's docker/apps startup has
+   reached a terminal state — plain unit ordering cannot see either, and
+   restarting middlewared while they run kills apps and dashboard reporting
+   for the whole boot. S3/B2 backup support is then active until the next
    reboot, when the cycle repeats.
 
 What you will observe: one middlewared restart shortly after every boot (a

patch/apply.sh

@@ -24,7 +24,7 @@
 # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
 PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 LOG="$PATCH_DIR/apply.log"
-VERSION="0.0.4"
+VERSION="0.2.1"
 
 # Rotate log at 512 KB to avoid unbounded growth on a system volume.
 # Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -329,6 +329,14 @@ fi
 # ix-* boot units still need midclt to answer.
 # Boot context is detected by the parent process being middlewared; manual
 # runs (install.sh, recovery) never trigger a restart.
+#
+# The unit runs wait_restart.sh, which blocks until boot has actually
+# settled (systemd job queue drained, docker/apps state terminal) before
+# restarting. systemd ordering alone (After=multi-user.target, ≤ v0.0.4)
+# fired while ix-reporting and the docker/apps startup were still in flight
+# and killed both — apps and dashboard stats stayed down until the next
+# boot. No Type=oneshot: a oneshot's start job would hold the boot queue
+# open against the `is-system-running --wait` inside the script.
 
 echo "--- deferred restart ---"
 
@@ -340,12 +348,10 @@ else
     # A failed unit from an earlier attempt this boot would block systemd-run.
     systemctl reset-failed truecloud-mw-restart.service 2>/dev/null
     if systemd-run --no-block --collect --unit=truecloud-mw-restart \
-           --property=Type=oneshot \
-           --property=After=multi-user.target \
-           --property=After=ix-postinit.service \
-           systemctl try-restart middlewared; then
+           /bin/bash "$PATCH_DIR/patch/wait_restart.sh"; then
         echo "OK: Scheduled deferred middlewared restart (unit: truecloud-mw-restart)."
-        echo "    Backend patch becomes active once boot completes."
+        echo "    It waits for boot to fully settle (apps started, reporting up),"
+        echo "    then restarts middlewared so the backend patch actually loads."
     else
         echo "WARNING: Could not schedule deferred restart — backend patch is on disk but NOT loaded."
         echo "  Activate manually: systemctl restart middlewared"

patch/wait_restart.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# patch/wait_restart.sh — payload of the transient `truecloud-mw-restart`
+# unit that apply.sh schedules in boot context (Step 3).
+#
+# Why not restart middlewared directly from the unit: systemd ordering
+# (`After=multi-user.target`, used up to v0.0.4) cannot see middlewared's
+# *internal* boot work. When the boot targets are reached, two things are
+# typically still in flight inside middlewared:
+#
+#   - ix-reporting.service's `midclt call reporting.start_service` (netdata,
+#     which feeds the dashboard hardware stats), and
+#   - the docker/apps startup task middlewared creates on its own
+#     system-ready event (`docker.state.start_service`).
+#
+# Restarting middlewared while those run kills them, and nothing retries
+# them until the next boot: every app stays down (`docker.status` FAILED),
+# the dashboard shows no stats, and middleware-internal service state (e.g.
+# the SMB backend) is left uninitialized. Observed on 25.10.4 with v0.0.4.
+#
+# So this script waits for both layers to settle before restarting. Every
+# wait is bounded and fails open: worst case the restart still happens, just
+# later — a restart on a settled system is harmless (docker, apps and
+# netdata are independent processes; only the middleware API blips).
+#
+# NOTE: the unit must NOT be Type=oneshot. A oneshot's start job stays in
+# the systemd job queue until the process exits, and `is-system-running
+# --wait` below waits for that same queue to drain — the unit would deadlock
+# on itself until the timeout. apply.sh schedules this with the default
+# service type, whose start job completes at fork.
+
+# 1. systemd layer: wait for the boot job queue to drain. This covers every
+#    ix-* oneshot still activating, including ix-reporting's in-flight midclt
+#    call. The exit code is irrelevant — a "degraded" boot (any unrelated
+#    failed unit) is still a finished boot. The timeout only guards against
+#    a boot that never settles (e.g. a unit stuck on a network wait).
+timeout 900 systemctl is-system-running --wait > /dev/null 2>&1
+
+# 2. middlewared layer: poll the docker state machine until it leaves the
+#    transitional states (PENDING/INITIALIZING/STOPPING/MIGRATING — see
+#    middlewared/plugins/docker/state_utils.py). An empty answer means
+#    midclt could not respond at all; keep waiting. Cap at 10 minutes.
+for _ in $(seq 1 120); do
+    _status=$(midclt call docker.status 2>/dev/null \
+                  | grep -oE '"status": "[A-Z_]+"' | cut -d'"' -f4)
+    case "$_status" in
+        RUNNING|STOPPED|UNCONFIGURED|FAILED|MIGRATION_FAILED) break ;;
+    esac
+    sleep 5
+done
+
+# 3. Grace period for middleware-internal ready-event tasks that expose no
+#    queryable state (smb.configure and friends). Bounded insurance.
+sleep 30
+
+exec systemctl try-restart middlewared