diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e079f7..218d5ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,36 @@ # Changelog +## v0.2.1 — 2026-07-09 + +### Fixed + +- **Deferred restart raced the rest of boot, leaving all apps and dashboard + stats down.** The `truecloud-mw-restart` unit introduced in v0.0.4 relied + on systemd ordering (`After=multi-user.target`, `After=ix-postinit.service`), + which cannot see middlewared's *internal* boot work. Observed on 25.10.4: + the restart fired two seconds into `ix-reporting.service`'s + `midclt call reporting.start_service` and before the docker/apps startup + task (created on middlewared's system-ready event) had run. Both were + killed, and nothing retries them until the next boot — every app stayed + down (`docker.status` FAILED, the apps dataset never mounted), netdata + never started (no dashboard hardware stats), and the SMB middleware + backend was left uninitialized. + + The transient unit now runs `patch/wait_restart.sh` instead of restarting + directly: it waits for the systemd boot job queue to drain + (`systemctl is-system-running --wait`, covering in-flight `ix-*` oneshots + such as ix-reporting), then polls `midclt call docker.status` until the + docker state machine leaves its transitional states, then allows a short + grace period for middleware-internal tasks with no queryable state before + issuing `systemctl try-restart middlewared`. The unit no longer sets + `Type=oneshot` — a oneshot's start job stays in the very queue the script + waits on and would deadlock on itself. All waits are bounded and fail + open: worst case the restart still happens, just later. + + Recovery on a boot that already hit this (without rebooting): + `midclt call reporting.start_service` and + `midclt call docker.state.start_service true`. + ## v0.2.0 — 2026-07-08 ### Changed diff --git a/README.md b/README.md index fd20e0e..1041705 100644 --- a/README.md +++ b/README.md @@ -66,9 +66,11 @@ directly. PREINIT scripts are executed *by* middlewared, which by then has already imported the stock modules — so after patching, `apply.sh` schedules a single -detached middlewared restart (transient systemd unit `truecloud-mw-restart`, -ordered after `multi-user.target`) that loads the patched modules once boot -completes. Expect one middlewared restart shortly after every boot; the UI +detached middlewared restart (transient systemd unit `truecloud-mw-restart` +running `patch/wait_restart.sh`) that loads the patched modules once boot has +*actually* settled: the script waits for the systemd boot job queue to drain +and for the docker/apps state machine to reach a terminal state before +restarting. Expect one middlewared restart shortly after every boot; the UI and API are briefly unavailable while it happens, and running services are not affected. @@ -113,10 +115,15 @@ Two different things must survive two different events: imported the stock modules in step 1 and never re-imports, so the on-disk patch alone is not enough. `apply.sh` detects it was invoked by middlewared and creates a transient systemd unit (`truecloud-mw-restart`, via - `systemd-run --no-block`, ordered after `multi-user.target`) — detached and - deferred so it cannot disrupt the remainder of the boot sequence. -5. **Once boot completes, middlewared restarts once** and imports the patched - modules from the overlay. S3/B2 backup support is now active until the next + `systemd-run --no-block`) running `patch/wait_restart.sh` — detached so it + cannot disrupt the remainder of the boot sequence. +5. **Once boot has settled, middlewared restarts once** and imports the + patched modules from the overlay. `wait_restart.sh` holds the restart until + the systemd boot job queue has drained (so in-flight `ix-*` units like + `ix-reporting` finish first) *and* middlewared's docker/apps startup has + reached a terminal state — plain unit ordering cannot see either, and + restarting middlewared while they run kills apps and dashboard reporting + for the whole boot. S3/B2 backup support is then active until the next reboot, when the cycle repeats. What you will observe: one middlewared restart shortly after every boot (a diff --git a/patch/apply.sh b/patch/apply.sh index 4540081..e2d52d7 100755 --- a/patch/apply.sh +++ b/patch/apply.sh @@ -24,7 +24,7 @@ # Derive PATCH_DIR from this script's location (parent of the patch/ directory). PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)" LOG="$PATCH_DIR/apply.log" -VERSION="0.0.4" +VERSION="0.2.1" # Rotate log at 512 KB to avoid unbounded growth on a system volume. # Keep two prior generations (.1 and .2) so the last three boots are always available. @@ -329,6 +329,14 @@ fi # ix-* boot units still need midclt to answer. # Boot context is detected by the parent process being middlewared; manual # runs (install.sh, recovery) never trigger a restart. +# +# The unit runs wait_restart.sh, which blocks until boot has actually +# settled (systemd job queue drained, docker/apps state terminal) before +# restarting. systemd ordering alone (After=multi-user.target, ≤ v0.0.4) +# fired while ix-reporting and the docker/apps startup were still in flight +# and killed both — apps and dashboard stats stayed down until the next +# boot. No Type=oneshot: a oneshot's start job would hold the boot queue +# open against the `is-system-running --wait` inside the script. echo "--- deferred restart ---" @@ -340,12 +348,10 @@ else # A failed unit from an earlier attempt this boot would block systemd-run. systemctl reset-failed truecloud-mw-restart.service 2>/dev/null if systemd-run --no-block --collect --unit=truecloud-mw-restart \ - --property=Type=oneshot \ - --property=After=multi-user.target \ - --property=After=ix-postinit.service \ - systemctl try-restart middlewared; then + /bin/bash "$PATCH_DIR/patch/wait_restart.sh"; then echo "OK: Scheduled deferred middlewared restart (unit: truecloud-mw-restart)." - echo " Backend patch becomes active once boot completes." + echo " It waits for boot to fully settle (apps started, reporting up)," + echo " then restarts middlewared so the backend patch actually loads." else echo "WARNING: Could not schedule deferred restart — backend patch is on disk but NOT loaded." echo " Activate manually: systemctl restart middlewared" diff --git a/patch/wait_restart.sh b/patch/wait_restart.sh new file mode 100755 index 0000000..325a3b7 --- /dev/null +++ b/patch/wait_restart.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# patch/wait_restart.sh — payload of the transient `truecloud-mw-restart` +# unit that apply.sh schedules in boot context (Step 3). +# +# Why not restart middlewared directly from the unit: systemd ordering +# (`After=multi-user.target`, used up to v0.0.4) cannot see middlewared's +# *internal* boot work. When the boot targets are reached, two things are +# typically still in flight inside middlewared: +# +# - ix-reporting.service's `midclt call reporting.start_service` (netdata, +# which feeds the dashboard hardware stats), and +# - the docker/apps startup task middlewared creates on its own +# system-ready event (`docker.state.start_service`). +# +# Restarting middlewared while those run kills them, and nothing retries +# them until the next boot: every app stays down (`docker.status` FAILED), +# the dashboard shows no stats, and middleware-internal service state (e.g. +# the SMB backend) is left uninitialized. Observed on 25.10.4 with v0.0.4. +# +# So this script waits for both layers to settle before restarting. Every +# wait is bounded and fails open: worst case the restart still happens, just +# later — a restart on a settled system is harmless (docker, apps and +# netdata are independent processes; only the middleware API blips). +# +# NOTE: the unit must NOT be Type=oneshot. A oneshot's start job stays in +# the systemd job queue until the process exits, and `is-system-running +# --wait` below waits for that same queue to drain — the unit would deadlock +# on itself until the timeout. apply.sh schedules this with the default +# service type, whose start job completes at fork. + +# 1. systemd layer: wait for the boot job queue to drain. This covers every +# ix-* oneshot still activating, including ix-reporting's in-flight midclt +# call. The exit code is irrelevant — a "degraded" boot (any unrelated +# failed unit) is still a finished boot. The timeout only guards against +# a boot that never settles (e.g. a unit stuck on a network wait). +timeout 900 systemctl is-system-running --wait > /dev/null 2>&1 + +# 2. middlewared layer: poll the docker state machine until it leaves the +# transitional states (PENDING/INITIALIZING/STOPPING/MIGRATING — see +# middlewared/plugins/docker/state_utils.py). An empty answer means +# midclt could not respond at all; keep waiting. Cap at 10 minutes. +for _ in $(seq 1 120); do + _status=$(midclt call docker.status 2>/dev/null \ + | grep -oE '"status": "[A-Z_]+"' | cut -d'"' -f4) + case "$_status" in + RUNNING|STOPPED|UNCONFIGURED|FAILED|MIGRATION_FAILED) break ;; + esac + sleep 5 +done + +# 3. Grace period for middleware-internal ready-event tasks that expose no +# queryable state (smb.configure and friends). Bounded insurance. +sleep 30 + +exec systemctl try-restart middlewared