try-restart returns as soon as middlewared is READY, and middlewared then
brings docker up — docker.configure_nvidia merges the stock nvidia sysext
over /usr at that point, detaching the overlay after the patched modules
have already been imported. Checking the disk there reports "missing" on a
perfectly healthy system, and the retry that followed would restart a
correctly-patched middlewared straight back into the same race, then log
ERROR when nothing was wrong.
The disk is the right oracle before the restart and the wrong one after it.
After the restart the overlay is re-mounted so the next restart finds patched
files, and whether this middlewared actually holds the patch is left to the
in-process alert, which is the only thing that can answer it exactly.
Patching at PREINIT and restarting minutes later is only sound while the
patched files are still on the live path when middlewared re-imports them,
and PREINIT cannot guarantee that. The overlay sits inside /usr, so anything
that remounts that hierarchy detaches it — a systemd-sysext merge/refresh
from another PREINIT hook, or middlewared's own docker.configure_nvidia at
runtime. Init scripts run sequentially in id order, so a hook registered
after this one always wins, and reordering them would not help because
docker.configure_nvidia fires long after PREINIT is done.
Observed on 25.10.6: the overlay was mounted at 16:41:56, a sysext refresh
unmerged and remerged /usr four seconds later, and the deferred restart at
16:47:24 loaded stock modules. Every B2 cloud_backup task then failed with
NotImplementedError for nineteen hours across four scheduled runs while
apply.log and hook_status.json both reported the patch active.
wait_restart.sh now re-applies immediately before restarting — after boot has
settled, which is also after every sysext merge and docker nvidia
configuration — verifies the marker is on the live path, restarts, and
verifies again, retrying once. It is no longer exec'd, so something can run
after the restart to find out what it loaded. apply.sh records the resolved
middlewared directory in .mw_dir for that check, and honours TRUECLOUD_REAPPLY
so the re-apply pass does not schedule a second restart.
_ensure_writable treated "one of our overlays is listed here" as "already
done", but it only reaches that check when the directory is not writable, and
a live overlay of ours always is — a shadowed overlay was indistinguishable
from a healthy one. It is now detached and re-mounted, reusing the upperdir so
files patched earlier in the boot survive, with a fresh workdir and a retry on
a private one, since overlayfs refuses a workdir a detached mount still holds.
Add a CRITICAL hourly alert for the case none of this can prevent: the patch
being on disk but not in the running process. apply.log can only report the
first. The alert asks the second question from inside middlewared, where the
patch's own stamps make it exact, and checks both halves since either can go
missing alone. It stays quiet when the kill switch is set or the providers
module has been retired as native, and is not muted by update_alerts_disabled.
wait_restart.sh also logs to apply.log now: journald retention on a busy box
is easily shorter than the interval between reboots, and the boot that caused
this had already rotated away by the time it was investigated.
The Gitea runner image executes jobs as root, and chmod(0) cannot make a file
unreadable for root (CAP_DAC_OVERRIDE) — the two tests failed there while
passing on GitHub's non-root runner. Skipping as root keeps the scenario
exercised everywhere it is constructible.
compat.yml's push paths listed tools/compat.py but not tools/compat_publish.py.
So the commit that taught the bot to refresh a stale report body fired no run, and
the report stayed stale until the next scheduled one -- caught by watching for the
run that never came. A fix nobody runs is a fix nobody has.
Two questions were sharing one answer.
"Have the findings changed?" gates COMMENTS. They notify, and a daily "still
broken, same as yesterday" is what teaches everyone to ignore the one that
finally matters.
"Is the body still true?" gates the BODY. Editing an issue body notifies nobody
on either forge, so keeping it honest is free.
Conflated, an unchanged fingerprint froze the body -- and the fingerprint ignores,
by design, everything that moves on its own: healthy rows, the hardware-verified
column, point releases, and how a row is LABELLED. So the master -> 27-dev relabel
would have shipped to the README and never to the issue anybody actually opens.
The report would have gone on saying "master (unreleased) BROKEN" -- the precise
false alarm the relabel exists to kill -- until iX happened to break something
else.
The body is now rewritten whenever it is stale; comments stay strictly a changelog
of real changes. Bodies are compared after normalising line endings, because a
forge that round-trips \r\n would otherwise trigger a silent rewrite every run and
leave the issue looking freshly touched every morning.
Two ways the matrix misled the person it exists for -- somebody deciding whether
to trust this with their backups.
master is not the next release. iX branches each major onto its own release/ line
and master rolls straight on to the one after: every recent commit on master
targets 27.0.0-BETA.1 while 26 is still in beta. So a BROKEN master row, rendered
"master (unreleased)", read as "the version you are about to install is broken"
when the breakage is a major release away on a line nobody can download. It is now
labelled from the newest major in the matrix plus one, so it becomes 28-dev by
itself once 27 branches.
The break, for the record, is NAS-141498 (2026-06-24), "Convert cloud_backup
plugin to the typesafe pattern": it re-signatures restic_backup and
get_restic_config, splitting entry/credentials out of the cloud_backup dict. Not
being chased while the 27 line churns.
The next maintenance release was never checked -- and it is the one that reaches
users. Shipped came from TS-* tags, unreleased from release/* branches carrying
-BETA/-RC. A branched-but-untagged MAINTENANCE release is neither: release/25.10.5
has no tag, and its line has already shipped, so the "a prerelease of a shipped
line is history" filter threw it out. It was invisible, and it is exactly what a
25.10.4 box gets on its next update; a break in it reaches real users before the
daily check ever looks, on the only line anybody runs.
A plain release/X.Y.Z branch now counts when its line HAS shipped and it sorts
NEWER than that line's newest tag. Both exclusions fall out of the same rule:
release/24.10-RC.2 sorts older than TS-24.10.2.4 (history), and the typo branch
release/25.20.2.2 is on a line with no tag at all (not a release line). This
surfaced two refs never checked before -- release/25.10.5 and release/24.10.2.5 --
both of which pass.
is_unreleased() keys off where a ref came from (a branch is by construction not
shipped) rather than hunting -BETA/-RC in the name. Otherwise release/25.10.5
counts as shipped and a break in it fails the build as a live outage, on a version
nobody is running yet.
`find_issue()` skipped pull requests by testing for the PRESENCE of the
`pull_request` key. GitHub omits that key on a plain issue. Gitea sends it as
`null`. So on Gitea every issue was thrown away as if it were a PR, the lookup
came back empty on every run, and the bot took the "nothing filed yet" branch and
opened a brand-new report instead of editing the one already open.
Nine of them piled up on the canonical forge. Four were filed AFTER the commit
that was supposed to stop exactly this -- the same failure it was written to
prevent, moved from comments to issues. It survived because `find_issue` was the
one function here with no test; the mirror deduped correctly, and the mirror is
what anybody would have looked at.
Test the value, not the key: it is the only form that is true on both forges.
Pinned by tests that carry each forge's payload shape by hand.
Also send both paging parameters. GitHub reads `per_page`, Gitea reads `limit`,
and each ignores the other's; Gitea's default page is 30, so the lookup would
have begun missing the report once the pile it was creating outgrew one page.
The README and how-it-works still said "TrueNAS 26: nested snapshots are not supported
yet" and "the third is not fixed, and is why 26 reports BROKEN". Both shipped in v0.7.0
and are now false — exactly the kind of stale claim that misleads somebody deciding
whether to trust this with their backups.
Adds docs/verification.md: what has ACTUALLY been run, as opposed to what the support
matrix proves. The matrix is static analysis — it shows the patch's assumptions still
hold, which is a strictly weaker claim than "a backup ran and a restore came back". The
new file records the three live tasks exercised on 25.10.4 (nested, nested+zvols,
non-nested), the md5 of the file that came back out of B2, the real orphan the collector
reclaimed from the pool, and what is NOT covered (24.10/25.04 unrun; master broken; no
reboot on v0.7.0). The README now points at it, next to the matrix it qualifies.
The how-it-works TrueNAS 26 section now explains the part that mattered: the public
pool.* queries are not like-for-like replacements for the deleted private zfs.* ones —
they apply a visibility policy hiding 84 of 270 datasets on a real pool, including live
app data — and the rule the module now follows (read the truth from ZFS, make changes
through middleware). Plus the divergence nothing warned about: 26 decides `recursive` by
a different rule than this patch decides `nested`, which orphaned one snapshot per zvol
on every run until ownership of the sweep was made unconditional.
Enumerate datasets and snapshots from ZFS, not middleware's filtered queries; own the
snapshot sweep unconditionally (TrueNAS 26 decides "recursive" by a different rule than
we decide "nested"); resolve the snapshot namespace at runtime by the same predicate
tools/compat.py checks.
Four adversarial audits, every finding fixed and mutation-pinned. Verified on a real
TrueNAS 26.0.0-BETA.1 install: 292-dataset backup, 0 orphans, 0 leaked mounts,
byte-identical restore of a four-level-deep child dataset that middleware's own API
hides.
The test suite is not bloated -- 3,701 lines of test code against 3,760 lines of
product code, one duplicate pair in 356 tests, 8% single-assertion tests. The waste was
not volume, it was four tests that looked like coverage and provided none:
- test_parents_are_mounted_before_children: the fixture was already in depth order, so
the sort it exists for was never exercised. Deleting `mounts.sort(key=_depth)` passed
the whole suite. It now uses datasets whose NAME order differs from their MOUNTPOINT
depth, which is the only case the sort is for.
- test_it_NEVER_touches_the_current_run: the "current" snapshot was a minute old, so the
age floor excluded it regardless and the same-snapname guard never ran. That guard
only matters for a run that OUTLIVES the floor -- which a first full upload easily
does, and where collecting it would yank the snapshot out from under a backup that is
still reading from it. Now tested with a 12-hour-old current run.
- the fingerprint/unknown test put the unreadable file in `providers`, which is not
broken -- so fingerprint() skipped the whole module via is_broken() and the filter
under test never executed. The blip has to land in the module that IS broken.
- test_the_real_truenas_versions was a byte-identical copy of
test_async_middleware_is_detected under a name promising more. Replaced with the fact
actually worth pinning: the flavour probe must read STOCK source, because apply.sh
re-runs on an already-patched overlay and our own SNAPSHOT_SYNC block is a plain
`def create_snapshot` -- reading it would report a 25.10 box as synchronous and inject
the wrong wrapper.
All four now fail when the code they name is broken.
Two of these were mine, from the previous round.
- mounted_snapshots still swallowed OSError. I said I had fixed it and had not: the
edit never matched, and I did not read it back. With the mount table unreadable the
GC loses its in-use protection entirely and can destroy the snapshots of a backup
that is still uploading (a first upload easily outlives the 1h age floor). It raises
now, and both behaviours are tested.
- The foreign-dataset check added last round had two bugs of its own. It ignored
`mounted`, so a locked/encrypted dataset from a sibling tree turned a working nightly
backup into a permanent failure — it belongs in `skipped`, exactly as an in-tree one
does. And it tested `mp.startswith(path + "/")`, so a foreign dataset mounted EXACTLY
at the backup path slipped through — the very hole the check was added to close, one
character wide, and the worse case of the two because it SHADOWS the base dataset's
own directory.
- _read_sidecar's new raise broke cleanup_all, which is what recover.sh and
uninstall.sh call — i.e. the code that must work when the box is ALREADY stuck. One
unreadable sidecar aborted it before it unmounted anything, leaving the staging tree
mounted, which pins the snapshots, which is the state recover.sh exists to escape. It
now reports and carries on — and does not delete a record it could not read.
- compat could report a FALSE OK: `defined` was collected by walking the whole file, so
any function named `delete` anywhere in it — on an unrelated class, or nested inside
another method — satisfied "this namespace defines delete". The runtime is stricter
(a plugin class on the service's MRO), so the two could disagree in the ok direction.
compat now looks in the class that declares the namespace. Same question on both
sides, which is what pick_snapshot_service's docstring has been claiming all along.
- apply.sh's compat preflight — the guard that refuses to patch a middleware whose
assumptions no longer hold, on every boot, on a live NAS — had no test at all. It
could be turned into a no-op eight different ways with the suite still green. The
SHIPPED heredoc is now extracted and driven directly against fake verdicts.
Also pinned: the Tap/Tap2 prefix collisions (a sweep that treats "Tap2/data@snap" as
part of Tap's tree DESTROYS another pool's snapshot), and the GC's in_use wiring.
355 tests. Verified on TrueNAS 26.0.0-BETA.1: 292-dataset backup, 0 orphans, 0 leaked
mounts, byte-identical restore of a 4-deep child dataset.
D1, the only cardinal-rule violation left. plan_staging scopes by dataset NAME, which
is right (a dataset with no mountpoint cannot be scoped by path). But ZFS lets any
dataset mount anywhere, so one from a DIFFERENT tree can sit inside the backup path:
Tank/photos mountpoint=/mnt/Tap/apps/photos
It holds data inside the path, and `zfs snapshot -r Tap@...` does NOT cover it —
recursion follows the dataset tree, not the directory tree. It fell out of the name
filter and vanished: not staged, not in `skipped`, no error. The backup reported
SUCCESS with that data missing. Stock has the same blind spot but refuses the nested
config outright; we are the ones relaxing that guard, so the hole is ours. It now
raises.
The test suite was the real weakness. apply.sh's injected blocks carry the
highest-consequence logic in the project — the run_in_thread hop, the flavour
selection, the finally-teardown, the re-raise — and were guarded only by substring
greps. Two of them passed on COMMENTS: `assert "raise" in block` was satisfied by a
comment reading "a cleanup that raises...", and `assert "cleanup_task" in block` by
"cleanup_task gets logger=None". Deleting the actual re-raise (restic then backs up the
UN-STAGED path — the silently-empty backup this module exists to prevent) and deleting
the actual cleanup call from the finally (~250 orphans per run) both left the suite
green. They are asserted structurally now, against the parsed block.
Eleven regressions the audit found surviving now fail the suite, including: a swallowed
staging failure, a missing teardown, an inverted flavour mapping, blocking work back on
the asyncio event loop, the host's deleted get_dataset_recursive, query_filesystems
quietly preferring the filtered middleware query, and a re-frozen `runner`/`sleep`/
`mounts_file` default (which would silently re-arm 19 tests reading the real mount
table on the NAS).
Also: _read_sidecar conflated "no sidecar" with "cannot read the sidecar", so
cleanup_task took the empty branch and UNLINKED the only record of a tree it could not
read. mounted_snapshots returned an empty set on error, silently switching off the GC's
protection for snapshots a concurrent run is using. Both raise now.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts, byte-identical restore of a 4-deep child dataset.
The most important finding is that the FIRST audit's fix was wrong.
_can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines
`delete` on the BASE class and dispatches to self.do_delete at call time, so a bound
`delete` exists on every CRUDService subclass whether or not it still implements one.
The check was therefore answering "is this a CRUDService?" — precisely the weaker "is
the namespace registered?" question its own docstring said must never be asked. It
would still have picked a gutted pool.snapshot and failed every delete. It now walks
the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining
delete/do_delete counts. The test double was equally wrong: it modelled a gutted
service as object(), a shape middlewared cannot produce, so the test passed against a
fake it could never have caught in the field. It is now CRUDService-shaped.
Also:
- The recursive delete's fast path returned [] without confirming anything was
destroyed. A delete that returns cleanly is not proof — iX has already gutted
pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read
"no survivors" as a clean sweep, dropped the sidecar (the only record), and would
have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the
by-name sweep trusts ZFS rather than the API's return value.
- When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not
symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record
clears), a lost record does not.
- _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist;
failing to write it must never be invisible.
- stage_nested now refuses UP FRONT when middleware has no usable snapshot delete,
rather than discovering it after restic has already run.
Tests. The autouse fixture added in the last commit did not work: `runner=_run`,
`mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__
at def time, so monkeypatching the module attribute never reached them. 19 tests were
still reading the real mount table — one matching name from running a real `umount` on
the NAS — and the retry loop really slept. All three are late-bound now; the suite
reads nothing outside tmp_path and runs in 1.1s.
Every mutation the audit reported as SURVIVING now fails the suite: the naive delete
check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager
service resolution, compat's method check, compat's unknown handling, a single-quoted
filtered query in apply.sh, and the get-service assumption.
Also: fingerprint() folded `unknown` problems into a broken module, so one transient
429 rewrote the bug report and the next clean run rewrote it back. Problems are
state-tagged; only definite breakage is digested.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep
child dataset.
`middleware` and the ZFS reader were being threaded through five functions as a
pair, and the snapshot namespace was re-resolved in each of them. That is one
collaborator, not two. _Snapshots owns both, resolves the namespace lazily (so the
read-only paths do not raise over a mutation they never make), and makes the rule the
module rests on structural instead of a comment people have to remember.
Deliberately internal: apply.sh injects calls to the public functions into middlewared
itself, so their signatures are a boot-time contract with a live NAS and are not worth
churning for tidiness.
Verified on TrueNAS 26.0.0-BETA.1 after the change: 292-dataset backup, 0 orphans,
0 leaked mounts, byte-identical restore of a 4-deep child dataset.
Four audits of the TrueNAS 26 branch. The findings, in severity order.
1. TrueNAS 26 orphaned a snapshot on every run, with no backstop.
Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours.
<= 25.10 its create_snapshot called get_dataset_recursive() — the same function this
module vendors — so "stock went recursive" and "we have something to stage" were the
same question. 26 uses filesystem.statfs: recursive = (path == the dataset's
mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint
datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage.
The patch then handed the snapshot back to stock, which destroys the parent only. No
staging tree meant no sidecar, and the GC only ever ran from stage_nested — so
nothing on the box would ever have found the children. Reproduced on the VM: one
orphan per zvol, every run, forever, backup green.
Ownership of the sweep is no longer conditional on staging (own_snapshot()).
2. The runtime resolved a NAMESPACE; compat.py verified a METHOD.
get_service() only proves a namespace is registered. compat checks the namespace AND
that it defines delete/do_delete. So if iX guts the method but keeps the service —
which they have already done to pool.snapshot.do_update on master — compat falls
through to zfs.snapshot and reports the box healthy, while the runtime picks
pool.snapshot and fails every delete. Both sides now ask "can this namespace
delete?", and a test binds the two lists together.
3. query_filesystems() silently dropped malformed rows — the one remaining
silent-omission path, and a direct contradiction of the cardinal rule. It raises now.
A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed.
4. The retry loop discarded the delete error and reported every survivor as
"(still busy?)" — naming the one cause that is benign and hiding the ones that are
permanent. It keeps and reports the real error.
Also: the staging-failure handler could lose the original exception if its own sweep
raised; get_service is now a checked assumption; normalise_dataset and two dead
MiddlewareCall properties removed; stale comments corrected.
Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148
snapshots) and passed here only because this box has no zfs binary — they would have
gone red on the NAS, which is the one machine the release process requires them green
on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of
the five fixes above now fails the suite; before, all 293 passed.
Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset
backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4
(pool.snapshot.delete honours recursive=True).
TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query,
zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested
module's critical path, so nested snapshots were BROKEN on 26.
Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10
and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported
release. tools/compat.py checks the same list the runtime uses, so what CI verifies
and what runs cannot drift apart.
Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is
the point of this commit. Those methods exist, are documented, and are covered by
iX's deprecation policy — and they are not like-for-like replacements. They apply a
visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets
on a real pool, including live application data. Staging from that view omits them
silently, and plan_staging never sees them, so they do not even reach the skipped
list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one
per hidden dataset on every run.
So: read the truth from ZFS, make changes through middleware. zfs list cannot be
filtered by policy and behaves identically on every release. A failing zfs list raises
rather than returning an empty list — "no datasets" and "the command broke" must never
look the same.
No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query,
which returns all 270 datasets. The bug existed only in this port.
Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a
292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical
restore of a four-level-deep child dataset that pool.dataset.query hides.
Two bugs, both of which would have failed silently on the first scheduled run:
- permissions were while the step pushes a branch and opens a PR.
It would have died with a 403 and I would have had a bot that never worked.
- it opened the PR on GITHUB, which is a one-way MIRROR. A PR merged there would be
clobbered by the next fleet-repos mirror push from Gitea. A bot opening PRs against
a mirror is a bot doing nothing, slowly.
Now: contents+pull-requests write, and the PR is opened on Gitea (canonical) via its
API. One long-lived PR, force-pushed in place -- a daily PR is the same mistake as a
daily comment, wearing a hat.
It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.
Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.
The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.
Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
whether to comment' is two chances to drift, and the Gitea one duplicated an issue
for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
main from CI. An unattended push to main is exactly what the release barrier exists
to prevent: a bot that can move main can move it somewhere nobody looked.
An edit matched the literal '## Unreleased' inside a backticked phrase in a prose
bullet and spliced a whole new section into the middle of it, splitting the sentence
in half. The release body IS this file, so that would have shipped to every user.
Tests now assert: no empty version section, versions descend, no heading is indented
inside a list item, and every bullet's bold phrases are balanced (ignoring code spans
-- '*args, **kwargs' is a literal, not markup).
A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record
of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the
recursive snapshot and its cleanup destroyed that record, leaving one snapshot per
descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would
ever have found them.
gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is
gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and
the collector only mops up what the record lost.
It deletes data on a name match, which is a weaker claim than a recorded fact, so the
selection is a pure function with the harshest tests here. A snapshot is collected
only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current
run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a
concurrent backup), and it is over an hour old.
Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it
selects exactly the orphans of the task being run and nothing else.
Found live, in the code written to prevent exactly this.
The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to
finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of
the survivor -- orphaning it permanently.
Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46
reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it
behind again, and then wrote its own snapshot over the record. Permanent orphan,
created by the safety net.
The sidecar is now a list. stage_nested carries forward whatever a reclaim could not
delete; cleanup_task sweeps every pending tree and writes back only the survivors.
cleanup_all reports them one per line instead of formatting a list into an f-string
at the user during uninstall.
Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots
with no straggler.
Found on real hardware: a 256-snapshot backup of /mnt/Tap swept 253 and left 3 with
'dataset is busy'.
ZFS AUTOMOUNTS <dataset>/.zfs/snapshot/<snap> when it is read, and keeps it mounted
for zfs_expire_snapshot seconds (300 default) after the last access. teardown()
unmounts OUR bind mounts but not the automount underneath, so zfs destroy refuses for
exactly the datasets restic read most recently. cleanup_task() then removed the
sidecar anyway -- destroying the only record those snapshots existed. Nothing would
ever have reclaimed them.
- release_snapdirs() unmounts ZFS's own automounts (deepest first) before deleting.
- delete_snapshot_tree() retries the transient busy and RETURNS what it could not
delete, instead of swallowing it.
- The sidecar is removed only on a confirmed-clean sweep -- including on the
staging-failure path, which used to remove it before the caller swept. A sidecar
left behind when the tree is gone costs one no-op delete; a sidecar removed while
the tree exists is unrecoverable.
Two issues with the same title already existed -- the old title embedded the list of
broken refs, so the issue's identity changed whenever that set changed. With an
order-dependent pick the bot would alternate between them, reopening one and
commenting on the other. Lowest number is stable regardless of how the API sorts.
The title embedded the list of broken refs, so the issue's identity changed whenever
that set changed -- and it did: when the async/sync port briefly made 26 look green,
the next run filed a SECOND issue for 'master' alone. A bot that spawns duplicates
gets muted, and then it is not a warning system any more.
The title is now fixed; the refs live in the body, which gets updated in place.
install.sh chmod +x's update.sh, and git recorded update.sh as 100644 -- so the chmod
was a TRACKED modification, and update.sh refuses to run over a dirty tree. Install
once and you could never update again. The error even told you to 'git checkout -- .',
which just undoes the exec bit so the next install can re-dirty it.
Found on the real box, which had been sitting on v0.4.1 for exactly this reason.
Fixed on both sides: the scripts install.sh chmods are executable in git (so the
chmod is a no-op), and update.sh's dirty check now looks at CONTENT, not mode --
git diff --numstat reports 0 0 for a mode-only change. A test asserts every script in
install.sh's chmod loop is already 100755 in git.
Gitea is canonical for development; GitHub is where users clone from and where the
box's read-only checkout points. The install instructions, the re-clone hint and the
'file an issue' link are all read by users, so they name GitHub. docs/releasing.md
still names Gitea, because that is a contributor doc about where the code is pushed.
release_notes.py 'notes v0.6.0-rc1' looked for a CHANGELOG section literally named
v0.6.0-rc1. check() already used base_version(); extract_notes() did not. So the
release workflow cut the tag, passed every gate, and then died extracting the body --
the candidate existed but was never published.
Caught in an rc, which is the entire point of having them.
Also: the Gitea publish and issue steps used jq, which is not guaranteed on a
self-hosted runner. A publish step that dies on a missing tool leaves a tag with no
release behind it, and a bug report that dies on one is a warning system that does
not warn. Both now use python3, which setup-python guarantees.
Install was at line 517 of 969, under the boot sequence, the snapshot lifecycle and
the release process. Someone deciding whether to trust this with their backups should
not have to scroll past any of that.
README is now 211 lines: what it does, the minimum version, install, the support
matrix, updating, uninstall. Everything else moved to docs/ (nested snapshots, how it
works, recovery, CLI, releasing).
A test enforces it: every internal link resolves, Install stays near the top, and the
README does not grow back. Moving Markdown breaks cross-references -- it broke eight
of them here, including one in a recovery doc, where the person following the link is
by definition already having a bad day.
That gate was removed as redundant -- the release job re-runs the full suite against
the tagged commit, and release_gate proves a candidate points at it. The notes
described a check that does not exist.
create_snapshot is module-global in plugins/cloud/snapshot.py, and cloud_sync.py
imports it as well as cloud_backup/sync.py. So the wrapper sat in the path of every
rclone/Storj CloudSync task with snapshot=true, and ran a zfs.dataset.query before
concluding it had nothing to do -- a new failure mode for jobs that worked before
this patch existed.
Worse: a CloudSync task that ever got staged would never be torn down. The teardown
is wired into cloud_backup's restic_backup finally, and CRUD_BLOCK deliberately
leaves CloudSync's guard intact, so the bind mounts would pin the snapshot forever.
The staging path now bails out unless the snapshot is named cloud_backup-*, before
any middleware call.
Separately: the async wrapper's finally dropped logger=, which the sync one passes.
run_in_thread forwards **kwargs, so a cleanup that failed to unmount a bind mount or
delete a snapshot tree logged nothing at all -- on the only platform anyone runs.
This gap was hiding a catastrophe. TrueNAS 26 deletes plugins/zfs_/dataset.py and
plugins/zfs_/snapshot.py outright, taking zfs.dataset.query, zfs.snapshot.query and
zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
Nothing about the five cloud_backup files reveals that, so every other check went
green -- including the one I had just added. The patch would have applied cleanly
and then failed on the first backup, or worse: snapshotted fine and failed to
DELETE, orphaning one snapshot per descendant dataset (250 on a real pool) on every
run, forever.
So 26 is BROKEN and the nested module will not apply there. The async/sync wrapper
work and the vendored get_dataset_recursive stay -- they are correct and necessary
-- but 26 is not supported until the ZFS calls are ported, and that needs a real 26
box to verify. Shipping a port nobody has run is the failure this project exists to
avoid.
Also: do_delete is recognised as delete (24.10/25.04 use the CRUDService
convention), which was reporting both as BROKEN -- a false verdict that would have
disabled nested snapshots on boxes where they work.
26 rewrites cloud_backup from async to synchronous AND deletes
get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's
namespace. Either is a broken backup found at restore time.
The nested module is now one synchronous implementation talking to middlewared via
call_sync, behind two thin wrappers. apply.sh reads which flavour the installed
middleware declares and injects the matching one: <= 25.10 reaches it through
'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls
it directly. The snapshot/bind-mount/failure logic exists once -- an async twin
would mean every future fix had to land twice.
A middleware whose three wrapped functions disagree about asyncness is refused,
not guessed at. get_dataset_recursive is vendored, removing the dependency on both
versions rather than asserting it.
master stays BROKEN on purpose: iX are still renaming middleware->context,
cloud_backup->entry and adding a required credentials param there. Chasing a
branch that moves daily is how you ship a patch nobody tested.
TrueCloud Backup does not exist before 24.10, so on anything older the patch would
attach to nothing and do nothing -- silently, while the user believed their backups
were set up. install.sh now reads system.version and refuses, naming the reason.
Also: the boot sequence now documents the preflight (and that an incompatible module
is skipped for one boot, NOT kill-switched); the troubleshooting table covers the
incompatibility warning; forge URLs point at Gitea.
The split-literal squash is implemented twice: inline in apply.sh's runtime probe
and as compat._squash in the static checker. That subtlety already caused one
silent bug (the probe concluded iX had removed the nesting guard, which means
'retire the module'). Both are now exercised against the same inputs, including
the split-across-literals form stock actually uses.
The audit found the new machinery could do more harm than the bugs it prevents.
- apply.sh reused the 'nothing left to do' exit -- which touches the PERMANENT
kill switch, cleared only by install.sh, never by update.sh -- for the
incompatible case. On TrueNAS 26 (providers ok, nested opt-out) both modules go
quiet, so the switch would fire and the release that fixed 26 could never
re-enable itself. Retirement and incompatibility now take different exits.
- A network blip, a re-export, or a conditional def all read as BROKEN. Each is
now 'unknown', which changes nothing, rather than evidence strong enough to
disable a module.
- 'native' outranked BROKEN everywhere but apply.sh, so a TrueNAS that reworded
the guard AND reshaped the functions rendered as good news.
- compat.py --tree read B2_BLOCK's own 'restic = True' as native support, so the
documented way to check a live box lied on every patched machine.
- The signature check was a name-subset test. It passed reorders, kw-only
conversions, and added required params -- and it had already passed a real bug:
restic_backup takes 4 args on 24.10/25.04, and the wrapper forwarded 5. Nested
backups have been raising TypeError on those releases the whole time. The
wrapper now forwards *args/**kwargs.
- release.sh --promote was unreachable: it died if the tag existed, the gate died
if it did not. The tests hid it by always tagging first.
The matrix is regenerated daily by CI rather than typed once and forgotten — a
support table that quietly goes stale is a false promise to someone deciding
whether to trust this with their backups.
The report body is full of backticks, so 'echo "${{ steps.report.outputs.body }}"'
pasted it into the shell text and bash executed create-snapshot, def and async as
commands. The report is built from iX's middleware source, so that was an injection
vector as well as a bug. inputs.tag on workflow_dispatch had the same shape.
Data goes through files, scalars through env:. Tests enforce it across every
workflow.
TrueNAS 26 rewrites cloud_backup from async to sync. Every block the nested
module injects is an async wrapper around an awaited original, so on 26 it hands
sync.py a coroutine where it unpacks a tuple.
tools/compat.py records what each module assumes and checks it two ways: CI runs
it against iX's source at every release line (including master and the current
BETA) and files a bug report when an unreleased line breaks; apply.sh runs it
against the middlewared actually installed and refuses to apply a module whose
assumptions no longer hold. Stock TrueNAS without a feature beats TrueNAS with a
broken one.
Workflows run on both forges; only the release/issue API calls differ.
The truecloud-mw-restart unit relied on After=multi-user.target /
After=ix-postinit.service, but systemd ordering cannot see middlewared's
internal boot work. On 25.10.4 the restart fired two seconds into
ix-reporting's reporting.start_service call and before the docker/apps
startup task ran, killing both for the whole boot: all apps down
(docker.status FAILED), no dashboard stats, SMB backend uninitialized.
The unit now runs patch/wait_restart.sh: drain the systemd boot job
queue (is-system-running --wait), poll docker.status until the state
machine leaves its transitional states, short grace period, then
try-restart. No Type=oneshot — a oneshot's start job sits in the very
queue the script waits on and would deadlock on itself. All waits are
bounded and fail open.
Document the full boot sequence (stock start, pool import, PREINIT
patching, deferred restart via truecloud-mw-restart), the reboot vs
OS-update survival table, the short unpatched window after boot, and
why manual apply.sh runs require an explicit middlewared restart.
Add a troubleshooting entry for backups failing with
NotImplementedError after a reboot, with ordered diagnostic commands.
On 24.10 (Electric Eel) credentials["provider"] is the type string with
account/key in credentials["attributes"]; 25.04+ moved them into a
provider dict. The injected get_restic_config only handled the newer
shape and raised TypeError on 24.10 at task creation (#1).
The method now detects the schema and reads credentials from the right
place on both. create_task.py list-credentials and list-tasks use the
same schema-agnostic lookup.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds. Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.
Bumps all scripts to v0.0.3.
The guard in the b2.py patch block skipped setting get_restic_config when
something caused B2RcloneRemote.__dict__ to already contain it at import
time (e.g. a TrueNAS version that adds a NotImplementedError stub). Remove
the guard and always assign, which is safe: the native-support kill switch
already prevents patching when TrueNAS ships a real implementation.
Also update the native-support check to distinguish a stub (source contains
NotImplementedError) from a working implementation, so a stub does not
trigger the kill switch and block all future patching.
apply.sh now inspects B2RcloneRemote.__dict__ before patching. If TrueNAS
has shipped get_restic_config natively, it sets the kill switch, unmounts
overlays, and logs a clear instruction to run uninstall.sh.
Also: drop all conditional 'if read-only' language — overlay is always
mounted unconditionally since /usr is always immutable on TrueNAS SCALE.
README updated with auto-disable behaviour and revised native-support table.
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.
- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
URL-fix logic now lives once in sitecustomize._tc_fix_restic_cmd.
apply.sh's restic.py BLOCK delegates to it instead of repeating
the ~40-line implementation.
Additional safety: the BLOCK now guards its get_restic_config
reference with try/except NameError, so a future TrueNAS that
restructures restic.py won't cause an import error.
README: updated Backend table and disclaimer to reflect graceful
degradation; added "If TrueNAS adds native support" section that
covers all five upgrade scenarios (safe pass-through, base-class
shadowing risk, schema-change risk, etc.).
restic 0.16.x changed the B2 URL format to use a colon between bucket and
path (b2:bucket:prefix) instead of a slash. The middlewared URL builder
produces b2:/bucket/path; restic then validates the full string after 'b2:'
as a bucket name, which fails because the slash is not in [a-z0-9-].
Fix the restic.py wrapper to strip the leading slash and replace the first
slash with a colon: b2:/bucket/path -> b2:bucket:path.
Also fix the hasattr bug in sitecustomize.py _patch_b2: hasattr() returns
True for methods inherited from the base class (which raises
NotImplementedError), causing the patch to be silently skipped. Use
'get_restic_config' not in cls.__dict__ instead.
Two bugs:
1. hasattr(B2RcloneRemote, "get_restic_config") returned True because the base
class defines the method (it just raises NotImplementedError). The method was
never added to B2RcloneRemote. Fixed: use __dict__ check instead.
2. "skip if TRUECLOUD_PATCH marker present" prevented a corrected patch block
from replacing a previously-written buggy one without clearing the overlay.
Fixed: always strip any existing TRUECLOUD_PATCH block and rewrite it fresh
using a Python heredoc. Each apply.sh run now self-corrects to the latest
version of the patch.
site.getsitepackages()[0] returns /usr/local/lib/python3.11/dist-packages/ on
TrueNAS 25.x but middlewared lives in /usr/lib/python3/dist-packages/.
sitecustomize.py was installed to the wrong directory and Python never loaded it.
Fix: derive SITE_PKG from middlewared.__file__ so the overlay and sitecustomize.py
land in the correct directory.
Also add direct patching of b2.py and restic.py in the overlay as the primary
backend approach — more reliable than an import hook since it works regardless
of Python's site initialisation configuration. apply.sh now also writes
hook_status.json at boot time so 'verify' shows OK without requiring a backup run.
Also fixes incorrect middlewared log path in README (/var/log/middlewared/middlewared.log
→ /var/log/middlewared.log) and simplifies the verify troubleshooting note.
TrueNAS 25.x mounts /usr as a read-only filesystem. Writing
sitecustomize.py to site-packages and patching the Angular bundle
both fail with EROFS.
Fix: mount a writable overlayfs on each target directory before
writing to it. Upper/work dirs live in /run (tmpfs), so overlays are
volatile per boot and are recreated by apply.sh on every PREINIT run
before middlewared starts.
apply.sh:
- Add _ensure_writable(dir, tag): probes writability; mounts overlay
in /run/truecloud-{tag}-{upper,work} if the directory is read-only;
detects if the overlay is already mounted (idempotent)
- Call _ensure_writable before site-packages writes (tag "sc")
- Detect webui dir with bash loop; call _ensure_writable before
patch_ui.py (tag "ui") — non-fatal if mount fails
uninstall.sh:
- Add overlay unmounting section after file restoration and before
rm -rf, so the lower layer's originals are exposed immediately
- Move _restore_failed exit 1 to after unmount so overlays are
cleaned up even on partial failure
- Update "no backup files" message for immutable OS context
TrueNAS 25.x changed how Angular emits the filterByProviders binding.
Previously a static inline array ("filterByProviders",["STORJ_IX"]),
it is now a pureFunction call:
pe(slot, factory, component.CloudSyncProviderName.Storj)
Add a _PATTERNS list tried in order, with a _match_pattern() helper.
The 25.x pureFunction pattern matches on the stable TypeScript enum
name (CloudSyncProviderName.Storj); the minified variable names and
slot index are matched with \w+ / \d+. Both patterns replace the
binding with ["STORJ_IX","S3","B2"]. MARKER and backup/restore logic
are unchanged.
- create_task.py list-tasks: crash on null credentials.provider
(`creds.get("provider", {})` returns None when key exists but is null;
switch to `(creds.get("provider") or {})`)
- sitecustomize.py: write hook_status.json after each module, not only
when both have loaded; S3-only users (B2 module never imported) now
get a status file from verify instead of "No status file found"
- README: add filesystem find + sqlite3 DB query to the emergency
recovery section so users can locate their clone path when middlewared
is down and midclt is unavailable
- Capture midclt output via $(...) instead of > /dev/null so that
failure detail (which midclt writes to stdout on TrueNAS) is
preserved and shown to the user on error rather than silently
discarded
- Expand update failure hint from a bare query command to an actionable
recovery path: show the midclt output, then print the exact delete
command with the known stale ID so the user can remove it and retry
- Extract hook comment string to _HOOK_COMMENT variable in both
install.sh and uninstall.sh; previously the literal string
'TrueCloud provider patch (S3/B2)' appeared three times across two
files with no shared constant — a silent mismatch on any divergence
would cause hook lookup to return empty with no error output
- Wrap midclt update and create calls with if/else error handlers;
previously a midclt failure under set -euo pipefail silently aborted
the script at "Updating path and enabling ..." with no diagnostic
or recovery guidance
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
substitution into sitecustomize.py; sed's & and | metacharacters
silently corrupt or truncate the output for paths containing those
chars; Python str.replace has no metacharacter issues; also write to
a tmp file and mv atomically so a failed substitution never leaves
an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
$PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
("TrueCloud provider patch (S3/B2)") instead of exact script path;
exact-path match breaks when the repo is moved after install —
uninstall leaves the stale hook registered (fires on every boot),
and reinstall creates a duplicate entry; install.sh now also updates
the script path on re-run so a moved repo self-corrects
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
Users who delete the cloned repo after install had no way to uninstall
without re-cloning. Now install.sh copies uninstall.sh to PATCH_DIR
alongside recover.sh, so the uninstall path is always the stable
/data/truecloud-patch/uninstall.sh. README updated to match.
- uninstall.sh: track _restore_failed separately from RESTORED so
"No backup files found" only prints when find returns nothing (not
when mv fails on existing backups); abort with exit 1 before rm -rf
when any restore fails, leaving PATCH_DIR and recover.sh intact
- install.sh: extend log-scan grep to catch ERROR: lines from
patch_ui.py (backup OSError was silently missed by WARNING:-only grep)
- install.sh: reword restart-failure message — hook IS already
registered and sitecustomize.py IS installed; patch activates on
next boot regardless
- apply.sh: replace `if $_can_install` with `[ "$_can_install" = true ]`
(explicit test, no implicit command lookup); drop 2>/dev/null on
install cp so OS error detail reaches the log
- install.sh: print 'verify' command immediately after successful restart
so users know to confirm the backend patch loaded before creating tasks
- README: correct --insecure description; it controls TLS to the TrueNAS
API (where the API key is transmitted), not the S3 endpoint — previous
wording implied it was safe to use for S3 self-signed certs
All findings from four consecutive full-codebase audit passes plus an
adversarial iXsystems-perspective audit. No new candidates surfaced in
the final clean-pass — branch declared complete.
Fixes:
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp previously fell through and could destroy the vendor file
- apply.sh: correct comment (keeps two prior log generations, not one)
- patch_ui.py: catch OSError on bundle backup with specific diagnostic
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return 'UI already patched' instead of misleading 'pattern not found'
- install.sh: early guard detects pipe-install (bash <(curl ...)) and
exits with a clear error pointing to the git clone workflow
- install.sh: scope WARNING grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: systemctl restart failure now surfaces a recovery hint
- create_task.py: add MITM risk warning to --insecure flag help text
- create_task.py: handle unexpected 2xx response schema in cmd_create
- uninstall.sh: add import-middlewared verification after Python detection
- uninstall.sh: mv failure in JS bundle restore loop no longer aborts
under set -e before cleanup; emits WARNING and continues
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
- apply.sh: gate sitecustomize.py install on backup success; a failed
backup cp (disk full, read-only mount) previously fell through and
overwrote the vendor file with no recovery path
- create_task.py: handle unexpected 2xx response schema in cmd_create;
bare KeyError on result['id'] is replaced with a diagnostic print
- uninstall.sh: mv inside while loop had no error handling; under
set -euo pipefail a failed mv aborted the script before rm -rf PATCH_DIR,
leaving the system in partial-uninstall limbo
- create_task.py: add MITM risk warning to --insecure flag help text;
common home-user pattern (self-signed cert) exposes API key in transit
- install.sh: replace bare systemctl restart with explicit failure check
that prints a recovery hint when middlewared fails to start post-install
- patch_ui.py: wrap shutil.copy2 backup in try/except OSError so a
permission or read-only filesystem error prints a specific diagnostic
instead of crashing the script with a generic 'exited non-zero' message
- install.sh: add early guard that detects pipe-install (bash <(curl ...))
and exits with a clear error pointing to the git clone workflow
- apply.sh: correct comment from 'one prior generation' to 'two prior
generations (.1 and .2)' — rotation has always kept three log files
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
in patch/patch_ui.py, preventing silent drift if a new path is added
- patch_ui.py: find_bundle now matches MARKER so already-patched files
return early and print 'UI already patched' instead of the misleading
'filterByProviders pattern not found' warning
- install.sh: scope warning grep to current run only (record log offset
before apply.sh, tail -c +N to read only new bytes)
- install.sh: fix misleading 'before continuing' wording on warning banner
- install.sh: fix grep anchor (^WARNING: missed [truecloud-patch] WARNING: lines)
- uninstall.sh: add import-middlewared verification after Python detection,
matching apply.sh fallback logic
flan