Two CI failures shared one cause: four jobs starting together on the
self-hosted runner.
act caches each action as one shared clone under /root/.cache/act/<hash> and
re-pulls it per job, so concurrent jobs fight over that directory and the
loser dies with "lstat .../<file>: no such file or directory" before any test
runs — a different victim each push. And the runner force-pulls its base image
per job, so four jobs meant four anonymous Docker Hub pulls per push; that hit
429 Too Many Requests and every job started failing before it began, including
the shell job nothing had touched.
Installing uv without an action only shrank the surface, since every job still
used actions/checkout. Concurrency is the ingredient, so this removes it: one
job cannot race itself whatever actions it uses, and one job is one pull. The
version sweep moves inside the job and still runs every version after one
fails, preserving what fail-fast: false bought.
act caches each action as one shared git clone under /root/.cache/act/<hash>
and re-pulls it per job. The three python matrix jobs start within the same
second on the self-hosted runner, race on that directory, and the loser dies
with "lstat /root/.cache/act/<hash>/.npmrc: no such file or directory" before
any test runs — a red main with zero suite output and a different victim each
push (3.12, then 3.11).
The action was only fetching a binary; the matrix interpreter is selected per
command by uvx --python. A run: step has no action-cache entry and cannot
race, and keeps the jobs parallel — serialising the matrix would cost 3x the
wall clock and still leave actions/checkout shared across four jobs. The uv
version is pinned under the same rule as ruff.
The accompanying test parses uses: directives rather than the raw text, so the
comment can still name the action it avoids, and uses re instead of PyYAML
because CI runs uvx pytest, whose environment holds pytest and nothing else.
create_task.py verify decides "loaded" by comparing middlewared's start time
against hook_status.json's patched_at. The post-restart re-mount restores the
same patch the boot pass already applied, but it runs after the restart — so
letting apply.sh re-stamp made patched_at newer than the process that had
correctly imported the patch, and verify reported FAIL while everything was
working. That would have fired on every boot where docker's nvidia sysext
merge detaches the overlay.
Found on hardware while exercising the candidate: a new lying status
introduced by the fix for a lying status. The snapshot lives in /run, not the
repo — a leftover file there would leave the tree dirty, which update.sh
refuses to run over.
try-restart returns as soon as middlewared is READY, and middlewared then
brings docker up — docker.configure_nvidia merges the stock nvidia sysext
over /usr at that point, detaching the overlay after the patched modules
have already been imported. Checking the disk there reports "missing" on a
perfectly healthy system, and the retry that followed would restart a
correctly-patched middlewared straight back into the same race, then log
ERROR when nothing was wrong.
The disk is the right oracle before the restart and the wrong one after it.
After the restart the overlay is re-mounted so the next restart finds patched
files, and whether this middlewared actually holds the patch is left to the
in-process alert, which is the only thing that can answer it exactly.
Patching at PREINIT and restarting minutes later is only sound while the
patched files are still on the live path when middlewared re-imports them,
and PREINIT cannot guarantee that. The overlay sits inside /usr, so anything
that remounts that hierarchy detaches it — a systemd-sysext merge/refresh
from another PREINIT hook, or middlewared's own docker.configure_nvidia at
runtime. Init scripts run sequentially in id order, so a hook registered
after this one always wins, and reordering them would not help because
docker.configure_nvidia fires long after PREINIT is done.
Observed on 25.10.6: the overlay was mounted at 16:41:56, a sysext refresh
unmerged and remerged /usr four seconds later, and the deferred restart at
16:47:24 loaded stock modules. Every B2 cloud_backup task then failed with
NotImplementedError for nineteen hours across four scheduled runs while
apply.log and hook_status.json both reported the patch active.
wait_restart.sh now re-applies immediately before restarting — after boot has
settled, which is also after every sysext merge and docker nvidia
configuration — verifies the marker is on the live path, restarts, and
verifies again, retrying once. It is no longer exec'd, so something can run
after the restart to find out what it loaded. apply.sh records the resolved
middlewared directory in .mw_dir for that check, and honours TRUECLOUD_REAPPLY
so the re-apply pass does not schedule a second restart.
_ensure_writable treated "one of our overlays is listed here" as "already
done", but it only reaches that check when the directory is not writable, and
a live overlay of ours always is — a shadowed overlay was indistinguishable
from a healthy one. It is now detached and re-mounted, reusing the upperdir so
files patched earlier in the boot survive, with a fresh workdir and a retry on
a private one, since overlayfs refuses a workdir a detached mount still holds.
Add a CRITICAL hourly alert for the case none of this can prevent: the patch
being on disk but not in the running process. apply.log can only report the
first. The alert asks the second question from inside middlewared, where the
patch's own stamps make it exact, and checks both halves since either can go
missing alone. It stays quiet when the kill switch is set or the providers
module has been retired as native, and is not muted by update_alerts_disabled.
wait_restart.sh also logs to apply.log now: journald retention on a busy box
is easily shorter than the interval between reboots, and the boot that caused
this had already rotated away by the time it was investigated.
The Gitea runner image executes jobs as root, and chmod(0) cannot make a file
unreadable for root (CAP_DAC_OVERRIDE) — the two tests failed there while
passing on GitHub's non-root runner. Skipping as root keeps the scenario
exercised everywhere it is constructible.
compat.yml's push paths listed tools/compat.py but not tools/compat_publish.py.
So the commit that taught the bot to refresh a stale report body fired no run, and
the report stayed stale until the next scheduled one -- caught by watching for the
run that never came. A fix nobody runs is a fix nobody has.
Two questions were sharing one answer.
"Have the findings changed?" gates COMMENTS. They notify, and a daily "still
broken, same as yesterday" is what teaches everyone to ignore the one that
finally matters.
"Is the body still true?" gates the BODY. Editing an issue body notifies nobody
on either forge, so keeping it honest is free.
Conflated, an unchanged fingerprint froze the body -- and the fingerprint ignores,
by design, everything that moves on its own: healthy rows, the hardware-verified
column, point releases, and how a row is LABELLED. So the master -> 27-dev relabel
would have shipped to the README and never to the issue anybody actually opens.
The report would have gone on saying "master (unreleased) BROKEN" -- the precise
false alarm the relabel exists to kill -- until iX happened to break something
else.
The body is now rewritten whenever it is stale; comments stay strictly a changelog
of real changes. Bodies are compared after normalising line endings, because a
forge that round-trips \r\n would otherwise trigger a silent rewrite every run and
leave the issue looking freshly touched every morning.
Two ways the matrix misled the person it exists for -- somebody deciding whether
to trust this with their backups.
master is not the next release. iX branches each major onto its own release/ line
and master rolls straight on to the one after: every recent commit on master
targets 27.0.0-BETA.1 while 26 is still in beta. So a BROKEN master row, rendered
"master (unreleased)", read as "the version you are about to install is broken"
when the breakage is a major release away on a line nobody can download. It is now
labelled from the newest major in the matrix plus one, so it becomes 28-dev by
itself once 27 branches.
The break, for the record, is NAS-141498 (2026-06-24), "Convert cloud_backup
plugin to the typesafe pattern": it re-signatures restic_backup and
get_restic_config, splitting entry/credentials out of the cloud_backup dict. Not
being chased while the 27 line churns.
The next maintenance release was never checked -- and it is the one that reaches
users. Shipped came from TS-* tags, unreleased from release/* branches carrying
-BETA/-RC. A branched-but-untagged MAINTENANCE release is neither: release/25.10.5
has no tag, and its line has already shipped, so the "a prerelease of a shipped
line is history" filter threw it out. It was invisible, and it is exactly what a
25.10.4 box gets on its next update; a break in it reaches real users before the
daily check ever looks, on the only line anybody runs.
A plain release/X.Y.Z branch now counts when its line HAS shipped and it sorts
NEWER than that line's newest tag. Both exclusions fall out of the same rule:
release/24.10-RC.2 sorts older than TS-24.10.2.4 (history), and the typo branch
release/25.20.2.2 is on a line with no tag at all (not a release line). This
surfaced two refs never checked before -- release/25.10.5 and release/24.10.2.5 --
both of which pass.
is_unreleased() keys off where a ref came from (a branch is by construction not
shipped) rather than hunting -BETA/-RC in the name. Otherwise release/25.10.5
counts as shipped and a break in it fails the build as a live outage, on a version
nobody is running yet.
`find_issue()` skipped pull requests by testing for the PRESENCE of the
`pull_request` key. GitHub omits that key on a plain issue. Gitea sends it as
`null`. So on Gitea every issue was thrown away as if it were a PR, the lookup
came back empty on every run, and the bot took the "nothing filed yet" branch and
opened a brand-new report instead of editing the one already open.
Nine of them piled up on the canonical forge. Four were filed AFTER the commit
that was supposed to stop exactly this -- the same failure it was written to
prevent, moved from comments to issues. It survived because `find_issue` was the
one function here with no test; the mirror deduped correctly, and the mirror is
what anybody would have looked at.
Test the value, not the key: it is the only form that is true on both forges.
Pinned by tests that carry each forge's payload shape by hand.
Also send both paging parameters. GitHub reads `per_page`, Gitea reads `limit`,
and each ignores the other's; Gitea's default page is 30, so the lookup would
have begun missing the report once the pile it was creating outgrew one page.
The README and how-it-works still said "TrueNAS 26: nested snapshots are not supported
yet" and "the third is not fixed, and is why 26 reports BROKEN". Both shipped in v0.7.0
and are now false — exactly the kind of stale claim that misleads somebody deciding
whether to trust this with their backups.
Adds docs/verification.md: what has ACTUALLY been run, as opposed to what the support
matrix proves. The matrix is static analysis — it shows the patch's assumptions still
hold, which is a strictly weaker claim than "a backup ran and a restore came back". The
new file records the three live tasks exercised on 25.10.4 (nested, nested+zvols,
non-nested), the md5 of the file that came back out of B2, the real orphan the collector
reclaimed from the pool, and what is NOT covered (24.10/25.04 unrun; master broken; no
reboot on v0.7.0). The README now points at it, next to the matrix it qualifies.
The how-it-works TrueNAS 26 section now explains the part that mattered: the public
pool.* queries are not like-for-like replacements for the deleted private zfs.* ones —
they apply a visibility policy hiding 84 of 270 datasets on a real pool, including live
app data — and the rule the module now follows (read the truth from ZFS, make changes
through middleware). Plus the divergence nothing warned about: 26 decides `recursive` by
a different rule than this patch decides `nested`, which orphaned one snapshot per zvol
on every run until ownership of the sweep was made unconditional.
Enumerate datasets and snapshots from ZFS, not middleware's filtered queries; own the
snapshot sweep unconditionally (TrueNAS 26 decides "recursive" by a different rule than
we decide "nested"); resolve the snapshot namespace at runtime by the same predicate
tools/compat.py checks.
Four adversarial audits, every finding fixed and mutation-pinned. Verified on a real
TrueNAS 26.0.0-BETA.1 install: 292-dataset backup, 0 orphans, 0 leaked mounts,
byte-identical restore of a four-level-deep child dataset that middleware's own API
hides.
The test suite is not bloated -- 3,701 lines of test code against 3,760 lines of
product code, one duplicate pair in 356 tests, 8% single-assertion tests. The waste was
not volume, it was four tests that looked like coverage and provided none:
- test_parents_are_mounted_before_children: the fixture was already in depth order, so
the sort it exists for was never exercised. Deleting `mounts.sort(key=_depth)` passed
the whole suite. It now uses datasets whose NAME order differs from their MOUNTPOINT
depth, which is the only case the sort is for.
- test_it_NEVER_touches_the_current_run: the "current" snapshot was a minute old, so the
age floor excluded it regardless and the same-snapname guard never ran. That guard
only matters for a run that OUTLIVES the floor -- which a first full upload easily
does, and where collecting it would yank the snapshot out from under a backup that is
still reading from it. Now tested with a 12-hour-old current run.
- the fingerprint/unknown test put the unreadable file in `providers`, which is not
broken -- so fingerprint() skipped the whole module via is_broken() and the filter
under test never executed. The blip has to land in the module that IS broken.
- test_the_real_truenas_versions was a byte-identical copy of
test_async_middleware_is_detected under a name promising more. Replaced with the fact
actually worth pinning: the flavour probe must read STOCK source, because apply.sh
re-runs on an already-patched overlay and our own SNAPSHOT_SYNC block is a plain
`def create_snapshot` -- reading it would report a 25.10 box as synchronous and inject
the wrong wrapper.
All four now fail when the code they name is broken.
Two of these were mine, from the previous round.
- mounted_snapshots still swallowed OSError. I said I had fixed it and had not: the
edit never matched, and I did not read it back. With the mount table unreadable the
GC loses its in-use protection entirely and can destroy the snapshots of a backup
that is still uploading (a first upload easily outlives the 1h age floor). It raises
now, and both behaviours are tested.
- The foreign-dataset check added last round had two bugs of its own. It ignored
`mounted`, so a locked/encrypted dataset from a sibling tree turned a working nightly
backup into a permanent failure — it belongs in `skipped`, exactly as an in-tree one
does. And it tested `mp.startswith(path + "/")`, so a foreign dataset mounted EXACTLY
at the backup path slipped through — the very hole the check was added to close, one
character wide, and the worse case of the two because it SHADOWS the base dataset's
own directory.
- _read_sidecar's new raise broke cleanup_all, which is what recover.sh and
uninstall.sh call — i.e. the code that must work when the box is ALREADY stuck. One
unreadable sidecar aborted it before it unmounted anything, leaving the staging tree
mounted, which pins the snapshots, which is the state recover.sh exists to escape. It
now reports and carries on — and does not delete a record it could not read.
- compat could report a FALSE OK: `defined` was collected by walking the whole file, so
any function named `delete` anywhere in it — on an unrelated class, or nested inside
another method — satisfied "this namespace defines delete". The runtime is stricter
(a plugin class on the service's MRO), so the two could disagree in the ok direction.
compat now looks in the class that declares the namespace. Same question on both
sides, which is what pick_snapshot_service's docstring has been claiming all along.
- apply.sh's compat preflight — the guard that refuses to patch a middleware whose
assumptions no longer hold, on every boot, on a live NAS — had no test at all. It
could be turned into a no-op eight different ways with the suite still green. The
SHIPPED heredoc is now extracted and driven directly against fake verdicts.
Also pinned: the Tap/Tap2 prefix collisions (a sweep that treats "Tap2/data@snap" as
part of Tap's tree DESTROYS another pool's snapshot), and the GC's in_use wiring.
355 tests. Verified on TrueNAS 26.0.0-BETA.1: 292-dataset backup, 0 orphans, 0 leaked
mounts, byte-identical restore of a 4-deep child dataset.
D1, the only cardinal-rule violation left. plan_staging scopes by dataset NAME, which
is right (a dataset with no mountpoint cannot be scoped by path). But ZFS lets any
dataset mount anywhere, so one from a DIFFERENT tree can sit inside the backup path:
Tank/photos mountpoint=/mnt/Tap/apps/photos
It holds data inside the path, and `zfs snapshot -r Tap@...` does NOT cover it —
recursion follows the dataset tree, not the directory tree. It fell out of the name
filter and vanished: not staged, not in `skipped`, no error. The backup reported
SUCCESS with that data missing. Stock has the same blind spot but refuses the nested
config outright; we are the ones relaxing that guard, so the hole is ours. It now
raises.
The test suite was the real weakness. apply.sh's injected blocks carry the
highest-consequence logic in the project — the run_in_thread hop, the flavour
selection, the finally-teardown, the re-raise — and were guarded only by substring
greps. Two of them passed on COMMENTS: `assert "raise" in block` was satisfied by a
comment reading "a cleanup that raises...", and `assert "cleanup_task" in block` by
"cleanup_task gets logger=None". Deleting the actual re-raise (restic then backs up the
UN-STAGED path — the silently-empty backup this module exists to prevent) and deleting
the actual cleanup call from the finally (~250 orphans per run) both left the suite
green. They are asserted structurally now, against the parsed block.
Eleven regressions the audit found surviving now fail the suite, including: a swallowed
staging failure, a missing teardown, an inverted flavour mapping, blocking work back on
the asyncio event loop, the host's deleted get_dataset_recursive, query_filesystems
quietly preferring the filtered middleware query, and a re-frozen `runner`/`sleep`/
`mounts_file` default (which would silently re-arm 19 tests reading the real mount
table on the NAS).
Also: _read_sidecar conflated "no sidecar" with "cannot read the sidecar", so
cleanup_task took the empty branch and UNLINKED the only record of a tree it could not
read. mounted_snapshots returned an empty set on error, silently switching off the GC's
protection for snapshots a concurrent run is using. Both raise now.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts, byte-identical restore of a 4-deep child dataset.
The most important finding is that the FIRST audit's fix was wrong.
_can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines
`delete` on the BASE class and dispatches to self.do_delete at call time, so a bound
`delete` exists on every CRUDService subclass whether or not it still implements one.
The check was therefore answering "is this a CRUDService?" — precisely the weaker "is
the namespace registered?" question its own docstring said must never be asked. It
would still have picked a gutted pool.snapshot and failed every delete. It now walks
the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining
delete/do_delete counts. The test double was equally wrong: it modelled a gutted
service as object(), a shape middlewared cannot produce, so the test passed against a
fake it could never have caught in the field. It is now CRUDService-shaped.
Also:
- The recursive delete's fast path returned [] without confirming anything was
destroyed. A delete that returns cleanly is not proof — iX has already gutted
pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read
"no survivors" as a clean sweep, dropped the sidecar (the only record), and would
have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the
by-name sweep trusts ZFS rather than the API's return value.
- When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not
symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record
clears), a lost record does not.
- _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist;
failing to write it must never be invisible.
- stage_nested now refuses UP FRONT when middleware has no usable snapshot delete,
rather than discovering it after restic has already run.
Tests. The autouse fixture added in the last commit did not work: `runner=_run`,
`mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__
at def time, so monkeypatching the module attribute never reached them. 19 tests were
still reading the real mount table — one matching name from running a real `umount` on
the NAS — and the retry loop really slept. All three are late-bound now; the suite
reads nothing outside tmp_path and runs in 1.1s.
Every mutation the audit reported as SURVIVING now fails the suite: the naive delete
check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager
service resolution, compat's method check, compat's unknown handling, a single-quoted
filtered query in apply.sh, and the get-service assumption.
Also: fingerprint() folded `unknown` problems into a broken module, so one transient
429 rewrote the bug report and the next clean run rewrote it back. Problems are
state-tagged; only definite breakage is digested.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep
child dataset.
`middleware` and the ZFS reader were being threaded through five functions as a
pair, and the snapshot namespace was re-resolved in each of them. That is one
collaborator, not two. _Snapshots owns both, resolves the namespace lazily (so the
read-only paths do not raise over a mutation they never make), and makes the rule the
module rests on structural instead of a comment people have to remember.
Deliberately internal: apply.sh injects calls to the public functions into middlewared
itself, so their signatures are a boot-time contract with a live NAS and are not worth
churning for tidiness.
Verified on TrueNAS 26.0.0-BETA.1 after the change: 292-dataset backup, 0 orphans,
0 leaked mounts, byte-identical restore of a 4-deep child dataset.
Four audits of the TrueNAS 26 branch. The findings, in severity order.
1. TrueNAS 26 orphaned a snapshot on every run, with no backstop.
Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours.
<= 25.10 its create_snapshot called get_dataset_recursive() — the same function this
module vendors — so "stock went recursive" and "we have something to stage" were the
same question. 26 uses filesystem.statfs: recursive = (path == the dataset's
mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint
datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage.
The patch then handed the snapshot back to stock, which destroys the parent only. No
staging tree meant no sidecar, and the GC only ever ran from stage_nested — so
nothing on the box would ever have found the children. Reproduced on the VM: one
orphan per zvol, every run, forever, backup green.
Ownership of the sweep is no longer conditional on staging (own_snapshot()).
2. The runtime resolved a NAMESPACE; compat.py verified a METHOD.
get_service() only proves a namespace is registered. compat checks the namespace AND
that it defines delete/do_delete. So if iX guts the method but keeps the service —
which they have already done to pool.snapshot.do_update on master — compat falls
through to zfs.snapshot and reports the box healthy, while the runtime picks
pool.snapshot and fails every delete. Both sides now ask "can this namespace
delete?", and a test binds the two lists together.
3. query_filesystems() silently dropped malformed rows — the one remaining
silent-omission path, and a direct contradiction of the cardinal rule. It raises now.
A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed.
4. The retry loop discarded the delete error and reported every survivor as
"(still busy?)" — naming the one cause that is benign and hiding the ones that are
permanent. It keeps and reports the real error.
Also: the staging-failure handler could lose the original exception if its own sweep
raised; get_service is now a checked assumption; normalise_dataset and two dead
MiddlewareCall properties removed; stale comments corrected.
Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148
snapshots) and passed here only because this box has no zfs binary — they would have
gone red on the NAS, which is the one machine the release process requires them green
on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of
the five fixes above now fails the suite; before, all 293 passed.
Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset
backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4
(pool.snapshot.delete honours recursive=True).
TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query,
zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested
module's critical path, so nested snapshots were BROKEN on 26.
Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10
and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported
release. tools/compat.py checks the same list the runtime uses, so what CI verifies
and what runs cannot drift apart.
Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is
the point of this commit. Those methods exist, are documented, and are covered by
iX's deprecation policy — and they are not like-for-like replacements. They apply a
visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets
on a real pool, including live application data. Staging from that view omits them
silently, and plan_staging never sees them, so they do not even reach the skipped
list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one
per hidden dataset on every run.
So: read the truth from ZFS, make changes through middleware. zfs list cannot be
filtered by policy and behaves identically on every release. A failing zfs list raises
rather than returning an empty list — "no datasets" and "the command broke" must never
look the same.
No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query,
which returns all 270 datasets. The bug existed only in this port.
Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a
292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical
restore of a four-level-deep child dataset that pool.dataset.query hides.
Two bugs, both of which would have failed silently on the first scheduled run:
- permissions were while the step pushes a branch and opens a PR.
It would have died with a 403 and I would have had a bot that never worked.
- it opened the PR on GITHUB, which is a one-way MIRROR. A PR merged there would be
clobbered by the next fleet-repos mirror push from Gitea. A bot opening PRs against
a mirror is a bot doing nothing, slowly.
Now: contents+pull-requests write, and the PR is opened on Gitea (canonical) via its
API. One long-lived PR, force-pushed in place -- a daily PR is the same mistake as a
daily comment, wearing a hat.
It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.
Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.
The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.
Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
whether to comment' is two chances to drift, and the Gitea one duplicated an issue
for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
main from CI. An unattended push to main is exactly what the release barrier exists
to prevent: a bot that can move main can move it somewhere nobody looked.
An edit matched the literal '## Unreleased' inside a backticked phrase in a prose
bullet and spliced a whole new section into the middle of it, splitting the sentence
in half. The release body IS this file, so that would have shipped to every user.
Tests now assert: no empty version section, versions descend, no heading is indented
inside a list item, and every bullet's bold phrases are balanced (ignoring code spans
-- '*args, **kwargs' is a literal, not markup).
A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record
of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the
recursive snapshot and its cleanup destroyed that record, leaving one snapshot per
descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would
ever have found them.
gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is
gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and
the collector only mops up what the record lost.
It deletes data on a name match, which is a weaker claim than a recorded fact, so the
selection is a pure function with the harshest tests here. A snapshot is collected
only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current
run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a
concurrent backup), and it is over an hour old.
Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it
selects exactly the orphans of the task being run and nothing else.
Found live, in the code written to prevent exactly this.
The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to
finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of
the survivor -- orphaning it permanently.
Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46
reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it
behind again, and then wrote its own snapshot over the record. Permanent orphan,
created by the safety net.
The sidecar is now a list. stage_nested carries forward whatever a reclaim could not
delete; cleanup_task sweeps every pending tree and writes back only the survivors.
cleanup_all reports them one per line instead of formatting a list into an f-string
at the user during uninstall.
Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots
with no straggler.
Found on real hardware: a 256-snapshot backup of /mnt/Tap swept 253 and left 3 with
'dataset is busy'.
ZFS AUTOMOUNTS <dataset>/.zfs/snapshot/<snap> when it is read, and keeps it mounted
for zfs_expire_snapshot seconds (300 default) after the last access. teardown()
unmounts OUR bind mounts but not the automount underneath, so zfs destroy refuses for
exactly the datasets restic read most recently. cleanup_task() then removed the
sidecar anyway -- destroying the only record those snapshots existed. Nothing would
ever have reclaimed them.
- release_snapdirs() unmounts ZFS's own automounts (deepest first) before deleting.
- delete_snapshot_tree() retries the transient busy and RETURNS what it could not
delete, instead of swallowing it.
- The sidecar is removed only on a confirmed-clean sweep -- including on the
staging-failure path, which used to remove it before the caller swept. A sidecar
left behind when the tree is gone costs one no-op delete; a sidecar removed while
the tree exists is unrecoverable.
Two issues with the same title already existed -- the old title embedded the list of
broken refs, so the issue's identity changed whenever that set changed. With an
order-dependent pick the bot would alternate between them, reopening one and
commenting on the other. Lowest number is stable regardless of how the API sorts.
The title embedded the list of broken refs, so the issue's identity changed whenever
that set changed -- and it did: when the async/sync port briefly made 26 look green,
the next run filed a SECOND issue for 'master' alone. A bot that spawns duplicates
gets muted, and then it is not a warning system any more.
The title is now fixed; the refs live in the body, which gets updated in place.
install.sh chmod +x's update.sh, and git recorded update.sh as 100644 -- so the chmod
was a TRACKED modification, and update.sh refuses to run over a dirty tree. Install
once and you could never update again. The error even told you to 'git checkout -- .',
which just undoes the exec bit so the next install can re-dirty it.
Found on the real box, which had been sitting on v0.4.1 for exactly this reason.
Fixed on both sides: the scripts install.sh chmods are executable in git (so the
chmod is a no-op), and update.sh's dirty check now looks at CONTENT, not mode --
git diff --numstat reports 0 0 for a mode-only change. A test asserts every script in
install.sh's chmod loop is already 100755 in git.
Gitea is canonical for development; GitHub is where users clone from and where the
box's read-only checkout points. The install instructions, the re-clone hint and the
'file an issue' link are all read by users, so they name GitHub. docs/releasing.md
still names Gitea, because that is a contributor doc about where the code is pushed.
release_notes.py 'notes v0.6.0-rc1' looked for a CHANGELOG section literally named
v0.6.0-rc1. check() already used base_version(); extract_notes() did not. So the
release workflow cut the tag, passed every gate, and then died extracting the body --
the candidate existed but was never published.
Caught in an rc, which is the entire point of having them.
Also: the Gitea publish and issue steps used jq, which is not guaranteed on a
self-hosted runner. A publish step that dies on a missing tool leaves a tag with no
release behind it, and a bug report that dies on one is a warning system that does
not warn. Both now use python3, which setup-python guarantees.
truecloud-patch bot
flan