The Gitea runner image executes jobs as root, and chmod(0) cannot make a file
unreadable for root (CAP_DAC_OVERRIDE) — the two tests failed there while
passing on GitHub's non-root runner. Skipping as root keeps the scenario
exercised everywhere it is constructible.
compat.yml's push paths listed tools/compat.py but not tools/compat_publish.py.
So the commit that taught the bot to refresh a stale report body fired no run, and
the report stayed stale until the next scheduled one -- caught by watching for the
run that never came. A fix nobody runs is a fix nobody has.
Two questions were sharing one answer.
"Have the findings changed?" gates COMMENTS. They notify, and a daily "still
broken, same as yesterday" is what teaches everyone to ignore the one that
finally matters.
"Is the body still true?" gates the BODY. Editing an issue body notifies nobody
on either forge, so keeping it honest is free.
Conflated, an unchanged fingerprint froze the body -- and the fingerprint ignores,
by design, everything that moves on its own: healthy rows, the hardware-verified
column, point releases, and how a row is LABELLED. So the master -> 27-dev relabel
would have shipped to the README and never to the issue anybody actually opens.
The report would have gone on saying "master (unreleased) BROKEN" -- the precise
false alarm the relabel exists to kill -- until iX happened to break something
else.
The body is now rewritten whenever it is stale; comments stay strictly a changelog
of real changes. Bodies are compared after normalising line endings, because a
forge that round-trips \r\n would otherwise trigger a silent rewrite every run and
leave the issue looking freshly touched every morning.
Two ways the matrix misled the person it exists for -- somebody deciding whether
to trust this with their backups.
master is not the next release. iX branches each major onto its own release/ line
and master rolls straight on to the one after: every recent commit on master
targets 27.0.0-BETA.1 while 26 is still in beta. So a BROKEN master row, rendered
"master (unreleased)", read as "the version you are about to install is broken"
when the breakage is a major release away on a line nobody can download. It is now
labelled from the newest major in the matrix plus one, so it becomes 28-dev by
itself once 27 branches.
The break, for the record, is NAS-141498 (2026-06-24), "Convert cloud_backup
plugin to the typesafe pattern": it re-signatures restic_backup and
get_restic_config, splitting entry/credentials out of the cloud_backup dict. Not
being chased while the 27 line churns.
The next maintenance release was never checked -- and it is the one that reaches
users. Shipped came from TS-* tags, unreleased from release/* branches carrying
-BETA/-RC. A branched-but-untagged MAINTENANCE release is neither: release/25.10.5
has no tag, and its line has already shipped, so the "a prerelease of a shipped
line is history" filter threw it out. It was invisible, and it is exactly what a
25.10.4 box gets on its next update; a break in it reaches real users before the
daily check ever looks, on the only line anybody runs.
A plain release/X.Y.Z branch now counts when its line HAS shipped and it sorts
NEWER than that line's newest tag. Both exclusions fall out of the same rule:
release/24.10-RC.2 sorts older than TS-24.10.2.4 (history), and the typo branch
release/25.20.2.2 is on a line with no tag at all (not a release line). This
surfaced two refs never checked before -- release/25.10.5 and release/24.10.2.5 --
both of which pass.
is_unreleased() keys off where a ref came from (a branch is by construction not
shipped) rather than hunting -BETA/-RC in the name. Otherwise release/25.10.5
counts as shipped and a break in it fails the build as a live outage, on a version
nobody is running yet.
`find_issue()` skipped pull requests by testing for the PRESENCE of the
`pull_request` key. GitHub omits that key on a plain issue. Gitea sends it as
`null`. So on Gitea every issue was thrown away as if it were a PR, the lookup
came back empty on every run, and the bot took the "nothing filed yet" branch and
opened a brand-new report instead of editing the one already open.
Nine of them piled up on the canonical forge. Four were filed AFTER the commit
that was supposed to stop exactly this -- the same failure it was written to
prevent, moved from comments to issues. It survived because `find_issue` was the
one function here with no test; the mirror deduped correctly, and the mirror is
what anybody would have looked at.
Test the value, not the key: it is the only form that is true on both forges.
Pinned by tests that carry each forge's payload shape by hand.
Also send both paging parameters. GitHub reads `per_page`, Gitea reads `limit`,
and each ignores the other's; Gitea's default page is 30, so the lookup would
have begun missing the report once the pile it was creating outgrew one page.
The README and how-it-works still said "TrueNAS 26: nested snapshots are not supported
yet" and "the third is not fixed, and is why 26 reports BROKEN". Both shipped in v0.7.0
and are now false — exactly the kind of stale claim that misleads somebody deciding
whether to trust this with their backups.
Adds docs/verification.md: what has ACTUALLY been run, as opposed to what the support
matrix proves. The matrix is static analysis — it shows the patch's assumptions still
hold, which is a strictly weaker claim than "a backup ran and a restore came back". The
new file records the three live tasks exercised on 25.10.4 (nested, nested+zvols,
non-nested), the md5 of the file that came back out of B2, the real orphan the collector
reclaimed from the pool, and what is NOT covered (24.10/25.04 unrun; master broken; no
reboot on v0.7.0). The README now points at it, next to the matrix it qualifies.
The how-it-works TrueNAS 26 section now explains the part that mattered: the public
pool.* queries are not like-for-like replacements for the deleted private zfs.* ones —
they apply a visibility policy hiding 84 of 270 datasets on a real pool, including live
app data — and the rule the module now follows (read the truth from ZFS, make changes
through middleware). Plus the divergence nothing warned about: 26 decides `recursive` by
a different rule than this patch decides `nested`, which orphaned one snapshot per zvol
on every run until ownership of the sweep was made unconditional.
Enumerate datasets and snapshots from ZFS, not middleware's filtered queries; own the
snapshot sweep unconditionally (TrueNAS 26 decides "recursive" by a different rule than
we decide "nested"); resolve the snapshot namespace at runtime by the same predicate
tools/compat.py checks.
Four adversarial audits, every finding fixed and mutation-pinned. Verified on a real
TrueNAS 26.0.0-BETA.1 install: 292-dataset backup, 0 orphans, 0 leaked mounts,
byte-identical restore of a four-level-deep child dataset that middleware's own API
hides.
The test suite is not bloated -- 3,701 lines of test code against 3,760 lines of
product code, one duplicate pair in 356 tests, 8% single-assertion tests. The waste was
not volume, it was four tests that looked like coverage and provided none:
- test_parents_are_mounted_before_children: the fixture was already in depth order, so
the sort it exists for was never exercised. Deleting `mounts.sort(key=_depth)` passed
the whole suite. It now uses datasets whose NAME order differs from their MOUNTPOINT
depth, which is the only case the sort is for.
- test_it_NEVER_touches_the_current_run: the "current" snapshot was a minute old, so the
age floor excluded it regardless and the same-snapname guard never ran. That guard
only matters for a run that OUTLIVES the floor -- which a first full upload easily
does, and where collecting it would yank the snapshot out from under a backup that is
still reading from it. Now tested with a 12-hour-old current run.
- the fingerprint/unknown test put the unreadable file in `providers`, which is not
broken -- so fingerprint() skipped the whole module via is_broken() and the filter
under test never executed. The blip has to land in the module that IS broken.
- test_the_real_truenas_versions was a byte-identical copy of
test_async_middleware_is_detected under a name promising more. Replaced with the fact
actually worth pinning: the flavour probe must read STOCK source, because apply.sh
re-runs on an already-patched overlay and our own SNAPSHOT_SYNC block is a plain
`def create_snapshot` -- reading it would report a 25.10 box as synchronous and inject
the wrong wrapper.
All four now fail when the code they name is broken.
Two of these were mine, from the previous round.
- mounted_snapshots still swallowed OSError. I said I had fixed it and had not: the
edit never matched, and I did not read it back. With the mount table unreadable the
GC loses its in-use protection entirely and can destroy the snapshots of a backup
that is still uploading (a first upload easily outlives the 1h age floor). It raises
now, and both behaviours are tested.
- The foreign-dataset check added last round had two bugs of its own. It ignored
`mounted`, so a locked/encrypted dataset from a sibling tree turned a working nightly
backup into a permanent failure — it belongs in `skipped`, exactly as an in-tree one
does. And it tested `mp.startswith(path + "/")`, so a foreign dataset mounted EXACTLY
at the backup path slipped through — the very hole the check was added to close, one
character wide, and the worse case of the two because it SHADOWS the base dataset's
own directory.
- _read_sidecar's new raise broke cleanup_all, which is what recover.sh and
uninstall.sh call — i.e. the code that must work when the box is ALREADY stuck. One
unreadable sidecar aborted it before it unmounted anything, leaving the staging tree
mounted, which pins the snapshots, which is the state recover.sh exists to escape. It
now reports and carries on — and does not delete a record it could not read.
- compat could report a FALSE OK: `defined` was collected by walking the whole file, so
any function named `delete` anywhere in it — on an unrelated class, or nested inside
another method — satisfied "this namespace defines delete". The runtime is stricter
(a plugin class on the service's MRO), so the two could disagree in the ok direction.
compat now looks in the class that declares the namespace. Same question on both
sides, which is what pick_snapshot_service's docstring has been claiming all along.
- apply.sh's compat preflight — the guard that refuses to patch a middleware whose
assumptions no longer hold, on every boot, on a live NAS — had no test at all. It
could be turned into a no-op eight different ways with the suite still green. The
SHIPPED heredoc is now extracted and driven directly against fake verdicts.
Also pinned: the Tap/Tap2 prefix collisions (a sweep that treats "Tap2/data@snap" as
part of Tap's tree DESTROYS another pool's snapshot), and the GC's in_use wiring.
355 tests. Verified on TrueNAS 26.0.0-BETA.1: 292-dataset backup, 0 orphans, 0 leaked
mounts, byte-identical restore of a 4-deep child dataset.
D1, the only cardinal-rule violation left. plan_staging scopes by dataset NAME, which
is right (a dataset with no mountpoint cannot be scoped by path). But ZFS lets any
dataset mount anywhere, so one from a DIFFERENT tree can sit inside the backup path:
Tank/photos mountpoint=/mnt/Tap/apps/photos
It holds data inside the path, and `zfs snapshot -r Tap@...` does NOT cover it —
recursion follows the dataset tree, not the directory tree. It fell out of the name
filter and vanished: not staged, not in `skipped`, no error. The backup reported
SUCCESS with that data missing. Stock has the same blind spot but refuses the nested
config outright; we are the ones relaxing that guard, so the hole is ours. It now
raises.
The test suite was the real weakness. apply.sh's injected blocks carry the
highest-consequence logic in the project — the run_in_thread hop, the flavour
selection, the finally-teardown, the re-raise — and were guarded only by substring
greps. Two of them passed on COMMENTS: `assert "raise" in block` was satisfied by a
comment reading "a cleanup that raises...", and `assert "cleanup_task" in block` by
"cleanup_task gets logger=None". Deleting the actual re-raise (restic then backs up the
UN-STAGED path — the silently-empty backup this module exists to prevent) and deleting
the actual cleanup call from the finally (~250 orphans per run) both left the suite
green. They are asserted structurally now, against the parsed block.
Eleven regressions the audit found surviving now fail the suite, including: a swallowed
staging failure, a missing teardown, an inverted flavour mapping, blocking work back on
the asyncio event loop, the host's deleted get_dataset_recursive, query_filesystems
quietly preferring the filtered middleware query, and a re-frozen `runner`/`sleep`/
`mounts_file` default (which would silently re-arm 19 tests reading the real mount
table on the NAS).
Also: _read_sidecar conflated "no sidecar" with "cannot read the sidecar", so
cleanup_task took the empty branch and UNLINKED the only record of a tree it could not
read. mounted_snapshots returned an empty set on error, silently switching off the GC's
protection for snapshots a concurrent run is using. Both raise now.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts, byte-identical restore of a 4-deep child dataset.
The most important finding is that the FIRST audit's fix was wrong.
_can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines
`delete` on the BASE class and dispatches to self.do_delete at call time, so a bound
`delete` exists on every CRUDService subclass whether or not it still implements one.
The check was therefore answering "is this a CRUDService?" — precisely the weaker "is
the namespace registered?" question its own docstring said must never be asked. It
would still have picked a gutted pool.snapshot and failed every delete. It now walks
the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining
delete/do_delete counts. The test double was equally wrong: it modelled a gutted
service as object(), a shape middlewared cannot produce, so the test passed against a
fake it could never have caught in the field. It is now CRUDService-shaped.
Also:
- The recursive delete's fast path returned [] without confirming anything was
destroyed. A delete that returns cleanly is not proof — iX has already gutted
pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read
"no survivors" as a clean sweep, dropped the sidecar (the only record), and would
have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the
by-name sweep trusts ZFS rather than the API's return value.
- When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not
symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record
clears), a lost record does not.
- _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist;
failing to write it must never be invisible.
- stage_nested now refuses UP FRONT when middleware has no usable snapshot delete,
rather than discovering it after restic has already run.
Tests. The autouse fixture added in the last commit did not work: `runner=_run`,
`mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__
at def time, so monkeypatching the module attribute never reached them. 19 tests were
still reading the real mount table — one matching name from running a real `umount` on
the NAS — and the retry loop really slept. All three are late-bound now; the suite
reads nothing outside tmp_path and runs in 1.1s.
Every mutation the audit reported as SURVIVING now fails the suite: the naive delete
check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager
service resolution, compat's method check, compat's unknown handling, a single-quoted
filtered query in apply.sh, and the get-service assumption.
Also: fingerprint() folded `unknown` problems into a broken module, so one transient
429 rewrote the bug report and the next clean run rewrote it back. Problems are
state-tagged; only definite breakage is digested.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep
child dataset.
`middleware` and the ZFS reader were being threaded through five functions as a
pair, and the snapshot namespace was re-resolved in each of them. That is one
collaborator, not two. _Snapshots owns both, resolves the namespace lazily (so the
read-only paths do not raise over a mutation they never make), and makes the rule the
module rests on structural instead of a comment people have to remember.
Deliberately internal: apply.sh injects calls to the public functions into middlewared
itself, so their signatures are a boot-time contract with a live NAS and are not worth
churning for tidiness.
Verified on TrueNAS 26.0.0-BETA.1 after the change: 292-dataset backup, 0 orphans,
0 leaked mounts, byte-identical restore of a 4-deep child dataset.
Four audits of the TrueNAS 26 branch. The findings, in severity order.
1. TrueNAS 26 orphaned a snapshot on every run, with no backstop.
Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours.
<= 25.10 its create_snapshot called get_dataset_recursive() — the same function this
module vendors — so "stock went recursive" and "we have something to stage" were the
same question. 26 uses filesystem.statfs: recursive = (path == the dataset's
mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint
datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage.
The patch then handed the snapshot back to stock, which destroys the parent only. No
staging tree meant no sidecar, and the GC only ever ran from stage_nested — so
nothing on the box would ever have found the children. Reproduced on the VM: one
orphan per zvol, every run, forever, backup green.
Ownership of the sweep is no longer conditional on staging (own_snapshot()).
2. The runtime resolved a NAMESPACE; compat.py verified a METHOD.
get_service() only proves a namespace is registered. compat checks the namespace AND
that it defines delete/do_delete. So if iX guts the method but keeps the service —
which they have already done to pool.snapshot.do_update on master — compat falls
through to zfs.snapshot and reports the box healthy, while the runtime picks
pool.snapshot and fails every delete. Both sides now ask "can this namespace
delete?", and a test binds the two lists together.
3. query_filesystems() silently dropped malformed rows — the one remaining
silent-omission path, and a direct contradiction of the cardinal rule. It raises now.
A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed.
4. The retry loop discarded the delete error and reported every survivor as
"(still busy?)" — naming the one cause that is benign and hiding the ones that are
permanent. It keeps and reports the real error.
Also: the staging-failure handler could lose the original exception if its own sweep
raised; get_service is now a checked assumption; normalise_dataset and two dead
MiddlewareCall properties removed; stale comments corrected.
Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148
snapshots) and passed here only because this box has no zfs binary — they would have
gone red on the NAS, which is the one machine the release process requires them green
on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of
the five fixes above now fails the suite; before, all 293 passed.
Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset
backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4
(pool.snapshot.delete honours recursive=True).
TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query,
zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested
module's critical path, so nested snapshots were BROKEN on 26.
Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10
and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported
release. tools/compat.py checks the same list the runtime uses, so what CI verifies
and what runs cannot drift apart.
Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is
the point of this commit. Those methods exist, are documented, and are covered by
iX's deprecation policy — and they are not like-for-like replacements. They apply a
visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets
on a real pool, including live application data. Staging from that view omits them
silently, and plan_staging never sees them, so they do not even reach the skipped
list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one
per hidden dataset on every run.
So: read the truth from ZFS, make changes through middleware. zfs list cannot be
filtered by policy and behaves identically on every release. A failing zfs list raises
rather than returning an empty list — "no datasets" and "the command broke" must never
look the same.
No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query,
which returns all 270 datasets. The bug existed only in this port.
Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a
292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical
restore of a four-level-deep child dataset that pool.dataset.query hides.
Two bugs, both of which would have failed silently on the first scheduled run:
- permissions were while the step pushes a branch and opens a PR.
It would have died with a 403 and I would have had a bot that never worked.
- it opened the PR on GITHUB, which is a one-way MIRROR. A PR merged there would be
clobbered by the next fleet-repos mirror push from Gitea. A bot opening PRs against
a mirror is a bot doing nothing, slowly.
Now: contents+pull-requests write, and the PR is opened on Gitea (canonical) via its
API. One long-lived PR, force-pushed in place -- a daily PR is the same mistake as a
daily comment, wearing a hat.
It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.
Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.
The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.
Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
whether to comment' is two chances to drift, and the Gitea one duplicated an issue
for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
main from CI. An unattended push to main is exactly what the release barrier exists
to prevent: a bot that can move main can move it somewhere nobody looked.
An edit matched the literal '## Unreleased' inside a backticked phrase in a prose
bullet and spliced a whole new section into the middle of it, splitting the sentence
in half. The release body IS this file, so that would have shipped to every user.
Tests now assert: no empty version section, versions descend, no heading is indented
inside a list item, and every bullet's bold phrases are balanced (ignoring code spans
-- '*args, **kwargs' is a literal, not markup).
A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record
of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the
recursive snapshot and its cleanup destroyed that record, leaving one snapshot per
descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would
ever have found them.
gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is
gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and
the collector only mops up what the record lost.
It deletes data on a name match, which is a weaker claim than a recorded fact, so the
selection is a pure function with the harshest tests here. A snapshot is collected
only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current
run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a
concurrent backup), and it is over an hour old.
Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it
selects exactly the orphans of the task being run and nothing else.
Found live, in the code written to prevent exactly this.
The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to
finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of
the survivor -- orphaning it permanently.
Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46
reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it
behind again, and then wrote its own snapshot over the record. Permanent orphan,
created by the safety net.
The sidecar is now a list. stage_nested carries forward whatever a reclaim could not
delete; cleanup_task sweeps every pending tree and writes back only the survivors.
cleanup_all reports them one per line instead of formatting a list into an f-string
at the user during uninstall.
Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots
with no straggler.
Found on real hardware: a 256-snapshot backup of /mnt/Tap swept 253 and left 3 with
'dataset is busy'.
ZFS AUTOMOUNTS <dataset>/.zfs/snapshot/<snap> when it is read, and keeps it mounted
for zfs_expire_snapshot seconds (300 default) after the last access. teardown()
unmounts OUR bind mounts but not the automount underneath, so zfs destroy refuses for
exactly the datasets restic read most recently. cleanup_task() then removed the
sidecar anyway -- destroying the only record those snapshots existed. Nothing would
ever have reclaimed them.
- release_snapdirs() unmounts ZFS's own automounts (deepest first) before deleting.
- delete_snapshot_tree() retries the transient busy and RETURNS what it could not
delete, instead of swallowing it.
- The sidecar is removed only on a confirmed-clean sweep -- including on the
staging-failure path, which used to remove it before the caller swept. A sidecar
left behind when the tree is gone costs one no-op delete; a sidecar removed while
the tree exists is unrecoverable.
Two issues with the same title already existed -- the old title embedded the list of
broken refs, so the issue's identity changed whenever that set changed. With an
order-dependent pick the bot would alternate between them, reopening one and
commenting on the other. Lowest number is stable regardless of how the API sorts.
The title embedded the list of broken refs, so the issue's identity changed whenever
that set changed -- and it did: when the async/sync port briefly made 26 look green,
the next run filed a SECOND issue for 'master' alone. A bot that spawns duplicates
gets muted, and then it is not a warning system any more.
The title is now fixed; the refs live in the body, which gets updated in place.
install.sh chmod +x's update.sh, and git recorded update.sh as 100644 -- so the chmod
was a TRACKED modification, and update.sh refuses to run over a dirty tree. Install
once and you could never update again. The error even told you to 'git checkout -- .',
which just undoes the exec bit so the next install can re-dirty it.
Found on the real box, which had been sitting on v0.4.1 for exactly this reason.
Fixed on both sides: the scripts install.sh chmods are executable in git (so the
chmod is a no-op), and update.sh's dirty check now looks at CONTENT, not mode --
git diff --numstat reports 0 0 for a mode-only change. A test asserts every script in
install.sh's chmod loop is already 100755 in git.
Gitea is canonical for development; GitHub is where users clone from and where the
box's read-only checkout points. The install instructions, the re-clone hint and the
'file an issue' link are all read by users, so they name GitHub. docs/releasing.md
still names Gitea, because that is a contributor doc about where the code is pushed.
release_notes.py 'notes v0.6.0-rc1' looked for a CHANGELOG section literally named
v0.6.0-rc1. check() already used base_version(); extract_notes() did not. So the
release workflow cut the tag, passed every gate, and then died extracting the body --
the candidate existed but was never published.
Caught in an rc, which is the entire point of having them.
Also: the Gitea publish and issue steps used jq, which is not guaranteed on a
self-hosted runner. A publish step that dies on a missing tool leaves a tag with no
release behind it, and a bug report that dies on one is a warning system that does
not warn. Both now use python3, which setup-python guarantees.
Install was at line 517 of 969, under the boot sequence, the snapshot lifecycle and
the release process. Someone deciding whether to trust this with their backups should
not have to scroll past any of that.
README is now 211 lines: what it does, the minimum version, install, the support
matrix, updating, uninstall. Everything else moved to docs/ (nested snapshots, how it
works, recovery, CLI, releasing).
A test enforces it: every internal link resolves, Install stays near the top, and the
README does not grow back. Moving Markdown breaks cross-references -- it broke eight
of them here, including one in a recovery doc, where the person following the link is
by definition already having a bad day.
That gate was removed as redundant -- the release job re-runs the full suite against
the tagged commit, and release_gate proves a candidate points at it. The notes
described a check that does not exist.
create_snapshot is module-global in plugins/cloud/snapshot.py, and cloud_sync.py
imports it as well as cloud_backup/sync.py. So the wrapper sat in the path of every
rclone/Storj CloudSync task with snapshot=true, and ran a zfs.dataset.query before
concluding it had nothing to do -- a new failure mode for jobs that worked before
this patch existed.
Worse: a CloudSync task that ever got staged would never be torn down. The teardown
is wired into cloud_backup's restic_backup finally, and CRUD_BLOCK deliberately
leaves CloudSync's guard intact, so the bind mounts would pin the snapshot forever.
The staging path now bails out unless the snapshot is named cloud_backup-*, before
any middleware call.
Separately: the async wrapper's finally dropped logger=, which the sync one passes.
run_in_thread forwards **kwargs, so a cleanup that failed to unmount a bind mount or
delete a snapshot tree logged nothing at all -- on the only platform anyone runs.
This gap was hiding a catastrophe. TrueNAS 26 deletes plugins/zfs_/dataset.py and
plugins/zfs_/snapshot.py outright, taking zfs.dataset.query, zfs.snapshot.query and
zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
Nothing about the five cloud_backup files reveals that, so every other check went
green -- including the one I had just added. The patch would have applied cleanly
and then failed on the first backup, or worse: snapshotted fine and failed to
DELETE, orphaning one snapshot per descendant dataset (250 on a real pool) on every
run, forever.
So 26 is BROKEN and the nested module will not apply there. The async/sync wrapper
work and the vendored get_dataset_recursive stay -- they are correct and necessary
-- but 26 is not supported until the ZFS calls are ported, and that needs a real 26
box to verify. Shipping a port nobody has run is the failure this project exists to
avoid.
Also: do_delete is recognised as delete (24.10/25.04 use the CRUDService
convention), which was reporting both as BROKEN -- a false verdict that would have
disabled nested snapshots on boxes where they work.
26 rewrites cloud_backup from async to synchronous AND deletes
get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's
namespace. Either is a broken backup found at restore time.
The nested module is now one synchronous implementation talking to middlewared via
call_sync, behind two thin wrappers. apply.sh reads which flavour the installed
middleware declares and injects the matching one: <= 25.10 reaches it through
'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls
it directly. The snapshot/bind-mount/failure logic exists once -- an async twin
would mean every future fix had to land twice.
A middleware whose three wrapped functions disagree about asyncness is refused,
not guessed at. get_dataset_recursive is vendored, removing the dependency on both
versions rather than asserting it.
master stays BROKEN on purpose: iX are still renaming middleware->context,
cloud_backup->entry and adding a required credentials param there. Chasing a
branch that moves daily is how you ship a patch nobody tested.
TrueCloud Backup does not exist before 24.10, so on anything older the patch would
attach to nothing and do nothing -- silently, while the user believed their backups
were set up. install.sh now reads system.version and refuses, naming the reason.
Also: the boot sequence now documents the preflight (and that an incompatible module
is skipped for one boot, NOT kill-switched); the troubleshooting table covers the
incompatibility warning; forge URLs point at Gitea.
The split-literal squash is implemented twice: inline in apply.sh's runtime probe
and as compat._squash in the static checker. That subtlety already caused one
silent bug (the probe concluded iX had removed the nesting guard, which means
'retire the module'). Both are now exercised against the same inputs, including
the split-across-literals form stock actually uses.
The audit found the new machinery could do more harm than the bugs it prevents.
- apply.sh reused the 'nothing left to do' exit -- which touches the PERMANENT
kill switch, cleared only by install.sh, never by update.sh -- for the
incompatible case. On TrueNAS 26 (providers ok, nested opt-out) both modules go
quiet, so the switch would fire and the release that fixed 26 could never
re-enable itself. Retirement and incompatibility now take different exits.
- A network blip, a re-export, or a conditional def all read as BROKEN. Each is
now 'unknown', which changes nothing, rather than evidence strong enough to
disable a module.
- 'native' outranked BROKEN everywhere but apply.sh, so a TrueNAS that reworded
the guard AND reshaped the functions rendered as good news.
- compat.py --tree read B2_BLOCK's own 'restic = True' as native support, so the
documented way to check a live box lied on every patched machine.
- The signature check was a name-subset test. It passed reorders, kw-only
conversions, and added required params -- and it had already passed a real bug:
restic_backup takes 4 args on 24.10/25.04, and the wrapper forwarded 5. Nested
backups have been raising TypeError on those releases the whole time. The
wrapper now forwards *args/**kwargs.
- release.sh --promote was unreachable: it died if the tag existed, the gate died
if it did not. The tests hid it by always tagging first.
flan
flan