Compare commits

...
22 Commits
Author SHA1 Message Date
flan 30b9f18166 release v0.7.0
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.13 (push) Failing after 16s
CI / python 3.12 (push) Failing after 17s
Release / release (push) Failing after 15s
CI / python 3.11 (push) Failing after 12s
TrueNAS compatibility / compat (push) Successful in 14s
2026-07-14 01:45:23 +00:00
flan 908b6e9f22 merge: TrueNAS 26 support
CI / shell (shellcheck + syntax) (push) Successful in 9s
CI / python 3.11 (push) Failing after 12s
CI / python 3.12 (push) Failing after 16s
CI / python 3.13 (push) Failing after 16s
TrueNAS compatibility / compat (push) Successful in 15s
Enumerate datasets and snapshots from ZFS, not middleware's filtered queries; own the
snapshot sweep unconditionally (TrueNAS 26 decides "recursive" by a different rule than
we decide "nested"); resolve the snapshot namespace at runtime by the same predicate
tools/compat.py checks.

Four adversarial audits, every finding fixed and mutation-pinned. Verified on a real
TrueNAS 26.0.0-BETA.1 install: 292-dataset backup, 0 orphans, 0 leaked mounts,
byte-identical restore of a four-level-deep child dataset that middleware's own API
hides.
2026-07-14 01:44:44 +00:00
flan df412eeff7 test: make three vacuous tests actually test something; drop one duplicate
The test suite is not bloated -- 3,701 lines of test code against 3,760 lines of
product code, one duplicate pair in 356 tests, 8% single-assertion tests. The waste was
not volume, it was four tests that looked like coverage and provided none:

- test_parents_are_mounted_before_children: the fixture was already in depth order, so
  the sort it exists for was never exercised. Deleting `mounts.sort(key=_depth)` passed
  the whole suite. It now uses datasets whose NAME order differs from their MOUNTPOINT
  depth, which is the only case the sort is for.

- test_it_NEVER_touches_the_current_run: the "current" snapshot was a minute old, so the
  age floor excluded it regardless and the same-snapname guard never ran. That guard
  only matters for a run that OUTLIVES the floor -- which a first full upload easily
  does, and where collecting it would yank the snapshot out from under a backup that is
  still reading from it. Now tested with a 12-hour-old current run.

- the fingerprint/unknown test put the unreadable file in `providers`, which is not
  broken -- so fingerprint() skipped the whole module via is_broken() and the filter
  under test never executed. The blip has to land in the module that IS broken.

- test_the_real_truenas_versions was a byte-identical copy of
  test_async_middleware_is_detected under a name promising more. Replaced with the fact
  actually worth pinning: the flavour probe must read STOCK source, because apply.sh
  re-runs on an already-patched overlay and our own SNAPSHOT_SYNC block is a plain
  `def create_snapshot` -- reading it would report a 25.10 box as synchronous and inject
  the wrong wrapper.

All four now fail when the code they name is broken.
2026-07-14 01:41:25 +00:00
flan 086b20ed23 fix: fourth audit — two regressions from the last fix, and the boot preflight had no test
Two of these were mine, from the previous round.

- mounted_snapshots still swallowed OSError. I said I had fixed it and had not: the
  edit never matched, and I did not read it back. With the mount table unreadable the
  GC loses its in-use protection entirely and can destroy the snapshots of a backup
  that is still uploading (a first upload easily outlives the 1h age floor). It raises
  now, and both behaviours are tested.

- The foreign-dataset check added last round had two bugs of its own. It ignored
  `mounted`, so a locked/encrypted dataset from a sibling tree turned a working nightly
  backup into a permanent failure — it belongs in `skipped`, exactly as an in-tree one
  does. And it tested `mp.startswith(path + "/")`, so a foreign dataset mounted EXACTLY
  at the backup path slipped through — the very hole the check was added to close, one
  character wide, and the worse case of the two because it SHADOWS the base dataset's
  own directory.

- _read_sidecar's new raise broke cleanup_all, which is what recover.sh and
  uninstall.sh call — i.e. the code that must work when the box is ALREADY stuck. One
  unreadable sidecar aborted it before it unmounted anything, leaving the staging tree
  mounted, which pins the snapshots, which is the state recover.sh exists to escape. It
  now reports and carries on — and does not delete a record it could not read.

- compat could report a FALSE OK: `defined` was collected by walking the whole file, so
  any function named `delete` anywhere in it — on an unrelated class, or nested inside
  another method — satisfied "this namespace defines delete". The runtime is stricter
  (a plugin class on the service's MRO), so the two could disagree in the ok direction.
  compat now looks in the class that declares the namespace. Same question on both
  sides, which is what pick_snapshot_service's docstring has been claiming all along.

- apply.sh's compat preflight — the guard that refuses to patch a middleware whose
  assumptions no longer hold, on every boot, on a live NAS — had no test at all. It
  could be turned into a no-op eight different ways with the suite still green. The
  SHIPPED heredoc is now extracted and driven directly against fake verdicts.

Also pinned: the Tap/Tap2 prefix collisions (a sweep that treats "Tap2/data@snap" as
part of Tap's tree DESTROYS another pool's snapshot), and the GC's in_use wiring.

355 tests. Verified on TrueNAS 26.0.0-BETA.1: 292-dataset backup, 0 orphans, 0 leaked
mounts, byte-identical restore of a 4-deep child dataset.
2026-07-14 01:34:13 +00:00
flan 677c90481c docs: record the third-audit findings in the changelog 2026-07-14 00:39:40 +00:00
flan 8a41d7d7ef fix: third audit — a cross-tree dataset was omitted silently, and the block tests passed on comments
D1, the only cardinal-rule violation left. plan_staging scopes by dataset NAME, which
is right (a dataset with no mountpoint cannot be scoped by path). But ZFS lets any
dataset mount anywhere, so one from a DIFFERENT tree can sit inside the backup path:

    Tank/photos   mountpoint=/mnt/Tap/apps/photos

It holds data inside the path, and `zfs snapshot -r Tap@...` does NOT cover it —
recursion follows the dataset tree, not the directory tree. It fell out of the name
filter and vanished: not staged, not in `skipped`, no error. The backup reported
SUCCESS with that data missing. Stock has the same blind spot but refuses the nested
config outright; we are the ones relaxing that guard, so the hole is ours. It now
raises.

The test suite was the real weakness. apply.sh's injected blocks carry the
highest-consequence logic in the project — the run_in_thread hop, the flavour
selection, the finally-teardown, the re-raise — and were guarded only by substring
greps. Two of them passed on COMMENTS: `assert "raise" in block` was satisfied by a
comment reading "a cleanup that raises...", and `assert "cleanup_task" in block` by
"cleanup_task gets logger=None". Deleting the actual re-raise (restic then backs up the
UN-STAGED path — the silently-empty backup this module exists to prevent) and deleting
the actual cleanup call from the finally (~250 orphans per run) both left the suite
green. They are asserted structurally now, against the parsed block.

Eleven regressions the audit found surviving now fail the suite, including: a swallowed
staging failure, a missing teardown, an inverted flavour mapping, blocking work back on
the asyncio event loop, the host's deleted get_dataset_recursive, query_filesystems
quietly preferring the filtered middleware query, and a re-frozen `runner`/`sleep`/
`mounts_file` default (which would silently re-arm 19 tests reading the real mount
table on the NAS).

Also: _read_sidecar conflated "no sidecar" with "cannot read the sidecar", so
cleanup_task took the empty branch and UNLINKED the only record of a tree it could not
read. mounted_snapshots returned an empty set on error, silently switching off the GC's
protection for snapshots a concurrent run is using. Both raise now.

Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts, byte-identical restore of a 4-deep child dataset.
2026-07-14 00:38:45 +00:00
flan 0fea5c40bd docs: record the second-audit findings in the changelog 2026-07-14 00:21:06 +00:00
flan ce6998a935 fix: second audit — the delete check did nothing on a real box, and five guards were untestable
The most important finding is that the FIRST audit's fix was wrong.

_can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines
`delete` on the BASE class and dispatches to self.do_delete at call time, so a bound
`delete` exists on every CRUDService subclass whether or not it still implements one.
The check was therefore answering "is this a CRUDService?" — precisely the weaker "is
the namespace registered?" question its own docstring said must never be asked. It
would still have picked a gutted pool.snapshot and failed every delete. It now walks
the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining
delete/do_delete counts. The test double was equally wrong: it modelled a gutted
service as object(), a shape middlewared cannot produce, so the test passed against a
fake it could never have caught in the field. It is now CRUDService-shaped.

Also:

- The recursive delete's fast path returned [] without confirming anything was
  destroyed. A delete that returns cleanly is not proof — iX has already gutted
  pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read
  "no survivors" as a clean sweep, dropped the sidecar (the only record), and would
  have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the
  by-name sweep trusts ZFS rather than the API's return value.

- When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not
  symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record
  clears), a lost record does not.

- _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist;
  failing to write it must never be invisible.

- stage_nested now refuses UP FRONT when middleware has no usable snapshot delete,
  rather than discovering it after restic has already run.

Tests. The autouse fixture added in the last commit did not work: `runner=_run`,
`mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__
at def time, so monkeypatching the module attribute never reached them. 19 tests were
still reading the real mount table — one matching name from running a real `umount` on
the NAS — and the retry loop really slept. All three are late-bound now; the suite
reads nothing outside tmp_path and runs in 1.1s.

Every mutation the audit reported as SURVIVING now fails the suite: the naive delete
check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager
service resolution, compat's method check, compat's unknown handling, a single-quoted
filtered query in apply.sh, and the get-service assumption.

Also: fingerprint() folded `unknown` problems into a broken module, so one transient
429 rewrote the bug report and the next clean run rewrote it back. Problems are
state-tagged; only definite breakage is digested.

Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep
child dataset.
2026-07-14 00:10:10 +00:00
flan 928d0d1973 refactor: one seam for snapshots — read from ZFS, mutate through middleware
`middleware` and the ZFS reader were being threaded through five functions as a
pair, and the snapshot namespace was re-resolved in each of them. That is one
collaborator, not two. _Snapshots owns both, resolves the namespace lazily (so the
read-only paths do not raise over a mutation they never make), and makes the rule the
module rests on structural instead of a comment people have to remember.

Deliberately internal: apply.sh injects calls to the public functions into middlewared
itself, so their signatures are a boot-time contract with a live NAS and are not worth
churning for tidiness.

Verified on TrueNAS 26.0.0-BETA.1 after the change: 292-dataset backup, 0 orphans,
0 leaked mounts, byte-identical restore of a 4-deep child dataset.
2026-07-13 23:33:30 +00:00
flan 413cd60ed4 fix: own the snapshot sweep unconditionally; align the runtime and the manifest
Four audits of the TrueNAS 26 branch. The findings, in severity order.

1. TrueNAS 26 orphaned a snapshot on every run, with no backstop.

Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours.
<= 25.10 its create_snapshot called get_dataset_recursive() — the same function this
module vendors — so "stock went recursive" and "we have something to stage" were the
same question. 26 uses filesystem.statfs: recursive = (path == the dataset's
mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint
datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage.

The patch then handed the snapshot back to stock, which destroys the parent only. No
staging tree meant no sidecar, and the GC only ever ran from stage_nested — so
nothing on the box would ever have found the children. Reproduced on the VM: one
orphan per zvol, every run, forever, backup green.

Ownership of the sweep is no longer conditional on staging (own_snapshot()).

2. The runtime resolved a NAMESPACE; compat.py verified a METHOD.

get_service() only proves a namespace is registered. compat checks the namespace AND
that it defines delete/do_delete. So if iX guts the method but keeps the service —
which they have already done to pool.snapshot.do_update on master — compat falls
through to zfs.snapshot and reports the box healthy, while the runtime picks
pool.snapshot and fails every delete. Both sides now ask "can this namespace
delete?", and a test binds the two lists together.

3. query_filesystems() silently dropped malformed rows — the one remaining
silent-omission path, and a direct contradiction of the cardinal rule. It raises now.
A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed.

4. The retry loop discarded the delete error and reported every survivor as
"(still busy?)" — naming the one cause that is benign and hiding the ones that are
permanent. It keeps and reports the real error.

Also: the staging-failure handler could lose the original exception if its own sweep
raised; get_service is now a checked assumption; normalise_dataset and two dead
MiddlewareCall properties removed; stale comments corrected.

Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148
snapshots) and passed here only because this box has no zfs binary — they would have
gone red on the NAS, which is the one machine the release process requires them green
on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of
the five fixes above now fails the suite; before, all 293 passed.

Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset
backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4
(pool.snapshot.delete honours recursive=True).
2026-07-13 23:30:48 +00:00
flan 605231b39f feat: TrueNAS 26 support; enumerate datasets and snapshots from ZFS, not middleware
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 13s
CI / python 3.13 (push) Successful in 17s
CI / python 3.12 (push) Successful in 18s
TrueNAS compatibility / compat (push) Successful in 9s
TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query,
zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested
module's critical path, so nested snapshots were BROKEN on 26.

Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10
and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported
release. tools/compat.py checks the same list the runtime uses, so what CI verifies
and what runs cannot drift apart.

Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is
the point of this commit. Those methods exist, are documented, and are covered by
iX's deprecation policy — and they are not like-for-like replacements. They apply a
visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets
on a real pool, including live application data. Staging from that view omits them
silently, and plan_staging never sees them, so they do not even reach the skipped
list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one
per hidden dataset on every run.

So: read the truth from ZFS, make changes through middleware. zfs list cannot be
filtered by policy and behaves identically on every release. A failing zfs list raises
rather than returning an empty list — "no datasets" and "the command broke" must never
look the same.

No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query,
which returns all 270 datasets. The bug existed only in this port.

Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a
292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical
restore of a four-level-deep child dataset that pool.dataset.query hides.
2026-07-13 22:43:20 +00:00
flan 7cc0826c2c The matrix bot would never have worked: wrong permissions, wrong forge
CI / shell (shellcheck + syntax) (push) Successful in 9s
CI / python 3.11 (push) Successful in 13s
CI / python 3.13 (push) Successful in 16s
CI / python 3.12 (push) Successful in 16s
TrueNAS compatibility / compat (push) Failing after 6s
Release / release (push) Successful in 16s
Two bugs, both of which would have failed silently on the first scheduled run:

- permissions were  while the step pushes a branch and opens a PR.
  It would have died with a 403 and I would have had a bot that never worked.
- it opened the PR on GITHUB, which is a one-way MIRROR. A PR merged there would be
  clobbered by the next fleet-repos mirror push from Gitea. A bot opening PRs against
  a mirror is a bot doing nothing, slowly.

Now: contents+pull-requests write, and the PR is opened on Gitea (canonical) via its
API. One long-lived PR, force-pushed in place -- a daily PR is the same mistake as a
daily comment, wearing a hat.
2026-07-13 20:28:30 +00:00
flan 82084b6806 The bug-report bot was spamming; make it say something only when there is something to say
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 14s
CI / python 3.12 (push) Successful in 17s
CI / python 3.13 (push) Successful in 17s
TrueNAS compatibility / compat (push) Successful in 11s
It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.

Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.

The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.

Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
  implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
  whether to comment' is two chances to drift, and the Gitea one duplicated an issue
  for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
  main from CI. An unattended push to main is exactly what the release barrier exists
  to prevent: a bot that can move main can move it somewhere nobody looked.
2026-07-13 20:20:35 +00:00
flan c6b252ac6b release v0.6.1
CI / shell (shellcheck + syntax) (push) Successful in 11s
CI / python 3.11 (push) Successful in 13s
CI / python 3.12 (push) Successful in 14s
CI / python 3.13 (push) Successful in 13s
TrueNAS compatibility / compat (push) Successful in 11s
Release / release (push) Successful in 14s
2026-07-13 19:59:38 +00:00
flan 841e0364fd CHANGELOG: repair a section spliced into the middle of a bullet, and guard it
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 16s
CI / python 3.12 (push) Successful in 15s
CI / python 3.13 (push) Successful in 14s
An edit matched the literal '## Unreleased' inside a backticked phrase in a prose
bullet and spliced a whole new section into the middle of it, splitting the sentence
in half. The release body IS this file, so that would have shipped to every user.

Tests now assert: no empty version section, versions descend, no heading is indented
inside a list item, and every bullet's bold phrases are balanced (ignoring code spans
-- '*args, **kwargs' is a literal, not markup).
2026-07-13 19:59:35 +00:00
flan 0d04c2cd1c Collect orphaned snapshots by name: the sidecar lives in tmpfs
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 13s
CI / python 3.12 (push) Successful in 16s
CI / python 3.13 (push) Successful in 16s
A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record
of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the
recursive snapshot and its cleanup destroyed that record, leaving one snapshot per
descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would
ever have found them.

gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is
gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and
the collector only mops up what the record lost.

It deletes data on a name match, which is a weaker claim than a recorded fact, so the
selection is a pure function with the harshest tests here. A snapshot is collected
only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current
run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a
concurrent backup), and it is over an hour old.

Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it
selects exactly the orphans of the task being run and nothing else.
2026-07-13 19:56:32 +00:00
flan 2b8ef107f7 The sidecar must carry EVERY pending tree, not just the newest
CI / shell (shellcheck + syntax) (push) Successful in 9s
CI / python 3.12 (push) Successful in 13s
CI / python 3.13 (push) Successful in 17s
CI / python 3.11 (push) Successful in 15s
TrueNAS compatibility / compat (push) Successful in 11s
Release / release (push) Successful in 15s
Found live, in the code written to prevent exactly this.

The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to
finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of
the survivor -- orphaning it permanently.

Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46
reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it
behind again, and then wrote its own snapshot over the record. Permanent orphan,
created by the safety net.

The sidecar is now a list. stage_nested carries forward whatever a reclaim could not
delete; cleanup_task sweeps every pending tree and writes back only the survivors.
cleanup_all reports them one per line instead of formatting a list into an f-string
at the user during uninstall.

Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots
with no straggler.
2026-07-13 19:35:44 +00:00
flan b50567e9a7 A few snapshots leaked on every nested run, forever
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 12s
CI / python 3.12 (push) Successful in 15s
CI / python 3.13 (push) Successful in 17s
TrueNAS compatibility / compat (push) Successful in 13s
Release / release (push) Successful in 14s
Found on real hardware: a 256-snapshot backup of /mnt/Tap swept 253 and left 3 with
'dataset is busy'.

ZFS AUTOMOUNTS <dataset>/.zfs/snapshot/<snap> when it is read, and keeps it mounted
for zfs_expire_snapshot seconds (300 default) after the last access. teardown()
unmounts OUR bind mounts but not the automount underneath, so zfs destroy refuses for
exactly the datasets restic read most recently. cleanup_task() then removed the
sidecar anyway -- destroying the only record those snapshots existed. Nothing would
ever have reclaimed them.

- release_snapdirs() unmounts ZFS's own automounts (deepest first) before deleting.
- delete_snapshot_tree() retries the transient busy and RETURNS what it could not
  delete, instead of swallowing it.
- The sidecar is removed only on a confirmed-clean sweep -- including on the
  staging-failure path, which used to remove it before the caller swept. A sidecar
  left behind when the tree is gone costs one no-op delete; a sidecar removed while
  the tree exists is unrecoverable.
2026-07-13 19:18:22 +00:00
flan 1b2407f6e2 compat: dedup the bug report deterministically (lowest issue number wins)
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 16s
CI / python 3.12 (push) Successful in 18s
CI / python 3.13 (push) Successful in 18s
TrueNAS compatibility / compat (push) Successful in 10s
Two issues with the same title already existed -- the old title embedded the list of
broken refs, so the issue's identity changed whenever that set changed. With an
order-dependent pick the bot would alternate between them, reopening one and
commenting on the other. Lowest number is stable regardless of how the API sorts.
2026-07-13 19:08:27 +00:00
flan ab0b66c47d compat: the bug-report title must be stable across ref-set changes
CI / shell (shellcheck + syntax) (push) Successful in 10s
CI / python 3.11 (push) Successful in 15s
CI / python 3.12 (push) Successful in 17s
CI / python 3.13 (push) Successful in 18s
TrueNAS compatibility / compat (push) Successful in 10s
The title embedded the list of broken refs, so the issue's identity changed whenever
that set changed -- and it did: when the async/sync port briefly made 26 look green,
the next run filed a SECOND issue for 'master' alone. A bot that spawns duplicates
gets muted, and then it is not a warning system any more.

The title is now fixed; the refs live in the body, which gets updated in place.
2026-07-13 19:02:57 +00:00
flan 469a2e4651 Installing the patch permanently blocked updating it
CI / shell (shellcheck + syntax) (push) Successful in 9s
CI / python 3.11 (push) Successful in 14s
CI / python 3.12 (push) Successful in 16s
CI / python 3.13 (push) Successful in 17s
TrueNAS compatibility / compat (push) Successful in 11s
Release / release (push) Successful in 14s
install.sh chmod +x's update.sh, and git recorded update.sh as 100644 -- so the chmod
was a TRACKED modification, and update.sh refuses to run over a dirty tree. Install
once and you could never update again. The error even told you to 'git checkout -- .',
which just undoes the exec bit so the next install can re-dirty it.

Found on the real box, which had been sitting on v0.4.1 for exactly this reason.

Fixed on both sides: the scripts install.sh chmods are executable in git (so the
chmod is a no-op), and update.sh's dirty check now looks at CONTENT, not mode --
git diff --numstat reports 0 0 for a mode-only change. A test asserts every script in
install.sh's chmod loop is already 100755 in git.
2026-07-13 18:58:46 +00:00
flan 518a22d87e docs: user-facing URLs point at GitHub, the user-facing repo
CI / shell (shellcheck + syntax) (push) Successful in 8s
CI / python 3.11 (push) Successful in 13s
CI / python 3.12 (push) Successful in 14s
CI / python 3.13 (push) Successful in 15s
TrueNAS compatibility / compat (push) Failing after 6s
Release / release (push) Successful in 14s
Gitea is canonical for development; GitHub is where users clone from and where the
box's read-only checkout points. The install instructions, the re-clone hint and the
'file an issue' link are all read by users, so they name GitHub. docs/releasing.md
still names Gitea, because that is a contributor doc about where the code is pushed.
2026-07-13 18:42:44 +00:00
23 changed files with 3975 additions and 329 deletions
+83 -75
View File
@@ -27,7 +27,10 @@ on:
- ".github/workflows/compat.yml"
permissions:
contents: read
# write, because the matrix refresh pushes a branch and opens a PR. It does NOT get
# to move `main` -- that is the whole reason it is a PR. See the refresh step below.
contents: write
pull-requests: write
issues: write
jobs:
@@ -130,17 +133,27 @@ jobs:
- name: matrix
run: cat /tmp/matrix.md
# Keep the README's table true. A support matrix that quietly goes stale is not
# a stale doc -- it is a false promise to somebody deciding whether to trust
# this with their backups.
# Keep the README's table true — as a PULL REQUEST, on the CANONICAL forge.
#
# Only ever touches the block between the COMPAT MATRIX markers, and only on
# the canonical host (Gitea) so the two forges cannot race each other. The
# `paths:` trigger above does not include README.md, so this cannot re-trigger
# itself; and a README change is documentation-only, which by design raises no
# update alert on anyone's box.
- name: refresh the README matrix
# Two things this gets right that the obvious version gets wrong:
#
# 1. It is a PR, not a push to main. This used to `git push origin HEAD:main`
# from CI. An unattended write to main is exactly what the release barrier
# exists to prevent — a bot that can move main can move it somewhere nobody
# looked. Nothing lands by itself.
#
# 2. It runs on GITEA, not GitHub. GitHub is a one-way MIRROR: a PR merged there
# would be silently clobbered by the next `fleet-repos mirror` push from Gitea.
# A bot opening PRs against a mirror is a bot doing nothing, slowly.
#
# A stale support matrix is not a stale doc — it is a false promise to somebody
# deciding whether to trust this with their backups. So it is refreshed daily; it
# just asks first.
- name: refresh the README matrix (PR on the canonical forge)
if: ${{ github.event_name == 'schedule' && !contains(github.server_url, 'github.com') }}
env:
TOKEN: ${{ secrets.GITEA_TOKEN || github.token }}
API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
run: |
python3 - <<'PY'
import json, sys
@@ -151,78 +164,73 @@ jobs:
print("changed" if compat.update_readme(rows) else "unchanged")
PY
if ! git diff --quiet -- README.md; then
git diff --quiet -- README.md && { echo "matrix unchanged — nothing to propose"; exit 0; }
git config user.name "truecloud-patch bot"
git config user.email "bot@onetick.ninja"
BRANCH=bot/compat-matrix
git checkout -B "$BRANCH"
git add README.md
git commit -m "docs: refresh the TrueNAS compatibility matrix"
git push origin HEAD:main
fi
git push -f origin "$BRANCH"
# A broken SHIPPED release is an outage: users are on it right now.
# ONE long-lived PR, force-pushed in place — not a new one every morning.
# (A daily PR is the same mistake as a daily comment, wearing a hat.)
BRANCH="$BRANCH" python3 - <<'PY'
import json, os, urllib.error, urllib.request
api, token, branch = os.environ["API"], os.environ["TOKEN"], os.environ["BRANCH"]
h = {"Authorization": f"token {token}", "Content-Type": "application/json"}
def call(url, method="GET", data=None):
r = urllib.request.Request(
url, method=method, headers=h,
data=json.dumps(data).encode() if data else None)
with urllib.request.urlopen(r) as resp: # noqa: S310
return json.load(resp) if resp.length != 0 else {}
existing = [
p for p in call(f"{api}/pulls?state=open")
if p["head"]["ref"] == branch
]
if existing:
print(f"PR #{existing[0]['number']} already open; the force-push updated it")
else:
pr = call(f"{api}/pulls", "POST", {
"head": branch, "base": "main",
"title": "docs: refresh the TrueNAS compatibility matrix",
"body": (
"The daily compatibility check found that the support matrix in "
"the README no longer matches iXsystems' actual middleware.\n\n"
"This only touches the block between the `COMPAT MATRIX` markers. "
"It is regenerated by `tools/compat.py --matrix --update-readme` "
"and force-pushed, so it always reflects the latest run."
),
})
print(f"opened PR #{pr['number']}")
PY
# ONE bug report, kept in sync. It is edited in place when the findings change and
# says NOTHING when they do not.
#
# The first version commented on every run and left 11 identical 3,000-character
# comments on one issue in a single day. A bot that repeats itself daily gets
# muted, and then the next real finding is scrolled past — which defeats the whole
# reason for building it.
#
# Runs on whichever forge it lands on; compat_publish.py handles both, so the two
# cannot drift.
- name: file / update / close the bug report
env:
TOKEN: ${{ secrets.GITEA_TOKEN || github.token }}
API: ${{ contains(github.server_url, 'github.com') && 'https://api.github.com' || format('{0}/api/v1', github.server_url) }}/repos/${{ github.repository }}
run: python3 tools/compat_publish.py --api "$API" --token "$TOKEN" --matrix /tmp/matrix.json
# A broken SHIPPED release is an outage: users are on it right now. Fails LAST, so
# the report is filed before the job goes red.
- name: fail if a shipped release is broken
if: ${{ steps.check.outputs.shipped_broken != '0' }}
run: |
echo "::error::The patch is broken on a SHIPPED TrueNAS release."
exit 1
- name: file a bug report (GitHub)
if: ${{ steps.report.outputs.broken == '1' && contains(github.server_url, 'github.com') }}
env:
GH_TOKEN: ${{ github.token }}
TITLE: "Incompatible with upcoming TrueNAS: ${{ steps.report.outputs.refs }}"
run: |
# One issue per set of broken refs, reopened/updated rather than duplicated
# daily -- a bot that files the same issue every morning gets muted, and
# then it is not a warning system any more.
existing="$(gh issue list --state all --search "$TITLE" \
--json number,title \
--jq '.[] | select(.title == env.TITLE) | .number' | head -1)"
if [ -n "$existing" ]; then
gh issue comment "$existing" --body-file /tmp/issue.md
gh issue reopen "$existing" 2>/dev/null || true
else
gh issue create --title "$TITLE" --body-file /tmp/issue.md
fi
- name: file a bug report (Gitea)
if: ${{ steps.report.outputs.broken == '1' && !contains(github.server_url, 'github.com') }}
env:
TOKEN: ${{ secrets.GITEA_TOKEN || github.token }}
API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
TITLE: "Incompatible with upcoming TrueNAS: ${{ steps.report.outputs.refs }}"
run: |
# python3, not jq: jq is not guaranteed on a self-hosted runner, and a bug
# report that dies on a missing tool is a warning system that does not warn.
python3 - <<'PY'
import json, os, urllib.error, urllib.request
api, token, title = os.environ["API"], os.environ["TOKEN"], os.environ["TITLE"]
with open("/tmp/issue.md", encoding="utf-8") as fh:
body = fh.read()
headers = {"Authorization": f"token {token}",
"Content-Type": "application/json"}
def call(url, method, data=None):
req = urllib.request.Request(
url, method=method, headers=headers,
data=json.dumps(data).encode() if data else None)
with urllib.request.urlopen(req) as r: # noqa: S310
return json.load(r) if r.length != 0 else {}
# Same title => same issue. Comment on it rather than filing a new one every
# morning: a bot that duplicates itself daily gets muted, and then it is not
# a warning system any more.
issues = call(f"{api}/issues?state=all&type=issues", "GET")
match = next((i for i in issues if i["title"] == title), None)
if match:
n = match["number"]
call(f"{api}/issues/{n}/comments", "POST", {"body": body})
call(f"{api}/issues/{n}", "PATCH", {"state": "open"})
print(f"commented on and reopened issue #{n}")
else:
made = call(f"{api}/issues", "POST", {"title": title, "body": body})
print(f"filed issue #{made['number']}")
PY
+195
View File
@@ -6,6 +6,157 @@ is deliberate: see [Releasing](docs/releasing.md). Twelve releases were cut on
live, every one of those interrupts every user. An alert people learn to ignore is
worse than no alert, because one day it carries a security fix.
## v0.7.0 — 2026-07-14
### Added
- **TrueNAS 26 support, verified on a real TrueNAS 26 install.** 26 deletes
`plugins/zfs_/` outright, taking the private `zfs.dataset.query`,
`zfs.snapshot.query` and `zfs.snapshot.delete` with it. Every one of those was on
the nested module's critical path, so nested snapshots were **BROKEN** on 26 and
`apply.sh` correctly refused to apply the module there.
Snapshot **deletion** now resolves its namespace at runtime — `pool.snapshot` on
25.10 and 26, `zfs.snapshot` on 24.10 and 25.04, because no single namespace spans
every supported release. `tools/compat.py` checks the same list the runtime uses,
so what CI verifies and what runs cannot drift apart.
Hardware-verified on TrueNAS 26.0.0-BETA.1: a 274-snapshot recursive backup of a
292-dataset pool, then a **byte-identical restore of a four-level-deep child
dataset**.
### Fixed
- **Enumeration no longer trusts middleware's dataset and snapshot queries — they
are filtered.** This is the important one, and it is the bug that a test VM caught
and no amount of source analysis ever could have.
The obvious port of the deleted private `zfs.dataset.query` was the public
`pool.dataset.query`. It exists, it is documented, it is covered by iX's
deprecation policy — and it is **not a like-for-like replacement**. It applies a
*visibility policy*: it hides the datasets TrueNAS considers its own — `ix-apps/*`,
`.system/*`, `.ix-virt/*`. On a real pool that is **84 of 270 datasets**, and
`ix-apps` holds **live application data**.
Staging from that view would have silently omitted every one of them. Worse,
`plan_staging()` would never have seen them, so they would not have appeared in its
`skipped` list either — no warning, no failure, just a green backup quietly missing
data. That is precisely the failure this module exists to prevent. The snapshot
query lies the same way (205 of 274), so the sweep would have orphaned one snapshot
per hidden dataset, on every run, forever.
The module now **reads the truth from ZFS and makes changes through middleware**:
enumeration is `zfs list`, which no policy can filter and which behaves identically
on every release; mutation stays a middleware call, so TrueNAS's own bookkeeping
stays consistent. A failing `zfs list` raises rather than returning an empty list —
"no datasets" and "the command broke" must never look the same.
**No shipped release is affected.** v0.6.1 and earlier call the *private*
`zfs.dataset.query`, which returns all 270 datasets. The bug existed only in the
unreleased TrueNAS 26 port.
- **The patch now owns the snapshot sweep even when it does not stage anything.**
Stock decides whether to take a *recursive* snapshot by its own rule, and on
TrueNAS 26 that rule stopped being ours.
Up to 25.10, stock's `create_snapshot` called `get_dataset_recursive()` — the same
function this module vendors — so "stock went recursive" and "we have something to
stage" were the *same question*, and stock's non-recursive delete was correct for
everything the patch declined to stage. TrueNAS 26 uses `filesystem.statfs`:
`recursive = (path == the dataset's mountpoint)`. The two rules now disagree for a
dataset whose only descendants are **ZVOLs** or **legacy/none-mountpoint** datasets
— stock snapshots it recursively, while the patch sees nothing to stage.
The patch then handed the snapshot back to stock, which destroys the parent only.
With no staging tree there was no sidecar, and the garbage collector only ever ran
from the staging path — so nothing on the box would ever have found the children.
Reproduced on the test VM: one orphaned snapshot per zvol, on every run, forever,
with the backup reporting success. Ownership of the sweep is no longer conditional
on staging.
- **The runtime resolved a *namespace*; the checker verified a *method*.** Those are
different questions, and the gap is a false "ok". `get_service()` only proves a
namespace is registered — it says nothing about whether `delete` still exists on it.
So if iX guts the method while keeping the service (they have already done exactly
that to `pool.snapshot.do_update` on master), `tools/compat.py` would fall through
to `zfs.snapshot`, report the box healthy, and let the patch apply — while the
runtime picked `pool.snapshot` and failed *every* delete, orphaning the whole tree.
Both sides now ask the same question, and a test binds the two lists together.
- `query_filesystems()` **dropped malformed `zfs list` rows silently** — the last
remaining silent-omission path, and a direct contradiction of this module's cardinal
rule. It raises now. A missing `zfs` binary raised `FileNotFoundError` rather than
`ZfsError`; also fixed.
- The snapshot retry loop **discarded the delete error** and reported every survivor
as "(still busy?)" — naming the one cause that is benign and self-healing, and
hiding the ones that are permanent. It keeps and reports the real error.
- The staging-failure handler could **lose the original exception** if its own cleanup
sweep raised. An error handler must not be able to lose the error.
- **A snapshot delete that returns cleanly is not proof that anything was deleted.**
The recursive sweep's fast path took the call's word for it and returned "no
survivors" — so `cleanup_task` read that as a clean sweep and removed the sidecar,
the only record the tree ever existed. Roughly 250 snapshots would have been orphaned
on every run, with nothing left able to find them, and the backup reporting success.
This is not a hypothetical about a well-behaved API: iX has already gutted
`pool.snapshot.do_update` on master into a no-op whose body is commented out and
which returns `None`. A source check still sees the `def`; a runtime check still sees
a callable method. Only asking ZFS can tell. The sweep now confirms against ZFS, and
where it *cannot* confirm it keeps owning the tree rather than claiming success — a
false survivor self-heals on the next run, a lost record never does.
- `_write_sidecar` **swallowed `OSError`**. The sidecar is the only thing that survives
a middlewared restart; failing to write it is not fatal, but it must never be
invisible. `_read_sidecar` had the mirror bug — it conflated "there is no sidecar"
with "I could not read the sidecar", and `cleanup_task` then took the empty branch
and **unlinked the only record** of a tree it had failed to read.
- **A dataset from another tree, mounted inside the backup path, was omitted
silently.** The staging plan scopes by dataset *name*, which is correct — a dataset
with no mountpoint cannot be scoped by path at all. But ZFS lets any dataset mount
anywhere, so one from an unrelated tree can sit inside the path:
Tank/photos mountpoint=/mnt/Tap/apps/photos
It holds data inside the backed-up path, and `zfs snapshot -r Tap@…` does **not**
cover it: recursion follows the dataset tree, not the directory tree. So there is no
snapshot of it to stage, and no way to capture it consistently with the rest. It fell
out of the name filter and vanished — not staged, not in `skipped`, no error, backup
green. Stock has the same blind spot, but stock also refuses the nested config
outright; this patch is what relaxes that guard, so the hole is this patch's to close.
It now refuses, and names the offending datasets.
## v0.6.1 — 2026-07-13
### Fixed
- **A reboot mid-backup orphaned the entire snapshot tree, permanently.** The sidecar
is the record of which snapshots a run pinned — and it lives in `/run`, which is
**tmpfs**. A reboot (or a crash) between taking the recursive snapshot and cleaning
it up destroyed that record, leaving one snapshot per descendant dataset — **250+ on
a real pool** — with nothing left pointing at them. Nothing would ever have found
them again.
`gc_stale_snapshots()` is the backstop: it identifies leftovers **by name**, so it
works when the record is gone. It runs at the start of every backup, after the
sidecar reclaim — the recorded path stays authoritative, and the collector only ever
mops up what the record lost.
Because it deletes data on a *name match* — a weaker claim than a recorded fact — the
selection is a **pure function** with the harshest tests in the suite. A snapshot is
collected only if **all** of these hold:
| | |
| --- | --- |
| name is exactly `<dataset>@<task>-<YYYYMMDDHHMMSS>` | so `cloud_backup-5` never matches `cloud_backup-50`, an `auto-*` periodic snapshot, or anything a human made |
| it is not the current run's | parent *and* children are excluded |
| **nothing is mounted from it** | an in-flight run pins its own snapshots — this, not the age guard, is what protects a concurrent backup |
| it is **over an hour old** | covers the seconds-long window where a live run has snapshotted but not yet mounted |
Verified against the real pool: of **4,728** snapshots — including **2,341** periodic
ones — it selects exactly the orphans of the task being run, and nothing else.
## v0.6.0 — 2026-07-13
### Added
@@ -66,6 +217,50 @@ worse than no alert, because one day it carries a security fix.
### Fixed
- **A few snapshots leaked on every nested run, forever.** Found on real hardware, in
the one place it could be: a 256-snapshot backup of `/mnt/Tap` swept 253 cleanly and
left **3 behind** with `dataset is busy`.
The cause is ZFS's own automount. Reading anything under
`<dataset>/.zfs/snapshot/<snap>/` makes ZFS **automount that snapshot**, and it stays
mounted for `zfs_expire_snapshot` seconds (**300** by default) after the last access.
`teardown()` unmounts *our* bind mounts — but not the automount underneath — so
`zfs destroy` refuses for exactly the datasets restic read most recently. Then
`cleanup_task()` removed the sidecar anyway, destroying the only record that those
snapshots existed. Nothing would ever have reclaimed them.
Three changes, and the third is the one that makes it safe rather than merely
unlikely:
- `release_snapdirs()` unmounts ZFS's own `.zfs/snapshot` automounts (deepest first)
before deleting, so the snapshots are not busy in the first place.
- `delete_snapshot_tree()` **retries** the transient busy, and **returns the
snapshots it could not delete** instead of swallowing them.
- **The sidecar is now removed only on a confirmed-clean sweep** — including on the
staging-failure path, which used to remove it *before* the caller swept. The
asymmetry is deliberate: a sidecar left behind when the tree is already gone costs
one no-op delete on the next run, while a sidecar removed while the tree still
exists is unrecoverable. Survivors are reclaimed by the next run.
**Expect the occasional straggler, and expect it to clean itself up.** On a
256-snapshot tree this reliably sweeps ~255 immediately and may leave **one**: it is
whatever restic read last, so its 300-second window has barely opened. That one is
logged, its sidecar is kept, and the next run reclaims it before doing anything else.
The leak is bounded at a single cycle rather than growing without limit — which is
the property that actually matters. Blocking a backup job for five minutes to chase
the last snapshot would be a worse trade, so it is not made.
- **Installing the patch permanently blocked updating it.** `install.sh` does
`chmod +x update.sh`, and git recorded `update.sh` as `100644` — so the chmod was a
*tracked modification*, and `update.sh` refuses to run over a dirty tree. Install
once and you could never update again; the error even told you to run
`git checkout -- .`, which just undoes the exec bit so the next install can re-dirty
it. A real box sat on an old version for exactly this reason.
Fixed on both sides: the scripts `install.sh` chmods are now executable in git (so
the chmod is a no-op), and `update.sh`'s dirty check now looks at **content**, not
file mode — `git diff --numstat` reports `0 0` for a mode-only change. A test
asserts every script in `install.sh`'s chmod loop is already `100755` in git.
- **Nested snapshots were broken on TrueNAS 24.10 and 25.04, and had been all
along.** `SYNC_BLOCK`'s wrapper spelled out the stock signature and forwarded five
arguments — but those releases declare `restic_backup(middleware, job,
+8 -2
View File
@@ -22,7 +22,7 @@ Clone it onto a **pool** (not the boot device — that is wiped on TrueNAS upgra
then run `install.sh` as root:
```bash
git clone https://git.onetick.ninja/flan/truenas-truecloud-patch.git \
git clone https://github.com/sudolulo/truenas-truecloud-patch.git \
/mnt/tank/truenas-truecloud-patch # replace `tank` with your pool
cd /mnt/tank/truenas-truecloud-patch
sudo bash install.sh
@@ -60,7 +60,7 @@ If something is wrong, the reason is in `apply.log` — start at
| 24.10.2.4 | ok | ok | — |
| 25.04.2.6 | ok | ok | — |
| 25.10.4 | ok | ok | nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m |
| 26.0.0-BETA.3 _(unreleased)_ | ok | **BROKEN** | — |
| 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — |
| master _(unreleased)_ | **BROKEN** | **BROKEN** | — |
| verdict | meaning |
@@ -77,6 +77,12 @@ source. It does not mean a human ran a backup on it — that is the
The table is **regenerated daily by CI** against iXsystems' actual middleware source
— it is not a claim somebody typed once and forgot.
**On TrueNAS 26:** the patch was run on a real TrueNAS **26.0.0-BETA.1** install — a
274-snapshot recursive backup of a 292-dataset pool, followed by a byte-identical
restore of a four-level-deep child dataset. The *Hardware-verified* column tracks the
newest beta iX has tagged (currently BETA.3), so it does not carry that mark: a build
nobody has actually run a backup on does not get credit for one.
**TrueNAS 26: nested snapshots are not supported yet, and upgrading will not break
you.** 26 rewrites `cloud_backup` and deletes the ZFS methods this module calls. On
26 `apply.sh` finds that the assumptions no longer hold and **does not apply the
+1 -1
View File
@@ -158,7 +158,7 @@ python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
| What you see | What it means |
|---|---|
| `[OK] providers`, `[OK]`/`[SKIP] nested_snapshots` | Fine. Nothing to do. |
| `WARNING: … pattern not found` (UI) | The Angular bundle changed. The UI dropdown reverts to Storj-only, but **backups keep working** — create tasks with `create_task.py` meanwhile, and [open an issue](https://git.onetick.ninja/flan/truenas-truecloud-patch/issues) with your TrueNAS version. |
| `WARNING: … pattern not found` (UI) | The Angular bundle changed. The UI dropdown reverts to Storj-only, but **backups keep working** — create tasks with `create_task.py` meanwhile, and [open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues) with your TrueNAS version. |
| `WARNING: truecloud-patch is NOT COMPATIBLE with this TrueNAS version` | This TrueNAS changed middleware underneath the patch, and the named module was **deliberately not applied** — see `incompatible.json` for exactly which assumption broke. TrueNAS is left stock, so nothing is half-patched. Check [TrueNAS compatibility](../README.md#truenas-compatibility), then `bash update.sh` once a release supports your version; it re-applies itself on the next boot. This is **not** the kill switch and needs no manual reset. |
| `[FAIL] providers` | **Your B2/S3 backups will not run.** middlewared is fine, but the credential/URL handling is gone. Open an issue with your version. |
| `[FAIL] nested_snapshots` | The stock guard is back, so tasks with `snapshot = true` on a nested dataset will fail validation. Turn the option off on those tasks until it's fixed. |
+34
View File
@@ -95,6 +95,40 @@ re-scan each time.
### Snapshot lifecycle
> **Two mechanisms clean up, and the second exists because the first can be destroyed.**
>
> 1. **The sidecar** records exactly which snapshots a run pinned, and is removed only
> on a confirmed-clean sweep. Precise, and it survives a middlewared restart.
> 2. **The garbage collector** finds leftovers by *name*, so it still works when the
> sidecar is gone — and it can be: **the sidecar lives in `/run`, which is tmpfs.** A
> reboot mid-backup takes it, and with it the only record of a 250-snapshot tree.
>
> The collector runs at the start of every backup, after the sidecar reclaim. It will
> only touch a snapshot named `<dataset>@<task>-<timestamp>` that is not the current
> run's, has **nothing mounted from it** (which is what protects a concurrently-running
> backup), and is **over an hour old**. Periodic `auto-*` snapshots, other tasks'
> snapshots, and anything you made by hand are structurally out of reach.
> **A snapshot may survive a run, and that is expected.** ZFS **automounts**
> `<dataset>/.zfs/snapshot/<snap>` the moment it is read, and holds it for
> `zfs_expire_snapshot` seconds (**300** by default) after the last access. So
> whatever restic read *last* is still pinned when we try to destroy it, and
> `zfs destroy` refuses with `dataset is busy`.
>
> The patch unmounts those automounts itself and retries, which clears ~255 of 256 on
> a real pool. The one that remains is **logged, its sidecar is kept, and the next run
> reclaims it before doing anything else** — so the leak is bounded at a single cycle
> instead of growing forever. Seeing one `could not delete snapshot … it will be
> reclaimed on the next run` in the log is normal. Seeing the count *grow* run over run
> is not, and would be a bug.
>
> This is why the sidecar is removed **only on a confirmed-clean sweep**: it is the
> only record those snapshots exist, and a run that dropped it while they were still
> around would orphan them permanently. That is precisely what happened before this was
> fixed.
`zfs.snapshot.delete` defaults to **`recursive=False`**, and stock
`restic_backup()` calls it with no options. Stock is safe only because its
validation means a *recursive* snapshot never actually happens in the field.
+1 -1
View File
@@ -109,7 +109,7 @@ If a module shows `[FAIL]`:
risk.
4. **If the detail says the module doesn't exist**, a TrueNAS update renamed
or restructured the internal API.
[Open an issue](https://git.onetick.ninja/flan/truenas-truecloud-patch/issues)
[Open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues)
with your TrueNAS version number and the full verify output.
---
+3 -3
View File
@@ -4,7 +4,7 @@
# Prerequisites: run as root on TrueNAS SCALE with middlewared running.
# Clone this repository to a persistent ZFS pool first:
#
# git clone https://git.onetick.ninja/flan/truenas-truecloud-patch \
# git clone https://github.com/sudolulo/truenas-truecloud-patch \
# /mnt/<pool>/truenas-truecloud-patch
# cd /mnt/<pool>/truenas-truecloud-patch && bash install.sh
#
@@ -18,7 +18,7 @@
set -euo pipefail
VERSION="0.6.0"
VERSION="0.7.0"
# The directory containing install.sh is the permanent install location.
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -72,7 +72,7 @@ done
if [ ! -f "$PATCH_DIR/patch/apply.sh" ]; then
echo "ERROR: patch files not found at $PATCH_DIR/patch/" >&2
echo "Run install.sh from a clone of the repository on a persistent pool:" >&2
echo " git clone https://git.onetick.ninja/flan/truenas-truecloud-patch \\" >&2
echo " git clone https://github.com/sudolulo/truenas-truecloud-patch \\" >&2
echo " /mnt/<pool>/truenas-truecloud-patch" >&2
echo " cd /mnt/<pool>/truenas-truecloud-patch && bash install.sh" >&2
exit 1
+31 -9
View File
@@ -32,7 +32,7 @@
# Derive PATCH_DIR from this script's location (parent of the patch/ directory).
PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
LOG="$PATCH_DIR/apply.log"
VERSION="0.6.0"
VERSION="0.7.0"
# Rotate log at 512 KB to avoid unbounded growth on a system volume.
# Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -535,8 +535,8 @@ if _tc_nested is not None:
# snapshot=true, not just ours. Two consequences, and the second is worse:
#
# * everything below is a NEW failure mode for tasks that worked before we
# were installed. A `zfs.dataset.query` that errors would break a
# CloudSync job we have no business touching.
# were installed. A `zfs list` that errors would break a CloudSync job
# we have no business touching.
# * if a CloudSync task ever were staged, nothing would ever tear it down:
# the teardown is wired into cloud_backup's restic_backup finally, and
# CRUD_BLOCK deliberately leaves CloudSync's nesting guard intact. The
@@ -556,18 +556,28 @@ if _tc_nested is not None:
# our staging plan would not -- silently omitting it from the backup.
# Read afterwards, an unsnapshotted dataset instead trips the isdir()
# check in plan_staging and fails the run loudly. Loud beats silent.
datasets = middleware.call_sync(
"zfs.dataset.query", [["type", "=", "FILESYSTEM"]]
)
# query_filesystems() reads ZFS directly. It deliberately does NOT use
# pool.dataset.query: that applies a visibility policy and hides
# TrueNAS-internal datasets (ix-apps/*, .system/*, .ix-virt/*) -- 84 of
# 270 on a real pool, including live app data. Staging from the filtered
# view omits them silently, which is the one thing this must never do.
datasets = _tc_nested.query_filesystems(middleware)
# OUR copy of get_dataset_recursive, not the host module's: TrueNAS 26
# deleted that helper (create_snapshot uses filesystem.statfs now), so
# calling it out of the module namespace is a NameError there.
dataset, nested = _tc_nested.get_dataset_recursive(datasets, path)
if not nested:
# No children: stock behaviour, untouched. Stock's `finally` owns
# the snapshot from here (its non-recursive delete is correct,
# because a non-nested snapshot has no children).
# Nothing to STAGE -- but we still own the SWEEP, and that is not a
# formality. Stock decides `recursive` by its own rule, and on 26 that
# rule is no longer ours: it snapshots recursively whenever the backup
# path IS the dataset's mountpoint (filesystem.statfs), while
# get_dataset_recursive() sees nothing to stage when the only
# descendants are ZVOLs or legacy/none-mountpoint datasets. Stock then
# deletes the PARENT ONLY. Without this, one snapshot per descendant is
# orphaned on every run, forever, with no sidecar and no GC to find it --
# and the backup still reports success.
_tc_nested.own_snapshot(middleware, name, snapshot, logger=_logger)
return snapshot, snap_path
staging_root = _tc_nested.stage_nested(
@@ -581,7 +591,19 @@ if _tc_nested is not None:
# stays None and its `finally` deletes NOTHING. Sweep the tree ourselves
# or leak the parent plus one snapshot per descendant dataset (160+ here)
# on every failed run.
#
# The sweep is itself wrapped: a cleanup that raises would REPLACE the
# original exception with its own, hiding why the backup actually failed.
# An error handler must not be able to lose the error.
try:
_tc_nested.delete_snapshot_tree(middleware, snapshot, logger=_logger)
except Exception as _tc_sweep_err:
if _logger:
_logger.error(
"truecloud-patch: could not sweep %s after a staging failure "
"(%r) -- it is orphaned and must be deleted by hand",
snapshot, _tc_sweep_err,
)
raise
return snapshot, staging_root
+1 -1
View File
@@ -52,7 +52,7 @@ import subprocess
import sys
import time
__version__ = "0.6.0"
__version__ = "0.7.0"
_PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
_STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")
+3 -3
View File
@@ -111,7 +111,7 @@ def main():
print(
"[truecloud-patch] WARNING: filterByProviders pattern not found in any JS bundle.\n"
"[truecloud-patch] The TrueNAS webui may have been restructured in this version.\n"
"[truecloud-patch] File an issue at https://git.onetick.ninja/flan/truenas-truecloud-patch\n"
"[truecloud-patch] File an issue at https://github.com/sudolulo/truenas-truecloud-patch\n"
f"[truecloud-patch] TrueNAS version info: {_tnversion()}"
)
return
@@ -136,7 +136,7 @@ def main():
f"[truecloud-patch] WARNING: {count} replacement(s) in {path}; "
f"expected exactly 1 — skipping write to avoid corrupting the bundle.\n"
f"[truecloud-patch] File an issue at "
f"https://git.onetick.ninja/flan/truenas-truecloud-patch"
f"https://github.com/sudolulo/truenas-truecloud-patch"
)
return
@@ -154,7 +154,7 @@ def main():
"[truecloud-patch] The UI is UNCHANGED and still works. This means the "
"pattern no longer fits this TrueNAS build.\n"
"[truecloud-patch] File an issue at "
"https://git.onetick.ninja/flan/truenas-truecloud-patch"
"https://github.com/sudolulo/truenas-truecloud-patch"
)
return
+922 -62
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -17,7 +17,7 @@
# bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
# systemctl restart middlewared
VERSION="0.6.0"
VERSION="0.7.0"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
Regular → Executable
View File
+322 -14
View File
@@ -140,19 +140,60 @@ class TestSnapshotLeak:
every path that creates one must also sweep the whole tree.
"""
def test_staging_failure_deletes_the_snapshot_tree(self):
# On a staging failure, sync.py's `snapshot, local_path = await
# create_snapshot(...)` never completes, so its local `snapshot` stays
# None and its finally deletes nothing. We must sweep it ourselves.
block = extract_blocks()["SNAPSHOT_ASYNC"]
assert "except Exception:" in block
assert "delete_snapshot_tree" in block
assert "raise" in block
# The behaviour these once asserted as substrings -- the sweep, the re-raise, the
# teardown in the finally -- is now asserted STRUCTURALLY, against the parsed
# block: see TestTheStagingFailurePathReallyReRaises and
# TestTheSyncBlockAlwaysTearsDown. As substring checks they were satisfied by
# COMMENTS ("a cleanup that raises...", "cleanup_task gets logger=None"), so
# deleting the actual `raise` and the actual cleanup call both left the suite
# green -- reinstating a silently-empty backup and ~250 orphans per run.
def test_sync_block_cleans_up_on_every_path(self):
block = extract_blocks()["SYNC_ASYNC"]
assert "finally:" in block
assert "cleanup_task" in block
def test_the_snapshot_block_still_owns_the_snapshot_when_not_staging(self):
# The TrueNAS 26 zvol/legacy orphan: stock decides `recursive` by its own rule
# (path == mountpoint) and deletes only the parent, so we must record the
# snapshot even on the path where we stage nothing.
for name in ("SNAPSHOT_ASYNC", "SNAPSHOT_SYNC"):
stage = functions(tree_of(name), "_tc_stage")[0]
assert calls_to(stage, "_tc_nested.own_snapshot"), (
f"{name} hands an unstaged snapshot back to stock, whose delete is "
f"non-recursive -- every zvol/legacy child is orphaned, every run"
)
def test_the_staging_plan_is_enumerated_from_ZFS(self):
for name in ("SNAPSHOT_ASYNC", "SNAPSHOT_SYNC"):
stage = functions(tree_of(name), "_tc_stage")[0]
assert calls_to(stage, "_tc_nested.query_filesystems"), (
"the staging plan must come from query_filesystems() (which reads ZFS "
"unfiltered); middleware's query hides ix-apps/*, .system/*, .ix-virt/*"
)
assert not calls_to(stage, "middleware.call_sync"), (
"the block calls middleware directly again -- its dataset/snapshot "
"queries are FILTERED and silently omit 84 of 270 datasets"
)
def test_the_vendored_helper_is_used_not_the_host_module(self):
# TrueNAS 26 DELETED get_dataset_recursive from plugins/cloud/snapshot.py, so
# calling it out of the host module's namespace is a NameError there.
for name in ("SNAPSHOT_ASYNC", "SNAPSHOT_SYNC"):
stage = functions(tree_of(name), "_tc_stage")[0]
assert calls_to(stage, "_tc_nested.get_dataset_recursive"), (
"must call OUR vendored copy: TrueNAS 26 deleted the host's"
)
def test_datasets_are_enumerated_AFTER_the_snapshot(self):
# A dataset created between the listing and the snapshot would be captured by
# the recursive snapshot but missing from the staging plan -- silently omitted.
# Read afterwards, it instead trips plan_staging's probe and fails loudly.
for name in ("SNAPSHOT_ASYNC", "SNAPSHOT_SYNC"):
src = extract_blocks()[name]
code = "\n".join(
ln for ln in src.splitlines() if not ln.lstrip().startswith("#")
)
# _tc_stage receives `snapshot` as a parameter -- i.e. it is taken by the
# caller, before any of this runs. If the enumeration ever moves ahead of
# create_snapshot it can only do so by leaving _tc_stage.
assert "def _tc_stage(middleware, path, name, snapshot, snap_path)" in code
assert "query_filesystems" in code
def test_crud_block_is_scoped_to_cloud_backup():
@@ -438,8 +479,275 @@ class TestOnlyOurOwnTasksAreTouched:
# The point is to add NO new failure mode to a CloudSync task. If any
# middleware call happened before the bail-out, we would already have broken
# the thing we are trying not to touch.
#
# Checked against whichever interactions the block ACTUALLY contains, not a
# fixed list: the dataset query moved behind `_tc_nested.query_filesystems()`
# when it switched to the public pool.* API, and a hardcoded
# `middleware.call_sync(` simply stopped being found -- a test that silently
# stops testing is worse than no test.
block = extract_blocks()[name]
gate = block.index('if not name.startswith("cloud_backup"):')
for call in ("middleware.call_sync(", "_tc_nested.stage_nested(",
"_tc_nested.delete_snapshot_tree("):
interactions = [
"middleware.call_sync(",
"_tc_nested.query_filesystems(",
"_tc_nested.stage_nested(",
"_tc_nested.delete_snapshot_tree(",
]
present = [c for c in interactions if c in block]
assert present, "found no middleware interaction at all -- the test is vacuous"
for call in present:
assert gate < block.index(call), f"{call} runs before the cloud_backup gate"
# ── structural assertions ────────────────────────────────────────────────────
#
# `assert "raise" in block` was TRUE because a COMMENT in the block says "a cleanup
# that raises would replace the original exception". `assert "cleanup_task" in block`
# was TRUE because a comment says "cleanup_task gets logger=None". Deleting the actual
# `raise`, and deleting the actual cleanup call from the `finally`, both left the suite
# green -- while reinstating, respectively, a silently-empty backup and ~250 orphaned
# snapshots per run.
#
# A test that a comment can satisfy is not a test. These parse the block and assert on
# the CODE.
def tree_of(name):
return ast.parse(textwrap.dedent(extract_blocks()[name]))
def functions(tree, name):
return [
n for n in ast.walk(tree)
if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef) and n.name == name
]
def calls_to(node, dotted):
"""Every Call in `node` whose callee renders as `dotted` (e.g. a.b.c)."""
out = []
for n in ast.walk(node):
if isinstance(n, ast.Call):
try:
if ast.unparse(n.func) == dotted:
out.append(n)
except Exception: # noqa: BLE001
pass
return out
class TestTheStagingFailurePathReallyReRaises:
"""If staging fails and we swallow it, restic backs up the UN-STAGED path.
That is the silently-empty backup this entire module exists to prevent: stock
points the tool at the parent's `.zfs/snapshot/`, where child datasets are
invisible. The exception MUST propagate.
"""
@pytest.mark.parametrize("name", ["SNAPSHOT_ASYNC", "SNAPSHOT_SYNC"])
def test_the_handler_sweeps_the_snapshot_and_re_raises(self, name):
stage = functions(tree_of(name), "_tc_stage")
assert stage, "_tc_stage is gone"
handlers = [
h for t in ast.walk(stage[0]) if isinstance(t, ast.Try)
for h in t.handlers
]
assert handlers, "the staging failure handler is gone"
sweeps = any(calls_to(h, "_tc_nested.delete_snapshot_tree") for h in handlers)
assert sweeps, (
"a staging failure no longer sweeps the snapshot. sync.py's `snapshot` "
"local stays None, so ITS finally deletes nothing -- the whole tree leaks "
"on every failed run."
)
# A bare `raise` directly in the handler body -- not one nested inside the
# defensive try/except that wraps the sweep.
reraises = any(
any(isinstance(s, ast.Raise) and s.exc is None for s in h.body)
for h in handlers
)
assert reraises, (
"the staging failure is SWALLOWED. restic then runs against the un-staged "
"path and uploads a near-empty tree, reporting SUCCESS."
)
class TestTheSyncBlockAlwaysTearsDown:
"""The teardown is what unmounts the staging tree and sweeps the snapshot.
It must run on EVERY exit from restic_backup -- success, failure, or exception --
or the bind mounts pin the snapshot and the tree is orphaned.
"""
@pytest.mark.parametrize("name", ["SYNC_ASYNC", "SYNC_SYNC"])
def test_cleanup_runs_in_a_finally(self, name):
fns = functions(tree_of(name), "restic_backup")
assert fns, "the restic_backup wrapper is gone"
tries = [t for t in ast.walk(fns[0]) if isinstance(t, ast.Try) and t.finalbody]
assert tries, "restic_backup no longer has a try/finally"
cleans = any(
"cleanup_task" in ast.unparse(stmt)
for t in tries for stmt in t.finalbody
)
assert cleans, (
"cleanup_task is not called in the finally. The staging tree is never torn "
"down, its bind mounts pin the snapshot, and ~250 snapshots leak per run."
)
class TestTheBlockingWorkNeverRunsOnTheEventLoop:
"""`zfs list` and `call_sync` are BLOCKING. On <=25.10 these blocks are async.
Running them directly on middlewared's event loop stalls the whole daemon.
"""
@pytest.mark.parametrize("name,fn", [
("SNAPSHOT_ASYNC", "create_snapshot"),
("SYNC_ASYNC", "restic_backup"),
])
def test_the_async_flavour_hops_to_a_thread(self, name, fn):
fns = functions(tree_of(name), fn)
assert fns and isinstance(fns[0], ast.AsyncFunctionDef)
assert calls_to(fns[0], "middleware.run_in_thread"), (
f"{name}.{fn} does the blocking work on the asyncio event loop"
)
@pytest.mark.parametrize("name,fn", [
("SNAPSHOT_SYNC", "create_snapshot"),
("SYNC_SYNC", "restic_backup"),
])
def test_the_sync_flavour_does_not(self, name, fn):
# On 26 stock already runs this in the thread pool; hopping again would be
# wrong (and there is no event loop to protect).
fns = functions(tree_of(name), fn)
assert fns and isinstance(fns[0], ast.FunctionDef)
assert not calls_to(fns[0], "middleware.run_in_thread")
def test_the_flavour_mapping_is_not_inverted():
# `_snapshot_block = SNAPSHOT_ASYNC if _flavour else SNAPSHOT_SYNC` -- inverting it
# injects an async wrapper on 26 (a coroutine gets unpacked as a tuple) or a sync
# one on 25.10 (the event loop blocks). Every nested backup breaks, both ways.
with open(APPLY_SH, encoding="utf-8") as fh:
code = " ".join(
ln for ln in fh.read().splitlines() if not ln.lstrip().startswith("#")
)
code = re.sub(r"\s+", " ", code) # the assignments are space-aligned
for block in ("SNAPSHOT", "CRUD", "SYNC"):
assert f"{block}_ASYNC if _flavour else {block}_SYNC" in code, (
f"the {block} flavour mapping is missing or inverted: _flavour is True for "
f"an ASYNC middleware, so it must select {block}_ASYNC"
)
# ── the compat preflight ─────────────────────────────────────────────────────
#
# This is the guard that stands between a broken middleware and a live NAS: at every
# boot, apply.sh checks the patch's assumptions against the middlewared actually
# installed, and REFUSES to apply a module whose assumptions no longer hold.
#
# It had no test. An audit turned it into a no-op eight different ways -- `verdict()`
# always returning 'ok', the broken branch never firing, the kill switch never honoured
# -- and the suite stayed green every time. The most consequential safety net in the
# project was unguarded.
def preflight_heredoc():
"""The preflight's Python, lifted out of apply.sh and made runnable.
Extracted, not reimplemented: a reimplementation would happily pass while the
SHIPPED preflight stayed broken, which is exactly the failure being guarded.
"""
with open(APPLY_SH, encoding="utf-8") as fh:
sh = fh.read()
# Line-based: the compat heredoc opens with `<<'PYEOF'` on the _tc_compat line and
# closes at the next bare PYEOF. (A regex that matched `<< 'PYEOF'` silently found
# the OTHER heredoc and ran a different script entirely.)
lines = sh.splitlines()
start = next(
i for i, ln in enumerate(lines)
if ln.startswith("_tc_compat=$(") and "<<'PYEOF'" in ln
)
end = next(i for i in range(start + 1, len(lines)) if lines[i].strip() == "PYEOF")
m = "\n".join(lines[start + 1:end])
assert m, "could not find the compat preflight heredoc in apply.sh"
return m
def run_preflight(result, tmp_path):
"""Run the SHIPPED preflight against a fake compat.check_tree result.
The heredoc does `import sys`, so a fake `sys` in the namespace is immediately
rebound to the real module -- drive the real one instead.
"""
import contextlib
import io
import sys
import types
src = preflight_heredoc()
fake = types.ModuleType("compat")
fake.check_tree = lambda _mw: result
saved_mod = sys.modules.get("compat")
saved_argv = sys.argv
sys.modules["compat"] = fake
sys.argv = ["x", "/patch", "/mw", str(tmp_path / "compat.json")]
buf = io.StringIO()
try:
with contextlib.redirect_stdout(buf):
exec(compile(src, "apply.sh:preflight", "exec"), {"__name__": "__main__"}) # noqa: S102
except SystemExit:
pass
finally:
sys.argv = saved_argv
if saved_mod is not None:
sys.modules["compat"] = saved_mod
else:
sys.modules.pop("compat", None)
return buf.getvalue().splitlines()
def _mod(ok=True, native=False, unknown=False, problems=()):
return {"ok": ok, "native": native, "unknown": unknown, "problems": list(problems)}
class TestTheBootPreflightRefusesABrokenMiddleware:
def test_a_healthy_tree_is_ok(self, tmp_path):
out = run_preflight({"providers": _mod(), "nested": _mod()}, tmp_path)
assert out[:2] == ["ok", "ok"]
def test_a_broken_module_is_reported_broken(self, tmp_path):
out = run_preflight({
"providers": _mod(),
"nested": _mod(ok=False, problems=[
{"id": "x", "detail": "gone", "why": "orphans every run"},
]),
}, tmp_path)
assert "broken" in out, (
"the preflight did not report a module whose assumptions FAILED. It would "
"be injected into a middleware it does not fit -- broken backups, "
"discovered at restore time."
)
def test_a_module_that_went_NATIVE_is_also_not_applied(self, tmp_path):
# 'native' answers "do we still need it?", 'ok' answers "is it safe to inject?".
# Applying a module TrueNAS now implements itself is not safe either.
out = run_preflight({
"providers": _mod(),
"nested": _mod(ok=False, native=True),
}, tmp_path)
assert "broken" in out
def test_an_UNKNOWN_verdict_is_not_reported_as_broken(self, tmp_path):
# A network error or an unreadable file is not iX deleting our symbols. Calling
# it broken would switch a working module off on a healthy box.
out = run_preflight({
"providers": _mod(unknown=True),
"nested": _mod(unknown=True),
}, tmp_path)
assert "broken" not in out
+256 -41
View File
@@ -51,24 +51,44 @@ GOOD = {
"async def restic_backup(middleware, job, cloud_backup, dry_run=False, "
"rate_limit=None):\n pass\n"
),
# The middlewared METHODS the injected code calls. TrueNAS 26 deleted both of
# these files, taking zfs.dataset.query / zfs.snapshot.query / zfs.snapshot.delete
# with them -- see TestMiddlewareMethodsWeCall.
"plugins/zfs_/dataset.py": (
"class ZFSDataset(CRUDService):\n"
# The middlewared METHODS the injected code calls. These now go through the
# PUBLIC pool.* API: TrueNAS 26 deleted plugins/zfs_/ outright, taking the whole
# private zfs.* service with it -- see TestMiddlewareMethodsWeCall.
#
# This default tree is a MODERN box (25.10/26): it has pool.snapshot and no
# zfs.snapshot. The older shape is built explicitly where it is tested.
# Not a plugin: a method on the middleware OBJECT. `snapshot_service()` resolves
# the snapshot namespace through it, so if it vanishes the module cannot sweep the
# snapshot it just took.
"utils/plugins.py": (
"class LoadPluginsMixin:\n"
" def get_service(self, name):\n pass\n"
),
"plugins/pool_/dataset.py": (
"class PoolDatasetService(CRUDService):\n"
" class Config:\n"
" namespace = 'zfs.dataset'\n"
" namespace = 'pool.dataset'\n"
" def query(self, filters, options):\n pass\n"
),
"plugins/zfs_/snapshot.py": (
"class ZFSSnapshot(CRUDService):\n"
"plugins/pool_/snapshot.py": (
"class PoolSnapshotService(CRUDService):\n"
" class Config:\n"
" namespace = 'zfs.snapshot'\n"
" namespace = 'pool.snapshot'\n"
" def query(self, filters, options):\n pass\n"
" def delete(self, id_, options={}):\n pass\n"
),
}
#: A 24.10/25.04 box: `pool.snapshot` does not exist yet and the snapshot CRUD
#: service still answers to the (then-public) `zfs.snapshot`.
ZFS_ERA_SNAPSHOT = (
"class ZFSSnapshot(CRUDService):\n"
" class Config:\n"
" namespace = 'zfs.snapshot'\n"
" def query(self, filters, options):\n pass\n"
" def do_delete(self, id_, options={}):\n pass\n"
)
def loader(files):
def load(path):
@@ -284,9 +304,23 @@ class TestAsyncFlavour:
broken["plugins/cloud_backup/sync.py"] = Unreadable("HTTP 429")
assert compat.async_flavour(loader(broken)) is None
def test_the_real_truenas_versions(self):
# Pinning the actual fact this whole port exists for.
assert compat.async_flavour(loader(GOOD)) is True
def test_it_reads_STOCK_source_not_our_own_injected_block(self):
# This was a byte-identical copy of test_async_middleware_is_detected under a
# name that promised more. The fact worth pinning: apply.sh re-runs on an
# ALREADY-PATCHED overlay, so the probe must cut our block off first -- our own
# SNAPSHOT_SYNC wrapper is a plain `def create_snapshot`, and reading it would
# report a 25.10 box as synchronous and inject the wrong flavour.
patched = dict(GOOD)
patched["plugins/cloud/snapshot.py"] = (
GOOD["plugins/cloud/snapshot.py"]
+ "\n# TRUECLOUD_PATCH\n"
+ 'def create_snapshot(middleware, path, name="x"):\n return "s", "p"\n'
)
assert compat.async_flavour(loader(patched)) is True, (
"the flavour probe read our own injected block and concluded the box is "
"synchronous -- it would then inject a sync wrapper into an async "
"middleware, and every nested backup would break"
)
class TestMiddlewareMethodsWeCall:
@@ -303,52 +337,86 @@ class TestMiddlewareMethodsWeCall:
per descendant dataset (250 on a real pool) on every run, forever.
"""
ZFS_SNAPSHOT = (
"class ZFSSnapshot(CRUDService):\n"
" class Config:\n"
" namespace = 'zfs.snapshot'\n"
" def query(self, filters, options):\n pass\n"
" def delete(self, id_, options={}):\n pass\n"
)
ZFS_DATASET = (
"class ZFSDataset(CRUDService):\n"
" class Config:\n"
" namespace = 'zfs.dataset'\n"
" def query(self, filters, options):\n pass\n"
)
POOL_SNAPSHOT = GOOD["plugins/pool_/snapshot.py"]
def _tree(self, **over):
files = dict(GOOD)
files["plugins/zfs_/snapshot.py"] = self.ZFS_SNAPSHOT
files["plugins/zfs_/dataset.py"] = self.ZFS_DATASET
files.update(over)
return files
#: A 26 box: pool.snapshot only.
def _modern(self, **over):
return self._tree(**over)
#: A 24.10/25.04 box: zfs.snapshot only -- plugins/pool_/snapshot.py does not
#: exist yet.
def _zfs_era(self, **over):
return self._tree(**{
"plugins/pool_/snapshot.py": None,
"plugins/zfs_/snapshot.py": ZFS_ERA_SNAPSHOT,
**over,
})
def test_present_methods_are_ok(self):
r = check_files(self._tree())
r = check_files(self._modern())
assert r[NESTED]["ok"], r[NESTED]["problems"]
def test_a_deleted_plugin_file_is_broken(self):
# Literally TrueNAS 26: plugins/zfs_/snapshot.py does not exist.
r = check_files(self._tree(**{"plugins/zfs_/snapshot.py": None}))
def test_the_OLD_zfs_era_snapshot_service_also_satisfies_the_call(self):
# 24.10 and 25.04 have no `pool.snapshot` at all -- the CRUD service is the
# then-public `zfs.snapshot`. Pinning only the modern spelling marked both of
# those releases BROKEN and would have switched nested snapshots OFF on boxes
# where they work perfectly. The runtime picks the same way; see
# pick_snapshot_service().
r = check_files(self._zfs_era())
assert r[NESTED]["ok"], r[NESTED]["problems"]
def test_it_is_broken_only_when_NEITHER_namespace_exists(self):
# The real failure: middleware drops the last spelling we know how to call.
r = check_files(self._tree(**{
"plugins/pool_/snapshot.py": None,
"plugins/zfs_/snapshot.py": None,
}))
assert is_broken(r[NESTED])
details = " ".join(p["detail"] for p in r[NESTED]["problems"])
assert "zfs.snapshot.delete" in details
assert "pool.snapshot.delete" in details
assert "zfs.snapshot.delete" in details, (
"the report must say BOTH spellings were tried, or whoever reads it will "
"think we simply never looked for the one their box has"
)
def test_we_do_NOT_depend_on_a_middleware_dataset_query_at_all(self):
# iX could delete plugins/pool_/dataset.py tomorrow and the patch would not
# care, because the staging plan is enumerated from ZFS, not from middleware.
#
# That is deliberate, and it was expensive to learn. `pool.dataset.query`
# exists and is correctly shaped -- and it LIES: it applies a visibility
# policy that hides ix-apps/*, .system/* and .ix-virt/* (84 of 270 datasets
# on the real pool, including live app data). No source check could ever
# have caught that; only running it could. So there is no assumption here
# left to break.
r = check_files(self._modern(**{"plugins/pool_/dataset.py": None}))
assert r[NESTED]["ok"], r[NESTED]["problems"]
ids = {c.id for c in compat.MIDDLEWARE_CALLS}
assert not any("dataset" in i or "query" in i for i in ids), (
"a dataset/snapshot QUERY assumption crept back into the manifest -- "
"middleware's queries are filtered; enumerate from ZFS"
)
def test_a_renamed_namespace_is_broken(self):
r = check_files(self._tree(**{
"plugins/zfs_/snapshot.py": self.ZFS_SNAPSHOT.replace(
"'zfs.snapshot'", "'zfs.resource.snapshot'"),
"plugins/pool_/snapshot.py": self.POOL_SNAPSHOT.replace(
"'pool.snapshot'", "'zfs.resource.snapshot'"),
"plugins/zfs_/snapshot.py": None,
}))
assert is_broken(r[NESTED])
def test_the_CRUDService_do_prefix_is_accepted(self):
# 24.10 and 25.04 declare `do_delete`; 25.10 renamed it to `delete`. BOTH
# answer to zfs.snapshot.delete. Accepting only the literal name reported the
# two older releases as broken -- a false BROKEN that would have switched off
# nested snapshots on boxes where they work perfectly.
r = check_files(self._tree(**{
"plugins/zfs_/snapshot.py": self.ZFS_SNAPSHOT.replace(
# A CRUDService exposes `delete` from a method NAMED `do_delete`. Both
# spellings are live across the matrix. Accepting only the literal name
# reported working releases as broken.
r = check_files(self._modern(**{
"plugins/pool_/snapshot.py": self.POOL_SNAPSHOT.replace(
"def delete(", "def do_delete("),
}))
assert r[NESTED]["ok"], r[NESTED]["problems"]
@@ -356,6 +424,153 @@ class TestMiddlewareMethodsWeCall:
def test_the_snapshot_delete_reason_names_the_orphan_risk(self):
# If this ever regresses, whoever reads the bug report must understand that
# it is not a cosmetic failure.
r = check_files(self._tree(**{"plugins/zfs_/snapshot.py": None}))
r = check_files(self._tree(**{
"plugins/pool_/snapshot.py": None,
"plugins/zfs_/snapshot.py": None,
}))
whys = " ".join(p["why"] for p in r[NESTED]["problems"])
assert "orphan" in whys
class TestTheMethodCheckIsNotJustANamespaceCheck:
"""compat must verify the METHOD, not merely that the namespace still exists.
Deleting the method check entirely used to leave all 304 tests green -- so the
"namespace AND method" claim was unenforced and silently revertible. It is the
half of the predicate that catches iX gutting a method while keeping its service,
which they have already done to `pool.snapshot.do_update` on master.
"""
def test_a_namespace_that_no_longer_defines_delete_is_broken(self):
gutted = (
"class PoolSnapshotService(CRUDService):\n"
" class Config:\n"
" namespace = 'pool.snapshot'\n"
" def query(self, filters, options):\n pass\n"
# do_delete is GONE -- the service is still registered and still a
# CRUDService, so it still INHERITS a callable `delete`.
)
r = check_files(with_(**{
"plugins/pool_/snapshot.py": gutted,
"plugins/zfs_/snapshot.py": None, # no fallback either
}))
assert is_broken(r[NESTED]), (
"a namespace with no delete must be BROKEN. Checking only that the "
"namespace exists would apply the patch to a box that cannot sweep its "
"own snapshots."
)
def test_the_alternative_still_saves_it_when_only_the_primary_is_gutted(self):
gutted = (
"class PoolSnapshotService(CRUDService):\n"
" class Config:\n"
" namespace = 'pool.snapshot'\n"
" def query(self, filters, options):\n pass\n"
)
r = check_files(with_(**{
"plugins/pool_/snapshot.py": gutted,
"plugins/zfs_/snapshot.py": ZFS_ERA_SNAPSHOT,
}))
assert r[NESTED]["ok"], r[NESTED]["problems"]
class TestUnreadableIsNeverOkAndNeverBroken:
"""A rate limit is not a regression, and it is not a clean bill of health either.
compat runs ~30 unauthenticated GitHub requests per matrix; 429 is a real outcome.
It also runs at BOOT against the installed tree, where a read can fail with EACCES.
* treating unreadable as BROKEN repaints the README, files a bug report, and
makes apply.sh refuse the module on a box where it works.
* treating it as OK injects a module whose delete may be gone.
Both mutations used to pass the whole suite.
"""
def test_both_spellings_unreadable_is_unknown_not_broken(self):
r = check_files(with_(**{
"plugins/pool_/snapshot.py": Unreadable("HTTP 429"),
"plugins/zfs_/snapshot.py": Unreadable("HTTP 429"),
}))
assert not is_broken(r[NESTED]), "a 429 is not iX deleting the snapshot service"
assert r[NESTED]["unknown"]
def test_an_unreadable_primary_with_a_healthy_alternative_is_ok(self):
r = check_files(with_(**{
"plugins/pool_/snapshot.py": Unreadable("HTTP 429"),
"plugins/zfs_/snapshot.py": ZFS_ERA_SNAPSHOT,
}))
assert r[NESTED]["ok"], r[NESTED]["problems"]
assert not r[NESTED]["unknown"], (
"one spelling answered the question; the other's 429 is irrelevant"
)
def test_a_missing_primary_with_an_unreadable_alternative_is_unknown(self):
# We cannot tell whether the box is broken. Saying either would be a guess.
r = check_files(with_(**{
"plugins/pool_/snapshot.py": None,
"plugins/zfs_/snapshot.py": Unreadable("HTTP 429"),
}))
assert not is_broken(r[NESTED])
assert r[NESTED]["unknown"]
class TestGetServiceIsChecked:
"""The runtime resolves the snapshot namespace through `middleware.get_service`.
It is not a plugin method, so the manifest had no way to express it and never
checked it. If it vanishes, `_can_delete` reports BOTH namespaces unusable and
every nested backup fails -- on a box the preflight had declared healthy.
"""
def test_a_middleware_without_get_service_is_broken(self):
r = check_files(with_(**{"utils/plugins.py": None}))
assert is_broken(r[NESTED])
details = " ".join(p["detail"] for p in r[NESTED]["problems"])
assert "get_service" in details
class TestATransientNetworkBlipDoesNotWakeAnybody:
"""The fingerprint must digest what iX BROKE, not what GitHub failed to serve.
`unknown` problems (a 429 on one of ~30 unauthenticated fetches, an EACCES at boot)
used to be folded into an already-broken module's problem list, so one blip flipped
the fingerprint, `compat_publish` rewrote the issue body, and the next clean run
rewrote it back. Daily churn is what teaches people to ignore the bot -- which is
the whole thing this fingerprint exists to prevent.
"""
def _rows(self, files):
return [{"ref": "master", "modules": check_files(files)}]
def test_an_unreadable_file_does_not_change_the_fingerprint_of_a_broken_ref(self):
# The blip must land in the SAME module that is broken. Put it in `providers`
# (which is healthy) and `fingerprint()` skips the whole module via
# `is_broken(m)` -- so the `state` filter under test never runs and the test
# passes no matter what the code does. `nested` is the broken one here, so the
# unreadable file goes in `nested` too.
broken = with_(**{
"plugins/cloud/snapshot.py":
"async def create_snapshot(name, path, middleware):\n return 1, 2\n",
})
clean = compat.fingerprint(self._rows(broken))
blipped = dict(broken)
blipped["plugins/cloud_backup/sync.py"] = Unreadable("HTTP 429") # nested
assert compat.fingerprint(self._rows(blipped)) == clean, (
"a rate-limited fetch changed the fingerprint, so the bot rewrites the "
"issue body and then rewrites it back tomorrow"
)
def test_a_REAL_new_finding_still_changes_it(self):
# ...and the anti-noise measure must not have made it deaf.
broken = with_(**{
"plugins/cloud/snapshot.py":
"async def create_snapshot(name, path, middleware):\n return 1, 2\n",
})
worse = dict(broken)
worse["plugins/cloud_backup/restic.py"] = (
"class ResticConfig:\n cmd: list\n\n"
"def get_restic_config(entry, credentials):\n pass\n"
)
assert compat.fingerprint(self._rows(worse)) != compat.fingerprint(self._rows(broken))
+37
View File
@@ -94,3 +94,40 @@ class TestTheReadmeStaysAReadme:
assert "24.10" in text[:text.index("## Install")], (
"the minimum TrueNAS version must be visible above the install steps"
)
class TestInstallDoesNotDirtyTheCheckout:
"""install.sh chmod +x's scripts. If git records them as 100644, that chmod is a
TRACKED MODIFICATION -- and update.sh refuses to run over a dirty tree.
So installing once permanently blocked updating, for every user, with a message
telling them to `git checkout -- .` (which would just undo the exec bit and let
the next install re-dirty it). Found on a real box that had been stuck on an old
version for exactly this reason.
Every script install.sh makes executable must already be executable in git.
"""
def test_every_chmodded_script_is_already_executable_in_git(self):
import re
import subprocess
with open(os.path.join(ROOT, "install.sh"), encoding="utf-8") as fh:
m = re.search(r"^for _exe in (.+?); do", fh.read(), re.M)
assert m, "could not find install.sh's chmod loop"
scripts = m.group(1).split()
out = subprocess.run(
["git", "ls-files", "-s", *scripts],
cwd=ROOT, capture_output=True, text=True, check=True,
).stdout
not_exec = [
line.split("\t")[-1] for line in out.strip().splitlines()
if not line.startswith("100755")
]
assert not not_exec, (
"install.sh chmod +x's these, but git records them as non-executable — "
"so installing dirties the checkout and update.sh then refuses to run:\n "
+ "\n ".join(not_exec)
)
+58
View File
@@ -9,6 +9,7 @@ across install.sh / uninstall.sh / recover.sh / apply.sh and nothing noticed.
"""
import os
import re
import sys
import pytest
@@ -233,3 +234,60 @@ class TestCandidateNotesResolveToTheBaseVersion:
def test_a_genuinely_missing_section_still_raises(self):
with pytest.raises(KeyError):
extract_notes(self.CHANGELOG, "v9.9.9-rc1")
class TestTheChangelogIsStructurallySound:
"""The release body IS this file, so a mangled section ships to every user.
It has been mangled once: an edit matched the literal `## Unreleased` inside a
backticked phrase in a prose bullet and spliced a whole new section into the middle
of it, splitting the sentence in half.
"""
def changelog(self):
with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
return fh.read()
def test_no_version_section_is_empty(self):
text = self.changelog()
for v in changelog_versions(text):
assert extract_notes(text, v).strip(), f"v{v} has an empty section"
def test_versions_are_in_descending_order(self):
from release_notes import version_tuple
versions = changelog_versions(self.changelog())
assert versions == sorted(versions, key=version_tuple, reverse=True), (
"CHANGELOG versions are out of order — a section was spliced in wrong"
)
def test_headings_are_at_the_start_of_a_line_and_not_inside_prose(self):
# A `### Fixed` that ends up indented under a bullet is a section nobody sees.
for i, line in enumerate(self.changelog().splitlines(), 1):
if line.lstrip().startswith(("## ", "### ")) and line != line.lstrip():
raise AssertionError(
f"line {i}: heading is indented, so it is inside a list item "
f"rather than being a section: {line!r}"
)
def test_every_bullet_that_opens_a_bold_phrase_closes_it(self):
# The splice cut `- **A stable release ... under \`## Unreleased` in half,
# leaving an unterminated ** and a dangling sentence.
#
# A bullet is the `- ` line plus everything up to the next top-level bullet or
# heading -- bold phrases routinely wrap across lines, so a per-line check
# would flag every long bullet in the file.
text = self.changelog()
bullets = re.split(r"^(?=- |#{2,3} )", text, flags=re.M)
bad = []
for b in bullets:
if not b.startswith("- "):
continue
# Code spans are not markup: `*args, **kwargs` is a literal, not a bold
# phrase, and counting its ** would flag a perfectly well-formed bullet.
prose = re.sub(r"`[^`]*`", "", b)
if prose.count("**") % 2:
bad.append(b.splitlines()[0][:70])
assert not bad, (
"unbalanced ** in a bullet — a section was probably spliced into the "
"middle of it:\n " + "\n ".join(bad)
)
File diff suppressed because it is too large Load Diff
+71 -4
View File
@@ -10,7 +10,8 @@ import re
import pytest
WORKFLOWS = os.path.join(os.path.dirname(__file__), "..", ".github", "workflows")
ROOT = os.path.join(os.path.dirname(__file__), "..")
WORKFLOWS = os.path.join(ROOT, ".github", "workflows")
def workflow_files():
@@ -84,11 +85,77 @@ class TestBothForges:
assert "if: ${{ contains(github.server_url, 'github.com') }}" in src
assert "if: ${{ !contains(github.server_url, 'github.com') }}" in src
def test_compat_files_an_issue_on_each_forge(self):
def test_compat_files_its_report_through_ONE_implementation(self):
# It used to be two near-identical shell steps, one per forge. Two copies of
# "find the issue, decide whether to comment, post it" is two chances to drift,
# and the Gitea one duplicated an issue for real.
with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
src = fh.read()
assert "file a bug report (GitHub)" in src
assert "file a bug report (Gitea)" in src
assert "tools/compat_publish.py" in src
assert "file a bug report (GitHub)" not in src
assert "file a bug report (Gitea)" not in src
class TestTheBotDoesNotSpam:
"""It left 11 identical 3,000-character comments on one issue in a single day.
A bot that repeats itself daily gets muted — and then the next REAL finding is
scrolled past, which defeats the entire reason for building it.
"""
def publisher(self):
with open(os.path.join(ROOT, "tools", "compat_publish.py"), encoding="utf-8") as fh:
return fh.read()
def test_it_compares_a_fingerprint_before_saying_anything(self):
src = self.publisher()
assert "extract_fingerprint" in src
assert "staying quiet" in src
def test_the_body_is_edited_in_place_not_appended_to(self):
src = self.publisher()
assert '"PATCH"' in src, "the issue body must be updated, not commented onto"
def test_it_closes_the_issue_when_everything_is_fixed(self):
src = self.publisher()
assert '"state": "closed"' in src
def test_the_matrix_refresh_opens_a_PR_rather_than_pushing_to_main(self):
# An unattended push to main from CI is exactly what the release barrier exists
# to prevent: a bot that can move main can move it somewhere nobody looked.
#
# Checked against CODE, not comments — the step's own commentary explains what
# it replaced, and that mention must not read as the thing itself.
with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
src = fh.read()
code = "\n".join(
ln for ln in src.splitlines() if not ln.lstrip().startswith("#")
)
assert "/pulls" in code, "the matrix refresh must open a PR"
assert "HEAD:main" not in code, "CI still pushes straight to main"
def test_the_matrix_PR_targets_the_CANONICAL_forge_not_the_mirror(self):
# GitHub is a one-way mirror: a PR merged there would be silently clobbered by
# the next `fleet-repos mirror` push from Gitea. A bot opening PRs against a
# mirror is a bot doing nothing, slowly.
with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
src = fh.read()
i = src.index("refresh the README matrix")
step = src[i:i + 400]
assert "!contains(github.server_url, 'github.com')" in step, (
"the matrix PR must be opened on Gitea (canonical), not GitHub (mirror)"
)
def test_the_workflow_has_the_permissions_its_steps_actually_need(self):
# It shipped with `contents: read` while the step pushed a branch and opened a
# PR — it would have died with a 403 on the first scheduled run, and I would
# have had a bot that silently never worked.
with open(os.path.join(WORKFLOWS, "compat.yml"), encoding="utf-8") as fh:
src = fh.read()
perms = src[src.index("permissions:"):src.index("jobs:")]
assert "contents: write" in perms, "pushing a branch needs contents: write"
assert "pull-requests: write" in perms, "opening a PR needs pull-requests: write"
assert "issues: write" in perms
class TestCompatCannotSilentlyPass:
+293 -60
View File
@@ -43,6 +43,7 @@ from __future__ import annotations
import argparse
import ast
import hashlib
import json
import os
import sys
@@ -132,9 +133,43 @@ ASSUMPTIONS = [
params=["middleware", "job", "cloud_backup"],
why="SYNC_BLOCK wraps it to tear down bind mounts in a finally",
),
Assumption(
# Not a plugin method -- a method on the middleware OBJECT itself, which the
# manifest had no way to express and therefore never checked.
#
# The nested module calls `middleware.get_service(<ns>)` to decide whether to
# sweep snapshots through `pool.snapshot` or `zfs.snapshot` (see
# SNAPSHOT_SERVICES). If it ever disappears, `_can_delete()` catches the
# AttributeError, reports BOTH namespaces unusable, and every nested backup
# fails -- loudly, but only at RUN time, on a box the preflight had already
# declared healthy. Checking it costs one file read.
"get-service", NESTED, "utils/plugins.py",
"LoadPluginsMixin.get_service", kind="method",
params=["self", "name"],
why="snapshot_service() resolves the snapshot namespace through it; without "
"it the module cannot sweep the snapshot it just took",
),
]
def accepted_spellings(name):
"""The method names that satisfy a call to `<namespace>.<name>`.
A CRUDService exposes `create`/`update`/`delete` from methods NAMED
`do_create`/`do_update`/`do_delete`. Both are live across the matrix: 24.10 and
25.04 declare `do_delete`, 25.10 renamed it to `delete`, and all of them answer
to `<ns>.delete`. Accepting only the literal name reported working releases as
BROKEN and would have switched nested snapshots off on boxes where they work.
"""
return (name, f"do_{name}")
#: The spellings that satisfy `<ns>.delete`. A test binds this to the runtime's
#: `truecloud_nested.DELETE_METHODS`, so the checker and the patch cannot come to
#: disagree about what "can delete" means on the same box.
DELETE_NAMES = accepted_spellings("delete")
class MiddlewareCall:
"""A middlewared METHOD the injected code calls at runtime.
@@ -161,87 +196,167 @@ class MiddlewareCall:
the whole design: declining is always the cheaper mistake.
"""
def __init__(self, ident, module, method, path, why=""):
def __init__(self, ident, module, method, path, why="", also=()):
self.id = ident
self.module = module
self.method = method # "zfs.snapshot.delete"
self.method = method # "pool.snapshot.delete"
self.path = path # plugin file that declares it
self.why = why
#: Equally acceptable spellings of the SAME call, as (method, path) pairs.
#:
#: No single snapshot namespace spans every supported release. 24.10 and
#: 25.04 expose the CRUD service as the public `zfs.snapshot`; 25.10
#: promoted it to `pool.snapshot` and demoted `zfs.snapshot` to private;
#: 26 deleted `plugins/zfs_/` entirely. Pinning either one alone marks
#: half the matrix BROKEN and declines to apply on versions that work
#: perfectly well.
#:
#: The call is satisfied if ANY option is present. The runtime picks the
#: same way -- see `pick_snapshot_service()` in the nested module -- so
#: what this checks and what the patch does cannot drift apart.
self.also = tuple(also)
@property
def namespace(self):
return self.method.rsplit(".", 1)[0]
def options(self):
"""Every (method, path) that would satisfy this call, best first."""
return ((self.method, self.path), *self.also)
@property
def name(self):
return self.method.rsplit(".", 1)[1]
@staticmethod
def namespace_of(method):
return method.rsplit(".", 1)[0]
@staticmethod
def name_of(method):
return method.rsplit(".", 1)[1]
#: Every middlewared method the nested module calls at runtime.
#: The middleware methods the nested module CALLS.
#:
#: These used to be the PRIVATE `zfs.*` service (`zfs.dataset.query`,
#: `zfs.snapshot.delete`, `zfs.snapshot.query`). TrueNAS 26 deleted
#: `plugins/zfs_/` outright and every one of them vanished -- silently, because a
#: private service carries no stability contract and nothing warned us. The patch
#: would have applied cleanly and then failed on the first backup.
#:
#: The replacements are the PUBLIC `pool.*` API, and switching to it is not merely
#: a TrueNAS 26 fix -- it is the correct call on every version:
#:
#: * It is public, documented, and covered by iX's deprecation policy, so it
#: cannot be deleted from under us the way `zfs.*` just was.
#: * The same methods, in the same files, exist on 24.10 through 26. One code
#: path, no version conditionals.
#: * Both spellings take `recursive`, so ONE call sweeps the whole tree instead
#: of ~250 individual deletes, any of which could be missed.
MIDDLEWARE_CALLS = [
MiddlewareCall(
"call-zfs-dataset-query", NESTED, "zfs.dataset.query",
"plugins/zfs_/dataset.py",
why="SNAPSHOT_BLOCK enumerates FILESYSTEM datasets to build the staging plan",
),
MiddlewareCall(
"call-zfs-snapshot-delete", NESTED, "zfs.snapshot.delete",
"plugins/zfs_/snapshot.py",
"call-snapshot-delete", NESTED, "pool.snapshot.delete",
"plugins/pool_/snapshot.py",
also=[("zfs.snapshot.delete", "plugins/zfs_/snapshot.py")],
why="delete_snapshot_tree() sweeps the recursive snapshot. Without it every "
"run orphans one snapshot per descendant dataset (250 on a real pool)",
),
MiddlewareCall(
"call-zfs-snapshot-query", NESTED, "zfs.snapshot.query",
"plugins/zfs_/snapshot.py",
why="delete_snapshot_tree()'s fallback sweep enumerates the tree by name",
),
]
# There is deliberately NO entry here for a dataset or snapshot QUERY.
#
# The patch used to call `zfs.dataset.query` / `zfs.snapshot.query` (private, and
# deleted in TrueNAS 26). The obvious port was to the public `pool.dataset.query` /
# `pool.snapshot.query` -- and that port was WRONG in a way no source check could
# ever have caught, because the methods are all present and correctly shaped.
#
# They are simply filtered. On a real box they return 205 of 274 datasets and 205
# of 274 snapshots, hiding `ix-apps/*`, `.system/*` and `.ix-virt/*` -- 84 of 270
# on the production pool, including live application data. Staging from that view
# silently omits them; sweeping from it orphans one snapshot per hidden dataset,
# forever.
#
# So the module enumerates from ZFS itself and there is no middleware assumption
# left to check. That is the point: the fewer things we assume about middleware,
# the less there is for iX to break. Only the MUTATION is still a middleware call,
# and that is the one entry above.
def check_call(c: MiddlewareCall, src: str | None,
method: str | None = None, path: str | None = None,
) -> tuple[str, str | None]:
"""Is `method` still registered by middlewared?
`method`/`path` name WHICH spelling of the call is being tried -- a call may
have several equally acceptable ones (see MiddlewareCall.also). They default
to the preferred spelling.
"""
method = method or c.method
path = path or c.path
namespace = MiddlewareCall.namespace_of(method)
name = MiddlewareCall.name_of(method)
def check_call(c: MiddlewareCall, src: str | None) -> tuple[str, str | None]:
"""Is `c.method` still registered by middlewared?"""
if src is None:
return "broken", (
f"{c.path} no longer exists, so `{c.method}` is gone"
f"{path} no longer exists, so `{method}` is gone"
)
try:
tree = ast.parse(_stock(src))
except SyntaxError as e:
return "unknown", f"{c.path} does not parse: {e}"
return "unknown", f"{path} does not parse: {e}"
# namespace = 'zfs.snapshot' on some Service class in this file...
namespaces = {
# Find the CLASS that declares this namespace, and look for the method THERE.
#
# Not anywhere in the file. `ast.walk` over the whole module made *any* function
# called `delete` satisfy the check -- one on an unrelated class, or even a nested
# local function inside `do_query`. That is a FALSE OK, and it breaks the one
# invariant this checker and the runtime share: `_defines_delete()` looks in
# `vars(klass)` for a PLUGIN class on the service's MRO. If iX gutted
# `PoolSnapshotService.do_delete` while some other class in the same file still had
# a `delete`, compat would say ok, apply.sh would patch, and the runtime would then
# correctly refuse `pool.snapshot`, fall through to a `zfs.snapshot` that does not
# exist on 26, and fail every nested backup on a box the preflight called healthy.
#
# Same question on both sides: does the class that OWNS this namespace define the
# method?
#
# A CRUDService exposes `create`/`update`/`delete` from methods NAMED
# `do_create`/`do_update`/`do_delete`. Both spellings are live: 24.10 and 25.04
# declare `do_delete`, 25.10 renamed it to `delete`, and all answer to
# `<ns>.delete`. Accepting only the literal name reported working releases as
# broken.
owners = []
all_namespaces = set()
for cls in (n for n in ast.walk(tree) if isinstance(n, ast.ClassDef)):
declared = {
n.value.value
for n in ast.walk(tree)
for n in ast.walk(cls)
if isinstance(n, ast.Assign)
and isinstance(n.value, ast.Constant)
and isinstance(n.value.value, str)
and any(isinstance(t, ast.Name) and t.id == "namespace" for t in n.targets)
}
if c.namespace not in namespaces:
all_namespaces |= declared
if namespace in declared:
owners.append(cls)
if not owners:
return "broken", (
f"{c.path} no longer declares namespace {c.namespace!r} "
f"(found: {sorted(namespaces) or 'none'}), so `{c.method}` is gone"
f"{path} no longer declares namespace {namespace!r} "
f"(found: {sorted(all_namespaces) or 'none'}), so `{method}` is gone"
)
# ...and it defines the method.
#
# A CRUDService exposes `create`/`update`/`delete` from methods NAMED
# `do_create`/`do_update`/`do_delete`. Both spellings are live right now:
# 24.10 and 25.04 declare `do_delete`, 25.10 renamed it to `delete`, and all
# three answer to `zfs.snapshot.delete`. Accepting only the literal name reported
# the two older releases as broken -- a false BROKEN that would have switched off
# nested snapshots on boxes where they work.
defined = {
n.name for n in ast.walk(tree)
if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef)
}
if c.name not in defined and f"do_{c.name}" not in defined:
return "broken", f"{c.path} no longer defines `{c.method}`"
wanted = accepted_spellings(name)
for cls in owners:
# Direct members of the class, not its nested scopes: a `def delete` inside
# another method is a local function, not a service method.
if any(
isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef) and n.name in wanted
for n in cls.body
):
return "ok", None
return "broken", (
f"{path} still declares namespace {namespace!r}, but its class no longer "
f"defines `{'` or `'.join(wanted)}` -- so `{method}` is gone"
)
#: Things that mean iX has done the job themselves and the module should RETIRE,
#: not break. Absence of the nesting guard = nested snapshots went native.
@@ -403,7 +518,10 @@ def check_source(a: Assumption, src: str | None) -> tuple[str, str | None]:
breaks a box that was working.
"""
if src is None:
return "broken", f"{a.path} does not exist"
# Name the SYMBOL, not just the file. Whoever reads the bug report needs to
# know what the patch can no longer reach, and "utils/plugins.py does not
# exist" does not tell them that `get_service` is gone.
return "broken", f"{a.path} does not exist, so `{a.symbol}` is gone"
try:
tree = ast.parse(src)
@@ -538,6 +656,7 @@ def check(loader, modules=None) -> dict:
out[a.module]["unknown"] = True
out[a.module]["problems"].append({
"id": a.id, "detail": f"could not read {a.path}: {e}", "why": a.why,
"state": "unknown",
})
continue
@@ -545,37 +664,57 @@ def check(loader, modules=None) -> dict:
if status == "broken":
out[a.module]["ok"] = False
out[a.module]["problems"].append({
"id": a.id, "detail": detail, "why": a.why,
"id": a.id, "detail": detail, "why": a.why, "state": "broken",
})
elif status == "unknown":
out[a.module]["unknown"] = True
out[a.module]["problems"].append({
"id": a.id, "detail": detail, "why": a.why,
"id": a.id, "detail": detail, "why": a.why, "state": "unknown",
})
# The methods the injected code CALLS, not just the symbols it wraps.
#
# A call may have several equally acceptable spellings, because no single
# snapshot namespace spans every supported release (24.10 has `zfs.snapshot`,
# 26 has only `pool.snapshot`). It is satisfied if ANY of them is present --
# exactly as the runtime resolves it -- and BROKEN only when they all vanish.
for c in MIDDLEWARE_CALLS:
if c.module not in out:
continue
satisfied, unknown, details = False, False, []
for method, path in c.options:
try:
text = src(c.path)
text = src(path)
except Unreadable as e:
out[c.module]["unknown"] = True
out[c.module]["problems"].append({
"id": c.id, "detail": f"could not read {c.path}: {e}", "why": c.why,
})
unknown = True
details.append(f"could not read {path}: {e}")
continue
status, detail = check_call(c, text)
if status == "broken":
out[c.module]["ok"] = False
out[c.module]["problems"].append({
"id": c.id, "detail": detail, "why": c.why,
})
elif status == "unknown":
status, detail = check_call(c, text, method, path)
if status == "ok":
satisfied = True
break
if status == "unknown":
unknown = True
details.append(detail)
if satisfied:
continue
# Every spelling failed. If we could not READ one of them we do not know
# that it is broken -- a rate-limited fetch is not a regression.
if unknown:
out[c.module]["unknown"] = True
out[c.module]["problems"].append({
"id": c.id, "detail": detail, "why": c.why,
"id": c.id, "detail": "; ".join(details), "why": c.why,
"state": "unknown",
})
else:
out[c.module]["ok"] = False
out[c.module]["problems"].append({
"id": c.id, "detail": "; ".join(details), "why": c.why,
"state": "broken",
})
for module, (path, phrase, native_when_present) in NATIVE_PROBES.items():
@@ -803,6 +942,10 @@ def is_broken(r: dict) -> bool:
#: a strictly weaker claim than "a restore worked". Add a row only after doing it.
HARDWARE_VERIFIED = {
"25.10.4": "nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m",
"26.0.0-BETA.1": (
"nested + providers; 274-snapshot recursive backup of a 292-dataset pool, "
"restored a 4-deep child dataset byte-identical"
),
}
_LEGEND = """
@@ -892,6 +1035,96 @@ def render_markdown(rows: list[dict]) -> str:
return "\n".join(out) + "\n" + _LEGEND
FINGERPRINT = "<!-- compat-fingerprint:"
def fingerprint(rows: list[dict]) -> str:
"""A stable digest of WHAT IS BROKEN, and nothing else.
The bug report must be updated when the findings change and stay silent when they
do not. Without this the workflow commented on every run -- it left **11 identical
3,000-character comments** on one issue in a single day, which is not a warning
system, it is a mute button with extra steps.
Deliberately excludes anything that moves on its own: the matrix's `ok` rows, the
hardware-verified column, and the exact TrueNAS point-release (`TS-25.10.4` ->
`TS-25.10.5` is not news). Only the broken (ref, module, problem-id) triples count.
"""
findings = sorted(
(r["ref"], mod, p["id"])
for r in rows
for mod, m in r["modules"].items()
if is_broken(m)
for p in m["problems"]
# `unknown` problems are things we could not READ (a 429, an EACCES), not
# things iX changed. On a ref that is broken for some other reason they would
# otherwise join the digest, so one transient network blip rewrites the issue
# body and the next clean run rewrites it back. That is the daily-noise
# failure this fingerprint exists to prevent, wearing a different hat.
if p.get("state", "broken") == "broken"
)
return hashlib.sha256(repr(findings).encode()).hexdigest()[:16]
def extract_fingerprint(body: str) -> str | None:
"""The fingerprint a previous run left in the issue body, if any."""
if not body:
return None
i = body.find(FINGERPRINT)
if i == -1:
return None
return body[i + len(FINGERPRINT):].split("-->", 1)[0].strip() or None
def render_issue(rows: list[dict]) -> str:
"""The bug report body: what is broken, why it matters, and nothing else up front.
Short by design. The full matrix and the healthy versions go in a fold -- somebody
opening this wants to know what broke and whether it can hurt them, not to re-read
a table they can see in the README.
"""
broken = [r for r in rows if any(is_broken(m) for m in r["modules"].values())]
out = [
"`tools/compat.py` checks what this patch assumes about middlewared against "
"iXsystems' actual source, every day. Those assumptions no longer hold on the "
"versions below.",
"",
"**This does not break anyone today.** `apply.sh` re-checks on every boot and "
"**declines to apply** a module whose assumptions fail, so TrueNAS is left "
"stock rather than half-patched. The cost is the module's feature, not a "
"broken backup.",
"",
]
for r in broken:
out.append(f"### `{r['ref']}`")
out.append("")
for mod, m in sorted(r["modules"].items()):
if not is_broken(m):
continue
out.append(f"**{mod}**")
out.append("")
for p in m["problems"]:
out.append(f"- {p['detail']}")
out.append(f" <br><sub>{p['why']}</sub>")
out.append("")
out += [
"<details><summary>Full support matrix</summary>",
"",
render_markdown(rows),
"</details>",
"",
"_Filed and kept up to date by "
"[`compat.yml`](.github/workflows/compat.yml). It edits this body when the "
"findings change, and stays quiet when they do not._",
"",
f"{FINGERPRINT} {fingerprint(rows)} -->",
]
return "\n".join(out)
def render_matrix(rows: list[dict]) -> str:
"""A support table.
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""Keep ONE bug report in sync with what compat.py currently finds.
WHY THIS IS NOT JUST "POST A COMMENT"
-------------------------------------
The first version commented on every run that found a break. In one day it left
**11 identical 3,000-character comments** on the same issue. That is not a warning
system; it is a mute button with extra steps. The next real finding would have been
scrolled past, which defeats the entire point of building it.
So:
* **The issue body is the current truth.** It is edited in place, never appended to.
* **Comments are a changelog of CHANGES.** A run whose findings are identical to the
last one says nothing at all -- no comment, no edit, no notification.
* A fingerprint of the findings (broken ref/module/problem triples only) is embedded
in the body. It deliberately ignores things that move on their own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so `TS-25.10.4` becoming
`TS-25.10.5` is not news, and does not wake anybody up.
* When everything is fixed, the issue is **closed** with a comment saying so.
Works against GitHub and Gitea, which differ only in the auth header and the issue
list URL. One implementation, so the two cannot drift.
python3 tools/compat_publish.py --api <url> --token <tok> --matrix /tmp/matrix.json
"""
from __future__ import annotations
import argparse
import json
import sys
import urllib.error
import urllib.request
sys.path.insert(0, __file__.rsplit("/", 1)[0])
from compat import ( # noqa: E402
extract_fingerprint,
fingerprint,
is_broken,
render_issue,
)
TITLE = "TrueNAS compatibility: the patch's assumptions no longer hold"
def _call(url, token, method="GET", data=None):
req = urllib.request.Request(
url, method=method,
headers={
# Gitea wants `token <t>`; GitHub accepts `Bearer <t>`. GitHub also
# accepts `token <t>`, so one header serves both.
"Authorization": f"token {token}",
"Content-Type": "application/json",
"Accept": "application/vnd.github+json",
},
data=json.dumps(data).encode() if data else None,
)
with urllib.request.urlopen(req) as r: # noqa: S310
return json.load(r) if r.length != 0 else {}
def find_issue(api, token, title):
"""The LOWEST-numbered issue with this title, open or closed.
Lowest, not "whichever the API returns first": two issues with the same title
existed once (an earlier version put the ref list in the title, so the identity
changed whenever that set changed), and an order-dependent pick would alternate
between them -- reopening one while commenting on the other.
"""
issues = _call(f"{api}/issues?state=all&per_page=100", token)
mine = [
i for i in issues
if i.get("title") == title and "pull_request" not in i # GitHub lists PRs here
]
return min(mine, key=lambda i: i["number"]) if mine else None
def main(argv):
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("--api", required=True, help="…/repos/<owner>/<repo>")
ap.add_argument("--token", required=True)
ap.add_argument("--matrix", required=True, help="compat.py --matrix --json output")
args = ap.parse_args(argv[1:])
with open(args.matrix, encoding="utf-8") as fh:
rows = json.load(fh)
broken = [r for r in rows if any(is_broken(m) for m in r["modules"].values())]
issue = find_issue(args.api, args.token, TITLE)
# ── everything is healthy ────────────────────────────────────────────────
if not broken:
if issue and issue["state"] == "open":
_call(f"{args.api}/issues/{issue['number']}/comments", args.token, "POST",
{"body": "All of the patch's assumptions hold again on every "
"checked TrueNAS version. Closing."})
_call(f"{args.api}/issues/{issue['number']}", args.token, "PATCH",
{"state": "closed"})
print(f"closed #{issue['number']} — nothing is broken any more")
else:
print("nothing broken; no open report to close")
return 0
body = render_issue(rows)
want = fingerprint(rows)
# ── nothing to file yet ──────────────────────────────────────────────────
if issue is None:
made = _call(f"{args.api}/issues", args.token, "POST",
{"title": TITLE, "body": body})
print(f"filed #{made['number']}")
return 0
have = extract_fingerprint(issue.get("body") or "")
n = issue["number"]
# ── the findings are UNCHANGED: say nothing ──────────────────────────────
#
# This is the whole point. A daily "still broken, same as yesterday" comment is
# what taught everyone to ignore the last one.
if have == want and issue["state"] == "open":
print(f"#{n} is already current ({want}) — staying quiet")
return 0
_call(f"{args.api}/issues/{n}", args.token, "PATCH", {"body": body, "state": "open"})
if have != want:
refs = ", ".join(f"`{r['ref']}`" for r in broken)
note = (
"The findings changed — the report above has been updated.\n\n"
f"Currently broken on: {refs}."
if have else
"This report is now kept up to date automatically: the body above always "
"reflects the current findings, and a comment is only added when they "
"change."
)
_call(f"{args.api}/issues/{n}/comments", args.token, "POST", {"body": note})
print(f"updated #{n}: {have} -> {want}")
else:
print(f"reopened #{n}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+1 -1
View File
@@ -3,7 +3,7 @@
set -euo pipefail
VERSION="0.6.0"
VERSION="0.7.0"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
_HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
Regular → Executable
+16 -4
View File
@@ -19,7 +19,7 @@
set -euo pipefail
VERSION="0.6.0"
VERSION="0.7.0"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
_PREV_FILE="$PATCH_DIR/.update_previous"
@@ -123,7 +123,7 @@ cd "$PATCH_DIR"
if ! git rev-parse --git-dir >/dev/null 2>&1; then
echo "ERROR: $PATCH_DIR is not a git clone — nothing to update." >&2
echo " Re-clone from https://git.onetick.ninja/flan/truenas-truecloud-patch" >&2
echo " Re-clone from https://github.com/sudolulo/truenas-truecloud-patch" >&2
exit 1
fi
@@ -136,9 +136,21 @@ fi
# A dirty tree means someone edited or scp'd files in place; merging over that
# silently loses their changes, or conflicts halfway through.
if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
#
# CONTENT changes only. A mode-only change (100644 -> 100755) is not somebody's work
# and must not block an update -- and it is not hypothetical: install.sh chmod +x's
# these very scripts, so on any version where git recorded one as 100644, INSTALLING
# dirtied the checkout and update.sh then refused to run. Install once, and updating
# was blocked forever, with an error telling the user to `git checkout -- .` (which
# merely undoes the exec bit so the next install can re-dirty it). A real box sat on
# an old version for exactly this reason.
#
# `git diff --numstat` reports "0 0 file" for a mode-only change, so anything with a
# nonzero insert or delete count is a genuine edit.
_dirty=$(git diff --numstat HEAD -- . | awk '$1 != 0 || $2 != 0 { print $3 }')
if [ -n "$_dirty" ]; then
echo "ERROR: the working tree has uncommitted changes:" >&2
git status --short --untracked-files=no >&2
printf ' M %s\n' $_dirty >&2
echo "" >&2
echo " Refusing to update over them. Commit, stash, or discard them first:" >&2
echo " git -C $PATCH_DIR checkout -- ." >&2