It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.
Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.
The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.
Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
whether to comment' is two chances to drift, and the Gitea one duplicated an issue
for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
main from CI. An unattended push to main is exactly what the release barrier exists
to prevent: a bot that can move main can move it somewhere nobody looked.
This gap was hiding a catastrophe. TrueNAS 26 deletes plugins/zfs_/dataset.py and
plugins/zfs_/snapshot.py outright, taking zfs.dataset.query, zfs.snapshot.query and
zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
Nothing about the five cloud_backup files reveals that, so every other check went
green -- including the one I had just added. The patch would have applied cleanly
and then failed on the first backup, or worse: snapshotted fine and failed to
DELETE, orphaning one snapshot per descendant dataset (250 on a real pool) on every
run, forever.
So 26 is BROKEN and the nested module will not apply there. The async/sync wrapper
work and the vendored get_dataset_recursive stay -- they are correct and necessary
-- but 26 is not supported until the ZFS calls are ported, and that needs a real 26
box to verify. Shipping a port nobody has run is the failure this project exists to
avoid.
Also: do_delete is recognised as delete (24.10/25.04 use the CRUDService
convention), which was reporting both as BROKEN -- a false verdict that would have
disabled nested snapshots on boxes where they work.
26 rewrites cloud_backup from async to synchronous AND deletes
get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's
namespace. Either is a broken backup found at restore time.
The nested module is now one synchronous implementation talking to middlewared via
call_sync, behind two thin wrappers. apply.sh reads which flavour the installed
middleware declares and injects the matching one: <= 25.10 reaches it through
'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls
it directly. The snapshot/bind-mount/failure logic exists once -- an async twin
would mean every future fix had to land twice.
A middleware whose three wrapped functions disagree about asyncness is refused,
not guessed at. get_dataset_recursive is vendored, removing the dependency on both
versions rather than asserting it.
master stays BROKEN on purpose: iX are still renaming middleware->context,
cloud_backup->entry and adding a required credentials param there. Chasing a
branch that moves daily is how you ship a patch nobody tested.
The audit found the new machinery could do more harm than the bugs it prevents.
- apply.sh reused the 'nothing left to do' exit -- which touches the PERMANENT
kill switch, cleared only by install.sh, never by update.sh -- for the
incompatible case. On TrueNAS 26 (providers ok, nested opt-out) both modules go
quiet, so the switch would fire and the release that fixed 26 could never
re-enable itself. Retirement and incompatibility now take different exits.
- A network blip, a re-export, or a conditional def all read as BROKEN. Each is
now 'unknown', which changes nothing, rather than evidence strong enough to
disable a module.
- 'native' outranked BROKEN everywhere but apply.sh, so a TrueNAS that reworded
the guard AND reshaped the functions rendered as good news.
- compat.py --tree read B2_BLOCK's own 'restic = True' as native support, so the
documented way to check a live box lied on every patched machine.
- The signature check was a name-subset test. It passed reorders, kw-only
conversions, and added required params -- and it had already passed a real bug:
restic_backup takes 4 args on 24.10/25.04, and the wrapper forwarded 5. Nested
backups have been raising TypeError on those releases the whole time. The
wrapper now forwards *args/**kwargs.
- release.sh --promote was unreachable: it died if the tag existed, the gate died
if it did not. The tests hid it by always tagging first.
The matrix is regenerated daily by CI rather than typed once and forgotten — a
support table that quietly goes stale is a false promise to someone deciding
whether to trust this with their backups.
flan