The README and how-it-works still said "TrueNAS 26: nested snapshots are not supported
yet" and "the third is not fixed, and is why 26 reports BROKEN". Both shipped in v0.7.0
and are now false — exactly the kind of stale claim that misleads somebody deciding
whether to trust this with their backups.
Adds docs/verification.md: what has ACTUALLY been run, as opposed to what the support
matrix proves. The matrix is static analysis — it shows the patch's assumptions still
hold, which is a strictly weaker claim than "a backup ran and a restore came back". The
new file records the three live tasks exercised on 25.10.4 (nested, nested+zvols,
non-nested), the md5 of the file that came back out of B2, the real orphan the collector
reclaimed from the pool, and what is NOT covered (24.10/25.04 unrun; master broken; no
reboot on v0.7.0). The README now points at it, next to the matrix it qualifies.
The how-it-works TrueNAS 26 section now explains the part that mattered: the public
pool.* queries are not like-for-like replacements for the deleted private zfs.* ones —
they apply a visibility policy hiding 84 of 270 datasets on a real pool, including live
app data — and the rule the module now follows (read the truth from ZFS, make changes
through middleware). Plus the divergence nothing warned about: 26 decides `recursive` by
a different rule than this patch decides `nested`, which orphaned one snapshot per zvol
on every run until ownership of the sweep was made unconditional.
A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record
of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the
recursive snapshot and its cleanup destroyed that record, leaving one snapshot per
descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would
ever have found them.
gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is
gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and
the collector only mops up what the record lost.
It deletes data on a name match, which is a weaker claim than a recorded fact, so the
selection is a pure function with the harshest tests here. A snapshot is collected
only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current
run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a
concurrent backup), and it is over an hour old.
Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it
selects exactly the orphans of the task being run and nothing else.
Found live, in the code written to prevent exactly this.
The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to
finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of
the survivor -- orphaning it permanently.
Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46
reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it
behind again, and then wrote its own snapshot over the record. Permanent orphan,
created by the safety net.
The sidecar is now a list. stage_nested carries forward whatever a reclaim could not
delete; cleanup_task sweeps every pending tree and writes back only the survivors.
cleanup_all reports them one per line instead of formatting a list into an f-string
at the user during uninstall.
Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots
with no straggler.
Gitea is canonical for development; GitHub is where users clone from and where the
box's read-only checkout points. The install instructions, the re-clone hint and the
'file an issue' link are all read by users, so they name GitHub. docs/releasing.md
still names Gitea, because that is a contributor doc about where the code is pushed.
Install was at line 517 of 969, under the boot sequence, the snapshot lifecycle and
the release process. Someone deciding whether to trust this with their backups should
not have to scroll past any of that.
README is now 211 lines: what it does, the minimum version, install, the support
matrix, updating, uninstall. Everything else moved to docs/ (nested snapshots, how it
works, recovery, CLI, releasing).
A test enforces it: every internal link resolves, Install stays near the top, and the
README does not grow back. Moving Markdown breaks cross-references -- it broke eight
of them here, including one in a recovery doc, where the person following the link is
by definition already having a bad day.
flan