The most important finding is that the FIRST audit's fix was wrong.
_can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines
`delete` on the BASE class and dispatches to self.do_delete at call time, so a bound
`delete` exists on every CRUDService subclass whether or not it still implements one.
The check was therefore answering "is this a CRUDService?" — precisely the weaker "is
the namespace registered?" question its own docstring said must never be asked. It
would still have picked a gutted pool.snapshot and failed every delete. It now walks
the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining
delete/do_delete counts. The test double was equally wrong: it modelled a gutted
service as object(), a shape middlewared cannot produce, so the test passed against a
fake it could never have caught in the field. It is now CRUDService-shaped.
Also:
- The recursive delete's fast path returned [] without confirming anything was
destroyed. A delete that returns cleanly is not proof — iX has already gutted
pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read
"no survivors" as a clean sweep, dropped the sidecar (the only record), and would
have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the
by-name sweep trusts ZFS rather than the API's return value.
- When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not
symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record
clears), a lost record does not.
- _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist;
failing to write it must never be invisible.
- stage_nested now refuses UP FRONT when middleware has no usable snapshot delete,
rather than discovering it after restic has already run.
Tests. The autouse fixture added in the last commit did not work: `runner=_run`,
`mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__
at def time, so monkeypatching the module attribute never reached them. 19 tests were
still reading the real mount table — one matching name from running a real `umount` on
the NAS — and the retry loop really slept. All three are late-bound now; the suite
reads nothing outside tmp_path and runs in 1.1s.
Every mutation the audit reported as SURVIVING now fails the suite: the naive delete
check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager
service resolution, compat's method check, compat's unknown handling, a single-quoted
filtered query in apply.sh, and the get-service assumption.
Also: fingerprint() folded `unknown` problems into a broken module, so one transient
429 rewrote the bug report and the next clean run rewrote it back. Problems are
state-tagged; only definite breakage is digested.
Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup
0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep
child dataset.
`middleware` and the ZFS reader were being threaded through five functions as a
pair, and the snapshot namespace was re-resolved in each of them. That is one
collaborator, not two. _Snapshots owns both, resolves the namespace lazily (so the
read-only paths do not raise over a mutation they never make), and makes the rule the
module rests on structural instead of a comment people have to remember.
Deliberately internal: apply.sh injects calls to the public functions into middlewared
itself, so their signatures are a boot-time contract with a live NAS and are not worth
churning for tidiness.
Verified on TrueNAS 26.0.0-BETA.1 after the change: 292-dataset backup, 0 orphans,
0 leaked mounts, byte-identical restore of a 4-deep child dataset.
Four audits of the TrueNAS 26 branch. The findings, in severity order.
1. TrueNAS 26 orphaned a snapshot on every run, with no backstop.
Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours.
<= 25.10 its create_snapshot called get_dataset_recursive() — the same function this
module vendors — so "stock went recursive" and "we have something to stage" were the
same question. 26 uses filesystem.statfs: recursive = (path == the dataset's
mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint
datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage.
The patch then handed the snapshot back to stock, which destroys the parent only. No
staging tree meant no sidecar, and the GC only ever ran from stage_nested — so
nothing on the box would ever have found the children. Reproduced on the VM: one
orphan per zvol, every run, forever, backup green.
Ownership of the sweep is no longer conditional on staging (own_snapshot()).
2. The runtime resolved a NAMESPACE; compat.py verified a METHOD.
get_service() only proves a namespace is registered. compat checks the namespace AND
that it defines delete/do_delete. So if iX guts the method but keeps the service —
which they have already done to pool.snapshot.do_update on master — compat falls
through to zfs.snapshot and reports the box healthy, while the runtime picks
pool.snapshot and fails every delete. Both sides now ask "can this namespace
delete?", and a test binds the two lists together.
3. query_filesystems() silently dropped malformed rows — the one remaining
silent-omission path, and a direct contradiction of the cardinal rule. It raises now.
A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed.
4. The retry loop discarded the delete error and reported every survivor as
"(still busy?)" — naming the one cause that is benign and hiding the ones that are
permanent. It keeps and reports the real error.
Also: the staging-failure handler could lose the original exception if its own sweep
raised; get_service is now a checked assumption; normalise_dataset and two dead
MiddlewareCall properties removed; stale comments corrected.
Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148
snapshots) and passed here only because this box has no zfs binary — they would have
gone red on the NAS, which is the one machine the release process requires them green
on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of
the five fixes above now fails the suite; before, all 293 passed.
Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset
backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4
(pool.snapshot.delete honours recursive=True).
TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query,
zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested
module's critical path, so nested snapshots were BROKEN on 26.
Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10
and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported
release. tools/compat.py checks the same list the runtime uses, so what CI verifies
and what runs cannot drift apart.
Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is
the point of this commit. Those methods exist, are documented, and are covered by
iX's deprecation policy — and they are not like-for-like replacements. They apply a
visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets
on a real pool, including live application data. Staging from that view omits them
silently, and plan_staging never sees them, so they do not even reach the skipped
list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one
per hidden dataset on every run.
So: read the truth from ZFS, make changes through middleware. zfs list cannot be
filtered by policy and behaves identically on every release. A failing zfs list raises
rather than returning an empty list — "no datasets" and "the command broke" must never
look the same.
No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query,
which returns all 270 datasets. The bug existed only in this port.
Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a
292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical
restore of a four-level-deep child dataset that pool.dataset.query hides.
Two bugs, both of which would have failed silently on the first scheduled run:
- permissions were while the step pushes a branch and opens a PR.
It would have died with a 403 and I would have had a bot that never worked.
- it opened the PR on GITHUB, which is a one-way MIRROR. A PR merged there would be
clobbered by the next fleet-repos mirror push from Gitea. A bot opening PRs against
a mirror is a bot doing nothing, slowly.
Now: contents+pull-requests write, and the PR is opened on Gitea (canonical) via its
API. One long-lived PR, force-pushed in place -- a daily PR is the same mistake as a
daily comment, wearing a hat.
It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.
Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.
The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.
Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
whether to comment' is two chances to drift, and the Gitea one duplicated an issue
for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
main from CI. An unattended push to main is exactly what the release barrier exists
to prevent: a bot that can move main can move it somewhere nobody looked.
An edit matched the literal '## Unreleased' inside a backticked phrase in a prose
bullet and spliced a whole new section into the middle of it, splitting the sentence
in half. The release body IS this file, so that would have shipped to every user.
Tests now assert: no empty version section, versions descend, no heading is indented
inside a list item, and every bullet's bold phrases are balanced (ignoring code spans
-- '*args, **kwargs' is a literal, not markup).
A reboot mid-backup orphaned the entire tree, permanently. The sidecar is the record
of which snapshots a run pinned -- and /run is tmpfs. A reboot or crash between the
recursive snapshot and its cleanup destroyed that record, leaving one snapshot per
descendant dataset (250+ on a real pool) with nothing pointing at them. Nothing would
ever have found them.
gc_stale_snapshots() identifies leftovers by NAME, so it works when the record is
gone. It runs after the sidecar reclaim -- the recorded path stays authoritative and
the collector only mops up what the record lost.
It deletes data on a name match, which is a weaker claim than a recorded fact, so the
selection is a pure function with the harshest tests here. A snapshot is collected
only if the name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS>, it is not the current
run's, NOTHING IS MOUNTED FROM IT (this, not the age guard, is what protects a
concurrent backup), and it is over an hour old.
Checked against the real pool: of 4728 snapshots including 2341 periodic ones, it
selects exactly the orphans of the task being run and nothing else.
Found live, in the code written to prevent exactly this.
The sidecar held ONE snapshot. So a run that reclaimed an older tree, failed to
finish reclaiming it, and then recorded its own snapshot OVERWROTE the only record of
the survivor -- orphaning it permanently.
Observed: job 24 left one snapshot busy and kept the sidecar (correct). Job 46
reclaimed it, hit ZFS's 300s automount window (the runs were minutes apart), left it
behind again, and then wrote its own snapshot over the record. Permanent orphan,
created by the safety net.
The sidecar is now a list. stage_nested carries forward whatever a reclaim could not
delete; cleanup_task sweeps every pending tree and writes back only the survivors.
cleanup_all reports them one per line instead of formatting a list into an f-string
at the user during uninstall.
Job 46 also confirms the automount fix itself: it swept all 256 of its own snapshots
with no straggler.
Found on real hardware: a 256-snapshot backup of /mnt/Tap swept 253 and left 3 with
'dataset is busy'.
ZFS AUTOMOUNTS <dataset>/.zfs/snapshot/<snap> when it is read, and keeps it mounted
for zfs_expire_snapshot seconds (300 default) after the last access. teardown()
unmounts OUR bind mounts but not the automount underneath, so zfs destroy refuses for
exactly the datasets restic read most recently. cleanup_task() then removed the
sidecar anyway -- destroying the only record those snapshots existed. Nothing would
ever have reclaimed them.
- release_snapdirs() unmounts ZFS's own automounts (deepest first) before deleting.
- delete_snapshot_tree() retries the transient busy and RETURNS what it could not
delete, instead of swallowing it.
- The sidecar is removed only on a confirmed-clean sweep -- including on the
staging-failure path, which used to remove it before the caller swept. A sidecar
left behind when the tree is gone costs one no-op delete; a sidecar removed while
the tree exists is unrecoverable.
Two issues with the same title already existed -- the old title embedded the list of
broken refs, so the issue's identity changed whenever that set changed. With an
order-dependent pick the bot would alternate between them, reopening one and
commenting on the other. Lowest number is stable regardless of how the API sorts.
The title embedded the list of broken refs, so the issue's identity changed whenever
that set changed -- and it did: when the async/sync port briefly made 26 look green,
the next run filed a SECOND issue for 'master' alone. A bot that spawns duplicates
gets muted, and then it is not a warning system any more.
The title is now fixed; the refs live in the body, which gets updated in place.
install.sh chmod +x's update.sh, and git recorded update.sh as 100644 -- so the chmod
was a TRACKED modification, and update.sh refuses to run over a dirty tree. Install
once and you could never update again. The error even told you to 'git checkout -- .',
which just undoes the exec bit so the next install can re-dirty it.
Found on the real box, which had been sitting on v0.4.1 for exactly this reason.
Fixed on both sides: the scripts install.sh chmods are executable in git (so the
chmod is a no-op), and update.sh's dirty check now looks at CONTENT, not mode --
git diff --numstat reports 0 0 for a mode-only change. A test asserts every script in
install.sh's chmod loop is already 100755 in git.
Gitea is canonical for development; GitHub is where users clone from and where the
box's read-only checkout points. The install instructions, the re-clone hint and the
'file an issue' link are all read by users, so they name GitHub. docs/releasing.md
still names Gitea, because that is a contributor doc about where the code is pushed.
release_notes.py 'notes v0.6.0-rc1' looked for a CHANGELOG section literally named
v0.6.0-rc1. check() already used base_version(); extract_notes() did not. So the
release workflow cut the tag, passed every gate, and then died extracting the body --
the candidate existed but was never published.
Caught in an rc, which is the entire point of having them.
Also: the Gitea publish and issue steps used jq, which is not guaranteed on a
self-hosted runner. A publish step that dies on a missing tool leaves a tag with no
release behind it, and a bug report that dies on one is a warning system that does
not warn. Both now use python3, which setup-python guarantees.
Install was at line 517 of 969, under the boot sequence, the snapshot lifecycle and
the release process. Someone deciding whether to trust this with their backups should
not have to scroll past any of that.
README is now 211 lines: what it does, the minimum version, install, the support
matrix, updating, uninstall. Everything else moved to docs/ (nested snapshots, how it
works, recovery, CLI, releasing).
A test enforces it: every internal link resolves, Install stays near the top, and the
README does not grow back. Moving Markdown breaks cross-references -- it broke eight
of them here, including one in a recovery doc, where the person following the link is
by definition already having a bad day.
That gate was removed as redundant -- the release job re-runs the full suite against
the tagged commit, and release_gate proves a candidate points at it. The notes
described a check that does not exist.
create_snapshot is module-global in plugins/cloud/snapshot.py, and cloud_sync.py
imports it as well as cloud_backup/sync.py. So the wrapper sat in the path of every
rclone/Storj CloudSync task with snapshot=true, and ran a zfs.dataset.query before
concluding it had nothing to do -- a new failure mode for jobs that worked before
this patch existed.
Worse: a CloudSync task that ever got staged would never be torn down. The teardown
is wired into cloud_backup's restic_backup finally, and CRUD_BLOCK deliberately
leaves CloudSync's guard intact, so the bind mounts would pin the snapshot forever.
The staging path now bails out unless the snapshot is named cloud_backup-*, before
any middleware call.
Separately: the async wrapper's finally dropped logger=, which the sync one passes.
run_in_thread forwards **kwargs, so a cleanup that failed to unmount a bind mount or
delete a snapshot tree logged nothing at all -- on the only platform anyone runs.
This gap was hiding a catastrophe. TrueNAS 26 deletes plugins/zfs_/dataset.py and
plugins/zfs_/snapshot.py outright, taking zfs.dataset.query, zfs.snapshot.query and
zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
Nothing about the five cloud_backup files reveals that, so every other check went
green -- including the one I had just added. The patch would have applied cleanly
and then failed on the first backup, or worse: snapshotted fine and failed to
DELETE, orphaning one snapshot per descendant dataset (250 on a real pool) on every
run, forever.
So 26 is BROKEN and the nested module will not apply there. The async/sync wrapper
work and the vendored get_dataset_recursive stay -- they are correct and necessary
-- but 26 is not supported until the ZFS calls are ported, and that needs a real 26
box to verify. Shipping a port nobody has run is the failure this project exists to
avoid.
Also: do_delete is recognised as delete (24.10/25.04 use the CRUDService
convention), which was reporting both as BROKEN -- a false verdict that would have
disabled nested snapshots on boxes where they work.
26 rewrites cloud_backup from async to synchronous AND deletes
get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's
namespace. Either is a broken backup found at restore time.
The nested module is now one synchronous implementation talking to middlewared via
call_sync, behind two thin wrappers. apply.sh reads which flavour the installed
middleware declares and injects the matching one: <= 25.10 reaches it through
'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls
it directly. The snapshot/bind-mount/failure logic exists once -- an async twin
would mean every future fix had to land twice.
A middleware whose three wrapped functions disagree about asyncness is refused,
not guessed at. get_dataset_recursive is vendored, removing the dependency on both
versions rather than asserting it.
master stays BROKEN on purpose: iX are still renaming middleware->context,
cloud_backup->entry and adding a required credentials param there. Chasing a
branch that moves daily is how you ship a patch nobody tested.
TrueCloud Backup does not exist before 24.10, so on anything older the patch would
attach to nothing and do nothing -- silently, while the user believed their backups
were set up. install.sh now reads system.version and refuses, naming the reason.
Also: the boot sequence now documents the preflight (and that an incompatible module
is skipped for one boot, NOT kill-switched); the troubleshooting table covers the
incompatibility warning; forge URLs point at Gitea.
The split-literal squash is implemented twice: inline in apply.sh's runtime probe
and as compat._squash in the static checker. That subtlety already caused one
silent bug (the probe concluded iX had removed the nesting guard, which means
'retire the module'). Both are now exercised against the same inputs, including
the split-across-literals form stock actually uses.
The audit found the new machinery could do more harm than the bugs it prevents.
- apply.sh reused the 'nothing left to do' exit -- which touches the PERMANENT
kill switch, cleared only by install.sh, never by update.sh -- for the
incompatible case. On TrueNAS 26 (providers ok, nested opt-out) both modules go
quiet, so the switch would fire and the release that fixed 26 could never
re-enable itself. Retirement and incompatibility now take different exits.
- A network blip, a re-export, or a conditional def all read as BROKEN. Each is
now 'unknown', which changes nothing, rather than evidence strong enough to
disable a module.
- 'native' outranked BROKEN everywhere but apply.sh, so a TrueNAS that reworded
the guard AND reshaped the functions rendered as good news.
- compat.py --tree read B2_BLOCK's own 'restic = True' as native support, so the
documented way to check a live box lied on every patched machine.
- The signature check was a name-subset test. It passed reorders, kw-only
conversions, and added required params -- and it had already passed a real bug:
restic_backup takes 4 args on 24.10/25.04, and the wrapper forwarded 5. Nested
backups have been raising TypeError on those releases the whole time. The
wrapper now forwards *args/**kwargs.
- release.sh --promote was unreachable: it died if the tag existed, the gate died
if it did not. The tests hid it by always tagging first.
The matrix is regenerated daily by CI rather than typed once and forgotten — a
support table that quietly goes stale is a false promise to someone deciding
whether to trust this with their backups.
The report body is full of backticks, so 'echo "${{ steps.report.outputs.body }}"'
pasted it into the shell text and bash executed create-snapshot, def and async as
commands. The report is built from iX's middleware source, so that was an injection
vector as well as a bug. inputs.tag on workflow_dispatch had the same shape.
Data goes through files, scalars through env:. Tests enforce it across every
workflow.
TrueNAS 26 rewrites cloud_backup from async to sync. Every block the nested
module injects is an async wrapper around an awaited original, so on 26 it hands
sync.py a coroutine where it unpacks a tuple.
tools/compat.py records what each module assumes and checks it two ways: CI runs
it against iX's source at every release line (including master and the current
BETA) and files a bug report when an unreleased line breaks; apply.sh runs it
against the middlewared actually installed and refuses to apply a module whose
assumptions no longer hold. Stock TrueNAS without a feature beats TrueNAS with a
broken one.
Workflows run on both forges; only the release/issue API calls differ.
subprocess is called in list form with only literal arguments -- nothing
user-supplied reaches the command line -- and the partial `git` path is moot in a
module that only ever runs as root inside middlewared.
Deliberately NOT tagged: this changes no behaviour, and cutting a release for two
noqa comments would raise an update alert on every user's box. main sits one commit
ahead of v0.5.1 until the next real change -- which is exactly the restraint the
alert's docs-only rule exists to encode.
middlewared's alert.load() imports every file in alert/source/ with NO try/except:
def load(self):
for module in load_modules(.../alert/source):
for cls in load_classes(module, AlertSource, (ThreadedAlertSource,)):
...
and it runs during setup. A module that raises on import therefore takes
middlewared's startup down with it -- exactly the class of failure this project
exists to avoid.
apply.sh now COMPILES the substituted alert source and refuses to write it if it
does not parse. An uninstalled alert is a missing convenience; a broken one is a
broken box.
@PATCH_DIR@ is also substituted with repr() rather than raw, so a repository path
containing a quote or backslash yields a valid Python literal instead of a syntax
error in the installed module.
The alert source no longer mutates sys.path. It loaded tools/release_notes.py via
sys.path.insert(0, ...), which shadows the stdlib for that interpreter -- and
ThreadedAlertSource runs in middlewared's thread pool, so mutating sys.path is a
race. It now loads by file path with importlib.
New tests guard every import-time failure mode: the module compiles, apply.sh
compiles before writing, awkward paths (quotes, backslashes) still produce valid
modules, nothing but imports/constants/classes runs at module scope, every
AlertClass name ends in "AlertClass" (AlertClassMeta raises NameError otherwise),
the alert text placeholders match the args passed, and no git command that writes
to .git is ever used.
152 tests, ruff and shellcheck -S style clean.
Raises a real alert in the TrueNAS UI bell -- not a log line nobody reads. On by
default, checked once a day. install.sh --no-update-alerts turns it off.
It does not nag
---------------
A release whose CHANGELOG contains only a "### Docs" section changed no code and
raises nothing. Anything else raises INFO; a "### Security" section raises WARNING.
The CHANGELOG's own section headings are the signal, and a security fix anywhere in
the range escalates the whole span -- a docs-only release sitting on top of a
security fix still reports as security rather than hiding it.
Why an AlertSource and not midclt
----------------------------------
TrueNAS cannot raise an alert from the CLI. midclt exposes only alert.dismiss,
alert.list, alert.list_categories, alert.list_policies and alert.restore -- alert
CREATION is internal to middlewared, and none of its ~60 one-shot classes is
generic enough to reuse. Registering an AlertSource is the only way.
It is also the least invasive thing this patch does. The providers and nested
modules both APPEND CODE TO STOCK middleware files; the alert source ADDS ONE FILE
and modifies none. It is the native mechanism -- the same one every built-in
TrueNAS alert uses -- and TrueNAS polls it itself, so there is no cron job and no
systemd timer.
- Fail-safe: every error path returns None; it cannot take middlewared down.
- Read-only: `git ls-remote` plus an HTTPS fetch of the CHANGELOG. It never writes
to .git, so it cannot leave root-owned objects behind the way a `git fetch` from
middlewared (running as root) would.
- Removed by uninstall.sh (mw_patch.revert_all).
- It only tells you; it never updates anything.
Verified against the real repo and remote, with middlewared stubbed:
on v0.4.1, only a README-only v0.4.2 available -> NO ALERT
on v0.4.0, v0.4.1 fixed real bugs -> INFO
on v0.3.2, v0.3.3 was the password fix -> SECURITY / WARNING
139 tests, ruff and shellcheck -S style clean.
The Updating section told you to run update.sh, but never said how to GET it. It
ships inside the patch, so any clone older than v0.4.0 does not have it -- the docs
described a script the reader did not possess. There is now an explicit bootstrap
step, including the fix for the "insufficient permission for adding an object to
repository database" failure that past `sudo git pull`s leave behind.
Rewrote "After a TrueNAS update". It never explained that apply.sh re-applies the
patch at every boot (so you never reinstall), and it soft-pedalled what a failure
costs: "fail-safe" means the BOX stays up, not that your backups keep running. A
[FAIL] providers is a broken backup, and the docs now say that plainly instead of
implying everything degrades gracefully.
Added a repo map -- patch/mw_patch.py and tools/release_notes.py were documented
nowhere -- and fixed `ruff check patch tests`, which skips tools/.
Every command and file path in the README was then verified to exist and run:
all five update.sh flags, the ruff/pytest commands, every file in the repo map,
and the truecloud_nested.py cleanup CLI.
132 tests, ruff and shellcheck -S style clean.
Release-candidate tags would have been installed as stable
-----------------------------------------------------------
git's version sort ranks v0.5.0-rc1 ABOVE v0.5.0 (verified empirically), and the
release workflow deliberately supports rc/beta/alpha tags. update.sh would have
offered an RC as "the newest release". Tag selection is now filtered to plain
vX.Y.Z.
update.sh would have died mid-update on an untracked file
----------------------------------------------------------
The dirty-tree guard uses --untracked-files=no, so an untracked file that the
TARGET tracks slips past it -- and `git checkout` then aborts. Under set -e the
script died with a raw git error, after already recording the rollback point.
Not hypothetical: a hand-copied patch/wait_restart.sh blocked a pull on a real box
in exactly this way. It is now detected up front, by name. Gitignored files are
correctly not treated as blockers, since git overwrites those silently.
Special case: if update.sh ITSELF is the blocker, it was hand-copied in to
bootstrap -- and "delete update.sh, then re-run update.sh" is impossible. It now
says so and prints the git commands that bootstrap it properly.
--rollback skipped that check entirely and would have hit the identical failure.
The check is now a shared function used by both paths, and rollback also validates
that the recorded revision still exists.
Also: install.sh's chmod aborted under set -e if a listed file was missing (the
file set changes between versions, so --rollback must not be killed by a name this
version happens to know about), and --to with no value was silently ignored.
Verified end to end in a throwaway clone: forward v0.4.1 -> v0.4.2 and rollback
back, with files appearing and disappearing correctly; both guards fire.
132 tests, ruff and shellcheck -S style clean.
Fetch a newer release and apply it, preserving the nested-snapshot opt-in setting.
bash update.sh # to the newest release, with a confirmation
bash update.sh --check # show what would happen; change nothing
bash update.sh --rollback # undo the last update
Deliberately NOT automated. This patch injects Python into middlewared and
re-applies itself at every boot, so an unattended pull would let any bad upstream
commit reach a box with no human in the loop and take effect on the next reboot.
v0.0.4 shipped exactly such a bug and took every app on the box down. The manual
step is the safety gate.
Design decisions worth keeping:
- Defaults to the newest RELEASE TAG, not main. main can be mid-refactor; a tag is
the tested artifact. --main exists but says so loudly.
- Tags ordered by version, not date. Date order silently downgrades the box the
first time a hotfix is tagged out of band: a v0.3.6 cut after v0.4.0 would sort
as "newest".
- Refuses to run over a dirty working tree rather than merging across hand-edited
or scp'd files. (Verified: the guard fires.)
- Shows the commits and release notes you do not have, read from the TARGET's
CHANGELOG via tools/release_notes.py -- not a second copy of the extractor.
- Records the previous revision BEFORE moving, so --rollback works even if
install.sh dies halfway.
- Repairs .git ownership, which past `sudo git pull`s leave root-owned and which
then breaks every later non-root git command.
update.sh is covered by the version-drift check, so it cannot go stale the way
create_task.py's __version__ did.
Tested end to end in a throwaway clone: detects v0.3.2 -> v0.3.5, lists missing
commits, handles already-up-to-date, and the dirty-tree guard fires.
132 tests, ruff and shellcheck clean.
delete_snapshot_tree tries one recursive delete first, then falls back to sweeping
the tree by name. The exception from the fast path was discarded.
That failure is usually benign -- stock's finally already removed the parent once
our mounts were released, which is exactly what the sweep exists to handle. But if
the cause were anything else, this was the only place it was ever visible, and it
went straight to /dev/null. The sweep would then report some different, downstream
symptom. It is now logged before falling through.
Also annotated the two remaining static-analysis findings as considered rather than
leaving them to be re-litigated every audit: subprocess is always invoked in list
form (no shell, so ZFS dataset names cannot inject), and a partial `systemctl` path
is moot in a script that only ever runs as root.
Extended ruleset (E,F,W,B,S,SIM,UP,C4,RET,ARG,A,ISC) and shellcheck -S style both
report zero. 132 tests.
The "strip the TRUECLOUD_PATCH block" logic existed twice -- in apply.sh's heredoc
and in an inline heredoc in uninstall.sh -- and the uninstall copy was the untested
one. That is precisely how the two could have drifted apart, with apply.sh
reverting one set of files and uninstall.sh another.
Both now call patch/mw_patch.py. 17 new tests cover it, including that
revert_nested never touches restic.py: that file carries a TRUECLOUD_PATCH block
too, but it belongs to the providers module, and removing it would silently break
B2 backups.
apply.sh imports it fail-safe -- on ImportError the backend patch is skipped and
middlewared starts stock, which is this script's whole design principle. The
import uses sys.path.append, never insert(0): prepending would give patch/
precedence over the stdlib for that interpreter, so a future patch/json.py would
shadow the real json module and break the boot.
Also: the README's create_task.py example still taught `--password <secret>`, which
is how a security fix quietly fails to land. It now shows --password-stdin.
132 tests, ruff and shellcheck clean.
Security
--------
create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that
JSON carries the restic repository password -- so it sat in the subprocess's argv,
which is world-readable via ps, for the duration of the call. That password is the
encryption key for the entire cloud backup repository.
It now talks to the middleware through truenas_api_client, the library that backs
midclt itself, so the password never leaves this process's memory. Verified on a
live box: list-tasks and list-credentials work through the new transport.
--password is also no longer required, because passing a secret as a CLI argument
writes it to shell history permanently. --password-stdin reads it from stdin, and
with neither flag the tool prompts via getpass. --password still works but warns.
Fixed
-----
uninstall.sh could leave every patch installed. It reverted by unmounting the
overlay -- but apply.sh only mounts one when the target directory is read-only. On
a writable /usr it patches the real files in place, and uninstall would remove the
boot hook, report success, and leave the patch applied. It now strips the appended
blocks from the middleware files explicitly. This also covers the case where the
overlay unmount fails.
create_task.py's __version__ had been stuck at 0.2.0 through three releases. The
version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it
missed the one file that actually shows a version to users (--version). The check
now covers __version__ too -- and caught this immediately.
118 tests, ruff and shellcheck clean.
apply.sh only ever ADDED patches; there was no revert path anywhere. Disabling
removed the opt-in marker and then merely skipped re-applying -- but the overlay
persists for the whole boot, so the previously patched cloud/{snapshot,crud}.py,
cloud_backup/sync.py and _truecloud_nested.py were all still on disk, and
middlewared re-imported them on the restart install.sh performs.
It printed "DISABLED (stock guard restored)" while the feature kept running until
the next reboot. Someone disabling it because they were worried about it would
have believed it was off.
apply.sh now reverts on every not-needed path (opt-out, or superseded by native
support): remove the module FIRST -- every injected block is guarded by
`if _tc_nested is not None`, so the stock guard comes back even if a later step
fails -- then strip the appended blocks from the three patched files.
restic.py also carries a TRUECLOUD_PATCH block but belongs to the providers
module; reverting it would silently break B2 backups, so it is explicitly
excluded. Verified: the three nested files restore byte-for-byte to stock, the
module is removed, and restic.py's block survives.
install.sh --disable also tears the staging tree down first, since those bind
mounts pin ZFS snapshots that could otherwise never be destroyed.
Updating WITHOUT the flag was always correct and is unchanged: the nested module
is never installed into middleware unless explicitly enabled.
109 tests, ruff and shellcheck clean.
The "Automated releases" entry was written under v0.3.0, but that commit landed
after the v0.3.0 tag -- the CHANGELOG was claiming the release contained something
it did not. Moved to its own version rather than left as a quiet inaccuracy.
Bumps VERSION to 0.3.1 across all four scripts, which the new consistency check
now enforces.
Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never
released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the
incident that took every app down.
Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is
the matching CHANGELOG.md section -- one source of truth for release notes, so
there is no second place for them to be wrong.
The workflow refuses to publish when:
- the tests, ruff, or bash -n fail (a tagged commit is what people install; it
must be at least as good as main)
- the tag does not match the VERSION= declared by every script
- CHANGELOG.md has no section for the tag, or the section is empty
That version check is not theoretical: VERSION= had drifted to three different
values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing
noticed until this release. tests/test_release_notes.py now asserts the scripts
agree with each other and with the newest CHANGELOG entry, so the drift cannot
come back.
workflow_dispatch takes an existing tag, so releases can be backfilled for tags
that were pushed before this existed.
106 tests, ruff and shellcheck clean.
Commit 47cdf72 shipped a pattern that matched one closing paren and emitted one,
netting an extra `)` in the Angular bundle:
c(2,"filterByProviders",["STORJ_IX","S3","B2"]))("required",!0)
^^ syntax error
The TrueNAS web UI went blank. Worse, MARKER was now present in the file, so
every later run reported "already patched" and skipped -- the patch could not
heal itself, and the bundle had to be hand-restored from the .pre-truecloud-patch
backup.
patch_ui.py now compares the parenthesis balance before and after substitution and
refuses to write if it changed. A bundle we cannot patch correctly is left exactly
as it was: an unpatched UI is a missing dropdown entry, a corrupted one is a dead
web UI.
Tests cover the real regression (verbatim 47cdf72 pattern) end to end: it still
matches, the balance still shifts, main() refuses, and the file on disk is
byte-for-byte unchanged. README documents the manual recovery for anyone who
already hit it.
93 tests, ruff and shellcheck clean.
An unattended scheduled backup of a live 252-dataset pool ran through the staging
tree end to end:
task 5 /mnt/Tap SUCCESS 18m14s
- 252 datasets recursively snapshotted; 173 bind mounts built and verified
- zero orphaned ZFS snapshots and zero stale mounts afterwards -- the failure
that would otherwise have accumulated 251 snapshots on every single run
- the same task previously stalled at 74% for over 12 hours reading live files
The README said the mount --bind staging step had not been exercised by a live
backup run. That is no longer true, so it is removed rather than left to
understate the state of the code.
The advice to verify your own first backup actually contains child-dataset data
stays -- that one is not boilerplate.
patch_ui.py rewrites minified third-party JavaScript by regex and had no tests.
It is the easiest place in this project to do real damage: a pattern that matches
nothing silently leaves the dropdown Storj-only, and one that consumes a paren
too many is a syntax error in the bundle that blanks the entire TrueNAS web UI.
Tests run the real patterns against verbatim snippets from a TrueNAS 25.x
chunk-*.js (the chained property(...)(...) form) and a 24.x literal array, and
assert: exactly one match, all three providers present, the paren balance is
UNCHANGED, surrounding code untouched, re-patching is a no-op, unrelated JS is
never matched, and every pattern stays anchored to filterByProviders.
The paren-balance assertion is the load-bearing one -- a plausible-but-wrong
pattern that eats both parens and re-emits none shifts the delta from 1 to 2 and
is caught.
Also restore the comment explaining why the 25.x pattern is shaped the way it is,
and correct the module docstring, which showed the binding with a single closing
paren; the real bundle wraps it in a chained property call.
90 tests, ruff and shellcheck clean.
delete_snapshot_tree removed the parent and every child snapshot individually.
On a real pool `zfs snapshot -r` creates one snapshot per descendant dataset --
252 on Tap -- so cleanup was 252 sequential middleware calls.
Slow, but the real problem is that it is not atomic: a job killed part-way
through the sweep leaves behind exactly the orphaned snapshots this function
exists to prevent.
zfs.snapshot.delete accepts {"recursive": True}, which destroys the parent and
all children in one call. Use that as the fast path and keep the name-by-name
sweep as the fallback -- it is still needed when the parent is already gone
(stock's finally can win the race once our mounts are released), which makes a
recursive delete fail while the children survive.
The test fake now emulates real `zfs destroy -r` semantics, so a test cannot pass
while the shipped code deletes only the parent.
76 tests, ruff and shellcheck clean.
flan
flan