It commented on every run that found a break. In one day it left ELEVEN identical
3,000-character comments on the same issue. That is not a warning system, it is a mute
button with extra steps -- and the next real finding would have been scrolled past,
which defeats the entire reason for building it.
Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of
changes. A fingerprint of the findings (broken ref/module/problem triples only) is
embedded in the body; a run whose findings match it says nothing at all. It closes the
issue when everything is fixed.
The fingerprint deliberately ignores anything that moves on its own -- healthy rows,
the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming
TS-25.10.5 is not news and does not wake anybody up.
Also:
- The two near-identical per-forge shell steps are gone, replaced by one tested
implementation (tools/compat_publish.py). Two copies of 'find the issue, decide
whether to comment' is two chances to drift, and the Gitea one duplicated an issue
for real.
- The README matrix refresh now opens a PULL REQUEST instead of pushing straight to
main from CI. An unattended push to main is exactly what the release barrier exists
to prevent: a bot that can move main can move it somewhere nobody looked.
release_notes.py 'notes v0.6.0-rc1' looked for a CHANGELOG section literally named
v0.6.0-rc1. check() already used base_version(); extract_notes() did not. So the
release workflow cut the tag, passed every gate, and then died extracting the body --
the candidate existed but was never published.
Caught in an rc, which is the entire point of having them.
Also: the Gitea publish and issue steps used jq, which is not guaranteed on a
self-hosted runner. A publish step that dies on a missing tool leaves a tag with no
release behind it, and a bug report that dies on one is a warning system that does
not warn. Both now use python3, which setup-python guarantees.
This gap was hiding a catastrophe. TrueNAS 26 deletes plugins/zfs_/dataset.py and
plugins/zfs_/snapshot.py outright, taking zfs.dataset.query, zfs.snapshot.query and
zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
Nothing about the five cloud_backup files reveals that, so every other check went
green -- including the one I had just added. The patch would have applied cleanly
and then failed on the first backup, or worse: snapshotted fine and failed to
DELETE, orphaning one snapshot per descendant dataset (250 on a real pool) on every
run, forever.
So 26 is BROKEN and the nested module will not apply there. The async/sync wrapper
work and the vendored get_dataset_recursive stay -- they are correct and necessary
-- but 26 is not supported until the ZFS calls are ported, and that needs a real 26
box to verify. Shipping a port nobody has run is the failure this project exists to
avoid.
Also: do_delete is recognised as delete (24.10/25.04 use the CRUDService
convention), which was reporting both as BROKEN -- a false verdict that would have
disabled nested snapshots on boxes where they work.
26 rewrites cloud_backup from async to synchronous AND deletes
get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's
namespace. Either is a broken backup found at restore time.
The nested module is now one synchronous implementation talking to middlewared via
call_sync, behind two thin wrappers. apply.sh reads which flavour the installed
middleware declares and injects the matching one: <= 25.10 reaches it through
'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls
it directly. The snapshot/bind-mount/failure logic exists once -- an async twin
would mean every future fix had to land twice.
A middleware whose three wrapped functions disagree about asyncness is refused,
not guessed at. get_dataset_recursive is vendored, removing the dependency on both
versions rather than asserting it.
master stays BROKEN on purpose: iX are still renaming middleware->context,
cloud_backup->entry and adding a required credentials param there. Chasing a
branch that moves daily is how you ship a patch nobody tested.
The audit found the new machinery could do more harm than the bugs it prevents.
- apply.sh reused the 'nothing left to do' exit -- which touches the PERMANENT
kill switch, cleared only by install.sh, never by update.sh -- for the
incompatible case. On TrueNAS 26 (providers ok, nested opt-out) both modules go
quiet, so the switch would fire and the release that fixed 26 could never
re-enable itself. Retirement and incompatibility now take different exits.
- A network blip, a re-export, or a conditional def all read as BROKEN. Each is
now 'unknown', which changes nothing, rather than evidence strong enough to
disable a module.
- 'native' outranked BROKEN everywhere but apply.sh, so a TrueNAS that reworded
the guard AND reshaped the functions rendered as good news.
- compat.py --tree read B2_BLOCK's own 'restic = True' as native support, so the
documented way to check a live box lied on every patched machine.
- The signature check was a name-subset test. It passed reorders, kw-only
conversions, and added required params -- and it had already passed a real bug:
restic_backup takes 4 args on 24.10/25.04, and the wrapper forwarded 5. Nested
backups have been raising TypeError on those releases the whole time. The
wrapper now forwards *args/**kwargs.
- release.sh --promote was unreachable: it died if the tag existed, the gate died
if it did not. The tests hid it by always tagging first.
The matrix is regenerated daily by CI rather than typed once and forgotten — a
support table that quietly goes stale is a false promise to someone deciding
whether to trust this with their backups.
Raises a real alert in the TrueNAS UI bell -- not a log line nobody reads. On by
default, checked once a day. install.sh --no-update-alerts turns it off.
It does not nag
---------------
A release whose CHANGELOG contains only a "### Docs" section changed no code and
raises nothing. Anything else raises INFO; a "### Security" section raises WARNING.
The CHANGELOG's own section headings are the signal, and a security fix anywhere in
the range escalates the whole span -- a docs-only release sitting on top of a
security fix still reports as security rather than hiding it.
Why an AlertSource and not midclt
----------------------------------
TrueNAS cannot raise an alert from the CLI. midclt exposes only alert.dismiss,
alert.list, alert.list_categories, alert.list_policies and alert.restore -- alert
CREATION is internal to middlewared, and none of its ~60 one-shot classes is
generic enough to reuse. Registering an AlertSource is the only way.
It is also the least invasive thing this patch does. The providers and nested
modules both APPEND CODE TO STOCK middleware files; the alert source ADDS ONE FILE
and modifies none. It is the native mechanism -- the same one every built-in
TrueNAS alert uses -- and TrueNAS polls it itself, so there is no cron job and no
systemd timer.
- Fail-safe: every error path returns None; it cannot take middlewared down.
- Read-only: `git ls-remote` plus an HTTPS fetch of the CHANGELOG. It never writes
to .git, so it cannot leave root-owned objects behind the way a `git fetch` from
middlewared (running as root) would.
- Removed by uninstall.sh (mw_patch.revert_all).
- It only tells you; it never updates anything.
Verified against the real repo and remote, with middlewared stubbed:
on v0.4.1, only a README-only v0.4.2 available -> NO ALERT
on v0.4.0, v0.4.1 fixed real bugs -> INFO
on v0.3.2, v0.3.3 was the password fix -> SECURITY / WARNING
139 tests, ruff and shellcheck -S style clean.
Fetch a newer release and apply it, preserving the nested-snapshot opt-in setting.
bash update.sh # to the newest release, with a confirmation
bash update.sh --check # show what would happen; change nothing
bash update.sh --rollback # undo the last update
Deliberately NOT automated. This patch injects Python into middlewared and
re-applies itself at every boot, so an unattended pull would let any bad upstream
commit reach a box with no human in the loop and take effect on the next reboot.
v0.0.4 shipped exactly such a bug and took every app on the box down. The manual
step is the safety gate.
Design decisions worth keeping:
- Defaults to the newest RELEASE TAG, not main. main can be mid-refactor; a tag is
the tested artifact. --main exists but says so loudly.
- Tags ordered by version, not date. Date order silently downgrades the box the
first time a hotfix is tagged out of band: a v0.3.6 cut after v0.4.0 would sort
as "newest".
- Refuses to run over a dirty working tree rather than merging across hand-edited
or scp'd files. (Verified: the guard fires.)
- Shows the commits and release notes you do not have, read from the TARGET's
CHANGELOG via tools/release_notes.py -- not a second copy of the extractor.
- Records the previous revision BEFORE moving, so --rollback works even if
install.sh dies halfway.
- Repairs .git ownership, which past `sudo git pull`s leave root-owned and which
then breaks every later non-root git command.
update.sh is covered by the version-drift check, so it cannot go stale the way
create_task.py's __version__ did.
Tested end to end in a throwaway clone: detects v0.3.2 -> v0.3.5, lists missing
commits, handles already-up-to-date, and the dirty-tree guard fires.
132 tests, ruff and shellcheck clean.
Security
--------
create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that
JSON carries the restic repository password -- so it sat in the subprocess's argv,
which is world-readable via ps, for the duration of the call. That password is the
encryption key for the entire cloud backup repository.
It now talks to the middleware through truenas_api_client, the library that backs
midclt itself, so the password never leaves this process's memory. Verified on a
live box: list-tasks and list-credentials work through the new transport.
--password is also no longer required, because passing a secret as a CLI argument
writes it to shell history permanently. --password-stdin reads it from stdin, and
with neither flag the tool prompts via getpass. --password still works but warns.
Fixed
-----
uninstall.sh could leave every patch installed. It reverted by unmounting the
overlay -- but apply.sh only mounts one when the target directory is read-only. On
a writable /usr it patches the real files in place, and uninstall would remove the
boot hook, report success, and leave the patch applied. It now strips the appended
blocks from the middleware files explicitly. This also covers the case where the
overlay unmount fails.
create_task.py's __version__ had been stuck at 0.2.0 through three releases. The
version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it
missed the one file that actually shows a version to users (--version). The check
now covers __version__ too -- and caught this immediately.
118 tests, ruff and shellcheck clean.
Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never
released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the
incident that took every app down.
Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is
the matching CHANGELOG.md section -- one source of truth for release notes, so
there is no second place for them to be wrong.
The workflow refuses to publish when:
- the tests, ruff, or bash -n fail (a tagged commit is what people install; it
must be at least as good as main)
- the tag does not match the VERSION= declared by every script
- CHANGELOG.md has no section for the tag, or the section is empty
That version check is not theoretical: VERSION= had drifted to three different
values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing
noticed until this release. tests/test_release_notes.py now asserts the scripts
agree with each other and with the newest CHANGELOG entry, so the drift cannot
come back.
workflow_dispatch takes an existing tag, so releases can be backfilled for tags
that were pushed before this existed.
106 tests, ruff and shellcheck clean.
flan
flan