middlewared's alert.load() imports every file in alert/source/ with NO try/except:
def load(self):
for module in load_modules(.../alert/source):
for cls in load_classes(module, AlertSource, (ThreadedAlertSource,)):
...
and it runs during setup. A module that raises on import therefore takes
middlewared's startup down with it -- exactly the class of failure this project
exists to avoid.
apply.sh now COMPILES the substituted alert source and refuses to write it if it
does not parse. An uninstalled alert is a missing convenience; a broken one is a
broken box.
@PATCH_DIR@ is also substituted with repr() rather than raw, so a repository path
containing a quote or backslash yields a valid Python literal instead of a syntax
error in the installed module.
The alert source no longer mutates sys.path. It loaded tools/release_notes.py via
sys.path.insert(0, ...), which shadows the stdlib for that interpreter -- and
ThreadedAlertSource runs in middlewared's thread pool, so mutating sys.path is a
race. It now loads by file path with importlib.
New tests guard every import-time failure mode: the module compiles, apply.sh
compiles before writing, awkward paths (quotes, backslashes) still produce valid
modules, nothing but imports/constants/classes runs at module scope, every
AlertClass name ends in "AlertClass" (AlertClassMeta raises NameError otherwise),
the alert text placeholders match the args passed, and no git command that writes
to .git is ever used.
152 tests, ruff and shellcheck -S style clean.
Raises a real alert in the TrueNAS UI bell -- not a log line nobody reads. On by
default, checked once a day. install.sh --no-update-alerts turns it off.
It does not nag
---------------
A release whose CHANGELOG contains only a "### Docs" section changed no code and
raises nothing. Anything else raises INFO; a "### Security" section raises WARNING.
The CHANGELOG's own section headings are the signal, and a security fix anywhere in
the range escalates the whole span -- a docs-only release sitting on top of a
security fix still reports as security rather than hiding it.
Why an AlertSource and not midclt
----------------------------------
TrueNAS cannot raise an alert from the CLI. midclt exposes only alert.dismiss,
alert.list, alert.list_categories, alert.list_policies and alert.restore -- alert
CREATION is internal to middlewared, and none of its ~60 one-shot classes is
generic enough to reuse. Registering an AlertSource is the only way.
It is also the least invasive thing this patch does. The providers and nested
modules both APPEND CODE TO STOCK middleware files; the alert source ADDS ONE FILE
and modifies none. It is the native mechanism -- the same one every built-in
TrueNAS alert uses -- and TrueNAS polls it itself, so there is no cron job and no
systemd timer.
- Fail-safe: every error path returns None; it cannot take middlewared down.
- Read-only: `git ls-remote` plus an HTTPS fetch of the CHANGELOG. It never writes
to .git, so it cannot leave root-owned objects behind the way a `git fetch` from
middlewared (running as root) would.
- Removed by uninstall.sh (mw_patch.revert_all).
- It only tells you; it never updates anything.
Verified against the real repo and remote, with middlewared stubbed:
on v0.4.1, only a README-only v0.4.2 available -> NO ALERT
on v0.4.0, v0.4.1 fixed real bugs -> INFO
on v0.3.2, v0.3.3 was the password fix -> SECURITY / WARNING
139 tests, ruff and shellcheck -S style clean.
The Updating section told you to run update.sh, but never said how to GET it. It
ships inside the patch, so any clone older than v0.4.0 does not have it -- the docs
described a script the reader did not possess. There is now an explicit bootstrap
step, including the fix for the "insufficient permission for adding an object to
repository database" failure that past `sudo git pull`s leave behind.
Rewrote "After a TrueNAS update". It never explained that apply.sh re-applies the
patch at every boot (so you never reinstall), and it soft-pedalled what a failure
costs: "fail-safe" means the BOX stays up, not that your backups keep running. A
[FAIL] providers is a broken backup, and the docs now say that plainly instead of
implying everything degrades gracefully.
Added a repo map -- patch/mw_patch.py and tools/release_notes.py were documented
nowhere -- and fixed `ruff check patch tests`, which skips tools/.
Every command and file path in the README was then verified to exist and run:
all five update.sh flags, the ruff/pytest commands, every file in the repo map,
and the truecloud_nested.py cleanup CLI.
132 tests, ruff and shellcheck -S style clean.
Release-candidate tags would have been installed as stable
-----------------------------------------------------------
git's version sort ranks v0.5.0-rc1 ABOVE v0.5.0 (verified empirically), and the
release workflow deliberately supports rc/beta/alpha tags. update.sh would have
offered an RC as "the newest release". Tag selection is now filtered to plain
vX.Y.Z.
update.sh would have died mid-update on an untracked file
----------------------------------------------------------
The dirty-tree guard uses --untracked-files=no, so an untracked file that the
TARGET tracks slips past it -- and `git checkout` then aborts. Under set -e the
script died with a raw git error, after already recording the rollback point.
Not hypothetical: a hand-copied patch/wait_restart.sh blocked a pull on a real box
in exactly this way. It is now detected up front, by name. Gitignored files are
correctly not treated as blockers, since git overwrites those silently.
Special case: if update.sh ITSELF is the blocker, it was hand-copied in to
bootstrap -- and "delete update.sh, then re-run update.sh" is impossible. It now
says so and prints the git commands that bootstrap it properly.
--rollback skipped that check entirely and would have hit the identical failure.
The check is now a shared function used by both paths, and rollback also validates
that the recorded revision still exists.
Also: install.sh's chmod aborted under set -e if a listed file was missing (the
file set changes between versions, so --rollback must not be killed by a name this
version happens to know about), and --to with no value was silently ignored.
Verified end to end in a throwaway clone: forward v0.4.1 -> v0.4.2 and rollback
back, with files appearing and disappearing correctly; both guards fire.
132 tests, ruff and shellcheck -S style clean.
Fetch a newer release and apply it, preserving the nested-snapshot opt-in setting.
bash update.sh # to the newest release, with a confirmation
bash update.sh --check # show what would happen; change nothing
bash update.sh --rollback # undo the last update
Deliberately NOT automated. This patch injects Python into middlewared and
re-applies itself at every boot, so an unattended pull would let any bad upstream
commit reach a box with no human in the loop and take effect on the next reboot.
v0.0.4 shipped exactly such a bug and took every app on the box down. The manual
step is the safety gate.
Design decisions worth keeping:
- Defaults to the newest RELEASE TAG, not main. main can be mid-refactor; a tag is
the tested artifact. --main exists but says so loudly.
- Tags ordered by version, not date. Date order silently downgrades the box the
first time a hotfix is tagged out of band: a v0.3.6 cut after v0.4.0 would sort
as "newest".
- Refuses to run over a dirty working tree rather than merging across hand-edited
or scp'd files. (Verified: the guard fires.)
- Shows the commits and release notes you do not have, read from the TARGET's
CHANGELOG via tools/release_notes.py -- not a second copy of the extractor.
- Records the previous revision BEFORE moving, so --rollback works even if
install.sh dies halfway.
- Repairs .git ownership, which past `sudo git pull`s leave root-owned and which
then breaks every later non-root git command.
update.sh is covered by the version-drift check, so it cannot go stale the way
create_task.py's __version__ did.
Tested end to end in a throwaway clone: detects v0.3.2 -> v0.3.5, lists missing
commits, handles already-up-to-date, and the dirty-tree guard fires.
132 tests, ruff and shellcheck clean.
delete_snapshot_tree tries one recursive delete first, then falls back to sweeping
the tree by name. The exception from the fast path was discarded.
That failure is usually benign -- stock's finally already removed the parent once
our mounts were released, which is exactly what the sweep exists to handle. But if
the cause were anything else, this was the only place it was ever visible, and it
went straight to /dev/null. The sweep would then report some different, downstream
symptom. It is now logged before falling through.
Also annotated the two remaining static-analysis findings as considered rather than
leaving them to be re-litigated every audit: subprocess is always invoked in list
form (no shell, so ZFS dataset names cannot inject), and a partial `systemctl` path
is moot in a script that only ever runs as root.
Extended ruleset (E,F,W,B,S,SIM,UP,C4,RET,ARG,A,ISC) and shellcheck -S style both
report zero. 132 tests.
The "strip the TRUECLOUD_PATCH block" logic existed twice -- in apply.sh's heredoc
and in an inline heredoc in uninstall.sh -- and the uninstall copy was the untested
one. That is precisely how the two could have drifted apart, with apply.sh
reverting one set of files and uninstall.sh another.
Both now call patch/mw_patch.py. 17 new tests cover it, including that
revert_nested never touches restic.py: that file carries a TRUECLOUD_PATCH block
too, but it belongs to the providers module, and removing it would silently break
B2 backups.
apply.sh imports it fail-safe -- on ImportError the backend patch is skipped and
middlewared starts stock, which is this script's whole design principle. The
import uses sys.path.append, never insert(0): prepending would give patch/
precedence over the stdlib for that interpreter, so a future patch/json.py would
shadow the real json module and break the boot.
Also: the README's create_task.py example still taught `--password <secret>`, which
is how a security fix quietly fails to land. It now shows --password-stdin.
132 tests, ruff and shellcheck clean.
Security
--------
create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that
JSON carries the restic repository password -- so it sat in the subprocess's argv,
which is world-readable via ps, for the duration of the call. That password is the
encryption key for the entire cloud backup repository.
It now talks to the middleware through truenas_api_client, the library that backs
midclt itself, so the password never leaves this process's memory. Verified on a
live box: list-tasks and list-credentials work through the new transport.
--password is also no longer required, because passing a secret as a CLI argument
writes it to shell history permanently. --password-stdin reads it from stdin, and
with neither flag the tool prompts via getpass. --password still works but warns.
Fixed
-----
uninstall.sh could leave every patch installed. It reverted by unmounting the
overlay -- but apply.sh only mounts one when the target directory is read-only. On
a writable /usr it patches the real files in place, and uninstall would remove the
boot hook, report success, and leave the patch applied. It now strips the appended
blocks from the middleware files explicitly. This also covers the case where the
overlay unmount fails.
create_task.py's __version__ had been stuck at 0.2.0 through three releases. The
version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it
missed the one file that actually shows a version to users (--version). The check
now covers __version__ too -- and caught this immediately.
118 tests, ruff and shellcheck clean.
apply.sh only ever ADDED patches; there was no revert path anywhere. Disabling
removed the opt-in marker and then merely skipped re-applying -- but the overlay
persists for the whole boot, so the previously patched cloud/{snapshot,crud}.py,
cloud_backup/sync.py and _truecloud_nested.py were all still on disk, and
middlewared re-imported them on the restart install.sh performs.
It printed "DISABLED (stock guard restored)" while the feature kept running until
the next reboot. Someone disabling it because they were worried about it would
have believed it was off.
apply.sh now reverts on every not-needed path (opt-out, or superseded by native
support): remove the module FIRST -- every injected block is guarded by
`if _tc_nested is not None`, so the stock guard comes back even if a later step
fails -- then strip the appended blocks from the three patched files.
restic.py also carries a TRUECLOUD_PATCH block but belongs to the providers
module; reverting it would silently break B2 backups, so it is explicitly
excluded. Verified: the three nested files restore byte-for-byte to stock, the
module is removed, and restic.py's block survives.
install.sh --disable also tears the staging tree down first, since those bind
mounts pin ZFS snapshots that could otherwise never be destroyed.
Updating WITHOUT the flag was always correct and is unchanged: the nested module
is never installed into middleware unless explicitly enabled.
109 tests, ruff and shellcheck clean.
The "Automated releases" entry was written under v0.3.0, but that commit landed
after the v0.3.0 tag -- the CHANGELOG was claiming the release contained something
it did not. Moved to its own version rather than left as a quiet inaccuracy.
Bumps VERSION to 0.3.1 across all four scripts, which the new consistency check
now enforces.
Audit
-----
- create_task.py verify failed on a DEFAULT install. hook_status.json emitted a
per-file entry for the nested module with ok:false whenever the feature was
switched off -- the default -- so verify printed [FAIL] and exited 1, right
after the README tells users to run it. Status is now per MODULE with an
`active` flag, and verify renders an inactive module as [SKIP].
- A partial apply suppressed the middlewared restart. The exit code conflated
"nothing applied" with "one module applied, one failed", so a failing providers
patch would prevent the restart that a freshly-applied nested patch needs,
leaving it on disk and never loaded. Exit 2 now means partial and the restart
still fires.
- The native-nested probe could never fire. It scanned crud.py for the guard
message, but our own injected block quotes that message, so once applied the
probe would always conclude the guard was still present. It now reads only the
stock portion of the file.
- recover.sh did not unmount staging trees, so an emergency recovery left bind
mounts pinning ZFS snapshots that could then never be destroyed.
- uninstall.sh deleted sidecar files without reading them. A sidecar is the only
record that an interrupted run's snapshot tree is still on disk; both scripts
now name the snapshot before clearing it.
- Removed a dead branch in the restart gate (unreachable: the kill switch exits).
Refactor
--------
- Staging teardown had been copy-pasted into uninstall.sh and recover.sh -- two
untested shell copies of the fiddly depth-ordering and lazy-umount logic. Both
now call `python3 patch/truecloud_nested.py cleanup`, so there is exactly one
implementation and it is the one under test.
- Dropped the in-memory ACTIVE dict. The sidecar file was already the source of
truth; a second in-process record could only desync -- and it is precisely the
middlewared-restart case (which empties it) that must not orphan a snapshot
tree. One record, on disk, or none.
Not done: the overlay-unmount loop is duplicated across apply.sh/uninstall.sh/
recover.sh. It is pre-existing, and apply.sh runs at PREINIT under a tight
timeout -- giving it a source dependency would trade 10 lines of duplication for
a boot-time failure mode.
74 tests, ruff and shellcheck clean.
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:
This option is only available for datasets that have no further nesting
That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.
The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.
Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.
Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays
The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.
Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.
Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.
apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.
create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.
recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds. Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.
Bumps all scripts to v0.0.3.
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.
- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
substitution into sitecustomize.py; sed's & and | metacharacters
silently corrupt or truncate the output for paths containing those
chars; Python str.replace has no metacharacter issues; also write to
a tmp file and mv atomically so a failed substitution never leaves
an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
$PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
("TrueCloud provider patch (S3/B2)") instead of exact script path;
exact-path match breaks when the repo is moved after install —
uninstall leaves the stale hook registered (fires on every boot),
and reinstall creates a duplicate entry; install.sh now also updates
the script path on re-run so a moved repo self-corrects
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.
- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
into sitecustomize.py via sed when writing to site-packages;
reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
If the patch ever prevents middlewared from starting, users now have a
clear escape hatch that requires no knowledge of Python internals:
bash /data/truecloud-patch/recover.sh
Or at a bare shell prompt:
touch /data/truecloud-patch/disabled
systemctl restart middlewared
sitecustomize.py checks for /data/truecloud-patch/disabled at Python
startup and skips the import hook entirely when the file exists.
apply.sh does the same so the PREINIT script also does nothing on reboot.
recover.sh is copied to /data/truecloud-patch/ by install.sh so it is
available even without the original repo directory.
README gains an "Emergency recovery" section above Troubleshooting.
flan
flan