15 Commits

Include renames

Author	SHA1	Message	Date
flan	086b20ed23	fix: fourth audit — two regressions from the last fix, and the boot preflight had no test ... Two of these were mine, from the previous round. - mounted_snapshots still swallowed OSError. I said I had fixed it and had not: the edit never matched, and I did not read it back. With the mount table unreadable the GC loses its in-use protection entirely and can destroy the snapshots of a backup that is still uploading (a first upload easily outlives the 1h age floor). It raises now, and both behaviours are tested. - The foreign-dataset check added last round had two bugs of its own. It ignored `mounted`, so a locked/encrypted dataset from a sibling tree turned a working nightly backup into a permanent failure — it belongs in `skipped`, exactly as an in-tree one does. And it tested `mp.startswith(path + "/")`, so a foreign dataset mounted EXACTLY at the backup path slipped through — the very hole the check was added to close, one character wide, and the worse case of the two because it SHADOWS the base dataset's own directory. - _read_sidecar's new raise broke cleanup_all, which is what recover.sh and uninstall.sh call — i.e. the code that must work when the box is ALREADY stuck. One unreadable sidecar aborted it before it unmounted anything, leaving the staging tree mounted, which pins the snapshots, which is the state recover.sh exists to escape. It now reports and carries on — and does not delete a record it could not read. - compat could report a FALSE OK: `defined` was collected by walking the whole file, so any function named `delete` anywhere in it — on an unrelated class, or nested inside another method — satisfied "this namespace defines delete". The runtime is stricter (a plugin class on the service's MRO), so the two could disagree in the ok direction. compat now looks in the class that declares the namespace. Same question on both sides, which is what pick_snapshot_service's docstring has been claiming all along. - apply.sh's compat preflight — the guard that refuses to patch a middleware whose assumptions no longer hold, on every boot, on a live NAS — had no test at all. It could be turned into a no-op eight different ways with the suite still green. The SHIPPED heredoc is now extracted and driven directly against fake verdicts. Also pinned: the Tap/Tap2 prefix collisions (a sweep that treats "Tap2/data@snap" as part of Tap's tree DESTROYS another pool's snapshot), and the GC's in_use wiring. 355 tests. Verified on TrueNAS 26.0.0-BETA.1: 292-dataset backup, 0 orphans, 0 leaked mounts, byte-identical restore of a 4-deep child dataset.	2026-07-14 01:34:13 +00:00
flan	ce6998a935	fix: second audit — the delete check did nothing on a real box, and five guards were untestable ... The most important finding is that the FIRST audit's fix was wrong. _can_delete() asked `callable(getattr(service, "delete"))`. But CRUDService defines `delete` on the BASE class and dispatches to self.do_delete at call time, so a bound `delete` exists on every CRUDService subclass whether or not it still implements one. The check was therefore answering "is this a CRUDService?" — precisely the weaker "is the namespace registered?" question its own docstring said must never be asked. It would still have picked a gutted pool.snapshot and failed every delete. It now walks the MRO and ignores middlewared.service.* plumbing, so only a PLUGIN class defining delete/do_delete counts. The test double was equally wrong: it modelled a gutted service as object(), a shape middlewared cannot produce, so the test passed against a fake it could never have caught in the field. It is now CRUDService-shaped. Also: - The recursive delete's fast path returned [] without confirming anything was destroyed. A delete that returns cleanly is not proof — iX has already gutted pool.snapshot.do_update on master into a no-op that returns None. cleanup_task read "no survivors" as a clean sweep, dropped the sidecar (the only record), and would have orphaned ~250 snapshots per run, silently. It confirms against ZFS now, and the by-name sweep trusts ZFS rather than the API's return value. - When ZFS cannot be read, the sweep no longer claims success. The two mistakes are not symmetric: a false survivor self-heals (sidecar kept, next run reclaims, record clears), a lost record does not. - _write_sidecar swallowed OSError. The sidecar is the only record the snapshots exist; failing to write it must never be invisible. - stage_nested now refuses UP FRONT when middleware has no usable snapshot delete, rather than discovering it after restic has already run. Tests. The autouse fixture added in the last commit did not work: `runner=_run`, `mounts_file="/proc/self/mounts"` and `sleep=time.sleep` are frozen into __defaults__ at def time, so monkeypatching the module attribute never reached them. 19 tests were still reading the real mount table — one matching name from running a real `umount` on the NAS — and the retry loop really slept. All three are late-bound now; the suite reads nothing outside tmp_path and runs in 1.1s. Every mutation the audit reported as SURVIVING now fails the suite: the naive delete check, the unconfirmed fast path, the malformed-row guard, a disconnected GC, eager service resolution, compat's method check, compat's unknown handling, a single-quoted filtered query in apply.sh, and the get-service assumption. Also: fingerprint() folded `unknown` problems into a broken module, so one transient 429 rewrote the bug report and the next clean run rewrote it back. Problems are state-tagged; only definite breakage is digested. Verified on TrueNAS 26.0.0-BETA.1: zvol-orphan case 0 orphans, 292-dataset backup 0 orphans / 0 leaked mounts / 0 stale sidecars, byte-identical restore of a 4-deep child dataset.	2026-07-14 00:10:10 +00:00
flan	413cd60ed4	fix: own the snapshot sweep unconditionally; align the runtime and the manifest ... Four audits of the TrueNAS 26 branch. The findings, in severity order. 1. TrueNAS 26 orphaned a snapshot on every run, with no backstop. Stock decides `recursive` by its own rule, and on 26 that rule is no longer ours. <= 25.10 its create_snapshot called get_dataset_recursive() — the same function this module vendors — so "stock went recursive" and "we have something to stage" were the same question. 26 uses filesystem.statfs: recursive = (path == the dataset's mountpoint). A dataset whose only descendants are ZVOLs or legacy/none-mountpoint datasets now gets a RECURSIVE snapshot while the patch sees nothing to stage. The patch then handed the snapshot back to stock, which destroys the parent only. No staging tree meant no sidecar, and the GC only ever ran from stage_nested — so nothing on the box would ever have found the children. Reproduced on the VM: one orphan per zvol, every run, forever, backup green. Ownership of the sweep is no longer conditional on staging (own_snapshot()). 2. The runtime resolved a NAMESPACE; compat.py verified a METHOD. get_service() only proves a namespace is registered. compat checks the namespace AND that it defines delete/do_delete. So if iX guts the method but keeps the service — which they have already done to pool.snapshot.do_update on master — compat falls through to zfs.snapshot and reports the box healthy, while the runtime picks pool.snapshot and fails every delete. Both sides now ask "can this namespace delete?", and a test binds the two lists together. 3. query_filesystems() silently dropped malformed rows — the one remaining silent-omission path, and a direct contradiction of the cardinal rule. It raises now. A missing `zfs` binary raised FileNotFoundError rather than ZfsError; also fixed. 4. The retry loop discarded the delete error and reported every survivor as "(still busy?)" — naming the one cause that is benign and hiding the ones that are permanent. It keeps and reports the real error. Also: the staging-failure handler could lose the original exception if its own sweep raised; get_service is now a checked assumption; normalise_dataset and two dead MiddlewareCall properties removed; stale comments corrected. Tests: five of them were shelling out to the REAL pool (`zfs list -r Tap`, 2148 snapshots) and passed here only because this box has no zfs binary — they would have gone red on the NAS, which is the one machine the release process requires them green on. An autouse fixture now makes that impossible. Mutation-tested: reverting any of the five fixes above now fails the suite; before, all 293 passed. Verified on TrueNAS 26.0.0-BETA.1 (zvol leak reproduced, then closed; 292-dataset backup, 0 orphans, byte-identical restore of a 4-deep hidden dataset) and on 25.10.4 (pool.snapshot.delete honours recursive=True).	2026-07-13 23:30:48 +00:00
flan	605231b39f	feat: TrueNAS 26 support; enumerate datasets and snapshots from ZFS, not middleware ... TrueNAS 26 deletes plugins/zfs_/ outright, taking the private zfs.dataset.query, zfs.snapshot.query and zfs.snapshot.delete with it. All three were on the nested module's critical path, so nested snapshots were BROKEN on 26. Snapshot deletion now resolves its namespace at runtime: pool.snapshot on 25.10 and 26, zfs.snapshot on 24.10 and 25.04. No single namespace spans every supported release. tools/compat.py checks the same list the runtime uses, so what CI verifies and what runs cannot drift apart. Enumeration does NOT move to pool.dataset.query / pool.snapshot.query, and that is the point of this commit. Those methods exist, are documented, and are covered by iX's deprecation policy — and they are not like-for-like replacements. They apply a visibility policy that hides ix-apps/*, .system/* and .ix-virt/*: 84 of 270 datasets on a real pool, including live application data. Staging from that view omits them silently, and plan_staging never sees them, so they do not even reach the skipped list. The snapshot query hides the same datasets' snapshots, so the sweep orphans one per hidden dataset on every run. So: read the truth from ZFS, make changes through middleware. zfs list cannot be filtered by policy and behaves identically on every release. A failing zfs list raises rather than returning an empty list — "no datasets" and "the command broke" must never look the same. No shipped release is affected: v0.6.1 and earlier use the private zfs.dataset.query, which returns all 270 datasets. The bug existed only in this port. Verified on a real TrueNAS 26.0.0-BETA.1 install: 274-snapshot recursive backup of a 292-dataset pool, zero orphaned snapshots, zero leaked mounts, and a byte-identical restore of a four-level-deep child dataset that pool.dataset.query hides.	2026-07-13 22:43:20 +00:00
flan	82084b6806	The bug-report bot was spamming; make it say something only when there is something to say ... It commented on every run that found a break. In one day it left ELEVEN identical 3,000-character comments on the same issue. That is not a warning system, it is a mute button with extra steps -- and the next real finding would have been scrolled past, which defeats the entire reason for building it. Now: the issue BODY is the current truth, edited in place. COMMENTS are a changelog of changes. A fingerprint of the findings (broken ref/module/problem triples only) is embedded in the body; a run whose findings match it says nothing at all. It closes the issue when everything is fixed. The fingerprint deliberately ignores anything that moves on its own -- healthy rows, the hardware-verified column, TrueNAS point releases -- so TS-25.10.4 becoming TS-25.10.5 is not news and does not wake anybody up. Also: - The two near-identical per-forge shell steps are gone, replaced by one tested implementation (tools/compat_publish.py). Two copies of 'find the issue, decide whether to comment' is two chances to drift, and the Gitea one duplicated an issue for real. - The README matrix refresh now opens a PULL REQUEST instead of pushing straight to main from CI. An unattended push to main is exactly what the release barrier exists to prevent: a bot that can move main can move it somewhere nobody looked.	2026-07-13 20:20:35 +00:00
flan	8c1b4c45f5	release: rc notes resolve to the base version; publish without jq ... release_notes.py 'notes v0.6.0-rc1' looked for a CHANGELOG section literally named v0.6.0-rc1. check() already used base_version(); extract_notes() did not. So the release workflow cut the tag, passed every gate, and then died extracting the body -- the candidate existed but was never published. Caught in an rc, which is the entire point of having them. Also: the Gitea publish and issue steps used jq, which is not guaranteed on a self-hosted runner. A publish step that dies on a missing tool leaves a tag with no release behind it, and a bug report that dies on one is a warning system that does not warn. Both now use python3, which setup-python guarantees.	2026-07-13 18:38:02 +00:00
flan	ecb64878ff	compat: check the middlewared METHODS we call, not just the symbols we wrap ... This gap was hiding a catastrophe. TrueNAS 26 deletes plugins/zfs_/dataset.py and plugins/zfs_/snapshot.py outright, taking zfs.dataset.query, zfs.snapshot.query and zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*). Nothing about the five cloud_backup files reveals that, so every other check went green -- including the one I had just added. The patch would have applied cleanly and then failed on the first backup, or worse: snapshotted fine and failed to DELETE, orphaning one snapshot per descendant dataset (250 on a real pool) on every run, forever. So 26 is BROKEN and the nested module will not apply there. The async/sync wrapper work and the vendored get_dataset_recursive stay -- they are correct and necessary -- but 26 is not supported until the ZFS calls are ported, and that needs a real 26 box to verify. Shipping a port nobody has run is the failure this project exists to avoid. Also: do_delete is recognised as delete (24.10/25.04 use the CRUDService convention), which was reporting both as BROKEN -- a false verdict that would have disabled nested snapshots on boxes where they work.	2026-07-13 18:25:14 +00:00
flan	498b2690e1	TrueNAS 26 support: one sync implementation, two wrappers ... 26 rewrites cloud_backup from async to synchronous AND deletes get_dataset_recursive(), which SNAPSHOT_BLOCK called out of the host module's namespace. Either is a broken backup found at restore time. The nested module is now one synchronous implementation talking to middlewared via call_sync, behind two thin wrappers. apply.sh reads which flavour the installed middleware declares and injects the matching one: <= 25.10 reaches it through 'await middleware.run_in_thread(...)', 26 is already in a worker thread and calls it directly. The snapshot/bind-mount/failure logic exists once -- an async twin would mean every future fix had to land twice. A middleware whose three wrapped functions disagree about asyncness is refused, not guessed at. get_dataset_recursive is vendored, removing the dependency on both versions rather than asserting it. master stays BROKEN on purpose: iX are still renaming middleware->context, cloud_backup->entry and adding a required credentials param there. Chasing a branch that moves daily is how you ship a patch nobody tested.	2026-07-13 18:18:28 +00:00
flan	ea090c7f72	Audit fixes: a compat verdict must never be able to brick a working box ... The audit found the new machinery could do more harm than the bugs it prevents. - apply.sh reused the 'nothing left to do' exit -- which touches the PERMANENT kill switch, cleared only by install.sh, never by update.sh -- for the incompatible case. On TrueNAS 26 (providers ok, nested opt-out) both modules go quiet, so the switch would fire and the release that fixed 26 could never re-enable itself. Retirement and incompatibility now take different exits. - A network blip, a re-export, or a conditional def all read as BROKEN. Each is now 'unknown', which changes nothing, rather than evidence strong enough to disable a module. - 'native' outranked BROKEN everywhere but apply.sh, so a TrueNAS that reworded the guard AND reshaped the functions rendered as good news. - compat.py --tree read B2_BLOCK's own 'restic = True' as native support, so the documented way to check a live box lied on every patched machine. - The signature check was a name-subset test. It passed reorders, kw-only conversions, and added required params -- and it had already passed a real bug: restic_backup takes 4 args on 24.10/25.04, and the wrapper forwarded 5. Nested backups have been raising TypeError on those releases the whole time. The wrapper now forwards *args/**kwargs. - release.sh --promote was unreachable: it died if the tag existed, the gate died if it did not. The tests hid it by always tagging first.	2026-07-13 17:58:15 +00:00
flan	f927773f81	docs: TrueNAS compatibility matrix and the two-stage release process ... The matrix is regenerated daily by CI rather than typed once and forgotten — a support table that quietly goes stale is a false promise to someone deciding whether to trust this with their backups.	2026-07-13 17:35:03 +00:00
flan	4ced730d65	Make Gitea canonical; derive the changelog URL from the remote instead of hard-coding GitHub	2026-07-13 17:21:30 +00:00
flan	4e0814028c	v0.5.0: TrueNAS alert when an update is available ... Raises a real alert in the TrueNAS UI bell -- not a log line nobody reads. On by default, checked once a day. install.sh --no-update-alerts turns it off. It does not nag --------------- A release whose CHANGELOG contains only a "### Docs" section changed no code and raises nothing. Anything else raises INFO; a "### Security" section raises WARNING. The CHANGELOG's own section headings are the signal, and a security fix anywhere in the range escalates the whole span -- a docs-only release sitting on top of a security fix still reports as security rather than hiding it. Why an AlertSource and not midclt ---------------------------------- TrueNAS cannot raise an alert from the CLI. midclt exposes only alert.dismiss, alert.list, alert.list_categories, alert.list_policies and alert.restore -- alert CREATION is internal to middlewared, and none of its ~60 one-shot classes is generic enough to reuse. Registering an AlertSource is the only way. It is also the least invasive thing this patch does. The providers and nested modules both APPEND CODE TO STOCK middleware files; the alert source ADDS ONE FILE and modifies none. It is the native mechanism -- the same one every built-in TrueNAS alert uses -- and TrueNAS polls it itself, so there is no cron job and no systemd timer. - Fail-safe: every error path returns None; it cannot take middlewared down. - Read-only: `git ls-remote` plus an HTTPS fetch of the CHANGELOG. It never writes to .git, so it cannot leave root-owned objects behind the way a `git fetch` from middlewared (running as root) would. - Removed by uninstall.sh (mw_patch.revert_all). - It only tells you; it never updates anything. Verified against the real repo and remote, with middlewared stubbed: on v0.4.1, only a README-only v0.4.2 available -> NO ALERT on v0.4.0, v0.4.1 fixed real bugs -> INFO on v0.3.2, v0.3.3 was the password fix -> SECURITY / WARNING 139 tests, ruff and shellcheck -S style clean.	2026-07-13 16:45:26 +00:00
flan	347c415aa7	v0.4.0: add update.sh ... Fetch a newer release and apply it, preserving the nested-snapshot opt-in setting. bash update.sh # to the newest release, with a confirmation bash update.sh --check # show what would happen; change nothing bash update.sh --rollback # undo the last update Deliberately NOT automated. This patch injects Python into middlewared and re-applies itself at every boot, so an unattended pull would let any bad upstream commit reach a box with no human in the loop and take effect on the next reboot. v0.0.4 shipped exactly such a bug and took every app on the box down. The manual step is the safety gate. Design decisions worth keeping: - Defaults to the newest RELEASE TAG, not main. main can be mid-refactor; a tag is the tested artifact. --main exists but says so loudly. - Tags ordered by version, not date. Date order silently downgrades the box the first time a hotfix is tagged out of band: a v0.3.6 cut after v0.4.0 would sort as "newest". - Refuses to run over a dirty working tree rather than merging across hand-edited or scp'd files. (Verified: the guard fires.) - Shows the commits and release notes you do not have, read from the TARGET's CHANGELOG via tools/release_notes.py -- not a second copy of the extractor. - Records the previous revision BEFORE moving, so --rollback works even if install.sh dies halfway. - Repairs .git ownership, which past `sudo git pull`s leave root-owned and which then breaks every later non-root git command. update.sh is covered by the version-drift check, so it cannot go stale the way create_task.py's __version__ did. Tested end to end in a throwaway clone: detects v0.3.2 -> v0.3.5, lists missing commits, handles already-up-to-date, and the dirty-tree guard fires. 132 tests, ruff and shellcheck clean.	2026-07-13 16:20:37 +00:00
flan	60b3ac4557	v0.3.3: keep the restic repo password out of argv and shell history ... Security -------- create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that JSON carries the restic repository password -- so it sat in the subprocess's argv, which is world-readable via ps, for the duration of the call. That password is the encryption key for the entire cloud backup repository. It now talks to the middleware through truenas_api_client, the library that backs midclt itself, so the password never leaves this process's memory. Verified on a live box: list-tasks and list-credentials work through the new transport. --password is also no longer required, because passing a secret as a CLI argument writes it to shell history permanently. --password-stdin reads it from stdin, and with neither flag the tool prompts via getpass. --password still works but warns. Fixed ----- uninstall.sh could leave every patch installed. It reverted by unmounting the overlay -- but apply.sh only mounts one when the target directory is read-only. On a writable /usr it patches the real files in place, and uninstall would remove the boot hook, report success, and leave the patch applied. It now strips the appended blocks from the middleware files explicitly. This also covers the case where the overlay unmount fails. create_task.py's __version__ had been stuck at 0.2.0 through three releases. The version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it missed the one file that actually shows a version to users (--version). The check now covers __version__ too -- and caught this immediately. 118 tests, ruff and shellcheck clean.	2026-07-13 15:43:24 +00:00
flan	eb91a337cd	Automate releases from tags ... Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the incident that took every app down. Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is the matching CHANGELOG.md section -- one source of truth for release notes, so there is no second place for them to be wrong. The workflow refuses to publish when: - the tests, ruff, or bash -n fail (a tagged commit is what people install; it must be at least as good as main) - the tag does not match the VERSION= declared by every script - CHANGELOG.md has no section for the tag, or the section is empty That version check is not theoretical: VERSION= had drifted to three different values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing noticed until this release. tests/test_release_notes.py now asserts the scripts agree with each other and with the newest CHANGELOG entry, so the drift cannot come back. workflow_dispatch takes an existing tag, so releases can be backfilled for tags that were pushed before this existed. 106 tests, ruff and shellcheck clean.	2026-07-13 15:05:46 +00:00