Compare commits
20
Commits
v0.7.0
...
74cf311785
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
74cf311785 | ||
|
|
520b2d3735 | ||
|
|
4371797f94 | ||
|
|
45c89fd001 | ||
|
|
670dbd25f6 | ||
|
|
827c4358fd | ||
|
|
ea00bd0685 | ||
|
|
5a3d4288a2 | ||
|
|
b364a17735 | ||
|
|
3e1de8ffd1 | ||
|
|
a7cbb3a994 | ||
|
|
d1216eeb0f | ||
|
|
6b7034a8cd | ||
|
|
6ce1206f01 | ||
|
|
e426045255 | ||
|
|
89eb3a16be | ||
|
|
862bdd3399 | ||
|
|
753c3f8cad | ||
|
|
52b11eada2 | ||
|
|
ca906f5ee4 |
@@ -0,0 +1,2 @@
|
|||||||
|
github: sudolulo
|
||||||
|
ko_fi: sudolulo
|
||||||
@@ -43,20 +43,21 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
- uses: actions/setup-python@v5
|
# uv-managed interpreters instead of actions/setup-python: the prebuilt-CPython
|
||||||
with:
|
# download path setup-python relies on does not work on the self-hosted Gitea
|
||||||
python-version: ${{ matrix.python }}
|
# runner (all three matrix jobs failed at setup there while passing on GitHub);
|
||||||
|
# uv works identically on both.
|
||||||
- name: install dev deps
|
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
|
||||||
run: python -m pip install --upgrade pip pytest ruff
|
|
||||||
|
|
||||||
- name: ruff
|
- name: ruff
|
||||||
run: ruff check patch tests tools
|
# Pinned: an unpinned ruff means any upstream release can turn main red
|
||||||
|
# with no code change.
|
||||||
|
run: uvx ruff@0.16.1 check patch tests tools
|
||||||
|
|
||||||
- name: pytest
|
- name: pytest
|
||||||
run: pytest tests -v
|
run: uvx --python ${{ matrix.python }} pytest tests -v
|
||||||
|
|
||||||
- name: verify injected middleware blocks compile
|
- name: verify injected middleware blocks compile
|
||||||
# Belt-and-braces: the *_BLOCK strings are appended into live middlewared
|
# Belt-and-braces: the *_BLOCK strings are appended into live middlewared
|
||||||
# modules. A syntax error there would break the box at boot.
|
# modules. A syntax error there would break the box at boot.
|
||||||
run: pytest tests/test_apply_blocks.py -v
|
run: uvx --python ${{ matrix.python }} pytest tests/test_apply_blocks.py -v
|
||||||
|
|||||||
@@ -24,6 +24,11 @@ on:
|
|||||||
paths:
|
paths:
|
||||||
# The manifest itself changed -- re-check immediately rather than waiting a day.
|
# The manifest itself changed -- re-check immediately rather than waiting a day.
|
||||||
- "tools/compat.py"
|
- "tools/compat.py"
|
||||||
|
# ...and so did the thing that PUBLISHES the finding. This was missing, and it
|
||||||
|
# showed: the commit teaching the bot to refresh a stale report body touched only
|
||||||
|
# compat_publish.py, so no run fired, and the report stayed stale until the next
|
||||||
|
# scheduled one. A fix nobody runs is a fix nobody has.
|
||||||
|
- "tools/compat_publish.py"
|
||||||
- ".github/workflows/compat.yml"
|
- ".github/workflows/compat.yml"
|
||||||
|
|
||||||
permissions:
|
permissions:
|
||||||
|
|||||||
@@ -124,7 +124,7 @@ jobs:
|
|||||||
echo "--- release body ---"
|
echo "--- release body ---"
|
||||||
cat /tmp/notes.md
|
cat /tmp/notes.md
|
||||||
|
|
||||||
# This repo is canonically hosted on Gitea (git.onetick.ninja) and mirrored to
|
# This repo is canonically hosted on Gitea (git.arch.fyi) and mirrored to
|
||||||
# GitHub, and BOTH run this workflow -- Gitea reads .github/workflows too. So
|
# GitHub, and BOTH run this workflow -- Gitea reads .github/workflows too. So
|
||||||
# the publish step has to work on whichever forge it lands on. Everything
|
# the publish step has to work on whichever forge it lands on. Everything
|
||||||
# above is forge-agnostic; only the "create a release" API differs.
|
# above is forge-agnostic; only the "create a release" API differs.
|
||||||
|
|||||||
@@ -6,6 +6,97 @@ is deliberate: see [Releasing](docs/releasing.md). Twelve releases were cut on
|
|||||||
live, every one of those interrupts every user. An alert people learn to ignore is
|
live, every one of those interrupts every user. An alert people learn to ignore is
|
||||||
worse than no alert, because one day it carries a security fix.
|
worse than no alert, because one day it carries a security fix.
|
||||||
|
|
||||||
|
## Unreleased
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **CI's python matrix is green on the self-hosted Gitea runner again.** The real
|
||||||
|
failure was that the Gitea runner image executes jobs as root, and the two
|
||||||
|
unreadable-sidecar tests build their scenario with `chmod(0)` — which cannot make
|
||||||
|
a file unreadable for root (`CAP_DAC_OVERRIDE`). Those two tests now skip as root
|
||||||
|
with that reason; GitHub's non-root runner still exercises them. The matrix also
|
||||||
|
moved to uv-managed interpreters (one toolchain across both runners) and ruff is
|
||||||
|
pinned to 0.16.1 so an upstream ruff release can't turn `main` red without a code
|
||||||
|
change.
|
||||||
|
- **README badges point at the public GitHub mirror** (workflow status and
|
||||||
|
releases) instead of the private forge. The release badge had also been reading
|
||||||
|
the stale Gitea v0.6.1 release instead of the current v0.7.0 on GitHub.
|
||||||
|
|
||||||
|
- **`master` is now labelled `27-dev`, because it is not the next release.** iX
|
||||||
|
branches each major onto its own `release/` line and master rolls straight on to the
|
||||||
|
one after — on 2026-07-14 every recent commit on master targeted `27.0.0-BETA.1`
|
||||||
|
while 26 was still in beta. So a **BROKEN** master row, rendered as
|
||||||
|
"master _(unreleased)_", read as *"the version you are about to install is broken"*
|
||||||
|
when the breakage was a major release away on a line nobody can download. In a table
|
||||||
|
whose entire job is helping somebody decide whether to trust this with their backups,
|
||||||
|
that is a false alarm in the worst possible place. The label is derived from the
|
||||||
|
newest major in the matrix plus one, so it rolls over to `28-dev` by itself once 27
|
||||||
|
branches.
|
||||||
|
|
||||||
|
For the record, the breakage is `NAS-141498` (2026-06-24), "Convert cloud_backup
|
||||||
|
plugin to the typesafe pattern": it re-signatures `restic_backup` and
|
||||||
|
`get_restic_config`, splitting `entry`/`credentials` out of the `cloud_backup` dict.
|
||||||
|
It is deliberately not being chased while the 27 line is still churning.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **The next maintenance release was never checked, and it is the one that reaches
|
||||||
|
users.** Shipped versions were discovered from `TS-*` tags and unreleased ones from
|
||||||
|
`release/*` branches carrying `-BETA`/`-RC`. A branched-but-untagged *maintenance*
|
||||||
|
release is neither: `release/25.10.5` has no tag, and its line has already shipped,
|
||||||
|
so the "a prerelease of a shipped line is history" filter discarded it. It was
|
||||||
|
invisible — and it is precisely what a 25.10.4 box gets on its next update. A break
|
||||||
|
there would have reached real users before the daily check ever looked at it, on the
|
||||||
|
only line anybody is actually running.
|
||||||
|
|
||||||
|
A plain `release/X.Y.Z` branch is now checked when its line **has** shipped and it
|
||||||
|
sorts **newer** than that line's newest tag. Both things that must stay out fall out
|
||||||
|
of the same rule: `release/24.10-RC.2` sorts older than `TS-24.10.2.4` (history, not
|
||||||
|
a warning), and iX's typo branch `release/25.20.2.2` is on a line that has no tag at
|
||||||
|
all, so it is not a release line. This immediately surfaced two refs that had never
|
||||||
|
been checked — `release/25.10.5` and `release/24.10.2.5` — both of which pass.
|
||||||
|
|
||||||
|
`is_unreleased()` now keys off where a ref came from (branch = not yet shipped)
|
||||||
|
rather than looking for `-BETA`/`-RC` in its name. Otherwise `release/25.10.5` would
|
||||||
|
count as shipped and a break in it would fail the build as a live outage — on a
|
||||||
|
version nobody is running yet.
|
||||||
|
|
||||||
|
- **An unchanged fingerprint froze the bug report's body, not just its comments.** Two
|
||||||
|
questions were sharing one answer. *Have the findings changed?* gates **comments** —
|
||||||
|
they notify, and a daily "still broken, same as yesterday" is what teaches everyone
|
||||||
|
to ignore the one that finally matters. *Is the body still true?* gates the **body** —
|
||||||
|
and editing an issue body notifies nobody on either forge, so keeping it honest is
|
||||||
|
free. Conflated, the report could never be corrected while the findings held steady,
|
||||||
|
and the fingerprint deliberately ignores everything that moves on its own — healthy
|
||||||
|
rows, the hardware-verified column, point releases, and how a row is labelled. The
|
||||||
|
`master` → `27-dev` relabel above would have reached the README and never the issue
|
||||||
|
anybody actually opens. The body is now rewritten whenever it is out of date (after
|
||||||
|
normalising line endings, so a forge round-tripping `\r\n` does not cause a rewrite
|
||||||
|
every run) and comments remain strictly a changelog of real changes.
|
||||||
|
|
||||||
|
- **A change to the publisher did not re-run the check.** `compat.yml`'s `push:` paths
|
||||||
|
listed `tools/compat.py` but not `tools/compat_publish.py` — so the very commit that
|
||||||
|
taught the bot to refresh a stale report body triggered no run, and the report stayed
|
||||||
|
stale until the next scheduled one. A fix nobody runs is a fix nobody has.
|
||||||
|
|
||||||
|
- **The compatibility bot filed a new duplicate bug report on every Gitea run.**
|
||||||
|
`find_issue()` skipped pull requests by testing for the *presence* of the
|
||||||
|
`pull_request` key. GitHub omits that key on a plain issue; Gitea sends it as
|
||||||
|
`null`. So on Gitea every issue was discarded as a PR, the lookup always came back
|
||||||
|
empty, and the bot took the "nothing filed yet" branch and opened a fresh report
|
||||||
|
each run — **nine copies on the canonical forge**, four of them filed *after* the
|
||||||
|
commit that was meant to stop precisely this. The mirror was fine, which is why it
|
||||||
|
went unnoticed: GitHub's payload shape is the one the filter was written against.
|
||||||
|
|
||||||
|
It is the same failure the anti-spam fix was written to prevent, moved from
|
||||||
|
comments to issues, and it survived because `find_issue` was the only function in
|
||||||
|
`compat_publish.py` with no test. It now has one, per forge, and the daily cron —
|
||||||
|
which had not yet run once — no longer accumulates a report a day.
|
||||||
|
|
||||||
|
The issue list is also requested with **both** paging parameters (`per_page` for
|
||||||
|
GitHub, `limit` for Gitea). Each forge ignores the other's, and Gitea's default page
|
||||||
|
is 30, so the lookup would have started missing the report again once the pile it
|
||||||
|
was creating grew past one page.
|
||||||
|
|
||||||
## v0.7.0 — 2026-07-14
|
## v0.7.0 — 2026-07-14
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,4 @@
|
|||||||
|
truenas-truecloud-patch
|
||||||
|
|
||||||
|
Parts of this project were written with AI assistance (Claude); all of it is
|
||||||
|
reviewed and tested before release.
|
||||||
@@ -1,5 +1,7 @@
|
|||||||
# truenas-truecloud-patch
|
# truenas-truecloud-patch
|
||||||
|
|
||||||
|
[](https://git.arch.fyi/flan/truenas-truecloud-patch/actions)
|
||||||
|
|
||||||
Extends TrueNAS SCALE's **TrueCloud Backup** to:
|
Extends TrueNAS SCALE's **TrueCloud Backup** to:
|
||||||
|
|
||||||
- back up to **Backblaze B2 and any S3-compatible provider**, not just Storj;
|
- back up to **Backblaze B2 and any S3-compatible provider**, not just Storj;
|
||||||
@@ -59,9 +61,11 @@ If something is wrong, the reason is in `apply.log` — start at
|
|||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| 24.10.2.4 | ok | ok | — |
|
| 24.10.2.4 | ok | ok | — |
|
||||||
| 25.04.2.6 | ok | ok | — |
|
| 25.04.2.6 | ok | ok | — |
|
||||||
| 25.10.4 | ok | ok | nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m |
|
| 25.10.6 | ok | ok | — |
|
||||||
|
| 24.10.2.5 _(unreleased)_ | ok | ok | — |
|
||||||
|
| 25.10.7 _(unreleased)_ | ok | ok | — |
|
||||||
| 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — |
|
| 26.0.0-BETA.3 _(unreleased)_ | ok | ok | — |
|
||||||
| master _(unreleased)_ | **BROKEN** | **BROKEN** | — |
|
| master _(27-dev)_ | **BROKEN** | **BROKEN** | — |
|
||||||
|
|
||||||
| verdict | meaning |
|
| verdict | meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -72,23 +76,44 @@ If something is wrong, the reason is in `apply.log` — start at
|
|||||||
"ok" means *the patch's assumptions hold*, checked automatically against iX's
|
"ok" means *the patch's assumptions hold*, checked automatically against iX's
|
||||||
source. It does not mean a human ran a backup on it — that is the
|
source. It does not mean a human ran a backup on it — that is the
|
||||||
**Hardware-verified** column, which is filled in by hand and only by doing it.
|
**Hardware-verified** column, which is filled in by hand and only by doing it.
|
||||||
|
|
||||||
|
**`master` is not the next release.** iX branches each major off to its own
|
||||||
|
`release/` line and master rolls straight on to the one after — so master is
|
||||||
|
`27-dev` while 26 is still in beta. A **BROKEN** master means iX has changed
|
||||||
|
something that will reach users *a major release from now*, not in the version you
|
||||||
|
are about to install. Read the numbered rows for that.
|
||||||
|
|
||||||
|
A row like `25.10.5 _(unreleased)_` is the next maintenance release: branched by iX,
|
||||||
|
not tagged yet, and the very next thing a 25.10.4 box gets. It is checked precisely
|
||||||
|
because it is the one unshipped ref that reaches real users without warning.
|
||||||
<!-- END COMPAT MATRIX -->
|
<!-- END COMPAT MATRIX -->
|
||||||
|
|
||||||
The table is **regenerated daily by CI** against iXsystems' actual middleware source
|
The table is **regenerated daily by CI** against iXsystems' actual middleware source
|
||||||
— it is not a claim somebody typed once and forgot.
|
— it is not a claim somebody typed once and forgot.
|
||||||
|
|
||||||
**On TrueNAS 26:** the patch was run on a real TrueNAS **26.0.0-BETA.1** install — a
|
It is also **static analysis**: it proves the patch's assumptions still hold, which is
|
||||||
274-snapshot recursive backup of a 292-dataset pool, followed by a byte-identical
|
a weaker claim than "a backup ran and a restore came back". For what has actually been
|
||||||
restore of a four-level-deep child dataset. The *Hardware-verified* column tracks the
|
run — which tasks, on which hardware, and the md5 of the file that came back — see
|
||||||
newest beta iX has tagged (currently BETA.3), so it does not carry that mark: a build
|
[docs/verification.md](docs/verification.md).
|
||||||
nobody has actually run a backup on does not get credit for one.
|
|
||||||
|
|
||||||
**TrueNAS 26: nested snapshots are not supported yet, and upgrading will not break
|
**TrueNAS 26 is supported** as of v0.7.0, and was verified on a real
|
||||||
you.** 26 rewrites `cloud_backup` and deletes the ZFS methods this module calls. On
|
**26.0.0-BETA.1** install: a 274-snapshot recursive backup of a 292-dataset pool, and a
|
||||||
26 `apply.sh` finds that the assumptions no longer hold and **does not apply the
|
byte-identical restore of a four-level-deep child dataset. The *Hardware-verified*
|
||||||
module**: TrueNAS is left stock, B2/S3 keeps working, nested datasets are simply not
|
column tracks the newest beta iX has tagged (currently BETA.3), so it does not carry
|
||||||
covered, and the reason is named in `apply.log`. A broken backup is worse than a
|
that mark — a build nobody has actually run a backup on does not get credit for one.
|
||||||
missing feature. Details: [How it works](docs/how-it-works.md#truenas-26).
|
|
||||||
|
26 rewrites `cloud_backup` from async to synchronous and deletes the private ZFS
|
||||||
|
methods this module used to call, so getting there took real work: the patch now
|
||||||
|
injects the wrapper flavour that matches the installed middleware, reads dataset and
|
||||||
|
snapshot lists **from ZFS rather than middleware** (whose queries hide TrueNAS's own
|
||||||
|
datasets — 84 of 270 on a real pool, including live app data), and owns the snapshot
|
||||||
|
sweep even when it stages nothing (26 decides `recursive` by a rule this patch does not
|
||||||
|
share, and would otherwise orphan one snapshot per zvol on every run).
|
||||||
|
|
||||||
|
**And if a future TrueNAS breaks it, you get a missing feature, not a broken backup.**
|
||||||
|
`apply.sh` re-checks the patch's assumptions at every boot and **refuses to apply a
|
||||||
|
module whose assumptions no longer hold** — TrueNAS is left stock, and the reason is
|
||||||
|
named in `apply.log`. Details: [How it works](docs/how-it-works.md#truenas-26).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -213,5 +238,7 @@ and restores the original UI bundle from backup.
|
|||||||
- Filing a TrueNAS bug? **Remove the patch first** and reproduce on a stock system.
|
- Filing a TrueNAS bug? **Remove the patch first** and reproduce on a stock system.
|
||||||
- Provided as-is, no warranty. See LICENSE.
|
- Provided as-is, no warranty. See LICENSE.
|
||||||
|
|
||||||
Parts of this project were written with AI assistance (Claude); all of it is reviewed
|
## Support
|
||||||
and tested before release. Bugs are mine.
|
|
||||||
|
If truenas-truecloud-patch is useful to you, consider supporting development via
|
||||||
|
[GitHub Sponsors](https://github.com/sponsors/sudolulo) or [Ko-fi](https://ko-fi.com/sudolulo).
|
||||||
|
|||||||
+62
-14
@@ -187,23 +187,71 @@ backup-breaking**, and none of them is visible from the `cloud_backup` files:
|
|||||||
| `get_dataset_recursive()` **deleted** from `plugins/cloud/snapshot.py` | `NameError` — the injected block called it out of the host module's namespace |
|
| `get_dataset_recursive()` **deleted** from `plugins/cloud/snapshot.py` | `NameError` — the injected block called it out of the host module's namespace |
|
||||||
| `plugins/zfs_/dataset.py` and `zfs_/snapshot.py` **deleted** | `zfs.dataset.query`, `zfs.snapshot.query` and `zfs.snapshot.delete` all vanish. 26 uses `filesystem.statfs` and `zfs.resource.*` |
|
| `plugins/zfs_/dataset.py` and `zfs_/snapshot.py` **deleted** | `zfs.dataset.query`, `zfs.snapshot.query` and `zfs.snapshot.delete` all vanish. 26 uses `filesystem.statfs` and `zfs.resource.*` |
|
||||||
|
|
||||||
The first two are fixed: the patch reads which flavour of `cloud_backup` your box
|
All three are fixed as of **v0.7.0**, and 26 is supported.
|
||||||
declares and injects the wrapper that matches (one implementation of the real logic,
|
|
||||||
two thin wrappers), and it carries its own copy of the deleted helper.
|
|
||||||
|
|
||||||
The third is **not** fixed, and is why 26 reports BROKEN. Porting it means rewriting
|
The first two were straightforward: the patch reads which flavour of `cloud_backup`
|
||||||
the module's ZFS calls onto 26's new API, and no single API spans 24.10 through 26 —
|
your box declares and injects the wrapper that matches (one implementation of the real
|
||||||
so it needs a real 26 box to verify against, not a plausible-looking diff. Shipping a
|
logic, two thin wrappers), and it carries its own copy of the deleted helper.
|
||||||
port nobody has run is exactly the failure this project exists to avoid.
|
|
||||||
|
|
||||||
It is also the row that would have hurt most. `zfs.snapshot.delete` is what sweeps the
|
The third was not, and it is the one that would have hurt most — `zfs.snapshot.delete`
|
||||||
recursive snapshot; without it, **every run would orphan one snapshot per descendant
|
is what sweeps the recursive snapshot, and without it **every run would orphan one
|
||||||
dataset — 250 on a real pool — forever.** The compatibility check caught it only
|
snapshot per descendant dataset (250 on a real pool), forever, while reporting
|
||||||
because it now asserts the middleware *methods the patch calls*, not just the symbols
|
success.**
|
||||||
it wraps.
|
|
||||||
|
|
||||||
`master` (development after 26) reports BROKEN too: iXsystems are still reshaping
|
The obvious port is to the public `pool.dataset.query` / `pool.snapshot.query`. Those
|
||||||
|
methods exist, are documented, and are covered by iX's deprecation policy — and they
|
||||||
|
are **not like-for-like replacements**. They apply a *visibility policy*: they hide the
|
||||||
|
datasets TrueNAS considers its own (`ix-apps/*`, `.system/*`, `.ix-virt/*`). On a real
|
||||||
|
pool that is **84 of 270 datasets, including live application data.** Staging from that
|
||||||
|
view would have omitted every one of them from the backup — and the planner would never
|
||||||
|
have seen them, so they would not have appeared in its "skipped" list either. A green
|
||||||
|
backup, quietly missing data. The snapshot query lies the same way, so the sweep would
|
||||||
|
have orphaned one snapshot per hidden dataset.
|
||||||
|
|
||||||
|
No source analysis could have caught that. The methods are all present and correctly
|
||||||
|
shaped. Only running it could, which is why it took a real 26 box.
|
||||||
|
|
||||||
|
So the module now follows one rule:
|
||||||
|
|
||||||
|
> **Read the truth from ZFS. Make changes through middleware.**
|
||||||
|
|
||||||
|
Enumeration is `zfs list` — no policy can filter it, and it behaves identically on every
|
||||||
|
release, which also means one code path instead of a version conditional. Mutation stays
|
||||||
|
a middleware call, so TrueNAS's own bookkeeping stays consistent; an exact-name delete
|
||||||
|
works fine even on a dataset the query hides. It is only enumeration that lies.
|
||||||
|
|
||||||
|
The snapshot *delete* still needs a namespace, and no single one spans every release —
|
||||||
|
24.10 and 25.04 have `zfs.snapshot`, 26 has only `pool.snapshot`, 25.10 has both. So it
|
||||||
|
is resolved at runtime, by asking whether the namespace can actually delete. `tools/compat.py`
|
||||||
|
asks the identical question against iX's source, and a test binds the two lists together,
|
||||||
|
so what CI verifies and what runs cannot drift apart.
|
||||||
|
|
||||||
|
### The one 26 changed that nothing warned about
|
||||||
|
|
||||||
|
Stock decides whether to take a **recursive** snapshot by its own rule, and on 26 that
|
||||||
|
rule stopped being ours:
|
||||||
|
|
||||||
|
| | decides `recursive` by |
|
||||||
|
| --- | --- |
|
||||||
|
| stock ≤ 25.10 | `get_dataset_recursive()` — the same function this patch vendors |
|
||||||
|
| **stock 26** | `filesystem.statfs`: `recursive = (path == the dataset's mountpoint)` |
|
||||||
|
| this patch | `get_dataset_recursive()` — is a mounted *filesystem* child under the path? |
|
||||||
|
|
||||||
|
Up to 25.10 those were the *same question*, so a snapshot the patch declined to stage
|
||||||
|
provably had no children and stock's non-recursive delete was correct. On 26 they
|
||||||
|
disagree: a dataset whose only descendants are **zvols** or **legacy-mountpoint**
|
||||||
|
datasets gets a recursive snapshot, while the patch sees nothing to stage. Stock then
|
||||||
|
destroys the parent only — and with no staging tree there was no sidecar, and the
|
||||||
|
garbage collector only ever ran from the staging path. Nothing on the box would ever
|
||||||
|
have found the children.
|
||||||
|
|
||||||
|
It was reproduced on a 26 VM (one orphan per zvol, every run, backup green) and closed:
|
||||||
|
**ownership of the sweep is no longer conditional on staging.**
|
||||||
|
|
||||||
|
`master` (development after 26) **does** report BROKEN: iXsystems are still reshaping
|
||||||
these functions there, renaming `middleware` → `context` and `cloud_backup` → `entry`
|
these functions there, renaming `middleware` → `context` and `cloud_backup` → `entry`
|
||||||
and adding a required `credentials` parameter. That is a moving target and is
|
and adding a required `credentials` parameter. That is a moving target and is
|
||||||
deliberately not chased; the check keeps reporting it until it settles into a beta,
|
deliberately not chased; the check keeps reporting it until it settles into a beta,
|
||||||
which is when it becomes worth fixing.
|
which is when it becomes worth fixing. Until then, a box running master would simply
|
||||||
|
not get the modules — `apply.sh` refuses to apply a module whose assumptions no longer
|
||||||
|
hold, and says why in `apply.log`.
|
||||||
|
|||||||
+1
-1
@@ -25,7 +25,7 @@ are worth calling out, because nothing else would catch what they catch:
|
|||||||
newest CHANGELOG entry. `VERSION=` had silently drifted to three different
|
newest CHANGELOG entry. `VERSION=` had silently drifted to three different
|
||||||
values across the scripts before anything checked.
|
values across the scripts before anything checked.
|
||||||
|
|
||||||
The project is hosted on **Gitea** (`git.onetick.ninja/flan/truenas-truecloud-patch`)
|
The project is hosted on **Gitea** (`git.arch.fyi/flan/truenas-truecloud-patch`)
|
||||||
and mirrored to GitHub. Both run the same workflows — Gitea reads
|
and mirrored to GitHub. Both run the same workflows — Gitea reads
|
||||||
`.github/workflows/` too — so a change is checked twice, on two independent runners.
|
`.github/workflows/` too — so a change is checked twice, on two independent runners.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# What has actually been run
|
||||||
|
|
||||||
|
The support matrix in the README is **static analysis**: it proves the patch's
|
||||||
|
assumptions about middlewared still hold. That is a strictly weaker claim than "a
|
||||||
|
backup ran and a restore came back". This file is the stronger claim, and it is
|
||||||
|
maintained by hand, because the only way to fill it in is to do it.
|
||||||
|
|
||||||
|
If you are deciding whether to trust this with your backups, read this file, not the
|
||||||
|
matrix.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## v0.7.0 — TrueNAS 25.10.4 (production hardware)
|
||||||
|
|
||||||
|
Six live TrueCloud tasks, all `snapshot = true`, backing up to Backblaze B2. Three were
|
||||||
|
exercised end to end, chosen to cover the three shapes the code handles differently:
|
||||||
|
|
||||||
|
| Task | Path | Shape | Result |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 5 | `/mnt/Tap` | 191 nested datasets staged, 282-snapshot recursive tree | SUCCESS |
|
||||||
|
| 7 | `/mnt/Tank/backups` | 215 filesystems **+ 2 zvols** | SUCCESS |
|
||||||
|
| 9 | `/mnt/Tank/flan` | **no** nested filesystem children | SUCCESS |
|
||||||
|
|
||||||
|
After every run: **0 orphaned snapshots, 0 leaked bind mounts, 0 stale sidecars.**
|
||||||
|
|
||||||
|
**The restore.** `apps/vaultwarden/data/config.json` — a file inside a *child* dataset,
|
||||||
|
which is exactly what stock TrueNAS cannot capture — was restored from B2 and compared
|
||||||
|
against the live file:
|
||||||
|
|
||||||
|
live f809df6ba231986b1ba824044228a03a 1808 bytes
|
||||||
|
restored f809df6ba231986b1ba824044228a03a 1808 bytes
|
||||||
|
=> byte-identical
|
||||||
|
|
||||||
|
**The collector earned its keep on real data.** The pool was already carrying an orphan:
|
||||||
|
`Tap/apps/prometheus@cloud_backup-5-20260713202355`, left behind by an earlier run when
|
||||||
|
ZFS's automount held the snapshot busy past all four retries. The first v0.7.0 run found
|
||||||
|
it by name, reclaimed it, and the pool's snapshot count went 2148 → 2147. That is the
|
||||||
|
garbage collector doing the job it was written for, against a leak that was already
|
||||||
|
there and that nothing else would ever have found.
|
||||||
|
|
||||||
|
**Boot path.** `apply.sh` is registered as a PREINIT `initshutdownscript`; it was
|
||||||
|
re-run against the live middleware and left exactly one `TRUECLOUD_PATCH` marker in
|
||||||
|
each patched module (a second copy stacked into a live middlewared module would break
|
||||||
|
the box at boot). It correctly detected the box as **async** (`cloud_backup is async
|
||||||
|
(TrueNAS <= 25.10)`) and injected the matching wrappers.
|
||||||
|
|
||||||
|
**Upgrade path.** `update.sh` was used to move the box from the release candidate to
|
||||||
|
the stable tag, in detached HEAD at `v0.7.0`, which is how a user's box actually
|
||||||
|
upgrades.
|
||||||
|
|
||||||
|
## v0.7.0 — TrueNAS 26.0.0-BETA.1 (VM)
|
||||||
|
|
||||||
|
A throwaway VM whose pool reproduces the production pool's *shape* — 292 datasets, 26
|
||||||
|
`legacy` mountpoints, nesting five deep — because every bug found on the real box came
|
||||||
|
from the shape of the pool, not the bytes in it. MinIO was not used; `rclone serve s3`
|
||||||
|
(already on the box) provided the S3 target, so no real B2 credential ever entered the
|
||||||
|
VM.
|
||||||
|
|
||||||
|
* 274-snapshot recursive backup of the 292-dataset pool. 0 orphans, 0 leaked mounts.
|
||||||
|
* Restored `ix-apps/app_mounts/vaultwarden/pgData` — **four levels deep, and a dataset
|
||||||
|
that middleware's own `pool.dataset.query` hides from itself** — byte-identical.
|
||||||
|
* The zvol-orphan case was **reproduced with the fix disabled** (one orphan per zvol,
|
||||||
|
every run, backup green), then **closed with it enabled**. See the CHANGELOG entry
|
||||||
|
for why TrueNAS 26 decides `recursive` by a different rule than this patch decides
|
||||||
|
`nested`.
|
||||||
|
|
||||||
|
## What is NOT covered
|
||||||
|
|
||||||
|
* **24.10 and 25.04** are `ok` in the matrix — the assumptions hold, checked against
|
||||||
|
iX's source — but nobody has run a backup on them. The matrix says so.
|
||||||
|
* **master** is BROKEN, and correctly reports so: iX renamed the leading parameters of
|
||||||
|
`get_restic_config` and `restic_backup`. It is not a shipped release; the daily
|
||||||
|
compatibility bot files it, and `apply.sh` would refuse to apply the modules on a box
|
||||||
|
running it.
|
||||||
|
* A **reboot** of the production box has not been done on v0.7.0. `apply.sh` was
|
||||||
|
re-executed by hand against the live middleware, which exercises the same code path,
|
||||||
|
but the PREINIT ordering itself has only been proven on earlier versions.
|
||||||
@@ -47,8 +47,8 @@ _VERSION_RE = re.compile(r'^VERSION="([^"]+)"', re.M)
|
|||||||
#: owner/repo out of any of:
|
#: owner/repo out of any of:
|
||||||
#: git@github.com:sudolulo/repo.git
|
#: git@github.com:sudolulo/repo.git
|
||||||
#: https://github.com/sudolulo/repo.git
|
#: https://github.com/sudolulo/repo.git
|
||||||
#: ssh://git@git.onetick.ninja:55214/flan/repo.git
|
#: ssh://git@git.arch.fyi:55214/flan/repo.git
|
||||||
#: https://git.onetick.ninja/flan/repo.git
|
#: https://git.arch.fyi/flan/repo.git
|
||||||
#: The SSH port is deliberately not captured: it is not the web port.
|
#: The SSH port is deliberately not captured: it is not the web port.
|
||||||
_REMOTE_RE = re.compile(
|
_REMOTE_RE = re.compile(
|
||||||
r"^(?:\w+://)?(?:[^@/]+@)?([^:/]+)(?::\d+)?[:/]([^/]+)/([^/]+?)(?:\.git)?/?$"
|
r"^(?:\w+://)?(?:[^@/]+@)?([^:/]+)(?::\d+)?[:/]([^/]+)/([^/]+?)(?:\.git)?/?$"
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ dangerous thing this file can say -- it means "TrueNAS does this now, retire the
|
|||||||
module" -- and it rests on nothing more than a substring match.
|
module" -- and it rests on nothing more than a substring match.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
@@ -23,6 +24,7 @@ import pytest
|
|||||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
|
||||||
|
|
||||||
import compat # noqa: E402
|
import compat # noqa: E402
|
||||||
|
import compat_publish # noqa: E402
|
||||||
from compat import ( # noqa: E402
|
from compat import ( # noqa: E402
|
||||||
NESTED,
|
NESTED,
|
||||||
PROVIDERS,
|
PROVIDERS,
|
||||||
@@ -574,3 +576,233 @@ class TestATransientNetworkBlipDoesNotWakeAnybody:
|
|||||||
"def get_restic_config(entry, credentials):\n pass\n"
|
"def get_restic_config(entry, credentials):\n pass\n"
|
||||||
)
|
)
|
||||||
assert compat.fingerprint(self._rows(worse)) != compat.fingerprint(self._rows(broken))
|
assert compat.fingerprint(self._rows(worse)) != compat.fingerprint(self._rows(broken))
|
||||||
|
|
||||||
|
|
||||||
|
class TestTheBotFindsItsOwnIssueOnBOTHForges:
|
||||||
|
"""`find_issue` decides "have I already filed this?" -- and it ran on two forges.
|
||||||
|
|
||||||
|
It used to skip pull requests with `"pull_request" not in i`. GitHub omits that key
|
||||||
|
on a plain issue; **Gitea sends it as `null`**. So on Gitea every issue looked like
|
||||||
|
a PR, the match list was always empty, and the bot took the "nothing filed yet"
|
||||||
|
branch on EVERY run: nine duplicate copies of the same report on the canonical
|
||||||
|
forge, four of them filed after the commit that was supposed to stop exactly this.
|
||||||
|
|
||||||
|
It is the same failure the spam fix was written to prevent, moved from comments to
|
||||||
|
issues -- and it survived because `find_issue` was the one function here with no
|
||||||
|
test. So the payload shapes are pinned, per forge, by hand.
|
||||||
|
"""
|
||||||
|
|
||||||
|
TITLE = compat_publish.TITLE
|
||||||
|
|
||||||
|
def _find(self, monkeypatch, payload):
|
||||||
|
monkeypatch.setattr(compat_publish, "_call", lambda *a, **k: payload)
|
||||||
|
return compat_publish.find_issue("https://forge/api", "tok", self.TITLE)
|
||||||
|
|
||||||
|
def test_gitea_sends_pull_request_as_null_and_the_issue_is_still_found(self, monkeypatch):
|
||||||
|
found = self._find(monkeypatch, [
|
||||||
|
{"number": 7, "title": self.TITLE, "state": "open", "pull_request": None},
|
||||||
|
{"number": 1, "title": self.TITLE, "state": "open", "pull_request": None},
|
||||||
|
])
|
||||||
|
assert found is not None, (
|
||||||
|
"find_issue missed a Gitea issue, so the bot files a NEW duplicate report "
|
||||||
|
"every run -- which is how nine of them piled up"
|
||||||
|
)
|
||||||
|
assert found["number"] == 1, "lowest-numbered wins"
|
||||||
|
|
||||||
|
def test_github_omits_the_key_entirely_and_the_issue_is_still_found(self, monkeypatch):
|
||||||
|
found = self._find(monkeypatch, [
|
||||||
|
{"number": 2, "title": self.TITLE, "state": "open"},
|
||||||
|
])
|
||||||
|
assert found is not None and found["number"] == 2
|
||||||
|
|
||||||
|
def test_a_real_PR_with_the_same_title_is_still_skipped_on_both(self, monkeypatch):
|
||||||
|
# The reason the filter exists at all: both forges list PRs on /issues, and
|
||||||
|
# commenting on a PR instead of the bug report would be worse than useless.
|
||||||
|
assert self._find(monkeypatch, [
|
||||||
|
{"number": 3, "title": self.TITLE, "state": "open", # Gitea PR
|
||||||
|
"pull_request": {"merged": False}},
|
||||||
|
{"number": 4, "title": self.TITLE, "state": "open", # GitHub PR
|
||||||
|
"pull_request": {"url": "https://api.github.com/..."}},
|
||||||
|
]) is None
|
||||||
|
|
||||||
|
def test_an_unrelated_issue_is_not_mistaken_for_the_report(self, monkeypatch):
|
||||||
|
assert self._find(monkeypatch, [
|
||||||
|
{"number": 1, "title": "TypeError when create B2 backup on Electric Eel",
|
||||||
|
"state": "closed", "pull_request": None},
|
||||||
|
]) is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestTheNextMaintenanceReleaseIsChecked:
|
||||||
|
"""`release/25.10.5` fell through every sieve, and it is the one that reaches users.
|
||||||
|
|
||||||
|
Shipped versions come from `TS-*` TAGS; unreleased ones come from `release/*`
|
||||||
|
BRANCHES that carry `-BETA`/`-RC`. A branched-but-untagged MAINTENANCE release is
|
||||||
|
neither: no tag, and its line (25.10) has already shipped, so the "prereleases of
|
||||||
|
a shipped line are history" filter threw it out. It was invisible.
|
||||||
|
|
||||||
|
That is backwards. `release/24.10-RC.2` is history -- nobody can install it. But
|
||||||
|
`release/25.10.5` is the FUTURE of a shipped line: it is what a 25.10.4 box gets
|
||||||
|
on its next update. A break there ships to real users before the daily check has
|
||||||
|
ever looked at it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
TAGS = ["TS-24.10.2.4", "TS-25.04.2.6", "TS-25.10.4"]
|
||||||
|
HEADS = [
|
||||||
|
"release/25.10.4.1",
|
||||||
|
"release/25.10.5", # branched, untagged -- the next maintenance release
|
||||||
|
"release/24.10-RC.2", # history: its line shipped long ago
|
||||||
|
"release/25.20.2.2", # iX's typo branch: 25.20 is not a TrueNAS version
|
||||||
|
"release/26.0.0-BETA.3",
|
||||||
|
"master",
|
||||||
|
]
|
||||||
|
|
||||||
|
def _refs(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
compat, "_ls_remote",
|
||||||
|
lambda remote, what: self.TAGS if what == "--tags" else self.HEADS)
|
||||||
|
return compat.discover_refs("origin")
|
||||||
|
|
||||||
|
def test_the_next_maintenance_release_is_checked(self, monkeypatch):
|
||||||
|
assert "release/25.10.5" in self._refs(monkeypatch), (
|
||||||
|
"the next thing a 25.10.4 box updates to is not checked, so a break in it "
|
||||||
|
"reaches users before the bot ever sees it"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_a_superseded_maintenance_branch_is_not(self, monkeypatch):
|
||||||
|
# 25.10.4.1 sorts OLDER than the newest tag TS-25.10.4? No -- it is NEWER, and
|
||||||
|
# both are on the 25.10 line, so only the newest branch on the line is taken.
|
||||||
|
refs = self._refs(monkeypatch)
|
||||||
|
assert "release/25.10.4.1" not in refs, "only the newest branch per line"
|
||||||
|
|
||||||
|
def test_the_typo_branch_stays_out(self, monkeypatch):
|
||||||
|
# 25.20 has no TS tag, so it is not a release line at all. A typo branch in the
|
||||||
|
# matrix reads as a real supported release we are silently broken on.
|
||||||
|
assert "release/25.20.2.2" not in self._refs(monkeypatch)
|
||||||
|
|
||||||
|
def test_a_prerelease_of_an_already_shipped_line_stays_out(self, monkeypatch):
|
||||||
|
assert "release/24.10-RC.2" not in self._refs(monkeypatch)
|
||||||
|
|
||||||
|
def test_an_untagged_branch_counts_as_UNRELEASED(self, monkeypatch):
|
||||||
|
# The exit code keys off this. Calling 25.10.5 "shipped" would fail the build
|
||||||
|
# as a live outage on a version nobody is running yet.
|
||||||
|
assert compat.is_unreleased("release/25.10.5")
|
||||||
|
assert compat.is_unreleased("master")
|
||||||
|
assert not compat.is_unreleased("TS-25.10.4")
|
||||||
|
|
||||||
|
|
||||||
|
class TestMasterIsNotTheNextRelease:
|
||||||
|
"""A red `master` row used to read as "the version you are about to install".
|
||||||
|
|
||||||
|
On 2026-07-14 master was 27-dev -- every recent commit targeted 27.0.0-BETA.1 --
|
||||||
|
while 26 was still in beta on its own branches. So `master BROKEN` meant "iX will
|
||||||
|
break us a major release from now", but the matrix said "master _(unreleased)_",
|
||||||
|
which any reader takes as the next thing out the door. For a table whose whole job
|
||||||
|
is helping somebody decide whether to trust this with their backups, that is a
|
||||||
|
false alarm in the worst possible place.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _rows(self, refs):
|
||||||
|
return [{"ref": r, "unreleased": compat.is_unreleased(r), "modules": {}}
|
||||||
|
for r in refs]
|
||||||
|
|
||||||
|
def test_master_is_labelled_with_the_major_AFTER_the_newest_known_one(self):
|
||||||
|
rows = self._rows(["TS-25.10.4", "release/26.0.0-BETA.3", "master"])
|
||||||
|
assert compat.dev_label(rows) == "27-dev"
|
||||||
|
|
||||||
|
def test_it_rolls_over_on_its_own_when_the_next_beta_branches(self):
|
||||||
|
# Derived, not hardcoded: when release/27.0.0-BETA.1 appears, master is 28-dev.
|
||||||
|
rows = self._rows(["TS-26.0.0", "release/27.0.0-BETA.1", "master"])
|
||||||
|
assert compat.dev_label(rows) == "28-dev"
|
||||||
|
|
||||||
|
def test_the_rendered_matrix_says_dev_not_unreleased(self):
|
||||||
|
healthy = check_files(with_())
|
||||||
|
rows = [
|
||||||
|
{"ref": r, "unreleased": compat.is_unreleased(r), "modules": healthy}
|
||||||
|
for r in ("TS-25.10.4", "release/26.0.0-BETA.3", "master")
|
||||||
|
]
|
||||||
|
md = compat.render_markdown(rows)
|
||||||
|
assert "master _(27-dev)_" in md
|
||||||
|
assert "master _(unreleased)_" not in md
|
||||||
|
# ...and the ordinary rows are untouched.
|
||||||
|
assert "| 25.10.4 |" in md
|
||||||
|
assert "| 26.0.0-BETA.3 _(unreleased)_ |" in md
|
||||||
|
|
||||||
|
|
||||||
|
class TestTheBodyIsTruthAndCommentsAreTheChangelog:
|
||||||
|
"""An unchanged fingerprint used to freeze the BODY, not just silence the comments.
|
||||||
|
|
||||||
|
Two different questions were sharing one answer. "Have the findings changed?" gates
|
||||||
|
COMMENTS -- they notify, and a daily "still broken, same as yesterday" is what
|
||||||
|
teaches people to ignore the one that finally matters. But "is the body still
|
||||||
|
true?" gates the BODY, and editing a body notifies nobody, so keeping it honest
|
||||||
|
costs nothing.
|
||||||
|
|
||||||
|
Conflated, an unchanged fingerprint meant the report could never be corrected --
|
||||||
|
and the fingerprint deliberately ignores everything that moves on its own, which
|
||||||
|
includes how a row is LABELLED. Relabelling master `27-dev` would have reached the
|
||||||
|
README and never the issue anybody opens.
|
||||||
|
"""
|
||||||
|
|
||||||
|
ROWS = [{"ref": "master", "unreleased": True,
|
||||||
|
"modules": check_files(with_(**{
|
||||||
|
"plugins/cloud_backup/restic.py":
|
||||||
|
"class ResticConfig:\n cmd: list\n\n"
|
||||||
|
"def get_restic_config(entry, credentials):\n pass\n",
|
||||||
|
}))}]
|
||||||
|
|
||||||
|
def _run(self, monkeypatch, tmp_path, existing_body):
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake(url, token, method="GET", data=None):
|
||||||
|
calls.append((method, url, data))
|
||||||
|
if url.endswith("/issues?state=all&per_page=100&limit=100"):
|
||||||
|
return [{"number": 1, "title": compat_publish.TITLE,
|
||||||
|
"state": "open", "body": existing_body,
|
||||||
|
"pull_request": None}]
|
||||||
|
return {"number": 1}
|
||||||
|
|
||||||
|
monkeypatch.setattr(compat_publish, "_call", fake)
|
||||||
|
matrix = tmp_path / "m.json"
|
||||||
|
matrix.write_text(json.dumps(self.ROWS))
|
||||||
|
compat_publish.main([
|
||||||
|
"prog", "--api", "https://forge/api", "--token", "t",
|
||||||
|
"--matrix", str(matrix)])
|
||||||
|
return calls
|
||||||
|
|
||||||
|
def _writes(self, calls):
|
||||||
|
patched = [c for c in calls if c[0] == "PATCH"]
|
||||||
|
commented = [c for c in calls if c[0] == "POST" and c[1].endswith("/comments")]
|
||||||
|
return patched, commented
|
||||||
|
|
||||||
|
def test_identical_body_and_findings_touches_nothing(self, monkeypatch, tmp_path):
|
||||||
|
body = compat.render_issue(self.ROWS)
|
||||||
|
patched, commented = self._writes(self._run(monkeypatch, tmp_path, body))
|
||||||
|
assert not patched and not commented, "a quiet run must be completely silent"
|
||||||
|
|
||||||
|
def test_a_relabel_refreshes_the_body_but_says_NOTHING(self, monkeypatch, tmp_path):
|
||||||
|
# Same findings (same fingerprint), different rendering -- the exact shape of
|
||||||
|
# the master -> 27-dev relabel.
|
||||||
|
stale = compat.render_issue(self.ROWS).replace("27-dev", "unreleased")
|
||||||
|
assert compat.extract_fingerprint(stale) == compat.fingerprint(self.ROWS)
|
||||||
|
|
||||||
|
patched, commented = self._writes(self._run(monkeypatch, tmp_path, stale))
|
||||||
|
assert patched, "the body was left stale, so the issue keeps telling lies"
|
||||||
|
assert "27-dev" in patched[0][2]["body"]
|
||||||
|
assert not commented, (
|
||||||
|
"a rendering change is not news -- commenting on it is how the bot gets "
|
||||||
|
"muted before the next real finding"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_a_REAL_findings_change_still_comments(self, monkeypatch, tmp_path):
|
||||||
|
# ...and the fix must not have made it mute.
|
||||||
|
stale = compat.render_issue(self.ROWS).replace(
|
||||||
|
compat.fingerprint(self.ROWS), "0" * 16)
|
||||||
|
patched, commented = self._writes(self._run(monkeypatch, tmp_path, stale))
|
||||||
|
assert patched and commented, "a genuine change must still notify"
|
||||||
|
|
||||||
|
def test_a_body_differing_only_by_CRLF_is_not_rewritten(self, monkeypatch, tmp_path):
|
||||||
|
# Forges round-trip line endings. Without normalising, every run would rewrite
|
||||||
|
# the body -- silent, but it churns updated_at and looks freshly touched daily.
|
||||||
|
body = compat.render_issue(self.ROWS).replace("\n", "\r\n")
|
||||||
|
patched, _ = self._writes(self._run(monkeypatch, tmp_path, body))
|
||||||
|
assert not patched
|
||||||
|
|||||||
@@ -1914,6 +1914,9 @@ class TestTheSnapshotRecordIsNeverLostToAnUnreadableFile:
|
|||||||
only record of a tree it had just failed to read.
|
only record of a tree it had just failed to read.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
@pytest.mark.skipif(os.geteuid() == 0, reason=
|
||||||
|
"chmod(0) cannot make a file unreadable for root (CAP_DAC_OVERRIDE); "
|
||||||
|
"the Gitea runner image executes jobs as root")
|
||||||
def test_an_unreadable_sidecar_raises_rather_than_reading_as_empty(self, tmp_path):
|
def test_an_unreadable_sidecar_raises_rather_than_reading_as_empty(self, tmp_path):
|
||||||
sc = tmp_path / "cloud_backup-5.snapshot"
|
sc = tmp_path / "cloud_backup-5.snapshot"
|
||||||
sc.write_text("Tap@snap\n")
|
sc.write_text("Tap@snap\n")
|
||||||
@@ -1927,6 +1930,9 @@ class TestTheSnapshotRecordIsNeverLostToAnUnreadableFile:
|
|||||||
def test_a_missing_sidecar_is_simply_empty(self, tmp_path):
|
def test_a_missing_sidecar_is_simply_empty(self, tmp_path):
|
||||||
assert tn._read_sidecar(str(tmp_path / "nope")) == []
|
assert tn._read_sidecar(str(tmp_path / "nope")) == []
|
||||||
|
|
||||||
|
@pytest.mark.skipif(os.geteuid() == 0, reason=
|
||||||
|
"chmod(0) cannot make a file unreadable for root (CAP_DAC_OVERRIDE); "
|
||||||
|
"the Gitea runner image executes jobs as root")
|
||||||
def test_cleanup_all_still_UNMOUNTS_when_a_sidecar_cannot_be_read(self, tmp_path):
|
def test_cleanup_all_still_UNMOUNTS_when_a_sidecar_cannot_be_read(self, tmp_path):
|
||||||
# cleanup_all is what recover.sh and uninstall.sh call, i.e. it runs precisely
|
# cleanup_all is what recover.sh and uninstall.sh call, i.e. it runs precisely
|
||||||
# when the box is already stuck. Its job is to get the mounts off. Aborting on
|
# when the box is already stuck. Its job is to get the mounts off. Aborting on
|
||||||
|
|||||||
+96
-13
@@ -871,6 +871,24 @@ def discover_refs(remote: str = REPO) -> list[str]:
|
|||||||
* UNRELEASED comes from the BRANCHES, because that is where a beta appears
|
* UNRELEASED comes from the BRANCHES, because that is where a beta appears
|
||||||
first: `release/26.0.0-BETA.3` had no tag yet while it was the newest beta.
|
first: `release/26.0.0-BETA.3` had no tag yet while it was the newest beta.
|
||||||
Catching breakage here, before it ships, is the whole point of this file.
|
Catching breakage here, before it ships, is the whole point of this file.
|
||||||
|
|
||||||
|
THE NEXT MAINTENANCE RELEASE IS ALSO A BRANCH, and it used to fall through both
|
||||||
|
sieves. `release/25.10.5` is branched but not yet tagged, and 25.10 has already
|
||||||
|
shipped -- so it is not in `shipped` (no tag) and it was excluded from `upcoming`
|
||||||
|
(its line is in `shipped_lines`). It was invisible. That is the one ref a 25.10.4
|
||||||
|
user is actually about to be upgraded onto, so a break there reaches people
|
||||||
|
BEFORE the daily check ever looks at it -- the exact hole this file exists to
|
||||||
|
close, on the only line anybody is running.
|
||||||
|
|
||||||
|
The rule that separates it from the two things we must NOT report:
|
||||||
|
|
||||||
|
* `release/24.10-RC.2` -- a prerelease of an already-shipped line. It is
|
||||||
|
history, not a warning; it sorts OLDER than TS-24.10.2.4, so it is dropped.
|
||||||
|
* `release/25.20.2.2` -- iX's typo branch. 25.20 never shipped, so it has no
|
||||||
|
TS tag, so it is not a line at all and is dropped.
|
||||||
|
|
||||||
|
...which is: a plain `release/X.Y.Z` branch counts only if its line HAS shipped
|
||||||
|
and it sorts NEWER than that line's newest tag. Both exclusions fall out of it.
|
||||||
"""
|
"""
|
||||||
tags = _ls_remote(remote, "--tags")
|
tags = _ls_remote(remote, "--tags")
|
||||||
heads = _ls_remote(remote, "--heads")
|
heads = _ls_remote(remote, "--heads")
|
||||||
@@ -878,26 +896,45 @@ def discover_refs(remote: str = REPO) -> list[str]:
|
|||||||
shipped = _newest_per_line([
|
shipped = _newest_per_line([
|
||||||
t for t in tags if t.startswith("TS-") and "-BETA" not in t and "-RC" not in t
|
t for t in tags if t.startswith("TS-") and "-BETA" not in t and "-RC" not in t
|
||||||
])
|
])
|
||||||
|
newest_shipped = {_version_of(t)[0][:2]: _version_of(t) for t in shipped}
|
||||||
|
|
||||||
|
release_heads = [h for h in heads if h.startswith("release/")]
|
||||||
|
|
||||||
# A prerelease of a line that has ALREADY shipped is history, not a warning:
|
# A prerelease of a line that has ALREADY shipped is history, not a warning:
|
||||||
# release/24.10-RC.2 still exists, and the nested module does not apply to it,
|
# release/24.10-RC.2 still exists, and the nested module does not apply to it,
|
||||||
# but 24.10 shipped long ago and TS-24.10.2.4 is fine. Reporting it would be a
|
# but 24.10 shipped long ago and TS-24.10.2.4 is fine. Reporting it would be a
|
||||||
# standing red row in the matrix for a version nobody can install.
|
# standing red row in the matrix for a version nobody can install.
|
||||||
shipped_lines = {_version_of(t)[0][:2] for t in shipped}
|
|
||||||
upcoming = [
|
upcoming = [
|
||||||
h for h in _newest_per_line([
|
h for h in _newest_per_line([
|
||||||
h for h in heads
|
h for h in release_heads if "-BETA" in h or "-RC" in h
|
||||||
if h.startswith("release/") and ("-BETA" in h or "-RC" in h)
|
|
||||||
])
|
])
|
||||||
if _version_of(h)[0][:2] not in shipped_lines
|
if _version_of(h)[0][:2] not in newest_shipped
|
||||||
]
|
]
|
||||||
|
|
||||||
return [*shipped, *upcoming, "master"]
|
# The next maintenance release of a line that HAS shipped: branched, untagged,
|
||||||
|
# and the very next thing those users get. See the docstring.
|
||||||
|
pending = []
|
||||||
|
for h in _newest_per_line([
|
||||||
|
h for h in release_heads if "-BETA" not in h and "-RC" not in h
|
||||||
|
]):
|
||||||
|
v = _version_of(h)
|
||||||
|
tagged = newest_shipped.get(v[0][:2])
|
||||||
|
if tagged and v > tagged:
|
||||||
|
pending.append(h)
|
||||||
|
|
||||||
|
return [*shipped, *pending, *upcoming, "master"]
|
||||||
|
|
||||||
|
|
||||||
def is_unreleased(ref: str) -> bool:
|
def is_unreleased(ref: str) -> bool:
|
||||||
"""master and any BETA/RC. Breakage here is early warning, not an outage."""
|
"""Anything iX has not TAGGED. Breakage here is early warning, not an outage.
|
||||||
return ref == "master" or "-BETA" in ref or "-RC" in ref
|
|
||||||
|
Keyed on where the ref came from, not on its name: `discover_refs` takes shipped
|
||||||
|
releases from `TS-*` TAGS and everything else from BRANCHES, so a `release/*` ref
|
||||||
|
is by construction something iX has not released yet. Testing for `-BETA`/`-RC`
|
||||||
|
instead would call `release/25.10.5` SHIPPED, and a break there would fail the
|
||||||
|
build as a live outage -- on a version nobody is running yet.
|
||||||
|
"""
|
||||||
|
return ref == "master" or ref.startswith("release/")
|
||||||
|
|
||||||
|
|
||||||
def matrix(refs=None, remote: str = REPO) -> list[dict]:
|
def matrix(refs=None, remote: str = REPO) -> list[dict]:
|
||||||
@@ -941,10 +978,15 @@ def is_broken(r: dict) -> bool:
|
|||||||
#: is static analysis of iX's source, which proves the patch's assumptions hold --
|
#: is static analysis of iX's source, which proves the patch's assumptions hold --
|
||||||
#: a strictly weaker claim than "a restore worked". Add a row only after doing it.
|
#: a strictly weaker claim than "a restore worked". Add a row only after doing it.
|
||||||
HARDWARE_VERIFIED = {
|
HARDWARE_VERIFIED = {
|
||||||
"25.10.4": "nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m",
|
"25.10.4": (
|
||||||
|
"v0.7.0: 3 live tasks — 191-dataset nested backup of /mnt/Tap, a "
|
||||||
|
"215-filesystem/2-zvol backup of /mnt/Tank/backups, and a non-nested one; "
|
||||||
|
"0 orphans, 0 leaked mounts, byte-identical restore; the collector also "
|
||||||
|
"reclaimed a real orphan the pool had been carrying"
|
||||||
|
),
|
||||||
"26.0.0-BETA.1": (
|
"26.0.0-BETA.1": (
|
||||||
"nested + providers; 274-snapshot recursive backup of a 292-dataset pool, "
|
"v0.7.0: 274-snapshot recursive backup of a 292-dataset pool; restored a "
|
||||||
"restored a 4-deep child dataset byte-identical"
|
"4-deep child dataset byte-identical; zvol-orphan case reproduced then closed"
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -958,6 +1000,16 @@ _LEGEND = """
|
|||||||
"ok" means *the patch's assumptions hold*, checked automatically against iX's
|
"ok" means *the patch's assumptions hold*, checked automatically against iX's
|
||||||
source. It does not mean a human ran a backup on it — that is the
|
source. It does not mean a human ran a backup on it — that is the
|
||||||
**Hardware-verified** column, which is filled in by hand and only by doing it.
|
**Hardware-verified** column, which is filled in by hand and only by doing it.
|
||||||
|
|
||||||
|
**`master` is not the next release.** iX branches each major off to its own
|
||||||
|
`release/` line and master rolls straight on to the one after — so master is
|
||||||
|
`27-dev` while 26 is still in beta. A **BROKEN** master means iX has changed
|
||||||
|
something that will reach users *a major release from now*, not in the version you
|
||||||
|
are about to install. Read the numbered rows for that.
|
||||||
|
|
||||||
|
A row like `25.10.5 _(unreleased)_` is the next maintenance release: branched by iX,
|
||||||
|
not tagged yet, and the very next thing a 25.10.4 box gets. It is checked precisely
|
||||||
|
because it is the one unshipped ref that reaches real users without warning.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
@@ -1005,17 +1057,42 @@ def update_readme(rows: list[dict], path: str = README) -> bool:
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def dev_label(rows: list[dict]) -> str:
|
||||||
|
"""What `master` is a development line FOR -- e.g. "27-dev".
|
||||||
|
|
||||||
|
`master` is NOT the next release, and labelling it "master _(unreleased)_" said
|
||||||
|
it was. On 2026-07-14 every recent commit on master targeted 27.0.0-BETA.1 while
|
||||||
|
26 was still in beta on its own `release/26.0.0-BETA.*` branches: master had
|
||||||
|
already rolled over to the major AFTER the one that has not shipped yet. So a red
|
||||||
|
`master` row read as "the version you are about to install is broken" when the
|
||||||
|
breakage was a year out, on a line nobody can even download. That is a false alarm
|
||||||
|
aimed squarely at the person deciding whether to trust this with their backups.
|
||||||
|
|
||||||
|
Derived, not hardcoded: the newest major we can see anywhere, plus one. iX branches
|
||||||
|
`release/N.0.0-BETA.1` off master and master immediately becomes N+1 -- so when 27
|
||||||
|
betas appear, this says 28-dev on its own.
|
||||||
|
"""
|
||||||
|
majors = [
|
||||||
|
v[0][0] for v in (_version_of(r["ref"]) for r in rows if r["ref"] != "master")
|
||||||
|
if v
|
||||||
|
]
|
||||||
|
return f"{max(majors) + 1}-dev" if majors else "unreleased"
|
||||||
|
|
||||||
|
|
||||||
def render_markdown(rows: list[dict]) -> str:
|
def render_markdown(rows: list[dict]) -> str:
|
||||||
"""The matrix, for the README."""
|
"""The matrix, for the README."""
|
||||||
out = [
|
out = [
|
||||||
"| TrueNAS | B2/S3 providers | Nested snapshots | Hardware-verified |",
|
"| TrueNAS | B2/S3 providers | Nested snapshots | Hardware-verified |",
|
||||||
"| --- | --- | --- | --- |",
|
"| --- | --- | --- | --- |",
|
||||||
]
|
]
|
||||||
|
dev = dev_label(rows)
|
||||||
for row in rows:
|
for row in rows:
|
||||||
m = row["modules"]
|
m = row["modules"]
|
||||||
ref = row["ref"]
|
ref = row["ref"]
|
||||||
label = ref.removeprefix("TS-").removeprefix("release/")
|
label = ref.removeprefix("TS-").removeprefix("release/")
|
||||||
if row["unreleased"]:
|
if ref == "master":
|
||||||
|
label = f"master _({dev})_"
|
||||||
|
elif row["unreleased"]:
|
||||||
label = f"{label} _(unreleased)_"
|
label = f"{label} _(unreleased)_"
|
||||||
|
|
||||||
cells = []
|
cells = []
|
||||||
@@ -1133,7 +1210,13 @@ def render_matrix(rows: list[dict]) -> str:
|
|||||||
hardware-verified column lives in COMPATIBILITY.md and is maintained by hand,
|
hardware-verified column lives in COMPATIBILITY.md and is maintained by hand,
|
||||||
because nothing else can honestly fill it in.
|
because nothing else can honestly fill it in.
|
||||||
"""
|
"""
|
||||||
w = max((len(r["ref"]) for r in rows), default=10)
|
dev = dev_label(rows)
|
||||||
|
# Same relabel as the README: a red `master` is a warning about the major AFTER
|
||||||
|
# next, and "master" alone reads as "the release you are about to install".
|
||||||
|
names = {r["ref"]: (f"master ({dev})" if r["ref"] == "master" else r["ref"])
|
||||||
|
for r in rows}
|
||||||
|
|
||||||
|
w = max((len(n) for n in names.values()), default=10)
|
||||||
lines = [
|
lines = [
|
||||||
f"{'TrueNAS'.ljust(w)} {'providers':<10} {'nested':<10}",
|
f"{'TrueNAS'.ljust(w)} {'providers':<10} {'nested':<10}",
|
||||||
f"{'-' * w} {'-' * 10} {'-' * 10}",
|
f"{'-' * w} {'-' * 10} {'-' * 10}",
|
||||||
@@ -1141,7 +1224,7 @@ def render_matrix(rows: list[dict]) -> str:
|
|||||||
for row in rows:
|
for row in rows:
|
||||||
m = row["modules"]
|
m = row["modules"]
|
||||||
lines.append(
|
lines.append(
|
||||||
f"{row['ref'].ljust(w)} "
|
f"{names[row['ref']].ljust(w)} "
|
||||||
f"{_verdict(m[PROVIDERS]):<10} {_verdict(m[NESTED]):<10}"
|
f"{_verdict(m[PROVIDERS]):<10} {_verdict(m[NESTED]):<10}"
|
||||||
)
|
)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
+44
-7
@@ -62,6 +62,19 @@ def _call(url, token, method="GET", data=None):
|
|||||||
return json.load(r) if r.length != 0 else {}
|
return json.load(r) if r.length != 0 else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _same(a, b):
|
||||||
|
"""Is the issue body already what we would write?
|
||||||
|
|
||||||
|
Compared after normalising line endings and trailing space: forges are free to
|
||||||
|
round-trip `\r\n`, and a body that only "differs" by that would be rewritten on
|
||||||
|
every single run -- a silent edit, but a pointless one that churns `updated_at`
|
||||||
|
and makes the issue look freshly touched every morning.
|
||||||
|
"""
|
||||||
|
def norm(s):
|
||||||
|
return "\n".join(line.rstrip() for line in (s or "").replace("\r\n", "\n").split("\n")).strip()
|
||||||
|
return norm(a) == norm(b)
|
||||||
|
|
||||||
|
|
||||||
def find_issue(api, token, title):
|
def find_issue(api, token, title):
|
||||||
"""The LOWEST-numbered issue with this title, open or closed.
|
"""The LOWEST-numbered issue with this title, open or closed.
|
||||||
|
|
||||||
@@ -69,11 +82,18 @@ def find_issue(api, token, title):
|
|||||||
existed once (an earlier version put the ref list in the title, so the identity
|
existed once (an earlier version put the ref list in the title, so the identity
|
||||||
changed whenever that set changed), and an order-dependent pick would alternate
|
changed whenever that set changed), and an order-dependent pick would alternate
|
||||||
between them -- reopening one while commenting on the other.
|
between them -- reopening one while commenting on the other.
|
||||||
|
|
||||||
|
Both forges list PRs alongside issues, but they SAY SO DIFFERENTLY: GitHub omits
|
||||||
|
the `pull_request` key on a plain issue, Gitea sends it as `null`. Testing for the
|
||||||
|
KEY therefore discards every Gitea issue as if it were a PR -- so this returned
|
||||||
|
None on every Gitea run, and the bot filed a brand-new duplicate report each time
|
||||||
|
instead of editing the one it already had. Test the VALUE; it is the only form
|
||||||
|
that is true on both.
|
||||||
"""
|
"""
|
||||||
issues = _call(f"{api}/issues?state=all&per_page=100", token)
|
issues = _call(f"{api}/issues?state=all&per_page=100&limit=100", token)
|
||||||
mine = [
|
mine = [
|
||||||
i for i in issues
|
i for i in issues
|
||||||
if i.get("title") == title and "pull_request" not in i # GitHub lists PRs here
|
if i.get("title") == title and not i.get("pull_request")
|
||||||
]
|
]
|
||||||
return min(mine, key=lambda i: i["number"]) if mine else None
|
return min(mine, key=lambda i: i["number"]) if mine else None
|
||||||
|
|
||||||
@@ -117,11 +137,24 @@ def main(argv):
|
|||||||
have = extract_fingerprint(issue.get("body") or "")
|
have = extract_fingerprint(issue.get("body") or "")
|
||||||
n = issue["number"]
|
n = issue["number"]
|
||||||
|
|
||||||
# ── the findings are UNCHANGED: say nothing ──────────────────────────────
|
# ── two different questions, and they were being answered with one answer ────
|
||||||
#
|
#
|
||||||
# This is the whole point. A daily "still broken, same as yesterday" comment is
|
# * IS THE BODY STILL TRUE? -> if not, rewrite it. Editing an issue body
|
||||||
# what taught everyone to ignore the last one.
|
# notifies NOBODY on either forge, so keeping it honest is free.
|
||||||
if have == want and issue["state"] == "open":
|
# * HAVE THE FINDINGS CHANGED? -> only then comment. Comments DO notify, and a
|
||||||
|
# daily "still broken, same as yesterday" is what teaches everyone to ignore
|
||||||
|
# the one that finally matters.
|
||||||
|
#
|
||||||
|
# Conflating them meant an unchanged FINGERPRINT froze the BODY. The fingerprint
|
||||||
|
# deliberately ignores everything that moves on its own -- healthy rows, the
|
||||||
|
# hardware-verified column, point releases, how a row is LABELLED -- so none of
|
||||||
|
# that could ever reach the report. Relabelling master `27-dev` (it is not the
|
||||||
|
# next release; a red row there was reading as "the version you are about to
|
||||||
|
# install is broken") would have shipped to the README and never to the issue
|
||||||
|
# anybody actually opens.
|
||||||
|
body_is_current = _same(issue.get("body"), body)
|
||||||
|
|
||||||
|
if have == want and issue["state"] == "open" and body_is_current:
|
||||||
print(f"#{n} is already current ({want}) — staying quiet")
|
print(f"#{n} is already current ({want}) — staying quiet")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
@@ -139,8 +172,12 @@ def main(argv):
|
|||||||
)
|
)
|
||||||
_call(f"{args.api}/issues/{n}/comments", args.token, "POST", {"body": note})
|
_call(f"{args.api}/issues/{n}/comments", args.token, "POST", {"body": note})
|
||||||
print(f"updated #{n}: {have} -> {want}")
|
print(f"updated #{n}: {have} -> {want}")
|
||||||
else:
|
elif issue["state"] != "open":
|
||||||
print(f"reopened #{n}")
|
print(f"reopened #{n}")
|
||||||
|
else:
|
||||||
|
# Same findings, new rendering. Silent by design: nothing has changed that
|
||||||
|
# anybody needs waking up for, but the report should not be telling lies.
|
||||||
|
print(f"#{n}: findings unchanged ({want}); body refreshed silently")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user