docs: TrueNAS compatibility matrix and the two-stage release process

The matrix is regenerated daily by CI rather than typed once and forgotten — a
support table that quietly goes stale is a false promise to someone deciding
whether to trust this with their backups.

.github/workflows/compat.yml

@@ -129,6 +129,35 @@ jobs:
       - name: matrix
         run: cat /tmp/matrix.md
 
+      # Keep the README's table true. A support matrix that quietly goes stale is not
+      # a stale doc -- it is a false promise to somebody deciding whether to trust
+      # this with their backups.
+      #
+      # Only ever touches the block between the COMPAT MATRIX markers, and only on
+      # the canonical host (Gitea) so the two forges cannot race each other. The
+      # `paths:` trigger above does not include README.md, so this cannot re-trigger
+      # itself; and a README change is documentation-only, which by design raises no
+      # update alert on anyone's box.
+      - name: refresh the README matrix
+        if: ${{ github.event_name == 'schedule' && !contains(github.server_url, 'github.com') }}
+        run: |
+          python3 - <<'PY'
+          import json, sys
+          sys.path.insert(0, "tools")
+          import compat
+          with open("/tmp/matrix.json") as fh:
+              rows = json.load(fh)
+          print("changed" if compat.update_readme(rows) else "unchanged")
+          PY
+
+          if ! git diff --quiet -- README.md; then
+            git config user.name  "truecloud-patch bot"
+            git config user.email "bot@onetick.ninja"
+            git add README.md
+            git commit -m "docs: refresh the TrueNAS compatibility matrix"
+            git push origin HEAD:main
+          fi
+
       # A broken SHIPPED release is an outage: users are on it right now.
       - name: fail if a shipped release is broken
         if: ${{ steps.check.outputs.shipped_broken != '0' }}

.gitignore

@@ -5,6 +5,9 @@
 /hook_status.json
 /disabled
 /nested_snapshots_enabled
+# Written by apply.sh when a module's assumptions no longer fit the installed
+# middlewared (see tools/compat.py). Evidence for the alert; not source.
+/incompatible.json
 
 # Python
 __pycache__/

CHANGELOG.md

@@ -37,6 +37,55 @@ worse than no alert, because one day it carries a security fix.
   detached HEAD, and `release.sh` now recognises that and says so, rather than
   emitting a confusing branch error.
 
+- **TrueNAS compatibility is now checked, not hoped for.**
+  [`tools/compat.py`](tools/compat.py) is a written-down record of everything each
+  module assumes about middlewared, checked in two places:
+
+  - **CI, daily** — against iXsystems' source at every release line *including
+    `master` and the current BETA/RC_. When an unreleased TrueNAS breaks the patch
+    it files a bug report automatically, so there is time to fix it before that
+    version reaches anyone. It also refreshes the README's support matrix, so the
+    table cannot quietly become a false promise.
+  - **`apply.sh`, at every boot** — against the middleware actually installed on the
+    box. **A module whose assumptions no longer hold is not applied.** Stock TrueNAS
+    without a feature beats TrueNAS with a broken backup.
+
+  This immediately found a real one: **TrueNAS 26 rewrites the entire `cloud_backup`
+  path from async to synchronous.** Every block the nested module injects is an
+  `async def` wrapping an `await`ed original, so on 26 it hands `sync.py` a coroutine
+  where it unpacks a tuple. Nobody would have found out until a restore failed. On
+  TrueNAS 26 the nested module now simply stays off.
+
+### Changed
+
+- **A stable release may not leave work stranded under `## Unreleased`.** Either it
+  is finished and belongs in the release, or the release is premature. Candidates
+  are exempt: an rc may legitimately have work queued behind it.
+
+- **`release.sh` refuses to run on an installed box.** The whole repo is cloned onto
+  every box, so this file is there too; `update.sh` pins the checkout to a tag in
+  detached HEAD, and `release.sh` now recognises that and says so, rather than
+  emitting a confusing branch error.
+
+- **Gitea (`git.onetick.ninja/flan/truenas-truecloud-patch`) is now canonical**, with
+  GitHub as a mirror. Both forges run the same workflows and publish the same
+  releases. The update alert now **derives the changelog URL from the `origin`
+  remote** instead of hard-coding GitHub — which matters more than it sounds: when
+  the changelog cannot be read, the alert deliberately fires *anyway* rather than
+  risk hiding a security fix, so a stale URL would not have disabled the alert, it
+  would have made it nag on every release including documentation-only ones.
+
+### Security
+
+- **Workflow expressions are no longer interpolated into shell.**
+  `echo "${{ steps.report.outputs.body }}"` pasted the compatibility report into the
+  script text, and the report is full of backticks — bash ran `create-snapshot`,
+  `def` and `async` as commands. Since that report is built from iXsystems' source,
+  anything landing in their tree would have executed on the runner. `inputs.tag` on
+  `workflow_dispatch` had the same shape, and that one is attacker-chosen. Data now
+  moves through files and scalars through `env:`; a test enforces it across every
+  workflow.
+
 ### Internal
 
 - Static-analysis annotations in `patch/alert_source.py` (`# noqa` placement). No

README.md

@@ -27,6 +27,66 @@ costs a fraction of the new Storj price.
 
 ---
 
+## TrueNAS compatibility
+
+<!-- BEGIN COMPAT MATRIX (generated by tools/compat.py --matrix --markdown) -->
+| TrueNAS | B2/S3 providers | Nested snapshots | Hardware-verified |
+| --- | --- | --- | --- |
+| 24.10.2.4 | ok | ok | — |
+| 25.04.2.6 | ok | ok | — |
+| 25.10.4 | ok | ok | nested + providers; 252-snapshot recursive backup of /mnt/Tap, 18m |
+| 26.0.0-BETA.3 _(unreleased)_ | ok | **BROKEN** | — |
+| master _(unreleased)_ | **BROKEN** | **BROKEN** | — |
+
+| verdict | meaning |
+| --- | --- |
+| **ok** | Every assumption the patch makes about middleware still holds. |
+| **BROKEN** | middleware changed underneath the patch. `apply.sh` **refuses to apply that module** on this version and leaves TrueNAS stock, so backups keep working — without the module's feature. |
+| **native** | TrueNAS does this itself now. The module retires; it is not a failure. |
+
+"ok" means *the patch's assumptions hold*, checked automatically against iX's
+source. It does not mean a human ran a backup on it — that is the
+**Hardware-verified** column, which is filled in by hand and only by doing it.
+<!-- END COMPAT MATRIX -->
+
+The table above is **regenerated daily by CI** — it is not a claim somebody typed
+once and forgot. **TrueCloud Backup does not exist before 24.10**, so earlier
+versions are absent rather than "unsupported".
+
+### ⚠️ TrueNAS 26 breaks nested snapshots (not yet released)
+
+TrueNAS 26 rewrites the whole `cloud_backup` path **from async to synchronous**.
+Every block the nested module injects is an `async def` wrapping an `await`ed
+original, so on 26 it would hand `sync.py` a coroutine where it unpacks a tuple.
+
+You do not need to do anything. `apply.sh` checks these assumptions against the
+middleware **actually installed on your box** at every boot, and will not apply a
+module that no longer fits. On TrueNAS 26 the nested module simply stays off:
+backups keep running, without nested-dataset coverage. A broken backup is worse
+than a missing feature.
+
+### How this is kept honest
+
+[`tools/compat.py`](tools/compat.py) is the single written-down record of what each
+module assumes about middleware. It is checked in two places:
+
+- **[CI](.github/workflows/compat.yml), daily** — against iXsystems' source at every
+  release line, *including `master` and the current BETA/RC*. When an unreleased
+  version breaks the patch, it files a bug report automatically. The point is to
+  find out while it is still a beta, rather than after it ships to you.
+- **`apply.sh`, at every boot** — against the middleware installed on your machine.
+  A module whose assumptions no longer hold is not applied.
+
+Regenerate the table with:
+
+```bash
+python3 tools/compat.py --matrix              # human-readable
+python3 tools/compat.py --matrix --markdown   # the table above
+python3 tools/compat.py --ref master          # one version, with reasons
+```
+
+---
+
 ## Before you install
 
 This project is unofficial and not affiliated with iXsystems. A few things worth
@@ -35,7 +95,8 @@ knowing:
 - It targets **internal middleware APIs** with no stability contract, so a
   TrueNAS update can break it. Every patch is fail-safe: if it can't apply,
   middlewared starts normally and the reason is logged to `apply.log`. Check the
-  log after an update.
+  log after an update. See [TrueNAS compatibility](#truenas-compatibility) for
+  which versions currently work.
 - If you file a TrueNAS bug report, **remove the patch first** and reproduce on a
   stock system.
 - **Test your restores.** True of any backup, but it matters more here — see
@@ -61,7 +122,10 @@ that capability natively. See [Native support](#if-truenas-adds-native-support).
 | `patch/create_task.py` | Create TrueCloud tasks with S3/B2 credentials; `verify` the patch state. |
 | `patch/alert_source.py` | The TrueNAS alert for "an update is available". Installed into `middlewared/alert/source/`. |
 | `patch/wait_restart.sh` | Waits for boot to actually settle before restarting middlewared. |
-| `tools/release_notes.py` | Extracts a version's CHANGELOG section; enforces version consistency. Used by CI. |
+| `release.sh` | Cut a release. Two stages, and the second is refused without the first. See [Releasing](#releasing). |
+| `tools/compat.py` | What each module assumes about middlewared — and the checker. Run by CI daily *and* by `apply.sh` at every boot. |
+| `tools/release_notes.py` | Extracts a version's CHANGELOG section; enforces version consistency. |
+| `tools/release_gate.py` | The barrier: a stable release must have been a release candidate on the same commit. |
 
 ## Development
 
@@ -86,9 +150,74 @@ are worth calling out, because nothing else would catch what they catch:
   newest CHANGELOG entry. `VERSION=` had silently drifted to three different
   values across the scripts before anything checked.
 
-Releases are automated: push a `vX.Y.Z` tag and the workflow runs the full suite,
-verifies the version matches, and cuts a GitHub release whose body **is** the
-matching `CHANGELOG.md` section — one source of truth for release notes.
+The project is hosted on **Gitea** (`git.onetick.ninja/flan/truenas-truecloud-patch`)
+and mirrored to GitHub. Both run the same workflows — Gitea reads
+`.github/workflows/` too — so a change is checked twice, on two independent runners.
+
+---
+
+## Releasing
+
+**Every release interrupts every user.** An update alert fires on each installed
+box (see [Update alerts](#update-alerts)), so a release that exists only to fix the
+last release teaches people to dismiss the alert — and one day that alert will be
+carrying a security fix. This project cut twelve releases in a single day once.
+Never again, and not by good intentions: by a gate.
+
+### The rule
+
+> A stable `vX.Y.Z` may only be published if a `vX.Y.Z-rcN` tag points at the
+> **same commit**.
+
+Release candidates are **invisible to users**: `update.sh` and the update alert both
+take the newest plain `vX.Y.Z` tag, so an `-rc` is never offered as an update. All
+the debugging therefore happens across `rc1`, `rc2`, `rc3` — at nobody's expense —
+instead of across `v0.5.0`, `v0.5.1`, `v0.5.2`, at everybody's.
+
+"The candidate passed, then I pushed one more little fix" is refused **by name**.
+That is not hypothetical; it is exactly how v0.5.1 happened.
+
+### Day to day
+
+You don't touch the release machinery. Write your changes under `## Unreleased` in
+`CHANGELOG.md` and push to `main`. `main` is a work surface — it is allowed to be
+mid-thought. Releasing is a separate, deliberate act.
+
+### Cutting a release
+
+```bash
+bash release.sh 0.6.0 --check      # what would ship? what is the next rc?
+
+bash release.sh 0.6.0 --rc         # promotes `## Unreleased` -> v0.6.0, stamps every
+                                   # VERSION=, tags v0.6.0-rc1, pushes. Users see nothing.
+
+#   ... install it on a real box. Exercise it. Break it. ...
+#   Found a bug? Fix it on main, then `bash release.sh 0.6.0 --rc` again -> rc2.
+
+bash release.sh 0.6.0 --promote    # publishes v0.6.0. REFUSED unless an rc points here.
+```
+
+### The gates, and where they live
+
+The logic is Python so it can be unit-tested; CI is the enforcement boundary
+because it is the only actor holding the token that publishes. `release.sh` runs the
+**same** code locally so you fail in 200 ms instead of after a push.
+
+| gate | enforces | where |
+| --- | --- | --- |
+| [`release_notes.py check`](tools/release_notes.py) | every script's `VERSION=` matches the tag; the CHANGELOG section exists and is non-empty; **nothing is stranded under `## Unreleased`** | `release.sh` + CI |
+| [`release_gate.py`](tools/release_gate.py) | **an rc points at this exact commit** | `release.sh` + CI |
+| the full suite | ruff, pytest, shellcheck, `bash -n` — re-run against the *tagged* commit | CI |
+
+A release's body **is** its `CHANGELOG.md` section — there is no second place to
+write release notes, and therefore no second place for them to go stale. Releases
+are published on both forges.
+
+### Why the alert doesn't nag
+
+A release whose CHANGELOG contains only a `### Docs` section changed no code, and
+raises **no alert**. Candidates raise no alert either. So the only thing that ever
+interrupts a user is a real, complete change — which is the entire point.
 
 ---
 

tools/compat.py

@@ -445,8 +445,37 @@ source. It does not mean a human ran a backup on it — that is the
 """
 
 
+#: The README's matrix lives between these. CI regenerates it daily, so a table
+#: claiming the patch works on a TrueNAS that iX has since changed cannot survive
+#: for longer than a day -- a stale support matrix is not a stale doc, it is a lie
+#: to somebody deciding whether to trust this with their backups.
+ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+README = os.path.join(ROOT_DIR, "README.md")
+BEGIN = "<!-- BEGIN COMPAT MATRIX (generated by tools/compat.py --matrix --markdown) -->"
+END = "<!-- END COMPAT MATRIX -->"
+
+
+def update_readme(rows: list[dict], path: str = README) -> bool:
+    """Rewrite the README's matrix block. True if it changed."""
+    with open(path, encoding="utf-8") as fh:
+        text = fh.read()
+
+    i, j = text.find(BEGIN), text.find(END)
+    if i == -1 or j == -1:
+        raise ValueError(f"{path} has no COMPAT MATRIX markers")
+
+    new = f"{BEGIN}\n{render_markdown(rows).rstrip()}\n{END}"
+    old = text[i:j + len(END)]
+    if old == new:
+        return False
+
+    with open(path, "w", encoding="utf-8") as fh:
+        fh.write(text[:i] + new + text[j + len(END):])
+    return True
+
+
 def render_markdown(rows: list[dict]) -> str:
-    """The matrix, for COMPATIBILITY.md and the README."""
+    """The matrix, for the README."""
     out = [
         "| TrueNAS | B2/S3 providers | Nested snapshots | Hardware-verified |",
         "| --- | --- | --- | --- |",
@@ -527,7 +556,9 @@ def main(argv):
                     help="check only this module (repeatable)")
     ap.add_argument("--json", action="store_true")
     ap.add_argument("--markdown", action="store_true",
-                    help="with --matrix: emit the table for COMPATIBILITY.md")
+                    help="with --matrix: emit the table as markdown")
+    ap.add_argument("--update-readme", action="store_true",
+                    help="with --matrix: rewrite the README's matrix block in place")
     args = ap.parse_args(argv[1:])
 
     if args.matrix:
@@ -536,6 +567,9 @@ def main(argv):
             print(json.dumps(rows, indent=2))
         elif args.markdown:
             print(render_markdown(rows))
+        elif args.update_readme:
+            changed = update_readme(rows)
+            print("README.md updated" if changed else "README.md already current")
         else:
             print(render_matrix(rows))
         # A broken UNRELEASED line (master, -BETA, -RC) is a warning, not a build