Compare commits

..
3 Commits
Author SHA1 Message Date
flan 8aa9038226 Fix: --disable-nested-snapshots did not disable anything until reboot
apply.sh only ever ADDED patches; there was no revert path anywhere. Disabling
removed the opt-in marker and then merely skipped re-applying -- but the overlay
persists for the whole boot, so the previously patched cloud/{snapshot,crud}.py,
cloud_backup/sync.py and _truecloud_nested.py were all still on disk, and
middlewared re-imported them on the restart install.sh performs.

It printed "DISABLED (stock guard restored)" while the feature kept running until
the next reboot. Someone disabling it because they were worried about it would
have believed it was off.

apply.sh now reverts on every not-needed path (opt-out, or superseded by native
support): remove the module FIRST -- every injected block is guarded by
`if _tc_nested is not None`, so the stock guard comes back even if a later step
fails -- then strip the appended blocks from the three patched files.

restic.py also carries a TRUECLOUD_PATCH block but belongs to the providers
module; reverting it would silently break B2 backups, so it is explicitly
excluded. Verified: the three nested files restore byte-for-byte to stock, the
module is removed, and restic.py's block survives.

install.sh --disable also tears the staging tree down first, since those bind
mounts pin ZFS snapshots that could otherwise never be destroyed.

Updating WITHOUT the flag was always correct and is unchanged: the nested module
is never installed into middleware unless explicitly enabled.

109 tests, ruff and shellcheck clean.
2026-07-13 15:23:27 +00:00
flan ba533dc8ae v0.3.1: automated releases + version-drift check
The "Automated releases" entry was written under v0.3.0, but that commit landed
after the v0.3.0 tag -- the CHANGELOG was claiming the release contained something
it did not. Moved to its own version rather than left as a quiet inaccuracy.

Bumps VERSION to 0.3.1 across all four scripts, which the new consistency check
now enforces.
2026-07-13 15:07:29 +00:00
flan eb91a337cd Automate releases from tags
Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never
released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the
incident that took every app down.

Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is
the matching CHANGELOG.md section -- one source of truth for release notes, so
there is no second place for them to be wrong.

The workflow refuses to publish when:
  - the tests, ruff, or bash -n fail (a tagged commit is what people install; it
    must be at least as good as main)
  - the tag does not match the VERSION= declared by every script
  - CHANGELOG.md has no section for the tag, or the section is empty

That version check is not theoretical: VERSION= had drifted to three different
values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing
noticed until this release. tests/test_release_notes.py now asserts the scripts
agree with each other and with the newest CHANGELOG entry, so the drift cannot
come back.

workflow_dispatch takes an existing tag, so releases can be backfilled for tags
that were pushed before this existed.

106 tests, ruff and shellcheck clean.
2026-07-13 15:05:46 +00:00
10 changed files with 564 additions and 18 deletions
+1 -1
View File
@@ -51,7 +51,7 @@ jobs:
run: python -m pip install --upgrade pip pytest ruff
- name: ruff
run: ruff check patch tests
run: ruff check patch tests tools
- name: pytest
run: pytest tests -v
+102
View File
@@ -0,0 +1,102 @@
name: Release
# Push a tag, get a release. The body always comes from CHANGELOG.md, so there is
# no second place to write release notes and therefore no second place for them to
# go stale.
#
# git tag -a v0.4.0 -m "v0.4.0" && git push origin v0.4.0
#
# workflow_dispatch re-cuts (or updates) the release for a tag that already
# exists, since re-pushing an existing tag triggers nothing.
#
# It checks out the TAG, because the tagged code is what people install and it has
# to pass its own tests. That means it only works for tags that actually contain
# this tooling (>= v0.3.0). Tags older than that were backfilled by hand.
on:
push:
tags: ["v*"]
workflow_dispatch:
inputs:
tag:
description: "Existing tag to create a release for (e.g. v0.2.1)"
required: true
type: string
permissions:
contents: write
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Resolve tag
id: tag
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "tag=${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
else
echo "tag=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
fi
- uses: actions/checkout@v4
with:
ref: ${{ steps.tag.outputs.tag }}
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.13"
# Never publish a release for code that does not pass its own tests. A
# tagged commit is what people install; it has to be at least as good as
# main.
- name: install dev deps
run: python -m pip install --upgrade pip pytest ruff
- name: ruff
run: ruff check patch tests tools
- name: pytest
run: pytest tests -q
- name: shell syntax
run: |
fail=0
while IFS= read -r f; do
bash -n "$f" || { echo "::error file=$f::bash syntax error"; fail=1; }
done < <(find . -name '*.sh' -not -path './.git/*')
exit $fail
# Catches the failure mode this repo actually had: VERSION= drifted to
# three different values across the scripts, and nothing noticed.
- name: version matches tag and CHANGELOG has a section
run: python3 tools/release_notes.py check "${{ steps.tag.outputs.tag }}"
- name: extract release notes from CHANGELOG
run: |
python3 tools/release_notes.py notes "${{ steps.tag.outputs.tag }}" > /tmp/notes.md
echo "--- release body ---"
cat /tmp/notes.md
- name: create or update the release
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ steps.tag.outputs.tag }}
run: |
# Pre-1.0 and any -rc/-beta suffix ship as prereleases, not "Latest".
prerelease=""
case "$TAG" in
*-rc*|*-beta*|*-alpha*) prerelease="--prerelease" ;;
esac
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG exists — updating notes."
gh release edit "$TAG" --notes-file /tmp/notes.md
else
# shellcheck disable=SC2086
gh release create "$TAG" \
--title "$TAG" \
--notes-file /tmp/notes.md \
$prerelease
fi
+48
View File
@@ -1,5 +1,53 @@
# Changelog
## v0.3.2 — 2026-07-13
### Fixed
- **`install.sh --disable-nested-snapshots` did not actually disable anything
until the next reboot.** `apply.sh` only ever *added* patches — there was no
revert path. Disabling removed the opt-in marker and then merely *skipped*
re-applying, but the overlay persists for the whole boot, so the previously
patched `plugins/cloud/{snapshot,crud}.py`, `plugins/cloud_backup/sync.py` and
`_truecloud_nested.py` were all still sitting there — and middlewared
re-imported them on the restart `install.sh` performs.
It printed *"DISABLED (stock guard restored)"* while the feature kept running.
Someone turning it off *because they were worried about it* would have believed
it was off.
`apply.sh` now actively reverts: it removes the module first (every injected
block is guarded by `if _tc_nested is not None`, so the stock guard is restored
even if a later step fails), then strips its appended blocks from the three
patched files. `restic.py` also carries a `TRUECLOUD_PATCH` block but belongs to
the *providers* module and is deliberately left alone — reverting it would break
B2 backups. `install.sh --disable` also tears down any staging tree first, since
those bind mounts pin ZFS snapshots that could otherwise never be destroyed.
Updating **without** the flag was always correct and is unchanged: the nested
module is never installed into middleware unless it is explicitly enabled.
## v0.3.1 — 2026-07-13
### Added
- **Automated releases.** Pushing a `v*` tag runs the full test suite and then
cuts a GitHub release whose body is the matching `CHANGELOG.md` section — so
release notes have exactly one source of truth, and no second place to go stale.
The workflow refuses to publish if the tests fail, if the tag does not match the
`VERSION=` declared by every script, or if the CHANGELOG has no section for it.
- **Version-drift check.** `VERSION=` had silently diverged to three different
values across `install.sh`, `uninstall.sh`, `recover.sh`, and `patch/apply.sh`,
and nothing noticed. CI now asserts every script agrees with the others and with
the newest CHANGELOG entry.
### Note
- Releases for `v0.2.0` and `v0.2.1` were backfilled — they had been tagged but
never released, so the releases page jumped v0.1.0 → v0.3.0 and hid the fix for
the boot race that took every app down.
## v0.3.0 — 2026-07-13
### Added
+10 -3
View File
@@ -18,7 +18,7 @@
set -euo pipefail
VERSION="0.3.0"
VERSION="0.3.2"
# The directory containing install.sh is the permanent install location.
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -160,10 +160,17 @@ case "$_nested_choice" in
off)
if [ -f "$_NESTED_MARKER" ]; then
rm -f "$_NESTED_MARKER"
echo "Nested-dataset snapshots: DISABLED (stock guard restored)."
# Tear down any staging tree first: those bind mounts PIN their ZFS
# snapshots, so leaving them would block those snapshots from ever
# being destroyed. apply.sh (below) then reverts the patched files.
python3 "$PATCH_DIR/patch/truecloud_nested.py" cleanup || \
echo " WARNING: staging mounts remain; unmount them manually."
echo "Nested-dataset snapshots: DISABLED."
echo " apply.sh will revert the patched middleware files and the stock"
echo " guard is restored when middlewared restarts (this script does that)."
echo " Any task that already has snapshot=true on a nested dataset will"
echo " fail validation on its next edit. Turn the option off on those"
echo " tasks, or re-run with --enable-nested-snapshots."
echo " tasks first, or re-run with --enable-nested-snapshots."
else
echo "Nested-dataset snapshots: already disabled."
fi
+66 -3
View File
@@ -32,7 +32,7 @@
# Derive PATCH_DIR from this script's location (parent of the patch/ directory).
PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
LOG="$PATCH_DIR/apply.log"
VERSION="0.3.0"
VERSION="0.3.2"
# Rotate log at 512 KB to avoid unbounded growth on a system volume.
# Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -477,6 +477,57 @@ def patch_file(path, block):
with open(path, "w", encoding="utf-8") as fh:
fh.write(base.rstrip("\n") + "\n" + block)
def unpatch_file(path):
"""Strip our appended block, restoring the stock file. True if it was patched."""
try:
with open(path, encoding="utf-8") as fh:
content = fh.read()
except OSError:
return False
idx = content.find("\n# TRUECLOUD_PATCH")
if idx == -1:
return False
try:
with open(path, "w", encoding="utf-8") as fh:
fh.write(content[:idx].rstrip("\n") + "\n")
except OSError:
return False
return True
def revert_nested(cloud_dir, sync_path):
"""Undo the nested patch. Returns the names of what was actually reverted.
Skipping the patch is NOT enough to disable the feature. The overlay persists
for the whole boot, so an earlier run this boot may already have written the
patched files -- and middlewared re-imports them on the restart that
install.sh performs. Without this, `install.sh --disable-nested-snapshots`
would report "disabled" while the feature kept running until the next reboot.
"""
reverted = []
# Remove the module FIRST. Every injected block is guarded by
# `if _tc_nested is not None`, so once it is gone they all no-op even if a
# later step here fails -- the guard is restored no matter what.
try:
os.unlink(os.path.join(cloud_dir, '_truecloud_nested.py'))
reverted.append('_truecloud_nested.py')
except OSError:
pass
# NB: restic.py also carries a TRUECLOUD_PATCH block, but that belongs to the
# providers module. Only these three are ours to revert.
for name, path in (
('crud.py', os.path.join(cloud_dir, 'crud.py')),
('sync.py', sync_path),
('snapshot.py', os.path.join(cloud_dir, 'snapshot.py')),
):
if unpatch_file(path):
reverted.append(name)
return reverted
b2_ok = restic_ok = False
nested_ok = False
@@ -517,16 +568,28 @@ else:
# guard LAST. If anything fails partway, the guard is still in place and the
# option stays unavailable -- we never expose "guard removed, traversal missing".
nested_detail = ''
if not nested_needed:
# Not just "skip": actively revert. The overlay lives for the whole boot, so a
# previously-applied patch is still sitting there and middlewared would
# re-import it on restart. See revert_nested().
if not nested_enabled:
nested_detail = 'disabled (opt-in; enable with: install.sh --enable-nested-snapshots)'
print('INFO: Nested-dataset snapshot support is disabled (opt-in feature).')
print('INFO: Enable with: bash install.sh --enable-nested-snapshots')
elif nested_native:
nested_detail = 'superseded: TrueNAS handles nested-dataset snapshots natively'
print('INFO: Nested module skipped — TrueNAS now handles nesting natively.')
elif not nested_needed:
else:
nested_detail = 'not needed'
print('INFO: Nested module skipped.')
reverted = revert_nested(cloud_dir, sync_path)
if reverted:
print('OK: Reverted a previously-applied nested patch (' + ', '.join(reverted) + ').')
print(' The stock nesting guard is restored once middlewared restarts.')
nested_detail += ' — previous patch reverted'
if not nested_enabled:
print('INFO: Enable with: bash install.sh --enable-nested-snapshots')
else:
try:
snapshot_py = os.path.join(cloud_dir, 'snapshot.py')
+1 -1
View File
@@ -17,7 +17,7 @@
# bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
# systemctl restart middlewared
VERSION="0.3.0"
VERSION="0.3.2"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
+48 -1
View File
@@ -263,10 +263,57 @@ class TestOptIn:
def test_patching_is_skipped_entirely_when_disabled(self):
# The guard-relaxing crud.py patch must be inside the enabled branch.
src = heredoc_source()
gate = src.index("if not nested_enabled:")
gate = src.index("if not nested_needed:")
crud = src.index("patch_file(crud_py, CRUD_BLOCK)")
assert gate < crud, "crud.py patch must sit inside the opt-in branch"
def test_disabling_REVERTS_the_patch_rather_than_merely_skipping_it(self):
"""Skipping is not disabling.
The overlay persists for the whole boot, so a patch applied by an earlier
run this boot is still on disk — and middlewared re-imports it on the
restart install.sh performs. Without an active revert,
`--disable-nested-snapshots` reports "disabled" while the feature keeps
running until the next reboot.
"""
src = heredoc_source()
assert "def unpatch_file(" in src
assert "def revert_nested(" in src
# The revert must run on every not-needed path (opt-out, superseded).
gate = src.index("if not nested_needed:")
revert = src.index("reverted = revert_nested(")
patch = src.index("patch_file(crud_py, CRUD_BLOCK)")
assert gate < revert < patch, "revert belongs in the not-needed branch"
def test_revert_removes_the_module_before_unpatching_files(self):
# Every injected block is guarded by `if _tc_nested is not None`, so
# deleting the module first means the guard is restored even if a later
# unpatch step fails.
src = heredoc_source()
body = src[src.index("def revert_nested("):src.index("def patch_file(") if
src.index("def patch_file(") > src.index("def revert_nested(") else len(src)]
body = src[src.index("def revert_nested("):]
body = body[:body.index("\n\n\n")] if "\n\n\n" in body else body
assert body.index("_truecloud_nested.py") < body.index("crud.py")
def test_revert_never_touches_the_providers_patch(self):
# restic.py also carries a TRUECLOUD_PATCH block, but it belongs to the
# providers module. Reverting it would silently break B2 backups.
src = heredoc_source()
body = src[src.index("def revert_nested("):]
body = body[:body.index("return reverted")]
# Comments legitimately *mention* restic.py to explain why it is excluded;
# what matters is that no code line touches it.
code = "\n".join(
ln for ln in body.splitlines() if not ln.lstrip().startswith("#")
)
assert "restic" not in code
assert "b2.py" not in code
# It must only ever revert these three, plus the module itself.
assert "crud.py" in code
assert "sync_path" in code
assert "snapshot.py" in code
def test_guard_is_relaxed_only_after_traversal_is_installed():
# Ordering in apply.sh is a safety property: copy module -> patch snapshot.py
+122
View File
@@ -0,0 +1,122 @@
"""Tests for the release automation.
The release workflow refuses to publish unless these hold, so a bad tag fails
loudly in CI instead of shipping a release whose notes are empty, wrong, or whose
scripts announce a different version than the tag.
That last one is not hypothetical: VERSION= drifted to three different values
across install.sh / uninstall.sh / recover.sh / apply.sh and nothing noticed.
"""
import os
import sys
import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
from release_notes import ( # noqa: E402
changelog_versions,
check,
extract_notes,
normalise,
script_versions,
)
REPO = os.path.join(os.path.dirname(__file__), "..")
SAMPLE = """\
# Changelog
## v0.3.0 — 2026-07-13
### Added
- the new thing
## v0.2.1 — 2026-07-09
### Fixed
- the old thing
## v0.2.0 — 2026-07-08
- first
"""
class TestExtractNotes:
def test_returns_only_that_versions_body(self):
body = extract_notes(SAMPLE, "v0.3.0")
assert "the new thing" in body
assert "the old thing" not in body
# The version heading itself is dropped (GitHub renders its own title),
# but sub-headings like "### Added" must survive.
assert not body.startswith("## v")
assert body.startswith("### Added")
def test_stops_at_the_next_version_heading(self):
body = extract_notes(SAMPLE, "v0.2.1")
assert "the old thing" in body
assert "first" not in body
def test_last_section_runs_to_end_of_file(self):
assert "first" in extract_notes(SAMPLE, "v0.2.0")
def test_accepts_the_tag_with_or_without_the_v(self):
assert extract_notes(SAMPLE, "0.3.0") == extract_notes(SAMPLE, "v0.3.0")
def test_unknown_version_raises_rather_than_returning_empty(self):
# An empty release body is worse than a failed release.
with pytest.raises(KeyError, match="no section"):
extract_notes(SAMPLE, "v9.9.9")
class TestChangelogVersions:
def test_lists_versions_newest_first(self):
assert changelog_versions(SAMPLE) == ["0.3.0", "0.2.1", "0.2.0"]
class TestAgainstTheRealRepo:
"""These run against the actual files, so drift breaks the build."""
def test_every_script_declares_a_version(self):
from release_notes import VERSIONED_FILES
found = script_versions(REPO)
missing = [f for f in VERSIONED_FILES if f not in found]
assert not missing, f"no VERSION= in: {missing}"
def test_all_scripts_agree_on_the_version(self):
versions = {normalise(v) for v in script_versions(REPO).values()}
assert len(versions) == 1, f"scripts disagree on version: {sorted(versions)}"
def test_the_current_version_has_a_changelog_section(self):
version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
body = extract_notes(fh.read(), version)
assert body, f"CHANGELOG.md has no content for v{version}"
def test_the_current_version_is_the_newest_changelog_entry(self):
version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
newest = changelog_versions(fh.read())[0]
assert newest == version, (
f"scripts say v{version} but the newest CHANGELOG entry is v{newest}"
)
def test_check_passes_for_the_current_version(self):
version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
assert check(version, REPO) == []
class TestCheckCatchesMistakes:
def test_reports_a_tag_that_no_script_matches(self):
problems = check("v9.9.9", REPO)
assert problems
assert any("declares VERSION" in p for p in problems)
def test_reports_a_missing_changelog_section(self):
problems = check("v9.9.9", REPO)
assert any("no section" in p for p in problems)
+157
View File
@@ -0,0 +1,157 @@
#!/usr/bin/env python3
"""Extract one version's section from CHANGELOG.md, and check version consistency.
Used by .github/workflows/release.yml so a release's body is always the changelog
entry -- there is no second place to write release notes, and therefore no second
place for them to be wrong.
python3 tools/release_notes.py notes v0.3.0 # -> the section body
python3 tools/release_notes.py version # -> version per the scripts
python3 tools/release_notes.py check v0.3.0 # -> exit 1 on any mismatch
"""
from __future__ import annotations
import os
import re
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CHANGELOG = os.path.join(ROOT, "CHANGELOG.md")
# Every script prints a version; they must all agree, and agree with the tag.
# They drifted to three different values once (0.0.4 / 0.2.1) before anything
# checked them.
VERSIONED_FILES = [
"install.sh",
"uninstall.sh",
"recover.sh",
os.path.join("patch", "apply.sh"),
]
_VERSION_RE = re.compile(r'^VERSION="([^"]+)"', re.M)
_HEADING_RE = re.compile(r"^##\s+v?(\d+\.\d+\.\d+[^\s]*)", re.M)
def normalise(v: str) -> str:
return v.strip().lstrip("v")
def script_versions(root: str = ROOT) -> dict[str, str]:
"""VERSION= as declared by each script."""
found = {}
for rel in VERSIONED_FILES:
path = os.path.join(root, rel)
try:
with open(path, encoding="utf-8") as fh:
m = _VERSION_RE.search(fh.read())
except OSError:
continue
if m:
found[rel] = m.group(1)
return found
def changelog_versions(text: str) -> list[str]:
"""Versions with a section in the changelog, newest first."""
return [normalise(v) for v in _HEADING_RE.findall(text)]
def extract_notes(text: str, version: str) -> str:
"""The body of one version's section, without its heading.
Raises KeyError if the version has no section -- a release with an empty or
wrong body is worse than a failed release.
"""
want = normalise(version)
lines = text.splitlines()
start = None
for i, line in enumerate(lines):
m = _HEADING_RE.match(line)
if m and normalise(m.group(1)) == want:
start = i + 1
break
if start is None:
raise KeyError(f"CHANGELOG.md has no section for v{want}")
end = len(lines)
for i in range(start, len(lines)):
if _HEADING_RE.match(lines[i]):
end = i
break
return "\n".join(lines[start:end]).strip()
def check(version: str, root: str = ROOT) -> list[str]:
"""Every reason this version is not releasable. Empty list means it is."""
want = normalise(version)
problems = []
versions = script_versions(root)
for rel, got in sorted(versions.items()):
if normalise(got) != want:
problems.append(f"{rel} declares VERSION={got!r}, tag is v{want}")
missing = [r for r in VERSIONED_FILES if r not in versions]
for rel in missing:
problems.append(f"{rel} has no VERSION= line")
try:
with open(os.path.join(root, "CHANGELOG.md"), encoding="utf-8") as fh:
text = fh.read()
except OSError as e:
problems.append(f"cannot read CHANGELOG.md: {e}")
return problems
try:
body = extract_notes(text, want)
except KeyError as e:
problems.append(str(e))
else:
if not body:
problems.append(f"CHANGELOG.md section for v{want} is empty")
return problems
def main(argv):
if len(argv) < 2:
print(__doc__, file=sys.stderr)
return 2
cmd = argv[1]
if cmd == "version":
versions = set(map(normalise, script_versions().values()))
if len(versions) != 1:
print(f"scripts disagree on version: {sorted(versions)}", file=sys.stderr)
return 1
print(versions.pop())
return 0
if len(argv) < 3:
print(f"usage: {argv[0]} {cmd} <version>", file=sys.stderr)
return 2
version = argv[2]
if cmd == "notes":
with open(CHANGELOG, encoding="utf-8") as fh:
print(extract_notes(fh.read(), version))
return 0
if cmd == "check":
problems = check(version)
for p in problems:
print(f"::error::{p}")
if problems:
return 1
print(f"v{normalise(version)} is consistent across scripts and CHANGELOG")
return 0
print(f"unknown command: {cmd}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main(sys.argv))
+1 -1
View File
@@ -3,7 +3,7 @@
set -euo pipefail
VERSION="0.3.0"
VERSION="0.3.2"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
_HOOK_COMMENT='TrueCloud provider patch (S3/B2)'