Compare commits

...
4 Commits
Author SHA1 Message Date
flan 60b3ac4557 v0.3.3: keep the restic repo password out of argv and shell history
Security
--------
create_task.py shelled out to `midclt call cloud_backup.create '<json>'`, and that
JSON carries the restic repository password -- so it sat in the subprocess's argv,
which is world-readable via ps, for the duration of the call. That password is the
encryption key for the entire cloud backup repository.

It now talks to the middleware through truenas_api_client, the library that backs
midclt itself, so the password never leaves this process's memory. Verified on a
live box: list-tasks and list-credentials work through the new transport.

--password is also no longer required, because passing a secret as a CLI argument
writes it to shell history permanently. --password-stdin reads it from stdin, and
with neither flag the tool prompts via getpass. --password still works but warns.

Fixed
-----
uninstall.sh could leave every patch installed. It reverted by unmounting the
overlay -- but apply.sh only mounts one when the target directory is read-only. On
a writable /usr it patches the real files in place, and uninstall would remove the
boot hook, report success, and leave the patch applied. It now strips the appended
blocks from the middleware files explicitly. This also covers the case where the
overlay unmount fails.

create_task.py's __version__ had been stuck at 0.2.0 through three releases. The
version-drift check added in v0.3.1 only looked at VERSION= in shell scripts, so it
missed the one file that actually shows a version to users (--version). The check
now covers __version__ too -- and caught this immediately.

118 tests, ruff and shellcheck clean.
2026-07-13 15:43:24 +00:00
flan 8aa9038226 Fix: --disable-nested-snapshots did not disable anything until reboot
apply.sh only ever ADDED patches; there was no revert path anywhere. Disabling
removed the opt-in marker and then merely skipped re-applying -- but the overlay
persists for the whole boot, so the previously patched cloud/{snapshot,crud}.py,
cloud_backup/sync.py and _truecloud_nested.py were all still on disk, and
middlewared re-imported them on the restart install.sh performs.

It printed "DISABLED (stock guard restored)" while the feature kept running until
the next reboot. Someone disabling it because they were worried about it would
have believed it was off.

apply.sh now reverts on every not-needed path (opt-out, or superseded by native
support): remove the module FIRST -- every injected block is guarded by
`if _tc_nested is not None`, so the stock guard comes back even if a later step
fails -- then strip the appended blocks from the three patched files.

restic.py also carries a TRUECLOUD_PATCH block but belongs to the providers
module; reverting it would silently break B2 backups, so it is explicitly
excluded. Verified: the three nested files restore byte-for-byte to stock, the
module is removed, and restic.py's block survives.

install.sh --disable also tears the staging tree down first, since those bind
mounts pin ZFS snapshots that could otherwise never be destroyed.

Updating WITHOUT the flag was always correct and is unchanged: the nested module
is never installed into middleware unless explicitly enabled.

109 tests, ruff and shellcheck clean.
2026-07-13 15:23:27 +00:00
flan ba533dc8ae v0.3.1: automated releases + version-drift check
The "Automated releases" entry was written under v0.3.0, but that commit landed
after the v0.3.0 tag -- the CHANGELOG was claiming the release contained something
it did not. Moved to its own version rather than left as a quiet inaccuracy.

Bumps VERSION to 0.3.1 across all four scripts, which the new consistency check
now enforces.
2026-07-13 15:07:29 +00:00
flan eb91a337cd Automate releases from tags
Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never
released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the
incident that took every app down.

Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is
the matching CHANGELOG.md section -- one source of truth for release notes, so
there is no second place for them to be wrong.

The workflow refuses to publish when:
  - the tests, ruff, or bash -n fail (a tagged commit is what people install; it
    must be at least as good as main)
  - the tag does not match the VERSION= declared by every script
  - CHANGELOG.md has no section for the tag, or the section is empty

That version check is not theoretical: VERSION= had drifted to three different
values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing
noticed until this release. tests/test_release_notes.py now asserts the scripts
agree with each other and with the newest CHANGELOG entry, so the drift cannot
come back.

workflow_dispatch takes an existing tag, so releases can be backfilled for tags
that were pushed before this existed.

106 tests, ruff and shellcheck clean.
2026-07-13 15:05:46 +00:00
12 changed files with 833 additions and 41 deletions
+1 -1
View File
@@ -51,7 +51,7 @@ jobs:
run: python -m pip install --upgrade pip pytest ruff run: python -m pip install --upgrade pip pytest ruff
- name: ruff - name: ruff
run: ruff check patch tests run: ruff check patch tests tools
- name: pytest - name: pytest
run: pytest tests -v run: pytest tests -v
+102
View File
@@ -0,0 +1,102 @@
name: Release
# Push a tag, get a release. The body always comes from CHANGELOG.md, so there is
# no second place to write release notes and therefore no second place for them to
# go stale.
#
# git tag -a v0.4.0 -m "v0.4.0" && git push origin v0.4.0
#
# workflow_dispatch re-cuts (or updates) the release for a tag that already
# exists, since re-pushing an existing tag triggers nothing.
#
# It checks out the TAG, because the tagged code is what people install and it has
# to pass its own tests. That means it only works for tags that actually contain
# this tooling (>= v0.3.0). Tags older than that were backfilled by hand.
on:
push:
tags: ["v*"]
workflow_dispatch:
inputs:
tag:
description: "Existing tag to create a release for (e.g. v0.2.1)"
required: true
type: string
permissions:
contents: write
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Resolve tag
id: tag
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "tag=${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
else
echo "tag=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
fi
- uses: actions/checkout@v4
with:
ref: ${{ steps.tag.outputs.tag }}
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.13"
# Never publish a release for code that does not pass its own tests. A
# tagged commit is what people install; it has to be at least as good as
# main.
- name: install dev deps
run: python -m pip install --upgrade pip pytest ruff
- name: ruff
run: ruff check patch tests tools
- name: pytest
run: pytest tests -q
- name: shell syntax
run: |
fail=0
while IFS= read -r f; do
bash -n "$f" || { echo "::error file=$f::bash syntax error"; fail=1; }
done < <(find . -name '*.sh' -not -path './.git/*')
exit $fail
# Catches the failure mode this repo actually had: VERSION= drifted to
# three different values across the scripts, and nothing noticed.
- name: version matches tag and CHANGELOG has a section
run: python3 tools/release_notes.py check "${{ steps.tag.outputs.tag }}"
- name: extract release notes from CHANGELOG
run: |
python3 tools/release_notes.py notes "${{ steps.tag.outputs.tag }}" > /tmp/notes.md
echo "--- release body ---"
cat /tmp/notes.md
- name: create or update the release
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ steps.tag.outputs.tag }}
run: |
# Pre-1.0 and any -rc/-beta suffix ship as prereleases, not "Latest".
prerelease=""
case "$TAG" in
*-rc*|*-beta*|*-alpha*) prerelease="--prerelease" ;;
esac
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG exists — updating notes."
gh release edit "$TAG" --notes-file /tmp/notes.md
else
# shellcheck disable=SC2086
gh release create "$TAG" \
--title "$TAG" \
--notes-file /tmp/notes.md \
$prerelease
fi
+80
View File
@@ -1,5 +1,85 @@
# Changelog # Changelog
## v0.3.3 — 2026-07-13
### Security
- **The restic repository password no longer passes through a process's argv.**
`create_task.py` shelled out to `midclt call cloud_backup.create '<json>'`, and
that JSON contains the repo password — so it appeared in the process's argv,
which is world-readable via `ps`, for the duration of the call. That password is
the encryption key for the entire cloud backup repository.
It now talks to the middleware through `truenas_api_client` (the library that
backs `midclt` itself), so the password never leaves the process's memory.
- **`--password` no longer required.** Passing a secret as a CLI argument writes it
to shell history permanently. `--password-stdin` reads it from stdin, and with
neither flag the tool prompts via `getpass`. `--password` still works but now
warns.
### Fixed
- **`uninstall.sh` could leave every patch installed.** It reverted by unmounting
the overlay — but `apply.sh` only mounts one when the target directory is
read-only. On a writable `/usr` it patches the real files in place, and uninstall
would remove the boot hook, report success, and leave the patch applied. It now
strips the appended blocks from the middleware files explicitly.
- **`create_task.py.__version__` had been stuck at `0.2.0`** for three releases.
The version-drift check added in v0.3.1 only looked at `VERSION=` in shell
scripts, so it missed the one file that actually shows a version to users
(`--version`). The check now covers `__version__` too — and caught this
immediately.
## v0.3.2 — 2026-07-13
### Fixed
- **`install.sh --disable-nested-snapshots` did not actually disable anything
until the next reboot.** `apply.sh` only ever *added* patches — there was no
revert path. Disabling removed the opt-in marker and then merely *skipped*
re-applying, but the overlay persists for the whole boot, so the previously
patched `plugins/cloud/{snapshot,crud}.py`, `plugins/cloud_backup/sync.py` and
`_truecloud_nested.py` were all still sitting there — and middlewared
re-imported them on the restart `install.sh` performs.
It printed *"DISABLED (stock guard restored)"* while the feature kept running.
Someone turning it off *because they were worried about it* would have believed
it was off.
`apply.sh` now actively reverts: it removes the module first (every injected
block is guarded by `if _tc_nested is not None`, so the stock guard is restored
even if a later step fails), then strips its appended blocks from the three
patched files. `restic.py` also carries a `TRUECLOUD_PATCH` block but belongs to
the *providers* module and is deliberately left alone — reverting it would break
B2 backups. `install.sh --disable` also tears down any staging tree first, since
those bind mounts pin ZFS snapshots that could otherwise never be destroyed.
Updating **without** the flag was always correct and is unchanged: the nested
module is never installed into middleware unless it is explicitly enabled.
## v0.3.1 — 2026-07-13
### Added
- **Automated releases.** Pushing a `v*` tag runs the full test suite and then
cuts a GitHub release whose body is the matching `CHANGELOG.md` section — so
release notes have exactly one source of truth, and no second place to go stale.
The workflow refuses to publish if the tests fail, if the tag does not match the
`VERSION=` declared by every script, or if the CHANGELOG has no section for it.
- **Version-drift check.** `VERSION=` had silently diverged to three different
values across `install.sh`, `uninstall.sh`, `recover.sh`, and `patch/apply.sh`,
and nothing noticed. CI now asserts every script agrees with the others and with
the newest CHANGELOG entry.
### Note
- Releases for `v0.2.0` and `v0.2.1` were backfilled — they had been tagged but
never released, so the releases page jumped v0.1.0 → v0.3.0 and hid the fix for
the boot race that took every app down.
## v0.3.0 — 2026-07-13 ## v0.3.0 — 2026-07-13
### Added ### Added
+10 -3
View File
@@ -18,7 +18,7 @@
set -euo pipefail set -euo pipefail
VERSION="0.3.0" VERSION="0.3.3"
# The directory containing install.sh is the permanent install location. # The directory containing install.sh is the permanent install location.
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)" PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -160,10 +160,17 @@ case "$_nested_choice" in
off) off)
if [ -f "$_NESTED_MARKER" ]; then if [ -f "$_NESTED_MARKER" ]; then
rm -f "$_NESTED_MARKER" rm -f "$_NESTED_MARKER"
echo "Nested-dataset snapshots: DISABLED (stock guard restored)." # Tear down any staging tree first: those bind mounts PIN their ZFS
# snapshots, so leaving them would block those snapshots from ever
# being destroyed. apply.sh (below) then reverts the patched files.
python3 "$PATCH_DIR/patch/truecloud_nested.py" cleanup || \
echo " WARNING: staging mounts remain; unmount them manually."
echo "Nested-dataset snapshots: DISABLED."
echo " apply.sh will revert the patched middleware files and the stock"
echo " guard is restored when middlewared restarts (this script does that)."
echo " Any task that already has snapshot=true on a nested dataset will" echo " Any task that already has snapshot=true on a nested dataset will"
echo " fail validation on its next edit. Turn the option off on those" echo " fail validation on its next edit. Turn the option off on those"
echo " tasks, or re-run with --enable-nested-snapshots." echo " tasks first, or re-run with --enable-nested-snapshots."
else else
echo "Nested-dataset snapshots: already disabled." echo "Nested-dataset snapshots: already disabled."
fi fi
+74 -11
View File
@@ -32,7 +32,7 @@
# Derive PATCH_DIR from this script's location (parent of the patch/ directory). # Derive PATCH_DIR from this script's location (parent of the patch/ directory).
PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)" PATCH_DIR="$(cd "$(dirname "$0")/.." && pwd)"
LOG="$PATCH_DIR/apply.log" LOG="$PATCH_DIR/apply.log"
VERSION="0.3.0" VERSION="0.3.3"
# Rotate log at 512 KB to avoid unbounded growth on a system volume. # Rotate log at 512 KB to avoid unbounded growth on a system volume.
# Keep two prior generations (.1 and .2) so the last three boots are always available. # Keep two prior generations (.1 and .2) so the last three boots are always available.
@@ -477,6 +477,57 @@ def patch_file(path, block):
with open(path, "w", encoding="utf-8") as fh: with open(path, "w", encoding="utf-8") as fh:
fh.write(base.rstrip("\n") + "\n" + block) fh.write(base.rstrip("\n") + "\n" + block)
def unpatch_file(path):
"""Strip our appended block, restoring the stock file. True if it was patched."""
try:
with open(path, encoding="utf-8") as fh:
content = fh.read()
except OSError:
return False
idx = content.find("\n# TRUECLOUD_PATCH")
if idx == -1:
return False
try:
with open(path, "w", encoding="utf-8") as fh:
fh.write(content[:idx].rstrip("\n") + "\n")
except OSError:
return False
return True
def revert_nested(cloud_dir, sync_path):
"""Undo the nested patch. Returns the names of what was actually reverted.
Skipping the patch is NOT enough to disable the feature. The overlay persists
for the whole boot, so an earlier run this boot may already have written the
patched files -- and middlewared re-imports them on the restart that
install.sh performs. Without this, `install.sh --disable-nested-snapshots`
would report "disabled" while the feature kept running until the next reboot.
"""
reverted = []
# Remove the module FIRST. Every injected block is guarded by
# `if _tc_nested is not None`, so once it is gone they all no-op even if a
# later step here fails -- the guard is restored no matter what.
try:
os.unlink(os.path.join(cloud_dir, '_truecloud_nested.py'))
reverted.append('_truecloud_nested.py')
except OSError:
pass
# NB: restic.py also carries a TRUECLOUD_PATCH block, but that belongs to the
# providers module. Only these three are ours to revert.
for name, path in (
('crud.py', os.path.join(cloud_dir, 'crud.py')),
('sync.py', sync_path),
('snapshot.py', os.path.join(cloud_dir, 'snapshot.py')),
):
if unpatch_file(path):
reverted.append(name)
return reverted
b2_ok = restic_ok = False b2_ok = restic_ok = False
nested_ok = False nested_ok = False
@@ -517,16 +568,28 @@ else:
# guard LAST. If anything fails partway, the guard is still in place and the # guard LAST. If anything fails partway, the guard is still in place and the
# option stays unavailable -- we never expose "guard removed, traversal missing". # option stays unavailable -- we never expose "guard removed, traversal missing".
nested_detail = '' nested_detail = ''
if not nested_enabled: if not nested_needed:
nested_detail = 'disabled (opt-in; enable with: install.sh --enable-nested-snapshots)' # Not just "skip": actively revert. The overlay lives for the whole boot, so a
print('INFO: Nested-dataset snapshot support is disabled (opt-in feature).') # previously-applied patch is still sitting there and middlewared would
print('INFO: Enable with: bash install.sh --enable-nested-snapshots') # re-import it on restart. See revert_nested().
elif nested_native: if not nested_enabled:
nested_detail = 'superseded: TrueNAS handles nested-dataset snapshots natively' nested_detail = 'disabled (opt-in; enable with: install.sh --enable-nested-snapshots)'
print('INFO: Nested module skipped — TrueNAS now handles nesting natively.') print('INFO: Nested-dataset snapshot support is disabled (opt-in feature).')
elif not nested_needed: elif nested_native:
nested_detail = 'not needed' nested_detail = 'superseded: TrueNAS handles nested-dataset snapshots natively'
print('INFO: Nested module skipped.') print('INFO: Nested module skipped — TrueNAS now handles nesting natively.')
else:
nested_detail = 'not needed'
print('INFO: Nested module skipped.')
reverted = revert_nested(cloud_dir, sync_path)
if reverted:
print('OK: Reverted a previously-applied nested patch (' + ', '.join(reverted) + ').')
print(' The stock nesting guard is restored once middlewared restarts.')
nested_detail += ' — previous patch reverted'
if not nested_enabled:
print('INFO: Enable with: bash install.sh --enable-nested-snapshots')
else: else:
try: try:
snapshot_py = os.path.join(cloud_dir, 'snapshot.py') snapshot_py = os.path.join(cloud_dir, 'snapshot.py')
+70 -23
View File
@@ -26,8 +26,9 @@ Create a task backed by a B2 credential (id=3):
--credential 3 \\ --credential 3 \\
--bucket my-bucket \\ --bucket my-bucket \\
--folder backups/tank \\ --folder backups/tank \\
--password "restic-repo-password" \\ --password-stdin \\
--keep-last 14 --keep-last 14
(pipe the password in: echo -n "s3cret" | python3 create_task.py create ... )
Create a task using an S3-compatible credential (Wasabi, R2, etc.): Create a task using an S3-compatible credential (Wasabi, R2, etc.):
python3 create_task.py create \\ python3 create_task.py create \\
@@ -36,7 +37,7 @@ Create a task using an S3-compatible credential (Wasabi, R2, etc.):
--credential 5 \\ --credential 5 \\
--bucket my-bucket \\ --bucket my-bucket \\
--folder backups \\ --folder backups \\
--password "restic-repo-password" --password-stdin
List existing TrueCloud Backup tasks: List existing TrueCloud Backup tasks:
python3 create_task.py list-tasks python3 create_task.py list-tasks
@@ -44,39 +45,49 @@ List existing TrueCloud Backup tasks:
import argparse import argparse
import calendar import calendar
import getpass
import json import json
import os import os
import subprocess import subprocess
import sys import sys
import time import time
__version__ = "0.2.0" __version__ = "0.3.3"
_PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
_STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json") _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")
def midclt_call(method, *args): def midclt_call(method, *args):
"""Call a middleware method locally via `midclt`, the supported JSON-RPC transport """Call a middleware method on the local host.
that replaces the deprecated /api/v2.0 REST API (removed in TrueNAS 26.04). Must run
on the TrueNAS host. Each arg is JSON-encoded (a dict for create; none for queries). Uses `truenas_api_client` -- the library that backs `midclt` itself -- rather
Exits with a clear message on failure.""" than shelling out to `midclt`.
cmd = ["midclt", "call", method] + [json.dumps(a) for a in args]
This is a SECURITY requirement, not a style choice. `midclt call <method>
<json>` puts its arguments in the process's **argv**, and `cloud_backup.create`
carries the restic repository password. argv is world-readable via `ps`, so
shelling out would expose the key to the entire backup repo to every local
user for the duration of the call. Going through the client library keeps it
in this process's memory.
"""
try: try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120) from truenas_api_client import Client
except FileNotFoundError: except ImportError:
print("ERROR: `midclt` not found — run this script ON the TrueNAS host.", print(
file=sys.stderr) "ERROR: `truenas_api_client` not importable — run this script ON the\n"
" TrueNAS host. (It ships with midclt.)",
file=sys.stderr,
)
sys.exit(1) sys.exit(1)
except subprocess.SubprocessError as exc:
print(f"ERROR: midclt call failed: {exc}", file=sys.stderr) try:
with Client() as client:
return client.call(method, *args)
except Exception as exc: # noqa: BLE001 - surface any middleware error verbatim
# Never echo `args` here: for cloud_backup.create it contains the password.
print(f"ERROR: {method}: {exc}", file=sys.stderr)
sys.exit(1) sys.exit(1)
if proc.returncode != 0:
print(f"ERROR: midclt {method}: {(proc.stderr or proc.stdout).strip()}",
file=sys.stderr)
sys.exit(1)
out = proc.stdout.strip()
return json.loads(out) if out else None
# ── Sub-commands ────────────────────────────────────────────────────────────── # ── Sub-commands ──────────────────────────────────────────────────────────────
@@ -213,6 +224,37 @@ def cmd_list_tasks(_args):
print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t.get('description', '')}") print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t.get('description', '')}")
def _resolve_password(args):
"""Get the restic repo password without writing it to the user's shell history.
That password is the key to the whole backup repository. `--password <secret>`
persists it in ~/.bash_history and exposes it in `ps` for the lifetime of the
shell command, so it is accepted but warned about; stdin and an interactive
prompt are the safe paths.
"""
if args.password_stdin:
if args.password:
print("ERROR: use either --password or --password-stdin, not both.",
file=sys.stderr)
sys.exit(1)
password = sys.stdin.readline().rstrip("\n")
elif args.password:
print(
"WARNING: --password puts the restic repository password in your shell\n"
" history. Prefer: echo -n 'pw' | ... --password-stdin",
file=sys.stderr,
)
password = args.password
else:
password = getpass.getpass("Restic repository password: ")
if not password:
print("ERROR: the restic repository password must not be empty.",
file=sys.stderr)
sys.exit(1)
return password
def cmd_create(args): def cmd_create(args):
parts = args.schedule.split() parts = args.schedule.split()
if len(parts) != 5: if len(parts) != 5:
@@ -223,6 +265,8 @@ def cmd_create(args):
sys.exit(1) sys.exit(1)
minute, hour, dom, month, dow = parts minute, hour, dom, month, dow = parts
password = _resolve_password(args)
body = { body = {
"description": args.name, "description": args.name,
"path": args.path, "path": args.path,
@@ -231,7 +275,7 @@ def cmd_create(args):
"bucket": args.bucket, "bucket": args.bucket,
"folder": args.folder, "folder": args.folder,
}, },
"password": args.password, "password": password,
"keep_last": args.keep_last, "keep_last": args.keep_last,
"transfer_setting": args.transfer_setting, "transfer_setting": args.transfer_setting,
"schedule": { "schedule": {
@@ -297,8 +341,11 @@ def main():
help="Bucket (S3) or container (B2) name") help="Bucket (S3) or container (B2) name")
c.add_argument("--folder", default="", c.add_argument("--folder", default="",
help="Path within the bucket (default: root)") help="Path within the bucket (default: root)")
c.add_argument("--password", required=True, c.add_argument("--password", default=None,
help="Restic repository encryption password (choose a strong one)") help="Restic repository password. UNSAFE: it lands in your shell "
"history. Prefer --password-stdin, or omit both and be prompted.")
c.add_argument("--password-stdin", action="store_true",
help="Read the restic repository password from stdin (recommended)")
c.add_argument("--keep-last", type=int, default=14, metavar="N", c.add_argument("--keep-last", type=int, default=14, metavar="N",
help="Snapshots to retain after each run (default: 14)") help="Snapshots to retain after each run (default: 14)")
c.add_argument("--schedule", default="0 2 * * *", c.add_argument("--schedule", default="0 2 * * *",
+1 -1
View File
@@ -17,7 +17,7 @@
# bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh # bash /mnt/tank/truenas-truecloud-patch/patch/apply.sh
# systemctl restart middlewared # systemctl restart middlewared
VERSION="0.3.0" VERSION="0.3.3"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)" PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
+48 -1
View File
@@ -263,10 +263,57 @@ class TestOptIn:
def test_patching_is_skipped_entirely_when_disabled(self): def test_patching_is_skipped_entirely_when_disabled(self):
# The guard-relaxing crud.py patch must be inside the enabled branch. # The guard-relaxing crud.py patch must be inside the enabled branch.
src = heredoc_source() src = heredoc_source()
gate = src.index("if not nested_enabled:") gate = src.index("if not nested_needed:")
crud = src.index("patch_file(crud_py, CRUD_BLOCK)") crud = src.index("patch_file(crud_py, CRUD_BLOCK)")
assert gate < crud, "crud.py patch must sit inside the opt-in branch" assert gate < crud, "crud.py patch must sit inside the opt-in branch"
def test_disabling_REVERTS_the_patch_rather_than_merely_skipping_it(self):
"""Skipping is not disabling.
The overlay persists for the whole boot, so a patch applied by an earlier
run this boot is still on disk — and middlewared re-imports it on the
restart install.sh performs. Without an active revert,
`--disable-nested-snapshots` reports "disabled" while the feature keeps
running until the next reboot.
"""
src = heredoc_source()
assert "def unpatch_file(" in src
assert "def revert_nested(" in src
# The revert must run on every not-needed path (opt-out, superseded).
gate = src.index("if not nested_needed:")
revert = src.index("reverted = revert_nested(")
patch = src.index("patch_file(crud_py, CRUD_BLOCK)")
assert gate < revert < patch, "revert belongs in the not-needed branch"
def test_revert_removes_the_module_before_unpatching_files(self):
# Every injected block is guarded by `if _tc_nested is not None`, so
# deleting the module first means the guard is restored even if a later
# unpatch step fails.
src = heredoc_source()
body = src[src.index("def revert_nested("):src.index("def patch_file(") if
src.index("def patch_file(") > src.index("def revert_nested(") else len(src)]
body = src[src.index("def revert_nested("):]
body = body[:body.index("\n\n\n")] if "\n\n\n" in body else body
assert body.index("_truecloud_nested.py") < body.index("crud.py")
def test_revert_never_touches_the_providers_patch(self):
# restic.py also carries a TRUECLOUD_PATCH block, but it belongs to the
# providers module. Reverting it would silently break B2 backups.
src = heredoc_source()
body = src[src.index("def revert_nested("):]
body = body[:body.index("return reverted")]
# Comments legitimately *mention* restic.py to explain why it is excluded;
# what matters is that no code line touches it.
code = "\n".join(
ln for ln in body.splitlines() if not ln.lstrip().startswith("#")
)
assert "restic" not in code
assert "b2.py" not in code
# It must only ever revert these three, plus the module itself.
assert "crud.py" in code
assert "sync_path" in code
assert "snapshot.py" in code
def test_guard_is_relaxed_only_after_traversal_is_installed(): def test_guard_is_relaxed_only_after_traversal_is_installed():
# Ordering in apply.sh is a safety property: copy module -> patch snapshot.py # Ordering in apply.sh is a safety property: copy module -> patch snapshot.py
+107
View File
@@ -0,0 +1,107 @@
"""Tests for create_task.py, focused on the restic repository password.
That password is the encryption key for the entire cloud backup repository. It
used to travel through `midclt call cloud_backup.create '<json>'` -- i.e. through
the subprocess's **argv**, which is world-readable via `ps` -- and `--password`
wrote it into the user's shell history forever.
"""
import importlib.util
import io
import os
import sys
import types
import pytest
SPEC = importlib.util.spec_from_file_location(
"create_task",
os.path.join(os.path.dirname(__file__), "..", "patch", "create_task.py"),
)
def load():
mod = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(mod)
return mod
class Args:
def __init__(self, password=None, password_stdin=False):
self.password = password
self.password_stdin = password_stdin
class TestPasswordNeverReachesArgv:
"""The whole reason this module talks to the client library."""
def test_midclt_call_spawns_no_subprocess(self):
import inspect
src = inspect.getsource(load().midclt_call)
assert "subprocess" not in src, (
"shelling out to `midclt` puts cloud_backup.create's JSON -- including "
"the restic repo password -- into argv, which any local user can read "
"with ps"
)
assert "truenas_api_client" in src
def test_errors_never_echo_the_call_arguments(self):
# A failed cloud_backup.create must not print the body back at the user;
# it contains the password.
import inspect
src = inspect.getsource(load().midclt_call)
assert "{args}" not in src
assert "args!r" not in src
class TestResolvePassword:
def test_reads_from_stdin(self, monkeypatch):
mod = load()
monkeypatch.setattr(sys, "stdin", io.StringIO("s3cret\n"))
assert mod._resolve_password(Args(password_stdin=True)) == "s3cret"
def test_strips_only_the_trailing_newline(self, monkeypatch):
# A password may legitimately contain spaces; only the line ending goes.
mod = load()
monkeypatch.setattr(sys, "stdin", io.StringIO(" pass word \n"))
assert mod._resolve_password(Args(password_stdin=True)) == " pass word "
def test_cli_password_still_works_but_warns(self, monkeypatch, capsys):
mod = load()
pw = mod._resolve_password(Args(password="cli-secret"))
assert pw == "cli-secret"
assert "shell" in capsys.readouterr().err.lower(), "must warn about history"
def test_prompts_when_neither_flag_given(self, monkeypatch):
mod = load()
monkeypatch.setattr(
mod, "getpass", types.SimpleNamespace(getpass=lambda _p: "prompted")
)
assert mod._resolve_password(Args()) == "prompted"
def test_rejects_both_flags(self, monkeypatch):
mod = load()
monkeypatch.setattr(sys, "stdin", io.StringIO("x\n"))
with pytest.raises(SystemExit):
mod._resolve_password(Args(password="a", password_stdin=True))
def test_rejects_an_empty_password(self, monkeypatch):
# An empty restic password would silently create an unencrypted-ish repo.
mod = load()
monkeypatch.setattr(sys, "stdin", io.StringIO("\n"))
with pytest.raises(SystemExit):
mod._resolve_password(Args(password_stdin=True))
class TestVersion:
def test_version_is_not_stale(self):
# __version__ sat at 0.2.0 through three releases because the drift check
# only looked at VERSION= in shell scripts. It covers this file now.
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
from release_notes import normalise, script_versions
repo = os.path.join(os.path.dirname(__file__), "..")
versions = {normalise(v) for v in script_versions(repo).values()}
assert len(versions) == 1, f"version drift: {sorted(versions)}"
+122
View File
@@ -0,0 +1,122 @@
"""Tests for the release automation.
The release workflow refuses to publish unless these hold, so a bad tag fails
loudly in CI instead of shipping a release whose notes are empty, wrong, or whose
scripts announce a different version than the tag.
That last one is not hypothetical: VERSION= drifted to three different values
across install.sh / uninstall.sh / recover.sh / apply.sh and nothing noticed.
"""
import os
import sys
import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
from release_notes import ( # noqa: E402
changelog_versions,
check,
extract_notes,
normalise,
script_versions,
)
REPO = os.path.join(os.path.dirname(__file__), "..")
SAMPLE = """\
# Changelog
## v0.3.0 — 2026-07-13
### Added
- the new thing
## v0.2.1 — 2026-07-09
### Fixed
- the old thing
## v0.2.0 — 2026-07-08
- first
"""
class TestExtractNotes:
def test_returns_only_that_versions_body(self):
body = extract_notes(SAMPLE, "v0.3.0")
assert "the new thing" in body
assert "the old thing" not in body
# The version heading itself is dropped (GitHub renders its own title),
# but sub-headings like "### Added" must survive.
assert not body.startswith("## v")
assert body.startswith("### Added")
def test_stops_at_the_next_version_heading(self):
body = extract_notes(SAMPLE, "v0.2.1")
assert "the old thing" in body
assert "first" not in body
def test_last_section_runs_to_end_of_file(self):
assert "first" in extract_notes(SAMPLE, "v0.2.0")
def test_accepts_the_tag_with_or_without_the_v(self):
assert extract_notes(SAMPLE, "0.3.0") == extract_notes(SAMPLE, "v0.3.0")
def test_unknown_version_raises_rather_than_returning_empty(self):
# An empty release body is worse than a failed release.
with pytest.raises(KeyError, match="no section"):
extract_notes(SAMPLE, "v9.9.9")
class TestChangelogVersions:
def test_lists_versions_newest_first(self):
assert changelog_versions(SAMPLE) == ["0.3.0", "0.2.1", "0.2.0"]
class TestAgainstTheRealRepo:
"""These run against the actual files, so drift breaks the build."""
def test_every_script_declares_a_version(self):
from release_notes import VERSIONED_FILES
found = script_versions(REPO)
missing = [f for f in VERSIONED_FILES if f not in found]
assert not missing, f"no VERSION= in: {missing}"
def test_all_scripts_agree_on_the_version(self):
versions = {normalise(v) for v in script_versions(REPO).values()}
assert len(versions) == 1, f"scripts disagree on version: {sorted(versions)}"
def test_the_current_version_has_a_changelog_section(self):
version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
body = extract_notes(fh.read(), version)
assert body, f"CHANGELOG.md has no content for v{version}"
def test_the_current_version_is_the_newest_changelog_entry(self):
version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
with open(os.path.join(REPO, "CHANGELOG.md"), encoding="utf-8") as fh:
newest = changelog_versions(fh.read())[0]
assert newest == version, (
f"scripts say v{version} but the newest CHANGELOG entry is v{newest}"
)
def test_check_passes_for_the_current_version(self):
version = next(iter({normalise(v) for v in script_versions(REPO).values()}))
assert check(version, REPO) == []
class TestCheckCatchesMistakes:
def test_reports_a_tag_that_no_script_matches(self):
problems = check("v9.9.9", REPO)
assert problems
assert any("declares VERSION" in p for p in problems)
def test_reports_a_missing_changelog_section(self):
problems = check("v9.9.9", REPO)
assert any("no section" in p for p in problems)
+160
View File
@@ -0,0 +1,160 @@
#!/usr/bin/env python3
"""Extract one version's section from CHANGELOG.md, and check version consistency.
Used by .github/workflows/release.yml so a release's body is always the changelog
entry -- there is no second place to write release notes, and therefore no second
place for them to be wrong.
python3 tools/release_notes.py notes v0.3.0 # -> the section body
python3 tools/release_notes.py version # -> version per the scripts
python3 tools/release_notes.py check v0.3.0 # -> exit 1 on any mismatch
"""
from __future__ import annotations
import os
import re
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CHANGELOG = os.path.join(ROOT, "CHANGELOG.md")
# Everything that announces a version must agree with everything else. They drifted
# to three different values once (0.0.4 / 0.2.1) before anything checked them --
# and create_task.py's __version__ then sat at 0.2.0 through three more releases,
# because the first version of this check only looked at VERSION= in shell scripts.
VERSIONED_FILES = [
"install.sh",
"uninstall.sh",
"recover.sh",
os.path.join("patch", "apply.sh"),
os.path.join("patch", "create_task.py"), # exposes `--version` to users
]
# `VERSION="x"` (shell) or `__version__ = "x"` (python).
_VERSION_RE = re.compile(r'^(?:VERSION=|__version__\s*=\s*)"([^"]+)"', re.M)
_HEADING_RE = re.compile(r"^##\s+v?(\d+\.\d+\.\d+[^\s]*)", re.M)
def normalise(v: str) -> str:
return v.strip().lstrip("v")
def script_versions(root: str = ROOT) -> dict[str, str]:
"""VERSION= as declared by each script."""
found = {}
for rel in VERSIONED_FILES:
path = os.path.join(root, rel)
try:
with open(path, encoding="utf-8") as fh:
m = _VERSION_RE.search(fh.read())
except OSError:
continue
if m:
found[rel] = m.group(1)
return found
def changelog_versions(text: str) -> list[str]:
"""Versions with a section in the changelog, newest first."""
return [normalise(v) for v in _HEADING_RE.findall(text)]
def extract_notes(text: str, version: str) -> str:
"""The body of one version's section, without its heading.
Raises KeyError if the version has no section -- a release with an empty or
wrong body is worse than a failed release.
"""
want = normalise(version)
lines = text.splitlines()
start = None
for i, line in enumerate(lines):
m = _HEADING_RE.match(line)
if m and normalise(m.group(1)) == want:
start = i + 1
break
if start is None:
raise KeyError(f"CHANGELOG.md has no section for v{want}")
end = len(lines)
for i in range(start, len(lines)):
if _HEADING_RE.match(lines[i]):
end = i
break
return "\n".join(lines[start:end]).strip()
def check(version: str, root: str = ROOT) -> list[str]:
"""Every reason this version is not releasable. Empty list means it is."""
want = normalise(version)
problems = []
versions = script_versions(root)
for rel, got in sorted(versions.items()):
if normalise(got) != want:
problems.append(f"{rel} declares VERSION={got!r}, tag is v{want}")
missing = [r for r in VERSIONED_FILES if r not in versions]
for rel in missing:
problems.append(f"{rel} has no VERSION= line")
try:
with open(os.path.join(root, "CHANGELOG.md"), encoding="utf-8") as fh:
text = fh.read()
except OSError as e:
problems.append(f"cannot read CHANGELOG.md: {e}")
return problems
try:
body = extract_notes(text, want)
except KeyError as e:
problems.append(str(e))
else:
if not body:
problems.append(f"CHANGELOG.md section for v{want} is empty")
return problems
def main(argv):
if len(argv) < 2:
print(__doc__, file=sys.stderr)
return 2
cmd = argv[1]
if cmd == "version":
versions = set(map(normalise, script_versions().values()))
if len(versions) != 1:
print(f"scripts disagree on version: {sorted(versions)}", file=sys.stderr)
return 1
print(versions.pop())
return 0
if len(argv) < 3:
print(f"usage: {argv[0]} {cmd} <version>", file=sys.stderr)
return 2
version = argv[2]
if cmd == "notes":
with open(CHANGELOG, encoding="utf-8") as fh:
print(extract_notes(fh.read(), version))
return 0
if cmd == "check":
problems = check(version)
for p in problems:
print(f"::error::{p}")
if problems:
return 1
print(f"v{normalise(version)} is consistent across scripts and CHANGELOG")
return 0
print(f"unknown command: {cmd}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main(sys.argv))
+58 -1
View File
@@ -3,7 +3,7 @@
set -euo pipefail set -euo pipefail
VERSION="0.3.0" VERSION="0.3.3"
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)" PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
_HOOK_COMMENT='TrueCloud provider patch (S3/B2)' _HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
@@ -91,6 +91,63 @@ if [ "$_ov_found" -eq 0 ]; then
fi fi
echo "" echo ""
# ── Revert file-level patches ─────────────────────────────────────────────────
# Unmounting the overlay is what normally reverts everything — the lower layer is
# the untouched /usr. But apply.sh only mounts an overlay when the directory is
# read-only; on a writable /usr it patches the real files in place. Uninstall
# would then remove the boot hook and report success while leaving every patch
# applied. Strip our appended blocks explicitly.
echo "Reverting any file-level patches ..."
python3 - <<'PYEOF'
import os
try:
import middlewared
except ImportError:
print(" middlewared not importable — nothing to revert.")
raise SystemExit(0)
mw = os.path.dirname(os.path.abspath(middlewared.__file__))
targets = [
os.path.join(mw, "rclone", "remote", "b2.py"),
os.path.join(mw, "plugins", "cloud_backup", "restic.py"),
os.path.join(mw, "plugins", "cloud_backup", "sync.py"),
os.path.join(mw, "plugins", "cloud", "crud.py"),
os.path.join(mw, "plugins", "cloud", "snapshot.py"),
]
reverted = []
for path in targets:
try:
with open(path, encoding="utf-8") as fh:
content = fh.read()
except OSError:
continue
idx = content.find("\n# TRUECLOUD_PATCH")
if idx == -1:
continue
try:
with open(path, "w", encoding="utf-8") as fh:
fh.write(content[:idx].rstrip("\n") + "\n")
reverted.append(os.path.basename(path))
except OSError as e:
print(f" WARNING: could not revert {path}: {e}")
nested = os.path.join(mw, "plugins", "cloud", "_truecloud_nested.py")
try:
os.unlink(nested)
reverted.append("_truecloud_nested.py")
except OSError:
pass
if reverted:
print(" Reverted: " + ", ".join(reverted))
else:
print(" Nothing to revert (overlay already removed, or never patched).")
PYEOF
echo ""
# ── Unmount nested-snapshot staging trees ───────────────────────────────────── # ── Unmount nested-snapshot staging trees ─────────────────────────────────────
# These bind mounts pin their ZFS snapshots, so they must go before anything # These bind mounts pin their ZFS snapshots, so they must go before anything
# tries to destroy those snapshots. Deepest first. # tries to destroy those snapshots. Deepest first.