Two CI failures shared one cause: four jobs starting together on the
self-hosted runner.
act caches each action as one shared clone under /root/.cache/act/<hash> and
re-pulls it per job, so concurrent jobs fight over that directory and the
loser dies with "lstat .../<file>: no such file or directory" before any test
runs — a different victim each push. And the runner force-pulls its base image
per job, so four jobs meant four anonymous Docker Hub pulls per push; that hit
429 Too Many Requests and every job started failing before it began, including
the shell job nothing had touched.
Installing uv without an action only shrank the surface, since every job still
used actions/checkout. Concurrency is the ingredient, so this removes it: one
job cannot race itself whatever actions it uses, and one job is one pull. The
version sweep moves inside the job and still runs every version after one
fails, preserving what fail-fast: false bought.
act caches each action as one shared git clone under /root/.cache/act/<hash>
and re-pulls it per job. The three python matrix jobs start within the same
second on the self-hosted runner, race on that directory, and the loser dies
with "lstat /root/.cache/act/<hash>/.npmrc: no such file or directory" before
any test runs — a red main with zero suite output and a different victim each
push (3.12, then 3.11).
The action was only fetching a binary; the matrix interpreter is selected per
command by uvx --python. A run: step has no action-cache entry and cannot
race, and keeps the jobs parallel — serialising the matrix would cost 3x the
wall clock and still leave actions/checkout shared across four jobs. The uv
version is pinned under the same rule as ruff.
The accompanying test parses uses: directives rather than the raw text, so the
comment can still name the action it avoids, and uses re instead of PyYAML
because CI runs uvx pytest, whose environment holds pytest and nothing else.
Releases were manual and had drifted: v0.2.0 and v0.2.1 were tagged but never
released, so the releases page jumped v0.1.0 -> v0.3.0 and hid the fix for the
incident that took every app down.
Pushing a v* tag now runs the full suite and cuts a GitHub release whose body is
the matching CHANGELOG.md section -- one source of truth for release notes, so
there is no second place for them to be wrong.
The workflow refuses to publish when:
- the tests, ruff, or bash -n fail (a tagged commit is what people install; it
must be at least as good as main)
- the tag does not match the VERSION= declared by every script
- CHANGELOG.md has no section for the tag, or the section is empty
That version check is not theoretical: VERSION= had drifted to three different
values across install.sh / uninstall.sh / recover.sh / apply.sh and nothing
noticed until this release. tests/test_release_notes.py now asserts the scripts
agree with each other and with the newest CHANGELOG entry, so the drift cannot
come back.
workflow_dispatch takes an existing tag, so releases can be backfilled for tags
that were pushed before this existed.
106 tests, ruff and shellcheck clean.
Opt-in
------
Nested-dataset snapshot support changes how backups read their source data, so
it is now off by default and gated behind a marker file:
install.sh --enable-nested-snapshots
install.sh --disable-nested-snapshots
With neither flag install.sh preserves the current setting, so a routine
`git pull && bash install.sh` can never silently flip it. When disabled,
apply.sh skips the patch entirely and the stock guard remains. uninstall.sh
tears down staging mounts and removes the marker.
Snapshot lifecycle
------------------
zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls
it with no options. Stock is safe only because its validation means recursive is
never True in the field. Enabling nested datasets makes recursive snapshots real:
the parent then has one child snapshot per descendant dataset (160+ on an Apps
pool), so stock's delete would orphan every child on EVERY successful run.
The patch now owns the lifecycle end to end:
- delete_snapshot_tree() sweeps the parent and all children, and is idempotent
against stock's finally winning the race once our mounts are released
- on a staging failure the tree is deleted here, because sync.py never completes
`snapshot, local_path = await create_snapshot(...)` and so its finally deletes
nothing at all
- the snapshot is recorded in a sidecar file before anything is mounted, so a
middlewared restart mid-backup cannot orphan it
- a crashed run's snapshot tree is reclaimed on the next run instead of being
overwritten and leaked
Silent-omission fix
-------------------
The dataset list is now enumerated AFTER the snapshot. Read beforehand it could
miss a dataset created in the gap, which the recursive snapshot would capture but
the staging plan would not -- silently omitting its data. Read afterwards, an
unsnapshotted dataset trips the staging check and fails the run loudly.
Also from the audit
-------------------
- plan_staging scopes by dataset name, so skipped-dataset warnings no longer
include every mountpoint-less dataset on the box, which buried the ones that
matter
- staging_root_for rejects "." / ".." components that would escape the staging
base, and resolves STAGING_BASE at call time rather than freezing it into a
default argument
- uninstall.sh no longer `rm -rf`s a tree that may still contain live bind
mounts, and unmounts by path depth rather than string length
- apply_plan takes an injectable isdir; verify_staged drops an unused parameter
- pin the shellcheck action instead of tracking @master
61 tests, ruff and shellcheck clean.
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:
This option is only available for datasets that have no further nesting
That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.
The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.
Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.
Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays
The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.
Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.
Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
flan
flan