flan/truenas-truecloud-patch
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases 12 Wiki Activity

Make nested snapshots opt-in; fix snapshot leaks found in audit

Browse Source 
Opt-in
------
Nested-dataset snapshot support changes how backups read their source data, so
it is now off by default and gated behind a marker file:

  install.sh --enable-nested-snapshots
  install.sh --disable-nested-snapshots

With neither flag install.sh preserves the current setting, so a routine
`git pull && bash install.sh` can never silently flip it. When disabled,
apply.sh skips the patch entirely and the stock guard remains. uninstall.sh
tears down staging mounts and removes the marker.

Snapshot lifecycle
------------------
zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls
it with no options. Stock is safe only because its validation means recursive is
never True in the field. Enabling nested datasets makes recursive snapshots real:
the parent then has one child snapshot per descendant dataset (160+ on an Apps
pool), so stock's delete would orphan every child on EVERY successful run.

The patch now owns the lifecycle end to end:

- delete_snapshot_tree() sweeps the parent and all children, and is idempotent
  against stock's finally winning the race once our mounts are released
- on a staging failure the tree is deleted here, because sync.py never completes
  `snapshot, local_path = await create_snapshot(...)` and so its finally deletes
  nothing at all
- the snapshot is recorded in a sidecar file before anything is mounted, so a
  middlewared restart mid-backup cannot orphan it
- a crashed run's snapshot tree is reclaimed on the next run instead of being
  overwritten and leaked

Silent-omission fix
-------------------
The dataset list is now enumerated AFTER the snapshot. Read beforehand it could
miss a dataset created in the gap, which the recursive snapshot would capture but
the staging plan would not -- silently omitting its data. Read afterwards, an
unsnapshotted dataset trips the staging check and fails the run loudly.

Also from the audit
-------------------
- plan_staging scopes by dataset name, so skipped-dataset warnings no longer
  include every mountpoint-less dataset on the box, which buried the ones that
  matter
- staging_root_for rejects "." / ".." components that would escape the staging
  base, and resolves STAGING_BASE at call time rather than freezing it into a
  default argument
- uninstall.sh no longer `rm -rf`s a tree that may still contain live bind
  mounts, and unmounts by path depth rather than string length
- apply_plan takes an injectable isdir; verify_staged drops an unused parameter
- pin the shellcheck action instead of tracking @master

61 tests, ruff and shellcheck clean.
This commit is contained in:
flan
2026-07-12 21:52:09 +00:00
parent a572eb2164
commit bb26edf351
10 changed files with 785 additions and 173 deletions
Download Patch File Download Diff File Expand all files Collapse all files
patch/truecloud_nested.py
+212 -62
View File
@@ -1,4 +1,4 @@
"""Nested-dataset snapshot support for TrueCloud Backup / Cloud Sync.
"""Nested-dataset snapshot support for TrueCloud Backup.
Why this exists
---------------
@@ -12,8 +12,8 @@ already takes a **recursive** ZFS snapshot, but it then points the backup tool
at the *parent* dataset's ``.zfs/snapshot/<snap>/`` directory -- and ZFS does
not expose child datasets through a parent's snapshot directory::
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries (children invisible)
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
/mnt/Tap/.zfs/snapshot/<snap>/apps/ -> 0 entries
/mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/ -> the real data
So without the guard, the backup tool would walk a near-empty tree, report
SUCCESS, and upload almost nothing. A backup that lies about succeeding is the
@@ -32,19 +32,24 @@ Cardinal safety rule
tree. Silently backing up an incomplete tree is precisely the failure this
feature exists to prevent, and it would be worse than not having the feature.
Notes
-----
* ZFS snapshots are immutable, so a plain ``mount --bind`` is inherently
read-only; no remount dance is needed.
* Bind-mounting ``.zfs/snapshot/<snap>`` pins the snapshot, so ``zfs destroy``
of that snapshot returns EBUSY until we unmount. Stock ``restic_backup()``
deletes the snapshot in its ``finally``, which therefore logs one benign
"Error deleting snapshot ... busy" warning; :func:`cleanup_task` then unmounts
and deletes the snapshot for real. See ``patch/apply.sh``.
* Staging roots live under a stable, per-task path so that the backup tool sees
the *same* path every run. Stock's ``.zfs/snapshot/<name>-<timestamp>/`` path
changes every run, which defeats restic's parent-snapshot detection; the
staging tree is an improvement on that.
Snapshot lifecycle -- read this before changing anything
--------------------------------------------------------
``zfs.snapshot.delete`` defaults to ``recursive=False``, and stock
``restic_backup()`` calls it with no options. Stock gets away with that because
its validation means ``recursive`` is never actually True in the field. Enabling
nested datasets makes recursive snapshots real, so the parent
(``Tap@snap``) has one child snapshot per descendant dataset (160+ here).
Deleting only the parent would orphan every child on **every successful run**.
Therefore this module owns the whole lifecycle:
* :func:`delete_snapshot_tree` sweeps the parent *and* every child snapshot, and
is idempotent -- it copes with stock's ``finally`` having already removed the
parent.
* The snapshot name is recorded in a sidecar file next to the staging root, not
only in memory, so a middlewared restart mid-backup cannot orphan it.
* Bind-mounting ``.zfs/snapshot/<snap>`` pins the snapshot, so stock's delete
fails with EBUSY and logs one benign warning; we unmount and then sweep.
"""
from __future__ import annotations
@@ -54,21 +59,27 @@ import os
import subprocess
__all__ = [
"StagingError",
"STAGING_BASE",
"ACTIVE",
"staging_root_for",
"plan_staging",
"current_mounts_under",
"STAGING_BASE",
"StagingError",
"apply_plan",
"verify_staged",
"cleanup_task",
"current_mounts_under",
"delete_snapshot_tree",
"plan_staging",
"sidecar_for",
"snapshot_tree_names",
"stage_nested",
"staging_root_for",
"teardown",
"verify_staged",
]
#: Where staging trees are assembled. tmpfs; bind mounts consume no space.
STAGING_BASE = "/run/truecloud-nested"
#: staging_root -> zfs snapshot name ("pool/ds@snap"), for cleanup.
#: staging_root -> zfs snapshot name. A cache; the sidecar file is the source of
#: truth, so that a middlewared restart cannot orphan a snapshot.
ACTIVE: dict[str, str] = {}
@@ -76,18 +87,76 @@ class StagingError(Exception):
"""Staging could not produce a complete tree. The backup must not proceed."""
def staging_root_for(name: str, base: str = STAGING_BASE) -> str:
"""Stable staging root for a task name (e.g. ``cloud_backup-5``)."""
safe = "".join(c if (c.isalnum() or c in "-_.") else "_" for c in name) or "task"
# ── pure helpers ──────────────────────────────────────────────────────────────
def staging_root_for(name: str, base: str | None = None) -> str:
"""Stable staging root for a task name (e.g. ``cloud_backup-5``).
``base`` defaults to :data:`STAGING_BASE` at CALL time, not at import time --
a ``base=STAGING_BASE`` default would freeze the value into the function
object and silently ignore any later override.
"""
if base is None:
base = STAGING_BASE
safe = "".join(c if (c.isalnum() or c in "-_.") else "_" for c in name)
# A component of "." or ".." would escape STAGING_BASE once joined.
if not safe or safe.strip(".") == "":
safe = "task"
return os.path.join(base, safe)
def sidecar_for(staging_root: str) -> str:
"""Path of the file recording which ZFS snapshot a staging tree pins."""
return staging_root + ".snapshot"
def _write_sidecar(staging_root: str, snapshot: str) -> None:
"""Record the pinned snapshot on disk. Blocking; call via run_in_thread."""
with contextlib.suppress(OSError):
os.makedirs(os.path.dirname(staging_root), exist_ok=True)
with open(sidecar_for(staging_root), "w", encoding="utf-8") as fh:
fh.write(snapshot)
def _read_sidecar(staging_root: str) -> str | None:
"""The snapshot a previous run recorded here, if any."""
try:
with open(sidecar_for(staging_root), encoding="utf-8") as fh:
return fh.read().strip() or None
except OSError:
return None
def _remove_sidecar(staging_root: str) -> None:
with contextlib.suppress(OSError):
os.unlink(sidecar_for(staging_root))
def _depth(path: str) -> int:
return len([p for p in path.split("/") if p])
def plan_staging(base_mountpoint, path, snapshot_name, datasets, staging_root,
isdir=os.path.isdir):
def snapshot_tree_names(snapshot: str, all_names) -> list[str]:
"""Every snapshot produced by ``zfs snapshot -r <dataset>@<snap>``.
That is the parent plus one per descendant dataset, all sharing the same
name after the ``@``. Pure, so the sweep logic is testable without ZFS.
"""
dataset, _, snapname = snapshot.partition("@")
if not snapname:
return []
parent = f"{dataset}@{snapname}"
prefix = dataset + "/"
suffix = "@" + snapname
return [
n for n in all_names
if n == parent or (n.startswith(prefix) and n.endswith(suffix))
]
def plan_staging(base_dataset, base_mountpoint, path, snapshot_name, datasets,
staging_root, isdir=os.path.isdir):
"""Compute the bind-mount plan for staging a nested tree. Pure function.
``datasets`` is a list of dicts shaped like ``zfs.dataset.query`` results:
@@ -96,9 +165,12 @@ def plan_staging(base_mountpoint, path, snapshot_name, datasets, staging_root,
Returns ``(mounts, skipped)`` where ``mounts`` is an ordered list of
``(source, target)`` pairs (parents before children) and ``skipped`` is a
list of ``(dataset_name, reason)``.
list of ``(dataset_name, reason)`` covering only datasets that are *in
scope* -- i.e. descendants of ``base_dataset``. Datasets elsewhere on the
system are ignored silently; reporting them would bury the ones that matter.
Raises StagingError if a descendant holds data we would silently omit.
Raises StagingError if an in-scope descendant holds data we would otherwise
silently omit.
"""
def snapdir(mountpoint):
return os.path.join(mountpoint, ".zfs", "snapshot", snapshot_name)
@@ -113,20 +185,30 @@ def plan_staging(base_mountpoint, path, snapshot_name, datasets, staging_root,
mounts = [(root_src, staging_root)]
skipped = []
prefix = path.rstrip("/") + "/"
ds_prefix = base_dataset.rstrip("/") + "/"
path_prefix = path.rstrip("/") + "/"
for ds in datasets:
name = ds.get("name", "")
# Scope by DATASET NAME, not mountpoint: a dataset with no mountpoint
# cannot be scoped by path, and scoping by path first would drag in
# every mountpoint-less dataset on the box (all of Tank/.system/*, ...).
if not name.startswith(ds_prefix):
continue
props = ds.get("properties", {})
mp = props.get("mountpoint", {}).get("value", "")
name = ds.get("name", "?")
if not mp or mp in ("none", "legacy", "-"):
skipped.append((name, f"mountpoint is {mp or 'unset'}"))
continue
if not mp.startswith(prefix):
continue # not a descendant of the backup path
mounted = props.get("mounted", {}).get("value", "yes")
if mounted == "no":
if not mp.startswith(path_prefix):
# A descendant dataset mounted outside the backed-up path is
# genuinely not part of this tree. Not an omission.
continue
if props.get("mounted", {}).get("value", "yes") == "no":
# An unmounted (e.g. locked/encrypted) dataset contributes nothing to
# the live tree either, so skipping matches stock semantics -- but it
# is a real gap and must be visible, never silent.
@@ -142,8 +224,7 @@ def plan_staging(base_mountpoint, path, snapshot_name, datasets, staging_root,
f"refusing to back up an incomplete tree"
)
target = os.path.join(staging_root, os.path.relpath(mp, path))
mounts.append((src, target))
mounts.append((src, os.path.join(staging_root, os.path.relpath(mp, path))))
# Parents before children, so each mountpoint exists before we mount onto it.
mounts.sort(key=lambda m: _depth(m[1]))
@@ -168,11 +249,14 @@ def current_mounts_under(root, mounts_file="/proc/self/mounts"):
return found
# ── mount / unmount ───────────────────────────────────────────────────────────
def _run(cmd):
return subprocess.run(cmd, capture_output=True, text=True, check=False)
def apply_plan(mounts, runner=_run):
def apply_plan(mounts, runner=_run, isdir=os.path.isdir):
"""Execute the bind-mount plan. Blocking; call via ``run_in_thread``.
Raises StagingError on the first failure, after rolling back what was
@@ -186,7 +270,7 @@ def apply_plan(mounts, runner=_run):
try:
os.makedirs(staging_root, exist_ok=True)
for src, target in mounts:
if not os.path.isdir(target):
if not isdir(target):
# Child mountpoint dirs come from the parent snapshot, which is
# read-only -- we cannot mkdir them. Only the root is ours.
raise StagingError(f"staging target {target!r} does not exist")
@@ -206,7 +290,7 @@ def apply_plan(mounts, runner=_run):
return staging_root
def verify_staged(mounts, runner=_run, ismount=os.path.ismount, listdir=os.listdir):
def verify_staged(mounts, ismount=os.path.ismount, listdir=os.listdir):
"""Assert the staged tree is real and complete. Raises StagingError if not.
This is the anti-regression guard: it is what stops this feature from ever
@@ -251,15 +335,61 @@ def teardown(staging_root, runner=_run, mounts_file="/proc/self/mounts"):
# ── async orchestration (middleware is duck-typed; no middlewared import) ─────
async def stage_nested(middleware, path, snapshot, base_mountpoint, task_name, logger=None):
async def delete_snapshot_tree(middleware, snapshot, logger=None):
"""Delete the parent snapshot AND every child created by ``zfs snapshot -r``.
``zfs.snapshot.delete`` is non-recursive by default and stock calls it with
no options, so relying on stock would orphan one snapshot per descendant
dataset on every run. Idempotent: tolerates the parent already being gone
(stock's ``finally`` may have won the race once our mounts were released).
"""
dataset = snapshot.partition("@")[0]
try:
snaps = await middleware.call(
"zfs.snapshot.query", [["name", "^", dataset]], {"select": ["name"]}
)
# An empty result means the tree is already gone -- delete nothing, and
# do not fall back to the parent, which would only log a spurious
# "does not exist" warning on every clean run.
names = snapshot_tree_names(snapshot, [s["name"] for s in snaps])
except Exception as e: # noqa: BLE001 - fall back to at least the parent
if logger:
logger.warning(
"truecloud-patch: could not enumerate snapshot tree for %s: %r",
snapshot, e,
)
names = [snapshot]
for name in names:
try:
await middleware.call("zfs.snapshot.delete", name)
except Exception as e: # noqa: BLE001 - already gone is fine
if logger:
logger.warning(
"truecloud-patch: could not delete snapshot %s: %r", name, e
)
async def stage_nested(middleware, path, snapshot, base_dataset, base_mountpoint,
task_name, datasets, logger=None):
"""Build a complete staging tree for `path` from the already-taken `snapshot`.
`snapshot` is a full ZFS snapshot name ("Tap@cloud_backup-5-2026...").
`datasets` is the FILESYSTEM dataset list. **It MUST have been enumerated
AFTER `snapshot` was taken.** A list read beforehand can miss a dataset
created in the gap: the recursive snapshot would capture it, but the staging
plan would not, and its data would be silently omitted from the backup.
Enumerated afterwards, an unsnapshotted dataset instead trips the isdir()
check in plan_staging and fails the run loudly.
Returns the staging root to hand to the backup tool.
Raises StagingError if the tree cannot be staged completely -- the caller
must let that propagate so the backup fails instead of silently uploading a
partial tree.
partial tree. The caller is responsible for deleting `snapshot` in that case
(see SNAPSHOT_BLOCK in apply.sh).
"""
snapshot_name = snapshot.split("@", 1)[1]
staging_root = staging_root_for(task_name)
@@ -267,23 +397,45 @@ async def stage_nested(middleware, path, snapshot, base_mountpoint, task_name, l
# A previous run may have crashed mid-flight; never build on top of that.
await middleware.run_in_thread(teardown, staging_root)
datasets = await middleware.call("zfs.dataset.query", [["type", "=", "FILESYSTEM"]])
# ...and if it left a sidecar behind, that snapshot tree is still on disk and
# nothing else will ever reclaim it. Sweep it before we overwrite the record,
# or a single crashed run orphans 160+ snapshots permanently.
stale = await middleware.run_in_thread(_read_sidecar, staging_root)
if stale and stale != snapshot:
if logger:
logger.warning(
"truecloud-patch: reclaiming snapshot tree from an earlier "
"interrupted run: %s", stale,
)
await delete_snapshot_tree(middleware, stale, logger=logger)
mounts, skipped = await middleware.run_in_thread(
plan_staging, base_mountpoint, path, snapshot_name, datasets, staging_root
)
if logger:
for name, reason in skipped:
logger.warning("truecloud-patch: not staging dataset %r: %s", name, reason)
# Record the snapshot BEFORE mounting anything, not after. middlewared can
# die at any point (this patch even schedules a restart at boot), and the
# sidecar is the only thing that survives it -- an in-process dict would take
# the sole record of a 160-snapshot tree with it. Writing it after apply_plan
# would leave exactly the crash window the sidecar exists to close.
await middleware.run_in_thread(_write_sidecar, staging_root, snapshot)
await middleware.run_in_thread(apply_plan, mounts)
try:
mounts, skipped = await middleware.run_in_thread(
plan_staging, base_dataset, base_mountpoint, path, snapshot_name,
datasets, staging_root,
)
if logger:
for name, reason in skipped:
logger.warning(
"truecloud-patch: not staging dataset %r: %s", name, reason
)
await middleware.run_in_thread(apply_plan, mounts)
await middleware.run_in_thread(verify_staged, mounts)
except Exception:
await middleware.run_in_thread(teardown, staging_root)
await middleware.run_in_thread(_remove_sidecar, staging_root)
raise
ACTIVE[staging_root] = snapshot
if logger:
logger.info(
"truecloud-patch: staged %d dataset(s) from %s at %s",
@@ -296,13 +448,14 @@ async def cleanup_task(middleware, task_name, logger=None):
"""Tear down a task's staging tree and delete the snapshot it pinned.
Safe to call unconditionally: a no-op when the task was never staged.
Stock `restic_backup()` deletes the snapshot in its own `finally`, which
fails with EBUSY while our bind mounts pin it (it logs a warning and moves
on). We unmount here and then delete the snapshot for real.
"""
staging_root = staging_root_for(task_name)
sidecar = sidecar_for(staging_root)
snapshot = ACTIVE.pop(staging_root, None)
if snapshot is None:
# Sidecar survives a middlewared restart; ACTIVE does not.
snapshot = _read_sidecar(staging_root)
if snapshot is None and not os.path.isdir(staging_root):
return # never staged; nothing to do
@@ -313,10 +466,7 @@ async def cleanup_task(middleware, task_name, logger=None):
logger.warning("truecloud-patch: staging teardown: %s", err)
if snapshot is not None:
try:
await middleware.call("zfs.snapshot.delete", snapshot)
except Exception as e: # noqa: BLE001 - cleanup must never mask the real error
if logger:
logger.warning(
"truecloud-patch: could not delete snapshot %s: %r", snapshot, e
)
await delete_snapshot_tree(middleware, snapshot, logger=logger)
with contextlib.suppress(OSError):
os.unlink(sidecar)