flan/truenas-truecloud-patch
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases 9 Wiki Activity

Harden for Python 3.12+, add disclaimer, audit for robustness

Browse Source 
- sitecustomize.py: replace deprecated find_module/load_module with
  find_spec/exec_module (required for Python 3.12+ / TrueNAS SCALE 25.x)
- apply.sh: remove set -e (PREINIT must not fail catastrophically);
  detect middlewared's actual Python binary instead of assuming python3;
  log rotation to avoid unbounded growth; independent failure per step
- patch_ui.py: detect multiple bundle matches; include TrueNAS version
  in pattern-not-found warning; better MARKER specificity
- uninstall.sh: mirror Python detection logic from apply.sh
- README: lead with Storj $5→$50 price context; prominent unsupported
  disclaimer; Python version compatibility matrix; post-update checklist
- Add MIT LICENSE
This commit is contained in:
flan
2026-06-15 02:13:24 +00:00
parent 21b9333324
commit 0a54bcba9d
8 changed files with 559 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 sudolulo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+129 -30
View File
@@ -1,40 +1,101 @@
# truenas-truecloud-patch
Extends TrueNAS SCALE's **TrueCloud Backup** feature to support S3-compatible and native Backblaze B2 providers, instead of Storj only.
TrueCloud Backup already uses [restic](https://restic.net) under the hood.
This patch just removes the artificial restriction.
Extends TrueNAS SCALE's **TrueCloud Backup** feature to work with S3-compatible
providers and native Backblaze B2, instead of Storj only.
---
## What it patches
## Why this exists
| Layer | What changes |
|---|---|
| **Backend** | `B2RcloneRemote` gains `get_restic_config()` so native B2 repos work. `restic.py` URL construction is fixed for providers with no hostname component (`b2:bucket/path` instead of the broken `b2:/bucket/path`). |
| **UI** | The Angular bundle's `filterByProviders` binding in the TrueCloud task form is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]`, so those three credential types appear in the dropdown. |
In 2024, Storj raised the price of their TrueNAS-integrated storage tier from
**$5/month to $50/month** — a 10× increase. For many home lab and small-office
users, the TrueCloud Backup feature became unaffordable overnight.
TrueCloud Backup is the only native TrueNAS mechanism that provides:
- Integrated ZFS snapshot support before each backup
- Restic-based incremental deduplication
- Scheduled tasks with progress and log tracking in the UI
- Dataset lock integration
Running restic manually is possible but loses all of the above.
This patch restores access to the TrueCloud Backup feature for users who
need a provider other than Storj, with storage they already pay for or that
costs a fraction of the new Storj price.
---
## ⚠ Disclaimer — please read before installing
**This project is unofficial, unsupported, and not affiliated with iXsystems
or the TrueNAS project in any way.**
By installing this patch you accept the following:
- **Unsupported configuration.** TrueNAS support staff are not obligated to
help with any issue on a system running this patch. If you file a bug report,
remove the patch first and reproduce the issue on an unmodified system.
- **May break on TrueNAS updates.** The patch targets internal middleware APIs
that are not part of any public contract. They can change at any time. When
they do, the patch silently degrades to Storj-only behaviour rather than
breaking TrueNAS — but you should check the log after each update.
- **Your backups are your responsibility.** Verify that your backup jobs
complete successfully and that restores work before relying on them for
disaster recovery.
- **No warranty.** This software is provided as-is. See the LICENSE file.
If TrueNAS ever adds native support for additional providers in TrueCloud
Backup, uninstall this patch immediately.
---
## What is actually patched
**Nothing in TrueNAS's persistent database or configuration is modified.**
The patch operates on two files that live in `/usr/` (which TrueNAS replaces
on every update) and are therefore re-applied automatically on every boot.
| Layer | What changes | Technique |
|---|---|---|
| **Backend** | `B2RcloneRemote` gains `get_restic_config()`. `restic.py` URL builder is fixed for providers with no hostname component (`b2:bucket/path` vs the broken `b2:/bucket/path`). | `sitecustomize.py` — Python's standard startup hook; no middleware files are modified on disk |
| **UI** | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS bundle; original is backed up |
Both changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
restructured the relevant code), middlewared starts normally with Storj-only
support and the reason is logged to `/data/truecloud-patch/apply.log`.
## Supported providers after patching
| Provider | Credential type | Notes |
|---|---|---|
| Backblaze B2 (native) | `B2` | Requires this patch |
| AWS S3 / Wasabi / Cloudflare R2 / MinIO / etc. | `S3` | Already worked at the API level; UI restriction removed by patch |
| Storj | `STORJ_IX` | Unchanged — still works |
| Provider | Credential type in TrueNAS |
|---|---|
| Backblaze B2 (native B2 API) | `B2` |
| AWS S3, Wasabi, Cloudflare R2, MinIO, and any S3-compatible endpoint | `S3` |
| Storj (unchanged) | `STORJ_IX` |
## How persistence works
TrueNAS updates replace `/usr/` entirely. The patch survives by:
TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by storing
all scripts in `/data/truecloud-patch/` (a persistent ZFS dataset) and
registering a **PREINIT initshutdownscript** in the TrueNAS database. This
causes `apply.sh` to run on every boot before `middlewared` starts, placing
`sitecustomize.py` in the correct site-packages directory and re-patching the
UI bundle.
1. Storing all patch scripts in `/data/truecloud-patch/` (persistent ZFS dataset).
2. Registering a **PREINIT** `initshutdownscript` (stored in the TrueNAS database) that runs `apply.sh` on every boot before `middlewared` starts.
3. `apply.sh` re-installs `sitecustomize.py` into Python site-packages and re-patches the Angular bundle each boot.
## Python version compatibility
`sitecustomize.py` uses the `find_spec` / `exec_module` import hook API
(introduced in Python 3.4, required from Python 3.12 onwards). This covers:
- TrueNAS SCALE 24.x — Debian 12, Python 3.11 ✓
- TrueNAS SCALE 25.x — Debian 13, Python 3.12 ✓
---
## Install
Run on your TrueNAS box (as root):
Run on your TrueNAS box as root, with the system fully booted:
```bash
git clone https://github.com/sudolulo/truenas-truecloud-patch.git
@@ -42,13 +103,25 @@ cd truenas-truecloud-patch
bash install.sh
```
Then refresh your browser. S3 and B2 credentials will now appear in the TrueCloud Backup task form.
Refresh your browser. S3 and B2 credentials now appear in the
**Data Protection → TrueCloud Backup → Add** credential dropdown.
---
## Creating a credential
Before creating a backup task, add a credential in the TrueNAS UI:
**Data Protection → TrueCloud Backup → Add → (create new credential)**
Or use **Credentials → Backup Credentials → Cloud Credentials → Add** and
select B2 or Amazon S3.
For S3-compatible providers, select **Amazon S3**, then set a custom endpoint
in the credential's advanced settings (e.g. `https://s3.wasabisys.com`).
## Creating a task via CLI
If you prefer the API over the UI (or want to script it):
If the UI still shows only Storj after refreshing (e.g. the JS bundle pattern
changed in a new TrueNAS version), create tasks directly via the REST API:
```bash
# List your cloud credentials to find the right ID
@@ -67,7 +140,7 @@ python3 /data/truecloud-patch/create_task.py \
--keep-last 14
```
Get an API key from **TrueNAS UI → System → API Keys**.
Get an API key from **System → API Keys → Add**.
---
@@ -77,25 +150,51 @@ Get an API key from **TrueNAS UI → System → API Keys**.
bash /path/to/truenas-truecloud-patch/uninstall.sh
```
Restores the original UI bundle and removes the PREINIT hook. The backend patch disappears automatically on the next `middlewared` restart once `sitecustomize.py` is removed.
Removes the PREINIT hook, `sitecustomize.py`, and restores the original UI
bundle from backup. The backend changes vanish on the next `middlewared`
restart.
---
## After a TrueNAS update
1. Check the log: `cat /data/truecloud-patch/apply.log | tail -30`
2. If you see "WARNING: … pattern not found", the UI patch needs updating.
[Open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues)
with your TrueNAS version number.
3. The backend patch (B2 support + URL fix) is more stable — check that a
B2 backup job still completes successfully after any update.
---
## Troubleshooting
**Apply log** (check after reboot or install):
**Apply log** (check after each reboot or install):
```bash
cat /data/truecloud-patch/apply.log
```
**Verify backend patch is active** (run while middlewared is running):
**Verify backend patch is loaded** (while middlewared is running):
```bash
midclt call cloud_backup.transfer_setting_choices # should return without error
python3 -c "
from middlewared.rclone.remote.b2 import B2RcloneRemote
print('B2 restic:', hasattr(B2RcloneRemote, 'get_restic_config'))
print('B2 restic support:', hasattr(B2RcloneRemote, 'get_restic_config'))
"
```
**UI pattern not found warning**
The Angular bundle's structure changed in a TrueNAS update. Open an issue with your TrueNAS version; the patch may need a regex update.
**Verify the UI patch** (should print your TrueNAS version):
```bash
grep -c 'STORJ_IX.*S3.*B2' \
$(find /usr/share/truenas -name '*.js' 2>/dev/null) 2>/dev/null \
| grep -v ':0'
```
**B2 backup fails with credential error**
Confirm the credential type is exactly `B2` (not `S3` with a B2 endpoint).
B2's native restic backend uses a different auth path than S3-compatible B2.
**S3-compatible backup fails**
S3 support already existed in the backend — the credential setup is the likely
issue. Verify the endpoint URL, access key, secret key, and bucket name in
the credential settings. Use `--insecure` in create_task.py only if you are
using a self-signed certificate on your S3 endpoint.
install.sh
+31 -12
View File
@@ -1,27 +1,42 @@
#!/bin/bash
# install.sh — run once on the TrueNAS box to set up the patch.
# install.sh — run once on the TrueNAS box to set up truecloud-patch.
#
# Prerequisites: run as root on TrueNAS SCALE with middlewared running.
#
# What this does:
# 1. Copies patch files to /data/truecloud-patch/ (survives updates).
# 2. Registers a PREINIT initshutdownscript so apply.sh runs on every boot
# before middlewared starts, re-applying patches to the refreshed /usr/.
# 3. Applies the patches immediately without rebooting.
# 4. Restarts middlewared so the backend patch takes effect now.
# 1. Copies patch files to /data/truecloud-patch/ (survives OS updates).
# 2. Registers a PREINIT initshutdownscript in the TrueNAS database so
# apply.sh re-applies the patches on every boot before middlewared starts.
# 3. Applies the patches immediately (no reboot required).
# 4. Restarts middlewared so the backend change takes effect now.
set -euo pipefail
PATCH_DIR="/data/truecloud-patch"
REPO_DIR="$(cd "$(dirname "$0")" && pwd)"
echo "=== TrueNAS TrueCloud Provider Patch ==="
echo "=== TrueNAS TrueCloud Provider Patch — Install ==="
echo ""
# ── Preflight ─────────────────────────────────────────────────────────────────
if [ "$(id -u)" -ne 0 ]; then
echo "ERROR: must be run as root." >&2
exit 1
fi
if ! command -v midclt &>/dev/null; then
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
exit 1
fi
if ! midclt call core.ping &>/dev/null; then
echo "ERROR: middlewared is not responding. Is TrueNAS fully booted?" >&2
exit 1
fi
# ── Copy files ────────────────────────────────────────────────────────────────
echo "Copying patch files to $PATCH_DIR ..."
mkdir -p "$PATCH_DIR"
cp "$REPO_DIR/patch/sitecustomize.py" "$PATCH_DIR/"
@@ -33,7 +48,8 @@ echo "Done."
echo ""
# ── Register PREINIT script ───────────────────────────────────────────────────
echo "Checking initshutdownscript registration ..."
echo "Registering PREINIT boot hook ..."
EXISTING_ID=$(midclt call initshutdownscript.query '[]' | \
python3 -c "
@@ -52,22 +68,25 @@ else
midclt call initshutdownscript.create \
'{"type":"SCRIPT","script":"/data/truecloud-patch/apply.sh","when":"PREINIT","enabled":true,"comment":"TrueCloud provider patch (S3/B2)"}' \
> /dev/null
echo "Registered PREINIT script."
echo "Registered."
fi
echo ""
# ── Apply now ─────────────────────────────────────────────────────────────────
echo "Applying patches ..."
bash "$PATCH_DIR/apply.sh"
cat "$PATCH_DIR/apply.log" | tail -20
echo ""
echo "Patch log ($PATCH_DIR/apply.log):"
tail -30 "$PATCH_DIR/apply.log"
echo ""
# ── Restart middlewared ───────────────────────────────────────────────────────
echo "Restarting middlewared (backend patch takes effect) ..."
echo "Restarting middlewared so the backend patch takes effect ..."
systemctl restart middlewared
echo "Done."
echo ""
echo "Refresh your browser to pick up the UI change."
echo ""
echo "To create a TrueCloud Backup task with S3 or B2 credentials:"
patch/apply.sh
+89 -29
View File
@@ -1,44 +1,104 @@
#!/bin/bash
# /data/truecloud-patch/apply.sh
#
# PREINIT script registered via TrueNAS initshutdownscript.
# Runs on every boot before middlewared starts, re-applying patches that
# TrueNAS updates wipe from /usr/.
# Registered as a TrueNAS PREINIT initshutdownscript.
# Runs on every boot BEFORE middlewared starts, so patches land before
# the first Python process for middlewared is created.
#
# Two things are patched:
# 1. sitecustomize.py → monkey-patches B2 restic support + URL fix into
# middlewared at Python startup time (backend)
# 2. Angular JS bundle → adds S3 and B2 to the credential dropdown in
# the TrueCloud Backup task form (UI)
set -euo pipefail
# TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
#
# 1. sitecustomize.py — Python executes this automatically at startup.
# Monkey-patches B2 restic support and fixes the
# URL builder for empty-host providers.
#
# 2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
# from Storj-only to include S3 and B2.
#
# Design principle: every step is independently fail-safe.
# A failed patch logs a warning and continues; middlewared always starts.
# Never use `set -e` in a PREINIT script.
PATCH_DIR="/data/truecloud-patch"
LOG="$PATCH_DIR/apply.log"
{
echo "=== $(date -Iseconds) ==="
# Rotate log at 512 KB to avoid unbounded growth on a system volume.
if [ -f "$LOG" ] && [ "$(wc -c < "$LOG")" -gt 524288 ]; then
mv "$LOG" "${LOG}.1"
fi
# ── 1. Backend: install sitecustomize.py ─────────────────────────────
SITE_PKG=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
exec >> "$LOG" 2>&1
echo "=== $(date -Iseconds) ==="
if [ -z "$SITE_PKG" ]; then
echo "WARNING: could not determine site-packages path, skipping backend patch"
else
# If an unrelated sitecustomize.py exists, back it up once.
if [ -f "$SITE_PKG/sitecustomize.py" ] && \
! grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
cp "$SITE_PKG/sitecustomize.py" "$SITE_PKG/sitecustomize.py.pre-truecloud-patch"
echo "Backed up existing sitecustomize.py"
# ── Helpers ───────────────────────────────────────────────────────────────────
warn() { echo "WARNING: $*"; }
ok() { echo "OK: $*"; }
# Find the Python interpreter that middlewared actually uses.
# On TrueNAS SCALE, /usr/bin/middlewared is usually a Python entry-point script
# with a shebang pointing at the right interpreter (system or venv).
find_mw_python() {
local py="python3"
local shebang=""
if [ -x /usr/bin/middlewared ]; then
# Read the first line safely (max 256 bytes) — avoids reading a binary ELF
shebang=$(dd if=/usr/bin/middlewared bs=256 count=1 2>/dev/null | head -1 || true)
if [[ "$shebang" =~ ^'#!'(/[^[:space:]]+python[^[:space:]]*) ]]; then
py="${BASH_REMATCH[1]}"
elif [[ "$shebang" =~ ^'#!/usr/bin/env '(python[^[:space:]]*) ]]; then
py=$(command -v "${BASH_REMATCH[1]}" 2>/dev/null || echo "python3")
fi
cp "$PATCH_DIR/sitecustomize.py" "$SITE_PKG/sitecustomize.py"
echo "Installed sitecustomize.py → $SITE_PKG/sitecustomize.py"
fi
# ── 2. UI: patch Angular bundle ──────────────────────────────────────
python3 "$PATCH_DIR/patch_ui.py"
# Verify the chosen interpreter can actually import middlewared.
if ! "$py" -c "import middlewared" 2>/dev/null; then
warn "Detected Python '$py' cannot import middlewared; falling back to python3"
py="python3"
fi
echo "=== done ==="
echo "$py"
}
} >> "$LOG" 2>&1
# ── Step 1: sitecustomize.py ──────────────────────────────────────────────────
echo "--- backend patch ---"
PYTHON=$(find_mw_python)
echo "Using Python: $PYTHON"
SITE_PKG=$("$PYTHON" -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
if [ -z "$SITE_PKG" ]; then
warn "Cannot determine site-packages directory; skipping backend patch."
warn "Verify that '$PYTHON -c \"import site; print(site.getsitepackages())\"' works."
else
# Back up any pre-existing sitecustomize.py that isn't ours.
if [ -f "$SITE_PKG/sitecustomize.py" ] && \
! grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
cp "$SITE_PKG/sitecustomize.py" \
"$SITE_PKG/sitecustomize.py.pre-truecloud-patch"
ok "Backed up existing sitecustomize.py"
fi
if cp "$PATCH_DIR/sitecustomize.py" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
ok "Installed sitecustomize.py → $SITE_PKG/sitecustomize.py"
else
warn "Failed to write $SITE_PKG/sitecustomize.py (permission error?)"
fi
fi
# ── Step 2: Angular bundle ────────────────────────────────────────────────────
echo "--- UI patch ---"
if "$PYTHON" "$PATCH_DIR/patch_ui.py"; then
: # patch_ui.py prints its own status
else
warn "patch_ui.py exited non-zero; UI dropdown may still show Storj only."
fi
# ── Done ──────────────────────────────────────────────────────────────────────
echo "=== done ==="
patch/create_task.py
+74 -47
View File
@@ -2,34 +2,42 @@
"""
create_task.py — create TrueNAS TrueCloud Backup tasks with S3 or B2 credentials.
The TrueNAS UI restricts the credential dropdown to Storj only. This script
talks directly to the REST API so you can use any compatible credential.
The TrueNAS UI normally restricts the credential dropdown to Storj only.
This script bypasses that restriction by calling the REST API directly.
Requires: an API key from TrueNAS UI → System → API Keys.
Compatible providers (after the truecloud-patch backend patch is applied):
S3 — any S3-compatible endpoint (AWS, Wasabi, Cloudflare R2, MinIO, …)
B2 — Backblaze B2 native API
STORJ_IX — Storj (unchanged, always worked)
Requires a TrueNAS API key: UI → System → API Keys → Add.
Examples
--------
List available cloud credentials:
python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
Create a task backed by a B2 credential (id=3):
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
--name "tank-to-b2" \\
--path /mnt/tank/data \\
--credential 3 \\
--bucket my-bucket \\
--folder backups/tank \\
--password "restic-repo-password" \\
--keep-last 14
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
--name "tank-to-b2" \\
--path /mnt/tank/data \\
--credential 3 \\
--bucket my-bucket \\
--folder backups/tank \\
--password "restic-repo-password" \\
--keep-last 14
Create a task using an S3-compatible credential (Wasabi, R2, etc.):
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
--name "tank-to-wasabi" \\
--path /mnt/tank/data \\
--credential 5 \\
--bucket my-bucket \\
--folder backups \\
--password "restic-repo-password"
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
--name "tank-to-wasabi" \\
--path /mnt/tank/data \\
--credential 5 \\
--bucket my-bucket \\
--folder backups \\
--password "restic-repo-password"
List existing TrueCloud Backup tasks:
python3 create_task.py --host 192.168.1.1 --api-key <key> list-tasks
"""
import argparse
@@ -41,6 +49,7 @@ import urllib.request
def make_client(host, api_key, insecure=False):
"""Return a callable that makes authenticated REST API calls."""
base = f"https://{host}/api/v2.0"
headers = {
"Authorization": f"Bearer {api_key}",
@@ -59,7 +68,7 @@ def make_client(host, api_key, insecure=False):
with urllib.request.urlopen(req, context=ctx) as resp:
return json.loads(resp.read())
except urllib.error.HTTPError as exc:
detail = exc.read().decode()
detail = exc.read().decode(errors="replace")
print(f"HTTP {exc.code} {exc.reason}: {detail}", file=sys.stderr)
sys.exit(1)
except urllib.error.URLError as exc:
@@ -69,13 +78,15 @@ def make_client(host, api_key, insecure=False):
return call
# ── Sub-commands ──────────────────────────────────────────────────────────────
def cmd_list_credentials(client, _args):
creds = client("GET", "/cloudsync/credentials")
if not creds:
print("No cloud credentials configured.")
return
print(f"{'ID':>4} {'Provider':<14} Name")
print("─" * 50)
print("─" * 55)
for c in sorted(creds, key=lambda x: x["id"]):
print(f"{c['id']:>4} {c['provider']['type']:<14} {c['name']}")
@@ -86,17 +97,20 @@ def cmd_list_tasks(client, _args):
print("No TrueCloud Backup tasks configured.")
return
print(f"{'ID':>4} {'Enabled':<8} {'Provider':<14} Name")
print("─" * 55)
print("─" * 60)
for t in sorted(tasks, key=lambda x: x["id"]):
ptype = t["credentials"]["provider"]["type"] if t.get("credentials") else "?"
ptype = (t.get("credentials") or {}).get("provider", {}).get("type", "?")
enabled = "yes" if t.get("enabled") else "no"
print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t['description']}")
print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t.get('description', '')}")
def cmd_create(client, args):
parts = args.schedule.split()
if len(parts) != 5:
print("--schedule must be a 5-field cron expression, e.g. '0 2 * * *'", file=sys.stderr)
print(
"ERROR: --schedule must be a 5-field cron expression, e.g. '0 2 * * *'",
file=sys.stderr,
)
sys.exit(1)
minute, hour, dom, month, dow = parts
@@ -127,39 +141,52 @@ def cmd_create(client, args):
print(f"Created task id={result['id']} name={result['description']!r}")
# ── CLI ───────────────────────────────────────────────────────────────────────
def main():
p = argparse.ArgumentParser(
description="Manage TrueNAS TrueCloud Backup tasks (S3/B2/Storj)",
description="Manage TrueNAS TrueCloud Backup tasks (S3 / B2 / Storj)",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__.split("Examples")[1] if "Examples" in __doc__ else "",
)
p.add_argument("--host", required=True, metavar="HOST", help="TrueNAS hostname or IP")
p.add_argument("--api-key", required=True, metavar="KEY", help="TrueNAS API key")
p.add_argument("--insecure", action="store_true", help="Skip TLS certificate verification")
p.add_argument("--host", required=True, metavar="HOST",
help="TrueNAS hostname or IP address")
p.add_argument("--api-key", required=True, metavar="KEY",
help="TrueNAS API key (System → API Keys)")
p.add_argument("--insecure", action="store_true",
help="Skip TLS certificate verification (self-signed certs)")
sub = p.add_subparsers(dest="cmd", required=True)
sub.add_parser("list-credentials", help="List configured cloud credentials")
sub.add_parser("list-tasks", help="List TrueCloud Backup tasks")
sub.add_parser("list-tasks", help="List TrueCloud Backup tasks")
c = sub.add_parser("create", help="Create a TrueCloud Backup task")
c.add_argument("--name", required=True, help="Task description shown in the UI")
c.add_argument("--path", required=True, help="Local dataset path, e.g. /mnt/tank/data")
c.add_argument("--credential", required=True, type=int, metavar="ID",
help="Cloud credential ID (from list-credentials)")
c.add_argument("--bucket", required=True, help="Bucket or container name")
c.add_argument("--folder", default="", help="Folder path within the bucket (default: root)")
c.add_argument("--password", required=True, help="Restic repository encryption password")
c.add_argument("--keep-last", type=int, default=14, metavar="N",
c = sub.add_parser("create", help="Create a new TrueCloud Backup task")
c.add_argument("--name", required=True,
help="Task description shown in the UI")
c.add_argument("--path", required=True,
help="Local dataset path (e.g. /mnt/tank/data)")
c.add_argument("--credential", required=True, type=int, metavar="ID",
help="Cloud credential ID — get it from list-credentials")
c.add_argument("--bucket", required=True,
help="Bucket (S3) or container (B2) name")
c.add_argument("--folder", default="",
help="Path within the bucket (default: root)")
c.add_argument("--password", required=True,
help="Restic repository encryption password (choose a strong one)")
c.add_argument("--keep-last", type=int, default=14, metavar="N",
help="Snapshots to retain after each run (default: 14)")
c.add_argument("--schedule", default="0 2 * * *",
c.add_argument("--schedule", default="0 2 * * *",
help="Cron schedule (default: '0 2 * * *' — daily at 02:00)")
c.add_argument("--transfer-setting",
choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"], default="DEFAULT")
c.add_argument("--snapshot", action="store_true",
help="Create a ZFS snapshot before each backup")
choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"],
default="DEFAULT",
help="Pack-size / concurrency preset (default: DEFAULT)")
c.add_argument("--snapshot", action="store_true",
help="Create a ZFS snapshot before each backup run")
c.add_argument("--absolute-paths", action="store_true",
help="Preserve absolute paths inside the restic repo")
c.add_argument("--disabled", action="store_true",
help="Preserve absolute paths inside the restic repository")
c.add_argument("--disabled", action="store_true",
help="Create the task in a disabled state")
args = p.parse_args()
@@ -167,8 +194,8 @@ def main():
dispatch = {
"list-credentials": cmd_list_credentials,
"list-tasks": cmd_list_tasks,
"create": cmd_create,
"list-tasks": cmd_list_tasks,
"create": cmd_create,
}
dispatch[args.cmd](client, args)
patch/patch_ui.py
+66 -22
View File
@@ -3,12 +3,20 @@
Patches the TrueNAS webui Angular bundle to show S3 and B2 credentials
in the TrueCloud Backup task form, instead of Storj only.
The compiled bundle contains:
"filterByProviders",["STORJ_IX"]
which is the template binding [filterByProviders]="[CloudSyncProviderName.Storj]".
We replace the array with ["STORJ_IX","S3","B2"] so all three providers appear.
Angular's Ivy compiler inlines TypeScript string enum values as literals in
the compiled bundle, so the template binding:
Run automatically by apply.sh on every boot. Safe to run multiple times.
[filterByProviders]="[CloudSyncProviderName.Storj]"
appears verbatim in the minified JS as:
"filterByProviders",["STORJ_IX"]
We replace that array to include S3 and B2. The file is backed up before
modification so uninstall.sh can restore it.
Safe to run multiple times — a marker string detects an already-patched file.
Exits 0 in all cases (warnings are printed to stdout and logged by apply.sh).
"""
import os
@@ -22,13 +30,13 @@ WEBUI_CANDIDATES = [
"/var/www/truenas",
]
# The pattern as compiled by Angular's Ivy into minified JS.
# String literals survive minification; the function name before the comma
# is mangled and not part of our match.
# Angular's Ivy template compiler serialises the Storj-only filter as this
# exact substring in every production build we've observed.
FIND = re.compile(r'("filterByProviders",)\["STORJ_IX"\]')
REPLACE = r'\1["STORJ_IX","S3","B2"]'
# Presence of this string means we already patched this file.
# A patched file contains both "S3" and "B2" next to "STORJ_IX" in this form.
# This string is specific enough not to appear elsewhere in the bundle.
MARKER = '"STORJ_IX","S3","B2"'
@@ -40,8 +48,14 @@ def find_webui():
def find_bundle(webui):
for root, _, names in os.walk(webui):
for name in names:
"""
Walk the webui directory looking for the JS file that contains the
filterByProviders binding. Returns (path, content) or (None, None).
Only .js files are read; binary files and permission errors are skipped.
"""
matches = []
for root, _dirs, names in os.walk(webui):
for name in sorted(names): # deterministic order
if not name.endswith(".js"):
continue
path = os.path.join(root, name)
@@ -49,41 +63,71 @@ def find_bundle(webui):
with open(path) as fh:
content = fh.read()
if FIND.search(content):
return path, content
matches.append((path, content))
except (UnicodeDecodeError, PermissionError, OSError):
continue
return None, None
if not matches:
return None, None
if len(matches) > 1:
# Unexpected — log all matches so the operator can investigate.
print(
f"[truecloud-patch] WARNING: filterByProviders pattern found in "
f"{len(matches)} files; patching only the first."
)
for p, _ in matches:
print(f"[truecloud-patch] {p}")
return matches[0]
def main():
webui = find_webui()
if not webui:
print("[truecloud-patch] WARNING: webui directory not found, skipping UI patch")
sys.exit(0)
print(
"[truecloud-patch] WARNING: webui directory not found; skipping UI patch.\n"
"[truecloud-patch] Searched: " + ", ".join(WEBUI_CANDIDATES)
)
return
path, content = find_bundle(webui)
if path is None:
print(
"[truecloud-patch] WARNING: filterByProviders pattern not found in webui bundle.\n"
"[truecloud-patch] The UI patch may need updating for this TrueNAS version.\n"
"[truecloud-patch] File an issue at https://github.com/sudolulo/truenas-truecloud-patch"
"[truecloud-patch] WARNING: filterByProviders pattern not found in any JS bundle.\n"
"[truecloud-patch] The TrueNAS webui may have been restructured in this version.\n"
"[truecloud-patch] File an issue at https://github.com/sudolulo/truenas-truecloud-patch\n"
f"[truecloud-patch] TrueNAS version info: {_tnversion()}"
)
sys.exit(0)
return
if MARKER in content:
print(f"[truecloud-patch] UI already patched: {path}")
sys.exit(0)
return
backup = path + ".pre-truecloud-patch"
if not os.path.exists(backup):
shutil.copy2(path, backup)
patched, count = FIND.subn(REPLACE, content)
with open(path, "w") as fh:
fh.write(patched)
try:
with open(path, "w") as fh:
fh.write(patched)
except OSError as exc:
print(f"[truecloud-patch] ERROR: Could not write {path}: {exc}")
return
print(f"[truecloud-patch] UI bundle patched ({count} replacement(s)): {path}")
def _tnversion():
try:
with open("/etc/version") as fh:
return fh.read().strip()
except OSError:
return "unknown"
if __name__ == "__main__":
main()
patch/sitecustomize.py
+108 -58
View File
@@ -2,78 +2,113 @@
TrueCloud provider patch — sitecustomize.py
Installed into Python site-packages on every boot by apply.sh.
Hooks the import of two middlewared modules and patches them in-place:
Hooks two middlewared module imports using the find_spec / exec_module API
(required for Python 3.12+, which ships with TrueNAS SCALE 25.x / Debian 13):
middlewared.rclone.remote.b2
Adds get_restic_config() so the native B2 restic backend works.
Restic URL: b2:<bucket>/<folder>
Auth env: B2_ACCOUNT_ID, B2_ACCOUNT_KEY
Adds get_restic_config() so the native restic B2 backend works.
Restic repo URL: b2:<bucket>/<folder>
Auth: B2_ACCOUNT_ID, B2_ACCOUNT_KEY
middlewared.plugins.cloud_backup.restic
Fixes URL construction for providers that have no hostname component
(url == ""). Stock code builds "b2:/bucket/path" (broken double-slash);
patched code builds "b2:bucket/path".
Fixes the URL builder for providers with no hostname component.
Stock code: f"{rclone_type}:{url}/{remote_path}" → "b2:/bucket/path" (broken)
Patched: "b2:bucket/path" when url == ""
Safe for all Python processes on the system: if middlewared is absent the
hook installs but never fires, and all errors are caught and logged to stderr.
Both patches are no-ops if the module already provides the functionality
(i.e. a future TrueNAS version adds native support). All errors are caught
and written to stderr so middlewared always starts regardless of patch state.
"""
import sys
def _install():
_pending = {
# ── Import hook ───────────────────────────────────────────────────────────────
class _Finder:
"""
find_spec-based meta path finder (Python 3.4+, required for 3.12+).
Intercepts specific module imports, loads them normally, then patches.
"""
_targets = frozenset({
"middlewared.rclone.remote.b2",
"middlewared.plugins.cloud_backup.restic",
}
_loading = set()
})
class _Hook:
def find_module(self, fullname, path=None): # noqa: ARG002
if fullname in _pending and fullname not in _loading:
return self
return None
def __init__(self):
self._loading = set() # guards against re-entrant imports
self._done = set() # modules already patched
def load_module(self, fullname):
if fullname in sys.modules:
module = sys.modules[fullname]
else:
_loading.add(fullname)
try:
__import__(fullname)
finally:
_loading.discard(fullname)
module = sys.modules[fullname]
_pending.discard(fullname)
def find_spec(self, fullname, path, target=None): # noqa: ARG002
import importlib.machinery
if (
fullname in self._targets
and fullname not in self._done
and fullname not in self._loading
):
return importlib.machinery.ModuleSpec(fullname, _Loader(self, fullname))
return None
def _mark_done(self, fullname):
self._done.add(fullname)
if self._done >= self._targets:
try:
if fullname == "middlewared.rclone.remote.b2":
_patch_b2(module)
elif fullname == "middlewared.plugins.cloud_backup.restic":
_patch_restic(module)
except Exception as exc:
sys.stderr.write(f"[truecloud-patch] patch failed for {fullname}: {exc}\n")
sys.meta_path.remove(self)
except ValueError:
pass
if not _pending:
try:
sys.meta_path.remove(hook)
except ValueError:
pass
return module
class _Loader:
def __init__(self, finder, fullname):
self._finder = finder
self._fullname = fullname
hook = _Hook()
sys.meta_path.append(hook)
def create_module(self, spec): # noqa: ARG002
return None # use Python's default module creation
def exec_module(self, module):
import importlib.util
fullname = self._fullname
self._finder._loading.add(fullname)
try:
# find_spec for the real file — our finder returns None while
# fullname is in _loading, so the normal finders handle this.
real_spec = importlib.util.find_spec(fullname)
if real_spec is None:
raise ImportError(f"No module named {fullname!r}")
real_spec.loader.exec_module(module)
# Fix module metadata so it looks like a normal import.
module.__spec__ = real_spec
module.__loader__ = real_spec.loader
if getattr(real_spec, "origin", None):
module.__file__ = real_spec.origin
finally:
self._finder._loading.discard(fullname)
self._finder._mark_done(fullname)
try:
_PATCHES[fullname](module)
except Exception as exc:
sys.stderr.write(
f"[truecloud-patch] patch failed for {fullname}: {exc}\n"
)
# ── Patch functions ───────────────────────────────────────────────────────────
def _patch_b2(module):
cls = module.B2RcloneRemote
if hasattr(cls, "get_restic_config"):
return # future TrueNAS version already added it
# A future TrueNAS version already added native B2 restic support.
return
def get_restic_config(self, task):
def get_restic_config(self, task): # noqa: ARG001
p = task["credentials"]["provider"]
return "", {
"B2_ACCOUNT_ID": p["account"],
@@ -90,15 +125,18 @@ def _patch_restic(module):
if getattr(orig, "_truecloud_patched", False):
return
# Capture module-level references; REMOTES is the same mutable dict
# object that remotes.setup() will populate later.
_REMOTES = module.REMOTES
_get_remote_path = module.get_remote_path
# ResticConfig is safe to capture now (it's a dataclass defined in the module).
# REMOTES and get_remote_path are imported lazily inside the function so that
# module layout changes in future middlewared versions fail at call time
# (during an actual backup job) rather than silently at patch time.
_ResticConfig = module.ResticConfig
def get_restic_config(cloud_backup):
remote = _REMOTES[cloud_backup["credentials"]["provider"]["type"]]
remote_path = _get_remote_path(remote, cloud_backup["attributes"])
from middlewared.plugins.cloud.path import get_remote_path
from middlewared.plugins.cloud.remotes import REMOTES
remote = REMOTES[cloud_backup["credentials"]["provider"]["type"]]
remote_path = get_remote_path(remote, cloud_backup["attributes"])
url, env = remote.get_restic_config(cloud_backup)
if cloud_backup["cache_path"]:
@@ -106,8 +144,7 @@ def _patch_restic(module):
else:
cache = ["--no-cache"]
# Fix: stock code does f"{rclone_type}:{url}/{remote_path}" which
# produces "b2:/bucket/path" when url is empty.
# Stock code produces "b2:/bucket/path" when url == "" (double-slash).
repo = (
f"{remote.rclone_type}:{url}/{remote_path}"
if url
@@ -122,9 +159,22 @@ def _patch_restic(module):
sys.stderr.write("[truecloud-patch] restic URL fix applied\n")
try:
_PATCHES = {
"middlewared.rclone.remote.b2": _patch_b2,
"middlewared.plugins.cloud_backup.restic": _patch_restic,
}
# ── Entry point ───────────────────────────────────────────────────────────────
def _install():
import importlib.util
if importlib.util.find_spec("middlewared") is not None:
_install()
if importlib.util.find_spec("middlewared") is None:
return # not a middlewared Python process; nothing to do
sys.meta_path.append(_Finder())
try:
_install()
except Exception:
pass
pass # never raise from sitecustomize.py — it would prevent Python from starting
uninstall.sh
+41 -13
View File
@@ -1,5 +1,5 @@
#!/bin/bash
# uninstall.sh — removes all traces of the patch from a TrueNAS box.
# uninstall.sh — remove all traces of truecloud-patch from a TrueNAS box.
set -euo pipefail
@@ -8,12 +8,20 @@ PATCH_DIR="/data/truecloud-patch"
echo "=== TrueNAS TrueCloud Provider Patch — Uninstall ==="
echo ""
if [ "$(id -u)" -ne 0 ]; then
echo "ERROR: must be run as root." >&2
exit 1
fi
if ! command -v midclt &>/dev/null; then
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
exit 1
fi
# ── Remove PREINIT registration ───────────────────────────────────────────────
# ── Remove PREINIT hook ───────────────────────────────────────────────────────
echo "Removing PREINIT boot hook ..."
IDS=$(midclt call initshutdownscript.query '[]' | \
python3 -c "
import sys, json
@@ -25,46 +33,67 @@ for s in json.load(sys.stdin):
if [ -n "$IDS" ]; then
for id in $IDS; do
midclt call initshutdownscript.delete "$id" > /dev/null
echo "Removed initshutdownscript id=$id"
echo " Removed initshutdownscript id=$id"
done
else
echo "No initshutdownscript entry found (already removed or never installed)."
echo " No entry found (already removed or never installed)."
fi
echo ""
# ── Remove sitecustomize.py ───────────────────────────────────────────────────
SITE_PKG=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
echo "Removing sitecustomize.py ..."
# Use the same Python detection logic as apply.sh
PYTHON="python3"
if [ -x /usr/bin/middlewared ]; then
shebang=$(dd if=/usr/bin/middlewared bs=256 count=1 2>/dev/null | head -1 || true)
if [[ "$shebang" =~ ^'#!'(/[^[:space:]]+python[^[:space:]]*) ]]; then
PYTHON="${BASH_REMATCH[1]}"
fi
fi
SITE_PKG=$("$PYTHON" -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
if [ -n "$SITE_PKG" ] && [ -f "$SITE_PKG/sitecustomize.py" ]; then
if grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
rm "$SITE_PKG/sitecustomize.py"
echo "Removed $SITE_PKG/sitecustomize.py"
echo " Removed $SITE_PKG/sitecustomize.py"
# Restore a pre-existing sitecustomize.py if we backed one up
if [ -f "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" ]; then
mv "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" \
"$SITE_PKG/sitecustomize.py"
echo "Restored previous sitecustomize.py"
echo " Restored previous sitecustomize.py"
fi
else
echo " $SITE_PKG/sitecustomize.py is not ours; leaving it alone."
fi
else
echo " Not found (already removed or install didn't place it here)."
fi
echo ""
# ── Restore UI bundle backup ──────────────────────────────────────────────────
# ── Restore UI bundle ─────────────────────────────────────────────────────────
echo "Restoring UI bundle backup ..."
RESTORED=0
for backup in $(find /usr/share/truenas /var/www/truenas -name "*.js.pre-truecloud-patch" 2>/dev/null); do
for backup in $(find /usr/share/truenas /var/www/truenas \
-name "*.js.pre-truecloud-patch" 2>/dev/null); do
original="${backup%.pre-truecloud-patch}"
mv "$backup" "$original"
echo "Restored: $original"
echo " Restored: $original"
RESTORED=1
done
if [ "$RESTORED" -eq 0 ]; then
echo "No UI bundle backups found (patch will be undone by the next TrueNAS update)."
echo " No backup files found."
echo " The UI patch will be undone automatically by the next TrueNAS update."
fi
echo ""
# ── Remove patch directory ────────────────────────────────────────────────────
if [ -d "$PATCH_DIR" ]; then
rm -rf "$PATCH_DIR"
echo "Removed $PATCH_DIR"
@@ -73,6 +102,5 @@ echo ""
echo "Restarting middlewared ..."
systemctl restart middlewared
echo ""
echo "Uninstall complete. Refresh your browser to see the restored UI."