Initial implementation: extend TrueCloud Backup to S3 and B2 providers

Patches middlewared at runtime via sitecustomize.py (no file edits to /usr/)
and widens the UI credential dropdown from Storj-only to S3+B2+Storj.
Persists across TrueNAS updates via PREINIT initshutdownscript stored in DB.
This commit is contained in:
2026-06-15 02:02:17 +00:00
commit 21b9333324
7 changed files with 693 additions and 0 deletions
+101
View File
@@ -0,0 +1,101 @@
# truenas-truecloud-patch
Extends TrueNAS SCALE's **TrueCloud Backup** feature to support S3-compatible and native Backblaze B2 providers, instead of Storj only.
TrueCloud Backup already uses [restic](https://restic.net) under the hood.
This patch just removes the artificial restriction.
---
## What it patches
| Layer | What changes |
|---|---|
| **Backend** | `B2RcloneRemote` gains `get_restic_config()` so native B2 repos work. `restic.py` URL construction is fixed for providers with no hostname component (`b2:bucket/path` instead of the broken `b2:/bucket/path`). |
| **UI** | The Angular bundle's `filterByProviders` binding in the TrueCloud task form is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]`, so those three credential types appear in the dropdown. |
## Supported providers after patching
| Provider | Credential type | Notes |
|---|---|---|
| Backblaze B2 (native) | `B2` | Requires this patch |
| AWS S3 / Wasabi / Cloudflare R2 / MinIO / etc. | `S3` | Already worked at the API level; UI restriction removed by patch |
| Storj | `STORJ_IX` | Unchanged — still works |
## How persistence works
TrueNAS updates replace `/usr/` entirely. The patch survives by:
1. Storing all patch scripts in `/data/truecloud-patch/` (persistent ZFS dataset).
2. Registering a **PREINIT** `initshutdownscript` (stored in the TrueNAS database) that runs `apply.sh` on every boot before `middlewared` starts.
3. `apply.sh` re-installs `sitecustomize.py` into Python site-packages and re-patches the Angular bundle each boot.
---
## Install
Run on your TrueNAS box (as root):
```bash
git clone https://github.com/sudolulo/truenas-truecloud-patch.git
cd truenas-truecloud-patch
bash install.sh
```
Then refresh your browser. S3 and B2 credentials will now appear in the TrueCloud Backup task form.
---
## Creating a task via CLI
If you prefer the API over the UI (or want to script it):
```bash
# List your cloud credentials to find the right ID
python3 /data/truecloud-patch/create_task.py \
--host 192.168.1.1 --api-key <key> list-credentials
# Create a task with a B2 credential (id=3)
python3 /data/truecloud-patch/create_task.py \
--host 192.168.1.1 --api-key <key> create \
--name "tank-to-b2" \
--path /mnt/tank/data \
--credential 3 \
--bucket my-bucket \
--folder backups/tank \
--password "restic-repo-password" \
--keep-last 14
```
Get an API key from **TrueNAS UI → System → API Keys**.
---
## Uninstall
```bash
bash /path/to/truenas-truecloud-patch/uninstall.sh
```
Restores the original UI bundle and removes the PREINIT hook. The backend patch disappears automatically on the next `middlewared` restart once `sitecustomize.py` is removed.
---
## Troubleshooting
**Apply log** (check after reboot or install):
```bash
cat /data/truecloud-patch/apply.log
```
**Verify backend patch is active** (run while middlewared is running):
```bash
midclt call cloud_backup.transfer_setting_choices # should return without error
python3 -c "
from middlewared.rclone.remote.b2 import B2RcloneRemote
print('B2 restic:', hasattr(B2RcloneRemote, 'get_restic_config'))
"
```
**UI pattern not found warning**
The Angular bundle's structure changed in a TrueNAS update. Open an issue with your TrueNAS version; the patch may need a regex update.
Executable
+74
View File
@@ -0,0 +1,74 @@
#!/bin/bash
# install.sh — run once on the TrueNAS box to set up the patch.
#
# What this does:
# 1. Copies patch files to /data/truecloud-patch/ (survives updates).
# 2. Registers a PREINIT initshutdownscript so apply.sh runs on every boot
# before middlewared starts, re-applying patches to the refreshed /usr/.
# 3. Applies the patches immediately without rebooting.
# 4. Restarts middlewared so the backend patch takes effect now.
set -euo pipefail
PATCH_DIR="/data/truecloud-patch"
REPO_DIR="$(cd "$(dirname "$0")" && pwd)"
echo "=== TrueNAS TrueCloud Provider Patch ==="
echo ""
if ! command -v midclt &>/dev/null; then
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
exit 1
fi
# ── Copy files ────────────────────────────────────────────────────────────────
echo "Copying patch files to $PATCH_DIR ..."
mkdir -p "$PATCH_DIR"
cp "$REPO_DIR/patch/sitecustomize.py" "$PATCH_DIR/"
cp "$REPO_DIR/patch/patch_ui.py" "$PATCH_DIR/"
cp "$REPO_DIR/patch/apply.sh" "$PATCH_DIR/"
cp "$REPO_DIR/patch/create_task.py" "$PATCH_DIR/"
chmod +x "$PATCH_DIR/apply.sh" "$PATCH_DIR/create_task.py"
echo "Done."
echo ""
# ── Register PREINIT script ───────────────────────────────────────────────────
echo "Checking initshutdownscript registration ..."
EXISTING_ID=$(midclt call initshutdownscript.query '[]' | \
python3 -c "
import sys, json
for s in json.load(sys.stdin):
if s.get('script') == '/data/truecloud-patch/apply.sh':
print(s['id'])
break
" 2>/dev/null || true)
if [ -n "$EXISTING_ID" ]; then
echo "Already registered (id=$EXISTING_ID). Ensuring it is enabled ..."
midclt call initshutdownscript.update "$EXISTING_ID" \
'{"enabled": true}' > /dev/null
else
midclt call initshutdownscript.create \
'{"type":"SCRIPT","script":"/data/truecloud-patch/apply.sh","when":"PREINIT","enabled":true,"comment":"TrueCloud provider patch (S3/B2)"}' \
> /dev/null
echo "Registered PREINIT script."
fi
echo ""
# ── Apply now ─────────────────────────────────────────────────────────────────
echo "Applying patches ..."
bash "$PATCH_DIR/apply.sh"
cat "$PATCH_DIR/apply.log" | tail -20
echo ""
# ── Restart middlewared ───────────────────────────────────────────────────────
echo "Restarting middlewared (backend patch takes effect) ..."
systemctl restart middlewared
echo "Done."
echo ""
echo "Refresh your browser to pick up the UI change."
echo ""
echo "To create a TrueCloud Backup task with S3 or B2 credentials:"
echo " python3 $PATCH_DIR/create_task.py --help"
Executable
+44
View File
@@ -0,0 +1,44 @@
#!/bin/bash
# /data/truecloud-patch/apply.sh
#
# PREINIT script registered via TrueNAS initshutdownscript.
# Runs on every boot before middlewared starts, re-applying patches that
# TrueNAS updates wipe from /usr/.
#
# Two things are patched:
# 1. sitecustomize.py → monkey-patches B2 restic support + URL fix into
# middlewared at Python startup time (backend)
# 2. Angular JS bundle → adds S3 and B2 to the credential dropdown in
# the TrueCloud Backup task form (UI)
set -euo pipefail
PATCH_DIR="/data/truecloud-patch"
LOG="$PATCH_DIR/apply.log"
{
echo "=== $(date -Iseconds) ==="
# ── 1. Backend: install sitecustomize.py ─────────────────────────────
SITE_PKG=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
if [ -z "$SITE_PKG" ]; then
echo "WARNING: could not determine site-packages path, skipping backend patch"
else
# If an unrelated sitecustomize.py exists, back it up once.
if [ -f "$SITE_PKG/sitecustomize.py" ] && \
! grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
cp "$SITE_PKG/sitecustomize.py" "$SITE_PKG/sitecustomize.py.pre-truecloud-patch"
echo "Backed up existing sitecustomize.py"
fi
cp "$PATCH_DIR/sitecustomize.py" "$SITE_PKG/sitecustomize.py"
echo "Installed sitecustomize.py → $SITE_PKG/sitecustomize.py"
fi
# ── 2. UI: patch Angular bundle ──────────────────────────────────────
python3 "$PATCH_DIR/patch_ui.py"
echo "=== done ==="
} >> "$LOG" 2>&1
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env python3
"""
create_task.py — create TrueNAS TrueCloud Backup tasks with S3 or B2 credentials.
The TrueNAS UI restricts the credential dropdown to Storj only. This script
talks directly to the REST API so you can use any compatible credential.
Requires: an API key from TrueNAS UI → System → API Keys.
Examples
--------
List available cloud credentials:
python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
Create a task backed by a B2 credential (id=3):
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
--name "tank-to-b2" \\
--path /mnt/tank/data \\
--credential 3 \\
--bucket my-bucket \\
--folder backups/tank \\
--password "restic-repo-password" \\
--keep-last 14
Create a task using an S3-compatible credential (Wasabi, R2, etc.):
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
--name "tank-to-wasabi" \\
--path /mnt/tank/data \\
--credential 5 \\
--bucket my-bucket \\
--folder backups \\
--password "restic-repo-password"
"""
import argparse
import json
import ssl
import sys
import urllib.error
import urllib.request
def make_client(host, api_key, insecure=False):
base = f"https://{host}/api/v2.0"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
ctx = ssl.create_default_context()
if insecure:
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
def call(method, path, body=None):
url = base + path
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, context=ctx) as resp:
return json.loads(resp.read())
except urllib.error.HTTPError as exc:
detail = exc.read().decode()
print(f"HTTP {exc.code} {exc.reason}: {detail}", file=sys.stderr)
sys.exit(1)
except urllib.error.URLError as exc:
print(f"Connection error: {exc.reason}", file=sys.stderr)
sys.exit(1)
return call
def cmd_list_credentials(client, _args):
creds = client("GET", "/cloudsync/credentials")
if not creds:
print("No cloud credentials configured.")
return
print(f"{'ID':>4} {'Provider':<14} Name")
print("─" * 50)
for c in sorted(creds, key=lambda x: x["id"]):
print(f"{c['id']:>4} {c['provider']['type']:<14} {c['name']}")
def cmd_list_tasks(client, _args):
tasks = client("GET", "/cloud_backup")
if not tasks:
print("No TrueCloud Backup tasks configured.")
return
print(f"{'ID':>4} {'Enabled':<8} {'Provider':<14} Name")
print("─" * 55)
for t in sorted(tasks, key=lambda x: x["id"]):
ptype = t["credentials"]["provider"]["type"] if t.get("credentials") else "?"
enabled = "yes" if t.get("enabled") else "no"
print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t['description']}")
def cmd_create(client, args):
parts = args.schedule.split()
if len(parts) != 5:
print("--schedule must be a 5-field cron expression, e.g. '0 2 * * *'", file=sys.stderr)
sys.exit(1)
minute, hour, dom, month, dow = parts
body = {
"description": args.name,
"path": args.path,
"credentials": args.credential,
"attributes": {
"bucket": args.bucket,
"folder": args.folder,
},
"password": args.password,
"keep_last": args.keep_last,
"transfer_setting": args.transfer_setting,
"schedule": {
"minute": minute,
"hour": hour,
"dom": dom,
"month": month,
"dow": dow,
},
"snapshot": args.snapshot,
"absolute_paths": args.absolute_paths,
"enabled": not args.disabled,
}
result = client("POST", "/cloud_backup", body)
print(f"Created task id={result['id']} name={result['description']!r}")
def main():
p = argparse.ArgumentParser(
description="Manage TrueNAS TrueCloud Backup tasks (S3/B2/Storj)",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
p.add_argument("--host", required=True, metavar="HOST", help="TrueNAS hostname or IP")
p.add_argument("--api-key", required=True, metavar="KEY", help="TrueNAS API key")
p.add_argument("--insecure", action="store_true", help="Skip TLS certificate verification")
sub = p.add_subparsers(dest="cmd", required=True)
sub.add_parser("list-credentials", help="List configured cloud credentials")
sub.add_parser("list-tasks", help="List TrueCloud Backup tasks")
c = sub.add_parser("create", help="Create a TrueCloud Backup task")
c.add_argument("--name", required=True, help="Task description shown in the UI")
c.add_argument("--path", required=True, help="Local dataset path, e.g. /mnt/tank/data")
c.add_argument("--credential", required=True, type=int, metavar="ID",
help="Cloud credential ID (from list-credentials)")
c.add_argument("--bucket", required=True, help="Bucket or container name")
c.add_argument("--folder", default="", help="Folder path within the bucket (default: root)")
c.add_argument("--password", required=True, help="Restic repository encryption password")
c.add_argument("--keep-last", type=int, default=14, metavar="N",
help="Snapshots to retain after each run (default: 14)")
c.add_argument("--schedule", default="0 2 * * *",
help="Cron schedule (default: '0 2 * * *' — daily at 02:00)")
c.add_argument("--transfer-setting",
choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"], default="DEFAULT")
c.add_argument("--snapshot", action="store_true",
help="Create a ZFS snapshot before each backup")
c.add_argument("--absolute-paths", action="store_true",
help="Preserve absolute paths inside the restic repo")
c.add_argument("--disabled", action="store_true",
help="Create the task in a disabled state")
args = p.parse_args()
client = make_client(args.host, args.api_key, args.insecure)
dispatch = {
"list-credentials": cmd_list_credentials,
"list-tasks": cmd_list_tasks,
"create": cmd_create,
}
dispatch[args.cmd](client, args)
if __name__ == "__main__":
main()
+89
View File
@@ -0,0 +1,89 @@
#!/usr/bin/env python3
"""
Patches the TrueNAS webui Angular bundle to show S3 and B2 credentials
in the TrueCloud Backup task form, instead of Storj only.
The compiled bundle contains:
"filterByProviders",["STORJ_IX"]
which is the template binding [filterByProviders]="[CloudSyncProviderName.Storj]".
We replace the array with ["STORJ_IX","S3","B2"] so all three providers appear.
Run automatically by apply.sh on every boot. Safe to run multiple times.
"""
import os
import re
import shutil
import sys
WEBUI_CANDIDATES = [
"/usr/share/truenas/webui",
"/usr/share/truenas-ui",
"/var/www/truenas",
]
# The pattern as compiled by Angular's Ivy into minified JS.
# String literals survive minification; the function name before the comma
# is mangled and not part of our match.
FIND = re.compile(r'("filterByProviders",)\["STORJ_IX"\]')
REPLACE = r'\1["STORJ_IX","S3","B2"]'
# Presence of this string means we already patched this file.
MARKER = '"STORJ_IX","S3","B2"'
def find_webui():
for d in WEBUI_CANDIDATES:
if os.path.isdir(d):
return d
return None
def find_bundle(webui):
for root, _, names in os.walk(webui):
for name in names:
if not name.endswith(".js"):
continue
path = os.path.join(root, name)
try:
with open(path) as fh:
content = fh.read()
if FIND.search(content):
return path, content
except (UnicodeDecodeError, PermissionError, OSError):
continue
return None, None
def main():
webui = find_webui()
if not webui:
print("[truecloud-patch] WARNING: webui directory not found, skipping UI patch")
sys.exit(0)
path, content = find_bundle(webui)
if path is None:
print(
"[truecloud-patch] WARNING: filterByProviders pattern not found in webui bundle.\n"
"[truecloud-patch] The UI patch may need updating for this TrueNAS version.\n"
"[truecloud-patch] File an issue at https://github.com/sudolulo/truenas-truecloud-patch"
)
sys.exit(0)
if MARKER in content:
print(f"[truecloud-patch] UI already patched: {path}")
sys.exit(0)
backup = path + ".pre-truecloud-patch"
if not os.path.exists(backup):
shutil.copy2(path, backup)
patched, count = FIND.subn(REPLACE, content)
with open(path, "w") as fh:
fh.write(patched)
print(f"[truecloud-patch] UI bundle patched ({count} replacement(s)): {path}")
if __name__ == "__main__":
main()
+130
View File
@@ -0,0 +1,130 @@
"""
TrueCloud provider patch — sitecustomize.py
Installed into Python site-packages on every boot by apply.sh.
Hooks the import of two middlewared modules and patches them in-place:
middlewared.rclone.remote.b2
Adds get_restic_config() so the native B2 restic backend works.
Restic URL: b2:<bucket>/<folder>
Auth env: B2_ACCOUNT_ID, B2_ACCOUNT_KEY
middlewared.plugins.cloud_backup.restic
Fixes URL construction for providers that have no hostname component
(url == ""). Stock code builds "b2:/bucket/path" (broken double-slash);
patched code builds "b2:bucket/path".
Safe for all Python processes on the system: if middlewared is absent the
hook installs but never fires, and all errors are caught and logged to stderr.
"""
import sys
def _install():
_pending = {
"middlewared.rclone.remote.b2",
"middlewared.plugins.cloud_backup.restic",
}
_loading = set()
class _Hook:
def find_module(self, fullname, path=None): # noqa: ARG002
if fullname in _pending and fullname not in _loading:
return self
return None
def load_module(self, fullname):
if fullname in sys.modules:
module = sys.modules[fullname]
else:
_loading.add(fullname)
try:
__import__(fullname)
finally:
_loading.discard(fullname)
module = sys.modules[fullname]
_pending.discard(fullname)
try:
if fullname == "middlewared.rclone.remote.b2":
_patch_b2(module)
elif fullname == "middlewared.plugins.cloud_backup.restic":
_patch_restic(module)
except Exception as exc:
sys.stderr.write(f"[truecloud-patch] patch failed for {fullname}: {exc}\n")
if not _pending:
try:
sys.meta_path.remove(hook)
except ValueError:
pass
return module
hook = _Hook()
sys.meta_path.append(hook)
def _patch_b2(module):
cls = module.B2RcloneRemote
if hasattr(cls, "get_restic_config"):
return # future TrueNAS version already added it
def get_restic_config(self, task):
p = task["credentials"]["provider"]
return "", {
"B2_ACCOUNT_ID": p["account"],
"B2_ACCOUNT_KEY": p["key"],
}
cls.get_restic_config = get_restic_config
cls.restic = True
sys.stderr.write("[truecloud-patch] B2 restic support enabled\n")
def _patch_restic(module):
orig = module.get_restic_config
if getattr(orig, "_truecloud_patched", False):
return
# Capture module-level references; REMOTES is the same mutable dict
# object that remotes.setup() will populate later.
_REMOTES = module.REMOTES
_get_remote_path = module.get_remote_path
_ResticConfig = module.ResticConfig
def get_restic_config(cloud_backup):
remote = _REMOTES[cloud_backup["credentials"]["provider"]["type"]]
remote_path = _get_remote_path(remote, cloud_backup["attributes"])
url, env = remote.get_restic_config(cloud_backup)
if cloud_backup["cache_path"]:
cache = ["--cache-dir", cloud_backup["cache_path"]]
else:
cache = ["--no-cache"]
# Fix: stock code does f"{rclone_type}:{url}/{remote_path}" which
# produces "b2:/bucket/path" when url is empty.
repo = (
f"{remote.rclone_type}:{url}/{remote_path}"
if url
else f"{remote.rclone_type}:{remote_path}"
)
cmd = ["restic"] + cache + ["--json", "-r", repo]
env["RESTIC_PASSWORD"] = cloud_backup["password"]
return _ResticConfig(cmd, env)
get_restic_config._truecloud_patched = True
module.get_restic_config = get_restic_config
sys.stderr.write("[truecloud-patch] restic URL fix applied\n")
try:
import importlib.util
if importlib.util.find_spec("middlewared") is not None:
_install()
except Exception:
pass
Executable
+78
View File
@@ -0,0 +1,78 @@
#!/bin/bash
# uninstall.sh — removes all traces of the patch from a TrueNAS box.
set -euo pipefail
PATCH_DIR="/data/truecloud-patch"
echo "=== TrueNAS TrueCloud Provider Patch — Uninstall ==="
echo ""
if ! command -v midclt &>/dev/null; then
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
exit 1
fi
# ── Remove PREINIT registration ───────────────────────────────────────────────
IDS=$(midclt call initshutdownscript.query '[]' | \
python3 -c "
import sys, json
for s in json.load(sys.stdin):
if s.get('script') == '/data/truecloud-patch/apply.sh':
print(s['id'])
" 2>/dev/null || true)
if [ -n "$IDS" ]; then
for id in $IDS; do
midclt call initshutdownscript.delete "$id" > /dev/null
echo "Removed initshutdownscript id=$id"
done
else
echo "No initshutdownscript entry found (already removed or never installed)."
fi
echo ""
# ── Remove sitecustomize.py ───────────────────────────────────────────────────
SITE_PKG=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
if [ -n "$SITE_PKG" ] && [ -f "$SITE_PKG/sitecustomize.py" ]; then
if grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
rm "$SITE_PKG/sitecustomize.py"
echo "Removed $SITE_PKG/sitecustomize.py"
# Restore a pre-existing sitecustomize.py if we backed one up
if [ -f "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" ]; then
mv "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" \
"$SITE_PKG/sitecustomize.py"
echo "Restored previous sitecustomize.py"
fi
fi
fi
echo ""
# ── Restore UI bundle backup ──────────────────────────────────────────────────
RESTORED=0
for backup in $(find /usr/share/truenas /var/www/truenas -name "*.js.pre-truecloud-patch" 2>/dev/null); do
original="${backup%.pre-truecloud-patch}"
mv "$backup" "$original"
echo "Restored: $original"
RESTORED=1
done
if [ "$RESTORED" -eq 0 ]; then
echo "No UI bundle backups found (patch will be undone by the next TrueNAS update)."
fi
echo ""
# ── Remove patch directory ────────────────────────────────────────────────────
if [ -d "$PATCH_DIR" ]; then
rm -rf "$PATCH_DIR"
echo "Removed $PATCH_DIR"
fi
echo ""
echo "Restarting middlewared ..."
systemctl restart middlewared
echo ""
echo "Uninstall complete. Refresh your browser to see the restored UI."