Harden for Python 3.12+, add disclaimer, audit for robustness
- sitecustomize.py: replace deprecated find_module/load_module with find_spec/exec_module (required for Python 3.12+ / TrueNAS SCALE 25.x) - apply.sh: remove set -e (PREINIT must not fail catastrophically); detect middlewared's actual Python binary instead of assuming python3; log rotation to avoid unbounded growth; independent failure per step - patch_ui.py: detect multiple bundle matches; include TrueNAS version in pattern-not-found warning; better MARKER specificity - uninstall.sh: mirror Python detection logic from apply.sh - README: lead with Storj $5→$50 price context; prominent unsupported disclaimer; Python version compatibility matrix; post-update checklist - Add MIT LICENSE
This commit is contained in:
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2025 sudolulo
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -1,40 +1,101 @@
|
|||||||
# truenas-truecloud-patch
|
# truenas-truecloud-patch
|
||||||
|
|
||||||
Extends TrueNAS SCALE's **TrueCloud Backup** feature to support S3-compatible and native Backblaze B2 providers, instead of Storj only.
|
Extends TrueNAS SCALE's **TrueCloud Backup** feature to work with S3-compatible
|
||||||
|
providers and native Backblaze B2, instead of Storj only.
|
||||||
TrueCloud Backup already uses [restic](https://restic.net) under the hood.
|
|
||||||
This patch just removes the artificial restriction.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## What it patches
|
## Why this exists
|
||||||
|
|
||||||
| Layer | What changes |
|
In 2024, Storj raised the price of their TrueNAS-integrated storage tier from
|
||||||
|---|---|
|
**$5/month to $50/month** — a 10× increase. For many home lab and small-office
|
||||||
| **Backend** | `B2RcloneRemote` gains `get_restic_config()` so native B2 repos work. `restic.py` URL construction is fixed for providers with no hostname component (`b2:bucket/path` instead of the broken `b2:/bucket/path`). |
|
users, the TrueCloud Backup feature became unaffordable overnight.
|
||||||
| **UI** | The Angular bundle's `filterByProviders` binding in the TrueCloud task form is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]`, so those three credential types appear in the dropdown. |
|
|
||||||
|
TrueCloud Backup is the only native TrueNAS mechanism that provides:
|
||||||
|
- Integrated ZFS snapshot support before each backup
|
||||||
|
- Restic-based incremental deduplication
|
||||||
|
- Scheduled tasks with progress and log tracking in the UI
|
||||||
|
- Dataset lock integration
|
||||||
|
|
||||||
|
Running restic manually is possible but loses all of the above.
|
||||||
|
This patch restores access to the TrueCloud Backup feature for users who
|
||||||
|
need a provider other than Storj, with storage they already pay for or that
|
||||||
|
costs a fraction of the new Storj price.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠ Disclaimer — please read before installing
|
||||||
|
|
||||||
|
**This project is unofficial, unsupported, and not affiliated with iXsystems
|
||||||
|
or the TrueNAS project in any way.**
|
||||||
|
|
||||||
|
By installing this patch you accept the following:
|
||||||
|
|
||||||
|
- **Unsupported configuration.** TrueNAS support staff are not obligated to
|
||||||
|
help with any issue on a system running this patch. If you file a bug report,
|
||||||
|
remove the patch first and reproduce the issue on an unmodified system.
|
||||||
|
|
||||||
|
- **May break on TrueNAS updates.** The patch targets internal middleware APIs
|
||||||
|
that are not part of any public contract. They can change at any time. When
|
||||||
|
they do, the patch silently degrades to Storj-only behaviour rather than
|
||||||
|
breaking TrueNAS — but you should check the log after each update.
|
||||||
|
|
||||||
|
- **Your backups are your responsibility.** Verify that your backup jobs
|
||||||
|
complete successfully and that restores work before relying on them for
|
||||||
|
disaster recovery.
|
||||||
|
|
||||||
|
- **No warranty.** This software is provided as-is. See the LICENSE file.
|
||||||
|
|
||||||
|
If TrueNAS ever adds native support for additional providers in TrueCloud
|
||||||
|
Backup, uninstall this patch immediately.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What is actually patched
|
||||||
|
|
||||||
|
**Nothing in TrueNAS's persistent database or configuration is modified.**
|
||||||
|
The patch operates on two files that live in `/usr/` (which TrueNAS replaces
|
||||||
|
on every update) and are therefore re-applied automatically on every boot.
|
||||||
|
|
||||||
|
| Layer | What changes | Technique |
|
||||||
|
|---|---|---|
|
||||||
|
| **Backend** | `B2RcloneRemote` gains `get_restic_config()`. `restic.py` URL builder is fixed for providers with no hostname component (`b2:bucket/path` vs the broken `b2:/bucket/path`). | `sitecustomize.py` — Python's standard startup hook; no middleware files are modified on disk |
|
||||||
|
| **UI** | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS bundle; original is backed up |
|
||||||
|
|
||||||
|
Both changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
|
||||||
|
restructured the relevant code), middlewared starts normally with Storj-only
|
||||||
|
support and the reason is logged to `/data/truecloud-patch/apply.log`.
|
||||||
|
|
||||||
## Supported providers after patching
|
## Supported providers after patching
|
||||||
|
|
||||||
| Provider | Credential type | Notes |
|
| Provider | Credential type in TrueNAS |
|
||||||
|---|---|---|
|
|---|---|
|
||||||
| Backblaze B2 (native) | `B2` | Requires this patch |
|
| Backblaze B2 (native B2 API) | `B2` |
|
||||||
| AWS S3 / Wasabi / Cloudflare R2 / MinIO / etc. | `S3` | Already worked at the API level; UI restriction removed by patch |
|
| AWS S3, Wasabi, Cloudflare R2, MinIO, and any S3-compatible endpoint | `S3` |
|
||||||
| Storj | `STORJ_IX` | Unchanged — still works |
|
| Storj (unchanged) | `STORJ_IX` |
|
||||||
|
|
||||||
## How persistence works
|
## How persistence works
|
||||||
|
|
||||||
TrueNAS updates replace `/usr/` entirely. The patch survives by:
|
TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by storing
|
||||||
|
all scripts in `/data/truecloud-patch/` (a persistent ZFS dataset) and
|
||||||
|
registering a **PREINIT initshutdownscript** in the TrueNAS database. This
|
||||||
|
causes `apply.sh` to run on every boot before `middlewared` starts, placing
|
||||||
|
`sitecustomize.py` in the correct site-packages directory and re-patching the
|
||||||
|
UI bundle.
|
||||||
|
|
||||||
1. Storing all patch scripts in `/data/truecloud-patch/` (persistent ZFS dataset).
|
## Python version compatibility
|
||||||
2. Registering a **PREINIT** `initshutdownscript` (stored in the TrueNAS database) that runs `apply.sh` on every boot before `middlewared` starts.
|
|
||||||
3. `apply.sh` re-installs `sitecustomize.py` into Python site-packages and re-patches the Angular bundle each boot.
|
`sitecustomize.py` uses the `find_spec` / `exec_module` import hook API
|
||||||
|
(introduced in Python 3.4, required from Python 3.12 onwards). This covers:
|
||||||
|
|
||||||
|
- TrueNAS SCALE 24.x — Debian 12, Python 3.11 ✓
|
||||||
|
- TrueNAS SCALE 25.x — Debian 13, Python 3.12 ✓
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
Run on your TrueNAS box (as root):
|
Run on your TrueNAS box as root, with the system fully booted:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/sudolulo/truenas-truecloud-patch.git
|
git clone https://github.com/sudolulo/truenas-truecloud-patch.git
|
||||||
@@ -42,13 +103,25 @@ cd truenas-truecloud-patch
|
|||||||
bash install.sh
|
bash install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Then refresh your browser. S3 and B2 credentials will now appear in the TrueCloud Backup task form.
|
Refresh your browser. S3 and B2 credentials now appear in the
|
||||||
|
**Data Protection → TrueCloud Backup → Add** credential dropdown.
|
||||||
|
|
||||||
---
|
## Creating a credential
|
||||||
|
|
||||||
|
Before creating a backup task, add a credential in the TrueNAS UI:
|
||||||
|
|
||||||
|
**Data Protection → TrueCloud Backup → Add → (create new credential)**
|
||||||
|
|
||||||
|
Or use **Credentials → Backup Credentials → Cloud Credentials → Add** and
|
||||||
|
select B2 or Amazon S3.
|
||||||
|
|
||||||
|
For S3-compatible providers, select **Amazon S3**, then set a custom endpoint
|
||||||
|
in the credential's advanced settings (e.g. `https://s3.wasabisys.com`).
|
||||||
|
|
||||||
## Creating a task via CLI
|
## Creating a task via CLI
|
||||||
|
|
||||||
If you prefer the API over the UI (or want to script it):
|
If the UI still shows only Storj after refreshing (e.g. the JS bundle pattern
|
||||||
|
changed in a new TrueNAS version), create tasks directly via the REST API:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# List your cloud credentials to find the right ID
|
# List your cloud credentials to find the right ID
|
||||||
@@ -67,7 +140,7 @@ python3 /data/truecloud-patch/create_task.py \
|
|||||||
--keep-last 14
|
--keep-last 14
|
||||||
```
|
```
|
||||||
|
|
||||||
Get an API key from **TrueNAS UI → System → API Keys**.
|
Get an API key from **System → API Keys → Add**.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -77,25 +150,51 @@ Get an API key from **TrueNAS UI → System → API Keys**.
|
|||||||
bash /path/to/truenas-truecloud-patch/uninstall.sh
|
bash /path/to/truenas-truecloud-patch/uninstall.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Restores the original UI bundle and removes the PREINIT hook. The backend patch disappears automatically on the next `middlewared` restart once `sitecustomize.py` is removed.
|
Removes the PREINIT hook, `sitecustomize.py`, and restores the original UI
|
||||||
|
bundle from backup. The backend changes vanish on the next `middlewared`
|
||||||
|
restart.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## After a TrueNAS update
|
||||||
|
|
||||||
|
1. Check the log: `cat /data/truecloud-patch/apply.log | tail -30`
|
||||||
|
2. If you see "WARNING: … pattern not found", the UI patch needs updating.
|
||||||
|
[Open an issue](https://github.com/sudolulo/truenas-truecloud-patch/issues)
|
||||||
|
with your TrueNAS version number.
|
||||||
|
3. The backend patch (B2 support + URL fix) is more stable — check that a
|
||||||
|
B2 backup job still completes successfully after any update.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
**Apply log** (check after reboot or install):
|
**Apply log** (check after each reboot or install):
|
||||||
```bash
|
```bash
|
||||||
cat /data/truecloud-patch/apply.log
|
cat /data/truecloud-patch/apply.log
|
||||||
```
|
```
|
||||||
|
|
||||||
**Verify backend patch is active** (run while middlewared is running):
|
**Verify backend patch is loaded** (while middlewared is running):
|
||||||
```bash
|
```bash
|
||||||
midclt call cloud_backup.transfer_setting_choices # should return without error
|
|
||||||
python3 -c "
|
python3 -c "
|
||||||
from middlewared.rclone.remote.b2 import B2RcloneRemote
|
from middlewared.rclone.remote.b2 import B2RcloneRemote
|
||||||
print('B2 restic:', hasattr(B2RcloneRemote, 'get_restic_config'))
|
print('B2 restic support:', hasattr(B2RcloneRemote, 'get_restic_config'))
|
||||||
"
|
"
|
||||||
```
|
```
|
||||||
|
|
||||||
**UI pattern not found warning**
|
**Verify the UI patch** (should print your TrueNAS version):
|
||||||
The Angular bundle's structure changed in a TrueNAS update. Open an issue with your TrueNAS version; the patch may need a regex update.
|
```bash
|
||||||
|
grep -c 'STORJ_IX.*S3.*B2' \
|
||||||
|
$(find /usr/share/truenas -name '*.js' 2>/dev/null) 2>/dev/null \
|
||||||
|
| grep -v ':0'
|
||||||
|
```
|
||||||
|
|
||||||
|
**B2 backup fails with credential error**
|
||||||
|
Confirm the credential type is exactly `B2` (not `S3` with a B2 endpoint).
|
||||||
|
B2's native restic backend uses a different auth path than S3-compatible B2.
|
||||||
|
|
||||||
|
**S3-compatible backup fails**
|
||||||
|
S3 support already existed in the backend — the credential setup is the likely
|
||||||
|
issue. Verify the endpoint URL, access key, secret key, and bucket name in
|
||||||
|
the credential settings. Use `--insecure` in create_task.py only if you are
|
||||||
|
using a self-signed certificate on your S3 endpoint.
|
||||||
|
|||||||
+31
-12
@@ -1,27 +1,42 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# install.sh — run once on the TrueNAS box to set up the patch.
|
# install.sh — run once on the TrueNAS box to set up truecloud-patch.
|
||||||
|
#
|
||||||
|
# Prerequisites: run as root on TrueNAS SCALE with middlewared running.
|
||||||
#
|
#
|
||||||
# What this does:
|
# What this does:
|
||||||
# 1. Copies patch files to /data/truecloud-patch/ (survives updates).
|
# 1. Copies patch files to /data/truecloud-patch/ (survives OS updates).
|
||||||
# 2. Registers a PREINIT initshutdownscript so apply.sh runs on every boot
|
# 2. Registers a PREINIT initshutdownscript in the TrueNAS database so
|
||||||
# before middlewared starts, re-applying patches to the refreshed /usr/.
|
# apply.sh re-applies the patches on every boot before middlewared starts.
|
||||||
# 3. Applies the patches immediately without rebooting.
|
# 3. Applies the patches immediately (no reboot required).
|
||||||
# 4. Restarts middlewared so the backend patch takes effect now.
|
# 4. Restarts middlewared so the backend change takes effect now.
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
PATCH_DIR="/data/truecloud-patch"
|
PATCH_DIR="/data/truecloud-patch"
|
||||||
REPO_DIR="$(cd "$(dirname "$0")" && pwd)"
|
REPO_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
|
||||||
echo "=== TrueNAS TrueCloud Provider Patch ==="
|
echo "=== TrueNAS TrueCloud Provider Patch — Install ==="
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
# ── Preflight ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if [ "$(id -u)" -ne 0 ]; then
|
||||||
|
echo "ERROR: must be run as root." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
if ! command -v midclt &>/dev/null; then
|
if ! command -v midclt &>/dev/null; then
|
||||||
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
|
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if ! midclt call core.ping &>/dev/null; then
|
||||||
|
echo "ERROR: middlewared is not responding. Is TrueNAS fully booted?" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
# ── Copy files ────────────────────────────────────────────────────────────────
|
# ── Copy files ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
echo "Copying patch files to $PATCH_DIR ..."
|
echo "Copying patch files to $PATCH_DIR ..."
|
||||||
mkdir -p "$PATCH_DIR"
|
mkdir -p "$PATCH_DIR"
|
||||||
cp "$REPO_DIR/patch/sitecustomize.py" "$PATCH_DIR/"
|
cp "$REPO_DIR/patch/sitecustomize.py" "$PATCH_DIR/"
|
||||||
@@ -33,7 +48,8 @@ echo "Done."
|
|||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Register PREINIT script ───────────────────────────────────────────────────
|
# ── Register PREINIT script ───────────────────────────────────────────────────
|
||||||
echo "Checking initshutdownscript registration ..."
|
|
||||||
|
echo "Registering PREINIT boot hook ..."
|
||||||
|
|
||||||
EXISTING_ID=$(midclt call initshutdownscript.query '[]' | \
|
EXISTING_ID=$(midclt call initshutdownscript.query '[]' | \
|
||||||
python3 -c "
|
python3 -c "
|
||||||
@@ -52,22 +68,25 @@ else
|
|||||||
midclt call initshutdownscript.create \
|
midclt call initshutdownscript.create \
|
||||||
'{"type":"SCRIPT","script":"/data/truecloud-patch/apply.sh","when":"PREINIT","enabled":true,"comment":"TrueCloud provider patch (S3/B2)"}' \
|
'{"type":"SCRIPT","script":"/data/truecloud-patch/apply.sh","when":"PREINIT","enabled":true,"comment":"TrueCloud provider patch (S3/B2)"}' \
|
||||||
> /dev/null
|
> /dev/null
|
||||||
echo "Registered PREINIT script."
|
echo "Registered."
|
||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Apply now ─────────────────────────────────────────────────────────────────
|
# ── Apply now ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
echo "Applying patches ..."
|
echo "Applying patches ..."
|
||||||
bash "$PATCH_DIR/apply.sh"
|
bash "$PATCH_DIR/apply.sh"
|
||||||
cat "$PATCH_DIR/apply.log" | tail -20
|
echo ""
|
||||||
|
echo "Patch log ($PATCH_DIR/apply.log):"
|
||||||
|
tail -30 "$PATCH_DIR/apply.log"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Restart middlewared ───────────────────────────────────────────────────────
|
# ── Restart middlewared ───────────────────────────────────────────────────────
|
||||||
echo "Restarting middlewared (backend patch takes effect) ..."
|
|
||||||
|
echo "Restarting middlewared so the backend patch takes effect ..."
|
||||||
systemctl restart middlewared
|
systemctl restart middlewared
|
||||||
echo "Done."
|
echo "Done."
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo "Refresh your browser to pick up the UI change."
|
echo "Refresh your browser to pick up the UI change."
|
||||||
echo ""
|
echo ""
|
||||||
echo "To create a TrueCloud Backup task with S3 or B2 credentials:"
|
echo "To create a TrueCloud Backup task with S3 or B2 credentials:"
|
||||||
|
|||||||
+89
-29
@@ -1,44 +1,104 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# /data/truecloud-patch/apply.sh
|
# /data/truecloud-patch/apply.sh
|
||||||
#
|
#
|
||||||
# PREINIT script registered via TrueNAS initshutdownscript.
|
# Registered as a TrueNAS PREINIT initshutdownscript.
|
||||||
# Runs on every boot before middlewared starts, re-applying patches that
|
# Runs on every boot BEFORE middlewared starts, so patches land before
|
||||||
# TrueNAS updates wipe from /usr/.
|
# the first Python process for middlewared is created.
|
||||||
#
|
#
|
||||||
# Two things are patched:
|
# TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
|
||||||
# 1. sitecustomize.py → monkey-patches B2 restic support + URL fix into
|
#
|
||||||
# middlewared at Python startup time (backend)
|
# 1. sitecustomize.py — Python executes this automatically at startup.
|
||||||
# 2. Angular JS bundle → adds S3 and B2 to the credential dropdown in
|
# Monkey-patches B2 restic support and fixes the
|
||||||
# the TrueCloud Backup task form (UI)
|
# URL builder for empty-host providers.
|
||||||
|
#
|
||||||
set -euo pipefail
|
# 2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
|
||||||
|
# from Storj-only to include S3 and B2.
|
||||||
|
#
|
||||||
|
# Design principle: every step is independently fail-safe.
|
||||||
|
# A failed patch logs a warning and continues; middlewared always starts.
|
||||||
|
# Never use `set -e` in a PREINIT script.
|
||||||
|
|
||||||
PATCH_DIR="/data/truecloud-patch"
|
PATCH_DIR="/data/truecloud-patch"
|
||||||
LOG="$PATCH_DIR/apply.log"
|
LOG="$PATCH_DIR/apply.log"
|
||||||
|
|
||||||
{
|
# Rotate log at 512 KB to avoid unbounded growth on a system volume.
|
||||||
echo "=== $(date -Iseconds) ==="
|
if [ -f "$LOG" ] && [ "$(wc -c < "$LOG")" -gt 524288 ]; then
|
||||||
|
mv "$LOG" "${LOG}.1"
|
||||||
|
fi
|
||||||
|
|
||||||
# ── 1. Backend: install sitecustomize.py ─────────────────────────────
|
exec >> "$LOG" 2>&1
|
||||||
SITE_PKG=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
|
echo "=== $(date -Iseconds) ==="
|
||||||
|
|
||||||
if [ -z "$SITE_PKG" ]; then
|
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||||
echo "WARNING: could not determine site-packages path, skipping backend patch"
|
|
||||||
else
|
warn() { echo "WARNING: $*"; }
|
||||||
# If an unrelated sitecustomize.py exists, back it up once.
|
ok() { echo "OK: $*"; }
|
||||||
if [ -f "$SITE_PKG/sitecustomize.py" ] && \
|
|
||||||
! grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
# Find the Python interpreter that middlewared actually uses.
|
||||||
cp "$SITE_PKG/sitecustomize.py" "$SITE_PKG/sitecustomize.py.pre-truecloud-patch"
|
# On TrueNAS SCALE, /usr/bin/middlewared is usually a Python entry-point script
|
||||||
echo "Backed up existing sitecustomize.py"
|
# with a shebang pointing at the right interpreter (system or venv).
|
||||||
|
find_mw_python() {
|
||||||
|
local py="python3"
|
||||||
|
local shebang=""
|
||||||
|
|
||||||
|
if [ -x /usr/bin/middlewared ]; then
|
||||||
|
# Read the first line safely (max 256 bytes) — avoids reading a binary ELF
|
||||||
|
shebang=$(dd if=/usr/bin/middlewared bs=256 count=1 2>/dev/null | head -1 || true)
|
||||||
|
|
||||||
|
if [[ "$shebang" =~ ^'#!'(/[^[:space:]]+python[^[:space:]]*) ]]; then
|
||||||
|
py="${BASH_REMATCH[1]}"
|
||||||
|
elif [[ "$shebang" =~ ^'#!/usr/bin/env '(python[^[:space:]]*) ]]; then
|
||||||
|
py=$(command -v "${BASH_REMATCH[1]}" 2>/dev/null || echo "python3")
|
||||||
fi
|
fi
|
||||||
|
|
||||||
cp "$PATCH_DIR/sitecustomize.py" "$SITE_PKG/sitecustomize.py"
|
|
||||||
echo "Installed sitecustomize.py → $SITE_PKG/sitecustomize.py"
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ── 2. UI: patch Angular bundle ──────────────────────────────────────
|
# Verify the chosen interpreter can actually import middlewared.
|
||||||
python3 "$PATCH_DIR/patch_ui.py"
|
if ! "$py" -c "import middlewared" 2>/dev/null; then
|
||||||
|
warn "Detected Python '$py' cannot import middlewared; falling back to python3"
|
||||||
|
py="python3"
|
||||||
|
fi
|
||||||
|
|
||||||
echo "=== done ==="
|
echo "$py"
|
||||||
|
}
|
||||||
|
|
||||||
} >> "$LOG" 2>&1
|
# ── Step 1: sitecustomize.py ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "--- backend patch ---"
|
||||||
|
|
||||||
|
PYTHON=$(find_mw_python)
|
||||||
|
echo "Using Python: $PYTHON"
|
||||||
|
|
||||||
|
SITE_PKG=$("$PYTHON" -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
|
||||||
|
|
||||||
|
if [ -z "$SITE_PKG" ]; then
|
||||||
|
warn "Cannot determine site-packages directory; skipping backend patch."
|
||||||
|
warn "Verify that '$PYTHON -c \"import site; print(site.getsitepackages())\"' works."
|
||||||
|
else
|
||||||
|
# Back up any pre-existing sitecustomize.py that isn't ours.
|
||||||
|
if [ -f "$SITE_PKG/sitecustomize.py" ] && \
|
||||||
|
! grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
||||||
|
cp "$SITE_PKG/sitecustomize.py" \
|
||||||
|
"$SITE_PKG/sitecustomize.py.pre-truecloud-patch"
|
||||||
|
ok "Backed up existing sitecustomize.py"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if cp "$PATCH_DIR/sitecustomize.py" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
||||||
|
ok "Installed sitecustomize.py → $SITE_PKG/sitecustomize.py"
|
||||||
|
else
|
||||||
|
warn "Failed to write $SITE_PKG/sitecustomize.py (permission error?)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Step 2: Angular bundle ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "--- UI patch ---"
|
||||||
|
|
||||||
|
if "$PYTHON" "$PATCH_DIR/patch_ui.py"; then
|
||||||
|
: # patch_ui.py prints its own status
|
||||||
|
else
|
||||||
|
warn "patch_ui.py exited non-zero; UI dropdown may still show Storj only."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Done ──────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "=== done ==="
|
||||||
|
|||||||
+74
-47
@@ -2,34 +2,42 @@
|
|||||||
"""
|
"""
|
||||||
create_task.py — create TrueNAS TrueCloud Backup tasks with S3 or B2 credentials.
|
create_task.py — create TrueNAS TrueCloud Backup tasks with S3 or B2 credentials.
|
||||||
|
|
||||||
The TrueNAS UI restricts the credential dropdown to Storj only. This script
|
The TrueNAS UI normally restricts the credential dropdown to Storj only.
|
||||||
talks directly to the REST API so you can use any compatible credential.
|
This script bypasses that restriction by calling the REST API directly.
|
||||||
|
|
||||||
Requires: an API key from TrueNAS UI → System → API Keys.
|
Compatible providers (after the truecloud-patch backend patch is applied):
|
||||||
|
S3 — any S3-compatible endpoint (AWS, Wasabi, Cloudflare R2, MinIO, …)
|
||||||
|
B2 — Backblaze B2 native API
|
||||||
|
STORJ_IX — Storj (unchanged, always worked)
|
||||||
|
|
||||||
|
Requires a TrueNAS API key: UI → System → API Keys → Add.
|
||||||
|
|
||||||
Examples
|
Examples
|
||||||
--------
|
--------
|
||||||
List available cloud credentials:
|
List available cloud credentials:
|
||||||
python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
|
python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
|
||||||
|
|
||||||
Create a task backed by a B2 credential (id=3):
|
Create a task backed by a B2 credential (id=3):
|
||||||
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
|
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
|
||||||
--name "tank-to-b2" \\
|
--name "tank-to-b2" \\
|
||||||
--path /mnt/tank/data \\
|
--path /mnt/tank/data \\
|
||||||
--credential 3 \\
|
--credential 3 \\
|
||||||
--bucket my-bucket \\
|
--bucket my-bucket \\
|
||||||
--folder backups/tank \\
|
--folder backups/tank \\
|
||||||
--password "restic-repo-password" \\
|
--password "restic-repo-password" \\
|
||||||
--keep-last 14
|
--keep-last 14
|
||||||
|
|
||||||
Create a task using an S3-compatible credential (Wasabi, R2, etc.):
|
Create a task using an S3-compatible credential (Wasabi, R2, etc.):
|
||||||
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
|
python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
|
||||||
--name "tank-to-wasabi" \\
|
--name "tank-to-wasabi" \\
|
||||||
--path /mnt/tank/data \\
|
--path /mnt/tank/data \\
|
||||||
--credential 5 \\
|
--credential 5 \\
|
||||||
--bucket my-bucket \\
|
--bucket my-bucket \\
|
||||||
--folder backups \\
|
--folder backups \\
|
||||||
--password "restic-repo-password"
|
--password "restic-repo-password"
|
||||||
|
|
||||||
|
List existing TrueCloud Backup tasks:
|
||||||
|
python3 create_task.py --host 192.168.1.1 --api-key <key> list-tasks
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
@@ -41,6 +49,7 @@ import urllib.request
|
|||||||
|
|
||||||
|
|
||||||
def make_client(host, api_key, insecure=False):
|
def make_client(host, api_key, insecure=False):
|
||||||
|
"""Return a callable that makes authenticated REST API calls."""
|
||||||
base = f"https://{host}/api/v2.0"
|
base = f"https://{host}/api/v2.0"
|
||||||
headers = {
|
headers = {
|
||||||
"Authorization": f"Bearer {api_key}",
|
"Authorization": f"Bearer {api_key}",
|
||||||
@@ -59,7 +68,7 @@ def make_client(host, api_key, insecure=False):
|
|||||||
with urllib.request.urlopen(req, context=ctx) as resp:
|
with urllib.request.urlopen(req, context=ctx) as resp:
|
||||||
return json.loads(resp.read())
|
return json.loads(resp.read())
|
||||||
except urllib.error.HTTPError as exc:
|
except urllib.error.HTTPError as exc:
|
||||||
detail = exc.read().decode()
|
detail = exc.read().decode(errors="replace")
|
||||||
print(f"HTTP {exc.code} {exc.reason}: {detail}", file=sys.stderr)
|
print(f"HTTP {exc.code} {exc.reason}: {detail}", file=sys.stderr)
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
except urllib.error.URLError as exc:
|
except urllib.error.URLError as exc:
|
||||||
@@ -69,13 +78,15 @@ def make_client(host, api_key, insecure=False):
|
|||||||
return call
|
return call
|
||||||
|
|
||||||
|
|
||||||
|
# ── Sub-commands ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def cmd_list_credentials(client, _args):
|
def cmd_list_credentials(client, _args):
|
||||||
creds = client("GET", "/cloudsync/credentials")
|
creds = client("GET", "/cloudsync/credentials")
|
||||||
if not creds:
|
if not creds:
|
||||||
print("No cloud credentials configured.")
|
print("No cloud credentials configured.")
|
||||||
return
|
return
|
||||||
print(f"{'ID':>4} {'Provider':<14} Name")
|
print(f"{'ID':>4} {'Provider':<14} Name")
|
||||||
print("─" * 50)
|
print("─" * 55)
|
||||||
for c in sorted(creds, key=lambda x: x["id"]):
|
for c in sorted(creds, key=lambda x: x["id"]):
|
||||||
print(f"{c['id']:>4} {c['provider']['type']:<14} {c['name']}")
|
print(f"{c['id']:>4} {c['provider']['type']:<14} {c['name']}")
|
||||||
|
|
||||||
@@ -86,17 +97,20 @@ def cmd_list_tasks(client, _args):
|
|||||||
print("No TrueCloud Backup tasks configured.")
|
print("No TrueCloud Backup tasks configured.")
|
||||||
return
|
return
|
||||||
print(f"{'ID':>4} {'Enabled':<8} {'Provider':<14} Name")
|
print(f"{'ID':>4} {'Enabled':<8} {'Provider':<14} Name")
|
||||||
print("─" * 55)
|
print("─" * 60)
|
||||||
for t in sorted(tasks, key=lambda x: x["id"]):
|
for t in sorted(tasks, key=lambda x: x["id"]):
|
||||||
ptype = t["credentials"]["provider"]["type"] if t.get("credentials") else "?"
|
ptype = (t.get("credentials") or {}).get("provider", {}).get("type", "?")
|
||||||
enabled = "yes" if t.get("enabled") else "no"
|
enabled = "yes" if t.get("enabled") else "no"
|
||||||
print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t['description']}")
|
print(f"{t['id']:>4} {enabled:<8} {ptype:<14} {t.get('description', '')}")
|
||||||
|
|
||||||
|
|
||||||
def cmd_create(client, args):
|
def cmd_create(client, args):
|
||||||
parts = args.schedule.split()
|
parts = args.schedule.split()
|
||||||
if len(parts) != 5:
|
if len(parts) != 5:
|
||||||
print("--schedule must be a 5-field cron expression, e.g. '0 2 * * *'", file=sys.stderr)
|
print(
|
||||||
|
"ERROR: --schedule must be a 5-field cron expression, e.g. '0 2 * * *'",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
minute, hour, dom, month, dow = parts
|
minute, hour, dom, month, dow = parts
|
||||||
|
|
||||||
@@ -127,39 +141,52 @@ def cmd_create(client, args):
|
|||||||
print(f"Created task id={result['id']} name={result['description']!r}")
|
print(f"Created task id={result['id']} name={result['description']!r}")
|
||||||
|
|
||||||
|
|
||||||
|
# ── CLI ───────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
p = argparse.ArgumentParser(
|
p = argparse.ArgumentParser(
|
||||||
description="Manage TrueNAS TrueCloud Backup tasks (S3/B2/Storj)",
|
description="Manage TrueNAS TrueCloud Backup tasks (S3 / B2 / Storj)",
|
||||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||||
|
epilog=__doc__.split("Examples")[1] if "Examples" in __doc__ else "",
|
||||||
)
|
)
|
||||||
p.add_argument("--host", required=True, metavar="HOST", help="TrueNAS hostname or IP")
|
p.add_argument("--host", required=True, metavar="HOST",
|
||||||
p.add_argument("--api-key", required=True, metavar="KEY", help="TrueNAS API key")
|
help="TrueNAS hostname or IP address")
|
||||||
p.add_argument("--insecure", action="store_true", help="Skip TLS certificate verification")
|
p.add_argument("--api-key", required=True, metavar="KEY",
|
||||||
|
help="TrueNAS API key (System → API Keys)")
|
||||||
|
p.add_argument("--insecure", action="store_true",
|
||||||
|
help="Skip TLS certificate verification (self-signed certs)")
|
||||||
|
|
||||||
sub = p.add_subparsers(dest="cmd", required=True)
|
sub = p.add_subparsers(dest="cmd", required=True)
|
||||||
|
|
||||||
sub.add_parser("list-credentials", help="List configured cloud credentials")
|
sub.add_parser("list-credentials", help="List configured cloud credentials")
|
||||||
sub.add_parser("list-tasks", help="List TrueCloud Backup tasks")
|
sub.add_parser("list-tasks", help="List TrueCloud Backup tasks")
|
||||||
|
|
||||||
c = sub.add_parser("create", help="Create a TrueCloud Backup task")
|
c = sub.add_parser("create", help="Create a new TrueCloud Backup task")
|
||||||
c.add_argument("--name", required=True, help="Task description shown in the UI")
|
c.add_argument("--name", required=True,
|
||||||
c.add_argument("--path", required=True, help="Local dataset path, e.g. /mnt/tank/data")
|
help="Task description shown in the UI")
|
||||||
c.add_argument("--credential", required=True, type=int, metavar="ID",
|
c.add_argument("--path", required=True,
|
||||||
help="Cloud credential ID (from list-credentials)")
|
help="Local dataset path (e.g. /mnt/tank/data)")
|
||||||
c.add_argument("--bucket", required=True, help="Bucket or container name")
|
c.add_argument("--credential", required=True, type=int, metavar="ID",
|
||||||
c.add_argument("--folder", default="", help="Folder path within the bucket (default: root)")
|
help="Cloud credential ID — get it from list-credentials")
|
||||||
c.add_argument("--password", required=True, help="Restic repository encryption password")
|
c.add_argument("--bucket", required=True,
|
||||||
c.add_argument("--keep-last", type=int, default=14, metavar="N",
|
help="Bucket (S3) or container (B2) name")
|
||||||
|
c.add_argument("--folder", default="",
|
||||||
|
help="Path within the bucket (default: root)")
|
||||||
|
c.add_argument("--password", required=True,
|
||||||
|
help="Restic repository encryption password (choose a strong one)")
|
||||||
|
c.add_argument("--keep-last", type=int, default=14, metavar="N",
|
||||||
help="Snapshots to retain after each run (default: 14)")
|
help="Snapshots to retain after each run (default: 14)")
|
||||||
c.add_argument("--schedule", default="0 2 * * *",
|
c.add_argument("--schedule", default="0 2 * * *",
|
||||||
help="Cron schedule (default: '0 2 * * *' — daily at 02:00)")
|
help="Cron schedule (default: '0 2 * * *' — daily at 02:00)")
|
||||||
c.add_argument("--transfer-setting",
|
c.add_argument("--transfer-setting",
|
||||||
choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"], default="DEFAULT")
|
choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"],
|
||||||
c.add_argument("--snapshot", action="store_true",
|
default="DEFAULT",
|
||||||
help="Create a ZFS snapshot before each backup")
|
help="Pack-size / concurrency preset (default: DEFAULT)")
|
||||||
|
c.add_argument("--snapshot", action="store_true",
|
||||||
|
help="Create a ZFS snapshot before each backup run")
|
||||||
c.add_argument("--absolute-paths", action="store_true",
|
c.add_argument("--absolute-paths", action="store_true",
|
||||||
help="Preserve absolute paths inside the restic repo")
|
help="Preserve absolute paths inside the restic repository")
|
||||||
c.add_argument("--disabled", action="store_true",
|
c.add_argument("--disabled", action="store_true",
|
||||||
help="Create the task in a disabled state")
|
help="Create the task in a disabled state")
|
||||||
|
|
||||||
args = p.parse_args()
|
args = p.parse_args()
|
||||||
@@ -167,8 +194,8 @@ def main():
|
|||||||
|
|
||||||
dispatch = {
|
dispatch = {
|
||||||
"list-credentials": cmd_list_credentials,
|
"list-credentials": cmd_list_credentials,
|
||||||
"list-tasks": cmd_list_tasks,
|
"list-tasks": cmd_list_tasks,
|
||||||
"create": cmd_create,
|
"create": cmd_create,
|
||||||
}
|
}
|
||||||
dispatch[args.cmd](client, args)
|
dispatch[args.cmd](client, args)
|
||||||
|
|
||||||
|
|||||||
+66
-22
@@ -3,12 +3,20 @@
|
|||||||
Patches the TrueNAS webui Angular bundle to show S3 and B2 credentials
|
Patches the TrueNAS webui Angular bundle to show S3 and B2 credentials
|
||||||
in the TrueCloud Backup task form, instead of Storj only.
|
in the TrueCloud Backup task form, instead of Storj only.
|
||||||
|
|
||||||
The compiled bundle contains:
|
Angular's Ivy compiler inlines TypeScript string enum values as literals in
|
||||||
"filterByProviders",["STORJ_IX"]
|
the compiled bundle, so the template binding:
|
||||||
which is the template binding [filterByProviders]="[CloudSyncProviderName.Storj]".
|
|
||||||
We replace the array with ["STORJ_IX","S3","B2"] so all three providers appear.
|
|
||||||
|
|
||||||
Run automatically by apply.sh on every boot. Safe to run multiple times.
|
[filterByProviders]="[CloudSyncProviderName.Storj]"
|
||||||
|
|
||||||
|
appears verbatim in the minified JS as:
|
||||||
|
|
||||||
|
"filterByProviders",["STORJ_IX"]
|
||||||
|
|
||||||
|
We replace that array to include S3 and B2. The file is backed up before
|
||||||
|
modification so uninstall.sh can restore it.
|
||||||
|
|
||||||
|
Safe to run multiple times — a marker string detects an already-patched file.
|
||||||
|
Exits 0 in all cases (warnings are printed to stdout and logged by apply.sh).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
@@ -22,13 +30,13 @@ WEBUI_CANDIDATES = [
|
|||||||
"/var/www/truenas",
|
"/var/www/truenas",
|
||||||
]
|
]
|
||||||
|
|
||||||
# The pattern as compiled by Angular's Ivy into minified JS.
|
# Angular's Ivy template compiler serialises the Storj-only filter as this
|
||||||
# String literals survive minification; the function name before the comma
|
# exact substring in every production build we've observed.
|
||||||
# is mangled and not part of our match.
|
|
||||||
FIND = re.compile(r'("filterByProviders",)\["STORJ_IX"\]')
|
FIND = re.compile(r'("filterByProviders",)\["STORJ_IX"\]')
|
||||||
REPLACE = r'\1["STORJ_IX","S3","B2"]'
|
REPLACE = r'\1["STORJ_IX","S3","B2"]'
|
||||||
|
|
||||||
# Presence of this string means we already patched this file.
|
# A patched file contains both "S3" and "B2" next to "STORJ_IX" in this form.
|
||||||
|
# This string is specific enough not to appear elsewhere in the bundle.
|
||||||
MARKER = '"STORJ_IX","S3","B2"'
|
MARKER = '"STORJ_IX","S3","B2"'
|
||||||
|
|
||||||
|
|
||||||
@@ -40,8 +48,14 @@ def find_webui():
|
|||||||
|
|
||||||
|
|
||||||
def find_bundle(webui):
|
def find_bundle(webui):
|
||||||
for root, _, names in os.walk(webui):
|
"""
|
||||||
for name in names:
|
Walk the webui directory looking for the JS file that contains the
|
||||||
|
filterByProviders binding. Returns (path, content) or (None, None).
|
||||||
|
Only .js files are read; binary files and permission errors are skipped.
|
||||||
|
"""
|
||||||
|
matches = []
|
||||||
|
for root, _dirs, names in os.walk(webui):
|
||||||
|
for name in sorted(names): # deterministic order
|
||||||
if not name.endswith(".js"):
|
if not name.endswith(".js"):
|
||||||
continue
|
continue
|
||||||
path = os.path.join(root, name)
|
path = os.path.join(root, name)
|
||||||
@@ -49,41 +63,71 @@ def find_bundle(webui):
|
|||||||
with open(path) as fh:
|
with open(path) as fh:
|
||||||
content = fh.read()
|
content = fh.read()
|
||||||
if FIND.search(content):
|
if FIND.search(content):
|
||||||
return path, content
|
matches.append((path, content))
|
||||||
except (UnicodeDecodeError, PermissionError, OSError):
|
except (UnicodeDecodeError, PermissionError, OSError):
|
||||||
continue
|
continue
|
||||||
return None, None
|
|
||||||
|
if not matches:
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
if len(matches) > 1:
|
||||||
|
# Unexpected — log all matches so the operator can investigate.
|
||||||
|
print(
|
||||||
|
f"[truecloud-patch] WARNING: filterByProviders pattern found in "
|
||||||
|
f"{len(matches)} files; patching only the first."
|
||||||
|
)
|
||||||
|
for p, _ in matches:
|
||||||
|
print(f"[truecloud-patch] {p}")
|
||||||
|
|
||||||
|
return matches[0]
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
webui = find_webui()
|
webui = find_webui()
|
||||||
if not webui:
|
if not webui:
|
||||||
print("[truecloud-patch] WARNING: webui directory not found, skipping UI patch")
|
print(
|
||||||
sys.exit(0)
|
"[truecloud-patch] WARNING: webui directory not found; skipping UI patch.\n"
|
||||||
|
"[truecloud-patch] Searched: " + ", ".join(WEBUI_CANDIDATES)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
path, content = find_bundle(webui)
|
path, content = find_bundle(webui)
|
||||||
if path is None:
|
if path is None:
|
||||||
print(
|
print(
|
||||||
"[truecloud-patch] WARNING: filterByProviders pattern not found in webui bundle.\n"
|
"[truecloud-patch] WARNING: filterByProviders pattern not found in any JS bundle.\n"
|
||||||
"[truecloud-patch] The UI patch may need updating for this TrueNAS version.\n"
|
"[truecloud-patch] The TrueNAS webui may have been restructured in this version.\n"
|
||||||
"[truecloud-patch] File an issue at https://github.com/sudolulo/truenas-truecloud-patch"
|
"[truecloud-patch] File an issue at https://github.com/sudolulo/truenas-truecloud-patch\n"
|
||||||
|
f"[truecloud-patch] TrueNAS version info: {_tnversion()}"
|
||||||
)
|
)
|
||||||
sys.exit(0)
|
return
|
||||||
|
|
||||||
if MARKER in content:
|
if MARKER in content:
|
||||||
print(f"[truecloud-patch] UI already patched: {path}")
|
print(f"[truecloud-patch] UI already patched: {path}")
|
||||||
sys.exit(0)
|
return
|
||||||
|
|
||||||
backup = path + ".pre-truecloud-patch"
|
backup = path + ".pre-truecloud-patch"
|
||||||
if not os.path.exists(backup):
|
if not os.path.exists(backup):
|
||||||
shutil.copy2(path, backup)
|
shutil.copy2(path, backup)
|
||||||
|
|
||||||
patched, count = FIND.subn(REPLACE, content)
|
patched, count = FIND.subn(REPLACE, content)
|
||||||
with open(path, "w") as fh:
|
|
||||||
fh.write(patched)
|
try:
|
||||||
|
with open(path, "w") as fh:
|
||||||
|
fh.write(patched)
|
||||||
|
except OSError as exc:
|
||||||
|
print(f"[truecloud-patch] ERROR: Could not write {path}: {exc}")
|
||||||
|
return
|
||||||
|
|
||||||
print(f"[truecloud-patch] UI bundle patched ({count} replacement(s)): {path}")
|
print(f"[truecloud-patch] UI bundle patched ({count} replacement(s)): {path}")
|
||||||
|
|
||||||
|
|
||||||
|
def _tnversion():
|
||||||
|
try:
|
||||||
|
with open("/etc/version") as fh:
|
||||||
|
return fh.read().strip()
|
||||||
|
except OSError:
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
||||||
|
|||||||
+108
-58
@@ -2,78 +2,113 @@
|
|||||||
TrueCloud provider patch — sitecustomize.py
|
TrueCloud provider patch — sitecustomize.py
|
||||||
Installed into Python site-packages on every boot by apply.sh.
|
Installed into Python site-packages on every boot by apply.sh.
|
||||||
|
|
||||||
Hooks the import of two middlewared modules and patches them in-place:
|
Hooks two middlewared module imports using the find_spec / exec_module API
|
||||||
|
(required for Python 3.12+, which ships with TrueNAS SCALE 25.x / Debian 13):
|
||||||
|
|
||||||
middlewared.rclone.remote.b2
|
middlewared.rclone.remote.b2
|
||||||
Adds get_restic_config() so the native B2 restic backend works.
|
Adds get_restic_config() so the native restic B2 backend works.
|
||||||
Restic URL: b2:<bucket>/<folder>
|
Restic repo URL: b2:<bucket>/<folder>
|
||||||
Auth env: B2_ACCOUNT_ID, B2_ACCOUNT_KEY
|
Auth: B2_ACCOUNT_ID, B2_ACCOUNT_KEY
|
||||||
|
|
||||||
middlewared.plugins.cloud_backup.restic
|
middlewared.plugins.cloud_backup.restic
|
||||||
Fixes URL construction for providers that have no hostname component
|
Fixes the URL builder for providers with no hostname component.
|
||||||
(url == ""). Stock code builds "b2:/bucket/path" (broken double-slash);
|
Stock code: f"{rclone_type}:{url}/{remote_path}" → "b2:/bucket/path" (broken)
|
||||||
patched code builds "b2:bucket/path".
|
Patched: "b2:bucket/path" when url == ""
|
||||||
|
|
||||||
Safe for all Python processes on the system: if middlewared is absent the
|
Both patches are no-ops if the module already provides the functionality
|
||||||
hook installs but never fires, and all errors are caught and logged to stderr.
|
(i.e. a future TrueNAS version adds native support). All errors are caught
|
||||||
|
and written to stderr so middlewared always starts regardless of patch state.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
|
|
||||||
def _install():
|
# ── Import hook ───────────────────────────────────────────────────────────────
|
||||||
_pending = {
|
|
||||||
|
class _Finder:
|
||||||
|
"""
|
||||||
|
find_spec-based meta path finder (Python 3.4+, required for 3.12+).
|
||||||
|
Intercepts specific module imports, loads them normally, then patches.
|
||||||
|
"""
|
||||||
|
|
||||||
|
_targets = frozenset({
|
||||||
"middlewared.rclone.remote.b2",
|
"middlewared.rclone.remote.b2",
|
||||||
"middlewared.plugins.cloud_backup.restic",
|
"middlewared.plugins.cloud_backup.restic",
|
||||||
}
|
})
|
||||||
_loading = set()
|
|
||||||
|
|
||||||
class _Hook:
|
def __init__(self):
|
||||||
def find_module(self, fullname, path=None): # noqa: ARG002
|
self._loading = set() # guards against re-entrant imports
|
||||||
if fullname in _pending and fullname not in _loading:
|
self._done = set() # modules already patched
|
||||||
return self
|
|
||||||
return None
|
|
||||||
|
|
||||||
def load_module(self, fullname):
|
def find_spec(self, fullname, path, target=None): # noqa: ARG002
|
||||||
if fullname in sys.modules:
|
import importlib.machinery
|
||||||
module = sys.modules[fullname]
|
if (
|
||||||
else:
|
fullname in self._targets
|
||||||
_loading.add(fullname)
|
and fullname not in self._done
|
||||||
try:
|
and fullname not in self._loading
|
||||||
__import__(fullname)
|
):
|
||||||
finally:
|
return importlib.machinery.ModuleSpec(fullname, _Loader(self, fullname))
|
||||||
_loading.discard(fullname)
|
return None
|
||||||
module = sys.modules[fullname]
|
|
||||||
|
|
||||||
_pending.discard(fullname)
|
|
||||||
|
|
||||||
|
def _mark_done(self, fullname):
|
||||||
|
self._done.add(fullname)
|
||||||
|
if self._done >= self._targets:
|
||||||
try:
|
try:
|
||||||
if fullname == "middlewared.rclone.remote.b2":
|
sys.meta_path.remove(self)
|
||||||
_patch_b2(module)
|
except ValueError:
|
||||||
elif fullname == "middlewared.plugins.cloud_backup.restic":
|
pass
|
||||||
_patch_restic(module)
|
|
||||||
except Exception as exc:
|
|
||||||
sys.stderr.write(f"[truecloud-patch] patch failed for {fullname}: {exc}\n")
|
|
||||||
|
|
||||||
if not _pending:
|
|
||||||
try:
|
|
||||||
sys.meta_path.remove(hook)
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return module
|
class _Loader:
|
||||||
|
def __init__(self, finder, fullname):
|
||||||
|
self._finder = finder
|
||||||
|
self._fullname = fullname
|
||||||
|
|
||||||
hook = _Hook()
|
def create_module(self, spec): # noqa: ARG002
|
||||||
sys.meta_path.append(hook)
|
return None # use Python's default module creation
|
||||||
|
|
||||||
|
def exec_module(self, module):
|
||||||
|
import importlib.util
|
||||||
|
|
||||||
|
fullname = self._fullname
|
||||||
|
self._finder._loading.add(fullname)
|
||||||
|
try:
|
||||||
|
# find_spec for the real file — our finder returns None while
|
||||||
|
# fullname is in _loading, so the normal finders handle this.
|
||||||
|
real_spec = importlib.util.find_spec(fullname)
|
||||||
|
if real_spec is None:
|
||||||
|
raise ImportError(f"No module named {fullname!r}")
|
||||||
|
|
||||||
|
real_spec.loader.exec_module(module)
|
||||||
|
|
||||||
|
# Fix module metadata so it looks like a normal import.
|
||||||
|
module.__spec__ = real_spec
|
||||||
|
module.__loader__ = real_spec.loader
|
||||||
|
if getattr(real_spec, "origin", None):
|
||||||
|
module.__file__ = real_spec.origin
|
||||||
|
finally:
|
||||||
|
self._finder._loading.discard(fullname)
|
||||||
|
|
||||||
|
self._finder._mark_done(fullname)
|
||||||
|
|
||||||
|
try:
|
||||||
|
_PATCHES[fullname](module)
|
||||||
|
except Exception as exc:
|
||||||
|
sys.stderr.write(
|
||||||
|
f"[truecloud-patch] patch failed for {fullname}: {exc}\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Patch functions ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def _patch_b2(module):
|
def _patch_b2(module):
|
||||||
cls = module.B2RcloneRemote
|
cls = module.B2RcloneRemote
|
||||||
|
|
||||||
if hasattr(cls, "get_restic_config"):
|
if hasattr(cls, "get_restic_config"):
|
||||||
return # future TrueNAS version already added it
|
# A future TrueNAS version already added native B2 restic support.
|
||||||
|
return
|
||||||
|
|
||||||
def get_restic_config(self, task):
|
def get_restic_config(self, task): # noqa: ARG001
|
||||||
p = task["credentials"]["provider"]
|
p = task["credentials"]["provider"]
|
||||||
return "", {
|
return "", {
|
||||||
"B2_ACCOUNT_ID": p["account"],
|
"B2_ACCOUNT_ID": p["account"],
|
||||||
@@ -90,15 +125,18 @@ def _patch_restic(module):
|
|||||||
if getattr(orig, "_truecloud_patched", False):
|
if getattr(orig, "_truecloud_patched", False):
|
||||||
return
|
return
|
||||||
|
|
||||||
# Capture module-level references; REMOTES is the same mutable dict
|
# ResticConfig is safe to capture now (it's a dataclass defined in the module).
|
||||||
# object that remotes.setup() will populate later.
|
# REMOTES and get_remote_path are imported lazily inside the function so that
|
||||||
_REMOTES = module.REMOTES
|
# module layout changes in future middlewared versions fail at call time
|
||||||
_get_remote_path = module.get_remote_path
|
# (during an actual backup job) rather than silently at patch time.
|
||||||
_ResticConfig = module.ResticConfig
|
_ResticConfig = module.ResticConfig
|
||||||
|
|
||||||
def get_restic_config(cloud_backup):
|
def get_restic_config(cloud_backup):
|
||||||
remote = _REMOTES[cloud_backup["credentials"]["provider"]["type"]]
|
from middlewared.plugins.cloud.path import get_remote_path
|
||||||
remote_path = _get_remote_path(remote, cloud_backup["attributes"])
|
from middlewared.plugins.cloud.remotes import REMOTES
|
||||||
|
|
||||||
|
remote = REMOTES[cloud_backup["credentials"]["provider"]["type"]]
|
||||||
|
remote_path = get_remote_path(remote, cloud_backup["attributes"])
|
||||||
url, env = remote.get_restic_config(cloud_backup)
|
url, env = remote.get_restic_config(cloud_backup)
|
||||||
|
|
||||||
if cloud_backup["cache_path"]:
|
if cloud_backup["cache_path"]:
|
||||||
@@ -106,8 +144,7 @@ def _patch_restic(module):
|
|||||||
else:
|
else:
|
||||||
cache = ["--no-cache"]
|
cache = ["--no-cache"]
|
||||||
|
|
||||||
# Fix: stock code does f"{rclone_type}:{url}/{remote_path}" which
|
# Stock code produces "b2:/bucket/path" when url == "" (double-slash).
|
||||||
# produces "b2:/bucket/path" when url is empty.
|
|
||||||
repo = (
|
repo = (
|
||||||
f"{remote.rclone_type}:{url}/{remote_path}"
|
f"{remote.rclone_type}:{url}/{remote_path}"
|
||||||
if url
|
if url
|
||||||
@@ -122,9 +159,22 @@ def _patch_restic(module):
|
|||||||
sys.stderr.write("[truecloud-patch] restic URL fix applied\n")
|
sys.stderr.write("[truecloud-patch] restic URL fix applied\n")
|
||||||
|
|
||||||
|
|
||||||
try:
|
_PATCHES = {
|
||||||
|
"middlewared.rclone.remote.b2": _patch_b2,
|
||||||
|
"middlewared.plugins.cloud_backup.restic": _patch_restic,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── Entry point ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def _install():
|
||||||
import importlib.util
|
import importlib.util
|
||||||
if importlib.util.find_spec("middlewared") is not None:
|
if importlib.util.find_spec("middlewared") is None:
|
||||||
_install()
|
return # not a middlewared Python process; nothing to do
|
||||||
|
sys.meta_path.append(_Finder())
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
_install()
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass # never raise from sitecustomize.py — it would prevent Python from starting
|
||||||
|
|||||||
+41
-13
@@ -1,5 +1,5 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# uninstall.sh — removes all traces of the patch from a TrueNAS box.
|
# uninstall.sh — remove all traces of truecloud-patch from a TrueNAS box.
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
@@ -8,12 +8,20 @@ PATCH_DIR="/data/truecloud-patch"
|
|||||||
echo "=== TrueNAS TrueCloud Provider Patch — Uninstall ==="
|
echo "=== TrueNAS TrueCloud Provider Patch — Uninstall ==="
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
if [ "$(id -u)" -ne 0 ]; then
|
||||||
|
echo "ERROR: must be run as root." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
if ! command -v midclt &>/dev/null; then
|
if ! command -v midclt &>/dev/null; then
|
||||||
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
|
echo "ERROR: midclt not found. Run this script on TrueNAS SCALE." >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ── Remove PREINIT registration ───────────────────────────────────────────────
|
# ── Remove PREINIT hook ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "Removing PREINIT boot hook ..."
|
||||||
|
|
||||||
IDS=$(midclt call initshutdownscript.query '[]' | \
|
IDS=$(midclt call initshutdownscript.query '[]' | \
|
||||||
python3 -c "
|
python3 -c "
|
||||||
import sys, json
|
import sys, json
|
||||||
@@ -25,46 +33,67 @@ for s in json.load(sys.stdin):
|
|||||||
if [ -n "$IDS" ]; then
|
if [ -n "$IDS" ]; then
|
||||||
for id in $IDS; do
|
for id in $IDS; do
|
||||||
midclt call initshutdownscript.delete "$id" > /dev/null
|
midclt call initshutdownscript.delete "$id" > /dev/null
|
||||||
echo "Removed initshutdownscript id=$id"
|
echo " Removed initshutdownscript id=$id"
|
||||||
done
|
done
|
||||||
else
|
else
|
||||||
echo "No initshutdownscript entry found (already removed or never installed)."
|
echo " No entry found (already removed or never installed)."
|
||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Remove sitecustomize.py ───────────────────────────────────────────────────
|
# ── Remove sitecustomize.py ───────────────────────────────────────────────────
|
||||||
SITE_PKG=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
|
|
||||||
|
echo "Removing sitecustomize.py ..."
|
||||||
|
|
||||||
|
# Use the same Python detection logic as apply.sh
|
||||||
|
PYTHON="python3"
|
||||||
|
if [ -x /usr/bin/middlewared ]; then
|
||||||
|
shebang=$(dd if=/usr/bin/middlewared bs=256 count=1 2>/dev/null | head -1 || true)
|
||||||
|
if [[ "$shebang" =~ ^'#!'(/[^[:space:]]+python[^[:space:]]*) ]]; then
|
||||||
|
PYTHON="${BASH_REMATCH[1]}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
SITE_PKG=$("$PYTHON" -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || true)
|
||||||
|
|
||||||
if [ -n "$SITE_PKG" ] && [ -f "$SITE_PKG/sitecustomize.py" ]; then
|
if [ -n "$SITE_PKG" ] && [ -f "$SITE_PKG/sitecustomize.py" ]; then
|
||||||
if grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
if grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
||||||
rm "$SITE_PKG/sitecustomize.py"
|
rm "$SITE_PKG/sitecustomize.py"
|
||||||
echo "Removed $SITE_PKG/sitecustomize.py"
|
echo " Removed $SITE_PKG/sitecustomize.py"
|
||||||
|
|
||||||
# Restore a pre-existing sitecustomize.py if we backed one up
|
|
||||||
if [ -f "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" ]; then
|
if [ -f "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" ]; then
|
||||||
mv "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" \
|
mv "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" \
|
||||||
"$SITE_PKG/sitecustomize.py"
|
"$SITE_PKG/sitecustomize.py"
|
||||||
echo "Restored previous sitecustomize.py"
|
echo " Restored previous sitecustomize.py"
|
||||||
fi
|
fi
|
||||||
|
else
|
||||||
|
echo " $SITE_PKG/sitecustomize.py is not ours; leaving it alone."
|
||||||
fi
|
fi
|
||||||
|
else
|
||||||
|
echo " Not found (already removed or install didn't place it here)."
|
||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Restore UI bundle backup ──────────────────────────────────────────────────
|
# ── Restore UI bundle ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
echo "Restoring UI bundle backup ..."
|
||||||
|
|
||||||
RESTORED=0
|
RESTORED=0
|
||||||
for backup in $(find /usr/share/truenas /var/www/truenas -name "*.js.pre-truecloud-patch" 2>/dev/null); do
|
for backup in $(find /usr/share/truenas /var/www/truenas \
|
||||||
|
-name "*.js.pre-truecloud-patch" 2>/dev/null); do
|
||||||
original="${backup%.pre-truecloud-patch}"
|
original="${backup%.pre-truecloud-patch}"
|
||||||
mv "$backup" "$original"
|
mv "$backup" "$original"
|
||||||
echo "Restored: $original"
|
echo " Restored: $original"
|
||||||
RESTORED=1
|
RESTORED=1
|
||||||
done
|
done
|
||||||
|
|
||||||
if [ "$RESTORED" -eq 0 ]; then
|
if [ "$RESTORED" -eq 0 ]; then
|
||||||
echo "No UI bundle backups found (patch will be undone by the next TrueNAS update)."
|
echo " No backup files found."
|
||||||
|
echo " The UI patch will be undone automatically by the next TrueNAS update."
|
||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Remove patch directory ────────────────────────────────────────────────────
|
# ── Remove patch directory ────────────────────────────────────────────────────
|
||||||
|
|
||||||
if [ -d "$PATCH_DIR" ]; then
|
if [ -d "$PATCH_DIR" ]; then
|
||||||
rm -rf "$PATCH_DIR"
|
rm -rf "$PATCH_DIR"
|
||||||
echo "Removed $PATCH_DIR"
|
echo "Removed $PATCH_DIR"
|
||||||
@@ -73,6 +102,5 @@ echo ""
|
|||||||
|
|
||||||
echo "Restarting middlewared ..."
|
echo "Restarting middlewared ..."
|
||||||
systemctl restart middlewared
|
systemctl restart middlewared
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "Uninstall complete. Refresh your browser to see the restored UI."
|
echo "Uninstall complete. Refresh your browser to see the restored UI."
|
||||||
|
|||||||
Reference in New Issue
Block a user