Switch to overlay-only patching; remove sitecustomize.py
Patches to b2.py and restic.py are applied directly in the overlayfs at PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders that succeeded or failed alongside the file patch every time, providing no genuine fallback. - Delete patch/sitecustomize.py entirely - apply.sh: remove sitecustomize install step; flatten if/elif/else structure; restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay tag 'sc' -> 'mw' - recover.sh: unmount overlays to restore original files immediately, no reboot required; kill-switch file prevents re-application on next boot - uninstall.sh: remove sitecustomize.py removal section; update overlay tag - install.sh: update preflight to check patch/apply.sh, not sitecustomize.py - README: remove sitecustomize references throughout; update recovery docs
This commit is contained in:
@@ -59,7 +59,7 @@ on every update) and are therefore re-applied automatically on every boot.
|
|||||||
|
|
||||||
| Layer | What changes | Technique |
|
| Layer | What changes | Technique |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | Direct file patch in the overlay (primary, delegates URL logic to `sitecustomize.py`) + `sitecustomize.py` import hook (belt-and-suspenders) |
|
| **Backend** | `B2RcloneRemote` gains `get_restic_config()` — skipped automatically if TrueNAS already provides one on the class. `restic.py` URL builder is fixed: strips the stray leading slash and converts the slash separator to a colon (`b2:bucket:path`), which is the format restic 0.16.x expects. URL wrapper is a no-op if the URL is already correctly formed. | Direct file patch in the overlay |
|
||||||
| **UI** | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS chunk; original is backed up |
|
| **UI** | The Angular bundle's `filterByProviders` binding is widened from `["STORJ_IX"]` to `["STORJ_IX","S3","B2"]` | In-place text replacement in the compiled JS chunk; original is backed up |
|
||||||
|
|
||||||
Both changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
|
Both changes are **fail-safe**: if a patch cannot be applied (e.g. TrueNAS
|
||||||
@@ -80,20 +80,14 @@ TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by keeping
|
|||||||
this repository on a **persistent ZFS pool** (your data pool, not `/tmp` or a
|
this repository on a **persistent ZFS pool** (your data pool, not `/tmp` or a
|
||||||
system path) and registering a **PREINIT initshutdownscript** in the TrueNAS
|
system path) and registering a **PREINIT initshutdownscript** in the TrueNAS
|
||||||
database. On every boot, `patch/apply.sh` runs from the repo before
|
database. On every boot, `patch/apply.sh` runs from the repo before
|
||||||
`middlewared` starts, placing `sitecustomize.py` in the correct site-packages
|
`middlewared` starts, patching `b2.py` and `restic.py` directly in the overlay
|
||||||
directory and re-patching the UI bundle.
|
and re-patching the UI bundle.
|
||||||
|
|
||||||
If `/usr` is a read-only filesystem, `apply.sh` handles this automatically by
|
If `/usr` is a read-only filesystem, `apply.sh` handles this automatically by
|
||||||
mounting a writable [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html)
|
mounting a writable [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html)
|
||||||
on top of the relevant directories. The overlay lives in `/run` (tmpfs) and is
|
on top of the relevant directories. The overlay lives in `/run` (tmpfs) and is
|
||||||
recreated on every boot. No extra configuration is needed.
|
recreated on every boot. No extra configuration is needed.
|
||||||
|
|
||||||
## Python version compatibility
|
|
||||||
|
|
||||||
`sitecustomize.py` uses the `find_spec` / `exec_module` import hook API
|
|
||||||
(Python 3.4+; the older `load_module` form was removed in Python 3.12).
|
|
||||||
Compatible with all Python versions shipped by TrueNAS SCALE.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
@@ -162,8 +156,8 @@ bash /mnt/tank/truenas-truecloud-patch/uninstall.sh
|
|||||||
```
|
```
|
||||||
|
|
||||||
Replace the path with your clone location. Removes the PREINIT hook,
|
Replace the path with your clone location. Removes the PREINIT hook,
|
||||||
`sitecustomize.py`, and restores the original UI bundle from backup. The
|
unmounts the overlay (restoring the original backend files immediately),
|
||||||
backend changes vanish on the next `middlewared` restart.
|
and restores the original UI bundle from backup.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -305,9 +299,8 @@ bash /mnt/tank/truenas-truecloud-patch/recover.sh
|
|||||||
```
|
```
|
||||||
|
|
||||||
Replace the path with your clone location. This creates a kill-switch file
|
Replace the path with your clone location. This creates a kill-switch file
|
||||||
(`disabled`) in the repo root. `sitecustomize.py` checks for it at Python
|
(`disabled`) in the repo root, unmounts the overlay so the original files are
|
||||||
startup; if present, the import hook is skipped entirely and middlewared starts
|
visible immediately, then restarts middlewared. No reboot required.
|
||||||
clean with Storj-only support. Nothing else on your system is affected.
|
|
||||||
|
|
||||||
If you cannot run a script and only have a bare shell prompt:
|
If you cannot run a script and only have a bare shell prompt:
|
||||||
|
|
||||||
@@ -407,10 +400,9 @@ cat /mnt/tank/truenas-truecloud-patch/apply.log
|
|||||||
```bash
|
```bash
|
||||||
python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
|
python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
|
||||||
```
|
```
|
||||||
This reads `hook_status.json` in your repo root. The file is written by
|
Reads `hook_status.json` written by `apply.sh` at boot. Reflects whether the
|
||||||
`apply.sh` at install/boot time and reflects whether the direct file patches
|
overlay patches to `b2.py` and `restic.py` were applied successfully. Does not
|
||||||
to `b2.py` and `restic.py` were applied successfully. Does not require
|
require `--host` or `--api-key`.
|
||||||
`--host` or `--api-key`.
|
|
||||||
|
|
||||||
**Middlewared log:**
|
**Middlewared log:**
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
+1
-1
@@ -20,7 +20,7 @@ set -euo pipefail
|
|||||||
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
|
PATCH_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
_HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
|
_HOOK_COMMENT='TrueCloud provider patch (S3/B2)'
|
||||||
|
|
||||||
if [ ! -f "$PATCH_DIR/patch/sitecustomize.py" ]; then
|
if [ ! -f "$PATCH_DIR/patch/apply.sh" ]; then
|
||||||
echo "ERROR: patch files not found at $PATCH_DIR/patch/" >&2
|
echo "ERROR: patch files not found at $PATCH_DIR/patch/" >&2
|
||||||
echo "Run install.sh from a clone of the repository on a persistent pool:" >&2
|
echo "Run install.sh from a clone of the repository on a persistent pool:" >&2
|
||||||
echo " git clone https://github.com/sudolulo/truenas-truecloud-patch \\" >&2
|
echo " git clone https://github.com/sudolulo/truenas-truecloud-patch \\" >&2
|
||||||
|
|||||||
+38
-67
@@ -7,7 +7,6 @@
|
|||||||
# TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
|
# TrueNAS updates replace /usr/ entirely; this script re-applies two patches:
|
||||||
#
|
#
|
||||||
# 1. Backend — b2.py and restic.py are patched directly in the overlay.
|
# 1. Backend — b2.py and restic.py are patched directly in the overlay.
|
||||||
# sitecustomize.py is also installed as belt-and-suspenders.
|
|
||||||
#
|
#
|
||||||
# 2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
|
# 2. Angular JS bundle — Widens the TrueCloud Backup credential dropdown
|
||||||
# from Storj-only to include S3 and B2.
|
# from Storj-only to include S3 and B2.
|
||||||
@@ -39,10 +38,9 @@ if [ -f "$PATCH_DIR/disabled" ]; then
|
|||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# On TrueNAS 25.x+, /usr is an immutable read-only filesystem.
|
# Mounts a writable overlayfs on $1 using /run (tmpfs) for the upper/work dirs
|
||||||
# This function mounts a writable overlayfs on $1 using /run (tmpfs) for the
|
# when the directory is read-only. The overlay is volatile per boot; this
|
||||||
# upper/work dirs. The overlay is volatile per boot; this PREINIT script
|
# PREINIT script recreates it on every boot before middlewared starts.
|
||||||
# recreates it on every boot before middlewared starts.
|
|
||||||
# Returns 0 if the directory is now writable, 1 if it could not be made so.
|
# Returns 0 if the directory is now writable, 1 if it could not be made so.
|
||||||
_ensure_writable() {
|
_ensure_writable() {
|
||||||
local dir="$1" tag="$2"
|
local dir="$1" tag="$2"
|
||||||
@@ -90,7 +88,7 @@ find_mw_python() {
|
|||||||
echo "WARNING: '$py' cannot import middlewared; falling back to python3" >&2
|
echo "WARNING: '$py' cannot import middlewared; falling back to python3" >&2
|
||||||
py="python3"
|
py="python3"
|
||||||
if ! "$py" -c "import middlewared" 2>/dev/null; then
|
if ! "$py" -c "import middlewared" 2>/dev/null; then
|
||||||
echo "WARNING: 'python3' also cannot import middlewared; sitecustomize.py may be installed in the wrong location" >&2
|
echo "WARNING: 'python3' also cannot import middlewared; backend patch will be skipped" >&2
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -105,10 +103,8 @@ PYTHON=$(find_mw_python)
|
|||||||
echo "Using Python: $PYTHON"
|
echo "Using Python: $PYTHON"
|
||||||
|
|
||||||
# Derive site-packages from where middlewared actually lives.
|
# Derive site-packages from where middlewared actually lives.
|
||||||
# On TrueNAS 25.x, middlewared is in /usr/lib/python3/dist-packages/, while
|
# getsitepackages()[0] may return the wrong directory; using middlewared.__file__
|
||||||
# getsitepackages()[0] typically returns /usr/local/lib/python3.11/dist-packages/
|
# ensures we patch files in the directory Python will actually read.
|
||||||
# — the wrong directory. Using middlewared.__file__ ensures we install
|
|
||||||
# sitecustomize.py and patch files in the directory Python will actually read.
|
|
||||||
SITE_PKG=$("$PYTHON" -c "
|
SITE_PKG=$("$PYTHON" -c "
|
||||||
import os
|
import os
|
||||||
try:
|
try:
|
||||||
@@ -137,52 +133,11 @@ _restic_ok=0
|
|||||||
if [ -z "$SITE_PKG" ]; then
|
if [ -z "$SITE_PKG" ]; then
|
||||||
echo "WARNING: Cannot determine site-packages directory; skipping backend patch."
|
echo "WARNING: Cannot determine site-packages directory; skipping backend patch."
|
||||||
echo " Run: $PYTHON -c \"import site; print(site.getsitepackages())\""
|
echo " Run: $PYTHON -c \"import site; print(site.getsitepackages())\""
|
||||||
|
elif ! _ensure_writable "$SITE_PKG" "mw"; then
|
||||||
|
echo "WARNING: Cannot make site-packages writable; skipping backend patch."
|
||||||
|
elif [ -z "$_MW_DIR" ]; then
|
||||||
|
echo "WARNING: Cannot determine middlewared directory; skipping backend patch."
|
||||||
else
|
else
|
||||||
_can_install=true
|
|
||||||
# On immutable OS, ensure site-packages is writable via overlay before
|
|
||||||
# attempting any writes.
|
|
||||||
if ! _ensure_writable "$SITE_PKG" "sc"; then
|
|
||||||
_can_install=false
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Back up any pre-existing sitecustomize.py that isn't ours.
|
|
||||||
if [ "$_can_install" = true ] && \
|
|
||||||
[ -f "$SITE_PKG/sitecustomize.py" ] && \
|
|
||||||
! grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
|
||||||
if cp "$SITE_PKG/sitecustomize.py" \
|
|
||||||
"$SITE_PKG/sitecustomize.py.pre-truecloud-patch"; then
|
|
||||||
echo "OK: Backed up existing sitecustomize.py"
|
|
||||||
else
|
|
||||||
echo "WARNING: Could not back up existing sitecustomize.py; skipping install to avoid data loss."
|
|
||||||
_can_install=false
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$_can_install" = true ]; then
|
|
||||||
# Substitute PATCH_DIR into the source so sitecustomize.py knows where
|
|
||||||
# to write hook_status.json and check the kill switch at runtime.
|
|
||||||
_sc_tmp="$SITE_PKG/sitecustomize.py.truecloud-tmp"
|
|
||||||
if TRUECLOUD_PATCH_DIR="$PATCH_DIR" \
|
|
||||||
"$PYTHON" -c "
|
|
||||||
import os, sys
|
|
||||||
d = os.environ['TRUECLOUD_PATCH_DIR']
|
|
||||||
with open(d + '/patch/sitecustomize.py', encoding='utf-8') as fh:
|
|
||||||
sys.stdout.write(fh.read().replace('/data/truecloud-patch', d))
|
|
||||||
" > "$_sc_tmp" && mv "$_sc_tmp" "$SITE_PKG/sitecustomize.py"; then
|
|
||||||
echo "OK: Installed sitecustomize.py → $SITE_PKG/sitecustomize.py"
|
|
||||||
else
|
|
||||||
rm -f "$_sc_tmp"
|
|
||||||
echo "WARNING: Failed to write $SITE_PKG/sitecustomize.py (permission error?)"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── Direct file patching ──────────────────────────────────────────────────
|
|
||||||
# Patch b2.py and restic.py directly in the overlay (primary approach).
|
|
||||||
# Each run strips any existing TRUECLOUD_PATCH block and rewrites it fresh,
|
|
||||||
# so a bugfix in the block takes effect immediately on the next apply.sh run
|
|
||||||
# without needing to manually clear the overlay.
|
|
||||||
|
|
||||||
if [ -n "$_MW_DIR" ] && [ "$_can_install" = true ]; then
|
|
||||||
_B2_PY="$_MW_DIR/rclone/remote/b2.py"
|
_B2_PY="$_MW_DIR/rclone/remote/b2.py"
|
||||||
_RESTIC_PY="$_MW_DIR/plugins/cloud_backup/restic.py"
|
_RESTIC_PY="$_MW_DIR/plugins/cloud_backup/restic.py"
|
||||||
|
|
||||||
@@ -231,7 +186,6 @@ import sys
|
|||||||
|
|
||||||
BLOCK = """
|
BLOCK = """
|
||||||
# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
|
# TRUECLOUD_PATCH — added by truenas-truecloud-patch/patch/apply.sh
|
||||||
# URL fix logic lives in sitecustomize._tc_fix_restic_cmd (single source of truth).
|
|
||||||
try:
|
try:
|
||||||
_tc_orig_get_restic_config = get_restic_config
|
_tc_orig_get_restic_config = get_restic_config
|
||||||
except NameError:
|
except NameError:
|
||||||
@@ -240,12 +194,34 @@ else:
|
|||||||
def get_restic_config(cloud_backup):
|
def get_restic_config(cloud_backup):
|
||||||
import dataclasses as _dc
|
import dataclasses as _dc
|
||||||
result = _tc_orig_get_restic_config(cloud_backup)
|
result = _tc_orig_get_restic_config(cloud_backup)
|
||||||
|
cmd = list(result.cmd)
|
||||||
|
for i, part in enumerate(cmd):
|
||||||
|
if part.startswith("--repo=") or part.startswith("--repository="):
|
||||||
|
pfx, _, url = part.partition("=")
|
||||||
|
pfx += "="
|
||||||
|
elif i and cmd[i - 1] in ("-r", "--repo", "--repository"):
|
||||||
|
pfx = None
|
||||||
|
url = part
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
scheme, sep, rest = url.partition(":")
|
||||||
|
if not sep:
|
||||||
|
break
|
||||||
|
changed = False
|
||||||
|
if rest.startswith("/") and not rest.startswith("//"):
|
||||||
|
rest = rest[1:]
|
||||||
|
changed = True
|
||||||
|
if scheme == "b2" and "/" in rest:
|
||||||
|
rest = rest.replace("/", ":", 1)
|
||||||
|
changed = True
|
||||||
|
if changed:
|
||||||
|
new_url = scheme + ":" + rest
|
||||||
|
cmd[i] = pfx + new_url if pfx is not None else new_url
|
||||||
try:
|
try:
|
||||||
import sitecustomize as _sc
|
return _dc.replace(result, cmd=cmd)
|
||||||
return _sc._tc_fix_restic_cmd(result, _dc)
|
except TypeError:
|
||||||
except Exception as _e:
|
return result._replace(cmd=cmd)
|
||||||
import sys as _sys
|
break
|
||||||
_sys.stderr.write(f"[truecloud-patch] restic URL fix failed: {_e}\n")
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
get_restic_config._truecloud_patched = True
|
get_restic_config._truecloud_patched = True
|
||||||
@@ -272,12 +248,7 @@ PYEOF
|
|||||||
else
|
else
|
||||||
echo "WARNING: restic.py not found at $_RESTIC_PY"
|
echo "WARNING: restic.py not found at $_RESTIC_PY"
|
||||||
fi
|
fi
|
||||||
elif [ -z "$_MW_DIR" ]; then
|
# Write hook_status.json so 'verify' reflects the current patch state.
|
||||||
echo "WARNING: Cannot determine middlewared directory; skipping direct file patch."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Write hook_status.json so 'verify' reflects the current patch state
|
|
||||||
# without requiring a backup run to trigger the import hook.
|
|
||||||
"$PYTHON" -c "
|
"$PYTHON" -c "
|
||||||
import json, os, sys, time
|
import json, os, sys, time
|
||||||
b2_ok = sys.argv[1] == '1'
|
b2_ok = sys.argv[1] == '1'
|
||||||
|
|||||||
@@ -1,285 +0,0 @@
|
|||||||
"""
|
|
||||||
TrueCloud provider patch — sitecustomize.py
|
|
||||||
Installed into Python site-packages on every boot by apply.sh.
|
|
||||||
|
|
||||||
Hooks two middlewared module imports using the find_spec / exec_module API
|
|
||||||
(required for Python 3.12+, which ships with TrueNAS SCALE 25.x / Debian 13):
|
|
||||||
|
|
||||||
middlewared.rclone.remote.b2
|
|
||||||
Adds get_restic_config() so the native restic B2 backend works.
|
|
||||||
Restic repo URL: b2:<bucket>:<folder> (colon separator, restic 0.16.x)
|
|
||||||
Auth: B2_ACCOUNT_ID, B2_ACCOUNT_KEY
|
|
||||||
|
|
||||||
middlewared.plugins.cloud_backup.restic
|
|
||||||
Fixes the URL builder for providers with no hostname component, and
|
|
||||||
converts the slash separator to a colon for B2 (restic 0.16.x format):
|
|
||||||
Stock code: f"{rclone_type}:{url}/{remote_path}" → "b2:/bucket/path"
|
|
||||||
Patched: "b2:bucket:path" (leading slash stripped, / → : for B2)
|
|
||||||
|
|
||||||
Both patches are no-ops if the module already provides the functionality
|
|
||||||
(i.e. a future TrueNAS version adds native support). All errors are caught
|
|
||||||
and written to stderr so middlewared always starts regardless of patch state.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import sys
|
|
||||||
|
|
||||||
|
|
||||||
# ── Import hook ───────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
class _Finder:
|
|
||||||
"""
|
|
||||||
find_spec-based meta path finder (Python 3.4+, required for 3.12+).
|
|
||||||
Intercepts specific module imports, loads them normally, then patches.
|
|
||||||
"""
|
|
||||||
|
|
||||||
_targets = frozenset({
|
|
||||||
"middlewared.rclone.remote.b2",
|
|
||||||
"middlewared.plugins.cloud_backup.restic",
|
|
||||||
})
|
|
||||||
|
|
||||||
def __init__(self):
|
|
||||||
self._loading = set() # guards against re-entrant imports
|
|
||||||
self._done = set() # modules already patched
|
|
||||||
|
|
||||||
def find_spec(self, fullname, path, target=None): # noqa: ARG002
|
|
||||||
if (
|
|
||||||
fullname in self._targets
|
|
||||||
and fullname not in self._done
|
|
||||||
and fullname not in self._loading
|
|
||||||
):
|
|
||||||
import importlib.machinery
|
|
||||||
import importlib.util
|
|
||||||
|
|
||||||
# Find the real file spec HERE, before Python adds the module to
|
|
||||||
# sys.modules. If we deferred this to exec_module, find_spec would
|
|
||||||
# short-circuit via sys.modules[fullname].__spec__ (our own spec) and
|
|
||||||
# exec_module would call itself recursively forever.
|
|
||||||
self._loading.add(fullname)
|
|
||||||
try:
|
|
||||||
real_spec = importlib.util.find_spec(fullname)
|
|
||||||
finally:
|
|
||||||
self._loading.discard(fullname)
|
|
||||||
|
|
||||||
if real_spec is None:
|
|
||||||
# Module absent in this Python installation (e.g. removed in a
|
|
||||||
# TrueNAS update). Record a FAIL so hook_status.json is still
|
|
||||||
# written and cmd_verify gives a diagnostic instead of "no file".
|
|
||||||
_record_status(fullname, ok=False,
|
|
||||||
detail="module not found in this Python installation")
|
|
||||||
self._mark_done(fullname)
|
|
||||||
return None
|
|
||||||
|
|
||||||
return importlib.machinery.ModuleSpec(
|
|
||||||
fullname,
|
|
||||||
_Loader(self, fullname, real_spec),
|
|
||||||
origin=real_spec.origin,
|
|
||||||
is_package=real_spec.submodule_search_locations is not None,
|
|
||||||
)
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _mark_done(self, fullname):
|
|
||||||
self._done.add(fullname)
|
|
||||||
if self._done >= self._targets:
|
|
||||||
try:
|
|
||||||
sys.meta_path.remove(self)
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
class _Loader:
|
|
||||||
def __init__(self, finder, fullname, real_spec):
|
|
||||||
self._finder = finder
|
|
||||||
self._fullname = fullname
|
|
||||||
self._real_spec = real_spec
|
|
||||||
|
|
||||||
def create_module(self, spec): # noqa: ARG002
|
|
||||||
return None # use Python's default module creation
|
|
||||||
|
|
||||||
def exec_module(self, module):
|
|
||||||
fullname = self._fullname
|
|
||||||
real_spec = self._real_spec
|
|
||||||
|
|
||||||
try:
|
|
||||||
real_spec.loader.exec_module(module)
|
|
||||||
# Fix module metadata so it looks like a normal import.
|
|
||||||
module.__spec__ = real_spec
|
|
||||||
module.__loader__ = real_spec.loader
|
|
||||||
if real_spec.origin:
|
|
||||||
module.__file__ = real_spec.origin
|
|
||||||
except Exception as exc:
|
|
||||||
# Mark done to prevent infinite retry loops, then surface the failure
|
|
||||||
# in hook_status.json so cmd_verify shows a diagnostic FAIL rather
|
|
||||||
# than "no status file found".
|
|
||||||
self._finder._mark_done(fullname)
|
|
||||||
_record_status(fullname, ok=False,
|
|
||||||
detail=f"module load failed: {exc}")
|
|
||||||
raise
|
|
||||||
self._finder._mark_done(fullname)
|
|
||||||
|
|
||||||
# exec_module succeeded — apply our patch.
|
|
||||||
# Must stay in sync with _Finder._targets.
|
|
||||||
try:
|
|
||||||
if fullname == "middlewared.rclone.remote.b2":
|
|
||||||
_patch_b2(module)
|
|
||||||
elif fullname == "middlewared.plugins.cloud_backup.restic":
|
|
||||||
_patch_restic(module)
|
|
||||||
except Exception as exc:
|
|
||||||
sys.stderr.write(
|
|
||||||
f"[truecloud-patch] patch failed for {fullname}: {exc}\n"
|
|
||||||
)
|
|
||||||
_record_status(fullname, ok=False, detail=str(exc))
|
|
||||||
|
|
||||||
|
|
||||||
# ── Patch functions ───────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def _tc_fix_restic_cmd(result, dataclasses):
|
|
||||||
"""Fix the restic repo URL: b2:/bucket/path → b2:bucket:path (restic 0.16.x)."""
|
|
||||||
cmd = list(result.cmd)
|
|
||||||
for i, part in enumerate(cmd):
|
|
||||||
if part.startswith("--repo=") or part.startswith("--repository="):
|
|
||||||
pfx, _, url = part.partition("=")
|
|
||||||
pfx += "="
|
|
||||||
elif i and cmd[i - 1] in ("-r", "--repo", "--repository"):
|
|
||||||
pfx = None
|
|
||||||
url = part
|
|
||||||
else:
|
|
||||||
continue
|
|
||||||
scheme, sep, rest = url.partition(":")
|
|
||||||
if not sep:
|
|
||||||
break
|
|
||||||
changed = False
|
|
||||||
if rest.startswith("/") and not rest.startswith("//"):
|
|
||||||
rest = rest[1:]
|
|
||||||
changed = True
|
|
||||||
if scheme == "b2" and "/" in rest:
|
|
||||||
rest = rest.replace("/", ":", 1)
|
|
||||||
changed = True
|
|
||||||
if changed:
|
|
||||||
new_url = scheme + ":" + rest
|
|
||||||
cmd[i] = pfx + new_url if pfx is not None else new_url
|
|
||||||
try:
|
|
||||||
return dataclasses.replace(result, cmd=cmd)
|
|
||||||
except TypeError:
|
|
||||||
return result._replace(cmd=cmd)
|
|
||||||
break
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def _patch_b2(module):
|
|
||||||
cls = module.B2RcloneRemote
|
|
||||||
|
|
||||||
if "get_restic_config" in cls.__dict__:
|
|
||||||
# A future TrueNAS version already added native B2 restic support.
|
|
||||||
_record_status("middlewared.rclone.remote.b2", ok=True,
|
|
||||||
detail="native support present; patch not needed")
|
|
||||||
return
|
|
||||||
|
|
||||||
def _b2_restic_config(task):
|
|
||||||
p = task["credentials"]["provider"]
|
|
||||||
missing = [f for f in ("account", "key") if f not in p]
|
|
||||||
if missing:
|
|
||||||
raise KeyError(
|
|
||||||
f"truecloud-patch: B2 provider missing field(s) {missing!r}; "
|
|
||||||
f"schema may have changed. Present: {sorted(p)!r}"
|
|
||||||
)
|
|
||||||
return "", {"B2_ACCOUNT_ID": p["account"], "B2_ACCOUNT_KEY": p["key"]}
|
|
||||||
|
|
||||||
cls.get_restic_config = staticmethod(_b2_restic_config)
|
|
||||||
cls.restic = True
|
|
||||||
sys.stderr.write("[truecloud-patch] B2 restic support enabled\n")
|
|
||||||
_record_status("middlewared.rclone.remote.b2", ok=True,
|
|
||||||
detail="method attached; credential fields verified at first backup")
|
|
||||||
|
|
||||||
|
|
||||||
def _patch_restic(module):
|
|
||||||
if getattr(module.get_restic_config, "_truecloud_patched", False):
|
|
||||||
_record_status("middlewared.plugins.cloud_backup.restic", ok=True,
|
|
||||||
detail="already patched in this process")
|
|
||||||
return
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
|
|
||||||
_orig = module.get_restic_config
|
|
||||||
|
|
||||||
def get_restic_config(cloud_backup):
|
|
||||||
return _tc_fix_restic_cmd(_orig(cloud_backup), dataclasses)
|
|
||||||
|
|
||||||
get_restic_config._truecloud_patched = True
|
|
||||||
module.get_restic_config = get_restic_config
|
|
||||||
sys.stderr.write("[truecloud-patch] restic B2 URL fix applied (b2:bucket:path)\n")
|
|
||||||
_record_status("middlewared.plugins.cloud_backup.restic", ok=True)
|
|
||||||
|
|
||||||
|
|
||||||
_STATUS_FILE = "/data/truecloud-patch/hook_status.json"
|
|
||||||
_hook_status: dict = {}
|
|
||||||
|
|
||||||
|
|
||||||
def _record_status(fullname: str, ok: bool, detail: str = "") -> None:
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import time
|
|
||||||
|
|
||||||
if fullname in _hook_status:
|
|
||||||
return # idempotent: first call wins
|
|
||||||
_hook_status[fullname] = {"ok": ok, "detail": detail}
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"patched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
|
||||||
"patches": _hook_status,
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
tmp = _STATUS_FILE + ".tmp"
|
|
||||||
with open(tmp, "w", encoding="utf-8") as fh:
|
|
||||||
json.dump(payload, fh, indent=2)
|
|
||||||
os.replace(tmp, _STATUS_FILE) # atomic on POSIX
|
|
||||||
except OSError:
|
|
||||||
pass # non-fatal — status file is informational only
|
|
||||||
|
|
||||||
|
|
||||||
# ── Entry point ───────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def _install():
|
|
||||||
import os
|
|
||||||
if os.path.exists("/data/truecloud-patch/disabled"):
|
|
||||||
return # kill switch
|
|
||||||
# Scope to the middlewared service process only — not midclt, debug scripts,
|
|
||||||
# or other tools that happen to share the same venv.
|
|
||||||
# Also covers python -m middlewared where argv[0] is the __main__.py path.
|
|
||||||
_argv0 = (sys.argv or [""])[0]
|
|
||||||
_base = os.path.basename(_argv0)
|
|
||||||
if not (
|
|
||||||
_base == "middlewared"
|
|
||||||
or (_base == "__main__.py"
|
|
||||||
and os.path.basename(os.path.dirname(_argv0)) == "middlewared")
|
|
||||||
):
|
|
||||||
return
|
|
||||||
import importlib.util
|
|
||||||
if importlib.util.find_spec("middlewared") is None:
|
|
||||||
return
|
|
||||||
# If apply.sh displaced an existing sitecustomize.py, exec it first so any
|
|
||||||
# vendor startup code (path additions, codec registrations, etc.) still runs.
|
|
||||||
# __file__ is absent in some embedded contexts; the empty fallback is safe.
|
|
||||||
_self = globals().get("__file__", "")
|
|
||||||
if _self:
|
|
||||||
_pre = _self + ".pre-truecloud-patch"
|
|
||||||
if os.path.isfile(_pre):
|
|
||||||
try:
|
|
||||||
import builtins
|
|
||||||
with open(_pre, encoding="utf-8") as _fh:
|
|
||||||
# Separate globals dict prevents the exec'd code from
|
|
||||||
# shadowing our names; sys.path changes still take effect
|
|
||||||
# via the shared sys module object.
|
|
||||||
exec( # noqa: S102
|
|
||||||
compile(_fh.read(), _pre, "exec"),
|
|
||||||
{"__builtins__": builtins, "__file__": _pre,
|
|
||||||
"__name__": "sitecustomize"},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
sys.meta_path.append(_Finder())
|
|
||||||
|
|
||||||
|
|
||||||
try:
|
|
||||||
_install()
|
|
||||||
except Exception:
|
|
||||||
pass # never raise from sitecustomize.py — it would prevent Python from starting
|
|
||||||
+21
-4
@@ -6,10 +6,11 @@
|
|||||||
# bash /mnt/tank/truenas-truecloud-patch/recover.sh
|
# bash /mnt/tank/truenas-truecloud-patch/recover.sh
|
||||||
#
|
#
|
||||||
# What it does:
|
# What it does:
|
||||||
# 1. Creates a "disabled" file in the repo root — sitecustomize.py checks for
|
# 1. Creates a "disabled" file in the repo root — apply.sh checks for this
|
||||||
# this file at startup and skips the import hook entirely, so middlewared
|
# file at boot and skips all patching, so the next boot is always clean.
|
||||||
# starts clean without any of our code running.
|
# 2. Unmounts any active truecloud overlays so the original /usr files are
|
||||||
# 2. Restarts middlewared.
|
# visible immediately (no reboot required).
|
||||||
|
# 3. Restarts middlewared against the unpatched files.
|
||||||
#
|
#
|
||||||
# To re-enable the patch after investigating:
|
# To re-enable the patch after investigating:
|
||||||
# rm /mnt/tank/truenas-truecloud-patch/disabled
|
# rm /mnt/tank/truenas-truecloud-patch/disabled
|
||||||
@@ -29,6 +30,22 @@ fi
|
|||||||
|
|
||||||
touch "$PATCH_DIR/disabled"
|
touch "$PATCH_DIR/disabled"
|
||||||
echo "Kill switch set: $PATCH_DIR/disabled created."
|
echo "Kill switch set: $PATCH_DIR/disabled created."
|
||||||
|
|
||||||
|
echo "Unmounting truecloud overlays ..."
|
||||||
|
_any=0
|
||||||
|
for _tag in mw ui; do
|
||||||
|
if mount | grep -qF "truecloud-${_tag} on "; then
|
||||||
|
_mnt=$(mount | grep "truecloud-${_tag} on " | awk '{print $3}' | head -1)
|
||||||
|
if umount "$_mnt" 2>/dev/null; then
|
||||||
|
echo " Unmounted: $_mnt"
|
||||||
|
_any=1
|
||||||
|
else
|
||||||
|
echo " WARNING: Could not unmount $_mnt — a reboot will restore original files."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[ "$_any" -eq 0 ] && echo " No overlays active."
|
||||||
|
|
||||||
echo "Restarting middlewared ..."
|
echo "Restarting middlewared ..."
|
||||||
if systemctl restart middlewared; then
|
if systemctl restart middlewared; then
|
||||||
echo ""
|
echo ""
|
||||||
|
|||||||
+2
-64
@@ -44,68 +44,6 @@ else
|
|||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Remove sitecustomize.py ───────────────────────────────────────────────────
|
|
||||||
|
|
||||||
echo "Removing sitecustomize.py ..."
|
|
||||||
|
|
||||||
# Use the same Python detection logic as apply.sh
|
|
||||||
PYTHON="python3"
|
|
||||||
if [ -x /usr/bin/middlewared ]; then
|
|
||||||
shebang=$(dd if=/usr/bin/middlewared bs=256 count=1 2>/dev/null | head -1 || true)
|
|
||||||
if [[ "$shebang" =~ ^'#!'(/[^[:space:]]+python[^[:space:]]*) ]]; then
|
|
||||||
PYTHON="${BASH_REMATCH[1]}"
|
|
||||||
elif [[ "$shebang" =~ ^'#!/usr/bin/env '(python[^[:space:]]*) ]]; then
|
|
||||||
PYTHON=$(command -v "${BASH_REMATCH[1]}" 2>/dev/null || echo "python3")
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! "$PYTHON" -c "import middlewared" 2>/dev/null; then
|
|
||||||
echo " WARNING: '$PYTHON' cannot import middlewared; falling back to python3"
|
|
||||||
PYTHON="python3"
|
|
||||||
if ! "$PYTHON" -c "import middlewared" 2>/dev/null; then
|
|
||||||
echo " WARNING: 'python3' also cannot import middlewared; sitecustomize.py may be removed from the wrong location."
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
SITE_PKG=$("$PYTHON" -c "
|
|
||||||
import os
|
|
||||||
try:
|
|
||||||
import middlewared
|
|
||||||
print(os.path.dirname(os.path.dirname(os.path.abspath(middlewared.__file__))))
|
|
||||||
except ImportError:
|
|
||||||
import site
|
|
||||||
print(site.getsitepackages()[0])
|
|
||||||
" 2>/dev/null || true)
|
|
||||||
|
|
||||||
if [ -n "$SITE_PKG" ] && [ -f "$SITE_PKG/sitecustomize.py" ]; then
|
|
||||||
if grep -q "truecloud-patch" "$SITE_PKG/sitecustomize.py" 2>/dev/null; then
|
|
||||||
if [ -f "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" ]; then
|
|
||||||
# mv atomically overwrites our file with the vendor original —
|
|
||||||
# safer than rm-then-mv if /usr is transiently read-only.
|
|
||||||
mv "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" \
|
|
||||||
"$SITE_PKG/sitecustomize.py"
|
|
||||||
echo " Restored previous sitecustomize.py"
|
|
||||||
else
|
|
||||||
rm "$SITE_PKG/sitecustomize.py"
|
|
||||||
echo " Removed $SITE_PKG/sitecustomize.py"
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo " $SITE_PKG/sitecustomize.py is not ours; leaving it alone."
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo " Not found (already removed or install didn't place it here)."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Handle orphaned backup when sitecustomize.py was removed (e.g. by a TrueNAS
|
|
||||||
# update) but the .pre-truecloud-patch file survived in the same directory.
|
|
||||||
if [ -n "$SITE_PKG" ] && [ ! -f "$SITE_PKG/sitecustomize.py" ] && \
|
|
||||||
[ -f "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" ]; then
|
|
||||||
mv "$SITE_PKG/sitecustomize.py.pre-truecloud-patch" \
|
|
||||||
"$SITE_PKG/sitecustomize.py"
|
|
||||||
echo " Restored orphaned sitecustomize.py backup"
|
|
||||||
fi
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# ── Restore UI bundle ─────────────────────────────────────────────────────────
|
# ── Restore UI bundle ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
echo "Restoring UI bundle backup ..."
|
echo "Restoring UI bundle backup ..."
|
||||||
@@ -131,11 +69,11 @@ if [ "$RESTORED" -eq 0 ]; then
|
|||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# ── Unmount overlays (TrueNAS 25.x immutable OS) ─────────────────────────────
|
# ── Unmount overlays ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
echo "Unmounting truecloud overlays (if any) ..."
|
echo "Unmounting truecloud overlays (if any) ..."
|
||||||
_ov_found=0
|
_ov_found=0
|
||||||
for _tag in sc ui; do
|
for _tag in mw ui; do
|
||||||
if mount | grep -qF "truecloud-${_tag} on "; then
|
if mount | grep -qF "truecloud-${_tag} on "; then
|
||||||
_ov_mnt=$(mount | grep "truecloud-${_tag} on " | awk '{print $3}' | head -1)
|
_ov_mnt=$(mount | grep "truecloud-${_tag} on " | awk '{print $3}' | head -1)
|
||||||
if umount "$_ov_mnt" 2>/dev/null; then
|
if umount "$_ov_mnt" 2>/dev/null; then
|
||||||
|
|||||||
Reference in New Issue
Block a user