create_task.py: migrate REST /api/v2.0 -> midclt (removed in TrueNAS 26.04)

The tool created cloud_backup tasks via POST /api/v2.0/cloud_backup, which is deprecated
and removed in TrueNAS 26.04. It now calls the middleware directly with midclt
(cloudsync.credentials.query / cloud_backup.query / cloud_backup.create), so it runs on
the TrueNAS host with no host address or API key. --host/--api-key/--insecure are kept
accepted-but-ignored for compatibility. Dropped the ssl/urllib HTTP client. v0.2.0.

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## v0.2.0 — 2026-07-08
+
+### Changed
+
+- **`create_task.py` now uses the TrueNAS middleware via `midclt` instead of the
+  deprecated `/api/v2.0` REST API**, which is removed in TrueNAS 26.04. Practical
+  effects:
+  - Run the script **on the TrueNAS host** — it uses the local middleware socket, so
+    it no longer needs a host address or API key.
+  - `--host`, `--api-key`, and `--insecure` are accepted but **ignored** (a deprecation
+    note is printed); they will be removed in a future release.
+  - `list-credentials` → `cloudsync.credentials.query`, `list-tasks` →
+    `cloud_backup.query`, `create` → `cloud_backup.create`.
+- Dropped the `ssl`/`urllib` HTTP client; no TLS certificate handling is needed anymore.
+
+## v0.1.0 — 2026-07-08
+
+### Added
+
+- `create --cache-path PATH` — sets the restic cache directory on the task.
+  Without a cache path, TrueNAS runs restic with `--no-cache`, which re-reads all
+  repository metadata from the provider on every run and is glacially slow on
+  large repos (a 564 GB dataset estimated **55 days** to a first backup). Tasks
+  created without `--cache-path` now print a warning explaining the consequence.
+
 ## v0.0.4 — 2026-07-06
 
 ### Fixed

README.md

@@ -91,17 +91,46 @@ support and the reason is logged to `apply.log` in your repo root.
 
 ## How persistence works
 
-TrueNAS SCALE updates replace `/usr/` entirely. The patch survives by keeping
-this repository on a **persistent ZFS pool** (your data pool, not `/tmp` or a
-system path) and registering a **PREINIT initshutdownscript** in the TrueNAS
-database — the one piece of state that survives both reboots and OS updates.
-On every boot, `patch/apply.sh` runs (executed by middlewared after pools are
-imported), mounts a writable
-[overlayfs](https://docs.kernel.org/filesystems/overlayfs.html) over the
-relevant directories (upper layer in `/run`, recreated each boot), patches
-`b2.py` and `restic.py` directly in that overlay, re-patches the UI bundle,
-and schedules the one-time deferred middlewared restart that loads the
-patched backend. No extra configuration is needed.
+Two different things must survive two different events:
+
+| Event | What would be lost | What makes it survive |
+|---|---|---|
+| **Reboot** | The overlay holding the patched files lives in `/run` (tmpfs) and vanishes | The PREINIT hook re-runs `apply.sh` on every boot and schedules one middlewared restart to load the result |
+| **TrueNAS update** | `/usr/` is replaced entirely; custom files in `/etc/` are wiped with the new boot environment | This repo lives on your **data pool**, and the hook registration lives in the **TrueNAS config database** — both survive updates. The first boot after an update is just a normal boot |
+
+### What happens on every boot
+
+1. **middlewared starts** with the stock (unpatched) modules. This is
+   unavoidable: PREINIT scripts are executed *by* middlewared
+   (`ix-preinit.service` → `midclt call initshutdownscript.execute_init_tasks`),
+   so nothing registered there can run before it.
+2. **Pools import** (`ix-zfs.service`), making `/mnt/<pool>` — and this
+   repository — available.
+3. **`apply.sh` runs** (`ix-preinit.service`): mounts the writable overlay
+   (upper layer in `/run`), patches `b2.py` and `restic.py` on disk inside it,
+   patches the UI bundle, and writes `apply.log` and `hook_status.json`.
+4. **A deferred restart is scheduled.** The middlewared that is running
+   imported the stock modules in step 1 and never re-imports, so the on-disk
+   patch alone is not enough. `apply.sh` detects it was invoked by middlewared
+   and creates a transient systemd unit (`truecloud-mw-restart`, via
+   `systemd-run --no-block`, ordered after `multi-user.target`) — detached and
+   deferred so it cannot disrupt the remainder of the boot sequence.
+5. **Once boot completes, middlewared restarts once** and imports the patched
+   modules from the overlay. S3/B2 backup support is now active until the next
+   reboot, when the cycle repeats.
+
+What you will observe: one middlewared restart shortly after every boot (a
+brief web UI/API blip; running services are unaffected). Between steps 3
+and 5 there is a short window — typically well under a minute — where the UI
+already shows S3/B2 (the JS bundle is read from disk per request) but the
+backend is still stock. A backup job that fires inside that window fails once
+with `NotImplementedError` and succeeds on its next run; see
+[Troubleshooting](#troubleshooting) if it persists beyond boot.
+
+Manual runs of `bash patch/apply.sh` never trigger the restart — that only
+happens in boot context. `install.sh` and `recover.sh` perform their own
+explicit restarts instead, which is why a manual re-apply must be followed by
+`systemctl restart middlewared`.
 
 ---
 
@@ -158,28 +187,34 @@ Check [CHANGELOG.md](CHANGELOG.md) to see what changed between versions.
 ## Creating a task via CLI
 
 If the UI still shows only Storj after refreshing (e.g. the JS bundle pattern
-changed in a new TrueNAS version), create tasks directly via the REST API:
+changed in a new TrueNAS version), create tasks directly. Run this **on the
+TrueNAS host** — it talks to the local middleware via `midclt`, so it needs no
+host address or API key:
 
 ```bash
 # Replace /mnt/tank/truenas-truecloud-patch with your clone path
 
 # List your cloud credentials to find the right ID
-python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py \
-    --host 192.168.1.1 --api-key <key> list-credentials
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py list-credentials
 
 # Create a task with a B2 credential (id=3)
-python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py \
-    --host 192.168.1.1 --api-key <key> create \
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py create \
     --name "tank-to-b2" \
     --path /mnt/tank/data \
     --credential 3 \
     --bucket my-bucket \
     --folder backups/tank \
     --password "restic-repo-password" \
+    --cache-path /mnt/tank/.restic-cache \
     --keep-last 14
 ```
 
-Get an API key from **System → API Keys → Add**.
+> **Always pass `--cache-path`.** Without it TrueNAS runs restic with `--no-cache`,
+> which re-fetches all repo metadata from the provider every run — glacially slow
+> on large repos. Point it at a writable dir on a pool with free space.
+
+> Versions ≤ 0.1.0 used the `/api/v2.0` REST API with `--host`/`--api-key`; those
+> flags are now accepted-but-ignored (REST is removed in TrueNAS 26.04).
 
 ---
 
@@ -342,6 +377,36 @@ If one or more entries show `[FAIL]`:
 
 ## Troubleshooting
 
+**Backups fail with `NotImplementedError` after a reboot**
+
+The traceback ends in `rclone/base.py` → `raise NotImplementedError` and
+contains no `_tc_` frames: the running middlewared is executing stock code.
+Either the deferred restart never fired, or the patch never landed on disk
+this boot. Diagnose in this order:
+
+```bash
+# Did apply.sh run this boot, at which version, and did it schedule the restart?
+tail -40 /mnt/tank/truenas-truecloud-patch/apply.log
+
+# Full check — compares the running process against the patch timestamp
+python3 /mnt/tank/truenas-truecloud-patch/patch/create_task.py verify
+
+# Did the deferred restart unit run, fail, or never get created?
+systemctl status truecloud-mw-restart.service
+journalctl -u truecloud-mw-restart.service --no-pager | tail -20
+```
+
+- `verify` reports the process started **before** the patch → the restart
+  didn't happen. `systemctl restart middlewared` fixes it immediately; the
+  journal output above tells you why it was missed.
+- `apply.log` shows the kill switch is active → `rm .../disabled`, then
+  `bash install.sh`.
+- `apply.log` has no entry for this boot → the hook didn't run; re-run
+  `bash install.sh` to re-register it.
+- `apply.log` header shows `[v0.0.3]` or older → update:
+  `git pull && bash install.sh` (v0.0.4 fixed patches not loading after
+  reboot).
+
 **Apply log** (check after each reboot or install):
 ```bash
 cat /mnt/tank/truenas-truecloud-patch/apply.log
@@ -369,9 +434,7 @@ grep -c 'STORJ_IX.*S3.*B2' \
     | grep -v ':0'
 ```
 
-**`create_task.py` SSL error connecting to TrueNAS**
-`create_task.py` talks to the **TrueNAS API**, not your S3 endpoint, and
-verifies its TLS certificate. If your NAS uses a self-signed certificate,
-pass `--insecure` — but be aware this disables certificate verification for
-the API call that transmits your TrueNAS API key. Adding your NAS certificate
-to your system's trust store is safer.
+**`create_task.py` — "midclt not found" or permission errors**
+`create_task.py` now talks to the local middleware via `midclt`, so run it **on
+the TrueNAS host** (not remotely) as a user with middleware access (root). There
+is no HTTPS/API-key call anymore, so there is no TLS certificate to configure.

patch/create_task.py

@@ -3,22 +3,24 @@
 create_task.py — create TrueNAS TrueCloud Backup tasks with S3 or B2 credentials.
 
 The TrueNAS UI normally restricts the credential dropdown to Storj only.
-This script bypasses that restriction by calling the REST API directly.
+This script bypasses that restriction by talking to the TrueNAS middleware
+directly via `midclt` (the /api/v2.0 REST API is removed in TrueNAS 26.04).
 
 Compatible providers (after the truecloud-patch backend patch is applied):
   S3     — any S3-compatible endpoint (AWS, Wasabi, Cloudflare R2, MinIO, …)
   B2     — Backblaze B2 native API
   STORJ_IX — Storj (unchanged, always worked)
 
-Requires a TrueNAS API key: UI → System → API Keys → Add.
+Run this ON the TrueNAS host — it uses the local middleware socket via `midclt`,
+so no host address or API key is needed.
 
 Examples
 --------
 List available cloud credentials:
-    python3 create_task.py --host 192.168.1.1 --api-key <key> list-credentials
+    python3 create_task.py list-credentials
 
 Create a task backed by a B2 credential (id=3):
-    python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
+    python3 create_task.py create \\
         --name "tank-to-b2" \\
         --path /mnt/tank/data \\
         --credential 3 \\
@@ -28,7 +30,7 @@ Create a task backed by a B2 credential (id=3):
         --keep-last 14
 
 Create a task using an S3-compatible credential (Wasabi, R2, etc.):
-    python3 create_task.py --host 192.168.1.1 --api-key <key> create \\
+    python3 create_task.py create \\
         --name "tank-to-wasabi" \\
         --path /mnt/tank/data \\
         --credential 5 \\
@@ -37,54 +39,44 @@ Create a task using an S3-compatible credential (Wasabi, R2, etc.):
         --password "restic-repo-password"
 
 List existing TrueCloud Backup tasks:
-    python3 create_task.py --host 192.168.1.1 --api-key <key> list-tasks
+    python3 create_task.py list-tasks
 """
 
 import argparse
 import calendar
 import json
 import os
-import ssl
 import subprocess
 import sys
 import time
-import urllib.error
-import urllib.request
 
-__version__ = "0.0.4"
+__version__ = "0.2.0"
 
 _PATCH_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 _STATUS_FILE = os.path.join(_PATCH_DIR, "hook_status.json")
 
 
-def make_client(host, api_key, insecure=False):
-    """Return a callable that makes authenticated REST API calls."""
-    base = f"https://{host}/api/v2.0"
-    headers = {
-        "Authorization": f"Bearer {api_key}",
-        "Content-Type": "application/json",
-    }
-    ctx = ssl.create_default_context()
-    if insecure:
-        ctx.check_hostname = False
-        ctx.verify_mode = ssl.CERT_NONE
-
-    def call(method, path, body=None):
-        url = base + path
-        data = json.dumps(body).encode() if body is not None else None
-        req = urllib.request.Request(url, data=data, headers=headers, method=method)
-        try:
-            with urllib.request.urlopen(req, context=ctx) as resp:
-                return json.loads(resp.read())
-        except urllib.error.HTTPError as exc:
-            detail = exc.read().decode(errors="replace")
-            print(f"HTTP {exc.code} {exc.reason}: {detail}", file=sys.stderr)
-            sys.exit(1)
-        except urllib.error.URLError as exc:
-            print(f"Connection error: {exc.reason}", file=sys.stderr)
-            sys.exit(1)
-
-    return call
+def midclt_call(method, *args):
+    """Call a middleware method locally via `midclt`, the supported JSON-RPC transport
+    that replaces the deprecated /api/v2.0 REST API (removed in TrueNAS 26.04). Must run
+    on the TrueNAS host. Each arg is JSON-encoded (a dict for create; none for queries).
+    Exits with a clear message on failure."""
+    cmd = ["midclt", "call", method] + [json.dumps(a) for a in args]
+    try:
+        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
+    except FileNotFoundError:
+        print("ERROR: `midclt` not found — run this script ON the TrueNAS host.",
+              file=sys.stderr)
+        sys.exit(1)
+    except subprocess.SubprocessError as exc:
+        print(f"ERROR: midclt call failed: {exc}", file=sys.stderr)
+        sys.exit(1)
+    if proc.returncode != 0:
+        print(f"ERROR: midclt {method}: {(proc.stderr or proc.stdout).strip()}",
+              file=sys.stderr)
+        sys.exit(1)
+    out = proc.stdout.strip()
+    return json.loads(out) if out else None
 
 
 # ── Sub-commands ──────────────────────────────────────────────────────────────
@@ -185,8 +177,8 @@ def _provider_type(cred):
     return p or "?"
 
 
-def cmd_list_credentials(client, _args):
-    creds = client("GET", "/cloudsync/credentials")
+def cmd_list_credentials(_args):
+    creds = midclt_call("cloudsync.credentials.query")
     if not creds:
         print("No cloud credentials configured.")
         return
@@ -196,8 +188,8 @@ def cmd_list_credentials(client, _args):
         print(f"{c['id']:>4}  {_provider_type(c):<14}  {c['name']}")
 
 
-def cmd_list_tasks(client, _args):
-    tasks = client("GET", "/cloud_backup")
+def cmd_list_tasks(_args):
+    tasks = midclt_call("cloud_backup.query")
     if not tasks:
         print("No TrueCloud Backup tasks configured.")
         return
@@ -209,7 +201,7 @@ def cmd_list_tasks(client, _args):
         print(f"{t['id']:>4}  {enabled:<8}  {ptype:<14}  {t.get('description', '')}")
 
 
-def cmd_create(client, args):
+def cmd_create(args):
     parts = args.schedule.split()
     if len(parts) != 5:
         print(
@@ -242,7 +234,18 @@ def cmd_create(client, args):
         "enabled": not args.disabled,
     }
 
-    result = client("POST", "/cloud_backup", body)
+    if args.cache_path:
+        body["cache_path"] = args.cache_path
+    else:
+        print(
+            "WARNING: no --cache-path given. TrueNAS will run restic with --no-cache, "
+            "which is very slow for large repositories (it re-reads all repo metadata "
+            "from the provider every run). Set --cache-path to a writable dir on a pool "
+            "with free space.",
+            file=sys.stderr,
+        )
+
+    result = midclt_call("cloud_backup.create", body)
     try:
         print(f"Created task id={result['id']}  name={result['description']!r}")
     except (KeyError, TypeError):
@@ -258,14 +261,12 @@ def main():
         epilog=__doc__.split("Examples")[1] if __doc__ and "Examples" in __doc__ else "",
     )
     p.add_argument("--version", "-V", action="version", version=f"truecloud-patch {__version__}")
-    p.add_argument("--host", default=None, metavar="HOST",
-                   help="TrueNAS hostname or IP address (required except for verify)")
-    p.add_argument("--api-key", default=None, metavar="KEY",
-                   help="TrueNAS API key — System → API Keys (required except for verify)")
-    p.add_argument("--insecure", action="store_true",
-                   help="Skip TLS certificate verification (self-signed certs). "
-                        "WARNING: exposes your API key to network interception. "
-                        "Prefer adding your cert to the trust store instead.")
+    # Deprecated & ignored: the tool now uses the local middleware via `midclt` (the
+    # /api/v2.0 REST API is removed in TrueNAS 26.04), so it must run ON the TrueNAS
+    # host and needs no host/API key. Kept accepted-but-ignored for compatibility.
+    p.add_argument("--host", default=None, help=argparse.SUPPRESS)
+    p.add_argument("--api-key", default=None, help=argparse.SUPPRESS)
+    p.add_argument("--insecure", action="store_true", help=argparse.SUPPRESS)
 
     sub = p.add_subparsers(dest="cmd", required=True)
 
@@ -290,6 +291,11 @@ def main():
                    help="Snapshots to retain after each run (default: 14)")
     c.add_argument("--schedule",    default="0 2 * * *",
                    help="Cron schedule (default: '0 2 * * *' — daily at 02:00)")
+    c.add_argument("--cache-path",  default="", metavar="PATH",
+                   help="restic cache directory (e.g. /mnt/pool/.restic-cache). "
+                        "STRONGLY recommended: without it TrueNAS runs restic with "
+                        "--no-cache, which re-fetches all repo metadata from the "
+                        "provider every run and is extremely slow on large repos.")
     c.add_argument("--transfer-setting",
                    choices=["DEFAULT", "PERFORMANCE", "FAST_STORAGE"],
                    default="DEFAULT",
@@ -307,16 +313,17 @@ def main():
         cmd_verify()
         return
 
-    if not args.host or not args.api_key:
-        p.error("--host and --api-key are required for this command")
+    if args.host or args.api_key or args.insecure:
+        print("NOTE: --host/--api-key/--insecure are deprecated and ignored; this tool "
+              "now uses the local middleware (midclt) and must run on the TrueNAS host.",
+              file=sys.stderr)
 
-    client = make_client(args.host, args.api_key, args.insecure)
     if args.cmd == "list-credentials":
-        cmd_list_credentials(client, args)
+        cmd_list_credentials(args)
     elif args.cmd == "list-tasks":
-        cmd_list_tasks(client, args)
+        cmd_list_tasks(args)
     elif args.cmd == "create":
-        cmd_create(client, args)
+        cmd_create(args)
 
 
 if __name__ == "__main__":