Commit Graph
67 Commits
Author SHA1 Message Date
flan f2d57420fb Update patch_ui.py 2026-07-12 20:49:03 -04:00
flan 150a241a0f Fix native-nested probe: guard message is split across string literals
The probe searched plugins/cloud/crud.py for the contiguous string
"no further nesting". Stock does not contain it. The message is split across
adjacent string literals:

    verrors.add(f"{name}.snapshot", "This option is only available for datasets that have no further "
                                    "nesting")

Python concatenates those at runtime, so the errmsg IS contiguous and CRUD_BLOCK's
runtime filter matches correctly -- but the SOURCE never contains the whole
phrase. The probe therefore found nothing, concluded iX had removed the guard, and
skipped the nested module as "already native" on every boot. apply.log would say
"TrueNAS now handles nesting natively" and the feature would never work.

It fails safe -- the stock guard stays in place, so no backup could be
misconfigured and no data was at risk -- but the module was 100% dead.

Verified against real middlewared on a live box: the probe returned native=yes
(wrong) before this change and native=no (correct) after.

The probe now strips whitespace and quote characters before matching, which is
robust to any wrapping or concatenation style. Added a regression test that
EXECUTES apply.sh's own probe code (extracted, not reimplemented -- a
reimplementation would pass while the shipped probe stayed broken) against the
real wrapped source, a single-line variant, and a three-way split.

75 tests, ruff and shellcheck clean.
2026-07-12 22:44:26 +00:00
flan 24f1f2c648 Fix post-merge audit findings; unify staging teardown
Audit
-----
- create_task.py verify failed on a DEFAULT install. hook_status.json emitted a
  per-file entry for the nested module with ok:false whenever the feature was
  switched off -- the default -- so verify printed [FAIL] and exited 1, right
  after the README tells users to run it. Status is now per MODULE with an
  `active` flag, and verify renders an inactive module as [SKIP].

- A partial apply suppressed the middlewared restart. The exit code conflated
  "nothing applied" with "one module applied, one failed", so a failing providers
  patch would prevent the restart that a freshly-applied nested patch needs,
  leaving it on disk and never loaded. Exit 2 now means partial and the restart
  still fires.

- The native-nested probe could never fire. It scanned crud.py for the guard
  message, but our own injected block quotes that message, so once applied the
  probe would always conclude the guard was still present. It now reads only the
  stock portion of the file.

- recover.sh did not unmount staging trees, so an emergency recovery left bind
  mounts pinning ZFS snapshots that could then never be destroyed.

- uninstall.sh deleted sidecar files without reading them. A sidecar is the only
  record that an interrupted run's snapshot tree is still on disk; both scripts
  now name the snapshot before clearing it.

- Removed a dead branch in the restart gate (unreachable: the kill switch exits).

Refactor
--------
- Staging teardown had been copy-pasted into uninstall.sh and recover.sh -- two
  untested shell copies of the fiddly depth-ordering and lazy-umount logic. Both
  now call `python3 patch/truecloud_nested.py cleanup`, so there is exactly one
  implementation and it is the one under test.

- Dropped the in-memory ACTIVE dict. The sidecar file was already the source of
  truth; a second in-process record could only desync -- and it is precisely the
  middlewared-restart case (which empties it) that must not orphan a snapshot
  tree. One record, on disk, or none.

Not done: the overlay-unmount loop is duplicated across apply.sh/uninstall.sh/
recover.sh. It is pre-existing, and apply.sh runs at PREINIT under a tight
timeout -- giving it a source dependency would trade 10 lines of duplication for
a boot-time failure mode.

74 tests, ruff and shellcheck clean.
2026-07-12 22:29:31 +00:00
flan c2e1976659 Merge nested-dataset snapshot support (v0.3.0)
Adds a second, opt-in module to the patch: TrueCloud Backup's "Take Snapshot"
option now works on datasets that have child datasets — i.e. every pool running
Apps, where each app is its own dataset and stock refuses the config with
"This option is only available for datasets that have no further nesting".

Stock's guard is correct and is not merely deleted. create_snapshot() already
takes a recursive ZFS snapshot but points restic at the PARENT dataset's
.zfs/snapshot/, where ZFS does not expose child datasets — so removing the check
would make restic walk a near-empty tree, report SUCCESS, and upload almost
nothing. The missing traversal is implemented instead: each descendant's own
.zfs/snapshot is bind-mounted into a staging tree and restic is pointed at that.
The guard is relaxed only after the machinery is in place.

The patch is now two modules (providers, nested) which retire independently as
TrueNAS ships each capability natively; the kill switch fires only when both are
done.

Also adds CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13; 67 tests),
including a pass that compile()s the Python source this patch appends into live
middlewared modules — previously unchecked, and a syntax error there breaks the
box at boot.

Nested snapshots are OFF by default (install.sh --enable-nested-snapshots).
The mount --bind staging step has not yet been exercised by a live backup run.
2026-07-12 22:17:48 +00:00
flan eca6eb3f3b Retire the two modules independently; calm the README down
The native-support check looked only for native B2 restic support and, on
finding it, set the kill switch and disabled the whole patch. That was fine when
providers were the only thing here. With nested-dataset snapshots in the patch it
is wrong: TrueNAS is likely to ship one capability long before the other, and a
single all-or-nothing kill switch would silently take a still-needed module down
with the superseded one.

apply.sh now treats the patch as two independent modules:

  providers  b2.py + restic.py + the UI credential dropdown
             native when B2RcloneRemote carries a real get_restic_config()

  nested     plugins/cloud/{snapshot,crud}.py + cloud_backup/sync.py (opt-in)
             native when the "no further nesting" validation is gone from
             plugins/cloud/crud.py

Each is detected and skipped on its own. The kill switch fires only once BOTH are
done (native, or nested was never enabled). The UI patch belongs to providers and
is skipped with it. The deferred middlewared restart now fires when ANY
still-needed module landed -- keying it off providers alone would have left a
freshly-patched nested module on disk and never loaded on a native-B2 box.
hook_status.json reports each module with an active flag and a reason.

README: dropped the warning boxes and the disclaimer's fear-bulleting in favour
of plain statements, documented the two-module design and the per-module
auto-disable, and added a Development section disclosing AI assistance. The one
caveat kept, as a plain sentence rather than a banner: the mount --bind staging
step has not yet been exercised by a live backup run.

67 tests, ruff and shellcheck clean.
2026-07-12 22:17:37 +00:00
flan a80de88078 Distinguish a missing snapshot from an unreadable one
os.path.isdir() returns False both when a snapshot directory does not exist and
when it cannot be stat'd. Staging aborted either way -- correct -- but reported
every case as "has no snapshot", which sends you hunting for a snapshot that is
sitting right there.

Found while dry-running the planner against a real recursive snapshot of Tap:
running as a non-root user, /mnt/Tap/apps/paperless/data is mode 0700 and the
probe reported "has no snapshot" when `zfs list` showed the snapshot present.
Middleware runs as root so this would not fire in production, but a backup
system must not misreport why it failed.

plan_staging now takes a probe() that classifies the path as ok / missing /
unreadable, and the error names which.

Dry-run results against Tap (250 datasets, real `zfs snapshot -r`):
- 170 mounted filesystems under /mnt/Tap/, and the plan produces exactly 170
  descendant mounts -- no omissions
- 18 legacy-mountpoint datasets reported as skipped, never dropped silently
- Tap/ix-apps mounts at /mnt/.ix-apps, correctly outside the backup path
- parent snapshot exposes 0 entries under /apps; the staged sources expose 71,
  and lidarr/config resolves with lidarr.db present
- snapshot_tree_names() identifies all 250; deleting only the parent (what stock
  does) leaves 249 orphans, and the sweep clears them
2026-07-12 22:04:56 +00:00
flan bb26edf351 Make nested snapshots opt-in; fix snapshot leaks found in audit
Opt-in
------
Nested-dataset snapshot support changes how backups read their source data, so
it is now off by default and gated behind a marker file:

  install.sh --enable-nested-snapshots
  install.sh --disable-nested-snapshots

With neither flag install.sh preserves the current setting, so a routine
`git pull && bash install.sh` can never silently flip it. When disabled,
apply.sh skips the patch entirely and the stock guard remains. uninstall.sh
tears down staging mounts and removes the marker.

Snapshot lifecycle
------------------
zfs.snapshot.delete defaults to recursive=False and stock restic_backup() calls
it with no options. Stock is safe only because its validation means recursive is
never True in the field. Enabling nested datasets makes recursive snapshots real:
the parent then has one child snapshot per descendant dataset (160+ on an Apps
pool), so stock's delete would orphan every child on EVERY successful run.

The patch now owns the lifecycle end to end:

- delete_snapshot_tree() sweeps the parent and all children, and is idempotent
  against stock's finally winning the race once our mounts are released
- on a staging failure the tree is deleted here, because sync.py never completes
  `snapshot, local_path = await create_snapshot(...)` and so its finally deletes
  nothing at all
- the snapshot is recorded in a sidecar file before anything is mounted, so a
  middlewared restart mid-backup cannot orphan it
- a crashed run's snapshot tree is reclaimed on the next run instead of being
  overwritten and leaked

Silent-omission fix
-------------------
The dataset list is now enumerated AFTER the snapshot. Read beforehand it could
miss a dataset created in the gap, which the recursive snapshot would capture but
the staging plan would not -- silently omitting its data. Read afterwards, an
unsnapshotted dataset trips the staging check and fails the run loudly.

Also from the audit
-------------------
- plan_staging scopes by dataset name, so skipped-dataset warnings no longer
  include every mountpoint-less dataset on the box, which buried the ones that
  matter
- staging_root_for rejects "." / ".." components that would escape the staging
  base, and resolves STAGING_BASE at call time rather than freezing it into a
  default argument
- uninstall.sh no longer `rm -rf`s a tree that may still contain live bind
  mounts, and unmounts by path depth rather than string length
- apply_plan takes an injectable isdir; verify_staged drops an unused parameter
- pin the shellcheck action instead of tracking @master

61 tests, ruff and shellcheck clean.
2026-07-12 21:52:09 +00:00
flan a572eb2164 Support ZFS snapshots on datasets with child datasets
TrueCloud Backup's "Take Snapshot" option is rejected on any path containing
child datasets:

  This option is only available for datasets that have no further nesting

That excludes every pool running Apps, where each app is its own dataset and
often has config/pgdata children. Without the option the backup reads live
files, so databases are captured mid-write and an app that continuously
rewrites its files can stall a run as restic chases a moving target.

The stock guard is correct and must not simply be removed. create_snapshot()
already takes a recursive ZFS snapshot, but points the backup tool at the
parent dataset's .zfs/snapshot/, and ZFS does not expose child datasets there:

  /mnt/Tap/.zfs/snapshot/<snap>/apps/                -> 0 entries
  /mnt/Tap/apps/lidarr/config/.zfs/snapshot/<snap>/  -> the real data

Deleting the check would make restic walk a near-empty tree, report success,
and upload almost nothing.

Implement the missing traversal instead. After the recursive snapshot is taken,
each descendant dataset's own .zfs/snapshot/<snap> is bind-mounted into a
staging tree mirroring the original layout, and the backup tool is pointed at
the staging root. The guard is relaxed only after that machinery is in place.

Safety properties:
- staging failure aborts the backup; a partial tree is never handed to restic
- a post-mount pass asserts every target is a mountpoint and the root is
  non-empty, so this cannot regress into the empty backup it exists to prevent
- apply.sh patches crud.py last, so a partial failure leaves the guard intact
  rather than exposing "guard removed, traversal missing"
- every injected block no-ops when _truecloud_nested is absent
- unmountable/locked datasets are skipped and reported, never dropped silently
- scoped to cloud_backup; cloudsync has no teardown wired in, so its guard stays

The staging root is stable per task, so restic can find its parent snapshot
between runs; stock's timestamped .zfs path changes every run and forces a
full re-scan.

Add CI (shellcheck, bash -n, ruff, pytest on 3.11-3.13), including tests that
compile the *_BLOCK strings, which are Python source appended to live
middlewared modules and were previously unchecked.

Also: sync stale version strings, untrack a committed .pyc, gitignore
__pycache__.
2026-07-12 21:20:22 +00:00
flan 4ded8cff3d Fix deferred restart racing boot: wait for boot to settle before restarting middlewared
The truecloud-mw-restart unit relied on After=multi-user.target /
After=ix-postinit.service, but systemd ordering cannot see middlewared's
internal boot work. On 25.10.4 the restart fired two seconds into
ix-reporting's reporting.start_service call and before the docker/apps
startup task ran, killing both for the whole boot: all apps down
(docker.status FAILED), no dashboard stats, SMB backend uninitialized.

The unit now runs patch/wait_restart.sh: drain the systemd boot job
queue (is-system-running --wait), poll docker.status until the state
machine leaves its transitional states, short grace period, then
try-restart. No Type=oneshot — a oneshot's start job sits in the very
queue the script waits on and would deadlock on itself. All waits are
bounded and fail open.
v0.2.1
2026-07-09 17:21:07 +00:00
flan da1be97377 create_task.py: migrate REST /api/v2.0 -> midclt (removed in TrueNAS 26.04)
The tool created cloud_backup tasks via POST /api/v2.0/cloud_backup, which is deprecated
and removed in TrueNAS 26.04. It now calls the middleware directly with midclt
(cloudsync.credentials.query / cloud_backup.query / cloud_backup.create), so it runs on
the TrueNAS host with no host address or API key. --host/--api-key/--insecure are kept
accepted-but-ignored for compatibility. Dropped the ssl/urllib HTTP client. v0.2.0.
v0.2.0
2026-07-08 15:09:38 -04:00
flan 8a66c85a7e create_task: add --cache-path (avoids restic --no-cache slowness); v0.1.0 v0.1.0 2026-07-08 01:14:19 -04:00
flan e8ff607234 README: explain reboot persistence in detail
Document the full boot sequence (stock start, pool import, PREINIT
patching, deferred restart via truecloud-mw-restart), the reboot vs
OS-update survival table, the short unpatched window after boot, and
why manual apply.sh runs require an explicit middlewared restart.

Add a troubleshooting entry for backups failing with
NotImplementedError after a reboot, with ordered diagnostic commands.
2026-07-07 15:27:16 +00:00
flan 04ac71deb3 Fix TypeError creating B2 tasks on TrueNAS 24.10: handle both credential schemas
On 24.10 (Electric Eel) credentials["provider"] is the type string with
account/key in credentials["attributes"]; 25.04+ moved them into a
provider dict. The injected get_restic_config only handled the newer
shape and raised TypeError on 24.10 at task creation (#1).

The method now detects the schema and reads credentials from the right
place on both. create_task.py list-credentials and list-tasks use the
same schema-agnostic lookup.
v0.0.4
2026-07-06 05:43:09 +00:00
flan 73233865e8 Fix backend patch never loading at boot: schedule deferred middlewared restart
PREINIT initshutdownscripts are executed by middlewared itself
(ix-preinit.service, ordered after ix-zfs pool import), so the running
process had already imported the stock modules when apply.sh patched
them in the overlay — S3/B2 support silently reverted on every reboot
until something restarted middlewared. install.sh masked the bug with
its explicit restart.

apply.sh now detects boot context (parent process is middlewared) and
schedules a single detached restart via a transient systemd unit
(truecloud-mw-restart, After=multi-user.target and ix-postinit.service).
Manual runs never trigger a restart.

create_task.py verify no longer trusts hook_status.json alone: it
compares the middlewared main-process start time (derived from
/proc/<pid>/stat and btime) against patched_at and reports FAIL when
the running process predates the patch.

recover.sh and uninstall.sh cancel a still-queued deferred restart
before their own; docs updated to match the real boot ordering.
2026-07-06 05:42:04 +00:00
flan ee190f558f Fix PREINIT 10-second timeout killing apply.sh before patches land
Registers the boot hook with timeout:120 so TrueNAS gives apply.sh
two minutes instead of the default ten seconds.  Also consolidates
apply.sh Python subprocess count from ~8 to 2, cutting startup
overhead from ~12-16s to ~2-4s.

Bumps all scripts to v0.0.3.
v0.0.3
2026-06-22 15:40:54 +00:00
flan c8de9971e7 Print version in all scripts; add Updating section to README 2026-06-19 19:04:48 +00:00
flan 5b9c888b72 Add CHANGELOG for v0.0.1 and v0.0.2 v0.0.2 2026-06-19 19:01:14 +00:00
flan 1c41df8fec Fix get_restic_config not applied when guard misfires; detect stubs in native check
The guard in the b2.py patch block skipped setting get_restic_config when
something caused B2RcloneRemote.__dict__ to already contain it at import
time (e.g. a TrueNAS version that adds a NotImplementedError stub). Remove
the guard and always assign, which is safe: the native-support kill switch
already prevents patching when TrueNAS ships a real implementation.

Also update the native-support check to distinguish a stub (source contains
NotImplementedError) from a working implementation, so a stub does not
trigger the kill switch and block all future patching.
2026-06-19 18:59:45 +00:00
flan 308fcc28ba README: describe overlay mechanism accurately v0.0.1 2026-06-16 21:10:42 +00:00
flan be6a99ac9d Fix false-positive native-support detection when /usr is writable 2026-06-16 20:06:17 +00:00
flan d0b0e79079 Fix apply.log missing on first install; keep repo on uninstall 2026-06-16 20:01:55 +00:00
flan e077a87300 Fix stale sitecustomize reference; remove duplicate step header 2026-06-16 18:02:48 +00:00
flan 82bffec75a README: warn clearly not to delete or move the repo after install 2026-06-16 18:02:00 +00:00
flan c23672d172 README: remove restore section and generic TrueCloud content 2026-06-16 17:53:04 +00:00
flan 96434d9994 Auto-disable when TrueNAS ships native B2 support; assume immutable /usr
apply.sh now inspects B2RcloneRemote.__dict__ before patching. If TrueNAS
has shipped get_restic_config natively, it sets the kill switch, unmounts
overlays, and logs a clear instruction to run uninstall.sh.

Also: drop all conditional 'if read-only' language — overlay is always
mounted unconditionally since /usr is always immutable on TrueNAS SCALE.

README updated with auto-disable behaviour and revised native-support table.
2026-06-16 17:45:34 +00:00
flan 7498d48b2d Switch to overlay-only patching; remove sitecustomize.py
Patches to b2.py and restic.py are applied directly in the overlayfs at
PREINIT boot time. The sitecustomize.py import hook was belt-and-suspenders
that succeeded or failed alongside the file patch every time, providing no
genuine fallback.

- Delete patch/sitecustomize.py entirely
- apply.sh: remove sitecustomize install step; flatten if/elif/else structure;
  restore self-contained URL-fix logic in the restic.py BLOCK; rename overlay
  tag 'sc' -> 'mw'
- recover.sh: unmount overlays to restore original files immediately, no
  reboot required; kill-switch file prevents re-application on next boot
- uninstall.sh: remove sitecustomize.py removal section; update overlay tag
- install.sh: update preflight to check patch/apply.sh, not sitecustomize.py
- README: remove sitecustomize references throughout; update recovery docs
2026-06-16 17:37:12 +00:00
flan f94e674561 README: remove version-specific claims, fix find path 2026-06-16 17:15:03 +00:00
flan 8c0ffab166 Eliminate URL-fix duplication; document native-support behaviour
URL-fix logic now lives once in sitecustomize._tc_fix_restic_cmd.
apply.sh's restic.py BLOCK delegates to it instead of repeating
the ~40-line implementation.

Additional safety: the BLOCK now guards its get_restic_config
reference with try/except NameError, so a future TrueNAS that
restructures restic.py won't cause an import error.

README: updated Backend table and disclaimer to reflect graceful
degradation; added "If TrueNAS adds native support" section that
covers all five upgrade scenarios (safe pass-through, base-class
shadowing risk, schema-change risk, etc.).
2026-06-16 17:11:08 +00:00
flan e8b0f961c7 Fix B2 restic URL: use colon separator (b2:bucket:path) for restic 0.16.x
restic 0.16.x changed the B2 URL format to use a colon between bucket and
path (b2:bucket:prefix) instead of a slash. The middlewared URL builder
produces b2:/bucket/path; restic then validates the full string after 'b2:'
as a bucket name, which fails because the slash is not in [a-z0-9-].

Fix the restic.py wrapper to strip the leading slash and replace the first
slash with a colon: b2:/bucket/path -> b2:bucket:path.

Also fix the hasattr bug in sitecustomize.py _patch_b2: hasattr() returns
True for methods inherited from the base class (which raises
NotImplementedError), causing the patch to be silently skipped. Use
'get_restic_config' not in cls.__dict__ instead.
2026-06-16 16:41:26 +00:00
flan c8a0c42762 apply.sh: fix b2.py patch guard and make patching self-correcting
Two bugs:
1. hasattr(B2RcloneRemote, "get_restic_config") returned True because the base
   class defines the method (it just raises NotImplementedError). The method was
   never added to B2RcloneRemote. Fixed: use __dict__ check instead.

2. "skip if TRUECLOUD_PATCH marker present" prevented a corrected patch block
   from replacing a previously-written buggy one without clearing the overlay.
   Fixed: always strip any existing TRUECLOUD_PATCH block and rewrite it fresh
   using a Python heredoc. Each apply.sh run now self-corrects to the latest
   version of the patch.
2026-06-16 15:36:36 +00:00
flan f556746310 Fix backend patch: use middlewared's actual site-packages dir, add direct file patching
site.getsitepackages()[0] returns /usr/local/lib/python3.11/dist-packages/ on
TrueNAS 25.x but middlewared lives in /usr/lib/python3/dist-packages/.
sitecustomize.py was installed to the wrong directory and Python never loaded it.

Fix: derive SITE_PKG from middlewared.__file__ so the overlay and sitecustomize.py
land in the correct directory.

Also add direct patching of b2.py and restic.py in the overlay as the primary
backend approach — more reliable than an import hook since it works regardless
of Python's site initialisation configuration. apply.sh now also writes
hook_status.json at boot time so 'verify' shows OK without requiring a backup run.

Also fixes incorrect middlewared log path in README (/var/log/middlewared/middlewared.log
→ /var/log/middlewared.log) and simplifies the verify troubleshooting note.
2026-06-16 15:25:32 +00:00
flan 08cc4254b7 README: document TrueNAS 25.x immutable OS, lazy module imports, log file location 2026-06-16 15:11:56 +00:00
flan 4d89fd6063 install.sh: show only current run's log output, not accumulated history 2026-06-16 14:56:09 +00:00
flan 3afca7075e Support TrueNAS 25.x immutable read-only /usr via overlayfs
TrueNAS 25.x mounts /usr as a read-only filesystem. Writing
sitecustomize.py to site-packages and patching the Angular bundle
both fail with EROFS.

Fix: mount a writable overlayfs on each target directory before
writing to it. Upper/work dirs live in /run (tmpfs), so overlays are
volatile per boot and are recreated by apply.sh on every PREINIT run
before middlewared starts.

apply.sh:
- Add _ensure_writable(dir, tag): probes writability; mounts overlay
  in /run/truecloud-{tag}-{upper,work} if the directory is read-only;
  detects if the overlay is already mounted (idempotent)
- Call _ensure_writable before site-packages writes (tag "sc")
- Detect webui dir with bash loop; call _ensure_writable before
  patch_ui.py (tag "ui") — non-fatal if mount fails

uninstall.sh:
- Add overlay unmounting section after file restoration and before
  rm -rf, so the lower layer's originals are exposed immediately
- Move _restore_failed exit 1 to after unmount so overlays are
  cleaned up even on partial failure
- Update "no backup files" message for immutable OS context
2026-06-16 14:54:21 +00:00
flan 9d9326c212 patch_ui: support TrueNAS 25.x Angular pureFunction binding format
TrueNAS 25.x changed how Angular emits the filterByProviders binding.
Previously a static inline array ("filterByProviders",["STORJ_IX"]),
it is now a pureFunction call:
  pe(slot, factory, component.CloudSyncProviderName.Storj)

Add a _PATTERNS list tried in order, with a _match_pattern() helper.
The 25.x pureFunction pattern matches on the stable TypeScript enum
name (CloudSyncProviderName.Storj); the minified variable names and
slot index are matched with \w+ / \d+. Both patterns replace the
binding with ["STORJ_IX","S3","B2"]. MARKER and backup/restore logic
are unchanged.
2026-06-16 14:45:33 +00:00
flan 52828d3457 Fix three pre-publish issues found in readiness audit
- create_task.py list-tasks: crash on null credentials.provider
  (`creds.get("provider", {})` returns None when key exists but is null;
  switch to `(creds.get("provider") or {})`)

- sitecustomize.py: write hook_status.json after each module, not only
  when both have loaded; S3-only users (B2 module never imported) now
  get a status file from verify instead of "No status file found"

- README: add filesystem find + sqlite3 DB query to the emergency
  recovery section so users can locate their clone path when middlewared
  is down and midclt is unavailable
2026-06-16 14:37:00 +00:00
flan 4730de75dd Improve midclt error handling in install.sh
- Capture midclt output via $(...) instead of > /dev/null so that
  failure detail (which midclt writes to stdout on TrueNAS) is
  preserved and shown to the user on error rather than silently
  discarded
- Expand update failure hint from a bare query command to an actionable
  recovery path: show the midclt output, then print the exact delete
  command with the known stale ID so the user can remove it and retry
2026-06-15 17:51:56 +00:00
flan d97b87bc5d Fix two audit findings in install/uninstall hook management
- Extract hook comment string to _HOOK_COMMENT variable in both
  install.sh and uninstall.sh; previously the literal string
  'TrueCloud provider patch (S3/B2)' appeared three times across two
  files with no shared constant — a silent mismatch on any divergence
  would cause hook lookup to return empty with no error output
- Wrap midclt update and create calls with if/else error handlers;
  previously a midclt failure under set -euo pipefail silently aborted
  the script at "Updating path and enabling ..." with no diagnostic
  or recovery guidance
2026-06-15 17:43:46 +00:00
flan 6a8ed7fa67 Fix four audit findings
- patch/apply.sh: replace sed with Python+env-var for PATCH_DIR
  substitution into sitecustomize.py; sed's & and | metacharacters
  silently corrupt or truncate the output for paths containing those
  chars; Python str.replace has no metacharacter issues; also write to
  a tmp file and mv atomically so a failed substitution never leaves
  an empty sitecustomize.py at the destination
- recover.sh: fix re-enable hint from $PATCH_DIR/apply.sh to
  $PATCH_DIR/patch/apply.sh (apply.sh moved into patch/ subdirectory)
- install.sh + uninstall.sh: match PREINIT hook on comment field
  ("TrueCloud provider patch (S3/B2)") instead of exact script path;
  exact-path match breaks when the repo is moved after install —
  uninstall leaves the stale hook registered (fires on every boot),
  and reinstall creates a duplicate entry; install.sh now also updates
  the script path on re-run so a moved repo self-corrects
2026-06-15 17:36:58 +00:00
flan 70e84038d6 Repo is the install location; all scripts self-locate
Users now clone to a persistent ZFS pool and the repo stays in place.
No files are copied on install — the PREINIT hook points directly into
the clone. Scripts derive PATCH_DIR from their own path at runtime.

- install.sh: PATCH_DIR=$(dirname $0); register patch/apply.sh as
  PREINIT target; chmod only, no cp; update pipe-install error message
- patch/apply.sh: PATCH_DIR=$(dirname $0)/..; substitute PATCH_DIR
  into sitecustomize.py via sed when writing to site-packages;
  reference patch_ui.py as patch/patch_ui.py
- recover.sh, uninstall.sh: PATCH_DIR=$(dirname $0)
- uninstall.sh: look for patch/apply.sh in PREINIT registry
- patch/create_task.py: _PATCH_DIR derived from __file__; apply.log
  path in error message derived from _PATCH_DIR
- patch/sitecustomize.py: /data/truecloud-patch remains as placeholder
  substituted by apply.sh on each install
- .gitignore: exclude runtime files (apply.log, hook_status.json, disabled)
- README: document clone-to-pool install; update all example paths
2026-06-15 17:07:11 +00:00
flan 4c79403491 Copy uninstall.sh to /data/truecloud-patch/ on install
Users who delete the cloned repo after install had no way to uninstall
without re-cloning. Now install.sh copies uninstall.sh to PATCH_DIR
alongside recover.sh, so the uninstall path is always the stable
/data/truecloud-patch/uninstall.sh. README updated to match.
2026-06-15 16:56:18 +00:00
flan fa6f6605b0 Fix four audit findings; simplify apply.sh boolean gate
- uninstall.sh: track _restore_failed separately from RESTORED so
  "No backup files found" only prints when find returns nothing (not
  when mv fails on existing backups); abort with exit 1 before rm -rf
  when any restore fails, leaving PATCH_DIR and recover.sh intact
- install.sh: extend log-scan grep to catch ERROR: lines from
  patch_ui.py (backup OSError was silently missed by WARNING:-only grep)
- install.sh: reword restart-failure message — hook IS already
  registered and sitecustomize.py IS installed; patch activates on
  next boot regardless
- apply.sh: replace `if $_can_install` with `[ "$_can_install" = true ]`
  (explicit test, no implicit command lookup); drop 2>/dev/null on
  install cp so OS error detail reaches the log
2026-06-15 16:26:00 +00:00
flan 0b597387bd Add cmd_verify prompt to install output; fix --insecure docs
- install.sh: print 'verify' command immediately after successful restart
  so users know to confirm the backend patch loaded before creating tasks
- README: correct --insecure description; it controls TLS to the TrueNAS
  API (where the API key is transmitted), not the S3 endpoint — previous
  wording implied it was safe to use for S3 self-signed certs
2026-06-15 16:14:35 +00:00
flan 13b61a8ece Merge audit/exhaustive-review: 9 quality/safety fixes from exhaustive review
All findings from four consecutive full-codebase audit passes plus an
adversarial iXsystems-perspective audit. No new candidates surfaced in
the final clean-pass — branch declared complete.

Fixes:
- apply.sh: gate sitecustomize.py install on backup success; a failed
  backup cp previously fell through and could destroy the vendor file
- apply.sh: correct comment (keeps two prior log generations, not one)
- patch_ui.py: catch OSError on bundle backup with specific diagnostic
- patch_ui.py: find_bundle now matches MARKER so already-patched files
  return 'UI already patched' instead of misleading 'pattern not found'
- install.sh: early guard detects pipe-install (bash <(curl ...)) and
  exits with a clear error pointing to the git clone workflow
- install.sh: scope WARNING grep to current run only (record log offset
  before apply.sh, tail -c +N to read only new bytes)
- install.sh: systemctl restart failure now surfaces a recovery hint
- create_task.py: add MITM risk warning to --insecure flag help text
- create_task.py: handle unexpected 2xx response schema in cmd_create
- uninstall.sh: add import-middlewared verification after Python detection
- uninstall.sh: mv failure in JS bundle restore loop no longer aborts
  under set -e before cleanup; emits WARNING and continues
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
2026-06-15 16:12:00 +00:00
flan 9481bf55f0 Fix three findings from final clean-pass audit
- apply.sh: gate sitecustomize.py install on backup success; a failed
  backup cp (disk full, read-only mount) previously fell through and
  overwrote the vendor file with no recovery path
- create_task.py: handle unexpected 2xx response schema in cmd_create;
  bare KeyError on result['id'] is replaced with a diagnostic print
- uninstall.sh: mv inside while loop had no error handling; under
  set -euo pipefail a failed mv aborted the script before rm -rf PATCH_DIR,
  leaving the system in partial-uninstall limbo
2026-06-15 16:10:09 +00:00
flan 8f64e4964c Fix two findings from adversarial IX-perspective audit
- create_task.py: add MITM risk warning to --insecure flag help text;
  common home-user pattern (self-signed cert) exposes API key in transit
- install.sh: replace bare systemctl restart with explicit failure check
  that prints a recovery hint when middlewared fails to start post-install
2026-06-15 16:06:26 +00:00
flan 7c9aa7159a Fix two confirmed findings from full codebase audit
- patch_ui.py: wrap shutil.copy2 backup in try/except OSError so a
  permission or read-only filesystem error prints a specific diagnostic
  instead of crashing the script with a generic 'exited non-zero' message
- install.sh: add early guard that detects pipe-install (bash <(curl ...))
  and exits with a clear error pointing to the git clone workflow
2026-06-15 16:03:19 +00:00
flan 2a3b15b4b2 Fix two findings from full codebase audit
- apply.sh: correct comment from 'one prior generation' to 'two prior
  generations (.1 and .2)' — rotation has always kept three log files
- uninstall.sh: add sync comment on find paths to match WEBUI_CANDIDATES
  in patch/patch_ui.py, preventing silent drift if a new path is added
2026-06-15 15:58:29 +00:00
flan cbd85af2a3 Fix six quality findings from iterative code review
- patch_ui.py: find_bundle now matches MARKER so already-patched files
  return early and print 'UI already patched' instead of the misleading
  'filterByProviders pattern not found' warning
- install.sh: scope warning grep to current run only (record log offset
  before apply.sh, tail -c +N to read only new bytes)
- install.sh: fix misleading 'before continuing' wording on warning banner
- install.sh: fix grep anchor (^WARNING: missed [truecloud-patch] WARNING: lines)
- uninstall.sh: add import-middlewared verification after Python detection,
  matching apply.sh fallback logic
2026-06-15 15:53:06 +00:00
flan 0ce2083c71 Remove speculative AttributeError guard in URL-fix wrapper 2026-06-15 15:10:16 +00:00