Previously all candidate thumbnails (up to 3000) were loaded into a
single dict before any processing started. At ~5 MB per decoded preview
image, 472 candidates = ~2.4 GB of thumbnail data alone, easily
exhausting a 4 GB container memory limit on CPU.
Now thumbnails are downloaded and processed in batches of 32. Each
image is pop()'d from the batch dict immediately after embedding so the
decoder memory is released before the next batch starts. Peak in-flight
thumbnail memory is now bounded to ~32 × 8 MB = ~256 MB regardless of
candidate pool size. GPU users benefit too — faster first results and
lower host RAM pressure during large runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
insightface 1.0.1 added a hard dep on the CPU onnxruntime package.
Combined with an incorrect override-dependencies entry in 0.2.10 that
forced onnxruntime (no platform marker) unconditionally, both packages
were installed into the venv on x86_64 Linux — the CPU package landed
last and overwrote onnxruntime-gpu, removing CUDAExecutionProvider
from the provider list.
Fix: declare the two packages as conflicting in uv's resolver so only
the correct one is installed per environment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- immich_api: `score or confidence` treated 0.0 score as falsy; use explicit None check
- diversity: same falsy-zero fix in _get_face_confidence
- diversity: _crop_face_from_thumbnail scale loop now filters by person_id (was
using first person's imageWidth/imageHeight regardless of target in group photos)
- jobs: partially-trained auto mode kept limit="auto" for adaptive stopping, then
caps result to remaining capacity (was converting to int, silently disabling FPS
adaptive threshold and early-stop)
- embeddings: _suppress_output finally block wraps first dup2 in try/finally so
stderr is always restored even if stdout restore raises OSError
- Dockerfile: ldconfig find uses python3.* glob instead of hardcoded python3.13
- scheduler: sleep until next_run instead of fixed 60s; eliminates late-fire jitter
and unnecessary wakeups on long schedules
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frigate /api/faces response has person names as top-level keys with
lists of filenames — {person: [file, ...], "train": [...]}. The old
code incorrectly looked inside data["train"] as if it were a dict of
persons, causing 'list object has no attribute items' on every run.
Immich get_people() now checks for 401 before raise_for_status() and
logs a clear "API key invalid or expired" message instead of the raw
requests exception string.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- .env.example: add VERBOSE, remove active AUTO_MODE=true (now TTY-detected),
comment out FORCE_CPU/ENABLE_CACHE default values, update CRON_SCHEDULE
section to document all three modes
- README: clarify that log file is always DEBUG regardless of VERBOSE
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The log file already captures DEBUG unconditionally. This env var wires
the same to the Rich console handler for troubleshooting without needing
to read the log file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv sync runs before winnow/ is COPY'd into the build stage, so the wheel
uv builds contains only dist-info (no Python files). The console_scripts
entry point sets sys.path[0] to its own directory (/app/.venv/bin), not
/app, so 'from winnow.cli import main' fails at container startup.
PYTHONPATH=/app makes the package importable regardless of how Python
is invoked (script, -m, entry point, docker exec). This is preferable
to re-ordering the COPY layers, which would bust the heavy uv sync cache
on every winnow/ source change.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bugs fixed (from high-effort review):
- InsightFace CPU fallback now passes providers=['CPUExecutionProvider'] and
wraps in _suppress_output() so broken GPU drivers don't cause fallback to
try the same broken provider again, and C-extension noise stays suppressed
- scheduler.py: BaseException → Exception (KeyboardInterrupt already re-raised;
winnow has no sys.exit() calls, so SystemExit would not occur, but Exception
is the correct scope)
- compose.yml: fix inverted AUTO_MODE comment (docker run -it enables
interactive mode via TTY, not non-interactive)
Model loading logging (embeddings.py):
- InsightFace: disk cache check, "not cached — downloading now (~300 MB)",
"loading into memory on GPU/CPU...", "ready on GPU/CPU (Xs)"
- SigLIP: same treatment; cache path derived dynamically from model_name
via HuggingFace slug convention (models--org--model) so it stays correct
if the model variant ever changes
Logging level audit (INFO/DEBUG/WARNING/ERROR):
- diversity.py: internal algo steps (clustering, medoids, adaptive threshold,
auto-stop decision) → DEBUG; final selection summaries stay INFO
- immich_api.py: "Fetching assets" and "Retained N assets" → DEBUG (callers
already print this to the console via rprint)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
embeddings.py: face embedding cache used 'immich' for lookup but
'insightface' for storage, so the cache was never hit for locally-
computed embeddings. Unified to 'insightface'/'siglip' throughout.
This affects all users since ENABLE_CACHE now defaults to true.
scheduler.py: INSIGHTFACE_HOME=/models/.insightface was having
'.insightface' appended again, making buffalo_l check always report
'will download'. Also catch BaseException (not just Exception) so a
SystemExit from a library call can't silently kill all future runs.
log_config.py: handlers.clear() abandoned open FileHandler fds on
each scheduled main() call. Close each handler properly before removal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
RETRY_REJECTED was silently read from the environment in _configure_person,
bypassing user control in interactive sessions. Now prompts the user with
the env var value as the default, so the setting is visible and overridable.
All other env vars in the interactive path are already correct:
YEARS_FILTER is a prompt default, ONLY_PEOPLE/SKIP_PEOPLE/MIN_FACE_COUNT
are auto_configure-only, and TRAINING_MODE/STRATEGY/OBJECT_CLASS are
always prompted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three container lifetime modes via CRON_SCHEDULE:
unset — run once on startup, exit
empty string — sleep infinity; use docker exec -it winnow winnow
cron expression — run on startup, then on schedule
This replaces the need for a separate MANUAL_MODE env var. The empty
string is a natural "I want the container alive but unscheduled" signal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The primary use case is headless Docker, so auto mode is now the default
whenever stdin has no TTY. Interactive mode activates when a terminal is
present (docker run -it, local shell). AUTO_MODE=true remains as an
explicit override for scripting with a pseudo-TTY.
Removes AUTO_MODE=true, stdin_open, tty, and FORCE_CPU=false from
compose.yml — none are needed for headless operation. Updates README
and the interactive-mode hint in the CLI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The static LD_LIBRARY_PATH only covered cudnn and cuda_runtime — missing
cublas, cufft, curand, cusolver, cusparse, nvjitlink, etc. onnxruntime-gpu
needs libcublasLt.so at minimum, so GPU mode silently fell back to CPU.
Replace with a one-shot ldconfig call over every nvidia site-packages lib/
dir, which covers all packages regardless of what gets installed.
Also: remove the ambiguous directory="" from preload_dlls (use auto-search
default) and add a clear warning when CUDAExecutionProvider is absent so
the user sees actionable guidance instead of silent CPU fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces a VARIANT=gpu|cpu build arg to the Dockerfile. The cpu
variant uses ubuntu:22.04 (no CUDA base), installs torch+cpu and
onnxruntime (no GPU deps) via a separate pyproject-cpu.toml / uv-cpu.lock,
and is published as :cpu (dev-cpu on the dev branch) via a new
build-cpu CI job. Saves ~2 GB over the default GPU image.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When stdin has no TTY (Docker without -it), IntPrompt/Confirm raise
EOFError and crash the container into a restart loop. Treat a non-TTY
stdin the same as AUTO_MODE=true so headless runs work without any env
var configuration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: ENV INSIGHTFACE_HOME=/models → /models/.insightface to
match compose.yml and .env.example; the old value caused InsightFace
to store models at /models/models/buffalo_l (double-appended subdir)
- entrypoint.sh: use /app/.venv/bin/winnow (installed entry point)
instead of python -m winnow.cli
- config.py: ENABLE_CACHE default false → true; embedding cache is
always beneficial in practice; users can opt out with ENABLE_CACHE=false
- compose.yml: comment out CRON_SCHEDULE so scheduling is opt-in;
flip ENABLE_CACHE to commented opt-out to reflect new default
- README.md: update ENABLE_CACHE default documentation to true
- tests/test_config.py: update default assertion to match
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- __init__.py: derive __version__ from importlib.metadata instead of
a hardcoded "0.1.0" that was six releases out of date
- scheduler.py: move winnow.cli import to module top (no more noqa);
clean up redundant bool variables in check_models
- tests/test_quality.py: 17 tests covering all five quality check
functions individually plus assess_quality integration cases
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- winnow/logging.py → winnow/log_config.py: avoids shadowing the stdlib
logging module; log file renamed from immich_export.log to winnow.log
- scheduler.py: run main() in-process instead of subprocess.run so
InsightFace and SigLIP models stay resident in memory across scheduled
runs (hundreds of MB load, previously reloaded every run)
- .gitignore: replaced 200-line boilerplate with ~30 project-relevant
patterns; removed Django/Flask/Redis/RabbitMQ/Scrapy/etc. noise
- .python-version: untracked (redundant with requires-python in
pyproject.toml; kept in .gitignore for local pyenv users)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- embeddings: initialize ctx_id=-1 before try block so the except
handler cannot NameError; move insightface_home out of try for the
same reason
- embeddings: replace contextlib.redirect_stdout/stderr (Python-level
only) with fd-level dup2 suppression — actually silences C extension
noise from InsightFace during model loading
- jobs: fix frigate_count==0 falling through `or` chain; use explicit
`is not None` check so a real zero is not treated as missing data
- diversity: thread person_id through select_diverse_assets →
_select_by_embedding → _get_face_bbox / _get_face_confidence /
_crop_face_from_thumbnail so group-photo assets embed the target
person's face rather than whichever person is listed first
- config: replace type() hack for ConfigManager with a proper class
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
onnxruntime-gpu has no arm64 wheels (manylinux_2_27_x86_64 /
manylinux_2_28_x86_64 only). uv sync --frozen failed on the arm64
image with exit code 2. Gated onnxruntime-gpu behind the x86_64
marker; arm64 and non-Linux use the CPU onnxruntime package. Added
required-environments so the lockfile is solved for both platforms.
Removed onnxruntime-gpu from override-dependencies (it had no marker
support and blocked arm64 resolution).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
curl|gpg --dearmor was silently dropping the deadsnakes key (gpg exits 0
on bad input), leaving apt unable to find python3.13. Reverted to
add-apt-repository with GNUPGHOME=$(mktemp -d) to isolate gpg from any
pre-existing agent socket. Removed --platform=\$BUILDPLATFORM from the
arm64 base so the image contains real arm64 binaries. License updated
from MIT to AGPL-3.0-or-later.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Title first, then badges on one consistent line. Removed redundant
Release and Lint badges. Fixed docs links to point to the wiki.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ubuntu 26.04 ships Python 3.14, not 3.13. Reverted arm64 to ubuntu:24.04.
Both architectures now add the deadsnakes PPA by fetching the GPG key via
curl and piping through gpg --dearmor — no gpg-agent, safe under QEMU.
Removes the per-arch conditional and ARG TARGETARCH dependency in RUN commands.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Docker automatic platform ARGs are only in scope for FROM instructions.
$TARGETARCH in RUN commands was always empty, so the deadsnakes PPA
conditional never ran and python3.13 could not be found on the Ubuntu
22.04 CUDA base.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All platforms now use Python 3.13. amd64 installs via deadsnakes PPA on
the Ubuntu 22.04 CUDA base; arm64 gets Python 3.13 natively from Ubuntu
26.04. Verified cp313 wheels exist for onnxruntime-gpu 1.26.0 and
torch 2.12.0+cu126. uv.lock regenerated under CPython 3.13.5.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
add-apt-repository ppa:deadsnakes/ppa fails on arm64 in GitHub Actions
because QEMU emulation doesn't support the GPG agent. Ubuntu 24.04 ships
Python 3.12 natively so the PPA is not needed. amd64 (CUDA/Ubuntu 22.04
base) still uses the PPA. PPA install is now gated on TARGETARCH=amd64.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flanandClaude Sonnet 4.6