When a person is at MAX_AUTO_IMAGES, winnow now replaces the
lowest-quality mapped training image in Frigate if a higher-confidence
candidate is available, keeping the training set always optimised.
Only files winnow uploaded (tracked via frigate_files mapping) are ever
replaced — manually added Frigate training images are never touched.
A concurrent-upload race condition is detected per-file: if N>1 new
files appear after one upload, the mapping is skipped rather than
guessed, logging at INFO level. The per-file snapshot approach is
retained over a batch approach because wrong mappings (which a batch
approach risks on race) are worse than no mapping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds ROCm (AMD GPU) and Intel GPU support as new :rocm and :intel image
variants. Full changelog in CHANGELOG.md under [0.2.12].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New image variants:
- :rocm — InsightFace via ROCmExecutionProvider, SigLIP via PyTorch ROCm 6.3
- :intel — InsightFace via OpenVINOExecutionProvider (onnxruntime-openvino);
Intel GPU compute runtime auto-installed from Intel graphics repo;
OPENVINO_DEVICE=GPU opts into Arc/iGPU inference (default: CPU)
Also adds:
- pyproject-rocm.toml + uv-rocm.lock, pyproject-intel.toml + uv-intel.lock
- compose.yml device passthrough snippets for AMD and Intel
- CI: build-rocm and build-intel jobs in docker-publish.yml; all four
variants built and tagged in release.yml
- README reworked: cleaner structure, GPU variant quick-start examples,
OPENVINO_DEVICE env var documented
- CHANGELOG entry and version bump to 0.2.12
Fix: IntPrompt in dict literal was eagerly evaluated in the no-embedding
fallback path of _get_strategy_choice, prompting users for a custom count
regardless of which strategy they picked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GHCR packages default private on first creation. Add a best-effort
gh api PATCH call at the end of both the multi-arch merge job and the
cpu build job so any new package version is immediately public without
requiring a manual UI step.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two small correctness fixes from post-commit code review:
- except Exception: continue swallowed network/auth errors silently; add
logger.debug so systematic failures are diagnosable in winnow.log
- batch_images.pop() regressed duplicate-asset-ID handling: if Immich
returns the same asset ID twice within the same 32-item batch window
(pagination edge case), the second occurrence got None and its embedding
was silently dropped. Switching back to .get() matches the old
thumbnail_map.get() behaviour. Peak memory is still bounded to _BATCH
images because batch_images goes out of scope between batches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously all candidate thumbnails (up to 3000) were loaded into a
single dict before any processing started. At ~5 MB per decoded preview
image, 472 candidates = ~2.4 GB of thumbnail data alone, easily
exhausting a 4 GB container memory limit on CPU.
Now thumbnails are downloaded and processed in batches of 32. Each
image is pop()'d from the batch dict immediately after embedding so the
decoder memory is released before the next batch starts. Peak in-flight
thumbnail memory is now bounded to ~32 × 8 MB = ~256 MB regardless of
candidate pool size. GPU users benefit too — faster first results and
lower host RAM pressure during large runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
insightface 1.0.1 added a hard dep on the CPU onnxruntime package.
Combined with an incorrect override-dependencies entry in 0.2.10 that
forced onnxruntime (no platform marker) unconditionally, both packages
were installed into the venv on x86_64 Linux — the CPU package landed
last and overwrote onnxruntime-gpu, removing CUDAExecutionProvider
from the provider list.
Fix: declare the two packages as conflicting in uv's resolver so only
the correct one is installed per environment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- immich_api: `score or confidence` treated 0.0 score as falsy; use explicit None check
- diversity: same falsy-zero fix in _get_face_confidence
- diversity: _crop_face_from_thumbnail scale loop now filters by person_id (was
using first person's imageWidth/imageHeight regardless of target in group photos)
- jobs: partially-trained auto mode kept limit="auto" for adaptive stopping, then
caps result to remaining capacity (was converting to int, silently disabling FPS
adaptive threshold and early-stop)
- embeddings: _suppress_output finally block wraps first dup2 in try/finally so
stderr is always restored even if stdout restore raises OSError
- Dockerfile: ldconfig find uses python3.* glob instead of hardcoded python3.13
- scheduler: sleep until next_run instead of fixed 60s; eliminates late-fire jitter
and unnecessary wakeups on long schedules
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frigate /api/faces response has person names as top-level keys with
lists of filenames — {person: [file, ...], "train": [...]}. The old
code incorrectly looked inside data["train"] as if it were a dict of
persons, causing 'list object has no attribute items' on every run.
Immich get_people() now checks for 401 before raise_for_status() and
logs a clear "API key invalid or expired" message instead of the raw
requests exception string.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- .env.example: add VERBOSE, remove active AUTO_MODE=true (now TTY-detected),
comment out FORCE_CPU/ENABLE_CACHE default values, update CRON_SCHEDULE
section to document all three modes
- README: clarify that log file is always DEBUG regardless of VERBOSE
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The log file already captures DEBUG unconditionally. This env var wires
the same to the Rich console handler for troubleshooting without needing
to read the log file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv sync runs before winnow/ is COPY'd into the build stage, so the wheel
uv builds contains only dist-info (no Python files). The console_scripts
entry point sets sys.path[0] to its own directory (/app/.venv/bin), not
/app, so 'from winnow.cli import main' fails at container startup.
PYTHONPATH=/app makes the package importable regardless of how Python
is invoked (script, -m, entry point, docker exec). This is preferable
to re-ordering the COPY layers, which would bust the heavy uv sync cache
on every winnow/ source change.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bugs fixed (from high-effort review):
- InsightFace CPU fallback now passes providers=['CPUExecutionProvider'] and
wraps in _suppress_output() so broken GPU drivers don't cause fallback to
try the same broken provider again, and C-extension noise stays suppressed
- scheduler.py: BaseException → Exception (KeyboardInterrupt already re-raised;
winnow has no sys.exit() calls, so SystemExit would not occur, but Exception
is the correct scope)
- compose.yml: fix inverted AUTO_MODE comment (docker run -it enables
interactive mode via TTY, not non-interactive)
Model loading logging (embeddings.py):
- InsightFace: disk cache check, "not cached — downloading now (~300 MB)",
"loading into memory on GPU/CPU...", "ready on GPU/CPU (Xs)"
- SigLIP: same treatment; cache path derived dynamically from model_name
via HuggingFace slug convention (models--org--model) so it stays correct
if the model variant ever changes
Logging level audit (INFO/DEBUG/WARNING/ERROR):
- diversity.py: internal algo steps (clustering, medoids, adaptive threshold,
auto-stop decision) → DEBUG; final selection summaries stay INFO
- immich_api.py: "Fetching assets" and "Retained N assets" → DEBUG (callers
already print this to the console via rprint)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
embeddings.py: face embedding cache used 'immich' for lookup but
'insightface' for storage, so the cache was never hit for locally-
computed embeddings. Unified to 'insightface'/'siglip' throughout.
This affects all users since ENABLE_CACHE now defaults to true.
scheduler.py: INSIGHTFACE_HOME=/models/.insightface was having
'.insightface' appended again, making buffalo_l check always report
'will download'. Also catch BaseException (not just Exception) so a
SystemExit from a library call can't silently kill all future runs.
log_config.py: handlers.clear() abandoned open FileHandler fds on
each scheduled main() call. Close each handler properly before removal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
RETRY_REJECTED was silently read from the environment in _configure_person,
bypassing user control in interactive sessions. Now prompts the user with
the env var value as the default, so the setting is visible and overridable.
All other env vars in the interactive path are already correct:
YEARS_FILTER is a prompt default, ONLY_PEOPLE/SKIP_PEOPLE/MIN_FACE_COUNT
are auto_configure-only, and TRAINING_MODE/STRATEGY/OBJECT_CLASS are
always prompted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three container lifetime modes via CRON_SCHEDULE:
unset — run once on startup, exit
empty string — sleep infinity; use docker exec -it winnow winnow
cron expression — run on startup, then on schedule
This replaces the need for a separate MANUAL_MODE env var. The empty
string is a natural "I want the container alive but unscheduled" signal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The primary use case is headless Docker, so auto mode is now the default
whenever stdin has no TTY. Interactive mode activates when a terminal is
present (docker run -it, local shell). AUTO_MODE=true remains as an
explicit override for scripting with a pseudo-TTY.
Removes AUTO_MODE=true, stdin_open, tty, and FORCE_CPU=false from
compose.yml — none are needed for headless operation. Updates README
and the interactive-mode hint in the CLI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The static LD_LIBRARY_PATH only covered cudnn and cuda_runtime — missing
cublas, cufft, curand, cusolver, cusparse, nvjitlink, etc. onnxruntime-gpu
needs libcublasLt.so at minimum, so GPU mode silently fell back to CPU.
Replace with a one-shot ldconfig call over every nvidia site-packages lib/
dir, which covers all packages regardless of what gets installed.
Also: remove the ambiguous directory="" from preload_dlls (use auto-search
default) and add a clear warning when CUDAExecutionProvider is absent so
the user sees actionable guidance instead of silent CPU fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces a VARIANT=gpu|cpu build arg to the Dockerfile. The cpu
variant uses ubuntu:22.04 (no CUDA base), installs torch+cpu and
onnxruntime (no GPU deps) via a separate pyproject-cpu.toml / uv-cpu.lock,
and is published as :cpu (dev-cpu on the dev branch) via a new
build-cpu CI job. Saves ~2 GB over the default GPU image.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When stdin has no TTY (Docker without -it), IntPrompt/Confirm raise
EOFError and crash the container into a restart loop. Treat a non-TTY
stdin the same as AUTO_MODE=true so headless runs work without any env
var configuration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: ENV INSIGHTFACE_HOME=/models → /models/.insightface to
match compose.yml and .env.example; the old value caused InsightFace
to store models at /models/models/buffalo_l (double-appended subdir)
- entrypoint.sh: use /app/.venv/bin/winnow (installed entry point)
instead of python -m winnow.cli
- config.py: ENABLE_CACHE default false → true; embedding cache is
always beneficial in practice; users can opt out with ENABLE_CACHE=false
- compose.yml: comment out CRON_SCHEDULE so scheduling is opt-in;
flip ENABLE_CACHE to commented opt-out to reflect new default
- README.md: update ENABLE_CACHE default documentation to true
- tests/test_config.py: update default assertion to match
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- __init__.py: derive __version__ from importlib.metadata instead of
a hardcoded "0.1.0" that was six releases out of date
- scheduler.py: move winnow.cli import to module top (no more noqa);
clean up redundant bool variables in check_models
- tests/test_quality.py: 17 tests covering all five quality check
functions individually plus assess_quality integration cases
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- winnow/logging.py → winnow/log_config.py: avoids shadowing the stdlib
logging module; log file renamed from immich_export.log to winnow.log
- scheduler.py: run main() in-process instead of subprocess.run so
InsightFace and SigLIP models stay resident in memory across scheduled
runs (hundreds of MB load, previously reloaded every run)
- .gitignore: replaced 200-line boilerplate with ~30 project-relevant
patterns; removed Django/Flask/Redis/RabbitMQ/Scrapy/etc. noise
- .python-version: untracked (redundant with requires-python in
pyproject.toml; kept in .gitignore for local pyenv users)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- embeddings: initialize ctx_id=-1 before try block so the except
handler cannot NameError; move insightface_home out of try for the
same reason
- embeddings: replace contextlib.redirect_stdout/stderr (Python-level
only) with fd-level dup2 suppression — actually silences C extension
noise from InsightFace during model loading
- jobs: fix frigate_count==0 falling through `or` chain; use explicit
`is not None` check so a real zero is not treated as missing data
- diversity: thread person_id through select_diverse_assets →
_select_by_embedding → _get_face_bbox / _get_face_confidence /
_crop_face_from_thumbnail so group-photo assets embed the target
person's face rather than whichever person is listed first
- config: replace type() hack for ConfigManager with a proper class
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
onnxruntime-gpu has no arm64 wheels (manylinux_2_27_x86_64 /
manylinux_2_28_x86_64 only). uv sync --frozen failed on the arm64
image with exit code 2. Gated onnxruntime-gpu behind the x86_64
marker; arm64 and non-Linux use the CPU onnxruntime package. Added
required-environments so the lockfile is solved for both platforms.
Removed onnxruntime-gpu from override-dependencies (it had no marker
support and blocked arm64 resolution).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
curl|gpg --dearmor was silently dropping the deadsnakes key (gpg exits 0
on bad input), leaving apt unable to find python3.13. Reverted to
add-apt-repository with GNUPGHOME=$(mktemp -d) to isolate gpg from any
pre-existing agent socket. Removed --platform=\$BUILDPLATFORM from the
arm64 base so the image contains real arm64 binaries. License updated
from MIT to AGPL-3.0-or-later.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Title first, then badges on one consistent line. Removed redundant
Release and Lint badges. Fixed docs links to point to the wiki.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ubuntu 26.04 ships Python 3.14, not 3.13. Reverted arm64 to ubuntu:24.04.
Both architectures now add the deadsnakes PPA by fetching the GPG key via
curl and piping through gpg --dearmor — no gpg-agent, safe under QEMU.
Removes the per-arch conditional and ARG TARGETARCH dependency in RUN commands.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Docker automatic platform ARGs are only in scope for FROM instructions.
$TARGETARCH in RUN commands was always empty, so the deadsnakes PPA
conditional never ran and python3.13 could not be found on the Ubuntu
22.04 CUDA base.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All platforms now use Python 3.13. amd64 installs via deadsnakes PPA on
the Ubuntu 22.04 CUDA base; arm64 gets Python 3.13 natively from Ubuntu
26.04. Verified cp313 wheels exist for onnxruntime-gpu 1.26.0 and
torch 2.12.0+cu126. uv.lock regenerated under CPython 3.13.5.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
add-apt-repository ppa:deadsnakes/ppa fails on arm64 in GitHub Actions
because QEMU emulation doesn't support the GPG agent. Ubuntu 24.04 ships
Python 3.12 natively so the PPA is not needed. amd64 (CUDA/Ubuntu 22.04
base) still uses the PPA. PPA install is now gated on TARGETARCH=amd64.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv lock --check in a separate job races against the update-lockfile
bot. Replace with uv lock inline in release.yml and drop the pre-job
from docker-publish.yml entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Store Immich face confidence scores per asset in frigate_uploaded_ids.json
- Add frigate_api.py: query GET /api/faces to count trained images per person
- Record Frigate training count as frigate_count in tracker for offline fallback
- MAX_AUTO_IMAGES cap now uses live Frigate count → cached frigate_count → local uploaded count
- Startup summary shows last known Frigate training count per person
- Migrate by_person entries from flat list to {asset_ids, scores, frigate_count} dict
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds context that winnow is especially useful for people who aren't
around enough for Frigate's live detections to supply adequate training
data on their own.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Advises users to review Frigate uploads after a run and remove bad
crops manually. Links to GitHub Issues for feedback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- entrypoint.sh: replace `uv run` with `/app/.venv/bin/python` to skip
uv's sync check which was re-downloading ruff and rebuilding the package
on every container startup
- compose.yml, .env.example: fix INSIGHTFACE_HOME /models → /models/.insightface;
InsightFace appends models/ to root, so /models produced /models/models/buffalo_l
- README.md: add Immich and Frigate badges from upstream
- CHANGELOG.md, pyproject.toml, uv.lock: bump to 0.2.1
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runtime stage was missing WORKDIR, so uv run started from / and
couldn't find the .venv or pyproject.toml, causing "No module named
'winnow'" on container startup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- pyproject.toml: version 0.1.0 → 0.2.0
- CHANGELOG.md: full [0.2.0] entry covering all changes since the fork —
headless operation, Docker/scheduling, object mode, people filtering,
quality controls, CI/CD, tests, docs, and all bug fixes
- uv.lock: regenerated after version bump
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add docs/setup.md, docs/troubleshooting.md, docs/faq.md as user-facing wiki
- Add .env.example with all env vars and inline comments
- Fix compose.override.yml: rename service if-curator → winnow, update volume paths
- Fix CHANGELOG.md: rename if-curator → winnow in release notes
- Link docs from README
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Describes the full pipeline (fetch → filter → embed → cluster → crop →
upload), explains why diversity matters and how the K-Medoids + FPS
selection works, covers both face and object modes in detail, and
documents all env vars in one place.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- LICENSE: add copyright line for Holden Salomon (retain original)
- pyproject.toml: update authors to Holden Salomon <holden@arch.fyi>
- README: lead with what the software does rather than fork attribution;
merge split env var tables into one unified reference; add lint/test
badges; move attribution to footer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- dependabot.yml: add uv ecosystem (updates pyproject.toml + uv.lock
together), github-actions ecosystem (keeps action versions current),
and group all Python deps into one weekly PR
- lint.yml, test.yml: extend triggers to dev branch so ruff and pytest
run on push/PR to dev, not just main
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- config.py: wire BLUR_THRESHOLD, MIN_CONFIDENCE, MAX_AUTO_IMAGES,
FACE_MARGIN, USE_FULL_RESOLUTION, ENABLE_FACE_ALIGNMENT to env vars
(were hardcoded class defaults, inaccessible in AUTO_MODE)
- jobs.py: add LIMIT env var for custom image count in auto mode;
overrides STRATEGY preset (mirrors interactive Custom Count option)
- compose.yml: document all env vars with inline comments grouped by
concern — mode/strategy, people filtering, image quality, caching,
tracker overrides, scheduling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Volume was mounted at /app/output but OUTPUT_DIR defaults to
./frigate_train (i.e. /app/frigate_train). Mount was never being used.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- upload_to_frigate() now only processes face-mode jobs; object crops are
saved to the output directory as the deliverable (Frigate has no API for
object classifier training data — it must be copied manually)
- execute_jobs() only wipes the output dir for face mode; object crops
accumulate across runs as intended
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- executor.py: wipe person output dir before each run so stale crops
from previous runs don't accumulate on disk
- compose.yml: move inline comment off CRON_SCHEDULE value (croniter
would parse the comment text as part of the expression)
- compose.yml: add note clarifying <pool> placeholder in volume paths
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: separate runtime stage from build stage so g++,
python3.12-dev, curl, gnupg are excluded from the final image
- uv sync --no-dev: drop ruff/pytest from production image
- Remove build.sh (replaced by CI) and root-level upload_tracker.py (stale duplicate)
- .gitignore: add *.log
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- get_people() and fetch_all_assets() only caught RequestException, leaving
JSONDecodeError unhandled if Immich returns a non-JSON 200 response
- cli.py thumbnail fetch used a raw header dict instead of get_headers()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv sync was after COPY if_curator/, so any source change invalidated
the 800MB dependency cache and re-downloaded everything. Now deps are
installed first and stay cached across code-only changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
upload_to_frigate() was scanning the output directory with os.listdir(),
picking up leftover files from previous runs and re-uploading them.
Use asset_map (populated by execute_jobs) to upload only current-run files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PIL opens JPEGs without applying EXIF rotation. Immich computes face
bounding boxes on orientation-corrected images, so portrait photos
produced misaligned crops (showing chins/eyes/foreheads instead of
whole faces). ImageOps.exif_transpose() normalizes orientation before
any coordinate math.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Hardcoded root="~/.insightface" ignored the INSIGHTFACE_HOME env var,
causing Buffalo_L to download into the container on every run instead
of persisting to the mounted volume.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix RGBA→JPEG error: convert all images to RGB before saving
- Upload tracker: per-person breakdown in JSON, reset_person(), get_person_summary()
- mark_rejected() only fires on face-detection failures (not all HTTP 400s)
- New env vars: DRY_RUN, RETRY_REJECTED, RESET_PERSON
- Tracker summary printed at startup showing uploaded/rejected counts per person
- Document new env vars in compose.yml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flanandClaude Sonnet 4.6