* refactor: rename CACHE_DIR to DATA_DIR, default path .if_cache → data
CACHE_DIR held both the embedding cache and the SQLite tracker DB, making
the name misleading. DATA_DIR is more accurate.
- Config reads DATA_DIR first; falls back to CACHE_DIR with a deprecation
warning so existing setups don't break on upgrade
- Default local path: data (was .if_cache)
- Docker default path: /app/data (was /app/.if_cache)
- Internal references (embeddings.py, upload_tracker.py) updated to DATA_DIR
- compose.yml, .env.example, README, wiki, and changelog updated
- Version bumped to 0.5.1
* chore: update lockfile
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
- release.yml: replace per-variant lockfile generation loop with a
single 'uv lock'; add idempotent release creation (skip if tag
already has a release so re-triggered runs don't 422)
- docker-publish.yml: remove stale uv-cpu/rocm/intel.lock entries
from paths-ignore (those files no longer exist)
- README: CACHE_DIR description now names winnow_tracker.db; tracker
description mentions SQLite
Two fixes found during post-refactor audit:
1. upload_tracker: remove COUNT(*) guard from _maybe_migrate. The guard
blocked re-migration when a previous run successfully committed both
JSON files but a PermissionError on the second rename() left it on
disk. On the next startup COUNT > 0 → early return → rejected IDs
permanently unimported. INSERT OR IGNORE is idempotent so re-running
migration is always safe; guard not needed.
Also wrap each rename() in its own try/except so a failure on one
file is logged and does not propagate uncaught.
2. reconcile: break early when new_count > target is detected in the
poll loop. Previously the loop ran all four delay intervals (1+2+4+8s)
before the post-loop > target branch fired, wasting up to 15 seconds
when a concurrent external upload was visible on the first poll.
- upload_tracker: partial migration now rolls back atomically on failure;
JSON renamed only after successful commit so failed runs retry cleanly
- upload_tracker: allowlist score_col in _pick_mapped_file to close
latent SQL injection surface
- config: move load_dotenv() from module import into _load() so no I/O
at import time and reset() fully resets env loading
- config: use is None checks for IMMICH_URL/OUTPUT_DIR config-file
fallback so explicitly empty env vars are not overridden by the file
- executor: skip reconcile when Frigate API is unreachable at upload
start — tracker baseline is incomplete and would mis-trigger the
external-upload guard, permanently losing file mappings
- reconcile: change polling break condition from >= to == target so
transient overshoots don't prematurely exit the loop and trigger
the external-upload guard
- jobs: apply capacity cap as the selection limit rather than truncating
post-selection by position, so the diversity algorithm works within
the right budget from the start
- Dockerfile: explicit gpu branch + exit 1 on unknown VARIANT instead
of silent fallback
_build_job no longer calls filter_already_uploaded internally; callers pass
pre-filtered assets so there's no double DB hit and the interactive path
restores the original prompt order (retry_rejected asked before strategy,
so post-dedup count informs the choice). Skip-count rprint restored in
auto_configure. Late 'import time' inside reconcile_frigate_mappings moved
to module level.
- rocm/intel/gpu extras are x86_64-only; aarch64 wheels don't exist so uv
failed to resolve them when required-environments includes aarch64
- test.yml: switch from --all-extras (broken by conflicts + missing wheels)
to --extra cpu which is cross-platform and sufficient for unit tests
- update-lockfile.yml: drop old file-swap loop; single pyproject means a
single uv lock run and a single uv.lock to commit
- Config: remove _ConfigAccessor and ConfigManager; use __getattr__ for lazy
loading on single _Config class; re-register self as _instance in __getattr__
so reset() always clears the correct object (item 1)
- upload_tracker: replace hand-rolled JSON store with sqlite3; auto-migrates
existing JSON on first run; remove dead record_frigate_file function;
connection re-opens when CACHE_DIR changes for test isolation (items 2, 8)
- diversity: move ThreadPoolExecutor import to module level; inject optional
fetch_fn parameter for testability (items 3, 6)
- pyproject: consolidate 4 variant files into extras (gpu/rocm/intel/cpu);
update Dockerfile to use --extra flag; delete variant pyproject/lock files;
uv.lock needs regen with `uv lock` after this change (item 4)
- jobs: extract _build_job helper to separate business logic from terminal I/O;
auto_configure delegates dedup/selection to _build_job (item 5)
- logging: convert f-string log calls to % interpolation throughout all winnow/
modules (item 7)
- reconcile: new module with reconcile_frigate_mappings and
enrich_asset_with_face_data extracted from executor.py (item 9)
- scheduler: print next scheduled run time after startup and after each run;
fix f-string logger.error call (item 10)
22.04 non-GPU amd64 bases were never updated when arm64 moved to 24.04.
Python 3.13 is still pulled from deadsnakes PPA (26.04 ships 3.14 natively).
intel stays on 22.04: the Intel GPU repo URL is pinned to the "jammy"
codename and cannot be bumped until Intel publishes 26.04 packages.
cache.py — model fingerprint auto-invalidation:
Replace hardcoded "buffalo_l_v1" version string with a fingerprint
derived from buffalo_l .onnx file sizes and mtimes. EmbeddingCache now
computes this at init time; stale embeddings from replaced or updated
model files are automatically invalidated. Falls back to the static
string before the model is downloaded.
Note: existing caches built against the old key will miss on the first
run after upgrade and recompute cleanly.
frigate_api.py — Frigate version check:
Add get_frigate_version() (GET /api/version). Called at the start of
upload_to_frigate(); warns if below v0.16 where the face training API
endpoints don't exist.
immich_api.py + cli.py — Immich version check:
Add get_immich_version() (GET /api/server/version). Called at startup
before get_people(); warns if below v1.106 where the face data and
merge APIs winnow depends on aren't guaranteed present.
Remaining TODO(frigate-api) annotations are left in place — they require
Frigate to expose per-file embeddings or a rebuild-complete signal before
they can be addressed.
Adds inline LIMITATION / TODO(frigate-api) comments at each specific
code site rather than a separate doc that would drift from the code.
frigate_api.py — recognize_face:
Mean-embedding limitation: score reflects the arithmetic mean of all
training embeddings. A bimodal set (frontals + profiles) has a mean
between clusters, making both ends look more novel than they are.
Fixable if Frigate exposes per-file embeddings for nearest-neighbour
comparison.
frigate_api.py — get_all_frigate_person_files:
"train" key exclusion is a hardcoded string. If Frigate adds other
special top-level keys in /api/faces they'll be silently treated as
person names. Needs a typed schema when Frigate documents the contract.
executor.py — recognize_face call site:
Async rebuild: each deletion triggers a background model rebuild in
Frigate. Subsequent recognize calls in the same run return None
(rebuild in progress), degrading quality replacement for later
candidates. Fixable with a rebuild-complete signal from Frigate.
executor.py — effective_count / manual file handling:
Manually-added files are invisible to diversity decisions. Winnow
observes their effect only indirectly via the Frigate score, not by
measuring their embedding distribution. Per-file embeddings from
Frigate would allow direct diversity measurement against the full set.
executor.py — Frigate version assumption:
All face training endpoints are v0.16+. No version check at startup;
failures on older versions are opaque 404s.
cache.py — MODEL_VERSIONS:
Version string is a hardcoded constant. Manual model file replacement
(custom weights, InsightFace update) won't invalidate cached embeddings.
Needs file-checksum-derived versioning or a CLEAR_EMBEDDING_CACHE flag.
diversity.py — thumbnail-resolution embeddings:
Diversity selection runs InsightFace on preview thumbnails; the actual
training crop comes from full-resolution originals. Negligible in
practice but degrades if Immich preview quality is low.
Tests cover the core ML pipeline algorithms in diversity.py — previously
untested. No network or model dependencies; all pure-function or
numpy-only paths:
- Face bbox and confidence extraction from Immich metadata, including
person_id filtering and missing-data edge cases
- Face crop scaling: verifies bbox coordinates are correctly scaled when
the thumbnail dimensions differ from the metadata image dimensions
- Near-duplicate dedup: removal below cosine threshold, quality-score
preference between duplicates, zero-quality-score treated as zero not
missing (falsy bug guard)
- K-Medoids: correct medoid count, distinctness, valid index range, and
full-N edge case
- Adaptive threshold: positive output, floor at 0.05 for identical
embeddings, single-point, scales with embedding spread
- Time-spread fallback: exact count, all-under-limit passthrough,
auto→30 default, first/last inclusion
- Cluster-aware selection: exact limit, subset invariant, auto-stop on
tight cluster, hard-example confidence weighting accepted
Remove torch/torchvision/transformers/ultralytics from all three variant
pyproject files (rocm/cpu/intel) — no longer needed since SigLIP and YOLO
were removed in v0.4.10. Update version to 0.4.10 and description.
NOTE: uv-rocm.lock, uv-cpu.lock, uv-intel.lock are now stale and must be
regenerated in their respective platform environments before the next Docker
build of those variants.
Also fix fetch_face_data() docstring to not mention embedding (removed field).
- immich_api.py: remove FaceData.embedding field and numpy import — Immich face
embeddings were never consumed after being fetched; bbox/confidence is all that's used
- embeddings.py: remove immich_embedding param from get_embedding() — never passed by
any caller; simplify docstring accordingly
- image_processing.py: wrap insightface_app.get() in warnings filter to suppress the
scikit-image FutureWarning about estimate being deprecated (already suppressed in
embeddings.py for the diversity path, was leaking from the crop-alignment path)
- README.md: remove two stale "object mode" / "object classification" fragments
After removing the object pipeline, several dead 'mode' artifacts remained:
- executor.py: unpack `config` from job even though it was no longer read
- jobs.py: set `"mode": "face"` in both configure paths (key never consumed)
- compose.yml: TRAINING_MODE=face env, OBJECT_CLASS comment, HF_HOME, stale MAX_AUTO_IMAGES default note
- Dockerfile: "and object classification" label, /models/huggingface mkdir, HF_HOME ENV
- cache.py: drop siglip MODEL_VERSIONS entry
- log_config.py: drop ultralytics/transformers/torch from noise silencers
- scheduler.py: drop HF_HOME and HuggingFace model check
- scripts/benchmark.py: drop bench_siglip and make_random_image
winnow is a face recognition training tool. Object mode required manual
file placement with no Frigate API, pulled in torch/torchvision/transformers/
ultralytics (~2 GB), and was architecturally misaligned with the project goal.
Removed:
- process_object_mode (YOLO inference), get_yolo_model
- SigLIP model stack (get_siglip_model, get_object_embedding, batch variant)
- entity_type branching throughout diversity, embeddings, jobs, executor
- TRAINING_MODE and OBJECT_CLASS env vars
- torch, torchvision, transformers, ultralytics dependencies
- pytorch index entries from pyproject.toml
- Object mode from README (Modes section, env var table, How It Works)
flanandgithub-actions[bot]