All _load() calls after the first return the cached dict instead of
re-reading disk. _save() updates both disk and cache atomically.
Drops per-person tracker reads from ~90 to ~1 in the upload loop.
Keyed by resolved file path so test isolation (unique tmp_path dirs)
is preserved with no fixture changes needed.
- _dedup_embeddings: pre-allocated (Q,D) buffer replaces vstack-on-keep,
dropping O(K²×D) copy overhead down to O(K×D) fill work
- _kmedoids: swap cost sum replaced with numpy fancy-index reduction,
~20-50x faster per swap evaluation
- _reconcile_frigate_mappings: O(L) load/save pairs collapsed to one
batch write via record_frigate_files_batch
Fetching up to MAX_PAGES*page_size (1M) assets before the 3000-item
diversity pool cap was applied could exhaust memory on large Immich
libraries. Early-exit once 5000 items are collected — the pool cap
of 3000 makes anything beyond that wasteful. Also filter null/non-dict
items from page responses at fetch time.
- _dedup_embeddings: rebuild kept_stack only on keep (was every iteration → O(N²))
- _dedup_embeddings: fix quality_score sort key to use explicit None check (falsy-zero)
- _select_by_embedding: add post-dedup pool < limit guard with warning
- executor: use full resp.text for 'face' keyword check; only truncate display snippet
- _safe_person_dir: avoid false "//" prefix when output_dir resolves to filesystem root
0.10 only removed burst shots (distance 0.01-0.05). Same-event photos with
similar pose and lighting sit at 0.10-0.20 and were passing through,
producing visually similar training images especially for people with small
datasets. 0.20 removes these while still preserving genuinely different
poses, expressions, and lighting conditions.
os.path.join silently discards the base when the second arg is absolute,
and '../..' sequences escape the output tree. _safe_person_dir() resolves
both paths with realpath and rejects any name that lands outside the output
directory, logging an error and skipping the job rather than touching an
unintended path.
- RESET_PERSON=* resets every tracked person (deletes their Frigate files
and clears the tracker). Any other value resets that specific person by
name, including someone literally named 'all'.
- Near-duplicate removal pass added before diversity clustering: greedily
drops candidates within 0.10 cosine distance of a higher-quality image,
eliminating burst-shot duplicates that FPS would otherwise pass through.
Burst shots produce embeddings that differ slightly (~0.01-0.05 cosine
distance) due to JPEG noise and minor lighting variation, so FPS does not
filter them. Add a greedy dedup pass after embedding collection: sort
candidates by quality score descending, then drop any candidate within
0.10 cosine distance of an already-kept image. The best frame from each
near-identical group survives; the rest are dropped before clustering.
- HTTP 500 errors from Frigate no longer echo the response body to the user
(Frigate's generic message says 'Try restarting Frigate' which is wrong —
500s on upload are almost always image-specific, not a health issue). The
detail is now logged at debug level. HTTP 400 detail is still shown since
'No face was detected' is genuinely useful.
- np.median on empty upper triangle (n=1 after quality filtering) no longer
emits RuntimeWarning; _compute_adaptive_threshold returns the floor (0.05)
immediately when there are no pairwise distances to sample.
- k-medoids cluster count floor raised to 1 (was 0 when n < 3), preventing
k=0 being passed to _kmedoids.
When Immich has multiple person records sharing a name (e.g. unmerged
face clusters), winnow would previously run separate jobs for each,
with the second job wiping the first job's output directory — resulting
in far fewer training images than expected.
New behaviour:
- At startup, duplicate names are detected and a warning is printed
showing asset counts for each duplicate.
- By default (MERGE_DUPLICATE_PEOPLE=false), only the person with the
most assets is processed; smaller duplicates are skipped cleanly.
- With MERGE_DUPLICATE_PEOPLE=true, the duplicates are permanently
merged inside Immich via PUT /api/people/{id}/merge (keeps the
largest), then the people list is re-fetched before jobs run.
Also adds an explicit comment in executor.py confirming that replacement
targets come exclusively from tracker-mapped files, so manually-added
Frigate training images are never selected for deletion.
Immich's /api/faces endpoint only returns bounding boxes, not facial
landmarks. This meant align_face() never fired and all crops fell back
to a plain bbox rectangle — producing partial crops (forehead-only,
off-angle faces) when Immich's detection was slightly off.
Now, when ENABLE_FACE_ALIGNMENT is true and InsightFace is loaded,
execute_jobs() passes the app to process_face_mode(). For each face,
it expands the Immich bbox by 50%, crops that search region, runs
InsightFace detection within it, and aligns the nearest face to the
standard ArcFace 112×112 format using norm_crop(). Falls back to bbox
crop if InsightFace finds no face in the search region.
The InsightFace model is already in GPU memory from the diversity/
embedding phase, so the singleton lookup adds no load cost.
CUDA 13.3 requires driver >= 575 but the host only has 570 (error 804).
CUDA 12.8.1 is the highest version supported by driver 570 and works
correctly with the NVIDIA Container Toolkit.
Add scripts/benchmark.py to measure InsightFace + SigLIP latency and
throughput across GPU and CPU modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
release.yml now calls docker-publish.yml via workflow_call instead of
re-running all four image builds independently. docker-publish.yml gains
workflow_call inputs (tag, version) for release context; branch trigger
is narrowed to dev only (main changes only land via tagged releases).
Each release previously built all four variants twice (~90 min) — once on
merge to main, once on tag push. Now it builds once.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CUDA 13.3 requires driver >= 575 but the host only has 570 (error 804).
CUDA 12.8.1 is the highest version supported by driver 570 and works
correctly with the NVIDIA Container Toolkit.
Add scripts/benchmark.py to measure InsightFace + SigLIP latency and
throughput across GPU and CPU modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
release.yml now calls docker-publish.yml via workflow_call instead of
re-running all four image builds independently. docker-publish.yml gains
workflow_call inputs (tag, version) for release context; branch trigger
is narrowed to dev only (main changes only land via tagged releases).
Each release previously built all four variants twice (~90 min) — once on
merge to main, once on tag push. Now it builds once.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously reset_person wiped the local tracker but left existing Frigate
training files as orphans, causing the next run to upload a full new batch
on top of them. Now deletes all winnow-managed files from Frigate first so
the next run starts truly clean. Manually-added Frigate files are never
touched.
Also fixes a spurious warning when FRIGATE_URL is unset: the deletion step
is now skipped at info level rather than logging a misleading error. Moves
the deferred import to top-level and eliminates a double disk read.
Bumps to 0.4.1. Also fixes ruff lint violations in executor.py (import
sort, line length) and promotes the "winnow only touches files it uploaded"
callout to the README intro.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- upload_tracker: extract _pick_mapped_file() private helper; get_lowest_quality_mapped_file
and get_most_redundant_mapped_file are now one-liners over the same body
- upload_tracker: remove_frigate_file now also prunes the corresponding frigate_scores entry,
preventing unbounded accumulation of orphaned score entries across replacement cycles
- frigate_api: get_frigate_face_counts delegates to get_all_frigate_person_files, eliminating
the duplicated "name != 'train' and isinstance(files, list)" filter body
- executor: cache has_frigate_scores(name) as person_has_fscores before the per-file loop;
refresh it after each remove_frigate_file call and after each scored upload, eliminating
two redundant disk reads per at-cap file iteration
- executor: casefold() both sides of the recognize_face person-name comparison so a Frigate
casing normalization or manual-registration casing mismatch does not silently suppress scoring
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frigate pre-upload scoring, quality replacement inversion, bootstrap fix,
FRIGATE_SCORE_CEILING / ENABLE_FRIGATE_SCORES, removal of post-upload gate.
See CHANGELOG.md [0.4.0] for the full list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flan
github-actions[bot]