Commit Graph
41 Commits
Author SHA1 Message Date
flan 9685c310af fix: symlink guard placement, fetch_all_assets raw count, config TOCTOU (#31)
executor.py:
- Move islink check into _safe_person_dir on the raw path, before realpath
  resolves it; the previous check at the rmtree site was unreachable dead code
  because realpath already followed any symlink

immich_api.py / jobs.py:
- fetch_all_assets now returns (assets, total_raw) where total_raw is the
  item count seen before non-dict filtering; callers use it for MIN_FACE_COUNT
  guard and display so transient non-dict API items can't incorrectly skip people
- Add WARNING when pagination stops because a page had items but all were non-dict

config.py:
- Cache _data_cfg.exists() in _data_cfg_exists so the dual-config warning
  and config_file selection always read from the same stat() result; previously
  two calls created a TOCTOU window where log and code could disagree

Bump version to 0.5.7
2026-06-14 20:27:16 -04:00
flan 84ebd91929 fix: quality replacement slot floor uses deleted file's score not failed candidate's (#27)
* fix: quality replacement slot floor uses deleted file's score not failed candidate's

When a blur-score replacement deletes a low-quality Frigate file but the
subsequent upload fails, min_quality_score_for_slot was set to candidate_score
(the good file that failed to upload). This filtered out any subsequent
candidate that didn't beat the failed upload, even if it was better than
the file we just deleted — leaving the freed slot unfilled unnecessarily.

The comment on the guard already documented the correct intent: 'require
the next candidate to beat the deleted file's score'. Fix: use target_score
(the deleted file's blur score) as the floor instead of candidate_score.

* chore: bump version to 0.5.4
2026-06-14 19:53:33 -04:00
flan 2e08504682 fix: audit hardening — input validation, error handling, and robustness (#25)
* fix: audit hardening — input validation, error handling, and robustness

- immich_api: guard person["id"] with .get() + early return on missing field
- immich_api: include page number in pagination exception log
- immich_api: validate faces response is a list before indexing
- executor: wrap Image.open() in try/except for non-image HTTP responses
- executor: strip leading 'v' from Frigate version before parsing (v0.16.0 was misread)
- config: wrap FRIGATE_SCORE_CEILING float() parse in try/except with warning
- config: warn when both DATA_DIR and legacy CWD config files exist simultaneously
- scheduler: wrap PID file write in try/except so /tmp failures don't crash startup
- scheduler: clamp sleep to 60s max to bound recovery time after NTP clock jumps
- frigate_api: log unexpected non-list type in get_frigate_person_files at DEBUG

* fix: LIMIT env var crash and symlink guard on person output dir

- jobs: wrap int(LIMIT) parse in try/except — bad value (e.g. "30.5", "all")
  now logs a warning and falls back to the default instead of crashing
- executor: check for symlink before shutil.rmtree on person_dir — prevents
  following a symlink out of OUTPUT_DIR on a shared volume

* chore: bump version to 0.5.3
2026-06-14 19:39:35 -04:00
flan cabdb9c0eb fix: address 9 code review findings
- upload_tracker: partial migration now rolls back atomically on failure;
  JSON renamed only after successful commit so failed runs retry cleanly
- upload_tracker: allowlist score_col in _pick_mapped_file to close
  latent SQL injection surface
- config: move load_dotenv() from module import into _load() so no I/O
  at import time and reset() fully resets env loading
- config: use is None checks for IMMICH_URL/OUTPUT_DIR config-file
  fallback so explicitly empty env vars are not overridden by the file
- executor: skip reconcile when Frigate API is unreachable at upload
  start — tracker baseline is incomplete and would mis-trigger the
  external-upload guard, permanently losing file mappings
- reconcile: change polling break condition from >= to == target so
  transient overshoots don't prematurely exit the loop and trigger
  the external-upload guard
- jobs: apply capacity cap as the selection limit rather than truncating
  post-selection by position, so the diversity algorithm works within
  the right budget from the start
- Dockerfile: explicit gpu branch + exit 1 on unknown VARIANT instead
  of silent fallback
2026-06-14 20:37:51 +00:00
flan e2a1924fb0 refactor: collapse Config proxy, migrate tracker to SQLite, split reconcile module
- Config: remove _ConfigAccessor and ConfigManager; use __getattr__ for lazy
  loading on single _Config class; re-register self as _instance in __getattr__
  so reset() always clears the correct object (item 1)
- upload_tracker: replace hand-rolled JSON store with sqlite3; auto-migrates
  existing JSON on first run; remove dead record_frigate_file function;
  connection re-opens when CACHE_DIR changes for test isolation (items 2, 8)
- diversity: move ThreadPoolExecutor import to module level; inject optional
  fetch_fn parameter for testability (items 3, 6)
- pyproject: consolidate 4 variant files into extras (gpu/rocm/intel/cpu);
  update Dockerfile to use --extra flag; delete variant pyproject/lock files;
  uv.lock needs regen with `uv lock` after this change (item 4)
- jobs: extract _build_job helper to separate business logic from terminal I/O;
  auto_configure delegates dedup/selection to _build_job (item 5)
- logging: convert f-string log calls to % interpolation throughout all winnow/
  modules (item 7)
- reconcile: new module with reconcile_frigate_mappings and
  enrich_asset_with_face_data extracted from executor.py (item 9)
- scheduler: print next scheduled run time after startup and after each run;
  fix f-string logger.error call (item 10)
2026-06-14 19:59:17 +00:00
flan 28424b0f16 fix: implement fixable limitations from annotation pass
cache.py — model fingerprint auto-invalidation:
  Replace hardcoded "buffalo_l_v1" version string with a fingerprint
  derived from buffalo_l .onnx file sizes and mtimes. EmbeddingCache now
  computes this at init time; stale embeddings from replaced or updated
  model files are automatically invalidated. Falls back to the static
  string before the model is downloaded.
  Note: existing caches built against the old key will miss on the first
  run after upgrade and recompute cleanly.

frigate_api.py — Frigate version check:
  Add get_frigate_version() (GET /api/version). Called at the start of
  upload_to_frigate(); warns if below v0.16 where the face training API
  endpoints don't exist.

immich_api.py + cli.py — Immich version check:
  Add get_immich_version() (GET /api/server/version). Called at startup
  before get_people(); warns if below v1.106 where the face data and
  merge APIs winnow depends on aren't guaranteed present.

Remaining TODO(frigate-api) annotations are left in place — they require
Frigate to expose per-file embeddings or a rebuild-complete signal before
they can be addressed.
2026-06-14 18:48:07 +00:00
flan 1d44df6e96 docs: annotate known limitations and Frigate API improvement hooks
Adds inline LIMITATION / TODO(frigate-api) comments at each specific
code site rather than a separate doc that would drift from the code.

frigate_api.py — recognize_face:
  Mean-embedding limitation: score reflects the arithmetic mean of all
  training embeddings. A bimodal set (frontals + profiles) has a mean
  between clusters, making both ends look more novel than they are.
  Fixable if Frigate exposes per-file embeddings for nearest-neighbour
  comparison.

frigate_api.py — get_all_frigate_person_files:
  "train" key exclusion is a hardcoded string. If Frigate adds other
  special top-level keys in /api/faces they'll be silently treated as
  person names. Needs a typed schema when Frigate documents the contract.

executor.py — recognize_face call site:
  Async rebuild: each deletion triggers a background model rebuild in
  Frigate. Subsequent recognize calls in the same run return None
  (rebuild in progress), degrading quality replacement for later
  candidates. Fixable with a rebuild-complete signal from Frigate.

executor.py — effective_count / manual file handling:
  Manually-added files are invisible to diversity decisions. Winnow
  observes their effect only indirectly via the Frigate score, not by
  measuring their embedding distribution. Per-file embeddings from
  Frigate would allow direct diversity measurement against the full set.

executor.py — Frigate version assumption:
  All face training endpoints are v0.16+. No version check at startup;
  failures on older versions are opaque 404s.

cache.py — MODEL_VERSIONS:
  Version string is a hardcoded constant. Manual model file replacement
  (custom weights, InsightFace update) won't invalidate cached embeddings.
  Needs file-checksum-derived versioning or a CLEAR_EMBEDDING_CACHE flag.

diversity.py — thumbnail-resolution embeddings:
  Diversity selection runs InsightFace on preview thumbnails; the actual
  training crop comes from full-resolution originals. Negligible in
  practice but degrades if Immich preview quality is low.
2026-06-14 18:41:37 +00:00
flan 274d50ee99 chore: remove remaining dead-mode references from executor, jobs, compose, Dockerfile
After removing the object pipeline, several dead 'mode' artifacts remained:

- executor.py: unpack `config` from job even though it was no longer read
- jobs.py: set `"mode": "face"` in both configure paths (key never consumed)
- compose.yml: TRAINING_MODE=face env, OBJECT_CLASS comment, HF_HOME, stale MAX_AUTO_IMAGES default note
- Dockerfile: "and object classification" label, /models/huggingface mkdir, HF_HOME ENV
2026-06-14 17:42:27 +00:00
flan 835016e0e3 feat: remove object mode pipeline (YOLO, SigLIP, TRAINING_MODE, OBJECT_CLASS)
winnow is a face recognition training tool. Object mode required manual
file placement with no Frigate API, pulled in torch/torchvision/transformers/
ultralytics (~2 GB), and was architecturally misaligned with the project goal.

Removed:
- process_object_mode (YOLO inference), get_yolo_model
- SigLIP model stack (get_siglip_model, get_object_embedding, batch variant)
- entity_type branching throughout diversity, embeddings, jobs, executor
- TRAINING_MODE and OBJECT_CLASS env vars
- torch, torchvision, transformers, ultralytics dependencies
- pytorch index entries from pyproject.toml
- Object mode from README (Modes section, env var table, How It Works)
2026-06-14 17:29:25 +00:00
flan 105099c819 fix: mark assets rejected when faces API confidence is below threshold
Previously, assets that passed embedding-phase selection but failed the
faces API confidence check in execute_jobs were silently skipped with no
tracker entry. They appeared as valid candidates on every future run,
were re-selected, and re-skipped in an endless cycle. Now they are
marked rejected so they are excluded from future runs. RETRY_REJECTED=true
clears them if Immich later re-processes the image.
2026-06-14 16:46:09 +00:00
flan 0afc9386c6 feat: dynamic Frigate score ceiling; consolidate quality replacement branches
FRIGATE_SCORE_CEILING now defaults to dynamic mode (unset): below-cap
candidates are skipped if their pre-upload Frigate score exceeds the
most-redundant tracked file's score. This catches conditions already
covered by manually-added Frigate images that winnow cannot track —
the embedding-based diversity selection has no visibility into those.
Set FRIGATE_SCORE_CEILING=0 to disable; a positive value (e.g. 0.85)
still acts as a fixed hard ceiling. First-run safety is unchanged
(pre_run_count==0 prevents recognize_face from being called).

The two quality replacement branches (Frigate-score and blur-score)
shared identical structure and are merged into a single code path
parameterised by score source and comparison direction.

Also raises MIN_FACE_COUNT default from 0 to 3 and updates the
config test to match.
2026-06-14 16:38:25 +00:00
flan ec074fd279 fix: remove unused record_frigate_file import (ruff F401) 2026-06-14 04:10:15 +00:00
flan 7dce4a5c71 perf: eliminate O(K²) dedup allocs, vectorize kmedoids cost, batch tracker writes
- _dedup_embeddings: pre-allocated (Q,D) buffer replaces vstack-on-keep,
  dropping O(K²×D) copy overhead down to O(K×D) fill work
- _kmedoids: swap cost sum replaced with numpy fancy-index reduction,
  ~20-50x faster per swap evaluation
- _reconcile_frigate_mappings: O(L) load/save pairs collapsed to one
  batch write via record_frigate_files_batch
2026-06-14 04:03:55 +00:00
flan 3c602e2ef6 fix: code review corrections — dedup O(N²), truncated rejection check, pool warning, path guard
- _dedup_embeddings: rebuild kept_stack only on keep (was every iteration → O(N²))
- _dedup_embeddings: fix quality_score sort key to use explicit None check (falsy-zero)
- _select_by_embedding: add post-dedup pool < limit guard with warning
- executor: use full resp.text for 'face' keyword check; only truncate display snippet
- _safe_person_dir: avoid false "//" prefix when output_dir resolves to filesystem root
2026-06-14 03:40:39 +00:00
flan 6e34d41036 Guard person-name path traversal in output directory construction
os.path.join silently discards the base when the second arg is absolute,
and '../..' sequences escape the output tree. _safe_person_dir() resolves
both paths with realpath and rejects any name that lands outside the output
directory, logging an error and skipping the job rather than touching an
unintended path.
2026-06-14 01:48:20 +00:00
flan eb3abe2cca Fix misleading HTTP 500 detail and RuntimeWarning on single-image selection
- HTTP 500 errors from Frigate no longer echo the response body to the user
  (Frigate's generic message says 'Try restarting Frigate' which is wrong —
  500s on upload are almost always image-specific, not a health issue). The
  detail is now logged at debug level. HTTP 400 detail is still shown since
  'No face was detected' is genuinely useful.
- np.median on empty upper triangle (n=1 after quality filtering) no longer
  emits RuntimeWarning; _compute_adaptive_threshold returns the floor (0.05)
  immediately when there are no pairwise distances to sample.
- k-medoids cluster count floor raised to 1 (was 0 when n < 3), preventing
  k=0 being passed to _kmedoids.
2026-06-14 01:23:46 +00:00
flan 2dd911e9ea Detect and handle duplicate Immich people with the same name
When Immich has multiple person records sharing a name (e.g. unmerged
face clusters), winnow would previously run separate jobs for each,
with the second job wiping the first job's output directory — resulting
in far fewer training images than expected.

New behaviour:
- At startup, duplicate names are detected and a warning is printed
  showing asset counts for each duplicate.
- By default (MERGE_DUPLICATE_PEOPLE=false), only the person with the
  most assets is processed; smaller duplicates are skipped cleanly.
- With MERGE_DUPLICATE_PEOPLE=true, the duplicates are permanently
  merged inside Immich via PUT /api/people/{id}/merge (keeps the
  largest), then the people list is re-fetched before jobs run.

Also adds an explicit comment in executor.py confirming that replacement
targets come exclusively from tracker-mapped files, so manually-added
Frigate training images are never selected for deletion.
2026-06-13 23:58:38 +00:00
flan 341c6b0e85 Use InsightFace for landmark-based face crop alignment
Immich's /api/faces endpoint only returns bounding boxes, not facial
landmarks. This meant align_face() never fired and all crops fell back
to a plain bbox rectangle — producing partial crops (forehead-only,
off-angle faces) when Immich's detection was slightly off.

Now, when ENABLE_FACE_ALIGNMENT is true and InsightFace is loaded,
execute_jobs() passes the app to process_face_mode(). For each face,
it expands the Immich bbox by 50%, crops that search region, runs
InsightFace detection within it, and aligns the nearest face to the
standard ArcFace 112×112 format using norm_crop(). Falls back to bbox
crop if InsightFace finds no face in the search region.

The InsightFace model is already in GPU memory from the diversity/
embedding phase, so the singleton lookup adds no load cost.
2026-06-13 23:13:50 +00:00
flanandClaude Sonnet 4.6 405413490b fix: RESET_PERSON now deletes managed Frigate files before clearing tracker
Previously reset_person wiped the local tracker but left existing Frigate
training files as orphans, causing the next run to upload a full new batch
on top of them. Now deletes all winnow-managed files from Frigate first so
the next run starts truly clean. Manually-added Frigate files are never
touched.

Also fixes a spurious warning when FRIGATE_URL is unset: the deletion step
is now skipped at info level rather than logging a misleading error. Moves
the deferred import to top-level and eliminates a double disk read.

Bumps to 0.4.1. Also fixes ruff lint violations in executor.py (import
sort, line length) and promotes the "winnow only touches files it uploaded"
callout to the README intro.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 19:58:08 +00:00
flanandClaude Sonnet 4.6 e67f2d9638 refactor: cleanup audit findings — dedup helpers, prune orphan scores, cache has_frigate_scores
- upload_tracker: extract _pick_mapped_file() private helper; get_lowest_quality_mapped_file
  and get_most_redundant_mapped_file are now one-liners over the same body
- upload_tracker: remove_frigate_file now also prunes the corresponding frigate_scores entry,
  preventing unbounded accumulation of orphaned score entries across replacement cycles
- frigate_api: get_frigate_face_counts delegates to get_all_frigate_person_files, eliminating
  the duplicated "name != 'train' and isinstance(files, list)" filter body
- executor: cache has_frigate_scores(name) as person_has_fscores before the per-file loop;
  refresh it after each remove_frigate_file call and after each scored upload, eliminating
  two redundant disk reads per at-cap file iteration
- executor: casefold() both sides of the recognize_face person-name comparison so a Frigate
  casing normalization or manual-registration casing mismatch does not silently suppress scoring

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 18:55:18 +00:00
flanandClaude Sonnet 4.6 6fcea587ff chore: bump version to 0.4.0, update changelog and all docs
Finalizes the 0.4.0 release:

- Version bumped to 0.4.0 in pyproject.toml
- CHANGELOG.md: add [0.4.0] section covering Frigate pre-upload scoring,
  quality replacement inversion, bootstrap fix, FRIGATE_SCORE_CEILING,
  ENABLE_FRIGATE_SCORES, removal of post-upload quality gate, and all
  doc/default corrections
- README.md: step 8 updated for dual-mode replacement, FRIGATE_SCORE_CEILING
  and ENABLE_FRIGATE_SCORES added to env var table, MIN_FACE_WIDTH and
  BLUR_THRESHOLD defaults corrected (50→90, 100→120)
- .env.example: FRIGATE_SCORE_THRESHOLD replaced with FRIGATE_SCORE_CEILING;
  QUALITY_REPLACEMENT line added; comments updated to match current semantics
- winnow/executor.py: bootstrap fix — recognize now called for all below-cap
  uploads when ENABLE_FRIGATE_SCORES=true (was gated on CEILING > 0)
- winnow/upload_tracker.py: frigate_scores schema comment corrected to
  pre-upload; get_most_redundant_mapped_file() added
- winnow/frigate_api.py: recognize_face returns (face_name, score)|None tuple
  so wrong-person scores never drive replacement or ceiling decisions
- winnow/config.py: FRIGATE_SCORE_THRESHOLD renamed to FRIGATE_SCORE_CEILING;
  ENABLE_FRIGATE_SCORES added
- tests/test_upload_tracker.py: 4 new tests for get_most_redundant_mapped_file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 18:35:11 +00:00
flanandClaude Sonnet 4.6 ed045f07dd fix: label InsightFace skip as "detection confidence" to distinguish from Frigate score
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 17:22:58 +00:00
flanandClaude Sonnet 4.6 110a45f467 perf: batch GET /api/faces; skip download on low confidence; batch gate tracker writes
Fetch all Frigate training files once before the upload loop instead of once
per person — for N people this reduces GET /api/faces calls from N to 1.
Falls back to per-person calls if the pre-fetch fails.

Check InsightFace detection confidence immediately after face enrichment,
before fetching the full-resolution image. Assets that fail MIN_CONFIDENCE
are skipped without downloading, saving potentially large image downloads.

Collapse the gate removal tracker writes from 3×N file ops into 2 total
via remove_and_reclassify_batch: one write to the uploaded tracker (remove
file mappings + remove from flat set) and one write to the rejected tracker.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 17:22:08 +00:00
flanandClaude Sonnet 4.6 785c9d4a22 feat: quality gate on by default; fix replacement/gate conflict; gate-failed → rejected
Dynamic floor now always active once Frigate scores exist — new images must
score at least as well as the weakest image already in the set, with no
config required. FRIGATE_SCORE_THRESHOLD adds an explicit absolute floor on
top. Gate active state is surfaced in normal output for both cases.

Quality replacement now pre-checks the gate threshold before deleting the
worst image. If the candidate would fail the gate, replacement is skipped
entirely rather than creating a net slot loss.

Gate-failed assets are reclassified as rejected (moved from uploaded_asset_ids
to rejected_asset_ids) so they are excluded from future runs without wasting
API calls on re-upload. RESET_PERSON still clears rejected records for a true
full reset. RETRY_REJECTED can recover them if the threshold is later lowered.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 17:10:30 +00:00
flanandClaude Sonnet 4.6 e0a5d98df6 fix: surface dynamic gate floor in normal output
When the dynamic threshold (min stored Frigate score) raises the effective
gate floor above the configured FRIGATE_SCORE_THRESHOLD, print it as a dim
info line rather than only logging at DEBUG/VERBOSE level.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 17:06:19 +00:00
flanandClaude Sonnet 4.6 ef5934d1af fix: stale frigate mapping reconciliation, recognize opt-out, cold start notice
- At upload start, diff tracker vs live Frigate file list and remove any
  mappings for files no longer present; corrects effective_count so manually
  deleted files don't permanently consume quota slots
- Add ENABLE_FRIGATE_SCORES config (default true); when false, skips all
  recognize_face calls and falls back to blur scores for quality replacement
- Print a dim notice when FRIGATE_SCORE_THRESHOLD is set but pre_run_count
  is zero, so users know the gate is deferred to the next run

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 16:51:44 +00:00
flanandClaude Sonnet 4.6 903d7f1054 fix: quality gate disabled at threshold 0; batch deletions; summary shows net count
- FRIGATE_SCORE_THRESHOLD=0.0 now fully disables the quality gate including the
  dynamic floor; a positive value is required to activate either
- Post-reconcile gate deletions are batched into one API call per person instead
  of one call per file
- Per-person summary reports gate removals and net uploaded count when the gate
  fires; grand summary includes total removed across all people
- .env.example comment updated to match the corrected opt-in behaviour

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 16:41:31 +00:00
flanandClaude Sonnet 4.6 f322eba380 feat: dynamic Frigate score threshold from stored set minimum
At the start of each person's upload phase, compute the minimum stored
Frigate recognition score across all currently mapped files. Use
max(config_threshold, dynamic_min) as the effective gate threshold so
new uploads must score at least as well as the weakest image already
in the training set.

Prevents overtraining well-recognised people: if all 80 images score
≥0.85, the dynamic threshold becomes ~0.85 and new additions that
score below that are removed rather than diluting a good training set.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 16:32:47 +00:00
flanandClaude Sonnet 4.6 26b598db98 feat: post-upload quality gate via FRIGATE_SCORE_THRESHOLD
When FRIGATE_SCORE_THRESHOLD > 0, images that score below the threshold
after upload are deleted from Frigate and removed from the tracker.
Skipped when pre_run_count == 0 (cold start — no class mean to compare
against yet). Deletion happens after reconciliation so the Frigate
filename is known. Disabled by default (0.0).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 16:29:29 +00:00
flanandClaude Sonnet 4.6 03be6ce2cb feat: store Frigate recognition scores and use them for quality replacement
After each successful upload, call POST /api/faces/recognize to get
Frigate's own confidence score (0-1) for the uploaded crop. Store it
in the tracker as frigate_scores alongside the existing blur score.

When quality replacement activates and frigate_scores are present,
use them for the replacement comparison instead of blur scores — an
image Frigate recognizes poorly is a worse training image than one it
recognizes well, regardless of sharpness. Falls back to blur scores
on first run before any frigate_scores are populated.

Also surfaces frigate_score in TRACE_CROP_SIZE output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 16:21:58 +00:00
flanandClaude Sonnet 4.6 19416cc7b8 feat: record crop pixel dimensions and add TRACE_CROP_SIZE lookup
Store (width, height) of each face crop in the tracker at upload time
alongside the existing blur score. Expose TRACE_CROP_SIZE=<px> to look
up which Immich asset produced a crop with that pixel dimension, making
it straightforward to trace unexpected or low-quality images visible in
Frigate back to their source.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 15:20:54 +00:00
flanandClaude Sonnet 4.6 12789e83f2 Fix four quality-replacement bugs found by code re-audit
C1/C4: Cap blur-score computation at 1440 px before calling assess_quality
so scores are always on the same Laplacian scale as the embedding path
(which operates on Immich preview thumbnails). Also converts the image to
RGB before scoring and stores 0.0 on assess_quality failure so files
uploaded without a score remain eligible for future quality replacement
instead of occupying a slot permanently.

C2: Fall back to the tracker's mapped-filename set as the pre-upload
baseline when the Frigate GET /api/faces endpoint is unreachable at upload
start. Previously, uploads that succeeded during a partial API outage were
never mapped in frigate_files, leaving get_tracked_frigate_file_count
permanently under-counting those files and allowing Frigate to exceed
MAX_AUTO_IMAGES over time.

C3: Track min_quality_score_for_slot when a quality-replacement delete
succeeds but the subsequent upload fails. This ensures the freed slot can
only be filled by a candidate that beats the deleted file's score, not just
the next file in iteration order (which could be lower quality than what
was deleted).

Add get_tracked_frigate_filenames() to upload_tracker and expand tracker
tests to cover the new function and exclude-parameter behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 06:52:52 +00:00
flanandClaude Sonnet 4.6 650dadd102 Cap quality replacement against tracked files only, not total Frigate count
Previously, effective_count and the jobs.py cap check used the total
Frigate file count (including manually-added files), so any file a user
curated by hand ate into winnow's managed quota. Now:

- get_tracked_frigate_file_count() returns len(frigate_files) from the
  tracker — only files winnow uploaded and reconciled
- effective_count in the upload loop uses this tracker count so
  manually-added files are invisible to the cap
- jobs.py capacity check uses len(frigate_files) instead of the live
  Frigate API count or cached frigate_count
- Frigate API call for known_frigate_files_at_start is now only used
  for the post-upload reconciliation diff, not for cap enforcement

Side-effect: fixes audit bug #1 — an unreachable Frigate GET no longer
zeroes effective_count and bypasses the cap, because the cap is now
read from the always-available local tracker.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 05:35:26 +00:00
flanandClaude Sonnet 4.6 12bccaa631 Fix TypeError when worst mapped file is None in quality replacement
When no mapped files exist for a person, worst is None and the old
f-string tried to subscript it before the conditional was evaluated.
Extracted worst_score_str as a local variable to avoid the crash.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 05:14:15 +00:00
flanandClaude Sonnet 4.6 11e7d4aad7 Implement quality score tracking and batch Frigate file mapping
- Track laplacian blur score through quality filtering pipeline
  (quality.py: blur_score on QualityResult; diversity.py: store on asset;
   executor.py: read via quality_score key)
- Replace per-file polling with post-person batch reconciliation:
  after all uploads for a person complete, poll Frigate (up to 15s)
  until the expected number of new files appear, then map by filename
  timestamp order (Frigate FIFO queue = upload order = timestamp order)
- Document race condition limitation: concurrent external uploads cause
  the batch to be skipped entirely (safe but files go unmapped); noted
  in code as requiring a Frigate API fix (return filename on upload)
- Add two assess_quality integration tests for blur_score

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 04:59:12 +00:00
flanandClaude Sonnet 4.6 615c3c3cc6 Fix ruff import ordering in executor.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 03:53:28 +00:00
flanandClaude Sonnet 4.6 a0083cb9fe Fix three bugs found by code audit
- Delete failure retry loop: when delete_frigate_person_files() fails,
  remove the file from the tracker so the next candidate targets a
  different worst file rather than re-attempting the same failed delete.

- Interactive mode quality replacement: _configure_person() never set
  config["quality_replacement"], causing the executor to always default
  to False and silently skip all uploads for at-cap interactive jobs.
  Now mirrors auto_configure by reading Config.QUALITY_REPLACEMENT.

- Silent mapping loss on API flap: after a successful upload, if the
  post-upload GET /api/faces returns None (transient API failure),
  the file was silently left unmapped. Now logs a warning so users
  know quality replacement won't target that file.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 03:52:04 +00:00
flanandClaude Sonnet 4.6 129c74720e Add quality replacement for Frigate face training images
When a person is at MAX_AUTO_IMAGES, winnow now replaces the
lowest-quality mapped training image in Frigate if a higher-confidence
candidate is available, keeping the training set always optimised.

Only files winnow uploaded (tracked via frigate_files mapping) are ever
replaced — manually added Frigate training images are never touched.
A concurrent-upload race condition is detected per-file: if N>1 new
files appear after one upload, the mapping is skipped rather than
guessed, logging at INFO level. The per-file snapshot approach is
retained over a batch approach because wrong mappings (which a batch
approach risks on race) are worse than no mapping.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 03:44:36 +00:00
flanandClaude Sonnet 4.6 f50d3fe1ab refactor: rename logging.py, in-process scheduler, trim .gitignore
- winnow/logging.py → winnow/log_config.py: avoids shadowing the stdlib
  logging module; log file renamed from immich_export.log to winnow.log
- scheduler.py: run main() in-process instead of subprocess.run so
  InsightFace and SigLIP models stay resident in memory across scheduled
  runs (hundreds of MB load, previously reloaded every run)
- .gitignore: replaced 200-line boilerplate with ~30 project-relevant
  patterns; removed Django/Flask/Redis/RabbitMQ/Scrapy/etc. noise
- .python-version: untracked (redundant with requires-python in
  pyproject.toml; kept in .gitignore for local pyenv users)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 05:30:37 +00:00
flanandClaude Sonnet 4.6 759579fc30 feat: confidence scores, Frigate-authoritative capacity cap
- Store Immich face confidence scores per asset in frigate_uploaded_ids.json
- Add frigate_api.py: query GET /api/faces to count trained images per person
- Record Frigate training count as frigate_count in tracker for offline fallback
- MAX_AUTO_IMAGES cap now uses live Frigate count → cached frigate_count → local uploaded count
- Startup summary shows last known Frigate training count per person
- Migrate by_person entries from flat list to {asset_ids, scores, frigate_count} dict

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 03:33:40 +00:00
flanandClaude Sonnet 4.6 eea7baa19f refactor: rename project to winnow
- Rename Python package directory if_curator/ → winnow/
- Update all imports, entry points, and CLI references
- Update pyproject.toml: name, scripts, package list, repository URL
- Update Dockerfile, compose.yml, entrypoint.sh, scheduler.py
- Update GitHub Actions workflow image names
- Update README

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 01:07:46 +00:00