- upload_tracker: replace executescript() in _migrate_schema_v2 with
individual execute() calls inside a transaction so a crash between DROP
and RENAME rolls back instead of permanently destroying tracked_assets
- frigate_api: _get_frigate_url now strips leading/trailing whitespace
before rstrip('/') so whitespace-only FRIGATE_URL is treated as unset
- executor: upload_to_frigate now uses _get_frigate_url() eliminating
double-slash upload paths when FRIGATE_URL has a trailing slash
- executor: corrupt thumbnail (resp.ok=True, Image.open fails) now calls
mark_rejected() so permanently broken assets are not retried forever
- upload_tracker: reset_person now uses _get_frigate_url() instead of
inline os.environ.get('FRIGATE_URL', '').strip()
- image_processing: _save_jpeg writes to a .tmp file and calls
os.replace() so a disk-full error never leaves a truncated JPEG
- cli: _handle_duplicate_people falls back to local deduplication when
all Immich merges fail, preventing two jobs from overwriting the same
Frigate folder
- config: _getenv_optional_float now delegates to _getenv_num() like
_getenv_optional_int, eliminating the inconsistent duplicate
- reconcile: _ts() uses rsplit('.', 1)[0] instead of .replace('.webp','')
so FIFO mapping works with any Frigate training-file extension
Correctness:
- fetch_face_data: only fall back to faces[0] when person_id is absent;
previously a missing person match injected a different person's bbox
- upload_tracker: change PK from (asset_id, status) to
(asset_id, person_name, status); old PK allowed INSERT OR REPLACE to
silently overwrite person_name when the same photo appeared in two
people's jobs, breaking quality-replacement JOINs; auto-migrates DBs
- filter_recent_assets: treat years=0 as "no age filter" instead of
falling through to Config.YEARS_FILTER via falsy `or`
- _is_module_available: return find_spec(...) is not None; find_spec
returns None (not raises) for absent top-level modules, so the
previous code always returned True
- execute_jobs error handler: use asset.get("id", "<unknown>") to avoid
a secondary KeyError propagating out of execute_jobs on malformed dicts
- upload_to_frigate: also mark_rejected on HTTP 422, not only HTTP 400
with "face" in body; other permanent errors left assets untracked and
retried forever
- reconcile_frigate_mappings: sort key lambda f: (_ts(f), f) makes order
deterministic when timestamps are equal or 0.0; set iteration order is
hash-randomised, stable sort preserves it
Reuse / cleanup:
- config.py: add _getenv_optional_int delegating to _getenv_num(name, None, int)
- jobs.py: _resolve_strategy uses _getenv_optional_int("LIMIT") instead
of inline os.environ.get + int() + warning duplicate of _getenv_num
- frigate_api.py: add _get_frigate_url() helper; eliminates 4× copy of
os.environ.get("FRIGATE_URL", "").rstrip("/")
- quality.py: extract blur_score_from_image(img, max_dim=1440) helper;
executor.py time-spread blur fallback now uses it instead of inlining
the resize+RGB+assess_quality sequence, keeping scale logic in one place
- _getenv_num: add raw.strip() + empty-string guard so numeric vars set
to "" (common Compose pattern for "use default") return the default
silently instead of warning "not a valid int/float"
- _getenv_bool: same guard so True-defaulted flags set to "" return
the configured default instead of silently returning False
- embeddings.py: replace inline FORCE_CPU bool parse with _getenv_bool
- cache.py: fix np.save extension bug from v0.5.13 — tmp path used
final+".tmp" (abc.npy.tmp) but np.save auto-appends .npy to paths not
ending in .npy, writing to abc.npy.tmp.npy instead; os.replace then
raised FileNotFoundError silently, making every cache write a no-op
and leaking *.npy.tmp.npy files. Fixed by inserting .tmp before .npy:
tmp = final[:-4] + ".tmp.npy"
- config.py: remove str(default) round-trip in _getenv_int/_getenv_float
— use raw = os.getenv(name); return default if raw is None else int(raw)
so a future float default can't cause a spurious "not a valid integer"
warning and return the wrong type
- executor.py: consolidate 4 progress.remove_task calls into one
try/finally around the per-job body; continue inside try/finally
executes the finally before the next iteration, making the invariant
structurally enforced rather than relying on discipline across 4 sites
YEARS_FILTER, MIN_FACE_WIDTH, MIN_FACE_COUNT, MAX_AUTO_IMAGES,
BLUR_THRESHOLD, MIN_CONFIDENCE, and FACE_MARGIN used bare int()/float()
calls with no error handler. A typo (trailing space, non-numeric value)
raised ValueError inside __getattr__, producing a cryptic traceback on
the first config access rather than at the validate() step. Values are
now parsed by _getenv_int/_getenv_float helpers that warn and fall back
to the documented default, matching the existing FRIGATE_SCORE_CEILING
pattern.
- executor.py: fix _safe_person_dir docstring — realpath+startswith is the
load-bearing traversal guard; islink is a supplementary early-exit for the
symlink sub-case only. The previous comment "checking after realpath would be
too late" implied islink was the primary guard, which is backwards.
- immich_api.py: move total_raw accumulation to after the dead-end-page break
so all-garbage pages don't inflate the count and produce misleading
"N total, 0 recent" output. Mixed pages (some valid, some non-dict) still
count page_count so transient schema issues don't shrink MIN_FACE_COUNT below
threshold. Add warning when a RequestException interrupts pagination mid-way
so operators know total_raw is a lower bound.
- config.py: eliminate residual TOCTOU — change `if config_file.exists():` to
`if _data_cfg_exists or config_file.exists():` so _data_cfg is never
stat'd twice (the v0.5.7 fix cached the first check but not the second).
executor.py:
- Move islink check into _safe_person_dir on the raw path, before realpath
resolves it; the previous check at the rmtree site was unreachable dead code
because realpath already followed any symlink
immich_api.py / jobs.py:
- fetch_all_assets now returns (assets, total_raw) where total_raw is the
item count seen before non-dict filtering; callers use it for MIN_FACE_COUNT
guard and display so transient non-dict API items can't incorrectly skip people
- Add WARNING when pagination stops because a page had items but all were non-dict
config.py:
- Cache _data_cfg.exists() in _data_cfg_exists so the dual-config warning
and config_file selection always read from the same stat() result; previously
two calls created a TOCTOU window where log and code could disagree
Bump version to 0.5.7
* fix: audit hardening — input validation, error handling, and robustness
- immich_api: guard person["id"] with .get() + early return on missing field
- immich_api: include page number in pagination exception log
- immich_api: validate faces response is a list before indexing
- executor: wrap Image.open() in try/except for non-image HTTP responses
- executor: strip leading 'v' from Frigate version before parsing (v0.16.0 was misread)
- config: wrap FRIGATE_SCORE_CEILING float() parse in try/except with warning
- config: warn when both DATA_DIR and legacy CWD config files exist simultaneously
- scheduler: wrap PID file write in try/except so /tmp failures don't crash startup
- scheduler: clamp sleep to 60s max to bound recovery time after NTP clock jumps
- frigate_api: log unexpected non-list type in get_frigate_person_files at DEBUG
* fix: LIMIT env var crash and symlink guard on person output dir
- jobs: wrap int(LIMIT) parse in try/except — bad value (e.g. "30.5", "all")
now logs a warning and falls back to the default instead of crashing
- executor: check for symlink before shutil.rmtree on person_dir — prevents
following a symlink out of OUTPUT_DIR on a shared volume
* chore: bump version to 0.5.3
* refactor: rename CACHE_DIR to DATA_DIR, default path .if_cache → data
CACHE_DIR held both the embedding cache and the SQLite tracker DB, making
the name misleading. DATA_DIR is more accurate.
- Config reads DATA_DIR first; falls back to CACHE_DIR with a deprecation
warning so existing setups don't break on upgrade
- Default local path: data (was .if_cache)
- Docker default path: /app/data (was /app/.if_cache)
- Internal references (embeddings.py, upload_tracker.py) updated to DATA_DIR
- compose.yml, .env.example, README, wiki, and changelog updated
- Version bumped to 0.5.1
* chore: update lockfile
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
- upload_tracker: partial migration now rolls back atomically on failure;
JSON renamed only after successful commit so failed runs retry cleanly
- upload_tracker: allowlist score_col in _pick_mapped_file to close
latent SQL injection surface
- config: move load_dotenv() from module import into _load() so no I/O
at import time and reset() fully resets env loading
- config: use is None checks for IMMICH_URL/OUTPUT_DIR config-file
fallback so explicitly empty env vars are not overridden by the file
- executor: skip reconcile when Frigate API is unreachable at upload
start — tracker baseline is incomplete and would mis-trigger the
external-upload guard, permanently losing file mappings
- reconcile: change polling break condition from >= to == target so
transient overshoots don't prematurely exit the loop and trigger
the external-upload guard
- jobs: apply capacity cap as the selection limit rather than truncating
post-selection by position, so the diversity algorithm works within
the right budget from the start
- Dockerfile: explicit gpu branch + exit 1 on unknown VARIANT instead
of silent fallback
- Config: remove _ConfigAccessor and ConfigManager; use __getattr__ for lazy
loading on single _Config class; re-register self as _instance in __getattr__
so reset() always clears the correct object (item 1)
- upload_tracker: replace hand-rolled JSON store with sqlite3; auto-migrates
existing JSON on first run; remove dead record_frigate_file function;
connection re-opens when CACHE_DIR changes for test isolation (items 2, 8)
- diversity: move ThreadPoolExecutor import to module level; inject optional
fetch_fn parameter for testability (items 3, 6)
- pyproject: consolidate 4 variant files into extras (gpu/rocm/intel/cpu);
update Dockerfile to use --extra flag; delete variant pyproject/lock files;
uv.lock needs regen with `uv lock` after this change (item 4)
- jobs: extract _build_job helper to separate business logic from terminal I/O;
auto_configure delegates dedup/selection to _build_job (item 5)
- logging: convert f-string log calls to % interpolation throughout all winnow/
modules (item 7)
- reconcile: new module with reconcile_frigate_mappings and
enrich_asset_with_face_data extracted from executor.py (item 9)
- scheduler: print next scheduled run time after startup and after each run;
fix f-string logger.error call (item 10)
FRIGATE_SCORE_CEILING now defaults to dynamic mode (unset): below-cap
candidates are skipped if their pre-upload Frigate score exceeds the
most-redundant tracked file's score. This catches conditions already
covered by manually-added Frigate images that winnow cannot track —
the embedding-based diversity selection has no visibility into those.
Set FRIGATE_SCORE_CEILING=0 to disable; a positive value (e.g. 0.85)
still acts as a fixed hard ceiling. First-run safety is unchanged
(pre_run_count==0 prevents recognize_face from being called).
The two quality replacement branches (Frigate-score and blur-score)
shared identical structure and are merged into a single code path
parameterised by score source and comparison direction.
Also raises MIN_FACE_COUNT default from 0 to 3 and updates the
config test to match.
When Immich has multiple person records sharing a name (e.g. unmerged
face clusters), winnow would previously run separate jobs for each,
with the second job wiping the first job's output directory — resulting
in far fewer training images than expected.
New behaviour:
- At startup, duplicate names are detected and a warning is printed
showing asset counts for each duplicate.
- By default (MERGE_DUPLICATE_PEOPLE=false), only the person with the
most assets is processed; smaller duplicates are skipped cleanly.
- With MERGE_DUPLICATE_PEOPLE=true, the duplicates are permanently
merged inside Immich via PUT /api/people/{id}/merge (keeps the
largest), then the people list is re-fetched before jobs run.
Also adds an explicit comment in executor.py confirming that replacement
targets come exclusively from tracker-mapped files, so manually-added
Frigate training images are never selected for deletion.
Finalizes the 0.4.0 release:
- Version bumped to 0.4.0 in pyproject.toml
- CHANGELOG.md: add [0.4.0] section covering Frigate pre-upload scoring,
quality replacement inversion, bootstrap fix, FRIGATE_SCORE_CEILING,
ENABLE_FRIGATE_SCORES, removal of post-upload quality gate, and all
doc/default corrections
- README.md: step 8 updated for dual-mode replacement, FRIGATE_SCORE_CEILING
and ENABLE_FRIGATE_SCORES added to env var table, MIN_FACE_WIDTH and
BLUR_THRESHOLD defaults corrected (50→90, 100→120)
- .env.example: FRIGATE_SCORE_THRESHOLD replaced with FRIGATE_SCORE_CEILING;
QUALITY_REPLACEMENT line added; comments updated to match current semantics
- winnow/executor.py: bootstrap fix — recognize now called for all below-cap
uploads when ENABLE_FRIGATE_SCORES=true (was gated on CEILING > 0)
- winnow/upload_tracker.py: frigate_scores schema comment corrected to
pre-upload; get_most_redundant_mapped_file() added
- winnow/frigate_api.py: recognize_face returns (face_name, score)|None tuple
so wrong-person scores never drive replacement or ceiling decisions
- winnow/config.py: FRIGATE_SCORE_THRESHOLD renamed to FRIGATE_SCORE_CEILING;
ENABLE_FRIGATE_SCORES added
- tests/test_upload_tracker.py: 4 new tests for get_most_redundant_mapped_file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- At upload start, diff tracker vs live Frigate file list and remove any
mappings for files no longer present; corrects effective_count so manually
deleted files don't permanently consume quota slots
- Add ENABLE_FRIGATE_SCORES config (default true); when false, skips all
recognize_face calls and falls back to blur scores for quality replacement
- Print a dim notice when FRIGATE_SCORE_THRESHOLD is set but pre_run_count
is zero, so users know the gate is deferred to the next run
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When FRIGATE_SCORE_THRESHOLD > 0, images that score below the threshold
after upload are deleted from Frigate and removed from the tracker.
Skipped when pre_run_count == 0 (cold start — no class mean to compare
against yet). Deletion happens after reconciliation so the Frigate
filename is known. Disabled by default (0.0).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frigate classifies images with Laplacian variance < 120 as "very blurry"
and its own docs recommend avoiding blurry training data. Winnow was
accepting images in the 100-120 range that Frigate considers too blurry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
50px crops produce ~2,500–4,225 total pixels — well below Frigate's own
camera capture range of 16k–50k px. 90px guarantees ≥8,100 total pixels
even when face margins are fully clipped by image edges.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a person is at MAX_AUTO_IMAGES, winnow now replaces the
lowest-quality mapped training image in Frigate if a higher-confidence
candidate is available, keeping the training set always optimised.
Only files winnow uploaded (tracked via frigate_files mapping) are ever
replaced — manually added Frigate training images are never touched.
A concurrent-upload race condition is detected per-file: if N>1 new
files appear after one upload, the mapping is skipped rather than
guessed, logging at INFO level. The per-file snapshot approach is
retained over a batch approach because wrong mappings (which a batch
approach risks on race) are worse than no mapping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: ENV INSIGHTFACE_HOME=/models → /models/.insightface to
match compose.yml and .env.example; the old value caused InsightFace
to store models at /models/models/buffalo_l (double-appended subdir)
- entrypoint.sh: use /app/.venv/bin/winnow (installed entry point)
instead of python -m winnow.cli
- config.py: ENABLE_CACHE default false → true; embedding cache is
always beneficial in practice; users can opt out with ENABLE_CACHE=false
- compose.yml: comment out CRON_SCHEDULE so scheduling is opt-in;
flip ENABLE_CACHE to commented opt-out to reflect new default
- README.md: update ENABLE_CACHE default documentation to true
- tests/test_config.py: update default assertion to match
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- embeddings: initialize ctx_id=-1 before try block so the except
handler cannot NameError; move insightface_home out of try for the
same reason
- embeddings: replace contextlib.redirect_stdout/stderr (Python-level
only) with fd-level dup2 suppression — actually silences C extension
noise from InsightFace during model loading
- jobs: fix frigate_count==0 falling through `or` chain; use explicit
`is not None` check so a real zero is not treated as missing data
- diversity: thread person_id through select_diverse_assets →
_select_by_embedding → _get_face_bbox / _get_face_confidence /
_crop_face_from_thumbnail so group-photo assets embed the target
person's face rather than whichever person is listed first
- config: replace type() hack for ConfigManager with a proper class
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flan
