* revert: replace SQLite tracker with JSON backend (v0.6.0)
The SQLite migration (v0.5.0) spawned 21 bug-fix releases in two days:
data-loss risk in the migration layer, schema PK conflicts on per-person
tracking, tracker isolation races under concurrent runs, and a disk-full
error that triggered duplicate Frigate uploads. The complexity cost
outweighs the benefit.
Restored the pre-SQL JSON tracker (frigate_uploaded_ids.json /
frigate_rejected_ids.json in DATA_DIR). Public API is identical — all
callers in executor.py, jobs.py, cli.py, and reconcile.py work unchanged.
Existing JSON files are read automatically; frigate_tracker.db can be
deleted once verified.
* fix: narrow corrupt-thumbnail exception to UnidentifiedImageError; restore IMMICH_URL empty-string fallback
* docs: rewrite v0.6.0 changelog, strip v0.5.x entries, fix README SQLite references
* chore: remove dead get_frigate_filename_for_asset (orphaned since FRIGATE_SCORE_THRESHOLD removal in v0.4.0)
* fix: sort imports in executor.py (ruff I001)
* refactor: rename CACHE_DIR to DATA_DIR, default path .if_cache → data
CACHE_DIR held both the embedding cache and the SQLite tracker DB, making
the name misleading. DATA_DIR is more accurate.
- Config reads DATA_DIR first; falls back to CACHE_DIR with a deprecation
warning so existing setups don't break on upgrade
- Default local path: data (was .if_cache)
- Docker default path: /app/data (was /app/.if_cache)
- Internal references (embeddings.py, upload_tracker.py) updated to DATA_DIR
- compose.yml, .env.example, README, wiki, and changelog updated
- Version bumped to 0.5.1
* chore: update lockfile
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
- Config: remove _ConfigAccessor and ConfigManager; use __getattr__ for lazy
loading on single _Config class; re-register self as _instance in __getattr__
so reset() always clears the correct object (item 1)
- upload_tracker: replace hand-rolled JSON store with sqlite3; auto-migrates
existing JSON on first run; remove dead record_frigate_file function;
connection re-opens when CACHE_DIR changes for test isolation (items 2, 8)
- diversity: move ThreadPoolExecutor import to module level; inject optional
fetch_fn parameter for testability (items 3, 6)
- pyproject: consolidate 4 variant files into extras (gpu/rocm/intel/cpu);
update Dockerfile to use --extra flag; delete variant pyproject/lock files;
uv.lock needs regen with `uv lock` after this change (item 4)
- jobs: extract _build_job helper to separate business logic from terminal I/O;
auto_configure delegates dedup/selection to _build_job (item 5)
- logging: convert f-string log calls to % interpolation throughout all winnow/
modules (item 7)
- reconcile: new module with reconcile_frigate_mappings and
enrich_asset_with_face_data extracted from executor.py (item 9)
- scheduler: print next scheduled run time after startup and after each run;
fix f-string logger.error call (item 10)
Tests cover the core ML pipeline algorithms in diversity.py — previously
untested. No network or model dependencies; all pure-function or
numpy-only paths:
- Face bbox and confidence extraction from Immich metadata, including
person_id filtering and missing-data edge cases
- Face crop scaling: verifies bbox coordinates are correctly scaled when
the thumbnail dimensions differ from the metadata image dimensions
- Near-duplicate dedup: removal below cosine threshold, quality-score
preference between duplicates, zero-quality-score treated as zero not
missing (falsy bug guard)
- K-Medoids: correct medoid count, distinctness, valid index range, and
full-N edge case
- Adaptive threshold: positive output, floor at 0.05 for identical
embeddings, single-point, scales with embedding spread
- Time-spread fallback: exact count, all-under-limit passthrough,
auto→30 default, first/last inclusion
- Cluster-aware selection: exact limit, subset invariant, auto-stop on
tight cluster, hard-example confidence weighting accepted
FRIGATE_SCORE_CEILING now defaults to dynamic mode (unset): below-cap
candidates are skipped if their pre-upload Frigate score exceeds the
most-redundant tracked file's score. This catches conditions already
covered by manually-added Frigate images that winnow cannot track —
the embedding-based diversity selection has no visibility into those.
Set FRIGATE_SCORE_CEILING=0 to disable; a positive value (e.g. 0.85)
still acts as a fixed hard ceiling. First-run safety is unchanged
(pre_run_count==0 prevents recognize_face from being called).
The two quality replacement branches (Frigate-score and blur-score)
shared identical structure and are merged into a single code path
parameterised by score source and comparison direction.
Also raises MIN_FACE_COUNT default from 0 to 3 and updates the
config test to match.
Finalizes the 0.4.0 release:
- Version bumped to 0.4.0 in pyproject.toml
- CHANGELOG.md: add [0.4.0] section covering Frigate pre-upload scoring,
quality replacement inversion, bootstrap fix, FRIGATE_SCORE_CEILING,
ENABLE_FRIGATE_SCORES, removal of post-upload quality gate, and all
doc/default corrections
- README.md: step 8 updated for dual-mode replacement, FRIGATE_SCORE_CEILING
and ENABLE_FRIGATE_SCORES added to env var table, MIN_FACE_WIDTH and
BLUR_THRESHOLD defaults corrected (50→90, 100→120)
- .env.example: FRIGATE_SCORE_THRESHOLD replaced with FRIGATE_SCORE_CEILING;
QUALITY_REPLACEMENT line added; comments updated to match current semantics
- winnow/executor.py: bootstrap fix — recognize now called for all below-cap
uploads when ENABLE_FRIGATE_SCORES=true (was gated on CEILING > 0)
- winnow/upload_tracker.py: frigate_scores schema comment corrected to
pre-upload; get_most_redundant_mapped_file() added
- winnow/frigate_api.py: recognize_face returns (face_name, score)|None tuple
so wrong-person scores never drive replacement or ceiling decisions
- winnow/config.py: FRIGATE_SCORE_THRESHOLD renamed to FRIGATE_SCORE_CEILING;
ENABLE_FRIGATE_SCORES added
- tests/test_upload_tracker.py: 4 new tests for get_most_redundant_mapped_file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frigate classifies images with Laplacian variance < 120 as "very blurry"
and its own docs recommend avoiding blurry training data. Winnow was
accepting images in the 100-120 range that Frigate considers too blurry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
50px crops produce ~2,500–4,225 total pixels — well below Frigate's own
camera capture range of 16k–50k px. 90px guarantees ≥8,100 total pixels
even when face margins are fully clipped by image edges.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
C1/C4: Cap blur-score computation at 1440 px before calling assess_quality
so scores are always on the same Laplacian scale as the embedding path
(which operates on Immich preview thumbnails). Also converts the image to
RGB before scoring and stores 0.0 on assess_quality failure so files
uploaded without a score remain eligible for future quality replacement
instead of occupying a slot permanently.
C2: Fall back to the tracker's mapped-filename set as the pre-upload
baseline when the Frigate GET /api/faces endpoint is unreachable at upload
start. Previously, uploads that succeeded during a partial API outage were
never mapped in frigate_files, leaving get_tracked_frigate_file_count
permanently under-counting those files and allowing Frigate to exceed
MAX_AUTO_IMAGES over time.
C3: Track min_quality_score_for_slot when a quality-replacement delete
succeeds but the subsequent upload fails. This ensures the freed slot can
only be filled by a candidate that beats the deleted file's score, not just
the next file in iteration order (which could be lower quality than what
was deleted).
Add get_tracked_frigate_filenames() to upload_tracker and expand tracker
tests to cover the new function and exclude-parameter behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, effective_count and the jobs.py cap check used the total
Frigate file count (including manually-added files), so any file a user
curated by hand ate into winnow's managed quota. Now:
- get_tracked_frigate_file_count() returns len(frigate_files) from the
tracker — only files winnow uploaded and reconciled
- effective_count in the upload loop uses this tracker count so
manually-added files are invisible to the cap
- jobs.py capacity check uses len(frigate_files) instead of the live
Frigate API count or cached frigate_count
- Frigate API call for known_frigate_files_at_start is now only used
for the post-upload reconciliation diff, not for cap enforcement
Side-effect: fixes audit bug #1 — an unreachable Frigate GET no longer
zeroes effective_count and bypasses the cap, because the cap is now
read from the always-available local tracker.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Config tests now verify QUALITY_REPLACEMENT defaults to True and
respects the QUALITY_REPLACEMENT=false env override.
CI: replace minimal disk cleanup with more aggressive removal
(Android SDK ~14GB, Swift, CodeQL, docker system prune) so the
NVIDIA GPU image build no longer exhausts runner disk space.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Track laplacian blur score through quality filtering pipeline
(quality.py: blur_score on QualityResult; diversity.py: store on asset;
executor.py: read via quality_score key)
- Replace per-file polling with post-person batch reconciliation:
after all uploads for a person complete, poll Frigate (up to 15s)
until the expected number of new files appear, then map by filename
timestamp order (Frigate FIFO queue = upload order = timestamp order)
- Document race condition limitation: concurrent external uploads cause
the batch to be skipped entirely (safe but files go unmapped); noted
in code as requiring a Frigate API fix (return filename on upload)
- Add two assess_quality integration tests for blur_score
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a person is at MAX_AUTO_IMAGES, winnow now replaces the
lowest-quality mapped training image in Frigate if a higher-confidence
candidate is available, keeping the training set always optimised.
Only files winnow uploaded (tracked via frigate_files mapping) are ever
replaced — manually added Frigate training images are never touched.
A concurrent-upload race condition is detected per-file: if N>1 new
files appear after one upload, the mapping is skipped rather than
guessed, logging at INFO level. The per-file snapshot approach is
retained over a batch approach because wrong mappings (which a batch
approach risks on race) are worse than no mapping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: ENV INSIGHTFACE_HOME=/models → /models/.insightface to
match compose.yml and .env.example; the old value caused InsightFace
to store models at /models/models/buffalo_l (double-appended subdir)
- entrypoint.sh: use /app/.venv/bin/winnow (installed entry point)
instead of python -m winnow.cli
- config.py: ENABLE_CACHE default false → true; embedding cache is
always beneficial in practice; users can opt out with ENABLE_CACHE=false
- compose.yml: comment out CRON_SCHEDULE so scheduling is opt-in;
flip ENABLE_CACHE to commented opt-out to reflect new default
- README.md: update ENABLE_CACHE default documentation to true
- tests/test_config.py: update default assertion to match
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- __init__.py: derive __version__ from importlib.metadata instead of
a hardcoded "0.1.0" that was six releases out of date
- scheduler.py: move winnow.cli import to module top (no more noqa);
clean up redundant bool variables in check_models
- tests/test_quality.py: 17 tests covering all five quality check
functions individually plus assess_quality integration cases
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flan
