release: v0.4.9

- How It Works step 9: "upload freely" → note novelty gate may skip below-cap candidates
- Persistence note: "Frigate rejections" → "rejected assets" (covers confidence skips too)
- RETRY_REJECTED description: explicitly covers all rejection types, not just Frigate
- Image Quality section: split into user-adjustable controls and calibrated image
  processing defaults with a support disclaimer to deter blind tuning

CHANGELOG.md

@@ -7,6 +7,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.9] - 2026-06-14
+
+### Changed
+
+- **`FRIGATE_SCORE_CEILING` is now dynamic by default**: previously defaulted to `0` (disabled). Now unset (default) enables a self-calibrating novelty gate — below-cap candidates are skipped if their pre-upload Frigate score exceeds the most-redundant tracked file's score. This catches conditions already covered by manually-added Frigate images that winnow cannot track. Set `FRIGATE_SCORE_CEILING=0` to disable entirely; set a positive value (e.g. `0.85`) for a fixed hard ceiling.
+- **Quality replacement branches consolidated**: the Frigate-score and blur-score replacement paths in the upload loop shared identical structure. Merged into a single code path parameterised by score source and comparison direction.
+- **`MIN_FACE_COUNT` default raised from `0` to `3`**: people with fewer than 3 tagged photos produce degenerate training sets; skipping them by default avoids noisy runs.
+- **`STRATEGY=adaptive`** is the new primary name for embedding-based diversity selection; `auto` remains a silent alias for backwards compatibility.
+- **`MERGE_DUPLICATE_PEOPLE` and `TRACE_CROP_SIZE`** added to the README env var table (were in the codebase but undocumented).
+- **CUDA version corrected** in the image tags table (was 13.3, actual base image is 12.8.1).
+
+## [0.4.8] - 2026-06-14
+
+### Changed
+
+- **Tracker write-through cache**: `upload_tracker` now keeps an in-memory copy of each JSON file keyed by its resolved path. All reads after the first hit the cache instead of disk; writes go to both disk and cache atomically. Cuts per-person disk I/O in the upload loop from ~90 reads to ~1, with no API or behaviour changes.
+
+## [0.4.7] - 2026-06-14
+
+### Changed
+
+- **`_dedup_embeddings` pre-allocated buffer**: replaced the grow-on-keep `np.vstack` pattern with a pre-allocated `(Q, D)` buffer filled row-by-row. Eliminates O(K²) copy work and the GC pressure from K intermediate heap allocations while keeping identical arithmetic for the similarity checks.
+- **`_kmedoids` cost computation vectorized**: the Python-level `sum(dist_matrix[i, medoids[labels[i]]] for i in range(n))` generator (called once per swap evaluation) is replaced with `dist_matrix[np.arange(n), np.array(medoids)[labels]].sum()` — a single numpy fancy-index + reduction, ~20–50× faster in the swap loop.
+- **`_reconcile_frigate_mappings` single-write batch**: previously called `record_frigate_file` once per uploaded file, each doing a full JSON load + save (O(L) disk round-trips per person). Now builds the full `{frigate_filename: asset_id}` mapping dict and writes it in one `record_frigate_files_batch` call (O(1) disk round-trip).
+
+## [0.4.6] - 2026-06-14
+
+### Fixed
+
+- **OOM when Immich returns many pages per person**: `fetch_all_assets` now stops fetching once 5000 assets have been collected — the diversity selection pool is already capped at 3000 items, so fetching up to 1,000,000 was wasteful and could exhaust memory on large libraries. 5000 provides ample headroom for the pool cap while bounding per-person memory to ~2 MB.
+- **Non-dict items in Immich asset pages silently skipped**: a malformed or partially-null Immich response page could include `null` or non-object items in the assets array. These are now filtered at fetch time rather than causing `AttributeError` downstream.
+
 ## [0.4.5] - 2026-06-14
 
 ### Fixed

README.md

@@ -43,26 +43,34 @@ Immich library
    • Objects → SigLIP (Vision Transformer)         → 768-dim vector
       │
       ▼
-5. Diversity selection
+5. Near-duplicate removal — greedy cosine-distance pass drops burst shots
+   and near-identical photos before clustering runs; the highest-quality
+   image from each near-duplicate group is kept
+      │
+      ▼
+6. Diversity selection
    • K-Medoids clustering → one representative per natural group
    • Farthest Point Sampling → fill remaining slots with maximally spread picks
-   • Hard example weighting — unusual angles and low-confidence detections
+   • Hard example weighting — low-confidence detections get a distance boost
-     are biased toward selection, since those are where models tend to fail
+     so unusual angles and harder looks are preferred over easy frontals
-   • Auto mode: stops when similarity to the existing set exceeds a threshold
+   • Adaptive mode: stops when the next candidate is too similar to those already
-     (20 % of median pairwise distance for faces, 10 % for objects)
+     selected (distance threshold = 20 % of median pairwise distance for
+     faces, 10 % for objects)
       │
       ▼
-6. Download full-resolution originals from Immich
+7. Download full-resolution originals from Immich
       │
       ▼
-7. Crop and process
+8. Crop and process
    • Face mode: EXIF-corrected, landmark-aligned 112×112 crop (ArcFace format)
    • Object mode: YOLOv9c detection → one crop per matched instance
       │
       ▼
-8. Deliver
+9. Deliver
    • Face mode: upload crops to Frigate's face registration API
-     ↳ below MAX_AUTO_IMAGES — upload freely
+     ↳ below MAX_AUTO_IMAGES — upload, unless the novelty gate
+       (FRIGATE_SCORE_CEILING) determines the candidate is already
+       covered by the current training set
      ↳ at cap + QUALITY_REPLACEMENT=true — with Frigate scoring active,
        swap the most redundant tracked image (highest pre-upload recognize
        score) if the candidate is more novel (lower score); falling back to
@@ -72,7 +80,7 @@ Immich library
    • Object mode: save crops to disk → place into your Frigate data directory
 ```
 
-Uploaded and rejected asset IDs are persisted across runs. The same image is never processed twice; Frigate rejections are permanently skipped unless `RETRY_REJECTED=true`.
+Uploaded and rejected asset IDs are persisted across runs. The same image is never processed twice; rejected assets are permanently skipped unless `RETRY_REJECTED=true`.
 
 ---
 
@@ -90,7 +98,7 @@ Uploaded and rejected asset IDs are persisted across runs. The same image is nev
 
 | Tag | Arch | Acceleration |
 | :-- | :-- | :-- |
-| `:latest` | amd64 + arm64 | NVIDIA CUDA 13.3 (amd64) · requires [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) |
+| `:latest` | amd64 + arm64 | NVIDIA CUDA 12.8 (amd64) · requires [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) |
 | `:rocm` | amd64 | AMD ROCm · pass `/dev/kfd` + `/dev/dri` |
 | `:intel` | amd64 | Intel Arc / iGPU via OpenVINO · pass `/dev/dri`, set `OPENVINO_DEVICE=GPU` |
 | `:cpu` | amd64 + arm64 | CPU only · ~2 GB smaller · no GPU required |
@@ -173,10 +181,10 @@ In scheduled mode the process (and loaded models) stays resident between runs. T
 | Variable | Default | Description |
 | :--- | :--- | :--- |
 | `TRAINING_MODE` | `face` | `face` — upload crops to Frigate; `object` — save crops to disk |
-| `STRATEGY` | `auto` | `auto` (embedding-based adaptive), `standard` (30 images), `broad` (100 images) |
+| `STRATEGY` | `adaptive` | `adaptive` — embedding-based diversity selection, stops when candidates become redundant; `standard` — fixed 30 images; `broad` — fixed 100 images |
 | `LIMIT` | *(unset)* | Exact image count — overrides `STRATEGY` |
 | `OBJECT_CLASS` | `dog` | Target class for object mode (any YOLO class: `dog`, `cat`, `car`, etc.) |
-| `AUTO_MODE` | *(auto)* | Force non-interactive mode in a terminal; auto-detected otherwise |
+| `AUTO_MODE` | *(auto)* | Skip interactive prompts and process all people unattended — auto-detected when no TTY is present (Docker, cron); set `true` to force in a terminal |
 | `VERBOSE` | `false` | Enable DEBUG-level console output (log file is always DEBUG) |
 
 ### People Filtering
@@ -185,23 +193,31 @@ In scheduled mode the process (and loaded models) stays resident between runs. T
 | :--- | :--- | :--- |
 | `ONLY_PEOPLE` | *(unset)* | Comma-separated whitelist — process only these people |
 | `SKIP_PEOPLE` | *(unset)* | Comma-separated list — skip these people |
-| `MIN_FACE_COUNT` | `0` | Skip people with fewer than N tagged assets in Immich |
+| `MIN_FACE_COUNT` | `3` | Skip people with fewer than N tagged assets in Immich |
+| `MERGE_DUPLICATE_PEOPLE` | `false` | When Immich has duplicate entries for the same person (same face split across multiple names), merge their asset pools before processing. Without this, each duplicate group emits a warning and is skipped |
 | `YEARS_FILTER` | `10` | Ignore images older than N years |
 
 ### Image Quality
 
 | Variable | Default | Description |
 | :--- | :--- | :--- |
+| `MAX_AUTO_IMAGES` | `80` | Maximum training images per person in Frigate |
+| `QUALITY_REPLACEMENT` | `true` | When at cap, swap a weaker tracked image for a better candidate. With Frigate scoring active, targets the most redundant image (highest pre-upload recognize score); otherwise uses blur score. Never touches manually added Frigate files. Set `false` to skip people at cap |
+
+#### Advanced Tuning *(calibrated — do not adjust)*
+
+These defaults are tuned for Frigate's ArcFace requirements. winnow will warn on launch if any are set. Image quality issues caused by non-default values will not be investigated.
+
+| Variable | Default | Description |
+| :--- | :--- | :--- |
+| `ENABLE_FRIGATE_SCORES` | `true` | Call Frigate's recognize endpoint pre-upload to store diversity scores used for quality replacement. Adds ~200 ms per upload. Disabling also disables the below-cap novelty gate |
+| `FRIGATE_SCORE_CEILING` | *(unset)* | Below-cap novelty gate. Unset: dynamic — skips candidates whose Frigate score exceeds the most-redundant tracked file's score, auto-calibrates each run. `0`: disable entirely. Positive value (e.g. `0.85`): fixed hard ceiling |
 | `MIN_FACE_WIDTH` | `90` | Minimum face crop width in pixels |
 | `FACE_MARGIN` | `0.15` | Padding around bounding box crop (fraction of face size) |
 | `ENABLE_FACE_ALIGNMENT` | `true` | Align to ArcFace 112×112 format using facial landmarks |
 | `USE_FULL_RESOLUTION` | `true` | Download full-resolution originals rather than preview thumbnails |
 | `MIN_CONFIDENCE` | `0.7` | Minimum Immich face detection confidence |
 | `BLUR_THRESHOLD` | `120.0` | Laplacian variance threshold — lower accepts more blur |
-| `MAX_AUTO_IMAGES` | `80` | Maximum training images per person in Frigate |
-| `QUALITY_REPLACEMENT` | `true` | When at cap, swap a weaker tracked image for a better candidate. With Frigate scoring active, targets the most redundant image (highest pre-upload recognize score); otherwise uses blur score. Never touches manually added Frigate files. Set `false` to skip people at cap |
-| `FRIGATE_SCORE_CEILING` | `0.0` | Skip uploads whose pre-upload Frigate recognize score exceeds this value — they are already well-covered. `0` disables; requires at least one prior run to have scores |
-| `ENABLE_FRIGATE_SCORES` | `true` | Call Frigate's recognize endpoint pre-upload to store diversity scores used for quality replacement. Adds ~200 ms per upload. Disable to use blur-score replacement only |
 
 ### GPU & Models
 
@@ -225,8 +241,9 @@ In scheduled mode the process (and loaded models) stays resident between runs. T
 | Variable | Default | Description |
 | :--- | :--- | :--- |
 | `DRY_RUN` | `false` | Preview selection without downloading or uploading |
-| `RETRY_REJECTED` | `false` | Re-attempt assets previously rejected by Frigate |
+| `RETRY_REJECTED` | `false` | Re-attempt all previously rejected assets (low-confidence skips, Frigate rejections, and other permanent exclusions) |
-| `RESET_PERSON` | *(unset)* | Clear upload history for one person and delete their winnow-managed Frigate training files so the next run starts fresh. Manually added Frigate files are never touched |
+| `RESET_PERSON` | *(unset)* | Set to a person's name to clear their upload history and delete their winnow-managed Frigate training files so the next run starts fresh. Set to `*` to reset all tracked people at once. Manually added Frigate files are never touched |
+| `TRACE_CROP_SIZE` | *(unset)* | Debug: print all tracked crops whose width or height matches this pixel value, then exit |
 
 ### Scheduling
 
@@ -247,7 +264,7 @@ uv run winnow
 
 Requires Python 3.13+ and [uv](https://astral.sh/uv). An NVIDIA, AMD, or Intel GPU is recommended — CPU mode works but embedding computation is slower.
 
-When run with a terminal attached, winnow starts an interactive session: select which people to process and choose a strategy (auto, standard, broad, or a custom count) per person. Without a TTY — Docker, cron, or `AUTO_MODE=true` — it processes all people automatically using the configured defaults.
+When run with a terminal attached, winnow starts an interactive session: select which people to process and choose a strategy (adaptive, standard, broad, or a custom count) per person. Without a TTY — Docker, cron, or `AUTO_MODE=true` — it processes all people unattended using the configured defaults.
 
 ---
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "winnow"
-version = "0.4.5"
+version = "0.4.9"
 description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition and object classification."
 license = "AGPL-3.0-or-later"
 requires-python = ">=3.13"

tests/test_config.py

@@ -18,7 +18,7 @@ def test_config_loads_defaults(monkeypatch):
     assert cfg.OUTPUT_DIR == "./frigate_train"
     assert cfg.YEARS_FILTER == 10
     assert cfg.MIN_FACE_WIDTH == 90
-    assert cfg.MIN_FACE_COUNT == 0
+    assert cfg.MIN_FACE_COUNT == 3
     assert cfg.BLUR_THRESHOLD == 120.0
     assert cfg.MIN_CONFIDENCE == 0.7
     assert cfg.MAX_AUTO_IMAGES == 80

uv.lock

@@ -2348,7 +2348,7 @@ wheels = [
 
 [[package]]
 name = "winnow"
-version = "0.4.4"
+version = "0.4.8"
 source = { editable = "." }
 dependencies = [
     { name = "croniter" },

winnow/cli.py

@@ -130,6 +130,18 @@ def _handle_duplicate_people(people: list[dict]) -> list[dict]:
     return people
 
 
+_UNSUPPORTED_VARS = [
+    "ENABLE_FRIGATE_SCORES",
+    "FRIGATE_SCORE_CEILING",
+    "MIN_FACE_WIDTH",
+    "FACE_MARGIN",
+    "ENABLE_FACE_ALIGNMENT",
+    "USE_FULL_RESOLUTION",
+    "MIN_CONFIDENCE",
+    "BLUR_THRESHOLD",
+]
+
+
 def main() -> None:
     """Entry point for winnow CLI."""
     try:
@@ -145,6 +157,17 @@ def main() -> None:
     [dim]Immich -> Frigate Training Data Curator[/dim]
         """)
 
+        set_unsupported = [v for v in _UNSUPPORTED_VARS if os.environ.get(v)]
+        if set_unsupported:
+            console.print(
+                f"[bold yellow]⚠  Advanced tuning vars set: "
+                f"{', '.join(set_unsupported)}[/bold yellow]"
+            )
+            console.print(
+                "[dim]  These defaults are calibrated for Frigate's ArcFace requirements. "
+                "Image quality issues caused by non-default values will not be investigated.[/dim]\n"
+            )
+
         ConfigManager.get().interactive_setup()
 
         try:

winnow/config.py

@@ -31,11 +31,11 @@ class _Config:
     MIN_CONFIDENCE: float = 0.7
     MAX_AUTO_IMAGES: int = 80
     QUALITY_REPLACEMENT: bool = True
-    FRIGATE_SCORE_CEILING: float = 0.0
+    FRIGATE_SCORE_CEILING: float | None = None
     ENABLE_FRIGATE_SCORES: bool = True
 
     # People filtering
-    MIN_FACE_COUNT: int = 0
+    MIN_FACE_COUNT: int = 3
     MERGE_DUPLICATE_PEOPLE: bool = False
 
     # Output quality
@@ -60,13 +60,14 @@ class _Config:
         self.OUTPUT_DIR = os.getenv("OUTPUT_DIR", "./frigate_train")
         self.YEARS_FILTER = int(os.getenv("YEARS_FILTER", "10"))
         self.MIN_FACE_WIDTH = int(os.getenv("MIN_FACE_WIDTH", "90"))
-        self.MIN_FACE_COUNT = int(os.getenv("MIN_FACE_COUNT", "0"))
+        self.MIN_FACE_COUNT = int(os.getenv("MIN_FACE_COUNT", "3"))
         self.MERGE_DUPLICATE_PEOPLE = os.getenv("MERGE_DUPLICATE_PEOPLE", "false").lower() in ("true", "1", "yes")
         self.BLUR_THRESHOLD = float(os.getenv("BLUR_THRESHOLD", "120.0"))
         self.MIN_CONFIDENCE = float(os.getenv("MIN_CONFIDENCE", "0.7"))
         self.MAX_AUTO_IMAGES = int(os.getenv("MAX_AUTO_IMAGES", "80"))
         self.QUALITY_REPLACEMENT = os.getenv("QUALITY_REPLACEMENT", "true").lower() in ("true", "1", "yes")
-        self.FRIGATE_SCORE_CEILING = float(os.getenv("FRIGATE_SCORE_CEILING", "0.0"))
+        _ceiling_env = os.getenv("FRIGATE_SCORE_CEILING", "").strip()
+        self.FRIGATE_SCORE_CEILING = float(_ceiling_env) if _ceiling_env else None
         self.ENABLE_FRIGATE_SCORES = os.getenv("ENABLE_FRIGATE_SCORES", "true").lower() in ("true", "1", "yes")
         self.FACE_MARGIN = float(os.getenv("FACE_MARGIN", "0.15"))
         self.USE_FULL_RESOLUTION = os.getenv("USE_FULL_RESOLUTION", "true").lower() in ("true", "1", "yes")

winnow/diversity.py

@@ -337,16 +337,19 @@ def _dedup_embeddings(
     order = sorted(range(len(candidates)), key=lambda i: quality_scores[i], reverse=True)
 
     kept_indices = []
-    kept_stack: np.ndarray | None = None  # rebuilt only when a new item is kept (not every iteration)
+    # Pre-allocate a max-size buffer and fill row-by-row — eliminates the O(K²)
+    # copy overhead from vstack-on-keep while keeping identical arithmetic.
+    kept_buf = np.empty((len(order), emb_normed.shape[1]), dtype=emb_normed.dtype)
+    n_kept = 0
 
     for i in order:
-        if kept_stack is not None:
+        if n_kept > 0:
-            sims = emb_normed[i] @ kept_stack.T
+            sims = emb_normed[i] @ kept_buf[:n_kept].T
             if np.any(sims > 1 - _DEDUP_THRESHOLD):
                 continue
+        kept_buf[n_kept] = emb_normed[i]
+        n_kept += 1
         kept_indices.append(i)
-        row = emb_normed[i : i + 1]
-        kept_stack = row if kept_stack is None else np.vstack([kept_stack, row])
 
     dropped = len(embeddings) - len(kept_indices)
     if dropped:
@@ -390,7 +393,7 @@ def _kmedoids(dist_matrix: np.ndarray, k: int, max_iter: int = 50) -> tuple[list
     # Iterative swap step
     medoids = list(medoids)
     labels = np.argmin(dist_matrix[:, medoids], axis=1)
-    cost = sum(dist_matrix[i, medoids[labels[i]]] for i in range(n))
+    cost = dist_matrix[np.arange(n), np.array(medoids)[labels]].sum()
 
     for _ in range(max_iter):
         improved = False
@@ -405,7 +408,7 @@ def _kmedoids(dist_matrix: np.ndarray, k: int, max_iter: int = 50) -> tuple[list
                 new_medoids = medoids.copy()
                 new_medoids[m_idx] = cand
                 new_labels = np.argmin(dist_matrix[:, new_medoids], axis=1)
-                new_cost = sum(dist_matrix[i, new_medoids[new_labels[i]]] for i in range(n))
+                new_cost = dist_matrix[np.arange(n), np.array(new_medoids)[new_labels]].sum()
                 if new_cost < cost:
                     medoids = new_medoids
                     labels = new_labels

winnow/executor.py

@@ -31,7 +31,7 @@ from .upload_tracker import (
     has_frigate_scores,
     mark_rejected,
     mark_uploaded,
-    record_frigate_file,
+    record_frigate_files_batch,
     remove_frigate_file,
 )
 
@@ -100,10 +100,12 @@ def _reconcile_frigate_mappings(
             except (ValueError, IndexError):
                 return 0.0
 
-        for (fname, asset_id), frigate_file in zip(uploaded, sorted(new_files, key=_ts)):
+        mappings = {
-            if asset_id:
+            frigate_file: asset_id
-                record_frigate_file(person_name, frigate_file, asset_id)
+            for (_, asset_id), frigate_file in zip(uploaded, sorted(new_files, key=_ts))
-        logger.debug(f"{person_name}: batch-mapped {target} Frigate file(s)")
+            if asset_id
+        }
+        record_frigate_files_batch(person_name, mappings)
     elif len(new_files) > target:
         logger.info(
             f"{person_name}: {len(new_files)} new Frigate files for {target} uploads"
@@ -229,6 +231,7 @@ def execute_jobs(jobs: list[dict]) -> None:
                                 f"[yellow]Skipped {asset['id']}"
                                 f" (detection confidence {conf:.2f} < {Config.MIN_CONFIDENCE})[/yellow]"
                             )
+                            mark_rejected(asset["id"], person_name=name)
                             progress.advance(job_task)
                             progress.advance(overall_task)
                             continue
@@ -480,14 +483,28 @@ def upload_to_frigate(jobs: list[dict]) -> None:
                         if _result is not None and (_result[0] or "").casefold() == name.casefold():
                             pre_fscore = _result[1]
 
-                # Ceiling check: skip if the existing training set already covers this
+                # Below-cap novelty gate: skip candidates already covered by the Frigate model,
-                # face condition well. Applies below cap only — at cap, replacement logic
+                # including conditions learned from manually-added images winnow can't track.
-                # drives the decision.
+                # pre_fscore is None on the first run (pre_run_count == 0 skips recognize_face
-                if not at_cap and Config.FRIGATE_SCORE_CEILING > 0 and pre_run_count > 0:
+                # above), so this block never fires on the first run without an extra guard.
-                    if pre_fscore is not None and pre_fscore > Config.FRIGATE_SCORE_CEILING:
+                if not at_cap and pre_fscore is not None:
+                    _ceiling = Config.FRIGATE_SCORE_CEILING
+                    if _ceiling is None:
+                        # Dynamic default: bar = most-redundant tracked file's Frigate score.
+                        # Falls back to uploading freely when no tracked scores exist yet.
+                        _bar = get_most_redundant_mapped_file(name)
+                        _skip = _bar is not None and pre_fscore > _bar[2]
+                        _bar_str = f"most redundant tracked {_bar[2]:.2f}" if _bar else ""
+                    elif _ceiling == 0.0:
+                        _skip = False  # explicitly disabled
+                        _bar_str = ""
+                    else:
+                        _skip = pre_fscore > _ceiling
+                        _bar_str = f"ceiling {_ceiling:.2f}"
+                    if _skip:
                         progress.console.print(
                             f"    [dim]⏭  {fname}: Frigate score {pre_fscore:.2f}"
-                            f" > ceiling {Config.FRIGATE_SCORE_CEILING:.2f}, already covered[/dim]"
+                            f" > {_bar_str}, already covered[/dim]"
                         )
                         progress.advance(upload_task)
                         continue
@@ -501,70 +518,50 @@ def upload_to_frigate(jobs: list[dict]) -> None:
                     using_fscore = person_has_fscores and Config.ENABLE_FRIGATE_SCORES
                     if using_fscore:
                         candidate_score = pre_fscore
-                        if candidate_score is None:
+                        get_target = get_most_redundant_mapped_file
-                            progress.console.print(
+                        score_label, better_note = "frigate", " (more novel)"
-                                f"    [dim]⏭  {fname}: Frigate recognize unavailable, skipping replacement[/dim]"
+                        no_score_msg = "Frigate recognize unavailable, skipping replacement"
-                            )
-                            progress.advance(upload_task)
-                            continue
-                        # Low score = more novel than the most redundant mapped file = replace
-                        target = get_most_redundant_mapped_file(name, exclude=failed_deletes)
-                        if target is None or candidate_score >= target[2]:
-                            target_score_str = f"{target[2]:.3f}" if target is not None else "N/A"
-                            progress.console.print(
-                                f"    [dim]⏭  {fname}: frigate {candidate_score:.3f} ≥ most redundant"
-                                f" {target_score_str}, not more novel[/dim]"
-                            )
-                            progress.advance(upload_task)
-                            continue
-                        target_frigate_file, _target_asset_id, target_score = target
-                        progress.console.print(
-                            f"    🔄 {fname}: frigate {candidate_score:.3f} < {target_score:.3f},"
-                            f" replacing {target_frigate_file} (more novel)"
-                        )
-                        if delete_frigate_person_files(name, [target_frigate_file]):
-                            remove_frigate_file(name, target_frigate_file)
-                            person_has_fscores = has_frigate_scores(name)
-                            effective_count -= 1
-                            # clear any blur-mode slot floor — Frigate uses a different score metric
-                            min_quality_score_for_slot = None
-                        else:
-                            logger.warning(f"Failed to delete {target_frigate_file} for {name}, skipping replacement")
-                            failed_deletes.add(target_frigate_file)
-                            progress.advance(upload_task)
-                            continue
                     else:
                         candidate_score = score_map.get(fname)
-                        if candidate_score is None:
+                        get_target = get_lowest_quality_mapped_file
-                            progress.console.print(
+                        score_label, better_note = "blur", ""
-                                f"    [dim]⏭  {fname}: no quality score, skipping replacement[/dim]"
+                        no_score_msg = "no quality score, skipping replacement"
-                            )
+
-                            progress.advance(upload_task)
+                    if candidate_score is None:
-                            continue
+                        progress.console.print(f"    [dim]⏭  {fname}: {no_score_msg}[/dim]")
-                        target = get_lowest_quality_mapped_file(name, exclude=failed_deletes)
+                        progress.advance(upload_task)
-                        if target is None or candidate_score <= target[2]:
+                        continue
-                            target_score_str = f"{target[2]:.3f}" if target is not None else "N/A"
+
-                            progress.console.print(
+                    target = get_target(name, exclude=failed_deletes)
-                                f"    [dim]⏭  {fname}: blur {candidate_score:.3f} ≤ worst"
+                    not_better = target is None or (
-                                f" {target_score_str}, skipping[/dim]"
+                        candidate_score >= target[2] if using_fscore else candidate_score <= target[2]
-                            )
+                    )
-                            progress.advance(upload_task)
+                    if not_better:
-                            continue
+                        target_str = f"{target[2]:.3f}" if target is not None else "N/A"
-                        target_frigate_file, _target_asset_id, target_score = target
+                        op = "<" if using_fscore else ">"
                         progress.console.print(
-                            f"    🔄 {fname}: blur {candidate_score:.3f} > {target_score:.3f},"
+                            f"    [dim]⏭  {fname}: {score_label} {candidate_score:.3f}"
-                            f" replacing {target_frigate_file}"
+                            f" not {op} {target_str}, skipping[/dim]"
                         )
-                        if delete_frigate_person_files(name, [target_frigate_file]):
+                        progress.advance(upload_task)
-                            remove_frigate_file(name, target_frigate_file)
+                        continue
-                            person_has_fscores = has_frigate_scores(name)
+
-                            effective_count -= 1
+                    target_frigate_file, _target_asset_id, target_score = target
-                            min_quality_score_for_slot = score_map.get(fname)
+                    op = "<" if using_fscore else ">"
-                        else:
+                    progress.console.print(
-                            logger.warning(f"Failed to delete {target_frigate_file} for {name}, skipping replacement")
+                        f"    🔄 {fname}: {score_label} {candidate_score:.3f} {op} {target_score:.3f},"
-                            failed_deletes.add(target_frigate_file)
+                        f" replacing {target_frigate_file}{better_note}"
-                            progress.advance(upload_task)
+                    )
-                            continue
+                    if delete_frigate_person_files(name, [target_frigate_file]):
+                        remove_frigate_file(name, target_frigate_file)
+                        person_has_fscores = has_frigate_scores(name)
+                        effective_count -= 1
+                        min_quality_score_for_slot = None if using_fscore else candidate_score
+                    else:
+                        logger.warning(f"Failed to delete {target_frigate_file} for {name}, skipping replacement")
+                        failed_deletes.add(target_frigate_file)
+                        progress.advance(upload_task)
+                        continue
 
                 for attempt in range(1, max_retries + 1):
                     try:

winnow/immich_api.py

@@ -14,6 +14,7 @@ from .config import Config, get_headers
 logger = logging.getLogger(__name__)
 
 MAX_PAGES = 1000  # Safety limit for pagination
+_MAX_ASSETS_PER_PERSON = 5000  # Stop fetching after this many — diversity pool is capped at 3000 anyway
 
 
 @dataclass
@@ -95,10 +96,10 @@ def fetch_all_assets(person: dict) -> list[dict]:
             if not page_assets:
                 break
 
-            assets.extend(page_assets)
+            assets.extend(a for a in page_assets if isinstance(a, dict))
             logger.debug(f"Fetched page {page}, total: {len(assets)}")
 
-            if len(page_assets) < page_size:
+            if len(page_assets) < page_size or len(assets) >= _MAX_ASSETS_PER_PERSON:
                 break
 
         except (requests.RequestException, ValueError) as e:

winnow/jobs.py

@@ -20,7 +20,7 @@ logger = logging.getLogger(__name__)
 
 # Strategy presets: (limit, mode_name)
 STRATEGY_PRESETS = {
-    "1": ("auto", "Auto Diversity"),
+    "1": ("auto", "Adaptive Diversity"),
     "2": (30, "Standard (30)"),
     "3": (100, "Broad (100)"),
 }
@@ -31,7 +31,7 @@ def _get_strategy_choice(has_embedding: bool, entity_type: str) -> tuple[int | s
     model_name = "InsightFace" if entity_type == "face" else "SigLIP"
 
     if has_embedding:
-        rprint("  [bold]1.[/bold] Auto (Objective Diversity) [green][Recommended][/green]")
+        rprint("  [bold]1.[/bold] Adaptive Diversity [green][Recommended][/green]")
         rprint("     [dim]• Dynamically selects images until redundancy starts[/dim]")
         rprint("  [bold]2.[/bold] Standard (30 images)")
         rprint("  [bold]3.[/bold] Broad (100 images)")
@@ -77,7 +77,8 @@ def _resolve_strategy(strategy: str, has_embedding: bool) -> tuple[int | str, st
         return int(custom_limit), "smart"
 
     strategy_map = {
-        "auto": ("auto", "smart"),
+        "adaptive": ("auto", "smart"),
+        "auto": ("auto", "smart"),  # legacy alias for adaptive
         "standard": (30, "smart"),
         "broad": (100, "smart"),
     }

winnow/upload_tracker.py

@@ -39,6 +39,11 @@ logger = logging.getLogger(__name__)
 UPLOAD_TRACKER_FILE = "frigate_uploaded_ids.json"
 REJECT_TRACKER_FILE = "frigate_rejected_ids.json"
 
+# Write-through in-memory cache keyed by the resolved file path.
+# Reduces per-call JSON reads from O(calls) to O(1) after the first load.
+# Keyed by full path so tests with isolated tmp dirs never share entries.
+_cache: dict[str, dict] = {}
+
 
 def _tracker_path(filename: str) -> Path:
     try:
@@ -50,18 +55,23 @@ def _tracker_path(filename: str) -> Path:
 
 def _load(filename: str) -> dict:
     path = _tracker_path(filename)
-    if not path.exists():
+    key = str(path)
-        return {}
+    if key in _cache:
-    try:
+        return _cache[key]
-        with open(path) as f:
+    data: dict = {}
-            return json.load(f)
+    if path.exists():
-    except (json.JSONDecodeError, OSError) as e:
+        try:
-        logger.warning(f"Could not load tracker {filename}: {e}")
+            with open(path) as f:
-        return {}
+                data = json.load(f)
+        except (json.JSONDecodeError, OSError) as e:
+            logger.warning(f"Could not load tracker {filename}: {e}")
+    _cache[key] = data
+    return data
 
 
 def _save(filename: str, data: dict) -> None:
     path = _tracker_path(filename)
+    _cache[str(path)] = data  # keep cache consistent with what we write
     path.parent.mkdir(parents=True, exist_ok=True)
     with open(path, "w") as f:
         json.dump(data, f, indent=2)
@@ -161,6 +171,19 @@ def record_frigate_file(person_name: str, frigate_filename: str, asset_id: str)
     logger.debug(f"Mapped Frigate file {frigate_filename} → {asset_id} ({person_name})")
 
 
+def record_frigate_files_batch(person_name: str, mappings: dict[str, str]) -> None:
+    """Record multiple Frigate filename → asset_id mappings in a single load/save."""
+    if not mappings:
+        return
+    data = _load(UPLOAD_TRACKER_FILE)
+    by_person = data.setdefault("by_person", {})
+    entry = _migrate_entry(by_person.get(person_name, {}))
+    entry["frigate_files"].update(mappings)
+    by_person[person_name] = entry
+    _save(UPLOAD_TRACKER_FILE, data)
+    logger.debug(f"Batch-mapped {len(mappings)} Frigate file(s) for {person_name}")
+
+
 def remove_frigate_file(person_name: str, frigate_filename: str) -> None:
     """Remove a Frigate filename from the mapping after it has been deleted.