perf: eliminate O(K²) dedup allocs, vectorize kmedoids cost, batch tracker writes

- _dedup_embeddings: pre-allocated (Q,D) buffer replaces vstack-on-keep,
  dropping O(K²×D) copy overhead down to O(K×D) fill work
- _kmedoids: swap cost sum replaced with numpy fancy-index reduction,
  ~20-50x faster per swap evaluation
- _reconcile_frigate_mappings: O(L) load/save pairs collapsed to one
  batch write via record_frigate_files_batch

CHANGELOG.md

@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.7] - 2026-06-14
+
+### Changed
+
+- **`_dedup_embeddings` pre-allocated buffer**: replaced the grow-on-keep `np.vstack` pattern with a pre-allocated `(Q, D)` buffer filled row-by-row. Eliminates O(K²) copy work and the GC pressure from K intermediate heap allocations while keeping identical arithmetic for the similarity checks.
+- **`_kmedoids` cost computation vectorized**: the Python-level `sum(dist_matrix[i, medoids[labels[i]]] for i in range(n))` generator (called once per swap evaluation) is replaced with `dist_matrix[np.arange(n), np.array(medoids)[labels]].sum()` — a single numpy fancy-index + reduction, ~20–50× faster in the swap loop.
+- **`_reconcile_frigate_mappings` single-write batch**: previously called `record_frigate_file` once per uploaded file, each doing a full JSON load + save (O(L) disk round-trips per person). Now builds the full `{frigate_filename: asset_id}` mapping dict and writes it in one `record_frigate_files_batch` call (O(1) disk round-trip).
+
+## [0.4.6] - 2026-06-14
+
+### Fixed
+
+- **OOM when Immich returns many pages per person**: `fetch_all_assets` now stops fetching once 5000 assets have been collected — the diversity selection pool is already capped at 3000 items, so fetching up to 1,000,000 was wasteful and could exhaust memory on large libraries. 5000 provides ample headroom for the pool cap while bounding per-person memory to ~2 MB.
+- **Non-dict items in Immich asset pages silently skipped**: a malformed or partially-null Immich response page could include `null` or non-object items in the assets array. These are now filtered at fetch time rather than causing `AttributeError` downstream.
+
+## [0.4.5] - 2026-06-14
+
+### Fixed
+
+- **Near-duplicate dedup O(N²) allocation**: `np.vstack(kept_normed)` was rebuilt on every loop iteration even for candidates that would be dropped; the stack is now rebuilt only when a new item is kept, reducing memory pressure significantly for large pools.
+- **`quality_score` falsy-zero in dedup sort**: the sort key used `c.get("quality_score") or 0.0`, which treated a legitimate `quality_score=0.0` identically to a missing key. Changed to an explicit `None` check so zero is preserved as-is, and object-mode candidates (which have no `quality_score`) continue to sort stably to the back.
+- **Post-dedup pool not re-checked against limit**: after near-duplicate removal the pool could silently shrink below the requested limit with no warning. A second `len < limit` guard now fires after dedup and emits the same "Only N embeddings" warning that the pre-dedup guard does.
+- **`mark_rejected` could miss plain-text 400 bodies longer than 100 bytes**: `error_detail = resp.text[:100]` was being searched for the keyword `"face"` to gate `mark_rejected()`, so a response body with `"face"` after byte 100 would never mark the asset rejected and it would be retried on every future run. The `"face"` check now uses the full response body; truncation is kept only for the displayed snippet.
+- **`_safe_person_dir` raised ValueError for all person names when `output_dir` resolved to `/`**: `base + os.sep` produced `"//"` when base was `"/"`, and valid paths like `/alice` don't start with `"//"`. Fixed by using `base` directly as the prefix when `base == os.sep`.
+
 ## [0.4.4] - 2026-06-14
 
 ### Added

README.md

@@ -2,6 +2,8 @@
 
 [![Docker](https://github.com/sudolulo/winnow/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/sudolulo/winnow/actions/workflows/docker-publish.yml) [![Test](https://github.com/sudolulo/winnow/actions/workflows/test.yml/badge.svg)](https://github.com/sudolulo/winnow/actions/workflows/test.yml) [![GitHub release](https://img.shields.io/github/v/release/sudolulo/winnow)](https://github.com/sudolulo/winnow/releases/latest) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](LICENSE) [![Immich](https://img.shields.io/badge/Immich-v1.106%2B-blueviolet)](https://immich.app) [![Frigate](https://img.shields.io/badge/Frigate-Ready-brightgreen)](https://frigate.video)
 
+> **Note:** winnow's approach to training Frigate face recognition is not an officially documented workflow — results may vary.
+
 > **Early Development — Use With Caution**
 > winnow is functional but still maturing. Features that modify your Frigate training data — quality replacement, stale mapping cleanup — can remove images from your dataset and are not yet battle-tested at scale. Review the logs after each run and keep backups of your Frigate face training directory until you are confident in the results.
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "winnow"
-version = "0.4.4"
+version = "0.4.7"
 description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition and object classification."
 license = "AGPL-3.0-or-later"
 requires-python = ">=3.13"

uv.lock

@@ -2348,7 +2348,7 @@ wheels = [
 
 [[package]]
 name = "winnow"
-version = "0.4.3"
+version = "0.4.5"
 source = { editable = "." }
 dependencies = [
     { name = "croniter" },

winnow/diversity.py

@@ -290,6 +290,11 @@ def _select_by_embedding(
         embeddings, valid_candidates, confidence_scores
     )
 
+    # Re-check after dedup: pool may have shrunk below limit
+    if limit != "auto" and len(valid_candidates) < limit:
+        logger.warning(f"Only {len(valid_candidates)} embeddings after near-duplicate removal. Returning all.")
+        return valid_candidates
+
     # --- Phase 6: Cluster-aware selection ---
     return _cluster_aware_selection(
         embeddings,
@@ -326,21 +331,25 @@ def _dedup_embeddings(
     norms = np.linalg.norm(emb_matrix, axis=1, keepdims=True)
     emb_normed = emb_matrix / np.maximum(norms, 1e-8)
 
-    # Sort by quality descending so the best image in each near-duplicate group wins
+    # Sort by quality descending so the best image in each near-duplicate group wins.
-    quality_scores = [c.get("quality_score") or 0.0 for c in candidates]
+    # Use explicit None check so a legitimate quality_score=0.0 isn't treated as missing.
+    quality_scores = [qs if (qs := c.get("quality_score")) is not None else 0.0 for c in candidates]
     order = sorted(range(len(candidates)), key=lambda i: quality_scores[i], reverse=True)
 
     kept_indices = []
-    kept_normed = []
+    # Pre-allocate a max-size buffer and fill row-by-row — eliminates the O(K²)
+    # copy overhead from vstack-on-keep while keeping identical arithmetic.
+    kept_buf = np.empty((len(order), emb_normed.shape[1]), dtype=emb_normed.dtype)
+    n_kept = 0
 
     for i in order:
-        if kept_normed:
+        if n_kept > 0:
-            kept_stack = np.vstack(kept_normed)
+            sims = emb_normed[i] @ kept_buf[:n_kept].T
-            sims = emb_normed[i] @ kept_stack.T
             if np.any(sims > 1 - _DEDUP_THRESHOLD):
                 continue
+        kept_buf[n_kept] = emb_normed[i]
+        n_kept += 1
         kept_indices.append(i)
-        kept_normed.append(emb_normed[i])
 
     dropped = len(embeddings) - len(kept_indices)
     if dropped:
@@ -384,7 +393,7 @@ def _kmedoids(dist_matrix: np.ndarray, k: int, max_iter: int = 50) -> tuple[list
     # Iterative swap step
     medoids = list(medoids)
     labels = np.argmin(dist_matrix[:, medoids], axis=1)
-    cost = sum(dist_matrix[i, medoids[labels[i]]] for i in range(n))
+    cost = dist_matrix[np.arange(n), np.array(medoids)[labels]].sum()
 
     for _ in range(max_iter):
         improved = False
@@ -399,7 +408,7 @@ def _kmedoids(dist_matrix: np.ndarray, k: int, max_iter: int = 50) -> tuple[list
                 new_medoids = medoids.copy()
                 new_medoids[m_idx] = cand
                 new_labels = np.argmin(dist_matrix[:, new_medoids], axis=1)
-                new_cost = sum(dist_matrix[i, new_medoids[new_labels[i]]] for i in range(n))
+                new_cost = dist_matrix[np.arange(n), np.array(new_medoids)[new_labels]].sum()
                 if new_cost < cost:
                     medoids = new_medoids
                     labels = new_labels

winnow/executor.py

@@ -32,6 +32,7 @@ from .upload_tracker import (
     mark_rejected,
     mark_uploaded,
     record_frigate_file,
+    record_frigate_files_batch,
     remove_frigate_file,
 )
 
@@ -46,7 +47,10 @@ def _safe_person_dir(output_dir: str, person_name: str) -> str:
     """
     candidate = os.path.realpath(os.path.join(output_dir, person_name))
     base = os.path.realpath(output_dir)
-    if not candidate.startswith(base + os.sep) and candidate != base:
+    # Use the base path as its own prefix when it's the filesystem root ("/"),
+    # otherwise append os.sep — avoids the false "//" double-slash when base == "/".
+    base_prefix = base if base == os.sep else base + os.sep
+    if not candidate.startswith(base_prefix) and candidate != base:
         raise ValueError(f"Person name {person_name!r} escapes output directory — skipping")
     return candidate
 
@@ -97,10 +101,12 @@ def _reconcile_frigate_mappings(
             except (ValueError, IndexError):
                 return 0.0
 
-        for (fname, asset_id), frigate_file in zip(uploaded, sorted(new_files, key=_ts)):
+        mappings = {
-            if asset_id:
+            frigate_file: asset_id
-                record_frigate_file(person_name, frigate_file, asset_id)
+            for (_, asset_id), frigate_file in zip(uploaded, sorted(new_files, key=_ts))
-        logger.debug(f"{person_name}: batch-mapped {target} Frigate file(s)")
+            if asset_id
+        }
+        record_frigate_files_batch(person_name, mappings)
     elif len(new_files) > target:
         logger.info(
             f"{person_name}: {len(new_files)} new Frigate files for {target} uploads"
@@ -603,15 +609,16 @@ def upload_to_frigate(jobs: list[dict]) -> None:
                             progress.console.print(
                                 f"    [red]✗ {fname}: HTTP {resp.status_code} (after {max_retries} attempts)[/red]"
                             )
+                            full_body = resp.text
                             try:
-                                error_detail = resp.json().get("message", resp.text[:100])
+                                error_detail = resp.json().get("message", full_body[:100])
                             except Exception:
-                                error_detail = resp.text[:100]
+                                error_detail = full_body[:100]
                             if resp.status_code == 400:
                                 progress.console.print(f"      [dim]{error_detail}[/dim]")
                             else:
                                 logger.debug(f"{fname} HTTP {resp.status_code}: {error_detail}")
-                            if resp.status_code == 400 and "face" in error_detail.lower():
+                            if resp.status_code == 400 and "face" in full_body.lower():
                                 asset_id = asset_map.get(fname)
                                 if asset_id:
                                     mark_rejected(asset_id, person_name=name)

winnow/immich_api.py

@@ -14,6 +14,7 @@ from .config import Config, get_headers
 logger = logging.getLogger(__name__)
 
 MAX_PAGES = 1000  # Safety limit for pagination
+_MAX_ASSETS_PER_PERSON = 5000  # Stop fetching after this many — diversity pool is capped at 3000 anyway
 
 
 @dataclass
@@ -95,10 +96,10 @@ def fetch_all_assets(person: dict) -> list[dict]:
             if not page_assets:
                 break
 
-            assets.extend(page_assets)
+            assets.extend(a for a in page_assets if isinstance(a, dict))
             logger.debug(f"Fetched page {page}, total: {len(assets)}")
 
-            if len(page_assets) < page_size:
+            if len(page_assets) < page_size or len(assets) >= _MAX_ASSETS_PER_PERSON:
                 break
 
         except (requests.RequestException, ValueError) as e:

winnow/upload_tracker.py

@@ -161,6 +161,19 @@ def record_frigate_file(person_name: str, frigate_filename: str, asset_id: str)
     logger.debug(f"Mapped Frigate file {frigate_filename} → {asset_id} ({person_name})")
 
 
+def record_frigate_files_batch(person_name: str, mappings: dict[str, str]) -> None:
+    """Record multiple Frigate filename → asset_id mappings in a single load/save."""
+    if not mappings:
+        return
+    data = _load(UPLOAD_TRACKER_FILE)
+    by_person = data.setdefault("by_person", {})
+    entry = _migrate_entry(by_person.get(person_name, {}))
+    entry["frigate_files"].update(mappings)
+    by_person[person_name] = entry
+    _save(UPLOAD_TRACKER_FILE, data)
+    logger.debug(f"Batch-mapped {len(mappings)} Frigate file(s) for {person_name}")
+
+
 def remove_frigate_file(person_name: str, frigate_filename: str) -> None:
     """Remove a Frigate filename from the mapping after it has been deleted.