fix: v0.5.21 — diversity cap, fetch rejection, tracker isolation, merge dedup, env parsing
- diversity: cap k-medoids seed count at target so _cluster_aware_selection never returns more images than requested (violated MAX_AUTO_IMAGES when remaining capacity was 1-4 slots); add early return for limit=0 to prevent k-medoids from running with a zero budget; slice return to target as a final guard - executor: mark_rejected() when fetch_full_image returns None so assets that can't be fetched (both original and preview) aren't retried every run - executor: wrap mark_uploaded() in its own try/except so a SQLite disk-full error after a successful HTTP 200 doesn't retry the Frigate POST (duplicate upload) — the upload succeeded; only the tracker write failed - cli: apply skip_ids deduplication to the re-fetched people list after a partial merge (some groups succeed, some fail) so unmerged duplicates don't produce two jobs for the same Frigate folder - jobs: strip whitespace from SKIP_PEOPLE/ONLY_PEOPLE elements on split so "Alice, Bob" (space after comma) correctly matches "Bob"
This commit is contained in:
@@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.5.21] - 2026-06-15
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`_cluster_aware_selection` now respects the requested limit**: the initial K-Medoids seed count `k` was never capped at `target`, so when the remaining capacity was 1–4 slots the function returned 5+ images instead of the requested count, silently violating `MAX_AUTO_IMAGES`. `k` is now `min(..., target)` and the returned list is sliced to `target` as a final guard. A new early-return for `limit == 0` prevents k-medoids from running at all and returning medoids for a zero-budget request.
|
||||
|
||||
- **`USE_FULL_RESOLUTION=True` path now marks assets rejected on persistent fetch failure**: when both the original and preview fallback in `fetch_full_image()` fail, the asset was silently re-selected and re-attempted on every future run. The full-res path now calls `mark_rejected()` on a `None` return, matching the behavior added in v0.5.20 for the thumbnail path.
|
||||
|
||||
- **`mark_uploaded` tracker failure no longer causes a duplicate Frigate upload**: `mark_uploaded()` was called inside the upload retry `try/except` block. A SQLite error (e.g. disk-full) after a successful HTTP 200 response would propagate to the retry handler, which would retry the POST and upload the same file twice. `mark_uploaded()` is now wrapped in its own `try/except`; a tracker write failure is logged and the upload loop breaks normally so Frigate never receives a duplicate.
|
||||
|
||||
- **`_handle_duplicate_people` deduplicates failed merges when some succeed**: when `MERGE_DUPLICATE_PEOPLE=true` and a mix of merges succeed and fail, `get_people()` was returned directly. The re-fetched list still contained the un-merged duplicate pairs, creating two jobs for the same Frigate folder. The re-fetched list is now filtered using the same skip-id logic applied in the all-fail path.
|
||||
|
||||
- **`SKIP_PEOPLE`/`ONLY_PEOPLE` now strip whitespace from each element**: `"Alice, Bob".split(",")` produces `[' Bob']` (with a leading space), which never matched person names from Immich. Both env vars now use a list comprehension with `.strip()` on each element, so space-padded comma-separated values work as expected.
|
||||
|
||||
## [0.5.20] - 2026-06-15
|
||||
|
||||
### Fixed
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "winnow"
|
||||
version = "0.5.20"
|
||||
version = "0.5.21"
|
||||
description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition."
|
||||
license = "AGPL-3.0-or-later"
|
||||
requires-python = ">=3.13"
|
||||
|
||||
+11
-1
@@ -125,7 +125,17 @@ def _handle_duplicate_people(people: list[dict]) -> list[dict]:
|
||||
|
||||
if merged_any:
|
||||
rprint(" [dim]Re-fetching people after merge...[/dim]")
|
||||
return get_people()
|
||||
fresh = get_people()
|
||||
# Filter out the smaller duplicate from any group whose merge failed — those
|
||||
# IDs still exist in Immich and would produce two jobs for the same folder.
|
||||
# IDs from groups that merged successfully are already gone from Immich, so
|
||||
# this filter is a no-op for them.
|
||||
skip_ids = {
|
||||
p["id"]
|
||||
for ps in duplicates.values()
|
||||
for p in sorted(ps, key=lambda x: x.get("assetCount", 0), reverse=True)[1:]
|
||||
}
|
||||
return [p for p in fresh if p.get("id") not in skip_ids]
|
||||
|
||||
# All merges failed — fall back to local deduplication (keep largest per name) so
|
||||
# downstream job creation never runs two jobs for the same Frigate folder.
|
||||
|
||||
+12
-2
@@ -494,8 +494,13 @@ def _cluster_aware_selection(
|
||||
auto_threshold = _compute_adaptive_threshold(emb_normed) if limit == "auto" else 0.0
|
||||
target = Config.MAX_AUTO_IMAGES if limit == "auto" else limit
|
||||
|
||||
# Short-circuit: nothing to select
|
||||
if limit != "auto" and target <= 0:
|
||||
return []
|
||||
|
||||
# --- Stage 1: K-Medoids clustering ---
|
||||
k = min(max(5, target // 4), max(1, n // 3), n) # e.g., 1-20 clusters
|
||||
# Cap k at target so we never seed more cluster representatives than requested.
|
||||
k = min(max(5, target // 4), max(1, n // 3), n, target) # e.g., 1-20 clusters
|
||||
logger.debug("Clustering %s embeddings into %s groups (K-Medoids)...", n, k)
|
||||
|
||||
# Compute full cosine distance matrix
|
||||
@@ -547,7 +552,12 @@ def _cluster_aware_selection(
|
||||
hard_count = sum(1 for c in selected_conf if c < 0.85)
|
||||
logger.info("Selection complete: %s images (%s hard examples with confidence < 0.85).", len(selected), hard_count)
|
||||
|
||||
return [candidates[i] for i in selected]
|
||||
# Slice to target: the while loop enforces this for non-auto mode, but
|
||||
# guard here too in case the medoid seed already exceeded target (small target).
|
||||
result = [candidates[i] for i in selected]
|
||||
if limit != "auto":
|
||||
result = result[:target]
|
||||
return result
|
||||
|
||||
|
||||
# =============================================================================
|
||||
|
||||
+20
-7
@@ -149,6 +149,10 @@ def execute_jobs(jobs: list[dict]) -> None:
|
||||
# Use full-resolution for final output when configured
|
||||
if use_full_res:
|
||||
img = fetch_full_image(asset["id"])
|
||||
if img is None:
|
||||
# Both original and preview fallback failed — mark rejected
|
||||
# so this asset isn't retried on every future run.
|
||||
mark_rejected(asset["id"], person_name=name)
|
||||
else:
|
||||
resp = requests.get(
|
||||
f"{Config.IMMICH_URL}/api/assets/{asset['id']}/thumbnail?size=preview&format=JPEG",
|
||||
@@ -498,13 +502,22 @@ def upload_to_frigate(jobs: list[dict]) -> None:
|
||||
|
||||
asset_id = asset_map.get(fname)
|
||||
if asset_id:
|
||||
mark_uploaded(
|
||||
asset_id,
|
||||
person_name=name,
|
||||
score=score_map.get(fname),
|
||||
crop_dims=dims_map.get(fname),
|
||||
frigate_score=pre_fscore,
|
||||
)
|
||||
try:
|
||||
mark_uploaded(
|
||||
asset_id,
|
||||
person_name=name,
|
||||
score=score_map.get(fname),
|
||||
crop_dims=dims_map.get(fname),
|
||||
frigate_score=pre_fscore,
|
||||
)
|
||||
except Exception as tracker_exc:
|
||||
# Upload to Frigate succeeded — don't retry on tracker
|
||||
# failure or we'd upload a duplicate to Frigate.
|
||||
logger.error(
|
||||
"Tracker write failed for %s — upload succeeded"
|
||||
" but asset may be re-selected next run: %s",
|
||||
fname, tracker_exc,
|
||||
)
|
||||
if pre_fscore is not None:
|
||||
person_has_fscores = True
|
||||
actually_uploaded.append((fname, asset_id))
|
||||
|
||||
+2
-2
@@ -229,8 +229,8 @@ def auto_configure(people: list[dict]) -> list[dict]:
|
||||
return []
|
||||
|
||||
strategy = os.environ.get("STRATEGY", "auto")
|
||||
skip = os.environ.get("SKIP_PEOPLE", "").split(",") if os.environ.get("SKIP_PEOPLE") else []
|
||||
only = os.environ.get("ONLY_PEOPLE", "").split(",") if os.environ.get("ONLY_PEOPLE") else []
|
||||
skip = [s.strip() for s in os.environ.get("SKIP_PEOPLE", "").split(",") if s.strip()]
|
||||
only = [s.strip() for s in os.environ.get("ONLY_PEOPLE", "").split(",") if s.strip()]
|
||||
|
||||
if only:
|
||||
valid_people = [p for p in valid_people if p["name"] in only]
|
||||
|
||||
Reference in New Issue
Block a user