Compare commits

...
13 Commits
Author SHA1 Message Date
flanandClaude Sonnet 4.6 b98fc94179 ci: remove lockfile verify race condition
uv lock --check in a separate job races against the update-lockfile
bot. Replace with uv lock inline in release.yml and drop the pre-job
from docker-publish.yml entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 03:43:05 +00:00
github-actions[bot] 762ee160c3 chore: update uv.lock 2026-06-12 03:34:10 +00:00
flanandClaude Sonnet 4.6 759579fc30 feat: confidence scores, Frigate-authoritative capacity cap
- Store Immich face confidence scores per asset in frigate_uploaded_ids.json
- Add frigate_api.py: query GET /api/faces to count trained images per person
- Record Frigate training count as frigate_count in tracker for offline fallback
- MAX_AUTO_IMAGES cap now uses live Frigate count → cached frigate_count → local uploaded count
- Startup summary shows last known Frigate training count per person
- Migrate by_person entries from flat list to {asset_ids, scores, frigate_count} dict

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 03:33:40 +00:00
flanandClaude Sonnet 4.6 0e416176c6 docs: clarify use case includes people who have never been to your property
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:14:26 +00:00
flanandClaude Sonnet 4.6 e856cba6d7 docs: explain use case for infrequent visitors
Adds context that winnow is especially useful for people who aren't
around enough for Frigate's live detections to supply adequate training
data on their own.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:14:16 +00:00
flanandClaude Sonnet 4.6 e4d5f603d8 docs: remove TrueNAS references, make volume paths platform agnostic
- compose.yml: generic /path/to/winnow/... placeholder paths
- docs/setup.md: replace TrueNAS section with generic GPU passthrough guide
- docs/faq.md: remove platform-specific path example
- README.md: soften crop quality disclaimer, remove TrueNAS compose note
- CHANGELOG.md: scrub TrueNAS mentions from release notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:13:29 +00:00
flanandClaude Sonnet 4.6 0a67699db4 docs: add early stage disclaimer and manual review note
Advises users to review Frigate uploads after a run and remove bad
crops manually. Links to GitHub Issues for feedback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:12:12 +00:00
flanandClaude Sonnet 4.6 f9b9cea84f docs: add lint fix to 0.2.1 changelog
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:11:35 +00:00
flanandClaude Sonnet 4.6 4b6bce9b74 fix: resolve ruff lint errors in scheduler and tests
- scheduler.py: sort import block (I001)
- tests/test_config.py: remove unused os, pytest imports (F401)
- tests/test_upload_tracker.py: remove unused json, os, tempfile imports (F401)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:11:16 +00:00
flanandClaude Sonnet 4.6 c3bf81d490 fix: direct venv python in entrypoint, fix INSIGHTFACE_HOME path, add badges
- entrypoint.sh: replace `uv run` with `/app/.venv/bin/python` to skip
  uv's sync check which was re-downloading ruff and rebuilding the package
  on every container startup
- compose.yml, .env.example: fix INSIGHTFACE_HOME /models → /models/.insightface;
  InsightFace appends models/ to root, so /models produced /models/models/buffalo_l
- README.md: add Immich and Frigate badges from upstream
- CHANGELOG.md, pyproject.toml, uv.lock: bump to 0.2.1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:10:01 +00:00
flanandClaude Sonnet 4.6 4529a63082 fix: add WORKDIR /app to runtime stage in Dockerfile
Runtime stage was missing WORKDIR, so uv run started from / and
couldn't find the .venv or pyproject.toml, causing "No module named
'winnow'" on container startup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:01:32 +00:00
flanandClaude Sonnet 4.6 ec9d2ecd0f chore: remove compose.override.yml from repo, add to .gitignore
Local dev override with machine-specific paths and build context;
not useful to other users.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 02:01:32 +00:00
flan 1df7c99c2e Update README.md 2026-06-11 21:47:10 -04:00
22 changed files with 219 additions and 96 deletions
+1 -1
View File
@@ -33,7 +33,7 @@ FORCE_CPU=false
ENABLE_CACHE=true
CACHE_DIR=/app/.if_cache
HF_HOME=/models/huggingface
INSIGHTFACE_HOME=/models
INSIGHTFACE_HOME=/models/.insightface
# ── Tracker overrides (one-shot — remove after use) ───────────────────────────
# DRY_RUN=true # Preview selection without downloading/uploading
-15
View File
@@ -15,22 +15,8 @@ env:
IMAGE_NAME: sudolulo/winnow
jobs:
verify-lockfile:
name: Verify uv.lock is current
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
- name: Check lockfile is up to date
run: uv lock --check
build:
name: Build (${{ matrix.platform }})
needs: verify-lockfile
runs-on: ${{ matrix.runner }}
strategy:
matrix:
@@ -140,4 +126,3 @@ jobs:
- name: Inspect image
run: |
docker buildx imagetools inspect ${{ steps.tags.outputs.tags }}
+6 -15
View File
@@ -14,22 +14,8 @@ concurrency:
cancel-in-progress: true
jobs:
verify-lockfile:
name: Verify uv.lock is current
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
- name: Check lockfile is up to date
run: uv lock --check
release:
name: Create GitHub Release & Build Image
needs: verify-lockfile
runs-on: ubuntu-latest
permissions:
contents: write
@@ -48,6 +34,12 @@ jobs:
with:
fetch-depth: 0
- name: Install uv
uses: astral-sh/setup-uv@v4
- name: Ensure uv.lock is current
run: uv lock
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
@@ -138,4 +130,3 @@ jobs:
tags: |
ghcr.io/sudolulo/winnow:latest
ghcr.io/sudolulo/winnow:${{ steps.tag.outputs.TAG }}
+1
View File
@@ -236,3 +236,4 @@ __marimo__/
# Streamlit
.streamlit/secrets.toml
compose.override.yml
+27 -3
View File
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.2.2] - 2026-06-12
### Added
- **Confidence scores in upload tracker**: Immich face confidence scores are now stored per asset in `frigate_uploaded_ids.json` under `by_person[name].scores`. Lays the groundwork for future replacement logic (remove low-confidence uploads when better images are found).
- **Frigate-authoritative capacity tracking**: At startup, `GET /api/faces` is queried on the Frigate host to retrieve the actual number of trained images per person from the `train` directory (pending/unclassified queue is excluded). This count is stored as `frigate_count` in the tracker JSON so it survives Frigate downtime.
- **Lifetime cap uses Frigate count**: `MAX_AUTO_IMAGES` is now enforced against Frigate's live training image count rather than the local uploaded-asset tally. Fallback priority: live Frigate API → last cached `frigate_count` in JSON → local uploaded count.
- **Startup summary shows Frigate count**: Tracker summary at startup now includes the last known Frigate training count per person (e.g. `78 uploaded, 2 rejected, 42 in Frigate`).
- **`winnow/frigate_api.py`**: new module encapsulating Frigate API helpers; currently exposes `get_frigate_face_counts()`.
### Changed
- `upload_tracker.py`: `by_person` entries migrated from flat list to `{asset_ids, scores, frigate_count}` dict. Old list format is read and migrated transparently on first write.
- `mark_uploaded()` now accepts an optional `score` keyword argument.
- `get_person_summary()` now returns `frigate_count` and `scores` fields alongside `uploaded` and `rejected`.
## [0.2.1] - 2026-06-12
### Fixed
- **Container startup reinstalling packages**: `entrypoint.sh` used `uv run`, which performs a sync check on every startup and re-downloaded `ruff` and rebuilt the package each time. Replaced with direct `.venv/bin/python` calls to skip the sync entirely.
- **InsightFace double `models/` path**: `INSIGHTFACE_HOME=/models` caused InsightFace to download Buffalo_L to `/models/models/buffalo_l` (InsightFace always appends `models/` to the root). Updated default in `compose.yml` and `.env.example` to `/models/.insightface`.
- **Lint errors in CI**: unused imports in `tests/test_config.py` and `tests/test_upload_tracker.py`, unsorted imports in `scheduler.py` — all would have failed the ruff CI check.
## [0.2.0] - 2026-06-12
First release of winnow. Forked from [if-curator](https://github.com/ds-sebastian/if_curator) by Sebastian and rewritten for headless Docker deployment.
@@ -21,14 +45,14 @@ First release of winnow. Forked from [if-curator](https://github.com/ds-sebastia
**Docker and scheduling**
- `Dockerfile` — multi-stage build (CUDA 12.9 on amd64, plain Ubuntu on arm64); runtime stage excludes build tools (g++, python3.12-dev, curl, gnupg)
- `compose.yml` — fully annotated with inline comments grouped by concern; TrueNAS volume paths in the example
- `compose.yml` — fully annotated with inline comments grouped by concern
- `entrypoint.sh` — runs the tool once on startup, then hands off to the scheduler if `CRON_SCHEDULE` is set
- `scheduler.py` — in-process cron scheduler that keeps the container (and loaded models) alive between runs
- `CRON_SCHEDULE` env var — standard cron expression for recurring runs; unset exits after first run
- `.dockerignore` — keeps `.venv`, `__pycache__`, test files, and logs out of the image context
- Multi-arch image: `linux/amd64` and `linux/arm64` built and merged into a single manifest on GHCR
- `tini` as PID 1 init process for correct signal handling
- Non-root container user (`appuser`, uid 568) matching TrueNAS default app UID
- Non-root container user (`appuser`, uid 568)
- `HEALTHCHECK` in Dockerfile
**Object mode**
@@ -73,7 +97,7 @@ First release of winnow. Forked from [if-curator](https://github.com/ds-sebastia
- 24 unit tests across four modules: `test_config`, `test_immich_api`, `test_jobs`, `test_upload_tracker`
**Documentation**
- `docs/setup.md` — step-by-step install guide with TrueNAS and bare-Docker sections
- `docs/setup.md` — step-by-step install and GPU passthrough guide
- `docs/troubleshooting.md` — common failure modes with fixes
- `docs/faq.md` — answers to questions new users will ask
- `.env.example` — copy-paste starting point with every env var and inline comments
+1
View File
@@ -68,6 +68,7 @@ RUN groupadd -g 568 apps && useradd -u 568 -g apps -m -s /bin/bash appuser \
&& mkdir -p /models/.insightface /models/huggingface \
&& chown -R appuser:apps /app /models
WORKDIR /app
USER appuser
ENV HF_HOME=/models/huggingface INSIGHTFACE_HOME=/models
+13 -2
View File
@@ -1,4 +1,5 @@
[![Publish Docker Image](https://github.com/sudolulo/winnow/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/sudolulo/winnow/actions/workflows/docker-publish.yml) [![Release](https://github.com/sudolulo/winnow/actions/workflows/release.yml/badge.svg)](https://github.com/sudolulo/winnow/actions/workflows/release.yml) [![Lint](https://github.com/sudolulo/winnow/actions/workflows/lint.yml/badge.svg)](https://github.com/sudolulo/winnow/actions/workflows/lint.yml) [![Test](https://github.com/sudolulo/winnow/actions/workflows/test.yml/badge.svg)](https://github.com/sudolulo/winnow/actions/workflows/test.yml)
[![Immich](https://img.shields.io/badge/Immich-v1.106%2B-violet?style=for-the-badge)](https://immich.app) [![Frigate](https://img.shields.io/badge/Frigate-Ready-green?style=for-the-badge)](https://frigate.video)
**Docs:** [Setup Guide](docs/setup.md) · [Troubleshooting](docs/troubleshooting.md) · [FAQ](docs/faq.md)
@@ -16,6 +17,8 @@ Frigate's face recognition model (ArcFace) and object classifier are only as goo
If you upload 100 photos from the same week, the model learns the lighting in your living room and the jacket you wore that month. It struggles the moment anything changes. What you actually want is a spread: different years, different lighting conditions, different angles, different contexts.
This is especially true for people who have never been to your property, or who visit rarely — family members, friends, anyone Frigate has never seen in person. Live detections alone will never build a reliable model for these people. Your photo library already has the data; winnow finds and delivers the right subset of it.
Finding that spread manually across a library of thousands of photos is not practical. `winnow` does it automatically.
---
@@ -68,6 +71,14 @@ Uploaded asset IDs are recorded so the same image is never uploaded twice, even
---
## Note on Crop Quality
winnow works well, but no automated pipeline is perfect. Occasionally a bad crop will slip through quality filtering — a partial face, someone in the background, a blurry frame. After a run it's worth a quick review in Frigate's face management UI to remove anything that doesn't belong.
Issues and feedback welcome via [GitHub Issues](https://github.com/sudolulo/winnow/issues).
---
## Modes
### Face Mode (default)
@@ -126,7 +137,7 @@ services:
capabilities: [gpu]
```
See [compose.yml](compose.yml) for the full annotated example including TrueNAS volume paths.
See [compose.yml](compose.yml) for the full annotated example.
### Scheduling Behaviour
@@ -227,4 +238,4 @@ Requires Python 3.12+ and [uv](https://astral.sh/uv/). An NVIDIA GPU is strongly
## Attribution
Based on [winnow](https://github.com/ds-sebastian/if_curator) by Sebastian, licensed MIT.
Based on [if_curator](https://github.com/ds-sebastian/if_curator) by Sebastian, licensed MIT.
-8
View File
@@ -1,8 +0,0 @@
services:
winnow:
build: .
network_mode: host
volumes:
- /code/winnow/models:/models
- /code/winnow/embeddings:/app/.if_cache
- /code/winnow/output:/app/frigate_train
+5 -5
View File
@@ -38,7 +38,7 @@ services:
- ENABLE_CACHE=true
- CACHE_DIR=/app/.if_cache
- HF_HOME=/models/huggingface
- INSIGHTFACE_HOME=/models
- INSIGHTFACE_HOME=/models/.insightface
# ── Tracker overrides (one-shot, remove after use) ────────────────────
# - DRY_RUN=true # Preview selection without downloading/uploading
@@ -52,10 +52,10 @@ services:
# - CRON_SCHEDULE=0 3 1 * *
# - CRON_SCHEDULE=*/30 * * * *
volumes:
# Replace <pool> with your TrueNAS pool name, e.g. /mnt/tank/winnow/...
- /mnt/<pool>/winnow/models:/models
- /mnt/<pool>/winnow/embeddings:/app/.if_cache
- /mnt/<pool>/winnow/output:/app/frigate_train
# Replace with absolute paths on your host, e.g. /opt/winnow/models
- /path/to/winnow/models:/models
- /path/to/winnow/cache:/app/.if_cache
- /path/to/winnow/output:/app/frigate_train
stdin_open: true
tty: true
restart: unless-stopped
+1 -1
View File
@@ -49,7 +49,7 @@ ARM builds (linux/arm64) use CPU-only — CUDA is not available on ARM.
## Does it work on Unraid / Proxmox / bare Docker?
Yes — the `compose.yml` uses standard Docker volume mounts. The TrueNAS paths in the example (`/mnt/<pool>/...`) are just an example; replace them with whatever paths suit your setup.
Yes — the `compose.yml` uses standard Docker volume mounts. Replace the example paths with whatever absolute paths suit your setup.
---
+16 -9
View File
@@ -40,7 +40,16 @@ API_KEY=your-immich-api-key
FRIGATE_URL=http://192.168.1.10:5000
```
Edit the volume paths in `compose.yml` to match your storage layout (replace `/mnt/<pool>` with your actual path).
Edit the volume paths in `compose.yml` to point to directories on your host where models, cache, and output crops should be stored:
```yaml
volumes:
- /your/path/to/models:/models
- /your/path/to/cache:/app/.if_cache
- /your/path/to/output:/app/frigate_train
```
These directories will be created automatically by Docker if they don't exist.
Start it:
@@ -74,15 +83,13 @@ Without `CRON_SCHEDULE`, the container runs once and exits.
---
## TrueNAS Scale
## GPU passthrough
The included `compose.yml` uses TrueNAS-style volume paths. Replace `<pool>` with your pool name:
To enable GPU acceleration, include the `deploy` block in `compose.yml` (already present in the example) and ensure the NVIDIA container toolkit is installed on your host:
```yaml
volumes:
- /mnt/tank/winnow/models:/models
- /mnt/tank/winnow/embeddings:/app/.if_cache
- /mnt/tank/winnow/output:/app/frigate_train
```bash
# Verify GPU is accessible to Docker
docker run --rm --gpus all nvidia/cuda:12.9.2-base-ubuntu22.04 nvidia-smi
```
GPU passthrough on TrueNAS requires the NVIDIA app to be installed from the TrueNAS catalog and the `deploy.resources.reservations.devices` block in `compose.yml` (already included).
CPU mode works without any GPU setup — set `FORCE_CPU=true` to disable GPU explicitly.
+2 -2
View File
@@ -4,13 +4,13 @@ export PYTHONUNBUFFERED=1
# 1. Run the job immediately on startup
echo "▶ Running on startup..."
uv run python -m winnow.cli
/app/.venv/bin/python -m winnow.cli
# 2. If a schedule exists, start the scheduler
if [ -n "${CRON_SCHEDULE:-}" ]; then
echo "▶ CRON_SCHEDULE set to: $CRON_SCHEDULE"
echo "▶ Switching to scheduled mode..."
exec uv run python3 /app/scheduler.py
exec /app/.venv/bin/python /app/scheduler.py
else
echo "▶ No schedule set, exiting."
fi
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "winnow"
version = "0.2.0"
version = "0.2.2"
description = "Immich to Frigate training sets"
license = "MIT"
requires-python = ">=3.12"
+4 -4
View File
@@ -1,9 +1,9 @@
#!/usr/bin/env python3
import os
import sys
import subprocess
import time
import logging
import os
import subprocess
import sys
import time
from pathlib import Path
try:
-2
View File
@@ -1,8 +1,6 @@
"""Smoke tests for configuration loading."""
import os
import pytest
def test_config_loads_defaults(monkeypatch):
-3
View File
@@ -1,8 +1,5 @@
"""Tests for upload tracker — mark, filter, reset, and summary logic."""
import json
import os
import tempfile
import pytest
Generated
+1 -1
View File
@@ -2289,7 +2289,7 @@ wheels = [
[[package]]
name = "winnow"
version = "0.2.0"
version = "0.2.2"
source = { editable = "." }
dependencies = [
{ name = "croniter" },
+9 -1
View File
@@ -48,7 +48,15 @@ def main() -> None:
if summary:
rprint("\n[dim]Tracker summary:[/dim]")
for person_name, counts in summary.items():
rprint(f" [dim]{person_name}: {counts['uploaded']} uploaded, {counts['rejected']} rejected[/dim]")
frigate_part = (
f", {counts['frigate_count']} in Frigate"
if counts.get("frigate_count") is not None
else ""
)
rprint(
f" [dim]{person_name}: {counts['uploaded']} uploaded,"
f" {counts['rejected']} rejected{frigate_part}[/dim]"
)
people = get_people()
if not people:
+9 -3
View File
@@ -58,6 +58,7 @@ def _enrich_asset_with_face_data(asset: dict, person: dict) -> dict:
# Inject into asset so process_face_mode can find it via asset["people"]
asset["people"] = [{"id": person_id, "faces": [face_info]}]
asset["face_confidence"] = face_data.confidence
return asset
@@ -96,8 +97,9 @@ def execute_jobs(jobs: list[dict]) -> None:
shutil.rmtree(person_dir)
os.makedirs(person_dir, exist_ok=True)
# Track filename → asset_id mapping for upload dedup
# Track filename → asset_id and filename → confidence score
asset_map: dict[str, str] = {}
score_map: dict[str, float | None] = {}
count = 0
for asset in assets:
@@ -132,11 +134,13 @@ def execute_jobs(jobs: list[dict]) -> None:
# Record which asset produced which output file
filename = f"{count}.jpg"
asset_map[filename] = asset["id"]
score_map[filename] = asset.get("face_confidence")
# Also record object-mode variant filenames
if mode == "object":
for f in sorted(os.listdir(person_dir)):
if f.startswith(f"{count}_") and f not in asset_map:
asset_map[f] = asset["id"]
score_map[f] = asset.get("face_confidence")
count += 1
else:
@@ -149,8 +153,9 @@ def execute_jobs(jobs: list[dict]) -> None:
progress.advance(job_task)
progress.advance(overall_task)
# Store asset_map on the job so upload_to_frigate can use it
# Store maps on the job so upload_to_frigate can use them
job["asset_map"] = asset_map
job["score_map"] = score_map
progress.remove_task(job_task)
@@ -230,6 +235,7 @@ def upload_to_frigate(jobs: list[dict]) -> None:
continue
asset_map = filename_to_asset_id.get(name, {})
score_map = job.get("score_map", {})
person_files = sorted(asset_map.keys())
if not person_files:
@@ -257,7 +263,7 @@ def upload_to_frigate(jobs: list[dict]) -> None:
# Mark this asset as uploaded so it's skipped on future runs
asset_id = asset_map.get(fname)
if asset_id:
mark_uploaded(asset_id, person_name=name)
mark_uploaded(asset_id, person_name=name, score=score_map.get(fname))
break
else:
+28
View File
@@ -0,0 +1,28 @@
"""Frigate API helpers for querying face training state."""
import logging
import os
import requests
logger = logging.getLogger(__name__)
def get_frigate_face_counts() -> dict[str, int] | None:
"""Return {person_name: training_image_count} from Frigate's train directory.
Returns None if FRIGATE_URL is not set or the API is unreachable, so callers
can distinguish "API unavailable" from "person has 0 images."
"""
frigate_url = os.environ.get("FRIGATE_URL", "").rstrip("/")
if not frigate_url:
return None
try:
resp = requests.get(f"{frigate_url}/api/faces", timeout=10)
resp.raise_for_status()
data = resp.json()
train = data.get("train", {})
return {name: len(files) for name, files in train.items() if isinstance(files, list)}
except Exception as e:
logger.warning(f"Could not query Frigate face counts: {e}")
return None
+33 -1
View File
@@ -11,9 +11,10 @@ from rich.table import Table
from .config import Config
from .diversity import select_diverse_assets
from .embeddings import is_embedding_available, load_embedding_model
from .frigate_api import get_frigate_face_counts
from .immich_api import fetch_all_assets, filter_recent_assets
from .logging import console
from .upload_tracker import filter_already_uploaded
from .upload_tracker import filter_already_uploaded, get_person_summary, update_frigate_count
logger = logging.getLogger(__name__)
@@ -236,6 +237,12 @@ def auto_configure(people: list[dict]) -> list[dict]:
f" ≥{min_face_count} assets (MIN_FACE_COUNT={min_face_count})"
)
frigate_counts = get_frigate_face_counts()
# Persist each count to tracker so the last known value survives Frigate downtime
if frigate_counts is not None:
for pname, count in frigate_counts.items():
update_frigate_count(pname, count)
upload_summary = get_person_summary()
jobs = []
for person in valid_people:
name = person["name"]
@@ -263,9 +270,34 @@ def auto_configure(people: list[dict]) -> list[dict]:
rprint(f" [dim]Skipping {name} (0 new images after dedup).[/dim]")
continue
# Enforce MAX_AUTO_IMAGES as a lifetime cap per person.
# Priority: live Frigate count → last cached Frigate count → local uploaded count.
person_summary = upload_summary.get(name, {})
if frigate_counts is not None:
already_uploaded = frigate_counts.get(name, 0)
else:
already_uploaded = (
person_summary.get("frigate_count")
or person_summary.get("uploaded", 0)
)
capacity = Config.MAX_AUTO_IMAGES - already_uploaded
if capacity <= 0:
rprint(
f" [dim]Skipping {name} (at lifetime cap:"
f" {already_uploaded}/{Config.MAX_AUTO_IMAGES} trained).[/dim]"
)
continue
has_embedding = is_embedding_available(entity_type)
limit, selection_mode = _resolve_strategy(strategy, has_embedding)
# Cap selection to remaining capacity
if limit == "auto":
if already_uploaded > 0:
limit = capacity # partially filled — select exactly what remains
else:
limit = min(limit, capacity)
if selection_mode == "skip":
continue
+61 -19
View File
@@ -8,6 +8,13 @@ Both are excluded from future candidate pools. To reset:
- All: delete both files
- One person: call reset_person("Name") or set RESET_PERSON=Name
- Rejects only: delete frigate_rejected_ids.json, or set RETRY_REJECTED=true
by_person schema (frigate_uploaded_ids.json):
{
"asset_ids": ["immich-id-1", ...], # all assets we attempted to upload
"scores": {"immich-id-1": 0.953}, # Immich face confidence at upload time
"frigate_count": 42 # last known Frigate training image count
}
"""
import json
@@ -55,7 +62,23 @@ def _load_flat(filename: str) -> set[str]:
return set(_load(filename).get(_flat_key(filename), []))
def _mark(filename: str, asset_id: str, person_name: str | None) -> None:
def _get_ids(entry: list | dict) -> list[str]:
"""Extract asset_ids from either the old list format or the new dict format."""
if isinstance(entry, list):
return entry
return entry.get("asset_ids", [])
def _migrate_entry(entry: list | dict) -> dict:
"""Ensure by_person entry is in the current dict format."""
if isinstance(entry, list):
return {"asset_ids": sorted(entry), "scores": {}}
entry.setdefault("asset_ids", [])
entry.setdefault("scores", {})
return entry
def _mark(filename: str, asset_id: str, person_name: str | None, score: float | None = None) -> None:
data = _load(filename)
flat_key = _flat_key(filename)
flat = set(data.get(flat_key, []))
@@ -63,9 +86,13 @@ def _mark(filename: str, asset_id: str, person_name: str | None) -> None:
data[flat_key] = sorted(flat)
if person_name:
by_person = data.setdefault("by_person", {})
person_ids = set(by_person.get(person_name, []))
person_ids.add(asset_id)
by_person[person_name] = sorted(person_ids)
entry = _migrate_entry(by_person.get(person_name, {}))
ids = set(entry["asset_ids"])
ids.add(asset_id)
entry["asset_ids"] = sorted(ids)
if score is not None:
entry["scores"][asset_id] = round(score, 4)
by_person[person_name] = entry
_save(filename, data)
@@ -79,8 +106,8 @@ def load_rejected_ids() -> set[str]:
return _load_flat(REJECT_TRACKER_FILE)
def mark_uploaded(asset_id: str, person_name: str | None = None) -> None:
_mark(UPLOAD_TRACKER_FILE, asset_id, person_name)
def mark_uploaded(asset_id: str, person_name: str | None = None, score: float | None = None) -> None:
_mark(UPLOAD_TRACKER_FILE, asset_id, person_name, score=score)
logger.debug(f"Marked {asset_id} as uploaded ({person_name})")
@@ -89,14 +116,25 @@ def mark_rejected(asset_id: str, person_name: str | None = None) -> None:
logger.debug(f"Marked {asset_id} as rejected ({person_name})")
def update_frigate_count(person_name: str, count: int) -> None:
"""Record Frigate's authoritative training image count for a person."""
data = _load(UPLOAD_TRACKER_FILE)
by_person = data.setdefault("by_person", {})
entry = _migrate_entry(by_person.get(person_name, {}))
entry["frigate_count"] = count
by_person[person_name] = entry
_save(UPLOAD_TRACKER_FILE, data)
def reset_person(person_name: str) -> None:
"""Remove all uploaded and rejected records for a given person."""
for filename in (UPLOAD_TRACKER_FILE, REJECT_TRACKER_FILE):
data = _load(filename)
flat_key = _flat_key(filename)
by_person = data.get("by_person", {})
person_ids = set(by_person.pop(person_name, []))
if person_ids:
entry = by_person.pop(person_name, None)
if entry is not None:
person_ids = set(_get_ids(entry))
flat = set(data.get(flat_key, [])) - person_ids
data[flat_key] = sorted(flat)
data["by_person"] = by_person
@@ -104,18 +142,22 @@ def reset_person(person_name: str) -> None:
logger.info(f"Reset tracking data for {person_name}")
def get_person_summary() -> dict[str, dict[str, int]]:
"""Return {person_name: {uploaded: N, rejected: N}} for display."""
uploaded_by = _load(UPLOAD_TRACKER_FILE).get("by_person", {})
rejected_by = _load(REJECT_TRACKER_FILE).get("by_person", {})
names = set(uploaded_by) | set(rejected_by)
return {
name: {
"uploaded": len(uploaded_by.get(name, [])),
"rejected": len(rejected_by.get(name, [])),
def get_person_summary() -> dict[str, dict]:
"""Return {person_name: {uploaded, rejected, frigate_count, scores}} for display/capacity."""
uploaded_data = _load(UPLOAD_TRACKER_FILE).get("by_person", {})
rejected_data = _load(REJECT_TRACKER_FILE).get("by_person", {})
names = set(uploaded_data) | set(rejected_data)
result = {}
for name in sorted(names):
u_entry = uploaded_data.get(name, {})
r_entry = rejected_data.get(name, {})
result[name] = {
"uploaded": len(_get_ids(u_entry)),
"rejected": len(_get_ids(r_entry)),
"frigate_count": u_entry.get("frigate_count") if isinstance(u_entry, dict) else None,
"scores": u_entry.get("scores", {}) if isinstance(u_entry, dict) else {},
}
for name in sorted(names)
}
return result
def filter_already_uploaded(