Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dfaa03de47 | ||
|
|
9d5741f626 | ||
|
|
d70a246a1c | ||
|
|
51cc7032eb | ||
|
|
105099c819 | ||
|
|
0afc9386c6 | ||
|
|
a59d05e7fd | ||
|
|
d0cb2e17b1 | ||
|
|
2dc26b5a3d | ||
|
|
fe4cfac7b5 | ||
|
|
ec074fd279 | ||
|
|
6018bf7222 | ||
|
|
4eb6e3169b | ||
|
|
f5ec9a0001 | ||
|
|
7dce4a5c71 | ||
|
|
56c802a245 |
@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.4.9] - 2026-06-14
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`FRIGATE_SCORE_CEILING` is now dynamic by default**: previously defaulted to `0` (disabled). Now unset (default) enables a self-calibrating novelty gate — below-cap candidates are skipped if their pre-upload Frigate score exceeds the most-redundant tracked file's score. This catches conditions already covered by manually-added Frigate images that winnow cannot track. Set `FRIGATE_SCORE_CEILING=0` to disable entirely; set a positive value (e.g. `0.85`) for a fixed hard ceiling.
|
||||||
|
- **Quality replacement branches consolidated**: the Frigate-score and blur-score replacement paths in the upload loop shared identical structure. Merged into a single code path parameterised by score source and comparison direction.
|
||||||
|
- **`MIN_FACE_COUNT` default raised from `0` to `3`**: people with fewer than 3 tagged photos produce degenerate training sets; skipping them by default avoids noisy runs.
|
||||||
|
- **`STRATEGY=adaptive`** is the new primary name for embedding-based diversity selection; `auto` remains a silent alias for backwards compatibility.
|
||||||
|
- **`MERGE_DUPLICATE_PEOPLE` and `TRACE_CROP_SIZE`** added to the README env var table (were in the codebase but undocumented).
|
||||||
|
- **CUDA version corrected** in the image tags table (was 13.3, actual base image is 12.8.1).
|
||||||
|
|
||||||
|
## [0.4.8] - 2026-06-14
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Tracker write-through cache**: `upload_tracker` now keeps an in-memory copy of each JSON file keyed by its resolved path. All reads after the first hit the cache instead of disk; writes go to both disk and cache atomically. Cuts per-person disk I/O in the upload loop from ~90 reads to ~1, with no API or behaviour changes.
|
||||||
|
|
||||||
## [0.4.7] - 2026-06-14
|
## [0.4.7] - 2026-06-14
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
@@ -43,26 +43,34 @@ Immich library
|
|||||||
• Objects → SigLIP (Vision Transformer) → 768-dim vector
|
• Objects → SigLIP (Vision Transformer) → 768-dim vector
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
5. Diversity selection
|
5. Near-duplicate removal — greedy cosine-distance pass drops burst shots
|
||||||
|
and near-identical photos before clustering runs; the highest-quality
|
||||||
|
image from each near-duplicate group is kept
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
6. Diversity selection
|
||||||
• K-Medoids clustering → one representative per natural group
|
• K-Medoids clustering → one representative per natural group
|
||||||
• Farthest Point Sampling → fill remaining slots with maximally spread picks
|
• Farthest Point Sampling → fill remaining slots with maximally spread picks
|
||||||
• Hard example weighting — unusual angles and low-confidence detections
|
• Hard example weighting — low-confidence detections get a distance boost
|
||||||
are biased toward selection, since those are where models tend to fail
|
so unusual angles and harder looks are preferred over easy frontals
|
||||||
• Auto mode: stops when similarity to the existing set exceeds a threshold
|
• Adaptive mode: stops when the next candidate is too similar to those already
|
||||||
(20 % of median pairwise distance for faces, 10 % for objects)
|
selected (distance threshold = 20 % of median pairwise distance for
|
||||||
|
faces, 10 % for objects)
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
6. Download full-resolution originals from Immich
|
7. Download full-resolution originals from Immich
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
7. Crop and process
|
8. Crop and process
|
||||||
• Face mode: EXIF-corrected, landmark-aligned 112×112 crop (ArcFace format)
|
• Face mode: EXIF-corrected, landmark-aligned 112×112 crop (ArcFace format)
|
||||||
• Object mode: YOLOv9c detection → one crop per matched instance
|
• Object mode: YOLOv9c detection → one crop per matched instance
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
8. Deliver
|
9. Deliver
|
||||||
• Face mode: upload crops to Frigate's face registration API
|
• Face mode: upload crops to Frigate's face registration API
|
||||||
↳ below MAX_AUTO_IMAGES — upload freely
|
↳ below MAX_AUTO_IMAGES — upload, unless the novelty gate
|
||||||
|
(FRIGATE_SCORE_CEILING) determines the candidate is already
|
||||||
|
covered by the current training set
|
||||||
↳ at cap + QUALITY_REPLACEMENT=true — with Frigate scoring active,
|
↳ at cap + QUALITY_REPLACEMENT=true — with Frigate scoring active,
|
||||||
swap the most redundant tracked image (highest pre-upload recognize
|
swap the most redundant tracked image (highest pre-upload recognize
|
||||||
score) if the candidate is more novel (lower score); falling back to
|
score) if the candidate is more novel (lower score); falling back to
|
||||||
@@ -72,7 +80,7 @@ Immich library
|
|||||||
• Object mode: save crops to disk → place into your Frigate data directory
|
• Object mode: save crops to disk → place into your Frigate data directory
|
||||||
```
|
```
|
||||||
|
|
||||||
Uploaded and rejected asset IDs are persisted across runs. The same image is never processed twice; Frigate rejections are permanently skipped unless `RETRY_REJECTED=true`.
|
Uploaded and rejected asset IDs are persisted across runs. The same image is never processed twice; rejected assets are permanently skipped unless `RETRY_REJECTED=true`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -90,7 +98,7 @@ Uploaded and rejected asset IDs are persisted across runs. The same image is nev
|
|||||||
|
|
||||||
| Tag | Arch | Acceleration |
|
| Tag | Arch | Acceleration |
|
||||||
| :-- | :-- | :-- |
|
| :-- | :-- | :-- |
|
||||||
| `:latest` | amd64 + arm64 | NVIDIA CUDA 13.3 (amd64) · requires [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) |
|
| `:latest` | amd64 + arm64 | NVIDIA CUDA 12.8 (amd64) · requires [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) |
|
||||||
| `:rocm` | amd64 | AMD ROCm · pass `/dev/kfd` + `/dev/dri` |
|
| `:rocm` | amd64 | AMD ROCm · pass `/dev/kfd` + `/dev/dri` |
|
||||||
| `:intel` | amd64 | Intel Arc / iGPU via OpenVINO · pass `/dev/dri`, set `OPENVINO_DEVICE=GPU` |
|
| `:intel` | amd64 | Intel Arc / iGPU via OpenVINO · pass `/dev/dri`, set `OPENVINO_DEVICE=GPU` |
|
||||||
| `:cpu` | amd64 + arm64 | CPU only · ~2 GB smaller · no GPU required |
|
| `:cpu` | amd64 + arm64 | CPU only · ~2 GB smaller · no GPU required |
|
||||||
@@ -173,10 +181,10 @@ In scheduled mode the process (and loaded models) stays resident between runs. T
|
|||||||
| Variable | Default | Description |
|
| Variable | Default | Description |
|
||||||
| :--- | :--- | :--- |
|
| :--- | :--- | :--- |
|
||||||
| `TRAINING_MODE` | `face` | `face` — upload crops to Frigate; `object` — save crops to disk |
|
| `TRAINING_MODE` | `face` | `face` — upload crops to Frigate; `object` — save crops to disk |
|
||||||
| `STRATEGY` | `auto` | `auto` (embedding-based adaptive), `standard` (30 images), `broad` (100 images) |
|
| `STRATEGY` | `adaptive` | `adaptive` — embedding-based diversity selection, stops when candidates become redundant; `standard` — fixed 30 images; `broad` — fixed 100 images |
|
||||||
| `LIMIT` | *(unset)* | Exact image count — overrides `STRATEGY` |
|
| `LIMIT` | *(unset)* | Exact image count — overrides `STRATEGY` |
|
||||||
| `OBJECT_CLASS` | `dog` | Target class for object mode (any YOLO class: `dog`, `cat`, `car`, etc.) |
|
| `OBJECT_CLASS` | `dog` | Target class for object mode (any YOLO class: `dog`, `cat`, `car`, etc.) |
|
||||||
| `AUTO_MODE` | *(auto)* | Force non-interactive mode in a terminal; auto-detected otherwise |
|
| `AUTO_MODE` | *(auto)* | Skip interactive prompts and process all people unattended — auto-detected when no TTY is present (Docker, cron); set `true` to force in a terminal |
|
||||||
| `VERBOSE` | `false` | Enable DEBUG-level console output (log file is always DEBUG) |
|
| `VERBOSE` | `false` | Enable DEBUG-level console output (log file is always DEBUG) |
|
||||||
|
|
||||||
### People Filtering
|
### People Filtering
|
||||||
@@ -185,23 +193,31 @@ In scheduled mode the process (and loaded models) stays resident between runs. T
|
|||||||
| :--- | :--- | :--- |
|
| :--- | :--- | :--- |
|
||||||
| `ONLY_PEOPLE` | *(unset)* | Comma-separated whitelist — process only these people |
|
| `ONLY_PEOPLE` | *(unset)* | Comma-separated whitelist — process only these people |
|
||||||
| `SKIP_PEOPLE` | *(unset)* | Comma-separated list — skip these people |
|
| `SKIP_PEOPLE` | *(unset)* | Comma-separated list — skip these people |
|
||||||
| `MIN_FACE_COUNT` | `0` | Skip people with fewer than N tagged assets in Immich |
|
| `MIN_FACE_COUNT` | `3` | Skip people with fewer than N tagged assets in Immich |
|
||||||
|
| `MERGE_DUPLICATE_PEOPLE` | `false` | When Immich has duplicate entries for the same person (same face split across multiple names), merge their asset pools before processing. Without this, each duplicate group emits a warning and is skipped |
|
||||||
| `YEARS_FILTER` | `10` | Ignore images older than N years |
|
| `YEARS_FILTER` | `10` | Ignore images older than N years |
|
||||||
|
|
||||||
### Image Quality
|
### Image Quality
|
||||||
|
|
||||||
| Variable | Default | Description |
|
| Variable | Default | Description |
|
||||||
| :--- | :--- | :--- |
|
| :--- | :--- | :--- |
|
||||||
|
| `MAX_AUTO_IMAGES` | `80` | Maximum training images per person in Frigate |
|
||||||
|
| `QUALITY_REPLACEMENT` | `true` | When at cap, swap a weaker tracked image for a better candidate. With Frigate scoring active, targets the most redundant image (highest pre-upload recognize score); otherwise uses blur score. Never touches manually added Frigate files. Set `false` to skip people at cap |
|
||||||
|
|
||||||
|
#### Advanced Tuning *(calibrated — do not adjust)*
|
||||||
|
|
||||||
|
These defaults are tuned for Frigate's ArcFace requirements. winnow will warn on launch if any are set. Image quality issues caused by non-default values will not be investigated.
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| `ENABLE_FRIGATE_SCORES` | `true` | Call Frigate's recognize endpoint pre-upload to store diversity scores used for quality replacement. Adds ~200 ms per upload. Disabling also disables the below-cap novelty gate |
|
||||||
|
| `FRIGATE_SCORE_CEILING` | *(unset)* | Below-cap novelty gate. Unset: dynamic — skips candidates whose Frigate score exceeds the most-redundant tracked file's score, auto-calibrates each run. `0`: disable entirely. Positive value (e.g. `0.85`): fixed hard ceiling |
|
||||||
| `MIN_FACE_WIDTH` | `90` | Minimum face crop width in pixels |
|
| `MIN_FACE_WIDTH` | `90` | Minimum face crop width in pixels |
|
||||||
| `FACE_MARGIN` | `0.15` | Padding around bounding box crop (fraction of face size) |
|
| `FACE_MARGIN` | `0.15` | Padding around bounding box crop (fraction of face size) |
|
||||||
| `ENABLE_FACE_ALIGNMENT` | `true` | Align to ArcFace 112×112 format using facial landmarks |
|
| `ENABLE_FACE_ALIGNMENT` | `true` | Align to ArcFace 112×112 format using facial landmarks |
|
||||||
| `USE_FULL_RESOLUTION` | `true` | Download full-resolution originals rather than preview thumbnails |
|
| `USE_FULL_RESOLUTION` | `true` | Download full-resolution originals rather than preview thumbnails |
|
||||||
| `MIN_CONFIDENCE` | `0.7` | Minimum Immich face detection confidence |
|
| `MIN_CONFIDENCE` | `0.7` | Minimum Immich face detection confidence |
|
||||||
| `BLUR_THRESHOLD` | `120.0` | Laplacian variance threshold — lower accepts more blur |
|
| `BLUR_THRESHOLD` | `120.0` | Laplacian variance threshold — lower accepts more blur |
|
||||||
| `MAX_AUTO_IMAGES` | `80` | Maximum training images per person in Frigate |
|
|
||||||
| `QUALITY_REPLACEMENT` | `true` | When at cap, swap a weaker tracked image for a better candidate. With Frigate scoring active, targets the most redundant image (highest pre-upload recognize score); otherwise uses blur score. Never touches manually added Frigate files. Set `false` to skip people at cap |
|
|
||||||
| `FRIGATE_SCORE_CEILING` | `0.0` | Skip uploads whose pre-upload Frigate recognize score exceeds this value — they are already well-covered. `0` disables; requires at least one prior run to have scores |
|
|
||||||
| `ENABLE_FRIGATE_SCORES` | `true` | Call Frigate's recognize endpoint pre-upload to store diversity scores used for quality replacement. Adds ~200 ms per upload. Disable to use blur-score replacement only |
|
|
||||||
|
|
||||||
### GPU & Models
|
### GPU & Models
|
||||||
|
|
||||||
@@ -225,8 +241,9 @@ In scheduled mode the process (and loaded models) stays resident between runs. T
|
|||||||
| Variable | Default | Description |
|
| Variable | Default | Description |
|
||||||
| :--- | :--- | :--- |
|
| :--- | :--- | :--- |
|
||||||
| `DRY_RUN` | `false` | Preview selection without downloading or uploading |
|
| `DRY_RUN` | `false` | Preview selection without downloading or uploading |
|
||||||
| `RETRY_REJECTED` | `false` | Re-attempt assets previously rejected by Frigate |
|
| `RETRY_REJECTED` | `false` | Re-attempt all previously rejected assets (low-confidence skips, Frigate rejections, and other permanent exclusions) |
|
||||||
| `RESET_PERSON` | *(unset)* | Clear upload history for one person and delete their winnow-managed Frigate training files so the next run starts fresh. Manually added Frigate files are never touched |
|
| `RESET_PERSON` | *(unset)* | Set to a person's name to clear their upload history and delete their winnow-managed Frigate training files so the next run starts fresh. Set to `*` to reset all tracked people at once. Manually added Frigate files are never touched |
|
||||||
|
| `TRACE_CROP_SIZE` | *(unset)* | Debug: print all tracked crops whose width or height matches this pixel value, then exit |
|
||||||
|
|
||||||
### Scheduling
|
### Scheduling
|
||||||
|
|
||||||
@@ -247,7 +264,7 @@ uv run winnow
|
|||||||
|
|
||||||
Requires Python 3.13+ and [uv](https://astral.sh/uv). An NVIDIA, AMD, or Intel GPU is recommended — CPU mode works but embedding computation is slower.
|
Requires Python 3.13+ and [uv](https://astral.sh/uv). An NVIDIA, AMD, or Intel GPU is recommended — CPU mode works but embedding computation is slower.
|
||||||
|
|
||||||
When run with a terminal attached, winnow starts an interactive session: select which people to process and choose a strategy (auto, standard, broad, or a custom count) per person. Without a TTY — Docker, cron, or `AUTO_MODE=true` — it processes all people automatically using the configured defaults.
|
When run with a terminal attached, winnow starts an interactive session: select which people to process and choose a strategy (adaptive, standard, broad, or a custom count) per person. Without a TTY — Docker, cron, or `AUTO_MODE=true` — it processes all people unattended using the configured defaults.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "winnow"
|
name = "winnow"
|
||||||
version = "0.4.7"
|
version = "0.4.9"
|
||||||
description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition and object classification."
|
description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition and object classification."
|
||||||
license = "AGPL-3.0-or-later"
|
license = "AGPL-3.0-or-later"
|
||||||
requires-python = ">=3.13"
|
requires-python = ">=3.13"
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ def test_config_loads_defaults(monkeypatch):
|
|||||||
assert cfg.OUTPUT_DIR == "./frigate_train"
|
assert cfg.OUTPUT_DIR == "./frigate_train"
|
||||||
assert cfg.YEARS_FILTER == 10
|
assert cfg.YEARS_FILTER == 10
|
||||||
assert cfg.MIN_FACE_WIDTH == 90
|
assert cfg.MIN_FACE_WIDTH == 90
|
||||||
assert cfg.MIN_FACE_COUNT == 0
|
assert cfg.MIN_FACE_COUNT == 3
|
||||||
assert cfg.BLUR_THRESHOLD == 120.0
|
assert cfg.BLUR_THRESHOLD == 120.0
|
||||||
assert cfg.MIN_CONFIDENCE == 0.7
|
assert cfg.MIN_CONFIDENCE == 0.7
|
||||||
assert cfg.MAX_AUTO_IMAGES == 80
|
assert cfg.MAX_AUTO_IMAGES == 80
|
||||||
|
|||||||
@@ -2348,7 +2348,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "winnow"
|
name = "winnow"
|
||||||
version = "0.4.5"
|
version = "0.4.8"
|
||||||
source = { editable = "." }
|
source = { editable = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "croniter" },
|
{ name = "croniter" },
|
||||||
|
|||||||
@@ -130,6 +130,18 @@ def _handle_duplicate_people(people: list[dict]) -> list[dict]:
|
|||||||
return people
|
return people
|
||||||
|
|
||||||
|
|
||||||
|
_UNSUPPORTED_VARS = [
|
||||||
|
"ENABLE_FRIGATE_SCORES",
|
||||||
|
"FRIGATE_SCORE_CEILING",
|
||||||
|
"MIN_FACE_WIDTH",
|
||||||
|
"FACE_MARGIN",
|
||||||
|
"ENABLE_FACE_ALIGNMENT",
|
||||||
|
"USE_FULL_RESOLUTION",
|
||||||
|
"MIN_CONFIDENCE",
|
||||||
|
"BLUR_THRESHOLD",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
"""Entry point for winnow CLI."""
|
"""Entry point for winnow CLI."""
|
||||||
try:
|
try:
|
||||||
@@ -145,6 +157,17 @@ def main() -> None:
|
|||||||
[dim]Immich -> Frigate Training Data Curator[/dim]
|
[dim]Immich -> Frigate Training Data Curator[/dim]
|
||||||
""")
|
""")
|
||||||
|
|
||||||
|
set_unsupported = [v for v in _UNSUPPORTED_VARS if os.environ.get(v)]
|
||||||
|
if set_unsupported:
|
||||||
|
console.print(
|
||||||
|
f"[bold yellow]⚠ Advanced tuning vars set: "
|
||||||
|
f"{', '.join(set_unsupported)}[/bold yellow]"
|
||||||
|
)
|
||||||
|
console.print(
|
||||||
|
"[dim] These defaults are calibrated for Frigate's ArcFace requirements. "
|
||||||
|
"Image quality issues caused by non-default values will not be investigated.[/dim]\n"
|
||||||
|
)
|
||||||
|
|
||||||
ConfigManager.get().interactive_setup()
|
ConfigManager.get().interactive_setup()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
|||||||
+5
-4
@@ -31,11 +31,11 @@ class _Config:
|
|||||||
MIN_CONFIDENCE: float = 0.7
|
MIN_CONFIDENCE: float = 0.7
|
||||||
MAX_AUTO_IMAGES: int = 80
|
MAX_AUTO_IMAGES: int = 80
|
||||||
QUALITY_REPLACEMENT: bool = True
|
QUALITY_REPLACEMENT: bool = True
|
||||||
FRIGATE_SCORE_CEILING: float = 0.0
|
FRIGATE_SCORE_CEILING: float | None = None
|
||||||
ENABLE_FRIGATE_SCORES: bool = True
|
ENABLE_FRIGATE_SCORES: bool = True
|
||||||
|
|
||||||
# People filtering
|
# People filtering
|
||||||
MIN_FACE_COUNT: int = 0
|
MIN_FACE_COUNT: int = 3
|
||||||
MERGE_DUPLICATE_PEOPLE: bool = False
|
MERGE_DUPLICATE_PEOPLE: bool = False
|
||||||
|
|
||||||
# Output quality
|
# Output quality
|
||||||
@@ -60,13 +60,14 @@ class _Config:
|
|||||||
self.OUTPUT_DIR = os.getenv("OUTPUT_DIR", "./frigate_train")
|
self.OUTPUT_DIR = os.getenv("OUTPUT_DIR", "./frigate_train")
|
||||||
self.YEARS_FILTER = int(os.getenv("YEARS_FILTER", "10"))
|
self.YEARS_FILTER = int(os.getenv("YEARS_FILTER", "10"))
|
||||||
self.MIN_FACE_WIDTH = int(os.getenv("MIN_FACE_WIDTH", "90"))
|
self.MIN_FACE_WIDTH = int(os.getenv("MIN_FACE_WIDTH", "90"))
|
||||||
self.MIN_FACE_COUNT = int(os.getenv("MIN_FACE_COUNT", "0"))
|
self.MIN_FACE_COUNT = int(os.getenv("MIN_FACE_COUNT", "3"))
|
||||||
self.MERGE_DUPLICATE_PEOPLE = os.getenv("MERGE_DUPLICATE_PEOPLE", "false").lower() in ("true", "1", "yes")
|
self.MERGE_DUPLICATE_PEOPLE = os.getenv("MERGE_DUPLICATE_PEOPLE", "false").lower() in ("true", "1", "yes")
|
||||||
self.BLUR_THRESHOLD = float(os.getenv("BLUR_THRESHOLD", "120.0"))
|
self.BLUR_THRESHOLD = float(os.getenv("BLUR_THRESHOLD", "120.0"))
|
||||||
self.MIN_CONFIDENCE = float(os.getenv("MIN_CONFIDENCE", "0.7"))
|
self.MIN_CONFIDENCE = float(os.getenv("MIN_CONFIDENCE", "0.7"))
|
||||||
self.MAX_AUTO_IMAGES = int(os.getenv("MAX_AUTO_IMAGES", "80"))
|
self.MAX_AUTO_IMAGES = int(os.getenv("MAX_AUTO_IMAGES", "80"))
|
||||||
self.QUALITY_REPLACEMENT = os.getenv("QUALITY_REPLACEMENT", "true").lower() in ("true", "1", "yes")
|
self.QUALITY_REPLACEMENT = os.getenv("QUALITY_REPLACEMENT", "true").lower() in ("true", "1", "yes")
|
||||||
self.FRIGATE_SCORE_CEILING = float(os.getenv("FRIGATE_SCORE_CEILING", "0.0"))
|
_ceiling_env = os.getenv("FRIGATE_SCORE_CEILING", "").strip()
|
||||||
|
self.FRIGATE_SCORE_CEILING = float(_ceiling_env) if _ceiling_env else None
|
||||||
self.ENABLE_FRIGATE_SCORES = os.getenv("ENABLE_FRIGATE_SCORES", "true").lower() in ("true", "1", "yes")
|
self.ENABLE_FRIGATE_SCORES = os.getenv("ENABLE_FRIGATE_SCORES", "true").lower() in ("true", "1", "yes")
|
||||||
self.FACE_MARGIN = float(os.getenv("FACE_MARGIN", "0.15"))
|
self.FACE_MARGIN = float(os.getenv("FACE_MARGIN", "0.15"))
|
||||||
self.USE_FULL_RESOLUTION = os.getenv("USE_FULL_RESOLUTION", "true").lower() in ("true", "1", "yes")
|
self.USE_FULL_RESOLUTION = os.getenv("USE_FULL_RESOLUTION", "true").lower() in ("true", "1", "yes")
|
||||||
|
|||||||
+61
-67
@@ -31,7 +31,6 @@ from .upload_tracker import (
|
|||||||
has_frigate_scores,
|
has_frigate_scores,
|
||||||
mark_rejected,
|
mark_rejected,
|
||||||
mark_uploaded,
|
mark_uploaded,
|
||||||
record_frigate_file,
|
|
||||||
record_frigate_files_batch,
|
record_frigate_files_batch,
|
||||||
remove_frigate_file,
|
remove_frigate_file,
|
||||||
)
|
)
|
||||||
@@ -232,6 +231,7 @@ def execute_jobs(jobs: list[dict]) -> None:
|
|||||||
f"[yellow]Skipped {asset['id']}"
|
f"[yellow]Skipped {asset['id']}"
|
||||||
f" (detection confidence {conf:.2f} < {Config.MIN_CONFIDENCE})[/yellow]"
|
f" (detection confidence {conf:.2f} < {Config.MIN_CONFIDENCE})[/yellow]"
|
||||||
)
|
)
|
||||||
|
mark_rejected(asset["id"], person_name=name)
|
||||||
progress.advance(job_task)
|
progress.advance(job_task)
|
||||||
progress.advance(overall_task)
|
progress.advance(overall_task)
|
||||||
continue
|
continue
|
||||||
@@ -483,14 +483,28 @@ def upload_to_frigate(jobs: list[dict]) -> None:
|
|||||||
if _result is not None and (_result[0] or "").casefold() == name.casefold():
|
if _result is not None and (_result[0] or "").casefold() == name.casefold():
|
||||||
pre_fscore = _result[1]
|
pre_fscore = _result[1]
|
||||||
|
|
||||||
# Ceiling check: skip if the existing training set already covers this
|
# Below-cap novelty gate: skip candidates already covered by the Frigate model,
|
||||||
# face condition well. Applies below cap only — at cap, replacement logic
|
# including conditions learned from manually-added images winnow can't track.
|
||||||
# drives the decision.
|
# pre_fscore is None on the first run (pre_run_count == 0 skips recognize_face
|
||||||
if not at_cap and Config.FRIGATE_SCORE_CEILING > 0 and pre_run_count > 0:
|
# above), so this block never fires on the first run without an extra guard.
|
||||||
if pre_fscore is not None and pre_fscore > Config.FRIGATE_SCORE_CEILING:
|
if not at_cap and pre_fscore is not None:
|
||||||
|
_ceiling = Config.FRIGATE_SCORE_CEILING
|
||||||
|
if _ceiling is None:
|
||||||
|
# Dynamic default: bar = most-redundant tracked file's Frigate score.
|
||||||
|
# Falls back to uploading freely when no tracked scores exist yet.
|
||||||
|
_bar = get_most_redundant_mapped_file(name)
|
||||||
|
_skip = _bar is not None and pre_fscore > _bar[2]
|
||||||
|
_bar_str = f"most redundant tracked {_bar[2]:.2f}" if _bar else ""
|
||||||
|
elif _ceiling == 0.0:
|
||||||
|
_skip = False # explicitly disabled
|
||||||
|
_bar_str = ""
|
||||||
|
else:
|
||||||
|
_skip = pre_fscore > _ceiling
|
||||||
|
_bar_str = f"ceiling {_ceiling:.2f}"
|
||||||
|
if _skip:
|
||||||
progress.console.print(
|
progress.console.print(
|
||||||
f" [dim]⏭ {fname}: Frigate score {pre_fscore:.2f}"
|
f" [dim]⏭ {fname}: Frigate score {pre_fscore:.2f}"
|
||||||
f" > ceiling {Config.FRIGATE_SCORE_CEILING:.2f}, already covered[/dim]"
|
f" > {_bar_str}, already covered[/dim]"
|
||||||
)
|
)
|
||||||
progress.advance(upload_task)
|
progress.advance(upload_task)
|
||||||
continue
|
continue
|
||||||
@@ -504,70 +518,50 @@ def upload_to_frigate(jobs: list[dict]) -> None:
|
|||||||
using_fscore = person_has_fscores and Config.ENABLE_FRIGATE_SCORES
|
using_fscore = person_has_fscores and Config.ENABLE_FRIGATE_SCORES
|
||||||
if using_fscore:
|
if using_fscore:
|
||||||
candidate_score = pre_fscore
|
candidate_score = pre_fscore
|
||||||
if candidate_score is None:
|
get_target = get_most_redundant_mapped_file
|
||||||
progress.console.print(
|
score_label, better_note = "frigate", " (more novel)"
|
||||||
f" [dim]⏭ {fname}: Frigate recognize unavailable, skipping replacement[/dim]"
|
no_score_msg = "Frigate recognize unavailable, skipping replacement"
|
||||||
)
|
|
||||||
progress.advance(upload_task)
|
|
||||||
continue
|
|
||||||
# Low score = more novel than the most redundant mapped file = replace
|
|
||||||
target = get_most_redundant_mapped_file(name, exclude=failed_deletes)
|
|
||||||
if target is None or candidate_score >= target[2]:
|
|
||||||
target_score_str = f"{target[2]:.3f}" if target is not None else "N/A"
|
|
||||||
progress.console.print(
|
|
||||||
f" [dim]⏭ {fname}: frigate {candidate_score:.3f} ≥ most redundant"
|
|
||||||
f" {target_score_str}, not more novel[/dim]"
|
|
||||||
)
|
|
||||||
progress.advance(upload_task)
|
|
||||||
continue
|
|
||||||
target_frigate_file, _target_asset_id, target_score = target
|
|
||||||
progress.console.print(
|
|
||||||
f" 🔄 {fname}: frigate {candidate_score:.3f} < {target_score:.3f},"
|
|
||||||
f" replacing {target_frigate_file} (more novel)"
|
|
||||||
)
|
|
||||||
if delete_frigate_person_files(name, [target_frigate_file]):
|
|
||||||
remove_frigate_file(name, target_frigate_file)
|
|
||||||
person_has_fscores = has_frigate_scores(name)
|
|
||||||
effective_count -= 1
|
|
||||||
# clear any blur-mode slot floor — Frigate uses a different score metric
|
|
||||||
min_quality_score_for_slot = None
|
|
||||||
else:
|
|
||||||
logger.warning(f"Failed to delete {target_frigate_file} for {name}, skipping replacement")
|
|
||||||
failed_deletes.add(target_frigate_file)
|
|
||||||
progress.advance(upload_task)
|
|
||||||
continue
|
|
||||||
else:
|
else:
|
||||||
candidate_score = score_map.get(fname)
|
candidate_score = score_map.get(fname)
|
||||||
if candidate_score is None:
|
get_target = get_lowest_quality_mapped_file
|
||||||
progress.console.print(
|
score_label, better_note = "blur", ""
|
||||||
f" [dim]⏭ {fname}: no quality score, skipping replacement[/dim]"
|
no_score_msg = "no quality score, skipping replacement"
|
||||||
)
|
|
||||||
progress.advance(upload_task)
|
if candidate_score is None:
|
||||||
continue
|
progress.console.print(f" [dim]⏭ {fname}: {no_score_msg}[/dim]")
|
||||||
target = get_lowest_quality_mapped_file(name, exclude=failed_deletes)
|
progress.advance(upload_task)
|
||||||
if target is None or candidate_score <= target[2]:
|
continue
|
||||||
target_score_str = f"{target[2]:.3f}" if target is not None else "N/A"
|
|
||||||
progress.console.print(
|
target = get_target(name, exclude=failed_deletes)
|
||||||
f" [dim]⏭ {fname}: blur {candidate_score:.3f} ≤ worst"
|
not_better = target is None or (
|
||||||
f" {target_score_str}, skipping[/dim]"
|
candidate_score >= target[2] if using_fscore else candidate_score <= target[2]
|
||||||
)
|
)
|
||||||
progress.advance(upload_task)
|
if not_better:
|
||||||
continue
|
target_str = f"{target[2]:.3f}" if target is not None else "N/A"
|
||||||
target_frigate_file, _target_asset_id, target_score = target
|
op = "<" if using_fscore else ">"
|
||||||
progress.console.print(
|
progress.console.print(
|
||||||
f" 🔄 {fname}: blur {candidate_score:.3f} > {target_score:.3f},"
|
f" [dim]⏭ {fname}: {score_label} {candidate_score:.3f}"
|
||||||
f" replacing {target_frigate_file}"
|
f" not {op} {target_str}, skipping[/dim]"
|
||||||
)
|
)
|
||||||
if delete_frigate_person_files(name, [target_frigate_file]):
|
progress.advance(upload_task)
|
||||||
remove_frigate_file(name, target_frigate_file)
|
continue
|
||||||
person_has_fscores = has_frigate_scores(name)
|
|
||||||
effective_count -= 1
|
target_frigate_file, _target_asset_id, target_score = target
|
||||||
min_quality_score_for_slot = score_map.get(fname)
|
op = "<" if using_fscore else ">"
|
||||||
else:
|
progress.console.print(
|
||||||
logger.warning(f"Failed to delete {target_frigate_file} for {name}, skipping replacement")
|
f" 🔄 {fname}: {score_label} {candidate_score:.3f} {op} {target_score:.3f},"
|
||||||
failed_deletes.add(target_frigate_file)
|
f" replacing {target_frigate_file}{better_note}"
|
||||||
progress.advance(upload_task)
|
)
|
||||||
continue
|
if delete_frigate_person_files(name, [target_frigate_file]):
|
||||||
|
remove_frigate_file(name, target_frigate_file)
|
||||||
|
person_has_fscores = has_frigate_scores(name)
|
||||||
|
effective_count -= 1
|
||||||
|
min_quality_score_for_slot = None if using_fscore else candidate_score
|
||||||
|
else:
|
||||||
|
logger.warning(f"Failed to delete {target_frigate_file} for {name}, skipping replacement")
|
||||||
|
failed_deletes.add(target_frigate_file)
|
||||||
|
progress.advance(upload_task)
|
||||||
|
continue
|
||||||
|
|
||||||
for attempt in range(1, max_retries + 1):
|
for attempt in range(1, max_retries + 1):
|
||||||
try:
|
try:
|
||||||
|
|||||||
+4
-3
@@ -20,7 +20,7 @@ logger = logging.getLogger(__name__)
|
|||||||
|
|
||||||
# Strategy presets: (limit, mode_name)
|
# Strategy presets: (limit, mode_name)
|
||||||
STRATEGY_PRESETS = {
|
STRATEGY_PRESETS = {
|
||||||
"1": ("auto", "Auto Diversity"),
|
"1": ("auto", "Adaptive Diversity"),
|
||||||
"2": (30, "Standard (30)"),
|
"2": (30, "Standard (30)"),
|
||||||
"3": (100, "Broad (100)"),
|
"3": (100, "Broad (100)"),
|
||||||
}
|
}
|
||||||
@@ -31,7 +31,7 @@ def _get_strategy_choice(has_embedding: bool, entity_type: str) -> tuple[int | s
|
|||||||
model_name = "InsightFace" if entity_type == "face" else "SigLIP"
|
model_name = "InsightFace" if entity_type == "face" else "SigLIP"
|
||||||
|
|
||||||
if has_embedding:
|
if has_embedding:
|
||||||
rprint(" [bold]1.[/bold] Auto (Objective Diversity) [green][Recommended][/green]")
|
rprint(" [bold]1.[/bold] Adaptive Diversity [green][Recommended][/green]")
|
||||||
rprint(" [dim]• Dynamically selects images until redundancy starts[/dim]")
|
rprint(" [dim]• Dynamically selects images until redundancy starts[/dim]")
|
||||||
rprint(" [bold]2.[/bold] Standard (30 images)")
|
rprint(" [bold]2.[/bold] Standard (30 images)")
|
||||||
rprint(" [bold]3.[/bold] Broad (100 images)")
|
rprint(" [bold]3.[/bold] Broad (100 images)")
|
||||||
@@ -77,7 +77,8 @@ def _resolve_strategy(strategy: str, has_embedding: bool) -> tuple[int | str, st
|
|||||||
return int(custom_limit), "smart"
|
return int(custom_limit), "smart"
|
||||||
|
|
||||||
strategy_map = {
|
strategy_map = {
|
||||||
"auto": ("auto", "smart"),
|
"adaptive": ("auto", "smart"),
|
||||||
|
"auto": ("auto", "smart"), # legacy alias for adaptive
|
||||||
"standard": (30, "smart"),
|
"standard": (30, "smart"),
|
||||||
"broad": (100, "smart"),
|
"broad": (100, "smart"),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -39,6 +39,11 @@ logger = logging.getLogger(__name__)
|
|||||||
UPLOAD_TRACKER_FILE = "frigate_uploaded_ids.json"
|
UPLOAD_TRACKER_FILE = "frigate_uploaded_ids.json"
|
||||||
REJECT_TRACKER_FILE = "frigate_rejected_ids.json"
|
REJECT_TRACKER_FILE = "frigate_rejected_ids.json"
|
||||||
|
|
||||||
|
# Write-through in-memory cache keyed by the resolved file path.
|
||||||
|
# Reduces per-call JSON reads from O(calls) to O(1) after the first load.
|
||||||
|
# Keyed by full path so tests with isolated tmp dirs never share entries.
|
||||||
|
_cache: dict[str, dict] = {}
|
||||||
|
|
||||||
|
|
||||||
def _tracker_path(filename: str) -> Path:
|
def _tracker_path(filename: str) -> Path:
|
||||||
try:
|
try:
|
||||||
@@ -50,18 +55,23 @@ def _tracker_path(filename: str) -> Path:
|
|||||||
|
|
||||||
def _load(filename: str) -> dict:
|
def _load(filename: str) -> dict:
|
||||||
path = _tracker_path(filename)
|
path = _tracker_path(filename)
|
||||||
if not path.exists():
|
key = str(path)
|
||||||
return {}
|
if key in _cache:
|
||||||
try:
|
return _cache[key]
|
||||||
with open(path) as f:
|
data: dict = {}
|
||||||
return json.load(f)
|
if path.exists():
|
||||||
except (json.JSONDecodeError, OSError) as e:
|
try:
|
||||||
logger.warning(f"Could not load tracker {filename}: {e}")
|
with open(path) as f:
|
||||||
return {}
|
data = json.load(f)
|
||||||
|
except (json.JSONDecodeError, OSError) as e:
|
||||||
|
logger.warning(f"Could not load tracker {filename}: {e}")
|
||||||
|
_cache[key] = data
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
def _save(filename: str, data: dict) -> None:
|
def _save(filename: str, data: dict) -> None:
|
||||||
path = _tracker_path(filename)
|
path = _tracker_path(filename)
|
||||||
|
_cache[str(path)] = data # keep cache consistent with what we write
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
with open(path, "w") as f:
|
with open(path, "w") as f:
|
||||||
json.dump(data, f, indent=2)
|
json.dump(data, f, indent=2)
|
||||||
|
|||||||
Reference in New Issue
Block a user