Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0e416176c6 | ||
|
|
e856cba6d7 | ||
|
|
e4d5f603d8 | ||
|
|
0a67699db4 | ||
|
|
f9b9cea84f | ||
|
|
4b6bce9b74 | ||
|
|
c3bf81d490 | ||
|
|
4529a63082 | ||
|
|
ec9d2ecd0f | ||
|
|
1df7c99c2e |
+1
-1
@@ -33,7 +33,7 @@ FORCE_CPU=false
|
|||||||
ENABLE_CACHE=true
|
ENABLE_CACHE=true
|
||||||
CACHE_DIR=/app/.if_cache
|
CACHE_DIR=/app/.if_cache
|
||||||
HF_HOME=/models/huggingface
|
HF_HOME=/models/huggingface
|
||||||
INSIGHTFACE_HOME=/models
|
INSIGHTFACE_HOME=/models/.insightface
|
||||||
|
|
||||||
# ── Tracker overrides (one-shot — remove after use) ───────────────────────────
|
# ── Tracker overrides (one-shot — remove after use) ───────────────────────────
|
||||||
# DRY_RUN=true # Preview selection without downloading/uploading
|
# DRY_RUN=true # Preview selection without downloading/uploading
|
||||||
|
|||||||
@@ -236,3 +236,4 @@ __marimo__/
|
|||||||
|
|
||||||
# Streamlit
|
# Streamlit
|
||||||
.streamlit/secrets.toml
|
.streamlit/secrets.toml
|
||||||
|
compose.override.yml
|
||||||
|
|||||||
+11
-3
@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.2.1] - 2026-06-12
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Container startup reinstalling packages**: `entrypoint.sh` used `uv run`, which performs a sync check on every startup and re-downloaded `ruff` and rebuilt the package each time. Replaced with direct `.venv/bin/python` calls to skip the sync entirely.
|
||||||
|
- **InsightFace double `models/` path**: `INSIGHTFACE_HOME=/models` caused InsightFace to download Buffalo_L to `/models/models/buffalo_l` (InsightFace always appends `models/` to the root). Updated default in `compose.yml` and `.env.example` to `/models/.insightface`.
|
||||||
|
- **Lint errors in CI**: unused imports in `tests/test_config.py` and `tests/test_upload_tracker.py`, unsorted imports in `scheduler.py` — all would have failed the ruff CI check.
|
||||||
|
|
||||||
## [0.2.0] - 2026-06-12
|
## [0.2.0] - 2026-06-12
|
||||||
|
|
||||||
First release of winnow. Forked from [if-curator](https://github.com/ds-sebastian/if_curator) by Sebastian and rewritten for headless Docker deployment.
|
First release of winnow. Forked from [if-curator](https://github.com/ds-sebastian/if_curator) by Sebastian and rewritten for headless Docker deployment.
|
||||||
@@ -21,14 +29,14 @@ First release of winnow. Forked from [if-curator](https://github.com/ds-sebastia
|
|||||||
|
|
||||||
**Docker and scheduling**
|
**Docker and scheduling**
|
||||||
- `Dockerfile` — multi-stage build (CUDA 12.9 on amd64, plain Ubuntu on arm64); runtime stage excludes build tools (g++, python3.12-dev, curl, gnupg)
|
- `Dockerfile` — multi-stage build (CUDA 12.9 on amd64, plain Ubuntu on arm64); runtime stage excludes build tools (g++, python3.12-dev, curl, gnupg)
|
||||||
- `compose.yml` — fully annotated with inline comments grouped by concern; TrueNAS volume paths in the example
|
- `compose.yml` — fully annotated with inline comments grouped by concern
|
||||||
- `entrypoint.sh` — runs the tool once on startup, then hands off to the scheduler if `CRON_SCHEDULE` is set
|
- `entrypoint.sh` — runs the tool once on startup, then hands off to the scheduler if `CRON_SCHEDULE` is set
|
||||||
- `scheduler.py` — in-process cron scheduler that keeps the container (and loaded models) alive between runs
|
- `scheduler.py` — in-process cron scheduler that keeps the container (and loaded models) alive between runs
|
||||||
- `CRON_SCHEDULE` env var — standard cron expression for recurring runs; unset exits after first run
|
- `CRON_SCHEDULE` env var — standard cron expression for recurring runs; unset exits after first run
|
||||||
- `.dockerignore` — keeps `.venv`, `__pycache__`, test files, and logs out of the image context
|
- `.dockerignore` — keeps `.venv`, `__pycache__`, test files, and logs out of the image context
|
||||||
- Multi-arch image: `linux/amd64` and `linux/arm64` built and merged into a single manifest on GHCR
|
- Multi-arch image: `linux/amd64` and `linux/arm64` built and merged into a single manifest on GHCR
|
||||||
- `tini` as PID 1 init process for correct signal handling
|
- `tini` as PID 1 init process for correct signal handling
|
||||||
- Non-root container user (`appuser`, uid 568) matching TrueNAS default app UID
|
- Non-root container user (`appuser`, uid 568)
|
||||||
- `HEALTHCHECK` in Dockerfile
|
- `HEALTHCHECK` in Dockerfile
|
||||||
|
|
||||||
**Object mode**
|
**Object mode**
|
||||||
@@ -73,7 +81,7 @@ First release of winnow. Forked from [if-curator](https://github.com/ds-sebastia
|
|||||||
- 24 unit tests across four modules: `test_config`, `test_immich_api`, `test_jobs`, `test_upload_tracker`
|
- 24 unit tests across four modules: `test_config`, `test_immich_api`, `test_jobs`, `test_upload_tracker`
|
||||||
|
|
||||||
**Documentation**
|
**Documentation**
|
||||||
- `docs/setup.md` — step-by-step install guide with TrueNAS and bare-Docker sections
|
- `docs/setup.md` — step-by-step install and GPU passthrough guide
|
||||||
- `docs/troubleshooting.md` — common failure modes with fixes
|
- `docs/troubleshooting.md` — common failure modes with fixes
|
||||||
- `docs/faq.md` — answers to questions new users will ask
|
- `docs/faq.md` — answers to questions new users will ask
|
||||||
- `.env.example` — copy-paste starting point with every env var and inline comments
|
- `.env.example` — copy-paste starting point with every env var and inline comments
|
||||||
|
|||||||
@@ -68,6 +68,7 @@ RUN groupadd -g 568 apps && useradd -u 568 -g apps -m -s /bin/bash appuser \
|
|||||||
&& mkdir -p /models/.insightface /models/huggingface \
|
&& mkdir -p /models/.insightface /models/huggingface \
|
||||||
&& chown -R appuser:apps /app /models
|
&& chown -R appuser:apps /app /models
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
USER appuser
|
USER appuser
|
||||||
ENV HF_HOME=/models/huggingface INSIGHTFACE_HOME=/models
|
ENV HF_HOME=/models/huggingface INSIGHTFACE_HOME=/models
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
[](https://github.com/sudolulo/winnow/actions/workflows/docker-publish.yml) [](https://github.com/sudolulo/winnow/actions/workflows/release.yml) [](https://github.com/sudolulo/winnow/actions/workflows/lint.yml) [](https://github.com/sudolulo/winnow/actions/workflows/test.yml)
|
[](https://github.com/sudolulo/winnow/actions/workflows/docker-publish.yml) [](https://github.com/sudolulo/winnow/actions/workflows/release.yml) [](https://github.com/sudolulo/winnow/actions/workflows/lint.yml) [](https://github.com/sudolulo/winnow/actions/workflows/test.yml)
|
||||||
|
[](https://immich.app) [](https://frigate.video)
|
||||||
|
|
||||||
**Docs:** [Setup Guide](docs/setup.md) · [Troubleshooting](docs/troubleshooting.md) · [FAQ](docs/faq.md)
|
**Docs:** [Setup Guide](docs/setup.md) · [Troubleshooting](docs/troubleshooting.md) · [FAQ](docs/faq.md)
|
||||||
|
|
||||||
@@ -16,6 +17,8 @@ Frigate's face recognition model (ArcFace) and object classifier are only as goo
|
|||||||
|
|
||||||
If you upload 100 photos from the same week, the model learns the lighting in your living room and the jacket you wore that month. It struggles the moment anything changes. What you actually want is a spread: different years, different lighting conditions, different angles, different contexts.
|
If you upload 100 photos from the same week, the model learns the lighting in your living room and the jacket you wore that month. It struggles the moment anything changes. What you actually want is a spread: different years, different lighting conditions, different angles, different contexts.
|
||||||
|
|
||||||
|
This is especially true for people who have never been to your property, or who visit rarely — family members, friends, anyone Frigate has never seen in person. Live detections alone will never build a reliable model for these people. Your photo library already has the data; winnow finds and delivers the right subset of it.
|
||||||
|
|
||||||
Finding that spread manually across a library of thousands of photos is not practical. `winnow` does it automatically.
|
Finding that spread manually across a library of thousands of photos is not practical. `winnow` does it automatically.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -68,6 +71,14 @@ Uploaded asset IDs are recorded so the same image is never uploaded twice, even
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Note on Crop Quality
|
||||||
|
|
||||||
|
winnow works well, but no automated pipeline is perfect. Occasionally a bad crop will slip through quality filtering — a partial face, someone in the background, a blurry frame. After a run it's worth a quick review in Frigate's face management UI to remove anything that doesn't belong.
|
||||||
|
|
||||||
|
Issues and feedback welcome via [GitHub Issues](https://github.com/sudolulo/winnow/issues).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Modes
|
## Modes
|
||||||
|
|
||||||
### Face Mode (default)
|
### Face Mode (default)
|
||||||
@@ -126,7 +137,7 @@ services:
|
|||||||
capabilities: [gpu]
|
capabilities: [gpu]
|
||||||
```
|
```
|
||||||
|
|
||||||
See [compose.yml](compose.yml) for the full annotated example including TrueNAS volume paths.
|
See [compose.yml](compose.yml) for the full annotated example.
|
||||||
|
|
||||||
### Scheduling Behaviour
|
### Scheduling Behaviour
|
||||||
|
|
||||||
@@ -227,4 +238,4 @@ Requires Python 3.12+ and [uv](https://astral.sh/uv/). An NVIDIA GPU is strongly
|
|||||||
|
|
||||||
## Attribution
|
## Attribution
|
||||||
|
|
||||||
Based on [winnow](https://github.com/ds-sebastian/if_curator) by Sebastian, licensed MIT.
|
Based on [if_curator](https://github.com/ds-sebastian/if_curator) by Sebastian, licensed MIT.
|
||||||
|
|||||||
@@ -1,8 +0,0 @@
|
|||||||
services:
|
|
||||||
winnow:
|
|
||||||
build: .
|
|
||||||
network_mode: host
|
|
||||||
volumes:
|
|
||||||
- /code/winnow/models:/models
|
|
||||||
- /code/winnow/embeddings:/app/.if_cache
|
|
||||||
- /code/winnow/output:/app/frigate_train
|
|
||||||
+5
-5
@@ -38,7 +38,7 @@ services:
|
|||||||
- ENABLE_CACHE=true
|
- ENABLE_CACHE=true
|
||||||
- CACHE_DIR=/app/.if_cache
|
- CACHE_DIR=/app/.if_cache
|
||||||
- HF_HOME=/models/huggingface
|
- HF_HOME=/models/huggingface
|
||||||
- INSIGHTFACE_HOME=/models
|
- INSIGHTFACE_HOME=/models/.insightface
|
||||||
|
|
||||||
# ── Tracker overrides (one-shot, remove after use) ────────────────────
|
# ── Tracker overrides (one-shot, remove after use) ────────────────────
|
||||||
# - DRY_RUN=true # Preview selection without downloading/uploading
|
# - DRY_RUN=true # Preview selection without downloading/uploading
|
||||||
@@ -52,10 +52,10 @@ services:
|
|||||||
# - CRON_SCHEDULE=0 3 1 * *
|
# - CRON_SCHEDULE=0 3 1 * *
|
||||||
# - CRON_SCHEDULE=*/30 * * * *
|
# - CRON_SCHEDULE=*/30 * * * *
|
||||||
volumes:
|
volumes:
|
||||||
# Replace <pool> with your TrueNAS pool name, e.g. /mnt/tank/winnow/...
|
# Replace with absolute paths on your host, e.g. /opt/winnow/models
|
||||||
- /mnt/<pool>/winnow/models:/models
|
- /path/to/winnow/models:/models
|
||||||
- /mnt/<pool>/winnow/embeddings:/app/.if_cache
|
- /path/to/winnow/cache:/app/.if_cache
|
||||||
- /mnt/<pool>/winnow/output:/app/frigate_train
|
- /path/to/winnow/output:/app/frigate_train
|
||||||
stdin_open: true
|
stdin_open: true
|
||||||
tty: true
|
tty: true
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|||||||
+1
-1
@@ -49,7 +49,7 @@ ARM builds (linux/arm64) use CPU-only — CUDA is not available on ARM.
|
|||||||
|
|
||||||
## Does it work on Unraid / Proxmox / bare Docker?
|
## Does it work on Unraid / Proxmox / bare Docker?
|
||||||
|
|
||||||
Yes — the `compose.yml` uses standard Docker volume mounts. The TrueNAS paths in the example (`/mnt/<pool>/...`) are just an example; replace them with whatever paths suit your setup.
|
Yes — the `compose.yml` uses standard Docker volume mounts. Replace the example paths with whatever absolute paths suit your setup.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+16
-9
@@ -40,7 +40,16 @@ API_KEY=your-immich-api-key
|
|||||||
FRIGATE_URL=http://192.168.1.10:5000
|
FRIGATE_URL=http://192.168.1.10:5000
|
||||||
```
|
```
|
||||||
|
|
||||||
Edit the volume paths in `compose.yml` to match your storage layout (replace `/mnt/<pool>` with your actual path).
|
Edit the volume paths in `compose.yml` to point to directories on your host where models, cache, and output crops should be stored:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
volumes:
|
||||||
|
- /your/path/to/models:/models
|
||||||
|
- /your/path/to/cache:/app/.if_cache
|
||||||
|
- /your/path/to/output:/app/frigate_train
|
||||||
|
```
|
||||||
|
|
||||||
|
These directories will be created automatically by Docker if they don't exist.
|
||||||
|
|
||||||
Start it:
|
Start it:
|
||||||
|
|
||||||
@@ -74,15 +83,13 @@ Without `CRON_SCHEDULE`, the container runs once and exits.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## TrueNAS Scale
|
## GPU passthrough
|
||||||
|
|
||||||
The included `compose.yml` uses TrueNAS-style volume paths. Replace `<pool>` with your pool name:
|
To enable GPU acceleration, include the `deploy` block in `compose.yml` (already present in the example) and ensure the NVIDIA container toolkit is installed on your host:
|
||||||
|
|
||||||
```yaml
|
```bash
|
||||||
volumes:
|
# Verify GPU is accessible to Docker
|
||||||
- /mnt/tank/winnow/models:/models
|
docker run --rm --gpus all nvidia/cuda:12.9.2-base-ubuntu22.04 nvidia-smi
|
||||||
- /mnt/tank/winnow/embeddings:/app/.if_cache
|
|
||||||
- /mnt/tank/winnow/output:/app/frigate_train
|
|
||||||
```
|
```
|
||||||
|
|
||||||
GPU passthrough on TrueNAS requires the NVIDIA app to be installed from the TrueNAS catalog and the `deploy.resources.reservations.devices` block in `compose.yml` (already included).
|
CPU mode works without any GPU setup — set `FORCE_CPU=true` to disable GPU explicitly.
|
||||||
|
|||||||
+2
-2
@@ -4,13 +4,13 @@ export PYTHONUNBUFFERED=1
|
|||||||
|
|
||||||
# 1. Run the job immediately on startup
|
# 1. Run the job immediately on startup
|
||||||
echo "▶ Running on startup..."
|
echo "▶ Running on startup..."
|
||||||
uv run python -m winnow.cli
|
/app/.venv/bin/python -m winnow.cli
|
||||||
|
|
||||||
# 2. If a schedule exists, start the scheduler
|
# 2. If a schedule exists, start the scheduler
|
||||||
if [ -n "${CRON_SCHEDULE:-}" ]; then
|
if [ -n "${CRON_SCHEDULE:-}" ]; then
|
||||||
echo "▶ CRON_SCHEDULE set to: $CRON_SCHEDULE"
|
echo "▶ CRON_SCHEDULE set to: $CRON_SCHEDULE"
|
||||||
echo "▶ Switching to scheduled mode..."
|
echo "▶ Switching to scheduled mode..."
|
||||||
exec uv run python3 /app/scheduler.py
|
exec /app/.venv/bin/python /app/scheduler.py
|
||||||
else
|
else
|
||||||
echo "▶ No schedule set, exiting."
|
echo "▶ No schedule set, exiting."
|
||||||
fi
|
fi
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "winnow"
|
name = "winnow"
|
||||||
version = "0.2.0"
|
version = "0.2.1"
|
||||||
description = "Immich to Frigate training sets"
|
description = "Immich to Frigate training sets"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
|
|||||||
+4
-4
@@ -1,9 +1,9 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import subprocess
|
|
||||||
import time
|
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -1,8 +1,6 @@
|
|||||||
"""Smoke tests for configuration loading."""
|
"""Smoke tests for configuration loading."""
|
||||||
|
|
||||||
import os
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
def test_config_loads_defaults(monkeypatch):
|
def test_config_loads_defaults(monkeypatch):
|
||||||
|
|||||||
@@ -1,8 +1,5 @@
|
|||||||
"""Tests for upload tracker — mark, filter, reset, and summary logic."""
|
"""Tests for upload tracker — mark, filter, reset, and summary logic."""
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import tempfile
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user