* refactor: rename CACHE_DIR to DATA_DIR, default path .if_cache → data CACHE_DIR held both the embedding cache and the SQLite tracker DB, making the name misleading. DATA_DIR is more accurate. - Config reads DATA_DIR first; falls back to CACHE_DIR with a deprecation warning so existing setups don't break on upgrade - Default local path: data (was .if_cache) - Docker default path: /app/data (was /app/.if_cache) - Internal references (embeddings.py, upload_tracker.py) updated to DATA_DIR - compose.yml, .env.example, README, wiki, and changelog updated - Version bumped to 0.5.1 * chore: update lockfile --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This commit is contained in:
+1
-1
@@ -34,7 +34,7 @@ STRATEGY=adaptive
|
|||||||
# ── Caching & Models ──────────────────────────────────────────────────────────
|
# ── Caching & Models ──────────────────────────────────────────────────────────
|
||||||
# FORCE_CPU=true # Disable GPU, fall back to CPU
|
# FORCE_CPU=true # Disable GPU, fall back to CPU
|
||||||
# ENABLE_CACHE=false # Disable embedding cache (default: true)
|
# ENABLE_CACHE=false # Disable embedding cache (default: true)
|
||||||
CACHE_DIR=/app/.if_cache
|
DATA_DIR=/app/data
|
||||||
INSIGHTFACE_HOME=/models/.insightface
|
INSIGHTFACE_HOME=/models/.insightface
|
||||||
|
|
||||||
# ── Tracker overrides (one-shot — remove after use) ───────────────────────────
|
# ── Tracker overrides (one-shot — remove after use) ───────────────────────────
|
||||||
|
|||||||
@@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.5.1] - 2026-06-14
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`CACHE_DIR` renamed to `DATA_DIR`**: the environment variable that sets the path for the embedding cache and SQLite tracker database is now called `DATA_DIR` (default: `data`; Docker default: `/app/data`). The old `CACHE_DIR` still works with a startup deprecation warning — rename it to `DATA_DIR` in your `.env` or `compose.yml` to silence the warning. The container-side default path changes from `/app/.if_cache` to `/app/data`; update your volume mount accordingly.
|
||||||
|
|
||||||
## [0.5.0] - 2026-06-14
|
## [0.5.0] - 2026-06-14
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ Immich library
|
|||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
2. Filter by recency (YEARS_FILTER) and skip already-uploaded
|
2. Filter by recency (YEARS_FILTER) and skip already-uploaded
|
||||||
and rejected assets (persistent tracker in CACHE_DIR)
|
and rejected assets (persistent tracker in DATA_DIR)
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
3. Quality filter — download preview thumbnails and reject:
|
3. Quality filter — download preview thumbnails and reject:
|
||||||
@@ -74,7 +74,7 @@ Immich library
|
|||||||
↳ at cap + QUALITY_REPLACEMENT=false — skip this person
|
↳ at cap + QUALITY_REPLACEMENT=false — skip this person
|
||||||
```
|
```
|
||||||
|
|
||||||
Uploaded and rejected asset IDs are persisted across runs in a SQLite database (`winnow_tracker.db` in `CACHE_DIR`). The same image is never processed twice; rejected assets are permanently skipped unless `RETRY_REJECTED=true`.
|
Uploaded and rejected asset IDs are persisted across runs in a SQLite database (`winnow_tracker.db` in `DATA_DIR`). The same image is never processed twice; rejected assets are permanently skipped unless `RETRY_REJECTED=true`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -103,7 +103,7 @@ services:
|
|||||||
- CRON_SCHEDULE=0 3 * * 0
|
- CRON_SCHEDULE=0 3 * * 0
|
||||||
volumes:
|
volumes:
|
||||||
- /path/to/models:/models # INSIGHTFACE_HOME — persists Buffalo_L model (~300 MB)
|
- /path/to/models:/models # INSIGHTFACE_HOME — persists Buffalo_L model (~300 MB)
|
||||||
- /path/to/cache:/app/.if_cache
|
- /path/to/data:/app/data
|
||||||
- /path/to/output:/app/frigate_train
|
- /path/to/output:/app/frigate_train
|
||||||
deploy:
|
deploy:
|
||||||
resources:
|
resources:
|
||||||
@@ -212,7 +212,7 @@ These defaults are tuned for Frigate's ArcFace requirements. winnow will warn on
|
|||||||
| `FORCE_CPU` | `false` | Disable GPU — fall back to CPU for all inference |
|
| `FORCE_CPU` | `false` | Disable GPU — fall back to CPU for all inference |
|
||||||
| `OPENVINO_DEVICE` | `CPU` | Intel variant only: set `GPU` to use Arc or iGPU; default runs on CPU |
|
| `OPENVINO_DEVICE` | `CPU` | Intel variant only: set `GPU` to use Arc or iGPU; default runs on CPU |
|
||||||
| `ENABLE_CACHE` | `true` | Cache computed embeddings to disk (speeds up re-runs on the same library) |
|
| `ENABLE_CACHE` | `true` | Cache computed embeddings to disk (speeds up re-runs on the same library) |
|
||||||
| `CACHE_DIR` | `.if_cache` | Path for embedding cache and upload tracker database (`winnow_tracker.db`) |
|
| `DATA_DIR` | `data` | Path for embedding cache and upload tracker database (`winnow_tracker.db`) |
|
||||||
| `INSIGHTFACE_HOME` | *(system)* | InsightFace model cache path (Buffalo_L) |
|
| `INSIGHTFACE_HOME` | *(system)* | InsightFace model cache path (Buffalo_L) |
|
||||||
|
|
||||||
### Output
|
### Output
|
||||||
|
|||||||
+2
-2
@@ -37,7 +37,7 @@ services:
|
|||||||
# - FORCE_CPU=true # Disable GPU, fall back to CPU
|
# - FORCE_CPU=true # Disable GPU, fall back to CPU
|
||||||
# - OPENVINO_DEVICE=GPU # Intel variant only: use Arc/iGPU instead of CPU (default: CPU)
|
# - OPENVINO_DEVICE=GPU # Intel variant only: use Arc/iGPU instead of CPU (default: CPU)
|
||||||
# - ENABLE_CACHE=false # Disable embedding cache (default: true)
|
# - ENABLE_CACHE=false # Disable embedding cache (default: true)
|
||||||
- CACHE_DIR=/app/.if_cache
|
- DATA_DIR=/app/data
|
||||||
- INSIGHTFACE_HOME=/models/.insightface
|
- INSIGHTFACE_HOME=/models/.insightface
|
||||||
|
|
||||||
# ── Tracker overrides (one-shot, remove after use) ────────────────────
|
# ── Tracker overrides (one-shot, remove after use) ────────────────────
|
||||||
@@ -57,7 +57,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
# Replace with absolute paths on your host, e.g. /opt/winnow/models
|
# Replace with absolute paths on your host, e.g. /opt/winnow/models
|
||||||
- /path/to/winnow/models:/models
|
- /path/to/winnow/models:/models
|
||||||
- /path/to/winnow/cache:/app/.if_cache
|
- /path/to/winnow/data:/app/data
|
||||||
- /path/to/winnow/output:/app/frigate_train
|
- /path/to/winnow/output:/app/frigate_train
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "winnow"
|
name = "winnow"
|
||||||
version = "0.5.0"
|
version = "0.5.1"
|
||||||
description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition."
|
description = "Selects diverse, high-quality photos from Immich as training data for Frigate face recognition."
|
||||||
license = "AGPL-3.0-or-later"
|
license = "AGPL-3.0-or-later"
|
||||||
requires-python = ">=3.13"
|
requires-python = ">=3.13"
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import pytest
|
|||||||
@pytest.fixture(autouse=True)
|
@pytest.fixture(autouse=True)
|
||||||
def isolated_cache(monkeypatch, tmp_path):
|
def isolated_cache(monkeypatch, tmp_path):
|
||||||
"""Point tracker at a temp directory so tests don't touch real cache files."""
|
"""Point tracker at a temp directory so tests don't touch real cache files."""
|
||||||
monkeypatch.setenv("CACHE_DIR", str(tmp_path))
|
monkeypatch.setenv("DATA_DIR", str(tmp_path))
|
||||||
from winnow.config import _Config
|
from winnow.config import _Config
|
||||||
_Config.reset()
|
_Config.reset()
|
||||||
# Also reset the SQLite connection so the next call opens the new path
|
# Also reset the SQLite connection so the next call opens the new path
|
||||||
|
|||||||
@@ -862,7 +862,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "winnow"
|
name = "winnow"
|
||||||
version = "0.5.0"
|
version = "0.5.1"
|
||||||
source = { editable = "." }
|
source = { editable = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "croniter" },
|
{ name = "croniter" },
|
||||||
|
|||||||
+12
-2
@@ -47,7 +47,7 @@ class _Config:
|
|||||||
ENABLE_FACE_ALIGNMENT: bool
|
ENABLE_FACE_ALIGNMENT: bool
|
||||||
|
|
||||||
ENABLE_CACHE: bool
|
ENABLE_CACHE: bool
|
||||||
CACHE_DIR: str
|
DATA_DIR: str
|
||||||
|
|
||||||
def __new__(cls) -> "_Config":
|
def __new__(cls) -> "_Config":
|
||||||
if cls._instance is None:
|
if cls._instance is None:
|
||||||
@@ -99,7 +99,17 @@ class _Config:
|
|||||||
self.USE_FULL_RESOLUTION = os.getenv("USE_FULL_RESOLUTION", "true").lower() in ("true", "1", "yes")
|
self.USE_FULL_RESOLUTION = os.getenv("USE_FULL_RESOLUTION", "true").lower() in ("true", "1", "yes")
|
||||||
self.ENABLE_FACE_ALIGNMENT = os.getenv("ENABLE_FACE_ALIGNMENT", "true").lower() in ("true", "1", "yes")
|
self.ENABLE_FACE_ALIGNMENT = os.getenv("ENABLE_FACE_ALIGNMENT", "true").lower() in ("true", "1", "yes")
|
||||||
self.ENABLE_CACHE = os.getenv("ENABLE_CACHE", "true").lower() in ("true", "1", "yes")
|
self.ENABLE_CACHE = os.getenv("ENABLE_CACHE", "true").lower() in ("true", "1", "yes")
|
||||||
self.CACHE_DIR = os.getenv("CACHE_DIR", ".if_cache")
|
_data_dir = os.getenv("DATA_DIR")
|
||||||
|
_cache_dir_legacy = os.getenv("CACHE_DIR")
|
||||||
|
if _data_dir:
|
||||||
|
self.DATA_DIR = _data_dir
|
||||||
|
elif _cache_dir_legacy:
|
||||||
|
logging.warning(
|
||||||
|
"CACHE_DIR is deprecated — rename it to DATA_DIR in your .env or compose.yml"
|
||||||
|
)
|
||||||
|
self.DATA_DIR = _cache_dir_legacy
|
||||||
|
else:
|
||||||
|
self.DATA_DIR = "data"
|
||||||
|
|
||||||
# Fall back to config file only when the env var is genuinely absent (None).
|
# Fall back to config file only when the env var is genuinely absent (None).
|
||||||
# An explicitly empty env var (IMMICH_URL="") takes priority over the file.
|
# An explicitly empty env var (IMMICH_URL="") takes priority over the file.
|
||||||
|
|||||||
@@ -214,7 +214,7 @@ def get_embedding(
|
|||||||
from .config import Config
|
from .config import Config
|
||||||
|
|
||||||
use_cache = Config.ENABLE_CACHE and asset_id is not None
|
use_cache = Config.ENABLE_CACHE and asset_id is not None
|
||||||
cache = get_cache(Config.CACHE_DIR) if use_cache else None
|
cache = get_cache(Config.DATA_DIR) if use_cache else None
|
||||||
|
|
||||||
if cache:
|
if cache:
|
||||||
cached = cache.get(asset_id, "insightface")
|
cached = cache.get(asset_id, "insightface")
|
||||||
|
|||||||
+10
-10
@@ -1,6 +1,6 @@
|
|||||||
"""Persistent tracker for Immich asset IDs uploaded/rejected by Frigate.
|
"""Persistent tracker for Immich asset IDs uploaded/rejected by Frigate.
|
||||||
|
|
||||||
Uses a local SQLite database (frigate_tracker.db) in CACHE_DIR.
|
Uses a local SQLite database (frigate_tracker.db) in DATA_DIR.
|
||||||
|
|
||||||
Schema
|
Schema
|
||||||
------
|
------
|
||||||
@@ -53,7 +53,7 @@ CREATE TABLE IF NOT EXISTS person_metadata (
|
|||||||
);
|
);
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# Module-level connection state — re-opened when CACHE_DIR changes (test isolation)
|
# Module-level connection state — re-opened when DATA_DIR changes (test isolation)
|
||||||
_conn: sqlite3.Connection | None = None
|
_conn: sqlite3.Connection | None = None
|
||||||
_conn_path: str | None = None
|
_conn_path: str | None = None
|
||||||
|
|
||||||
@@ -61,15 +61,15 @@ _conn_path: str | None = None
|
|||||||
def _get_conn() -> sqlite3.Connection:
|
def _get_conn() -> sqlite3.Connection:
|
||||||
"""Return (or create) the module-level SQLite connection.
|
"""Return (or create) the module-level SQLite connection.
|
||||||
|
|
||||||
Re-opens the connection when Config.CACHE_DIR has changed — this provides
|
Re-opens the connection when Config.DATA_DIR has changed — this provides
|
||||||
test isolation when the isolated_cache fixture sets a new tmp directory and
|
test isolation when the isolated_cache fixture sets a new tmp directory and
|
||||||
calls _Config.reset().
|
calls _Config.reset().
|
||||||
"""
|
"""
|
||||||
global _conn, _conn_path
|
global _conn, _conn_path
|
||||||
|
|
||||||
from .config import Config
|
from .config import Config
|
||||||
cache_dir = Config.CACHE_DIR
|
data_dir = Config.DATA_DIR
|
||||||
db_path = str(Path(cache_dir) / _DB_NAME)
|
db_path = str(Path(data_dir) / _DB_NAME)
|
||||||
|
|
||||||
if _conn is not None and _conn_path != db_path:
|
if _conn is not None and _conn_path != db_path:
|
||||||
try:
|
try:
|
||||||
@@ -79,7 +79,7 @@ def _get_conn() -> sqlite3.Connection:
|
|||||||
_conn = None
|
_conn = None
|
||||||
|
|
||||||
if _conn is None:
|
if _conn is None:
|
||||||
Path(cache_dir).mkdir(parents=True, exist_ok=True)
|
Path(data_dir).mkdir(parents=True, exist_ok=True)
|
||||||
_conn = sqlite3.connect(db_path, check_same_thread=False)
|
_conn = sqlite3.connect(db_path, check_same_thread=False)
|
||||||
_conn.row_factory = sqlite3.Row
|
_conn.row_factory = sqlite3.Row
|
||||||
_conn.execute("PRAGMA journal_mode=WAL")
|
_conn.execute("PRAGMA journal_mode=WAL")
|
||||||
@@ -87,7 +87,7 @@ def _get_conn() -> sqlite3.Connection:
|
|||||||
_conn.executescript(_DDL)
|
_conn.executescript(_DDL)
|
||||||
_conn.commit()
|
_conn.commit()
|
||||||
_conn_path = db_path
|
_conn_path = db_path
|
||||||
_maybe_migrate(cache_dir, _conn)
|
_maybe_migrate(data_dir, _conn)
|
||||||
|
|
||||||
return _conn
|
return _conn
|
||||||
|
|
||||||
@@ -96,9 +96,9 @@ def _get_conn() -> sqlite3.Connection:
|
|||||||
# JSON → SQLite migration
|
# JSON → SQLite migration
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def _maybe_migrate(cache_dir: str, conn: sqlite3.Connection) -> None:
|
def _maybe_migrate(data_dir: str, conn: sqlite3.Connection) -> None:
|
||||||
"""If the old JSON files exist and DB is empty, migrate and rename them."""
|
"""If the old JSON files exist and DB is empty, migrate and rename them."""
|
||||||
base = Path(cache_dir)
|
base = Path(data_dir)
|
||||||
upload_json = base / _UPLOAD_JSON
|
upload_json = base / _UPLOAD_JSON
|
||||||
reject_json = base / _REJECT_JSON
|
reject_json = base / _REJECT_JSON
|
||||||
|
|
||||||
@@ -110,7 +110,7 @@ def _maybe_migrate(cache_dir: str, conn: sqlite3.Connection) -> None:
|
|||||||
# (e.g. a PermissionError on the second rename would have left the first file's
|
# (e.g. a PermissionError on the second rename would have left the first file's
|
||||||
# data committed but the second file un-renamed and un-migrated).
|
# data committed but the second file un-renamed and un-migrated).
|
||||||
|
|
||||||
logger.info("Migrating JSON tracker files to SQLite in %s", cache_dir)
|
logger.info("Migrating JSON tracker files to SQLite in %s", data_dir)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
with conn:
|
with conn:
|
||||||
|
|||||||
Reference in New Issue
Block a user