Depend on adscrub as a library for ad-stripping (v0.4.0)
adscrub (flan/adscrub) stays a separate, standalone product -- own repo, schema, CLI. hark adds it as a path dependency (editable, ../adscrub -- see pyproject.toml [tool.uv.sources]) and calls its functions directly rather than duplicating any code. hark's own episodes/shows/ad_segments schema was deliberately shaped to match adscrub's column-for-column, so adscrub's schema-coupled functions (pending_episodes, scan_episode, transcribe_episode, detect_pending, cut_pending, ...) work unchanged against hark's own conn. hark cli.py's chapters/transcribe/detect-ads/cut subcommands call straight into the adscrub package -- no hark-side chapters.py/transcribe.py/detect.py/cut.py exists. podcast_feed.py is hark's own file (schema-specific: show_id/feed_url naming plus token auth, none of which adscrub's own feed.py has), used by new token-gated /feed/<show_id>/<token> and /audio/<episode_id>/<token>.<ext> routes on hark web -- unauthenticated since a podcast app can't do the dashboard's cookie login, gated instead by a random per-show feed_token. 110 tests pass. Known gap, not solved: the path dependency doesn't resolve in the Docker build context yet (needs a real packaging decision -- git dependency+deploy key, vendored wheel, or a multi-repo build script) -- documented in the Dockerfile, README, and docs/PLAN.md rather than papered over.
This commit is contained in:
@@ -9,3 +9,5 @@ dist/
|
||||
# Local data
|
||||
hark.db
|
||||
hark.db-*
|
||||
auth.db
|
||||
data/
|
||||
|
||||
@@ -7,6 +7,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.4.0] - 2026-07-11
|
||||
|
||||
### Added
|
||||
|
||||
- Ad-stripping via `flan/adscrub`, added as a **library dependency**
|
||||
(`[tool.uv.sources]` path dependency, editable) — not a code merge. hark's
|
||||
`episodes`/`shows`/`ad_segments` schema was deliberately shaped to match
|
||||
adscrub's own, so adscrub's schema-coupled functions work unchanged against
|
||||
hark's database: `hark chapters`/`transcribe`/`detect-ads`/`cut` call
|
||||
straight into the `adscrub` package. No duplicated pipeline code exists in
|
||||
this repo.
|
||||
- `hark web` now also serves `GET /feed/<show_id>/<token>` (regenerated clean
|
||||
RSS, via hark's own `podcast_feed.py`) and `GET /audio/<episode_id>/<token>.<ext>`
|
||||
(locally-cut episodes) — unauthenticated (no cookie login, since a podcast
|
||||
app can't do that) but gated by a per-show random token. `--base-url`/
|
||||
`$HARK_BASE_URL` controls what's embedded in generated links; warns if left
|
||||
at the unreachable `localhost` default.
|
||||
- `compose.gpu.yaml`: requests the host's GPU via the `nvidia` Docker runtime;
|
||||
hark's own `gpu` extra passes through to `adscrub[gpu]`.
|
||||
|
||||
### Changed
|
||||
|
||||
- Dependency, not a merge — corrected mid-session (see below) after the wrong
|
||||
approach was initially built and pushed, then reverted.
|
||||
|
||||
### Fixed
|
||||
|
||||
- An earlier pass in this same session **fully merged** adscrub's source files
|
||||
into `src/hark/` and pushed it to `main` — the wrong architecture (the
|
||||
intent was always two separate products, with hark depending on adscrub as
|
||||
a library). Reverted via `git revert -m 1` (history-preserving, not a
|
||||
force-push/reset) once caught, then rebuilt correctly as a dependency. Also:
|
||||
this merge/revert/rebuild happened concurrently with another Claude session
|
||||
actively working in this same `~/hark` checkout (uncommitted `claims.py`
|
||||
work) — branch switches were done carefully to avoid disturbing it. See
|
||||
memory `feedback_shared_working_dir` for the general lesson.
|
||||
|
||||
## [0.3.7] - 2026-07-10
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -15,25 +15,63 @@ A homelab web service (NOT a mobile app, NOT an AntennaPod fork) that:
|
||||
similarity.
|
||||
3. **Episode scoring (later):** metric-based interestingness ratings, tiltmeter-style
|
||||
(auditable, defined metrics, calibrated against the owner's actual listening).
|
||||
4. **Ad-stripping (added 2026-07-11):** finds ad spans (chapter markers, or Whisper +
|
||||
LLM classification) and cuts them out, covering *every* subscription, not just the
|
||||
genre-curated shows #1-#3 track. **This is provided by depending on `flan/adscrub` as
|
||||
a library, not by duplicating its code** — adscrub is a separate, standalone product.
|
||||
See "Architecture decisions" below before touching anything ad-stripping-related.
|
||||
|
||||
Origin: ideas #2 and #3 in a private ideas repo (git.onetick.ninja) — read
|
||||
`~/project-ideas/README.md` for the full assessments and reasoning.
|
||||
`~/project-ideas/README.md` for the full assessments and reasoning. The ad-stripping
|
||||
feature's own origin (AntennaPod's long-open feature request, why LLM-over-transcript
|
||||
beats fingerprinting/crowdsourcing) is in adscrub's own repo history.
|
||||
|
||||
## Architecture decisions (already made — don't relitigate)
|
||||
|
||||
- Standalone service on the homelab, shaped like tiltmeter: scheduled ingest → pipeline →
|
||||
SQLite → API/UI. The owner's player stays AntennaPod.
|
||||
- **Input integration:** AntennaPod syncs subscriptions + play history to Nextcloud (gpodder
|
||||
sync app) on truenas; hark reads from that API. OPML import as fallback. (Not wired in M0.)
|
||||
sync app) on truenas; hark reads from that API. OPML import as fallback. (Not wired yet —
|
||||
see M3 in docs/PLAN.md; ad-stripping still uses the manual `feeds.txt`/`resolve` flow too.)
|
||||
- **Output integration:** hark generates custom RSS feeds (e.g. "top episodes about topics
|
||||
you like", "best of candidate shows") that get subscribed to in AntennaPod like any podcast.
|
||||
No app modification anywhere.
|
||||
you like", "best of candidate shows", ad-stripped versions of any subscription) that get
|
||||
subscribed to in AntennaPod like any podcast. No app modification anywhere.
|
||||
- Feed URLs resolve via the keyless iTunes Search API; Podcast Index API can be added later
|
||||
(needs a registered key). Episode metadata comes from plain RSS.
|
||||
- Topic extraction: LLM extraction from episode title/description, canonicalized against
|
||||
Wikidata. Transcripts/Whisper are explicitly OUT of scope until much later — these genres
|
||||
name their subject in the metadata.
|
||||
Wikidata — these genres name their subject in the metadata, so this doesn't need
|
||||
transcripts even though transcription is now available (see below).
|
||||
- Topics can belong to multiple genres (Titanic = history + disaster); never force one bucket.
|
||||
- **adscrub is a dependency, not a merge — this is deliberate and non-negotiable.**
|
||||
`flan/adscrub` is its own product: own repo, own schema, own CLI, deployable and useful
|
||||
standalone. hark depends on it (`[tool.uv.sources]` path dependency, editable — see
|
||||
pyproject.toml) and calls its functions directly. hark's `episodes`/`shows`/`ad_segments`
|
||||
schema is deliberately shaped to match adscrub's own column names *specifically so*
|
||||
adscrub's schema-coupled functions (`pending_episodes`, `scan_episode`,
|
||||
`transcribe_episode`, `detect_pending`, `cut_pending`, ...) work unchanged against hark's
|
||||
`conn` — call them from hark's cli.py directly. **Do not copy adscrub's source files into
|
||||
this repo.** That mistake was actually made once (2026-07-11), pushed to main, and had to
|
||||
be reverted via `git revert` once caught — see CHANGELOG 0.4.0. The only hark-owned
|
||||
ad-stripping code should be: the schema migration, cli.py's argparse wiring, and
|
||||
`podcast_feed.py` (genuinely schema-specific — adscrub's own feed-building code targets a
|
||||
different schema and has no token-auth concept, so it isn't reusable as-is).
|
||||
- **Whisper transcription** (via adscrub) is cached process-wide, keyed by model size —
|
||||
hark's ad-span detection and (later) M4 episode-scoring should request the *same* model
|
||||
size and run sequentially through the cron-scheduled pipeline (they do) so exactly one
|
||||
model is ever resident in VRAM. This works automatically only because hark calls
|
||||
adscrub's actual `load_model()` function, not a copy — don't undermine it by ever adding a
|
||||
second, hark-owned copy of that caching logic. GPU: `code` has a real RTX 2070 SUPER,
|
||||
Docker's `nvidia` runtime is registered; `compose.gpu.yaml` requests it, hark's own `gpu`
|
||||
extra passes through to `adscrub[gpu]`.
|
||||
- **Feed/audio route auth:** `/feed/<show_id>/<token>` and `/audio/<episode_id>/<token>.<ext>`
|
||||
are unauthenticated (no cookie login — a podcast app can't do that) but gated by a random
|
||||
per-show `feed_token` embedded in the URL, compared with `secrets.compare_digest`. Not the
|
||||
dashboard's session system, and not wide open either.
|
||||
- **Known unsolved gap:** the adscrub path dependency doesn't resolve in the Docker build
|
||||
(build context only has hark's own files). Don't quietly work around this by copying
|
||||
adscrub's source into the build context or removing the dependency — it's a real packaging
|
||||
decision (git dependency + deploy key, vendored wheel, multi-repo build script) that needs
|
||||
to actually be made, not paved over. See docs/PLAN.md open questions.
|
||||
|
||||
## Conventions
|
||||
|
||||
|
||||
+30
-6
@@ -1,9 +1,26 @@
|
||||
# hark: pipeline + web frontend in one image.
|
||||
#
|
||||
# Default command serves the login-walled web UI over the databases in
|
||||
# /app/data; every pipeline stage is available as a one-shot command, e.g.:
|
||||
# Default command serves the dashboard (login-walled) + feed/audio routes
|
||||
# (token-gated) over the databases in /app/data; every pipeline stage is
|
||||
# available as a one-shot command, e.g.:
|
||||
# docker compose run --rm hark ingest
|
||||
# docker compose run --rm hark canon
|
||||
# docker compose run --rm hark chapters
|
||||
# docker compose run --rm hark transcribe
|
||||
# docker compose run --rm hark detect-ads
|
||||
# docker compose run --rm hark cut
|
||||
#
|
||||
# KNOWN GAP, not solved here: hark depends on adscrub as a local path
|
||||
# dependency (../adscrub, editable — see pyproject.toml [tool.uv.sources]).
|
||||
# This build context only COPYs hark's own files, so `uv sync` below will
|
||||
# fail to resolve that dependency as written. Real fix is a packaging
|
||||
# decision (git dependency + deploy key, vendoring a built adscrub wheel into
|
||||
# this context, or a small multi-repo build script) — see docs/PLAN.md open
|
||||
# questions. Don't paper over this by quietly dropping the dependency.
|
||||
#
|
||||
# Build with --build-arg GPU=1 (or `docker compose -f compose.yaml -f compose.gpu.yaml
|
||||
# build`) to pull in the cuBLAS/cuDNN extra for faster-whisper's CUDA path — only
|
||||
# needed on a host that actually passes a GPU through (see CLAUDE.md).
|
||||
|
||||
FROM python:3.13-slim
|
||||
|
||||
@@ -11,24 +28,31 @@ COPY --from=ghcr.io/astral-sh/uv:0.7 /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
ENV UV_LINK_MODE=copy UV_COMPILE_BYTECODE=1
|
||||
ARG GPU=0
|
||||
|
||||
# dependency layer first: rebuilds only when the lockfile changes
|
||||
COPY pyproject.toml uv.lock README.md ./
|
||||
RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen --no-dev --no-install-project
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
if [ "$GPU" = "1" ]; then uv sync --frozen --no-dev --no-install-project --extra gpu; \
|
||||
else uv sync --frozen --no-dev --no-install-project; fi
|
||||
|
||||
COPY src ./src
|
||||
RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen --no-dev
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
if [ "$GPU" = "1" ]; then uv sync --frozen --no-dev --extra gpu; \
|
||||
else uv sync --frozen --no-dev; fi
|
||||
|
||||
ENV PATH="/app/.venv/bin:$PATH" \
|
||||
HARK_DB=/app/data/hark.db \
|
||||
HARK_AUTH_DB=/app/data/auth.db
|
||||
HARK_AUTH_DB=/app/data/auth.db \
|
||||
HARK_DATA_DIR=/app/data
|
||||
|
||||
# gosu drops from root to the unprivileged `hark` user after the entrypoint
|
||||
# fixes ownership of /app/data — Docker creates bind mounts and anonymous
|
||||
# volumes as root, which this user can't write to on its own. uid/gid 568
|
||||
# matches TrueNAS SCALE's standard "apps" account, so files land owned by
|
||||
# the same user/group as every other app on that host; harmless elsewhere.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends gosu \
|
||||
# ffmpeg is for adscrub's cut.py, called as a library (see cli.py).
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends gosu ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& groupadd --gid 568 hark \
|
||||
&& useradd --system --uid 568 --gid 568 --no-create-home hark
|
||||
|
||||
@@ -5,10 +5,19 @@ Cross-podcast topic index and discovery service for subject-per-episode genres
|
||||
real-world case/event/person they cover, so you can ask "who covered the Dyatlov
|
||||
Pass incident?" and compare treatments across shows.
|
||||
|
||||
See `docs/PLAN.md` for milestones. Current state (0.3.5, M1 complete): feed
|
||||
resolution, episode ingest, LLM topic extraction with Wikidata canonicalization,
|
||||
the cross-show topic index, and a full web UI — deployed live. M2 (discovery) is
|
||||
next.
|
||||
hark also strips ads from every subscription (not just the genre-curated shows
|
||||
above) by depending on [adscrub](https://git.onetick.ninja/flan/adscrub) — a
|
||||
separate, standalone product — as a library: `hark chapters`/`transcribe`/
|
||||
`detect-ads`/`cut` call straight into adscrub's functions, since hark's own
|
||||
`episodes`/`ad_segments` schema was deliberately shaped to match adscrub's, so
|
||||
adscrub's schema-coupled functions work unchanged against hark's database.
|
||||
Nothing here is a copy of adscrub's code — see CLAUDE.md and docs/PLAN.md for
|
||||
the integration design and why it's a dependency, not a merge.
|
||||
|
||||
See `docs/PLAN.md` for milestones. Current state (0.4.0): feed resolution,
|
||||
episode ingest, LLM topic extraction with Wikidata canonicalization, the
|
||||
cross-show topic index, a full web UI, and adscrub-backed ad-stripping —
|
||||
deployed live. M2 (discovery) is next for the topic-index side.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -22,11 +31,32 @@ uv run hark canon # retry Wikidata canonicalization for unmatched t
|
||||
uv run hark stats # counts per show
|
||||
uv run hark topics # topics ranked by cross-show coverage
|
||||
uv run hark who "dyatlov" # who covered X (label substring or Wikidata QID)
|
||||
|
||||
# ad-stripping pipeline (backed by the adscrub library) — every show, not just feeds.txt's
|
||||
uv run hark chapters # scan chapter markers for ad spans (free — no transcription)
|
||||
uv run hark transcribe # Whisper the rest
|
||||
uv run hark detect-ads # LLM ad-span classification (needs $ANTHROPIC_API_KEY)
|
||||
uv run hark cut # ffmpeg out the ad spans
|
||||
```
|
||||
|
||||
The database defaults to `./hark.db`; override with `--db` or `$HARK_DB`.
|
||||
Show names live in `feeds.txt`, one per line, `#` for comments.
|
||||
|
||||
## Setup
|
||||
|
||||
adscrub is a **path dependency** (`../adscrub`, editable — see
|
||||
`pyproject.toml`'s `[tool.uv.sources]`), so `flan/adscrub` needs to be checked
|
||||
out as a sibling of this repo before `uv sync` will resolve it:
|
||||
|
||||
```
|
||||
cd .. && git clone ssh://git@git.onetick.ninja:55214/flan/adscrub.git
|
||||
cd hark && uv sync
|
||||
```
|
||||
|
||||
This works for local development; it does **not** yet work for the Docker
|
||||
build (the build context only has hark's own files) — see the Dockerfile's
|
||||
"KNOWN GAP" comment and docs/PLAN.md's open questions. Not solved yet.
|
||||
|
||||
## Web UI
|
||||
|
||||
`hark web` serves the topic index (default `0.0.0.0:8710`): a home dashboard
|
||||
@@ -41,13 +71,27 @@ separate `auth.db` (`--auth-db` / `$HARK_AUTH_DB`) so replacing `hark.db` with
|
||||
a fresh data snapshot never logs anyone out. Set `HARK_COOKIE_SECURE=1` when
|
||||
serving behind a TLS-terminating proxy.
|
||||
|
||||
The same server also answers `GET /feed/<show_id>/<token>` (the cleaned RSS
|
||||
feed) and `GET /audio/<episode_id>/<token>.<ext>` (locally-cut episodes) —
|
||||
deliberately *not* behind the login wall, since a podcast app can't do cookie
|
||||
login. Instead each show gets a random `feed_token` (auto-generated,
|
||||
`shows.feed_token`) that has to appear in the URL; wrong or missing token is a
|
||||
404, not a redirect to `/login`. `--base-url`/`$HARK_BASE_URL` must be set to
|
||||
wherever the podcast player can actually reach this server — it's embedded in
|
||||
every generated audio link, and `web` warns if left at the unreachable
|
||||
`localhost` default.
|
||||
|
||||
In Docker: `docker compose up -d` (mounts `./data`, serves :8710); pipeline
|
||||
stages run as one-shots, e.g. `docker compose run --rm hark ingest`.
|
||||
Transcription runs CPU-only by default — see `compose.gpu.yaml` and CLAUDE.md
|
||||
for the GPU deploy path.
|
||||
|
||||
Extraction calls the Anthropic API (default model `claude-opus-4-8`; override
|
||||
with `--model` or `$HARK_MODEL`) and canonicalizes labels against Wikidata so
|
||||
aliases merge ("BTK" = "Dennis Rader"). Runs are idempotent and resumable:
|
||||
processed episodes are marked and skipped, failures are retried next run.
|
||||
Ad-span classification is a separate model default (`--model`/`$HARK_AD_MODEL`)
|
||||
since it's a differently-shaped task.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# GPU override: requests the host's RTX 2070 SUPER via the nvidia Docker
|
||||
# runtime (confirmed registered on `code` — see CLAUDE.md) and builds the
|
||||
# image with the cuBLAS/cuDNN extra (passed through from adscrub[gpu], see
|
||||
# pyproject.toml) so faster-whisper actually uses it.
|
||||
#
|
||||
# docker compose -f compose.yaml -f compose.gpu.yaml build
|
||||
# docker compose -f compose.yaml -f compose.gpu.yaml run --rm hark transcribe
|
||||
services:
|
||||
hark:
|
||||
build:
|
||||
args:
|
||||
GPU: "1"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: ["gpu"]
|
||||
+11
-2
@@ -1,10 +1,18 @@
|
||||
# Web UI + one-shot pipeline commands.
|
||||
# docker compose up -d # serve web UI :8710
|
||||
# Dashboard + feed/audio routes + one-shot pipeline commands.
|
||||
# docker compose up -d # serve :8710
|
||||
# docker compose run --rm hark ingest # poll feeds once
|
||||
# docker compose run --rm hark canon # retry Wikidata matches
|
||||
# docker compose run --rm hark chapters # chapter-sourced ad spans
|
||||
# docker compose run --rm hark transcribe # Whisper — CPU by default, see compose.gpu.yaml
|
||||
# docker compose run --rm hark detect-ads # LLM ad-span classification
|
||||
# docker compose run --rm hark cut # ffmpeg out the ad spans
|
||||
#
|
||||
# HARK_ADMIN_TOKEN bootstraps the admin login on a fresh auth db; set a real
|
||||
# password at /account afterwards (the token stops working once one exists).
|
||||
# HARK_BASE_URL must be set to wherever the podcast player can actually reach
|
||||
# this container (not localhost) — it's embedded in generated feeds' links.
|
||||
# NOTE: build doesn't actually work yet — see Dockerfile's KNOWN GAP comment
|
||||
# about the adscrub path dependency not resolving in this build context.
|
||||
services:
|
||||
hark:
|
||||
build: .
|
||||
@@ -14,6 +22,7 @@ services:
|
||||
environment:
|
||||
HARK_ADMIN_TOKEN: ${HARK_ADMIN_TOKEN:-}
|
||||
HARK_COOKIE_SECURE: ${HARK_COOKIE_SECURE:-0}
|
||||
HARK_BASE_URL: ${HARK_BASE_URL:-http://localhost:8710}
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
restart: unless-stopped
|
||||
|
||||
+68
-3
@@ -43,6 +43,53 @@ Not in the original milestone list — added mid-stream on explicit request, ahe
|
||||
- Two full audit passes (security + code-quality, then a screenshot-driven UX pass against
|
||||
the real dataset) — see CHANGELOG 0.3.1 and 0.3.5 for what each one caught.
|
||||
|
||||
## Ad-stripping via adscrub (done, 0.4.0)
|
||||
|
||||
Not in the original milestone list. `flan/adscrub` is a separate, standalone
|
||||
product (its own repo, schema, CLI, deployable alone) that does chapter-marker
|
||||
scanning, Whisper transcription, LLM ad-span classification, and ffmpeg
|
||||
cutting. hark depends on it **as a library** rather than duplicating its code
|
||||
— two products, not a merge (an earlier session actually did a full code
|
||||
merge here; it was reverted per explicit correction — see CHANGELOG 0.4.0 and
|
||||
CLAUDE.md for why, and don't repeat that mistake).
|
||||
|
||||
- **Why a dependency, not two fully separate schemas:** hark's own
|
||||
`episodes` gained `chapters_url`/`chapters_scanned_at`/`transcript_path`/
|
||||
`llm_detected_at`/`cut_path`, and `shows` gained `feed_token`; new
|
||||
`ad_segments` table. These were deliberately shaped to match adscrub's own
|
||||
schema column-for-column, so adscrub's schema-coupled functions
|
||||
(`pending_episodes`, `scan_episode`, `transcribe_episode`, `detect_pending`,
|
||||
`cut_pending`, ...) work **unchanged** against hark's own `conn` — hark's
|
||||
CLI (`chapters`/`transcribe`/`detect-ads`/`cut`) calls them directly. No
|
||||
hark-side `chapters.py`/`transcribe.py`/`detect.py`/`cut.py` files exist;
|
||||
that would just be duplicated code with its own drift risk.
|
||||
- **What's genuinely hark's own code:** the CLI wiring (cli.py), the schema
|
||||
migration, and `podcast_feed.py` (feed-building — adscrub's own `feed.py`
|
||||
targets its `feeds`/`feed_id` schema and has no token concept, so this one
|
||||
isn't reusable as-is; not worth generalizing adscrub's version for one
|
||||
consumer).
|
||||
- **Serving:** `hark web` also answers `/feed/<show_id>/<token>` and
|
||||
`/audio/<episode_id>/<token>.<ext>` — unauthenticated (a podcast app can't
|
||||
do the dashboard's cookie login) but gated by a random per-show
|
||||
`feed_token`, not wide open.
|
||||
- **One shared Whisper model:** adscrub's `transcribe.load_model()` caches
|
||||
one model process-wide, keyed by model size. Since hark's CLI calls that
|
||||
same function (not a copy), ad-span detection and future M4 episode-scoring
|
||||
share one cached instance for free — *as long as* both ask for the same
|
||||
model size and run sequentially (they do — cron-scheduled batch, not
|
||||
concurrent requests). A future scoring feature wanting a genuinely
|
||||
different model size would need to decide that as a real tradeoff.
|
||||
- **GPU:** `code` has a real RTX 2070 SUPER, Docker's `nvidia` runtime is
|
||||
registered; `compose.gpu.yaml` requests it, and hark's own `gpu` extra just
|
||||
passes through to `adscrub[gpu]` rather than duplicating the cuBLAS/cuDNN
|
||||
package list.
|
||||
- **Known gap, not solved:** the path dependency (`../adscrub`, editable)
|
||||
only works for local dev. The Docker build context doesn't include
|
||||
adscrub's source, so `docker compose build` doesn't actually work yet —
|
||||
needs a real packaging decision (git dependency + deploy key, vendoring a
|
||||
built wheel into the build context, or a small multi-repo build script).
|
||||
See open questions below.
|
||||
|
||||
## M2 — discovery
|
||||
|
||||
- Embedding similarity over episode topics → related shows, notable back-catalog episodes.
|
||||
@@ -52,12 +99,20 @@ Not in the original milestone list — added mid-stream on explicit request, ahe
|
||||
|
||||
- Read subscriptions/history from Nextcloud gpodder sync (truenas).
|
||||
- Generate custom RSS feeds as the recommendation delivery channel.
|
||||
- Note: this is also how new shows should reach the ad-stripping pipeline
|
||||
(currently manual via `feeds.txt`/`hark resolve`) — wiring gpodder sync in
|
||||
properly is what actually delivers "every subscription gets ad-stripped,"
|
||||
not just "every show you've typed into feeds.txt." Deliberately deferred,
|
||||
not built alongside the adscrub dependency work — a real API integration
|
||||
project of its own.
|
||||
|
||||
## M4 — episode scoring (tiltmeter-style)
|
||||
|
||||
- Defined interestingness metrics, calibration loop against owner ratings.
|
||||
- Per-topic treatment comparison (depth, sensationalism) — needs transcripts for fidelity;
|
||||
revisit Whisper here, not before.
|
||||
- Per-topic treatment comparison (depth, sensationalism) — needs transcripts for
|
||||
fidelity. Whisper is already available via the adscrub dependency (see above);
|
||||
this milestone should call `adscrub.transcribe`'s functions the same way the
|
||||
ad-stripping pipeline does, not stand up its own transcription path.
|
||||
|
||||
## Seed shows (feeds.txt)
|
||||
|
||||
@@ -69,4 +124,14 @@ Resolve their real feed URLs via iTunes Search API at runtime — do not hand-co
|
||||
|
||||
- ~~Hosting~~ Resolved 2026-07-10: private Gitea (`flan/hark`); revisit GitHub if it goes public.
|
||||
- Which LLM/provider for extraction (M1 decision).
|
||||
- GPU/Whisper feasibility on this LXC (M4 decision; CUDA device nodes may not be exposed).
|
||||
- ~~GPU/Whisper feasibility~~ Resolved 2026-07-11: `code` has a real GPU, Docker's
|
||||
`nvidia` runtime is registered — see the adscrub dependency section above.
|
||||
- **How to make the adscrub path dependency work in the Docker build.** Not solved;
|
||||
the build context currently only has hark's own files. Options: git dependency
|
||||
(needs a deploy key baked into the build, or a build secret), vendor a built
|
||||
adscrub wheel into the build context, or a small script that builds both repos
|
||||
together. Don't guess at this — it's a real infra decision.
|
||||
- `hark detect-ads` currently defaults to `claude-opus-4-8`; revisit cost vs.
|
||||
accuracy on ad-span boundaries once run against real transcripts.
|
||||
- When to actually wire the gpodder/Nextcloud subscription sync (M3) so ad-stripping
|
||||
covers real subscriptions instead of the manually-curated `feeds.txt` list.
|
||||
|
||||
+166
-2
@@ -1,4 +1,14 @@
|
||||
"""hark command line: resolve, ingest, extract, stats, topics, who."""
|
||||
"""hark command line: resolve, ingest, extract, chapters, transcribe, detect-ads,
|
||||
cut, stats, topics, who, web.
|
||||
|
||||
chapters/transcribe/detect-ads/cut call straight into the `adscrub` package
|
||||
(a separate product, depended on as a library — see pyproject.toml) rather
|
||||
than through any hark-side reimplementation: hark's episodes/ad_segments
|
||||
schema was deliberately shaped to match adscrub's own, so adscrub's
|
||||
schema-coupled functions (pending_episodes, scan_episode, transcribe_episode,
|
||||
detect_pending, cut_pending, ...) work unchanged against hark's `conn`. Only
|
||||
the CLI wiring here is hark's own code.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -9,6 +19,11 @@ from typing import Callable
|
||||
|
||||
import httpx
|
||||
|
||||
from adscrub import chapters as ad_chapters
|
||||
from adscrub import cut as ad_cut
|
||||
from adscrub import detect as ad_detect
|
||||
from adscrub import transcribe as ad_transcribe
|
||||
|
||||
from . import __version__, db, extract, ingest, pipeline, resolve, wikidata
|
||||
|
||||
DEFAULT_DB = os.environ.get("HARK_DB", "hark.db")
|
||||
@@ -78,6 +93,116 @@ def cmd_ingest(args: argparse.Namespace) -> int:
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
def cmd_chapters(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
episodes = ad_chapters.pending_episodes(conn)
|
||||
if not episodes:
|
||||
print("no episodes with an unscanned chapters_url", file=sys.stderr)
|
||||
return 1
|
||||
found = 0
|
||||
with make_client() as client:
|
||||
for ep in episodes:
|
||||
try:
|
||||
n = ad_chapters.scan_episode(conn, client, ep)
|
||||
except httpx.HTTPError as exc:
|
||||
print(f" FAIL {ep['title']}: {exc}")
|
||||
continue
|
||||
found += n
|
||||
print(f" ok {ep['title']}: {n} ad span(s) from chapters")
|
||||
print(f"found {found} chapter-sourced ad span(s) across {len(episodes)} episode(s)")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_transcribe(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
pending = ad_transcribe.pending_episodes(conn, args.limit)
|
||||
if args.dry_run:
|
||||
total_pending = len(ad_transcribe.pending_episodes(conn))
|
||||
print(f"pending episodes: {total_pending}"
|
||||
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
|
||||
return 0
|
||||
if not pending:
|
||||
print("no episodes pending transcription", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
errors = 0
|
||||
with make_client() as client:
|
||||
for ep in pending:
|
||||
try:
|
||||
path = ad_transcribe.transcribe_episode(conn, ep, client, model_size=args.model)
|
||||
except (httpx.HTTPError, OSError) as exc:
|
||||
errors += 1
|
||||
print(f" FAIL {ep['title']}: {exc}")
|
||||
continue
|
||||
print(f" ok {ep['title']} -> {path}")
|
||||
remaining = len(ad_transcribe.pending_episodes(conn))
|
||||
print(f"transcribed {len(pending) - errors} episode(s) ({errors} failed, {remaining} still pending)")
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
def cmd_detect_ads(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
pending = ad_detect.pending_episodes(conn, args.limit)
|
||||
if args.dry_run:
|
||||
total_pending = len(ad_detect.pending_episodes(conn))
|
||||
print(f"pending episodes: {total_pending}"
|
||||
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
|
||||
return 0
|
||||
if not pending:
|
||||
print("no episodes pending ad-span detection", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
import anthropic # deferred: other commands must work without a key
|
||||
|
||||
try:
|
||||
client = anthropic.Anthropic()
|
||||
except anthropic.AnthropicError as exc:
|
||||
print(f"anthropic client: {exc}", file=sys.stderr)
|
||||
print("hint: export ANTHROPIC_API_KEY first (it lives in rbw, not in a file)",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
|
||||
detector = ad_detect.ClaudeAdDetector(client, model=args.model)
|
||||
|
||||
def report(r: ad_detect.DetectResult) -> None:
|
||||
if r.error:
|
||||
print(f" FAIL {r.title}: {r.error}")
|
||||
else:
|
||||
print(f" ok {r.title}: {r.found} ad span(s) from transcript")
|
||||
|
||||
results = ad_detect.detect_pending(conn, detector, limit=args.limit, on_result=report)
|
||||
errors = sum(1 for r in results if r.error)
|
||||
remaining = len(ad_detect.pending_episodes(conn))
|
||||
print(f"detected across {len(results) - errors} episode(s) ({errors} failed, {remaining} still pending)")
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
def cmd_cut(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
pending = ad_cut.pending_episodes(conn, args.limit)
|
||||
if args.dry_run:
|
||||
total_pending = len(ad_cut.pending_episodes(conn))
|
||||
print(f"pending episodes: {total_pending}"
|
||||
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
|
||||
return 0
|
||||
if not pending:
|
||||
print("no episodes pending cutting", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
def report(r: ad_cut.CutResult) -> None:
|
||||
if r.error:
|
||||
print(f" FAIL {r.title}: {r.error}")
|
||||
else:
|
||||
print(f" ok {r.title}: removed {r.ad_seconds:.1f}s of ads")
|
||||
|
||||
with make_client() as client:
|
||||
results = ad_cut.cut_pending(conn, client, limit=args.limit, on_result=report)
|
||||
errors = sum(1 for r in results if r.error)
|
||||
remaining = len(ad_cut.pending_episodes(conn))
|
||||
print(f"cut {len(results) - errors} episode(s) ({errors} failed, {remaining} still pending)")
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
def cmd_extract(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
pending = pipeline.pending_episodes(conn, args.limit)
|
||||
@@ -224,6 +349,7 @@ def cmd_web(args: argparse.Namespace) -> int:
|
||||
bind=args.bind,
|
||||
admin_token=os.environ.get("HARK_ADMIN_TOKEN"),
|
||||
cookie_secure=os.environ.get("HARK_COOKIE_SECURE", "0") == "1",
|
||||
base_url=args.base_url,
|
||||
)
|
||||
return 0
|
||||
|
||||
@@ -236,9 +362,12 @@ def cmd_stats(args: argparse.Namespace) -> int:
|
||||
episodes = conn.execute("SELECT COUNT(*) FROM episodes").fetchone()[0]
|
||||
topics = conn.execute("SELECT COUNT(*) FROM topics").fetchone()[0]
|
||||
links = conn.execute("SELECT COUNT(*) FROM episode_topics").fetchone()[0]
|
||||
segments = conn.execute("SELECT COUNT(*) FROM ad_segments").fetchone()[0]
|
||||
cut_count = conn.execute("SELECT COUNT(*) FROM episodes WHERE cut_path IS NOT NULL").fetchone()[0]
|
||||
print(f"shows: {shows['n']} ({shows['resolved']} resolved)")
|
||||
print(f"episodes: {episodes}")
|
||||
print(f"topics: {topics} ({links} episode links)")
|
||||
print(f"ad_segments: {segments} ({cut_count} episodes cut)")
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT COALESCE(s.title, s.query) AS name, COUNT(e.id) AS n,
|
||||
@@ -273,6 +402,34 @@ def main(argv: list[str] | None = None) -> int:
|
||||
p = sub.add_parser("ingest", help="fetch resolved feeds and upsert episodes")
|
||||
p.set_defaults(func=cmd_ingest)
|
||||
|
||||
p = sub.add_parser("chapters", help="scan episodes' existing chapter markers for ad spans")
|
||||
p.set_defaults(func=cmd_chapters)
|
||||
|
||||
p = sub.add_parser(
|
||||
"transcribe", help="transcribe episodes with no chapter-sourced ad spans"
|
||||
)
|
||||
p.add_argument("--limit", type=int, help="max episodes to process this run")
|
||||
p.add_argument("--model", default=os.environ.get("HARK_WHISPER_MODEL", ad_transcribe.DEFAULT_MODEL),
|
||||
help=f"faster-whisper model size (default: $HARK_WHISPER_MODEL or "
|
||||
f"{ad_transcribe.DEFAULT_MODEL})")
|
||||
p.add_argument("--dry-run", action="store_true",
|
||||
help="only report how many episodes are pending")
|
||||
p.set_defaults(func=cmd_transcribe)
|
||||
|
||||
p = sub.add_parser("detect-ads", help="classify ad spans from transcripts with a Claude model")
|
||||
p.add_argument("--limit", type=int, help="max episodes to process this run")
|
||||
p.add_argument("--model", default=os.environ.get("HARK_AD_MODEL", ad_detect.DEFAULT_MODEL),
|
||||
help=f"Claude model id (default: $HARK_AD_MODEL or {ad_detect.DEFAULT_MODEL})")
|
||||
p.add_argument("--dry-run", action="store_true",
|
||||
help="only report how many episodes are pending")
|
||||
p.set_defaults(func=cmd_detect_ads)
|
||||
|
||||
p = sub.add_parser("cut", help="cut ad spans out of episode audio with ffmpeg")
|
||||
p.add_argument("--limit", type=int, help="max episodes to process this run")
|
||||
p.add_argument("--dry-run", action="store_true",
|
||||
help="only report how many episodes are pending")
|
||||
p.set_defaults(func=cmd_cut)
|
||||
|
||||
p = sub.add_parser("extract", help="extract episode topics with a Claude model")
|
||||
p.add_argument("--limit", type=int, help="max episodes to process this run")
|
||||
p.add_argument("--model", default=DEFAULT_MODEL,
|
||||
@@ -295,12 +452,19 @@ def main(argv: list[str] | None = None) -> int:
|
||||
p = sub.add_parser("canon", help="retry Wikidata canonicalization for unmatched topics")
|
||||
p.set_defaults(func=cmd_canon)
|
||||
|
||||
p = sub.add_parser("web", help="serve the web frontend (login-walled)")
|
||||
p = sub.add_parser(
|
||||
"web", help="serve the dashboard (login-walled) + feed/audio routes (token-gated)"
|
||||
)
|
||||
p.add_argument("--bind", default=os.environ.get("HARK_BIND", "0.0.0.0:8710"),
|
||||
help="host:port (default: $HARK_BIND or 0.0.0.0:8710)")
|
||||
p.add_argument("--auth-db", default=os.environ.get("HARK_AUTH_DB", "auth.db"),
|
||||
help="auth database path, kept separate from hark.db "
|
||||
"(default: $HARK_AUTH_DB or auth.db)")
|
||||
p.add_argument("--base-url", default=os.environ.get("HARK_BASE_URL", "http://localhost:8710"),
|
||||
help="externally-reachable URL this server is served at — embedded "
|
||||
"in generated feeds' audio links, so it must resolve from "
|
||||
"wherever the podcast player runs, not just from this host "
|
||||
"(default: $HARK_BASE_URL or http://localhost:8710)")
|
||||
p.set_defaults(func=cmd_web)
|
||||
|
||||
p = sub.add_parser("topics", help="list topics by cross-show coverage")
|
||||
|
||||
+58
-11
@@ -2,12 +2,22 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# topics / topic_genres / episode_topics are created now but only populated by
|
||||
# M1 extraction; wikidata_id, confidence, and source stay NULL until then.
|
||||
#
|
||||
# feed_token / chapters_* / transcript_path / llm_detected_at / cut_path / ad_segments
|
||||
# are hark's own ad-stripping tracking — hark uses adscrub as a library (see
|
||||
# pyproject.toml [tool.uv.sources]) for the reusable pipeline logic, but the state of
|
||||
# *which of hark's own episodes* have been scanned/transcribed/detected/cut lives here,
|
||||
# in hark's own schema, since that's tied to hark's own shows/episodes rows, not
|
||||
# adscrub's separate database. feed_token gates the unauthenticated /feed and /audio
|
||||
# routes (podcast apps can't do the dashboard's cookie login); every show gets one via
|
||||
# _backfill_feed_tokens, not just ones added after this feature.
|
||||
SCHEMA = """
|
||||
PRAGMA foreign_keys = ON;
|
||||
|
||||
@@ -20,23 +30,29 @@ CREATE TABLE IF NOT EXISTS shows (
|
||||
author TEXT,
|
||||
description TEXT,
|
||||
image_url TEXT,
|
||||
feed_token TEXT,
|
||||
last_fetched_at TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
|
||||
updated_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS episodes (
|
||||
id INTEGER PRIMARY KEY,
|
||||
show_id INTEGER NOT NULL REFERENCES shows(id) ON DELETE CASCADE,
|
||||
guid TEXT NOT NULL,
|
||||
title TEXT,
|
||||
description TEXT,
|
||||
pubdate TEXT,
|
||||
duration_seconds INTEGER,
|
||||
audio_url TEXT,
|
||||
extracted_at TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
|
||||
updated_at TEXT,
|
||||
id INTEGER PRIMARY KEY,
|
||||
show_id INTEGER NOT NULL REFERENCES shows(id) ON DELETE CASCADE,
|
||||
guid TEXT NOT NULL,
|
||||
title TEXT,
|
||||
description TEXT,
|
||||
pubdate TEXT,
|
||||
duration_seconds INTEGER,
|
||||
audio_url TEXT,
|
||||
extracted_at TEXT,
|
||||
chapters_url TEXT,
|
||||
chapters_scanned_at TEXT,
|
||||
transcript_path TEXT,
|
||||
llm_detected_at TEXT,
|
||||
cut_path TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
|
||||
updated_at TEXT,
|
||||
UNIQUE (show_id, guid)
|
||||
);
|
||||
|
||||
@@ -61,7 +77,22 @@ CREATE TABLE IF NOT EXISTS episode_topics (
|
||||
PRIMARY KEY (episode_id, topic_id)
|
||||
);
|
||||
|
||||
-- Ad spans for an episode, however they were found. Multiple sources can
|
||||
-- coexist (a chapter-sourced span later confirmed by transcript classification)
|
||||
-- — dedup/precedence is a cut-time concern (overlap-merge), not a schema one.
|
||||
CREATE TABLE IF NOT EXISTS ad_segments (
|
||||
id INTEGER PRIMARY KEY,
|
||||
episode_id INTEGER NOT NULL REFERENCES episodes(id) ON DELETE CASCADE,
|
||||
start_second REAL NOT NULL,
|
||||
end_second REAL NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
confidence REAL,
|
||||
reason TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_episodes_show_pubdate ON episodes (show_id, pubdate);
|
||||
CREATE INDEX IF NOT EXISTS idx_ad_segments_episode ON ad_segments (episode_id);
|
||||
"""
|
||||
|
||||
|
||||
@@ -69,6 +100,12 @@ CREATE INDEX IF NOT EXISTS idx_episodes_show_pubdate ON episodes (show_id, pubda
|
||||
# so they are bolted on here for databases created by older versions.
|
||||
_MIGRATIONS = (
|
||||
("episodes", "extracted_at", "ALTER TABLE episodes ADD COLUMN extracted_at TEXT"),
|
||||
("episodes", "chapters_url", "ALTER TABLE episodes ADD COLUMN chapters_url TEXT"),
|
||||
("episodes", "chapters_scanned_at", "ALTER TABLE episodes ADD COLUMN chapters_scanned_at TEXT"),
|
||||
("episodes", "transcript_path", "ALTER TABLE episodes ADD COLUMN transcript_path TEXT"),
|
||||
("episodes", "llm_detected_at", "ALTER TABLE episodes ADD COLUMN llm_detected_at TEXT"),
|
||||
("episodes", "cut_path", "ALTER TABLE episodes ADD COLUMN cut_path TEXT"),
|
||||
("shows", "feed_token", "ALTER TABLE shows ADD COLUMN feed_token TEXT"),
|
||||
)
|
||||
|
||||
|
||||
@@ -79,11 +116,21 @@ def _migrate(conn: sqlite3.Connection) -> None:
|
||||
conn.execute(ddl)
|
||||
|
||||
|
||||
def _backfill_feed_tokens(conn: sqlite3.Connection) -> None:
|
||||
"""Every show needs a feed_token to serve /feed and /audio — including shows
|
||||
that existed before this column did, not just ones added going forward."""
|
||||
for row in conn.execute("SELECT id FROM shows WHERE feed_token IS NULL"):
|
||||
conn.execute(
|
||||
"UPDATE shows SET feed_token = ? WHERE id = ?", (secrets.token_urlsafe(24), row["id"])
|
||||
)
|
||||
|
||||
|
||||
def connect(path: str | Path) -> sqlite3.Connection:
|
||||
conn = sqlite3.connect(path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript(SCHEMA)
|
||||
_migrate(conn)
|
||||
_backfill_feed_tokens(conn)
|
||||
conn.commit()
|
||||
return conn
|
||||
|
||||
|
||||
+26
-6
@@ -16,7 +16,8 @@ import httpx
|
||||
from .db import utcnow
|
||||
|
||||
# Fields compared to decide whether an existing episode row needs an update.
|
||||
_EPISODE_FIELDS = ("title", "description", "pubdate", "duration_seconds", "audio_url")
|
||||
_EPISODE_FIELDS = ("title", "description", "pubdate", "duration_seconds", "audio_url",
|
||||
"chapters_url")
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -27,6 +28,7 @@ class ParsedEpisode:
|
||||
pubdate: str | None
|
||||
duration_seconds: int | None
|
||||
audio_url: str | None
|
||||
chapters_url: str | None
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -81,6 +83,22 @@ def _pubdate(entry) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def _chapters_url(entry) -> str | None:
|
||||
"""Podcasting 2.0 <podcast:chapters url="..."/> if the feed declares it.
|
||||
|
||||
feedparser exposes elements from namespaces it doesn't recognize under a
|
||||
key built from the document's own prefix — feeds using the conventional
|
||||
"podcast" prefix land in entry["podcast_chapters"]["url"] (confirmed
|
||||
against feedparser 6.x). Kept as hark's own small helper rather than
|
||||
imported from adscrub — this is generic RSS parsing, not adscrub's
|
||||
ad-detection logic, so it doesn't belong on the dependency boundary.
|
||||
"""
|
||||
chapters = entry.get("podcast_chapters")
|
||||
if isinstance(chapters, dict):
|
||||
return chapters.get("url") or chapters.get("href")
|
||||
return None
|
||||
|
||||
|
||||
def parse_feed(content: bytes | str) -> ParsedFeed:
|
||||
parsed = feedparser.parse(content)
|
||||
episodes = []
|
||||
@@ -97,6 +115,7 @@ def parse_feed(content: bytes | str) -> ParsedFeed:
|
||||
pubdate=_pubdate(entry),
|
||||
duration_seconds=parse_duration(entry.get("itunes_duration")),
|
||||
audio_url=audio_url,
|
||||
chapters_url=_chapters_url(entry),
|
||||
)
|
||||
)
|
||||
feed = parsed.feed
|
||||
@@ -127,11 +146,12 @@ def upsert_episodes(
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO episodes
|
||||
(show_id, guid, title, description, pubdate, duration_seconds, audio_url)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
(show_id, guid, title, description, pubdate, duration_seconds,
|
||||
audio_url, chapters_url)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(show_id, ep.guid, ep.title, ep.description, ep.pubdate,
|
||||
ep.duration_seconds, ep.audio_url),
|
||||
ep.duration_seconds, ep.audio_url, ep.chapters_url),
|
||||
)
|
||||
inserted += 1
|
||||
elif any(row[field] != getattr(ep, field) for field in _EPISODE_FIELDS):
|
||||
@@ -139,11 +159,11 @@ def upsert_episodes(
|
||||
"""
|
||||
UPDATE episodes
|
||||
SET title = ?, description = ?, pubdate = ?, duration_seconds = ?,
|
||||
audio_url = ?, updated_at = ?
|
||||
audio_url = ?, chapters_url = ?, updated_at = ?
|
||||
WHERE id = ?
|
||||
""",
|
||||
(ep.title, ep.description, ep.pubdate, ep.duration_seconds,
|
||||
ep.audio_url, utcnow(), row["id"]),
|
||||
ep.audio_url, ep.chapters_url, utcnow(), row["id"]),
|
||||
)
|
||||
updated += 1
|
||||
return inserted, updated
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Build a cleaned RSS feed (feedgen) pointing at cut_path episodes.
|
||||
|
||||
This is the only integration point any podcast player needs: subscribe to
|
||||
`/feed/<show_id>/<token>` instead of the original feed URL. Episodes with a
|
||||
cut_path are served locally at `/audio/<episode_id>/<token>.<ext>`; everything
|
||||
else still points at its original audio_url unchanged — an episode nobody has
|
||||
cut (no ads found, or not processed yet) doesn't need a local copy at all.
|
||||
|
||||
The token gates both routes (see web.py) since a podcast app can't do the
|
||||
dashboard's cookie-session login — see CLAUDE.md for why this is a per-show
|
||||
token embedded in the URL rather than either fully open or a second auth
|
||||
system. This is hark's own file, not imported from adscrub: adscrub's own
|
||||
feed.py builds against its `feeds`/`feed_id` schema and has no token concept
|
||||
at all, so there's no reusable piece here beyond feedgen itself (already a
|
||||
direct hark dependency).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from feedgen.feed import FeedGenerator
|
||||
|
||||
|
||||
def _parse_pubdate(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def build_feed(conn: sqlite3.Connection, show: sqlite3.Row, base_url: str) -> bytes:
|
||||
fg = FeedGenerator()
|
||||
fg.title(show["title"] or show["query"])
|
||||
fg.link(href=show["feed_url"] or f"{base_url}/feed/{show['id']}/{show['feed_token']}",
|
||||
rel="self")
|
||||
fg.description(show["description"] or show["title"] or show["query"])
|
||||
if show["image_url"]:
|
||||
fg.image(show["image_url"])
|
||||
|
||||
episodes = conn.execute(
|
||||
"SELECT * FROM episodes WHERE show_id = ? ORDER BY pubdate DESC", (show["id"],)
|
||||
).fetchall()
|
||||
for ep in episodes:
|
||||
length = 0
|
||||
if ep["cut_path"]:
|
||||
cut_path = Path(ep["cut_path"])
|
||||
audio_url = f"{base_url}/audio/{ep['id']}/{show['feed_token']}{cut_path.suffix}"
|
||||
if cut_path.is_file():
|
||||
length = cut_path.stat().st_size
|
||||
else:
|
||||
audio_url = ep["audio_url"]
|
||||
if not audio_url:
|
||||
continue # nothing playable to link — skip rather than emit a dead enclosure
|
||||
fe = fg.add_entry()
|
||||
fe.id(ep["guid"])
|
||||
fe.title(ep["title"] or "(untitled)")
|
||||
fe.description(ep["description"] or "")
|
||||
pubdate = _parse_pubdate(ep["pubdate"])
|
||||
if pubdate:
|
||||
fe.pubDate(pubdate)
|
||||
fe.enclosure(audio_url, length, "audio/mpeg")
|
||||
|
||||
return fg.rss_str(pretty=True)
|
||||
+98
-9
@@ -1,12 +1,23 @@
|
||||
"""Web frontend: the cross-show topic index behind a login wall.
|
||||
"""Web frontend: the cross-show topic index behind a login wall, plus the
|
||||
ad-stripped podcast feed/audio routes (unauthenticated, token-gated).
|
||||
|
||||
Security model follows the influence-registry spec: the whole site is gated
|
||||
Security model follows the influence-registry spec: the dashboard is gated
|
||||
by server-side sessions carried in an HttpOnly cookie; only /login, /logout
|
||||
and /healthz are reachable unauthenticated; fail-closed — with no admin
|
||||
password and no HARK_ADMIN_TOKEN the site cannot be entered at all.
|
||||
Passwords are stretched (iterated salted SHA-256) and compared in constant
|
||||
time; changing the password revokes every session.
|
||||
|
||||
/feed/<show_id>/<token> and /audio/<episode_id>/<token>.<ext> are also
|
||||
unauthenticated, but for a different reason: a podcast app can't do the
|
||||
dashboard's cookie login. They're gated instead by a per-show random token
|
||||
(shows.feed_token, compared with secrets.compare_digest) embedded in the URL
|
||||
— same idea as the tokened private-feed URLs most self-hosted podcast tools
|
||||
use, not a second login system. The RSS itself is built by podcast_feed.py
|
||||
(hark's own — adscrub's own feed-building code targets a different schema
|
||||
and has no token concept); the ad-detection/cutting pipeline that populates
|
||||
cut_path comes from the adscrub package (see cli.py, pyproject.toml).
|
||||
|
||||
Auth state lives in its own SQLite file (auth.db), NOT in hark.db — data
|
||||
snapshots pushed from the pipeline replace hark.db wholesale and must never
|
||||
wipe accounts or sessions.
|
||||
@@ -31,7 +42,7 @@ from http import HTTPStatus
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
from . import __version__
|
||||
from . import __version__, podcast_feed
|
||||
from .extract import GENRES as GENRES_FILTER
|
||||
|
||||
PW_ITERS = 120_000
|
||||
@@ -241,8 +252,12 @@ def page(title: str, body: str, user: str | None = None) -> str:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class App:
|
||||
def __init__(self, db_path: str | Path, auth: Auth, cookie_secure: bool = False):
|
||||
def __init__(
|
||||
self, db_path: str | Path, auth: Auth, cookie_secure: bool = False,
|
||||
base_url: str = "http://localhost:8710",
|
||||
):
|
||||
self.db_path = str(db_path)
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.auth = auth
|
||||
self.cookie_secure = cookie_secure
|
||||
|
||||
@@ -720,7 +735,10 @@ class Handler(BaseHTTPRequestHandler):
|
||||
|
||||
def respond(self, status: int, body: str, content_type="text/html; charset=utf-8",
|
||||
extra_headers: dict | None = None):
|
||||
data = body.encode()
|
||||
self.respond_bytes(status, body.encode(), content_type, extra_headers)
|
||||
|
||||
def respond_bytes(self, status: int, data: bytes, content_type: str,
|
||||
extra_headers: dict | None = None):
|
||||
self.send_response(status)
|
||||
self._security_headers()
|
||||
self.send_header("Content-Type", content_type)
|
||||
@@ -769,6 +787,67 @@ class Handler(BaseHTTPRequestHandler):
|
||||
user["username"],
|
||||
))
|
||||
|
||||
# -- ad-stripped podcast feed/audio (unauthenticated, token-gated) --------
|
||||
|
||||
def _plain_404(self) -> None:
|
||||
self.respond_bytes(404, b"not found", "text/plain; charset=utf-8")
|
||||
|
||||
def _serve_feed(self, route: str) -> None:
|
||||
# route == "/feed/<show_id>/<token>"
|
||||
parts = route.split("/")
|
||||
if len(parts) != 4:
|
||||
return self._plain_404()
|
||||
try:
|
||||
show_id = int(parts[2])
|
||||
except ValueError:
|
||||
return self._plain_404()
|
||||
token = parts[3]
|
||||
conn = self.app.db()
|
||||
try:
|
||||
show = conn.execute("SELECT * FROM shows WHERE id = ?", (show_id,)).fetchone()
|
||||
if show is None or not show["feed_token"] or not secrets.compare_digest(
|
||||
show["feed_token"], token
|
||||
):
|
||||
return self._plain_404()
|
||||
body = podcast_feed.build_feed(conn, show, self.app.base_url)
|
||||
finally:
|
||||
conn.close()
|
||||
return self.respond_bytes(200, body, "application/rss+xml; charset=utf-8")
|
||||
|
||||
def _serve_audio(self, route: str) -> None:
|
||||
# route == "/audio/<episode_id>/<token>.<ext>"
|
||||
parts = route.split("/", 3)
|
||||
if len(parts) != 4:
|
||||
return self._plain_404()
|
||||
try:
|
||||
episode_id = int(parts[2])
|
||||
except ValueError:
|
||||
return self._plain_404()
|
||||
token = parts[3].split(".", 1)[0]
|
||||
conn = self.app.db()
|
||||
try:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT e.cut_path, s.feed_token FROM episodes e
|
||||
JOIN shows s ON s.id = e.show_id
|
||||
WHERE e.id = ?
|
||||
""",
|
||||
(episode_id,),
|
||||
).fetchone()
|
||||
finally:
|
||||
conn.close()
|
||||
if row is None or not row["feed_token"] or not secrets.compare_digest(
|
||||
row["feed_token"], token
|
||||
):
|
||||
return self._plain_404()
|
||||
# cut_path is set by adscrub's cut.py, never user-supplied — no
|
||||
# traversal surface to guard against here, unlike a URL-derived
|
||||
# filename would need.
|
||||
cut_path = Path(row["cut_path"]) if row["cut_path"] else None
|
||||
if cut_path is None or not cut_path.is_file():
|
||||
return self._plain_404()
|
||||
return self.respond_bytes(200, cut_path.read_bytes(), "audio/mpeg")
|
||||
|
||||
# -- routing -------------------------------------------------------------
|
||||
|
||||
def do_GET(self):
|
||||
@@ -783,6 +862,10 @@ class Handler(BaseHTTPRequestHandler):
|
||||
return self.respond(200, STYLE, "text/css; charset=utf-8")
|
||||
if route == "/login":
|
||||
return self.respond(200, page("login", LOGIN_PAGE.format(err="")))
|
||||
if route.startswith("/feed/"):
|
||||
return self._serve_feed(route)
|
||||
if route.startswith("/audio/"):
|
||||
return self._serve_audio(route)
|
||||
|
||||
user = app.auth.session_user(self.cookie_token())
|
||||
if user is None:
|
||||
@@ -857,9 +940,10 @@ class Handler(BaseHTTPRequestHandler):
|
||||
|
||||
|
||||
def make_server(db_path: str | Path, auth_path: str | Path, bind: str = "0.0.0.0:8710",
|
||||
admin_token: str | None = None, cookie_secure: bool = False) -> ThreadingHTTPServer:
|
||||
admin_token: str | None = None, cookie_secure: bool = False,
|
||||
base_url: str = "http://localhost:8710") -> ThreadingHTTPServer:
|
||||
auth = Auth(auth_path, admin_token=admin_token)
|
||||
app = App(db_path, auth, cookie_secure=cookie_secure)
|
||||
app = App(db_path, auth, cookie_secure=cookie_secure, base_url=base_url)
|
||||
host, _, port = bind.rpartition(":")
|
||||
try:
|
||||
port_num = int(port)
|
||||
@@ -870,8 +954,13 @@ def make_server(db_path: str | Path, auth_path: str | Path, bind: str = "0.0.0.0
|
||||
|
||||
|
||||
def serve(db_path: str | Path, auth_path: str | Path, bind: str, admin_token: str | None,
|
||||
cookie_secure: bool) -> None:
|
||||
server = make_server(db_path, auth_path, bind, admin_token, cookie_secure)
|
||||
cookie_secure: bool, base_url: str = "http://localhost:8710") -> None:
|
||||
if "localhost" in base_url or "127.0.0.1" in base_url:
|
||||
print(f"warning: --base-url is {base_url!r} — a podcast player running "
|
||||
f"anywhere but this exact machine won't be able to reach cut audio "
|
||||
f"links embedded in the generated feed. Set --base-url/$HARK_BASE_URL "
|
||||
f"to this host's actual reachable address.")
|
||||
server = make_server(db_path, auth_path, bind, admin_token, cookie_secure, base_url)
|
||||
print(f"hark web on http://{bind} (db={db_path}, auth={auth_path})")
|
||||
if not admin_token:
|
||||
print("note: no HARK_ADMIN_TOKEN set — login is impossible until a "
|
||||
|
||||
Vendored
+3
-1
@@ -1,5 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
|
||||
xmlns:podcast="https://podcastindex.org/namespace/1.0">
|
||||
<channel>
|
||||
<title>Example Case Show</title>
|
||||
<link>https://example.com/show</link>
|
||||
@@ -15,6 +16,7 @@
|
||||
<description>An unidentified man found on Somerton Beach in 1948.</description>
|
||||
<pubDate>Wed, 01 Jan 2025 06:00:00 GMT</pubDate>
|
||||
<itunes:duration>01:02:03</itunes:duration>
|
||||
<podcast:chapters url="https://example.com/ep1-chapters.json" type="application/json+chapters"/>
|
||||
<enclosure url="https://example.com/audio/ep1.mp3" length="100" type="audio/mpeg"/>
|
||||
</item>
|
||||
<item>
|
||||
|
||||
Vendored
+3
-1
@@ -1,5 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
|
||||
xmlns:podcast="https://podcastindex.org/namespace/1.0">
|
||||
<channel>
|
||||
<title>Example Case Show</title>
|
||||
<link>https://example.com/show</link>
|
||||
@@ -23,6 +24,7 @@
|
||||
<description>An unidentified man found on Somerton Beach in 1948.</description>
|
||||
<pubDate>Wed, 01 Jan 2025 06:00:00 GMT</pubDate>
|
||||
<itunes:duration>01:02:03</itunes:duration>
|
||||
<podcast:chapters url="https://example.com/ep1-chapters.json" type="application/json+chapters"/>
|
||||
<enclosure url="https://example.com/audio/ep1.mp3" length="100" type="audio/mpeg"/>
|
||||
</item>
|
||||
<item>
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
import json
|
||||
|
||||
from adscrub import cut as ad_cut
|
||||
from adscrub import detect as ad_detect
|
||||
from adscrub import transcribe as ad_transcribe
|
||||
|
||||
from hark import cli, db
|
||||
from hark.extract import NullExtractor
|
||||
|
||||
@@ -121,3 +127,167 @@ def test_topics_empty_db(tmp_path, capsys):
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), "topics"])
|
||||
assert rc == 1
|
||||
assert "hark extract" in capsys.readouterr().err
|
||||
|
||||
|
||||
# --- ad-stripping pipeline (calls straight into the adscrub package) ---
|
||||
|
||||
|
||||
def test_chapters_with_nothing_to_scan_fails(tmp_path, capsys):
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), "chapters"])
|
||||
assert rc == 1
|
||||
assert "no episodes" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_transcribe_with_nothing_pending_fails(tmp_path, capsys):
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), "transcribe"])
|
||||
assert rc == 1
|
||||
assert "no episodes pending" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_transcribe_dry_run_reports_pending(tmp_path, capsys):
|
||||
path = tmp_path / "t.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
|
||||
conn.execute("INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'ep', 'http://a/1.mp3')")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
rc = cli.main(["--db", str(path), "transcribe", "--dry-run"])
|
||||
assert rc == 0
|
||||
assert "pending episodes: 1" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_transcribe_success_path_calls_adscrub_directly(tmp_path, capsys, monkeypatch):
|
||||
path = tmp_path / "t.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
|
||||
conn.execute("INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'Ep One', 'http://a/1.mp3')")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
def fake_transcribe_episode(conn, ep, client, model_size=None):
|
||||
conn.execute("UPDATE episodes SET transcript_path = 'x.json' WHERE id = ?", (ep["id"],))
|
||||
conn.commit()
|
||||
return "x.json"
|
||||
|
||||
# patched on the adscrub module itself -- hark.cli's `ad_transcribe` name
|
||||
# is the same module object, not a copy, so this is what hark's CLI calls.
|
||||
monkeypatch.setattr(ad_transcribe, "transcribe_episode", fake_transcribe_episode)
|
||||
|
||||
rc = cli.main(["--db", str(path), "transcribe"])
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "ok Ep One -> x.json" in out
|
||||
assert "transcribed 1 episode(s) (0 failed, 0 still pending)" in out
|
||||
|
||||
|
||||
def test_detect_ads_with_nothing_pending_fails(tmp_path, capsys):
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), "detect-ads"])
|
||||
assert rc == 1
|
||||
assert "no episodes pending" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_detect_ads_dry_run_reports_pending(tmp_path, capsys):
|
||||
path = tmp_path / "t.db"
|
||||
transcript_path = tmp_path / "t.json"
|
||||
transcript_path.write_text(json.dumps([{"start": 0.0, "end": 1.0, "text": "hi"}]))
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (show_id, guid, title, transcript_path) VALUES (1, 'g1', 'ep', ?)",
|
||||
(str(transcript_path),),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
rc = cli.main(["--db", str(path), "detect-ads", "--dry-run"])
|
||||
assert rc == 0
|
||||
assert "pending episodes: 1" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_detect_ads_success_path_calls_adscrub_directly(tmp_path, capsys, monkeypatch):
|
||||
path = tmp_path / "t.db"
|
||||
transcript_path = tmp_path / "t.json"
|
||||
transcript_path.write_text(json.dumps(
|
||||
[{"start": 0.0, "end": 5.0, "text": "a"}, {"start": 5.0, "end": 8.0, "text": "ad"}]
|
||||
))
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (show_id, guid, title, transcript_path) VALUES (1, 'g1', 'Ep One', ?)",
|
||||
(str(transcript_path),),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
class FakeMessages:
|
||||
def parse(self, **kwargs):
|
||||
class Response:
|
||||
parsed_output = ad_detect._Detection(
|
||||
ad_spans=[ad_detect._Span(start_segment=1, end_segment=1, reason="ad")]
|
||||
)
|
||||
return Response()
|
||||
|
||||
class FakeAnthropic:
|
||||
def __init__(self):
|
||||
self.messages = FakeMessages()
|
||||
|
||||
import anthropic
|
||||
monkeypatch.setattr(anthropic, "Anthropic", FakeAnthropic)
|
||||
|
||||
rc = cli.main(["--db", str(path), "detect-ads"])
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "ok Ep One: 1 ad span(s) from transcript" in out
|
||||
assert "detected across 1 episode(s) (0 failed, 0 still pending)" in out
|
||||
|
||||
|
||||
def test_cut_with_nothing_pending_fails(tmp_path, capsys):
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), "cut"])
|
||||
assert rc == 1
|
||||
assert "no episodes pending cutting" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_cut_dry_run_reports_pending(tmp_path, capsys):
|
||||
path = tmp_path / "t.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'ep', 'http://a/1.mp3')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
rc = cli.main(["--db", str(path), "cut", "--dry-run"])
|
||||
assert rc == 0
|
||||
assert "pending episodes: 1" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_cut_success_path_calls_adscrub_directly(tmp_path, capsys, monkeypatch):
|
||||
path = tmp_path / "t.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'Ep One', 'http://a/1.mp3')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
def fake_cut_episode(conn, ep, client, data_dir=None):
|
||||
conn.execute("UPDATE episodes SET cut_path = 'x.mp3' WHERE id = ?", (ep["id"],))
|
||||
conn.commit()
|
||||
return "x.mp3", 5.0
|
||||
|
||||
monkeypatch.setattr(ad_cut, "cut_episode", fake_cut_episode)
|
||||
|
||||
rc = cli.main(["--db", str(path), "cut"])
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "ok Ep One: removed 5.0s of ads" in out
|
||||
assert "cut 1 episode(s) (0 failed, 0 still pending)" in out
|
||||
|
||||
+75
-2
@@ -1,3 +1,4 @@
|
||||
import re
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
@@ -10,12 +11,23 @@ def conn(tmp_path):
|
||||
return db.connect(tmp_path / "test.db")
|
||||
|
||||
|
||||
def _schema_without_column(table: str, column: str) -> str:
|
||||
"""db.SCHEMA with one column's definition line stripped — used to simulate
|
||||
a pre-migration database. Matches on the column name token, not exact
|
||||
whitespace, so it doesn't silently no-op if SCHEMA's alignment changes."""
|
||||
pattern = rf"\n\s*{re.escape(column)}\s+\S+,"
|
||||
new_schema, n = re.subn(pattern, "", db.SCHEMA, count=1)
|
||||
assert n == 1, f"{column!r} not found in SCHEMA — test fixture is stale"
|
||||
return new_schema
|
||||
|
||||
|
||||
def test_connect_creates_tables(conn):
|
||||
tables = {
|
||||
row["name"]
|
||||
for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'")
|
||||
}
|
||||
assert {"shows", "episodes", "topics", "topic_genres", "episode_topics"} <= tables
|
||||
assert {"shows", "episodes", "topics", "topic_genres", "episode_topics",
|
||||
"ad_segments"} <= tables
|
||||
|
||||
|
||||
def test_connect_is_idempotent(tmp_path):
|
||||
@@ -43,7 +55,7 @@ def test_migration_adds_extracted_at_to_old_db(tmp_path):
|
||||
"""A 0.1.0-era database (no extracted_at) gains the column on connect."""
|
||||
path = tmp_path / "old.db"
|
||||
old = sqlite3.connect(path)
|
||||
old.executescript(db.SCHEMA.replace(" extracted_at TEXT,\n", ""))
|
||||
old.executescript(_schema_without_column("episodes", "extracted_at"))
|
||||
old.execute("INSERT INTO shows (query) VALUES ('a')")
|
||||
old.execute("INSERT INTO episodes (show_id, guid) VALUES (1, 'g')")
|
||||
old.commit()
|
||||
@@ -54,6 +66,67 @@ def test_migration_adds_extracted_at_to_old_db(tmp_path):
|
||||
assert row["extracted_at"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("column", [
|
||||
"chapters_url", "chapters_scanned_at", "transcript_path", "llm_detected_at", "cut_path",
|
||||
])
|
||||
def test_migration_adds_ad_pipeline_columns_to_old_episodes(tmp_path, column):
|
||||
"""A pre-ad-pipeline database gains these columns on connect — the
|
||||
ad_segments table itself is CREATE TABLE IF NOT EXISTS so it doesn't need
|
||||
a migration, only columns bolted onto the pre-existing episodes table do."""
|
||||
path = tmp_path / "old.db"
|
||||
old = sqlite3.connect(path)
|
||||
old.executescript(_schema_without_column("episodes", column))
|
||||
old.execute("INSERT INTO shows (query) VALUES ('a')")
|
||||
old.execute("INSERT INTO episodes (show_id, guid) VALUES (1, 'g')")
|
||||
old.commit()
|
||||
old.close()
|
||||
|
||||
conn = db.connect(path)
|
||||
row = conn.execute(f"SELECT {column} FROM episodes").fetchone()
|
||||
assert row[column] is None
|
||||
|
||||
|
||||
def test_migration_adds_feed_token_to_old_shows(tmp_path):
|
||||
path = tmp_path / "old.db"
|
||||
old = sqlite3.connect(path)
|
||||
old.executescript(_schema_without_column("shows", "feed_token"))
|
||||
old.execute("INSERT INTO shows (query) VALUES ('a')")
|
||||
old.commit()
|
||||
old.close()
|
||||
|
||||
conn = db.connect(path)
|
||||
row = conn.execute("SELECT feed_token FROM shows").fetchone()
|
||||
assert row["feed_token"] is not None # backfilled, not just added-and-null
|
||||
|
||||
|
||||
def test_backfill_feed_tokens_gives_every_show_a_unique_token(tmp_path):
|
||||
# backfill runs at connect() time, same as migrations — a show inserted on
|
||||
# an already-open connection doesn't get one until the next reconnect,
|
||||
# which matches real operation (every CLI command opens a fresh connection).
|
||||
path = tmp_path / "test.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('a')")
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('b')")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
conn = db.connect(path)
|
||||
tokens = [row["feed_token"] for row in conn.execute("SELECT feed_token FROM shows")]
|
||||
assert all(tokens)
|
||||
assert len(set(tokens)) == 2
|
||||
|
||||
|
||||
def test_ad_segments_cascade_on_episode_delete(conn):
|
||||
conn.execute("INSERT INTO shows (query) VALUES ('a')")
|
||||
conn.execute("INSERT INTO episodes (show_id, guid) VALUES (1, 'ep-1')")
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source)"
|
||||
" VALUES (1, 0, 30, 'chapter')"
|
||||
)
|
||||
conn.execute("DELETE FROM episodes WHERE id = 1")
|
||||
assert conn.execute("SELECT COUNT(*) FROM ad_segments").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_utcnow_format():
|
||||
value = db.utcnow()
|
||||
assert len(value) == 20 and value.endswith("Z") and value[10] == "T"
|
||||
|
||||
@@ -47,7 +47,9 @@ def test_parse_feed(fixtures):
|
||||
assert ep1.pubdate == "2025-01-01T06:00:00Z"
|
||||
assert ep1.duration_seconds == 3723
|
||||
assert ep1.audio_url == "https://example.com/audio/ep1.mp3"
|
||||
assert ep1.chapters_url == "https://example.com/ep1-chapters.json"
|
||||
assert ep2.duration_seconds == 2700
|
||||
assert ep2.chapters_url is None
|
||||
# no <guid> → enclosure URL stands in
|
||||
assert ep3.guid == "https://example.com/audio/ep3.mp3"
|
||||
assert ep3.duration_seconds == 1800
|
||||
@@ -80,6 +82,8 @@ def test_ingest_inserts_then_noop(conn, fixtures):
|
||||
assert (first.inserted, first.updated, first.total) == (3, 0, 3)
|
||||
assert (second.inserted, second.updated, second.total) == (0, 0, 3)
|
||||
assert conn.execute("SELECT COUNT(*) FROM episodes").fetchone()[0] == 3
|
||||
row = conn.execute("SELECT chapters_url FROM episodes WHERE guid = 'ep-001'").fetchone()
|
||||
assert row["chapters_url"] == "https://example.com/ep1-chapters.json"
|
||||
|
||||
|
||||
def test_ingest_updates_changed_episodes(conn, fixtures):
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Token-gated /feed and /audio routes: real HTTP against a server on an
|
||||
ephemeral port, same pattern as test_web.py. Kept in its own file since these
|
||||
routes are deliberately unauthenticated (no cookie login — see web.py's
|
||||
module docstring) and that's worth testing in isolation from the dashboard's
|
||||
login-wall behavior.
|
||||
"""
|
||||
|
||||
import http.client
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from hark import db, web
|
||||
|
||||
TOKEN = "test-token-abc123"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def server(tmp_path):
|
||||
conn = db.connect(tmp_path / "hark.db")
|
||||
conn.execute(
|
||||
"INSERT INTO shows (query, title, feed_url, description, image_url, feed_token)"
|
||||
" VALUES ('q', 'Show A', 'http://original/feed', 'A show', 'http://original/art.png', ?)",
|
||||
(TOKEN,),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (show_id, guid, title, description, pubdate, audio_url)"
|
||||
" VALUES (1, 'ep-1', 'Ep 1', 'desc', '2026-01-01T00:00:00Z', 'http://original/ep1.mp3')"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
cut_dir = tmp_path / "cut"
|
||||
cut_dir.mkdir()
|
||||
cut_path = cut_dir / "2.mp3"
|
||||
cut_path.write_bytes(b"cut-bytes")
|
||||
conn = db.connect(tmp_path / "hark.db")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (show_id, guid, title, audio_url, cut_path)"
|
||||
" VALUES (1, 'ep-2', 'Ep 2', 'http://original/ep2.mp3', ?)",
|
||||
(str(cut_path),),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
srv = web.make_server(
|
||||
tmp_path / "hark.db", tmp_path / "auth.db", bind="127.0.0.1:0",
|
||||
admin_token="letmein", base_url="http://myhost:8710",
|
||||
)
|
||||
thread = threading.Thread(target=srv.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
yield srv
|
||||
srv.shutdown()
|
||||
|
||||
|
||||
def request(srv, path):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", srv.server_address[1], timeout=5)
|
||||
conn.request("GET", path)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
conn.close()
|
||||
return resp, data
|
||||
|
||||
|
||||
def test_feed_route_no_login_required(server):
|
||||
resp, data = request(server, f"/feed/1/{TOKEN}")
|
||||
assert resp.status == 200
|
||||
assert resp.getheader("Content-Type") == "application/rss+xml; charset=utf-8"
|
||||
assert b"Show A" in data
|
||||
|
||||
|
||||
def test_feed_route_wrong_token_404s(server):
|
||||
resp, _ = request(server, "/feed/1/wrong-token")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_feed_route_unknown_show_404s(server):
|
||||
resp, _ = request(server, f"/feed/999/{TOKEN}")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_feed_route_passthrough_for_uncut_episode(server):
|
||||
_resp, data = request(server, f"/feed/1/{TOKEN}")
|
||||
assert b'url="http://original/ep1.mp3"' in data
|
||||
|
||||
|
||||
def test_feed_route_points_cut_episode_at_local_audio(server):
|
||||
_resp, data = request(server, f"/feed/1/{TOKEN}")
|
||||
assert f'url="http://myhost:8710/audio/2/{TOKEN}.mp3"'.encode() in data
|
||||
|
||||
|
||||
def test_audio_route_serves_cut_file_with_correct_token(server):
|
||||
resp, data = request(server, f"/audio/2/{TOKEN}.mp3")
|
||||
assert resp.status == 200
|
||||
assert resp.getheader("Content-Type") == "audio/mpeg"
|
||||
assert data == b"cut-bytes"
|
||||
|
||||
|
||||
def test_audio_route_wrong_token_404s(server):
|
||||
resp, _ = request(server, "/audio/2/wrong-token.mp3")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_audio_route_uncut_episode_404s(server):
|
||||
# ep-1 has no cut_path — nothing to serve locally for it
|
||||
resp, _ = request(server, f"/audio/1/{TOKEN}.mp3")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_audio_route_unknown_episode_404s(server):
|
||||
resp, _ = request(server, f"/audio/999/{TOKEN}.mp3")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_unknown_route_falls_through_to_dashboard_login_wall(server):
|
||||
# confirms /feed and /audio prefix checks don't swallow unrelated routes —
|
||||
# an unauthenticated request to anything else still hits the normal
|
||||
# dashboard dispatcher, which redirects to /login (not a 404, since that
|
||||
# dispatcher's own not_found() is gated behind the session check too).
|
||||
resp, _ = request(server, "/nonsense")
|
||||
assert resp.status == 303
|
||||
assert resp.getheader("Location") == "/login"
|
||||
|
||||
|
||||
class _FakeServer:
|
||||
def serve_forever(self):
|
||||
pass
|
||||
|
||||
|
||||
def test_serve_warns_when_base_url_is_localhost(tmp_path, capsys, monkeypatch):
|
||||
monkeypatch.setattr(web, "make_server", lambda *a, **k: _FakeServer())
|
||||
web.serve(
|
||||
tmp_path / "hark.db", tmp_path / "auth.db", "127.0.0.1:0", None, False,
|
||||
base_url="http://localhost:8710",
|
||||
)
|
||||
assert "warning:" in capsys.readouterr().out.lower()
|
||||
|
||||
|
||||
def test_serve_no_warning_for_a_configured_hostname(tmp_path, capsys, monkeypatch):
|
||||
# note: matching the exact "warning:" prefix (not a bare "warning"
|
||||
# substring) since pytest's tmp_path embeds the test's own function name,
|
||||
# and a name containing "warning" would otherwise leak into the printed
|
||||
# db path and self-sabotage this assertion.
|
||||
monkeypatch.setattr(web, "make_server", lambda *a, **k: _FakeServer())
|
||||
web.serve(
|
||||
tmp_path / "hark.db", tmp_path / "auth.db", "127.0.0.1:0", None, False,
|
||||
base_url="http://truenas.local:8710",
|
||||
)
|
||||
assert "warning:" not in capsys.readouterr().out.lower()
|
||||
Reference in New Issue
Block a user