Depend on adscrub as a library for ad-stripping (v0.4.0)

adscrub (flan/adscrub) stays a separate, standalone product -- own
repo, schema, CLI. hark adds it as a path dependency (editable,
../adscrub -- see pyproject.toml [tool.uv.sources]) and calls its
functions directly rather than duplicating any code.

hark's own episodes/shows/ad_segments schema was deliberately shaped
to match adscrub's column-for-column, so adscrub's schema-coupled
functions (pending_episodes, scan_episode, transcribe_episode,
detect_pending, cut_pending, ...) work unchanged against hark's own
conn. hark cli.py's chapters/transcribe/detect-ads/cut subcommands
call straight into the adscrub package -- no hark-side
chapters.py/transcribe.py/detect.py/cut.py exists.

podcast_feed.py is hark's own file (schema-specific: show_id/feed_url
naming plus token auth, none of which adscrub's own feed.py has), used
by new token-gated /feed/<show_id>/<token> and
/audio/<episode_id>/<token>.<ext> routes on hark web -- unauthenticated
since a podcast app can't do the dashboard's cookie login, gated
instead by a random per-show feed_token.

110 tests pass. Known gap, not solved: the path dependency doesn't
resolve in the Docker build context yet (needs a real packaging
decision -- git dependency+deploy key, vendored wheel, or a
multi-repo build script) -- documented in the Dockerfile, README, and
docs/PLAN.md rather than papered over.
This commit is contained in:
flan
2026-07-11 04:33:03 +00:00
parent c55efc12b1
commit 7c94632bf6
19 changed files with 1076 additions and 53 deletions
+2
View File
@@ -9,3 +9,5 @@ dist/
# Local data
hark.db
hark.db-*
auth.db
data/
+37
View File
@@ -7,6 +7,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.4.0] - 2026-07-11
### Added
- Ad-stripping via `flan/adscrub`, added as a **library dependency**
(`[tool.uv.sources]` path dependency, editable) — not a code merge. hark's
`episodes`/`shows`/`ad_segments` schema was deliberately shaped to match
adscrub's own, so adscrub's schema-coupled functions work unchanged against
hark's database: `hark chapters`/`transcribe`/`detect-ads`/`cut` call
straight into the `adscrub` package. No duplicated pipeline code exists in
this repo.
- `hark web` now also serves `GET /feed/<show_id>/<token>` (regenerated clean
RSS, via hark's own `podcast_feed.py`) and `GET /audio/<episode_id>/<token>.<ext>`
(locally-cut episodes) — unauthenticated (no cookie login, since a podcast
app can't do that) but gated by a per-show random token. `--base-url`/
`$HARK_BASE_URL` controls what's embedded in generated links; warns if left
at the unreachable `localhost` default.
- `compose.gpu.yaml`: requests the host's GPU via the `nvidia` Docker runtime;
hark's own `gpu` extra passes through to `adscrub[gpu]`.
### Changed
- Dependency, not a merge — corrected mid-session (see below) after the wrong
approach was initially built and pushed, then reverted.
### Fixed
- An earlier pass in this same session **fully merged** adscrub's source files
into `src/hark/` and pushed it to `main` — the wrong architecture (the
intent was always two separate products, with hark depending on adscrub as
a library). Reverted via `git revert -m 1` (history-preserving, not a
force-push/reset) once caught, then rebuilt correctly as a dependency. Also:
this merge/revert/rebuild happened concurrently with another Claude session
actively working in this same `~/hark` checkout (uncommitted `claims.py`
work) — branch switches were done carefully to avoid disturbing it. See
memory `feedback_shared_working_dir` for the general lesson.
## [0.3.7] - 2026-07-10
### Fixed
+44 -6
View File
@@ -15,25 +15,63 @@ A homelab web service (NOT a mobile app, NOT an AntennaPod fork) that:
similarity.
3. **Episode scoring (later):** metric-based interestingness ratings, tiltmeter-style
(auditable, defined metrics, calibrated against the owner's actual listening).
4. **Ad-stripping (added 2026-07-11):** finds ad spans (chapter markers, or Whisper +
LLM classification) and cuts them out, covering *every* subscription, not just the
genre-curated shows #1-#3 track. **This is provided by depending on `flan/adscrub` as
a library, not by duplicating its code** — adscrub is a separate, standalone product.
See "Architecture decisions" below before touching anything ad-stripping-related.
Origin: ideas #2 and #3 in a private ideas repo (git.onetick.ninja) — read
`~/project-ideas/README.md` for the full assessments and reasoning.
`~/project-ideas/README.md` for the full assessments and reasoning. The ad-stripping
feature's own origin (AntennaPod's long-open feature request, why LLM-over-transcript
beats fingerprinting/crowdsourcing) is in adscrub's own repo history.
## Architecture decisions (already made — don't relitigate)
- Standalone service on the homelab, shaped like tiltmeter: scheduled ingest → pipeline →
SQLite → API/UI. The owner's player stays AntennaPod.
- **Input integration:** AntennaPod syncs subscriptions + play history to Nextcloud (gpodder
sync app) on truenas; hark reads from that API. OPML import as fallback. (Not wired in M0.)
sync app) on truenas; hark reads from that API. OPML import as fallback. (Not wired yet —
see M3 in docs/PLAN.md; ad-stripping still uses the manual `feeds.txt`/`resolve` flow too.)
- **Output integration:** hark generates custom RSS feeds (e.g. "top episodes about topics
you like", "best of candidate shows") that get subscribed to in AntennaPod like any podcast.
No app modification anywhere.
you like", "best of candidate shows", ad-stripped versions of any subscription) that get
subscribed to in AntennaPod like any podcast. No app modification anywhere.
- Feed URLs resolve via the keyless iTunes Search API; Podcast Index API can be added later
(needs a registered key). Episode metadata comes from plain RSS.
- Topic extraction: LLM extraction from episode title/description, canonicalized against
Wikidata. Transcripts/Whisper are explicitly OUT of scope until much later — these genres
name their subject in the metadata.
Wikidata — these genres name their subject in the metadata, so this doesn't need
transcripts even though transcription is now available (see below).
- Topics can belong to multiple genres (Titanic = history + disaster); never force one bucket.
- **adscrub is a dependency, not a merge — this is deliberate and non-negotiable.**
`flan/adscrub` is its own product: own repo, own schema, own CLI, deployable and useful
standalone. hark depends on it (`[tool.uv.sources]` path dependency, editable — see
pyproject.toml) and calls its functions directly. hark's `episodes`/`shows`/`ad_segments`
schema is deliberately shaped to match adscrub's own column names *specifically so*
adscrub's schema-coupled functions (`pending_episodes`, `scan_episode`,
`transcribe_episode`, `detect_pending`, `cut_pending`, ...) work unchanged against hark's
`conn` — call them from hark's cli.py directly. **Do not copy adscrub's source files into
this repo.** That mistake was actually made once (2026-07-11), pushed to main, and had to
be reverted via `git revert` once caught — see CHANGELOG 0.4.0. The only hark-owned
ad-stripping code should be: the schema migration, cli.py's argparse wiring, and
`podcast_feed.py` (genuinely schema-specific — adscrub's own feed-building code targets a
different schema and has no token-auth concept, so it isn't reusable as-is).
- **Whisper transcription** (via adscrub) is cached process-wide, keyed by model size —
hark's ad-span detection and (later) M4 episode-scoring should request the *same* model
size and run sequentially through the cron-scheduled pipeline (they do) so exactly one
model is ever resident in VRAM. This works automatically only because hark calls
adscrub's actual `load_model()` function, not a copy — don't undermine it by ever adding a
second, hark-owned copy of that caching logic. GPU: `code` has a real RTX 2070 SUPER,
Docker's `nvidia` runtime is registered; `compose.gpu.yaml` requests it, hark's own `gpu`
extra passes through to `adscrub[gpu]`.
- **Feed/audio route auth:** `/feed/<show_id>/<token>` and `/audio/<episode_id>/<token>.<ext>`
are unauthenticated (no cookie login — a podcast app can't do that) but gated by a random
per-show `feed_token` embedded in the URL, compared with `secrets.compare_digest`. Not the
dashboard's session system, and not wide open either.
- **Known unsolved gap:** the adscrub path dependency doesn't resolve in the Docker build
(build context only has hark's own files). Don't quietly work around this by copying
adscrub's source into the build context or removing the dependency — it's a real packaging
decision (git dependency + deploy key, vendored wheel, multi-repo build script) that needs
to actually be made, not paved over. See docs/PLAN.md open questions.
## Conventions
+30 -6
View File
@@ -1,9 +1,26 @@
# hark: pipeline + web frontend in one image.
#
# Default command serves the login-walled web UI over the databases in
# /app/data; every pipeline stage is available as a one-shot command, e.g.:
# Default command serves the dashboard (login-walled) + feed/audio routes
# (token-gated) over the databases in /app/data; every pipeline stage is
# available as a one-shot command, e.g.:
# docker compose run --rm hark ingest
# docker compose run --rm hark canon
# docker compose run --rm hark chapters
# docker compose run --rm hark transcribe
# docker compose run --rm hark detect-ads
# docker compose run --rm hark cut
#
# KNOWN GAP, not solved here: hark depends on adscrub as a local path
# dependency (../adscrub, editable — see pyproject.toml [tool.uv.sources]).
# This build context only COPYs hark's own files, so `uv sync` below will
# fail to resolve that dependency as written. Real fix is a packaging
# decision (git dependency + deploy key, vendoring a built adscrub wheel into
# this context, or a small multi-repo build script) — see docs/PLAN.md open
# questions. Don't paper over this by quietly dropping the dependency.
#
# Build with --build-arg GPU=1 (or `docker compose -f compose.yaml -f compose.gpu.yaml
# build`) to pull in the cuBLAS/cuDNN extra for faster-whisper's CUDA path — only
# needed on a host that actually passes a GPU through (see CLAUDE.md).
FROM python:3.13-slim
@@ -11,24 +28,31 @@ COPY --from=ghcr.io/astral-sh/uv:0.7 /uv /uvx /bin/
WORKDIR /app
ENV UV_LINK_MODE=copy UV_COMPILE_BYTECODE=1
ARG GPU=0
# dependency layer first: rebuilds only when the lockfile changes
COPY pyproject.toml uv.lock README.md ./
RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen --no-dev --no-install-project
RUN --mount=type=cache,target=/root/.cache/uv \
if [ "$GPU" = "1" ]; then uv sync --frozen --no-dev --no-install-project --extra gpu; \
else uv sync --frozen --no-dev --no-install-project; fi
COPY src ./src
RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen --no-dev
RUN --mount=type=cache,target=/root/.cache/uv \
if [ "$GPU" = "1" ]; then uv sync --frozen --no-dev --extra gpu; \
else uv sync --frozen --no-dev; fi
ENV PATH="/app/.venv/bin:$PATH" \
HARK_DB=/app/data/hark.db \
HARK_AUTH_DB=/app/data/auth.db
HARK_AUTH_DB=/app/data/auth.db \
HARK_DATA_DIR=/app/data
# gosu drops from root to the unprivileged `hark` user after the entrypoint
# fixes ownership of /app/data — Docker creates bind mounts and anonymous
# volumes as root, which this user can't write to on its own. uid/gid 568
# matches TrueNAS SCALE's standard "apps" account, so files land owned by
# the same user/group as every other app on that host; harmless elsewhere.
RUN apt-get update && apt-get install -y --no-install-recommends gosu \
# ffmpeg is for adscrub's cut.py, called as a library (see cli.py).
RUN apt-get update && apt-get install -y --no-install-recommends gosu ffmpeg \
&& rm -rf /var/lib/apt/lists/* \
&& groupadd --gid 568 hark \
&& useradd --system --uid 568 --gid 568 --no-create-home hark
+48 -4
View File
@@ -5,10 +5,19 @@ Cross-podcast topic index and discovery service for subject-per-episode genres
real-world case/event/person they cover, so you can ask "who covered the Dyatlov
Pass incident?" and compare treatments across shows.
See `docs/PLAN.md` for milestones. Current state (0.3.5, M1 complete): feed
resolution, episode ingest, LLM topic extraction with Wikidata canonicalization,
the cross-show topic index, and a full web UI — deployed live. M2 (discovery) is
next.
hark also strips ads from every subscription (not just the genre-curated shows
above) by depending on [adscrub](https://git.onetick.ninja/flan/adscrub) — a
separate, standalone product — as a library: `hark chapters`/`transcribe`/
`detect-ads`/`cut` call straight into adscrub's functions, since hark's own
`episodes`/`ad_segments` schema was deliberately shaped to match adscrub's, so
adscrub's schema-coupled functions work unchanged against hark's database.
Nothing here is a copy of adscrub's code — see CLAUDE.md and docs/PLAN.md for
the integration design and why it's a dependency, not a merge.
See `docs/PLAN.md` for milestones. Current state (0.4.0): feed resolution,
episode ingest, LLM topic extraction with Wikidata canonicalization, the
cross-show topic index, a full web UI, and adscrub-backed ad-stripping —
deployed live. M2 (discovery) is next for the topic-index side.
## Usage
@@ -22,11 +31,32 @@ uv run hark canon # retry Wikidata canonicalization for unmatched t
uv run hark stats # counts per show
uv run hark topics # topics ranked by cross-show coverage
uv run hark who "dyatlov" # who covered X (label substring or Wikidata QID)
# ad-stripping pipeline (backed by the adscrub library) — every show, not just feeds.txt's
uv run hark chapters # scan chapter markers for ad spans (free — no transcription)
uv run hark transcribe # Whisper the rest
uv run hark detect-ads # LLM ad-span classification (needs $ANTHROPIC_API_KEY)
uv run hark cut # ffmpeg out the ad spans
```
The database defaults to `./hark.db`; override with `--db` or `$HARK_DB`.
Show names live in `feeds.txt`, one per line, `#` for comments.
## Setup
adscrub is a **path dependency** (`../adscrub`, editable — see
`pyproject.toml`'s `[tool.uv.sources]`), so `flan/adscrub` needs to be checked
out as a sibling of this repo before `uv sync` will resolve it:
```
cd .. && git clone ssh://git@git.onetick.ninja:55214/flan/adscrub.git
cd hark && uv sync
```
This works for local development; it does **not** yet work for the Docker
build (the build context only has hark's own files) — see the Dockerfile's
"KNOWN GAP" comment and docs/PLAN.md's open questions. Not solved yet.
## Web UI
`hark web` serves the topic index (default `0.0.0.0:8710`): a home dashboard
@@ -41,13 +71,27 @@ separate `auth.db` (`--auth-db` / `$HARK_AUTH_DB`) so replacing `hark.db` with
a fresh data snapshot never logs anyone out. Set `HARK_COOKIE_SECURE=1` when
serving behind a TLS-terminating proxy.
The same server also answers `GET /feed/<show_id>/<token>` (the cleaned RSS
feed) and `GET /audio/<episode_id>/<token>.<ext>` (locally-cut episodes) —
deliberately *not* behind the login wall, since a podcast app can't do cookie
login. Instead each show gets a random `feed_token` (auto-generated,
`shows.feed_token`) that has to appear in the URL; wrong or missing token is a
404, not a redirect to `/login`. `--base-url`/`$HARK_BASE_URL` must be set to
wherever the podcast player can actually reach this server — it's embedded in
every generated audio link, and `web` warns if left at the unreachable
`localhost` default.
In Docker: `docker compose up -d` (mounts `./data`, serves :8710); pipeline
stages run as one-shots, e.g. `docker compose run --rm hark ingest`.
Transcription runs CPU-only by default — see `compose.gpu.yaml` and CLAUDE.md
for the GPU deploy path.
Extraction calls the Anthropic API (default model `claude-opus-4-8`; override
with `--model` or `$HARK_MODEL`) and canonicalizes labels against Wikidata so
aliases merge ("BTK" = "Dennis Rader"). Runs are idempotent and resumable:
processed episodes are marked and skipped, failures are retried next run.
Ad-span classification is a separate model default (`--model`/`$HARK_AD_MODEL`)
since it's a differently-shaped task.
## Development
+19
View File
@@ -0,0 +1,19 @@
# GPU override: requests the host's RTX 2070 SUPER via the nvidia Docker
# runtime (confirmed registered on `code` — see CLAUDE.md) and builds the
# image with the cuBLAS/cuDNN extra (passed through from adscrub[gpu], see
# pyproject.toml) so faster-whisper actually uses it.
#
# docker compose -f compose.yaml -f compose.gpu.yaml build
# docker compose -f compose.yaml -f compose.gpu.yaml run --rm hark transcribe
services:
hark:
build:
args:
GPU: "1"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: ["gpu"]
+11 -2
View File
@@ -1,10 +1,18 @@
# Web UI + one-shot pipeline commands.
# docker compose up -d # serve web UI :8710
# Dashboard + feed/audio routes + one-shot pipeline commands.
# docker compose up -d # serve :8710
# docker compose run --rm hark ingest # poll feeds once
# docker compose run --rm hark canon # retry Wikidata matches
# docker compose run --rm hark chapters # chapter-sourced ad spans
# docker compose run --rm hark transcribe # Whisper — CPU by default, see compose.gpu.yaml
# docker compose run --rm hark detect-ads # LLM ad-span classification
# docker compose run --rm hark cut # ffmpeg out the ad spans
#
# HARK_ADMIN_TOKEN bootstraps the admin login on a fresh auth db; set a real
# password at /account afterwards (the token stops working once one exists).
# HARK_BASE_URL must be set to wherever the podcast player can actually reach
# this container (not localhost) — it's embedded in generated feeds' links.
# NOTE: build doesn't actually work yet — see Dockerfile's KNOWN GAP comment
# about the adscrub path dependency not resolving in this build context.
services:
hark:
build: .
@@ -14,6 +22,7 @@ services:
environment:
HARK_ADMIN_TOKEN: ${HARK_ADMIN_TOKEN:-}
HARK_COOKIE_SECURE: ${HARK_COOKIE_SECURE:-0}
HARK_BASE_URL: ${HARK_BASE_URL:-http://localhost:8710}
volumes:
- ./data:/app/data
restart: unless-stopped
+68 -3
View File
@@ -43,6 +43,53 @@ Not in the original milestone list — added mid-stream on explicit request, ahe
- Two full audit passes (security + code-quality, then a screenshot-driven UX pass against
the real dataset) — see CHANGELOG 0.3.1 and 0.3.5 for what each one caught.
## Ad-stripping via adscrub (done, 0.4.0)
Not in the original milestone list. `flan/adscrub` is a separate, standalone
product (its own repo, schema, CLI, deployable alone) that does chapter-marker
scanning, Whisper transcription, LLM ad-span classification, and ffmpeg
cutting. hark depends on it **as a library** rather than duplicating its code
— two products, not a merge (an earlier session actually did a full code
merge here; it was reverted per explicit correction — see CHANGELOG 0.4.0 and
CLAUDE.md for why, and don't repeat that mistake).
- **Why a dependency, not two fully separate schemas:** hark's own
`episodes` gained `chapters_url`/`chapters_scanned_at`/`transcript_path`/
`llm_detected_at`/`cut_path`, and `shows` gained `feed_token`; new
`ad_segments` table. These were deliberately shaped to match adscrub's own
schema column-for-column, so adscrub's schema-coupled functions
(`pending_episodes`, `scan_episode`, `transcribe_episode`, `detect_pending`,
`cut_pending`, ...) work **unchanged** against hark's own `conn` — hark's
CLI (`chapters`/`transcribe`/`detect-ads`/`cut`) calls them directly. No
hark-side `chapters.py`/`transcribe.py`/`detect.py`/`cut.py` files exist;
that would just be duplicated code with its own drift risk.
- **What's genuinely hark's own code:** the CLI wiring (cli.py), the schema
migration, and `podcast_feed.py` (feed-building — adscrub's own `feed.py`
targets its `feeds`/`feed_id` schema and has no token concept, so this one
isn't reusable as-is; not worth generalizing adscrub's version for one
consumer).
- **Serving:** `hark web` also answers `/feed/<show_id>/<token>` and
`/audio/<episode_id>/<token>.<ext>` — unauthenticated (a podcast app can't
do the dashboard's cookie login) but gated by a random per-show
`feed_token`, not wide open.
- **One shared Whisper model:** adscrub's `transcribe.load_model()` caches
one model process-wide, keyed by model size. Since hark's CLI calls that
same function (not a copy), ad-span detection and future M4 episode-scoring
share one cached instance for free — *as long as* both ask for the same
model size and run sequentially (they do — cron-scheduled batch, not
concurrent requests). A future scoring feature wanting a genuinely
different model size would need to decide that as a real tradeoff.
- **GPU:** `code` has a real RTX 2070 SUPER, Docker's `nvidia` runtime is
registered; `compose.gpu.yaml` requests it, and hark's own `gpu` extra just
passes through to `adscrub[gpu]` rather than duplicating the cuBLAS/cuDNN
package list.
- **Known gap, not solved:** the path dependency (`../adscrub`, editable)
only works for local dev. The Docker build context doesn't include
adscrub's source, so `docker compose build` doesn't actually work yet —
needs a real packaging decision (git dependency + deploy key, vendoring a
built wheel into the build context, or a small multi-repo build script).
See open questions below.
## M2 — discovery
- Embedding similarity over episode topics → related shows, notable back-catalog episodes.
@@ -52,12 +99,20 @@ Not in the original milestone list — added mid-stream on explicit request, ahe
- Read subscriptions/history from Nextcloud gpodder sync (truenas).
- Generate custom RSS feeds as the recommendation delivery channel.
- Note: this is also how new shows should reach the ad-stripping pipeline
(currently manual via `feeds.txt`/`hark resolve`) — wiring gpodder sync in
properly is what actually delivers "every subscription gets ad-stripped,"
not just "every show you've typed into feeds.txt." Deliberately deferred,
not built alongside the adscrub dependency work — a real API integration
project of its own.
## M4 — episode scoring (tiltmeter-style)
- Defined interestingness metrics, calibration loop against owner ratings.
- Per-topic treatment comparison (depth, sensationalism) — needs transcripts for fidelity;
revisit Whisper here, not before.
- Per-topic treatment comparison (depth, sensationalism) — needs transcripts for
fidelity. Whisper is already available via the adscrub dependency (see above);
this milestone should call `adscrub.transcribe`'s functions the same way the
ad-stripping pipeline does, not stand up its own transcription path.
## Seed shows (feeds.txt)
@@ -69,4 +124,14 @@ Resolve their real feed URLs via iTunes Search API at runtime — do not hand-co
- ~~Hosting~~ Resolved 2026-07-10: private Gitea (`flan/hark`); revisit GitHub if it goes public.
- Which LLM/provider for extraction (M1 decision).
- GPU/Whisper feasibility on this LXC (M4 decision; CUDA device nodes may not be exposed).
- ~~GPU/Whisper feasibility~~ Resolved 2026-07-11: `code` has a real GPU, Docker's
`nvidia` runtime is registered — see the adscrub dependency section above.
- **How to make the adscrub path dependency work in the Docker build.** Not solved;
the build context currently only has hark's own files. Options: git dependency
(needs a deploy key baked into the build, or a build secret), vendor a built
adscrub wheel into the build context, or a small script that builds both repos
together. Don't guess at this — it's a real infra decision.
- `hark detect-ads` currently defaults to `claude-opus-4-8`; revisit cost vs.
accuracy on ad-span boundaries once run against real transcripts.
- When to actually wire the gpodder/Nextcloud subscription sync (M3) so ad-stripping
covers real subscriptions instead of the manually-curated `feeds.txt` list.
+166 -2
View File
@@ -1,4 +1,14 @@
"""hark command line: resolve, ingest, extract, stats, topics, who."""
"""hark command line: resolve, ingest, extract, chapters, transcribe, detect-ads,
cut, stats, topics, who, web.
chapters/transcribe/detect-ads/cut call straight into the `adscrub` package
(a separate product, depended on as a library — see pyproject.toml) rather
than through any hark-side reimplementation: hark's episodes/ad_segments
schema was deliberately shaped to match adscrub's own, so adscrub's
schema-coupled functions (pending_episodes, scan_episode, transcribe_episode,
detect_pending, cut_pending, ...) work unchanged against hark's `conn`. Only
the CLI wiring here is hark's own code.
"""
from __future__ import annotations
@@ -9,6 +19,11 @@ from typing import Callable
import httpx
from adscrub import chapters as ad_chapters
from adscrub import cut as ad_cut
from adscrub import detect as ad_detect
from adscrub import transcribe as ad_transcribe
from . import __version__, db, extract, ingest, pipeline, resolve, wikidata
DEFAULT_DB = os.environ.get("HARK_DB", "hark.db")
@@ -78,6 +93,116 @@ def cmd_ingest(args: argparse.Namespace) -> int:
return 1 if errors else 0
def cmd_chapters(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
episodes = ad_chapters.pending_episodes(conn)
if not episodes:
print("no episodes with an unscanned chapters_url", file=sys.stderr)
return 1
found = 0
with make_client() as client:
for ep in episodes:
try:
n = ad_chapters.scan_episode(conn, client, ep)
except httpx.HTTPError as exc:
print(f" FAIL {ep['title']}: {exc}")
continue
found += n
print(f" ok {ep['title']}: {n} ad span(s) from chapters")
print(f"found {found} chapter-sourced ad span(s) across {len(episodes)} episode(s)")
return 0
def cmd_transcribe(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
pending = ad_transcribe.pending_episodes(conn, args.limit)
if args.dry_run:
total_pending = len(ad_transcribe.pending_episodes(conn))
print(f"pending episodes: {total_pending}"
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
return 0
if not pending:
print("no episodes pending transcription", file=sys.stderr)
return 1
errors = 0
with make_client() as client:
for ep in pending:
try:
path = ad_transcribe.transcribe_episode(conn, ep, client, model_size=args.model)
except (httpx.HTTPError, OSError) as exc:
errors += 1
print(f" FAIL {ep['title']}: {exc}")
continue
print(f" ok {ep['title']} -> {path}")
remaining = len(ad_transcribe.pending_episodes(conn))
print(f"transcribed {len(pending) - errors} episode(s) ({errors} failed, {remaining} still pending)")
return 1 if errors else 0
def cmd_detect_ads(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
pending = ad_detect.pending_episodes(conn, args.limit)
if args.dry_run:
total_pending = len(ad_detect.pending_episodes(conn))
print(f"pending episodes: {total_pending}"
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
return 0
if not pending:
print("no episodes pending ad-span detection", file=sys.stderr)
return 1
import anthropic # deferred: other commands must work without a key
try:
client = anthropic.Anthropic()
except anthropic.AnthropicError as exc:
print(f"anthropic client: {exc}", file=sys.stderr)
print("hint: export ANTHROPIC_API_KEY first (it lives in rbw, not in a file)",
file=sys.stderr)
return 1
detector = ad_detect.ClaudeAdDetector(client, model=args.model)
def report(r: ad_detect.DetectResult) -> None:
if r.error:
print(f" FAIL {r.title}: {r.error}")
else:
print(f" ok {r.title}: {r.found} ad span(s) from transcript")
results = ad_detect.detect_pending(conn, detector, limit=args.limit, on_result=report)
errors = sum(1 for r in results if r.error)
remaining = len(ad_detect.pending_episodes(conn))
print(f"detected across {len(results) - errors} episode(s) ({errors} failed, {remaining} still pending)")
return 1 if errors else 0
def cmd_cut(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
pending = ad_cut.pending_episodes(conn, args.limit)
if args.dry_run:
total_pending = len(ad_cut.pending_episodes(conn))
print(f"pending episodes: {total_pending}"
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
return 0
if not pending:
print("no episodes pending cutting", file=sys.stderr)
return 1
def report(r: ad_cut.CutResult) -> None:
if r.error:
print(f" FAIL {r.title}: {r.error}")
else:
print(f" ok {r.title}: removed {r.ad_seconds:.1f}s of ads")
with make_client() as client:
results = ad_cut.cut_pending(conn, client, limit=args.limit, on_result=report)
errors = sum(1 for r in results if r.error)
remaining = len(ad_cut.pending_episodes(conn))
print(f"cut {len(results) - errors} episode(s) ({errors} failed, {remaining} still pending)")
return 1 if errors else 0
def cmd_extract(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
pending = pipeline.pending_episodes(conn, args.limit)
@@ -224,6 +349,7 @@ def cmd_web(args: argparse.Namespace) -> int:
bind=args.bind,
admin_token=os.environ.get("HARK_ADMIN_TOKEN"),
cookie_secure=os.environ.get("HARK_COOKIE_SECURE", "0") == "1",
base_url=args.base_url,
)
return 0
@@ -236,9 +362,12 @@ def cmd_stats(args: argparse.Namespace) -> int:
episodes = conn.execute("SELECT COUNT(*) FROM episodes").fetchone()[0]
topics = conn.execute("SELECT COUNT(*) FROM topics").fetchone()[0]
links = conn.execute("SELECT COUNT(*) FROM episode_topics").fetchone()[0]
segments = conn.execute("SELECT COUNT(*) FROM ad_segments").fetchone()[0]
cut_count = conn.execute("SELECT COUNT(*) FROM episodes WHERE cut_path IS NOT NULL").fetchone()[0]
print(f"shows: {shows['n']} ({shows['resolved']} resolved)")
print(f"episodes: {episodes}")
print(f"topics: {topics} ({links} episode links)")
print(f"ad_segments: {segments} ({cut_count} episodes cut)")
rows = conn.execute(
"""
SELECT COALESCE(s.title, s.query) AS name, COUNT(e.id) AS n,
@@ -273,6 +402,34 @@ def main(argv: list[str] | None = None) -> int:
p = sub.add_parser("ingest", help="fetch resolved feeds and upsert episodes")
p.set_defaults(func=cmd_ingest)
p = sub.add_parser("chapters", help="scan episodes' existing chapter markers for ad spans")
p.set_defaults(func=cmd_chapters)
p = sub.add_parser(
"transcribe", help="transcribe episodes with no chapter-sourced ad spans"
)
p.add_argument("--limit", type=int, help="max episodes to process this run")
p.add_argument("--model", default=os.environ.get("HARK_WHISPER_MODEL", ad_transcribe.DEFAULT_MODEL),
help=f"faster-whisper model size (default: $HARK_WHISPER_MODEL or "
f"{ad_transcribe.DEFAULT_MODEL})")
p.add_argument("--dry-run", action="store_true",
help="only report how many episodes are pending")
p.set_defaults(func=cmd_transcribe)
p = sub.add_parser("detect-ads", help="classify ad spans from transcripts with a Claude model")
p.add_argument("--limit", type=int, help="max episodes to process this run")
p.add_argument("--model", default=os.environ.get("HARK_AD_MODEL", ad_detect.DEFAULT_MODEL),
help=f"Claude model id (default: $HARK_AD_MODEL or {ad_detect.DEFAULT_MODEL})")
p.add_argument("--dry-run", action="store_true",
help="only report how many episodes are pending")
p.set_defaults(func=cmd_detect_ads)
p = sub.add_parser("cut", help="cut ad spans out of episode audio with ffmpeg")
p.add_argument("--limit", type=int, help="max episodes to process this run")
p.add_argument("--dry-run", action="store_true",
help="only report how many episodes are pending")
p.set_defaults(func=cmd_cut)
p = sub.add_parser("extract", help="extract episode topics with a Claude model")
p.add_argument("--limit", type=int, help="max episodes to process this run")
p.add_argument("--model", default=DEFAULT_MODEL,
@@ -295,12 +452,19 @@ def main(argv: list[str] | None = None) -> int:
p = sub.add_parser("canon", help="retry Wikidata canonicalization for unmatched topics")
p.set_defaults(func=cmd_canon)
p = sub.add_parser("web", help="serve the web frontend (login-walled)")
p = sub.add_parser(
"web", help="serve the dashboard (login-walled) + feed/audio routes (token-gated)"
)
p.add_argument("--bind", default=os.environ.get("HARK_BIND", "0.0.0.0:8710"),
help="host:port (default: $HARK_BIND or 0.0.0.0:8710)")
p.add_argument("--auth-db", default=os.environ.get("HARK_AUTH_DB", "auth.db"),
help="auth database path, kept separate from hark.db "
"(default: $HARK_AUTH_DB or auth.db)")
p.add_argument("--base-url", default=os.environ.get("HARK_BASE_URL", "http://localhost:8710"),
help="externally-reachable URL this server is served at — embedded "
"in generated feeds' audio links, so it must resolve from "
"wherever the podcast player runs, not just from this host "
"(default: $HARK_BASE_URL or http://localhost:8710)")
p.set_defaults(func=cmd_web)
p = sub.add_parser("topics", help="list topics by cross-show coverage")
+58 -11
View File
@@ -2,12 +2,22 @@
from __future__ import annotations
import secrets
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
# topics / topic_genres / episode_topics are created now but only populated by
# M1 extraction; wikidata_id, confidence, and source stay NULL until then.
#
# feed_token / chapters_* / transcript_path / llm_detected_at / cut_path / ad_segments
# are hark's own ad-stripping tracking — hark uses adscrub as a library (see
# pyproject.toml [tool.uv.sources]) for the reusable pipeline logic, but the state of
# *which of hark's own episodes* have been scanned/transcribed/detected/cut lives here,
# in hark's own schema, since that's tied to hark's own shows/episodes rows, not
# adscrub's separate database. feed_token gates the unauthenticated /feed and /audio
# routes (podcast apps can't do the dashboard's cookie login); every show gets one via
# _backfill_feed_tokens, not just ones added after this feature.
SCHEMA = """
PRAGMA foreign_keys = ON;
@@ -20,23 +30,29 @@ CREATE TABLE IF NOT EXISTS shows (
author TEXT,
description TEXT,
image_url TEXT,
feed_token TEXT,
last_fetched_at TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT
);
CREATE TABLE IF NOT EXISTS episodes (
id INTEGER PRIMARY KEY,
show_id INTEGER NOT NULL REFERENCES shows(id) ON DELETE CASCADE,
guid TEXT NOT NULL,
title TEXT,
description TEXT,
pubdate TEXT,
duration_seconds INTEGER,
audio_url TEXT,
extracted_at TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT,
id INTEGER PRIMARY KEY,
show_id INTEGER NOT NULL REFERENCES shows(id) ON DELETE CASCADE,
guid TEXT NOT NULL,
title TEXT,
description TEXT,
pubdate TEXT,
duration_seconds INTEGER,
audio_url TEXT,
extracted_at TEXT,
chapters_url TEXT,
chapters_scanned_at TEXT,
transcript_path TEXT,
llm_detected_at TEXT,
cut_path TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
updated_at TEXT,
UNIQUE (show_id, guid)
);
@@ -61,7 +77,22 @@ CREATE TABLE IF NOT EXISTS episode_topics (
PRIMARY KEY (episode_id, topic_id)
);
-- Ad spans for an episode, however they were found. Multiple sources can
-- coexist (a chapter-sourced span later confirmed by transcript classification)
-- — dedup/precedence is a cut-time concern (overlap-merge), not a schema one.
CREATE TABLE IF NOT EXISTS ad_segments (
id INTEGER PRIMARY KEY,
episode_id INTEGER NOT NULL REFERENCES episodes(id) ON DELETE CASCADE,
start_second REAL NOT NULL,
end_second REAL NOT NULL,
source TEXT NOT NULL,
confidence REAL,
reason TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
CREATE INDEX IF NOT EXISTS idx_episodes_show_pubdate ON episodes (show_id, pubdate);
CREATE INDEX IF NOT EXISTS idx_ad_segments_episode ON ad_segments (episode_id);
"""
@@ -69,6 +100,12 @@ CREATE INDEX IF NOT EXISTS idx_episodes_show_pubdate ON episodes (show_id, pubda
# so they are bolted on here for databases created by older versions.
_MIGRATIONS = (
("episodes", "extracted_at", "ALTER TABLE episodes ADD COLUMN extracted_at TEXT"),
("episodes", "chapters_url", "ALTER TABLE episodes ADD COLUMN chapters_url TEXT"),
("episodes", "chapters_scanned_at", "ALTER TABLE episodes ADD COLUMN chapters_scanned_at TEXT"),
("episodes", "transcript_path", "ALTER TABLE episodes ADD COLUMN transcript_path TEXT"),
("episodes", "llm_detected_at", "ALTER TABLE episodes ADD COLUMN llm_detected_at TEXT"),
("episodes", "cut_path", "ALTER TABLE episodes ADD COLUMN cut_path TEXT"),
("shows", "feed_token", "ALTER TABLE shows ADD COLUMN feed_token TEXT"),
)
@@ -79,11 +116,21 @@ def _migrate(conn: sqlite3.Connection) -> None:
conn.execute(ddl)
def _backfill_feed_tokens(conn: sqlite3.Connection) -> None:
"""Every show needs a feed_token to serve /feed and /audio — including shows
that existed before this column did, not just ones added going forward."""
for row in conn.execute("SELECT id FROM shows WHERE feed_token IS NULL"):
conn.execute(
"UPDATE shows SET feed_token = ? WHERE id = ?", (secrets.token_urlsafe(24), row["id"])
)
def connect(path: str | Path) -> sqlite3.Connection:
conn = sqlite3.connect(path)
conn.row_factory = sqlite3.Row
conn.executescript(SCHEMA)
_migrate(conn)
_backfill_feed_tokens(conn)
conn.commit()
return conn
+26 -6
View File
@@ -16,7 +16,8 @@ import httpx
from .db import utcnow
# Fields compared to decide whether an existing episode row needs an update.
_EPISODE_FIELDS = ("title", "description", "pubdate", "duration_seconds", "audio_url")
_EPISODE_FIELDS = ("title", "description", "pubdate", "duration_seconds", "audio_url",
"chapters_url")
@dataclass
@@ -27,6 +28,7 @@ class ParsedEpisode:
pubdate: str | None
duration_seconds: int | None
audio_url: str | None
chapters_url: str | None
@dataclass
@@ -81,6 +83,22 @@ def _pubdate(entry) -> str | None:
return None
def _chapters_url(entry) -> str | None:
"""Podcasting 2.0 <podcast:chapters url="..."/> if the feed declares it.
feedparser exposes elements from namespaces it doesn't recognize under a
key built from the document's own prefix — feeds using the conventional
"podcast" prefix land in entry["podcast_chapters"]["url"] (confirmed
against feedparser 6.x). Kept as hark's own small helper rather than
imported from adscrub — this is generic RSS parsing, not adscrub's
ad-detection logic, so it doesn't belong on the dependency boundary.
"""
chapters = entry.get("podcast_chapters")
if isinstance(chapters, dict):
return chapters.get("url") or chapters.get("href")
return None
def parse_feed(content: bytes | str) -> ParsedFeed:
parsed = feedparser.parse(content)
episodes = []
@@ -97,6 +115,7 @@ def parse_feed(content: bytes | str) -> ParsedFeed:
pubdate=_pubdate(entry),
duration_seconds=parse_duration(entry.get("itunes_duration")),
audio_url=audio_url,
chapters_url=_chapters_url(entry),
)
)
feed = parsed.feed
@@ -127,11 +146,12 @@ def upsert_episodes(
conn.execute(
"""
INSERT INTO episodes
(show_id, guid, title, description, pubdate, duration_seconds, audio_url)
VALUES (?, ?, ?, ?, ?, ?, ?)
(show_id, guid, title, description, pubdate, duration_seconds,
audio_url, chapters_url)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""",
(show_id, ep.guid, ep.title, ep.description, ep.pubdate,
ep.duration_seconds, ep.audio_url),
ep.duration_seconds, ep.audio_url, ep.chapters_url),
)
inserted += 1
elif any(row[field] != getattr(ep, field) for field in _EPISODE_FIELDS):
@@ -139,11 +159,11 @@ def upsert_episodes(
"""
UPDATE episodes
SET title = ?, description = ?, pubdate = ?, duration_seconds = ?,
audio_url = ?, updated_at = ?
audio_url = ?, chapters_url = ?, updated_at = ?
WHERE id = ?
""",
(ep.title, ep.description, ep.pubdate, ep.duration_seconds,
ep.audio_url, utcnow(), row["id"]),
ep.audio_url, ep.chapters_url, utcnow(), row["id"]),
)
updated += 1
return inserted, updated
+65
View File
@@ -0,0 +1,65 @@
"""Build a cleaned RSS feed (feedgen) pointing at cut_path episodes.
This is the only integration point any podcast player needs: subscribe to
`/feed/<show_id>/<token>` instead of the original feed URL. Episodes with a
cut_path are served locally at `/audio/<episode_id>/<token>.<ext>`; everything
else still points at its original audio_url unchanged — an episode nobody has
cut (no ads found, or not processed yet) doesn't need a local copy at all.
The token gates both routes (see web.py) since a podcast app can't do the
dashboard's cookie-session login — see CLAUDE.md for why this is a per-show
token embedded in the URL rather than either fully open or a second auth
system. This is hark's own file, not imported from adscrub: adscrub's own
feed.py builds against its `feeds`/`feed_id` schema and has no token concept
at all, so there's no reusable piece here beyond feedgen itself (already a
direct hark dependency).
"""
from __future__ import annotations
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from feedgen.feed import FeedGenerator
def _parse_pubdate(value: str | None) -> datetime | None:
if not value:
return None
return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
def build_feed(conn: sqlite3.Connection, show: sqlite3.Row, base_url: str) -> bytes:
fg = FeedGenerator()
fg.title(show["title"] or show["query"])
fg.link(href=show["feed_url"] or f"{base_url}/feed/{show['id']}/{show['feed_token']}",
rel="self")
fg.description(show["description"] or show["title"] or show["query"])
if show["image_url"]:
fg.image(show["image_url"])
episodes = conn.execute(
"SELECT * FROM episodes WHERE show_id = ? ORDER BY pubdate DESC", (show["id"],)
).fetchall()
for ep in episodes:
length = 0
if ep["cut_path"]:
cut_path = Path(ep["cut_path"])
audio_url = f"{base_url}/audio/{ep['id']}/{show['feed_token']}{cut_path.suffix}"
if cut_path.is_file():
length = cut_path.stat().st_size
else:
audio_url = ep["audio_url"]
if not audio_url:
continue # nothing playable to link — skip rather than emit a dead enclosure
fe = fg.add_entry()
fe.id(ep["guid"])
fe.title(ep["title"] or "(untitled)")
fe.description(ep["description"] or "")
pubdate = _parse_pubdate(ep["pubdate"])
if pubdate:
fe.pubDate(pubdate)
fe.enclosure(audio_url, length, "audio/mpeg")
return fg.rss_str(pretty=True)
+98 -9
View File
@@ -1,12 +1,23 @@
"""Web frontend: the cross-show topic index behind a login wall.
"""Web frontend: the cross-show topic index behind a login wall, plus the
ad-stripped podcast feed/audio routes (unauthenticated, token-gated).
Security model follows the influence-registry spec: the whole site is gated
Security model follows the influence-registry spec: the dashboard is gated
by server-side sessions carried in an HttpOnly cookie; only /login, /logout
and /healthz are reachable unauthenticated; fail-closed — with no admin
password and no HARK_ADMIN_TOKEN the site cannot be entered at all.
Passwords are stretched (iterated salted SHA-256) and compared in constant
time; changing the password revokes every session.
/feed/<show_id>/<token> and /audio/<episode_id>/<token>.<ext> are also
unauthenticated, but for a different reason: a podcast app can't do the
dashboard's cookie login. They're gated instead by a per-show random token
(shows.feed_token, compared with secrets.compare_digest) embedded in the URL
— same idea as the tokened private-feed URLs most self-hosted podcast tools
use, not a second login system. The RSS itself is built by podcast_feed.py
(hark's own — adscrub's own feed-building code targets a different schema
and has no token concept); the ad-detection/cutting pipeline that populates
cut_path comes from the adscrub package (see cli.py, pyproject.toml).
Auth state lives in its own SQLite file (auth.db), NOT in hark.db — data
snapshots pushed from the pipeline replace hark.db wholesale and must never
wipe accounts or sessions.
@@ -31,7 +42,7 @@ from http import HTTPStatus
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from . import __version__
from . import __version__, podcast_feed
from .extract import GENRES as GENRES_FILTER
PW_ITERS = 120_000
@@ -241,8 +252,12 @@ def page(title: str, body: str, user: str | None = None) -> str:
# ---------------------------------------------------------------------------
class App:
def __init__(self, db_path: str | Path, auth: Auth, cookie_secure: bool = False):
def __init__(
self, db_path: str | Path, auth: Auth, cookie_secure: bool = False,
base_url: str = "http://localhost:8710",
):
self.db_path = str(db_path)
self.base_url = base_url.rstrip("/")
self.auth = auth
self.cookie_secure = cookie_secure
@@ -720,7 +735,10 @@ class Handler(BaseHTTPRequestHandler):
def respond(self, status: int, body: str, content_type="text/html; charset=utf-8",
extra_headers: dict | None = None):
data = body.encode()
self.respond_bytes(status, body.encode(), content_type, extra_headers)
def respond_bytes(self, status: int, data: bytes, content_type: str,
extra_headers: dict | None = None):
self.send_response(status)
self._security_headers()
self.send_header("Content-Type", content_type)
@@ -769,6 +787,67 @@ class Handler(BaseHTTPRequestHandler):
user["username"],
))
# -- ad-stripped podcast feed/audio (unauthenticated, token-gated) --------
def _plain_404(self) -> None:
self.respond_bytes(404, b"not found", "text/plain; charset=utf-8")
def _serve_feed(self, route: str) -> None:
# route == "/feed/<show_id>/<token>"
parts = route.split("/")
if len(parts) != 4:
return self._plain_404()
try:
show_id = int(parts[2])
except ValueError:
return self._plain_404()
token = parts[3]
conn = self.app.db()
try:
show = conn.execute("SELECT * FROM shows WHERE id = ?", (show_id,)).fetchone()
if show is None or not show["feed_token"] or not secrets.compare_digest(
show["feed_token"], token
):
return self._plain_404()
body = podcast_feed.build_feed(conn, show, self.app.base_url)
finally:
conn.close()
return self.respond_bytes(200, body, "application/rss+xml; charset=utf-8")
def _serve_audio(self, route: str) -> None:
# route == "/audio/<episode_id>/<token>.<ext>"
parts = route.split("/", 3)
if len(parts) != 4:
return self._plain_404()
try:
episode_id = int(parts[2])
except ValueError:
return self._plain_404()
token = parts[3].split(".", 1)[0]
conn = self.app.db()
try:
row = conn.execute(
"""
SELECT e.cut_path, s.feed_token FROM episodes e
JOIN shows s ON s.id = e.show_id
WHERE e.id = ?
""",
(episode_id,),
).fetchone()
finally:
conn.close()
if row is None or not row["feed_token"] or not secrets.compare_digest(
row["feed_token"], token
):
return self._plain_404()
# cut_path is set by adscrub's cut.py, never user-supplied — no
# traversal surface to guard against here, unlike a URL-derived
# filename would need.
cut_path = Path(row["cut_path"]) if row["cut_path"] else None
if cut_path is None or not cut_path.is_file():
return self._plain_404()
return self.respond_bytes(200, cut_path.read_bytes(), "audio/mpeg")
# -- routing -------------------------------------------------------------
def do_GET(self):
@@ -783,6 +862,10 @@ class Handler(BaseHTTPRequestHandler):
return self.respond(200, STYLE, "text/css; charset=utf-8")
if route == "/login":
return self.respond(200, page("login", LOGIN_PAGE.format(err="")))
if route.startswith("/feed/"):
return self._serve_feed(route)
if route.startswith("/audio/"):
return self._serve_audio(route)
user = app.auth.session_user(self.cookie_token())
if user is None:
@@ -857,9 +940,10 @@ class Handler(BaseHTTPRequestHandler):
def make_server(db_path: str | Path, auth_path: str | Path, bind: str = "0.0.0.0:8710",
admin_token: str | None = None, cookie_secure: bool = False) -> ThreadingHTTPServer:
admin_token: str | None = None, cookie_secure: bool = False,
base_url: str = "http://localhost:8710") -> ThreadingHTTPServer:
auth = Auth(auth_path, admin_token=admin_token)
app = App(db_path, auth, cookie_secure=cookie_secure)
app = App(db_path, auth, cookie_secure=cookie_secure, base_url=base_url)
host, _, port = bind.rpartition(":")
try:
port_num = int(port)
@@ -870,8 +954,13 @@ def make_server(db_path: str | Path, auth_path: str | Path, bind: str = "0.0.0.0
def serve(db_path: str | Path, auth_path: str | Path, bind: str, admin_token: str | None,
cookie_secure: bool) -> None:
server = make_server(db_path, auth_path, bind, admin_token, cookie_secure)
cookie_secure: bool, base_url: str = "http://localhost:8710") -> None:
if "localhost" in base_url or "127.0.0.1" in base_url:
print(f"warning: --base-url is {base_url!r} — a podcast player running "
f"anywhere but this exact machine won't be able to reach cut audio "
f"links embedded in the generated feed. Set --base-url/$HARK_BASE_URL "
f"to this host's actual reachable address.")
server = make_server(db_path, auth_path, bind, admin_token, cookie_secure, base_url)
print(f"hark web on http://{bind} (db={db_path}, auth={auth_path})")
if not admin_token:
print("note: no HARK_ADMIN_TOKEN set — login is impossible until a "
+3 -1
View File
@@ -1,5 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
xmlns:podcast="https://podcastindex.org/namespace/1.0">
<channel>
<title>Example Case Show</title>
<link>https://example.com/show</link>
@@ -15,6 +16,7 @@
<description>An unidentified man found on Somerton Beach in 1948.</description>
<pubDate>Wed, 01 Jan 2025 06:00:00 GMT</pubDate>
<itunes:duration>01:02:03</itunes:duration>
<podcast:chapters url="https://example.com/ep1-chapters.json" type="application/json+chapters"/>
<enclosure url="https://example.com/audio/ep1.mp3" length="100" type="audio/mpeg"/>
</item>
<item>
+3 -1
View File
@@ -1,5 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
xmlns:podcast="https://podcastindex.org/namespace/1.0">
<channel>
<title>Example Case Show</title>
<link>https://example.com/show</link>
@@ -23,6 +24,7 @@
<description>An unidentified man found on Somerton Beach in 1948.</description>
<pubDate>Wed, 01 Jan 2025 06:00:00 GMT</pubDate>
<itunes:duration>01:02:03</itunes:duration>
<podcast:chapters url="https://example.com/ep1-chapters.json" type="application/json+chapters"/>
<enclosure url="https://example.com/audio/ep1.mp3" length="100" type="audio/mpeg"/>
</item>
<item>
+170
View File
@@ -1,3 +1,9 @@
import json
from adscrub import cut as ad_cut
from adscrub import detect as ad_detect
from adscrub import transcribe as ad_transcribe
from hark import cli, db
from hark.extract import NullExtractor
@@ -121,3 +127,167 @@ def test_topics_empty_db(tmp_path, capsys):
rc = cli.main(["--db", str(tmp_path / "t.db"), "topics"])
assert rc == 1
assert "hark extract" in capsys.readouterr().err
# --- ad-stripping pipeline (calls straight into the adscrub package) ---
def test_chapters_with_nothing_to_scan_fails(tmp_path, capsys):
rc = cli.main(["--db", str(tmp_path / "t.db"), "chapters"])
assert rc == 1
assert "no episodes" in capsys.readouterr().err
def test_transcribe_with_nothing_pending_fails(tmp_path, capsys):
rc = cli.main(["--db", str(tmp_path / "t.db"), "transcribe"])
assert rc == 1
assert "no episodes pending" in capsys.readouterr().err
def test_transcribe_dry_run_reports_pending(tmp_path, capsys):
path = tmp_path / "t.db"
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
conn.execute("INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'ep', 'http://a/1.mp3')")
conn.commit()
conn.close()
rc = cli.main(["--db", str(path), "transcribe", "--dry-run"])
assert rc == 0
assert "pending episodes: 1" in capsys.readouterr().out
def test_transcribe_success_path_calls_adscrub_directly(tmp_path, capsys, monkeypatch):
path = tmp_path / "t.db"
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
conn.execute("INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'Ep One', 'http://a/1.mp3')")
conn.commit()
conn.close()
def fake_transcribe_episode(conn, ep, client, model_size=None):
conn.execute("UPDATE episodes SET transcript_path = 'x.json' WHERE id = ?", (ep["id"],))
conn.commit()
return "x.json"
# patched on the adscrub module itself -- hark.cli's `ad_transcribe` name
# is the same module object, not a copy, so this is what hark's CLI calls.
monkeypatch.setattr(ad_transcribe, "transcribe_episode", fake_transcribe_episode)
rc = cli.main(["--db", str(path), "transcribe"])
assert rc == 0
out = capsys.readouterr().out
assert "ok Ep One -> x.json" in out
assert "transcribed 1 episode(s) (0 failed, 0 still pending)" in out
def test_detect_ads_with_nothing_pending_fails(tmp_path, capsys):
rc = cli.main(["--db", str(tmp_path / "t.db"), "detect-ads"])
assert rc == 1
assert "no episodes pending" in capsys.readouterr().err
def test_detect_ads_dry_run_reports_pending(tmp_path, capsys):
path = tmp_path / "t.db"
transcript_path = tmp_path / "t.json"
transcript_path.write_text(json.dumps([{"start": 0.0, "end": 1.0, "text": "hi"}]))
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
conn.execute(
"INSERT INTO episodes (show_id, guid, title, transcript_path) VALUES (1, 'g1', 'ep', ?)",
(str(transcript_path),),
)
conn.commit()
conn.close()
rc = cli.main(["--db", str(path), "detect-ads", "--dry-run"])
assert rc == 0
assert "pending episodes: 1" in capsys.readouterr().out
def test_detect_ads_success_path_calls_adscrub_directly(tmp_path, capsys, monkeypatch):
path = tmp_path / "t.db"
transcript_path = tmp_path / "t.json"
transcript_path.write_text(json.dumps(
[{"start": 0.0, "end": 5.0, "text": "a"}, {"start": 5.0, "end": 8.0, "text": "ad"}]
))
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
conn.execute(
"INSERT INTO episodes (show_id, guid, title, transcript_path) VALUES (1, 'g1', 'Ep One', ?)",
(str(transcript_path),),
)
conn.commit()
conn.close()
class FakeMessages:
def parse(self, **kwargs):
class Response:
parsed_output = ad_detect._Detection(
ad_spans=[ad_detect._Span(start_segment=1, end_segment=1, reason="ad")]
)
return Response()
class FakeAnthropic:
def __init__(self):
self.messages = FakeMessages()
import anthropic
monkeypatch.setattr(anthropic, "Anthropic", FakeAnthropic)
rc = cli.main(["--db", str(path), "detect-ads"])
assert rc == 0
out = capsys.readouterr().out
assert "ok Ep One: 1 ad span(s) from transcript" in out
assert "detected across 1 episode(s) (0 failed, 0 still pending)" in out
def test_cut_with_nothing_pending_fails(tmp_path, capsys):
rc = cli.main(["--db", str(tmp_path / "t.db"), "cut"])
assert rc == 1
assert "no episodes pending cutting" in capsys.readouterr().err
def test_cut_dry_run_reports_pending(tmp_path, capsys):
path = tmp_path / "t.db"
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
conn.execute(
"INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'ep', 'http://a/1.mp3')"
)
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
)
conn.commit()
conn.close()
rc = cli.main(["--db", str(path), "cut", "--dry-run"])
assert rc == 0
assert "pending episodes: 1" in capsys.readouterr().out
def test_cut_success_path_calls_adscrub_directly(tmp_path, capsys, monkeypatch):
path = tmp_path / "t.db"
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('Show A')")
conn.execute(
"INSERT INTO episodes (show_id, guid, title, audio_url) VALUES (1, 'g1', 'Ep One', 'http://a/1.mp3')"
)
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
)
conn.commit()
conn.close()
def fake_cut_episode(conn, ep, client, data_dir=None):
conn.execute("UPDATE episodes SET cut_path = 'x.mp3' WHERE id = ?", (ep["id"],))
conn.commit()
return "x.mp3", 5.0
monkeypatch.setattr(ad_cut, "cut_episode", fake_cut_episode)
rc = cli.main(["--db", str(path), "cut"])
assert rc == 0
out = capsys.readouterr().out
assert "ok Ep One: removed 5.0s of ads" in out
assert "cut 1 episode(s) (0 failed, 0 still pending)" in out
+75 -2
View File
@@ -1,3 +1,4 @@
import re
import sqlite3
import pytest
@@ -10,12 +11,23 @@ def conn(tmp_path):
return db.connect(tmp_path / "test.db")
def _schema_without_column(table: str, column: str) -> str:
"""db.SCHEMA with one column's definition line stripped — used to simulate
a pre-migration database. Matches on the column name token, not exact
whitespace, so it doesn't silently no-op if SCHEMA's alignment changes."""
pattern = rf"\n\s*{re.escape(column)}\s+\S+,"
new_schema, n = re.subn(pattern, "", db.SCHEMA, count=1)
assert n == 1, f"{column!r} not found in SCHEMA — test fixture is stale"
return new_schema
def test_connect_creates_tables(conn):
tables = {
row["name"]
for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'")
}
assert {"shows", "episodes", "topics", "topic_genres", "episode_topics"} <= tables
assert {"shows", "episodes", "topics", "topic_genres", "episode_topics",
"ad_segments"} <= tables
def test_connect_is_idempotent(tmp_path):
@@ -43,7 +55,7 @@ def test_migration_adds_extracted_at_to_old_db(tmp_path):
"""A 0.1.0-era database (no extracted_at) gains the column on connect."""
path = tmp_path / "old.db"
old = sqlite3.connect(path)
old.executescript(db.SCHEMA.replace(" extracted_at TEXT,\n", ""))
old.executescript(_schema_without_column("episodes", "extracted_at"))
old.execute("INSERT INTO shows (query) VALUES ('a')")
old.execute("INSERT INTO episodes (show_id, guid) VALUES (1, 'g')")
old.commit()
@@ -54,6 +66,67 @@ def test_migration_adds_extracted_at_to_old_db(tmp_path):
assert row["extracted_at"] is None
@pytest.mark.parametrize("column", [
"chapters_url", "chapters_scanned_at", "transcript_path", "llm_detected_at", "cut_path",
])
def test_migration_adds_ad_pipeline_columns_to_old_episodes(tmp_path, column):
"""A pre-ad-pipeline database gains these columns on connect — the
ad_segments table itself is CREATE TABLE IF NOT EXISTS so it doesn't need
a migration, only columns bolted onto the pre-existing episodes table do."""
path = tmp_path / "old.db"
old = sqlite3.connect(path)
old.executescript(_schema_without_column("episodes", column))
old.execute("INSERT INTO shows (query) VALUES ('a')")
old.execute("INSERT INTO episodes (show_id, guid) VALUES (1, 'g')")
old.commit()
old.close()
conn = db.connect(path)
row = conn.execute(f"SELECT {column} FROM episodes").fetchone()
assert row[column] is None
def test_migration_adds_feed_token_to_old_shows(tmp_path):
path = tmp_path / "old.db"
old = sqlite3.connect(path)
old.executescript(_schema_without_column("shows", "feed_token"))
old.execute("INSERT INTO shows (query) VALUES ('a')")
old.commit()
old.close()
conn = db.connect(path)
row = conn.execute("SELECT feed_token FROM shows").fetchone()
assert row["feed_token"] is not None # backfilled, not just added-and-null
def test_backfill_feed_tokens_gives_every_show_a_unique_token(tmp_path):
# backfill runs at connect() time, same as migrations — a show inserted on
# an already-open connection doesn't get one until the next reconnect,
# which matches real operation (every CLI command opens a fresh connection).
path = tmp_path / "test.db"
conn = db.connect(path)
conn.execute("INSERT INTO shows (query) VALUES ('a')")
conn.execute("INSERT INTO shows (query) VALUES ('b')")
conn.commit()
conn.close()
conn = db.connect(path)
tokens = [row["feed_token"] for row in conn.execute("SELECT feed_token FROM shows")]
assert all(tokens)
assert len(set(tokens)) == 2
def test_ad_segments_cascade_on_episode_delete(conn):
conn.execute("INSERT INTO shows (query) VALUES ('a')")
conn.execute("INSERT INTO episodes (show_id, guid) VALUES (1, 'ep-1')")
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source)"
" VALUES (1, 0, 30, 'chapter')"
)
conn.execute("DELETE FROM episodes WHERE id = 1")
assert conn.execute("SELECT COUNT(*) FROM ad_segments").fetchone()[0] == 0
def test_utcnow_format():
value = db.utcnow()
assert len(value) == 20 and value.endswith("Z") and value[10] == "T"
+4
View File
@@ -47,7 +47,9 @@ def test_parse_feed(fixtures):
assert ep1.pubdate == "2025-01-01T06:00:00Z"
assert ep1.duration_seconds == 3723
assert ep1.audio_url == "https://example.com/audio/ep1.mp3"
assert ep1.chapters_url == "https://example.com/ep1-chapters.json"
assert ep2.duration_seconds == 2700
assert ep2.chapters_url is None
# no <guid> → enclosure URL stands in
assert ep3.guid == "https://example.com/audio/ep3.mp3"
assert ep3.duration_seconds == 1800
@@ -80,6 +82,8 @@ def test_ingest_inserts_then_noop(conn, fixtures):
assert (first.inserted, first.updated, first.total) == (3, 0, 3)
assert (second.inserted, second.updated, second.total) == (0, 0, 3)
assert conn.execute("SELECT COUNT(*) FROM episodes").fetchone()[0] == 3
row = conn.execute("SELECT chapters_url FROM episodes WHERE guid = 'ep-001'").fetchone()
assert row["chapters_url"] == "https://example.com/ep1-chapters.json"
def test_ingest_updates_changed_episodes(conn, fixtures):
+149
View File
@@ -0,0 +1,149 @@
"""Token-gated /feed and /audio routes: real HTTP against a server on an
ephemeral port, same pattern as test_web.py. Kept in its own file since these
routes are deliberately unauthenticated (no cookie login — see web.py's
module docstring) and that's worth testing in isolation from the dashboard's
login-wall behavior.
"""
import http.client
import threading
import pytest
from hark import db, web
TOKEN = "test-token-abc123"
@pytest.fixture
def server(tmp_path):
conn = db.connect(tmp_path / "hark.db")
conn.execute(
"INSERT INTO shows (query, title, feed_url, description, image_url, feed_token)"
" VALUES ('q', 'Show A', 'http://original/feed', 'A show', 'http://original/art.png', ?)",
(TOKEN,),
)
conn.execute(
"INSERT INTO episodes (show_id, guid, title, description, pubdate, audio_url)"
" VALUES (1, 'ep-1', 'Ep 1', 'desc', '2026-01-01T00:00:00Z', 'http://original/ep1.mp3')"
)
conn.commit()
conn.close()
cut_dir = tmp_path / "cut"
cut_dir.mkdir()
cut_path = cut_dir / "2.mp3"
cut_path.write_bytes(b"cut-bytes")
conn = db.connect(tmp_path / "hark.db")
conn.execute(
"INSERT INTO episodes (show_id, guid, title, audio_url, cut_path)"
" VALUES (1, 'ep-2', 'Ep 2', 'http://original/ep2.mp3', ?)",
(str(cut_path),),
)
conn.commit()
conn.close()
srv = web.make_server(
tmp_path / "hark.db", tmp_path / "auth.db", bind="127.0.0.1:0",
admin_token="letmein", base_url="http://myhost:8710",
)
thread = threading.Thread(target=srv.serve_forever, daemon=True)
thread.start()
yield srv
srv.shutdown()
def request(srv, path):
conn = http.client.HTTPConnection("127.0.0.1", srv.server_address[1], timeout=5)
conn.request("GET", path)
resp = conn.getresponse()
data = resp.read()
conn.close()
return resp, data
def test_feed_route_no_login_required(server):
resp, data = request(server, f"/feed/1/{TOKEN}")
assert resp.status == 200
assert resp.getheader("Content-Type") == "application/rss+xml; charset=utf-8"
assert b"Show A" in data
def test_feed_route_wrong_token_404s(server):
resp, _ = request(server, "/feed/1/wrong-token")
assert resp.status == 404
def test_feed_route_unknown_show_404s(server):
resp, _ = request(server, f"/feed/999/{TOKEN}")
assert resp.status == 404
def test_feed_route_passthrough_for_uncut_episode(server):
_resp, data = request(server, f"/feed/1/{TOKEN}")
assert b'url="http://original/ep1.mp3"' in data
def test_feed_route_points_cut_episode_at_local_audio(server):
_resp, data = request(server, f"/feed/1/{TOKEN}")
assert f'url="http://myhost:8710/audio/2/{TOKEN}.mp3"'.encode() in data
def test_audio_route_serves_cut_file_with_correct_token(server):
resp, data = request(server, f"/audio/2/{TOKEN}.mp3")
assert resp.status == 200
assert resp.getheader("Content-Type") == "audio/mpeg"
assert data == b"cut-bytes"
def test_audio_route_wrong_token_404s(server):
resp, _ = request(server, "/audio/2/wrong-token.mp3")
assert resp.status == 404
def test_audio_route_uncut_episode_404s(server):
# ep-1 has no cut_path — nothing to serve locally for it
resp, _ = request(server, f"/audio/1/{TOKEN}.mp3")
assert resp.status == 404
def test_audio_route_unknown_episode_404s(server):
resp, _ = request(server, f"/audio/999/{TOKEN}.mp3")
assert resp.status == 404
def test_unknown_route_falls_through_to_dashboard_login_wall(server):
# confirms /feed and /audio prefix checks don't swallow unrelated routes —
# an unauthenticated request to anything else still hits the normal
# dashboard dispatcher, which redirects to /login (not a 404, since that
# dispatcher's own not_found() is gated behind the session check too).
resp, _ = request(server, "/nonsense")
assert resp.status == 303
assert resp.getheader("Location") == "/login"
class _FakeServer:
def serve_forever(self):
pass
def test_serve_warns_when_base_url_is_localhost(tmp_path, capsys, monkeypatch):
monkeypatch.setattr(web, "make_server", lambda *a, **k: _FakeServer())
web.serve(
tmp_path / "hark.db", tmp_path / "auth.db", "127.0.0.1:0", None, False,
base_url="http://localhost:8710",
)
assert "warning:" in capsys.readouterr().out.lower()
def test_serve_no_warning_for_a_configured_hostname(tmp_path, capsys, monkeypatch):
# note: matching the exact "warning:" prefix (not a bare "warning"
# substring) since pytest's tmp_path embeds the test's own function name,
# and a name containing "warning" would otherwise leak into the printed
# db path and self-sabotage this assertion.
monkeypatch.setattr(web, "make_server", lambda *a, **k: _FakeServer())
web.serve(
tmp_path / "hark.db", tmp_path / "auth.db", "127.0.0.1:0", None, False,
base_url="http://truenas.local:8710",
)
assert "warning:" not in capsys.readouterr().out.lower()