Add M4 cut + serve: end-to-end ad-free feed pipeline (v0.4.0)

adscrub cut: merges overlapping ad spans from any source (chapter, LLM
-- no dedup rule needed, overlap-merging handles it), ffmpeg-extracts
the surviving audio, concatenates with -c copy (no re-encode). Episode
duration comes from ffprobe on the real file, not RSS metadata.

adscrub serve: stdlib http.server (dependency-free, same approach as
hark's web.py) regenerates a cleaned RSS feed live at GET /feed/<id>.
Cut episodes serve locally at /audio/<id>.<ext>; everything else keeps
its original audio_url unchanged -- nothing gets a local copy unless
it was actually cut. No login wall (trusted-network machine-consumed
feed, not a browsable dashboard). Warns instead of silently producing
a broken feed if --base-url is left at the unreachable localhost
default.

Split download_audio/probe_duration out of transcribe.py into a new
audio.py, since cut.py needed the same downloaded-audio cache.

Docker default CMD now runs adscrub serve (port 8711); pipeline stages
stay one-shot docker compose run commands.

Full pipeline (ingest -> chapters -> transcribe -> detect -> cut ->
serve) is now built end-to-end, all mocked/fixture-tested -- real
audio quality and AntennaPod compatibility are still unverified.
This commit is contained in:
flan
2026-07-10 23:57:18 +00:00
parent f06eb0c4b6
commit 016a4ad0c1
17 changed files with 916 additions and 89 deletions
+24
View File
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.4.0] - 2026-07-10
### Added
- Ad cutting (`adscrub cut`): merges overlapping ad spans from any source
(chapter, LLM — no "which source wins" rule needed, overlap-merging handles it),
then `ffmpeg`-extracts the surviving audio and concatenates with `-c copy` (no
re-encode, no quality loss). Episode duration comes from `ffprobe` on the real
file, not RSS metadata.
- Feed serving (`adscrub serve`): stdlib `http.server` (dependency-free, same
approach as hark's web.py), regenerates a cleaned RSS feed live from the DB at
`GET /feed/<id>`. Cut episodes are served locally at `/audio/<id>.<ext>`;
everything else still points at its original `audio_url` — nothing gets a local
copy unless it was actually cut. No login wall (machine-consumed feed on a
trusted network, not a browsable dashboard).
- `--base-url` is required to make sense of generated audio links (embedded in
every cut episode's enclosure); `serve` warns loudly if left at the
unreachable `localhost` default instead of failing silently into a broken feed.
- Docker: default `CMD` now runs `adscrub serve` (port 8711, `restart:
unless-stopped`), matching hark/tiltmeter's long-running-service shape;
pipeline stages remain one-shot `docker compose run --rm` commands.
- Shared `audio.py` module: `download_audio`/`probe_duration`, split out of
transcribe.py since cut.py needed the same downloaded-audio cache.
## [0.3.0] - 2026-07-10
### Added
+10 -3
View File
@@ -63,15 +63,22 @@ a distinct auth/web layer, deployment identity, or feed-registration UX that wou
just get thrown away on merge) — keep it a thin CLI + SQLite + pipeline, matching
hark's own M0/M1 shape, so a later merge is a module import, not a rewrite.
The full pipeline (ingest → chapters → transcribe → detect → cut → serve) is now
built end-to-end as of 0.4.0, which is exactly the trigger PLAN.md's M5 names for
actually making the merge-or-stay-standalone call — **that's the owner's decision, not
something to resolve unprompted.** Don't preemptively start merging repos.
## Conventions
- Python 3.12+, `uv` + `pyproject.toml`, src layout. SQLite for storage. Keep
dependencies minimal; don't add faster-whisper/torch etc. until M2 actually needs them.
dependencies minimal — GPU runtime libs are an optional `gpu` extra rather than a
base dependency (see M2), and there's no torch dependency anywhere (CUDA detection
goes through `ctranslate2.get_cuda_device_count()` instead).
- CHANGELOG.md in Keep a Changelog format; SemVer.
- **No AI/Claude attribution in commit messages** (no Co-Authored-By). Disclose AI use
in the README instead. Commit messages describe actual changes, concise; never
reference prompts or instructions.
- Significant multi-commit features go on a feature branch; small increments can go on
main while the project is pre-0.1.
- Significant multi-commit features go on a feature branch; small increments are fine
directly on main.
- Remote: private Gitea repo `flan/adscrub` (origin, SSH). Do not create additional
remotes or mirrors unprompted.
+8 -4
View File
@@ -1,9 +1,11 @@
# adscrub: pipeline CLI in one image. No web frontend yet (see docs/PLAN.md M4).
# adscrub: pipeline CLI + feed server in one image.
#
# Every pipeline stage is a one-shot command, e.g.:
# Default command serves the cleaned feed(s) over HTTP; every pipeline stage is
# also available as a one-shot command, e.g.:
# docker compose run --rm adscrub ingest
# docker compose run --rm adscrub chapters
# docker compose run --rm adscrub transcribe
# docker compose run --rm adscrub cut
#
# Build with --build-arg GPU=1 (or `docker compose -f compose.yaml -f compose.gpu.yaml
# build`) to pull in the cuBLAS/cuDNN extra for faster-whisper's CUDA path — only
@@ -29,7 +31,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
else uv sync --frozen --no-dev; fi
ENV PATH="/app/.venv/bin:$PATH" \
ADSCRUB_DB=/app/data/adscrub.db
ADSCRUB_DB=/app/data/adscrub.db \
ADSCRUB_DATA_DIR=/app/data
# gosu drops from root to the unprivileged `adscrub` user after the entrypoint fixes
# ownership of /app/data. uid/gid 568 matches TrueNAS SCALE's standard "apps" account,
@@ -43,6 +46,7 @@ COPY docker-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
VOLUME ["/app/data"]
EXPOSE 8711
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["adscrub", "stats"]
CMD ["adscrub", "serve", "--bind", "0.0.0.0:8711"]
+14 -7
View File
@@ -23,9 +23,14 @@ on, and a feed-level proxy works with any podcast app, not just one.
episode/listener, so a fingerprint/crowdsourced-timestamp database (SponsorBlock
style) can't catch them. The model points at transcript segment indices, not
raw timestamps, so stored spans are always grounded in Whisper's own output.
5. **Cut** (not built — M4) — `ffmpeg` out the ad spans, write the clean audio.
6. **Serve** (not built — M4/M5) — re-host a cleaned RSS feed pointing at the cut
audio; this is the only thing the podcast player ever sees.
5. **Cut** (done) — `ffmpeg` extracts the surviving (non-ad) spans and concatenates
them with no re-encoding (`-c copy` — no quality loss). Ad spans from any source
(chapter, LLM) are merged before cutting, so overlapping/duplicate detections
collapse automatically rather than needing a "which source wins" rule.
6. **Serve** (done) — re-hosts a cleaned RSS feed (`GET /feed/<id>`); cut episodes
point at locally-served audio (`/audio/<id>.<ext>`), everything else still points
at its original URL unchanged. This is the only thing the podcast player ever
sees — point AntennaPod at `/feed/<id>` instead of the original feed.
See [docs/PLAN.md](docs/PLAN.md) for the full milestone breakdown.
@@ -38,14 +43,16 @@ uv run adscrub ingest # fetch it, upsert epi
uv run adscrub chapters # scan chapter markers for ad spans
uv run adscrub transcribe # Whisper the rest
uv run adscrub detect # LLM ad-span classification
uv run adscrub cut # ffmpeg out the ad spans
uv run adscrub serve --base-url http://this-host:8711 # serve the cleaned feed(s)
uv run adscrub stats # counts
```
`detect` needs `$ANTHROPIC_API_KEY` set (get it from rbw, not a file — same
convention as hark).
`cut` / `serve` are registered subcommands that report "not built yet" until
their milestones land — see docs/PLAN.md.
convention as hark). `serve`'s `--base-url` must be wherever the podcast player can
actually reach this host — it's embedded in every generated audio link, so
`localhost` only works if the player runs on the same machine (it prints a warning
if left at that default).
Transcription runs CPU-only by default. `code` does have a real GPU (RTX 2070
SUPER) and Docker here has the `nvidia` runtime registered, but that's only wired
+16 -6
View File
@@ -1,12 +1,22 @@
# Pipeline one-shot commands. No web frontend yet (see docs/PLAN.md M4).
# docker compose run --rm adscrub add-feed https://feeds.example.com/show
# docker compose run --rm adscrub ingest
# docker compose run --rm adscrub chapters
# docker compose run --rm adscrub transcribe # CPU by default — see compose.gpu.yaml
# docker compose run --rm adscrub stats
# docker compose up -d serves the feed(s) :8711
# docker compose run --rm adscrub add-feed https://feeds.example.com/show
# docker compose run --rm adscrub ingest
# docker compose run --rm adscrub chapters
# docker compose run --rm adscrub transcribe # CPU by default — see compose.gpu.yaml
# docker compose run --rm adscrub detect
# docker compose run --rm adscrub cut
#
# ADSCRUB_BASE_URL must be set to wherever the podcast player can actually
# reach this container (not localhost) — it's embedded in generated feeds'
# audio links.
services:
adscrub:
build: .
image: adscrub:latest
ports:
- "8711:8711"
environment:
ADSCRUB_BASE_URL: ${ADSCRUB_BASE_URL:-http://localhost:8711}
volumes:
- ./data:/app/data
restart: unless-stopped
+29 -12
View File
@@ -57,27 +57,44 @@ Milestones. Each one ships something usable and gets a CHANGELOG version.
Same fix applied to M1's `chapters_scanned_at` for the same reason (a free
HTTP re-fetch, not a billing bug, but the same defect class).
## M4 — cut + re-hosted feed
## M4 — cut + re-hosted feed (done, 0.4.0)
- `ffmpeg` extraction of the surviving (non-ad) spans, concat back together, write to
`episodes.cut_path`.
- `feedgen`-based feed regeneration: same episodes, `cut_path` audio instead of the
original `audio_url`, served over HTTP. This is the only integration point any
podcast player needs — subscribe to this feed's URL instead of the original.
- Docker deploy on TrueNAS, same shape as hark/tiltmeter (scheduled ingest → pipeline →
serve), once the pipeline actually produces something worth deploying.
- `ffmpeg` extraction of the surviving (non-ad) spans (`cut.compute_keep_spans` merges
overlapping ad_segments from any source, then takes the complement), concat back
together via the concat demuxer (`-c copy` — no re-encode, no quality loss), write to
`episodes.cut_path`. Episode duration comes from `ffprobe` on the actual downloaded
file, not RSS metadata (which is often wrong/missing).
- `feedgen`-based feed regeneration (`adscrub serve`, stdlib `http.server` — dependency
-free by design, same as hark's web.py): `GET /feed/<feed_id>` regenerates the RSS
live from the DB on every request; episodes with a `cut_path` point at
`/audio/<id>.<ext>` (served from `data/cut/`), everything else still points straight
at its original `audio_url` — an episode nobody's cut (no ads found, or not
processed yet) needs no local copy at all. This is the only integration point any
podcast player needs — subscribe to `/feed/<id>` instead of the original feed URL.
- No login wall on the server (unlike hark's dashboard) — this is a machine-consumed
feed on a trusted homelab network, not a browsable UI over someone's listening
habits. Revisit if it ever needs to be reachable from outside a trusted network.
- Docker: default `CMD` now runs `adscrub serve` (long-running, port 8711, matching
hark/tiltmeter's shape); pipeline stages stay one-shot `docker compose run --rm`
commands. `$ADSCRUB_BASE_URL` must be set to wherever the podcast player can actually
reach the container — `serve` prints a warning if left at the `localhost` default.
## M5 — hark module decision
- Once M4 is working end-to-end, decide whether to fold this into `flan/hark` as a
module (shared feed-ingest code, one deployed service) or keep it standalone. Don't
pre-build shared infrastructure for this before M4 proves the pipeline works —
premature merging risks coupling two still-changing pipelines.
- Now that M4 works end-to-end, the actual decision point: fold this into `flan/hark`
as a module (shared feed-ingest code, one deployed service) or keep it standalone?
**Owner call, not decided here** — don't pre-build shared infrastructure before this
is actually decided.
## Open questions (owner input needed, don't block on these)
- M5's hark-merge decision (see above).
- M3 currently defaults to `claude-opus-4-8`; revisit cost vs. accuracy on ad-span
boundaries once it's run against real transcripts (a cheaper model may be plenty
for a fairly mechanical "find the sponsor read" task).
- Real-world validation of the M1 chapters-URL parsing against an actual subscribed
feed, not just the synthetic test fixture.
- Real-world end-to-end validation: this has only been tested with synthetic fixtures
and mocked ffmpeg/Whisper/Claude calls — actual audio quality/timing accuracy after
a real cut, and whether AntennaPod accepts the regenerated feed without complaint,
are both unverified.
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "adscrub"
version = "0.3.0"
version = "0.4.0"
description = "Self-hosted podcast ad-detection and removal proxy"
readme = "README.md"
requires-python = ">=3.12"
+40
View File
@@ -0,0 +1,40 @@
"""Shared audio-file helpers: download + duration probing.
Split out of transcribe.py since both transcribe (M2) and cut (M4) need the
same downloaded-episode-audio cache — neither owns it more than the other.
"""
from __future__ import annotations
import os
import subprocess
from pathlib import Path
import httpx
DEFAULT_DATA_DIR = Path(os.environ.get("ADSCRUB_DATA_DIR", "data"))
def download_audio(client: httpx.Client, audio_url: str, dest: Path) -> Path:
"""Fetch episode audio to dest if not already cached there."""
if dest.exists():
return dest
dest.parent.mkdir(parents=True, exist_ok=True)
tmp = dest.with_suffix(dest.suffix + ".part")
with client.stream("GET", audio_url) as resp:
resp.raise_for_status()
with open(tmp, "wb") as fh:
for chunk in resp.iter_bytes():
fh.write(chunk)
tmp.rename(dest)
return dest
def probe_duration(path: Path) -> float:
"""Audio duration in seconds via ffprobe."""
result = subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1", str(path)],
capture_output=True, text=True, check=True,
)
return float(result.stdout.strip())
+53 -20
View File
@@ -1,4 +1,4 @@
"""adscrub command line: add-feed, ingest, chapters, transcribe, detect, stats (cut/serve: M4+)."""
"""adscrub command line: add-feed, ingest, chapters, transcribe, detect, cut, serve, stats."""
from __future__ import annotations
@@ -8,16 +8,11 @@ import sys
import httpx
from . import __version__, chapters, db, detect, ingest, transcribe
from . import __version__, chapters, cut, db, detect, feed, ingest, transcribe
DEFAULT_DB = os.environ.get("ADSCRUB_DB", "adscrub.db")
USER_AGENT = f"adscrub/{__version__} (homelab podcast ad-removal proxy)"
_NOT_BUILT_YET = {
"cut": "M4",
"serve": "M4/M5",
}
def make_client() -> httpx.Client:
return httpx.Client(
@@ -135,11 +130,43 @@ def cmd_detect(args: argparse.Namespace) -> int:
return 1 if errors else 0
def cmd_cut(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
pending = cut.pending_episodes(conn, args.limit)
if args.dry_run:
total_pending = len(cut.pending_episodes(conn))
print(f"pending episodes: {total_pending}"
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
return 0
if not pending:
print("no episodes pending cutting", file=sys.stderr)
return 1
def report(r: cut.CutResult) -> None:
if r.error:
print(f" FAIL {r.title}: {r.error}")
else:
print(f" ok {r.title}: removed {r.ad_seconds:.1f}s of ads")
with make_client() as client:
results = cut.cut_pending(conn, client, limit=args.limit, on_result=report)
errors = sum(1 for r in results if r.error)
remaining = len(cut.pending_episodes(conn))
print(f"cut {len(results) - errors} episode(s) ({errors} failed, {remaining} still pending)")
return 1 if errors else 0
def cmd_serve(args: argparse.Namespace) -> int:
feed.serve(args.db, args.data_dir, args.base_url, args.bind)
return 0
def cmd_stats(args: argparse.Namespace) -> int:
conn = db.connect(args.db)
feeds = conn.execute("SELECT COUNT(*) FROM feeds").fetchone()[0]
episodes = conn.execute("SELECT COUNT(*) FROM episodes").fetchone()[0]
segments = conn.execute("SELECT COUNT(*) FROM ad_segments").fetchone()[0]
cut_count = conn.execute("SELECT COUNT(*) FROM episodes WHERE cut_path IS NOT NULL").fetchone()[0]
by_source = conn.execute(
"SELECT source, COUNT(*) AS n FROM ad_segments GROUP BY source"
).fetchall()
@@ -148,18 +175,10 @@ def cmd_stats(args: argparse.Namespace) -> int:
print(f"ad_segments: {segments}")
for row in by_source:
print(f" {row['source']:<10} {row['n']}")
print(f"cut: {cut_count}")
return 0
def _cmd_not_built_yet(name: str):
def handler(args: argparse.Namespace) -> int:
milestone = _NOT_BUILT_YET[name]
print(f"`adscrub {name}` is not built yet ({milestone}) — see docs/PLAN.md",
file=sys.stderr)
return 1
return handler
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="adscrub", description="Self-hosted podcast ad-detection and removal proxy."
@@ -200,13 +219,27 @@ def main(argv: list[str] | None = None) -> int:
help="only report how many episodes are pending")
p.set_defaults(func=cmd_detect)
p = sub.add_parser("cut", help="cut ad spans out of episode audio with ffmpeg")
p.add_argument("--limit", type=int, help="max episodes to process this run")
p.add_argument("--dry-run", action="store_true",
help="only report how many episodes are pending")
p.set_defaults(func=cmd_cut)
p = sub.add_parser("serve", help="serve the cleaned feed(s) + cut audio over HTTP")
p.add_argument("--bind", default=os.environ.get("ADSCRUB_BIND", "0.0.0.0:8711"),
help="host:port (default: $ADSCRUB_BIND or 0.0.0.0:8711)")
p.add_argument("--base-url", default=os.environ.get("ADSCRUB_BASE_URL", "http://localhost:8711"),
help="externally-reachable URL this server is served at — embedded in "
"generated feeds' audio links, so it must resolve from wherever the "
"podcast player runs, not just from this host "
"(default: $ADSCRUB_BASE_URL or http://localhost:8711)")
p.add_argument("--data-dir", default=os.environ.get("ADSCRUB_DATA_DIR", "data"),
help="directory holding cut/ audio (default: $ADSCRUB_DATA_DIR or data)")
p.set_defaults(func=cmd_serve)
p = sub.add_parser("stats", help="print database counts")
p.set_defaults(func=cmd_stats)
for name in _NOT_BUILT_YET:
p = sub.add_parser(name, help=f"not built yet ({_NOT_BUILT_YET[name]})")
p.set_defaults(func=_cmd_not_built_yet(name))
args = parser.parse_args(argv)
return args.func(args)
+147 -5
View File
@@ -1,14 +1,156 @@
"""M4: cut ad_segments out of the downloaded audio with ffmpeg.
Not implemented yet. Approach: ffmpeg -ss/-to segment extraction for the
surviving (non-ad) spans, concat-demuxer them back together, write the
result to episodes.cut_path. See docs/PLAN.md M4.
Ad spans can come from more than one source for the same episode (a chapter
marker and an LLM-flagged span might both cover roughly the same ad break, or
overlap partially). Rather than pick a "winning" source, merge overlapping
spans at cut time — this is the same idea PLAN.md flagged in M3 ("dedup is a
pipeline concern, not a schema one"), delivered here.
Approach: ffmpeg -ss/-to stream-copy extraction of each surviving (non-ad)
span, then the concat demuxer glues them back together — no re-encoding, so
no quality loss and no cost proportional to episode length.
"""
from __future__ import annotations
import shutil
import sqlite3
import subprocess
import tempfile
from dataclasses import dataclass
from pathlib import Path
from typing import Callable
import httpx
from .audio import DEFAULT_DATA_DIR, download_audio, probe_duration
from .db import utcnow
def cut_ads(conn: sqlite3.Connection, episode: sqlite3.Row, audio_path: str) -> str:
raise NotImplementedError("M4: audio cutting not built yet — see docs/PLAN.md")
def compute_keep_spans(
ad_spans: list[tuple[float, float]], duration: float
) -> list[tuple[float, float]]:
"""Merge overlapping/adjacent ad spans, then return the complementary spans to keep."""
if not ad_spans:
return [(0.0, duration)]
merged: list[list[float]] = []
for start, end in sorted(ad_spans):
if merged and start <= merged[-1][1]:
merged[-1][1] = max(merged[-1][1], end)
else:
merged.append([start, end])
keep = []
cursor = 0.0
for start, end in merged:
start = max(0.0, min(start, duration))
end = max(0.0, min(end, duration))
if start > cursor:
keep.append((cursor, start))
cursor = max(cursor, end)
if cursor < duration:
keep.append((cursor, duration))
return keep
def cut_audio(audio_path: Path, keep_spans: list[tuple[float, float]], output_path: Path) -> None:
"""Write the audio restricted to keep_spans to output_path."""
output_path.parent.mkdir(parents=True, exist_ok=True)
if len(keep_spans) == 1 and keep_spans[0][0] == 0.0:
shutil.copyfile(audio_path, output_path) # nothing to cut
return
with tempfile.TemporaryDirectory() as tmp:
segment_paths = []
for i, (start, end) in enumerate(keep_spans):
seg_path = Path(tmp) / f"seg_{i}{audio_path.suffix}"
subprocess.run(
["ffmpeg", "-y", "-i", str(audio_path), "-ss", str(start), "-to", str(end),
"-c", "copy", str(seg_path)],
capture_output=True, check=True,
)
segment_paths.append(seg_path)
concat_list = Path(tmp) / "concat.txt"
concat_list.write_text("".join(f"file '{p}'\n" for p in segment_paths))
subprocess.run(
["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", str(concat_list),
"-c", "copy", str(output_path)],
capture_output=True, check=True,
)
@dataclass
class CutResult:
episode_id: int
title: str
ad_seconds: float = 0.0
error: str | None = None
def pending_episodes(conn: sqlite3.Connection, limit: int | None = None) -> list[sqlite3.Row]:
"""Episodes with at least one ad span found, not yet cut."""
query = """
SELECT * FROM episodes
WHERE cut_path IS NULL
AND EXISTS (SELECT 1 FROM ad_segments WHERE episode_id = episodes.id)
ORDER BY id
"""
if limit:
query += " LIMIT ?"
return conn.execute(query, (limit,)).fetchall()
return conn.execute(query).fetchall()
def cut_episode(
conn: sqlite3.Connection,
episode: sqlite3.Row,
client: httpx.Client,
data_dir: Path = DEFAULT_DATA_DIR,
) -> tuple[Path, float]:
"""Download (if needed), cut ad spans out, update the episode row.
Returns (cut_path, ad_seconds_removed).
"""
audio_path = download_audio(
client, episode["audio_url"], data_dir / "audio" / f"{episode['id']}.mp3"
)
duration = probe_duration(audio_path)
ad_spans = [
(row["start_second"], row["end_second"])
for row in conn.execute(
"SELECT start_second, end_second FROM ad_segments WHERE episode_id = ?",
(episode["id"],),
)
]
keep_spans = compute_keep_spans(ad_spans, duration)
ad_seconds = duration - sum(end - start for start, end in keep_spans)
output_path = data_dir / "cut" / f"{episode['id']}{audio_path.suffix}"
cut_audio(audio_path, keep_spans, output_path)
conn.execute(
"UPDATE episodes SET cut_path = ?, updated_at = ? WHERE id = ?",
(str(output_path), utcnow(), episode["id"]),
)
conn.commit()
return output_path, ad_seconds
def cut_pending(
conn: sqlite3.Connection,
client: httpx.Client,
data_dir: Path = DEFAULT_DATA_DIR,
limit: int | None = None,
on_result: Callable[[CutResult], None] | None = None,
) -> list[CutResult]:
results: list[CutResult] = []
for row in pending_episodes(conn, limit):
result = CutResult(episode_id=row["id"], title=row["title"] or "")
try:
_path, ad_seconds = cut_episode(conn, row, client, data_dir)
result.ad_seconds = ad_seconds
except Exception as exc: # noqa: BLE001 — per-episode isolation
result.error = str(exc)
results.append(result)
if on_result:
on_result(result)
return results
+140 -6
View File
@@ -1,15 +1,149 @@
"""M4/M5: re-host a cleaned feed (feedgen) pointing at cut_path episodes.
Not implemented yet. This is the only integration point AntennaPod (or any
podcast app) ever sees: subscribe to this feed's URL instead of the
original — same shape as hark's own "output integration" (custom RSS feeds
subscribed to like any podcast, no app-side changes). See docs/PLAN.md M4-M5.
This is the only integration point any podcast player needs: subscribe to
`/feed/<feed_id>` instead of the original feed URL. Episodes with a cut_path
are served locally at `/audio/<id>.<ext>`; everything else still points at its
original audio_url unchanged — an episode nobody has cut (no ads found, or
not processed yet) doesn't need a local copy at all.
Dependency-free HTTP by design, same as hark's web.py: stdlib http.server, no
framework. Unlike hark's browsable dashboard this has no login wall — it's a
machine-consumed feed for one owner on a homelab network, not a searchable UI
with a filesystem of someone's listening habits behind it. Revisit if this
ever needs to be reachable from outside a trusted network (see docs/PLAN.md).
"""
from __future__ import annotations
import sqlite3
import urllib.parse
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from feedgen.feed import FeedGenerator
from . import __version__, db
def build_feed(conn: sqlite3.Connection, feed_id: int) -> bytes:
raise NotImplementedError("M4/M5: proxy feed generation not built yet — see docs/PLAN.md")
def _parse_pubdate(value: str | None) -> datetime | None:
if not value:
return None
return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
def build_feed(conn: sqlite3.Connection, feed: sqlite3.Row, base_url: str) -> bytes:
fg = FeedGenerator()
fg.title(feed["title"] or feed["source_url"])
fg.link(href=feed["source_url"], rel="self")
fg.description(feed["description"] or feed["title"] or feed["source_url"])
if feed["image_url"]:
fg.image(feed["image_url"])
episodes = conn.execute(
"SELECT * FROM episodes WHERE feed_id = ? ORDER BY pubdate DESC", (feed["id"],)
).fetchall()
for ep in episodes:
length = 0
if ep["cut_path"]:
cut_path = Path(ep["cut_path"])
audio_url = f"{base_url}/audio/{ep['id']}{cut_path.suffix}"
if cut_path.is_file():
length = cut_path.stat().st_size
else:
audio_url = ep["audio_url"]
if not audio_url:
continue # nothing playable to link — skip rather than emit a dead enclosure
fe = fg.add_entry()
fe.id(ep["guid"])
fe.title(ep["title"] or "(untitled)")
fe.description(ep["description"] or "")
pubdate = _parse_pubdate(ep["pubdate"])
if pubdate:
fe.pubDate(pubdate)
fe.enclosure(audio_url, length, "audio/mpeg")
return fg.rss_str(pretty=True)
class Handler(BaseHTTPRequestHandler):
db_path: str
data_dir: Path
base_url: str
server_version = f"adscrub/{__version__}"
def log_message(self, fmt, *args): # quiet access log
pass
def respond(self, status: int, data: bytes, content_type: str) -> None:
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(data)
def not_found(self) -> None:
self.respond(404, b"not found", "text/plain; charset=utf-8")
def do_GET(self):
route = urllib.parse.urlsplit(self.path).path.rstrip("/") or "/"
if route == "/healthz":
return self.respond(200, b"ok", "text/plain; charset=utf-8")
if route.startswith("/feed/"):
try:
feed_id = int(route.rsplit("/", 1)[1])
except ValueError:
return self.not_found()
conn = db.connect(self.db_path)
try:
feed = conn.execute("SELECT * FROM feeds WHERE id = ?", (feed_id,)).fetchone()
if feed is None:
return self.not_found()
body = build_feed(conn, feed, self.base_url)
finally:
conn.close()
return self.respond(200, body, "application/rss+xml; charset=utf-8")
if route.startswith("/audio/"):
# strip any path components the client sent — only ever look inside
# data_dir/cut, never let the URL choose an arbitrary filesystem path
name = Path(route[len("/audio/"):]).name
cut_dir = (self.data_dir / "cut").resolve()
candidate = (cut_dir / name).resolve()
if cut_dir not in candidate.parents or not candidate.is_file():
return self.not_found()
return self.respond(200, candidate.read_bytes(), "audio/mpeg")
return self.not_found()
def make_server(
db_path: str | Path, data_dir: str | Path, base_url: str, bind: str = "0.0.0.0:8711"
) -> ThreadingHTTPServer:
host, _, port = bind.rpartition(":")
try:
port_num = int(port)
except ValueError:
raise SystemExit(f"invalid --bind {bind!r}: expected host:port or :port")
handler = type("BoundHandler", (Handler,), {
"db_path": str(db_path), "data_dir": Path(data_dir), "base_url": base_url.rstrip("/"),
})
return ThreadingHTTPServer((host or "0.0.0.0", port_num), handler)
def serve(db_path: str | Path, data_dir: str | Path, base_url: str, bind: str) -> None:
if "localhost" in base_url or "127.0.0.1" in base_url:
print(f"warning: --base-url is {base_url!r} — a podcast player running "
f"anywhere but this exact machine won't be able to reach cut audio "
f"links embedded in the generated feed. Set --base-url/$ADSCRUB_BASE_URL "
f"to this host's actual reachable address.")
server = make_server(db_path, data_dir, base_url, bind)
print(f"adscrub serving on {bind} (feeds at {base_url.rstrip('/')}/feed/<id>)")
try:
server.serve_forever()
except KeyboardInterrupt:
server.shutdown()
+1 -16
View File
@@ -18,10 +18,10 @@ from pathlib import Path
import httpx
from .audio import DEFAULT_DATA_DIR, download_audio
from .db import utcnow
DEFAULT_MODEL = os.environ.get("ADSCRUB_WHISPER_MODEL", "small")
DEFAULT_DATA_DIR = Path(os.environ.get("ADSCRUB_DATA_DIR", "data"))
_model = None
_model_size = None
@@ -48,21 +48,6 @@ def load_model(model_size: str = DEFAULT_MODEL):
return _model
def download_audio(client: httpx.Client, audio_url: str, dest: Path) -> Path:
"""Fetch episode audio to dest if not already cached there."""
if dest.exists():
return dest
dest.parent.mkdir(parents=True, exist_ok=True)
tmp = dest.with_suffix(dest.suffix + ".part")
with client.stream("GET", audio_url) as resp:
resp.raise_for_status()
with open(tmp, "wb") as fh:
for chunk in resp.iter_bytes():
fh.write(chunk)
tmp.rename(dest)
return dest
def transcribe_episode(
conn: sqlite3.Connection,
episode: sqlite3.Row,
+13
View File
@@ -0,0 +1,13 @@
import subprocess
from adscrub import audio
def test_probe_duration_parses_ffprobe_output(monkeypatch):
def fake_run(cmd, **kwargs):
assert cmd[0] == "ffprobe"
assert cmd[-1] == "/tmp/fake.mp3"
return subprocess.CompletedProcess(cmd, 0, stdout="123.456\n", stderr="")
monkeypatch.setattr(subprocess, "run", fake_run)
assert audio.probe_duration("/tmp/fake.mp3") == 123.456
+52 -8
View File
@@ -1,6 +1,6 @@
import json
from adscrub import cli, db, detect, transcribe
from adscrub import cli, cut, db, detect, transcribe
def test_add_feed_then_stats(tmp_path, capsys):
@@ -37,13 +37,6 @@ def test_chapters_with_nothing_to_scan_fails(tmp_path, capsys):
assert "no episodes" in capsys.readouterr().err
def test_not_built_yet_commands_report_milestone(tmp_path, capsys):
for name, milestone in [
("cut", "M4"), ("serve", "M4/M5"),
]:
rc = cli.main(["--db", str(tmp_path / "t.db"), name])
assert rc == 1
assert milestone in capsys.readouterr().err
def test_transcribe_with_nothing_pending_fails(tmp_path, capsys):
@@ -150,6 +143,57 @@ def test_detect_success_path(tmp_path, capsys, monkeypatch):
assert "detected across 1 episode(s) (0 failed, 0 still pending)" in out
def test_cut_with_nothing_pending_fails(tmp_path, capsys):
rc = cli.main(["--db", str(tmp_path / "t.db"), "cut"])
assert rc == 1
assert "no episodes pending cutting" in capsys.readouterr().err
def test_cut_dry_run_reports_pending(tmp_path, capsys):
path = tmp_path / "t.db"
conn = db.connect(path)
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed')")
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (1, 'g1', 'ep', 'http://a/1.mp3')"
)
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
)
conn.commit()
conn.close()
rc = cli.main(["--db", str(path), "cut", "--dry-run"])
assert rc == 0
assert "pending episodes: 1" in capsys.readouterr().out
def test_cut_success_path(tmp_path, capsys, monkeypatch):
path = tmp_path / "t.db"
conn = db.connect(path)
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed')")
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (1, 'g1', 'Ep One', 'http://a/1.mp3')"
)
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
)
conn.commit()
conn.close()
def fake_cut_episode(conn, ep, client, data_dir=None):
conn.execute("UPDATE episodes SET cut_path = 'x.mp3' WHERE id = ?", (ep["id"],))
conn.commit()
return "x.mp3", 5.0
monkeypatch.setattr(cut, "cut_episode", fake_cut_episode)
rc = cli.main(["--db", str(path), "cut"])
assert rc == 0
out = capsys.readouterr().out
assert "ok Ep One: removed 5.0s of ads" in out
assert "cut 1 episode(s) (0 failed, 0 still pending)" in out
def test_version(capsys):
try:
cli.main(["--version"])
+189
View File
@@ -0,0 +1,189 @@
import subprocess
import httpx
import pytest
from adscrub import cut, db
AUDIO_URL = "https://example.com/audio/ep1.mp3"
@pytest.fixture
def conn(tmp_path):
return db.connect(tmp_path / "test.db")
def seed_episode(conn, ad_spans=()):
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed')")
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (1, 'ep-1', 'Ep 1', ?)",
(AUDIO_URL,),
)
conn.commit()
ep = conn.execute("SELECT * FROM episodes WHERE guid = 'ep-1'").fetchone()
for start, end, source in ad_spans:
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source)"
" VALUES (?, ?, ?, ?)",
(ep["id"], start, end, source),
)
conn.commit()
return conn.execute("SELECT * FROM episodes WHERE guid = 'ep-1'").fetchone()
# --- compute_keep_spans ---
def test_compute_keep_spans_no_ads():
assert cut.compute_keep_spans([], 100.0) == [(0.0, 100.0)]
def test_compute_keep_spans_ad_in_middle():
assert cut.compute_keep_spans([(40.0, 60.0)], 100.0) == [(0.0, 40.0), (60.0, 100.0)]
def test_compute_keep_spans_ad_at_start_and_end():
assert cut.compute_keep_spans([(0.0, 10.0), (90.0, 100.0)], 100.0) == [(10.0, 90.0)]
def test_compute_keep_spans_merges_overlapping_spans_from_different_sources():
# a chapter-sourced span and an llm-sourced span covering roughly the same break
spans = [(40.0, 65.0), (60.0, 70.0)]
assert cut.compute_keep_spans(spans, 100.0) == [(0.0, 40.0), (70.0, 100.0)]
def test_compute_keep_spans_merges_adjacent_spans():
assert cut.compute_keep_spans([(10.0, 20.0), (20.0, 30.0)], 100.0) == [
(0.0, 10.0), (30.0, 100.0)
]
def test_compute_keep_spans_clamps_out_of_range_end():
assert cut.compute_keep_spans([(90.0, 150.0)], 100.0) == [(0.0, 90.0)]
def test_compute_keep_spans_entire_episode_is_ads():
assert cut.compute_keep_spans([(0.0, 100.0)], 100.0) == []
# --- cut_audio ---
def test_cut_audio_no_ads_copies_file_without_invoking_ffmpeg(tmp_path, monkeypatch):
def fail_if_called(*a, **k):
raise AssertionError("ffmpeg should not be invoked when there's nothing to cut")
monkeypatch.setattr(subprocess, "run", fail_if_called)
src = tmp_path / "in.mp3"
src.write_bytes(b"original-audio-bytes")
dest = tmp_path / "out" / "out.mp3"
cut.cut_audio(src, [(0.0, 100.0)], dest)
assert dest.read_bytes() == b"original-audio-bytes"
def test_cut_audio_invokes_ffmpeg_per_segment_then_concat(tmp_path, monkeypatch):
calls = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
# ffmpeg's real job is producing the output file at the end of argv
with open(cmd[-1], "wb") as fh:
fh.write(b"x")
return subprocess.CompletedProcess(cmd, 0)
monkeypatch.setattr(subprocess, "run", fake_run)
src = tmp_path / "in.mp3"
src.write_bytes(b"original")
dest = tmp_path / "out.mp3"
cut.cut_audio(src, [(0.0, 40.0), (60.0, 100.0)], dest)
assert dest.exists()
# 2 segment extractions + 1 concat
assert len(calls) == 3
assert calls[0][:2] == ["ffmpeg", "-y"]
assert "-ss" in calls[0] and "0.0" in calls[0]
assert "-ss" in calls[1] and "60.0" in calls[1]
assert calls[2][3] == "concat"
# --- pending_episodes ---
def test_pending_episodes_requires_ad_segments(conn):
seed_episode(conn) # no ad spans at all
assert cut.pending_episodes(conn) == []
def test_pending_episodes_includes_episode_with_ad_spans(conn):
ep = seed_episode(conn, ad_spans=[(10.0, 20.0, "chapter")])
assert [e["id"] for e in cut.pending_episodes(conn)] == [ep["id"]]
conn.execute("UPDATE episodes SET cut_path = '/x.mp3' WHERE id = ?", (ep["id"],))
conn.commit()
assert cut.pending_episodes(conn) == []
# --- cut_episode / cut_pending ---
def audio_client():
def handler(request):
return httpx.Response(200, content=b"fake-mp3-bytes")
return httpx.Client(transport=httpx.MockTransport(handler))
def fake_cut_audio(audio_path, keep_spans, output_path):
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_bytes(b"cut")
def test_cut_episode_updates_row_and_returns_ad_seconds(conn, tmp_path, monkeypatch):
ep = seed_episode(conn, ad_spans=[(10.0, 20.0, "chapter")])
monkeypatch.setattr(cut, "probe_duration", lambda path: 100.0)
monkeypatch.setattr(cut, "cut_audio", fake_cut_audio)
with audio_client() as client:
path, ad_seconds = cut.cut_episode(conn, ep, client, data_dir=tmp_path)
assert ad_seconds == 10.0
assert path == tmp_path / "cut" / f"{ep['id']}.mp3"
assert path.read_bytes() == b"cut"
row = conn.execute("SELECT cut_path FROM episodes WHERE id = ?", (ep["id"],)).fetchone()
assert row["cut_path"] == str(path)
def test_cut_pending_isolates_per_episode_failures(conn, tmp_path, monkeypatch):
ep1 = seed_episode(conn, ad_spans=[(10.0, 20.0, "chapter")])
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed2')")
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (2, 'ep-2', 'Ep 2', ?)",
(AUDIO_URL,),
)
conn.commit()
ep2 = conn.execute("SELECT * FROM episodes WHERE guid = 'ep-2'").fetchone()
conn.execute(
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (?, 0, 5, 'chapter')",
(ep2["id"],),
)
conn.commit()
def fake_probe_duration(path):
if "2" in str(path):
raise RuntimeError("ffprobe failed")
return 100.0
monkeypatch.setattr(cut, "probe_duration", fake_probe_duration)
monkeypatch.setattr(cut, "cut_audio", fake_cut_audio)
with audio_client() as client:
results = {r.title: r for r in cut.cut_pending(conn, client, data_dir=tmp_path)}
assert results["Ep 1"].error is None
assert results["Ep 2"].error is not None
assert cut.pending_episodes(conn) == [
conn.execute("SELECT * FROM episodes WHERE guid = 'ep-2'").fetchone()
]
+178
View File
@@ -0,0 +1,178 @@
"""feed.py tests: build_feed content, and real HTTP against a server on an
ephemeral port — same pattern as hark's test_web.py."""
import http.client
import threading
import pytest
from adscrub import db, feed
@pytest.fixture
def conn(tmp_path):
conn = db.connect(tmp_path / "test.db")
conn.execute(
"INSERT INTO feeds (source_url, title, description, image_url)"
" VALUES ('http://original/feed', 'Show A', 'A show', 'http://original/art.png')"
)
conn.commit()
return conn
# --- build_feed ---
def test_build_feed_passthrough_for_uncut_episode(conn):
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, description, pubdate, audio_url)"
" VALUES (1, 'ep-1', 'Ep 1', 'desc', '2026-01-01T00:00:00Z', 'http://original/ep1.mp3')"
)
conn.commit()
feed_row = conn.execute("SELECT * FROM feeds WHERE id = 1").fetchone()
xml = feed.build_feed(conn, feed_row, "http://myhost:8711").decode()
assert "<title>Show A</title>" in xml
assert 'url="http://original/ep1.mp3"' in xml
assert "myhost" not in xml # untouched episode keeps its original URL
def test_build_feed_points_cut_episodes_at_local_audio_route(conn, tmp_path):
# cut.py names the file after the episode id (see cut_episode) — build_feed
# reconstructs the same URL from ep['id'] + the stored cut_path's suffix,
# so the fixture's filename here must match episode id 1 to be realistic.
cut_path = tmp_path / "cut" / "1.mp3"
cut_path.parent.mkdir(parents=True)
cut_path.write_bytes(b"cut-audio-bytes")
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, audio_url, cut_path)"
" VALUES (1, 'ep-1', 'Ep 1', 'http://original/ep1.mp3', ?)",
(str(cut_path),),
)
conn.commit()
feed_row = conn.execute("SELECT * FROM feeds WHERE id = 1").fetchone()
xml = feed.build_feed(conn, feed_row, "http://myhost:8711").decode()
assert 'url="http://myhost:8711/audio/1.mp3"' in xml
assert f'length="{len(b"cut-audio-bytes")}"' in xml
assert "original/ep1.mp3" not in xml
def test_build_feed_skips_episode_with_no_audio(conn):
conn.execute("INSERT INTO episodes (feed_id, guid, title) VALUES (1, 'ep-1', 'No audio')")
conn.commit()
feed_row = conn.execute("SELECT * FROM feeds WHERE id = 1").fetchone()
xml = feed.build_feed(conn, feed_row, "http://myhost:8711").decode()
assert "No audio" not in xml
# --- HTTP server ---
@pytest.fixture
def server(tmp_path):
conn = db.connect(tmp_path / "adscrub.db")
conn.execute(
"INSERT INTO feeds (source_url, title) VALUES ('http://original/feed', 'Show A')"
)
conn.execute(
"INSERT INTO episodes (feed_id, guid, title, audio_url)"
" VALUES (1, 'ep-1', 'Ep 1', 'http://original/ep1.mp3')"
)
conn.commit()
conn.close()
cut_dir = tmp_path / "data" / "cut"
cut_dir.mkdir(parents=True)
(cut_dir / "audio.mp3").write_bytes(b"cut-bytes")
srv = feed.make_server(
tmp_path / "adscrub.db", tmp_path / "data", "http://myhost:8711", bind="127.0.0.1:0"
)
thread = threading.Thread(target=srv.serve_forever, daemon=True)
thread.start()
yield srv
srv.shutdown()
def request(srv, path):
conn = http.client.HTTPConnection("127.0.0.1", srv.server_address[1], timeout=5)
conn.request("GET", path)
resp = conn.getresponse()
data = resp.read()
conn.close()
return resp, data
def test_healthz(server):
resp, data = request(server, "/healthz")
assert resp.status == 200
assert data == b"ok"
def test_feed_route_serves_generated_rss(server):
resp, data = request(server, "/feed/1")
assert resp.status == 200
assert resp.getheader("Content-Type") == "application/rss+xml; charset=utf-8"
assert b"Show A" in data
def test_feed_route_unknown_id_404s(server):
resp, _ = request(server, "/feed/999")
assert resp.status == 404
def test_feed_route_non_numeric_id_404s(server):
resp, _ = request(server, "/feed/not-a-number")
assert resp.status == 404
def test_audio_route_serves_cut_file(server):
resp, data = request(server, "/audio/audio.mp3")
assert resp.status == 200
assert resp.getheader("Content-Type") == "audio/mpeg"
assert data == b"cut-bytes"
def test_audio_route_missing_file_404s(server):
resp, _ = request(server, "/audio/nope.mp3")
assert resp.status == 404
def test_audio_route_rejects_path_traversal(server):
# Path(...).name strips any directory components regardless of how many
# ".." segments precede the final part, so this can never escape cut_dir.
resp, data = request(server, "/audio/../../../../etc/passwd")
assert resp.status == 404
assert b"root:" not in data
def test_audio_route_rejects_encoded_path_traversal(server):
resp, data = request(server, "/audio/..%2F..%2F..%2Fetc%2Fpasswd")
assert resp.status == 404
assert b"root:" not in data
def test_unknown_route_404s(server):
resp, _ = request(server, "/nonsense")
assert resp.status == 404
# --- serve() base_url warning ---
class _FakeServer:
def serve_forever(self):
pass
def test_serve_warns_when_base_url_is_localhost(tmp_path, capsys, monkeypatch):
monkeypatch.setattr(feed, "make_server", lambda *a, **k: _FakeServer())
feed.serve(tmp_path / "db", tmp_path / "data", "http://localhost:8711", "127.0.0.1:0")
assert "warning" in capsys.readouterr().out.lower()
def test_serve_no_warning_for_a_real_hostname(tmp_path, capsys, monkeypatch):
monkeypatch.setattr(feed, "make_server", lambda *a, **k: _FakeServer())
feed.serve(tmp_path / "db", tmp_path / "data", "http://truenas.local:8711", "127.0.0.1:0")
assert "warning" not in capsys.readouterr().out.lower()
Generated
+1 -1
View File
@@ -4,7 +4,7 @@ requires-python = ">=3.12"
[[package]]
name = "adscrub"
version = "0.3.0"
version = "0.4.0"
source = { editable = "." }
dependencies = [
{ name = "anthropic" },