Add M4 cut + serve: end-to-end ad-free feed pipeline (v0.4.0)
adscrub cut: merges overlapping ad spans from any source (chapter, LLM -- no dedup rule needed, overlap-merging handles it), ffmpeg-extracts the surviving audio, concatenates with -c copy (no re-encode). Episode duration comes from ffprobe on the real file, not RSS metadata. adscrub serve: stdlib http.server (dependency-free, same approach as hark's web.py) regenerates a cleaned RSS feed live at GET /feed/<id>. Cut episodes serve locally at /audio/<id>.<ext>; everything else keeps its original audio_url unchanged -- nothing gets a local copy unless it was actually cut. No login wall (trusted-network machine-consumed feed, not a browsable dashboard). Warns instead of silently producing a broken feed if --base-url is left at the unreachable localhost default. Split download_audio/probe_duration out of transcribe.py into a new audio.py, since cut.py needed the same downloaded-audio cache. Docker default CMD now runs adscrub serve (port 8711); pipeline stages stay one-shot docker compose run commands. Full pipeline (ingest -> chapters -> transcribe -> detect -> cut -> serve) is now built end-to-end, all mocked/fixture-tested -- real audio quality and AntennaPod compatibility are still unverified.
This commit is contained in:
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.4.0] - 2026-07-10
|
||||
|
||||
### Added
|
||||
|
||||
- Ad cutting (`adscrub cut`): merges overlapping ad spans from any source
|
||||
(chapter, LLM — no "which source wins" rule needed, overlap-merging handles it),
|
||||
then `ffmpeg`-extracts the surviving audio and concatenates with `-c copy` (no
|
||||
re-encode, no quality loss). Episode duration comes from `ffprobe` on the real
|
||||
file, not RSS metadata.
|
||||
- Feed serving (`adscrub serve`): stdlib `http.server` (dependency-free, same
|
||||
approach as hark's web.py), regenerates a cleaned RSS feed live from the DB at
|
||||
`GET /feed/<id>`. Cut episodes are served locally at `/audio/<id>.<ext>`;
|
||||
everything else still points at its original `audio_url` — nothing gets a local
|
||||
copy unless it was actually cut. No login wall (machine-consumed feed on a
|
||||
trusted network, not a browsable dashboard).
|
||||
- `--base-url` is required to make sense of generated audio links (embedded in
|
||||
every cut episode's enclosure); `serve` warns loudly if left at the
|
||||
unreachable `localhost` default instead of failing silently into a broken feed.
|
||||
- Docker: default `CMD` now runs `adscrub serve` (port 8711, `restart:
|
||||
unless-stopped`), matching hark/tiltmeter's long-running-service shape;
|
||||
pipeline stages remain one-shot `docker compose run --rm` commands.
|
||||
- Shared `audio.py` module: `download_audio`/`probe_duration`, split out of
|
||||
transcribe.py since cut.py needed the same downloaded-audio cache.
|
||||
|
||||
## [0.3.0] - 2026-07-10
|
||||
|
||||
### Added
|
||||
|
||||
@@ -63,15 +63,22 @@ a distinct auth/web layer, deployment identity, or feed-registration UX that wou
|
||||
just get thrown away on merge) — keep it a thin CLI + SQLite + pipeline, matching
|
||||
hark's own M0/M1 shape, so a later merge is a module import, not a rewrite.
|
||||
|
||||
The full pipeline (ingest → chapters → transcribe → detect → cut → serve) is now
|
||||
built end-to-end as of 0.4.0, which is exactly the trigger PLAN.md's M5 names for
|
||||
actually making the merge-or-stay-standalone call — **that's the owner's decision, not
|
||||
something to resolve unprompted.** Don't preemptively start merging repos.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Python 3.12+, `uv` + `pyproject.toml`, src layout. SQLite for storage. Keep
|
||||
dependencies minimal; don't add faster-whisper/torch etc. until M2 actually needs them.
|
||||
dependencies minimal — GPU runtime libs are an optional `gpu` extra rather than a
|
||||
base dependency (see M2), and there's no torch dependency anywhere (CUDA detection
|
||||
goes through `ctranslate2.get_cuda_device_count()` instead).
|
||||
- CHANGELOG.md in Keep a Changelog format; SemVer.
|
||||
- **No AI/Claude attribution in commit messages** (no Co-Authored-By). Disclose AI use
|
||||
in the README instead. Commit messages describe actual changes, concise; never
|
||||
reference prompts or instructions.
|
||||
- Significant multi-commit features go on a feature branch; small increments can go on
|
||||
main while the project is pre-0.1.
|
||||
- Significant multi-commit features go on a feature branch; small increments are fine
|
||||
directly on main.
|
||||
- Remote: private Gitea repo `flan/adscrub` (origin, SSH). Do not create additional
|
||||
remotes or mirrors unprompted.
|
||||
|
||||
+8
-4
@@ -1,9 +1,11 @@
|
||||
# adscrub: pipeline CLI in one image. No web frontend yet (see docs/PLAN.md M4).
|
||||
# adscrub: pipeline CLI + feed server in one image.
|
||||
#
|
||||
# Every pipeline stage is a one-shot command, e.g.:
|
||||
# Default command serves the cleaned feed(s) over HTTP; every pipeline stage is
|
||||
# also available as a one-shot command, e.g.:
|
||||
# docker compose run --rm adscrub ingest
|
||||
# docker compose run --rm adscrub chapters
|
||||
# docker compose run --rm adscrub transcribe
|
||||
# docker compose run --rm adscrub cut
|
||||
#
|
||||
# Build with --build-arg GPU=1 (or `docker compose -f compose.yaml -f compose.gpu.yaml
|
||||
# build`) to pull in the cuBLAS/cuDNN extra for faster-whisper's CUDA path — only
|
||||
@@ -29,7 +31,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
else uv sync --frozen --no-dev; fi
|
||||
|
||||
ENV PATH="/app/.venv/bin:$PATH" \
|
||||
ADSCRUB_DB=/app/data/adscrub.db
|
||||
ADSCRUB_DB=/app/data/adscrub.db \
|
||||
ADSCRUB_DATA_DIR=/app/data
|
||||
|
||||
# gosu drops from root to the unprivileged `adscrub` user after the entrypoint fixes
|
||||
# ownership of /app/data. uid/gid 568 matches TrueNAS SCALE's standard "apps" account,
|
||||
@@ -43,6 +46,7 @@ COPY docker-entrypoint.sh /usr/local/bin/
|
||||
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
|
||||
|
||||
VOLUME ["/app/data"]
|
||||
EXPOSE 8711
|
||||
|
||||
ENTRYPOINT ["docker-entrypoint.sh"]
|
||||
CMD ["adscrub", "stats"]
|
||||
CMD ["adscrub", "serve", "--bind", "0.0.0.0:8711"]
|
||||
|
||||
@@ -23,9 +23,14 @@ on, and a feed-level proxy works with any podcast app, not just one.
|
||||
episode/listener, so a fingerprint/crowdsourced-timestamp database (SponsorBlock
|
||||
style) can't catch them. The model points at transcript segment indices, not
|
||||
raw timestamps, so stored spans are always grounded in Whisper's own output.
|
||||
5. **Cut** (not built — M4) — `ffmpeg` out the ad spans, write the clean audio.
|
||||
6. **Serve** (not built — M4/M5) — re-host a cleaned RSS feed pointing at the cut
|
||||
audio; this is the only thing the podcast player ever sees.
|
||||
5. **Cut** (done) — `ffmpeg` extracts the surviving (non-ad) spans and concatenates
|
||||
them with no re-encoding (`-c copy` — no quality loss). Ad spans from any source
|
||||
(chapter, LLM) are merged before cutting, so overlapping/duplicate detections
|
||||
collapse automatically rather than needing a "which source wins" rule.
|
||||
6. **Serve** (done) — re-hosts a cleaned RSS feed (`GET /feed/<id>`); cut episodes
|
||||
point at locally-served audio (`/audio/<id>.<ext>`), everything else still points
|
||||
at its original URL unchanged. This is the only thing the podcast player ever
|
||||
sees — point AntennaPod at `/feed/<id>` instead of the original feed.
|
||||
|
||||
See [docs/PLAN.md](docs/PLAN.md) for the full milestone breakdown.
|
||||
|
||||
@@ -38,14 +43,16 @@ uv run adscrub ingest # fetch it, upsert epi
|
||||
uv run adscrub chapters # scan chapter markers for ad spans
|
||||
uv run adscrub transcribe # Whisper the rest
|
||||
uv run adscrub detect # LLM ad-span classification
|
||||
uv run adscrub cut # ffmpeg out the ad spans
|
||||
uv run adscrub serve --base-url http://this-host:8711 # serve the cleaned feed(s)
|
||||
uv run adscrub stats # counts
|
||||
```
|
||||
|
||||
`detect` needs `$ANTHROPIC_API_KEY` set (get it from rbw, not a file — same
|
||||
convention as hark).
|
||||
|
||||
`cut` / `serve` are registered subcommands that report "not built yet" until
|
||||
their milestones land — see docs/PLAN.md.
|
||||
convention as hark). `serve`'s `--base-url` must be wherever the podcast player can
|
||||
actually reach this host — it's embedded in every generated audio link, so
|
||||
`localhost` only works if the player runs on the same machine (it prints a warning
|
||||
if left at that default).
|
||||
|
||||
Transcription runs CPU-only by default. `code` does have a real GPU (RTX 2070
|
||||
SUPER) and Docker here has the `nvidia` runtime registered, but that's only wired
|
||||
|
||||
+16
-6
@@ -1,12 +1,22 @@
|
||||
# Pipeline one-shot commands. No web frontend yet (see docs/PLAN.md M4).
|
||||
# docker compose run --rm adscrub add-feed https://feeds.example.com/show
|
||||
# docker compose run --rm adscrub ingest
|
||||
# docker compose run --rm adscrub chapters
|
||||
# docker compose run --rm adscrub transcribe # CPU by default — see compose.gpu.yaml
|
||||
# docker compose run --rm adscrub stats
|
||||
# docker compose up -d serves the feed(s) :8711
|
||||
# docker compose run --rm adscrub add-feed https://feeds.example.com/show
|
||||
# docker compose run --rm adscrub ingest
|
||||
# docker compose run --rm adscrub chapters
|
||||
# docker compose run --rm adscrub transcribe # CPU by default — see compose.gpu.yaml
|
||||
# docker compose run --rm adscrub detect
|
||||
# docker compose run --rm adscrub cut
|
||||
#
|
||||
# ADSCRUB_BASE_URL must be set to wherever the podcast player can actually
|
||||
# reach this container (not localhost) — it's embedded in generated feeds'
|
||||
# audio links.
|
||||
services:
|
||||
adscrub:
|
||||
build: .
|
||||
image: adscrub:latest
|
||||
ports:
|
||||
- "8711:8711"
|
||||
environment:
|
||||
ADSCRUB_BASE_URL: ${ADSCRUB_BASE_URL:-http://localhost:8711}
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
restart: unless-stopped
|
||||
|
||||
+29
-12
@@ -57,27 +57,44 @@ Milestones. Each one ships something usable and gets a CHANGELOG version.
|
||||
Same fix applied to M1's `chapters_scanned_at` for the same reason (a free
|
||||
HTTP re-fetch, not a billing bug, but the same defect class).
|
||||
|
||||
## M4 — cut + re-hosted feed
|
||||
## M4 — cut + re-hosted feed (done, 0.4.0)
|
||||
|
||||
- `ffmpeg` extraction of the surviving (non-ad) spans, concat back together, write to
|
||||
`episodes.cut_path`.
|
||||
- `feedgen`-based feed regeneration: same episodes, `cut_path` audio instead of the
|
||||
original `audio_url`, served over HTTP. This is the only integration point any
|
||||
podcast player needs — subscribe to this feed's URL instead of the original.
|
||||
- Docker deploy on TrueNAS, same shape as hark/tiltmeter (scheduled ingest → pipeline →
|
||||
serve), once the pipeline actually produces something worth deploying.
|
||||
- `ffmpeg` extraction of the surviving (non-ad) spans (`cut.compute_keep_spans` merges
|
||||
overlapping ad_segments from any source, then takes the complement), concat back
|
||||
together via the concat demuxer (`-c copy` — no re-encode, no quality loss), write to
|
||||
`episodes.cut_path`. Episode duration comes from `ffprobe` on the actual downloaded
|
||||
file, not RSS metadata (which is often wrong/missing).
|
||||
- `feedgen`-based feed regeneration (`adscrub serve`, stdlib `http.server` — dependency
|
||||
-free by design, same as hark's web.py): `GET /feed/<feed_id>` regenerates the RSS
|
||||
live from the DB on every request; episodes with a `cut_path` point at
|
||||
`/audio/<id>.<ext>` (served from `data/cut/`), everything else still points straight
|
||||
at its original `audio_url` — an episode nobody's cut (no ads found, or not
|
||||
processed yet) needs no local copy at all. This is the only integration point any
|
||||
podcast player needs — subscribe to `/feed/<id>` instead of the original feed URL.
|
||||
- No login wall on the server (unlike hark's dashboard) — this is a machine-consumed
|
||||
feed on a trusted homelab network, not a browsable UI over someone's listening
|
||||
habits. Revisit if it ever needs to be reachable from outside a trusted network.
|
||||
- Docker: default `CMD` now runs `adscrub serve` (long-running, port 8711, matching
|
||||
hark/tiltmeter's shape); pipeline stages stay one-shot `docker compose run --rm`
|
||||
commands. `$ADSCRUB_BASE_URL` must be set to wherever the podcast player can actually
|
||||
reach the container — `serve` prints a warning if left at the `localhost` default.
|
||||
|
||||
## M5 — hark module decision
|
||||
|
||||
- Once M4 is working end-to-end, decide whether to fold this into `flan/hark` as a
|
||||
module (shared feed-ingest code, one deployed service) or keep it standalone. Don't
|
||||
pre-build shared infrastructure for this before M4 proves the pipeline works —
|
||||
premature merging risks coupling two still-changing pipelines.
|
||||
- Now that M4 works end-to-end, the actual decision point: fold this into `flan/hark`
|
||||
as a module (shared feed-ingest code, one deployed service) or keep it standalone?
|
||||
**Owner call, not decided here** — don't pre-build shared infrastructure before this
|
||||
is actually decided.
|
||||
|
||||
## Open questions (owner input needed, don't block on these)
|
||||
|
||||
- M5's hark-merge decision (see above).
|
||||
- M3 currently defaults to `claude-opus-4-8`; revisit cost vs. accuracy on ad-span
|
||||
boundaries once it's run against real transcripts (a cheaper model may be plenty
|
||||
for a fairly mechanical "find the sponsor read" task).
|
||||
- Real-world validation of the M1 chapters-URL parsing against an actual subscribed
|
||||
feed, not just the synthetic test fixture.
|
||||
- Real-world end-to-end validation: this has only been tested with synthetic fixtures
|
||||
and mocked ffmpeg/Whisper/Claude calls — actual audio quality/timing accuracy after
|
||||
a real cut, and whether AntennaPod accepts the regenerated feed without complaint,
|
||||
are both unverified.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "adscrub"
|
||||
version = "0.3.0"
|
||||
version = "0.4.0"
|
||||
description = "Self-hosted podcast ad-detection and removal proxy"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Shared audio-file helpers: download + duration probing.
|
||||
|
||||
Split out of transcribe.py since both transcribe (M2) and cut (M4) need the
|
||||
same downloaded-episode-audio cache — neither owns it more than the other.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
DEFAULT_DATA_DIR = Path(os.environ.get("ADSCRUB_DATA_DIR", "data"))
|
||||
|
||||
|
||||
def download_audio(client: httpx.Client, audio_url: str, dest: Path) -> Path:
|
||||
"""Fetch episode audio to dest if not already cached there."""
|
||||
if dest.exists():
|
||||
return dest
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
||||
with client.stream("GET", audio_url) as resp:
|
||||
resp.raise_for_status()
|
||||
with open(tmp, "wb") as fh:
|
||||
for chunk in resp.iter_bytes():
|
||||
fh.write(chunk)
|
||||
tmp.rename(dest)
|
||||
return dest
|
||||
|
||||
|
||||
def probe_duration(path: Path) -> float:
|
||||
"""Audio duration in seconds via ffprobe."""
|
||||
result = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1", str(path)],
|
||||
capture_output=True, text=True, check=True,
|
||||
)
|
||||
return float(result.stdout.strip())
|
||||
+53
-20
@@ -1,4 +1,4 @@
|
||||
"""adscrub command line: add-feed, ingest, chapters, transcribe, detect, stats (cut/serve: M4+)."""
|
||||
"""adscrub command line: add-feed, ingest, chapters, transcribe, detect, cut, serve, stats."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -8,16 +8,11 @@ import sys
|
||||
|
||||
import httpx
|
||||
|
||||
from . import __version__, chapters, db, detect, ingest, transcribe
|
||||
from . import __version__, chapters, cut, db, detect, feed, ingest, transcribe
|
||||
|
||||
DEFAULT_DB = os.environ.get("ADSCRUB_DB", "adscrub.db")
|
||||
USER_AGENT = f"adscrub/{__version__} (homelab podcast ad-removal proxy)"
|
||||
|
||||
_NOT_BUILT_YET = {
|
||||
"cut": "M4",
|
||||
"serve": "M4/M5",
|
||||
}
|
||||
|
||||
|
||||
def make_client() -> httpx.Client:
|
||||
return httpx.Client(
|
||||
@@ -135,11 +130,43 @@ def cmd_detect(args: argparse.Namespace) -> int:
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
def cmd_cut(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
pending = cut.pending_episodes(conn, args.limit)
|
||||
if args.dry_run:
|
||||
total_pending = len(cut.pending_episodes(conn))
|
||||
print(f"pending episodes: {total_pending}"
|
||||
+ (f" (would process {len(pending)} this run)" if args.limit else ""))
|
||||
return 0
|
||||
if not pending:
|
||||
print("no episodes pending cutting", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
def report(r: cut.CutResult) -> None:
|
||||
if r.error:
|
||||
print(f" FAIL {r.title}: {r.error}")
|
||||
else:
|
||||
print(f" ok {r.title}: removed {r.ad_seconds:.1f}s of ads")
|
||||
|
||||
with make_client() as client:
|
||||
results = cut.cut_pending(conn, client, limit=args.limit, on_result=report)
|
||||
errors = sum(1 for r in results if r.error)
|
||||
remaining = len(cut.pending_episodes(conn))
|
||||
print(f"cut {len(results) - errors} episode(s) ({errors} failed, {remaining} still pending)")
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
def cmd_serve(args: argparse.Namespace) -> int:
|
||||
feed.serve(args.db, args.data_dir, args.base_url, args.bind)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_stats(args: argparse.Namespace) -> int:
|
||||
conn = db.connect(args.db)
|
||||
feeds = conn.execute("SELECT COUNT(*) FROM feeds").fetchone()[0]
|
||||
episodes = conn.execute("SELECT COUNT(*) FROM episodes").fetchone()[0]
|
||||
segments = conn.execute("SELECT COUNT(*) FROM ad_segments").fetchone()[0]
|
||||
cut_count = conn.execute("SELECT COUNT(*) FROM episodes WHERE cut_path IS NOT NULL").fetchone()[0]
|
||||
by_source = conn.execute(
|
||||
"SELECT source, COUNT(*) AS n FROM ad_segments GROUP BY source"
|
||||
).fetchall()
|
||||
@@ -148,18 +175,10 @@ def cmd_stats(args: argparse.Namespace) -> int:
|
||||
print(f"ad_segments: {segments}")
|
||||
for row in by_source:
|
||||
print(f" {row['source']:<10} {row['n']}")
|
||||
print(f"cut: {cut_count}")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_not_built_yet(name: str):
|
||||
def handler(args: argparse.Namespace) -> int:
|
||||
milestone = _NOT_BUILT_YET[name]
|
||||
print(f"`adscrub {name}` is not built yet ({milestone}) — see docs/PLAN.md",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
return handler
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="adscrub", description="Self-hosted podcast ad-detection and removal proxy."
|
||||
@@ -200,13 +219,27 @@ def main(argv: list[str] | None = None) -> int:
|
||||
help="only report how many episodes are pending")
|
||||
p.set_defaults(func=cmd_detect)
|
||||
|
||||
p = sub.add_parser("cut", help="cut ad spans out of episode audio with ffmpeg")
|
||||
p.add_argument("--limit", type=int, help="max episodes to process this run")
|
||||
p.add_argument("--dry-run", action="store_true",
|
||||
help="only report how many episodes are pending")
|
||||
p.set_defaults(func=cmd_cut)
|
||||
|
||||
p = sub.add_parser("serve", help="serve the cleaned feed(s) + cut audio over HTTP")
|
||||
p.add_argument("--bind", default=os.environ.get("ADSCRUB_BIND", "0.0.0.0:8711"),
|
||||
help="host:port (default: $ADSCRUB_BIND or 0.0.0.0:8711)")
|
||||
p.add_argument("--base-url", default=os.environ.get("ADSCRUB_BASE_URL", "http://localhost:8711"),
|
||||
help="externally-reachable URL this server is served at — embedded in "
|
||||
"generated feeds' audio links, so it must resolve from wherever the "
|
||||
"podcast player runs, not just from this host "
|
||||
"(default: $ADSCRUB_BASE_URL or http://localhost:8711)")
|
||||
p.add_argument("--data-dir", default=os.environ.get("ADSCRUB_DATA_DIR", "data"),
|
||||
help="directory holding cut/ audio (default: $ADSCRUB_DATA_DIR or data)")
|
||||
p.set_defaults(func=cmd_serve)
|
||||
|
||||
p = sub.add_parser("stats", help="print database counts")
|
||||
p.set_defaults(func=cmd_stats)
|
||||
|
||||
for name in _NOT_BUILT_YET:
|
||||
p = sub.add_parser(name, help=f"not built yet ({_NOT_BUILT_YET[name]})")
|
||||
p.set_defaults(func=_cmd_not_built_yet(name))
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
return args.func(args)
|
||||
|
||||
|
||||
+147
-5
@@ -1,14 +1,156 @@
|
||||
"""M4: cut ad_segments out of the downloaded audio with ffmpeg.
|
||||
|
||||
Not implemented yet. Approach: ffmpeg -ss/-to segment extraction for the
|
||||
surviving (non-ad) spans, concat-demuxer them back together, write the
|
||||
result to episodes.cut_path. See docs/PLAN.md M4.
|
||||
Ad spans can come from more than one source for the same episode (a chapter
|
||||
marker and an LLM-flagged span might both cover roughly the same ad break, or
|
||||
overlap partially). Rather than pick a "winning" source, merge overlapping
|
||||
spans at cut time — this is the same idea PLAN.md flagged in M3 ("dedup is a
|
||||
pipeline concern, not a schema one"), delivered here.
|
||||
|
||||
Approach: ffmpeg -ss/-to stream-copy extraction of each surviving (non-ad)
|
||||
span, then the concat demuxer glues them back together — no re-encoding, so
|
||||
no quality loss and no cost proportional to episode length.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
import httpx
|
||||
|
||||
from .audio import DEFAULT_DATA_DIR, download_audio, probe_duration
|
||||
from .db import utcnow
|
||||
|
||||
|
||||
def cut_ads(conn: sqlite3.Connection, episode: sqlite3.Row, audio_path: str) -> str:
|
||||
raise NotImplementedError("M4: audio cutting not built yet — see docs/PLAN.md")
|
||||
def compute_keep_spans(
|
||||
ad_spans: list[tuple[float, float]], duration: float
|
||||
) -> list[tuple[float, float]]:
|
||||
"""Merge overlapping/adjacent ad spans, then return the complementary spans to keep."""
|
||||
if not ad_spans:
|
||||
return [(0.0, duration)]
|
||||
merged: list[list[float]] = []
|
||||
for start, end in sorted(ad_spans):
|
||||
if merged and start <= merged[-1][1]:
|
||||
merged[-1][1] = max(merged[-1][1], end)
|
||||
else:
|
||||
merged.append([start, end])
|
||||
|
||||
keep = []
|
||||
cursor = 0.0
|
||||
for start, end in merged:
|
||||
start = max(0.0, min(start, duration))
|
||||
end = max(0.0, min(end, duration))
|
||||
if start > cursor:
|
||||
keep.append((cursor, start))
|
||||
cursor = max(cursor, end)
|
||||
if cursor < duration:
|
||||
keep.append((cursor, duration))
|
||||
return keep
|
||||
|
||||
|
||||
def cut_audio(audio_path: Path, keep_spans: list[tuple[float, float]], output_path: Path) -> None:
|
||||
"""Write the audio restricted to keep_spans to output_path."""
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if len(keep_spans) == 1 and keep_spans[0][0] == 0.0:
|
||||
shutil.copyfile(audio_path, output_path) # nothing to cut
|
||||
return
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
segment_paths = []
|
||||
for i, (start, end) in enumerate(keep_spans):
|
||||
seg_path = Path(tmp) / f"seg_{i}{audio_path.suffix}"
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-i", str(audio_path), "-ss", str(start), "-to", str(end),
|
||||
"-c", "copy", str(seg_path)],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
segment_paths.append(seg_path)
|
||||
concat_list = Path(tmp) / "concat.txt"
|
||||
concat_list.write_text("".join(f"file '{p}'\n" for p in segment_paths))
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", str(concat_list),
|
||||
"-c", "copy", str(output_path)],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CutResult:
|
||||
episode_id: int
|
||||
title: str
|
||||
ad_seconds: float = 0.0
|
||||
error: str | None = None
|
||||
|
||||
|
||||
def pending_episodes(conn: sqlite3.Connection, limit: int | None = None) -> list[sqlite3.Row]:
|
||||
"""Episodes with at least one ad span found, not yet cut."""
|
||||
query = """
|
||||
SELECT * FROM episodes
|
||||
WHERE cut_path IS NULL
|
||||
AND EXISTS (SELECT 1 FROM ad_segments WHERE episode_id = episodes.id)
|
||||
ORDER BY id
|
||||
"""
|
||||
if limit:
|
||||
query += " LIMIT ?"
|
||||
return conn.execute(query, (limit,)).fetchall()
|
||||
return conn.execute(query).fetchall()
|
||||
|
||||
|
||||
def cut_episode(
|
||||
conn: sqlite3.Connection,
|
||||
episode: sqlite3.Row,
|
||||
client: httpx.Client,
|
||||
data_dir: Path = DEFAULT_DATA_DIR,
|
||||
) -> tuple[Path, float]:
|
||||
"""Download (if needed), cut ad spans out, update the episode row.
|
||||
|
||||
Returns (cut_path, ad_seconds_removed).
|
||||
"""
|
||||
audio_path = download_audio(
|
||||
client, episode["audio_url"], data_dir / "audio" / f"{episode['id']}.mp3"
|
||||
)
|
||||
duration = probe_duration(audio_path)
|
||||
ad_spans = [
|
||||
(row["start_second"], row["end_second"])
|
||||
for row in conn.execute(
|
||||
"SELECT start_second, end_second FROM ad_segments WHERE episode_id = ?",
|
||||
(episode["id"],),
|
||||
)
|
||||
]
|
||||
keep_spans = compute_keep_spans(ad_spans, duration)
|
||||
ad_seconds = duration - sum(end - start for start, end in keep_spans)
|
||||
|
||||
output_path = data_dir / "cut" / f"{episode['id']}{audio_path.suffix}"
|
||||
cut_audio(audio_path, keep_spans, output_path)
|
||||
|
||||
conn.execute(
|
||||
"UPDATE episodes SET cut_path = ?, updated_at = ? WHERE id = ?",
|
||||
(str(output_path), utcnow(), episode["id"]),
|
||||
)
|
||||
conn.commit()
|
||||
return output_path, ad_seconds
|
||||
|
||||
|
||||
def cut_pending(
|
||||
conn: sqlite3.Connection,
|
||||
client: httpx.Client,
|
||||
data_dir: Path = DEFAULT_DATA_DIR,
|
||||
limit: int | None = None,
|
||||
on_result: Callable[[CutResult], None] | None = None,
|
||||
) -> list[CutResult]:
|
||||
results: list[CutResult] = []
|
||||
for row in pending_episodes(conn, limit):
|
||||
result = CutResult(episode_id=row["id"], title=row["title"] or "")
|
||||
try:
|
||||
_path, ad_seconds = cut_episode(conn, row, client, data_dir)
|
||||
result.ad_seconds = ad_seconds
|
||||
except Exception as exc: # noqa: BLE001 — per-episode isolation
|
||||
result.error = str(exc)
|
||||
results.append(result)
|
||||
if on_result:
|
||||
on_result(result)
|
||||
return results
|
||||
|
||||
+140
-6
@@ -1,15 +1,149 @@
|
||||
"""M4/M5: re-host a cleaned feed (feedgen) pointing at cut_path episodes.
|
||||
|
||||
Not implemented yet. This is the only integration point AntennaPod (or any
|
||||
podcast app) ever sees: subscribe to this feed's URL instead of the
|
||||
original — same shape as hark's own "output integration" (custom RSS feeds
|
||||
subscribed to like any podcast, no app-side changes). See docs/PLAN.md M4-M5.
|
||||
This is the only integration point any podcast player needs: subscribe to
|
||||
`/feed/<feed_id>` instead of the original feed URL. Episodes with a cut_path
|
||||
are served locally at `/audio/<id>.<ext>`; everything else still points at its
|
||||
original audio_url unchanged — an episode nobody has cut (no ads found, or
|
||||
not processed yet) doesn't need a local copy at all.
|
||||
|
||||
Dependency-free HTTP by design, same as hark's web.py: stdlib http.server, no
|
||||
framework. Unlike hark's browsable dashboard this has no login wall — it's a
|
||||
machine-consumed feed for one owner on a homelab network, not a searchable UI
|
||||
with a filesystem of someone's listening habits behind it. Revisit if this
|
||||
ever needs to be reachable from outside a trusted network (see docs/PLAN.md).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
import urllib.parse
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
from feedgen.feed import FeedGenerator
|
||||
|
||||
from . import __version__, db
|
||||
|
||||
|
||||
def build_feed(conn: sqlite3.Connection, feed_id: int) -> bytes:
|
||||
raise NotImplementedError("M4/M5: proxy feed generation not built yet — see docs/PLAN.md")
|
||||
def _parse_pubdate(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def build_feed(conn: sqlite3.Connection, feed: sqlite3.Row, base_url: str) -> bytes:
|
||||
fg = FeedGenerator()
|
||||
fg.title(feed["title"] or feed["source_url"])
|
||||
fg.link(href=feed["source_url"], rel="self")
|
||||
fg.description(feed["description"] or feed["title"] or feed["source_url"])
|
||||
if feed["image_url"]:
|
||||
fg.image(feed["image_url"])
|
||||
|
||||
episodes = conn.execute(
|
||||
"SELECT * FROM episodes WHERE feed_id = ? ORDER BY pubdate DESC", (feed["id"],)
|
||||
).fetchall()
|
||||
for ep in episodes:
|
||||
length = 0
|
||||
if ep["cut_path"]:
|
||||
cut_path = Path(ep["cut_path"])
|
||||
audio_url = f"{base_url}/audio/{ep['id']}{cut_path.suffix}"
|
||||
if cut_path.is_file():
|
||||
length = cut_path.stat().st_size
|
||||
else:
|
||||
audio_url = ep["audio_url"]
|
||||
if not audio_url:
|
||||
continue # nothing playable to link — skip rather than emit a dead enclosure
|
||||
fe = fg.add_entry()
|
||||
fe.id(ep["guid"])
|
||||
fe.title(ep["title"] or "(untitled)")
|
||||
fe.description(ep["description"] or "")
|
||||
pubdate = _parse_pubdate(ep["pubdate"])
|
||||
if pubdate:
|
||||
fe.pubDate(pubdate)
|
||||
fe.enclosure(audio_url, length, "audio/mpeg")
|
||||
|
||||
return fg.rss_str(pretty=True)
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
db_path: str
|
||||
data_dir: Path
|
||||
base_url: str
|
||||
server_version = f"adscrub/{__version__}"
|
||||
|
||||
def log_message(self, fmt, *args): # quiet access log
|
||||
pass
|
||||
|
||||
def respond(self, status: int, data: bytes, content_type: str) -> None:
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.send_header("X-Content-Type-Options", "nosniff")
|
||||
self.send_header("Cache-Control", "no-store")
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def not_found(self) -> None:
|
||||
self.respond(404, b"not found", "text/plain; charset=utf-8")
|
||||
|
||||
def do_GET(self):
|
||||
route = urllib.parse.urlsplit(self.path).path.rstrip("/") or "/"
|
||||
|
||||
if route == "/healthz":
|
||||
return self.respond(200, b"ok", "text/plain; charset=utf-8")
|
||||
|
||||
if route.startswith("/feed/"):
|
||||
try:
|
||||
feed_id = int(route.rsplit("/", 1)[1])
|
||||
except ValueError:
|
||||
return self.not_found()
|
||||
conn = db.connect(self.db_path)
|
||||
try:
|
||||
feed = conn.execute("SELECT * FROM feeds WHERE id = ?", (feed_id,)).fetchone()
|
||||
if feed is None:
|
||||
return self.not_found()
|
||||
body = build_feed(conn, feed, self.base_url)
|
||||
finally:
|
||||
conn.close()
|
||||
return self.respond(200, body, "application/rss+xml; charset=utf-8")
|
||||
|
||||
if route.startswith("/audio/"):
|
||||
# strip any path components the client sent — only ever look inside
|
||||
# data_dir/cut, never let the URL choose an arbitrary filesystem path
|
||||
name = Path(route[len("/audio/"):]).name
|
||||
cut_dir = (self.data_dir / "cut").resolve()
|
||||
candidate = (cut_dir / name).resolve()
|
||||
if cut_dir not in candidate.parents or not candidate.is_file():
|
||||
return self.not_found()
|
||||
return self.respond(200, candidate.read_bytes(), "audio/mpeg")
|
||||
|
||||
return self.not_found()
|
||||
|
||||
|
||||
def make_server(
|
||||
db_path: str | Path, data_dir: str | Path, base_url: str, bind: str = "0.0.0.0:8711"
|
||||
) -> ThreadingHTTPServer:
|
||||
host, _, port = bind.rpartition(":")
|
||||
try:
|
||||
port_num = int(port)
|
||||
except ValueError:
|
||||
raise SystemExit(f"invalid --bind {bind!r}: expected host:port or :port")
|
||||
handler = type("BoundHandler", (Handler,), {
|
||||
"db_path": str(db_path), "data_dir": Path(data_dir), "base_url": base_url.rstrip("/"),
|
||||
})
|
||||
return ThreadingHTTPServer((host or "0.0.0.0", port_num), handler)
|
||||
|
||||
|
||||
def serve(db_path: str | Path, data_dir: str | Path, base_url: str, bind: str) -> None:
|
||||
if "localhost" in base_url or "127.0.0.1" in base_url:
|
||||
print(f"warning: --base-url is {base_url!r} — a podcast player running "
|
||||
f"anywhere but this exact machine won't be able to reach cut audio "
|
||||
f"links embedded in the generated feed. Set --base-url/$ADSCRUB_BASE_URL "
|
||||
f"to this host's actual reachable address.")
|
||||
server = make_server(db_path, data_dir, base_url, bind)
|
||||
print(f"adscrub serving on {bind} (feeds at {base_url.rstrip('/')}/feed/<id>)")
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
server.shutdown()
|
||||
|
||||
@@ -18,10 +18,10 @@ from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from .audio import DEFAULT_DATA_DIR, download_audio
|
||||
from .db import utcnow
|
||||
|
||||
DEFAULT_MODEL = os.environ.get("ADSCRUB_WHISPER_MODEL", "small")
|
||||
DEFAULT_DATA_DIR = Path(os.environ.get("ADSCRUB_DATA_DIR", "data"))
|
||||
|
||||
_model = None
|
||||
_model_size = None
|
||||
@@ -48,21 +48,6 @@ def load_model(model_size: str = DEFAULT_MODEL):
|
||||
return _model
|
||||
|
||||
|
||||
def download_audio(client: httpx.Client, audio_url: str, dest: Path) -> Path:
|
||||
"""Fetch episode audio to dest if not already cached there."""
|
||||
if dest.exists():
|
||||
return dest
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
||||
with client.stream("GET", audio_url) as resp:
|
||||
resp.raise_for_status()
|
||||
with open(tmp, "wb") as fh:
|
||||
for chunk in resp.iter_bytes():
|
||||
fh.write(chunk)
|
||||
tmp.rename(dest)
|
||||
return dest
|
||||
|
||||
|
||||
def transcribe_episode(
|
||||
conn: sqlite3.Connection,
|
||||
episode: sqlite3.Row,
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
import subprocess
|
||||
|
||||
from adscrub import audio
|
||||
|
||||
|
||||
def test_probe_duration_parses_ffprobe_output(monkeypatch):
|
||||
def fake_run(cmd, **kwargs):
|
||||
assert cmd[0] == "ffprobe"
|
||||
assert cmd[-1] == "/tmp/fake.mp3"
|
||||
return subprocess.CompletedProcess(cmd, 0, stdout="123.456\n", stderr="")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||
assert audio.probe_duration("/tmp/fake.mp3") == 123.456
|
||||
+52
-8
@@ -1,6 +1,6 @@
|
||||
import json
|
||||
|
||||
from adscrub import cli, db, detect, transcribe
|
||||
from adscrub import cli, cut, db, detect, transcribe
|
||||
|
||||
|
||||
def test_add_feed_then_stats(tmp_path, capsys):
|
||||
@@ -37,13 +37,6 @@ def test_chapters_with_nothing_to_scan_fails(tmp_path, capsys):
|
||||
assert "no episodes" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_not_built_yet_commands_report_milestone(tmp_path, capsys):
|
||||
for name, milestone in [
|
||||
("cut", "M4"), ("serve", "M4/M5"),
|
||||
]:
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), name])
|
||||
assert rc == 1
|
||||
assert milestone in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_transcribe_with_nothing_pending_fails(tmp_path, capsys):
|
||||
@@ -150,6 +143,57 @@ def test_detect_success_path(tmp_path, capsys, monkeypatch):
|
||||
assert "detected across 1 episode(s) (0 failed, 0 still pending)" in out
|
||||
|
||||
|
||||
def test_cut_with_nothing_pending_fails(tmp_path, capsys):
|
||||
rc = cli.main(["--db", str(tmp_path / "t.db"), "cut"])
|
||||
assert rc == 1
|
||||
assert "no episodes pending cutting" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_cut_dry_run_reports_pending(tmp_path, capsys):
|
||||
path = tmp_path / "t.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (1, 'g1', 'ep', 'http://a/1.mp3')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
rc = cli.main(["--db", str(path), "cut", "--dry-run"])
|
||||
assert rc == 0
|
||||
assert "pending episodes: 1" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_cut_success_path(tmp_path, capsys, monkeypatch):
|
||||
path = tmp_path / "t.db"
|
||||
conn = db.connect(path)
|
||||
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (1, 'g1', 'Ep One', 'http://a/1.mp3')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (1, 0, 5, 'chapter')"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
def fake_cut_episode(conn, ep, client, data_dir=None):
|
||||
conn.execute("UPDATE episodes SET cut_path = 'x.mp3' WHERE id = ?", (ep["id"],))
|
||||
conn.commit()
|
||||
return "x.mp3", 5.0
|
||||
|
||||
monkeypatch.setattr(cut, "cut_episode", fake_cut_episode)
|
||||
|
||||
rc = cli.main(["--db", str(path), "cut"])
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "ok Ep One: removed 5.0s of ads" in out
|
||||
assert "cut 1 episode(s) (0 failed, 0 still pending)" in out
|
||||
|
||||
|
||||
def test_version(capsys):
|
||||
try:
|
||||
cli.main(["--version"])
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
import subprocess
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from adscrub import cut, db
|
||||
|
||||
AUDIO_URL = "https://example.com/audio/ep1.mp3"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conn(tmp_path):
|
||||
return db.connect(tmp_path / "test.db")
|
||||
|
||||
|
||||
def seed_episode(conn, ad_spans=()):
|
||||
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (1, 'ep-1', 'Ep 1', ?)",
|
||||
(AUDIO_URL,),
|
||||
)
|
||||
conn.commit()
|
||||
ep = conn.execute("SELECT * FROM episodes WHERE guid = 'ep-1'").fetchone()
|
||||
for start, end, source in ad_spans:
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source)"
|
||||
" VALUES (?, ?, ?, ?)",
|
||||
(ep["id"], start, end, source),
|
||||
)
|
||||
conn.commit()
|
||||
return conn.execute("SELECT * FROM episodes WHERE guid = 'ep-1'").fetchone()
|
||||
|
||||
|
||||
# --- compute_keep_spans ---
|
||||
|
||||
|
||||
def test_compute_keep_spans_no_ads():
|
||||
assert cut.compute_keep_spans([], 100.0) == [(0.0, 100.0)]
|
||||
|
||||
|
||||
def test_compute_keep_spans_ad_in_middle():
|
||||
assert cut.compute_keep_spans([(40.0, 60.0)], 100.0) == [(0.0, 40.0), (60.0, 100.0)]
|
||||
|
||||
|
||||
def test_compute_keep_spans_ad_at_start_and_end():
|
||||
assert cut.compute_keep_spans([(0.0, 10.0), (90.0, 100.0)], 100.0) == [(10.0, 90.0)]
|
||||
|
||||
|
||||
def test_compute_keep_spans_merges_overlapping_spans_from_different_sources():
|
||||
# a chapter-sourced span and an llm-sourced span covering roughly the same break
|
||||
spans = [(40.0, 65.0), (60.0, 70.0)]
|
||||
assert cut.compute_keep_spans(spans, 100.0) == [(0.0, 40.0), (70.0, 100.0)]
|
||||
|
||||
|
||||
def test_compute_keep_spans_merges_adjacent_spans():
|
||||
assert cut.compute_keep_spans([(10.0, 20.0), (20.0, 30.0)], 100.0) == [
|
||||
(0.0, 10.0), (30.0, 100.0)
|
||||
]
|
||||
|
||||
|
||||
def test_compute_keep_spans_clamps_out_of_range_end():
|
||||
assert cut.compute_keep_spans([(90.0, 150.0)], 100.0) == [(0.0, 90.0)]
|
||||
|
||||
|
||||
def test_compute_keep_spans_entire_episode_is_ads():
|
||||
assert cut.compute_keep_spans([(0.0, 100.0)], 100.0) == []
|
||||
|
||||
|
||||
# --- cut_audio ---
|
||||
|
||||
|
||||
def test_cut_audio_no_ads_copies_file_without_invoking_ffmpeg(tmp_path, monkeypatch):
|
||||
def fail_if_called(*a, **k):
|
||||
raise AssertionError("ffmpeg should not be invoked when there's nothing to cut")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", fail_if_called)
|
||||
|
||||
src = tmp_path / "in.mp3"
|
||||
src.write_bytes(b"original-audio-bytes")
|
||||
dest = tmp_path / "out" / "out.mp3"
|
||||
cut.cut_audio(src, [(0.0, 100.0)], dest)
|
||||
assert dest.read_bytes() == b"original-audio-bytes"
|
||||
|
||||
|
||||
def test_cut_audio_invokes_ffmpeg_per_segment_then_concat(tmp_path, monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake_run(cmd, **kwargs):
|
||||
calls.append(cmd)
|
||||
# ffmpeg's real job is producing the output file at the end of argv
|
||||
with open(cmd[-1], "wb") as fh:
|
||||
fh.write(b"x")
|
||||
return subprocess.CompletedProcess(cmd, 0)
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||
|
||||
src = tmp_path / "in.mp3"
|
||||
src.write_bytes(b"original")
|
||||
dest = tmp_path / "out.mp3"
|
||||
cut.cut_audio(src, [(0.0, 40.0), (60.0, 100.0)], dest)
|
||||
|
||||
assert dest.exists()
|
||||
# 2 segment extractions + 1 concat
|
||||
assert len(calls) == 3
|
||||
assert calls[0][:2] == ["ffmpeg", "-y"]
|
||||
assert "-ss" in calls[0] and "0.0" in calls[0]
|
||||
assert "-ss" in calls[1] and "60.0" in calls[1]
|
||||
assert calls[2][3] == "concat"
|
||||
|
||||
|
||||
# --- pending_episodes ---
|
||||
|
||||
|
||||
def test_pending_episodes_requires_ad_segments(conn):
|
||||
seed_episode(conn) # no ad spans at all
|
||||
assert cut.pending_episodes(conn) == []
|
||||
|
||||
|
||||
def test_pending_episodes_includes_episode_with_ad_spans(conn):
|
||||
ep = seed_episode(conn, ad_spans=[(10.0, 20.0, "chapter")])
|
||||
assert [e["id"] for e in cut.pending_episodes(conn)] == [ep["id"]]
|
||||
|
||||
conn.execute("UPDATE episodes SET cut_path = '/x.mp3' WHERE id = ?", (ep["id"],))
|
||||
conn.commit()
|
||||
assert cut.pending_episodes(conn) == []
|
||||
|
||||
|
||||
# --- cut_episode / cut_pending ---
|
||||
|
||||
|
||||
def audio_client():
|
||||
def handler(request):
|
||||
return httpx.Response(200, content=b"fake-mp3-bytes")
|
||||
|
||||
return httpx.Client(transport=httpx.MockTransport(handler))
|
||||
|
||||
|
||||
def fake_cut_audio(audio_path, keep_spans, output_path):
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(b"cut")
|
||||
|
||||
|
||||
def test_cut_episode_updates_row_and_returns_ad_seconds(conn, tmp_path, monkeypatch):
|
||||
ep = seed_episode(conn, ad_spans=[(10.0, 20.0, "chapter")])
|
||||
monkeypatch.setattr(cut, "probe_duration", lambda path: 100.0)
|
||||
monkeypatch.setattr(cut, "cut_audio", fake_cut_audio)
|
||||
|
||||
with audio_client() as client:
|
||||
path, ad_seconds = cut.cut_episode(conn, ep, client, data_dir=tmp_path)
|
||||
|
||||
assert ad_seconds == 10.0
|
||||
assert path == tmp_path / "cut" / f"{ep['id']}.mp3"
|
||||
assert path.read_bytes() == b"cut"
|
||||
|
||||
row = conn.execute("SELECT cut_path FROM episodes WHERE id = ?", (ep["id"],)).fetchone()
|
||||
assert row["cut_path"] == str(path)
|
||||
|
||||
|
||||
def test_cut_pending_isolates_per_episode_failures(conn, tmp_path, monkeypatch):
|
||||
ep1 = seed_episode(conn, ad_spans=[(10.0, 20.0, "chapter")])
|
||||
conn.execute("INSERT INTO feeds (source_url) VALUES ('http://feed2')")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, audio_url) VALUES (2, 'ep-2', 'Ep 2', ?)",
|
||||
(AUDIO_URL,),
|
||||
)
|
||||
conn.commit()
|
||||
ep2 = conn.execute("SELECT * FROM episodes WHERE guid = 'ep-2'").fetchone()
|
||||
conn.execute(
|
||||
"INSERT INTO ad_segments (episode_id, start_second, end_second, source) VALUES (?, 0, 5, 'chapter')",
|
||||
(ep2["id"],),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
def fake_probe_duration(path):
|
||||
if "2" in str(path):
|
||||
raise RuntimeError("ffprobe failed")
|
||||
return 100.0
|
||||
|
||||
monkeypatch.setattr(cut, "probe_duration", fake_probe_duration)
|
||||
monkeypatch.setattr(cut, "cut_audio", fake_cut_audio)
|
||||
|
||||
with audio_client() as client:
|
||||
results = {r.title: r for r in cut.cut_pending(conn, client, data_dir=tmp_path)}
|
||||
|
||||
assert results["Ep 1"].error is None
|
||||
assert results["Ep 2"].error is not None
|
||||
assert cut.pending_episodes(conn) == [
|
||||
conn.execute("SELECT * FROM episodes WHERE guid = 'ep-2'").fetchone()
|
||||
]
|
||||
@@ -0,0 +1,178 @@
|
||||
"""feed.py tests: build_feed content, and real HTTP against a server on an
|
||||
ephemeral port — same pattern as hark's test_web.py."""
|
||||
|
||||
import http.client
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from adscrub import db, feed
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conn(tmp_path):
|
||||
conn = db.connect(tmp_path / "test.db")
|
||||
conn.execute(
|
||||
"INSERT INTO feeds (source_url, title, description, image_url)"
|
||||
" VALUES ('http://original/feed', 'Show A', 'A show', 'http://original/art.png')"
|
||||
)
|
||||
conn.commit()
|
||||
return conn
|
||||
|
||||
|
||||
# --- build_feed ---
|
||||
|
||||
|
||||
def test_build_feed_passthrough_for_uncut_episode(conn):
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, description, pubdate, audio_url)"
|
||||
" VALUES (1, 'ep-1', 'Ep 1', 'desc', '2026-01-01T00:00:00Z', 'http://original/ep1.mp3')"
|
||||
)
|
||||
conn.commit()
|
||||
feed_row = conn.execute("SELECT * FROM feeds WHERE id = 1").fetchone()
|
||||
|
||||
xml = feed.build_feed(conn, feed_row, "http://myhost:8711").decode()
|
||||
assert "<title>Show A</title>" in xml
|
||||
assert 'url="http://original/ep1.mp3"' in xml
|
||||
assert "myhost" not in xml # untouched episode keeps its original URL
|
||||
|
||||
|
||||
def test_build_feed_points_cut_episodes_at_local_audio_route(conn, tmp_path):
|
||||
# cut.py names the file after the episode id (see cut_episode) — build_feed
|
||||
# reconstructs the same URL from ep['id'] + the stored cut_path's suffix,
|
||||
# so the fixture's filename here must match episode id 1 to be realistic.
|
||||
cut_path = tmp_path / "cut" / "1.mp3"
|
||||
cut_path.parent.mkdir(parents=True)
|
||||
cut_path.write_bytes(b"cut-audio-bytes")
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, audio_url, cut_path)"
|
||||
" VALUES (1, 'ep-1', 'Ep 1', 'http://original/ep1.mp3', ?)",
|
||||
(str(cut_path),),
|
||||
)
|
||||
conn.commit()
|
||||
feed_row = conn.execute("SELECT * FROM feeds WHERE id = 1").fetchone()
|
||||
|
||||
xml = feed.build_feed(conn, feed_row, "http://myhost:8711").decode()
|
||||
assert 'url="http://myhost:8711/audio/1.mp3"' in xml
|
||||
assert f'length="{len(b"cut-audio-bytes")}"' in xml
|
||||
assert "original/ep1.mp3" not in xml
|
||||
|
||||
|
||||
def test_build_feed_skips_episode_with_no_audio(conn):
|
||||
conn.execute("INSERT INTO episodes (feed_id, guid, title) VALUES (1, 'ep-1', 'No audio')")
|
||||
conn.commit()
|
||||
feed_row = conn.execute("SELECT * FROM feeds WHERE id = 1").fetchone()
|
||||
xml = feed.build_feed(conn, feed_row, "http://myhost:8711").decode()
|
||||
assert "No audio" not in xml
|
||||
|
||||
|
||||
# --- HTTP server ---
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def server(tmp_path):
|
||||
conn = db.connect(tmp_path / "adscrub.db")
|
||||
conn.execute(
|
||||
"INSERT INTO feeds (source_url, title) VALUES ('http://original/feed', 'Show A')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO episodes (feed_id, guid, title, audio_url)"
|
||||
" VALUES (1, 'ep-1', 'Ep 1', 'http://original/ep1.mp3')"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
cut_dir = tmp_path / "data" / "cut"
|
||||
cut_dir.mkdir(parents=True)
|
||||
(cut_dir / "audio.mp3").write_bytes(b"cut-bytes")
|
||||
|
||||
srv = feed.make_server(
|
||||
tmp_path / "adscrub.db", tmp_path / "data", "http://myhost:8711", bind="127.0.0.1:0"
|
||||
)
|
||||
thread = threading.Thread(target=srv.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
yield srv
|
||||
srv.shutdown()
|
||||
|
||||
|
||||
def request(srv, path):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", srv.server_address[1], timeout=5)
|
||||
conn.request("GET", path)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
conn.close()
|
||||
return resp, data
|
||||
|
||||
|
||||
def test_healthz(server):
|
||||
resp, data = request(server, "/healthz")
|
||||
assert resp.status == 200
|
||||
assert data == b"ok"
|
||||
|
||||
|
||||
def test_feed_route_serves_generated_rss(server):
|
||||
resp, data = request(server, "/feed/1")
|
||||
assert resp.status == 200
|
||||
assert resp.getheader("Content-Type") == "application/rss+xml; charset=utf-8"
|
||||
assert b"Show A" in data
|
||||
|
||||
|
||||
def test_feed_route_unknown_id_404s(server):
|
||||
resp, _ = request(server, "/feed/999")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_feed_route_non_numeric_id_404s(server):
|
||||
resp, _ = request(server, "/feed/not-a-number")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_audio_route_serves_cut_file(server):
|
||||
resp, data = request(server, "/audio/audio.mp3")
|
||||
assert resp.status == 200
|
||||
assert resp.getheader("Content-Type") == "audio/mpeg"
|
||||
assert data == b"cut-bytes"
|
||||
|
||||
|
||||
def test_audio_route_missing_file_404s(server):
|
||||
resp, _ = request(server, "/audio/nope.mp3")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
def test_audio_route_rejects_path_traversal(server):
|
||||
# Path(...).name strips any directory components regardless of how many
|
||||
# ".." segments precede the final part, so this can never escape cut_dir.
|
||||
resp, data = request(server, "/audio/../../../../etc/passwd")
|
||||
assert resp.status == 404
|
||||
assert b"root:" not in data
|
||||
|
||||
|
||||
def test_audio_route_rejects_encoded_path_traversal(server):
|
||||
resp, data = request(server, "/audio/..%2F..%2F..%2Fetc%2Fpasswd")
|
||||
assert resp.status == 404
|
||||
assert b"root:" not in data
|
||||
|
||||
|
||||
def test_unknown_route_404s(server):
|
||||
resp, _ = request(server, "/nonsense")
|
||||
assert resp.status == 404
|
||||
|
||||
|
||||
# --- serve() base_url warning ---
|
||||
|
||||
|
||||
class _FakeServer:
|
||||
def serve_forever(self):
|
||||
pass
|
||||
|
||||
|
||||
def test_serve_warns_when_base_url_is_localhost(tmp_path, capsys, monkeypatch):
|
||||
monkeypatch.setattr(feed, "make_server", lambda *a, **k: _FakeServer())
|
||||
feed.serve(tmp_path / "db", tmp_path / "data", "http://localhost:8711", "127.0.0.1:0")
|
||||
assert "warning" in capsys.readouterr().out.lower()
|
||||
|
||||
|
||||
def test_serve_no_warning_for_a_real_hostname(tmp_path, capsys, monkeypatch):
|
||||
monkeypatch.setattr(feed, "make_server", lambda *a, **k: _FakeServer())
|
||||
feed.serve(tmp_path / "db", tmp_path / "data", "http://truenas.local:8711", "127.0.0.1:0")
|
||||
assert "warning" not in capsys.readouterr().out.lower()
|
||||
Reference in New Issue
Block a user