The SIM118 unsafe fix was reverted with a noqa where it broke sqlite3.Row
access (Row has keys() but no .get()). README CI badge now points at the
public GitHub mirror workflow.
manual joins GROUND_TRUTH_SOURCES (seeds both libraries, confirms a
campaign) and CUT_SOURCES (cut). A human marking a span as an ad is the
strongest evidence there is; hark's UI writes these when an operator marks
or corrects an ad by hand.
probe_variance re-probes with a larger window when a divergence never
realigns in the first, so a long/mid-roll ad's end can be found instead of
discarding the detection. stream_index_episodes/pending_stream_index_ids
fingerprint un-downloaded episodes by streaming (fetch-and-discard), so the
index reaches the whole corpus without storing audio; quarantined URLs are
excluded and per-episode failures are skipped.
hold_cut clears cut_path (so the feed falls back to the original audio)
and stamps a new cut_held_at column, so a cut removing an implausible share
of an episode (is_anomalous_cut) is held for review rather than served
gutted; pending_episodes excludes held episodes so they are not re-cut.
is_audio_gone/mark_audio_gone + a new episodes.audio_gone_at column let a
permanently-gone (404/410) audio URL be quarantined instead of retried
forever — a few dead URLs at the head of the queue otherwise trip the
consecutive-failure abort before any live episode is reached, stalling the
whole corpus. is_anomalous_cut flags a cut removing >35% of an episode as a
likely false positive worth review rather than a silent gutted episode.
fpcalc over a whole episode is the tier's only real cost and happens once per
episode. Separating it from matching lets the pipeline index a bounded slice
of new episodes each cycle while matching everything already cached, so the
whole-corpus backfill spreads across cycles instead of stalling for hours on
first run.
index_episodes draws from a pending queue of episodes with local audio and no
cached fingerprint, which shrinks as they are indexed. apply_fingerprints and
fingerprint_episode gain indexed_only, matching only cache hits with no
download or fpcalc and skipping an un-indexed episode rather than erroring.
A caller that already ran probe_variance in its own loop should not have to
fetch the episode a second time just to persist the result. dai_episode is
now that store step plus the probe; hark's scheduled dai-probe hands the
DAIProbeResult straight in, so server-inserted ads it already found seed the
fingerprint library instead of being thrown away each cycle.
A region that aligns to a confirmed recording but carries no words is a music
bed, a sting, or room tone that happened to recur. It is the one error an
audio-only tier cannot see, because it never reads, and it is what the 14s
region in the end-to-end cut turned out to be.
Measured on Casefile: 12 regions totalling 177 seconds removed at 0.00%
recall cost, 89.6% before and after. It costs nothing because ads talk; the
only regions it takes are the ones with nothing to say.
Skipped when no transcript exists, so the tier still runs before
transcription. This works where the min-density guard did not: density was a
proxy for whether a match was real and had no knee, while speech tests
directly for whether the thing is an ad.
The new cached_fingerprint test inserted into episode_fingerprints without
calling ensure_schema first; the cache tables are created lazily, so the
insert hit a missing table. Committed failing because the verifying pytest
ran in a pipeline, where the exit code comes from tail rather than pytest.
find_campaigns and discover_recurring checked that an episode's audio existed
before looking at the cache, so a corpus that had been indexed and then had
its audio discarded looked empty — requiring exactly the thing the index
exists to replace. Both now go through cached_fingerprint, which reads
episode_fingerprints first and falls back to local audio only for episodes
never indexed. Verified on 40 real episodes with no audio files present.
fpcalc takes the audio on stdin and returns a byte-identical fingerprint to
the file path, so an episode can be indexed straight off the network and
discarded. A fingerprint is about 184 times smaller than its audio, measured
at 6.3 GB of episodes against 34 MB of fingerprints, which is the difference
between a two-terabyte corpus and one under a gigabyte.
What the stream loses is duration, since fpcalc cannot seek and reports zero.
It is derived from the frame count instead, using a rate measured over 122
real episodes; the error on real episodes is about 0.02 percent. That is fine
for what the index answers, which is whether audio recurs rather than exactly
where. Cutting keeps using the real file and ffprobe, because it needs the
audio regardless and the same 0.1 percent is around four seconds on a
two-hour episode.
Feed-declared duration is not a substitute. Episode 1 declares 5,896 seconds
against 6,393 seconds of audio; that eight-minute gap is dynamic ad insertion
giving every listener a differently sized file, so the feed's number describes
nobody's copy.
discover_recurring reports recurring regions per episode, which is the wrong
unit for spending a model budget: twelve episodes carrying one sponsor read
are one thing to learn, not twelve, and fingerprinting recognises the other
eleven for free once any of them is confirmed.
find_campaigns links recurring regions across episodes by the alignment that
matched them and merges them with a union-find, so each connected component
is one ad recording. A component counts as known when any member overlaps a
ground-truth span. select_seed_episodes is then greedy set cover over the
unknown ones: the fewest episodes that confirm every recording the library
does not already have.
A cluster spanning more than STOP_EPISODE_FRACTION of the feed is dropped as
the show's own content. Union-find chains transitively, so one shared music
bed can fuse unrelated regions into a component covering every episode; on 40
Casual Criminalist episodes the largest cluster reached 40/40 and pulled
silent and music-only regions into the selection. With the ceiling: 11
campaigns, 7 episodes to read.
A recording running in more than that fraction of a feed stays invisible to
self-recurrence. Real campaigns fragment into variants that each stay a
minority, which is what makes that survivable.
discover_recurring gains episode_ids so a caller holding many feeds in one
database can scope recurrence to a single show. hark has ~70 unrelated feeds
in one database, and recurrence is measured against whatever set it is given,
so pooling them would build one enormous index and compute the stop-list over
a corpus that shares no ad pool.
Also records that AdSpanDetector.detect gained a second argument. It is a
breaking change for implementors outside this package: LayeredDetector passes
skip positionally, so a two-argument detect() now raises TypeError. hark's own
_PrecomputedDetector hit exactly this.
Cuts the acoustic-fingerprint work as 0.8.0: the fingerprint tier, dai
persistence, cold-start discovery, coverage-aware LLM prompting, the
frequency-intersect-editorial stop-list and source-filtered cutting.
Bumps __init__.__version__ alongside pyproject, which has drifted before,
and corrects two statements in the release notes that later commits in the
same branch had made false.
The previous commit claimed silence-snapping fixed a cut that "ate the
opening of 'It was 405 on the morning of Thursday, June 19, 2014'". That
diagnosis was wrong and is corrected here rather than left in the history to
mislead. Those frames match a confirmed ad recording from another episode
densely, gaps of one to three frames, and an episode's narration is unique to
it and cannot match another episode's audio. The region is the ad's outro bed
and Whisper had timestamped the segment early. Whisper segment starts are not
evidence of speech onset.
snap_spans_to_silence is kept on its own merit: an edge is a guess about where
a break ends, moving edges inward bounds the risk of running into speech, it
tightened three of five edges on a real cut, and by construction it can only
shrink what gets removed.
Sparse heavily-bridged regions looked like a false-positive signature, so a
minimum match-density guard was measured against ground truth. It has no
knee: 0.35 costs 1.0pp recall to remove 220s of outside-LLM time, 0.55 costs
2.0pp, 0.75 collapses recall to 60.2%. Outside-LLM time is not a synonym for
false positives either, since much of it is ads the model missed. Not shipped.
The Dockerfile installed ffmpeg but not libchromaprint-tools, so in any
deploy the fingerprint and discover tiers were inert rather than broken:
fpcalc_available() returned False, the command exited tidily, and a
healthy-looking image silently never matched an ad.
Cutting a real episode (rather than trusting detection metrics) showed a
fingerprint match ends where the ad stops being recognisable, not where the
break ends. One edge sat 2.3s inside the resumed narration and the cut ate
the opening of "It was 405 on the morning of Thursday, June 19, 2014".
Snapping to the nearest silence was tried first and measured worse, 2.3s to
2.88s clipped, because the closest silence was a pause inside the narration.
Direction is the fix: starts only move later, ends only move earlier, so a
span shrinks and never grows and every error leaves a sliver of ad rather
than deleting a sentence.
This only helps where silence exists. On that episode it tightened three of
five edges and left the 2.31s clip untouched, since the ad-to-narration
transition has no detectable pause. That residual is a detection-edge
problem, likely BRIDGE_FRAMES extending a run, and is not solved here.
cut selected every ad_segments row regardless of source, so the discovery
tiers added in the last two commits would have silently begun deleting
audio: dai spans whose end is only an upper bound, and recur spans of which
roughly one in ten is not an ad.
Cutting is now limited to CUT_SOURCES (chapter, llm, repeat, fpmatch). The
line is not evidence versus inference — repeat and fpmatch are inference and
are exactly what the cheap tiers exist to cut. It is whether the tier pins
the span's edges: those four ground their boundaries in publisher markers,
transcript segment timestamps, or audio alignment, while dai and recur find
ads well without saying where they stop. Good seeds, bad scissors.
pending_episodes filters on the same list, so an episode carrying only
discovery spans is not treated as pending — that would rewrite the file
unchanged and mark it cut, retiring it before a real span ever arrived.
Override with `adscrub cut --sources`.
Two more pipeline gaps.
The model was re-reading transcript that cheaper tiers had already covered.
detect_episode now computes covered_segment_indices across every source and
_chunks omits those segments, so an episode already 60% covered costs ~40%
of the tokens it did. Omissions leave an elision marker, since without one
the segments either side of a removed ad read as adjacent and the model sees
a seam that isn't there; indices stay global so spans still ground against
the full transcript. AdSpanDetector.detect takes an optional skip argument,
which free tiers ignore on purpose.
The seeded tier could only recognise campaigns something else had already
confirmed, leaving a new feed with no way to get cheap. discover_recurring
matches a feed against itself: audio recurring across unrelated episodes is
the inserted material. Verified on 40 unlabelled Casual Criminalist
episodes — recurring audio in 40/40, 174 regions, reading as real ads. It
recovered that feed's DAI inserts and not its host-read Shopify spot, the
expected split. Spans are source='recur' (inference, never library-seeding).
RECUR_MIN_EPISODES is 8 on arithmetic, not taste: a campaign needs two
episodes to recur but must stay under the frequency threshold, so discovery
cannot work below about seven.
Also records a measured correction: being host-read does not defeat
fingerprinting, being re-read does. A spot recorded once and re-rolled
matches like any insert; this feed re-records Shopify per episode, so it
appears in 33/40 episodes and matches 0/39 while a DAI insert matches 2/39
and an editorial control 0/39.
Two pipeline gaps found while validating the fingerprint tier.
dai.py proved which bytes were server-inserted and then discarded the
finding — nothing reached ad_segments. dai_episode now stores a `dai` span
(bytes converted via the file's average byte rate, trimmed, capped at one
plausible break, confidence 0.5) plus an `adscrub dai` command, giving ad
discovery with no transcript and no model. The span's start is evidence and
its end only an upper bound, so it seeds the audio library but deliberately
not repeats' text library, where a sloppy boundary would teach the matcher
editorial wording.
The stop-list dropped any value present in >30% of source episodes as
presumed silence, which also deletes a sponsor that runs in most episodes
(Flexcar: 27/40 Casual Criminalist episodes). It is now an intersection:
frequency proposes, known non-ad audio vetoes. Measured on Casefile —
frequency alone 88.9% recall, editorial alone 76.8% (too aggressive:
Chromaprint values collide between ad and ordinary speech), intersection
89.6% with 669 values stopped. Better recall than either, and the
ubiquitous sponsor survives.
Also adds the MIN_REGION_FRAMES emit floor (~10s), swept against ground
truth: 0.2pp recall for ~12% less false-positive time.
Match an episode's audio against Chromaprint fingerprints of ad recordings
already confirmed elsewhere in the corpus (llm/chapter spans), so a campaign
confirmed once is cut with no transcript and no model. Runs before
transcription as a sibling audio stage (not a transcript AdSpanDetector),
which is the cost lever repeats can't pull.
- fingerprint.py: build_library over confirmed-ad audio (cached in
ad_fingerprints), AudioFingerprintDetector with diagonal exact-frame
matching + silence stop-list, whole-episode fingerprint cache
(episode_fingerprints) so re-scans re-run only matching, per-episode and
bulk apply. fpmatch spans are inference and never seed the library.
- CLI: adscrub fingerprint, cheapest-first after repeats.
- tests: 18 cases (matching logic + DB plumbing/cache/idempotency/isolation).
- Requires fpcalc (Chromaprint); clear error if absent.
Measured leave-one-out (Casefile, 82 eps): 89.1% of confirmed ad duration
recovered from audio alone, 0 episodes fully missed, 0/82 non-ad control
false-match. Cross-show on The Casual Criminalist: mechanism generalises
(recurring audio is real ads), and a library from one show catches another's
ads when they share a DAI campaign.
A feed could stream an unbounded body and fill the disk. Enforce a byte cap
(default 1 GiB, ADSCRUB_MAX_AUDIO_MB) two ways: reject an oversized declared
Content-Length up front, and abort mid-stream once the running total exceeds
the cap. Either way the .part file is unlinked so a rejected download leaves
nothing behind. The cap is a keyword arg defaulting to the module constant, so
existing callers are unaffected.
Browser UAs under-trigger targeting: megaphone.fm reported no
divergence with Chrome/Safari signatures, then diverged cleanly once
probed as Apple Podcasts vs. Spotify. An ad server has no reason to
personalize traffic that doesn't look like a real podcast client.
A shared httpx.Client's cookie jar auto-replayed the first fetch's
Set-Cookie on the second fetch, making the ad server see the same
listener both times regardless of User-Agent. Caught on real data:
a shared-client run reported acast.com as unchanged on an episode a
raw two-curl test had already shown genuine divergence on.
Fetches audio_url twice with different User-Agents and compares the
bytes. If a platform's ad server varies by these signals, the
divergence point is a provable ad boundary with zero transcription
or classification. Confirmed live against an Acast-hosted show.
For each pending episode, compares the repeat tier's found ad-break
count against the show's typical count. Episodes where the count
doesn't match are ranked first, since a mismatch is more likely to
mean something was actually missed. Not a skip: leave-one-out
validation found exact-count matches still had real recall gaps, so
a match only means "process later," never "skip."
It takes the first N episodes by id. There is no pending-queue — re-scanning is free
and the library grows — so the pipeline runs this unbounded. A limit inside a loop
would rescan the same head forever and never reach the tail; the help text now says
so rather than leaving that to be discovered.
pyproject said 0.5.1 while __init__ said 0.6.0 — and 0.5.2 never bumped it either, so
every install since has reported a version two releases stale. The version a consumer
resolves comes from here, not from __init__.
hark filters by a per-show ad_stripping_enabled toggle that adscrub knows nothing
about, and already bypasses detect_pending for exactly that reason. Mirror that shape
here rather than making hark reimplement the tier or adscrub learn about the filter.
apply_repeats() is now this in a loop.
Ads arrive in batches. The ad server rotates a small pool of campaigns, and we
download each episode once, server-side, out of that pool — so the same reads
recur near-verbatim across the episodes we fetched in the same period.
Measured leave-one-out over the live corpus (82 episodes, 286 confirmed spans):
93.5% of confirmed ad segments are recoverable from ad reads confirmed in OTHER
episodes, and 51 of 80 episodes are >=95% covered, with no model called. On 5-word
shingles at 0.4 overlap, not whole-segment equality: Whisper segments the same ad
read differently between episodes, and exact matching scores 70% where shingles
score 93.5%.
This is not the fingerprinting CLAUDE.md rejects, and that ruling stands. What it
rejects is a GLOBAL, CROWDSOURCED ad database — strangers' ads are not our ads, so
there is nothing to share. It was never an argument against matching our corpus
against itself. The thing there was "nothing to fingerprint against" has been sitting
in our own ad_segments table the whole time. CLAUDE.md now records the narrowing and
the evidence, so this doesn't read as a violation to whoever finds it next.
LayeredDetector composes the tiers into one AdSpanDetector, so every call site takes
the layering with nothing else changing. Spans keep their own source, overlaps are
allowed, cut.py already merges them. No tier knows about another; no caller branches.
An empty library degrades to exactly the old behaviour, with no special case.
Two things this turned up:
ClaudeAdDetector was truncating every transcript to `body[:20000]` — the first ~28%
of an episode — and then marking it llm_detected_at. The cost instinct behind that
was right; the implementation threw the episode away. A rendered transcript is
~88,000 chars, so the model saw segments 0-235 of 840: every mid-roll and every
end-tag sat past the cliff, unseen, and the episode never came back. A truncation
that also marks the work complete is worse than no detection — it launders a 28% look
as a finished one. Chunked now: same per-call ceiling, whole episode covered.
And detection recall was worse than anyone could see. Of the segments the repeat tier
flags that the LLM did not, 31% carry an unambiguous brand/CTA marker — against 29%
of the segments the LLM did flag. Identical density: same content, not false
positives. 62% of episodes had at least one provably-missed ad still in the audio.
repeat spans are inference, not evidence, and never enter the library. Feeding them
back makes the detector bootstrap off its own guesses — caught on real data, where a
second sweep went 958 -> 993 spans as each pass's inferences became the next pass's
evidence and drifted. The unit test had missed it; the corpus did not.
The 0.5.1 fallback kept CUDA-visible-but-unusable deploys running, but it
degraded silently: the GPU idles at 0% while faster-whisper's CPU int8 path
saturates ~4 cores at roughly 10x the wall time, with nothing in the logs.
Print the cause, the consequence and the fix when it trips, and announce the
chosen device at model load.
get_cuda_device_count() only checks driver/device visibility, not
whether the runtime libraries are actually loadable. A hark deploy hit
this in production: every episode failed at first real inference with
"Library libcublas.so.12 is not found or cannot be loaded" instead of
falling back to CPU the way this module intends. transcribe_episode()
now catches that failure once per process and retries on CPU.
Without one the code was legally all-rights-reserved — nobody could use, fork or contribute
to it. AGPL-3.0 matches hark, its main consumer, and the other self-hosted services.
Outside readers cannot reach git.onetick.ninja, so cross-reference hark by its GitHub URL
and record that Gitea stays canonical while GitHub is a read-only mirror.
Extracted the segment-index-to-timestamp grounding/validation
ClaudeAdDetector.detect() did inline into a public function, so a
non-LLM AdSpanDetector fed pre-computed spans gets the same
bounds-checking without reimplementing it. Behavior unchanged.
detect.py was the odd sibling module, only exposing the bulk
detect_pending (plus a private _store) with no per-episode function a
caller could use to build its own pending-episode selection. Needed by
hark's per-show ad-stripping toggle. detect_pending now calls it
internally; behavior unchanged.
test_cut_pending_isolates_per_episode_failures matched a bare "2" against
the full audio file path to target episode 2's failure — but pytest's
auto-numbered tmp_path can itself contain that digit, so the test failed
depending on run order. Now matches the deterministic per-episode filename.
README/CLAUDE.md/PLAN.md still described the hark relationship as an open
question; M5 was actually decided (library dependency, not a merge) — bring
them in line with what hark's own docs already say.
pyproject.toml's version was bumped at each milestone (0.1.0 -> 0.4.0)
but __init__.py's __version__ constant never was, so the CLI's
--version flag and every outbound User-Agent header have claimed to
be adscrub/0.1.0 this whole time.
adscrub cut: merges overlapping ad spans from any source (chapter, LLM
-- no dedup rule needed, overlap-merging handles it), ffmpeg-extracts
the surviving audio, concatenates with -c copy (no re-encode). Episode
duration comes from ffprobe on the real file, not RSS metadata.
adscrub serve: stdlib http.server (dependency-free, same approach as
hark's web.py) regenerates a cleaned RSS feed live at GET /feed/<id>.
Cut episodes serve locally at /audio/<id>.<ext>; everything else keeps
its original audio_url unchanged -- nothing gets a local copy unless
it was actually cut. No login wall (trusted-network machine-consumed
feed, not a browsable dashboard). Warns instead of silently producing
a broken feed if --base-url is left at the unreachable localhost
default.
Split download_audio/probe_duration out of transcribe.py into a new
audio.py, since cut.py needed the same downloaded-audio cache.
Docker default CMD now runs adscrub serve (port 8711); pipeline stages
stay one-shot docker compose run commands.
Full pipeline (ingest -> chapters -> transcribe -> detect -> cut ->
serve) is now built end-to-end, all mocked/fixture-tested -- real
audio quality and AntennaPod compatibility are still unverified.
flan
flan