Adds acoustic ad recognition, so a campaign confirmed once is cut with no transcript and no model.
What this adds
fingerprint — matches episode AUDIO against Chromaprint fingerprints of ad recordings already confirmed in the corpus. Runs before transcription, so unlike repeats it saves the Whisper cost too.
dai — persists DAI probe divergences, which the probe previously proved and then discarded. Ad discovery with no transcript, no model.
discover — cold start for a feed with nothing confirmed: match the feed against itself.
Coverage-aware prompting — the model is only sent transcript no cheaper tier has covered.
cut is source-filtered — only tiers that pin their span EDGES may remove audio.
Measured (real corpus, leave-one-out)
89.6% of confirmed ad duration recovered from audio alone (Casefile, 82 eps, 286 ads), 0 episodes fully missed, 0/82 non-ad control false-match.
Cross-show on The Casual Criminalist (40 eps, zero labels): recurring ad audio in 40/40, and 13/40 share the Flexcar DAI campaign with Casefile — a library from one show recognises another show's ads.
End-to-end cut verified on real audio: 6392.7s -> 5945.1s, arithmetic exact, 4/5 removed spans verifiably ads.
Things measured and then rejected
Recorded so they aren't re-attempted blind:
Editorial-only stop-list: 76.8% recall (74,075 values stopped vs 1,173). Frequency ∩ editorial wins at 89.6%.
Nearest-silence cut snapping: measured worse than shrink-only; the closest silence is often a pause inside speech.
Match-density guard: no knee — 0.35 costs 1.0pp recall, 0.75 collapses to 60.2%.
Known limits
Host-read spots that are re-read per episode can't match (measured: 33/40 episodes carry one, 0/39 matched). repeats covers that case.
Cold-start recur spans are inference and are not cut by default.
A cross-show music-bed false positive is unsolved.
Not yet wired into hark, and not yet run against live data.
155 tests.
Adds acoustic ad recognition, so a campaign confirmed once is cut with no transcript and no model.
## What this adds
- **`fingerprint`** — matches episode AUDIO against Chromaprint fingerprints of ad recordings already confirmed in the corpus. Runs *before* transcription, so unlike `repeats` it saves the Whisper cost too.
- **`dai`** — persists DAI probe divergences, which the probe previously proved and then discarded. Ad discovery with no transcript, no model.
- **`discover`** — cold start for a feed with nothing confirmed: match the feed against itself.
- **Coverage-aware prompting** — the model is only sent transcript no cheaper tier has covered.
- **`cut` is source-filtered** — only tiers that pin their span EDGES may remove audio.
## Measured (real corpus, leave-one-out)
- **89.6%** of confirmed ad duration recovered from audio alone (Casefile, 82 eps, 286 ads), 0 episodes fully missed, 0/82 non-ad control false-match.
- Cross-show on The Casual Criminalist (40 eps, **zero labels**): recurring ad audio in 40/40, and 13/40 share the Flexcar DAI campaign with Casefile — a library from one show recognises another show's ads.
- End-to-end cut verified on real audio: 6392.7s -> 5945.1s, arithmetic exact, 4/5 removed spans verifiably ads.
## Things measured and then *rejected*
Recorded so they aren't re-attempted blind:
- **Editorial-only stop-list**: 76.8% recall (74,075 values stopped vs 1,173). Frequency ∩ editorial wins at 89.6%.
- **Nearest-silence cut snapping**: measured worse than shrink-only; the closest silence is often a pause *inside* speech.
- **Match-density guard**: no knee — 0.35 costs 1.0pp recall, 0.75 collapses to 60.2%.
## Known limits
- Host-read spots that are re-*read* per episode can't match (measured: 33/40 episodes carry one, 0/39 matched). `repeats` covers that case.
- Cold-start `recur` spans are inference and are not cut by default.
- A cross-show music-bed false positive is unsolved.
- Not yet wired into hark, and not yet run against live data.
155 tests.
Match an episode's audio against Chromaprint fingerprints of ad recordings
already confirmed elsewhere in the corpus (llm/chapter spans), so a campaign
confirmed once is cut with no transcript and no model. Runs before
transcription as a sibling audio stage (not a transcript AdSpanDetector),
which is the cost lever repeats can't pull.
- fingerprint.py: build_library over confirmed-ad audio (cached in
ad_fingerprints), AudioFingerprintDetector with diagonal exact-frame
matching + silence stop-list, whole-episode fingerprint cache
(episode_fingerprints) so re-scans re-run only matching, per-episode and
bulk apply. fpmatch spans are inference and never seed the library.
- CLI: adscrub fingerprint, cheapest-first after repeats.
- tests: 18 cases (matching logic + DB plumbing/cache/idempotency/isolation).
- Requires fpcalc (Chromaprint); clear error if absent.
Measured leave-one-out (Casefile, 82 eps): 89.1% of confirmed ad duration
recovered from audio alone, 0 episodes fully missed, 0/82 non-ad control
false-match. Cross-show on The Casual Criminalist: mechanism generalises
(recurring audio is real ads), and a library from one show catches another's
ads when they share a DAI campaign.
Two pipeline gaps found while validating the fingerprint tier.
dai.py proved which bytes were server-inserted and then discarded the
finding — nothing reached ad_segments. dai_episode now stores a `dai` span
(bytes converted via the file's average byte rate, trimmed, capped at one
plausible break, confidence 0.5) plus an `adscrub dai` command, giving ad
discovery with no transcript and no model. The span's start is evidence and
its end only an upper bound, so it seeds the audio library but deliberately
not repeats' text library, where a sloppy boundary would teach the matcher
editorial wording.
The stop-list dropped any value present in >30% of source episodes as
presumed silence, which also deletes a sponsor that runs in most episodes
(Flexcar: 27/40 Casual Criminalist episodes). It is now an intersection:
frequency proposes, known non-ad audio vetoes. Measured on Casefile —
frequency alone 88.9% recall, editorial alone 76.8% (too aggressive:
Chromaprint values collide between ad and ordinary speech), intersection
89.6% with 669 values stopped. Better recall than either, and the
ubiquitous sponsor survives.
Also adds the MIN_REGION_FRAMES emit floor (~10s), swept against ground
truth: 0.2pp recall for ~12% less false-positive time.
Two more pipeline gaps.
The model was re-reading transcript that cheaper tiers had already covered.
detect_episode now computes covered_segment_indices across every source and
_chunks omits those segments, so an episode already 60% covered costs ~40%
of the tokens it did. Omissions leave an elision marker, since without one
the segments either side of a removed ad read as adjacent and the model sees
a seam that isn't there; indices stay global so spans still ground against
the full transcript. AdSpanDetector.detect takes an optional skip argument,
which free tiers ignore on purpose.
The seeded tier could only recognise campaigns something else had already
confirmed, leaving a new feed with no way to get cheap. discover_recurring
matches a feed against itself: audio recurring across unrelated episodes is
the inserted material. Verified on 40 unlabelled Casual Criminalist
episodes — recurring audio in 40/40, 174 regions, reading as real ads. It
recovered that feed's DAI inserts and not its host-read Shopify spot, the
expected split. Spans are source='recur' (inference, never library-seeding).
RECUR_MIN_EPISODES is 8 on arithmetic, not taste: a campaign needs two
episodes to recur but must stay under the frequency threshold, so discovery
cannot work below about seven.
Also records a measured correction: being host-read does not defeat
fingerprinting, being re-read does. A spot recorded once and re-rolled
matches like any insert; this feed re-records Shopify per episode, so it
appears in 33/40 episodes and matches 0/39 while a DAI insert matches 2/39
and an editorial control 0/39.
cut selected every ad_segments row regardless of source, so the discovery
tiers added in the last two commits would have silently begun deleting
audio: dai spans whose end is only an upper bound, and recur spans of which
roughly one in ten is not an ad.
Cutting is now limited to CUT_SOURCES (chapter, llm, repeat, fpmatch). The
line is not evidence versus inference — repeat and fpmatch are inference and
are exactly what the cheap tiers exist to cut. It is whether the tier pins
the span's edges: those four ground their boundaries in publisher markers,
transcript segment timestamps, or audio alignment, while dai and recur find
ads well without saying where they stop. Good seeds, bad scissors.
pending_episodes filters on the same list, so an episode carrying only
discovery spans is not treated as pending — that would rewrite the file
unchanged and mark it cut, retiring it before a real span ever arrived.
Override with `adscrub cut --sources`.
The Dockerfile installed ffmpeg but not libchromaprint-tools, so in any
deploy the fingerprint and discover tiers were inert rather than broken:
fpcalc_available() returned False, the command exited tidily, and a
healthy-looking image silently never matched an ad.
Cutting a real episode (rather than trusting detection metrics) showed a
fingerprint match ends where the ad stops being recognisable, not where the
break ends. One edge sat 2.3s inside the resumed narration and the cut ate
the opening of "It was 405 on the morning of Thursday, June 19, 2014".
Snapping to the nearest silence was tried first and measured worse, 2.3s to
2.88s clipped, because the closest silence was a pause inside the narration.
Direction is the fix: starts only move later, ends only move earlier, so a
span shrinks and never grows and every error leaves a sliver of ad rather
than deleting a sentence.
This only helps where silence exists. On that episode it tightened three of
five edges and left the 2.31s clip untouched, since the ad-to-narration
transition has no detectable pause. That residual is a detection-edge
problem, likely BRIDGE_FRAMES extending a run, and is not solved here.
The previous commit claimed silence-snapping fixed a cut that "ate the
opening of 'It was 405 on the morning of Thursday, June 19, 2014'". That
diagnosis was wrong and is corrected here rather than left in the history to
mislead. Those frames match a confirmed ad recording from another episode
densely, gaps of one to three frames, and an episode's narration is unique to
it and cannot match another episode's audio. The region is the ad's outro bed
and Whisper had timestamped the segment early. Whisper segment starts are not
evidence of speech onset.
snap_spans_to_silence is kept on its own merit: an edge is a guess about where
a break ends, moving edges inward bounds the risk of running into speech, it
tightened three of five edges on a real cut, and by construction it can only
shrink what gets removed.
Sparse heavily-bridged regions looked like a false-positive signature, so a
minimum match-density guard was measured against ground truth. It has no
knee: 0.35 costs 1.0pp recall to remove 220s of outside-LLM time, 0.55 costs
2.0pp, 0.75 collapses recall to 60.2%. Outside-LLM time is not a synonym for
false positives either, since much of it is ads the model missed. Not shipped.
Cuts the acoustic-fingerprint work as 0.8.0: the fingerprint tier, dai
persistence, cold-start discovery, coverage-aware LLM prompting, the
frequency-intersect-editorial stop-list and source-filtered cutting.
Bumps __init__.__version__ alongside pyproject, which has drifted before,
and corrects two statements in the release notes that later commits in the
same branch had made false.
discover_recurring gains episode_ids so a caller holding many feeds in one
database can scope recurrence to a single show. hark has ~70 unrelated feeds
in one database, and recurrence is measured against whatever set it is given,
so pooling them would build one enormous index and compute the stop-list over
a corpus that shares no ad pool.
Also records that AdSpanDetector.detect gained a second argument. It is a
breaking change for implementors outside this package: LayeredDetector passes
skip positionally, so a two-argument detect() now raises TypeError. hark's own
_PrecomputedDetector hit exactly this.
Some required checks failed
Lint Python / lint (pull_request) Canceled after 0s
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Adds acoustic ad recognition, so a campaign confirmed once is cut with no transcript and no model.
What this adds
fingerprint— matches episode AUDIO against Chromaprint fingerprints of ad recordings already confirmed in the corpus. Runs before transcription, so unlikerepeatsit saves the Whisper cost too.dai— persists DAI probe divergences, which the probe previously proved and then discarded. Ad discovery with no transcript, no model.discover— cold start for a feed with nothing confirmed: match the feed against itself.cutis source-filtered — only tiers that pin their span EDGES may remove audio.Measured (real corpus, leave-one-out)
Things measured and then rejected
Recorded so they aren't re-attempted blind:
Known limits
repeatscovers that case.recurspans are inference and are not cut by default.155 tests.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.