- /stats: corpus ad-stripping impact + per-account listening summary
- Per-show "chapters" mode: mark ads as Podcasting-2.0 <podcast:chapters>
instead of cutting; honored in the recommendation feed as well
- /feeds.opml: bulk export of all subscribable feed URLs
- /search genre filter across topics, episode titles, and transcripts
- Share the episode->genre predicate (queries.episode_in_genre) and the
cut-vs-chapters enclosure resolution across both feed builders
A live probe found 2960 episode actions on the Nextcloud GpodderSync server
while listen_actions was empty: an early sync-history advanced the cursor
past all history without ingesting it, so every later run fetched since=now
and got nothing. sync-history now re-fetches from since=0 whenever
listen_actions is empty (self-heal) + gains --full. This was a hark bug,
not the AntennaPod/Nextcloud config issue first assumed.
Topic pages show a coverage span + chronological sort. Recommendation feeds
take a ?genre filter (per-genre URLs on the /feeds hub). Episode pages link
multi-part series siblings (series.py title heuristic). Adds a read-only
JSON API (/api/episode, /api/topic), 401-JSON when unauthenticated. Fixes a
recommendation-feed key bug (episode_id) that would have 500'd on real
recommendations, and adds a subscribed-shows fallback so the feed is never
empty when listening history hasn't synced.
Surfaces and closes the loop on the ad pipeline. Episode pages show the ads
found/removed (timestamp, tier, reason, time saved) and let an admin
hand-mark or remove spans (adscrub 0.18.0 'manual' ground truth -> cut +
library seed + re-cut). Adds a personal /recommended/<token> RSS feed (from
scoring), a /feeds hub listing every subscribable feed URL, and full-text
transcript search over an FTS5 index built by a new index-transcripts stage.
Bumps adscrub 0.17.0 -> 0.18.0.
The streaming fingerprint index is a corpus-coverage backfill with no span
output for the current cycle; sitting it mid-cycle (before transcribe/
fp-match/cut) delayed the ad-stripped audio the player receives behind a
slow network backfill. Moved it after cut — its fingerprints are matched
next cycle, a one-cycle lag that's nothing for a whole-corpus backfill.
/pipeline gains a 'What the LLM would run' panel: for ad detection it lists
the actual seed episodes detect-ads would read (the campaign set-cover, the
fewest covering every unread campaign) with per-episode cost, and states
that the other pending episodes are caught free by fingerprinting once
these are confirmed. seeds --count persists that set to llm_ad_seeds so the
read-only web can show it. Updates docs/PLAN.md with the ad-pipeline
maturation and current status.
Adds a /pipeline page showing every stage's cadence, gate, last run, and a
colour-coded status badge (ran / error (exit N) / idle / needs key or
budget), plus ad-tier roles, work queues, and today's LLM spend. run_cycle
now persists every stage's outcome (pipeline_runs gains last_status/
last_seen/last_exit, migrated in place) so the dashboard shows real state;
the web app reads it through a guarded query. The home pipeline banner is
now a compact one-liner linking to the dashboard.
Adds a compare pipeline stage gated on a key + the comparisons budget
(per-topic cap), lighting up treatment comparison once funded. Adds a
bounded streaming fingerprint index (fingerprint --index --stream) so
coverage reaches the whole ~27.8k corpus without storing audio. DAI probing
does per-platform 3 and skips platforms proven non-DAI so budget lands where
insertion actually happens; long ads get their end found (adscrub 0.17.0's
escalating probe). compare exits 0 on empty. Bumps adscrub 0.16.0 -> 0.17.0.
cmd_cut now HOLDS an anomalous cut (adscrub 0.16.0 hold_cut): reverts
cut_path so the feed serves the original and stamps cut_held_at, so a
likely false-positive cut never reaches the player. Adds 'verify-inference'
(+ a free pipeline stage): per-source span count, median duration, and
suspect-short share as a drift signal, with --sample to render spans for
review. Adds opt-in ntfy alerting (alert.py, HARK_NTFY_URL) that pages on a
cycle error or any stage exiting non-zero, deduped hourly. Bumps adscrub
0.15.0 -> 0.16.0.
Fixes a transcription head-of-line stall: a show's expired-CDN back
catalogue (repeated 410 Gone) tripped the consecutive-failure abort every
cycle, producing 0 new transcripts while 26.5k stayed pending. cmd_transcribe
now quarantines a permanently-gone URL (adscrub 0.15.0) via a new
audio_gone_at migration and doesn't count it toward the abort. transcribe/
cut/detect-ads now exit 0 on an empty queue (nothing-to-do != failure).
Adds 'seeds --count' + an ad-seeds stage surfacing unread ad campaigns for
free, and flags anomalous (>35%) cut fractions for review. Bumps adscrub to
0.15.0; documents the recur tier as intentionally unwired.
detect-ads draws from HARK_LLM_ADS_BUDGET, the topic-index/comparison LLM
work (extract, and a future compare) from HARK_LLM_COMPARISONS_BUDGET —
two independent daily pools metered in llm_spend(day, category), replacing
the single ads-first HARK_LLM_DAILY_BUDGET (kept as an ads-only alias). The
old single-column llm_spend is migrated in place. extract becomes a
pipeline stage, gated on a key + the comparisons budget and enforcing it
per episode, running before canon so new topics canonicalise the same cycle.
run_cycle streams a per-stage heartbeat (→ start, ran on finish, exit code
on failure) via an optional log hook, so a long stage (whisper) is
observable in real time instead of the cycle's outcomes arriving in one
burst at the end. rotate_log copytruncates the pipeline log between cycles
past HARK_LOG_MAX_BYTES (default 25 MB), keeping one .1 backup; in-place
truncation is required because the container holds the log open O_APPEND.
orchestrator shells out via python -m hark; without a __main__.py that fails
and every pipeline stage errors. Adds the entry point and a test that a real
stage spawn returns 0.
The pipeline was a 1755-char shell string in the container's compose config:
unversioned, untestable, and changed by string surgery on production.
orchestrator.py makes the orchestration first-class code — stage order,
cadence, and gating are now tested and shipped in the image, changed by
rebuild instead of editing a live box. It stays single-process, one stage at
a time, which is the right shape for one SQLite writer.
Each stage declares a cadence and its gates. Free stages always run.
detect-ads is gated on both a key and a daily budget, so a key alone never
spends. llm_budget.py meters spend per UTC day from an estimate of the
transcript actually sent, conservative so it stops sooner rather than later,
and detect-ads stops before the next episode once the cap is reached. The
session-as-extractor drop files are still loaded and archived every cycle, so
the no-key path is unchanged.
fingerprint gains --index for bounded indexing and defaults to indexed-only
matching, so the one-time fpcalc backfill spreads across cycles instead of
stalling the loop. Pins adscrub 0.14.0.
dai-probe ran every cycle and discarded what it found. It now stores each
divergence via adscrub 0.13.0's store_probe_result, seeding the fingerprint
library with server-inserted ads for free. Persists only when the episode's
audio is on disk to convert the probe's byte offsets to seconds, so it is
silent rather than failed otherwise.
The deploy pairs this with a loop-command change that runs fingerprint and a
key-gated detect-ads in the slow gate; the subscription drop-file path is
unchanged, so detection works with or without an API key.
Picks up transcript corroboration for fpmatch spans, and adds the tests seeds
shipped without: only selected episodes are emitted, covered regions are
omitted while indices stay global across the omission, the hand-back contract
is printed, and an empty selection exits non-zero.
hark seeds is the subscription path's twin of detect-ads. Both answer what
is worth reading with the same campaign set cover; they differ only in who
does the reading. seeds emits the selected episodes' transcripts for a Claude
Code session, which hands results back through load-ad-detections.
Segments are rendered with adscrub's own chunk renderer rather than a
hark-side copy, so the indices a reader points at are the ones
spans_from_segment_indices grounds against on the way back in. Regions an
earlier tier already covered are omitted, as on the API path.
Verified end to end on 40 real episodes with no audio files on disk, working
purely from cached fingerprints: emitted two episodes covering five
campaigns, read one, loaded three spans, and the episode and its campaigns
dropped out of the next selection.
Also stops fingerprint downloading audio it was not asked to. It selected
every episode with an audio_url, about 27,865 here, and adscrub downloads
what is not cached, so an unguarded sweep would have pulled terabytes from
podcast CDNs. Local audio only now, with --download to opt in.
pending_episodes counts every transcribed episode the model has not read,
a definition written before anything could tell which were worth reading.
On this corpus that is 1,262 episodes, roughly $132, for data that is both
largely redundant and measurably lossy: fingerprinting already recognises a
campaign everywhere once any one episode of it is confirmed, and adscrub's
own validation found a provably-missed ad in 62% of episodes.
detect-ads now selects through adscrub's campaign set cover: the fewest
episodes confirming every recording the library does not have. On 40 Casual
Criminalist episodes that is 7 episodes for 11 campaigns. The queue shrinks
as campaigns are confirmed and grows only for genuinely new ones.
Nothing is marked processed and no episode is retired. Setting
llm_detected_at without reading would be the 0.6.0 bug on purpose. Episodes
that are not selected are simply not needed yet, and --all-pending restores
the old sweep. With too little audio for self-recurrence the command falls
back to that sweep rather than reading nothing.
The header still described adscrub as a local ../adscrub path dependency; it
became a git source on 2026-07-13. Records what the staging is actually still
for, which is building from uncommitted local adscrub work.
The image installed ffmpeg but not libchromaprint-tools, so hark fingerprint
and discover-ads would have been inert in the deploy rather than broken:
fpcalc_available() returns False, the command exits tidily, and a
healthy-looking container silently never matches an ad. Same defect adscrub
0.8.0 fixed in its own image.
Bumps __version__ alongside pyproject, which has drifted in sibling repos.
Moves the git pin from 59f9c837 (0.7.2) to 9f05892 (0.8.0), which is what
brings adscrub.fingerprint into reach and satisfies the AdSpanDetector
protocol change _PrecomputedDetector was updated for. Verified against the
real GitHub-resolved dependency rather than a local editable override.
hark fingerprint matches episode audio against ad recordings already
confirmed in the corpus. It costs no tokens and needs no transcript, so it
can run while the Claude-driven pipeline stays disabled. discover-ads is the
cold start for a single show, finding ads by matching that show's episodes
against each other.
Both go per-episode rather than through adscrub's bulk helpers, for the same
reason cmd_repeats does: the bulk versions run their own episode query with
no way to restrict it to ad-stripping-enabled shows. discover-ads is scoped
to one show deliberately, since recurrence is measured against whatever set
it is given and this database holds ~70 unrelated feeds.
_PrecomputedDetector gains the skip argument AdSpanDetector.detect grew in
adscrub 0.8.0. LayeredDetector passes it positionally, so without this every
load-ad-detections record failed with a TypeError.
REQUIRES adscrub >= 0.8.0. cli.py imports adscrub.fingerprint at module
scope, so with the currently pinned 0.7.2 every hark command fails to start.
Do not merge until the adscrub dependency resolves 0.8.0.
Breadcrumbs and active-nav state on every page; consistent pagination on
/topic/<id>, /shows, and /search's episode results (all were unbounded or
hard-capped before); claims comparisons now render in full on /topic/<id>
(the data is topic-scoped, not episode-scoped) with a trimmed pointer-back
view on /episode/<id> instead of repeating every show's claims on each of
its episodes; /notable and /admin/users split their stacked unrelated
sections into tabs. Plus a copy button on feed/invite URLs, inline
subscribe/unsubscribe on /shows, a two-step confirm on account removal, a
sort control on /topics, aggregate genre pills and a combined settings row
on /show/<id>, and a subscribed-shows/top-genres summary on /account.
select_sample() used to exclude an episode forever after its first
probe. Wrong now that we know results are noisy: acast.com flipped
from diverged to identical on an otherwise-identical re-test.
Episodes stay eligible, least-probed-first, until --min-trials
attempts (default 3) are recorded.
Matches adscrub 0.7.1's probe_variance() signature change
(client_factory instead of a shared client) — a shared client's
cookie jar was silently defeating the comparison.
hosting.py classifies a show's hosting platform from its episodes'
audio_url (the registrable domain, e.g. "acast.com"). dai_probe.py
selects up to --per-platform untested episodes per distinct platform
and runs adscrub's dual-fetch probe against each, storing every
attempt in the new dai_probes table so results are comparable across
hosts rather than clustering on whichever shows sync first.
Wraps adscrub 0.6.1's repeats.prioritize_pending(): episodes where
the repeat tier's found ad-break count doesn't match the show's
typical count go first, so --limit spends budget on the episodes
most likely to actually need it.
Wraps adscrub 0.6.0's repeat tier. Ads arrive in batches — the ad server rotates a
small pool of campaigns and we download each episode once, server-side, from that
pool — so the same reads recur near-verbatim across episodes fetched in the same
period. Leave-one-out on the live corpus: 93.5% of confirmed ad segments are
recoverable from ad reads confirmed in OTHER episodes, with no model called.
On the live database: 958 ad spans across 339 episodes in 3.7 seconds, and 259 of
those episodes had never been seen by the LLM at all. At the LLM tier's ~$0.23 an
episode that queue would have cost ~$60 to work through.
Per-episode via adscrub's repeat_episode(), mirroring how detect-ads uses
detect_episode(), so hark's per-show ad_stripping_enabled toggle actually takes
effect. Idempotent, because the library grows and re-scanning is the point: an
episode scanned when ten ad reads were known deserves another look once a thousand
are. It never sets llm_detected_at — a free pass that never read the words must not
retire an episode from the model.
Also picks up adscrub's fix for ClaudeAdDetector truncating every transcript to its
first 20,000 characters and then marking the episode detected. Latent here (the
deployed pipeline loads pre-computed detections rather than calling detect-ads), but
one ANTHROPIC_API_KEY away from silently under-detecting the whole corpus.
0.17.2-0.17.4 were built with the default GPU=0, omitting the cuBLAS/cuDNN
wheels, while the deployed app still reserves the RTX 2070 SUPER. The container
could see the device but not use it, so faster-whisper picked CUDA, failed at
first inference on the missing libcublas.so.12, and fell back to CPU int8 --
~3.7 cores saturated against a 27k-episode backlog with the GPU at 0% util.
Pin adscrub 0.5.2 so that fallback logs loudly, and warn when the build script
is producing a CPU-only image.
The deployed transcribe container was crashing on every episode with
Library libcublas.so.12 is not found or cannot be loaded. adscrub
0.5.1 catches that and falls back to CPU. No changes in hark's own
source.
A real fleet-agent batch dropped in production used the shorthand pair
form instead of the documented {start_segment, end_segment, reason}
dict. Every record in it failed with 'list' object has no attribute
'get', the file was never renamed to loaded-*, and the deployed loop
retried and failed on it every cycle indefinitely.
recanonicalize() swept every unmatched topic unconditionally, one live
Wikidata lookup each with its own retry/backoff. The deployed pipeline
runs canon sequentially ahead of the fast loop's pending-file loads and
transcribe/cut steps, so a large backlog (514 unmatched topics after the
2026-07-12 gpodder sync) blocked the entire loop for hours with no
visible progress in the log.
/admin/users gained a "Show ratings" section: a "Refresh now" button
running the same two steps hark rate-shows does (itunes_id backfill,
then Taddy ratings if $HARK_TADDY_USER_ID/$HARK_TADDY_API_KEY are
set) — for when SSH/Shell access to the deployed container isn't
convenient, matching the reason /admin/users itself exists. Manual
only; automatic/scheduled runs stay a deliberate follow-up.
Also documents the first live verification: hark rate-shows --limit 5
against the real catalog matched 4/5 shows on the first attempt with
no fix needed — the query shape, field names, and auth headers all
held up as built.
Three changes, no functional behavior change beyond request count:
- RATINGS_STALE_DAYS 30 -> 90: a coarse popularity tier doesn't move
fast enough to justify monthly re-checks, let alone the old cadence.
- New RATINGS_MISS_STALE_DAYS (180): a confirmed "Taddy doesn't have
this show" is even less likely to change soon than an existing
match's tier, and now waits separately/longer before retrying.
- Batch re-fetching via getMultiplePodcastSeries (up to 25 shows per
request) for any show already matched to a known Taddy uuid, instead
of one getPodcastSeries lookup each. This is the one that actually
matters long-run: after a catalog's initial backfill, nearly every
subsequent hark rate-shows run is re-checking already-matched shows,
so this turns steady-state consumption from ~1 request/show into
~1 request per 25 shows. A batch failure fails that whole batch
(retried, re-batched, next run) rather than falling back to
per-show requests, which would defeat the point of batching.
Podchaser's rating fields turned out to need a paid tier — never
deployed against it. Replaced with Taddy's free-tier GraphQL API
(500 requests/month, no card), the only genuinely free option found
with a real per-show signal after also checking iTunes Search
(verified empirically: no rating fields for podcasts), Spotify Web
API (verified via official docs: no popularity field on Show
objects), PodcastIndex.org (verified via its OpenAPI spec: no
per-show rating field), and Listen Notes (Listen Score/Global Rank
explicitly Pro-tier-only per its own pricing page).
Auth is simpler too: two static headers (X-USER-ID, X-API-KEY), no
OAuth token exchange. But the signal itself is different in kind —
Taddy's popularityRank is a coarse tier ("TOP_1000", etc.) against
its 4M+-podcast index, not a star average, so it's mapped to a 0-5
score and fed through the same Bayesian-shrinkage machinery personal
affinity already uses. Expect this to be genuinely sparse for hark's
mostly-niche catalog — most shows won't crack even TOP_100000 — which
the existing None-collapsing design already handles without a special
case; a real Taddy match with no tier is now stored distinctly from
"not found at all" so show_ratings stays informative either way.
$HARK_PODCHASER_CLIENT_ID/$HARK_PODCHASER_CLIENT_SECRET ->
$HARK_TADDY_USER_ID/$HARK_TADDY_API_KEY throughout cli.py/docs.
ratings.py shipped assuming a single api_key used directly as a
Bearer token — confirmed wrong on further research (corroborated
across the account settings docs, the authorization guide, and a
requestAccessToken mutation reference, since api-docs.podchaser.com
itself 403'd every direct fetch attempt).
Podchaser actually uses OAuth2 client-credentials: client_id +
client_secret (from the account's own API settings page) are
exchanged for a Bearer access token via a requestAccessToken
*mutation* — there's no separate REST token endpoint, it's GraphQL
end to end. PodchaserRatingsSource now does that exchange lazily on
first fetch() (not in __init__, so a batch with nothing to refresh
never authenticates for nothing) and reuses the token for the
lifetime of the instance — Podchaser's own docs say tokens are valid
for about a year.
$HARK_PODCHASER_API_KEY -> $HARK_PODCHASER_CLIENT_ID +
$HARK_PODCHASER_CLIENT_SECRET throughout cli.py/docs.
flan