- /stats: corpus ad-stripping impact + per-account listening summary
- Per-show "chapters" mode: mark ads as Podcasting-2.0 <podcast:chapters>
instead of cutting; honored in the recommendation feed as well
- /feeds.opml: bulk export of all subscribable feed URLs
- /search genre filter across topics, episode titles, and transcripts
- Share the episode->genre predicate (queries.episode_in_genre) and the
cut-vs-chapters enclosure resolution across both feed builders
A live probe found 2960 episode actions on the Nextcloud GpodderSync server
while listen_actions was empty: an early sync-history advanced the cursor
past all history without ingesting it, so every later run fetched since=now
and got nothing. sync-history now re-fetches from since=0 whenever
listen_actions is empty (self-heal) + gains --full. This was a hark bug,
not the AntennaPod/Nextcloud config issue first assumed.
Topic pages show a coverage span + chronological sort. Recommendation feeds
take a ?genre filter (per-genre URLs on the /feeds hub). Episode pages link
multi-part series siblings (series.py title heuristic). Adds a read-only
JSON API (/api/episode, /api/topic), 401-JSON when unauthenticated. Fixes a
recommendation-feed key bug (episode_id) that would have 500'd on real
recommendations, and adds a subscribed-shows fallback so the feed is never
empty when listening history hasn't synced.
Surfaces and closes the loop on the ad pipeline. Episode pages show the ads
found/removed (timestamp, tier, reason, time saved) and let an admin
hand-mark or remove spans (adscrub 0.18.0 'manual' ground truth -> cut +
library seed + re-cut). Adds a personal /recommended/<token> RSS feed (from
scoring), a /feeds hub listing every subscribable feed URL, and full-text
transcript search over an FTS5 index built by a new index-transcripts stage.
Bumps adscrub 0.17.0 -> 0.18.0.
The streaming fingerprint index is a corpus-coverage backfill with no span
output for the current cycle; sitting it mid-cycle (before transcribe/
fp-match/cut) delayed the ad-stripped audio the player receives behind a
slow network backfill. Moved it after cut — its fingerprints are matched
next cycle, a one-cycle lag that's nothing for a whole-corpus backfill.
/pipeline gains a 'What the LLM would run' panel: for ad detection it lists
the actual seed episodes detect-ads would read (the campaign set-cover, the
fewest covering every unread campaign) with per-episode cost, and states
that the other pending episodes are caught free by fingerprinting once
these are confirmed. seeds --count persists that set to llm_ad_seeds so the
read-only web can show it. Updates docs/PLAN.md with the ad-pipeline
maturation and current status.
Adds a /pipeline page showing every stage's cadence, gate, last run, and a
colour-coded status badge (ran / error (exit N) / idle / needs key or
budget), plus ad-tier roles, work queues, and today's LLM spend. run_cycle
now persists every stage's outcome (pipeline_runs gains last_status/
last_seen/last_exit, migrated in place) so the dashboard shows real state;
the web app reads it through a guarded query. The home pipeline banner is
now a compact one-liner linking to the dashboard.
Adds a compare pipeline stage gated on a key + the comparisons budget
(per-topic cap), lighting up treatment comparison once funded. Adds a
bounded streaming fingerprint index (fingerprint --index --stream) so
coverage reaches the whole ~27.8k corpus without storing audio. DAI probing
does per-platform 3 and skips platforms proven non-DAI so budget lands where
insertion actually happens; long ads get their end found (adscrub 0.17.0's
escalating probe). compare exits 0 on empty. Bumps adscrub 0.16.0 -> 0.17.0.
cmd_cut now HOLDS an anomalous cut (adscrub 0.16.0 hold_cut): reverts
cut_path so the feed serves the original and stamps cut_held_at, so a
likely false-positive cut never reaches the player. Adds 'verify-inference'
(+ a free pipeline stage): per-source span count, median duration, and
suspect-short share as a drift signal, with --sample to render spans for
review. Adds opt-in ntfy alerting (alert.py, HARK_NTFY_URL) that pages on a
cycle error or any stage exiting non-zero, deduped hourly. Bumps adscrub
0.15.0 -> 0.16.0.
Fixes a transcription head-of-line stall: a show's expired-CDN back
catalogue (repeated 410 Gone) tripped the consecutive-failure abort every
cycle, producing 0 new transcripts while 26.5k stayed pending. cmd_transcribe
now quarantines a permanently-gone URL (adscrub 0.15.0) via a new
audio_gone_at migration and doesn't count it toward the abort. transcribe/
cut/detect-ads now exit 0 on an empty queue (nothing-to-do != failure).
Adds 'seeds --count' + an ad-seeds stage surfacing unread ad campaigns for
free, and flags anomalous (>35%) cut fractions for review. Bumps adscrub to
0.15.0; documents the recur tier as intentionally unwired.
detect-ads draws from HARK_LLM_ADS_BUDGET, the topic-index/comparison LLM
work (extract, and a future compare) from HARK_LLM_COMPARISONS_BUDGET —
two independent daily pools metered in llm_spend(day, category), replacing
the single ads-first HARK_LLM_DAILY_BUDGET (kept as an ads-only alias). The
old single-column llm_spend is migrated in place. extract becomes a
pipeline stage, gated on a key + the comparisons budget and enforcing it
per episode, running before canon so new topics canonicalise the same cycle.
run_cycle streams a per-stage heartbeat (→ start, ran on finish, exit code
on failure) via an optional log hook, so a long stage (whisper) is
observable in real time instead of the cycle's outcomes arriving in one
burst at the end. rotate_log copytruncates the pipeline log between cycles
past HARK_LOG_MAX_BYTES (default 25 MB), keeping one .1 backup; in-place
truncation is required because the container holds the log open O_APPEND.
orchestrator shells out via python -m hark; without a __main__.py that fails
and every pipeline stage errors. Adds the entry point and a test that a real
stage spawn returns 0.
The pipeline was a 1755-char shell string in the container's compose config:
unversioned, untestable, and changed by string surgery on production.
orchestrator.py makes the orchestration first-class code — stage order,
cadence, and gating are now tested and shipped in the image, changed by
rebuild instead of editing a live box. It stays single-process, one stage at
a time, which is the right shape for one SQLite writer.
Each stage declares a cadence and its gates. Free stages always run.
detect-ads is gated on both a key and a daily budget, so a key alone never
spends. llm_budget.py meters spend per UTC day from an estimate of the
transcript actually sent, conservative so it stops sooner rather than later,
and detect-ads stops before the next episode once the cap is reached. The
session-as-extractor drop files are still loaded and archived every cycle, so
the no-key path is unchanged.
fingerprint gains --index for bounded indexing and defaults to indexed-only
matching, so the one-time fpcalc backfill spreads across cycles instead of
stalling the loop. Pins adscrub 0.14.0.
dai-probe ran every cycle and discarded what it found. It now stores each
divergence via adscrub 0.13.0's store_probe_result, seeding the fingerprint
library with server-inserted ads for free. Persists only when the episode's
audio is on disk to convert the probe's byte offsets to seconds, so it is
silent rather than failed otherwise.
The deploy pairs this with a loop-command change that runs fingerprint and a
key-gated detect-ads in the slow gate; the subscription drop-file path is
unchanged, so detection works with or without an API key.
Picks up transcript corroboration for fpmatch spans, and adds the tests seeds
shipped without: only selected episodes are emitted, covered regions are
omitted while indices stay global across the omission, the hand-back contract
is printed, and an empty selection exits non-zero.
hark seeds is the subscription path's twin of detect-ads. Both answer what
is worth reading with the same campaign set cover; they differ only in who
does the reading. seeds emits the selected episodes' transcripts for a Claude
Code session, which hands results back through load-ad-detections.
Segments are rendered with adscrub's own chunk renderer rather than a
hark-side copy, so the indices a reader points at are the ones
spans_from_segment_indices grounds against on the way back in. Regions an
earlier tier already covered are omitted, as on the API path.
Verified end to end on 40 real episodes with no audio files on disk, working
purely from cached fingerprints: emitted two episodes covering five
campaigns, read one, loaded three spans, and the episode and its campaigns
dropped out of the next selection.
Also stops fingerprint downloading audio it was not asked to. It selected
every episode with an audio_url, about 27,865 here, and adscrub downloads
what is not cached, so an unguarded sweep would have pulled terabytes from
podcast CDNs. Local audio only now, with --download to opt in.
pending_episodes counts every transcribed episode the model has not read,
a definition written before anything could tell which were worth reading.
On this corpus that is 1,262 episodes, roughly $132, for data that is both
largely redundant and measurably lossy: fingerprinting already recognises a
campaign everywhere once any one episode of it is confirmed, and adscrub's
own validation found a provably-missed ad in 62% of episodes.
detect-ads now selects through adscrub's campaign set cover: the fewest
episodes confirming every recording the library does not have. On 40 Casual
Criminalist episodes that is 7 episodes for 11 campaigns. The queue shrinks
as campaigns are confirmed and grows only for genuinely new ones.
Nothing is marked processed and no episode is retired. Setting
llm_detected_at without reading would be the 0.6.0 bug on purpose. Episodes
that are not selected are simply not needed yet, and --all-pending restores
the old sweep. With too little audio for self-recurrence the command falls
back to that sweep rather than reading nothing.
The header still described adscrub as a local ../adscrub path dependency; it
became a git source on 2026-07-13. Records what the staging is actually still
for, which is building from uncommitted local adscrub work.
The image installed ffmpeg but not libchromaprint-tools, so hark fingerprint
and discover-ads would have been inert in the deploy rather than broken:
fpcalc_available() returns False, the command exits tidily, and a
healthy-looking container silently never matches an ad. Same defect adscrub
0.8.0 fixed in its own image.
Bumps __version__ alongside pyproject, which has drifted in sibling repos.
Moves the git pin from 59f9c837 (0.7.2) to 9f05892 (0.8.0), which is what
brings adscrub.fingerprint into reach and satisfies the AdSpanDetector
protocol change _PrecomputedDetector was updated for. Verified against the
real GitHub-resolved dependency rather than a local editable override.
hark fingerprint matches episode audio against ad recordings already
confirmed in the corpus. It costs no tokens and needs no transcript, so it
can run while the Claude-driven pipeline stays disabled. discover-ads is the
cold start for a single show, finding ads by matching that show's episodes
against each other.
Both go per-episode rather than through adscrub's bulk helpers, for the same
reason cmd_repeats does: the bulk versions run their own episode query with
no way to restrict it to ad-stripping-enabled shows. discover-ads is scoped
to one show deliberately, since recurrence is measured against whatever set
it is given and this database holds ~70 unrelated feeds.
_PrecomputedDetector gains the skip argument AdSpanDetector.detect grew in
adscrub 0.8.0. LayeredDetector passes it positionally, so without this every
load-ad-detections record failed with a TypeError.
REQUIRES adscrub >= 0.8.0. cli.py imports adscrub.fingerprint at module
scope, so with the currently pinned 0.7.2 every hark command fails to start.
Do not merge until the adscrub dependency resolves 0.8.0.
Breadcrumbs and active-nav state on every page; consistent pagination on
/topic/<id>, /shows, and /search's episode results (all were unbounded or
hard-capped before); claims comparisons now render in full on /topic/<id>
(the data is topic-scoped, not episode-scoped) with a trimmed pointer-back
view on /episode/<id> instead of repeating every show's claims on each of
its episodes; /notable and /admin/users split their stacked unrelated
sections into tabs. Plus a copy button on feed/invite URLs, inline
subscribe/unsubscribe on /shows, a two-step confirm on account removal, a
sort control on /topics, aggregate genre pills and a combined settings row
on /show/<id>, and a subscribed-shows/top-genres summary on /account.
select_sample() used to exclude an episode forever after its first
probe. Wrong now that we know results are noisy: acast.com flipped
from diverged to identical on an otherwise-identical re-test.
Episodes stay eligible, least-probed-first, until --min-trials
attempts (default 3) are recorded.
Matches adscrub 0.7.1's probe_variance() signature change
(client_factory instead of a shared client) — a shared client's
cookie jar was silently defeating the comparison.
hosting.py classifies a show's hosting platform from its episodes'
audio_url (the registrable domain, e.g. "acast.com"). dai_probe.py
selects up to --per-platform untested episodes per distinct platform
and runs adscrub's dual-fetch probe against each, storing every
attempt in the new dai_probes table so results are comparable across
hosts rather than clustering on whichever shows sync first.
Wraps adscrub 0.6.1's repeats.prioritize_pending(): episodes where
the repeat tier's found ad-break count doesn't match the show's
typical count go first, so --limit spends budget on the episodes
most likely to actually need it.
Wraps adscrub 0.6.0's repeat tier. Ads arrive in batches — the ad server rotates a
small pool of campaigns and we download each episode once, server-side, from that
pool — so the same reads recur near-verbatim across episodes fetched in the same
period. Leave-one-out on the live corpus: 93.5% of confirmed ad segments are
recoverable from ad reads confirmed in OTHER episodes, with no model called.
On the live database: 958 ad spans across 339 episodes in 3.7 seconds, and 259 of
those episodes had never been seen by the LLM at all. At the LLM tier's ~$0.23 an
episode that queue would have cost ~$60 to work through.
Per-episode via adscrub's repeat_episode(), mirroring how detect-ads uses
detect_episode(), so hark's per-show ad_stripping_enabled toggle actually takes
effect. Idempotent, because the library grows and re-scanning is the point: an
episode scanned when ten ad reads were known deserves another look once a thousand
are. It never sets llm_detected_at — a free pass that never read the words must not
retire an episode from the model.
Also picks up adscrub's fix for ClaudeAdDetector truncating every transcript to its
first 20,000 characters and then marking the episode detected. Latent here (the
deployed pipeline loads pre-computed detections rather than calling detect-ads), but
one ANTHROPIC_API_KEY away from silently under-detecting the whole corpus.
0.17.2-0.17.4 were built with the default GPU=0, omitting the cuBLAS/cuDNN
wheels, while the deployed app still reserves the RTX 2070 SUPER. The container
could see the device but not use it, so faster-whisper picked CUDA, failed at
first inference on the missing libcublas.so.12, and fell back to CPU int8 --
~3.7 cores saturated against a 27k-episode backlog with the GPU at 0% util.
Pin adscrub 0.5.2 so that fallback logs loudly, and warn when the build script
is producing a CPU-only image.
The deployed transcribe container was crashing on every episode with
Library libcublas.so.12 is not found or cannot be loaded. adscrub
0.5.1 catches that and falls back to CPU. No changes in hark's own
source.
A real fleet-agent batch dropped in production used the shorthand pair
form instead of the documented {start_segment, end_segment, reason}
dict. Every record in it failed with 'list' object has no attribute
'get', the file was never renamed to loaded-*, and the deployed loop
retried and failed on it every cycle indefinitely.
recanonicalize() swept every unmatched topic unconditionally, one live
Wikidata lookup each with its own retry/backoff. The deployed pipeline
runs canon sequentially ahead of the fast loop's pending-file loads and
transcribe/cut steps, so a large backlog (514 unmatched topics after the
2026-07-12 gpodder sync) blocked the entire loop for hours with no
visible progress in the log.
/admin/users gained a "Show ratings" section: a "Refresh now" button
running the same two steps hark rate-shows does (itunes_id backfill,
then Taddy ratings if $HARK_TADDY_USER_ID/$HARK_TADDY_API_KEY are
set) — for when SSH/Shell access to the deployed container isn't
convenient, matching the reason /admin/users itself exists. Manual
only; automatic/scheduled runs stay a deliberate follow-up.
Also documents the first live verification: hark rate-shows --limit 5
against the real catalog matched 4/5 shows on the first attempt with
no fix needed — the query shape, field names, and auth headers all
held up as built.
Three changes, no functional behavior change beyond request count:
- RATINGS_STALE_DAYS 30 -> 90: a coarse popularity tier doesn't move
fast enough to justify monthly re-checks, let alone the old cadence.
- New RATINGS_MISS_STALE_DAYS (180): a confirmed "Taddy doesn't have
this show" is even less likely to change soon than an existing
match's tier, and now waits separately/longer before retrying.
- Batch re-fetching via getMultiplePodcastSeries (up to 25 shows per
request) for any show already matched to a known Taddy uuid, instead
of one getPodcastSeries lookup each. This is the one that actually
matters long-run: after a catalog's initial backfill, nearly every
subsequent hark rate-shows run is re-checking already-matched shows,
so this turns steady-state consumption from ~1 request/show into
~1 request per 25 shows. A batch failure fails that whole batch
(retried, re-batched, next run) rather than falling back to
per-show requests, which would defeat the point of batching.
Podchaser's rating fields turned out to need a paid tier — never
deployed against it. Replaced with Taddy's free-tier GraphQL API
(500 requests/month, no card), the only genuinely free option found
with a real per-show signal after also checking iTunes Search
(verified empirically: no rating fields for podcasts), Spotify Web
API (verified via official docs: no popularity field on Show
objects), PodcastIndex.org (verified via its OpenAPI spec: no
per-show rating field), and Listen Notes (Listen Score/Global Rank
explicitly Pro-tier-only per its own pricing page).
Auth is simpler too: two static headers (X-USER-ID, X-API-KEY), no
OAuth token exchange. But the signal itself is different in kind —
Taddy's popularityRank is a coarse tier ("TOP_1000", etc.) against
its 4M+-podcast index, not a star average, so it's mapped to a 0-5
score and fed through the same Bayesian-shrinkage machinery personal
affinity already uses. Expect this to be genuinely sparse for hark's
mostly-niche catalog — most shows won't crack even TOP_100000 — which
the existing None-collapsing design already handles without a special
case; a real Taddy match with no tier is now stored distinctly from
"not found at all" so show_ratings stays informative either way.
$HARK_PODCHASER_CLIENT_ID/$HARK_PODCHASER_CLIENT_SECRET ->
$HARK_TADDY_USER_ID/$HARK_TADDY_API_KEY throughout cli.py/docs.
ratings.py shipped assuming a single api_key used directly as a
Bearer token — confirmed wrong on further research (corroborated
across the account settings docs, the authorization guide, and a
requestAccessToken mutation reference, since api-docs.podchaser.com
itself 403'd every direct fetch attempt).
Podchaser actually uses OAuth2 client-credentials: client_id +
client_secret (from the account's own API settings page) are
exchanged for a Bearer access token via a requestAccessToken
*mutation* — there's no separate REST token endpoint, it's GraphQL
end to end. PodchaserRatingsSource now does that exchange lazily on
first fetch() (not in __init__, so a batch with nothing to refresh
never authenticates for nothing) and reuses the token for the
lifetime of the instance — Podchaser's own docs say tokens are valid
for about a year.
$HARK_PODCHASER_API_KEY -> $HARK_PODCHASER_CLIENT_ID +
$HARK_PODCHASER_CLIENT_SECRET throughout cli.py/docs.
"Interesting" now depends on the listener's own listening history plus
external show ratings, on explicit request — and neither needed an
LLM call anywhere: the whole computation is pure SQL/Python
arithmetic over data hark already has, which is both the cheapest
possible answer and the most auditable one.
- scoring.py (new): completion ratio per genre/topic from
listen_actions, Bayesian-shrunk toward the user's own average so a
single play doesn't swing a score to 0 or 1; combined with an
external show rating the same way. Every component returned
alongside the combined score, not hidden behind it. No listening
history at all collapses cleanly to pure external-rating ranking —
every component just reads None and the weighted average
renormalizes to whatever's present, no special-cased branch.
- ratings.py (new): Podchaser's free-tier GraphQL API, matched by RSS
feed URL then Apple Podcast ID — both exact-identifier lookups, no
fuzzy title matching. Cached in a new show_ratings table so a
rate-limited API isn't hit on every page view.
- resolve.py: backfill_itunes_ids() fills in the id for shows
registered via add_show_by_feed_url() (gpodder sync, OPML import —
most of a real catalog), which never set it the way resolve_show()
does. Verified by exact feed-URL match against the search result,
never a bare title guess.
- cli.py: new `hark rate-shows` command runs the backfill then
refreshes ratings; ratings half no-ops with a hint if
$HARK_PODCHASER_API_KEY isn't set.
- views.py: /notable leads with the new ranked list and a "your
genres" mirror, replacing its own long-standing "interim, not M4
yet" framing; the older contested-topics/rare-coverage sections
stay below it.
Per-topic treatment comparison (depth/sensationalism) — PLAN's other
M4 bullet — stays explicitly out of scope: it needs transcripts and
real LLM judgment, unlike the rest of this.
/admin/users was only reachable by typing the URL — the header showed
the logged-in username but never linked anywhere admin-only, despite
otherwise trying to be the one place every page is reachable from.
page() now takes is_admin and renders a "users" nav link (matching
that page's own heading, not "admin" — the default bootstrap account
is itself named "admin", so "admin admin account" read as a stutter)
whenever the logged-in account is one. Every authenticated view, plus
the 404/403/503 error pages, now passes it through.
--base-url/$HARK_BASE_URL was the only way to set the URL invite links
and the podcast feed/audio routes get built from — wrong hostname
meant a redeploy to fix. /admin/users now has a "Server settings"
section to set or reset an override live, stored in a new generic
settings key/value table in auth.db (not hark.db, for the same reason
accounts/sessions live there — a pipeline-pushed snapshot replaces
hark.db wholesale and must never wipe server config either).
App.base_url is now a property that re-reads the override on every
access rather than a fixed constructor value, so an edit takes effect
on the very next request with no restart. hark user invite (CLI)
checks the same override; an explicit --base-url flag still wins
outright over it.
Also bumps the CHANGELOG/docs for the six real bugs found and fixed in
the preceding systematic audit pass (ingest.py's failure isolation,
pipeline.py's topic-merge collision, the subscribe()/
record_subscription_changes() quota race, and three instances of the
same falsy-zero --limit bug), none of which had a version bump yet.
Fourth instance of the same falsy-zero bug this session: 'if limit:'
skipped the SQL LIMIT clause entirely for limit=0, returning every
topic instead of none. Reachable via 'hark topics --limit 0'. Found
during a deliberate re-sweep for this exact pattern across the
codebase after fixing the same bug three times over in
claims.pending_topics(), cli._filter_enabled(), and (implicitly,
already correct) pipeline.pending_episodes().
Same falsy-zero footgun just fixed in claims.pending_topics(): a
truthy 'if limit else filtered' treated limit=0 as 'no limit' rather
than 'zero results', so 'hark transcribe/detect-ads/cut --limit 0'
would process every enabled pending episode instead of none. Fixed by
just slicing — filtered[:limit] already handles limit=None correctly
via Python's own slice semantics, no conditional needed.
Both read the current user_shows count, then conditionally INSERT, as
separate statements with no lock held between them. The web server is
threaded (ThreadingHTTPServer) and gpodder-sync opens a fresh
connection per request, so two concurrent calls for the same user near
the cap (a double-click, two tabs, or the same account syncing from
two AntennaPod installs at once) could each read a count under
MAX_SHOWS_PER_USER before either committed, letting both through and
landing above it — the same class of concurrent-mutation hazard
_toggle_show_flag's own comment already calls out, just spanning
several statements instead of one atomic UPDATE.
Both now open with BEGIN IMMEDIATE, taking the write lock up front so
a second concurrent call blocks until the first commits or rolls back
instead of reading stale state. Verified against the actual race with
two real threads and a barrier — the new tests reliably fail without
the fix and reliably pass with it.
Two compounding bugs: the cap check used a truthy 'if limit and ...'
(limit=0 is falsy, so the break condition was never even evaluated),
and it ran *after* appending the current item, so even a corrected
'is not None' check still let one item leak through before the first
break. pipeline.pending_episodes() gets this right via SQL LIMIT,
which naturally returns zero rows for LIMIT 0; this hand-rolled cap
had drifted from that behavior. compare_pending() inherits the fix
since it delegates straight to pending_topics().
Two different real-world entities can share the exact same Wikidata
label text (e.g. 'Mercury' the planet vs. the element — Canonicalizer
just returns the raw wbsearchentities hit_label, no disambiguation).
recanonicalize() already guards against merging across such a
collision, but upsert_topic() — the normal extraction hot path used
by every extract_pending()/load_extractions() run, not just the
offline recovery pass — did not: when a new topic's QID lookup missed
but its label matched an existing row that already carried a
*different* QID, it silently reused that row, discarding the new
topic's real QID entirely and attributing it to the wrong entity.
Now it disambiguates the label (matching recanonicalize()'s own
'(QID)' suffix convention) and inserts a distinct row instead of
merging into an unrelated topic.
parse_feed()/upsert_episodes()/the shows-table update ran outside
ingest_show()'s try/except, so an exception there (a locked database,
a feedparser edge case) would propagate uncaught out of ingest_all()'s
list comprehension and abort every show still queued after it.
Worse: ingest_all() reuses one connection across every show, so a
partial upsert left uncommitted here would still be pending when a
later show's own conn.commit() ran — silently persisting this show's
broken write alongside an unrelated show's good one. Now the whole
parse+upsert+update block is wrapped, with an explicit rollback on
failure, matching the per-item isolation every other batch command in
this codebase already has (extract/transcribe/detect-ads/compare).
The passwordless-known-username branch skipped stretch() entirely
(nothing to check a hash against before falling through to the
admin-token comparison) while the unknown-username and has-a-password
branches both call it once — an unauthenticated caller could distinguish
"this username exists but has no password set yet" from the other two
cases purely by response time, without needing to guess anything.
stretch() now runs on every branch; the passwordless path's result is
discarded, kept only for timing parity.
Audited queries.py/templates.py alongside auth.py (today's web.py
split) — no other issues found.
resolve_all() committed only once, after the whole loop, and had no
per-name error handling — unlike every other batch command in this
codebase (extract/transcribe/detect-ads/compare), which all isolate
failures per item. A real feeds.txt run hitting a transient network
error partway through (the iTunes Search API call for one show) would
propagate uncaught and roll back every show already resolved earlier in
that same run. Now catches httpx.HTTPError per name (treated the same
as a genuine miss) and commits after each successful upsert.
Audited opml.py/nextcloud.py/podcast_feed.py/discover.py/wikidata.py/
extract.py alongside this — no other issues found, no changes needed.
Last and largest piece: the App class (dashboard/topic/show/episode
pages, subscribe/unsubscribe, admin toggles, invite management — ~730
lines) moves to views.py. web.py is now HTTP plumbing only (Handler,
make_server/serve, gpodder-sync request handlers) at 585 lines, down
from ~1900 before this three-part split — auth.py (263), queries.py
(170), templates.py (233), views.py (778).
Two real bugs caught by testing before this landed, not by the move
itself: cookie_attrs() referenced COOKIE, and view_home() referenced
parse_iso, both still only defined in the old web.py rather than
imported into the new module they'd moved to — the test suite's own
server-backed HTTP tests caught both immediately (connection resets
from the request-handler thread dying), verified again with a direct
smoke test hitting every page route after the fix.
Pure move otherwise: web.py re-exports everything cli.py/tests
reference as web.<name>, same as the previous two steps.
templates.py: STYLE/PAGE/HEADER/LOGIN_PAGE/INVITE_PAGE and the
presentation helpers that build HTML snippets from data (esc, plural,
page, relative_time, pagination_html, index_status_html,
pipeline_status_html, conf, episode_cell, topic_pills, topic_table).
queries.py: the read-only hark.db query helpers shared across pages
(topics_query/topics_count, related_topics/related_shows,
contested_topics, rare_genre_episodes, paginate).
Pure move, no behavior change — web.py re-exports everything cli.py and
the test suite already reference as web.<name>.
web.py had grown to ~1900 lines mixing auth, HTML templates, view logic,
and HTTP routing. First extraction: the Auth class and everything it
needs (schema, migrations, stretch/constant_eq, iso/parse_iso/utcnow) —
pure move, no behavior change. web.py re-exports the names cli.py/tests
already reference as web.Auth etc.
Fixed test_web.py's fast_stretch fixture to monkeypatch auth.PW_ITERS
instead of web.PW_ITERS — stretch() reads PW_ITERS from its own module's
global scope, so patching a re-exported binding elsewhere wouldn't
actually have affected it once the function moved.
Highest-leverage gap identified for this codebase: decent type hint
coverage already existed, but nothing was checking them. Added as a dev
dependency + a third CI workflow alongside lint.yml/test.yml, scoped to
src/ only (tests/ has dozens of harmless resp.getheader()-without-assert
call sites from http.client's own Optional[str] stubs — not worth
asserting around when the test already checked resp.status == 200 first).
Fixed everything it found in src/: several sqlite3.Cursor.lastrowid uses
assumed non-None (true after a real INSERT, now asserted explicitly
instead of just assumed), a narrowing gap in load_comparisons() and
index_status_html() the checker couldn't follow through an early-return/
indirect-bool pattern, and Handler.log_message/log_error had a renamed
parameter (fmt vs BaseHTTPRequestHandler's own format) that broke the
override's type compatibility without changing runtime behavior.
Also fixed a stale CLAUDE.md bullet still describing adscrub as a local
path dependency after the git-source switch.
The 0.15.0 migration added invite_token as `ALTER TABLE users ADD COLUMN
invite_token TEXT UNIQUE` — SQLite rejects a UNIQUE column on ADD COLUMN
entirely, so this would have thrown on every real pre-0.15.0 auth.db the
moment the deployed app restarted (caught by testing against a copy of
the actual production auth.db before deploying, not by the test suite,
which only ever exercised fresh schemas). Fixed by dropping the inline
UNIQUE and enforcing it via a separate CREATE UNIQUE INDEX statement
instead, run after the column-adding migrations (not inside AUTH_SCHEMA)
since invite_token doesn't exist yet on an upgrading database until then.
The switch to a git-source adscrub dependency (previous commit) broke
scripts/build-image.sh's build: the base image has no git binary, so
uv sync couldn't fetch it at all. Installs git for the frozen syncs to
resolve normally, then overrides adscrub with an editable install of the
staged local copy (uv pip install -e /adscrub, run last so the second
uv sync --frozen doesn't undo it) — same override the README's
side-by-side-dev instructions use, so a build via the script always
reflects the adscrub commit actually checked out locally, not whatever
uv.lock has pinned.
hark user invite / the new admin-only /admin/users page create accounts
with a single-use /invite/<token> link instead of handing out the shared
$HARK_ADMIN_TOKEN bootstrap — scoped to exactly one account, safe to send
a specific friend. The web page exists because this project's own deploy
has no container shell access, so CLI-only user management wasn't
actually usable day to day; both surfaces call the same Auth methods.
Non-admin accounts are capped at 10 subscriptions (MAX_SHOWS_PER_USER),
enforced identically on the web UI and AntennaPod-sync paths. The sync
path can't report a per-item rejection, so an over-quota feed_url is
silently skipped rather than logged and re-offered as "added" next sync.
Also fixed: Auth.set_password ran an unscoped DELETE FROM sessions,
which would have logged out every account whenever any one of them
changed their password — harmless before multi-user, a real bug now.
The adscrub dependency was a filesystem path (../adscrub), so a fresh clone could not be
installed and CI could not pass — Test CI has been red on main since the repo went public.
adscrub is public now, so depend on it by git URL; uv.lock pins the exact commit, and
side-by-side development still works via an editable override.
Add the missing LICENSE (AGPL-3.0, matching winnow/tendril) — without one the code was
legally unusable by anyone.
Lead the README with the dashboard screenshot and a four-bullet summary rather than five
dense paragraphs, and drop the now-obsolete sibling-clone setup step.
Stop hardcoding a private registry and hostnames: REGISTRY_IMAGE is overridable and
defaults to a local tag; the feed test uses an example host.
Outside readers cannot clone from git.onetick.ninja, so the adscrub sibling-checkout
instructions were unusable for anyone but me. Cross-reference adscrub by its GitHub URL
and record that Gitea stays canonical while GitHub is a read-only mirror.
Each account gets its own subscription list (user_shows) and listen
history (listen_actions.user_id, including in its own UNIQUE constraint);
shows/episodes/transcripts/ad_segments stay global and shared, so a show
two accounts both subscribe to is still only ever transcribed/ad-detected
once. gpodder_server.py and web.py's HTTP handlers thread the
authenticated user_id through instead of operating on the global tables
unscoped, so each account's AntennaPod install only ever sees its own
subscriptions/history.
hark user add/list/remove manages accounts (auth.db only). is_admin gates
the two genuinely global show-level toggles (ad-stripping, topic-index) —
everyone else just curates their own /shows list or subscribes via
AntennaPod directly.
Also fixed: auth.db never actually enabled PRAGMA foreign_keys (a
per-connection setting, not a schema property), so sessions.user_id's own
ON DELETE CASCADE was never enforced. Found while building hark user
remove, which depends on it.
scripts/build-image.sh resolved this 2026-07-11 (see docs/PLAN.md); the
README section never caught up and still pointed at a Dockerfile
"KNOWN GAP" comment that no longer exists.
hark load-ad-detections loads pre-computed ad-span detections the same way
load/load-comparisons already do for extraction and claims comparison — no
$ANTHROPIC_API_KEY needed. detect-ads had never actually been automated:
the deployed fast loop only ran transcribe/cut, so production ad-stripping
was chapter-markers-only this whole time while transcripts piled up
unprocessed. claude-fleet's hark-pipeline job gains a third section to
close that gap.
_PrecomputedDetector adapts session output to adscrub's AdSpanDetector
protocol, reusing detect_episode()/spans_from_segment_indices() (adscrub
0.5.0) unchanged.
Also: several UI strings told the owner to run `hark compare`, a CLI
command not exposed in the web UI at all. Comparison has been automated
since 0.12.0; the copy never caught up.
gpodder_server.py implements the exact four endpoints AntennaPod's own
NextcloudSyncService.java calls, confirmed against AntennaPod's real
source rather than guessed from the server side. Its login()/logout() are
no-ops, so pointing AntennaPod's existing "Nextcloud" sync setting at
hark directly works with zero app changes -- no fork needed for the
subscription/history sync half of M3.
New subscription_changes table gives hark its own timestamped add/remove
history for incremental since= replay. Also fixed a real gap:
listen_actions never captured the protocol's `started` field, which
AntennaPod's client requires for a play action to round-trip -- both the
sync-history client path and the new server path now go through one
shared record_episode_actions() instead of duplicating the insert logic.
Most of the 67 shows the 2026-07-12 gpodder sync added aren't subject-
per-episode genre shows at all (news, politics, personal finance) —
running extraction on one just burns session-as-X effort for a
guaranteed-empty result every time. topic_index_enabled now gates
pipeline.pending_episodes() separately from ad_stripping_enabled (which
is meant to cover every subscription, a different question). Defaults on
for hand-curated hark-resolve shows, off for anything added via the bare
feed-URL path (sync-subscriptions/import-opml/discover --add) pending a
look at the show page. Existing databases get a one-time migration
backfill; /shows flags unreviewed shows so a large sync batch is easy to
work through.
M3: hark sync-subscriptions reads Nextcloud's GPodder Sync app and
registers subscribed feeds hark doesn't already track, so ad-stripping
covers real subscriptions instead of the manually-curated feeds.txt list.
hark sync-history pulls play-history events into a new listen_actions
table for future M4 scoring. hark import-opml is the same registration
path from a file. The deployed transcribe service runs both syncs once
at container start.
M2: hark discover does a cheap iTunes Search sweep for candidate shows
not yet tracked (report-only unless --add). A new /notable page surfaces
two interim signals distinct from the existing coverage-count ranking:
topics whose shows' claims diverge the most, and episodes in hark's
rarest genres — explicitly labeled provisional pending M4's real scoring.
resolve.add_show_by_feed_url() is the shared "register a show hark
already has a feed URL for" path behind all three new entry points.
Previously the only way to see how much work the transcribe/detect/cut/
compare pipeline had left was to query hark.db directly or run a CLI
--dry-run. Adds a home-page status banner, per-show progress counts on the
show page, and a comparison-availability note on topic pages.
count_pending_topics() gives the dashboard a read-only-connection-safe way
to count topics ready for comparison — pending_topics() itself calls
ensure_schema() (a write) and can't run against hark.web's read-only
connection on a database where topic_comparisons hasn't been created yet.
LD_LIBRARY_PATH loaded the library fine under plain docker run, but still
failed once deployed with GPU device reservations attached — the NVIDIA
container runtime's own environment injection likely overrides it rather
than merging with it. Register the same paths via ldconfig instead, which
dlopen() consults independent of any environment variable.
Screenshots are from a local instance running against a snapshot of the
live database, with feed tokens regenerated before capture so no real
per-show secret ends up in the repo.
hark fsck --fix clears transcript_path pointers whose file no longer
exists, run automatically at the start of every transcribe cycle. Found
in production: 209 of 212 entries in the deployed hark.db pointed at
transcript files an earlier local-database cleanup had deleted without
invalidating the pointers.
Also wires the deployed transcribe service to pick up and load a
pending-comparisons.jsonl dropped into the shared data volume, so a
session-generated comparison batch no longer needs one-off container
access to reach the live database.
nvidia-cublas-cu12/nvidia-cudnn-cu12 (adscrub's gpu extra) are pip wheels
that bundle their .so files under site-packages, not any path the dynamic
linker searches by default. ctranslate2 needs LD_LIBRARY_PATH pointed at
them, or transcription crashes at inference time even though the packages
are installed and the model loaded successfully.
The hark user's --no-create-home meant huggingface_hub's default cache dir
resolved to an unwritable /home/hark, so every model load re-hit the HF Hub
API instead of using a cache, exhausting the rate limit within seconds of
container start and keeping it exhausted. Point HF_HOME at /app/data so it's
writable and persists across restarts. Also add a consecutive-failure
circuit breaker to hark transcribe, matching hark detect-ads, so a future
outage backs off instead of burning through the whole pending list.
Show-name-vs-show_id key collision in claims comparison, missing
per-record isolation in load_comparisons, a dropped consecutive-
failure circuit breaker in detect-ads, a toggle race condition, and
a null-title print regression. Plus a missing episode_topics index
that made the new co-occurrence features full-scan on every page
view, and a hark transcribe --cross-show-only flag so the priority
transcription subset is a real CLI feature instead of a throwaway
script.
Each topic page now lists other topics that co-occur in the same
episodes, ranked by shared episode count. Also fixes topic 236's
label ("Fred Wes" -> "Fred West"), mirrored from a typo in Wikidata's
own entity label rather than hark's own extraction.
Each show page now lists other shows ranked by shared topic count.
Uses data M1 extraction already produces instead of standing up an
embedding model for the originally planned similarity approach.
docker build . could never resolve adscrub's local path dependency
since the build context only ever had hark's own files. Added
scripts/build-image.sh to stage git-archive-clean copies of both
repos side by side and build against that; Dockerfile/compose.yaml
updated to match.
The ad-stripping pipeline ran unconditionally against every show with
no way to exclude one (real compute per episode, especially
transcription). shows.ad_stripping_enabled (on by default) now gates
chapters/transcribe/detect-ads/cut, toggled from a button on the show
page. That page also now shows the show's /feed/<id>/<token> URL
directly so it can be copied into a podcast player -- it always
existed but was never surfaced anywhere.
detect-ads/cut switch from adscrub's bulk orchestrators to per-episode
calls in a hark-side loop so the toggle actually takes effect.
claims.py was built additively while web.py/cli.py were mid-merge from
the adscrub port; now that merge has landed, wire it in fully: hark
compare/load-comparisons CLI commands, and a new /episode/<id> page
(linked from every episode listing) showing shared vs. unique claims
per topic, or why there's nothing to show yet.
Also fixes __init__.py's __version__ being stuck at 0.3.7 after
pyproject.toml moved to 0.4.0 for the ad-stripping merge.
adscrub (flan/adscrub) stays a separate, standalone product -- own
repo, schema, CLI. hark adds it as a path dependency (editable,
../adscrub -- see pyproject.toml [tool.uv.sources]) and calls its
functions directly rather than duplicating any code.
hark's own episodes/shows/ad_segments schema was deliberately shaped
to match adscrub's column-for-column, so adscrub's schema-coupled
functions (pending_episodes, scan_episode, transcribe_episode,
detect_pending, cut_pending, ...) work unchanged against hark's own
conn. hark cli.py's chapters/transcribe/detect-ads/cut subcommands
call straight into the adscrub package -- no hark-side
chapters.py/transcribe.py/detect.py/cut.py exists.
podcast_feed.py is hark's own file (schema-specific: show_id/feed_url
naming plus token auth, none of which adscrub's own feed.py has), used
by new token-gated /feed/<show_id>/<token> and
/audio/<episode_id>/<token>.<ext> routes on hark web -- unauthenticated
since a podcast app can't do the dashboard's cookie login, gated
instead by a random per-show feed_token.
110 tests pass. Known gap, not solved: the path dependency doesn't
resolve in the Docker build context yet (needs a real packaging
decision -- git dependency+deploy key, vendored wheel, or a
multi-repo build script) -- documented in the Dockerfile, README, and
docs/PLAN.md rather than papered over.
Merged in from the standalone adscrub repo now that two things are true:
hark is going to use Whisper anyway, and ad-stripping needs to cover
every subscription, not just the genre-curated shows the topic index
tracks. Full pipeline: hark chapters (free chapter-marker scan) ->
hark transcribe (faster-whisper, GPU auto-detected via
ctranslate2.get_cuda_device_count(), no torch needed) -> hark
detect-ads (Claude structured outputs, model points at transcript
segment indices not raw timestamps) -> hark cut (ffmpeg -c copy,
overlapping ad spans from any source merged before cutting).
Schema: episodes gains chapters_url/chapters_scanned_at/
transcript_path/llm_detected_at/cut_path; shows gains feed_token; new
ad_segments table. All additive/backfilled -- a pre-merge hark.db
upgrades in place on next connect.
hark web now also answers GET /feed/<show_id>/<token> (regenerated
clean RSS) and GET /audio/<episode_id>/<token>.<ext> (locally-cut
episodes) -- unauthenticated since a podcast app can't do the
dashboard's cookie login, but gated by a random per-show token in the
URL rather than wide open. --base-url/ controls what's
embedded in generated links; warns if left at the unreachable
localhost default.
One shared transcribe.py / one cached Whisper model for both this
pipeline and future episode-scoring use, as long as both ask for the
same model size and run sequentially through the cron-scheduled
pipeline (they do) -- documented as a discipline to maintain, not
something merging the codebases guaranteed for free.
compose.gpu.yaml requests the host's GPU via the nvidia Docker
runtime; base install/Dockerfile stay CPU-safe by default (gpu extra
opt-in). All 140 tests pass (adscrub's suite ported + adapted to
shows/show_id + new token-auth route tests); real ffmpeg/Whisper/
Claude calls are mocked, matching the existing test conventions.
flan