Coding agents occasionally do real damage: they delete data, take a service down, leak a credential into a transcript, or burn a week of quota, and the lesson is usually forgotten by the next session. ai-incidents runs unattended over your own Claude Code and opencode transcripts, uses deterministic rules to pick out the moments that look like trouble, has a model judge which of them were real incidents, and keeps a severity-ranked markdown ledger of what broke, what it cost and the lesson, with a record of everything it excluded and why.

examples/sample-ledger.md is a curated excerpt from real use, and examples/sample-report.md shows the matching run report.

Install

pipx install git+https://github.com/sudolulo/ai-incidents   # or: uv tool install git+https://...
ai-incidents init --ledger ~/ai-incidents-ledger --git      # writes ~/.config/ai-incidents/config.toml
ai-incidents run --dry-run                                  # see what would be judged; changes nothing

From a checkout, pipx install . or uv tool install . works the same way. Python 3.11 or later, no other dependencies. The default judge is the Claude Code CLI (claude), which must be logged in; see Choosing a judge for local models.

Quick start

ai-incidents run --dry-run      # scan, and show what would be sent to the judge; no model call
ai-incidents run                # first run: record existing sessions as a baseline, judge nothing
ai-incidents run                # later runs: judge only sessions that are new or have grown

The first run takes a baseline: it records every existing session as already judged, so you do not pay to judge your whole history on day one. To judge history instead, run ai-incidents run --backfill (or set first_run = "backfill"); it works through the backlog max_candidates at a time, one batch per run.

Then schedule it. A systemd user timer and a crontab line are in examples/systemd/ and examples/cron.txt. Nightly is plenty: runs are incremental, and a night with nothing to judge makes no model call at all.

A run prints one line per source and a headline:

claude-code: 12 new session(s), 10 clean, 2 with candidates
opencode: 3 new session(s), 3 clean, 0 with candidates
Candidates: 5
committed 240f3cb
ai-incidents 2026-09-21: 1 filed (1 high), 2 candidate(s) excluded
HIGH   Deleted 48,211 live production orders with the wrong connection string
report: /home/you/ai-incidents-ledger/reports/2026-09-21.md

What it writes

Everything goes into the ledger directory (a git repository, if you want history):

File	What
`incidents.md`	The ledger. One entry per incident: title, date, severity, then What, Cost and Lesson. Most severe first. Hand edits and hand-written entries are kept.
`reports/YYYY-MM-DD.md`	One per judged run: what the pre-filter found, each confirmed incident with the reason it cleared the bar, each excluded candidate with the reason it did not, and the judge's token use.
`latest.md`	A copy of the newest report.
`index.json`	A machine index (fingerprint, title, date, severity, first seen). Regenerated from `incidents.md` on every write; do not edit it.

An entry looks like this:

## Committed 14 embedded git repositories to `main` with `git add -A` · 2026-07-14 · MEDIUM
- **What:** ran `git add -A && git commit && git push` in a worktree where a scheduled review job had left 14 throwaway clones ...
- **Cost:** `main` carried 14 broken submodule references that broke cloning until a follow-up commit removed them ...
- **Lesson:** `git add -A` is not a review step: stage explicit paths anywhere a tool may have written, and read the warnings from commands that succeed.

The tool's own run state (which sessions have been judged) lives outside the ledger, in ~/.local/state/ai-incidents/state.json by default.

Configuration

One TOML file, ~/.config/ai-incidents/config.toml by default (or $AI_INCIDENTS_CONFIG, or -c PATH). examples/config.toml documents every key. Unknown keys are an error, so a typo cannot quietly switch something off. ai-incidents config prints the effective settings, including exactly which paths the tool will write.

Table	Key	Default	Meaning
`[[source]]`	`type`	`claude-code`, `opencode`	Transcript format. Omit every `[[source]]` to read both defaults.
	`path`	`~/.claude/projects`, `~/.local/share/opencode/opencode*.db`	Where the transcripts are. The opencode path is a glob.
	`name`	the type	Needed only to tell two sources of one type apart.
`[scan]`	`max_candidates`	`50`	Most candidates shown to the judge per run; the rest wait for the next run.
	`max_per_session`	`12`	Most candidates from one session.
	`first_run`	`baseline`	`baseline` or `backfill` (see Quick start).
	`exclude`	`[]`	Globs on transcript paths to skip.
	`extra_destructive`, `extra_benign`, `extra_alarm`	`[]`	Extra pre-filter regexes (for example your own alert names).
`[judge]`	`backend`	`claude`	`claude`, `openai` or `command`.
	`model`	`sonnet`	Model name or alias for the backend.
	`timeout`	`900`	Seconds before the judge call is abandoned.
	`prompt_file`	built in	Replace the judging rubric.
	`base_url`, `api_key_env`, `json_mode`		`openai` backend. `api_key_env` names an environment variable; keys never go in the file.
	`command`		`command` backend: argv of a program that reads the prompt on stdin.
`[ledger]`	`dir`	(required)	The ledger directory.
	`order`	`severity`	`severity` or `newest`.
`[git]`	`enabled`	`true`	Commit if `ledger.dir` is a git repository.
	`push`	`false`	Push after committing.
	`expected_remote_url`		Refuse to push if the remote has been repointed.
`[state]`	`file`	`~/.local/state/ai-incidents/state.json`	Run state.
	`seen_list`		Optional plain-text list of judged Claude Code transcripts, for cleanup jobs.
`[privacy]`	`redact`	`true`	Mask secret-shaped strings before the judge and before the ledger.
`[hooks]`	`notify_cmd`		Runs after a run that filed something.
	`on_success_cmd`		Runs after every successful run, quiet ones included: a heartbeat.
	`on_failure_cmd`		Runs after a failed run.

Commands and exit codes

Command	Does
`ai-incidents run`	Scan, judge if there is anything to judge, write, commit, record. `--dry-run`, `--dry-run --with-judge`, `--backfill`, `--notify-cmd CMD`, `--no-git`, `--push` / `--no-push`.
`ai-incidents scan [--show]`	Show what the pre-filter finds. Reads only; records nothing.
`ai-incidents init --ledger DIR [--git]`	Write a starter config and create the ledger directory.
`ai-incidents reindex`	Re-rank `incidents.md` and rebuild `index.json` after hand edits.
`ai-incidents config`	Print the effective configuration.

Exit codes: 0 success (a quiet run is a success), 1 the run failed (judge, git, or state error), 2 bad usage or configuration, 3 another run holds the lock.

Hooks

Hooks replace any built-in notifier. Each is a command line, split the way a shell would split it but not run through one; wrap it in sh -c '...' if you want pipes. The run summary arrives on stdin, and these variables are set: AI_INCIDENTS_STATUS (ok or failed), AI_INCIDENTS_HEADLINE, AI_INCIDENTS_FILED, AI_INCIDENTS_HIGH, AI_INCIDENTS_MEDIUM, AI_INCIDENTS_LOW, AI_INCIDENTS_EXCLUDED, AI_INCIDENTS_REPORT. A hook that fails is logged and ignored.

[hooks]
notify_cmd = "curl -s -H 'Title: ai-incidents' --data-binary @- https://ntfy.sh/your-topic"
on_success_cmd = "curl -fsS -m 10 https://hc-ping.com/your-uuid"

Choosing a judge

[judge]                               # default: the Claude Code CLI you already use
backend = "claude"
model = "sonnet"

[judge]                               # a local model through any OpenAI-compatible server
backend = "openai"
base_url = "http://localhost:11434/v1"    # Ollama; llama.cpp's server is http://localhost:8080/v1
model = "qwen3:32b"

[judge]                               # anything that reads a prompt on stdin
backend = "command"
command = ["llm", "-m", "some-model"]

The judge must answer with a JSON object in the format the rubric describes. Larger models follow it more reliably; a malformed answer fails the run safely (see Failure modes).

Permission envelope

The tool reads private transcripts, so what it may do is deliberately narrow, and enforced in code rather than by convention:

Transcripts are read-only. Claude Code JSONL files are opened for reading. opencode's SQLite database is opened with mode=ro and query_only. (SQLite's WAL mode has every reader coordinate through a -shm index file next to the database, which a reader may create or update; that is lock bookkeeping, not data. For a database owned by another user, the tool opens it immutable=1 instead, so it never leaves a sidecar file that user cannot write.)
The judge only reads. It never sees a transcript directly: it receives the pre-filtered, redacted excerpts as text on stdin (never on the command line, where other local users could read them) and answers with JSON. With the default backend, claude -p runs with --tools "" (no built-in tools), --strict-mcp-config (no MCP servers), and --no-session-persistence (its own session is not written back into your transcripts), in an empty temporary directory. With the openai backend it is one HTTP request. The command backend runs whatever you configure, so the envelope there is yours to keep.
The only writes are the ledger. Every file write goes through one guard that allows exactly the ledger directory and the state file (plus its lock, and the optional seen-list). Anything else raises before a byte is written. The ledger text is rendered by the tool from validated, single-line JSON fields; the model never chooses a path or writes markdown structure.
Git is narrow. Commits name their files explicitly, so nothing else in the ledger repository (staged or not) is swept in. Pushing is off by default; when on, it pushes the current branch to its own name on the configured remote: no force, no tags, no detached HEAD, and optionally only to an expected_remote_url. Git is never allowed to prompt.
Hooks are yours. They run the commands you configure and are outside the envelope by definition. Nothing from a transcript is ever put on a hook's command line.

The test suite checks these: a full run over both transcript formats must leave every file outside the ledger and state unchanged, writes outside the guard must fail, the judge's argv must not contain the candidates, and a commit must not include files the tool did not write.

Privacy

Transcripts contain everything you and your agents did: code, file contents, command output, and now and then a secret that should never have been printed. Know where they go:

Excerpts leave the machine for the judge. Only the pre-filtered moments are sent (a few hundred characters per turn, the flagged commands and failures, and the title and first line of each existing ledger entry), not whole transcripts. They go to whichever model you configure. With the default claude backend that is Anthropic, the same provider your Claude Code sessions already use; opencode sessions may have run against a different provider, so sending them to Claude is a new disclosure.
A local model keeps everything local. Point the openai backend at llama.cpp, Ollama, vLLM or LM Studio on your own hardware and no excerpt leaves the machine.
Redaction is best-effort. Common token formats, private keys, Authorization headers, URL credentials and password=-style assignments are masked before the judge sees them and again before anything reaches the ledger. A bare password with nothing around it cannot be recognised. The rubric also tells the judge never to copy a secret or a third party's name into its answer.
The ledger is private by default. It describes your mistakes, your infrastructure and your projects. Review it before you publish it; examples/sample-ledger.md shows what a scrubbed excerpt looks like.
ai-incidents run --dry-run -v prints the candidates exactly as they would be sent, without sending them.

Design

Why it runs unattended

Nobody writes a postmortem for their own agent session. The session that caused the damage is usually the one that cleaned it up, and by the next day the lesson survives only in a transcript nobody will reread, until the transcript itself is deleted (Claude Code removes old transcripts after cleanupPeriodDays, 30 by default). A recorder that depends on someone remembering to run it has the same problem as the postmortem. So it runs on a timer, reads what was actually said and done, and costs nothing on the nights when nothing happened.

Why a model judges instead of rules

Rules are good at finding candidate moments and bad at deciding whether they mattered. The same git reset --hard is routine cleanup in one session and the loss of a day's unpushed work in another; rm -rf of a scratch directory and of a data directory look alike to a regex. Deciding needs the surrounding conversation and the consequences, which is a judgement.

So there are two stages, each doing what it is good at. The pre-filter is deterministic, free, and tuned for recall. It reads what sessions said (the agent admitting a mistake, the user objecting, an alarm, the agent blaming someone else's code) and, just as important, what they did: destructive commands, notable failures, and warnings printed by commands that exited 0. Reading behaviour is not optional. The sweep this tool came from originally matched only confessions, and on its first real test it missed all three incidents of that day, because none of the sessions happened to say one of the magic phrases; one of them committed 14 repositories to main from a command that exited 0 and announced the damage in a warning nobody read. The judge is a model, tuned for precision: strict about what counts, told that a digest of destructive commands is not an accusation, and told that confidently blaming an upstream project for your own bug is an incident in its own right.

The pre-filter also acts as a gate. When there are no candidates, no model is loaded. An agent that starts up only to discover there is no work still pays for the whole start-up, and a nightly job that does that every night is the kind of unmetered cost this ledger exists to catch.

Why exclusions are recorded

Every run report lists each candidate the judge excluded, with a one-line reason. Without that, the bar for "incident" is invisible and cannot be argued with: you cannot tell a quiet week from a judge that has become lax, or a noisy ledger from one that has become credulous. The exclusions are where you notice the judge drifting, where you find a real incident it waved through, and where you learn which pre-filter patterns only produce noise. They also make a quiet run a real result: "five candidates, all routine, here is why" is a finding; silence is not.

Severity is the cost

Severity is defined by what an incident cost, not by how alarming the mistake looked. HIGH is data loss, an outage, or a real credential exposure. MEDIUM is wasted rework, a bad decision pushed to a shared branch, or a debug spiral. LOW is a self-inflicted mess that was cleaned up cheaply. The agent catching its own mistake before it cost anything is not an incident at all. The ledger is ordered by severity, then date, because its reader wants the most expensive lessons first; the chronology is in reports/.

Failure modes

What goes wrong	What happens
The judge fails: it times out, exits non-zero, hits a quota, or answers with prose instead of a verdict (`claude -p` exits 0 while printing a spend-limit message).	The answer is validated, not the exit code, so the run fails with exit 1 and records nothing for the candidate sessions. The next run shows them again. Clean sessions are still recorded.
The judge returns a malformed incident (no lesson, an unknown severity).	That incident is rejected and listed in the report under "Loose ends"; the rest of the verdict is used. Candidates it never mentions are listed too.
The pre-filter misses something.	Recall is bounded by what a session said or did in a way the patterns recognise. A wrong decision that involved no destructive command, no failure and no complaint is invisible. `extra_*` patterns let you widen it; the exclusions show you what it currently catches.
The judge invents or overstates detail.	It is told to judge only from the excerpts, and each entry links back to its sessions in the report. The ledger is plain markdown: correct it by hand and run `reindex`.
The same incident surfaces twice (a resumed session, two sessions about one mistake).	The judge sees the existing ledger and marks duplicates, and an entry whose title matches an existing one is never filed twice. A duplicate under a different title can still slip through; delete it by hand.
More candidates than the budget.	Whole sessions are deferred to the next run, never split; clean sessions later in the scan are still recorded. A backfill takes several runs.
Two runs overlap.	The second exits with code 3 without touching anything.
The push fails.	The ledger is already committed locally and the sessions recorded; the run exits 1, and the next run pushes.
A transcript is deleted before it is judged.	It is simply gone. Schedule runs more often than your transcript retention. If a cleanup job of your own deletes transcripts, point it at `seen_list` so it only removes what has been judged.
A secret in an unrecognised format.	It can reach the judge and, if the judge repeats it, the ledger. Keep the ledger private and review before publishing.

Compatibility

Claude Code 2.x: transcripts under ~/.claude/projects/ (or $CLAUDE_CONFIG_DIR), and the claude CLI for the default judge, which needs a build with --tools and --no-session-persistence. Tested with 2.1.
opencode 1.x: the SQLite store (opencode.db, or opencode-<channel>.db) under ~/.local/share/opencode/. Tested on 1.18.22 or later. The older JSON-file storage is not read. A port for opencode 2.x is planned.
Python 3.11 to 3.13, on Linux and macOS.

License

MIT. See NOTICE for attribution.