Video 003 · Sep 2026
Can AI pick my best stream clips?
I built a clip detector for my streams. Fourteen signals: laughter, loudness, chat spikes, my face on camera. Then I checked it against 336 clips my editor had already said yes or no to. It was a coin flip.
So I tried something lazier, and it works better. Claude, shown 48 clips my editor had already graded, picks the one he kept about three times out of four.
Why the detector was a coin flip
Every signal fires on most of the candidates, because firing is how a moment becomes a candidate in the first place. So none of them can tell a good clip from a bad one.
It also spent three months learning which signals my editor rewards, from his grades. The learned version scored 0.505. It learned nothing.
What worked: an LLM as a judge
- Show Claude 48 clips my editor already graded. 24 he kept, 24 he binned, no more than 4 from any one stream. It only gets the game's name and the first 600 characters of the transcript. It never sees the video.
- Ask it to grade the next clip the same way. It answers with the chance he'd keep it, from 0 to 100.
- Check it against what he actually picked. On 287 clips it hadn't seen as examples, Claude Opus 5 scored an AUC of 0.76, and 0.763 on a rerun. In plain English: give it a clip he kept and a clip he binned, and it picks the kept one about three times out of four.
- Only send him the clips Claude rates. About 60% of a stream reaches him now. Most of that is the top scores, plus a fifth picked at random, so Claude gets marked on clips it didn't choose too.
One thing made it worse: giving Claude the detector's signals as well. That dropped it to 0.71.
The examples get rebuilt from his newest grades every run. A frozen copy of the first set scores the same clips alongside, so I can see whether the rebuild is actually learning anything, which is more than I checked the first time.
What's in the repo
The prompt and a script that builds it from your own graded clips, scores new ones and measures the AUC on grades it held back. The prompt my pipeline runs has my editor's name in it; the public copy doesn't. His grades aren't in there either. They're his calls on my stream footage.