1A lot of guessing
Every episode needs a title, and most of ours come from gut feel. That works fine, but you can't check it, and I kept wondering whether the titles we liked were the ones that actually did well. After a while that started to feel like a question I could answer.
The idea is simple. Let LLMs do what they're good at, which is generating lots of plausible alternatives. Then let a show-specific ranking model do what it's better at, which is learning from a real archive of how past episodes performed. Separate the brainstorming from the choosing. It sounds obvious once you say it out loud, which is usually a sign it was worth formalizing.
2The archive
The study uses 1,193 Sickboy episodes, performance metrics from two platforms and 869 episode transcripts. It comes with a formal runbook, a map from each claim to the artifact behind it, and frozen requirements. Partly that's because I wanted the work to be reproducible. Partly it's because I'd been burned before by analysis I couldn't retrace six months later.
Beyond the paper, the project is a full reproducibility package. Candidate generation, evaluation, ablations, transcript gates, visualization scripts and production-pipeline runners all sit in the same place. It reads less like a one-off analysis and more like a small research operating system for the metadata problem I'd been complaining about for years.
Table 1. What the study covers.
| Component | Value |
|---|---|
| Episodes in archive | 1,193 |
| Transcripts available | 869 |
| Model variants evaluated | 84 |
| Opening-alignment sample | 1,128 episodes |
3How it works
The pipeline has three stages. It generates many candidates from the transcript, then scores them with a learning-to-rank approach grounded in the show's historical performance. Finally it returns a shortlist, instead of pretending a single “best” title emerged from pure mathematical truth. That last step is important. Editorial judgment doesn't get replaced. It gets confined to a much better neighborhood.
- Stage 1
Generate
Use transcript-informed prompts to produce many title and description candidates.
- Stage 2
Rank
Score candidates with a show-specific model trained on historical performance data.
- Stage 3
Editorial Pick
Choose from the top cluster with human judgment instead of asking the model to cosplay taste.
I also wanted this to survive contact with our actual workflow, so there's a practical app layer alongside the research scripts: a Gradio tool the team can use on publish day. That felt more honest than a project that only lives in a PDF.
4How well it works
The best configuration finds the highest-performing episode 34% of the time and puts it in the top three 64% of the time, with NDCG@5 reaching 0.909. That's materially better than random and, more importantly, good enough to change how we pick titles. If your top five options are tightly clustered and grounded in the show's history, the final choice can safely go back to brand voice and human instinct.
The second finding is funnier. Ask LLMs to rank their own title candidates by likely performance and they hover around 51% pairwise accuracy. That's a coin flip with better prose. The system works best when the model writes and then steps aside.
The opening-alignment analysis surprised me most. Episodes whose title topic comes up earlier in the audio tend to get lower average consumption. That makes sense once you think about it from the listener's side. If the title promises the answer and the episode delivers it straight away, you may have built an exit ramp instead of an entrance.
Table 2. The main results.
| Metric | Value |
|---|---|
| Top-1 hit rate | 34% |
| Top-3 hit rate | 64% |
| NDCG@5 | 0.909 |
| LLM pairwise ranking accuracy | ~51% |
5Who gets the last word
What I like about this project is that it avoids both easy extremes. It isn't “AI will replace editors,” and it isn't “AI is useless slop.” The claim is more practical: AI is useful for generating options, historical data is useful for narrowing them, and people are still needed to choose the one that sounds like the show. I've sat in those meetings. The model can't do what the team does, but the team makes better choices when the options are better.
There's also something satisfying about proving that the model that writes a title isn't automatically the best judge of it. That's a narrow metadata result, and a pretty decent life principle.

