lately

I wondered if a model could help us pick titles, so I trained one

Brian Stever

2026 · Python, ranking models, LLMs, transcripts, LaTeX

The short version. We've titled 1,193 Sickboy episodes, mostly on instinct, and I always wondered whether we were getting it right. So I tried to find out. This project is an AI-assisted ranking framework: LLMs generate title and description candidates from episode transcripts, and a show-specific model trained on historical performance data ranks them. Across 84 model variants and 869 transcripts, the best configuration identifies the top-performing episode 34% of the time and puts the winner in the top three 64% of the time. The funniest finding is also the most useful: LLMs are decent brainstorming partners and mediocre judges of their own work.

1A lot of guessing

Every episode needs a title, and most of ours come from gut feel. That works fine, but you can't check it, and I kept wondering whether the titles we liked were the ones that actually did well. After a while that started to feel like a question I could answer.

The idea is simple. Let LLMs do what they're good at, which is generating lots of plausible alternatives. Then let a show-specific ranking model do what it's better at, which is learning from a real archive of how past episodes performed. Separate the brainstorming from the choosing. It sounds obvious once you say it out loud, which is usually a sign it was worth formalizing.

2The archive

The study uses 1,193 Sickboy episodes, performance metrics from two platforms and 869 episode transcripts. It comes with a formal runbook, a map from each claim to the artifact behind it, and frozen requirements. Partly that's because I wanted the work to be reproducible. Partly it's because I'd been burned before by analysis I couldn't retrace six months later.

Beyond the paper, the project is a full reproducibility package. Candidate generation, evaluation, ablations, transcript gates, visualization scripts and production-pipeline runners all sit in the same place. It reads less like a one-off analysis and more like a small research operating system for the metadata problem I'd been complaining about for years.

Table 1. What the study covers.

ComponentValue
Episodes in archive1,193
Transcripts available869
Model variants evaluated84
Opening-alignment sample1,128 episodes

3How it works

The pipeline has three stages. It generates many candidates from the transcript, then scores them with a learning-to-rank approach grounded in the show's historical performance. Finally it returns a shortlist, instead of pretending a single “best” title emerged from pure mathematical truth. That last step is important. Editorial judgment doesn't get replaced. It gets confined to a much better neighborhood.

  1. Stage 1

    Generate

    Use transcript-informed prompts to produce many title and description candidates.

  2. Stage 2

    Rank

    Score candidates with a show-specific model trained on historical performance data.

  3. Stage 3

    Editorial Pick

    Choose from the top cluster with human judgment instead of asking the model to cosplay taste.

Figure 1. The generate, rank and review loop. The main move is keeping candidate generation separate from candidate selection.

I also wanted this to survive contact with our actual workflow, so there's a practical app layer alongside the research scripts: a Gradio tool the team can use on publish day. That felt more honest than a project that only lives in a PDF.

4How well it works

The best configuration finds the highest-performing episode 34% of the time and puts it in the top three 64% of the time, with NDCG@5 reaching 0.909. That's materially better than random and, more importantly, good enough to change how we pick titles. If your top five options are tightly clustered and grounded in the show's history, the final choice can safely go back to brand voice and human instinct.

The second finding is funnier. Ask LLMs to rank their own title candidates by likely performance and they hover around 51% pairwise accuracy. That's a coin flip with better prose. The system works best when the model writes and then steps aside.

The opening-alignment analysis surprised me most. Episodes whose title topic comes up earlier in the audio tend to get lower average consumption. That makes sense once you think about it from the listener's side. If the title promises the answer and the episode delivers it straight away, you may have built an exit ramp instead of an entrance.

Table 2. The main results.

MetricValue
Top-1 hit rate34%
Top-3 hit rate64%
NDCG@50.909
LLM pairwise ranking accuracy~51%

5Who gets the last word

What I like about this project is that it avoids both easy extremes. It isn't “AI will replace editors,” and it isn't “AI is useless slop.” The claim is more practical: AI is useful for generating options, historical data is useful for narrowing them, and people are still needed to choose the one that sounds like the show. I've sat in those meetings. The model can't do what the team does, but the team makes better choices when the options are better.

There's also something satisfying about proving that the model that writes a title isn't automatically the best judge of it. That's a narrow metadata result, and a pretty decent life principle.