CaptionBench
- Clips
- 196
- Reviewed captions
- 1,342
- Window judgments
- 5,431
- Agreement κ
- 0.947
// 01 — why captions matter now
A caption used to be metadata.
Captions now drive retrieval, clipping and search directly over caption corpora, where the caption is the primary artefact rather than a label attached to one. A caption that reads well but places an action three seconds off from when it actually happens is a real defect.
Anyone choosing a captioner needs two answers: which model fits their footage, and when a model gets something wrong, what fixes it. Reference-based metrics answer neither.
// 02 — method
Rewrite, then grade.
- 196 clips
Short-form footage, mostly under thirty seconds, across the kinds of video people actually caption.
- Six models
Qwen3.8-Max · Muse Spark 1.2 · Kimi K3 · Seed 2.1 Turbo · Gemini 3.1 Pro · Qwen3.8-27B
- Two passes
Each caption graded once as a whole summary and again segment-by-segment against the timeline.
- Six failure types
Every rejection tagged with a category and a severity; critical means more than 45% of the text had to be replaced.
- κ = 0.947
Inter-rater agreement across 1,144 double-reviewed captions.
- 5,431 windows
Window-level judgments, so timing errors are measurable rather than anecdotal.
// 03 — the metrics didn’t know
BLEU and CIDEr scored good and bad captions alike.
Area under the curve at separating accepted from rejected captions. 0.5 is a coin flip.
| BLEU-4 | 0.51 |
|---|---|
| CIDEr | 0.53 |
// 04 — distinct failure modes
Similar scores, different problems.
Each needs a different fix: less assertion, more coverage, or a different model.
Over-assertive: invents detail, but has excellent timeline accuracy.
Conservative but incomplete: rarely wrong, often missing.
2.6× the best model, and 41.7% of its accepted captions hid a rejected segment.
// 05 — the expensive error
Camera and shot errors hide in summaries.
Reviewing segment-by-segment surfaced 4.7× more camera/shot failures than reading the caption as a whole. It’s also the category most often marked critical (21.6%).
| Full-summary review | 1.0× |
|---|---|
| Segment-by-segment | 4.7× |
// 06 — what it produced
Corrections, not just scores.
- 1,342reviewed captions
- 454rejected outputs with human corrections
- 5,431window-level judgments
- 6failure categories, each with a severity