Researchers Build a Test to See If AI Can Actually Summarise a 16-Minute Video
A new benchmark called LVSum reveals how badly today's AI video tools lose track of when things happen in long recordings.
Key points
- Apple ML Research introduced LVSum, a benchmark for testing how well AI summarises long videos, in a 2024 research paper.
- LVSum contains 72 videos across 13 subject areas, with an average length of 16 minutes per video.
- Each video in the dataset carries up to 10 human-written summaries that include specific timestamps, meaning the AI must say not just what happened but when.
- Current multimodal large language models, the AI systems that can read and watch video at the same time, struggle to keep events in the right order over long recordings.
Imagine asking a colleague to summarise a 16-minute meeting recording. You want them to tell you what was decided, yes, but also when the key moment happened, so you can scrub straight to it. That basic skill turns out to be surprisingly hard for AI.
That is the gap LVSum was built to measure.
The benchmark, described in a paper from Apple ML Research, puts AI systems through a structured test. Researchers gathered 72 videos spread across 13 different subject areas, from cooking demonstrations to legal proceedings. Each video runs about 16 minutes on average. Human annotators then wrote up to 10 separate summaries per video, each one pegged to specific timestamps in the footage.
The timestamps are the whole point. A summary that says "the speaker discussed the contract" is almost useless if you need to find that moment quickly. LVSum asks whether an AI can do better, producing what researchers call temporally grounded summaries, meaning text that tells you both the content and exactly when it occurs.
The models being tested are multimodal large language models, AI systems trained to process video, images and text together rather than text alone. Think of them as the technology behind tools that can watch a clip and describe it. Existing systems of this type already handle short clips reasonably well. Long video is a different problem entirely.
Over 16 minutes, a model must hold a great deal in memory and keep the sequence of events straight. The research found that maintaining what the paper calls "temporal fidelity", meaning an accurate sense of what came before what, is where current systems break down most visibly.
What does this mean for ordinary users?
Right now, if you use an AI tool to summarise a long meeting recording, a lecture or a documentary, you should treat the output as a rough guide rather than a reliable record. The AI may get the facts broadly right but place them in the wrong order or miss time cues entirely. Always spot-check against the original before sharing any AI-generated summary with colleagues or clients.
LVSum does not itself fix the problem. It is a measuring stick, a standardised way for researchers to compare how different AI systems perform on the same set of videos. Benchmarks like this tend to drive improvement: once there is an agreed test, labs can see clearly where their models fall short and work toward a specific target.
The dataset covers 13 domains deliberately, because an AI that only summarises cooking videos well is not much practical use. Diversity in the test forces models to generalise.
The paper stops short of naming which commercial products were evaluated. That detail, and the full scoring breakdown, sits in the complete research text.
