When we first ran a vision-language model over our shot library, the descriptions came back accurate. They also came back useless.
Accurate is easy. Show a VLM a still and it will tell you what's in it: "Two people in a dimly lit room, one seated, one standing, neutral expressions". That's not wrong. But it's true of about a thousand shots in our corpus, which means a search for "the moment a character realizes the truth" will surface that description and nine hundred and ninety-nine others just like it. The VLM has done its job. Retrieval has not.
This article is about the gap between accurate and useful in shot annotation, and the trick that closed it for us. The trick is not exotic. It's a pattern Anthropic published for document chunks, applied to a different domain, with a couple of subdomain-specific twists. The idea is to give the annotator more than the picture — the dialogue, the surrounding shots, the narrative situation — and let it write a description grounded in all of it.
The result is the difference between what's in this frame and what this frame is doing in the story. For retrieval, that distinction is everything.
Why visual-only annotations fail at retrieval
The default approach to shot annotation is straightforward: pass the keyframe to the VLM, get back a description, store the description, embed it for vector search. The shot is the unit of work. Each annotation is independent of every other annotation.
This works, in the sense that it produces text that's about the picture. The text is correct. It's also, almost without exception, indistinguishable from a hundred other shots' worth of correct text.
Here's the underlying problem. A still image, in isolation, is ambiguous in ways a paragraph of text isn't. Two people in a dimly lit room could be a confrontation, a confession, a reconciliation, a quiet conversation, a moment of reluctant cooperation, a charged silence between former lovers, an interrogation. The VLM can see the visual signal — two people, dim light, postures — but it can't see the narrative function. It writes whatever generic description fits the visuals, which means the same description fits dozens of similar visual setups.
For browsing, this is fine. A human flipping through annotations would notice that some of them are confrontations and some of them are confessions, by reading enough of the surrounding context. For embedding-based retrieval, it's a disaster. The query "find me a moment of reluctant reconciliation" embeds into a vector that's looking for words like "reconciliation", "reluctant", "history". The annotations don't have those words, because the VLM never had any reason to write them. The retrieval misses.
The fix has to be either (a) better at extracting narrative function from a still image, or (b) giving the annotator more than a still image. Option (a) requires a fundamentally different model than the VLM stack we have. Option (b) is a prompt-engineering and pipeline problem.
We went with option (b).
The two signals the VLM was missing
The two signals that turn an image into a scene are the dialogue and the context.
Dialogue is the most obvious. A shot of someone whispering across a table is one thing if they're saying "I love you" and another thing if they're saying "I know what you did." The visual content is identical. The narrative function is opposite. The VLM has access to neither piece of information when it's only looking at a keyframe.
Context is what's around the shot — the previous shot, the next shot, the dialogue earlier in the scene, the dialogue later. A shot of someone walking through a doorway in a hotel hallway means one thing if the previous shot was "they pulled up to the hotel" and another thing if the previous shot was "they fled the hotel three days ago and they're back to retrieve something they left." The visual content of the doorway shot is the same. The narrative weight is completely different.
A VLM, looking at the doorway shot in isolation, can only describe the doorway. It can't reach for the framing that makes the shot interesting because the framing isn't in the shot — it's in the rest of the scene. The annotation is going to be "a person walking through a hotel hallway doorway, neutrally framed". Useless for retrieval.
Adding either dialogue or context helps. Adding both together is where the change in description quality stops being a small edit and starts being a category shift.
Subtitle extraction and alignment
The pipeline change is unglamorous. We pull subtitle tracks out of the source video, parse them into their text and their timestamps, and align each subtitle line to the shot it falls inside.
Most film files include subtitle tracks as separate streams, alongside the video and audio. The standard tool ecosystem can extract them — probe the file to list the streams, pick the right subtitle track, dump it as a SubRip file. The SubRip format gives you start time, end time, and the text. From there, mapping each subtitle line to a shot is straightforward: for each subtitle, find the shot whose start and end timestamps cover the subtitle's start time.
A few practical wrinkles:
The first wrinkle is track selection. A film often has multiple subtitle tracks — different languages, "forced" tracks for non-English bits in an English film, "SDH" tracks with sound effects for hearing-impaired viewers. The pipeline has to pick the right one. We default to the most complete dialogue track in the language we want, which usually means inspecting track metadata and falling back on heuristics when the metadata is incomplete.
The second wrinkle is timing offset. If the keyframes were extracted from a trimmed copy of the video (we strip credits off the front), the subtitle timestamps won't line up. The pipeline has to apply the same trim offset to the subtitles before alignment.
The third wrinkle is missing tracks. Not every film has subtitles in the language we want. Some films are silent or near-silent. The pipeline has to handle the no-subtitle case gracefully — leave the dialogue field empty, let the rest of the annotation continue, accept that some shots will be context-richer than others.
None of this is hard engineering. The wrinkles add up to a couple of days of work to get right. It's the kind of work that's easy to skip until you realize it's the difference between a useful annotation and a useless one.
Bilingual subtitles and language detection
Many films have bilingual subtitle tracks — one stream that interleaves the original-language dialogue with English subtitles below each line. These are common in international film distribution, and they're a problem if the pipeline doesn't separate them.
A bilingual subtitle, fed naively into the dialogue field, contains both the original-language text and its English translation, jumbled together. The VLM annotation that uses this dialogue ends up with a strange mix of the two languages, sometimes citing the English meaning, sometimes citing characters of another script. The embedding of the resulting annotation also gets confused — it's trying to represent two languages at once.
The fix is to detect and split the bilingual stream into two separate fields: one for the original-language dialogue, one for the English translation. The detection is easy when the two languages use different scripts (Chinese characters and Latin characters, for example) — a per-line check on whether the line contains characters from a CJK range tells you which language the line belongs to. When both languages share a script (English subtitles for a French film), the detection requires a language-identification model, which we sidestep by relying on the SubRip file's per-track language tag when it's available.
We store both fields separately in the shot record. Downstream, the description-generation step can use whichever language the VLM is most confident in (English, in our case, because it's the safest assumption for a generalist VLM) and the embedding step can use both (multilingual embeddings handle mixed-language inputs gracefully). Cross-language retrieval — a query in English finding a shot whose original dialogue is in Cantonese — works because of the multilingual embedding step.
Contextual Retrieval, applied to shots
The trick that changed everything is borrowed from a paper Anthropic published on document retrieval. Their problem was that document chunks, indexed in isolation, often miss context that lives in the rest of the document. Their fix was to add a short context-prefix to each chunk before embedding — a sentence or two, generated by an LLM, that places the chunk within the document. They called this Contextual Retrieval, and the gains on retrieval quality were significant.
The pattern carries over to shots almost too cleanly. A shot is a "chunk" of a scene. The "document" is the scene that contains the shot — the surrounding shots, the dialogue across all of them, the metadata about the film and its position in the narrative. Annotating a shot in isolation gets you what the VLM can see in the keyframe. Annotating a shot with the surrounding scene as context gets you what the story is doing at that moment.
We apply the pattern with a separate LLM call after the VLM stage. The VLM produces the visual description and the structured fields. Then we hand the LLM the visual description, plus the dialogue from this shot, plus the dialogue from the previous and next few shots, plus the visual descriptions of those neighboring shots. We ask the LLM to write a short narrative-aware prefix — a sentence that says what the shot is doing, given everything around it. We prepend that prefix to the visual description before embedding.
The cost is a second API call per shot, on top of the VLM call. Roughly twice the API spend at this stage. We considered whether to skip it. We didn't. The retrieval-quality difference is large enough that paying twice for the annotation step is the right trade.
The output reads differently. A shot that the VLM described as "two figures, dimly lit, one seated, one standing" becomes, with the contextual prefix added: "a tense reconciliation between two characters with a long history; the standing figure has just delivered news the seated figure didn't want to hear; two figures, dimly lit, one seated, one standing." The visual description is unchanged at the end. The narrative prefix at the front is what makes the shot findable.
What the before-and-after looks like
We don't include real annotations in this post — those live in the production index — but the structure of the change is consistent across the corpus. A few generic examples illustrate the shape.
A shot of a single character looking out a window:
- Visual-only: "A person looking out a window in soft daylight, three-quarter framing, medium shot."
- With dialogue: "A character mid-sentence about leaving the city; person looking out a window in soft daylight."
- With dialogue + context: "A moment of decision after a long argument; the character has just resolved to leave; person looking out a window in soft daylight, three-quarter framing, medium shot."
A shot of two characters at a table:
- Visual-only: "Two figures seated at a table in low warm lighting, framed in a two-shot."
- With dialogue: "Two characters discussing a betrayal; seated at a table in low warm lighting."
- With dialogue + context: "A confession scene late in the second act; one character is about to admit to the other that they were responsible for an earlier loss; two figures seated at a table in low warm lighting, two-shot framing."
The visual-only versions are accurate but blend together. The contextual versions are searchable in ways the visual-only versions can't be — a query about "a confession late in the act" hits the contextual version directly and never lands on the visual-only version.
What this enables for retrieval
Two things, both meaningful.
The first is narrative-level queries. With contextual annotations, a user can search for "the moment a character realizes the truth", "a quiet confrontation between siblings", "a reconciliation that doesn't quite land", and the retrieval will surface shots whose narrative function matches. Visual-only annotations don't carry the words to match these queries; contextual annotations do.
The second is cross-language retrieval. Because the dialogue field captures the original-language script when it's not English, and the embeddings are multilingual, a user typing English queries can find shots whose original dialogue is in Cantonese, Japanese, French, or any of the other languages our corpus contains. The narrative function survives the language transit. A query about "an apology that comes too late" lands on shots in any language where that beat is happening.
Neither of these queries works against a visual-only annotation pipeline. Both work cleanly against a context-aware one. The difference, if you're building a retrieval system over film, is the difference between a database that supports the queries an actual director or editor would write and a database that supports queries about lighting and camera angle.
Closing
The pattern that connects every section of this post is something like: a shot in isolation underspecifies its meaning. Pixels carry less semantic load than words. To get retrieval-grade annotations, you have to give the annotator the context the audience would have when watching the shot in sequence. That's the dialogue, the surrounding shots, the position in the scene.
The cost is real — extra extraction steps, an extra LLM call, careful handling of language and timing. The payoff is the difference between a knowledge base that an AI agent can actually use and a knowledge base that the agent could use if you also wired in the rest of the scene at query time. We'd rather pay for the context once at ingestion than pay for it on every query.
If you're building an annotation pipeline over media, treat the visual-only baseline as a baseline, not a finished product. Whatever the next layer of context is — dialogue, audio, surrounding chunks, document position — adding it at ingestion is what closes the gap between accurate and useful.
If you've built something similar and your trade-offs landed differently, we read every reply.
All illustrations generated using Genkee AI.