When we built our film knowledge base, the architectural choices that mattered weren't the ones that get a lot of attention in technical blog posts. The big-name models and named libraries — those weren't where we spent our time. We spent our time on the smaller decisions: how long is "long"? When should we sample one frame versus three? What does the annotation schema actually need to capture? What's the cheapest model that doesn't lose retrieval quality?
There were eight of these decision points we kept circling back to. Each one was a real fork — pick A and you get one set of trade-offs; pick B and you get a different set. Most weren't obvious in advance. A few of them changed our minds halfway through.
This post is about those eight forks. Less about what we picked, more about what the trade-off actually was once we got close enough to see it. If you're building anything similar — a video knowledge base, a multi-modal RAG system, a pipeline that turns unstructured media into searchable structured data — you'll hit a lot of the same forks. Maybe in a different order. Maybe with different constraints. The dimensions you'll have to weigh will be similar.
We're going to skip the post-hoc justification format ("why we chose X") because it's mostly useless. The interesting story isn't that we picked one tool over another. The interesting story is what mattered when we picked, what we missed, and what we'd weigh differently next time.
For the previous post in this series, we wrote a high-level overview of the whole pipeline — what's in it, why it exists, what it powers. This post sits underneath that one. If you skipped the overview, the eight decisions still make sense as a list, but the why of building a film knowledge base in the first place lives there.
Now: the eight forks.
1. Shot detection: heuristic or neural?
The first stage of any video knowledge base has to find the boundaries between shots. Every downstream stage depends on this. Get it wrong and the rest of the pipeline operates on the wrong unit of work.
There are two camps. One uses classical frame-difference heuristics: compute pixel-level deltas between adjacent frames and call it a cut when the delta crosses a threshold. The other uses a neural network trained specifically on shot boundaries. Both produce a list of timestamps. Both are open source.
We tested both on the same set of films. The trade-off looks like this:
The heuristic detector is fast, has no GPU dependency, and returns sensible results on simple cuts. It struggles on dissolves and wipes — gradual transitions that lift pixel deltas above the threshold for many frames in a row, which the algorithm reads as a series of separate cuts. A single dissolve becomes three or four "shots". It also misses whip pans, where a fast camera move creates a brief blur that can read as a continuation of the previous shot when it's actually two. Tuning the threshold for one type of transition makes the other worse.
The neural detector is slower, needs a GPU, and has its own real failure mode — it occasionally splits a single shot when there's a sudden visual change inside the shot (someone turning on a light, a door opening). But it handles dissolves and whip pans correctly out of the box, with no parameter tuning. And the cost of running it, on the kind of consumer GPU you'd already need for the rest of the pipeline, is small.
We picked the neural option. The reason, distilled, is that shot boundaries are the unit of work for everything downstream. If half your dissolves get read as triple cuts, every downstream stage processes them three times. Annotation cost goes up. Embedding cost goes up. The retrieval surface gets noisier because there are now three versions of the same dissolve sitting in the index. The neural detector's cost is a one-time hit at ingestion; the heuristic's failures keep paying down the pipeline.
The lesson that generalizes: when one stage's output is the unit of work for many downstream stages, the value of getting that stage right is multiplied. The heuristic was cheaper in isolation but expensive in aggregate.
What we'd do differently: we tested on a set of films that had relatively clean cuts. Animated films, with their stylized transitions and flashbulbs and graphic cuts, would have stressed both detectors harder. Starting over, we'd fold a couple of animated films into the test set early to see which detector breaks first.
2. Long shots: lower the threshold, or add a second stage?
Even after the shot detector runs, some "shots" are still too long. A two-minute single take has multiple compositions inside it: the camera sits with one composition, dollies into another, then settles. Treating it as one shot loses everything that's interesting about it.
The cleanest fix is also a fork. Either (a) lower the shot detector's threshold so it cuts more aggressively, or (b) leave the shot detector alone and add a second pass that re-examines long shots and sub-divides them when something interesting happens.
The (a) approach has the advantage of being simple — one stage, one parameter. But lowering the threshold over-cuts the rest of the film. Short shots that should be one shot become two. The collateral damage is large.
The (b) approach is what we landed on. The second pass only looks at shots above a length threshold, and only cuts when a semantic change happens — a meaningful shift in what's in the frame, not a meaningful change in pixel values. Long shots get the careful treatment they need; short shots are left alone.
The interesting design question inside (b) is what counts as a "semantic shift". Pixel histograms work for some kinds of shift (a character walks into a different lighting condition) but not others (the camera dollies but the lighting stays the same). Image embeddings hold up better but are slower. There's a real cost-quality dial here, and the right point on the dial depends on how many long shots are in your corpus and how patient you are with the ingestion pipeline.
Empirically, long shots in classical cinema are a small percentage of total shots — most films have plenty of cuts. So spending more compute per long shot is fine. The wall-clock cost is small.
The lesson: don't fix one stage's failure by making another stage worse. The shot detector wasn't wrong; it was correctly identifying that the long take was one shot. The fact that we needed sub-shots for retrieval is a different problem, and it deserves a different stage. The temptation to fold both problems into one detector is real, and it usually produces a detector that's bad at both jobs.
What we'd do differently: we landed on the two-pass approach late, after running the pipeline on a few films with the single-pass version and noticing that long takes were getting under-annotated. If we'd run the test set against retrieval queries before claiming the ingestion was done, we would have caught this in week one instead of week three. The lesson there isn't about long shots specifically — it's about the value of running the full pipeline including retrieval validation, not just the ingestion side.
3. Keyframes: midpoint, or length-conditional?
Once you have shot boundaries, you have to pick which frames to actually annotate. You probably can't afford to annotate every frame; vision-language models are expensive enough that even one frame per shot makes you think about the bill. The question becomes: how many frames per shot, and where in the shot do you sample?
The naive answer is one frame at the midpoint of every shot. It's simple and it's defensible — the midpoint is the part of the shot the audience spends the most time looking at, on average. We started here.
The thing the midpoint sample misses is the long shot. A short shot really is dominated by its midpoint composition; nothing much happens in three seconds. A long shot is different. A thirty-second take might open with a wide framing, dolly to a close-up, then pull back. The midpoint is one of these compositions; the rest are gone.
The fix is a length-conditional sampling strategy. Short shots get one frame at the midpoint. Long shots get multiple frames sampled across the shot's duration, so each composition inside the shot gets its own annotation row. The threshold for "long" is a parameter we tuned by looking at what fraction of total shots got the multi-sample treatment — too aggressive and the annotation cost goes up disproportionately; too conservative and you keep losing long-take information.
A side issue: shot boundaries sometimes catch black frames during transitions. If you sample the midpoint and your midpoint happens to be a transition black frame, you've burned a VLM call on a black square. Adding a black-frame filter — checking pixel mean against a threshold and skipping frames that are too dark — saved us a meaningful amount of compute and produced cleaner annotations.
The decision logic looks like this:
The lesson: non-uniform sampling beats uniform sampling when your data has bimodal structure. Most shots are short; a few are long. A uniform "one frame per shot" strategy is wrong for the long ones. A uniform "three frames per shot" strategy wastes most of its budget on short shots that don't need three. Length-conditional is harder to explain in a slide but it dominates either uniform strategy on cost-quality.
What we'd do differently: we initially under-counted how often long shots appear in films. They're rare per film, but the films we were processing have a long tail of long takes (Wong Kar-wai and Tarkovsky, predictably). We should have looked at the shot-length histogram of our corpus before tuning the threshold. We didn't, and our first pass over-tuned for "average" films and under-served the auteurs.
4. VLM choice: flagship or budget?
This is the decision that costs real money, and it's the one where the quality-versus-cost trade-off is the most directly visible.
You're going to call a vision-language model on every keyframe in your corpus. If your corpus is large, that's a lot of calls. The flagship VLMs — the ones that lead the leaderboards on visual reasoning benchmarks — charge somewhere in the range of one or two cents per image at full resolution. The budget VLMs from cloud providers charge somewhere in the range of one or two hundredths of a cent. That's roughly two orders of magnitude.
For a small corpus, this doesn't matter; you'd run the flagship and forget about the bill. For a knowledge-base-scale corpus, it's the dominant line item.
The question is what you give up at the budget tier. We tested the same prompts and the same images on a flagship VLM and a budget VLM, side by side, on a few hundred shots.
The flagship produced more elegant prose. Its descriptions read better as standalone text — richer vocabulary, better narrative phrasing, more variation. If you were generating descriptions for human consumption — say, a film studies database meant to be browsed by film school students — the flagship would clearly be the right choice.
The budget VLM produced less elegant prose. Its descriptions were structurally similar but less polished — more "two characters in a kitchen at night, dimly lit, one speaking" and less "two figures, half-seen against the warm glow of a hanging bulb, one mid-sentence". The structural information — what's in the frame, what the camera is doing, the lighting — was about the same. The vocabulary was thinner. The narrative phrasing was less novel.
What matters for retrieval is structural information, not narrative phrasing. A query about "low-angle shot in a kitchen at night" hits the same shots regardless of which VLM described them. The only retrieval-quality gap we could measure between flagship and budget descriptions was small — and even that gap closed almost completely once we added the contextual prefix step (covered in section 7).
So we took the budget VLM and put the savings into more iterations of the rest of the pipeline.
The lesson: match model quality to what's downstream. If the VLM's output is being read by humans, prose quality matters and you should pay for the flagship. If the output is being embedded for vector retrieval, structural information dominates and the budget option is fine. Most teams default to the flagship out of caution; the cautious choice is the expensive one, and most of what you're paying for never reaches end users.
5. Annotation schema: free text, or structured fields?
Once you've picked a VLM, the next fork is what to ask it to produce. Free-form text? Structured JSON with named fields? Some hybrid?
The free-text camp argues that structured fields cap the model's expressive range. A great VLM can describe a shot in ways no fixed schema captures — emotional context, off-screen implications, callbacks to earlier scenes. Forcing the model into named fields throws that signal away.
The structured camp argues that retrieval works on consistent fields, not freeform prose. If you want to filter for "low-angle shot, golden hour, single character", you need the angle / lighting / subject-count to be in fields the query layer can match against. Free text leaves all of that buried in prose that's only accessible through embedding-based retrieval.
We landed on structured fields with one freeform field for the description itself. The structured fields cover the dimensions retrieval queries need to filter on: framing, camera angle, lighting, mood, what's in the frame, what the camera is doing. The freeform description captures everything that doesn't fit cleanly into a field — narrative texture, off-screen implications, things a future query type might want to find.
The interesting question wasn't whether to structure; it was which structure. We borrowed heavily from the categorization systems professional shot databases like ShotDeck use. They've already done the hard work of figuring out what cinematographers and editors actually want to filter on. Re-deriving that from first principles would have been a months-long detour into film school territory.
A subtle point: the schema itself is a queryable artifact. When the VLM emits the same field for the same dimension across millions of shots, downstream retrieval can do exact-match filtering, not just embedding similarity. That's the difference between "find me shots that feel like this" and "find me shots that are in golden hour with a single character at a low angle". Both are useful. Different queries need different ones.
The lesson: structure is what makes retrieval composable. Embedding similarity is great for "feel" queries; structured fields are great for "rule" queries; together they let you compose. A schema-less corpus is fine for a research toy. A production knowledge base needs the structure even if it costs a little expressive range.
What we'd do differently: we picked the schema once and didn't expect to revise it. In practice, two fields turned out to be poorly designed (one was too coarse, one was too redundant with another field), and revising them required re-running the VLM stage on every shot — expensive. Starting over, we'd version the schema from day one and treat schema migrations as a real workflow.
6. Camera motion: neural model, or optical flow?
A real camera moves. Static, pan, tilt, zoom, tracking, dolly, push-in, pull-out, compound. The shot's motion is part of what makes it the shot it is. A close-up on a static shot reads completely differently from the same close-up on a tracking shot.
You have two main options for analyzing this. One is a neural model trained on camera motion classification — feed it a clip, get a label. The other is classical optical flow — compute the dense vector field of pixel motion between adjacent frames, then classify based on the global structure of that field.
We initially planned to use the neural option. The leaderboards looked good, the labels were more granular, and the engineering effort to deploy was the same as any other model. We started building the integration.
Then we ran the numbers. The neural model needed several gigabytes of GPU memory, which collided with the other models we were running concurrently on the same machine. Inference time per shot was non-trivial. And the optical-flow approach, when we tested it as a baseline, hit somewhere around ninety percent of the neural model's accuracy on the cases that mattered for retrieval (the basic motion categories: static, pan, tilt, zoom, tracking).
We dropped the neural plan and went with optical flow. The remaining accuracy gap — mostly on rare compound motions — wasn't worth the memory and latency hit. We weren't building a research benchmark; we were building a production retrieval system, and the retrieval system never asked "is this a 'rolling crane' or a 'compound dolly'". It asked "is this a tracking shot or a static shot".
The lesson: the right model is the one that matches the query distribution, not the one that wins benchmarks. We had been instinctively reaching for the more sophisticated option because more sophisticated felt safer. In practice, the simpler option was a closer fit to what retrieval actually needed.
A second lesson: GPU memory is a shared resource at the system level, not the model level. The neural camera-motion model on its own would have been fine. The neural camera-motion model running concurrently with a vision-language model and an image embedder on the same consumer GPU was a problem. Before benchmarking accuracy, benchmark whether you can even fit the model alongside everything else.
What we'd do differently: we started building the neural integration before measuring whether the simpler approach was good enough. That's a couple of days of work we didn't need. The general pattern — measure the dumb baseline before reaching for the clever option — is something every engineering team learns and re-learns.
7. Description quality: per-shot, or context-aware?
This is the decision that ended up mattering most for retrieval quality. It's also the one that's least obvious in advance.
The default approach to annotating a shot is: feed the keyframe to the VLM, ask for a description, store the description. The shot is the unit of work. Each shot gets its own annotation, in isolation, with no awareness of what's around it.
This works. The descriptions are accurate. They're also, almost without exception, useless for the kind of retrieval we wanted.
Here's the failure mode. A shot of two people in a room with low lighting, viewed in isolation, looks like "two figures, dimly lit, one seated, one standing". That's accurate. It's also true of about a thousand shots in the corpus. A user searching for "the moment a character realizes the truth about their father" doesn't want the thousand-shot description. They want the one that says "a moment of confrontation between father and son, the son just having learned the family secret".
The fix is to give the VLM (or a separate LLM) more context than the single frame. The contextual approach reads the surrounding shots and the dialogue alongside the target shot, and writes a description grounded in the narrative situation, not just the visual content. Anthropic published a similar pattern for document chunks; we adapted it for shots.
The before-and-after on retrieval quality was dramatic. Descriptions stopped reading like museum captions ("two figures in a dimly lit room") and started reading like a script supervisor's notes ("a tense reconciliation between two characters with a long, fraught history"). Queries that meant nothing under the per-shot regime started returning the right shots under the contextual regime.
The cost is real. Every shot's description now requires not just the VLM's read of the keyframe, but also an LLM call that reads the surrounding context. Roughly twice the API calls. Latency goes up.
But it's the difference between a knowledge base an agent can actually use and one the agent could use if you also wired in the dialogue and the surrounding shots at query time. We'd rather pay for the context once at ingestion than pay for it on every query.
The lesson: context isn't optional for retrieval-grade descriptions of media. A shot in isolation is ambiguous in a way a paragraph of text isn't. Pixels carry less semantic load than words. To get retrieval-grade annotations, you have to give the annotator the context the audience would have when watching the shot in sequence. Article 3 in this series goes into the specifics with side-by-side comparisons.
8. Retrieval layer: single modality, or hybrid?
The last fork is on the query side. You have shot descriptions (text), keyframes (images), and structured fields (metadata). When a user asks for something, which of those do you query against?
The single-modality camp argues for keeping it simple. Pick the modality that maps best to your dominant query type. If users are mostly typing "find me a shot like this description", text retrieval is enough. If they're mostly uploading reference stills, image retrieval is enough. Adding more modalities is engineering complexity for marginal lift.
The hybrid camp argues that different queries need different modalities, and forcing the user to pick is bad UX. A storyboard agent might submit a text description for one query and a reference still for the next, and it shouldn't have to know which modality the underlying KB prefers. The retrieval layer should handle that.
We went hybrid. The cost is a more complex retrieval layer — three index types (text vector, image vector, structured-field), a fusion step that ranks across them, and weight tuning to figure out how much each modality contributes to the final ranking. But the agent on top of the KB benefits enormously. It can run text queries and image queries in parallel, fold structured filters into either, and not have to make a meta-decision about which approach to use.
The interesting design question inside hybrid retrieval is the fusion logic. Text scores and image scores aren't comparable on the same scale; one is a cosine distance in one embedding space, the other is a cosine distance in a different space. Naively averaging them produces nonsense. The standard fix is reciprocal rank fusion or learned-weight fusion; we use a variant that weights each modality by how much signal the query has for it (long detailed text query → text dominates; reference image with sparse text → image dominates).
The lesson: the cost of hybrid retrieval is borne once by the system; the cost of single-modality retrieval is borne every time a user wants to query the other modality. Building hybrid up front is more work; not building it produces a system that fights the user every time their query doesn't fit the chosen modality. For a retrieval system that an AI agent sits on top of — and that agent will throw arbitrary query types at — hybrid is the answer.
What we'd do differently: we under-invested in fusion-weight tuning. The default weights work; the tuned weights work better; the best results come from per-query weight adaptation, which we haven't built yet. Starting over, we'd treat fusion as its own real workstream rather than a "we'll figure it out" item at the end of the retrieval design.
A meta-pattern across the eight forks
Six of these turned out to be cases where the obvious choice — the heuristic, the uniform strategy, the flagship VLM, the neural model, the per-shot annotation, the single-modality query — was wrong, and the less-obvious choice was right. Two of them were cases where the obvious choice was right.
We didn't know that going in. We picked the obvious answer first on most of these, and noticed it was wrong by running the system end to end and looking at where the output failed. The pattern that connects all of them is something like: measure where the failures actually live, and fix them at the closest stage to where the cost is. Don't make stage one worse to fix stage four. Don't add the expensive model just because it benchmarks better. Don't ask the VLM for prose when the index needs structured fields.
This is the kind of advice that's easy to nod at and hard to apply. The forks always look different in the moment than they do in retrospect. We're sure there are forks we missed entirely, where we picked the wrong thing and never noticed because we never measured the right thing. Article 5 in this series is about the failures we did catch; we'd love to know what's in our blind spots that other teams have caught.
If you've built something similar and your trade-offs landed differently, we read every reply.
All illustrations generated using Genkee AI.