We tried generating a storyboard with off-the-shelf AI tools last summer. The brief was simple: a noir scene where a private detective walks into a bar, spots someone he hasn't seen in years, walks over, sits down. Six panels. Should be easy.
What we got back was technically a storyboard. It had six panels. The detective was in them. The bar was in them. But the angles were the kind you'd expect from a generic stock-photo collection. Panel one was a head-on medium shot. Panel two was the same medium shot, slightly different. The third panel — the moment the detective sees her — was framed from somewhere above and behind, where no audience member would ever sit. The close-up at the end looked like a portrait taken in a different lighting setup than the rest.
The model knew the words. It knew "low-angle hero shot" and "Dutch tilt" and "tracking dolly", because every film-theory blog on the internet had taught it those words. But it had no instinct for when to use which one. It didn't know what a noir bar entrance is supposed to feel like. It didn't know that the third panel — the moment the detective sees her — is the one the entire scene is built around, and that the framing of that single panel decides whether the scene works.
We sat with that draft for a long time. The fix wasn't more prompt engineering. It wasn't a better image generator. The model didn't lack pixels or vocabulary. It lacked taste. And taste, when we thought about it, isn't something you teach a language model from generic training data. Taste comes from having watched a lot of good films and remembering specific shots from specific scenes.
So we decided to give the model a memory of films. A real one. A library of classic shots, indexed and searchable, so that when the agent proposes "the moment the detective sees her", it's reaching for an actual shot in an actual film instead of inventing one from training-data fog.
That decision turned into a few weeks of pipeline work. This is what we built and why.
Why current AI storyboards fall flat
We kept seeing roughly three failure modes in AI-generated storyboards. They show up in different products in different proportions, but the same patterns repeat.
Shot vocabulary without judgment
Every modern LLM has read enough film theory to know the names of the standard shots. Wide, medium, close. Low angle, high angle, eye level. Establishing, over-the-shoulder, two-shot, insert. Ask it to write a storyboard and it will use these words correctly, in the sense that the words are spelled right and used in plausible-sounding sentences.
But it has no opinion about when to use them. A confrontation in a kitchen and a confession in a bedroom will get suggested with similar shot choices. The model can't tell that the bedroom confession needs to start tight on the speaker and pull back to reveal the listener. It just writes "medium shot" and moves on.
Image-level continuity collapse
This one's been written about a lot — we've covered it in earlier posts on CANVAS and Story2Board. The character's hair color drifts between panels. A prop disappears and reappears. The room rearranges itself when the story returns to it three shots later. There are good papers proposing fixes; some of those fixes are starting to land in production tools.
But the underlying problem is harder than the papers usually acknowledge: continuity isn't only about pixels matching. It's about whether the shot language of consecutive frames forms a coherent scene.
No scene-level grammar
This is the failure mode we care about most, and it's the one no one talks about. AI storyboard tools today treat each panel as an independent generation request. You write a prompt for panel one, get an image. You write a prompt for panel two, get another image. Even when the tool is clever about character consistency, it has no notion that panels one through six are supposed to be one scene — a scene with rhythm, with information that gets revealed in a specific order, with shot sizes that escalate or de-escalate based on what's happening emotionally.
A real director thinks about a scene as a whole and chooses each shot in service of the scene. AI tools today think one panel at a time.
The fix we landed on is straightforward in concept and weird in execution: treat the AI agent like a junior director who's watched a lot of films, and give it the ability to remember specific shots and pull them up when relevant. It's RAG. But for cinema. The interesting bit is what "knowledge" means when the documents are video.
That's what the rest of this post is about.
The shape of the knowledge base
Most articles about knowledge bases lead with the size. We're going to lead with the shape, because the shape turned out to matter more.
The current Story2Board library contains 270,000+ annotated shots across 250 acclaimed films.
What's in our film knowledge base, qualitatively:
- A curated set of classic films, picked for canonical taste rather than maximum volume. The list looks like what you'd see if you asked a serious film school what to watch first.
- Every film cut into individual shots — the unit of cinematic grammar. A shot is one camera, one continuous take, one piece of compositional intent.
- Every shot tagged with a structured set of cinematic attributes: composition, framing, camera angle, lighting setup, mood, what's in the frame, what the camera is doing.
- Every shot's dialogue pulled from the soundtrack and time-aligned to its frames. So the same kitchen scene with the same camera angle can be either a confrontation or a cooking lesson, depending on what's being said. The KB knows the difference.
- Every shot indexed twice — once by the text of its description and once by the image itself. So queries from either side land sensibly.
- All of this living in a database with vector search, on a single consumer PC. No GPU farm. No distributed cluster. Just a tower under a desk and a few weeks of pipeline work.
The headline isn't the count of shots in the library. The headline is that every shot has rich metadata, and every shot is searchable from either text or image, and the agent can mix those query types in a single retrieval pass.
That last property — hybrid retrieval — is what makes the KB feel different from a tag-based shot database. ShotDeck has a great tag-based shot database. We didn't want to rebuild ShotDeck. We wanted to build something a language model could query.
The pipeline, end to end
Here's the full pipeline, end to end. We'll keep this high-level and focus on how the stages connect.
Preprocess and decode. Strip the credits off the front and end of the film, so the pipeline doesn't waste time on title cards and copyright text. Decode the video on the GPU rather than the CPU, because video decoding is one of the things modern GPUs do significantly faster than even a strong CPU.
Shot detection. Use a neural model to find the boundaries between shots. We tried a classical frame-difference detector first and abandoned it within a day — it over-cuts on dissolves (a single dissolve becomes three "shots") and under-cuts on whip pans (a fast camera move gets read as one shot when it's actually two). The neural model handles both. We tested both on the same set of films before committing.
Long-shot semantic split. Even after the shot detector runs, some shots are too long. A two-minute Wong Kar-wai single-take has multiple compositions inside it; treating it as one shot loses everything that's interesting about the take. We re-examine any shot above a length threshold and sub-divide it when the composition shifts mid-shot.
Keyframe selection. Pick representative frames per shot. Long shots get sampled at multiple positions inside the shot, because one frame at the midpoint loses information. Short shots get a single midpoint frame. We also drop pure-black transition frames so the annotator doesn't get confused by them.
Subtitle extraction. Pull dialogue tracks (when present) and align them to shot boundaries. This is the cheapest stage and one of the most useful. Sometimes the dialogue is the only signal that a shot is a confession rather than a confrontation. Without it, descriptions read like museum captions.
Vision-language annotation. A vision-language model reads each keyframe and produces structured cinematic tags. This is the most expensive stage by far, both in cost and in latency. Most of the engineering effort in this stage went into picking the right model (we'll get to that in the next section), running it concurrently without hitting rate limits, and recovering gracefully when calls fail.
Camera motion analysis. Optical-flow analysis classifies each shot as static, panning, tilting, zooming, tracking, or compound. We initially planned to use a neural model for this, but the optical-flow approach turned out to be cheaper, faster, and accurate enough for our purposes.
Narrative-aware context prefix. For each shot, an LLM reads the surrounding scene and dialogue, and writes a short prefix that captures what's happening narratively, not just what's visually present. This is the stage where descriptions go from "two people in a room with low lighting" to "a moment of reluctant reconciliation between two characters with a long history". It's one of the most important stages for retrieval quality.
Dual-modal embedding. Compute a text embedding (over the description plus dialogue plus narrative prefix) and an image embedding (over the keyframe). Index both.
Store. Push everything into a Postgres database with a vector-search extension. Build an approximate-nearest-neighbor index. Add structured-field filters so queries can mix vector similarity with metadata constraints (camera angle, lighting, era, mood).
A typical feature-length film takes about an hour to process end to end on the consumer hardware we used. We ran several films a day, in batches, over a few weeks. Most of the work was unattended — the pipeline would run for hours, we'd check in, fix whatever broke, and let it keep running.
Six decisions that shaped the stack
Of the dozens of choices that went into this build, six were the ones we kept coming back to.
1. Curated, not crawled. It was tempting to scale up by scraping random YouTube clips. There's a lot of video on YouTube. We could've multiplied the data ten or twenty times. But "good cinema" and "any cinema" are different signals, and a knowledge base teaches the agent whichever signal dominates the data. We'd rather have a smaller library where every example is good than a bigger one where the median is mediocre. Retrieval quality is bounded by the worst examples in the corpus, not the count.
2. A neural shot detector, not a heuristic one. We tested a classical frame-difference scene-detection library and a neural shot detector on the same films. The classical detector over-cut on dissolves — a single dissolve became three "shots", because the algorithm read the in-between frames as cuts. It also under-cut on whip pans. The neural detector got both right. The cost is higher, but the difference matters when downstream stages are using shot boundaries as their unit of work.
3. Two keyframe strategies, not one. A long single-take has multiple compositions inside it; sampling once at the midpoint loses the rest. Long shots get sampled at multiple positions across the shot. Short shots get a single midpoint frame. The cost difference is small — long shots are rare relative to short ones — and the gain is that long shots get annotated as the multi-composition takes they actually are.
4. A budget cloud VLM, not a flagship one. Cost is the bottleneck when you're running a vision-language model on every keyframe in a large library. The budget option we used is roughly two orders of magnitude cheaper than running the same workload through a flagship VLM. The quality gap, on cinematic shot description, was small enough that we took the savings and put them into more iterations of the pipeline. We tried both. The flagship produced slightly more elegant prose; the budget option produced slightly less elegant prose with the same structural information. Structural information is what matters for retrieval. We took the savings.
5. Contextual Retrieval, but for shots. Anthropic's Contextual Retrieval paper proposes adding document-level context to each chunk before embedding. The idea is that a chunk in isolation is often ambiguous, and a short context prefix grounds it. We do the same for shots, but the "document" is the surrounding scene and the dialogue. Without it, descriptions of any individual shot tend to read like a museum caption — accurate, decontextualized, useless for retrieval. With it, they read more like a script supervisor's notes. This is the single biggest change to retrieval quality across the entire pipeline.
6. Index both text and image, not one. Text queries hit the description embedding. Image queries (when the user uploads a reference still) hit the image embedding. These are different vectors, and they catch different things. Text is good at "find me a low-angle shot in a kitchen at night"; image is good at "find me a shot that feels like this one I'm holding up". A hybrid retrieval layer mixes both into a single ranked list. The user doesn't have to pick which kind of query they're making — the agent figures out which signal to weight, based on what was provided.
From knowledge base to storyboard agent
The knowledge base sits on disk. The interesting part is what the agent does with it.
When a user describes a scene to our storyboard agent, the agent calls a tool we wrote — a film-search tool. That tool runs three queries in parallel:
- a text query against shot descriptions and dialogue
- an image query, if the user attached a reference still
- a structured filter against shot metadata (camera angle, lighting, era, mood)
The film-search tool merges the three result lists into a single ranked output, weighting each query type by how much signal it has. If the user gave a long, detailed scene description and no image, the text query dominates. If they uploaded a reference still and asked for "something like this", the image query dominates. If they specified hard constraints — "low-angle shot, golden hour, single character" — the structured filter rules out everything that doesn't match.
The agent gets back a handful of candidate shots. Each one comes with its full metadata and a URL to the keyframe. The agent picks one (or a few) and uses them as concrete references when proposing each frame of the storyboard.
This changes the conversation. Without the KB, when the agent says "we should open with a low-angle hero shot", the user has to take its word for it. The recommendation is only as trustworthy as a generic LLM's read of generic film theory. With the KB, the agent says "we should open with a low-angle hero shot, like this moment in Lawrence of Arabia where the camera looks up at the figure against the desert horizon to make him look monumental." And it shows the reference. The user can look at the reference and say "yes, that's the energy", or "no, I want something darker", or "different — give me something more like Blade Runner". The agent goes back, re-queries, and comes back with new options.
The output is a storyboard panel grounded in retrieved film references. The co-director explains why a reference fits, while the director decides whether to use it. The approved direction stays connected to the scene and shot as the board is generated and revised.
This is the loop the knowledge base exists to power. Without it, the agent would be drawing shot vocabulary from the same training-data fog as everyone else's tool. With it, the co-director can ground a shot recommendation in a specific retrieved film reference and explain the reasoning behind it.
The retrieval layer is in production, but its coverage is finite. A library of 250 films cannot represent every genre, period, or visual tradition, so the co-director presents references as options for the director to accept, reject, or refine.
A few things we got wrong
We made plenty of mistakes during this build. A few of the more memorable ones, before we wrap up.
We added a thread-count flag to ffmpeg, expecting things would speed up. They got slower. It took us half a day to figure out we were already running multiple worker processes, and adding threads inside each worker just made the workers fight each other for CPU cores. Removing the flag made the pipeline noticeably faster.
We imported PyTorch in the same Python process as ffprobe, and ffprobe stopped working. PyTorch's import-time DLL setup modifies the search path in a way that breaks subprocess invocations of system binaries. The fix was to keep the PyTorch and ffprobe stages in separate processes. The debugging session was less fun than the fix.
The cloud VLM API would silently stall for long stretches at random. Our first few times encountering this, we killed the process — assumed it was hung. It wasn't. It was waiting. The API would resume on its own. We added a longer timeout and stopped killing things prematurely.
Our database inserts weren't idempotent. We ran the same film through the pipeline twice during development and ended up with duplicate rows in the index. Catch-then-fix; the lesson stays.
There are more. The general theme is that pipelines like this fail in ways that are obvious in retrospect and invisible at design time, and the only way to find them is to run the thing on real data and watch what breaks.
Go deeper on retrieval
This overview covers the complete pipeline. The companion article Vector Search Is Easy. Vector Search That Helps an AI Agent Is Harder explains the dual-modal embeddings, hybrid ranking, index tuning, multilingual retrieval, and schema migration in detail.
Knowledge bases used to be a thing for legal discovery and customer-support search. We built one for cinema. The result is an AI that, instead of hallucinating shot vocabulary from a generic LLM, reaches for a specific shot in a specific film and says "something like this".
That's a different kind of AI directing. And honestly, it changed the output more than we expected. The first time we ran the agent against the new KB and watched it cite In the Mood for Love in a recommendation about composition, we had a small moment. It felt less like talking to a chatbot and more like talking to someone who'd watched the films we cared about.
Whether this generalizes — whether retrieval over real films becomes table stakes for AI storyboard tools the way RAG became table stakes for chatbots — is the question we're working on next.
To see how the retrieval layer fits into shot planning, explore the AI storyboard generator workflow.