The first time you wire up a vector database and run a query that returns the right result, it feels like magic. Type "a tense reconciliation scene" — get back a tense reconciliation scene. The system worked. Ship it.
The second time, the system returns four shots about cooking and one about a car chase. The query was "a tense reconciliation scene". The embedding similarity between your query and those shots was real, in the sense that some math happened. The relevance was not.
This article is about the gap between "vector search works" and "vector search actually answers the queries an AI agent will throw at it". The gap is wider than you'd think, and most of it is engineering, not modeling.
For context, Story2Board's film-reference library contains 270,000+ annotated shots across 250 acclaimed films. The records combine shot descriptions, available dialogue, structured metadata, and keyframes. The retrieval layer needs to handle queries from an AI storyboard agent that's stitching shots together into recommendations. The agent doesn't care which retrieval modality is "best"; it wants the right shots for whatever query it just composed.
What follows is the engineering that turned that goal into a working layer.
Two embeddings, not one
The first decision is what to embed.
Most teams default to a single text embedding — embed the description, store the vector, query the vector with text. This works for the simple case. The simple case is rare.
For a media library, you really have two distinct things to retrieve over: the textual description of each shot (what we wrote about it) and the visual content of the shot itself (what the camera saw). These embed into different spaces. Text embeddings are good at capturing semantics from words; image embeddings are good at capturing visual similarity. A query about "a low-angle shot of a figure against the sky" has a strong text signal. A query that's just "find me something like this still I uploaded" has only a visual signal.
We index both. Two embeddings per shot. Both stored in the same database, both indexed, both queryable independently. The retrieval layer decides which one (or which weighted combination) to use based on what the query has.
The cost is real. Two embedding models running on every shot at ingestion. Two index columns at storage. Two index builds at maintenance time. The benefit is that the system handles both query types — and the hybrid case where they're combined — without forcing the user (or the agent) to pick one.
A subtle decision inside this: the text embedding is computed over more than just the visual description. It runs over a concatenation of the narrative-aware context prefix, the visual description, and the dialogue. The image embedding runs over the keyframe alone. Both choices reflect the same principle: embed the thing the query will ask about. Queries about narrative function need text embeddings that capture narrative function; queries about visual look need image embeddings that capture visual look.
Hybrid query design
Once you have multiple modalities, you have to decide how to combine them.
A user query — or an agent query — usually has different amounts of signal in different modalities. A long detailed text description has strong text signal and no image signal. A reference still with a one-word note has strong image signal and weak text signal. A tag-style query like "low-angle, golden hour, single character" has strong structured-field signal and weak signal in either embedding.
The wrong way to fuse these is to weight them equally. Equal-weight fusion produces the worst of all worlds — text-heavy queries get diluted by irrelevant image scores, image-heavy queries get drowned by sparse text matches.
The right way is to weight each modality by how much signal the query has in it. If the text query is long and specific, text dominates. If a reference image is attached, image weight goes up. If structured filters are present, they short-circuit the candidate set before any vector retrieval happens. The fusion is per-query adaptive, not a global hyperparameter.
The implementation pattern we landed on is roughly this:
- Run each query type independently against its index, getting a top-N candidate list with scores.
- Apply structured-field filters (if any) as hard constraints — anything that doesn't match the camera-angle or lighting filter is dropped from all candidate lists.
- Use reciprocal rank fusion across the surviving candidates, weighted by per-query signal strength.
- Return the top-K from the fused ranked list.
Reciprocal rank fusion is one of those algorithms that reads like a hack and works like a charm. It avoids the score-normalization problem (text and image vector scores aren't on the same scale) by using rank instead of raw score. The math is one line and the behavior holds up across query types.
There's a per-query weight tuning step that we under-invested in. The default weights work. Tuned weights work better. Per-query learned weights — where the fusion logic adapts to query characteristics — work best. We're still on the second rung.
Choosing an index — the boring decision that matters
The index choice is the part of vector retrieval that most articles skip and most production systems get wrong.
The two main families are flat indexes (compute exact distance to every vector at query time) and approximate-nearest-neighbor indexes (use a graph or tree structure to skip most distance computations). For small corpora, a flat index is fine and gives exact results. For large corpora, the wall-clock cost of flat retrieval is unacceptable, so you need an ANN index.
Within the ANN family, the practical choice is a graph-based index — there are several implementations available across the major vector database extensions, all of them based on roughly similar ideas. The shape of the trade-off is:
- Higher graph parameters (more neighbors per node, longer build-time exploration) → better recall, slower build, more memory.
- Lower graph parameters → faster build, less memory, recall drops on hard queries.
The defaults in most vector libraries are reasonable for general-purpose use and slightly conservative for retrieval-heavy production systems. We tuned the build parameters up modestly — recall mattered more than build speed, and build was a one-time cost at ingestion.
There's also a query-time parameter that controls how many candidate neighbors get explored at retrieval. This is the fast knob. Turning it up improves recall at a small latency cost. Tune this with a representative query set and watch where recall plateaus.
A practical note: vector index parameters are not as universal as the database documentation suggests. The right values depend on dimension, distance metric, dataset distribution, and recall target. Don't trust the defaults blindly. Run a small evaluation suite — even a few dozen queries with known correct answers — and pick parameters that hit your target recall.
How a graph-based ANN index works at query time, in one picture:
The memory cliff at index build time
The single biggest operational surprise in our build was index memory.
A vector index, especially a graph-based one, holds a lot of state in memory while it's being built. The memory requirement scales with the corpus size and the index parameters. For a small corpus, this doesn't matter — the build fits comfortably in machine memory. For a large corpus with high-dimension vectors, the build memory can exceed available RAM, at which point the build either swaps (catastrophically slow) or fails outright.
Our first attempt at building the index on the full corpus crashed for exactly this reason. The build allocated more memory than the machine had, the kernel killed it, and the partial state had to be cleared and restarted. We hadn't sized the memory requirement up front; we'd assumed the build would work because every smaller test had worked.
The fix has a few parts:
The first part is sizing the build. Most vector index implementations document a formula for build-time memory, but the formula is usually a lower bound. Multiply it by a generous safety factor before assuming you have enough RAM.
The second part is building incrementally. Some index implementations allow appending vectors to an existing index without rebuilding from scratch. If yours does, an incremental build is much friendlier on memory than a single bulk build.
The third part is separating the build host from the query host. If you only have memory pressure during builds, you can build on a beefier machine and copy the index to a smaller serving machine. The serving machine's memory needs are much lower than the build machine's.
We landed on a combination: a one-time bulk build on a host with adequate memory, followed by incremental appends as new films were added. The bulk build is the expensive part; the appends are cheap.
Cross-language retrieval
A nice property that falls out of using multilingual text embeddings is that queries in one language can find shots whose dialogue (or description) is in another language.
Our corpus contains shots with dialogue in Cantonese, Mandarin, Japanese, French, Korean, English, and a few others. The text embedding is computed over a concatenation that includes the dialogue in its original language and the description in English. The multilingual embedding model handles the mixed-language input gracefully, mapping semantically similar content into nearby regions of the vector space regardless of script.
The result is that a query like "a regretful apology that comes too late" — typed in English — surfaces shots in any language where that beat is present. Without multilingual embeddings, that query would only find shots whose description happened to contain the English word "regret". With them, the semantic content of the query matches the semantic content of the shot's narrative description, and language barriers stop being barriers.
The cost is that the embedding model is larger and slower than monolingual alternatives. For ingestion, this matters — you're embedding hundreds of thousands of items, and the per-item cost is non-trivial. For query time, the embedding cost is amortized across cached models and is small relative to the database round-trip.
If your corpus spans languages, treat multilingual embeddings as the default and budget for the size cost. The retrieval-quality difference is large enough that it's almost always worth it.
Evaluating retrieval quality
The last piece — and the one most teams skip — is having a way to measure whether retrieval is actually working.
The temptation is to "look at some queries and see if the results look right". This works for spot-checking but fails as a real evaluation. Retrieval quality is a number, and you need a way to compute it.
The pattern we settled on is a small evaluation set: a few dozen queries, each with a hand-curated list of correct shot IDs. The retrieval system runs each query and we compute precision-at-K and recall-at-K against the expected results. Changes to the system — a new index, tuned parameters, a different fusion strategy — get evaluated against this set before going live.
The evaluation set is hard to build. Each query needs a thoughtful curation of which shots should match, which means watching enough of the corpus to know what's there. We treated this as a one-time cost early in the build, and we update it slowly as the corpus grows.
The set is small enough that it's not a full benchmark — it's a smoke test. But a smoke test that catches regressions is dramatically better than no smoke test, and the alternative is shipping a worse retrieval system without noticing.
A second evaluation pattern: agent-driven evaluation. Once we had an agent on top of the retrieval layer, we could run end-to-end tasks ("storyboard a noir bar entrance") and grade the agent's outputs. The agent's grade is a function of retrieval quality, so improvements in retrieval show up as improvements in agent output. This is a coarser signal but it's grounded in the actual user-facing behavior.
Schema migration without re-embedding everything
The last engineering challenge is what happens when you change the embedding strategy mid-flight.
We did this once. Halfway through the build, we made a change to what goes into the text embedding by adding dialogue and a narrative-aware context prefix. The embedding for every shot annotated under the old scheme was now subtly wrong: the retrieval space had shifted, and old vectors didn't match the new query distribution as well.
The naive fix is to re-embed every shot. This works, but it's expensive — every shot needs another inference pass through the embedding model, which at corpus scale is hours of compute and a non-trivial bill.
The fix we used is two columns: a new embedding column for the new scheme, populated only for shots annotated after the change, and the old column kept around for shots annotated before. At query time, the retrieval layer prefers the new column when it exists and falls back to the old column when it doesn't. The schema looks like a join with a coalesce, and the query plan is the same as a single-column lookup.
This is a transitional pattern. Eventually we backfill the old shots with new embeddings, but we don't have to do it on a tight schedule. The system works during the transition.
The general lesson: build the embedding column as a versioned thing from day one. Adding a _v2 suffix is cheap up front; renaming columns and back-filling under load is expensive.
Closing
The pattern across all of these — choice of embeddings, fusion logic, index parameters, memory sizing, evaluation, migration — is that vector retrieval is mostly an engineering problem once the modeling questions are settled. The cool part of vector retrieval is the math. The expensive part is everything around the math: how it's stored, how it's queried, how it's measured, how it changes.
If you're building a vector retrieval system that needs to actually answer the queries an AI agent will throw at it, treat the engineering as primary and the modeling as already mostly solved. The off-the-shelf embedding models and ANN indexes are good enough. The defaults are not.
If you've built something similar and your trade-offs landed differently, we read every reply.
Story2Board uses this retrieval layer when its AI co-director searches real-film references while planning shots. See how those references fit into the full AI storyboard generator workflow.