The GEST-Engine is not a photorealistic video model. That is exactly why it is useful.
The paper describes a system that takes a formal Graph of Events in Space and Time, grounds it inside a controllable game world, and captures multi-actor video with dense annotations. Instead of asking a neural model to remember who is holding what, the system keeps that state explicit. Instead of hoping two actors coordinate correctly, it schedules their events. Instead of labeling video afterward, it captures labels while the world runs.
For Story2Board, this is the important lesson: a storyboard should not be only a stack of generated images. It should also have an event graph underneath it.
The paper in one sentence
The GEST-Engine turns natural-language or procedurally generated event graphs into deterministic multi-actor simulation videos, while exporting frame-aligned labels such as depth, segmentation, pose, spatial relations, event timing, and captions.
That is a different goal from text-to-video generation. A text-to-video model tries to sample plausible pixels. The GEST-Engine tries to make every frame traceable back to a semantic specification.
The paper argues that this explicit world model gives three properties that implicit video models still struggle with:
| Property | Why explicit state helps |
|---|---|
| Object permanence | the system knows which object exists and who holds it |
| Multi-actor coordination | actors are scheduled through explicit temporal constraints |
| Temporal consistency | every event has a position in the story graph and frame timeline |
Those three properties are also the hard parts of storyboard continuity.
A video starts as a graph
The core representation is GEST: a Graph of Events in Space and Time. A GEST is a structured story specification. It says which actors exist, which objects exist, where things happen, which actions occur, and how events relate to one another.
That graph is not a prompt. It is closer to a production contract.
In a storyboard workflow, this matters because many visual mistakes are not image mistakes. They are state mistakes.
| Storyboard failure | Graph-level cause |
|---|---|
| A cup disappears between panels | prop state was not tracked |
| Two actors swap positions | blocking was not anchored |
| A reaction happens before the action | temporal order was underspecified |
| A door changes side of the room | location state was not persistent |
| A character holds a phone in one shot and nothing in the next | object possession was not part of the shot contract |
The GEST-Engine solves these problems in a simulation setting. Story2Board does not need to copy the game-engine stack, but it should copy the principle: keep semantic state outside the pixels.
The pipeline has four hard boundaries
The system turns a GEST into video through four stages:
| Stage | Job |
|---|---|
| Graph parsing and validation | read the GEST, check requirements, select valid episodes |
| Entity and action grounding | bind abstract actors, objects, and actions to concrete simulator entities |
| Temporal orchestration | resolve before, after, and same-time constraints into executable segments |
| Execution and capture | run actions in the game engine and collect aligned artifacts |
This separation is practical. The system does not let the story generator directly poke the simulator. It validates the story first. It grounds entities second. It schedules time third. Only then does it execute.
That is the same shape a serious storyboard agent needs. Gemini, the Story2Board co-director, can plan a shot, but the plan should pass through a structured state layer before image generation:
| Story2Board layer | Equivalent GEST idea |
|---|---|
| Script beat extraction | event graph creation |
| Character and prop cards | entity declarations |
| Shot plan | action and camera intent |
| Continuity checks | graph validation |
| Panel generation | execution and capture |
The takeaway is not that Story2Board should become GTA San Andreas. The takeaway is that a story generation system should have a formal checkpoint before rendering.
Temporal orchestration is the part most tools skip
The paper's temporal system is more interesting than it first sounds. The GEST supports relations such as before, after, and same_time, drawing on Allen-style interval logic. The engine then uses a Floyd-Warshall transitive closure to resolve implied order.
In plain language: if Alice enters before Bob speaks, and Bob speaks before Carla reacts, the system can infer that Alice enters before Carla reacts. That sounds obvious, but image and video models do not naturally preserve that logic across multiple shots.
Storyboard tools usually encode time weakly. A panel has an order, maybe a duration, and maybe a camera note. But multi-actor timing is richer than panel order:
| Timing question | Why it matters |
|---|---|
| Does the reaction overlap the action? | changes the shot design |
| Does one actor enter while another is already seated? | changes blocking |
| Does the prop exchange happen before the close-up? | affects continuity |
| Does dialogue begin during movement or after stillness? | affects pacing |
The GEST-Engine makes timing executable. That is the bar to aim for if we want AI storyboards to become reliable previsualization documents rather than attractive stills.
The LLM Director is constrained by tools
The paper includes an agentic text-to-GEST system. An LLM Director plans a story, but it does not directly mutate the graph. It uses tool calls against a programmatic state backend. The backend validates changes, tracks actors and events, checks capability constraints, and assembles an executable GEST.
That design is important. It avoids the common failure mode where an LLM writes a beautiful but impossible script.
The paper reports that earlier prompt-and-parse pipelines struggled with duplicated actors, dangling references, impossible actions, temporal cycles, and context overflow from the large capability registry. The tool-based Director is the correction: let the LLM decide intent, but let code own state.
That maps directly to Story2Board:
| LLM should own | Code should own |
|---|---|
| story intent | entity identity |
| dramatic emphasis | prop possession |
| shot purpose | temporal ordering |
| camera suggestion | schema validity |
| scene rewrite | continuity constraints |
This is also why the paper connects naturally to our MAGNET article. Characters can have goals and intentions, but the system still needs a durable world state beneath them.
Labels come free because the world is known
The GEST-Engine exports dense artifacts during a single simulation pass:
| Artifact | Why it matters |
|---|---|
| RGB video | the visible synthetic scene |
| Dense depth | geometric supervision |
| Texture-level instance segmentation | object and surface labels |
| 18-joint skeletal pose per actor | actor motion ground truth |
| Pairwise spatial-relation graphs | frame-level relationships between entities |
| 2D bounding boxes | detection and tracking supervision |
| Event-to-frame mappings | exact alignment between story graph and video |
| Natural-language descriptions | text paired with executable story state |
The phrase "zero marginal annotation cost" is doing real work here. Once the world is executing deterministically, the annotations are not a separate human labeling project. They are a byproduct of the simulation.
For Story2Board, the equivalent is not necessarily segmentation masks. The equivalent is traceability:
| Storyboard artifact | Traceable back to |
|---|---|
| panel image | shot record |
| character appearance | character card |
| prop position | prop state |
| camera move | camera intent and path |
| continuity warning | graph comparison |
| revision history | changed event or constraint |
If a generated panel changes a prop, the system should know which graph fact was violated. That is more useful than simply regenerating the image.
The scale is modest but strategically useful
The paper builds on GTA San Andreas through Multi Theft Auto. That choice is pragmatic rather than glamorous. The authors get a mature open world, controllable actors, many animations, low hardware requirements, and parallel execution.
The reported asset capacity is substantial:
| Resource | Scale |
|---|---|
| Object models | 733 |
| Object categories | 32 |
| Individual animations | 2,500+ |
| Animation categories | 101 |
| Actor skins | 312 |
| Game contexts | 70+ |
| Currently configured environments | 11 |
| Implemented granular action types | 46 |
The engineering effort is also large. The report describes roughly 38,000 lines of Lua for the engine core, 8,500 lines of C++ for native screenshot and pathfinding pieces, 53,000 lines of Python for generation and the text-to-GEST system, and a small JavaScript visualizer.
This matters because it keeps the paper honest. Explicit world models are not free. They require schemas, validators, capability registries, editors, adapters, capture systems, and batch infrastructure.
The benefit is control.
How it connects to recent world-model papers
The GEST-Engine sits in the same larger direction as World Narrative Model and ABot-3DWorld, but it attacks a different layer.
| System | Core control object |
|---|---|
| World Narrative Model | editable physical production state before video rendering |
| ABot-3DWorld 0 | spatial primitive and panoramic exploration for reconstructable worlds |
| GEST-Engine | event graph that deterministically drives multi-actor simulation |
WNM is about controlling video through a world blueprint. ABot-3DWorld is about exploring and reconstructing 3D space. GEST-Engine is about making story events executable and observable.
That last part is exactly what storyboard systems need more of. Shot continuity is not only about the same-looking face or the same-looking room. It is also about whether the event logic survived rendering.
What Story2Board should borrow
The best Story2Board version of GEST is not a full simulator. It is an event graph under the storyboard.
A practical graph could store:
| Graph field | Storyboard use |
|---|---|
| Actors | identity, role, current location |
| Props | ownership, visibility, state changes |
| Locations | persistent set layout and camera constraints |
| Events | actions, reactions, dialogue beats |
| Temporal edges | before, after, overlap, same-time |
| Camera notes | shot size, angle, movement, subject priority |
| Validation flags | impossible timing, missing prop, inconsistent blocking |
This would give Gemini a more reliable substrate. The co-director could still speak naturally, but the system would keep an executable memory of what has to remain true.
That connects to our CANVAS breakdown, where visual continuity is treated as story state, and to our Story2Board consistency article, where pixel-level consistency is only one layer of the problem.
Where the paper is limited
The GEST-Engine is powerful, but its limits are clear.
First, it inherits the look and physics of an old game engine. That is fine for synthetic supervision and diagnostic benchmarks, but it is not final visual quality for modern filmmaking.
Second, manual mapping work matters. The platform may contain 2,500+ animations and 733 objects, but the currently configured subset is smaller. Every new environment, action, and object affordance needs engineering.
Third, camera control exists but is not the center of the paper. For storyboard use, camera language needs to be richer than a spectator workaround. Shot size, lens behavior, camera motivation, eyeline, and coverage logic need first-class treatment.
Fourth, synthetic determinism can hide real-world ambiguity. A real set has messy hands, imperfect blocking, reflections, occlusions, soft cloth, missed marks, and actor improvisation. The graph is a strong contract, not a complete substitute for production judgment.
The product lesson
The GEST-Engine makes a useful argument: the future of AI video planning is not only better pixels. It is better state.
For storyboard tools, that means the central document should not be just images plus captions. It should be a structured graph that knows:
| Question | Why it matters |
|---|---|
| Who is in the scene? | identity and continuity |
| What are they doing? | action planning |
| What do they hold? | prop state |
| Where are they? | blocking and location memory |
| What happens first? | temporal logic |
| Which frame proves it? | traceable review |
That is the bridge from AI image generation to usable previsualization.
The GEST-Engine is a synthetic data system, but the design lesson is broader. A storyboard becomes more useful when every panel can answer: which event created me, which state do I preserve, and which next shot depends on me?
References
Cudlenco, N., Masala, M., & Leordeanu, M. (2026). The GEST-Engine: From Event Graphs to Synthetic Video. arXiv preprint arXiv:2607.12231. https://arxiv.org/abs/2607.12231