SceneAssistant: what a visual feedback loop actually buys

Somewhere in SceneAssistant's system prompt there's a line telling the agent to build the scene one piece at a time. "Do NOT create and place all the desired objects at once." Then, twenty pages later in the appendix, the same authors report that their agent creates everything at once anyway.

That gap is the most useful thing in the paper.

SceneAssistant (Luo et al., 2026) is a vision-language agent that builds 3D scenes from a text description by looking at a render of what it just made and correcting it. Peking University and NVIDIA, released with code. If you're building anything where a model inspects its own output — generated frames, layouts, charts, UI — the interesting question isn't whether the loop works. It's which parts of the loop hold when the model stops cooperating.

What the agent is allowed to see

Every step, the agent gets one image: the render of the scene as it stands right now. Not the previous five renders. Not the render from before its last mistake. One.

The reasoning behind that is stated plainly in the paper. Feeding every historical image into a long-horizon run overloads the context, and older images have limited bearing on the current decision. So they drop them. They also drop the agent's own earlier reasoning, its plans, and any stored memory, on the grounds that the information they hand it is reliable while its previous thinking may not be.

The last row matters more than it looks. In the released code, the system message is written, sent once with the next prompt, and then set back to empty. A collision warning the agent ignores does not follow it around. It gets one shot at being heard.

This is a different bet from the one WorldWeaver makes with persistent world state registers, where the whole design goal is that state survives across frames. SceneAssistant's position is that if you can re-render the truth on demand, you don't need to remember it. Both can be right. They're solving different shapes of problem — one is converging on a single scene, the other is maintaining continuity across a sequence.

Thirteen verbs instead of a JSON blob

The agent doesn't write Blender Python. It calls from a fixed set of thirteen actions: create, duplicate, delete, translate, place, rotate, scale, four camera moves, a floor texture generator, and finish.

The paper's argument for this is about attention budget, not capability. A vision model asked to emit correct Blender syntax spends its effort on syntax. The authors wanted it spending that effort on where the chair goes.

They tested the claim. One ablation, NoActionAPI, keeps the entire loop — same visual feedback, same steps — but makes the agent output the whole scene as one JSON structure instead of calling actions. Same expressive power in principle. In practice the scores split in an odd way, which we'll get to.

There's a detail in Create worth stealing. When the agent creates an object, that object lands at the center of the scene and stays there. It isn't placed. The agent has no idea what a generated "book" will look like — horizontal, vertical, broken — until it sees the next render. So the design refuses to let it guess: make the thing, look at the thing, then decide where the thing goes.

The instruction that didn't hold

Now the gap.

The system prompt is direct about incremental work. Build the scene piece by piece. Keep what's already there correct before adding more. And a hard number: at most three kinds of object touched in a single batch.

Here's what the appendix says happened:

"SceneAssistant often tends to initialize all required objects first and subsequently attempts to adjust their positions and orientations simultaneously. This batch-processing behavior significantly increases task complexity. Despite explicitly instructing the agent in the prompt to complete the scene step-by-step, our method still struggles with highly complex scenes containing numerous objects."

Despite explicitly instructing the agent in the prompt. That clause is doing a lot of work.

We've watched the same thing. An instruction that asks a model to be patient competes against every gradient in it that wants to complete the task in one pass. When the two disagree, the instruction loses, and it loses quietly — the agent doesn't announce that it's ignoring you, it just does everything at once and the output degrades in a way that's hard to trace back to a prompt line.

What the code enforces instead

One rule in the same system did hold, and it's the one that isn't a request.

Create and Duplicate can't share a batch with Rotate, Place, Scale, or Translate. If the agent tries, a validator in the released code checks the batch before anything executes, finds both action types present, and refuses the entire batch. Nothing runs. The scene doesn't move. The agent gets told why on the next step, once.

And the refusal isn't free. The step counter has already advanced, and a run only gets twenty steps. Break the rule and you've spent 5% of your budget on nothing.

Read that table top to bottom and there's a clean split. The three rules with no enforcement path are the three the paper reports problems with. The three with an enforcement path — a refusal, a correction, a detector — are the ones that behave.

The floor rule is the cheapest version of this. Objects below Z=0 get lifted automatically. Nobody asks the model to be careful about it, so the model never has to be. The collision rule sits in the middle: a BVH check computes the answer and hands it over as text, because as the ablations show, the agent won't get there on its own.

What the loop is actually worth

Thirty scenes, ten human raters each, scored 1 to 10 on spatial layout and object quality, plus a preference vote. The authors skip automated metrics on purpose, and their reasoning is fair: a low collision count doesn't tell you whether the scene matches what someone asked for.

Two things stand out.

Taking away the feedback loop hurts layout more than taking away the action APIs. 6.26 against 7.01. Given the paper spends a full section arguing for the API design, the loop turning out to be the more load-bearing half is a useful correction to have in print.

The second one is stranger. NoActionAPI scores 7.005 on layout, within striking distance of the full system's 7.600 — but its preference rate collapses to 35.91%, almost half. People looked at scenes with nearly the same layout score and didn't want them. The gap shows up in object quality, 6.591 against 7.277, which the authors attribute to the agent burning attention on JSON strings instead of noticing that an asset came out wrong. Layout correctness and "would a person pick this" are not the same axis, and the second one is the one that pays.

Against the earlier indoor baselines the margins are wider — 6.888 for SceneAssistant versus 5.800 for SceneWeaver and 4.475 for Holodeck — but those comparisons carry more asterisks. Different asset libraries, different renderers, one of them running in Unity. The ablations are cleaner because everything but the one variable is held still.

Where the loop goes blind

The limitations section is unusually honest, and worth reading before anyone gets excited about self-correcting agents.

The agent sometimes can't tell whether a chair faces a table. Not a subtle judgment about composition — the basic question of which way a thing points, from a rendered picture, with the object labeled.

It also won't move the camera. Four camera actions exist. The agent reaches for two of them, ViewScene and FocusOn, and when a render is ambiguous it mostly doesn't try a different angle. The authors describe this as overconfidence in its own judgment despite poor visual feedback, which is a polite way of saying the agent doesn't know when it can't see.

And when the 3D generator produces something malformed, the agent frequently doesn't notice. There's a delete-and-retry path for exactly this case, but using it requires spotting the defect first. The stated consequence: getting one good scene can take several full reruns of the same prompt.

Put those together and the failure has a single shape. The model is confident about an image it hasn't understood. It won't ask for a better look, and it won't flag its own uncertainty. Which is fine when the cost of a wrong action is one more step in a twenty-step loop. It stops being fine when the wrong action costs something real.

How we hold the same line

We're building the same capability into Story2Board — the assistant can read back a frame it just generated and talk about what's in it — and this paper shaped where we put the brakes.

Our first gate predates the paper and matches its finding exactly. The assistant plans one shot per turn. Not three, not a scene's worth. If a director wants a batch, they have to say so, and the assistant has to warn them what batching costs in quality before it runs. That constraint doesn't live in a paragraph of instructions where the model can quietly outvote it; it's part of the protocol, the same way the batch validator is. We arrived at it from the same symptom the paper describes: told to work shot by shot, a model reads a script and starts laying out twelve panels, and each one is a little worse than the one it would have made if you'd asked for it alone.

The second gate is about what happens after the assistant looks. In SceneAssistant, an agent that misreads a render wastes a step. In our product, an agent that misreads a frame and redraws it spends the director's credits on a picture nobody asked for — and by the paper's own account, the misreading is the likely case, not the edge case. So the assistant reports what it sees and stops. It can say a character's jacket doesn't match the previous panel. It can't act on that alone. The director decides whether to redraw.

The third thing we took is smaller and more concrete. The paper's ablation on visual prompting — stripping the object name labels and the coordinate axis overlay off the render — degrades the agent's spatial control badly. The overlay isn't decoration. It's what lets the model say this object rather than guessing. When we ask an assistant to point at something in a generated frame, we should expect to owe it the same scaffolding, not assume it can ground a description in raw pixels.

What we'd want tested next

The paper closes by suggesting the action API could collect action-scene pairs to train a dedicated agent. That's the obvious direction and probably the right one.

The thing we'd rather see measured is smaller. The system prompt tells the agent to check the render before acting again. The appendix says it often doesn't. Nobody ran the ablation where that instruction gets an enforcement path — where an agent that acts twice without looking gets its second batch refused, the same way a mixed Create batch is refused. Given that every other soft rule in this system failed and every hard rule held, that's the experiment sitting right there.

Twenty steps, thirteen verbs, one image at a time. The loop is real and it earns its place in the numbers. It just doesn't earn the right to be trusted unsupervised, and the authors are straight about why.

Reference

Luo, J., Tang, J., Lu, R., & Zeng, G. (2026). SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation. arXiv:2603.12238. https://arxiv.org/abs/2603.12238

Code released by the authors at github.com/ROUJINN/SceneAssistant. Prompt text and enforcement behavior described above were read from that repository; file contents may change after publication.

All diagrams were created by the Story2Board team for this analysis.