ReactWorld
Structuring Interaction Responses for Embodied Video Generation
From intent to response. One initial frame and a task instruction; interaction roles and local dynamics are predicted inside the generator.
Who, where, and how?
Watch the interaction. Explore the responses behind its motion.
Task instruction
Video and dynamic responses play in sync.
Explore attention & interventions ↓Model the
response.
Embodied video models need to predict how objects respond to a robot’s actions. These responses are often local, brief, and critical to the task: a cloth corner lifts before the rest unfolds, and an indirectly affected object responds differently from the manipulation target.
ReactWorld structures this process into interaction roles, future response strength, and interval motion. A response feedback adapter uses these predictions to update object features relative to a zero-dynamics reference and to guide temporal attention with displacement.
Read the project abstract
ReactWorld anticipates structured interaction responses from the video generator’s intermediate features within each denoising forward pass. The interaction response field predictor describes participant roles, future motion response strength, motion support, and displacement. The response feedback adapter constructs dynamics-dependent object increments by subtracting a zero-dynamics reference, and turns predicted displacement into a positional bias for temporal attention. Experiments in the current manuscript report a RBench average of 77.51, together with EZS-Bench Domain and average scores of 86.98 and 81.87.
Same task. Different responses.
Compare the visible task progress, contact, and object response across methods.
The shared player restarts the displayed videos from the same timestamp. Shorter clips hold their final frame. Individual controls remain available. The first cell shows a still frame; the five method videos share one playback clock.
Anticipate. Update. Read.
Predicted response fields drive two sequential feedback operations within one DiT forward pass.
How responses guide generation
Inspect displacement-guided attention, then change the feedback and observe the generated motion.
最终 Fig.3 槽位：A（000 / 056 / 076）；56→60 · target_q01_x033_y008；04_LowerFadeLate；01_Whitespace。这里未用旧预览替代最终图。
What happens when displacement feedback changes?
Qualitative interventions apply a leftward or rightward offset to the internal displacement condition. The videos show how the generated motion changes with that condition.
From interaction to evidence.
Benchmark scores and controlled ablations complement the video comparisons.
Task-aware object responses
In the current RBench table, ReactWorld improves all five task-category scores over PhysisForcing. The complete table also reports results for four robot embodiments.
Response feedback contributes
Adding prediction heads alone is less effective than feeding the predicted responses back into generation. The full model also improves over the variant without displacement bias.
Explore component ablations
All variants share the same initialization, training data, and budget. Except for Video-FT, the variants use the same response supervision.
PSS and TAC are RBench interaction-quality metrics. Domain and Quality are EZS-Bench scores. Higher is better for all four.
Citation
The citation and release links will be added with the paper.