Method

Compositional Simulation operates through a closed-loop real–sim–real pipeline that transforms a small amount of real-world data into large-scale, photorealistic training data for robot policies:

1. Real2Sim Alignment & Paired Data Collection: We strictly align the classical simulator with the real-world platform along three dimensions—background and object appearance, camera intrinsics/extrinsics, and object positions. Real-world trajectories are then replayed in simulation to form ($\mathcal{V}_{\text{sim}}$, $\mathcal{V}_{\text{real}}$, $\mathcal{A}$) tuples that share the same action sequence.

2. Compositional Dynamic Video Generation: A DiT-based neural simulator is trained to translate simulation videos into real-world video representations. It estimates scores conditioned jointly on Control Dynamics (actions) and Visual Dynamics (simulated observations), which are composed during sampling through Dynamic Guidance to ensure both photorealism and action consistency.

3. Rule-Based Large-Scale Data Synthesis: In RoboTwin we synthesize diverse trajectories via GPT-5-driven compositions of action primitives under semantic and physical constraints. The classical simulator produces clean ($\mathcal{V}_{\text{sim}}$, $\mathcal{A}$) pairs that span broad spatial and object variations.

4. Real-World Policy Deployment: Every simulated video is passed through the trained neural simulator, producing Pseudo-Real data that retains the original actions but looks real. Combined with a handful of real demonstrations, these data train robot policies (Diffusion Policy) that achieve high success rates and strong spatial/object generalization.

Tasks

We evaluate Compositional Simulation on five representative dual-arm manipulation tasks spanning diverse interaction patterns: Shake Bottle, Stack Blocks Two, Move Playing-Card Away, Place Mouse Pad, and Handover Bottle. To rigorously test generalization, we further introduce two scene variants—Cluttered (with distractors) and Colored Background (altered desktop textures)—as well as out-of-distribution object substitutions (e.g., Coca-Cola, Sprite, Nongfu Spring Oriental Leaf Tea bottles; red playing cards) that never appear in the real-world training demonstrations.

Results

We first evaluate the photorealism of Sim2Real neural simulation, then assess downstream policy performance under multiple data-mixture regimes. Our full pipeline (Ours-Full) achieves the best scores on all six video quality metrics, and policies trained with 10 Real + 200 Pseudo Real substantially outperform all baselines across every task and distribution.

Videos

We present real-world execution demos and Sim2Real neural simulation comparisons. In the comparison videos, simulated inputs and pseudo-real outputs are shown side by side, demonstrating how our neural simulator transforms classical simulation into photorealistic data while preserving action consistency.

Generalization

Compositional Simulation preserves real-world spatial statistics and object semantics, enabling Diffusion Policy to generalize well to unseen spatial layouts and object categories. Under new-object substitutions (Coca-Cola, Sprite, Nongfu Spring Oriental Leaf Tea, red playing cards), classical simulation brings essentially no improvement, whereas our Pseudo-Real data yields a substantial boost in success rate.

More Visualizations

We present additional qualitative comparisons spanning Real2Sim alignment quality and Sim2Real neural simulation, further confirming that our framework preserves action semantics while faithfully reproducing real-world appearance across diverse manipulation tasks.

Citation

If you find this project useful, welcome to cite us.

@article{qin2026comsim,
    title={Building Scalable Real-World Robot Data Generation via Compositional Simulation},
    author={Yiran Qin and Jiahua Ma and Li Kang and Wenzhan Li and Yihang Jiao and Xin Wen and Xiufeng Song and Heng Zhou and Jiwen Yu and Zhenfei Yin and Xihui Liu and Philip Torr and Yilun Du and Ruimao Zhang},
    journal={arXiv preprint arXiv:2504.xxxxx},
    year={2026}
}