Video-Coder is a code-driven video generation agent that writes and executes code to construct dynamic 3D scenes, trained with supervised fine-tuning and reinforcement learning using execution and video rewards to improve controllable video generation.
Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, we introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides structure and motion signals to downstream video generators. To support this paradigm, we design a dedicated data pipeline and curate two high-quality datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, which contain diverse…
video generationtemporal consistencyspatial structuremotion dynamicscontrollable scene compositioncode-driven video generation agentdynamic 3D sceneDraft Videodownstream video generatorssupervised fine-tuning
- Upvotes
- 6
- Authors
- 10
- Published
- Nov 6, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.