About this paper
Counterfactual supervision over pixel transformations enables language models to read motion dynamics from frozen predictive video encoders, yielding V-JEPA-LM models that outperform open video LLMs across benchmarks.
World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of…
world-model video encodersV-JEPAlanguage modellinear probefrozen V-JEPA featuresVLMdynamics subspacepredictive encodersframe-wise text-aligned encodersSigLIP
- Upvotes
- 13
- Authors
- 3
- Published
- Nov 13, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations was first seen on Hugging Face · Daily papers on Nov 16, 2026. Since then it has appeared in 15 checks on 2 different days. Its best position was #7 of 30, reached on Nov 17, 01:28 UTC. It dropped off the list after Nov 17, 13:28 UTC.