About this paper
Interactive Visual Gym enables closed-loop evaluation and training of visual interaction models by jointly simulating task environments and human users, exposing limitations in state-of-the-art VLMs and improving performance through closed-loop training.
Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation…
visual interaction modelsclosed-loop evaluationclosed-loop trainingjoint simulationexecutable task environmentshuman-aligned action spacespersona-conditioned user simulationout-of-plan behaviorsIVG-Benchvision-language models (VLMs)
- Upvotes
- 13
- Authors
- 14
- Published
- Nov 5, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds was first seen on Hugging Face · Daily papers on Nov 12, 2026. Since then it has appeared in 41 checks on 5 different days. Its best position was #14 of 30, reached on Nov 13, 07:28 UTC. It dropped off the list after Nov 16, 01:28 UTC.