About this paper
Two visual focusing losses align a Vision Transformer's multi-head attention with image regions, producing specialized spatial attention heads and improving dense prediction such as zero-shot segmentation without labels.
Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse…
self-supervised learningVision TransformerDINOvisual focusing loss functionsmulti-head attention mechanismspatial attention head specializationattention headsdense prediction taskszero-shot image segmentation
- Upvotes
- 4
- Authors
- 3
- Published
- Nov 9, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization was first seen on Hugging Face · Daily papers on Nov 10, 2026. Since then it has appeared in 13 checks on 2 different days. Its best position was #25 of 30, reached on Nov 10, 15:28 UTC. It dropped off the list after Nov 11, 17:28 UTC.