About this paper
Jointly selecting layers and modules in transformer foundation models yields better probing features than depth-only or final-layer approaches, and simple fixed strategies recover much of the gain.
Which representation should be probed in a transformer-based foundation model? Recent work shows that the standard practice of probing the final layer is often suboptimal, as intermediate layers can provide better features. Yet, representations are still largely studied along depth alone, treating each transformer block as an atomic unit. In this work, we analyze transformer representations layer by layer, module by module. Through more than 144,000 linear probing evaluations of vision foundation models (VFMs) and large language models (LLMs), we show that layer and module choices should be considered jointly: the best layer-module pair yields substantially larger gains over the…
transformer-based foundation modelslinear probingvision foundation modelslarge language modelsintermediate layerstransformer blocklayer-module pairfinal-layer residual streammodule performancefixed probing strategies
- Upvotes
- 3
- Authors
- 5
- Published
- Nov 8, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Probing Transformer Representations Layer by Layer, Module by Module was first seen on Hugging Face · Daily papers on Nov 13, 2026. Since then it has appeared in 1 check on 1 different day. Its best position was #30 of 30, reached on Nov 13, 11:28 UTC. It dropped off the list after Nov 13, 11:28 UTC.