About this paper
An exact tiled attention algorithm integrates additive relative-position terms into FlashAttention-style computation using offset-aware tiling, product reuse, and offset-based gradient sharding to speed up and reduce memory for long-context relative-position transformers.
Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms. Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as sequences scale. To address this bottleneck, we propose an exact attention algorithm that incorporates…
relative-position attention mechanismsRoPEtransformer architecturesattention scorescontent-position termsbias-position termsper-token rotationsquerieskeyspairwise offset-dependent computation
- Upvotes
- 11
- Authors
- 3
- Published
- Oct 31, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Fast and Exact Relative-Position Attention via Structure-Aware Kernels was first seen on Hugging Face · Daily papers on Nov 9, 2026. Since then it has appeared in 20 checks on 3 different days. Its best position was #10 of 30, reached on Nov 10, 21:28 UTC. It dropped off the list after Nov 11, 01:28 UTC.