Trendoday

Fast and Exact Relative-Position Attention via Structure-Aware Kernels

Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typica

  • Upvotes11
–Rank now
#10Peak rank
3Days on list
294Score · 7d
Nov 9, 2026First seen
Rank · 7 days

About this paper

An exact tiled attention algorithm integrates additive relative-position terms into FlashAttention-style computation using offset-aware tiling, product reuse, and offset-based gradient sharding to speed up and reduce memory for long-context relative-position transformers.

Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms. Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as sequences scale. To address this bottleneck, we propose an exact attention algorithm that incorporates…

relative-position attention mechanismsRoPEtransformer architecturesattention scorescontent-position termsbias-position termsper-token rotationsquerieskeyspairwise offset-dependent computation
Upvotes
11
Authors
3
Published
Oct 31, 2026

Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.

Trending history

Fast and Exact Relative-Position Attention via Structure-Aware Kernels was first seen on Hugging Face · Daily papers on Nov 9, 2026. Since then it has appeared in 20 checks on 3 different days. Its best position was #10 of 30, reached on Nov 10, 21:28 UTC. It dropped off the list after Nov 11, 01:28 UTC.

Rank history

Position on Hugging Face · Daily papers at every check (lower is better).

#1#11#20#30 11:2820:5806:2815:5801:28 Nov 9, 11:28 UTC · #17Nov 9, 13:28 UTC · #15Nov 9, 15:28 UTC · #13Nov 9, 17:28 UTC · #12Nov 9, 19:28 UTC · #13Nov 9, 21:28 UTC · #13Nov 9, 23:28 UTC · #14Nov 10, 01:28 UTC · #12Nov 10, 03:28 UTC · #13Nov 10, 05:28 UTC · #12Nov 10, 07:28 UTC · #14Nov 10, 09:28 UTC · #15Nov 10, 11:28 UTC · #14Nov 10, 13:28 UTC · #16Nov 10, 15:28 UTC · #13Nov 10, 17:28 UTC · #12Nov 10, 19:28 UTC · #11Nov 10, 21:28 UTC · #10Nov 10, 23:28 UTC · #10Nov 11, 01:28 UTC · #10

Last 14 days

Best rank per day (UTC).

Nov 3today

Metric history

Platform metric as reported by Hugging Face.

071522 11:2820:5806:2815:5801:28 Nov 9, 11:28 UTC · 11Nov 9, 13:28 UTC · 13Nov 9, 15:28 UTC · 15Nov 9, 17:28 UTC · 17Nov 9, 19:28 UTC · 17Nov 9, 21:28 UTC · 19Nov 9, 23:28 UTC · 19Nov 10, 01:28 UTC · 19Nov 10, 03:28 UTC · 19Nov 10, 05:28 UTC · 19Nov 10, 07:28 UTC · 19Nov 10, 09:28 UTC · 21Nov 10, 11:28 UTC · 21Nov 10, 13:28 UTC · 21Nov 10, 15:28 UTC · 21Nov 10, 17:28 UTC · 21Nov 10, 19:28 UTC · 22Nov 10, 21:28 UTC · 22Nov 10, 23:28 UTC · 22Nov 11, 01:28 UTC · 22

Questions

Is Fast and Exact Relative-Position Attention via Structure-Aware Kernels trending on HF Daily Papers right now?

Not at the moment. It was last on the list on Nov 11, 01:28 UTC, after 3 days of trending.

What is the highest rank Fast and Exact Relative-Position Attention via Structure-Aware Kernels reached?

#10 on Hugging Face · Daily papers, on Nov 10, 21:28 UTC. Trendoday checks this list every 120 minutes.

How long has Fast and Exact Relative-Position Attention via Structure-Aware Kernels been trending?

It was first seen on Nov 9, 2026 and has been on the list on 3 days (20 checks) since.