On-Policy Morphing into Hybrid Attention Models
Hybrid attention models combine full and linear attention to achieve a better quality--efficiency trade-off. Converting pretrained Transformers into such models requires identifying which layers shoul
- Upvotes6