RTPurbo: 100-Step Sparse Attention for LLMs

In this AI Research Roundup episode, Alex discusses the paper: 'Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps' Long-context large language model inference is severely bottlenecked by the quadratic computational cost of full attention. To solve this, the authors introduce RTPurbo, a highly efficient sparse attention framework that converts full attention models into sparse ones in under one hundred training steps. RTPurbo identifies specialized retrieval heads, projects key-value representations into a lightweight 16-dimensional space, and uses dynamic top-p selection to optimize the active token budget. This methodology avoids expensive native sparse training while delivering an incredibly efficient, low-cost pipeline for long-context LLM decoding. Paper URL: https://arxiv.org/abs/2605.16928 #AI #MachineLearning #DeepLearning #LLM #SparseAttention #RTPurbo #Transformers

Видео RTPurbo: 100-Step Sparse Attention for LLMs канала AI Research Roundup