7 ms·

Native Sparse Attention

139 points by CalmStorm 1y ago

Was submitted as "DeepSeek won the best paper award at ACL 2025" Here is the awards page: https://cspaper.org/topic/116/record-breaking-acl-2025-crowns-four-game-changing-papers-on-speed-fairness-safety-for-next-gen-llms-and-beyond https://cspaper.org/topic/116/record-breaking-acl-2025-crown...

israrkhan 1y ago
Well deserved
- laughingcurve 1y ago
  Yea I agree. It’s sad to find so much of the comments are focused on reinventing reality and jingoism instead of scientific discussion on the merits and technicals. I’ll return tomorrow and hope for better comments.
CalmStorm 1y ago
For the first time, it introduced native sparse attention into the full training process, achieving up to 11× inference speedup while maintaining model performance.
sabaimran 1y ago
> Despite being sparse, NSA surpasses Full Attention baseline on average across general benchmarks, long-context tasks, and reasoning evaluation. Isn't it very notable that the latency improvement didn't have a performance loss? I'm not super familiar with all the technical aspects, but that seems like it should be one of the main focuses of the paper.
- ethan_smith 1y ago
  The performance maintenance (or even improvement) isn't surprising - sparse attention can reduce noise by focusing only on relevant tokens. Traditional full attention dilutes focus by attending to everything equally, while NSA's pruning approach mimics how humans selectively process information.
- laughingcurve 1y ago
  Yes that’s what makes it so interesting and novel you nailed it
gnabgib 1y ago
Title: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention The awards page for ACL seems to disagree with this editorialized title: https://2025.aclweb.org/program/awards/ https://2025.aclweb.org/program/awards/
- fourdnet 1y ago
  The ACL webpage has not been updated yet. Here are the announcement slides: https://cspaper.org/topic/116/record-breaking-acl-2025-crowns-four-game-changing-papers-on-speed-fairness-safety-for-next-gen-llms-and-beyond https://cspaper.org/topic/116/record-breaking-acl-2025-crown...
  - aspenmayer 1y ago
    The page that the person you’re replying to does have this so it may not be updated, or they were looking in the wrong place originally, or both: > Industry Track Awards > Best Paper > Speed Without Sacrifice: Fine-Tuning Language Models with Medusa and Knowledge Distillation in Travel Applications > Daniel Zagyva, Emmanouil Stergiadis, Laurens van der Maas, Aleksandra Dokic, Eran Fainman, Ilya Gusev, Moran Beladev Per TFA, the paper we’re looking for is this one: > Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention > Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng I’m not finding it by author on the page you linked but I think it’s this reference by title: > DeepSeek × PKU × UW — Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention I did find it on this page: https://2025.aclweb.org/program/main_papers/ https://2025.aclweb.org/program/main_papers/
pyuser583 1y ago
I'd say award for best title is a tie between: "Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems"; "Finding Needles in Images: Can Multi-modal LLMs Locate Fine Details?"; and "Steering off Course: Reliability Challenges in Steering Language Models."
noosphr 1y ago
Deep seek papers are a must to read for anyone who wants to understand how to make LLMs operate at hyper scale. All western labs hide their best results, or at most release summaries that are about as meaningful as the answers Cleo used to give on stack exchange: https://math.stackexchange.com/questions/562694/integral-int-11-frac1x-sqrt-frac1x1-x-ln-left-frac2-x22-x1/563063#563063 https://math.stackexchange.com/questions/562694/integral-int... I have a suspicion with how quiet all the major players got after the two weeks after deepseek R1 was released that they were reading and implementing everything in the papers that came with it as fast as humanly possible.
- Art9681 1y ago
  None of the major players have ever been quiet. DeepSeek enjoyed about a week or two's worth of press before its spotlight was stolent from the next great model. It never held the top spot, ever, mind you. So I don't understand why you think major players had to say anything about it, when the model was neither first, second or third in real world capability, and why they would have to say anything about it when DeepSeek service processes maybe an 1/8 of what OpenAI, Google or Claude in any given span of time. I applaud their open efforts. But being "altruistic" and being best are two different things.
  - sothatsit 1y ago
    DeepSeek's contributions to training efficiency improvements were as, if not more, important than the models themselves. A lot of the worry people had about DeepSeek was related to people questioning the moat of the big AI players, since DeepSeek was able to train a competitive model with so much less compute. Their innovations in training efficiency were almost guaranteed to have been heavily considered by the big AI labs. For example, Dario Amodei talks about the efficiency improvements being the real important contribution of DeepSeek V3 here: https://www.darioamodei.com/post/on-deepseek-and-export-controls https://www.darioamodei.com/post/on-deepseek-and-export-cont... > DeepSeek's team did this via some genuine and impressive innovations, mostly focused on engineering efficiency. There were particularly innovative improvements in the management of an aspect called the "Key-Value cache", and in enabling a method called "mixture of experts" to be pushed further than it had before.
ninjin 1y ago
Link to the published paper rather than the preprint (update link?): https://aclanthology.org/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126
visarga 1y ago
I am always skeptical of RNN approaches but this paper is just sparsifying the input, it is not compressing any size input to a fixed memory. I am hopeful maybe this is a big break. 11x inference speedup with no degradation from an algorithmic improvement. Is it really that good? almost too good to be true. Adoption in the next 6 months will tell us the truth.
tony_borlini 1y ago
DeepSeek and the Sparse Attention Revolution: How a Research Paper is Redefining AI Efficiency https://deep.liveblog365.com/en/index-en.html?post=50 https://deep.liveblog365.com/en/index-en.html?post=50

139 points by CalmStorm 1y ago

israrkhan 1y ago

Well deserved

laughingcurve 1y ago
Yea I agree. It’s sad to find so much of the comments are focused on reinventing reality and jingoism instead of scientific discussion on the merits and technicals. I’ll return tomorrow and hope for better comments.

CalmStorm 1y ago

For the first time, it introduced native sparse attention into the full training process, achieving up to 11× inference speedup while maintaining model performance.

sabaimran 1y ago

> Despite being sparse, NSA surpasses Full Attention baseline on average across general benchmarks, long-context tasks, and reasoning evaluation. Isn't it very notable that the latency improvement didn't have a performance loss? I'm not super familiar with all the technical aspects, but that seems like it should be one of the main focuses of the paper.

ethan_smith 1y ago
The performance maintenance (or even improvement) isn't surprising - sparse attention can reduce noise by focusing only on relevant tokens. Traditional full attention dilutes focus by attending to everything equally, while NSA's pruning approach mimics how humans selectively process information.
laughingcurve 1y ago
Yes that’s what makes it so interesting and novel you nailed it

gnabgib 1y ago

Title: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention The awards page for ACL seems to disagree with this editorialized title: https://2025.aclweb.org/program/awards/ https://2025.aclweb.org/program/awards/

fourdnet 1y ago
The ACL webpage has not been updated yet. Here are the announcement slides: https://cspaper.org/topic/116/record-breaking-acl-2025-crowns-four-game-changing-papers-on-speed-fairness-safety-for-next-gen-llms-and-beyond https://cspaper.org/topic/116/record-breaking-acl-2025-crown...
- aspenmayer 1y ago
  The page that the person you’re replying to does have this so it may not be updated, or they were looking in the wrong place originally, or both: > Industry Track Awards > Best Paper > Speed Without Sacrifice: Fine-Tuning Language Models with Medusa and Knowledge Distillation in Travel Applications > Daniel Zagyva, Emmanouil Stergiadis, Laurens van der Maas, Aleksandra Dokic, Eran Fainman, Ilya Gusev, Moran Beladev Per TFA, the paper we’re looking for is this one: > Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention > Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng I’m not finding it by author on the page you linked but I think it’s this reference by title: > DeepSeek × PKU × UW — Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention I did find it on this page: https://2025.aclweb.org/program/main_papers/ https://2025.aclweb.org/program/main_papers/

pyuser583 1y ago

I'd say award for best title is a tie between: "Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems"; "Finding Needles in Images: Can Multi-modal LLMs Locate Fine Details?"; and "Steering off Course: Reliability Challenges in Steering Language Models."

noosphr 1y ago

Deep seek papers are a must to read for anyone who wants to understand how to make LLMs operate at hyper scale. All western labs hide their best results, or at most release summaries that are about as meaningful as the answers Cleo used to give on stack exchange: https://math.stackexchange.com/questions/562694/integral-int-11-frac1x-sqrt-frac1x1-x-ln-left-frac2-x22-x1/563063#563063 https://math.stackexchange.com/questions/562694/integral-int... I have a suspicion with how quiet all the major players got after the two weeks after deepseek R1 was released that they were reading and implementing everything in the papers that came with it as fast as humanly possible.

Art9681 1y ago
None of the major players have ever been quiet. DeepSeek enjoyed about a week or two's worth of press before its spotlight was stolent from the next great model. It never held the top spot, ever, mind you. So I don't understand why you think major players had to say anything about it, when the model was neither first, second or third in real world capability, and why they would have to say anything about it when DeepSeek service processes maybe an 1/8 of what OpenAI, Google or Claude in any given span of time. I applaud their open efforts. But being "altruistic" and being best are two different things.
- sothatsit 1y ago
  DeepSeek's contributions to training efficiency improvements were as, if not more, important than the models themselves. A lot of the worry people had about DeepSeek was related to people questioning the moat of the big AI players, since DeepSeek was able to train a competitive model with so much less compute. Their innovations in training efficiency were almost guaranteed to have been heavily considered by the big AI labs. For example, Dario Amodei talks about the efficiency improvements being the real important contribution of DeepSeek V3 here: https://www.darioamodei.com/post/on-deepseek-and-export-controls https://www.darioamodei.com/post/on-deepseek-and-export-cont... > DeepSeek's team did this via some genuine and impressive innovations, mostly focused on engineering efficiency. There were particularly innovative improvements in the management of an aspect called the "Key-Value cache", and in enabling a method called "mixture of experts" to be pushed further than it had before.

ninjin 1y ago

Link to the published paper rather than the preprint (update link?): https://aclanthology.org/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126

visarga 1y ago

I am always skeptical of RNN approaches but this paper is just sparsifying the input, it is not compressing any size input to a fixed memory. I am hopeful maybe this is a big break. 11x inference speedup with no degradation from an algorithmic improvement. Is it really that good? almost too good to be true. Adoption in the next 6 months will tell us the truth.

tony_borlini 1y ago

DeepSeek and the Sparse Attention Revolution: How a Research Paper is Redefining AI Efficiency https://deep.liveblog365.com/en/index-en.html?post=50 https://deep.liveblog365.com/en/index-en.html?post=50