4 ms·

Like this? https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT

adityashankar 11mo ago
yes!, thanks for the link!
moffkalast 11mo ago
GGUF when? /s
deepdarkforest 11mo ago
> which does pose interesting questions over nvidia's throne... > Zebra-Llama is a family of hybrid large language models (LLMs) proposed by AMD that... Hmmm
jychang 11mo ago
Or like this: https://api-docs.deepseek.com/news/news251201 https://api-docs.deepseek.com/news/news251201 I don't know what's so special about this paper. - They claim to use MLA to reduce KV cache by 90%. Yeah, Deepseek invented that for Deepseek V2 (and also V3 and Deepseek R1 etc) - They claim to use a hybrid linear attention architecture. So does Deepseek V3.2 and that was weeks ago. Or Granite 4, if you want to go even further back. Or Kimi Linear. Or Qwen3-Next. - They claimed to save a lot of money not doing a full pre-train run for millions of dollars. Well, so did Deepseek V3.2... Deepseek hasn't done a full $5.6mil full pretraining run since Deepseek V3 in 2024. Deepseek R1 is just a $294k post train on top of the expensive V3 pretrain run. Deepseek V3.2 is just a hybrid linear attention post-train run - i don't know the exact price, but it's probably just a few hundred thousand dollars as well. Hell, GPT-5, o3, o4-mini, and gpt-4o are all post-trains on top of the same expensive pre-train run for gpt-4o in 2024. That's why they all have the same information cutoff date. I don't really see anything new or interesting in this paper that isn't already something Deepseek V3.2 has already sort of done (just on a bigger scale). Not exactly the same, but is there anything amazingly new that's not in Deepseek V3.2?
- T-A 11mo ago
  From your link: DeepSeek-V3.2 Release 2025/12/01 From Zebra-Llama's arXiv page: Submitted on 22 May 2025
  - jychang 11mo ago
    That's still behind the times. Even the ancient dinosaur IBM had released a Mamba model [1] before this paper was even put out. > Granite-4.0-Tiny-Base-Preview is a 7B-parameter hybrid mixture-of-experts (MoE) language model featuring a 128k token context window. The architecture leverages Mamba-2, superimposed with a softmax attention for enhanced expressiveness, with no positional encoding for better length generalization. Release Date: May 2nd, 2025 I mean, good for them for shipping, I guess. But seriously, I expect any postgrad student to be able to train a similar model with some rented GPUs. They literally teach MLA to undergrads in the basic LLM class at Stanford [2] so this isn't some exactly some obscure never-heard-of concept. [1] https://huggingface.co/ibm-granite/granite-4.0-tiny-base-preview https://huggingface.co/ibm-granite/granite-4.0-tiny-base-pre... [2] https://youtu.be/Q5baLehv5So?t=6075 https://youtu.be/Q5baLehv5So?t=6075
  - Palmik 11mo ago
    DeepSeek's MLA paper was published in 2024: https://arxiv.org/abs/2405.04434 https://arxiv.org/abs/2405.04434 DeepSeek's Sparse Attention paper was published in February: https://arxiv.org/abs/2502.11089 https://arxiv.org/abs/2502.11089 DeepSeek 3.2 Exp (combining MLA and DSA) was released in September. You also had several other Chinese hybrid models, like Qwen3 Next and Minimax M1.
- nickpsecurity 11mo ago
  "Deepseek hasn't done a full $5.6mil full " Don't forget the billion dollars or so of GPU's they had access to that they left out of that accounting. Also, the R&D cost of the Meta model they originally used. Then, they added $5.6 million on top of that.
- credit_guy 11mo ago
  Here's what's important about this paper. It is written by AMD researchers. It shows AMD is investing in AI research. Is this the same level of achievement as DeepSeek 3.2. Most likely not. Do they have novel ideas? Difficult to say, there are hundreds of new ideas being tried in this space. Is this worthless? Most certainly not. In order to make progress in this domain (as in any other), you first need to get your feet wet. You need to play with the various components, and see how they fit together. The idea in this paper is that you can combine somehow SSMs (like Mamba) and LLMs (like LLama). The examples they give are absolute toys compared to DeepSeek 3.2 (the largest is 8 billion parameters, while DeepSeek 3.2 has 671 billion parameters). The comparison you are trying to make simply does not apply. The good news for all of us is that AMD is working in this space.

adityashankar 11mo ago

yes!, thanks for the link!

moffkalast 11mo ago

GGUF when? /s

deepdarkforest 11mo ago

> which does pose interesting questions over nvidia's throne... > Zebra-Llama is a family of hybrid large language models (LLMs) proposed by AMD that... Hmmm

jychang 11mo ago

Or like this: https://api-docs.deepseek.com/news/news251201 https://api-docs.deepseek.com/news/news251201 I don't know what's so special about this paper. - They claim to use MLA to reduce KV cache by 90%. Yeah, Deepseek invented that for Deepseek V2 (and also V3 and Deepseek R1 etc) - They claim to use a hybrid linear attention architecture. So does Deepseek V3.2 and that was weeks ago. Or Granite 4, if you want to go even further back. Or Kimi Linear. Or Qwen3-Next. - They claimed to save a lot of money not doing a full pre-train run for millions of dollars. Well, so did Deepseek V3.2... Deepseek hasn't done a full $5.6mil full pretraining run since Deepseek V3 in 2024. Deepseek R1 is just a $294k post train on top of the expensive V3 pretrain run. Deepseek V3.2 is just a hybrid linear attention post-train run - i don't know the exact price, but it's probably just a few hundred thousand dollars as well. Hell, GPT-5, o3, o4-mini, and gpt-4o are all post-trains on top of the same expensive pre-train run for gpt-4o in 2024. That's why they all have the same information cutoff date. I don't really see anything new or interesting in this paper that isn't already something Deepseek V3.2 has already sort of done (just on a bigger scale). Not exactly the same, but is there anything amazingly new that's not in Deepseek V3.2?

T-A 11mo ago
From your link: DeepSeek-V3.2 Release 2025/12/01 From Zebra-Llama's arXiv page: Submitted on 22 May 2025
- jychang 11mo ago
  That's still behind the times. Even the ancient dinosaur IBM had released a Mamba model [1] before this paper was even put out. > Granite-4.0-Tiny-Base-Preview is a 7B-parameter hybrid mixture-of-experts (MoE) language model featuring a 128k token context window. The architecture leverages Mamba-2, superimposed with a softmax attention for enhanced expressiveness, with no positional encoding for better length generalization. Release Date: May 2nd, 2025 I mean, good for them for shipping, I guess. But seriously, I expect any postgrad student to be able to train a similar model with some rented GPUs. They literally teach MLA to undergrads in the basic LLM class at Stanford [2] so this isn't some exactly some obscure never-heard-of concept. [1] https://huggingface.co/ibm-granite/granite-4.0-tiny-base-preview https://huggingface.co/ibm-granite/granite-4.0-tiny-base-pre... [2] https://youtu.be/Q5baLehv5So?t=6075 https://youtu.be/Q5baLehv5So?t=6075
- Palmik 11mo ago
  DeepSeek's MLA paper was published in 2024: https://arxiv.org/abs/2405.04434 https://arxiv.org/abs/2405.04434 DeepSeek's Sparse Attention paper was published in February: https://arxiv.org/abs/2502.11089 https://arxiv.org/abs/2502.11089 DeepSeek 3.2 Exp (combining MLA and DSA) was released in September. You also had several other Chinese hybrid models, like Qwen3 Next and Minimax M1.
nickpsecurity 11mo ago
"Deepseek hasn't done a full $5.6mil full " Don't forget the billion dollars or so of GPU's they had access to that they left out of that accounting. Also, the R&D cost of the Meta model they originally used. Then, they added $5.6 million on top of that.
credit_guy 11mo ago
Here's what's important about this paper. It is written by AMD researchers. It shows AMD is investing in AI research. Is this the same level of achievement as DeepSeek 3.2. Most likely not. Do they have novel ideas? Difficult to say, there are hundreds of new ideas being tried in this space. Is this worthless? Most certainly not. In order to make progress in this domain (as in any other), you first need to get your feet wet. You need to play with the various components, and see how they fit together. The idea in this paper is that you can combine somehow SSMs (like Mamba) and LLMs (like LLama). The examples they give are absolute toys compared to DeepSeek 3.2 (the largest is 8 billion parameters, while DeepSeek 3.2 has 671 billion parameters). The comparison you are trying to make simply does not apply. The good news for all of us is that AMD is working in this space.