4 ms·

> With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. That's not how you would do it with Cerebra

> With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would get 16k GB200 aka ~600M$, obviously way off. Cerebras uses a different architecture than Nvidia, where the HBM is not directly packaged with the chips, this is handled by a different system so you can scale memory and compute separately. Specifically you can use something like MemoryX to act as your HBM which will be high speed interconnected to the chips SRAM, see [1]. I'm not at all an expert in Cerebras, but IIRC you can connect up to like 2PB of memory to a single Cererbas chip, so almost 1000x the FP16 model. [1]: https://www.cerebras.ai/blog/announcing-the-cerebras-architecture-for-extreme-scale-ai https://www.cerebras.ai/blog/announcing-the-cerebras-archite...

throwawaymaths 1y ago
what are the bandwidth/latency of memoryX? those are the key parameters for inference
- Zenst 1y ago
  Well MemoryX compared to H100 HBM3 the key details are that MemoryX has lower latency, but also far lower bandwidth. However the memory on Cerebras is scales a lot more over NVidia. You need a cluster of H100's to create a model, as only way to scale the memory, Cerbras is more suited to that aspect, Nvidia do their scaling in tooling, with Cerbras doing theirs in design via there silicon approach. That's my take on it all, not many apples to oranges comparisons to work from on these two system for even rolling down the same slope.
  - perfobotto 1y ago
    No way an offchip HBM has same or better bandwidth then onchip
    - 0xCMP 1y ago
      > MemoryX has lower latency, but also far lower bandwidth
  - deleted 1y ago
    [deleted]
imtringued 1y ago
Yeah sure, but if you do that you are heavily dropping the token/s for a single user. The only way to recover from that is continuous batching. This could still be interesting if the KV caches of all users fit in SRAM though.
- sailingparrot 1y ago
  > but if you do that you are heavily dropping the token/s for a single user. I don’t follow what you are saying and what “that” is specifically. Assuming it’s referencing using HBM and not just SRAM, this is not optional on a GPU, SRAM is many order of magnitudes too small. Data is constantly flowing between HBM and SRAM by design, and to get data in/out of your GPU you have to go through HBM first, you can’t skip that. And while it is quite massive on a Cerebras system it is also still too small for very large models.
aurareturn 1y ago
That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would get 16k GB200 aka ~600M$, obviously way off. Yes but Cerebras achieves its speed by using SRAM.
- sailingparrot 1y ago
  There is no way not to use SRAM on a GPU/Cerebras/most accelerators. This is where the cores fetch the data. But that doesn’t mean you are only using SRAM, that would be impractical. Just like using a CPU just by storing stuff in the L3 cache and never going to the RAM. Unless I am missing something from the original link, I don’t know how you got to the conclusion that they only used SRAM.
  - aurareturn 1y ago
    I know Groq chips load the entire model into SRAM. That's why it can be so fast. So if Cerebras uses HBM to store the model but stream weights into SRAM, I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. The whole point of having a wafer chip is that you limit the need to reach out to external parts for memory since that's the slow part.
    - sailingparrot 1y ago
      > I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. I don’t think you can look at those things binarily. 44GB of SRAM is still a massive amount. You don’t need infinite SRAM to get better performances. There is a reason NVidia is increasing the L2 cache size with every generation rather than just sticking with 32MB if it really changed nothing to have a bit more. The more SRAM you have the more you are able to mask communication behind computation. You can imagine with 44GB being able to load the weights of layer N+1 into SRAM while computing layer N, thereby entirely negating the penalty of going to HBM (same idea as FSDP).

throwawaymaths 1y ago

what are the bandwidth/latency of memoryX? those are the key parameters for inference

Zenst 1y ago
Well MemoryX compared to H100 HBM3 the key details are that MemoryX has lower latency, but also far lower bandwidth. However the memory on Cerebras is scales a lot more over NVidia. You need a cluster of H100's to create a model, as only way to scale the memory, Cerbras is more suited to that aspect, Nvidia do their scaling in tooling, with Cerbras doing theirs in design via there silicon approach. That's my take on it all, not many apples to oranges comparisons to work from on these two system for even rolling down the same slope.
- perfobotto 1y ago
  No way an offchip HBM has same or better bandwidth then onchip
  - 0xCMP 1y ago
    > MemoryX has lower latency, but also far lower bandwidth
- deleted 1y ago
  [deleted]

imtringued 1y ago

Yeah sure, but if you do that you are heavily dropping the token/s for a single user. The only way to recover from that is continuous batching. This could still be interesting if the KV caches of all users fit in SRAM though.

sailingparrot 1y ago
> but if you do that you are heavily dropping the token/s for a single user. I don’t follow what you are saying and what “that” is specifically. Assuming it’s referencing using HBM and not just SRAM, this is not optional on a GPU, SRAM is many order of magnitudes too small. Data is constantly flowing between HBM and SRAM by design, and to get data in/out of your GPU you have to go through HBM first, you can’t skip that. And while it is quite massive on a Cerebras system it is also still too small for very large models.

aurareturn 1y ago

That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would get 16k GB200 aka ~600M$, obviously way off. Yes but Cerebras achieves its speed by using SRAM.

sailingparrot 1y ago
There is no way not to use SRAM on a GPU/Cerebras/most accelerators. This is where the cores fetch the data. But that doesn’t mean you are only using SRAM, that would be impractical. Just like using a CPU just by storing stuff in the L3 cache and never going to the RAM. Unless I am missing something from the original link, I don’t know how you got to the conclusion that they only used SRAM.
- aurareturn 1y ago
  I know Groq chips load the entire model into SRAM. That's why it can be so fast. So if Cerebras uses HBM to store the model but stream weights into SRAM, I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. The whole point of having a wafer chip is that you limit the need to reach out to external parts for memory since that's the slow part.
  - sailingparrot 1y ago
    > I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. I don’t think you can look at those things binarily. 44GB of SRAM is still a massive amount. You don’t need infinite SRAM to get better performances. There is a reason NVidia is increasing the L2 cache size with every generation rather than just sticking with 32MB if it really changed nothing to have a bit more. The more SRAM you have the more you are able to mask communication behind computation. You can imagine with 44GB being able to load the weights of layer N+1 into SRAM while computing layer N, thereby entirely negating the penalty of going to HBM (same idea as FSDP).