5 ms·

How much memory is on the chip, and what kind is it? Under what circumstances does the chip need to access external memory? What type of communication interfa

How much memory is on the chip, and what kind is it? Under what circumstances does the chip need to access external memory? What type of communication interfaces does this chip have? Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?

verdverm 6y ago
Tom's Hardware has some nice tables comparing the specs: https://www.tomshardware.com/news/cerebras-wafer-scale-engine-2-worlds-largest-chip-7nm-850000-cores https://www.tomshardware.com/news/cerebras-wafer-scale-engin... (more than the IEEE) This thing pulls 15-20kW of juice!
- teruakohatu 6y ago
  I did a double that when I realised that was Kilowatts not Watts. This chip uses more energy in an hour than the average household (in my country at least) does in a day. It may be a very large wafer but dissipating that heat is still very impressive.
  - kllrnohj 6y ago
    It sounds like a lot but it almost isn't? Like this is ~50x bigger than an Nvidia A100, and the A100 pulls up to 400w. 50 * 400 ~= 20kW. So in terms of thermal density it's in-line with existing GPUs. That said, I'd be fascinated to see the cooling solution. Is it just a massive copper heatsink & a boatload of airflow? Typical approaches of using heatpipes to expand the heatsink won't really work with something this big after all. Or is it a massive waterblock with multiple inlets/outlets so it can hit up a stack of radiators? How do they get even mounting pressure across that large of an area?
    - mvanaltvorst 6y ago
      There's an image on their website[1], pretty huge water pumps. "To solve the 70-year-old problem of wafer-scale, we needed not only to yield a big chip, but to invent new mechanisms for powering, packaging, and cooling it. The traditional method of powering a chip from its edges creates too much dissipation at a large chip’s center. To prevent this, CS-2’s innovative design delivers power perpendicularly to each core. To uniformly cool the entire wafer, pumps inside CS-2 move water across the back of the WSE-2, then into a heat exchanger where the internal water is cooled by either cold datacenter water or air." [1]: https://cerebras.net/product/ https://cerebras.net/product/
- meepmorp 6y ago
  > This thing pulls 15-20kW of juice! If you look at the wafer she’s holding at the top, it’s seemingly segmented into a 12x7 grid of roughly chip-sized rectangles. That’s 84 “CPUs” at 200-240 watts each, which is pretty well in line with discrete server CPUs. The amount of heat coming off this thing must be amazing, though.
typon 6y ago
40GB of SRAM. Not quite big enough to fit the big models like GPT3
- MuffinFlavored 6y ago
  SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm https://computer.howstuffworks.com/question452.htm
- etaioinshrdlu 6y ago
  It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications. The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available. However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip. So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains. That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.
- claytonius 6y ago
  I don’t think it’s straightforward to do a head to head comparison. from: https://www.youtube.com/watch?v=yso2S2Svdlg https://www.youtube.com/watch?v=yso2S2Svdlg @ 25:14 James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?" Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."
IshKebab 6y ago
It's 40 GB of SRAM. I doubt it supports external memory. > Also, if the chip is the size of a wafer, is it appropriate to call it a Chip? Good question. I think it is. I mean the word "chip" isn't really that well defined (is HBM one chip?), but given that they sell it as a single unit and you can't really cut it in half I think it's one chip.
- NaturalPhallacy 6y ago
  Looking at the picture, the thing is a platter. Not a chip. It's a really cool picture too.
  - IshKebab 6y ago
    It's a wafer, but that doesn't preclude it from being a chip. Where's you definition of chip coming from.
    - imtringued 6y ago
      Considering the chipped edges, it is not far off.

verdverm 6y ago

Tom's Hardware has some nice tables comparing the specs: https://www.tomshardware.com/news/cerebras-wafer-scale-engine-2-worlds-largest-chip-7nm-850000-cores https://www.tomshardware.com/news/cerebras-wafer-scale-engin... (more than the IEEE) This thing pulls 15-20kW of juice!

teruakohatu 6y ago
I did a double that when I realised that was Kilowatts not Watts. This chip uses more energy in an hour than the average household (in my country at least) does in a day. It may be a very large wafer but dissipating that heat is still very impressive.
- kllrnohj 6y ago
  It sounds like a lot but it almost isn't? Like this is ~50x bigger than an Nvidia A100, and the A100 pulls up to 400w. 50 * 400 ~= 20kW. So in terms of thermal density it's in-line with existing GPUs. That said, I'd be fascinated to see the cooling solution. Is it just a massive copper heatsink & a boatload of airflow? Typical approaches of using heatpipes to expand the heatsink won't really work with something this big after all. Or is it a massive waterblock with multiple inlets/outlets so it can hit up a stack of radiators? How do they get even mounting pressure across that large of an area?
  - mvanaltvorst 6y ago
    There's an image on their website[1], pretty huge water pumps. "To solve the 70-year-old problem of wafer-scale, we needed not only to yield a big chip, but to invent new mechanisms for powering, packaging, and cooling it. The traditional method of powering a chip from its edges creates too much dissipation at a large chip’s center. To prevent this, CS-2’s innovative design delivers power perpendicularly to each core. To uniformly cool the entire wafer, pumps inside CS-2 move water across the back of the WSE-2, then into a heat exchanger where the internal water is cooled by either cold datacenter water or air." [1]: https://cerebras.net/product/ https://cerebras.net/product/
meepmorp 6y ago
> This thing pulls 15-20kW of juice! If you look at the wafer she’s holding at the top, it’s seemingly segmented into a 12x7 grid of roughly chip-sized rectangles. That’s 84 “CPUs” at 200-240 watts each, which is pretty well in line with discrete server CPUs. The amount of heat coming off this thing must be amazing, though.

typon 6y ago

40GB of SRAM. Not quite big enough to fit the big models like GPT3

MuffinFlavored 6y ago
SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm https://computer.howstuffworks.com/question452.htm
etaioinshrdlu 6y ago
It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications. The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available. However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip. So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains. That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.
claytonius 6y ago
I don’t think it’s straightforward to do a head to head comparison. from: https://www.youtube.com/watch?v=yso2S2Svdlg https://www.youtube.com/watch?v=yso2S2Svdlg @ 25:14 James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?" Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."

IshKebab 6y ago

It's 40 GB of SRAM. I doubt it supports external memory. > Also, if the chip is the size of a wafer, is it appropriate to call it a Chip? Good question. I think it is. I mean the word "chip" isn't really that well defined (is HBM one chip?), but given that they sell it as a single unit and you can't really cut it in half I think it's one chip.

NaturalPhallacy 6y ago
Looking at the picture, the thing is a platter. Not a chip. It's a really cool picture too.
- IshKebab 6y ago
  It's a wafer, but that doesn't preclude it from being a chip. Where's you definition of chip coming from.
  - imtringued 6y ago
    Considering the chipped edges, it is not far off.