3 ms·

40GB of SRAM. Not quite big enough to fit the big models like GPT3

by typon 6y ago

MuffinFlavored 6y ago
SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm https://computer.howstuffworks.com/question452.htm
etaioinshrdlu 6y ago
It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications. The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available. However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip. So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains. That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.
claytonius 6y ago
I don’t think it’s straightforward to do a head to head comparison. from: https://www.youtube.com/watch?v=yso2S2Svdlg https://www.youtube.com/watch?v=yso2S2Svdlg @ 25:14 James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?" Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."
- frogblast 6y ago
  Are there any good resources out there describing in practice how existing training workloads are distributed among GPUs? (using tensorflow, pytorch, or whatever else?). I'm curious how the problem effectively gets sliced.
  - singhrac 6y ago
    SOTA on the biggest language models (which is where effectively the largest models are) is here: https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/ https://www.microsoft.com/en-us/research/blog/zero-infinity-...

40GB of SRAM. Not quite big enough to fit the big models like GPT3

by typon 6y ago

40GB of SRAM. Not quite big enough to fit the big models like GPT3

SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm https://computer.howstuffworks.com/question452.htm

etaioinshrdlu 6y ago

It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications. The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available. However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip. So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains. That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.

claytonius 6y ago

I don’t think it’s straightforward to do a head to head comparison. from: https://www.youtube.com/watch?v=yso2S2Svdlg https://www.youtube.com/watch?v=yso2S2Svdlg @ 25:14 James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?" Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."

frogblast 6y ago
Are there any good resources out there describing in practice how existing training workloads are distributed among GPUs? (using tensorflow, pytorch, or whatever else?). I'm curious how the problem effectively gets sliced.
- singhrac 6y ago
  SOTA on the biggest language models (which is where effectively the largest models are) is here: https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/ https://www.microsoft.com/en-us/research/blog/zero-infinity-...