6 ms·

I believe pricing was mid 6 figures per machine. They're also like 8U and water cooled I believe. I doubt it would be possible to deploy one outside of a fairly

I believe pricing was mid 6 figures per machine. They're also like 8U and water cooled I believe. I doubt it would be possible to deploy one outside of a fairly top tier colo facility where they have the ability to support water cooling. Also imagine learning a new CUDA but that is designed for another completely different compute model.

initplus 2y ago
Yeah you can see the cooling requirements by looking at their product images. https://cerebras.ai/wp-content/uploads/2021/04/Cerebras_Product_CS1-1.png https://cerebras.ai/wp-content/uploads/2021/04/Cerebras_Prod... Thing is nearly all cooling. And look at the diameter on the water cooling pipes. Airflow guides on the fans are solid steel. Apparently the chip itself measures 21.5cm^2. Insane.
bboygravity 2y ago
That means it'll be close to affordable in 3 to 5 years if we follow the curve we've been on for the past decades.
- schoen 2y ago
  How have power and cooling been doing with respect to chip improvements? Have power requirements per operation been coming down rapidly, as other features have improved? My recollection from PC CPUs is that we've gotten many more operations per second, and many more operations per second per dollar, but that the power and corresponding cooling requirements for the CPUs have tended to go up as well. I don't really know what power per operation has looked like there. (I guess it's clearly improved, though, because it seems like the power consumption of a desktop PC has only increased by a single order of magnitude, while the computational capacity has increased by more than that.) A reason that I wonder about this in this context is that people are saying that the power and cooling requirements for these devices are currently enormous (by individual or hobbyist standards, not by data center standards!). If we imagine a Moore's Law-style improvement where the hardware itself becomes 1/10 or 1/100 of its current price, would we expect the overall power consumption to be similarly reduced, or to remain closer to its current levels?
  - chaxor 2y ago
    Mooers law in the consumer space seems to be pretty much asymptoting now, as indicated by Apple's amazing Macbooks with an astounding 8GB of RAM. Data center compute is arguable, as it tends to be catered to some niche, making it confusing (cerebras as an example vs GPU datacenters vs more standard HPC). Also Clusters and even GPUs don't really fit in to Mooers law as originally framed.
    - saagarjha 2y ago
      Apple doesn’t sell those anymore.
      - chaxor 2y ago
        Aw man, are they selling only 4GB ones now? More seriously, even 16GB was essentially the 'norm' in consumer PCs about 15 years ago.
- dheera 2y ago
  It will also mean 405B models will be uninteresting in 3 to 5 years if we follow the curve we've been on for the past decades.
  - int_19h 2y ago
    I don't think they'll be uninteresting. They won't be cutting-edge anymore, sure, but much of the more practical applications of AI that we see today don't run on today's cutting-edge models, either. We're always going to have a certain compute budget, and if a smaller model does the job fine, why wouldn't you use it, and use the rest for something else (or use all of it to run the smaller model faster).
- dgfl 2y ago
  Not really. These are wafer-scale chips, which (as far as I'm aware) were first introduced by Cerebras. Cost reduction for cutting-edge products in the semiconductor industry has historically been driven by 1) reducing transistor size (by following the Dennard scaling laws), and 2) a variety of techniques (e.g. high-k dielectrics and strained silicon, or FinFETs and now GAAFETs) to improve transistor performance further. These techniques added more steps during manufacturing, but they were inexpensive enough that they allowed to reduce $/transistor still. In the last few years, we've had to pull off ever more expensive tricks which stopped the $/transistor progress. This is why the phrase "Moore's law is dead" has been circulating for a while. In any case, higher performance transistors means that you can get the same functionality for less power and a smaller area, meaning that iso-functionality chips are cheaper to build in bulk. This is especially true for older nodes, e.g. look at the absurdly low price of most microcontrollers. On the other hand, $/wafer is mostly a volume-related metric based on less scalable technology and more conventional manufacturing (relatively speaking). Cerebra's innovation was in making a wafer-scale chip possible, which is conventionally hard due to unavoidable manufacturing defects. But crucially, such a product (by definition) cannot scale like any other circuit produced so far. It may for sure drop in price in the future, especially once it gets obsolete. But I don't expect it to ever reach consumer level prices.
  - adrian_b 2y ago
    Wafer-scale chips have been attempted for many decades, but none of the previous attempts before Cerebras has resulted in a successful commercial product. The main reason why Cerebras has succeeded and the previous attempts have failed is not technical, but the existence of market demand. Before ML/AI training and inference, there has been no application where wafer-scale chips could provide enough additional performance to make their high cost worthwhile.
    - ryao 2y ago
      Cerebras has a patent on the technique used to etch across scribe lines. Is there any prior work that would invalidate that patent? By the way, I am a software developer, so you will not see me challenging their patent. I am just curious.
trsohmers 2y ago
Based on their S1 filing and public statements, the average cost per WSE system for their (~90% of their total revenue) largest customer is ~$1.36M, and I’ve heard “retail” pricing of $2.5M per system. They are also 15U and due to power and additional support equipment take up an entire rack. The other thing people don’t seem to be getting in this thread that just to hold the weights for 405B at FP16 requires 19 of their systems since it is SRAM only… rounding up to 20 to account for program code + KV cache for the user context would mean 20 systems/racks, so well over $20M. The full rack (including support equipment) also consumes 23kW, so we are talking nearly half a megawatt and ~$30M for them to be getting this performance on Llama 405B
- danpalmer 2y ago
  Thank you, far better answer than mine! Those are indeed wild numbers, although interestingly "only" 23kw, I'd expect the same level of compute in GPUs to be quite a lot more than that, or at least higher power density.
  - YetAnotherNick 2y ago
    You get ~400TFLOP/s in H100 for 350W. You need (2 * token/s * param count) FLOP/s. For 405b, 969tok/s you just need 784 TFLOP/s which is just 2 H100s. The limiting factor with GPU for inference is memory bandwidth. For 969 tok/s in int8, you need 392 TB/s memory bandwidth or 200 H100s.
    - latchkey 2y ago
      Memory bandwidth and memory size. Along with power/cooling density. Hence why you see AMD's MI325x coming out with 256GB HBM3e, but it is the same FLOPs as a 300x. 6TB/s too, which outperforms H200's, by a lot. You can see the direction AMD is going with this... https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html https://www.amd.com/en/products/accelerators/instinct/mi300/...
    - Const-me 2y ago
      > For 969 tok/s in int8, you need 392 TB/s memory bandwidth I think that math is only valid for batch size = 1. When these 969 tokens/second come from multiple sessions of the same batch, loaded model tensor elements are reused to compute many tokens for the entire batch. With large enough batches, you can even saturate compute throughput of the GPU instead of bottlenecking on memory bandwidth.
szundi 2y ago
Parent wishes 70b not 405b though
wkat4242 2y ago
Yeah but what is in a 4090 is also comparable to a whole rack of servers a decade ago. The tech will get smaller.

initplus 2y ago

Yeah you can see the cooling requirements by looking at their product images. https://cerebras.ai/wp-content/uploads/2021/04/Cerebras_Product_CS1-1.png https://cerebras.ai/wp-content/uploads/2021/04/Cerebras_Prod... Thing is nearly all cooling. And look at the diameter on the water cooling pipes. Airflow guides on the fans are solid steel. Apparently the chip itself measures 21.5cm^2. Insane.

bboygravity 2y ago

That means it'll be close to affordable in 3 to 5 years if we follow the curve we've been on for the past decades.

schoen 2y ago
How have power and cooling been doing with respect to chip improvements? Have power requirements per operation been coming down rapidly, as other features have improved? My recollection from PC CPUs is that we've gotten many more operations per second, and many more operations per second per dollar, but that the power and corresponding cooling requirements for the CPUs have tended to go up as well. I don't really know what power per operation has looked like there. (I guess it's clearly improved, though, because it seems like the power consumption of a desktop PC has only increased by a single order of magnitude, while the computational capacity has increased by more than that.) A reason that I wonder about this in this context is that people are saying that the power and cooling requirements for these devices are currently enormous (by individual or hobbyist standards, not by data center standards!). If we imagine a Moore's Law-style improvement where the hardware itself becomes 1/10 or 1/100 of its current price, would we expect the overall power consumption to be similarly reduced, or to remain closer to its current levels?
- chaxor 2y ago
  Mooers law in the consumer space seems to be pretty much asymptoting now, as indicated by Apple's amazing Macbooks with an astounding 8GB of RAM. Data center compute is arguable, as it tends to be catered to some niche, making it confusing (cerebras as an example vs GPU datacenters vs more standard HPC). Also Clusters and even GPUs don't really fit in to Mooers law as originally framed.
  - saagarjha 2y ago
    Apple doesn’t sell those anymore.
    - chaxor 2y ago
      Aw man, are they selling only 4GB ones now? More seriously, even 16GB was essentially the 'norm' in consumer PCs about 15 years ago.
dheera 2y ago
It will also mean 405B models will be uninteresting in 3 to 5 years if we follow the curve we've been on for the past decades.
- int_19h 2y ago
  I don't think they'll be uninteresting. They won't be cutting-edge anymore, sure, but much of the more practical applications of AI that we see today don't run on today's cutting-edge models, either. We're always going to have a certain compute budget, and if a smaller model does the job fine, why wouldn't you use it, and use the rest for something else (or use all of it to run the smaller model faster).
dgfl 2y ago
Not really. These are wafer-scale chips, which (as far as I'm aware) were first introduced by Cerebras. Cost reduction for cutting-edge products in the semiconductor industry has historically been driven by 1) reducing transistor size (by following the Dennard scaling laws), and 2) a variety of techniques (e.g. high-k dielectrics and strained silicon, or FinFETs and now GAAFETs) to improve transistor performance further. These techniques added more steps during manufacturing, but they were inexpensive enough that they allowed to reduce $/transistor still. In the last few years, we've had to pull off ever more expensive tricks which stopped the $/transistor progress. This is why the phrase "Moore's law is dead" has been circulating for a while. In any case, higher performance transistors means that you can get the same functionality for less power and a smaller area, meaning that iso-functionality chips are cheaper to build in bulk. This is especially true for older nodes, e.g. look at the absurdly low price of most microcontrollers. On the other hand, $/wafer is mostly a volume-related metric based on less scalable technology and more conventional manufacturing (relatively speaking). Cerebra's innovation was in making a wafer-scale chip possible, which is conventionally hard due to unavoidable manufacturing defects. But crucially, such a product (by definition) cannot scale like any other circuit produced so far. It may for sure drop in price in the future, especially once it gets obsolete. But I don't expect it to ever reach consumer level prices.
- adrian_b 2y ago
  Wafer-scale chips have been attempted for many decades, but none of the previous attempts before Cerebras has resulted in a successful commercial product. The main reason why Cerebras has succeeded and the previous attempts have failed is not technical, but the existence of market demand. Before ML/AI training and inference, there has been no application where wafer-scale chips could provide enough additional performance to make their high cost worthwhile.
  - ryao 2y ago
    Cerebras has a patent on the technique used to etch across scribe lines. Is there any prior work that would invalidate that patent? By the way, I am a software developer, so you will not see me challenging their patent. I am just curious.

trsohmers 2y ago

Based on their S1 filing and public statements, the average cost per WSE system for their (~90% of their total revenue) largest customer is ~$1.36M, and I’ve heard “retail” pricing of $2.5M per system. They are also 15U and due to power and additional support equipment take up an entire rack. The other thing people don’t seem to be getting in this thread that just to hold the weights for 405B at FP16 requires 19 of their systems since it is SRAM only… rounding up to 20 to account for program code + KV cache for the user context would mean 20 systems/racks, so well over $20M. The full rack (including support equipment) also consumes 23kW, so we are talking nearly half a megawatt and ~$30M for them to be getting this performance on Llama 405B

danpalmer 2y ago
Thank you, far better answer than mine! Those are indeed wild numbers, although interestingly "only" 23kw, I'd expect the same level of compute in GPUs to be quite a lot more than that, or at least higher power density.
- YetAnotherNick 2y ago
  You get ~400TFLOP/s in H100 for 350W. You need (2 * token/s * param count) FLOP/s. For 405b, 969tok/s you just need 784 TFLOP/s which is just 2 H100s. The limiting factor with GPU for inference is memory bandwidth. For 969 tok/s in int8, you need 392 TB/s memory bandwidth or 200 H100s.
  - latchkey 2y ago
    Memory bandwidth and memory size. Along with power/cooling density. Hence why you see AMD's MI325x coming out with 256GB HBM3e, but it is the same FLOPs as a 300x. 6TB/s too, which outperforms H200's, by a lot. You can see the direction AMD is going with this... https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html https://www.amd.com/en/products/accelerators/instinct/mi300/...
  - Const-me 2y ago
    > For 969 tok/s in int8, you need 392 TB/s memory bandwidth I think that math is only valid for batch size = 1. When these 969 tokens/second come from multiple sessions of the same batch, loaded model tensor elements are reused to compute many tokens for the entire batch. With large enough batches, you can even saturate compute throughput of the GPU instead of bottlenecking on memory bandwidth.

szundi 2y ago

Parent wishes 70b not 405b though

wkat4242 2y ago

Yeah but what is in a 4090 is also comparable to a whole rack of servers a decade ago. The tech will get smaller.