5 ms·

Not for what they are using it for. It is $1m+/chip and they can fit 1 of them in a rack. Rack space in DC's is a premium asset. The density isn't there. AI mod

Not for what they are using it for. It is $1m+/chip and they can fit 1 of them in a rack. Rack space in DC's is a premium asset. The density isn't there. AI models need tons of memory (this product annoucement is case in point) and they don't have it, nor do they have a way to get it since they are last in line at the fabs. Their only chance is an aquihire, but nvidia just spent $20b on groq instead. Dead man walking.

p1esk 8mo ago
The real question is what’s their perf/dollar vs nvidia?
- xnx 8mo ago
  Or Google TPUs.
  - latchkey 8mo ago
    TPUs don't have enough memory either, but they have really great interconnects, so they can build a nice high density cluster. Compare the photos of a Cerebras deployment to a TPU deployment. https://www.nextplatform.com/wp-content/uploads/2023/07/cerebras-g42-cg1-system-logo.jpg https://www.nextplatform.com/wp-content/uploads/2023/07/cere... https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iOLs2FEQxQvM/v0/-1x-1.webp https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iOLs2FEQxQv... The difference is striking.
    - p1esk 8mo ago
      Oh wow the cabling in the first link is really sloppy!
- latchkey 8mo ago
  Exactly. They won't ever tell you. It is never published. Let's not forget that the CEO is an SEC felon who got caught trying to pull a fast one.
- zozbot234 8mo ago
  I guess it depends what you mean by "perf". If you optimize everything for the absolutely lowest latency given your power budget, your throughput is going to suck - and vice versa. Throughput is ultimately what matters when everything about AI is so clearly power-constrained, latency is a distraction. So TPU-like custom chips are likely the better choice.
  - p1esk 8mo ago
    By perf I mean how much does it cost to serve 1T model to 1M users at 50 tokens/sec.
    - zozbot234 8mo ago
      All 1T models are not equal. E.g. how many active parameters? what's the native quantization? how long is the max context? Also, it's quite likely that some smaller models in common use are even sub-1T. If your model is light enough, the lower throughput doesn't necessarily hurt you all that much and you can enjoy the lightning-fast speed.
      - p1esk 8mo ago
        Just pick some reasonable values. Also, keep in mind that this hardware must still be useful 3 years from now. What’s going to happen to cerebras in 3 years? What about nvidia? Which one is a safer bet? On the other hand, competition is good - nvidia can’t have the whole pie forever.
        zozbot234 8mo ago
        > Just pick some reasonable values. And that's the point - what's "reasonable" depends on the hardware and is far from fixed. Some users here are saying that this model is "blazing fast" but a bit weaker than expected, and one might've guessed as much. > On the other hand, competition is good - nvidia can’t have the whole pie forever. Sure, but arguably the closest thing to competition for nVidia is TPUs and future custom ASICs that will likely save a lot on energy used per model inference, while not focusing all that much on being super fast.
- energy123 8mo ago
  That's coupling two different usecases. Many coding usecases care about tokens/second, not tokens/dollar.
spwa4 8mo ago
Oh don't worry. Ever since the power issue started developing rack space is no longer at a premium. Or at least, it's no longer the limiting factor. Power is.
- latchkey 8mo ago
  The dirty secret is that there is plenty of power. But, it isn't all in one place and it is often stranded in DC's that can't do the density needed for AI compute. Training models needs everything in one DC, inference doesn't.
  - spwa4 8mo ago
    Which DCs are these?
boredatoms 8mo ago
Power/cooling is the premium. Can always build a bigger hall
- latchkey 8mo ago
  Exactly my point. Their architecture requires someone to invest the capex / opex to also build another hall.
arisAlexis 8mo ago
How do you know the price of a unit ?
- latchkey 8mo ago
  I remembered $1m from when I was in their booth at SC24, but when I just looked, I was wrong. It is worse... https://www.datacenterdynamics.com/en/news/cerebras-unveils-four-trillion-transistor-giant-chip-targets-generative-ai/ https://www.datacenterdynamics.com/en/news/cerebras-unveils-...
  - arisAlexis 8mo ago
    You have no idea of the value this brings. Asml machines cost dozens of times more. So?
    - latchkey 8mo ago
      Ok, tell me the value.

p1esk 8mo ago

The real question is what’s their perf/dollar vs nvidia?

xnx 8mo ago
Or Google TPUs.
- latchkey 8mo ago
  TPUs don't have enough memory either, but they have really great interconnects, so they can build a nice high density cluster. Compare the photos of a Cerebras deployment to a TPU deployment. https://www.nextplatform.com/wp-content/uploads/2023/07/cerebras-g42-cg1-system-logo.jpg https://www.nextplatform.com/wp-content/uploads/2023/07/cere... https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iOLs2FEQxQvM/v0/-1x-1.webp https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iOLs2FEQxQv... The difference is striking.
  - p1esk 8mo ago
    Oh wow the cabling in the first link is really sloppy!
latchkey 8mo ago
Exactly. They won't ever tell you. It is never published. Let's not forget that the CEO is an SEC felon who got caught trying to pull a fast one.
zozbot234 8mo ago
I guess it depends what you mean by "perf". If you optimize everything for the absolutely lowest latency given your power budget, your throughput is going to suck - and vice versa. Throughput is ultimately what matters when everything about AI is so clearly power-constrained, latency is a distraction. So TPU-like custom chips are likely the better choice.
- p1esk 8mo ago
  By perf I mean how much does it cost to serve 1T model to 1M users at 50 tokens/sec.
  - zozbot234 8mo ago
    All 1T models are not equal. E.g. how many active parameters? what's the native quantization? how long is the max context? Also, it's quite likely that some smaller models in common use are even sub-1T. If your model is light enough, the lower throughput doesn't necessarily hurt you all that much and you can enjoy the lightning-fast speed.
    - p1esk 8mo ago
      Just pick some reasonable values. Also, keep in mind that this hardware must still be useful 3 years from now. What’s going to happen to cerebras in 3 years? What about nvidia? Which one is a safer bet? On the other hand, competition is good - nvidia can’t have the whole pie forever.
      - zozbot234 8mo ago
        > Just pick some reasonable values. And that's the point - what's "reasonable" depends on the hardware and is far from fixed. Some users here are saying that this model is "blazing fast" but a bit weaker than expected, and one might've guessed as much. > On the other hand, competition is good - nvidia can’t have the whole pie forever. Sure, but arguably the closest thing to competition for nVidia is TPUs and future custom ASICs that will likely save a lot on energy used per model inference, while not focusing all that much on being super fast.
energy123 8mo ago
That's coupling two different usecases. Many coding usecases care about tokens/second, not tokens/dollar.

spwa4 8mo ago

Oh don't worry. Ever since the power issue started developing rack space is no longer at a premium. Or at least, it's no longer the limiting factor. Power is.

latchkey 8mo ago
The dirty secret is that there is plenty of power. But, it isn't all in one place and it is often stranded in DC's that can't do the density needed for AI compute. Training models needs everything in one DC, inference doesn't.
- spwa4 8mo ago
  Which DCs are these?

boredatoms 8mo ago

Power/cooling is the premium. Can always build a bigger hall

latchkey 8mo ago
Exactly my point. Their architecture requires someone to invest the capex / opex to also build another hall.

arisAlexis 8mo ago

How do you know the price of a unit ?

latchkey 8mo ago
I remembered $1m from when I was in their booth at SC24, but when I just looked, I was wrong. It is worse... https://www.datacenterdynamics.com/en/news/cerebras-unveils-four-trillion-transistor-giant-chip-targets-generative-ai/ https://www.datacenterdynamics.com/en/news/cerebras-unveils-...
- arisAlexis 8mo ago
  You have no idea of the value this brings. Asml machines cost dozens of times more. So?
  - latchkey 8mo ago
    Ok, tell me the value.