3 ms·

edit: I should have just checked their website instead of guessing. Apparently WSE has significant fabrication challenges, which makes what Cerebras has accompl

edit: I should have just checked their website instead of guessing. Apparently WSE has significant fabrication challenges, which makes what Cerebras has accomplished all the more impressive. But it is still surprising that no one else has attempted this in the HPC field. I had guessed that Cerebras had made some trade-offs in process in order to make it work at scale, but then they aren't actually building these devices at scale (yet).

sanxiyn 2y ago
Cerebras is known to use TSMC, so your speculation about boutique fab is incorrect. https://cerebras.ai/press-release/cerebras-systems-smashes-the-2-5-trillion-transistor-mark-with-new-second-generation-wafer-scale-engine/ https://cerebras.ai/press-release/cerebras-systems-smashes-t...
johntb86 2y ago
They're using TSMC 5-nm for WSE-3: https://spectrum.ieee.org/cerebras-chip-cs3 https://spectrum.ieee.org/cerebras-chip-cs3
throwawaymaths 2y ago
It shouldn't be surprising. It's hard as fuck, and we don't know if it's worth it yet (there's something to be said for "if your compute dies you send your remote hands to swap out a high four figure component" and not "decommission a high-six-figure node"
- ChainOfFools 2y ago
  I hope I didn't make it sound like it was easy, at least I don't think I said that anywhere. It doesn't really matter how hard something is to do (short of it being trivially proven impossible), it matters whether there's a good enough chance that the payoff exceeds the cost. And actually there have been attempts to do it, I mentioned in an earlier version of my comment that Gene Amdahl had attemped to make WSE work something like 20 years ago, without success - but also without the clear profitability story of AI to attract the same mountains of cash being thrown around today. What's surprising is not that it is hard, or that it's hard as fuck, but that given the potentially stratospheric rewards for success there have not been more attempts in this direction.
  - throwawaymaths 2y ago
    What makes you think you couldn't do as well on something less radical? IIRC cerebras' design was originally for HPC workloads, so even it may not necessarily be optimized for LLMs

sanxiyn 2y ago

Cerebras is known to use TSMC, so your speculation about boutique fab is incorrect. https://cerebras.ai/press-release/cerebras-systems-smashes-the-2-5-trillion-transistor-mark-with-new-second-generation-wafer-scale-engine/ https://cerebras.ai/press-release/cerebras-systems-smashes-t...

johntb86 2y ago

They're using TSMC 5-nm for WSE-3: https://spectrum.ieee.org/cerebras-chip-cs3 https://spectrum.ieee.org/cerebras-chip-cs3

throwawaymaths 2y ago

It shouldn't be surprising. It's hard as fuck, and we don't know if it's worth it yet (there's something to be said for "if your compute dies you send your remote hands to swap out a high four figure component" and not "decommission a high-six-figure node"

ChainOfFools 2y ago
I hope I didn't make it sound like it was easy, at least I don't think I said that anywhere. It doesn't really matter how hard something is to do (short of it being trivially proven impossible), it matters whether there's a good enough chance that the payoff exceeds the cost. And actually there have been attempts to do it, I mentioned in an earlier version of my comment that Gene Amdahl had attemped to make WSE work something like 20 years ago, without success - but also without the clear profitability story of AI to attract the same mountains of cash being thrown around today. What's surprising is not that it is hard, or that it's hard as fuck, but that given the potentially stratospheric rewards for success there have not been more attempts in this direction.
- throwawaymaths 2y ago
  What makes you think you couldn't do as well on something less radical? IIRC cerebras' design was originally for HPC workloads, so even it may not necessarily be optimized for LLMs