3 ms·
> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model I don't see any evidence that this is possible. From
by cubefox 2mo ago
> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
- IsTom 2mo agoI think it's enough that a single layer fits on each chip if you can daisy-chain them with good interconnects.
- wmf 2mo agoThe methods for splitting weights across multiple chips are well established. Groq/Cerebras can't hold a model on one chip either.
- pyrolistical 2mo agoUmm I have an extra 35, do you have layer 6?
- octoberfranklin 2mo agoThey pipeline-parallelize across multiple chips. DeepSeek v4 Pro will be 30 chips.