3 ms·
You can't distribute LLM training. It requires high capacity low latency links. Even PCIe is a bottleneck
by flangola7 3y ago
You can't distribute LLM training. It requires high capacity low latency links. Even PCIe is a bottleneck
- CamperBob2 3y agoThat's my point -- there is no more important research problem in computer science right now, IMO, than figuring out how to distribute training. I don't believe that it is provably impossible, or at least I haven't seen any such formal proof.
- flangola7 3y agoHopefully nations will get that on lockdown before it becomes feasible. We do not need the digital equivalent of distributed uranium enrichment.
- CamperBob2 3y agoWell, that's certainly... a point of view.