3 ms·
8 GPUs do a pretty bang-up job of doing 8 months of compute in a month :). I think the broader point is that the last x0 years of ML research show that more co
by kristjansson 3y ago
8 GPUs do a pretty bang-up job of doing 8 months of compute in a month :).
I think the broader point is that the last x0 years of ML research show that more compute is better, both for iteration speed and for resulting performance. Distribution is just the natural outgrowth of that imperative once it reaches the limit of a single device/node. If Cerebras can address models at today's scale on one device, the immediate next step is "what can N of these devices do together to build models at tomorrow's scale"
[0]: https://www.cerebras.net/condor-galaxy-1 https://www.cerebras.net/condor-galaxy-1
- TeMPOraL 3y agoFair enough :). I still think work on improving single-core/device performance is worthwhile, as distribution will always strictly not-better, and almost always strictly worse, due to coordination costs reducing efficiency. If two Cerberas can be glued together and achieve roughly 2x of their performance, it's still going to be more efficient than achieving equivalent performance from many more regular GPUs. Getting the hardware fast enough so that you need just one device for your problem - that's a special case that will yield extra win.