6 ms·
This honestly feels kind of silly. Back-of-the-envelope, this training run (13B model on 380B tokens) could be done in two months on a single 8x H100 node using
by apsec112 2y ago
This honestly feels kind of silly. Back-of-the-envelope, this training run (13B model on 380B tokens) could be done in two months on a single 8x H100 node using off-the-shelf software, at a cost of around $35K from a cloud compute provider. They don't seem to list training time, but this cluster appears to use ~3.5 MW of power, so it's going to burn something like $500/hr just in electricity costs.
- LeoPanthera 2y agoBut the whole point of this was to show that it could be done on CPUs during the worldwide shortage of GPUs. Doing it in the CPU was the whole point.
- fnordpiglet 2y agoIt could be done using abacuses too. A global shortage of GPUs is probably better addressed by producing more GPUs and making techniques more efficient.
- slt2021 2y agoI would love to see human abacus calculating gradients like in 3 body problem https://i.imgur.com/pdz7xGh.jpeg https://i.imgur.com/pdz7xGh.jpeg
- fnordpiglet 2y agoYes best scene on tv ever.
- Eisenstein 2y agoYou are right. I assume that you have a fab ready to go and are licensed to produce CUDA capable GPUs?
- fnordpiglet 2y agoI’m certain they’re being built.
- rad_gruchalski 2y agoso it's being addressed and in the meantime someone has some fun
- fnordpiglet 2y agoYeah man I’m down with using super computer clusters for anything. That’s just awesome. But the results seem more like marketing literature for GPU manufacturers to help rationalize more fab spend.
- mat_epice 2y agoThe system uses 30MW, but this job used a portion of it that would consume 2.6MW. There isn't really a figure for how much compute time it takes to train this thing, but 8x H100s have 32PF of AI compute among them. This job had 2,100 (half precision[1]) PF-in-fugaku / 158,956 nodes-in-fugaku * 13,824 nodes-in-job = 182 PF-in-job, implying it can get the job done 5.6x faster, or a little over ten days at the most optimistic. Electricity costs for these nodes for ten days looks fairly similar to the rental costs of 8xH100s for 60 days according to my research. Lambda labs seems to have very cheap instances for 8xH100, but AWS and its ilk are much higher. However, the comparison is a little weird, as Fugaku is also a few years old now, and the contemporary GPU at the time of its release was the A100 (1/13th of an H100). The next Fujitsu chip may well narrow the power/performance gap between itself and (say) Blackwell or whatever is current at the time. [1] https://www.fujitsu.com/global/about/innovation/fugaku/specifications/ https://www.fujitsu.com/global/about/innovation/fugaku/speci...
- numpad0 2y agoI'm sure it's used for all sorts of industrial purposes, but Fugaku/K seemed a bit of solution in search of problem since NEC backed off(~2009). They were supposed to supply vector processing cluster interconnected with Fujitsu supplied scalar cluster that became K, but cancelled it stating vector processing is an outdated concept not worth pursuing during cash crunch the company was going through. I can't seem to find a lot of technical documents regarding NEC SX processors online, but I could find outrageously priced PCIe accelerator cards(released 2021, $18k, 300W, 4.91TF FP32, 48GB VRAM) that are supposed to be its descendants, while Fugaku sits there GPU-less, maybe such is the life.
- Trapais 2y agoFor comparison, here's 8B [Nemotron](https://huggingface.co/nvidia https://huggingface.co/nvidia): > 1,024 A100s were used for 19 days to train the model. > NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing 3.8 Trillion tokens of text.