4 ms·
Question: Suppose you wanted to train an LLM using say 100 GPUs, but for some reason you could not obtain the GPUs. You could however obtain CPUs. Each CPU lack
by tzs 2y ago
Question: Suppose you wanted to train an LLM using say 100 GPUs, but for some reason you could not obtain the GPUs. You could however obtain CPUs. Each CPU lacks the massive parallelism of each GPU, so might have a throughout of say 1/100th the TFLOPS of the GPU.
Could you use 10000 CPUs and get a similar level of performance that 100 GPUs would have given?
- lostmsu 2y agoPlease, refer to the article discussed here: https://news.ycombinator.com/item?id=41808013 https://news.ycombinator.com/item?id=41808013 TL;DR; 9950x peak fp16 is 10 Tops, 4090 peak fp16 is 650 Tops, and H100 is about 2 Pops.