4 ms·

Honest question: why is there so much PC hardware in the mix here? Why don't we have PCI + infiniband backends with GPUs and a little tiny orchestrating ARM con

Honest question: why is there so much PC hardware in the mix here? Why don't we have PCI + infiniband backends with GPUs and a little tiny orchestrating ARM controller and just let them all coordinate with each other? Is it just "momentum" from previous designs and/or lack of "market" for specialized GPU controllers?

jononor 2y ago
Keeping the GPUs feed is a actually a rather demanding job for deep learning training. I do not have experience with LLM/NLP, but for image and audio workloads one can struggle to reach full utilization of even a RTX2/3/4xxx GPU with a typical 4-8 core CPU. It does not take much to be bottlenecked by the CPU and/or IO.
bick_nyers 2y ago
Are you asking why pay extra for a CPU and RAM? Not everything can be done on a GPU, for example, .png decompression. If you really analyzed your training code and preprocessed your data substantially you could probably get away with very lightweight CPU/RAM resources but I think the reality is that it's such a minor contribution of cost to the overall system (GPU are expensive) that wasting development cycles on that degree of optimization isn't strictly necessary. When you're a hyperscaler you are likely chasing those fractions of a percent of cost efficiency though. To use my original example, you would likely want to preprocess your .png to either .webp (multi-threaded lossless) or .jpeg (lossy), but likely it wouldn't make sense to turn it into a GPU decompressible format as you would save on CPU cost during training but would pay more in storage (and maybe transfer) cost. Edit: To be more clear, if the CPU work is bottlenecking training, you want to optimize that as much as possible by preprocessing your data/tweaking training scripts. What I'm discussing here is the gap between "fast enough" and "faster": CPU is not fast enough for training < CPU is exactly fast enough for training < CPU is faster than needed for training
ianburrell 2y ago
Cause when you have quarter million dollars of GPU on each machine, it is dumb to worry about few thousand for the controlling hardware. Too risky to use something new. Another problem is that all the hardware, drivers, and experience for GPU are on PC. It would take a lot of work to get running on ARM since would be starting from scratch. Then more work to get it stable. All to save a little on processor.

jononor 2y ago

Keeping the GPUs feed is a actually a rather demanding job for deep learning training. I do not have experience with LLM/NLP, but for image and audio workloads one can struggle to reach full utilization of even a RTX2/3/4xxx GPU with a typical 4-8 core CPU. It does not take much to be bottlenecked by the CPU and/or IO.

bick_nyers 2y ago

Are you asking why pay extra for a CPU and RAM? Not everything can be done on a GPU, for example, .png decompression. If you really analyzed your training code and preprocessed your data substantially you could probably get away with very lightweight CPU/RAM resources but I think the reality is that it's such a minor contribution of cost to the overall system (GPU are expensive) that wasting development cycles on that degree of optimization isn't strictly necessary. When you're a hyperscaler you are likely chasing those fractions of a percent of cost efficiency though. To use my original example, you would likely want to preprocess your .png to either .webp (multi-threaded lossless) or .jpeg (lossy), but likely it wouldn't make sense to turn it into a GPU decompressible format as you would save on CPU cost during training but would pay more in storage (and maybe transfer) cost. Edit: To be more clear, if the CPU work is bottlenecking training, you want to optimize that as much as possible by preprocessing your data/tweaking training scripts. What I'm discussing here is the gap between "fast enough" and "faster": CPU is not fast enough for training < CPU is exactly fast enough for training < CPU is faster than needed for training

ianburrell 2y ago

Cause when you have quarter million dollars of GPU on each machine, it is dumb to worry about few thousand for the controlling hardware. Too risky to use something new. Another problem is that all the hardware, drivers, and experience for GPU are on PC. It would take a lot of work to get running on ARM since would be starting from scratch. Then more work to get it stable. All to save a little on processor.