3 ms·

Now I understand why ML uses floats for precision. It wasn't a choice, it was because graphics code uses them. Another piece in the "why is ML so inefficient"

by mannyv 3y ago

Now I understand why ML uses floats for precision. It wasn't a choice, it was because graphics code uses them. Another piece in the "why is ML so inefficient" puzzle! I wonder what that memory copying overhead is IRL. If it's like normal stuff it'll be brutal. I mean, they offload tcp processing into hardware to avoid that. This is way more data, though it is done in bigger chunks.

choppaface 3y ago
For a lot of larger modern networks the GPU compute time in computing gradients and doing the backwards pass is so slow that copying float data over the pcie bus is no bottleneck. I.e. copying a minibatch of float images is still plenty fast because the gradients / SGD iteration is so slow and requires so much compute (even with mixed precision). For shallower networks, there can be an advantage to copying just the original compressed data to GPU memory and then doing decompression etc there. But modern GPUs haven’t adopted pcie 5 yet because the raw compute is more important. Lastly tensor cores have had a big impact, depending on the network they can be fast enough to be very under-utilized.
- touisteur 3y ago
  I would say that some one modern GPU has adopted PCIe 5.0, the NVIDIA H100 - if you want more (FP32) 'compute' (and no HBM...) in one GPU your sourceable choice is the L40 which didn't get PCIe 5.0. This feels like another market/product segmentation (that and L40 didn't get nvlink).
oivey 3y ago
I don’t think the choice of using floating point numbers is particularly inefficient. If frameworks were fixed point by default it would be tricky to get the dynamic ranges right all the way through a network. The math in the training assumes the numbers are continuous, too.
Dylan16807 3y ago
What's inefficient about floats? ML seems to get a big benefit out of having access to multiple/many orders of magnitude.
mannyv 3y ago
Floats are bigger, and math with them is more difficult. But really, I wondered why the cpu-bound LLMs do quantization, which from what I understand is the process of reducing the precision of the weights to use less memory. Does the lack of precision make a difference? It's unclear. If that's the case, then why use FP at all? If precision doesn't matter, then the extra precision is just making the process use more resources for no real reason. And likely orders of magnitude more resources than required. I mean, this field wasn't started by people who understood performance. They used tools and built something...but there's no 'why.' This is what the tools did, so they did that. Here's why that might be important: on a normal CPU, accessing data one way can be orders of magnitude faster than another way...but you have to be aware. Would you like to reduce your LLM costs by orders of magnitude?

by mannyv 3y ago

choppaface 3y ago

For a lot of larger modern networks the GPU compute time in computing gradients and doing the backwards pass is so slow that copying float data over the pcie bus is no bottleneck. I.e. copying a minibatch of float images is still plenty fast because the gradients / SGD iteration is so slow and requires so much compute (even with mixed precision). For shallower networks, there can be an advantage to copying just the original compressed data to GPU memory and then doing decompression etc there. But modern GPUs haven’t adopted pcie 5 yet because the raw compute is more important. Lastly tensor cores have had a big impact, depending on the network they can be fast enough to be very under-utilized.

touisteur 3y ago
I would say that some one modern GPU has adopted PCIe 5.0, the NVIDIA H100 - if you want more (FP32) 'compute' (and no HBM...) in one GPU your sourceable choice is the L40 which didn't get PCIe 5.0. This feels like another market/product segmentation (that and L40 didn't get nvlink).

oivey 3y ago

I don’t think the choice of using floating point numbers is particularly inefficient. If frameworks were fixed point by default it would be tricky to get the dynamic ranges right all the way through a network. The math in the training assumes the numbers are continuous, too.

Dylan16807 3y ago

What's inefficient about floats? ML seems to get a big benefit out of having access to multiple/many orders of magnitude.

mannyv 3y ago

Floats are bigger, and math with them is more difficult. But really, I wondered why the cpu-bound LLMs do quantization, which from what I understand is the process of reducing the precision of the weights to use less memory. Does the lack of precision make a difference? It's unclear. If that's the case, then why use FP at all? If precision doesn't matter, then the extra precision is just making the process use more resources for no real reason. And likely orders of magnitude more resources than required. I mean, this field wasn't started by people who understood performance. They used tools and built something...but there's no 'why.' This is what the tools did, so they did that. Here's why that might be important: on a normal CPU, accessing data one way can be orders of magnitude faster than another way...but you have to be aware. Would you like to reduce your LLM costs by orders of magnitude?