4 ms·

Which is extra important because, if someone is still running AI/ML code on the CPU rather than GPU in 2022, I suspect there must be very good reasons for doing

Which is extra important because, if someone is still running AI/ML code on the CPU rather than GPU in 2022, I suspect there must be very good reasons for doing so. So that makes it even less likely that this code can be ported in an economical way.

deleted 4y ago
[deleted]
rocqua 4y ago
I can very well imagine there is lots of non AI/ML code that smart people could make a lot faster on a GPU. I doubt there are anywhere near enough smart people to do that. I also imagine Nvidea might like to overstate the total potential.
- nobodyandproud 4y ago
  It’s not that difficult. The biggest hurdles are learning the dev tech stack and setting up the development environment. The last time I tried out CUDA, it took me a couple of days to correctly setup the environment; just to spend an hour playing around parallelizing math operations. The best way I could describe the first impression dev experience: It’s a huge hack, adding non-standard language extensions to C/C++ And so relying on their own compiler or linker (iirc), instead of the standard C++. Couple this with NVidia not playing nice with OpenCL to force you into their playground, and I lost interest pretty quickly.
  - rocqua 4y ago
    Besides that hack, getting your head around the way to optimize code for a GPU takes a while. Then the real challenge is taking code that is based on random access to gigabytes of RAM, and making that code work with any kind of performance on the GPU performance model. There are cases where the random access to RAM are very deep in the algorithm. Consider e.g. sparse matrix multiplication.
    - nobodyandproud 4y ago
      “optimal” is a different goalpost than “a lot faster”. Some easy-to-parallelize math and vector operations can yield impressive results without it being anywhere near optimal. I do agree that optimal requires mastery of GPU programming; but I also think GPU programming can yield good results without said mastery, and be quite accessible If anything it should be easier than writing raw multi-threading code, because GPUs aren’t (weren’t?) general purpose calculation units. So more bare metal concerns, but less to reason about overall.
moffkalast 4y ago
> I suspect there must be very good reasons for doing so. GPU instances being absurdly pricey is not a good enough reason?
tiffanyh 4y ago
Don’t CGI rendering farms (Pixar, etc) use thousands of CPUs to render their movies (not GPUs) because the cost/perf is better.
- jb1991 4y ago
  It’s much much more complicated than that. They do typically use CPUs but there are many reasons for it. Starting with the fact that the path tracing algorithm is quite different when it runs on a CPU than on a GPU because thread divergence is a fundamental aspect of that formula. Second, the vast memory requirements are much easier to handle on a CPU than a GPU. There’s also the decades of man hours spent on the CPU code that is not so instantly ported to GPUs. Honestly this list goes on and on.
  - nopenopenopeno 4y ago
    Yeah, it’s really not that complicated. I think you covered all the reasons pretty straightforwardly.
    - jb1991 4y ago
      Well it's complicated when you consider it's taken many years of research to get where they are at currently, which is still far from where some have considered going.
fxtentacle 4y ago
The reason for us is rental price. Customers don't notice the difference between 0.01s inference on GPU and 0.1s inference on CPU. Renting the necessary high-CPU servers from Hetzner is like $0.1 per hour. I'm not aware of any provider that would offer the equivalent of a 3090 for less than $2 per hour. So it's 20x cheaper to do inference on CPU. Also, we tried to get into NVIDIA Inception and were rejected for the company being too small, so even if we wanted to do GPU deployment, we wouldn't be able to buy 3090s for testing. Scalpers / eBay are not an option in Germany because for company purchases, I need a proper tax invoice from another VAT-registered company.
- querez 4y ago
  If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an inference-only GPU like the T4 or the A2? From what you're saying it seems to me like they're made for use cases exactly like yours.
  - fxtentacle 4y ago
    We use the 3090 for testing and development, because it is "similar enough" to a V100. For deployment, we were evaluating A100 and V100. (You're not allowed to put a 3090 into a datacenter.) Since it's a 900 mio parameter model, T4 and A2 don't have enough RAM, I can barely train this thing on a 3090 ^_^
    - david-gpu 4y ago
      Once models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.
- blagie 4y ago
  I've had good luck getting A4000 cards from small vendors, at least as of last summer. ATLGaming was the shop we went with last. It's one person, and was able to source cards at MSRP. Very professional and excellent service. I don't know about EU and VAT, but they did handle US enterprise clients well. At the time, the price of the A4000 was similar to scalper-prices for similarly-performing gaming cards, but: - Had ECC RAM and was generally designed for ML/enterprise/data workloads - Was more power-efficient - Had 16GB RAM, which you won't get on anything short of an A3090 I'd taking a consumer card at MSRP over the A4000, but not at street price. For what I'm doing, an A3060 Ti would probably even have been fine. It has an MSRP of $330, but a street price of $500-$1000 depending on the month you look. The enterprise cards tend to sell at MSRP (which is just around a grand for the A4000). Note that NVidia reuses part numbers. With "4000," there is also an older NVIDIA Quadro RTX 4000 (no A) selling for the same price on eBay / Amazon / etc. as the Ampere-based A4000.

deleted 4y ago

[deleted]

rocqua 4y ago

I can very well imagine there is lots of non AI/ML code that smart people could make a lot faster on a GPU. I doubt there are anywhere near enough smart people to do that. I also imagine Nvidea might like to overstate the total potential.

nobodyandproud 4y ago
It’s not that difficult. The biggest hurdles are learning the dev tech stack and setting up the development environment. The last time I tried out CUDA, it took me a couple of days to correctly setup the environment; just to spend an hour playing around parallelizing math operations. The best way I could describe the first impression dev experience: It’s a huge hack, adding non-standard language extensions to C/C++ And so relying on their own compiler or linker (iirc), instead of the standard C++. Couple this with NVidia not playing nice with OpenCL to force you into their playground, and I lost interest pretty quickly.
- rocqua 4y ago
  Besides that hack, getting your head around the way to optimize code for a GPU takes a while. Then the real challenge is taking code that is based on random access to gigabytes of RAM, and making that code work with any kind of performance on the GPU performance model. There are cases where the random access to RAM are very deep in the algorithm. Consider e.g. sparse matrix multiplication.
  - nobodyandproud 4y ago
    “optimal” is a different goalpost than “a lot faster”. Some easy-to-parallelize math and vector operations can yield impressive results without it being anywhere near optimal. I do agree that optimal requires mastery of GPU programming; but I also think GPU programming can yield good results without said mastery, and be quite accessible If anything it should be easier than writing raw multi-threading code, because GPUs aren’t (weren’t?) general purpose calculation units. So more bare metal concerns, but less to reason about overall.

moffkalast 4y ago

> I suspect there must be very good reasons for doing so. GPU instances being absurdly pricey is not a good enough reason?

tiffanyh 4y ago

Don’t CGI rendering farms (Pixar, etc) use thousands of CPUs to render their movies (not GPUs) because the cost/perf is better.

jb1991 4y ago
It’s much much more complicated than that. They do typically use CPUs but there are many reasons for it. Starting with the fact that the path tracing algorithm is quite different when it runs on a CPU than on a GPU because thread divergence is a fundamental aspect of that formula. Second, the vast memory requirements are much easier to handle on a CPU than a GPU. There’s also the decades of man hours spent on the CPU code that is not so instantly ported to GPUs. Honestly this list goes on and on.
- nopenopenopeno 4y ago
  Yeah, it’s really not that complicated. I think you covered all the reasons pretty straightforwardly.
  - jb1991 4y ago
    Well it's complicated when you consider it's taken many years of research to get where they are at currently, which is still far from where some have considered going.

fxtentacle 4y ago

The reason for us is rental price. Customers don't notice the difference between 0.01s inference on GPU and 0.1s inference on CPU. Renting the necessary high-CPU servers from Hetzner is like $0.1 per hour. I'm not aware of any provider that would offer the equivalent of a 3090 for less than $2 per hour. So it's 20x cheaper to do inference on CPU. Also, we tried to get into NVIDIA Inception and were rejected for the company being too small, so even if we wanted to do GPU deployment, we wouldn't be able to buy 3090s for testing. Scalpers / eBay are not an option in Germany because for company purchases, I need a proper tax invoice from another VAT-registered company.

querez 4y ago
If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an inference-only GPU like the T4 or the A2? From what you're saying it seems to me like they're made for use cases exactly like yours.
- fxtentacle 4y ago
  We use the 3090 for testing and development, because it is "similar enough" to a V100. For deployment, we were evaluating A100 and V100. (You're not allowed to put a 3090 into a datacenter.) Since it's a 900 mio parameter model, T4 and A2 don't have enough RAM, I can barely train this thing on a 3090 ^_^
  - david-gpu 4y ago
    Once models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.
blagie 4y ago
I've had good luck getting A4000 cards from small vendors, at least as of last summer. ATLGaming was the shop we went with last. It's one person, and was able to source cards at MSRP. Very professional and excellent service. I don't know about EU and VAT, but they did handle US enterprise clients well. At the time, the price of the A4000 was similar to scalper-prices for similarly-performing gaming cards, but: - Had ECC RAM and was generally designed for ML/enterprise/data workloads - Was more power-efficient - Had 16GB RAM, which you won't get on anything short of an A3090 I'd taking a consumer card at MSRP over the A4000, but not at street price. For what I'm doing, an A3060 Ti would probably even have been fine. It has an MSRP of $330, but a street price of $500-$1000 depending on the month you look. The enterprise cards tend to sell at MSRP (which is just around a grand for the A4000). Note that NVidia reuses part numbers. With "4000," there is also an older NVIDIA Quadro RTX 4000 (no A) selling for the same price on eBay / Amazon / etc. as the Ampere-based A4000.