3 ms·

The reason for us is rental price. Customers don't notice the difference between 0.01s inference on GPU and 0.1s inference on CPU. Renting the necessary high-CP

The reason for us is rental price. Customers don't notice the difference between 0.01s inference on GPU and 0.1s inference on CPU. Renting the necessary high-CPU servers from Hetzner is like $0.1 per hour. I'm not aware of any provider that would offer the equivalent of a 3090 for less than $2 per hour. So it's 20x cheaper to do inference on CPU. Also, we tried to get into NVIDIA Inception and were rejected for the company being too small, so even if we wanted to do GPU deployment, we wouldn't be able to buy 3090s for testing. Scalpers / eBay are not an option in Germany because for company purchases, I need a proper tax invoice from another VAT-registered company.

querez 4y ago
If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an inference-only GPU like the T4 or the A2? From what you're saying it seems to me like they're made for use cases exactly like yours.
- fxtentacle 4y ago
  We use the 3090 for testing and development, because it is "similar enough" to a V100. For deployment, we were evaluating A100 and V100. (You're not allowed to put a 3090 into a datacenter.) Since it's a 900 mio parameter model, T4 and A2 don't have enough RAM, I can barely train this thing on a 3090 ^_^
  - david-gpu 4y ago
    Once models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.
blagie 4y ago
I've had good luck getting A4000 cards from small vendors, at least as of last summer. ATLGaming was the shop we went with last. It's one person, and was able to source cards at MSRP. Very professional and excellent service. I don't know about EU and VAT, but they did handle US enterprise clients well. At the time, the price of the A4000 was similar to scalper-prices for similarly-performing gaming cards, but: - Had ECC RAM and was generally designed for ML/enterprise/data workloads - Was more power-efficient - Had 16GB RAM, which you won't get on anything short of an A3090 I'd taking a consumer card at MSRP over the A4000, but not at street price. For what I'm doing, an A3060 Ti would probably even have been fine. It has an MSRP of $330, but a street price of $500-$1000 depending on the month you look. The enterprise cards tend to sell at MSRP (which is just around a grand for the A4000). Note that NVidia reuses part numbers. With "4000," there is also an older NVIDIA Quadro RTX 4000 (no A) selling for the same price on eBay / Amazon / etc. as the Ampere-based A4000.

querez 4y ago

If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an inference-only GPU like the T4 or the A2? From what you're saying it seems to me like they're made for use cases exactly like yours.

fxtentacle 4y ago
We use the 3090 for testing and development, because it is "similar enough" to a V100. For deployment, we were evaluating A100 and V100. (You're not allowed to put a 3090 into a datacenter.) Since it's a 900 mio parameter model, T4 and A2 don't have enough RAM, I can barely train this thing on a 3090 ^_^
- david-gpu 4y ago
  Once models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.

blagie 4y ago

I've had good luck getting A4000 cards from small vendors, at least as of last summer. ATLGaming was the shop we went with last. It's one person, and was able to source cards at MSRP. Very professional and excellent service. I don't know about EU and VAT, but they did handle US enterprise clients well. At the time, the price of the A4000 was similar to scalper-prices for similarly-performing gaming cards, but: - Had ECC RAM and was generally designed for ML/enterprise/data workloads - Was more power-efficient - Had 16GB RAM, which you won't get on anything short of an A3090 I'd taking a consumer card at MSRP over the A4000, but not at street price. For what I'm doing, an A3060 Ti would probably even have been fine. It has an MSRP of $330, but a street price of $500-$1000 depending on the month you look. The enterprise cards tend to sell at MSRP (which is just around a grand for the A4000). Note that NVidia reuses part numbers. With "4000," there is also an older NVIDIA Quadro RTX 4000 (no A) selling for the same price on eBay / Amazon / etc. as the Ampere-based A4000.