3 ms·

AVX-512 neural net inference on inexpensive, CPU-only cloud compute instances: https://NN-512.com https://NN-512.com An AVX-512 Skylake-X cloud compute instanc

AVX-512 neural net inference on inexpensive, CPU-only cloud compute instances: https://NN-512.com https://NN-512.com An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month at Vultr (https://www.vultr.com/products/cloud-compute/ https://www.vultr.com/products/cloud-compute/), and you can do about 18 DenseNet121 inferences per CPU-core per second (in series, not batched) using tools like NN-512 GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute https://www.linode.com/pricing/#row--compute). It's really hard to keep that GPU saturated, which is what you need to do to get your money's worth As AVX-512 becomes better supported by Intel and AMD chips, it becomes more attractive as an alternative to expensive GPU instances for workloads with small amounts of inference mixed with other computation

rckoepke 6y ago
What does this have to do with https://gpu.land/ https://gpu.land/ ? I've been very impressed with NN-512 in your past postings but I'm failing to see the direct salience w.r.t the top level post. Perhaps readers would benefit from an apples-to-apples comparison of V100 to some CPU in a training per dollar metric? Preferably using something like MLPerf. You do mention inference but I think most people looking at https://gpu.land/ https://gpu.land/ will be far more interested in training rather than inference. I think the most direct competitor to this ShowHN would be https://vast.ai/ https://vast.ai/
- ilmoi 6y ago
  You are correct that vast.ai is pretty close. The biggest difference is probably security / guaranteed uptime. With vast you're getting what it says on the tin - a machine from a marketplace. Could come from anyone / anywhere. No idea what else is running on it. Ours are hosted in a professional DC, managed and secured as they should be. If anyone's curious, there's a detailed comparison page with other platforms here - https://gpu.land/versus https://gpu.land/versus
- perennate 6y ago
  I think Linode/Paperspace/AWS/GCP/etc. are closer to "direct competitors". I would be much more comfortable trusting a single entity that owns its servers than a GPU rental marketplace like vast.ai.
- 37ef_ced3 6y ago
  The suggestion is that you may not want to use a GPU if you can use AVX-512 CPUs for your machine learning workload. Reducing the cost of the GPU helps, but it's still relatively expensive
- sharemywin 6y ago
  Here's an article about ROBLOX running BERT on cpus versus GPUs. https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requests-cpus/ https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-... They're not training so I think that's the difference.
dindresto 6y ago
"AVX512 instructions cannot be sustained on a high clock speed for very long -- the processor downclocks right away if you send too many of them within a short time frame, probably for thermal reasons. This will probably be much more true for laptop processors." https://github.com/pytorch/pytorch/issues/4825#issuecomment-521706208 https://github.com/pytorch/pytorch/issues/4825#issuecomment-... How much does this apply to the Xeon CPUs found in servers?

rckoepke 6y ago

What does this have to do with https://gpu.land/ https://gpu.land/ ? I've been very impressed with NN-512 in your past postings but I'm failing to see the direct salience w.r.t the top level post. Perhaps readers would benefit from an apples-to-apples comparison of V100 to some CPU in a training per dollar metric? Preferably using something like MLPerf. You do mention inference but I think most people looking at https://gpu.land/ https://gpu.land/ will be far more interested in training rather than inference. I think the most direct competitor to this ShowHN would be https://vast.ai/ https://vast.ai/

ilmoi 6y ago
You are correct that vast.ai is pretty close. The biggest difference is probably security / guaranteed uptime. With vast you're getting what it says on the tin - a machine from a marketplace. Could come from anyone / anywhere. No idea what else is running on it. Ours are hosted in a professional DC, managed and secured as they should be. If anyone's curious, there's a detailed comparison page with other platforms here - https://gpu.land/versus https://gpu.land/versus
perennate 6y ago
I think Linode/Paperspace/AWS/GCP/etc. are closer to "direct competitors". I would be much more comfortable trusting a single entity that owns its servers than a GPU rental marketplace like vast.ai.
37ef_ced3 6y ago
The suggestion is that you may not want to use a GPU if you can use AVX-512 CPUs for your machine learning workload. Reducing the cost of the GPU helps, but it's still relatively expensive
sharemywin 6y ago
Here's an article about ROBLOX running BERT on cpus versus GPUs. https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requests-cpus/ https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-... They're not training so I think that's the difference.

dindresto 6y ago

"AVX512 instructions cannot be sustained on a high clock speed for very long -- the processor downclocks right away if you send too many of them within a short time frame, probably for thermal reasons. This will probably be much more true for laptop processors." https://github.com/pytorch/pytorch/issues/4825#issuecomment-521706208 https://github.com/pytorch/pytorch/issues/4825#issuecomment-... How much does this apply to the Xeon CPUs found in servers?