5 ms·
I was in the same boat and this is good advice! I stopped using gpu's, "Vectorized inference isn’t bad at all!". This soo much, I was blinded with gpu speed, u
by ParanoidShroom 6y ago
I was in the same boat and this is good advice!
I stopped using gpu's, "Vectorized inference isn’t bad at all!". This soo much, I was blinded with gpu speed, using tensorflow builds with avx optimization is actually pretty fast.
My discovery:
+ Stop expensive GPU's for inference and switch to avx optimized tensorflow builds.
+ Cleaned up the inference pipeline and reduced complexity.
+ Buying compute instance for a year or more provides a discount.
- I never got pruning to work without a significant loss increase.
- Tried spot instances with gpu's that are cheaper. Random kills and spinning up new instances took too long loading my code. The discount is a lot, but I couldn't reliable get it up. Users where getting more timeouts. I bailed and just used cpu inference. The gpu was being underutilized, using cpu only increased the inference to around 2-3 seconds. With the price trade off it was a more simpel,cheaper and easier solution.
- jwr 6y agoAlso, consider physical servers from providers like Hetzner. These can be several times cheaper than EC2.
- papaf 6y agoI use Hetzner for quite a lot for personal projects and can recommend them for reliability and predictable costs. I've done reasonably high CPU tasks like compiling Android images on the larger Cloud instances. However, this morning I was playing around with Scaleway bare metal [1] and General Purpose instances [2] -- I am thinking of making a switch for high CPU tasks. [1] https://www.scaleway.com/en/bare-metal-servers/ https://www.scaleway.com/en/bare-metal-servers/ [2] https://www.scaleway.com/en/virtual-instances/general-purpose/ https://www.scaleway.com/en/virtual-instances/general-purpos...
- jwr 6y agoInteresting! These look very good indeed. I will have to try them. The main point is that physical servers are much cheaper than VMs and provide significantly better performance as well (see my benchmarking and comparison: https://jan.rychter.com/enblog/cloud-server-cpu-performance-comparison-2019-12-12 https://jan.rychter.com/enblog/cloud-server-cpu-performance-...).
- pluc 6y agoI worked on an unrelated market study - look at Upcloud and Raptr as well.
- GordonS 6y agoI was just looking at Hetzner yesterday, looking to host a HA Postgres setup. Their block storage volumes look interesting, but I couldn't find any information on performance guarantees, or even claims. Anyone have an idea about performance (IOPS or MB/s)?
- StavrosK 6y agoI use them but don't have that info off the top of my head. However, you can easily make an account, get a VPS with a volume and benchmark it in a few minutes for a few cents.