3 ms·
Interesting. Much of my work here is on production GCP workloads. We've landed on C2s and cpu optimizations for our inference engine but hadn't really considere
by ramoz 6y ago
Interesting. Much of my work here is on production GCP workloads. We've landed on C2s and cpu optimizations for our inference engine but hadn't really considered NVLINK. Now wondering if I can distribute batch inference across multi-gpus.
We'll be on premium support soon ... hoping we can get access to folks like yourself for some of this.
- boulos 6y agoI should clarify: A100s are likely a bad price/performance trade off for inference. The T4 part is designed for that, but doesn’t have NVLINK. Do you have models >16 GB that you’re trying to do real-time inference against? Feel free to send me an email regardless! (In my profile) Edit: https://news.ycombinator.com/item?id=23800049 https://news.ycombinator.com/item?id=23800049 was my writeup for a recent Ask HN about cost efficient inference.
- zetazzed 6y agoA100s support MIG (multi instance GPUs), so you can carve each A100 up into 7 separate GPUs for inference of small models. If your inference workload is too small to feed this beast all at once, it can be pretty handy: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html https://docs.nvidia.com/datacenter/tesla/mig-user-guide/inde...