5 ms·
Disclosure: I work on Google Cloud (but you don’t need to rent GPUs from us). Absolutely, but as last year’s discussion highlights, a bunch of GPUs connected v
by boulos 6y ago
Disclosure: I work on Google Cloud (but you don’t need to rent GPUs from us).
Absolutely, but as last year’s discussion highlights, a bunch of GPUs connected via NVLINK kind of gives you the aggregate memory of the set for some of these database applications (large-scale ML training has also gone this way).
That’s why our A100 system design is 16 A100s in a single host. 16x40 GB gives you 640 GB of aggregate memory, which is pretty attractive for many applications.
The question as always is cost vs benefit. If there’s something that a GPU backed < noun > can do that you “couldn’t” with a large Intel/AMD cpu box, or is actually a large integer multiple cheaper, it’s probably worth the development effort.
- ramoz 6y agoInteresting. Much of my work here is on production GCP workloads. We've landed on C2s and cpu optimizations for our inference engine but hadn't really considered NVLINK. Now wondering if I can distribute batch inference across multi-gpus. We'll be on premium support soon ... hoping we can get access to folks like yourself for some of this.
- boulos 6y agoI should clarify: A100s are likely a bad price/performance trade off for inference. The T4 part is designed for that, but doesn’t have NVLINK. Do you have models >16 GB that you’re trying to do real-time inference against? Feel free to send me an email regardless! (In my profile) Edit: https://news.ycombinator.com/item?id=23800049 https://news.ycombinator.com/item?id=23800049 was my writeup for a recent Ask HN about cost efficient inference.
- zetazzed 6y agoA100s support MIG (multi instance GPUs), so you can carve each A100 up into 7 separate GPUs for inference of small models. If your inference workload is too small to feed this beast all at once, it can be pretty handy: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html https://docs.nvidia.com/datacenter/tesla/mig-user-guide/inde...