4 ms·
I should clarify: A100s are likely a bad price/performance trade off for inference. The T4 part is designed for that, but doesn’t have NVLINK. Do you have mode
by boulos 6y ago
I should clarify: A100s are likely a bad price/performance trade off for inference. The T4 part is designed for that, but doesn’t have NVLINK.
Do you have models >16 GB that you’re trying to do real-time inference against?
Feel free to send me an email regardless! (In my profile)
Edit: https://news.ycombinator.com/item?id=23800049 https://news.ycombinator.com/item?id=23800049 was my writeup for a recent Ask HN about cost efficient inference.
- zetazzed 6y agoA100s support MIG (multi instance GPUs), so you can carve each A100 up into 7 separate GPUs for inference of small models. If your inference workload is too small to feed this beast all at once, it can be pretty handy: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html https://docs.nvidia.com/datacenter/tesla/mig-user-guide/inde...