7 ms·
If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an in
by querez 4y ago
If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an inference-only GPU like the T4 or the A2? From what you're saying it seems to me like they're made for use cases exactly like yours.
- fxtentacle 4y agoWe use the 3090 for testing and development, because it is "similar enough" to a V100. For deployment, we were evaluating A100 and V100. (You're not allowed to put a 3090 into a datacenter.) Since it's a 900 mio parameter model, T4 and A2 don't have enough RAM, I can barely train this thing on a 3090 ^_^
- david-gpu 4y agoOnce models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.