3 ms·
Any data on inference speed? I’ve found that the non quantized model was much faster on GPU than the quantized versions due to lower GPU utilization.
by Der_Einzige 3y ago
Any data on inference speed? I’ve found that the non quantized model was much faster on GPU than the quantized versions due to lower GPU utilization.
- api 3y agoIt's a RAM tradeoff. If you have enough GPU RAM to load the non-quantized model it may be faster.