3 ms·
If you want to not quantize at all, you need to double it for fp16—16GB.
by lolinder 3y ago
If you want to not quantize at all, you need to double it for fp16—16GB.
- cjbprime 3y agoYes, but I think it's standard to do inference at q8, not fp16.