3 ms·
Is there a reason all these language models rarely have any indication of the amount of GPU ram required to run them?
by DrSiemer 3y ago
Is there a reason all these language models rarely have any indication of the amount of GPU ram required to run them?
- Reubend 3y agoIt's because you can quantize to different extents, at the expense of a small amount of accuracy. The original model probably needs much more vRAM than most people will actually end up needing in prod.
- wsgeorge 3y agoI would guess that a) they assume their users will have a lot of GPU ram, b) actual running costs will depend on what inference engine/framework you're using (for example, some GGUF quants are very cheap).