3 ms·
This is wrong. You need big memory during inference too. The difference there is you can use tricks like quantisation and offloading to CPU to reduce it somewh
by nl 3y ago
This is wrong. You need big memory during inference too.
The difference there is you can use tricks like quantisation and offloading to CPU to reduce it somewhat at the cost of accuracy and/or speed.
- brrrrrm 3y agoTraining is 3x the memory used by inference, and usually run at a much larger batch size
- pama 3y agoNot sure what you mean by wrong. I have never encountered a case yet when training an LLM (no matter what architecture) would require limited memory and was pointing out that the typical memory requirements for training are much higher yet than the typical requirements for inference.