3 ms·
I was playing with this with the llama model was released - and provided I was loading from memory and not disk it was actually only negligbly slower than the m
by gpm 3y ago
I was playing with this with the llama model was released - and provided I was loading from memory and not disk it was actually only negligbly slower than the model was without sharding (for e.g. the 8gb models that I could load entirely on my GPU). Each layer took a lot more time to evaluate than it took to copy the weights onto the GPU (RTX 2070 super).
- godelski 3y agoYes, if you can fit it in RAM then the loss isn't too bad if you have the memory pinned and are taking advantage of some of Nvidia's optimizations (which should be defaulted to on). Inference usually isn't too bad but training is heavier. Things have also massively improved over the last few years. The efficiency also really increases with reduced floating point types and quantized models.