3 ms·
In addition to what mono said, llama.cpp allocates everything up front with "--mlock" Llama.cpp (and MLC) have to read the all the model weights from RAM for e
by brucethemoose2 3y ago
In addition to what mono said, llama.cpp allocates everything up front with "--mlock"
Llama.cpp (and MLC) have to read the all the model weights from RAM for every token. Batching aside, there's no way around that.
- Art9681 3y agoMlock is an optional parameter: github.com/ggerganov/llama.cpp/tree/master/examples/main#mlock
- deleted 3y ago[deleted]