3 ms·
Putting the whole model in memory is far faster then swapping to disk.
by vorticalbox 2mo ago
Putting the whole model in memory is far faster then swapping to disk.
- giancarlostoro 2mo agoFor local inference the cost of "speed" is not that bad I would think? I wouldn't mind a bit of a delay if it means I can run much larger models on my Mac.
- pertymcpert 2mo agoIt's pretty painful to have speeds < 30 tok/sec though. Especially if you're used to API providers at higher speeds. It makes any interactive work almost impossible to do efficiently because you have no choice but to context switch after every request.
- giancarlostoro 2mo agoI assume it will get better over time, and does it improve in speed if you use a larger buffer? Say instead of 2GB you go with 6GB? I imagine it would, and you might need to stream drastically less no?
- gitpusher42 2mo agoYeah, correct! You can set up this engine to get more expert cache slots (e.g 32 instead of 16) to get a better hit rate and better tok/s. it will be 3.5gb instead of 2gb.