4 ms·
Would it be possible to mmap the model parameters, allowing us to run even larger models? How much of an performance impact would that have?
by philippta 3y ago
Would it be possible to mmap the model parameters, allowing us to run even larger models? How much of an performance impact would that have?
- fancy_pantser 3y agoSure, this is common e.g. https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613
- Retr0id 3y agommap isn't magic, it's just one of many mechanisms for getting data off disk and into memory
- keivanalizadeh 3y agoAnd how does mmap gonna help with the performance when the bottleneck is flash to DRAM bandwidth?