3 ms·
Nope, you can use mmap to virtually map it to memory, and then you don’t have to hold the whole thing in RAM at once. I have spent the last 4-5 weeks working on
by malux85 3y ago
Nope, you can use mmap to virtually map it to memory, and then you don’t have to hold the whole thing in RAM at once. I have spent the last 4-5 weeks working on this and optimising it
You can see some information here:
https://justine.lol/mmap/ https://justine.lol/mmap/
- simonw 3y agoHow does that work? Does it mean that for every token generated it has to page areas of disk into RAM and then back out again?
- not2b 3y agoYou'd have to have enough RAM to hold the whole model in memory or performance will be awful, mmap is just a way to get faster startup (if the mmap'd file looks exactly like the in-memory representation) and easier sharing (the mmap region can be shared read-only memory that multiple processes use).