3 ms·
It doesn’t make sense to load the weights on the fly- that is gigabits of memory that has to be shuffled around. Instead, you have a long running process that s
by mattnewton 3y ago
It doesn’t make sense to load the weights on the fly- that is gigabits of memory that has to be shuffled around. Instead, you have a long running process that serves up lots of predictions
(edit: someday soon, probably to multiple clients too!)
- jameshart 3y agoLocal UI applications are long running processes normally
- nightfly 3y agoSo better to have GiBs of memory consumed by it constantly?
- mattnewton 3y agoIf you don’t have that memory to spare you can’t run this locally anyways, and keeping it in memory is the only way to have a fast experience. Paying the model loading cost repeatedly sucks.
- jameshart 3y agoWhy would linking llama.cpp into a UI application lead to incurring the model loading cost repeatedly?
- mattnewton 3y agoIt would be loaded repeatedly if the ui is opened and closed repeatedly. You can achieve the same “long running server + short running ui window” with multiple threads or processes all linked into one binary if you want of course. This way (with a separate server) seems simpler to me (and has the added benefit that multiple applications could easily call into the “server” if needed)