4 ms·
Why would linking llama.cpp into a UI application lead to incurring the model loading cost repeatedly?
by jameshart 3y ago
Why would linking llama.cpp into a UI application lead to incurring the model loading cost repeatedly?
- mattnewton 3y agoIt would be loaded repeatedly if the ui is opened and closed repeatedly. You can achieve the same “long running server + short running ui window” with multiple threads or processes all linked into one binary if you want of course. This way (with a separate server) seems simpler to me (and has the added benefit that multiple applications could easily call into the “server” if needed)