4 ms·

So better to have GiBs of memory consumed by it constantly?

• mattnewton 3y ago
  If you don’t have that memory to spare you can’t run this locally anyways, and keeping it in memory is the only way to have a fast experience. Paying the model loading cost repeatedly sucks.
  • jameshart 3y ago
    Why would linking llama.cpp into a UI application lead to incurring the model loading cost repeatedly?
    • mattnewton 3y ago
      It would be loaded repeatedly if the ui is opened and closed repeatedly. You can achieve the same “long running server + short running ui window” with multiple threads or processes all linked into one binary if you want of course. This way (with a separate server) seems simpler to me (and has the added benefit that multiple applications could easily call into the “server” if needed)