Woo! This sounds like it will make it easier to run in normal mode (ie, not interactive) and manage the chat history yourself if there's less penalty for a full

Woo! This sounds like it will make it easier to run in normal mode (ie, not interactive) and manage the chat history yourself if there's less penalty for a full program reload. Currently my perl irc bot wrapper for llama.cpp just open2's the program in interactive mode (-i) and reads/writes to stdout/stdin of llama.cpp to get the loading time savings of having it manage history and keep state. In one-shot there'd still be the "extra" inference time of processing the full history each time instead of saving state like interactive does but the memory load time matters just as much. For me personally this matters most because right now when llama.cpp runs out of the 2048 tokens it segfaults and this causes difficulties. In interactive mode if it goes off rails and generates 1000 tokens of nonsense then that nonsense is taking up tokens for the next line from chat. In normal mode where it just runs once and all history has to be manually supplied this can be avoided.