3 ms·

> it’s certainly not able to be deployed in production Why not?

• sillysaurusx 4mo ago
  It’s a PITA to offer a language model as a service. You’d need a beefy server, at minimum. This particular use case might work, since no one can write fast enough to consume too many tokens — the whole session should fit in the context window. But you’ll need to handle all the people connecting to your service indefinitely, which will become expensive for a hobby project. But sure, theoretically you could deploy it if you have resources. I’m not sure what you’d use to create instances of chat sessions, or if llama.cpp offers an API you can build the app on top of (probably) or whether that’s a workable solution.
  • fragmede 4mo ago
    Throw it up on Openrouter?