3 ms·
I'll jump in before the haterade engine wakes up -- great bit of engineering work here! I can't imagine a better level of abstracting away the unnecessary stuff
by upghost 2y ago
I'll jump in before the haterade engine wakes up -- great bit of engineering work here! I can't imagine a better level of abstracting away the unnecessary stuff while still retaining that level of manual control.
The only thing I don't see is setup for local/in-house LLMs, but it's easy enough to spoof OpenAI calls if necessary.
- yoeven 2y agoThank you!! Local model is something I'm looking into with gguf files & llama.cpp, still pretty experimental, you can check out the branch here: https://github.com/JigsawStack/omiai/tree/feat/local-models https://github.com/JigsawStack/omiai/tree/feat/local-models
- upghost 2y agoNice! To be clear for my usecase I didn't mean calling the local LLM directly, rather simply being able to point to an OpenAI compatible API would be fine. It seems like wrangling a native model would be a lot of extra complexity, but you seem to have a very good abstraction story here. Actually I probably will take a peek at the code to see how you are doing these abstractions layers, because the user facing API is certainly very clean...!