2 ms·

Thank you!! Local model is something I'm looking into with gguf files & llama.cpp, still pretty experimental, you can check out the branch here: https://github.

by yoeven 2y ago

Thank you!! Local model is something I'm looking into with gguf files & llama.cpp, still pretty experimental, you can check out the branch here: https://github.com/JigsawStack/omiai/tree/feat/local-models https://github.com/JigsawStack/omiai/tree/feat/local-models

upghost 2y ago
Nice! To be clear for my usecase I didn't mean calling the local LLM directly, rather simply being able to point to an OpenAI compatible API would be fine. It seems like wrangling a native model would be a lot of extra complexity, but you seem to have a very good abstraction story here. Actually I probably will take a peek at the code to see how you are doing these abstractions layers, because the user facing API is certainly very clean...!