3 ms·

I see you’re using gpt4all; do you have a supported way to change the model being used for local inference? A number of apps that are designed for OpenAI’s com

I see you’re using gpt4all; do you have a supported way to change the model being used for local inference? A number of apps that are designed for OpenAI’s completion/chat APIs can simply point to the endpoints served by llama-cpp-python [0], and function in (largely) the same way, while using the various models and quants supported by llama.cpp. That would allow folks to run larger models on the hardware of their choice (including Apple Silicon with Metal acceleration or NVIDIA GPUs) or using other proxies like openrouter.io. I enjoy openrouter.io myself because it supports Anthropic’s 100k models. [0]: https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python

110 3y ago
No, we don't yet. Lots of developer folks want to try different models, we want to provide simple to use, but deep assistance. Kind of unsure what to focus on given our limited resources.
- vunderba 3y ago
  I really like the idea of running a dedicated server that serves up various large language models via a standardized API, and then Khoj could just be pointed at one. Depending on the notes and the type of conversation I want to have, that would even allow for Khoj to swap models on the fly.
  - sabaimran 3y ago
    We're planning to work on this!
syntaxing 3y ago
The point of gpt4all is that you can change the model with minimal breaking. You should be able to change this line https://github.com/khoj-ai/khoj/blob/master/src/khoj/processor/conversation/gpt4all/chat_model.py#L19 https://github.com/khoj-ai/khoj/blob/master/src/khoj/process... to the model you want. You'll need to build your own local image with docker-compose but should be relatively straight forward.
- sabaimran 3y ago
  Yeah, the gpt4all project is super neat. If folks are inclined enough, it should be fairly straightforward for you to clone the Khoj project and swap out the model used. You'd have to update the model type in a few places, but should be easy enough just with normal string/keyword search. Then run it directly from inside your machine. You will, however, have to go in and modify the prompt structure to fit the model's expectation. Some guidance on that in this PR with Falcon: https://github.com/khoj-ai/khoj/pull/330/files#diff-7fa1139669b060850b065ee0791a3a8afee1e244fd3b2bbbe7918bf0be2314ed https://github.com/khoj-ai/khoj/pull/330/files#diff-7fa11396... I'll provide my insight from experimentation integrating Llama V2/GPT4All into Khoj -- Falcon 7b is probably the runner up in models that can be supported on consumer hardware, and it really wasn't good enough (for me) on my machine to be useful. The token consumption with personal notes context is too large, and the content too variable for a small model like that to be able to understand it. It's fine if you're just doing normal question-answering back and forth, but you don't need Khoj for that.

110 3y ago

No, we don't yet. Lots of developer folks want to try different models, we want to provide simple to use, but deep assistance. Kind of unsure what to focus on given our limited resources.

vunderba 3y ago
I really like the idea of running a dedicated server that serves up various large language models via a standardized API, and then Khoj could just be pointed at one. Depending on the notes and the type of conversation I want to have, that would even allow for Khoj to swap models on the fly.
- sabaimran 3y ago
  We're planning to work on this!

syntaxing 3y ago

The point of gpt4all is that you can change the model with minimal breaking. You should be able to change this line https://github.com/khoj-ai/khoj/blob/master/src/khoj/processor/conversation/gpt4all/chat_model.py#L19 https://github.com/khoj-ai/khoj/blob/master/src/khoj/process... to the model you want. You'll need to build your own local image with docker-compose but should be relatively straight forward.

sabaimran 3y ago
Yeah, the gpt4all project is super neat. If folks are inclined enough, it should be fairly straightforward for you to clone the Khoj project and swap out the model used. You'd have to update the model type in a few places, but should be easy enough just with normal string/keyword search. Then run it directly from inside your machine. You will, however, have to go in and modify the prompt structure to fit the model's expectation. Some guidance on that in this PR with Falcon: https://github.com/khoj-ai/khoj/pull/330/files#diff-7fa1139669b060850b065ee0791a3a8afee1e244fd3b2bbbe7918bf0be2314ed https://github.com/khoj-ai/khoj/pull/330/files#diff-7fa11396... I'll provide my insight from experimentation integrating Llama V2/GPT4All into Khoj -- Falcon 7b is probably the runner up in models that can be supported on consumer hardware, and it really wasn't good enough (for me) on my machine to be useful. The token consumption with personal notes context is too large, and the content too variable for a small model like that to be able to understand it. It's fine if you're just doing normal question-answering back and forth, but you don't need Khoj for that.