43 ms·
The point of gpt4all is that you can change the model with minimal breaking. You should be able to change this line https://github.com/khoj-ai/khoj/blob/master/
by syntaxing 3y ago
The point of gpt4all is that you can change the model with minimal breaking. You should be able to change this line https://github.com/khoj-ai/khoj/blob/master/src/khoj/processor/conversation/gpt4all/chat_model.py#L19 https://github.com/khoj-ai/khoj/blob/master/src/khoj/process... to the model you want. You'll need to build your own local image with docker-compose but should be relatively straight forward.
- sabaimran 3y agoYeah, the gpt4all project is super neat. If folks are inclined enough, it should be fairly straightforward for you to clone the Khoj project and swap out the model used. You'd have to update the model type in a few places, but should be easy enough just with normal string/keyword search. Then run it directly from inside your machine. You will, however, have to go in and modify the prompt structure to fit the model's expectation. Some guidance on that in this PR with Falcon: https://github.com/khoj-ai/khoj/pull/330/files#diff-7fa1139669b060850b065ee0791a3a8afee1e244fd3b2bbbe7918bf0be2314ed https://github.com/khoj-ai/khoj/pull/330/files#diff-7fa11396... I'll provide my insight from experimentation integrating Llama V2/GPT4All into Khoj -- Falcon 7b is probably the runner up in models that can be supported on consumer hardware, and it really wasn't good enough (for me) on my machine to be useful. The token consumption with personal notes context is too large, and the content too variable for a small model like that to be able to understand it. It's fine if you're just doing normal question-answering back and forth, but you don't need Khoj for that.