3 ms·
"Continue works with any LLM, including local models using ggml" Which of the local models have you seen the best results with? UPDATE: Found a note here http
by simonw 3y ago
"Continue works with any LLM, including local models using ggml"
Which of the local models have you seen the best results with?
UPDATE: Found a note here https://github.com/continuedev/continue/blob/ce76e391775034c1faec3198f59b2d5f90f61150/docs/docs/customization.md#local-models-with-ggml https://github.com/continuedev/continue/blob/ce76e391775034c... that says "While these models don't yet perform as well, they are free, entirely private, and run offline." and points to the documentation here for how to run one, but doesn't really recommend a specific model: https://github.com/continuedev/ggml-server-example https://github.com/continuedev/ggml-server-example
- sestinj 3y agoI've had the best luck with WizardLM-7B. It runs at ~4 tokens/sec on my M1 and has decent outputs for simple situations. While it can't handle complex refactors like gpt4, I believe there's room to improve both via prompting and codebase-specific fine-tuning. And good point! I've just added this as the recommendation in the README.
- petespeed 3y agoI tried to change config.py to use ggml setting, but I did not see any request going to local llama-cpp.server. It keeps on asking for openai key. The local llama-cpp.server is up and I was able to make swagger calls for completion. Assuming that may be I missed something, I deleted ~/.continue and trying to start from scratch (vs code message: Starting Continue Server ...). I do see FastAPI is up on http://localhost:65432/ http://localhost:65432/ still vscode dialog shows same message. Where I can see the logs what its trying to do? What am I missing? Anyone else able to use local ggml?
- sestinj 3y agoHey there. I realized that the documentation is less clear than it should be on this, so I've made an update: https://continue.dev/docs/customization#local-models-with-ggml https://continue.dev/docs/customization#local-models-with-gg... The additional thing you need to do is set "default_model" to "ggml" in ~/.continue/config.py.