3 ms·
I've had the best luck with WizardLM-7B. It runs at ~4 tokens/sec on my M1 and has decent outputs for simple situations. While it can't handle complex refactors
by sestinj 3y ago
I've had the best luck with WizardLM-7B. It runs at ~4 tokens/sec on my M1 and has decent outputs for simple situations. While it can't handle complex refactors like gpt4, I believe there's room to improve both via prompting and codebase-specific fine-tuning.
And good point! I've just added this as the recommendation in the README.
- petespeed 3y agoI tried to change config.py to use ggml setting, but I did not see any request going to local llama-cpp.server. It keeps on asking for openai key. The local llama-cpp.server is up and I was able to make swagger calls for completion. Assuming that may be I missed something, I deleted ~/.continue and trying to start from scratch (vs code message: Starting Continue Server ...). I do see FastAPI is up on http://localhost:65432/ http://localhost:65432/ still vscode dialog shows same message. Where I can see the logs what its trying to do? What am I missing? Anyone else able to use local ggml?
- sestinj 3y agoHey there. I realized that the documentation is less clear than it should be on this, so I've made an update: https://continue.dev/docs/customization#local-models-with-ggml https://continue.dev/docs/customization#local-models-with-gg... The additional thing you need to do is set "default_model" to "ggml" in ~/.continue/config.py.