3 ms·

I've been using [continue](https://continue.dev/ https://continue.dev/) alongside Ollama. My go-to llm has been [deepseek-coder 7b](https://ollama.com/library/d

I've been using [continue](https://continue.dev/ https://continue.dev/) alongside Ollama. My go-to llm has been [deepseek-coder 7b](https://ollama.com/library/deepseek-coder https://ollama.com/library/deepseek-coder). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.

manca 3y ago
I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.
- 0x008 3y ago
  I tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.
  - 3abiton 3y ago
    Same, CodeLlama 70B is known to suck. Deepseek is the best for coding so far in my experience, Mixtral 8x7B is another great contender (to be frank, for most tasks). Miqu is making a buzz, but so far I haven't tested it personally yet.
  - gpjt 3y ago
    What quantization were you using? I've been getting some weird results with 34b quantized to 4 bits -- glitching, dropped tokens, generating Java rather than Python as requested. But 7b, even at 4 bits, works OK. Posted about it earlier on this evening: https://www.gilesthomas.com/2024/02/llm-quantisation-weirdness https://www.gilesthomas.com/2024/02/llm-quantisation-weirdne...
jwells89 3y ago
deepseek-coder 6.7b is seriously impressive for how quickly it runs on an M1 Max. There’s a few spots where it still doesn’t fare quite as well as ChatGPT but it’s a small tradeoff considering that it’s fully local and doesn’t even spin up my laptop’s fans.
anticensor 3y ago
No markdown here sadly.
ranguna 3y ago
Have you tried dolphin mistral 7B 2.6? Part of its training data was code.

manca 3y ago

I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.

0x008 3y ago
I tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.
- 3abiton 3y ago
  Same, CodeLlama 70B is known to suck. Deepseek is the best for coding so far in my experience, Mixtral 8x7B is another great contender (to be frank, for most tasks). Miqu is making a buzz, but so far I haven't tested it personally yet.
- gpjt 3y ago
  What quantization were you using? I've been getting some weird results with 34b quantized to 4 bits -- glitching, dropped tokens, generating Java rather than Python as requested. But 7b, even at 4 bits, works OK. Posted about it earlier on this evening: https://www.gilesthomas.com/2024/02/llm-quantisation-weirdness https://www.gilesthomas.com/2024/02/llm-quantisation-weirdne...

jwells89 3y ago

deepseek-coder 6.7b is seriously impressive for how quickly it runs on an M1 Max. There’s a few spots where it still doesn’t fare quite as well as ChatGPT but it’s a small tradeoff considering that it’s fully local and doesn’t even spin up my laptop’s fans.

anticensor 3y ago

No markdown here sadly.

ranguna 3y ago

Have you tried dolphin mistral 7B 2.6? Part of its training data was code.