4 ms·
Note that they don't compare with deepseek coder 6.7b, which is vastly superior to much bigger coding models. Surpassing codellama 7b is not that big of a deal
by tarruda 3y ago
Note that they don't compare with deepseek coder 6.7b, which is vastly superior to much bigger coding models. Surpassing codellama 7b is not that big of a deal today.
The most impressive thing about these results is how good the 1.3B deepseek coder is.
- danielbln 3y agoDeepseek-coder-6.7B really is a quite surprisingly capable model. It's easy to give it a spin with ollama via `ollama run deepseek-coder:6.7b`.
- triyambakam 3y agoThanks for the tip with ollama
- behnamoh 3y agoollama is actually not a great way to run these models as it makes it difficult to change server parameters and doesn't use `mlock` to keep the models in memory.
- discordance 3y agoIf you do: 1. ollama run deepseek-coder:6.7b 2. pip install litellm 3. litellm --model deepseek-coder:6.7b You will have a local OpenAI compatible API for it.
- a_wild_dandan 3y agoThis is phenomenal. And runs fast! The 33b version might be my MacBook's new coding daily driver.
- tarruda 3y ago4-bit quantized 33b runs great on a mp pro with m3 max chip
- a_wild_dandan 3y agoI'm using the 5-bit quant with llama.cpp and it's excellent on my M2 96GB MacBook! Running this model + Mixtral will be fun.
- unshavedyak 3y agoHow are you using it? I need to find some sane way to use this stuff from Helix/terminal..
- a_wild_dandan 3y agoThere are many workflows, with hardware-dependent requirements. Three which work for my MacBook: 1. Clone & make llama.cpp. It's a CLI program that runs models, e.g. `./main -m <local-model-file.gguf> -p <prompt>`. 2. Another CLI option is `ollama`, which I believe can download/cache models for you. 3. A GUI like LM Studio provides a wonderful interface for configuring, and interacting with, your models. LM Studio also provides a model catalog for you to pick from. Assuming that your hardware is sufficient, options 1 & 2 should satisfy your terminal needs. Option 3 is an excellent playground for trying new models/configurations/etc. Models are heavy. To fit one in your silicon and run it quickly, you'll want to use a quantized model. It's a model's "distilled" version -- say 80% smaller for a 0.1% accuracy loss. TheBloke on HuggingFace is one specialist in distilling. After finding a model you like, you can download some flavor of quantization he made, e.g: `huggingface-cli download TheBloke/neural-chat-7B-v3-3-GGUF neural-chat-7b-v3-3.Q4_K_M.gguf --local-dir .`; then use your favorite model runner (e.g. llama.cpp) to run it. Hope that gets you started. Cheers!
- jyap 3y agoDeepseek Coder Instruct 6.7b has been my local LLM (M1 series MBP) for a while now and that was my first thought… They selectively chose benchmark results to look impressive (which is typical). I tested out StableLM Zephyr 3B when that came out and it was extremely underwhelming/unusable. Based on this, Stable Code 3B doesn’t look to be worth trying out. Guessing if they could put out a 7B model which beat Deepseek Coder 6.7B they would have.
- sroussey 3y agoWhat do you use it for?
- jyap 3y agoGolang grinding Leetcode coding buddy. And general coding buddy PR reviewer. The results are on par or better than ChatGPT 3.5. I often use it to delve deeper such as “is there an alternative way to write this?” Or “how does this code look?” If you have an M-series Mac I recommend trying out LM Studio. Really eye opening and I’m excited to see how things progress.
- a_wild_dandan 3y agoDo you know how Deepseek 33b compares to 6.7b? I'm trying 33b on my (96GB) MacBook just because I have plenty of spare (V)RAM. But I'll run the smaller model if the benefits are marginal in other peoples' experience.
- eyegor 3y agoThe 1.3b model is amazing for real time code complete, it's fast enough to be a better intellisense. Another model you should try is magicoder 6.7b ds (based on deepseek coder). After playing with it for a couple weeks, I think it gives slightly better results than the equivalent deepseek model. Repo https://github.com/ise-uiuc/magicoder https://github.com/ise-uiuc/magicoder Models https://huggingface.co/models?search=Magicoder-s-ds https://huggingface.co/models?search=Magicoder-s-ds
- hskalin 3y agoHow do you use these models with your editor? (E. vscode or Emacs etc)
- tarruda 3y agoAn easy way is to use an OpenAI compatible server, which means you can use any GPT plugin to integrate with your editor
- eyegor 3y agoI run tabby [0] which uses llama.cpp under the hood and they ship a vscode extension [1]. Going above 1.3b, I find the latency too distracting (but the highest end gpu I have nearby is some 16gb rtx quadro card that's a couple years old, and usually I'm running a consumer 8gb card instead). [0] https://tabby.tabbyml.com/ https://tabby.tabbyml.com/ [1] https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby https://marketplace.visualstudio.com/items?itemName=TabbyML....
- tarasglek 3y agoWould you mind sharing your tabby invocation to launch this model?
- varjag 3y agoNot sure what you guys are doing with it but even at 33B it's laughably dumb compared to Mixtral.