4 ms·
For anyone looking to try this out on consumer grade hardware here's a q4 version [0]. From initial testing it's subjectively a bit behind deepseek-coder-instru
by eyegor 3y ago
For anyone looking to try this out on consumer grade hardware here's a q4 version [0]. From initial testing it's subjectively a bit behind deepseek-coder-instruct [1] at the same size and quantization. Deepseeks model is near magic when it behaves (1-2 tries), spitting out nicely formatted code fence blocks in markdown that I can naively render in real time to get a local chatgpt like experience. Magicoder can do this too, but it usually takes 3-4 tries and it tends to "opt out" and ask for more info pretty frequently. Of course if you have a ton of vram, use the 33b-instruct variant instead.
After more testing, I think it's a toss up on most coding tasks but Magicoder tends to give subjectively better responses to "bad prompts". That is, prompts where you don't put effort into writing clear instructions. For example, one of my "bad prompt" tests is
> how to enable shared gpu memory in wsl2 docker container
A good response to this would discuss the nvidia container toolkit, maybe something about port forwarding, etc. But this isn't a prompt most models can give good responses to. Both of these models can handle it, even at 7b, but Magicoder gives more information.
[0] https://huggingface.co/LoneStriker/Magicoder-S-DS-6.7B-4.0bpw-h6-exl2 https://huggingface.co/LoneStriker/Magicoder-S-DS-6.7B-4.0bp...
[1] https://huggingface.co/bartowski/deepseek-coder-6.7b-instruct-exl2/tree/4_0 https://huggingface.co/bartowski/deepseek-coder-6.7b-instruc...
- Hedepig 3y ago> tends to give subjectively better responses to "bad prompts". I wonder if a first pass with another model to expand these so called bad prompts into better prompts would work.
- tarruda 3y ago> spitting out nicely formatted code fence blocks in markdown that I can naively render in real time to get a local chatgpt like experience. Not sure if understood, but in my experience almost every 7B instruct model does this if you add something like "respond with markdown" to the system prompt. Chatbot-UI (A ChatGPT UI clone) handles markdown nicely and does code rendering in real time.
- 3abiton 3y agoHow worth it though? From my understanding it is trained on GPT3 code snippet outputs, is the Q4 model any good?
- azeirah 3y agoFrom my experience when using llama.cpp, using min-p sampling gives far better results than the defaults, for example (note this is using deepseek-coder 33b) Prompt: "Write the fibonacci function in Python3" min_p 0.05 def fibonacci(n:int, memo={}) -> int: if n <=2 : return 1 # base case 1, 2 => 1 if n not in memo: memo[n] = fibonacci(n-1, memo) + fibonacci(n-2,memo) # store the result return memo[n] # Test cases print("Test case 1") print(f"The 5th Fibonacci is {fibonacci(5)} ") print(f"The 9th Fibonacci is {fibonacci(9)} ") Default sampling settings def fibonacci(n): if n == 0: return 0 elif n ==1: return 1 else : return (fibonacci(n - 1) + fibonacci(n -2)) n = int(input()) print(fibonacci(n)) Seed is the same. So if you're using these models locally, make sure you're using min-p sampling. All other samplers are genuinely very suboptimal. For more in-depth info about _why_: https://www.reddit.com/r/LocalLLaMA/comments/17vonjo/your_settings_are_probably_hurting_your_model_why/ https://www.reddit.com/r/LocalLLaMA/comments/17vonjo/your_se...