4 ms·
I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak. If I were Phind, I'd be looking at Deepseek 33B instead. Whi
by brucethemoose2 3y ago
I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak.
If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.
- mewpmewp2 3y agoDoes this run on 4090 16gb vram? What's best that can run fast on 4090 laptop?
- brucethemoose2 3y agoYour options are: - Hybrid offloading with llama.cpp, but with slow inference. - Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context. 30B-34B is more of a sweetspot for 24GB of VRAM. If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.
- rushingcreek 3y agoWe've found that CodeLlama-70B is a much more capable base model than DeepSeek-33B. I'd love to hear your feedback on Phind-70B specifically.
- brucethemoose2 3y agoYeah I will have to test it out, though TBH I am more inclined to run models locally. As I mentioned, being such an extensive continuation train can (sometimes) totally change the capabilities of a model.
- rickette 3y agoDeepseek 33B is great. Also runs well on a modern (beefy) MBP.
- cvak 3y agodo you just run it from https://github.com/deepseek-ai/DeepSeek-Coder https://github.com/deepseek-ai/DeepSeek-Coder ?
- shapenamer 3y agoAfter running a bunch of models on my own PC (a pretty good one), I have to say by FAR the best results for coding has been with Deepseek models. However, I just spent 20 minutes playing with this Phind 70B model and it's totally nailing the questions I'm asking it. Pretty impressed.
- johnfn 3y agoIs this related to the post? Phind has introduced their own model. Codellama 70B isn't related to Phind's model, other than presumably the "70B" size.
- rushingcreek 3y agoPhind-70B is an extensive fine-tune on top of CodeLlama-70B
- brucethemoose2 3y agoYeah, and I'd go so far as to call it a continued pretrain with that many tokens. More like a whole new model than a traditional finetune.