4 ms·

I use deepseek-coder-7b-instruct-v1.5 & DeepSeek-Coder-V2-Lite-Instruct when I want speed & codestral-22B-v0.1 when I want smartness. All of those are FIM capa

by trissi 2y ago

I use deepseek-coder-7b-instruct-v1.5 & DeepSeek-Coder-V2-Lite-Instruct when I want speed & codestral-22B-v0.1 when I want smartness. All of those are FIM capable, but especially deepseek-v2-lite is very picky with its prompt template so make sure you use it correctly... Depending on your hardware codestral-22B might be fast enough for everything, but for me it's a bit to slow... If you can run it deepseek v2 non-light is amazing, but it requires loads of VRAM

thot_experiment 2y ago
IIRC the codestral fim tokens aren't properly implemented in llama.cpp/ollama, what backend are you using to run them? id probably have to drop down to iq2_xxs or something for the full fat deepseek but I'll definitely look into codestral, I'm a big fan of mixtral, hopefully a MoE code model with FIM comes along soon. EDIT: nvm, my mistake looks like it works fine https://github.com/ollama/ollama/issues/5403 https://github.com/ollama/ollama/issues/5403