4 ms·
Not personally surprised that a MoE model performs so well. I used Mixtral a lot for coding Rust, and it had qualities no other model had except GPT 3.5 and la
by emporas 2y ago
Not personally surprised that a MoE model performs so well.
I used Mixtral a lot for coding Rust, and it had qualities no other model had except GPT 3.5 and later Claude Sonet. The funny thing is Mixtral was based on Llama 2 which was not trained on code that much.
DeepSeek v3: 671B parameters on total, and 37B activated sounds very good even though impossible to run locally.
Question if some people happen to know: For each query it activates just that many of parameters, 37B, and no more?
- coolspot 2y agoIt activates only 37B per query, but you don’t know which ones ahead of time, so you gotta store all 671B in (V)RAM.
- cma 2y agoBut you don't need cluster networking or nvlink so much like with splitting out llama 405B. You could even split them out with friends over internet levels of bandwidth.
- int_19h 2y agoMistral LMs are not LLaMA derivatives.
- wolfgangK 2y agoDeepSeek v3 can run on CPU & RAM : https://www.reddit.com/r/LocalLLaMA/comments/1hqidbs/deepseek_v3_running_on_llamacpp_wishes_you_a/ https://www.reddit.com/r/LocalLLaMA/comments/1hqidbs/deepsee... Epyc Gen4 and 12 memory channels of DDR5 @4800 should give you 7 to 9 t/s.