3 ms·
IMO, it would be more interesting to have a 3-way comparison of price/performance between DeepSeek 671b running on : 1. M3 Ultra 512 2. AMD Epyc (which Gen ? A
by wolfgangK 2y ago
IMO, it would be more interesting to have a 3-way comparison of price/performance between DeepSeek 671b running on :
1. M3 Ultra 512
2. AMD Epyc (which Gen ? AVX512 and DDR5 might make a difference in both performance and cost , Gen 4 or Gen 5 have 8 or 9 t/s https://github.com/ggml-org/llama.cpp/discussions/11733 https://github.com/ggml-org/llama.cpp/discussions/11733 )
2. AMD Epyc + 4090 or 5090 running KTransformers (over 10 t/s decode ? https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#benchmark-results https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...)