4 ms·

I don't get something. So I tried hosting this model myself. But the amount of minimum GPU RAM needed is 400gb+ Which even with the cheapest GPU providers wi

I don't get something. So I tried hosting this model myself. But the amount of minimum GPU RAM needed is 400gb+ Which even with the cheapest GPU providers will be at least USD 15/hour How is everyone running these models?

simonw 2y ago
Using the smaller distilled versions. I'm running this one, which only needs 20GB of VRAM (or regular RAM on Apple Silicon): https://ollama.com/library/deepseek-r1:32b https://ollama.com/library/deepseek-r1:32b
- punkpeye 2y ago
  Do we know how do these distilled versions perform in benchmarks?
  - simonw 2y ago
    DeepSeek published a bunch of benchmarks when they released the models: https://github.com/deepseek-ai/DeepSeek-R1?tab=readme-ov-file#4-evaluation-results https://github.com/deepseek-ai/DeepSeek-R1?tab=readme-ov-fil... I'd like to see detailed benchmarks run by other unaffiliated organizations.
    - punkpeye 2y ago
      This is very useful. Thank you. so basically there is not much reason to go beyond DeepSeek-R1-Distill-Qwen-32B, at least for coding tasks
      - punkpeye 2y ago
        Just had a chance to play around with 32B model https://glama.ai/models/deepseek-r1-distill-qwen-32b https://glama.ai/models/deepseek-r1-distill-qwen-32b I am using it with Cline VSCode extension to write code. It works impressively well for a model this size. Thanks again for sharing those benchmarks!
- cchance 2y ago
  Keep in mind the distilled versions are NOT shrunken versions of deepseek-r1 their just finetunes of Qwen and Llama i believe, and they are no where near as good as real r1 (the 400g version) or even the 133g quants.
rsanek 2y ago
you can also find the model via OpenRouter https://openrouter.ai/deepseek/deepseek-r1 https://openrouter.ai/deepseek/deepseek-r1
buyucu 2y ago
there are smaller distillations all the way down to 1.5b parameters. I'm running 7b on my laptop. https://ollama.com/library/deepseek-r1 https://ollama.com/library/deepseek-r1

simonw 2y ago

Using the smaller distilled versions. I'm running this one, which only needs 20GB of VRAM (or regular RAM on Apple Silicon): https://ollama.com/library/deepseek-r1:32b https://ollama.com/library/deepseek-r1:32b

punkpeye 2y ago
Do we know how do these distilled versions perform in benchmarks?
- simonw 2y ago
  DeepSeek published a bunch of benchmarks when they released the models: https://github.com/deepseek-ai/DeepSeek-R1?tab=readme-ov-file#4-evaluation-results https://github.com/deepseek-ai/DeepSeek-R1?tab=readme-ov-fil... I'd like to see detailed benchmarks run by other unaffiliated organizations.
  - punkpeye 2y ago
    This is very useful. Thank you. so basically there is not much reason to go beyond DeepSeek-R1-Distill-Qwen-32B, at least for coding tasks
    - punkpeye 2y ago
      Just had a chance to play around with 32B model https://glama.ai/models/deepseek-r1-distill-qwen-32b https://glama.ai/models/deepseek-r1-distill-qwen-32b I am using it with Cline VSCode extension to write code. It works impressively well for a model this size. Thanks again for sharing those benchmarks!
cchance 2y ago
Keep in mind the distilled versions are NOT shrunken versions of deepseek-r1 their just finetunes of Qwen and Llama i believe, and they are no where near as good as real r1 (the 400g version) or even the 133g quants.

rsanek 2y ago

you can also find the model via OpenRouter https://openrouter.ai/deepseek/deepseek-r1 https://openrouter.ai/deepseek/deepseek-r1

buyucu 2y ago

there are smaller distillations all the way down to 1.5b parameters. I'm running 7b on my laptop. https://ollama.com/library/deepseek-r1 https://ollama.com/library/deepseek-r1