4 ms·

How to Set Up DeepSeek with Ollama and Docker (Step-by-Step Guide)

16 points by sajiron 2y ago

sam_goody 2y ago
> Prerequisites: A system with sufficient GPU/CPU resources. The article doesn't give a hint what those resources are, but I assume that my macbook doesn't have it. So if I wanted to setup Ollama on digitalOcean [which offers both gpus and cpus], is there a good guide? How would that compare to running locally? How would that compare (privacy, quality, ease, etc) to huggingface or others?
- sajiron 2y ago
  Good point, I didn’t mention the hardware requirements in the article. Ollama runs best with a dedicated GPU, but it can work on CPUs (just much slower). If you're on a MacBook with an M1 or higher chip, it should work, but performance won’t be great. I haven’t set up Ollama on DigitalOcean yet, but since they offer GPU instances, it should be a solid option. I’ll test it and write a guide soon. Compared to running locally or using something like Hugging Face, DigitalOcean gives more power and flexibility but requires setup, while Hugging Face is easier but less customizable. I’ll share an update once I’ve tested the DigitalOcean setup
- K0balt 2y ago
  For most people, this guide will help you to run a DeepSeek trained distill on your machine, not the actual DSR1 model.
  - sajiron 2y ago
    Thanks for pointing that out! Yes, this guide is mainly for running a DeepSeek-trained distilled model, which I forgot to mention.
K0balt 2y ago
For people without 768gb+ of RAM and 64 cores+, this guide helps you set up systems to run the R1 distills, not the actual DeepSeek model itself which requires a pretty beefy machine to run. (It is possible to built an AMD epic based machine that can put out around 4 t/s on a reasonable q4 of DeepSeek R1 for <$3000, with CPU inference) The distills offer a similar type of functionality implemented in other models such as Quen and LLAMA3… but are emphatically -not- DSR1, which is a very efficient and powerful SOTA model in it’s own right, even without the R1 reasoning fine tuning. To be clear, for most people, this guide is not about running DeepSeek R1 , but rather about running DeepSeek trained distills of other popular models.

16 points by sajiron 2y ago

> Prerequisites: A system with sufficient GPU/CPU resources. The article doesn't give a hint what those resources are, but I assume that my macbook doesn't have it. So if I wanted to setup Ollama on digitalOcean [which offers both gpus and cpus], is there a good guide? How would that compare to running locally? How would that compare (privacy, quality, ease, etc) to huggingface or others?

sajiron 2y ago
Good point, I didn’t mention the hardware requirements in the article. Ollama runs best with a dedicated GPU, but it can work on CPUs (just much slower). If you're on a MacBook with an M1 or higher chip, it should work, but performance won’t be great. I haven’t set up Ollama on DigitalOcean yet, but since they offer GPU instances, it should be a solid option. I’ll test it and write a guide soon. Compared to running locally or using something like Hugging Face, DigitalOcean gives more power and flexibility but requires setup, while Hugging Face is easier but less customizable. I’ll share an update once I’ve tested the DigitalOcean setup
K0balt 2y ago
For most people, this guide will help you to run a DeepSeek trained distill on your machine, not the actual DSR1 model.
- sajiron 2y ago
  Thanks for pointing that out! Yes, this guide is mainly for running a DeepSeek-trained distilled model, which I forgot to mention.

K0balt 2y ago

For people without 768gb+ of RAM and 64 cores+, this guide helps you set up systems to run the R1 distills, not the actual DeepSeek model itself which requires a pretty beefy machine to run. (It is possible to built an AMD epic based machine that can put out around 4 t/s on a reasonable q4 of DeepSeek R1 for <$3000, with CPU inference) The distills offer a similar type of functionality implemented in other models such as Quen and LLAMA3… but are emphatically -not- DSR1, which is a very efficient and powerful SOTA model in it’s own right, even without the R1 reasoning fine tuning. To be clear, for most people, this guide is not about running DeepSeek R1 , but rather about running DeepSeek trained distills of other popular models.