7 ms·
Given these initial results, I'm now experimenting with running DeepSeek-R1-Distill-Qwen-32B for some coding tasks on my laptop via Ollama - their version of th
by simonw 2y ago
Given these initial results, I'm now experimenting with running DeepSeek-R1-Distill-Qwen-32B for some coding tasks on my laptop via Ollama - their version of that needs about 20GB of RAM on my M2. https://www.ollama.com/library/deepseek-r1:32b https://www.ollama.com/library/deepseek-r1:32b
It's impressive!
I'm finding myself running it against a few hundred lines of code mainly to read its chain of thought - it's good for things like refactoring where it will think through everything that needs to be updated.
Even if the code it writes has mistakes, the thinking helps spot bits of the code I may have otherwise forgotten to look at.
- miohtama 2y agoAlso even if the answer is incorrect, you can still cook the eggs on the laptop :)
- lawlessone 2y agoi spent a months salary on these eggs and can no longer afford to cook them :(
- the_arun 2y agoHey, where are you getting the eggs? I am unable to find them in the market.
- qingcharles 2y agoSold my GPU, bought chickens.
- belter 2y agoThe eggs cost more than the laptop...
- blakesterz 2y agoIs DeepSeek really that big of a deal that everyone else should worry?
- m11a 2y agoA lot of the niceness about DeepSeek-R1's usage in coding is that you can see the thought process, which (IME) has been more useful than the final answer. It may well be that o1's chain of thought reasoning trace is also quite good. But they hide it as a trade secret and supposedly ban users for trying to access it, so it's hard to know.
- m11a 2y agoOne example from today: I had a coding bug which I asked R1 about. The final answer wasn't correct, but adapting an idea from the CoT trace helped me fix the bug. o1's answer was also incorrect. Interestingly though, R1 struggled in part because it needed the value of some parameters I didn't provide, and instead it made an incorrect assumption about its value. This was apparent in the CoT trace, but the model didn't mention this in its final answer. If I wasn't able to see the trace, I'd not know what was lacking in my prompt, and how to make the model do better. I presume OpenAI kept their traces a secret to prevent their competitors from training models with it, but IMO they strategically err'd in doing so. If o1's traces were public, I think the hype around DS-R1 would be relatively less (and maybe more limited to the lower training costs and the MIT license, and not so much its performance and usefulness.)
- manmal 2y agoDo you use Continue.dev or similar tools to load code into the context, or do you copypaste into their web chat?
- gwd 2y ago> I presume OpenAI kept their traces a secret to prevent their competitors from training models with it At some point there was a paper they'd written about it, and IIRC the logic presented was like this: - We (the OpenAI safety people) want to be able to have insight into what o1 is actually thinking, not a self-censored "people are watching me" version of its thinking. - o1 knows all kinds of potentially harmful information, like how to make bombs, how to cook meth, how to manipulate someone, etc, which could "cause harm" if seen by an end-user So the options as they saw it were: 1. RLHF both the internal thinking and the final output. In this case the thought process would avoid saying things that might "cause harm", and so could be shown to the user. But they would have a less clear picture of what the LLM was "actually" thinking, and the potential state space of exploration would be limited due to the self-censorship. 2. Only RLHF the final output. In this case, they can have a clearer picture into what the LLM is "actually" thinking (and the LLM could potentially explore the state space more fully without risking about causing harm), but thought process could internally mention things which they don't want the user to see. OpenAI went with #2. Not sure what DeepSeek has done -- whether they have RLHF'd the CoT as well, or just not worried as much about it.
- lacedeconstruct 2y agoThe chain of thought is incredibly useful, I almost dont care about the answer now I just follow what I think is interesting from the way it broke the problem down, I tend to get tunnel vision when working for a long time on something so its a great way to revise my work and make sure I am not misunderstanding something
- rtsil 2y agoYesterday, I had it think for 194 seconds. At some point near the end, it said "This is getting frustrating!"
- bronco21016 2y agoI must not be hunting the right keywords but I was trying to figure this out earlier. How do you set how much time it “thinks”? If you let it run too long does the context window fill and it’s unable to do anymore?
- throwup238 2y agoIt looks like their API is OpenAI compatible but their docs say that they don’t support the `reasoning_effort` parameter yet. > max_tokens:The maximum length of the final response after the CoT output is completed, defaulting to 4K, with a maximum of 8K. Note that the CoT output can reach up to 32K tokens, and the parameter to control the CoT length (reasoning_effort) will be available soon. [1] [1] https://api-docs.deepseek.com/guides/reasoning_model https://api-docs.deepseek.com/guides/reasoning_model
- m3kw9 2y agoWhat does distil qwen 32b mean? It uses qwen for what?
- buyucu 2y agodeepseek fine-tuned qwen32b with data generated by deepseek671b
- buyucu 2y agoDeepSeek-R1-Distill-Qwen-32B is my new default model on my home server. previously it was aya-32b.
- xenospn 2y agoWhat do you use it at home for?
- whitehexagon 2y agoAgreed, I switched from qwq now to the same model. I'm running it under ollama on a M1 Asahi Linux and it seems maybe twice the speed (not very scientific but not sure how to time the token generation), and more, dare I say smarter? than qwq, and maybe a tad less RAM. It still over ponders, but not as bad as some of the pages and pages of, 'that looks wrong, maybe I should try...' circles with qwq, but which was already so impressive. I'm quite new to this, how are you feeding in so much text? just copy/paste? I'd love to be able to run some of my Zig code through it, but I haven't managed to get Zig running under Asahi so far.
- mike31fr 2y agoNoob question (I only learned how to use ollama a few days ago): what is the easiest way to run this DeepSeek-R1-Distill-Qwen-32B model that is not listed on ollama (or any other non-listed model) on my computer ?
- marpstar 2y agoI'm using it inside of LM Studio (https://lmstudio.ai https://lmstudio.ai), which has a "Discovery" tab where you can download models.
- nyrikki 2y agoollama run deepseek-r1:32b They dropped the Qwen/Llama terms from the string https://ollama.com/library/deepseek-r1 https://ollama.com/library/deepseek-r1
- rahimnathwani 2y agoWhenever they have an alias like this, they usually (always?) have a model with the same checksum but a more descriptive name, e.g. the checksum 38056bbcbb2d corresponds with both of these: https://ollama.com/library/deepseek-r1:32b https://ollama.com/library/deepseek-r1:32b https://ollama.com/library/deepseek-r1:32b-qwen-distill-q4_K_M https://ollama.com/library/deepseek-r1:32b-qwen-distill-q4_K... I prefer to use the longer name, so I know which model I'm running. In this particular case, it's confusing that they grouped the qwen and llama fine tunes with R1, because they're not R1.
- codingdave 2y agoIf you are specifically running it for coding, I'm satisfied with using it via continue.dev in VS Code. You can download a bunch of models with ollama, configure them into continue, and then there is a drop-down to switch models. I find myself swapping to smaller models for syntax reminders, and larger models for beefier questions. I only use it for chatting about the code - while this setup also lets the AI edit your code, I don't find the code good enough to risk it. I get more value from reading the thought process, evaluating it, and the cherry picking which bits of its code I really want. In any case, if that sounds like the experience you want and you already run ollama, you would just need to install the continue.dev VS Code extension, and then go to its settings to configure which models you want in the drop-down.
- brandall10 2y agoIf you have a bit more memory, use the 6 bit quant, takes up about 26gb and has been shown to be very minimally lossy as opposed to 4bit. Also serve it as MLX from LMStudio, will speed things up 30% or so so your 6bit will have similar perf to the 4bit. Getting about 12-13 tok/sec on my M3 Max 48gb.
- matwood 2y agoCan you link to the model you’re talking about? I can’t find the exact one using your description. Thanks!
- evrenesat 2y agohttps://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-32B-6bit https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwe...
- thomasskis 2y agoEXO is also great for running the 6bit deepseek, plus it’s super handy to serve from all your devices simultaneously. If your dev team all has M3 Max 48gb machines, sharing the compute lets you all run bigger models and your tools can point at your local API endpoint to keep configs simple. Our enterprise internal IT has a low friction way to request a Mac Studio (192GB) for our team and it’s a wonderful central EXO endpoint. (Life saver when we’re generally GPU poor)