4 ms·
Ask HN: Deepseek v3 how can I check on local computer?
deepseek v3 how can I check on my local computer and is my hardware capable of running this model (64 GB RAM, and 4080 video card)
- dtnewman 2y agoCheck out exo labs blog. https://blog.exolabs.net/day-2/ https://blog.exolabs.net/day-2/
- lavren1974 2y agoOllama doesn't have this model yet.
- talldayo 2y agoYour computer can run it, but you have to layer the model across CPU and GPU memory. Your bottleneck will be PCIe speed, which probably won't be a huge issue for a 4080 on smaller quants.
- smallerize 2y agoHow are you going to fit 671 billion parameters into 64 + 16 gigs of RAM? That's less than one bit per parameter.
- talldayo 2y agoDeepseek v3 is an MoE model - not every parameter is activated at the same time.
- smallerize 2y agoBut different experts will probably be activated between every token. You will have to load data from disk to generate the next token. That's painfully slow.
- theGeatZhopa 2y agoWhat would you suggest instead? I also thinking about.
- smallerize 2y agoNot to be flippant, but paying for access to an API is going to be way more cost-effective than buying sufficient hardware to run this locally. You might try waiting to see if someone distills this model down to a smaller one, or sparsifies it by deleting a bunch of parameters and fine-tuning, or comes up with a 1.5-bit version. But you're going to need 2x RAM to run this one even very optimistically. 4x would give better quality outputs. And it will still be fairly slow unless you throw more GPU hardware at it.
- dcreater 2y agoShort answer: You cant run it locally. Its 670B parameters.. Long(er) answer: check the reddit thread on r/LocalLLaMa