10 ms·
Complete hardware and software setup for running Deepseek-R1 locally
- nexus_six 2y agoWhat would the context length look like for this setup? How quick would the 6-7 tps degrade once you hit say 20k tokens?
- _giorgio_ 2y ago> the generation speed on this build is 6 to 8 tokens per second > ...if you want Q8 you'll need >700GB of GPU memory, which will probably cost $100k+
- alias_neo 2y agoI clicked through because that $6000 price tag seemed insane, achievable even. Now it makes sense. Still undecided how I feel about having the ability to use all that quality in the full size model if one could only retrieve it at 6-8 tokens per second.
- permanent 2y ago-- copied Complete hardware + software setup for running Deepseek-R1 locally. The actual model, no distillations, and Q8 quantization for full quality. Total cost, $6,000. All download and part links below: Motherboard: Gigabyte MZ73-LM0 or MZ73-LM1. We want 2 EPYC sockets to get a massive 24 channels of DDR5 RAM to max out that memory size and bandwidth. https://t.co/GCYsoYaKvZ https://t.co/GCYsoYaKvZ CPU: 2x any AMD EPYC 9004 or 9005 CPU. LLM generation is bottlenecked by memory bandwidth, so you don't need a top-end one. Get the 9115 or even the 9015 if you really want to cut costs https://t.co/TkbfSFBioq https://t.co/TkbfSFBioq RAM: This is the big one. We are going to need 768GB (to fit the model) across 24 RAM channels (to get the bandwidth to run it fast enough). That means 24 x 32GB DDR5-RDIMM modules. Example kits: https://t.co/pJDnjxnfjg https://t.co/pJDnjxnfjg https://t.co/ULXQen6TEc https://t.co/ULXQen6TEc Case: You can fit this in a standard tower case, but make sure it has screw mounts for a full server motherboard, which most consumer cases won't. The Enthoo Pro 2 Server will take this motherboard: https://t.co/m1KoTor49h https://t.co/m1KoTor49h PSU: The power use of this system is surprisingly low! (<400W) However, you will need lots of CPU power cables for 2 EPYC CPUs. The Corsair HX1000i has enough, but you might be able to find a cheaper option: https://t.co/y6ug3LKd2k https://t.co/y6ug3LKd2k Heatsink: This is a tricky bit. AMD EPYC is socket SP5, and most heatsinks for SP5 assume you have a 2U/4U server blade, which we don't for this build. You probably have to go to Ebay/Aliexpress for this. I can vouch for this one: https://t.co/51cUykOuWG https://t.co/51cUykOuWG And if you find the fans that come with that heatsink noisy, replacing with 1 or 2 of these per heatsink instead will be efficient and whisper-quiet: https://t.co/CaEwtoxRZj https://t.co/CaEwtoxRZj And finally, the SSD: Any 1TB or larger SSD that can fit R1 is fine. I recommend NVMe, just because you'll have to copy 700GB into RAM when you start the model, lol. No link here, if you got this far I assume you can find one yourself! And that's your system! Put it all together and throw Linux on it. Also, an important tip: Go into the BIOS and set the number of NUMA groups to 0. This will ensure that every layer of the model is interleaved across all RAM chips, doubling our throughput. Don't forget! Now, software. Follow the instructions here to install llama.cpp https://t.co/jIkQksXZzu https://t.co/jIkQksXZzu Next, the model. Time to download 700 gigabytes of weights from @huggingface! Grab every file in the Q8_0 folder here: https://t.co/9ni1Miw73O https://t.co/9ni1Miw73O Believe it or not, you're almost done. There are more elegant ways to set it up, but for a quick demo, just do this. llama-cli -m ./DeepSeek-R1.Q8_0-00001-of-00015.gguf --temp 0.6 -no-cnv -c 16384 -p "<｜User｜>How many Rs are there in strawberry?<｜Assistant｜>" If all goes well, you should witness a short load period followed by the stream of consciousness as a state-of-the-art local LLM begins to ponder your question: And once it passes that test, just use llama-server to host the model and pass requests in from your other software. You now have frontier-level intelligence hosted entirely on your local machine, all open-source and free to use! And if you got this far: Yes, there's no GPU in this build! If you want to host on GPU for faster generation speed, you can! You'll just lose a lot of quality from quantization, or if you want Q8 you'll need >700GB of GPU memory, which will probably cost $100k+
- ijk 2y agoI'd assume that the existing llama.cpp ability to split layers out to the GPU still applies, so you could have some fraction in VRAM and speed up those layers. The memory bandwidth might be an issue, and it would be a pretty small percentage of the model, but I'd guess the speedup would be apparent. Maybe not worth the few thousand for the card + more power/cooling/space, of course.
- SlavikCA 2y ago2x CPU system may be slower for LLM than 1x CPU system. Because in 2x CPU system, the model may have to be passed via NUMA, which has 10% - 30% of memory bandwidth bandwidth
- ryao 2y agoIt would be interesting to see the performance if someone built a single socket version. I have a parts list here if anyone wants to try it: https://news.ycombinator.com/item?id=42868360 https://news.ycombinator.com/item?id=42868360
- kfcjligmom 2y ago[dead]
- RistrettoMike 2y agoAmen.
- the_sleaze_ 2y agobehind the internet bubble was the modern internet.
- recursive 2y agoSick burn bro
- kfcjligmom 2y ago[dead]
- recursive 2y agoI can't find the video, but I have an idea what's going on. The quality of the output from these things is very inconsistent. Sometimes it seems to have surprising "insight". Sometimes it's incoherent nonsense. People that want to be impressed cherry-pick the good results. People that expect things to "just work" notice the duds more. You can find very good and very bad results. If you're a starry-eyed technologist, you'll publicize the good outliers because of the potential they represent. If you're a skeptic, you'll point out the seemingly brain-dead failures.
- __MatrixMan__ 2y agoThis was posted yesterday: https://news.ycombinator.com/item?id=42860770 https://news.ycombinator.com/item?id=42860770 And since X sucks today just about as much as it sucked yesterday: https://nitter.poast.org/carrigmat/status/1884244369907278106 https://nitter.poast.org/carrigmat/status/188424436990727810...
- alias_neo 2y agoI don't know if it's just me but I can't seem to view the nitter link, says the tweet isn't found, I can't use the OP Twitter link because I can't see anything without logging in.
- __MatrixMan__ 2y agoI ran across a thread once indicating that the guy who hosts that instance has a bit of a fight on his hands keeping it usable. Frequently fights hoards of bots, that sort of thing. Your signal probably got mistaken for some noise. We need Nitter and BitTorrent to have a baby, p2p so we can share the load.
- ForOldHack 2y agoThey did, and it works well ipfs.
- __MatrixMan__ 2y agoIPFS would be a good layer to build this on top of, but IPFS itself doesn't handle the correspondence between URL and CID. As it is, web DOMs are too variable to handle via content addressing. This would involve eliminating parts of the page that are unlikely to be stable (i.e. ads, pagination based on screen size, session specific details like usernames), and use that for the CID. That way the only time that users end up storing separate copies of the page is when it has actually been changed substantially. You'd probably also need some code to fluff the normalized content back up into something that makes sense for the device you're reading it on.
- horsawlarway 2y ago6 to 8 t/s is decent for a GPU-less build, but speaking from experience... it will feel slow. Usable - but slow. Especially because these models <think> for a long bit before actually answering, so they generate for a longer period. (to be clear - I find the <think> section useful, but it also means waiting for more tokens) Personally - I end up moving down to lower quality models (quants/less params) until I hit about 15 tokens/second. For chat, that seems to be the magical spot where I stop caring and it's "fast enough" to keep me engaged. For inline code helpers (ex - copilot) you really need to be up near 30 tokens/second to make it feel fast enough to be helpful.
- k__ 2y agoWould DeepSeek-V3 be cheaper?
- sigmoid10 2y agoOnly in the sense that it will generate fewer tokens.
- guerrilla 2y agoThis is the same I get from a Core i5-9400 for smaller models. Is there no prosumer board that can take that much RAM? There must be a ThreadRipper that can do it, right? Why did he need EPYC?
- hollerith 2y agoIt's not just the amount of RAM: RAM bandwidth also matters.
- UncleOxidant 2y agoRAM channels is key here. That board has 24 channels of DDR5 RAM. A lot of lowend boards only have 2 channels. Most come in at 4. Some high end boards have 8, so 24 channels is quite wide.
- UncleOxidant 2y ago
- sgt101 2y agoHmm - if only macpro's had memory that could be upgraded like in the olden days.
- SV_BubbleTime 2y agoIf it could, it wouldn’t be so tightly coupled and then identical to everything else out there.
- bitwize 2y agoIt also wouldn't be as fast. Soldered RAM is faster than socketed RAM. RAM that is in the CPU package is faster still. Apple is going in the direction of total integration. Louis Rossmann and iFixit will hem and haw, but soon there will be a MacBook whose motherboard, besides cooling, PSU, and ports, consists simply of a single component that houses CPU, GPU, RAM, storage, I/O port controllers, radios (wifi, Bluetooth, etc.), firmware for all of the above, and a security module plus keys, all directly on the CPU bus, and it will be glorious. It will absolutely lap any PC laptop, and in single-core performance will smoke even high-end AMD Epyc beastbox builds because of the aggressive elimination of inter-component latency. But there won't be a need to fix it. If it breaks you just recycle it and buy new, being sure to sync your data back from Apple Cloud -- but it probably won't break. Kinda like how unibody cars are both safer and more reliable than the much more fixable cars of the 60s, even if they crumple like tinfoil and must be totaled upon experiencing any sort of impact.
- menaerus 2y agoApple M1 Max LPDDR6-6400 measured latency is 111 ns so it's not true that soldered RAM is faster than socketed RAM. Source: first figure from https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc...
- sgt101 2y agoI wonder if that's really set in stone or just a stance that suited Apples marketing for a few years. Maybe there's a business case for some bigger memory Macs now!
- drooby 2y agoThe amount of compute required for R1 to determine that strawberry has 3 Rs is hilarious.. I'm not so convinced that the Nvidia panic is justified.
- disgruntledphd2 2y agoI mean, to be fair, given the way tokenization works, this isn't that surprising. I do agree that it's very funny though.
- cookingrobot 2y agoI think it’s like asking someone “how many times does your pen change direction when writing the word strawberry on paper”. You need to think really hard to get to an answer, because that’s more fine grained than the way you usually think about words and letters.
- layer8 2y agoI wondered how I should feel about that thought process. However, I might have a similar thought process when inebriated.
- ceejayoz 2y agoThe Nvidia panic stems, in part, from the possibility that this is just the first of many potential significant optimization leaps.
- drooby 2y agoIdgi, optimization leaps are virtually guaranteed. I think this is a deeper question about the bounds of human desire... which seem virtually limitless. We seem to have an unlimited appetite for answering questions on the complexity of existence. Pair that with arms race issues, and you have an obvious need for massive compute regardless of how efficient the algorithms are.
- kovacs 2y ago
- indeed30 2y agoSo, can somebody in the know speculate about how Deepseek (or OpenAI, or whoever really) is actually running their API? If I wanted to run a production-grade service using the full Deepseek model, with good tokens/sec and the ability to serve concurrent requests, what sort of hardware are we looking at?
- MurkyLabs 2y agoRacks and Racks of servers (likely nVidia HGX H100/H200 8-GPU server) connected at at least 100GB (but more likely 400gb and 800gb) links. The servers alone start at about $350k. Then you need to supply power, cooling, networking and a technical team to support the program.
- ssahoo 2y agoI have been using deepseek-r1:1.5b/8b on a MacBook m1 pro max. The performance has been pretty good, at par with the o1. So last 2 days i have been running them side by side. I'm satisfied with the results and performance. That's a not very cheap 2k hardware and it does the job.
- mysteria 2y agoWould adding a single GPU help with prompt processing here? When I run a llama.cpp GPU build with no layers offloaded prompt processing is still way faster as all the matrix multiplies are done on the accelerator. The actual memory bound inference continues to run on the CPU. Since tensor cores are so fast you still come out ahead when you send the weights over PCIE to the GPU and return the completed products back to main memory.
- ryao 2y agoMaybe. It likely depends on whether GEMM can operate at/near full speed with streaming weights over PCI-E. I am not sure how to stream the weights over PCI-E for use by CUDA/PTX code offhand. It would be a R&D project.
- ComputerGuru 2y agoCPU-only is, very unfortunately, infeasible for reasoning models. This setup would be great for deepseek v3 or (more fittingly) the 405B llama 3.1 model, but 6-7 tokens per second on a reasoning model is 100% getting (well) into seconds-per-token territory if you consider only the final answer. (You don’t have to take it from me: if CPU were good enough, AMD’s valuation would be 100x its current value.)
- samvher 2y agoGiven what we just saw in terms of the DeepSeek team squeezing a lot of extra performance out of more efficient implementation on GPU, and the model still being optimized for GPU rather than CPU - is it unreasonable to think that in the $6k setup described, some performance might still be left on the table that could be squeezed out with some better optimization for these particular CPUs?
- telotortium 2y agoMaybe a little, but FLOPs and memory bandwidth don't lie.
- snovv_crash 2y agoNo, because the bottleneck is RAM bandwidth. This is already quantized and otherwise is essentially random so can't be compressed in any meaningful way.
- niwtsol 2y agoIs anyone aware of a site that shows various builds or off the shelf systems (new and old) and how they handle various models? Like I’d love to see the above vs a Mac Studio 198gb vs an old m1 studio vs other models. I don’t have $6000, but what is a good happy medium I could get to on a used system with a smaller model.
- mv4 2y agonot everyone has X access, so [0] direct link with login https://x.com/carrigmat/status/1884244369907278106 https://x.com/carrigmat/status/1884244369907278106 [1] alt link without login but with ads https://threadreaderapp.com/thread/1884244369907278106.html https://threadreaderapp.com/thread/1884244369907278106.html Edit: someone posted xcancel link above - no ads https://xcancel.com/carrigmat/status/1884244369907278106 https://xcancel.com/carrigmat/status/1884244369907278106
- Jotalea 2y agoMy Ryzen 7 3700U is giving it all with the 7b model.
- 999900000999 2y agoI'm so hyped for this. It's going to take some time, but the farce is gone. We'll have parity to Chat GPT on consumer hardware soon enough. 6k is still too much. I suspect the community will be able to get this down to 2K. I'm tempted to cancel my Chat GPT subscription!
- seanp2k2 2y agoOrders of magnitude make the difference. “Consumer-level” will be once it’s around $300-700ish in a nice little box like an Intel NUC or similar. Ubiquiti just did this with their AI-Key to help classify stuff on their video camera surveillance platform.
- 999900000999 2y ago2K is reasonable. Not every single person needs to have it. But if someone in your circle has the needed hardware...
- binary132 2y agoMy theory is that in the future we will have much more “friends circle cloud” type ops, where that hardware cost is spread out among a small community and access is private. What it won’t look like is every Tom, Dick, and Harry running their own $10k hardware to have a chuckle at the naughty jokes and grade-C+ programmer IDE assistance offered by open-source LLMs.
- swiftcoder 2y agoDoes anyone have the performance delta between running this on a 768 GB setup like this where the whole thing fits in RAM, versus running it on an M4 Mac with the maxed out 128 GB?
- qingcharles 2y agoMy other question is.. can you jam 768GB of RAM into an M2 Ultra Mac Pro? And M4 Ultra Mac Pros are probably only weeks away too.
- qingcharles 2y agoOK, well, turns out you can't upgrade the RAM in the M2 Mac Pros and they top out at 192GB spec from factory, so that's that.
- ryao 2y agoIt might be possible to desolder the chips and solder larger capacity ones, but it is a risky thing to do, especially since there is no guarantee that it will work out. That said, a similar upgrade has been done on the raspberry pi 4, so it is theoretically possible: https://hackaday.com/2023/03/05/upgrade-ram-on-your-pi-4-the-fun-way/ https://hackaday.com/2023/03/05/upgrade-ram-on-your-pi-4-the...
- twoodfin 2y agoNo, you cannot. All M-series Macs to date—including the Mac Pro—have RAM fixed at manufacturing.
- oynqr 2y agoRunning the ollama 671b 4 bit quant on a 7950X3D with 128GiB RAM, I get like 1-2 t/s.
- monobot12 2y agoIf you don't mind a speed of 1 token per second, you can run the largest R1 model on a 2021 iMac, as I just did.
- 42772827 2y agoIs deepseek r-1 as fashioned here censored?
- m3kw9 2y agoThis is more like an experiment more than practical use with the stated 6 toks/s. Paying 6gs for that and days of setup when the next model may come in a month
- JonChesterfield 2y agoWorking through this now. The directions are to download the contents of the Q8_0 at https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main. That turns out to be a git lfs repo. `git clone` followed by `git lfs pull` is downloading all of it which will have to do. (fetchinclude = DeepSeek-R1-Q8_0 seems to be limiting it to the directory of interest). If there's a cleverer way to get the files please reply - I looked for a torrent and failed to find one. Not completely clear what changing numa nodes per socket from 1 to 0 does, possibly gives linux less information about when to migrate threads across x64 cores? (didn't upset llvm compile time so I'll leave it on nsp0)
- kgwgk 2y agoCan’t you use the download icons in each object at https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q8_0 https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De... ?
- JonChesterfield 2y agoThe icons resolve to things like https://huggingface.co/unsloth/DeepSeek-R1-GGUF/resolve/main/DeepSeek-R1-Q8_0/DeepSeek-R1.Q8_0-00002-of-00015.gguf https://huggingface.co/unsloth/DeepSeek-R1-GGUF/resolve/main... which wget understands. Presumably there's a greater-than-average risk of corruption in transit when the files are big and git does some sort of integrity checking that one would lose out on? It's the verify-local-data feature I'm really missing from torrent here.
- wg0 2y agoOr - Have your own "OpenAI" at home. Train, fine-tune, distill in the cloud if and when necessary. Now this is basically the moment of Apache/ngnix being free and open source. You then have share hosting phenomenon out of it.
- morphle 2y agoAs this is HN, I'm curious if there is anyone here on HN who is interested in starting a business hosting these large open source LLM's? I just finished a test of running this Deepseek-R1 768 GB model locally on a cluster of computers with 800 GB/s memory bandwidth (faster than the machine in the twitter post) and I can now extrapolate to a cluster with 6000 GB/s aggregate memory bandwidth and I'm sure we can reach higher speeds than Groq and Cerebras [1] on these large models. We might even be cheap enough in OPEX to retrain these models. Would anyone with cofounder or commercial skills be willing to set up this hosting service with me, it will take less than $30K investment but could be profitable in weeks? [1] https://hc2024.hotchips.org/assets/program/conference/day2/72_HC2024.Cerebras.Sean.v03.final.pdf https://hc2024.hotchips.org/assets/program/conference/day2/7...
- ForOldHack 2y agoThe Azure(tm) and AWS version of rent-a-second are in the works as we speak. So yes, rent-a-brain/vegetable and no, I will bet you $40k you will not beat either AWS ot Microsoft to the punch. Zero chance of that. They will have their excess computational power with extremely discounted electric rates in place before Friday morning.
- morphle 2y agoI think the important metric will be if we can compete against the price of AWS or Microsoft in running large LLMs, not their time to market. Competing on cost against overpriced hyperscalers is not very hard, and $30K is a small investment, not a gamble. If it would fail, worst case you would only loose $3000-$6500 or so.
- DoingIsLearning 2y ago> $40K is a small investment, not a gamble. If it would fail, worst case you would only lose $3000-$6500 or so. As someone not familiar with investment sourcing or SME financing. Could you break down the maths/accounting? How do you go from sinking 40k in a business to losing 6.5k if you turn the lights off at the end?
- ForOldHack 2y ago$6000. Oh that's impressive concedering my gaming PC cost all of $1100. What a deal! Can I get two? Or three? How about training? Is that free? Air? Is air free? Does the economic model ( profit, profita and profit ) depend on stealing published works? Is that intelligent? Once the brain dead greedy MBAs get involved, is just how much you can steal. It should all be sold short, as we watch the world burn.
- culi 2y agoHope you're doing okay... $6k is much less than the millions that would be required to run anything by OpenAI. And it's a first pass. It could get much lower by the end of the year
- erichocean 2y agoUnless I'm missing something, the full 1TB bandwidth isn't used because the memory layout is wrong. But that's fixable. Since it's memory bound, it might be possible to reach 15 tok/sec with this build.
- r14c 2y agoI'm curious, what's wrong with the memory layout? You mean the ollama settings?
- deleted 2y ago[deleted]
- tantalor 2y agoOff topic, but I'd be fine with not seeing direct links to Twitter on here anymore. It's not very useful or user friendly. Similar to Pinterest, Instagram, or TikTok. Screenshots or mirrors (without the login requirement) are okay.
- dkjaudyeqooe 2y agoA "rollup" of a twitter thread through a 3rd party service is a good solution
- ryao 2y agoLike this? https://threadreaderapp.com/thread/1884244369907278106.html https://threadreaderapp.com/thread/1884244369907278106.html
- faeyanpiraat 2y agoThis has more ads than content
- deleted 2y ago[deleted]
- tingle 2y agoReplacing 'twitter' in the URL by 'xcancel' can be helpful: https://xcancel.com/carrigmat/status/1884244369907278106 https://xcancel.com/carrigmat/status/1884244369907278106
- joecool1029 2y agopoast's nitter instance works as well: https://nitter.poast.org/carrigmat/status/1884244369907278106 https://nitter.poast.org/carrigmat/status/188424436990727810... (I use this firefox extension to automatically rewrite twitter links into it: https://addons.mozilla.org/en-US/firefox/addon/privacy-redirect/ https://addons.mozilla.org/en-US/firefox/addon/privacy-redir... )
- ryao 2y agoA better setup on paper would be this CPU and motherboard combination with 12x64GB DIMMs: https://www.newegg.com/p/N82E16819113866 https://www.newegg.com/p/N82E16819113866 https://www.newegg.com/supermicro-h13ssl-nt-amd-epyc-9004-series-processors/p/N82E16813183820 https://www.newegg.com/supermicro-h13ssl-nt-amd-epyc-9004-se... As for memory, these two kits should work (both are needed for the full 12 DIMMs): https://www.newegg.com/owc-256gb/p/1X5-005D-001G0 https://www.newegg.com/owc-256gb/p/1X5-005D-001G0 https://www.newegg.com/owc-512gb/p/1X5-005D-001G4 https://www.newegg.com/owc-512gb/p/1X5-005D-001G4 Since it would be a 2DPC configuration, the memory would be limited to 4400MT/sec unless you overclock it. That would give 422.4GB/sec, which should be enough to run the full model at 11 tokens per second according to a simple napkin math calculation. In practice, it might not run that fast. If the memory is overclocked, getting to 16 tokens per second might be possible (according to napkin math). The subtotal for the linked parts alone is $5,139.98. It should stay below $6000 even after adding the other things needed, although perhaps it would be more after tax. Note that I have not actually built this to know how it works in practice. My description here is purely hypothetical.
- bildung 2y agoI think the point of the two socket solution is the doubled memory bandwith. You propose using just a single one of the same CPU, or am I missing something?
- ryao 2y agollama.cpp’s token generation speed does not scale with multiple CPU sockets just like it does not scale with multiple GPUs. Matthew Carrigan wrote: > Also, an important tip: Go into the BIOS and set the number of NUMA groups to 0. This will ensure that every layer of the model is interleaved across all RAM chips, doubling our throughput. Don't forget! This does not actually make sense. It is well known that there is a penalty for accessing memory attached to a different CPU. You don’t get more bandwidth from disabling the NUMA node information and his token generation performance reflects that. If there was a doubling effect from using two CPU sockets, he should be getting twice the performance, but he is not. Additionally, llama.cpp’s NUMA support is suboptimal, so he is likely taking a performance hit: https://github.com/ggerganov/llama.cpp/issues/11333 https://github.com/ggerganov/llama.cpp/issues/11333 When llama.cpp fixes its NUMA support, using two sockets should be no worse than using one socket, but it will not become better unless some new way of doing the calculations is devised that benefits from NUMA. This might be possible (particularly if you can get GEMV to run faster using NUMA), but it is not how things are implemented right now.
- mrbonner 2y agoIs there a build that would allow me to run llama 3.3 local? Something around $2500 or below.
- sheepscreek 2y agoUpdate: Scratch that. Two of them together would only be able to run something half as big (~400B parameters) and cost as much as this rig. Maybe the next gen of DIGITS could do it. Keeping that in mind, this rig is pretty darn impressive for $6k! This is going to drop by half when Nvidia starts shipping DIGITS. I think we’re all going to want one. It’ll probably have a much bigger impact than Apple VisionPro, that costs the same. I can already think of using it as a much more intelligent local Siri/Alexa to control devices. It’s something that can actually keep the kids engaged with useful trivia/knowlege (better than watching mindless trash on YT) or it can just humour me whenever I want - all without needing to worry about privacy.
- UncleOxidant 2y ago> "Complete hardware + software setup for running Deepseek-R1 locally. The actual model, no distillations, and Q8 quantization for full quality. Total cost, $6,000." These guys say they got really good results with 2.51-bit quantization of the original R1. The original has 671B params weighing in at 720GB - that's what they're running on this $6000 setup. According to these guys[0] they get really good results at 2.51bit quantization which would be this model[1] which is still 671B params, but weighs in at 212GB. [0] https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic [1] https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
- segmondy 2y agoThat's not what they are running. the dynamic quants are Q1, Q2. The $6000 is running Q8.
- UncleOxidant 2y agoYes, exactly. What I'm saying is that the blog referenced in [0] says you can get good results with 2.51 quantization. The $6000 rig is running Q8. You can probably get similar results with a lesser rig if you use the quantized model.
- benlivengood 2y agoDo you actually need dual EPYCs? You can get Lenovo T620s on eBay for $1500 and fill them up with 1TB of RAM
- hollerith 2y agoYes, you need dual epycs: https://news.ycombinator.com/item?id=42868067 https://news.ycombinator.com/item?id=42868067
- Larrikin 2y agoBluesky link: https://bsky.app/profile/carrigmat.bsky.social/post/3lgsoqsxip224 https://bsky.app/profile/carrigmat.bsky.social/post/3lgsoqsx...