11 ms·

How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server

463 points by walterbell 2y ago

wanse_yung 2y ago
[flagged]
- leobg 2y ago
  This is a verbatim quote from Jason Calacanis.
- deleted 2y ago
  [deleted]
teruakohatu 2y ago
If you are going to go to that effort, adding a second NMVE drive, and doing RAID 0 across them, will improve the speed of getting the model into RAM.
- merb 2y ago
  than you will way more memory, since you can only do software raid a nvme drive.
  - davidmurdoch 2y ago
    About how much memory overhead would that require?
  - mgerdts 2y ago
    I’ve found that striping across two drives like the 980 Pro described here or WD SN850 Black drives easily gets direct IO read speeds over 12 GB/s on threadripper pro systems. This assumes a stripe size somewhere around 1 -2 MiB. This means that most reads will not need to be split and high queue depth sequential reads and random reads keep both drives busy. With careful alignment of IOs, performance approaches 2x of one drive’s performance. IO takes CPU cycles but I’ve not seen evidence that striping impacts that. Memory overhead is minimal, as the stripe to read from is done via simple math from a tiny data structure.
    - merb 2y ago
      well this really depends on what you use mdraid/zfs and as you said alignment. but only for reads. if you just use dumb mdraid (without too much optimizations configured) (raid 10) + xfs and read/write on top of it you will end up with quite a high memory + cpu usage. but i/o still will be insanly fast. I've written this more that it works, but its not a put another drive in and done solution. but if you just want to dumb a second drive into it and use mdraid/zfs you will have an overhead. of course if somebody tunes it and builds the application around it you can trim down the overhead significantly.
H8crilA 2y ago
Do we have any estimate on the size of OpenAI top of the line models? Would they also fit in ~512GB of (V)RAM? Also, this whole self hosting of LLMs is a bit like cloud. Yes, you can do it, but it's a lot easier to pay for API access. And not just for small users. Personally I don't even bother self hosting transcription models which are so small that they can run on nearly any hardware.
- exe34 2y ago
  it would make sense if you don't want somebody else to have access to all your code and customer data.
- yapyap 2y ago
  Is the size of OpenAI‘s top of the line models even relevant? Last I checked they weren’t open source in the slightest.
- GreenWatermelon 2y ago
  It's nice because a company can optionaly provide a SOTA reasoning model for their clients, without having to go through a middleman e.g. HR co. Can provide an LLM for their HRMS system for a small 2000$ investment. Not 2000/month, just a one time 2000 investment.
  - NeutralCrane 2y ago
    No one will be doing anything practical with a local version of Deepseek on a $2000 server. The token throughout of this thing is like, 1 token every 4 seconds. It would take nearly a full minute just to produce a standard “Roses are red, violets are blue” poem. There’s absolutely no practical usage that you can use that for. It’s cool that you can do it, and it’s a step in the right direction, but self-hosting these wont be a viable alternative to using providers like OpenAI for business applications for a while.
    - bbzylstra 2y ago
      The OP said they were getting 3-4 TPS on their $2000 rig.
buyucu 2y ago
related reddit thread: https://old.reddit.com/r/LocalLLaMA/comments/1idseqb/deepseek_r1_671b_over_2_toksec_without_gpu_on/ https://old.reddit.com/r/LocalLLaMA/comments/1idseqb/deepsee...
christophilus 2y ago
Aside: it’s pretty amazing what $2K will buy. It’s been a minute since I built my desktop, and this has given me the itch to upgrade. Any suggestions on building a low-power desktop that still yields decent performance?
- Havoc 2y ago
  >Any suggestions on building a low-power desktop that still yields decent performance? You don't for now. The bottleneck is mem throughput. That's why people using CPU for LLM are running xeon-ish/epyc setups...lots of mem channels. The APU class gear along the lines of Halo Strix is probably the path closest to lower power but it's not going to do 500gb of ram and still doesn't have enough throughput for big models
- baobun 2y ago
  Hard to know what ranges you have in mind with "decent performance" and "low-power". I think your best bet might be a Ryzen U-series mini PC. Or perhaps an APU barebone. The ATX platform is not ideal from a power-efficiency perspective (whether inherently or from laziness or conspiracy from mobo and PSU makers, I do not know). If you want the flexibility or scale, you pay the price of course but first make sure it's what you want. I wouldn't look at discrete graphics unless you have specific needs (really high-end gaming, workstation, LLMs, etc) - the integrated graphics of last few years can both drive your 4k monitors and play recent games at 1080p smoothly, albeit perhaps not simultaneously ;) Lenovo Tiny mq has some really impressive flavors (ECC support at the cost of CPU vendor-lock on PRO models) and there's the whole roster of Chinese competitors and up-and-comers if you're feeling adventerous. Believe me you can still get creative if you want to scratch the builder itch - thermals is generally what keeps these systems from really roaring (:
- spaceport 2y ago
  Not to be that yt'r that shills my videos all over, but you did ask for a low powered desktop build and this $350 one I put together is still my favorite. The 3060 12GB with llama 3.2 vision 11b is a very fun box that is low idle power (intel rules) to leave on 24/7 and have it run some additional services like HA. https://youtu.be/iflTQFn0jx4 https://youtu.be/iflTQFn0jx4
1ba9115454 2y ago
I can't imagine this setup will get more than 1 token per second. I would love to see Deepseek running on premise with a decent TPS.
- thomquaid 2y ago
  It says 4.25 TPS in the first para.
  - ricardobeat 2y ago
    Honest mistake. Some people think HN is just a series of short tweets and haven’t realized they are links yet!
    - thomquaid 2y ago
      4.25 is enough tps for a lot of use cases.
    - 4ndrewl 2y ago
      It's the modern way. Why read when you can just imagine facts straight out of your own brain.
      - plagiarist 2y ago
        I agree but also found your comment funny in the context of LLMs. People love getting facts straight out of their models.
  - weatherlight 2y ago
    That's still pretty slow, considering there's that "thinking" phase.
    - thomquaid 2y ago
      True, but 4.25 is the number we all want to know.
sylware 2y ago
Well, I read this, now I am sure: as of today, deepseek handling of LLMs is the less wrong, and by far.
geertj 2y ago
This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM. This [1] X thread runs the 671B model in the original Q8 at 6-8 TPS for $6K using a dual socket Epyc server motherboard using 768GB of RAM. I think this could be made cheaper by getting slower RAM but since this is RAM bandwidth limited that would likely reduce TPS. I’d be curious if this would just be a linear slowdown proportional to the RAM MHz or whether CAS latency plays into it as well. [1] https://x.com/carrigmat/status/1884244369907278106?s=46&t=5DsSie6D9vxgUafSFmo6EQ https://x.com/carrigmat/status/1884244369907278106?s=46&t=5D...
- nielsole 2y ago
  I've been running the unsloth 200GB dynamic quantisation with 8k context on my 64GB Ryzen 7 5800G. CPU and iGPU utilization were super low, because it basically has to read the entire model from disk. (Looks like it needs ~40GB of actual memory that it cannot easily mmap from disk) With a Samsung 970 Evo Plus that gave me 2.5GB/s read speed. That came out at 0.15 tps Not bad for completely underspecced hardware. Given the model has only so few active parameters per token (~40B), it is likely that just being able to hold it in memory absolve the largest bottleneck. I guess with a single consumer PCIe4.0x16 graphics card you could get at most 1tps just because of the PCIe transfer speed? Maybe CPU processing can be faster simply because DDR transfer is faster than transfer to the graphics card.
  - deleted 2y ago
    [deleted]
  - TeMPOraL 2y ago
    To add another datapoint, I've been running the 131GB (140GB on disk) 1.58 bit dynamic quant from Unsloth with 4k context on my 32GB Ryzen 7 2700X (8 cores, 3.70 GHz), and achieved exactly the same speed - around 0.15 tps on average, sometimes dropping to 0.11, tps occasionally going up to 0.16 tps. Roughly 1/2 of your specs, roughly 1/2 smaller quant, same tps. I've had to disable the overload safeties in LM Studio and tweak with some loader parameters to get the model to run mostly from disk (NVMe SSD), but once it did, it also used very little CPU! I tried offloading to GPU, but my RTX 4070 Ti (12GB VRAM) can take at most 4 layers, and it turned out to make no difference in tps. My RAM is DDR4, maybe switching to DDR5 would improve things? Testing that would require replacing everything but the GPU, though, as my motherboard is too old :/.
brunohaid 2y ago
Still surprised that the $3000 NVIDIA Digits doesn’t come up more often in that and also the gung-ho market cap discussion. I was an AI sceptic until 6 months ago, but that’s probably going to be my dev setup from spring onwards - running DeepSeek on it locally, with a nice RAG to pull in local documentation and datasheets, plus a curl plugin. https://www.nvidia.com/en-us/project-digits/ https://www.nvidia.com/en-us/project-digits/
- fake-name 2y ago
  It'll probably be more relevant when you can actually buy the things. It's just vaporware until then.
  - brunohaid 2y ago
    Call me naive, but I somehow trust them to deliver in time/specs? It’s also a more general comment around „AI desktop appliance“ vs homebuilts. I’d rather give NVIDIA/AMD $3k for a well adjusted local box than tinkering too much or feeding the next tech moloch, and have a hunch I’m not the only one feeling that way. Once it’s possible of course.
    - fake-name 2y ago
      Oh, if it's anything close to what they claim, I'll probably buy one as well, but I certainly do not expect them to deliver on time.
      - Cane_P 2y ago
        DIGITS isn't that impressive... It is a RTX 5070 Ti laptop GPU (992 TOPS, clocked less than 1% higher, to reach 1000 TOPS/1 PFLOP. As a reference RTX 5090 desktop have 3352 TOPS, more than 3x...), with 128 GB of unified memory. Just because Jensen calls it a super computer and gives it a DGX-1 design, doesn't make it one. In the Cleo Abram interview [1], Jensen said that DIGITS is 6 times more powerful than the first DGX-1. According to this PDF [2], DGX-1 had 170 TFLOPS of FP16 (half precision). 170x6=1020 TFLOP (~1 PFLOP). Yes DIGITS is suppose to have 1 PFLOP, but according to the presentation, it should be in FP4... He also said that it will draw 10k times less power. But DGX-1 had a TDP of 3.5kW [3] and I highly doubt DIGITS will draw 3500/10000=0.35W... the GPU alone will have a peak TDP that is more like 200 times higher than that. I mean, we all know that NVIDIA does fudge the numbers in charts. Like comparing FP8 from last generation, to FP4 on this. But this is extreme. Having said that. Do I believe that they can deliver a laptop (in another form factor) and it will perform 1 PFLOP of FP4. Of course! Like I said, it is nothing special. Both Apple and AMD have unified memory in relatively cheap systems. 1. https://youtu.be/7ARBJQn6QkM https://youtu.be/7ARBJQn6QkM 2. http://images.nvidia.com/content/technologies/deep-learning/pdf/Datasheet-DGX1.pdf http://images.nvidia.com/content/technologies/deep-learning/... 3. https://images.nvidia.com/content/pdf/dgx1-v100-system-architecture-whitepaper.pdf https://images.nvidia.com/content/pdf/dgx1-v100-system-archi...
cma 2y ago
He's running quantized Q4 671b. However, MoE doesn't need cluster networking so you could probably run the full thing on two of them unquantized. Maybe the router could be all resident in GPU RAM instead of in contrast offloading a larger percentage of everything there, or is that already how it is set up in his gpu offload config?
sAggymEllenZ 2y ago
[dead]
notsylver 2y ago
I think it would be more interesting doing this with smaller models (33b-70b) and see if you could get 5-10 tokens/sec on a budget. I've desperately wanted something locally thats around the same level of 4o, but I'm not in a hurry to spend $3k on an overpriced GPU or $2k on this
- firtoz 2y ago
  Would it be something like this? > OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi https://x.com/geerlingguy/status/1884994878477623485 https://x.com/geerlingguy/status/1884994878477623485 I haven't tried it myself or haven't verified the creds, but seems exciting at least
  - gliptic 2y ago
    That's 1.2 t/s for the 14B Qwen finetune, not the real R1. Unless you go with the GPU with the extra cost, but hardly anyone but Jeff Geerling is going to run a dedicated GPU on a Pi.
  - etra0 2y ago
    it's using a Raspberry Pi with a.... USD$1k gpu, which kinda defeat the purpose of using the RPI in the first place imo. or well, I guess you save a bit on power usage.
    - firtoz 2y ago
      Oh, I was naive to think that the Pi was capable of some kind of magic (sweaty smile emoji goes here)
    - unethical_ban 2y ago
      I suppose it makes sense, for extremely GPU centric applications, that the pi be used essentially as a controller for the 3090.
- gliptic 2y ago
  Your best bet for 33B is already having a computer and buying a used RTX 3090 for <$1k. I don't think there's currently any cheap options for 70B that would give you >5. High memory bandwidth is just too expensive. Strix Halo might give you >5 once it comes out, but will probably be significantly more than $1k for 64 GB RAM.
isoprophlex 2y ago
Online, R1 costs what, $2/MTok? This rig does >4 tok/s, which is ~15-20 ktok/hr, or $0.04/hr when purchased through a provider. You're probably spending $0.20/hr on power (1 kW) alone. Cool achievement, but to me it doesn't make a lot of sense (besides privacy...)
- rightbyte 2y ago
  > Cool achievement, but to me it doesn't make a lot of sense (besides privacy...) I would argue that is enough and that this is awesome. It was a long time ago I wanted to do a tech hack like this much.
  - isoprophlex 2y ago
    Well thinking about it a bit more, it would be so cool if you could A) somehow continuously interact with the running model, ambient-computing style. Say have the thing observe you as you work, letting it store memories. B) allowing it to process those memories when it chooses to/whenever it's not getting any external input/when it is "sleeping" and C) (this is probably very difficult) have it change it's own weights somehow due to whatever it does in A+B. THAT, in a privacy friendly self-hosted package, i'd pay serious money for
    - Willingham 2y ago
      I imagine it could solve crimes if it watched millions of hours of security footage…scary thought. Possibly it could arrest us before we even commit a crime through prediction like that black mirror episode.
      - falcor84 2y ago
        Oh, you're thinking of "Hated in the Nation"? More relevant would possibly be "Minority Report" (set in 2054) and Hulu's "Class of '09", in which the FBI starts deploying a crime prediction AI in their version of 2025. Quite scary. As the meme has it, it seems that we're getting ready to create the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus.
alecco 2y ago
Affiliate link spam.
- codetrotter 2y ago
  No. Affiliate link spam is when someone fills a page with content they stole, or it’s a bunch of nonsense that is stuffed to the brim with keywords, and they combine that with affiliate links. Someone getting a dollar or two in return for you following an affiliate link after you read something they put real time and effort into to make it valuable info for others is not “affiliate link spam”.
  - jjallen 2y ago
    I’m fine with useful content linking to affiliate links too. I am still confused at the ram specs required and those they linked to being off by a factor of 2 though. IF the setup is not realistic or accurate then that wouldn’t be cool.
    - alecco 2y ago
      Few people would spend $6k to run a model locally on CPU. But lower it to $2k and you might get some sweet affiliate link commissions. And make it fuzzy so they don't get it is running so quantized it probably is useless.
  - alecco 2y ago
    This is a lame me-too page to make money with affiliate links. The actual specs were in the linked original tweet. So, IMHO, it is affiliate link spam.
almostgotcaught 2y ago
Those EPYCs he's advertising as $700 are either engineering samples or used. New he's off by 2:1.
zajio1am 2y ago
> 512GB 2400 ECC RAM $400 Is this really that cheap? Looking at several local (CZ) eshops, i cannot find 32 GB DDR4 ECC RDIMM cheaper than $75, which will be $1200 for 512 GB.
- formerly_proven 2y ago
  Used server hardware is much more expensive in the EU generally, because the market is much smaller (fewer data centers to begin with, longer cycles to reduce costs and EU WEEE mandatory scrapping instead of reuse).
  - deleted 2y ago
    [deleted]
jjallen 2y ago
He links to a ram kit that is 8x 32Gb but says it should have 512 Gb ram. What gives? Also with 8 ram slots you would obviously need more than 256. Is the setup disingenuous to get people excited about the post or what is going on here?
- dave7 2y ago
  The board recommended has 16 ram slots.
  - jjallen 2y ago
    The ram kit is still 8x32gb so the price is lower than it actually would be.
deleted 2y ago
[deleted]
ilaksh 2y ago
Has anyone run any benchmarks on the quantized (non-distilled) versions?
pama 2y ago
What is the fastest documented way so far to serve the full R1 or V3 models (Q8, not Q4) if the main purpose is inference with many parallel queries and maximizing the total tokens per sec? Did anyone document and benchmark efficient distributed service setups?
- manmal 2y ago
  The top comment in this thread mentions a 6k setup, which likely could be used with vLLM with more tinkering. AFAIK vLLM‘s batched inference is great.
- snovv_crash 2y ago
  You need enough VRAM to hold the whole thing plus context. So probably a bunch of H100s, or MI300s.
jeswin 2y ago
Maybe Intel or AMD should bring back a Larrabee style CPU which can use say 48 sockets of DDR/CAMM2 sticks.
easygenes 2y ago
This is neat, but what I really want to see is someone running it on 8x 3090/4090/5090 and what is the most practical configuration for that.
- deoxykev 2y ago
  8x 3090 will net you around 10-12tok/s
  - bick_nyers 2y ago
    It would not be that slow as it is an MoE model with 37b activated parameters. Still, 8x3090 gives you ~2.25 bits per weight, which is not a healthy quantization. Doing bifurcation to get up to 16x3090 would be necessary for lightning fast inference with 4bit quants. At that point though it becomes very hard to build a system due to PCIE lanes, signal integrity, the volume of space you require, the heat generated, and the power requirements. This is the advantage of moving up to Quadro cards, half the power for 2-4x the VRAM (top end Blackwell Quadro expected to be 96GB).
    - deoxykev 2y ago
      Yeah, there is a clear bottleneck somewhere in llama.cpp. Even high end hardware is struggling to get good numbers. The theoretical limit should be higher, but it's not yet. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/11474#issuecomment-2629590835 https://github.com/ggerganov/llama.cpp/issues/11474#issuecom...
  - rdlw 2y ago
    Is it possible that eight graphics cards is the most practical configuration? How do you even set that up? I guess server mobos have crazy numbers of PCIe slots?
- MaxikCZ 2y ago
  I have been searching for a single example of someone running it like this (or 8x P40 and alike), and found nothing..
crispyambulance 2y ago
Kind of embarrassed to ask, I use AI a lot, I haven't really understood how the nuts and bolts work (other than at a 5th-grader 30000ft level)... So, when I use a "full" AI like chatGPT4o, I ask it questions and it has a firm grip on a vast amount of knowledge, like, whole-internet/search-engine scope knowledge. If I run an AI "locally", on even a muscular server, it obviously does NOT have vast amounts of stored information about everything. So what use is it to run locally? Can I just talk to it as though it were a very smart person who, tragically, knows nothing? I mean, I suppose I could point it to a NAS box full of pdf's and ask questions about that narrow range of knowledge, or maybe get one of those downloaded wikipedia stores. Is that what folks are doing? It seems like you would really need a lot content for the AI to even be remotely useable like the online versions.
- londons_explore 2y ago
  Most of an AI's knowledge is inside the weights, so when you run it locally, it has all that knowledge inside! Some AI services allow the use of 'tools', and some of those tools can search the web, calculate numbers, reserve restaurants, etc. However, you'll typically see it doing that in the UI. Local models can do that too, but it's typically a bit more setup.
- throwaway3b03 2y ago
  I've also been away from the tech (and AI scene) for a few years now. And I mostly stayed away from LLMs. But I'm certain that all the content is baked into the model, during training. When you query the model locally (since I suppose you don't train it yourself), you get all that knowledge that's baked into the model weights. So I would assume the locally queried output to be comparable with the output you get from an online service (they probably use slightly better models, I don't think they release their latest ones to the public).
- MuffinFlavored 2y ago
  I ask this all the time. locally running an LLM seems super hobbyist to me. like tweaking terminal font sizes on fringe BSD distros kind of thing
assimpleaspossi 2y ago
Are there any security concerns over DeepSeek as there are over TikTok?
- Kuinox 2y ago
  It's a local model, what security concern would you have ?
  - jappgar 2y ago
    I think this is unlikely but a local model could generate malicious code. You would have to run it manually though.
  - dinosaurdynasty 2y ago
    They also have an app that connects to their datacenter with R1. Also barely anyone can actually run the real R1 locally.
    - Kuinox 2y ago
      We are speaking about a 2k$ server here.
- Svoka 2y ago
  Local model - no. Using deepseek.com - absolutely. Do not put anything private there.
huijzer 2y ago
What is a bit weird about AI currently is that you basically always want to run the best model, but the price of the hardware is a bit ridiculous. In the 1990s, it was possible to run Linux on scrappy hardware. You could also always run other “building blocks” like Python, Docker, or C++ easily. But the newest AI models require an order of magnitude more RAM than my system or the systems I typically rent have. So I’m curious to people here, has this in the history of software happened before? Maybe computer games are a good example. There people would also have to upgrade their system to run the latest games.
- svilen_dobrev 2y ago
  well, if there was e.g. a model trained for coding - i.e. specialization as such, having models trained mostly for this or that - instead of everything incl. Shakespeare, the kitchen sink and the cockroaches biology under it, that would make those runable on much low level hardware. But there is only one, The-Big-Deal.. in many incarnations.
- spamizbad 2y ago
  Like AI, there were exciting classes of applications in the 70s, 80s and 90s that mandated pricier hardware. Anything 3D related, running multi-user systems, higher end CAD/EDA tooling, and running any server that actually got put under “real” load (more than 20 users). If anything this isn’t so bad: $4K in 2025 dollars is an affordable desktop computer from the 90s.
  - lukeschlather 2y ago
    The thing is I'm not that interested in running something that will run on a $4K rig. I'm a little frustrated by articles like this, because they claim to be running "R1" but it's a quantized version and/or it has a small context window... it's not meaningfully R1. I think to actually run R1 properly you need more like $250k. But it's hard to tell because most of the stuff posted is people trying to do duct tape and bailing wire solutions.
    - adenta 2y ago
jonotime 2y ago
I'm also kind of new to this and coming from coding with ChatGPT. Isnt the time to first token important? He is sitting there for minutes waiting for a response. Shouldnt that be a concern?
- HarHarVeryFunny 2y ago
  I'd rather wait to get a good response, than get a quick response that is much less useful, and it's the nature of these "reasoning" models that they reason before responding. Yesterday I was comparing DeepSeek-R1 (NVidia hosted version) with both Sonnet 3.5 (regarded by most as most capable coder) and the new Gemini 2.0 flash, and the wait was worth it. I was trying to get all three to create a web page with a horizontally scrolling timeline with associated clickable photos... Gemini got to about 90% success after half a dozen prompts, after which it became a frustrating game of whack-a-mole trying to get it to fix the remaining 10% without introducing new bugs - I gave up after ~30min. Sonnet 3.5 looked promising at first, generating based on a sketch I gave it, but also only got to 90%, then hit daily usage limit after a few attempts to complete the task. DeepSeek-R took a while to generate it, but nailed it on first attempt.
  - jonotime 2y ago
    Interesting. So in my use, I rarely see gpt get it right on the first pass but thats mostly due to interpretation of the question. I'm ruling out the times when it hallucinates calls to functions that dont exist. Lets say I ask for some function that calculates some matrix math in python. It will spit out something but I dont like what it did. So I will say, now dont us any calls to that library you pulled in, and also allow for these types of inputs. Add exception handling... So response time is important since its a conversation, no matter how correct the response is. When you say deep seek "nailed it on the first attempt" do you mean it was without bugs? Or do you mean it worked how you imagined? Or what exactly?
    - HarHarVeryFunny 2y ago
      DeepSeek-R generated a working web page on first attempt, based on a single brief prompt I gave it. With Sonnet 3.5, given the same brief prompt I gave DeepSeek-R, it took a half dozen feedback steps to get to 90%. Trying a hand drawn sketch input to Sonnet instead was quicker - impressive first attempt, but iterative attempts to fix it failed before I hit the usage limit. Gemini was the slowest to work with, and took a lot of feedback to get to the "almost there" stage, after which it floundered. The AI companies seem to want to move in the direction of autonomous agents (with reasoning) that you hand a task off to that they'll work on while you do something else. I guess that'd be useful if they are close to human level and can make meaningful progress without feedback, and I suppose today's slow-responding reasoning models can be seen as a step in that direction. I think I'd personally prefer something fast enough responding to use as a capable "pair programmer", rather than an autonomous agent trying to be an independent team member (at least until the AI gets MUCH better), but in either case being able to do what's being asked is what matters. If the fast/interactive AI only gets me 90% complete (then wastes my time floundering until I figure out it's just not capable of the task), then the slower but more capable model seems preferable as long as it's significantly better.
waynenilsen 2y ago
Someone please productize this
- loxias 2y ago
  "Okay" Can you describe for me what the product does? It's not that hard to make a turnkey "just add power" appliance that does nothing but spit out tokens. Some sort of "ollama appliance", which just sits on your network and provides LLM functionality for your home lab? But beyond that, what would your mythical dream product do?
  - waynenilsen 2y ago
    i'm talking about something you plug in and get on your network then you can chat with it with a typical chat interface and it should expose an openai compatible api automatic updates etc etc the AI home appliance great for privacy
- gatienboquet 2y ago
  https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox
jonotime 2y ago
Any idea what the power draw is? Resting power, vs resting with the model loaded in memory, vs full power computation. In case you may want to also run this as your desktop for basic web browsing etc when not LLMing.
waltercool 2y ago
Interesting and surprising to see, but 3-4 t/s is not practical overall. Anything lower than 10 t/s is going to give lot of wait time considering the reasoning wait time
jauntywundrkind 2y ago
Is there any hope that desktops get 64GB dimms? 48GB dimms have been a nice boost but are we gonna get more anytime soon? I'd love so much if quad-channel Strix Halo could get up to 256gb of memory. 192GB (4x48) won't be too bad, and at 8533MT/s, should provide competitive-ish throughput to these massive Epyc systems. Of course the $6k 24 channel 3200MHz is 4.4x more throughput but it does have a field of dimms to get there, and high power consumption.
la64710 2y ago
Basically three components : ollama , openwebui and deepseek. Nice!
sroussey 2y ago
If you really want to cheap—run it in your browser via webgpu.
- sroussey 2y ago
  I got downvoted, but you really can run it in the browser. ONNX version: https://huggingface.co/onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNX https://huggingface.co/onnx-community/DeepSeek-R1-Distill-Qw...
- zubiaur 2y ago
  Not comparable. That is a quantized distill.
  - sroussey 2y ago
    Oh yes, for sure. Don’t mean otherwise.
lhl 2y ago
Last fall I built a new workstation with an EPYC 9274F (24C Zen4 4.1-4.3GHz, $2400), 384GB 12 x 32GB DDR5-4800 RDIMM ($1600), and a Gigabyte MZ33-AR0 motherboard. I'm slowly populating with GPUs (including using C-Payne MCIO gen5 adapters), not focused on memory, but I did spend some time recently poking at it. I spent extra on the 9274F because of some published benchmarks [1] that showed that the 9274F had STREAM TRIAD results of 395 GB/s (on 460.8 GB/s of theoretical peak memory bandwidth), however sadly, my results have been nowhere near that. I did testing with LIKWID, Sysbench, and llama-bench, and even w/ an updated BIOS and NUMA tweaks, I was getting <1/2 the Fujitsu benchmark numbers: Results for results-f31-l3-srat: { "likwid_copy": 172.293857421875, "likwid_stream": 173.132177734375, "likwid_triad": 172.4758203125, "sysbench_memory_read_gib": 191.199125, "llama_llama-2-7b.Q4_0": { "tokens_per_second": 38.361456, "model_size_gb": 3.5623703002929688, "mbw": 136.6577115303955 } } For those interested in all the system details/running their own tests (also MLC and PMBW results among others): https://github.com/AUGMXNT/speed-benchmarking/tree/main/epyc-mbw-testing https://github.com/AUGMXNT/speed-benchmarking/tree/main/epyc... [1] https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-performance-report-primergy-rx1440-m2-ww-en.pdf https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-perfor...
- menaerus 2y ago
  Assuming that you populated the channels correctly, which I believe you did, I can only think that this issue could be related to the motherboard itself or RAM. I think you could start by measuring the single-core RAM bandwidth and latency. Since the CPU is clocked quite high, figures you should be getting are I guess around ~100ns, but probably less than that, and 40-ish GB/s of BW. If those figures do not match then it could be either a motherboard (HW) or BIOS (SW) issue or RAM stick issue. If those figures closely match then it's not a RAM issue but a motherboard (BIOS or HW) and you could continue debugging by adding more and more cores to the experiment to understand at which point you hit the saturation point for the bandwidth. It could be a power issue with the mobo.
  - lhl 2y ago
    Yeah, that channels are populated correctly. As you can see from the mlc-results.txt, the latency looks fine: mlc --idle_latency Intel(R) Memory Latency Checker - v3.11b Command line parameters: --idle_latency Using buffer size of 1800.000MiB Each iteration took 424.8 base frequency clocks ( 104.9 ns) As does the per-channel --bandwidth_matrix results: Numa node Numa node 0 1 2 3 4 5 6 7 0 45999.8 46036.3 50490.7 50529.7 50421.0 50427.6 50433.5 52118.2 1 46099.1 46129.9 52768.3 52122.3 52086.5 52767.6 52122.6 52093.4 2 46006.3 46095.3 52117.0 52097.2 50385.2 52088.5 50396.1 52077.4 3 46092.6 46091.5 52153.6 52123.4 52140.3 52134.8 52078.8 52076.1 4 45718.9 46053.1 52087.3 52124.0 52144.8 50544.5 50492.7 52125.1 5 46093.7 46107.4 52082.0 52091.2 52147.5 52759.1 52163.7 52179.9 6 45915.9 45988.2 50412.8 50411.3 50490.8 50473.9 52136.1 52084.9 7 46134.4 46017.2 52088.9 52114.1 52125.0 52152.9 52056.6 52115.1 I've tried various NUMA configurations (from 1 domain to a per-CCD config) and it doesn't seem to make much difference. Updating from the board-delivered F14 to the latest 9004 F31 BIOS (the F33 releases bricked the board and required using a BIOS flasher for manual recover) gave marginal (5-10%) improvement, but nothing major. While 1DPC, the memory is 2R (but still registers at 4800), training on every boot. The PMBW graph is probably the most useful behavior chart: https://github.com/AUGMXNT/speed-benchmarking/blob/main/epyc-mbw-testing/nps1-vs-nps4.png https://github.com/AUGMXNT/speed-benchmarking/blob/main/epyc... Since I'm not so concerned with CPU inference, I feel like the debugging/testing I've done is... the amount I'm going to do, which is enough to at least characterize, if not fix the performance. I might write up a more step-by-step guide at some point to help others but for now the testing scripts are there - I think most people who are looking at theoretical MBW should probably do their own real-world testing as it seems to vary a lot more than GPU bandwidth.
jbritton 2y ago
Does it make any sense to have specialized models, which could possibly be a lot smaller. Say a model that just translates between English and Spanish, or maybe a model that just understands unix utilities and bash. I don’t know if limiting the training content affects the ultimate output quality or model size.
- walterbell 2y ago
  Some enterprises have trained small specialized models based on proprietary data. https://www.maginative.com/article/nvidia-leverages-ai-to-assist-with-semiconductor-design/ https://www.maginative.com/article/nvidia-leverages-ai-to-as... > NVIDIA researchers customized LLaMA by training it on 24 billion tokens derived from internal documents, code, and other textual data related to chip design. This advanced “pretraining” tuned the model to understand the nuances of hardware engineering. The team then “fine-tuned” ChipNeMo on over 1,000 real-world examples of potential assistance applications collected from NVIDIA’s designers. 2023 paper, https://research.nvidia.com/publication/2023-10_chipnemo-domain-adapted-llms-chip-design https://research.nvidia.com/publication/2023-10_chipnemo-dom... > Our results show that these domain adaptation techniques enable significant LLM performance improvements over general-purpose base models across the three evaluated applications, enabling up to 5x model size reduction with similar or better performance on a range of design tasks. 2024 paper, https://developer.nvidia.com/blog/streamlining-data-processing-for-domain-adaptive-pretraining-with-nvidia-nemo-curator/ https://developer.nvidia.com/blog/streamlining-data-processi... > Domain-adaptive pretraining (DAPT) of large language models (LLMs) is an important step towards building domain-specific models. These models demonstrate greater capabilities in domain-specific tasks compared to their off-the-shelf open or commercial counterparts.
zelcon 2y ago
Or just wait for the NVIDIA Digits PC later this year which will cost the ~same amount and can fit on your desk
- dzogchen 2y ago
  That one can handle up to 200B parameters according to NVIDIA.
  - zelcon 2y ago
    That's a shame. I suppose you'll need 4 of them with RDMA to run a 671B, but somehow that seems better to me than trying to run it on DDR4 RAM like the OP is saying. I have a system with 230G of DDR4 RAM, and running even small models on it is atrociously slow.
spaceport 2y ago
Hi HN, Garage youtuber here. Wanted to add in some stats on the wattages/ram. Idle wattage: 60w (well below what I expected, this is w/o GPUs plugged in) Loaded wattage: 260w RAM Speed I am running currently: 2400 (V likely 3200 has a decent perf impact)
ant6n 2y ago
Ha, I'd recently asked about this here as well, just using some high memory AMD setup to infer. Another thing I wonder is whether using a bunch of Geforce 4060 TI with 16GB could be useful - they cost only around 500 EUR. If VRAM is the bottleneck, perhaps a couple of these could really help with inference (unless they become GPU bound, like too slow).
rahimnathwani 2y ago
Wow! Only $2k with no quantization. hit between 4.25 to 3.5 TPS (tokens per second) on the Q4 671b full model
- kristianp 2y ago
  I think it is quantised, they actually said no distillation.
  - rahimnathwani 2y ago
    I think you're right. The instructions say ollama pull deepseek-r1:671b This will pull down 400GB: https://ollama.com/library/deepseek-r1:671b https://ollama.com/library/deepseek-r1:671b But the Huggingface repo has 163 files of ~4.3GB each, so around 700GB: https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main
ptsneves 2y ago
Went through the steps and ran it on a similar r6a.16xlarge and the model seems to only load after the first prompt. After that it takes maybe more than half an hour trying to load the model and still no answer. The context size in the post is also not validated in my experiment with the above. With 512GB of ram I cannot use more than 4k context size without the model outright refusing to load. I am new to model setups so I might have missed something.
bofadeez 2y ago
so this $2k investment can substitute for $0.69 in API calls per day
daft_pink 2y ago
Is it possible to run the large Llama 3 models on this as well? Just curious if this is a one trick pony or not.
natas 2y ago
okay, but does it run crysis?
PopSmoke 2y ago
is there a way for deepseek to use gpu vram from 10 3070s? i have a lan setup of 10 pcs

463 points by walterbell 2y ago

wanse_yung 2y ago

[flagged]

leobg 2y ago
This is a verbatim quote from Jason Calacanis.
deleted 2y ago
[deleted]

teruakohatu 2y ago

If you are going to go to that effort, adding a second NMVE drive, and doing RAID 0 across them, will improve the speed of getting the model into RAM.

merb 2y ago
than you will way more memory, since you can only do software raid a nvme drive.
- davidmurdoch 2y ago
  About how much memory overhead would that require?
- mgerdts 2y ago
  I’ve found that striping across two drives like the 980 Pro described here or WD SN850 Black drives easily gets direct IO read speeds over 12 GB/s on threadripper pro systems. This assumes a stripe size somewhere around 1 -2 MiB. This means that most reads will not need to be split and high queue depth sequential reads and random reads keep both drives busy. With careful alignment of IOs, performance approaches 2x of one drive’s performance. IO takes CPU cycles but I’ve not seen evidence that striping impacts that. Memory overhead is minimal, as the stripe to read from is done via simple math from a tiny data structure.
  - merb 2y ago
    well this really depends on what you use mdraid/zfs and as you said alignment. but only for reads. if you just use dumb mdraid (without too much optimizations configured) (raid 10) + xfs and read/write on top of it you will end up with quite a high memory + cpu usage. but i/o still will be insanly fast. I've written this more that it works, but its not a put another drive in and done solution. but if you just want to dumb a second drive into it and use mdraid/zfs you will have an overhead. of course if somebody tunes it and builds the application around it you can trim down the overhead significantly.

H8crilA 2y ago

Do we have any estimate on the size of OpenAI top of the line models? Would they also fit in ~512GB of (V)RAM? Also, this whole self hosting of LLMs is a bit like cloud. Yes, you can do it, but it's a lot easier to pay for API access. And not just for small users. Personally I don't even bother self hosting transcription models which are so small that they can run on nearly any hardware.

exe34 2y ago
it would make sense if you don't want somebody else to have access to all your code and customer data.
yapyap 2y ago
Is the size of OpenAI‘s top of the line models even relevant? Last I checked they weren’t open source in the slightest.
GreenWatermelon 2y ago
It's nice because a company can optionaly provide a SOTA reasoning model for their clients, without having to go through a middleman e.g. HR co. Can provide an LLM for their HRMS system for a small 2000$ investment. Not 2000/month, just a one time 2000 investment.
- NeutralCrane 2y ago
  No one will be doing anything practical with a local version of Deepseek on a $2000 server. The token throughout of this thing is like, 1 token every 4 seconds. It would take nearly a full minute just to produce a standard “Roses are red, violets are blue” poem. There’s absolutely no practical usage that you can use that for. It’s cool that you can do it, and it’s a step in the right direction, but self-hosting these wont be a viable alternative to using providers like OpenAI for business applications for a while.
  - bbzylstra 2y ago
    The OP said they were getting 3-4 TPS on their $2000 rig.

buyucu 2y ago

related reddit thread: https://old.reddit.com/r/LocalLLaMA/comments/1idseqb/deepseek_r1_671b_over_2_toksec_without_gpu_on/ https://old.reddit.com/r/LocalLLaMA/comments/1idseqb/deepsee...

christophilus 2y ago

Aside: it’s pretty amazing what $2K will buy. It’s been a minute since I built my desktop, and this has given me the itch to upgrade. Any suggestions on building a low-power desktop that still yields decent performance?

Havoc 2y ago
>Any suggestions on building a low-power desktop that still yields decent performance? You don't for now. The bottleneck is mem throughput. That's why people using CPU for LLM are running xeon-ish/epyc setups...lots of mem channels. The APU class gear along the lines of Halo Strix is probably the path closest to lower power but it's not going to do 500gb of ram and still doesn't have enough throughput for big models
baobun 2y ago
Hard to know what ranges you have in mind with "decent performance" and "low-power". I think your best bet might be a Ryzen U-series mini PC. Or perhaps an APU barebone. The ATX platform is not ideal from a power-efficiency perspective (whether inherently or from laziness or conspiracy from mobo and PSU makers, I do not know). If you want the flexibility or scale, you pay the price of course but first make sure it's what you want. I wouldn't look at discrete graphics unless you have specific needs (really high-end gaming, workstation, LLMs, etc) - the integrated graphics of last few years can both drive your 4k monitors and play recent games at 1080p smoothly, albeit perhaps not simultaneously ;) Lenovo Tiny mq has some really impressive flavors (ECC support at the cost of CPU vendor-lock on PRO models) and there's the whole roster of Chinese competitors and up-and-comers if you're feeling adventerous. Believe me you can still get creative if you want to scratch the builder itch - thermals is generally what keeps these systems from really roaring (:
spaceport 2y ago
Not to be that yt'r that shills my videos all over, but you did ask for a low powered desktop build and this $350 one I put together is still my favorite. The 3060 12GB with llama 3.2 vision 11b is a very fun box that is low idle power (intel rules) to leave on 24/7 and have it run some additional services like HA. https://youtu.be/iflTQFn0jx4 https://youtu.be/iflTQFn0jx4

1ba9115454 2y ago

I can't imagine this setup will get more than 1 token per second. I would love to see Deepseek running on premise with a decent TPS.

thomquaid 2y ago
It says 4.25 TPS in the first para.
- ricardobeat 2y ago
  Honest mistake. Some people think HN is just a series of short tweets and haven’t realized they are links yet!
  - thomquaid 2y ago
    4.25 is enough tps for a lot of use cases.
  - 4ndrewl 2y ago
    It's the modern way. Why read when you can just imagine facts straight out of your own brain.
    - plagiarist 2y ago
      I agree but also found your comment funny in the context of LLMs. People love getting facts straight out of their models.
- weatherlight 2y ago
  That's still pretty slow, considering there's that "thinking" phase.
  - thomquaid 2y ago
    True, but 4.25 is the number we all want to know.

sylware 2y ago

Well, I read this, now I am sure: as of today, deepseek handling of LLMs is the less wrong, and by far.

geertj 2y ago

This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM. This [1] X thread runs the 671B model in the original Q8 at 6-8 TPS for $6K using a dual socket Epyc server motherboard using 768GB of RAM. I think this could be made cheaper by getting slower RAM but since this is RAM bandwidth limited that would likely reduce TPS. I’d be curious if this would just be a linear slowdown proportional to the RAM MHz or whether CAS latency plays into it as well. [1] https://x.com/carrigmat/status/1884244369907278106?s=46&t=5DsSie6D9vxgUafSFmo6EQ https://x.com/carrigmat/status/1884244369907278106?s=46&t=5D...

nielsole 2y ago
I've been running the unsloth 200GB dynamic quantisation with 8k context on my 64GB Ryzen 7 5800G. CPU and iGPU utilization were super low, because it basically has to read the entire model from disk. (Looks like it needs ~40GB of actual memory that it cannot easily mmap from disk) With a Samsung 970 Evo Plus that gave me 2.5GB/s read speed. That came out at 0.15 tps Not bad for completely underspecced hardware. Given the model has only so few active parameters per token (~40B), it is likely that just being able to hold it in memory absolve the largest bottleneck. I guess with a single consumer PCIe4.0x16 graphics card you could get at most 1tps just because of the PCIe transfer speed? Maybe CPU processing can be faster simply because DDR transfer is faster than transfer to the graphics card.
- deleted 2y ago
  [deleted]
- TeMPOraL 2y ago
  To add another datapoint, I've been running the 131GB (140GB on disk) 1.58 bit dynamic quant from Unsloth with 4k context on my 32GB Ryzen 7 2700X (8 cores, 3.70 GHz), and achieved exactly the same speed - around 0.15 tps on average, sometimes dropping to 0.11, tps occasionally going up to 0.16 tps. Roughly 1/2 of your specs, roughly 1/2 smaller quant, same tps. I've had to disable the overload safeties in LM Studio and tweak with some loader parameters to get the model to run mostly from disk (NVMe SSD), but once it did, it also used very little CPU! I tried offloading to GPU, but my RTX 4070 Ti (12GB VRAM) can take at most 4 layers, and it turned out to make no difference in tps. My RAM is DDR4, maybe switching to DDR5 would improve things? Testing that would require replacing everything but the GPU, though, as my motherboard is too old :/.

brunohaid 2y ago

Still surprised that the $3000 NVIDIA Digits doesn’t come up more often in that and also the gung-ho market cap discussion. I was an AI sceptic until 6 months ago, but that’s probably going to be my dev setup from spring onwards - running DeepSeek on it locally, with a nice RAG to pull in local documentation and datasheets, plus a curl plugin. https://www.nvidia.com/en-us/project-digits/ https://www.nvidia.com/en-us/project-digits/

fake-name 2y ago
It'll probably be more relevant when you can actually buy the things. It's just vaporware until then.
- brunohaid 2y ago
  Call me naive, but I somehow trust them to deliver in time/specs? It’s also a more general comment around „AI desktop appliance“ vs homebuilts. I’d rather give NVIDIA/AMD $3k for a well adjusted local box than tinkering too much or feeding the next tech moloch, and have a hunch I’m not the only one feeling that way. Once it’s possible of course.
  - fake-name 2y ago
    Oh, if it's anything close to what they claim, I'll probably buy one as well, but I certainly do not expect them to deliver on time.
    - Cane_P 2y ago
      DIGITS isn't that impressive... It is a RTX 5070 Ti laptop GPU (992 TOPS, clocked less than 1% higher, to reach 1000 TOPS/1 PFLOP. As a reference RTX 5090 desktop have 3352 TOPS, more than 3x...), with 128 GB of unified memory. Just because Jensen calls it a super computer and gives it a DGX-1 design, doesn't make it one. In the Cleo Abram interview [1], Jensen said that DIGITS is 6 times more powerful than the first DGX-1. According to this PDF [2], DGX-1 had 170 TFLOPS of FP16 (half precision). 170x6=1020 TFLOP (~1 PFLOP). Yes DIGITS is suppose to have 1 PFLOP, but according to the presentation, it should be in FP4... He also said that it will draw 10k times less power. But DGX-1 had a TDP of 3.5kW [3] and I highly doubt DIGITS will draw 3500/10000=0.35W... the GPU alone will have a peak TDP that is more like 200 times higher than that. I mean, we all know that NVIDIA does fudge the numbers in charts. Like comparing FP8 from last generation, to FP4 on this. But this is extreme. Having said that. Do I believe that they can deliver a laptop (in another form factor) and it will perform 1 PFLOP of FP4. Of course! Like I said, it is nothing special. Both Apple and AMD have unified memory in relatively cheap systems. 1. https://youtu.be/7ARBJQn6QkM https://youtu.be/7ARBJQn6QkM 2. http://images.nvidia.com/content/technologies/deep-learning/pdf/Datasheet-DGX1.pdf http://images.nvidia.com/content/technologies/deep-learning/... 3. https://images.nvidia.com/content/pdf/dgx1-v100-system-architecture-whitepaper.pdf https://images.nvidia.com/content/pdf/dgx1-v100-system-archi...

cma 2y ago

He's running quantized Q4 671b. However, MoE doesn't need cluster networking so you could probably run the full thing on two of them unquantized. Maybe the router could be all resident in GPU RAM instead of in contrast offloading a larger percentage of everything there, or is that already how it is set up in his gpu offload config?

sAggymEllenZ 2y ago

[dead]

notsylver 2y ago

I think it would be more interesting doing this with smaller models (33b-70b) and see if you could get 5-10 tokens/sec on a budget. I've desperately wanted something locally thats around the same level of 4o, but I'm not in a hurry to spend $3k on an overpriced GPU or $2k on this

firtoz 2y ago
Would it be something like this? > OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi https://x.com/geerlingguy/status/1884994878477623485 https://x.com/geerlingguy/status/1884994878477623485 I haven't tried it myself or haven't verified the creds, but seems exciting at least
- gliptic 2y ago
  That's 1.2 t/s for the 14B Qwen finetune, not the real R1. Unless you go with the GPU with the extra cost, but hardly anyone but Jeff Geerling is going to run a dedicated GPU on a Pi.
- etra0 2y ago
  it's using a Raspberry Pi with a.... USD$1k gpu, which kinda defeat the purpose of using the RPI in the first place imo. or well, I guess you save a bit on power usage.
  - firtoz 2y ago
    Oh, I was naive to think that the Pi was capable of some kind of magic (sweaty smile emoji goes here)
  - unethical_ban 2y ago
    I suppose it makes sense, for extremely GPU centric applications, that the pi be used essentially as a controller for the 3090.
gliptic 2y ago
Your best bet for 33B is already having a computer and buying a used RTX 3090 for <$1k. I don't think there's currently any cheap options for 70B that would give you >5. High memory bandwidth is just too expensive. Strix Halo might give you >5 once it comes out, but will probably be significantly more than $1k for 64 GB RAM.

isoprophlex 2y ago

Online, R1 costs what, $2/MTok? This rig does >4 tok/s, which is ~15-20 ktok/hr, or $0.04/hr when purchased through a provider. You're probably spending $0.20/hr on power (1 kW) alone. Cool achievement, but to me it doesn't make a lot of sense (besides privacy...)

rightbyte 2y ago
> Cool achievement, but to me it doesn't make a lot of sense (besides privacy...) I would argue that is enough and that this is awesome. It was a long time ago I wanted to do a tech hack like this much.
- isoprophlex 2y ago
  Well thinking about it a bit more, it would be so cool if you could A) somehow continuously interact with the running model, ambient-computing style. Say have the thing observe you as you work, letting it store memories. B) allowing it to process those memories when it chooses to/whenever it's not getting any external input/when it is "sleeping" and C) (this is probably very difficult) have it change it's own weights somehow due to whatever it does in A+B. THAT, in a privacy friendly self-hosted package, i'd pay serious money for
  - Willingham 2y ago
    I imagine it could solve crimes if it watched millions of hours of security footage…scary thought. Possibly it could arrest us before we even commit a crime through prediction like that black mirror episode.
    - falcor84 2y ago
      Oh, you're thinking of "Hated in the Nation"? More relevant would possibly be "Minority Report" (set in 2054) and Hulu's "Class of '09", in which the FBI starts deploying a crime prediction AI in their version of 2025. Quite scary. As the meme has it, it seems that we're getting ready to create the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus.

alecco 2y ago

Affiliate link spam.

codetrotter 2y ago
No. Affiliate link spam is when someone fills a page with content they stole, or it’s a bunch of nonsense that is stuffed to the brim with keywords, and they combine that with affiliate links. Someone getting a dollar or two in return for you following an affiliate link after you read something they put real time and effort into to make it valuable info for others is not “affiliate link spam”.
- jjallen 2y ago
  I’m fine with useful content linking to affiliate links too. I am still confused at the ram specs required and those they linked to being off by a factor of 2 though. IF the setup is not realistic or accurate then that wouldn’t be cool.
  - alecco 2y ago
    Few people would spend $6k to run a model locally on CPU. But lower it to $2k and you might get some sweet affiliate link commissions. And make it fuzzy so they don't get it is running so quantized it probably is useless.
- alecco 2y ago
  This is a lame me-too page to make money with affiliate links. The actual specs were in the linked original tweet. So, IMHO, it is affiliate link spam.

almostgotcaught 2y ago

Those EPYCs he's advertising as $700 are either engineering samples or used. New he's off by 2:1.

zajio1am 2y ago

> 512GB 2400 ECC RAM $400 Is this really that cheap? Looking at several local (CZ) eshops, i cannot find 32 GB DDR4 ECC RDIMM cheaper than $75, which will be $1200 for 512 GB.

formerly_proven 2y ago
Used server hardware is much more expensive in the EU generally, because the market is much smaller (fewer data centers to begin with, longer cycles to reduce costs and EU WEEE mandatory scrapping instead of reuse).
- deleted 2y ago
  [deleted]

jjallen 2y ago

He links to a ram kit that is 8x 32Gb but says it should have 512 Gb ram. What gives? Also with 8 ram slots you would obviously need more than 256. Is the setup disingenuous to get people excited about the post or what is going on here?

dave7 2y ago
The board recommended has 16 ram slots.
- jjallen 2y ago
  The ram kit is still 8x32gb so the price is lower than it actually would be.

deleted 2y ago

[deleted]

ilaksh 2y ago

Has anyone run any benchmarks on the quantized (non-distilled) versions?

pama 2y ago

What is the fastest documented way so far to serve the full R1 or V3 models (Q8, not Q4) if the main purpose is inference with many parallel queries and maximizing the total tokens per sec? Did anyone document and benchmark efficient distributed service setups?

manmal 2y ago
The top comment in this thread mentions a 6k setup, which likely could be used with vLLM with more tinkering. AFAIK vLLM‘s batched inference is great.
snovv_crash 2y ago
You need enough VRAM to hold the whole thing plus context. So probably a bunch of H100s, or MI300s.

jeswin 2y ago

Maybe Intel or AMD should bring back a Larrabee style CPU which can use say 48 sockets of DDR/CAMM2 sticks.

easygenes 2y ago

This is neat, but what I really want to see is someone running it on 8x 3090/4090/5090 and what is the most practical configuration for that.

deoxykev 2y ago
8x 3090 will net you around 10-12tok/s
- bick_nyers 2y ago
  It would not be that slow as it is an MoE model with 37b activated parameters. Still, 8x3090 gives you ~2.25 bits per weight, which is not a healthy quantization. Doing bifurcation to get up to 16x3090 would be necessary for lightning fast inference with 4bit quants. At that point though it becomes very hard to build a system due to PCIE lanes, signal integrity, the volume of space you require, the heat generated, and the power requirements. This is the advantage of moving up to Quadro cards, half the power for 2-4x the VRAM (top end Blackwell Quadro expected to be 96GB).
  - deoxykev 2y ago
    Yeah, there is a clear bottleneck somewhere in llama.cpp. Even high end hardware is struggling to get good numbers. The theoretical limit should be higher, but it's not yet. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/11474#issuecomment-2629590835 https://github.com/ggerganov/llama.cpp/issues/11474#issuecom...
- rdlw 2y ago
  Is it possible that eight graphics cards is the most practical configuration? How do you even set that up? I guess server mobos have crazy numbers of PCIe slots?
MaxikCZ 2y ago
I have been searching for a single example of someone running it like this (or 8x P40 and alike), and found nothing..

crispyambulance 2y ago

Kind of embarrassed to ask, I use AI a lot, I haven't really understood how the nuts and bolts work (other than at a 5th-grader 30000ft level)... So, when I use a "full" AI like chatGPT4o, I ask it questions and it has a firm grip on a vast amount of knowledge, like, whole-internet/search-engine scope knowledge. If I run an AI "locally", on even a muscular server, it obviously does NOT have vast amounts of stored information about everything. So what use is it to run locally? Can I just talk to it as though it were a very smart person who, tragically, knows nothing? I mean, I suppose I could point it to a NAS box full of pdf's and ask questions about that narrow range of knowledge, or maybe get one of those downloaded wikipedia stores. Is that what folks are doing? It seems like you would really need a lot content for the AI to even be remotely useable like the online versions.

londons_explore 2y ago
Most of an AI's knowledge is inside the weights, so when you run it locally, it has all that knowledge inside! Some AI services allow the use of 'tools', and some of those tools can search the web, calculate numbers, reserve restaurants, etc. However, you'll typically see it doing that in the UI. Local models can do that too, but it's typically a bit more setup.
throwaway3b03 2y ago
I've also been away from the tech (and AI scene) for a few years now. And I mostly stayed away from LLMs. But I'm certain that all the content is baked into the model, during training. When you query the model locally (since I suppose you don't train it yourself), you get all that knowledge that's baked into the model weights. So I would assume the locally queried output to be comparable with the output you get from an online service (they probably use slightly better models, I don't think they release their latest ones to the public).
MuffinFlavored 2y ago
I ask this all the time. locally running an LLM seems super hobbyist to me. like tweaking terminal font sizes on fringe BSD distros kind of thing

assimpleaspossi 2y ago

Are there any security concerns over DeepSeek as there are over TikTok?

Kuinox 2y ago
It's a local model, what security concern would you have ?
- jappgar 2y ago
  I think this is unlikely but a local model could generate malicious code. You would have to run it manually though.
- dinosaurdynasty 2y ago
  They also have an app that connects to their datacenter with R1. Also barely anyone can actually run the real R1 locally.
  - Kuinox 2y ago
    We are speaking about a 2k$ server here.
Svoka 2y ago
Local model - no. Using deepseek.com - absolutely. Do not put anything private there.

huijzer 2y ago

What is a bit weird about AI currently is that you basically always want to run the best model, but the price of the hardware is a bit ridiculous. In the 1990s, it was possible to run Linux on scrappy hardware. You could also always run other “building blocks” like Python, Docker, or C++ easily. But the newest AI models require an order of magnitude more RAM than my system or the systems I typically rent have. So I’m curious to people here, has this in the history of software happened before? Maybe computer games are a good example. There people would also have to upgrade their system to run the latest games.

svilen_dobrev 2y ago
well, if there was e.g. a model trained for coding - i.e. specialization as such, having models trained mostly for this or that - instead of everything incl. Shakespeare, the kitchen sink and the cockroaches biology under it, that would make those runable on much low level hardware. But there is only one, The-Big-Deal.. in many incarnations.
spamizbad 2y ago
Like AI, there were exciting classes of applications in the 70s, 80s and 90s that mandated pricier hardware. Anything 3D related, running multi-user systems, higher end CAD/EDA tooling, and running any server that actually got put under “real” load (more than 20 users). If anything this isn’t so bad: $4K in 2025 dollars is an affordable desktop computer from the 90s.
- lukeschlather 2y ago
  The thing is I'm not that interested in running something that will run on a $4K rig. I'm a little frustrated by articles like this, because they claim to be running "R1" but it's a quantized version and/or it has a small context window... it's not meaningfully R1. I think to actually run R1 properly you need more like $250k. But it's hard to tell because most of the stuff posted is people trying to do duct tape and bailing wire solutions.
  - adenta 2y ago

jonotime 2y ago

I'm also kind of new to this and coming from coding with ChatGPT. Isnt the time to first token important? He is sitting there for minutes waiting for a response. Shouldnt that be a concern?

HarHarVeryFunny 2y ago
I'd rather wait to get a good response, than get a quick response that is much less useful, and it's the nature of these "reasoning" models that they reason before responding. Yesterday I was comparing DeepSeek-R1 (NVidia hosted version) with both Sonnet 3.5 (regarded by most as most capable coder) and the new Gemini 2.0 flash, and the wait was worth it. I was trying to get all three to create a web page with a horizontally scrolling timeline with associated clickable photos... Gemini got to about 90% success after half a dozen prompts, after which it became a frustrating game of whack-a-mole trying to get it to fix the remaining 10% without introducing new bugs - I gave up after ~30min. Sonnet 3.5 looked promising at first, generating based on a sketch I gave it, but also only got to 90%, then hit daily usage limit after a few attempts to complete the task. DeepSeek-R took a while to generate it, but nailed it on first attempt.
- jonotime 2y ago
  Interesting. So in my use, I rarely see gpt get it right on the first pass but thats mostly due to interpretation of the question. I'm ruling out the times when it hallucinates calls to functions that dont exist. Lets say I ask for some function that calculates some matrix math in python. It will spit out something but I dont like what it did. So I will say, now dont us any calls to that library you pulled in, and also allow for these types of inputs. Add exception handling... So response time is important since its a conversation, no matter how correct the response is. When you say deep seek "nailed it on the first attempt" do you mean it was without bugs? Or do you mean it worked how you imagined? Or what exactly?
  - HarHarVeryFunny 2y ago
    DeepSeek-R generated a working web page on first attempt, based on a single brief prompt I gave it. With Sonnet 3.5, given the same brief prompt I gave DeepSeek-R, it took a half dozen feedback steps to get to 90%. Trying a hand drawn sketch input to Sonnet instead was quicker - impressive first attempt, but iterative attempts to fix it failed before I hit the usage limit. Gemini was the slowest to work with, and took a lot of feedback to get to the "almost there" stage, after which it floundered. The AI companies seem to want to move in the direction of autonomous agents (with reasoning) that you hand a task off to that they'll work on while you do something else. I guess that'd be useful if they are close to human level and can make meaningful progress without feedback, and I suppose today's slow-responding reasoning models can be seen as a step in that direction. I think I'd personally prefer something fast enough responding to use as a capable "pair programmer", rather than an autonomous agent trying to be an independent team member (at least until the AI gets MUCH better), but in either case being able to do what's being asked is what matters. If the fast/interactive AI only gets me 90% complete (then wastes my time floundering until I figure out it's just not capable of the task), then the slower but more capable model seems preferable as long as it's significantly better.

waynenilsen 2y ago

Someone please productize this

loxias 2y ago
"Okay" Can you describe for me what the product does? It's not that hard to make a turnkey "just add power" appliance that does nothing but spit out tokens. Some sort of "ollama appliance", which just sits on your network and provides LLM functionality for your home lab? But beyond that, what would your mythical dream product do?
- waynenilsen 2y ago
  i'm talking about something you plug in and get on your network then you can chat with it with a typical chat interface and it should expose an openai compatible api automatic updates etc etc the AI home appliance great for privacy
gatienboquet 2y ago
https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox

jonotime 2y ago

Any idea what the power draw is? Resting power, vs resting with the model loaded in memory, vs full power computation. In case you may want to also run this as your desktop for basic web browsing etc when not LLMing.

waltercool 2y ago

Interesting and surprising to see, but 3-4 t/s is not practical overall. Anything lower than 10 t/s is going to give lot of wait time considering the reasoning wait time

jauntywundrkind 2y ago

Is there any hope that desktops get 64GB dimms? 48GB dimms have been a nice boost but are we gonna get more anytime soon? I'd love so much if quad-channel Strix Halo could get up to 256gb of memory. 192GB (4x48) won't be too bad, and at 8533MT/s, should provide competitive-ish throughput to these massive Epyc systems. Of course the $6k 24 channel 3200MHz is 4.4x more throughput but it does have a field of dimms to get there, and high power consumption.

la64710 2y ago

Basically three components : ollama , openwebui and deepseek. Nice!

sroussey 2y ago

If you really want to cheap—run it in your browser via webgpu.

sroussey 2y ago
I got downvoted, but you really can run it in the browser. ONNX version: https://huggingface.co/onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNX https://huggingface.co/onnx-community/DeepSeek-R1-Distill-Qw...
zubiaur 2y ago
Not comparable. That is a quantized distill.
- sroussey 2y ago
  Oh yes, for sure. Don’t mean otherwise.

lhl 2y ago

Last fall I built a new workstation with an EPYC 9274F (24C Zen4 4.1-4.3GHz, $2400), 384GB 12 x 32GB DDR5-4800 RDIMM ($1600), and a Gigabyte MZ33-AR0 motherboard. I'm slowly populating with GPUs (including using C-Payne MCIO gen5 adapters), not focused on memory, but I did spend some time recently poking at it. I spent extra on the 9274F because of some published benchmarks [1] that showed that the 9274F had STREAM TRIAD results of 395 GB/s (on 460.8 GB/s of theoretical peak memory bandwidth), however sadly, my results have been nowhere near that. I did testing with LIKWID, Sysbench, and llama-bench, and even w/ an updated BIOS and NUMA tweaks, I was getting <1/2 the Fujitsu benchmark numbers: Results for results-f31-l3-srat: { "likwid_copy": 172.293857421875, "likwid_stream": 173.132177734375, "likwid_triad": 172.4758203125, "sysbench_memory_read_gib": 191.199125, "llama_llama-2-7b.Q4_0": { "tokens_per_second": 38.361456, "model_size_gb": 3.5623703002929688, "mbw": 136.6577115303955 } } For those interested in all the system details/running their own tests (also MLC and PMBW results among others): https://github.com/AUGMXNT/speed-benchmarking/tree/main/epyc-mbw-testing https://github.com/AUGMXNT/speed-benchmarking/tree/main/epyc... [1] https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-performance-report-primergy-rx1440-m2-ww-en.pdf https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-perfor...

menaerus 2y ago
Assuming that you populated the channels correctly, which I believe you did, I can only think that this issue could be related to the motherboard itself or RAM. I think you could start by measuring the single-core RAM bandwidth and latency. Since the CPU is clocked quite high, figures you should be getting are I guess around ~100ns, but probably less than that, and 40-ish GB/s of BW. If those figures do not match then it could be either a motherboard (HW) or BIOS (SW) issue or RAM stick issue. If those figures closely match then it's not a RAM issue but a motherboard (BIOS or HW) and you could continue debugging by adding more and more cores to the experiment to understand at which point you hit the saturation point for the bandwidth. It could be a power issue with the mobo.
- lhl 2y ago
  Yeah, that channels are populated correctly. As you can see from the mlc-results.txt, the latency looks fine: mlc --idle_latency Intel(R) Memory Latency Checker - v3.11b Command line parameters: --idle_latency Using buffer size of 1800.000MiB Each iteration took 424.8 base frequency clocks ( 104.9 ns) As does the per-channel --bandwidth_matrix results: Numa node Numa node 0 1 2 3 4 5 6 7 0 45999.8 46036.3 50490.7 50529.7 50421.0 50427.6 50433.5 52118.2 1 46099.1 46129.9 52768.3 52122.3 52086.5 52767.6 52122.6 52093.4 2 46006.3 46095.3 52117.0 52097.2 50385.2 52088.5 50396.1 52077.4 3 46092.6 46091.5 52153.6 52123.4 52140.3 52134.8 52078.8 52076.1 4 45718.9 46053.1 52087.3 52124.0 52144.8 50544.5 50492.7 52125.1 5 46093.7 46107.4 52082.0 52091.2 52147.5 52759.1 52163.7 52179.9 6 45915.9 45988.2 50412.8 50411.3 50490.8 50473.9 52136.1 52084.9 7 46134.4 46017.2 52088.9 52114.1 52125.0 52152.9 52056.6 52115.1 I've tried various NUMA configurations (from 1 domain to a per-CCD config) and it doesn't seem to make much difference. Updating from the board-delivered F14 to the latest 9004 F31 BIOS (the F33 releases bricked the board and required using a BIOS flasher for manual recover) gave marginal (5-10%) improvement, but nothing major. While 1DPC, the memory is 2R (but still registers at 4800), training on every boot. The PMBW graph is probably the most useful behavior chart: https://github.com/AUGMXNT/speed-benchmarking/blob/main/epyc-mbw-testing/nps1-vs-nps4.png https://github.com/AUGMXNT/speed-benchmarking/blob/main/epyc... Since I'm not so concerned with CPU inference, I feel like the debugging/testing I've done is... the amount I'm going to do, which is enough to at least characterize, if not fix the performance. I might write up a more step-by-step guide at some point to help others but for now the testing scripts are there - I think most people who are looking at theoretical MBW should probably do their own real-world testing as it seems to vary a lot more than GPU bandwidth.

jbritton 2y ago

Does it make any sense to have specialized models, which could possibly be a lot smaller. Say a model that just translates between English and Spanish, or maybe a model that just understands unix utilities and bash. I don’t know if limiting the training content affects the ultimate output quality or model size.

walterbell 2y ago
Some enterprises have trained small specialized models based on proprietary data. https://www.maginative.com/article/nvidia-leverages-ai-to-assist-with-semiconductor-design/ https://www.maginative.com/article/nvidia-leverages-ai-to-as... > NVIDIA researchers customized LLaMA by training it on 24 billion tokens derived from internal documents, code, and other textual data related to chip design. This advanced “pretraining” tuned the model to understand the nuances of hardware engineering. The team then “fine-tuned” ChipNeMo on over 1,000 real-world examples of potential assistance applications collected from NVIDIA’s designers. 2023 paper, https://research.nvidia.com/publication/2023-10_chipnemo-domain-adapted-llms-chip-design https://research.nvidia.com/publication/2023-10_chipnemo-dom... > Our results show that these domain adaptation techniques enable significant LLM performance improvements over general-purpose base models across the three evaluated applications, enabling up to 5x model size reduction with similar or better performance on a range of design tasks. 2024 paper, https://developer.nvidia.com/blog/streamlining-data-processing-for-domain-adaptive-pretraining-with-nvidia-nemo-curator/ https://developer.nvidia.com/blog/streamlining-data-processi... > Domain-adaptive pretraining (DAPT) of large language models (LLMs) is an important step towards building domain-specific models. These models demonstrate greater capabilities in domain-specific tasks compared to their off-the-shelf open or commercial counterparts.

zelcon 2y ago

Or just wait for the NVIDIA Digits PC later this year which will cost the ~same amount and can fit on your desk

dzogchen 2y ago
That one can handle up to 200B parameters according to NVIDIA.
- zelcon 2y ago
  That's a shame. I suppose you'll need 4 of them with RDMA to run a 671B, but somehow that seems better to me than trying to run it on DDR4 RAM like the OP is saying. I have a system with 230G of DDR4 RAM, and running even small models on it is atrociously slow.

spaceport 2y ago

Hi HN, Garage youtuber here. Wanted to add in some stats on the wattages/ram. Idle wattage: 60w (well below what I expected, this is w/o GPUs plugged in) Loaded wattage: 260w RAM Speed I am running currently: 2400 (V likely 3200 has a decent perf impact)

ant6n 2y ago

Ha, I'd recently asked about this here as well, just using some high memory AMD setup to infer. Another thing I wonder is whether using a bunch of Geforce 4060 TI with 16GB could be useful - they cost only around 500 EUR. If VRAM is the bottleneck, perhaps a couple of these could really help with inference (unless they become GPU bound, like too slow).

rahimnathwani 2y ago

Wow! Only $2k with no quantization. hit between 4.25 to 3.5 TPS (tokens per second) on the Q4 671b full model

kristianp 2y ago
I think it is quantised, they actually said no distillation.
- rahimnathwani 2y ago
  I think you're right. The instructions say ollama pull deepseek-r1:671b This will pull down 400GB: https://ollama.com/library/deepseek-r1:671b https://ollama.com/library/deepseek-r1:671b But the Huggingface repo has 163 files of ~4.3GB each, so around 700GB: https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main

ptsneves 2y ago

Went through the steps and ran it on a similar r6a.16xlarge and the model seems to only load after the first prompt. After that it takes maybe more than half an hour trying to load the model and still no answer. The context size in the post is also not validated in my experiment with the above. With 512GB of ram I cannot use more than 4k context size without the model outright refusing to load. I am new to model setups so I might have missed something.

bofadeez 2y ago

so this $2k investment can substitute for $0.69 in API calls per day

daft_pink 2y ago

Is it possible to run the large Llama 3 models on this as well? Just curious if this is a one trick pony or not.

natas 2y ago

okay, but does it run crysis?

PopSmoke 2y ago

is there a way for deepseek to use gpu vram from 10 3070s? i have a lan setup of 10 pcs